Been doing alot of this over the years. My best resources/tools thus far:
1. N64 Analyzer - A tool I made myself which is built on top of Mupen64Plus. You enter an address and then the analyzer logs all instructions in one frame, until that address is written. It knows MIPS IV assembly well enough to trace back all instructions involved in calculating the final value. Those instructions are then shown and can be debugged with breakpoints etc...
It’s extremely useful for tracing floating point instructions, which always goes through alot of temporary addresses, making them super tedious to debug with normal tools.
2. Modified Bizhawk - A few years ago I commited some improvements to the BizHawk N64 debugger, making it much faster to use breakpoints. The remaining issue though is that BizHawk is not built to handle mid-frame pausing, making debugging/stepping impossible. Nevertheless, while working on my commit, I did implement and build a custom version of BizHawk with a working debugger, even though it’s a bit hackish and doesn’t support all BizHawk features. Lua scripts don`t work well with it for example.
I still use Nemu sometimes though.
3. Source code online with labels/documentation - The hacking community and others have uploaded text files of the source code, divided into subroutines with labels and documentation of what it does and which parameters it takes. Very incomplete of course but still very useful and interesting.
I will add some links later when I have access to my desktop.
