I am going to start writing some ROMs that will test EVERY GODDAMN ASPECT OF THE NES, no matter how tiny or seemingly insignificant. I will start off with the CPU and then move on to the PPU, APU etc. However, I do need your help in ideas for the actual tests. Here is our chance to improve on NEStress.
So far I am going to implement testing on;
1. All Addressing modes including zero page wrapping and indirect bug
2. All Opcodes including all flags
3. Trapping read/write only registers/memory/ROM
4. CPU RAM/Register mirroring
Work has already started so please post sooner rather than later. I'll make the ROM downloadable from my emulator site soon I hope.
Writing the most basic ROMs will be tricky since they can't depend on much. A subset of the CPU instruction that's easy to implement properly would be good to stick to. The first ones will need a particular order of verification where each allows more instructions to be used in the next, a "bootstrapping" process.
Before the PPU is verified, you could use the APU or even just a memory location to report the result. The APU is very simple to use for reporting result codes and doesn't require any polling as the PPU does. It is possible to write code that doesn't depend on the PPU being implemented, by using a timeout loop while waiting for the VBL status bit. This would allow the tests to be converted to an NSF (using an automated tool) and run on an NSF player without modification. Write the code in a way that separates the reporting from the tests, so it can be improved without having to rewrite tests.
Don't just test for positive behavior, also test for things that should not occur. These must be chosen by predicting things emulators are likely to get wrong, or finding common errors in past emulator development; they can't be generated mechanically from the NES specification.
It's not possible to exhaustively test everything, since that would involve every possible set of program code, mapper, and joypad input. Again, the things to test depend on what is likely to be implemented incorrectly in an emulator. For example, there's little point in checking that an lda at address $8005 works if an lda at $8004 already worked. But there is a point in checking that an lda works at $bfff which crosses to another bank on a cartridge with a mapper. It's probably of little benefit (compared to the cost of writing the test) to check every single instruction for bank-crossing problems, though.
Try to make it possible to easily run the ROMs in an automated fashion, so emulator authors can add an automatic test and run it often. I suppose NEStress-style testing could be run as a movie, but checking the result would require a bit of code to parse the nametable for the result.
You can test the tests (and watch the watchers) by intentionally breaking your emulator in specific ways. It's fun to do this and see the test ROMs knowing exactly what error your introduced. This is important because tests are often fairly intricate and easy to mess up.
A couple of CPU testscome to mind (not sure if NEStress already checks these):
- Extra clock when branch "crosses page" (very poorly described in most documentation). This should cover these cases, of which I don't even remember which count as page crossing:
$4fe: bne $4f0
$4fe: bne $510
$4ff: bne $4f0
$4ff: bne $510
$500: bne $4f0
$500: bne $510
You could make use of the VBL flag as a time reference and run the branch thousands of times making any error in timing detectable as whether the VBL occurred or not.
- The value of bit 4 of the byte pushed on the stack containing the status flags when NMI, IRQ, PHP, and BRK occur.
AFAIK, you cannot test EVERY CPU opcode in a single way. You can, however, build an image output (or math) for each opcode correctly emulated. Even NEStress cannot test the things - I had problems to identify an error in a certain opcode. In fact, it wasn't checking the instruction itself, but the instruction *right before* using the addressing mode (can't remember which one).
Test ROMs are ALWAYS welcome, no doubt... but think about something more logical than psycological... ^_^;;