How would I automate the verification of my emu?

How would I automate the verification of my emu?
by jwdonal on 2010-07-06 (#63846)

Hello all,

My emu is becoming extremely complex. Especially now that I've implemented a bunch of mappers. The code is very modular but there's always the chance that making a change in one module will drastically affect another - and not necessarily in a good way.

Whenever I make a fairly large number of changes to my design (especially if it's a PPU/CPU core change/fix) I very tediously go through and test a subset of all of the games that I have available and ensure that I didn't break one that was previously working. As you can imagine this takes a considerable amount of time....and to be honest, it's starting to wear me out.

Can any of you suggest a better way of verifying my emu? Keep in my that I have the fairly unique case that my emu is implemented in hardware (FPGA) not software. Right now I have the ability to load ROMs over a UART port on the board. I could at a minimum write a script to load the various ROMs one after another...but is there a way to automatically have a script determine that the game actually "worked" without me have to look at (and/or play) each one?

Maybe I shouldn't even worry about actual games right now...maybe just the available CPU/PPU test ROMs (e.g. Blargg's)?

Any suggestions would be very much appreciated.

Thanks!

Jonathon

by ReaperSMS on 2010-07-06 (#63847)

Your best bet is probably to look for expected output over the serial connection, by way of hacking the test roms to output some known value to it, and running them in sequence.

Graphical glitches that are not detectable by the roms are probably best handled with actual eyeball tests, possibly ones driven by a special cased state machine to test, say, the PPU in isolation. Those can make it a lot easier to get the exact cycle timing for certain events verified, and speed tests up by an order of magnitude or so if they involve the bus.

Brute force RAM testing in particular is best done with a state machine that mimics the 6502's bus behavior, but with the address/data stream and pass/fail logic done at full bus speed, rather than the 1/10th or worse you'll get with an assembly loop.

by blargg on 2010-07-07 (#63861)

So your host machine can send stuff to the FPGA; can it send things back to the host? Can you have the FPGA do other things while emulating, like periodically stop emulation, checksum all emulator state, and send that back to the host?

by jwdonal on 2010-07-07 (#63879)

blargg wrote:
So your host machine can send stuff to the FPGA; can it send things back to the host?

Sure, that's not a problem. As of right now it only communicates back over the UART though.

blargg wrote:
Can you have the FPGA do other things while emulating

Absolutely. Pretty much anything within reason.

blargg wrote:
like periodically stop emulation

Yes, although making sure that every module, every flip-flop, and every RAM is completely 100% stopped would be quite an effort (and very tedious).

blargg wrote:
checksum all emulator state

I'm not sure that this is possible (most unfortunately). There is no way to take like a "snapshot" of the entire state of every single bit in the FPGA at a particular point in time. I wish there was though! Haha. That would be most ideal.

I'm thinking that maybe Reaper is right and this is going to turn out to be more of a brute force effort with ROM hacking and/or RAM checks.

by tepples on 2010-07-08 (#63886)

jwdonal wrote:
blargg wrote:
like periodically stop emulation

Yes, although making sure that every module, every flip-flop, and every RAM is completely 100% stopped would be quite an effort (and very tedious).

Perhaps you could handle it like the reset button on a Famicom or like the paused-waiting-for-coins state on a PlayChoice: only stop the CPU and let the raster effects glitch. While the CPU is stopped, you can dump RAM from the device to the PC and compare it to RAM from a successful run on Nintendulator.

by tokumaru on 2010-07-08 (#63888)

tepples wrote:
While the CPU is stopped, you can dump RAM from the device to the PC and compare it to RAM from a successful run on Nintendulator.

But it might still not match 100% if there are memory locations that weren't used by the program and were left uninitialized... Unless he initialized the memory the same way as Nintendulator (and this way is not filling it with random values).

by tepples on 2010-07-08 (#63889)

tokumaru wrote:
Unless he initialized the memory the same way as Nintendulator (and this way is not filling it with random values).

I don't think a PowerPak's menu fills RAM with random values either. But if one could use an access log such as that created by some FCEU variants to discern used from unused bytes, it would work for a test ROM not specifically designed to test power-up state. Besides, a well-behaved test ROM could just specify that zeroes in unused RAM locations are part of its output.

by jwdonal on 2010-07-08 (#63902)

tepples wrote:
...only stop the CPU and let the raster effects glitch. While the CPU is stopped, you can dump RAM from the device to the PC and compare it to RAM from a successful run on Nintendulator.

Hey now, that's a pretty neato idea. And yeah, I think I would be able to write out a memory address access log without a problem. That's pretty cool. I didn't know that it was even OK to stop only the CPU and not stop the PPU as well - although I hadn't really even thought about it until now. Thanks for the suggestions! Now all that's left is to implement it!