Test-ROM thoughts

This is an archive of a topic from NESdev BBS, taken in mid-October 2019 before a server upgrade.
View original topic

Test-ROM thoughts
by oRBIT2002 on 2010-04-29 (#60575)

During all these years I've been working on my emulator I've tried quite a few test-ROMs that some skilled people have written.
What they all seem to have in common is lack of human understandable-documentation. Tests may fail but I usually haven't got a clue what the problem is all about.
Yesterday I tried a bit with "NESStress". It actually comes with a textfile describing the tests (in a line or two), but it really doesn't tell me what's wrong with my emulator. So, I noticed I failed a few tests, but I simply don't understand what's wrong so I don't know what to fix.

What I really want is a detailed (but still readable) description of the tests, what they do, and the expected end result.
For example, documentation concerning a frame/vbl/timing-test should clearly state what it expects to pass the test. Which scanline is expected? Which state should all flags have etc.

I am not sure about the purpose of this post, but perhaps an idea for possible future test-ROM coders. :-)

What do you think? Am I the only one that thinks the tests are usually somewhat hard to understand?

by Zepper on 2010-04-29 (#60590)

...

by sahib on 2010-04-29 (#60596)

I always found the PPU testing part of NEStress to be quite cryptic. I do recommend using blarggs' ppu tests though, which comes with a neat little .txt-file which is a bit more verbose than NEStress' Error or OK!

When I was debugging my CPU and I got in to a trench, I'd always trace operations instruction-for-instructions and compare it with an emulator like Nintendulator, to see if they differed. And as Zepper suggested, just look at the disassembly, and see if the expected operation differ from that of your emulator.

by blargg on 2010-04-29 (#60598)

The tests' purpose is to test some behavior, and make it clear what behavior they're testing. It's not to describe the correct behavior; that's what the Wiki is for. That said, I'm sure my tests often fail at making clear what behavior they're testing. BTW, I've found Nestress to give wrong results, one reason I started writing my own tests.

Any tests in particular that are unclear?

by MatthewCallis on 2010-04-29 (#60601)

Many times blargg's test ROMs are obtained from ROM sets or ROM sites and don't come with source or any text documents. This was a problem I had early on.

by oRBIT2002 on 2010-04-29 (#60602)

blargg wrote:
The tests' purpose is to test some behavior, and make it clear what behavior they're testing. It's not to describe the correct behavior; that's what the Wiki is for. That said, I'm sure my tests often fail at making clear what behavior they're testing. BTW, I've found Nestress to give wrong results, one reason I started writing my own tests.

Any tests in particular that are unclear?

Your tests are awesome Blargg, however I keep failing them all the time. I've obviously got my basic timings wrong. Your tests seems to be extremly cycle-accurate, if the emulator is 1 or 2 cycles off, the test fails(?).

by byuu on 2010-04-29 (#60603)

Quote:
So, I noticed I failed a few tests, but I simply don't understand what's wrong so I don't know what to fix.

The people writing the tests have a deep understanding of the behavior in question, whereas they have no idea what your knowledge set is. Does the term "DRAM refresh" confuse you? Do we expect you to look that up on Wikipedia, or should we make each of thousands of tests dozens of pages long in their explanations?

Documentation is also error-prone. You can't run your explanation through an emulator to make sure there are no mistakes in it. The more we have to explain, the more likely we say something wrong and it throws you off.

We could certainly do better, just explaining things from our side a little.

by tepples on 2010-04-29 (#60635)

byuu wrote:
Does the term "DRAM refresh" confuse you? Do we expect you to look that up on Wikipedia

That isn't a bad idea. That's why I made a list of URLs of Wikipedia articles to read for background information. Such a list might be useful to include with a test suite so that emulator authors can know what they don't know.

by oRBIT2002 on 2010-04-30 (#60649)

From the nesstress.txt file:
"*PPU $3FFF-$0000 Wrap Around.
To test if the address is correctly reset to $0000 after
access to $3FFF. One emulator actually crashed during this test."

What's the deal here? What happens if you read/write at $3FFF and the PPU increment bits are set to 32? How would the wrap around work?

by Disch on 2010-04-30 (#60656)

Wraparound works the same way it always works. The value wraps around =P

The PPU address is actually 15 bits wide, so it would wrap at $7FFF->$0000, though, not $3FFF->$0000 (that extra bit controls the fine Y scroll). Wrapping can be accomplished by ANDing with $7FFF after you increment the address (though it's not entirely necessary to do that, but whatever)

When you read/write $2007, only the low 14 bits of the address are used, so you effectively AND the address with $3FFF (or you could say that $4000-7FFF mirrors $0000-3FFF -- it's the same thing)

So if the PPU is incrementing by 1:

Code:
ppu address - address to access $2007
-------------------------------------
$3FFE - $3FFE
$3FFF - $3FFF
$4000 - $0000
$4001 - $0001
...
$7FFF - $3FFF
$0000 - $0000

$3FFF

by tineras on 2010-07-21 (#64589)

I did not want to start a new thread to ask this question.

Can someone point me to some good test roms? I looked a bit through the forms and on the main page, but I'd like to get a consensus of what is good so I can test against a 'gold standard' and get things right.

I have the most of the cpu code implemented and a good portion of gpu code as well. I am starting to draw stuff to the screen, but there are obvious errors in my code somewhere. There is a resemblance of the correct image on the screen, but with a lot of graphical artifacts.

I would like to start out with some very basic tests to make sure everything is operating how it should be (basic cpu functions) and them move on to the more complex ones.

Anyway, I'm not really looking for help with a specific problem, just some good test roms. If anyone has any suggestions, I would really appreciate it.

by Zepper on 2010-07-21 (#64599)

http://wiki.nesdev.com/w/index.php/Emulator_tests

by tineras on 2010-07-21 (#64600)

Zepper wrote:
http://wiki.nesdev.com/w/index.php/Emulator_tests

Awesome. Thank you. I have only glanced at the wiki a couple of times and didn't notice them there.

by cpow on 2010-07-21 (#64602)

tineras wrote:
Zepper wrote:
http://wiki.nesdev.com/w/index.php/Emulator_tests

Awesome. Thank you. I have only glanced at the wiki a couple of times and didn't notice them there.

I keep all the test ROMs [and some of my crap ROMs] I test NESICIDE against in my git repo on www.gitorious.org:

http://www.gitorious.org/nesicide/nesicide2-master/trees/master/test_roms

[Click on the "Download master as tar.gz" button on the right side of the screen to get it all.]

And I also keep a status of test results on my wiki:

http://wiki.nesicide.com/doku.php?id=test_roms_status

I probably should have asked blargg and the other test ROM authors before putting all their hard work there, but I think it is all publicly available on other sites so I hope they're okay with that.

I'm not saying you have to keep a publicly visible test ROM status file, either...but in my opinion it gives potential users and/or development helpers the chance to see where a)your emulator might do a better job at emulating some esoteric system behavior that they'd like to take advantage of or b)they can help by using their knowledge of how to get tests to pass to improve your emulator.[/url]

by blargg on 2010-07-21 (#64605)

Posting emulator test results is also an objective way of evaluating an emulator. Simply seeing it stated that "sprite hit timing is accurate" isn't very meaningful. I suppose I should be more diligent about versioning my tests.

by essial on 2010-07-21 (#64608)

blargg you should join our team to help us make things as accurate as possible

by cpow on 2010-07-21 (#64610)

blargg wrote:
Posting emulator test results is also an objective way of evaluating an emulator. Simply seeing it stated that "sprite hit timing is accurate" isn't very meaningful. I suppose I should be more diligent about versioning my tests.

I used to keep your old versions around [i had six or seven instruction test versions] but then my pass/fail counts were polluted with repeats. I have left a couple of them in there, specifically the MMC3 test suites that you recently updated because I passed the scanline timing test in the old suite but not the new. I just haven't got back to MMC3 yet to figure out what's going on there.

git is a *great* and *easy* VCS. I use SVN/ClearCase professionally but now have joined a few others at work that use git instead.

by Zepper on 2010-07-21 (#64613)

@NESICIDE: I tried in both Internet Explorer and Firefox. WinRAR says it's a corrupted file (around 12 megs).

by essial on 2010-07-21 (#64614)

Poor windows users

You can browse individually here:
http://www.gitorious.org/nesicide/nesic ... /test_roms

HEAD is pretty darn big now :p (no pun intended, seriously)

by cpow on 2010-07-21 (#64615)

Zepper wrote:
@NESICIDE: I tried in both Internet Explorer and Firefox. WinRAR says it's a corrupted file (around 12 megs).

WinZIP worked for me...do you have that?

Or, you can install git and get the repo...

by cpow on 2010-07-21 (#64616)

essial wrote:
Poor windows users

You can browse individually here:
http://www.gitorious.org/nesicide/nesic ... /test_roms

HEAD is pretty darn big now :p (no pun intended, seriously)

Maybe we should set up a separate repo just for tests...

by essial on 2010-07-21 (#64617)

NESICIDE wrote:
essial wrote:
Poor windows users

You can browse individually here:
http://www.gitorious.org/nesicide/nesic ... /test_roms

HEAD is pretty darn big now :p (no pun intended, seriously)

Maybe we should set up a separate repo just for tests...

No :p
Honestly before too long tests will be managed inside of the IDE.. you'll see what I mean in a few weeks hopefully

by Zepper on 2010-07-21 (#64618)

essial wrote:
Poor windows users

by essial on 2010-07-21 (#64619)

Two can play at that game :p

by Zepper on 2010-07-21 (#64621)

- Right, but I don't mind. I still love Windows. Yup, I already tried Linux/Ubuntu. Back to the topic now, plz? ^_^;;

by tokumaru on 2010-07-21 (#64622)

essial wrote:
Two can play at that game :p

Zepper's is much better... It makes fun of the people who are bashing an OS, like you just did, not the competing OS itself.

I say we stop the OS war now. Yeah, Linux is great and all, but not everyone has the time and the competence necessary to configure it, so as long as Windows allows you to point-and-click through tasks that in Linux require a shitload of complex written commands, it will have a large user base.

by essial on 2010-07-21 (#64623)

Last post before getting back OT.. I have all 3 OSs on my machine (yes, OSx86 :p). I've recommend and installed Windows for more people than Linux -- and in both situations they've been happy so far. I just love trolling sometimes :p

by tepples on 2010-07-22 (#64636)

tineras wrote:
I have only glanced at the wiki a couple of times and didn't notice [[Emulator tests]] there.

If you failed to find something, it's likely that we failed to make it findable. Where on the wiki did you expect to find a link to this page?

NESICIDE wrote:
git is a *great* and *easy* VCS.

How well does it perform in a mixed Linux/Windows environment? I was thinking of using Mercurial; is git noticeably better?

essial wrote:
Poor windows users

Not everybody has a beefy enough PC to run VirtualBox.

by qeed on 2010-07-22 (#64637)

git is one of the fastest src control evah, check out this video http://www.youtube.com/watch?v=4XpnKHJAok8 One of his designs goals was to make it fast enough to manage the kernel.