a new alternative to iNES

Even if I've spent quite some time, in the past year, to improve compatibility in the NES emulation of MESS (by adding new mappers, by adding support for UNIF files and by implementing preliminary iNES 2.0 support), I have never been really happy with the current NES formats.
Being my background mainly related to MAME, I've always been under the impression that gluing together the different chip contents + an 'arbitrary' header was not the best solution. Especially, given the large number of ROMs floating around with the wrong header (while working on pirate mappers, I have found tons of chinese roms generically assigned to Mapper 4 or 0, which in fact required different features to work), and the generic lack of consensus on some mapper numbers for obscure chinese carts.

This is why, starting from today, MESS publicly supports loading zipfiles containing separate PRG and CHR files, in addition to iNES and UNIF!
The matching between the ROM files and the correspondent boards is obtained through a .xml database which gets read when carts are loaded and which is public for anyone to use (as long as new findings about obscure boards are made public so that anyone can use them): you can get a copy from this link (click on 'plain' and save the ~2.5MB xml file)

http://git.redump.net/cgit.cgi/mess/tree/hash/nes.xml

Before getting into the xml details, I would like to point out that only 3 things are required to add support for these files in any emu:
1. an xml parser, to read info from nes.xml about the board type, the memory sizes, the mirroring, the dispwitches, the peripherals, etc.
2. the capability to load separate files from inside a zipfile based on checksums or filenames (like MAME, Nebula and FBA do for arcade romsets)
3. a lookup table to pass from the board type [read in nes.xml] to the correct mapper emulation. NEStopia has a lookup table of this sort in src/core/boards/NstBoard.h, if a reference is needed

once the file is loaded in memory and the emulator has chosen the appropriate read/write handlers, emulation can go on as it would usually do after loading an iNES file to memory.

Now let's focus on the xml format. Let's examine a typical entry (in this case for Aerobic Studio)



Each game is in a <software> node with a "name" attribute (a shortname like MAME uses: the emulator will then look for the files inside a train3as.zip file) and a "supported" attribute (this is MESS specific, but other emus can strip it out with e.g. a perl script).

Then, we have the basic info which could be shown in the emu UI:
- <description> for the complete name of the game
- <year> & <publisher> for the corresponding info
the commented out fields (serial, release date & alt titles) might be re-added later, I'm not sure (this xml format is used for many systems in MESS, so some compromise had to be accepted)

Finally, <part> includes everything an emulator needs to know to emulate this specific piece of software (the "name" and "interface" attributes are internally used by MESS to know this is NES and not another console).

Ideally, an emulator could start any file of the xml list by simply parsing the part node:
- from <features>, you would find out: which "pcb" the game uses (in this case a Bandai CNROM variant containing an additional sound chip), so that you can choose the required mapper; if the game has hardwired "mirroring", horizontal in this case; if the game supports particular peripherals; etc. for some particular pcb there might be specific features, e.g. the pin settings for protected CNROM games, or the pin connections for Konami VRC-2, VRC-4 and VRC-6 games, or the chip revision for MMC1 and MMC3 (to handle different MMC3 IRQ or Mapper 1 vs. Mapper 155 emulation)...
- from <dataarea>, you would find out which chips were present on the pcb: in the example we have the expected "chr" and "prg", plus a "adpcm" area for the content of the audio chip, if we ever manage to dump it. Other possible 'dataarea' are "vram" (if present on the board), "wram" and "bwram" for PRG RAM with or without battery, "mapper_ram" and "mapper_bram" for mappers which have internal RAM with or without battery like MMC5, MMC6 & Taito X1-005. At start, an emu should simply load prg & chr, and take note of the other dataarea to setup VRAM & WRAM like it would do by parsing the iNES header
- from <dipswitch>, you would read if the cart had switches on it (this is needed for a few pirate multicarts)

Here are other three examples for Excitebike, Shin Satomi Hakkenden and Taito Grand Prix, respectively



Notice that, as shown by Excitebike, the "prg" dataarea is always at least 32KB wide, with smaller roms reloaded/mirrored where necessary

A few more remarks:

Concerning accuracy of the info in the xml list, the first part of the list is very precise, being based on the latest version of Bootgod's database.
On the other hand, the second part is not so complete (yet): it contains files (around 2500) from various other collections (GoodNES, nointro and NonGoodNES), with pcb, mirroring and vram/wram settings taken from the headers. As such, there might be mistakes which I hope to fix little by little... Moreover, additional info are mostly missing (I will add them next, but I thought checksums had definitely priority over Manufacturers and Release Dates ) and some Alt sets might actually be bad dumps which got incorrectly labeled, and they will need further investigations to be sorted out.
However, I think the current list is not that bas as a starting point, given that 95% of the games shows something when loaded

Concerning included file, you might notice that the xml list does not contain hacks or translations: this is done in purpose, since in my opinion these should be made available as IPS patches (to be applied at loading time, after the separate prg & chr have been loaded into memory). On the other hand, dumps from pirate carts are included.

Finally, concerning the file availability (a not so trivial question, given that it's what basically killed the nice UNIF format), conversion from iNES & UNIF is pretty easy since it mainly consists of splitting PRG and CHR chunks from the existing files: MESS is already capable of dumping out the data when the user loads one of his files, if required, and rommanager programs like clrmame or romvault can use the xml list to create zip archives with the correct names from the split files. Also, there is a group of users actively splitting the files so that ready-to-use zipfiles might appear soon-ish through rom collector websites. I'm not sure this will be enough to gain some momentum,
but I hope I can find some more people not exactly happy with the iNES format and the arbitrary mapper numbers

Of course, I'd be interested in feedback and suggestions (as well as critics, of course) and I'm ready to answer to any question which might arise.

That's all folks. Thanks for reading. Cheers.

A bit of constructive criticism:


In other words, "Now supports Pasofami split format!"


So it's almost Pasofami, just with the board spec in a central XML database instead of a .prm file. But then how can a homebrew game add itself to the database? Should a zipfile include its own nes.xml? Or did you intend it only for ROM images of games that have been released commercially or otherwise satisfy English Wikipedia's notability criteria?


Shouldn't text in XML be in UTF-8 rather than Shit-JIS?


Who registers software names and publisher names?


"no License" sounds like something Tengen or Color Dreams might do


If you want to handle them like MAME handles clones, a zipfile with an IPS in it would have the main ROM as its parent. But then IPS isn't so good at handling disassemble-patch-reassemble hacks such as the newer SMB1 hacks based on Doppelganger's SMBDis. Those would be better served by xdelta than by IPS, as KaioShin of Romhacking.net suggested to me.