A note to all NES-emulator authors: ROM header wishlist

I have a wish for everyone that's coding a NES-emulator..
I know some emulators uses checksums for detecting certain ROMs and to automatically adjust the experience. STOP doing this, for several reasons:
1) I don't like it. Personally I think it feels like "cheating" in some way..
2) It's bad for people doing homebrew/ROM-hacking (like myself). Try hacking VS-ROMs and play them in FCEUX and see what happens. Nothing, they won't work properly because checksums doesn't match the builtin database and you don't get correct VS-functionality.

What do you guys think about this? Am I the only one upset about this stuff?

This could be avoided if every emulator author agreed upon a way to denote all information about the game, and if every user subsequently updated their sets to reflect said format.

But that's not possible even in a scene with two emulators available for a given system, let alone over 9,000.

It's like herding cats.

'nuff said...

While NES2.0 is basically not deployed at all (except for in the things that I myself have released), at least Nestopia does use that header to select the correct PPU for Vs System games.

I guess it's very practical for gamers that doesn't need to bother about the settings for "odd" ROMs.. But I still think the options should be there. FCEUX is a pretty big source for my frustration. How do you develop something for VS-Zapper? The only way is that you start a VS-Zapper game, inject your own code (copy'n' paste to RAM), or hack your own ROM to match a CRC-value already in the VS-Zapper database.
I think Nintendulator behave in a similar way..

You think incorrectly - the only place Nintendulator has CRC checks is in its MMC5 mapper code for automatic detection of SRAM size, and that only happens if it can't find NES 2.0 header data.

For Vs System games, the only stuff it autoselects is the PPU (2C04 variants for palette, 2C05 variants for $2000/$2001 swap and the special bits read back from $2002) by looking at NES 2.0 header data; "Zapper" and "Vs Zapper" are completely different controllers that you have to select and configure separately.

For MMC6 (Startropics) you can look for a write to the WRAM register to set certain bits which other games won't. I think Startropics 1 and 2 both write #$20 to $A001 during startup which is related to SRAM block enabling. Regular MMC3 games don't set those bits. It's possible some do. But if they don't then that's one way to detect those games.

When you plug in a game cartridge to a real NES, the cartridge has significant amounts of "information" on it besides the bytes of the ROM image. That information includes the exact functioning specifications of every single mapper chip, plus how they're all connected, and more. A real NES never has to guess what variety of mapper chip is on your particular board, or whether there's a resistor at R5. It knows. Why should emulators not be allowed to know the same things?

As far as modified ROMs go, there have been some efforts made in Bizhawk to help out with those sorts of things. If there's something in particular that you need help with for a particular project, I'll see what I can do.

He specifically doesn't like using a checksum to ID a ROM image. Emulators ideally should be told about special cases in the header. The problem is that iNES has been around forever and expanded apon. Many people still have and distribute roms with bad or older headers. So it's up to the emulator to just fail or try to do additional checks to allow things to successfully load.

iNES 2.0 probably nails a number of special cases but the problem is the games that need this info already exist with older iNES headers that don't mention the additional information. So unless the user updates their header, it won't work. In theory though if you are a hacker, you should update the header and then the checksum will pass for the older header roms, and your hacked rom will specifically tell the emulator what to do. So it all works if the emulator supports the newer iNES header and acts on it.

There's also nothing wrong with a checksum/hash to detect certain games. However you'll have problems with hacks if you are hashing *everything*. Instead you could pick a portion of rom to hash that is unlikely to change in a hack, but still unique.

Wrong. The NES doesn't know shit, as far as it's concerned all carts have 32KB of PRG and 8KB of CHR! It's the cart itself that does all the magic, and since the cart (board, mapper, wiring, etc.) is missing from the emulator + ROM combination, the emulator has to get that information from somewhere in order to properly simulate the cart. Using the contents of the ROM to know what cart to simulate is a poor way of solving the problem, for the reasons oRBIT2002 mentioned.

I thought about this approach for the Game Boy Advance (no information on whether the memory is SRAM, EEPROM or Flash, nor what size.) You can usually tell what it is based on what the game tries to do (barring copy protections.)

The bigger problem I faced was with things like save states and auto-rewind snapshots. Dynamically changing the amount of RAM the cartridge supports means you have to be extra careful when serializing and unserializing data. A bit heavy-handed to just always dump 128KB of "any RAM" into all save states ...

Yay, let's argue semantics instead of fixing the problem. The emulated system, by which I mean the combination of the deck and the inserted cartridge, does not do any guessing, any heuristics, any fuzzy logic. While it's true that the NES deck itself does not know anything about the internals of the cart, all relevant internals are exposed by the sequence of data reads from the PPU and CPU buses, as well as cart interrupt timing. There's no uncertainty.

While it's true that hashing doesn't match what the console does, hashing does mach certain facts of life: Whenever a particular PRG ROM+CHR ROM combo was encountered on manufactured cartridges, it was always on a particular board (or one from a short list of particular board.) The most correct solution would be to put information in the ROM header, I agree; but that's not done, so pointless to talk about it from the point of view of what emulators should implement.

but we don't emulate "the combination of the deck and the inserted cartridge", we emulate the deck and we try to handle the best we can what comes in the cart

this discussion is meaningful for me given that, even if in MESS we have a database of known good dumps, we also keep supporting iNES/iNES2.0/UNIF to allow homebrew software to run

the problem is that either we establish some sort of 'central authority' which assigns in a meaningful way the mappers and keeps doing this until the firms in Taiwan and HK quit producing new carts, or there is no solution

just look at the 5-6 known cases of mapper #s which have been assigned to 2 incompatible boards...

All that's saying is that we make a best effort which doesn't always succeed. The intent and direction of effort is to emulate the combination of the deck and the inserted cartridge.



I think UNIF is a bit better than iNES in this respect, just from the freedom of board choice; any string of reasonable length and content is allowed, so inadvertent collisions are much less likely than when you only have a few numbers to work with. Of course, neither UNIF or iNES 2.0 is used much at all in the wild, so...

But therein lies the cat herding problem.

Who decides what information is worthy of going into the header, and how it's marked up?

Maybe I want to know what controllers a game is compatible with, so I can auto-select them. Maybe everyone assumes 8KB PRG is enough for Galaxian, and then a 4KB PRG ROM is discovered later on. You never know.

To me, the problem is trying to fit the whole game into a single file, and compressing all of this rich metadata about the PCB into this 16-byte header that has to work around a legacy mid-'90s header format.

I'd rather separate the ROM and the description of the PCB itself. And then allow that description to be stored in an extensible format (JSON, XML, ...) so that when something new comes up, we can support it.

From there, I'd probably keep a database of every commercial game internally, so that all commercial games can just load and play, regardless of whether there's an iNES 1.0 or 2.0 header, or not. And then require the external description to load homebrew. If it has to be one file, store the ROM and the description both inside a ZIP archive.

But now you have another problem. All the Javascript emulators are going to want to use JSON. MESS is going to want to use its own proprietary XML format. Nestopia has its own XML format. I use BML myself.

Even if we agreed to store external board descriptions in an extensible format, and even if we all agreed on a specific markup language, and even if we all somehow agreed on what tags to make and what to name them, we'd still run into issues where certain parsers had bugs, so a file that doesn't end in a line feed works on emulator A, but not emulator B.

Nobody who has already made homebrew would go back to make this board layout, and few people making new homebrew would want to make layouts for the dozen different emulators.

So I've pretty much come to the conclusion that there is no solution to this problem.

Support iNES 1.0, support iNES 2.0, support as many external board markups as you can, support an internal database, cross your fingers and hope for the best.

This pretty much sums it all up. Everyone just needs to live with it.

You'll never have an end-all solution that covers everything. The best you can do is have a dual approach: self-contained format(s) for describing new games, and a database of known games. The former allows a determined person to make a game work. The latter accounts for all the existing files out there to make them work without having to convert them to a "correct" format.

We've been trying to start a mapper registry. First there's the grid to document what we have, and then there's a "planes" proposal to segregate new dumps from the Far East into NES 2.0 mappers 512-767.


One of them is #34, but that's easy to solve: CHR RAM or 8 KiB CHR ROM = BNROM, and anything bigger = that NINA board. Others are disambiguated with iNES submappers.


I asked kevtris about this and he said that in these cases, an overdump shall be considered a "good dump" in the iNES or NES 2.0 format. For example, Slappin' (from the 2011 compo) has a 4 KiB CHR ROM repeated twice, and Galaxian has an 8 KiB PRG ROM repeated twice.

I agree completely. Implement NES 2.0, and optionally implement .NES.INI too (the ROM images I have written relased so far, and that I will do so in the future, all include such a file; if different emulators use their own format, a program can be written for conversion in both directions).

Luckily, in most cases, a simple script (possibly even online) could be written to convert from one format to the other




yeah, but I was using those as an example of what happens when there is no centralized effort to keep consistency as it has happened in the past... and sometimes still happens, with a few chinese dumpers (not all of them, luckily) marking as plain mapper 4 any waixing mapper variant they dump and release, so that it does not work in most emus

> and optionally implement .NES.INI too

Ooooh, another new format. And one that doesn't support trees. Don't forget to also include a MESS internal database entry, a Nestopia XML file, and a higan manifest.bml as well. Is anyone using JSON or YAML yet? Gotta be somebody out there.

> Luckily, in most cases, a simple script (possibly even online) could be written to convert from one format to the other

Tough to say. MESS' SNES DB does not capture any ROM/RAM address mapping/mirroring. You would not be able to convert your entries to mine. Similarly, I don't capture the writing on top of chips, so I couldn't turn my entries into yours (well, they'd probably work anyway, since that info is not used by emulators.)

But ... you make an interesting point. Just as a hypothetical, what if we had a community effort to form a massive textual database (in some common extensible format, it doesn't matter which), that each emulator author could derive from? Pull out the info they need, and format it into their preferred markup language with their preferred variable names.

The idea would be that absolutely anything and everything would go into it. If anyone finds certain info relevant, we could add it. Peoples' derivations could discard what they consider unimportant. Not only could you generate XML/JSON/YAML/BML, you could also generate iNES 1.0/UNIF/iNES 2.0 headers from the information. Even iNES 2.0 fans should consider it a boon, as it'd be a really fast way to propagate any fixes to bad iNES 2.0 headers.

It would be trivial to include unlicensed games as well, by having a flag to differentiate them from licensed games. Any volunteers that try and run some homebrew which doesn't work, could quickly submit a new board definition, and all emulators participating would benefit.

Distinguishing among original licensed games, original unlicensed games, and original homebrew games could be done with a flag called something like CIC Type. It'd take values such as "3193" (for licensed North American games), "Rabbit" (for Tengen games), "CIClone" (for the majority of homebrew), "passthrough" (for Game Genie, HES games, and that Noah's Ark game for Super NES), or some designator for a CIC stun circuit. But then that still doesn't solve how to designate ROM hacks of licensed games, which are usually built on donors.

If there's a centralized database, just have a unique ID for every board/tv format/PPU/whatever combination, then use the reserved bits in the iNES 1.0 header to indicate that there's a database entry. e.g.:

- byte 11 - some unique value to indicate that the rom is described in the database. Or maybe use 4 bits for that purpose and 4 bits as a checksum/CRC to reduce the chance of some DiskDude! like corruption?
- bytes 12-14 - database id. Lower 16 bits would be primary board type (i.e., existing iNES mapper assignment), upper 8 bits would define variant/different configurations.

That's basically what NES2.0 is, except that NES2.0 has less goofy indirection.

The mapper variant part, yes. But NES 2.0 attempts to further define the ROM in the header, and there will invariably be something that doesn't fit into the format. The idea here is to reduce the board definition to a simple ID, where anything can be defined in the database. Is it the same as using NES 2.0 and ignoring everything except the mapper variant field? Yeah, I suppose it is.


How so?

Hence why the remaining bits are explicitly marked as 0 in the header for future use, if we figure out what needs doing. It is possible we'll run out, but we've got a significant amount of headroom with no indication we'll need to use it.

If the concept that you want to contain is, say, "This a Vs. System game that uses the RP2C04-03 without any additional mapping hardware", in NES2.0 you set the bits that exactly correspond. Providing a SQL unique column index, or whatever you want to think of it, instead says "Please look up UID 26987", which in turn tells you that "it's a RP2C04-03 without any additional hardware". You could sort the UIDs such that they're actually a bitmap, but then that's basically NES2.0.

In any case, there's enough information in Nestopia's database (and thus also in the export from nescartdb) to supply everything but the NES2.0 submapper field for a header-fixing tool. It's just no-one's done it, either due to not wanting to touch it with a ten foot pole or not having it occur to them.

So would you dedicate some of that space to defining different VRC4 revisions? How about something like whether or not expansion audio is used and how it's mixed on the cart?


That and because iNES is good enough for the vast majority of cases/users.

IDs are flawed and unnecessary, and you'll have different people defining their own conflicting IDs, just like with mappers.

Map database entries to SHA256 hashes. Barring a future intentional exploit that allows SHA256 hash collisions (which we don't have yet, and if there ever is one, we can use stronger hashes), there are more possible ROM combinations than there are atoms in the universe.

And now we don't need a header at all, so we can leave it with DISKDUDE or whatever other people insist on keeping there.

> Hence why the remaining bits are explicitly marked as 0 in the header for future use, if we figure out what needs doing.

I want to know what controllers a game supports, and which ones are the preferred input methods, if applicable. Where can I put that into the iNES 2.0 header? I'd also like to note what the default overscan clipping should be for each game, to avoid showing scrolling artifacts on the edge of the screen. Is there space left for that?

What do you do with homebrew and hacks? Fallback to iNES/other header and hope for the best?

@byuu
The NES 2.0 file represents the function of everything inside the Game Pak shell. Things like controllers and overscan go in the manual. (Yes, a few NES era manuals mentioned overscan. I can provide references if you want.) So what you need is a machine-readable version of things commonly found in the manual.

In most cases, if you stick to the overscan amount that the scaled mode of PocketNES implements (8 left and right, 16 top, 11 bottom), you should end up with 240x213 pixels of clean video. Nesticle used a heuristic that if the game sets the clip bit to cut off the left 8 pixels, the emulator would also cut off the right 8.

EDIT: A scan of the Jeopardy! 25th Anniversary Edition manual shows controller information on page 4 and overscan at the bottom of page 18: "This game has been programmed to utilize the full TV screen. Since some older model TV sets have rounded screens, a portion of the image may be blocked out." The transcription of the Super Mario Bros./Duck Hunt manual has similar language about overscan.

@James
Homebrew and hacks can fall back to NES 2.0. If they use something not describable in NES 2.0, they can come with a database entry in the same format as the entries associated with hash values.

Submappers. Already done for VRC4. I'm tentatively working through the N163, but I don't know enough to make an official stance. I'm not certain what the point of explicitly marking a "no sound" version of VRC7 for TTA2 (separate from the question of A3 vs A4 for the register selection).

I don't understand why keeping people from intentionally making their ROM collide with something else is useful. The ability to make a collision even seems useful to me, if you have a game that isn't describable using the conventional headers. The bit that started this entire thread is for exactly this reason: iNES1 headers are insufficient, and overly-precise hashes prevent useful intentional collisions.

I guess the answer is: because there's two different objectives: One is denoting "This is the Real Original Game" and for that you want to prevent people from distributing broken things (But is that a real threat, or just an imagined one? CRC32 is enough to prevent almost any accidental collision). The other is "use this data as though it were on this PCB" and that's theoretically what the headers are for.

No. .NES.INI is a separate file for each game, not a database (this is why trees aren't needed). Things needed for emulation should be in the NES 2.0 headers; if the header is bad, it should be corrected with an external tool rather than within the emulator itself. The emulator itself shouldn't do cheating stuff like checksums and so on; it should use NES 2.0 headers if available. The .NES.INI is supposed to contain extra data which usually isn't needed for the game to work (although the format allows such things to be provided in case some emulator needs it).

However, a SQL database can be made containing game checksums and then given the PRG ROM and CHR ROM files (and possibly other data), can map this data to a .NES.INI and to a NES 2.0 header. But this should be an external tool, not part of the emulator. I know about programming in SQLite so I can know how such thing can be done. You could have seven fields: game title, PRG ROM checksum, PRG ROM size, CHR ROM checksum, CHR ROM size, NES 2.0 header, and .NES.INI data.

I don't know what XML format Nestopia uses, or what it uses it for.

For the emulator to use hashes instead of the actual data in the header, is a klugy way which I think is no good (for several reasons). Sometimes it might be necessary, but usually the header should be corrected before loading in the emulator!!!!!!!! Nevertheless, a few emulators have some features that are badly written and need a checksum to know how to run them (ideally, no emulator should require this, but sometimes there is no choice); for these things, .NES.INI does provide a solution (if the .NES.INI exists and contains a bootgod ID of the game, that should be used instead of the checksum). However, if the header is simply incorrect, you should not do this; in fact, the .NES.INI specification specifically says that you shouldn't correct the iNES header based on the bootgod ID.

That is precisely what the .NES.INI format is designed for; things like controllers and overscan. It is not intended to correct bad headers. The NES 2.0 header should represent the function of the cartridge and should do so accurately (if it is wrong, it should be corrected externally and not within the emulator).

It is not intended that any emulator must support the .NES.INI format; it is a completely optional file (although I provide it in my own software for the Famicom, despite not yet being supported by anything; emulators will just read the .NES and work fine, although you might need to configure the input device manually).

On "no sound" Famicom cartridges, 2A03 audio is simply passed through untouched (pins 45 and 46 bridged together), so audio output is 1.0 * 2A03 output. On cartridges with expansion audio, 2A03 output is mixed with expansion audio at some ratio, so audio output will be x * 2A03 + y * expansion. I'm unsure if the values of x and y differ for boards with the same mapper (e.g., is Just Breed audio mixed in differently than Metal Slader Glory audio is?).

In any event, knowing how audio is passed through the cartridge is necessary to accurately emulate audio output.

Upon further reflection, I think you're right about this. The iNES format is sufficient for the vast majority of ROMs. There is little incentive for users to switch formats or distribute new ROMs, so this has to be handled without any header changes.

Though I think it should be CRC-64 instead of SHA-256. Mainly because the implementation is easier, but a little bit because we can't agree on everything here.

Seriously, though, let's just do this. How do we get started?

> What do you do with homebrew and hacks? Fallback to iNES/other header and hope for the best?

Add them to the database.

> The NES 2.0 file represents the function of everything inside the Game Pak shell. Things like controllers and overscan go in the manual.

Every PCB attribute known so far. Who knows what Hong Kong will come out with next.

By having an infinitely extensible format, you never have to worry about running out of bits. After that, you could argue binary vs text, embedded in the ROM vs separate.

You're already using submappers to denote VRC pinouts, so there's no longer even a 1:1 correspondence of bits to functionality. What the bits mean change based on other bits. And you already have a pseudo-database inside the emulator to tell you that mapper #(rand % 256) == VRC4 anyway. Codification into a database really isn't much more work at this point.

> I don't understand why keeping people from intentionally making their ROM collide with something else is useful.

Mainly for preservation. I am trying to document every licensed SNES image, and for that, I need hashes that verify images are pristine originals.

> But is that a real threat, or just an imagined one?

You'd have to ask historians. We're probably somewhat well off in that even if a fake did appear in the timespan between a hashing algorithm being broken and a stronger one appearing to replace it, the real one would presumably have more copies around anyway, so in theory it should be easy to spot.

Still, no harm in some due diligence. I have my own master set, and will rehash the originals with SHA3+.

> No. .NES.INI is a separate file for each game, not a database (this is why trees aren't needed)

My BML files are separate for each game as well, and can also be in a database.

Trees are useful for a single game as well. They help show parent-child relationships.

Here's a template for a DSP-1 board in my format:



In an actual generated manifest, there would be a separate tree for the information (preferred controllers, catalog name (for sorting), actual name (in native language), publisher, serial#, etc.

> Though I think it should be CRC-64 instead of SHA-256.

The beauty of this is that we could store as many hash formats as we wanted, and the emulator author could pluck out the fields they wanted. Bonus points if the emulator has a "sync with online database" option in the menu for updates.

I think this is stupid to ever use a database for the emulator to decide. Use the NES 2.0 headers; if they aren't available, use an external program to convert them.

byuu....you have the best emu nobody uses because it doesn't play any roms. I know I've downloaded meaning to convert ROMS and just left ti in the dust. But really, CRC crap and it telling you what ROM version, good bad, or nonexistent...who really gives 2 craps? Especially when it annoys you. It wouldn't be too bad if it was ALL in the background, and didn't have to annoy the piss out of you with the message you have to load 5 times to read it all. Headers are the absolute best all-in-one solution. If I'm on a desert island with my laptop and want to play games, fuck CRC checks and fuck databases. Self contained formats are the only ones worth 2 damns, end of it.

Nice. Taking this further, settings like per-game overscan and turbo rate are like emulating a separate game room for each game. There you can adjust the TV's internal controls and have a NES advantage with the turbo controls set just how you like it for that game, and not have them disturbed when playing another game in a different game room.

3gengames, just a friendly reminder, I never read any of your posts:



If it wasn't about disparaging me (which would be a first), then please accept my apologies and ignore this message.
If it was, you're wasting your effort here. Might I recommend you try 4chan? There's a much more receptive audience to trolling there.

> I think this is stupid to ever use a database for the emulator to decide. Use the NES 2.0 headers; if they aren't available, use an external program to convert them.

To each their own. By the way, if you develop an emulator, and if it gets popular enough, I'll save you the trouble of learning this lesson yourself:
People went crazy when I made my emulator require an external program that translated games to the format I wanted. You're suggesting the same thing, just with iNES2 instead of BML.
I ended up integrating the conversion tool right back into the emulator anyway. And now everyone pretends that said integration isn't there, because it was missing from a single release.

> Nice. Taking this further, settings like per-game overscan and turbo rate are like emulating a separate game room for each game.

There's always RetroCopy. You can have your own 3D rendered video game movie theater room to play your games on.

Anyway, I like the idea of per-game overscan. I don't want to see artifacts on vertical scrolling in Mega Man, but cutting off 16 pixels from the top of the screen looks terrible in horizontal scrolling games. I wouldn't recommend including subjective data into binary headers inside games themselves, but it's a nice addition to extensible metadata in a database. Let each emulator author choose if they want to act on it or not.

Thanks, then we don't have anybody justifying garbage. You call it trolling, I call it being the only person who will call you out on your emulator forcing the worst standards ever seen in any emulator, failed or not.

Am I the only one who thinks that should be left out of the ROM and instead just make the user select the peripherals? I know this is what's done on Mega Drive emulators, no idea why it can't happen elsewhere. I mean, on real hardware you can be using the wrong set of peripherals and nothing would stop you if you forget to swap them out...

Something that could be done is store the peripherals used when the ROM is closed and restore them when the ROM is reopened again (which in fact will ensure that what the user decides is the ideal is used, not what the emulator thinks it is). Of course make this a setting. Keeping track by filename should be enough, this is already done for savegames and savestates anyway.


For the only thing intentional collisions are useful is to make it harder for emulators to attempt to emulate the game correctly (as they can't tell what they have to emulate, at least without trial and error). I guess this makes sense if you want to ensure nobody can play it anywhere except real hardware, but eventually emulator authors will find a way around it.

EDIT: typo

> Am I the only one who thinks that should be left out of the ROM and instead just make the user select the peripherals?

I think it should absolutely be left out of the ROM.

But inside some sort of information markup or database, it's a nice idea.
This is just something that makes the emulator more friendly. Duck Hunt only works with the Zapper. Why make your user select the Zapper when they load Duck Hunt, and then swap back to a controller when they load Mario Bros? I picked that example intentionally, as there's the infamous SMB/DH combo. So when the user goes to choose a controller for SMB/DH, why show them the Oeka Kids tablet? What if on a whim they played Oeka Kids, but didn't know it used the tablet? What if they play Tokimeki Memorial and don't even know it supports a mouse? They may not have a copy of the manual, as very few people are scanning those in. Yeah, their fault, but still. We can help them with that.

Same reason I include a cheat code database with my emulator. No reason to make users go on Google looking for codes that may be for a different country or ROM revision, or just may not work at all.

We can make users do lots of work. Usually users tend to like having to do less work.

> I mean, on real hardware you can be using the wrong set of peripherals and nothing would stop you if you forget to swap them out...

Sure, and you can run NTSC games on PAL (maybe with an adapter) and vice versa to get lovely error messages. And some people want to see those errors for some reason, and so we should allow them to if they want that.

> Something that could be done is store the peripherals used when the ROM is closed and restore them when the ROM is reopened again

Exactly. And you could also include overscan clipping, preferred video filter (Zelda 3 looks great with HQ2x, DKC looks terrible with it), custom controller bindings, cheat codes, etc.

Now what if, the very first time a user loaded a game, we looked up the best default settings for them, and then let them configure the settings from there?

> but eventually emulator authors will find a way around it.

Custom polynomials for a per-emulator-build checksum routine would do quite nicely

I used to think that this information was completely unnecessary, but after giving some thought to it, emulators are often about enhancing the experience. We get savestates, video filters, cheats... luxuries we can't possibly have on the actual console. So why would automatic controller configuration be any different?

I do agree that this should be left out of the ROM though (it should be in an external file, a database somewhere, etc.), and be completely optional and overridable. After all, there are legitimate uses for unusual controller configurations, like enabling cheats and so on.


Perfect solution, IMO. I see no problem in loading the initial settings from a database though, if you enable this option.

There's a mouse, and then there's a mouse. The Super NES mouse and the Subor mouse have different protocols, and I've had to reply to e-mails from players who couldn't get the mouse to work in an emulator because Thwaite supports only the Super NES mouse. ("Try it in Nintendulator." "It worked; thank you.") How many mice are there?


And sometimes you have to make users do work to satisfy the requirements of the law.


Especially homebrew developers testing their work. A game that I developed in 2009 had separate builds for 50 Hz and 60 Hz machines and would freeze on a "REGION CODING FAILED!" screen if the TV doesn't match to discourage people from cheating by playing the NTSC version on a PAL NES or Dendy to get 20% more time to react. Another game that I developed two years later autodetects the TV system but has speed correction in the same build.


Then you might be infringing a Nintendo patent.

This looks like it might be something similar to the mapper 768.0 which I have defined, but I don't really know if that is it.

I don't know how many; could you add information of Subor mouse into wiki? (I recommend the use of the Famicom keyboard and SNES mouse in new projects (these are the official Nintendo protocols), although it would be useful to have the other information too.)

Occasionally I think about the state of the .NES 2.0 header. And while it seems like it would work, it also feels like a kluge.

I know that there is probably no chance of this idea being adopted, and that people (including myself) wouldn't be thrilled with "yet another NES header."

Regardless, I feel like it's better to at least get the idea out there, feel free to shoot it down (but please, provide reasons so it can be a learning discussion).

Given the current state of NES format, and the strong desire for backwards compatibility, here's what I would consider ideal.

A structure which looks like this:



and can be located anywhere within the file, but probably the end of the file for maximum compatibility. Here's the concept:

When loading a ROM, you search the file for a magic value. If I were implementing this standard, I would enforce that it must be on a 4-byte boundary so make the search efficient. If found, you read the version number (so you know the size of the structure) and the checksum. Then you CRC what you found. If that matches the checksum, this is your header, you can just ignore the iNES 1.0 header entirely from that point forward.

There are several benefits to this approach:

* 100% backwards compatible, should not break old emulators
* easy to apply to existing ROMs, you just append it to the ROM file.
* versioned, so we can add new fields as the community strives for more and more accurate emulation.
* because it can be located anywhere, it's size is not limited. So if a new version wants to add some new bytes to the structure, it's trivial to do so.
* provides an opportunity to recreate the structure in a more sane way (no more munging together bit fields to create values). For example, I would just a uint16_t for mapper numbers, I would use a full uint32_t for PRG and CHR sizes (but with the standard spelling out how to handle non-power of 2 sizes).

The only counter argument I can foresee at the moment is "the loading is relatively complicated in comparison to just "read the first 16 bytes and parse." My counter argument to that is that iNES 1.0 already is fairly complex to parse correctly as it has been tweaked over time and requires mashing together bit fields for things like mapper numbers and such. This becomes even more complex as iNES 2.0 comes into play. When comparing the two schemes, a simple linear search followed by a checksum is pretty simple when compared to the bit fiddling currently needed to handle things correctly.

Thoughts?
proxy

Only because you asked for feedback ...

> and can be located anywhere within the file

What if two tags appear in the same file? Some emulators will invariably go with the first, others with the last, regardless of what you say in the specification.

> I would enforce that it must be on a 4-byte boundary so make the search efficient.

Other emulators would happily ignore this, and then homebrew makers would ignore it, and then you'd look like the broken implementation for enforcing this rule.

> 100% backwards compatible, should not break old emulators

You have to insert the data somewhere into existing ROMs. If you put it at the end, you will most certainly break some emulators that rely on file sizes instead of header info. Eg one might say "read iNES PRG ROM size, and assume CHR ROM size == remaining file size."

Remember that there are more NES emulators than there are cars. If it can be done, it's been done

> easy to apply to existing ROMs, you just append it to the ROM file.

Breaks checksums of files, breaks existing emulator CRC32 lookups.

> versioned, so we can add new fields as the community strives for more and more accurate emulation.

Versioning creates as many problems as it solves. But for something like this, I don't see much choice. Especially when everyone keeps insisting on non-extensible data formats.

> because it can be located anywhere, it's size is not limited. So if a new version wants to add some new bytes to the structure, it's trivial to do so.

The header should just specify the packet size.

...

Anyway, please don't be disheartened in the critiques. Always fun to hear new ideas!

My favorite part of your idea is that it could be embedded in new homebrew, while still keeping the PRG/CHR size exactly right. That would truly be 100% seamless with existing emulators.

I did, and it is always appreciated. You thought of some things which I hadn't, which is a good thing . I do have some responses though.



That would have to be addressed on a spec level. First matched tag seems reasonable to me as it would be the easiest for an author to implement. Also, it's what other similar specifications have done (multiboot, ACPI tables, etc) with much success.

With any spec, there is a reasonable expectation and requirement for people to follow it for it to be useful. While it is true that some people will of course not follow it correctly. I could say the same thing about any spec, even iNES 1.0 and 2.0. People could easily choose to interpret some bits the wrong way and not play the game correctly.

There is a silver lining though, most (if not all) NES emulator authors are members of this forum, so it is at least possible to get the majority of emulators on board with doing it right, which is enough for it to work.



Good point, I would have hoped that anyone trying this spec out would do it right, but definitely some people would not. Fortunately, that is nothing more than an optimization which could be dropped. Any alignment would work, just doing it by uint32_t's make it 4 times faster to locate it.



Interesting point. I know that GoodNES hashing everything after the first 16 bytes, so it would be effected (that's how it determines overdumps). It's a problem, but, I don't think it's a major problem, here's why:

#1 it is no different as far as these emulators are concerned from an overdump, there is just some extra data at the end.
#2 emulators which ignore the iNES headers and use hashes instead are fairly new and likely still in development to some degree, so we could work with the authors to get it right.
#3 truly legacy emulators (i mean the ones from the 90's) definitely rely on the iNES 1.0 header, so they won't be effected.
#4 every emulator which makes decisions based on hashes would be insane not to fall back on iNES headers if there isn't a match found. That's what would happen here. It's not perfect, but most games are playable under those conditions.
#5 taking into consideration point #4, those same emulators would have to continually update there database of checksums to keep up with new releases or they risk not being able to play new dumps/home brews. If they are keeping up, they can either add these new "overdumps", or update there emulator to handle it more gracefully.
#6 regarding "read iNES PRG ROM size, and assume CHR ROM size == remaining file size.", that's interesting, and certainly someone has done it. But the resultant CHR ROM size wouldn't be a valid size (not a multiple of 8192). So they would pretty much have to truncate it since they would not have a whole page at the end.
#7 I have a (admitadly stalled) "Open GoodNES" project which aims to be an open source / open DB alternative GoodNES. This may be a good reason to resume work on it!



This is pretty much the same as the previous point.



Agreed, I think it's necessary, but can be done carefully to make it easy to get right.



Agreed! Now that I think of it, it should be done similar to Microsoft APIs. Size is the version. The structure should have a size field instead of a version field. And you simply never remove fields, only append to the end (potentially deprecating previous ones). This has worked for almost 30 years and if there is one thing they succeed at, it's maintaining compatibility with older versions of there API.

All in all, you make some good critiques, but I don't think any of them would be deal breakers. The need is not to get every emulator to do it right, but only the most popular ones. The rest will follow suite if that happens.

I'm surprised byuu didn't ask what would happen if the game itself happens to have something in it that matches the tag starting the metadata. That's pretty much guaranteed to break everything. Heck, it could even be used to ensure emulators can't run the ROMs properly.

That is why I hate this way of doing it. If more header is needed to be added to NES 2.0 (such as to make NES 2.3 or to make NES 3.0), then add a header extension table address field in the header. However I also think many things don't need to be included in the ROM image file, and probably everything that does need to be included is already a part of NES 2.0, so this won't be necessary at all.

I recommend that you never use the database of checksums to determine the file if the header is NES 2.0 format (as indicated by the version bits). Probably best is don't use it at all unless the user enables and loads the database anyways. NES 2.0 headers should always be assumed correct.

I agree that this is the biggest issue with proxy's suggestion (much more than a dev stupidly assuming that he can ignore the CHR size in the header, when the format explicitly requires it for a reason and for the record any Vimm lair overdump would fail in such an emu as well, since that website was appending a 0x10 watermark on its roms, based on the fact that the format only specifies the PRG/CHR chunks and not the whole rom size, so you can load larger files without most emu even noticing it)

MESS took a more radical approach, actually. We prefer users to adopt for non-homebrew entries separate PRG and CHR files, so that we can throw in the wc and flush away headers (both correct and incorrect), and rely only on the database. Given that years of attempts to spread correct headers have proven useless because most users don't even want to know what headers are and why they need a correct one for the games to work, why not to remove entirely the problem and just have to handle good and bad dumps?

iNES is of course fine for homebrew, because one hopes the creator has setup the parameters correct, but for the rest...

I ensure you that we still get reports of people asking why half of their nointro set does not load in MESS, just because they downloaded it in 6 years ago when the set was first built and distributed by website *without* any headers
and people ask why these can't just load like SNES, GB and MD roms (but at least to this I can't reply to check their Atari Lynx and Atari 7800 romsets which have the same problem with headers )

Well, I suppose you could have an emulator to do this if you want to, but if you take this approach, you should probably do the following:

• Make the database external to the executable but in the same directory and provided with the emulator.
• Provide information about the database format in order to convert between iNES format and separate PRG/CHR format; you could also provide an extra program to do this.
• If the database includes the relevant information, also allow .NES.INI file to be created from an entry in the database; this could be done by the same program as the one to convert to/from iNES.

Although doing it your way is OK if you want to, I don't like that way; I think that that the user ought to understand these things when downloading a bad dump or if they are dumping a cartridge they own by themself, to know what the correct settings are (there is bootgod database to help with this).

Note that GameBoy ROMs already include headers; if you copy it from a cartridge the header will already be there, because Nintendo put them there!

about the first 2 points, did you ever attempt to make a search about MESS before posting?
the database is already external, and in xml format (so one can rely on existing parsers to read info from it)
it also was announced to the community long ago: viewtopic.php?f=3&t=6558 and everyone is free to use any info from it whenever they like (even if I'd appreciate people to acknowledge the source of info and share back any fix)
the updated address for the db is: http://git.redump.net/mame/tree/hash/nes.xml

same goes for the info in all the other databases we include with MESS: http://git.redump.net/mame/tree/hash/ (non-NES lists still lack proper documentation of controllers that can be used with a given game, but it will be added in due time)


about the 3rd point: I've been toying for a couple of years with the idea of adding support for a per-game xml file but we are back to the header point... despite trying for a few years, we never succeeded having people to fix headers in they roms, why should they now add an xml (or bml or ini or whatever additional file) to their roms? how would you force them to do that?





you might want to re-read my previous post. all the cases I mentioned (NES, Lynx and A7800) use a format which is not a straight dump from the carts, but rely on bits added by the dumper (or the emu author) in order to setup mappings of the ROM inside the emu memory. As such, even if you get a perfect dump from a cart you own, you cannot use it in emus which don't use hash databases because the binary dump does not contain the mapping data needed for emulation (and typically you blame emulators, not without a point)
OTOH, of course GB roms have an header too, but it's part of the original binary you extract from the cart so it has not to be added (artificially) by whoever spreads the dumps and does not confuse users...

That is good idea; thanks.

In my case the .NES.INI contains only optional information; the game should work without it. The information in the INI might be used to display a title, make a simpler cartridge (or at least give you more options) than what is specified in the iNES header, automatically select the correct input device, dynamically optimize the emulator, add enhancements for the game, use for querying in a database, etc. It is not a requirement. (My own released .NES files include a .NES.INI too, and no programs currently read them; that is perfectly fine. My format is meant to be completely optional on both ends (I am not trying to force anyone to do anything).)

I also suppose that you could use XSLT and XQuery to convert the database entries into an individual XML file for each game if you want to, and if you find such things useful; I also suppose that it would be possible to convert some of the data in such a file to/from .NES format and .NES.INI format, since much of it is the information in the NES 2.0 header while some (such as the <feature name="peripheral"> command) corresponds to things in the .NES.INI format.

Note that although I like to have the PRG and CHR ROM and header and stuff in one iNES file, I would rather have the disk images in a separate file for each side (I have decided to give these files the .QDI extension so you don't confuse them with .FDS files that have all disk sides in one file), so that you can write-protect them individually, make copies of the disks, use FDS software that can write files on other disks (maybe no commercial FDS games do this, but homebrew software might), etc.

I don't know about Lynx and A7800, but OK I understand you.

If it were just the magic which had to be correct you'd have a point. But that's why there are 3 things which need to match. The uint32_t magic value, the uint32_t size value AND the uint32_t checksum. That's a total number of values = 2^96, which makes it have 1/79,228,162,514,300,000,000,000,000,000 odds of happening by random chance.

Many standards have used this approach before with great success. And these aren't minor standards they are things like:

* Multiboot - What GRUB uses to load Linux and other OSes
* ACPI - What the OS uses to identify the power management features of your OS are[/list]

There are actually tons of systems which use this "taged packet with a checksum" approach. While it is not literally impossible for there to be a collision, it is quite close to it. Your odds of getting a CRC32 collision on the ROM's PRG/CHR themselves in a database lookup are FAR greater.

Two tags each with valid checksums? That'd be pretty tricky to make work, might even be like trying to create a file that gives a certain hash.

Plus we're talking hundred-K files. Premature optimization here.


Excellent point. Existing ROM sets have the problem more or less solved, through databases and headers. New ones can't rely on databases, and iNES handling isn't fully consistent or thorough. New ones can have this in their ROM data.

The size value will mostly be a small value with its upper and lower bits clear. The values being matched are also not distributed evenly, with things like 0x00 and 0xFF being more common than others. A while back I did an analysis of NES ROMs and found certain three- and four-byte sequences that were drastically less likely than others. I used this in choosing a three-byte magic value for the bootloader format.

Yea, the two tags thing, the odds of it happening by random are astronomical. The only way I could see it happening is if tools fail to check for an existing one before applying a new one. Which like i said, the spec could just say "Use the first". The second one would do no harm.

I Agreed about the alignment requirement. While nice in concept when thinking in terms of efficiency, it is simply not needed and only really serves to be something which be done wrong. I would be happy to ditch that part .



I hadn't thought about that, and I like that perspective. Having it as part of all new ROMs would be completely harmless since they won't be in any DBs anyway. Only serves as extra information for any emulators which choose to implement this kind of standard. After that addition of old ROMs could happen at any pace depending on adoption.

Though I do feel that the database lookup issue is not as bad as it sounds. Ever emulator I've ever seen will at least fall back on built in headers if the ROM isn't in the DB. And that will work for almost all games.



Fair enough, the odds aren't quire 1/(2^96) due to non-linear distributions. Picking a good magic value is certainly important. Additionally to improve unqueness, there are several equally viable options.

* make the magic value bigger.
* have a magic footer value too at the end of the structure.

Either one of these would be very effective in ensuring no collisions. And like I mentioned earlier, the odds of a CRC32 collision on PRG/CHR hashes is already FAR more likely and that doesn't seem to be an issue.

I think the only concern would be utilities, but a utility adding a second one without checking for the first would be fine since the checksum for the first would almost surely be wrong afterwards. Thus there would only be one valid tag in the file. Even a malicious utility would have to somehow have both checksums correct, which like I mentioned is very tricky, since modifying one changes the correct value for the other.

That still doesn't prevent the "fake header on purpose" issue.

Why have a checksum in the file itself?

I understand having checksums in a database so you know whether you've found the "good" ROM, but you can't do this with a hash is stored in the file itself. It's just as easy to build the hash for a bad ROM dump as a good one.

The only result I can think of is that homebrewers and romhackers would be annoyed by having to regenerate the hash every time they change their file. It's just redundant information. What's the point?

To address both of these, I think there has been a misunderstanding in what the checksum in my proposal would be fore. The checksum is not for the file in its entirety, or to verify ROM integrity. It is a checksum of the tag structure itself to avoid random data being mistaken for a valid tag. The actual content of the tag would be pretty much equivalent to all of the things which we would want in iNES 2.0, but better organized.

Whether or not that includes PRG/CHR checksums is a valid debate to be had, but not what I was proposing.



I don't think fake headers are a valid concern. What prevents someone from putting invalid info in the iNES header format to make the game unplayable except in emulators which use a database lookup? Should we disregard that iNES format because it could be abused like that as well?

Oh, I understand now. You want to place the "header" at some random location in the middle of the file and linearly search for it? Thus the checksum ensures that you didn't find the wrong thing by mistake. This is really weird to me; it just seems like to a solution to a problem that you created for yourself.

Why not just use a chunky structure like RIFF where unknown chunk types can be safely skipped because the size is part of the chunk header? (Or a table of contents, or XML, etc.) There are a lot of clean solutions to the skippable-data problem. Trying to append new data onto the end of existing stuff inherently creates its own set of problems, why not just start fresh with something properly extensible?

Or perhaps propose extensions to NES 2.0 to allow arbitrary metadata to be added?

Because UNIF had plenty of problems.

The reason why is because it is a particularly clean way to add new meta-data to iNES files in a way which is both extensible and backwards compatible. While it may seem weird to you, it is actually a fairly common technique. As mentioned in my previous posts, things like the multiboot standard (for OS kernels), ACPI for hardware power management, I think ID3v2 uses it (but it typically at the file start), and many others use it.

Since the goal would be to have minimal impact on emulators which use iNES headers. I could have just said "it must be at the very end of the file", which is also a viable option and would probably be the case for most implementations. But imposing such a limitation now means that I have to do one of the 2 options:

1. mandate the size of the structure so people know the read the last N bytes of the file. This makes it less extensible. By not doing this, version == size, and we can just add new fields to the structure as the need arises.

2. have implementations try to read a block of size N1, then N2, then N3 from the end of the file if there are 3 revisions.

By having it allowed to be "anywhere" in the file and the standard specifies a trivial search and verify algorithm. The structure can be versioned safely without breaking previous version support.



This has been tried, it's called UNIF and I'm actually the maintainer of that standard. Unfortunately, while I think it would have been pretty fantastic if it became widely adopted, it has enough problems that it is not considered worth it to convert people's collections to a new wildly different format. My current proposal is trivial to implement, backwards compatible, and can be applied to existing ROMs without having to worry very much about emulator support. You can always fall back on the legacy iNES tags if you don't support it.



iNES 2.0 (and even the current state of iNES 1.0) is very much a messy standard. It is kludgy, requires bit twiddling to get correct values and is limited in size (16 bytes total, that's all you get, and that's not enough IMO). You may have noticed that almost 100% of byu's well thought out critique involves the fact that extra non-header data placed in the file and the concerns it raises about compatibility with emulators which use databases of hashes and ignore iNES headers anyway.

So if we are going to add arbitrary meta-data to the file itself. We may was well do it right have a proper versioned structure with has enough space to represent the data in an obvious and easy to use format.

After looking it up, the Multiboot Standard appears to use it because it does not have an alternative. I'm not sure what to look up for ACPI, I'm not familiar with it. ID3 tags (v1 and 2) as far as I know only appear at specific locations in the file and do not require a linear search or a checksum?

I'm very familiar with techniques of placing extra data at the end of a file, but these don't require a linear search, or a checksum. In theory, data appended to the end of an iNES should not cause a problem, though I'm sure there are emulators/cases that will do stupid things (as has been mentioned). A 4-byte number telling you how much appended data there is and X bytes of "magic" at the end of a file should be sufficient to identify the extra data's presence and location in the file. In this case, since the location is already known, would there there any reason for a checksum? In my view it would only make it harder to build files, but provide no additional security.

I also think the linear search may not be germane to an efficient implementation on certain systems (e.g. flash cart). It's very easy where you have enough temporary space to hold the entire file, but if you don't this might require another pass through the file, drastically increasing load times. (Depending on whether random-access is available too, even a tag indicator on the end of the file could be a problem.)



It seems weird to reject iNES 2 for the problems it inherits from iNES 1 and at the same time wanting to reuse iNES 1 for your format though. Given how strong a candidate iNES 2 seems right now... do you not think there are any worthwhile ways it could be improved without throwing it out?

A lot of the unfortunate iNES 1 legacy is easily washed away by simply not using the legacy features and getting the header correct. We've been making a lot of good progress on mapper definitions on the wiki as well, so we're in better shape than ever toward having a coherent central database of mappers. With iNES 2.0 there is still unallocated space in the 16-byte header; there could easily be a bit for "extra data after end of ROM data", or whatever we need here.

What are its practical deficiencies, though? I think there are things like the Jaleco baseball games, where there is a sample ROM that should probably be included in the .NES file (though so far only WAV rips exist, so we're not even to the point where we have the data to include, yet). What other kinds of extra data can you think of? (I know we can't think of everything, obviously...)

Also, what kinds of extra data would you propose to include that can be ignored (i.e. fall back to iNES 1.0)? There are things like Game Genie cheats (I think was mentioned earlier in this thread) that could easily be included as chunky metadata, but aren't essential to emulation. I dunno how opposed people are to that sort of thing; possibly they would be seen as just another way to make a mess of the files...

Though, another thing that might actually be a fun use of a linear-search method for tagging in a homebrew is to actually embed it in the PRG or CHR data directly. That would keep it out of any place it could do harm to an old emulator, but still host all the fancy new metadata.

proxy: True, NES 2.0 requires a few bit operations to parse. But they're no more complicated than the bit manipulations needed to emulate an NES in the first place, and they're only done once at load time. And encouraging emulator authors to allow metadata embedded in the ROM to override the header would just make it easier for certain developers to include a false header as a speed bump to emulating dumped carts. Given what boards exist as of 2013, I'd be inclined to prefer sticking with NES 2.0.

When GBA homebrew was starting out, the GNU assembler had no ".incbin" directive, plus I wanted a way to tweak art and audio assets without having to relink the whole program. So I used linear search for my GBFS library. A game using GBFS would append one or more archives to a binary and linearly search cart address space for the archive signature at 256-byte intervals.

Are you suggesting having the magic value as as the very last value in the file? If present, the preceding value is the size? I suppose that would work but doesn't strike me as particularly simpler.



This is a valid point. But I think the idea of having a trailing magic value also would tend to be implemented in multiple passes of the file since I would imagine some systems don't have a trivial API for getting a file's size. At the very least it would involve a seek to the end - sizeof(uint32_t), then a seek to the end - (sizeof(uint32_t) * 2), then a seek to where the meta data starts. Doesn't strike me as much better .



I think you misunderstand. My proposed system is designed to replace iNES, but remain backwards compatible. The iNES header (version 1.x or 2.x) would be present and as complete as possible. But if the emulator supports the tag system I propose, it would ignore the iNES header and get all of its information from the tag instead. There would be some redundancy of information of course. But it would also maintain a high degree of compatibility.



I am precisely suggesting not using the legacy features of iNES. The tag system I suggest would replace it entirely with a more extensible and more forward thinking design.



The iNES header is mostly very functional, but it can be a pain in the ass:

* Having to do complex operations to get trivial values like "PRG size" or "Mapper Number".

* Some ROMs have PRG/CHR sizes which don't fit into the multiple of 16384 or multiple 8192 categories. They are rare, but we could easily dispense with such restrictions.

* Just look at Byte 10 (RAM size) of iNES 2.0. It's sufficient, for now, but complex, involves a lookup table because lots of information was stuffed into 4 bits. What if some oddball cart is discovered that doesn't fit the mold?

* Even with spare bits, very limited space. While not necessary (and I'm sure some would disagree), it would be very nice if there were a UTF-8 field containing the original name of the ROM. Or perhaps Region/Country codes. Or Perhaps manufacturer.

More than anything, the idea is to provide a technique to say, "if the NES ROM file format were being developed today, based on what we know now. What would it look like?"

Like I said, I have no expectation that my idea would be adopted by everyone, but it would be an opportunity to "remake" the ROM meta data is a more clear and simple way. So it's worth putting it out there, and fueling a discussion.

That is interesting and would certainly be considered fair game . I Like it!

If depends on the particular system, but I've never seen the unknown-file-size limitation except when dealing with strict C code. Without exception, I can't think of any file system I've worked with (embedded included) that didn't have an API for getting a file's size. The method of fseek+ftell is usually only resorted to when trying to write cross-platform C code within the limitations of the stdio.h API. At least, that's been my experience with that method.

Going to the end of the file to read the magic/etc. would indeed involve seeking though, and this is where the random access requirement is pretty important. If you don't have enough RAM to hold the whole file (probably not terribly convenient on the NES), hopefully at least you can jump around in the file without undue delay. This is probably true of CF cards, but I don't know for sure. I suspect in cases like the PowerPak a couple of seeks would still be a lot faster than having to read the whole file to find a tag.


As for stuffing metadata into PRG/CHR, I actually often stick text messages near the end of my ROMs for anybody who wants to poke around in there with a hex editor, so this kinda thing might actually give a regular user a chance to find it with their emulator.

I do agree that iNES 1.0 parsing is somewhat onerous, especially in the cases where iNES 1.0 underspecifies and has to be resolved due to known things about the existing ROMs. I think iNES 2.0 is trying to alleviate those problems of underspecification, the new fields for larger PRG/CHR, for example. It still has all the bit-packing (and more), though, which you seem to dislike.

I don't find the bit packing that much of a pain, personally. Decoding each value more or less becomes one line of code with a shift and an and. Also bit-packing is something you can easily do by hand or by an assembler if you're writing homebrew. This is not true of CRCs, which generally can't be done directly by an assembler or by hand, which is the immediate objection in the back of my mind when I see it proposed as part of the file format. Putting a CRC into a file requires a specialized tool for generating it.

I agree that the organization of all the bits and stuff in an iNES 1 or 2 is more or less abitrary/random, and if iNES 2.0 wasn't trying to maintain some backward compatibility it would certainly be packed a lot differently, but I feel this legacy packing organization is reasonable enough to accept in the name of backward compatibility.

There's a difference between messing with a header that was added just for emulators and messing with data that was part of the original ROM itself (rather than metadata). Yes, you can modify the ROM, but that doesn't sound good for preservation and such (it wouldn't be a good dump anymore).

Which systems might those be? POSIX systems have stat(), and Windows has _wstat(). Besides, NES ROMs are small enough that you could just read the entire ROM file into RAM. Are you designing a format for emulators, a format for the successor to the PowerPak, or a single format for both?


The emulator has to do complex operations anyway to get trivial values like "current VRAM address given values written to $2005". I agree with rainwarrior that the bit packing is no worse than what you already have to deal with when emulating an NES.


RAM sizes are a bit shift expression, not a lookup table. Try this:



From the spec as amended: "Sizes that are not a power of two, such as the 5120 byte battery-backed RAM of Taito's X1-017 (mapper 82), are rounded up."


It'd look like a PKZIP file containing a PRG ROM file, a CHR ROM file, and XML metadata. At least that's what the designer of WSZ (Winamp skin package), SMZIP (StepMania package), ODT (OpenOffice Writer document), DOCX (Microsoft Office 2007 Word document), JAR (Java archive), or APK (Android installer package) would have come up with. I'm aware that a zipped format wouldn't be practical on the successor to the PowerPak.

Thread author: please retitle this thread to something mentioning NES ROM headers, since that's what it's about. The current thread is very vague, only one step away from "Thread".

Now we're getting somewhere :D

I actually like the concept of separate PRG and CHR files .

I'm not sure I would go with XML as the meta-data format, but that's a minor detail. Though I'd probably go with a more trivial container format lie ar or something.

This is basically what I wanted to add to MESS in addition to the internal database, for homebrew
The problem would be to convince users to add such a xml to their zipfiles or rom sites / authors to spread these files instead of the old ones... As of late I got quite negative on the possibility to replace iNES1.0 by any means, so I basically lost most motivation to pursue this line of work

I'm pretty sure there are a few torrents with the NES set for MESS which give you a pretty complete set of split PRG/CHR to play with

And then, if you run linux, you can split the entire NES ROM set into separate files in probably a couple seconds with a 2 minute shell script write up and my NES ROM splitter.

If it ain't broken...don't fix it....we already fixed it, guys...chill down, this will never happen.

> I'm not sure I would go with XML as the meta-data format, but that's a minor detail.

Not really. It's not even just language (XML vs JSON vs YAML, someone even said INI here), there's also how you name every element inside of it. It's much more than a small detail. That has the potential to bikeshed for decades.

It's unlikely enough that an author is going to make ONE board markup file, let alone one for each NES emulator. So the only workaround there will be some kind of user-contributed database where anyone can make the board markup and contribute it for homebrew games. Won't get 100% coverage, but any popular emulator + popular homebrew should eventually work. And again, this is only an issue for homebrew that won't work with iNES 1.0 headers.

> Though I'd probably go with a more trivial container format lie ar or something.

I solved that one already, in a way that works for flash carts too.

Compressed archives are for distribution, because there's no way to download a folder all at once with ordinary browsers. But that's all they are is folders. For nearly any other media type on the planet, people decompress archives they download first. PNGs, MP4s, source code, etc. People don't expect those programs to load ZIP files containing their media. Browsers like Safari go so far as to auto-decompress ZIP archives for you. So when you load the game, just uncompress it to an actual folder. And now you can optionally store mutable data (such as SRAM) inside the folder, if the user wants that. Or store it somewhere else if they want.

If you're fundamentally opposed to a specific container format, then tell the user to uncompress the archive first. Or use a tool like the one I made (purify) to auto-extract game ZIP files to folders.

If people are really concerned about the extra disk space used (after all, the entire NES library eats up a whopping 300MB of disk space ... whereas we are lucky if the user has 1000000MB of disk space these days), then use something like NTFS' native folder compression.

You'll get a lot of resistance still because change == bad to most people, so it's up to you. Offer an option to not extract to disk if all you're concerned about is popularity. Use blargg's fex to support a half-dozen of the most popular compression formats and call it a day. What I like to do is abstract things. Write an API that takes either a folder or archive name, and then it gives you the same API to load files from it. So your emulator is written like it's working with a folder, even though it may actually be an archive.

> As of late I got quite negative on the possibility to replace iNES1.0 by any means

Well, you'll never replace it for everyone.

So again, what I've come up with is to allow iNES 1.0, but also offer your own method. For instance:

> I'm pretty sure there are a few torrents with the NES set for MESS which give you a pretty complete set of split PRG/CHR to play with

FitzRoy put together a torrent for what my emulator uses. Complete library with split PRG+CHR+manifest.

People who support the idea can have a whole library in said format. And people who don't can keep loading their zipped iNES files.

Unless you are keeping the ROMs in a pendrive or a SD card, as those are generally some variant of FAT (completely forget about compression in this case, or any even remotely interesting features, for that matter).

Yeah, in that case you'll have to eat up an entire 1.8% of the available storage space. Twice as much as when zipped.

It's definitely not a universal solution. Only the people who can afford to throw away an extra one-hundredth of their $10 pen drive space can afford this luxury.

You'd definitely be making your emulator for the top 1% only at that point. So it would be great to offer fex+decompress to RAM only as an option for everyone else. I only mentioned the decompress to folder idea as an option to counter the argument over which container format to use (ZIP vs tar vs 7z vs custom.)

I don't like separate PRG and CHR ROM files for games on cartridge, although for games on disk I would like each disk side to be a separate file (you could put them in a ZIP archive if you want to, I suppose, although I would also like to be able to work with them when not in a ZIP archive).

I also don't like having to put all the information in one database in order for the emulator to load a file (this makes it very difficult to distribute patches and homebrews), although if an emulator requires that, there could be a way to convert a NES 2.0 header and .NES.INI into such a database format. Other formats can also be converted in case someone uses XML for metadata or whatever, you can convert this too.

I don't like that MESS and MAME have these features (there are also other things I dislike about MAME, such as the license conditions).

My own recommendation is to distribute a .NES file (or a .FDS, or a collection of .QDI), using either iNES 1.0 or iNES 2.0 headers (there are some things which old iNES headers aren't sufficient for, so NES 2.0 is used), and then optionally provide a separate .NES.INI to go with it (if you prefer other formats (such as XML), you could provide those as long as it is possible to convert between them). If any emulators need their own format, a converter program can be made which will convert the formats into whatever is needed by those specific emulators.

Sure, you could use a ZIP archive containing the PRG ROM, CHR ROM, and XML metadata, if you prefer; it should be not too difficult to convert between that format and the format I prefer.

But I really think we should make the emulator that fully supports NES 2.0 headers (including high mapper numbers and submapper numbers), and a C plugin API for implementing mappers (and possibly input devices too); as it turns out, I have defined such an API.

There's no reason to replace the iNES format we have now. Separating PRG and CHR into separate files is pointless. With very little effort you know exactly where the PRG and CHR data is in the iNES file. iNES 1.0 was limited but it's since been expanded and I really don't see any specific example why anyone is going to go along with a new format. One file per game is enough. We don't need to separate everything or add some new header/xml, or whatever else.

UNIF never exactly caught on even though many said it was superior. Lets name the cases where iNES falls short typically.

Startropics/MMC6. Was that addressed in iNES 2.0?
Non-stand MMC1 Boards SUROM/SXROM. iNES 2.0 addresses that with PRGRAM size.
What else?

MMC3 revisions A and B. They have different working IRQs.

Yes; this is good, to use NES 2.0 headers for these purposes please, since NES 2.0 implements all the stuff we need. (External metadata files might still be useful for optional features, which don't needed to emulate the game, though, such as: title, author, input devices, licenses, etc.; and an emulator need not read this file, although it might be useful to support it to automatically select input device or the switches for the Vs.System, but this isn't necessary.) Even though it is good to have one file per game if it is one cartridge, I think each cartridge, and each side of each disk, should be in separate files. (The current format puts all disk sides in one file too, and I think that is a bad idea and makes the system very limited. You could support both the combined and separated disk image formats in an emulator, for maximum compatibility and usefulness.)

These kind of things might use NES 2.0 submapper numbers; you should really implement NES 2.0 and a plugin interface.

MMC6-style PRG RAM enable and TEROM/TFROM fixed mirroring were already defined as submappers. I've defined another submapper to mean MMC3A behavior. I've numbered it 4, and MMC3A + fixed mirroring is 6. Any objections?

> What else?

VRC board "protection" pinouts (Goemon, Contra, etc.)

just two small remarks: separate PRG/CHR is the most natural approach when you come from arcade emulation (chips are separate, not glued together, and their content is often mapped to different CPUs so that it does not make any sense to put pieces together). you might like or dislike it, but it is not strange at all.

also, you might want to notice that MAME/MESS transparently handle having zip content uncompressed in a folder since 1999 (maybe earlier too), so it's not like zipping files is mandatory in any emulator.



which parts? the fact you have to share back the changes? the fact you cannot sell the exe? the fact you cannot like to closed source dll?
MAME license it's plain BSD license + non-commercial clause, nothing particularly odd.



I'm all for supporting iNES2.0 headers. MESS supports them, for instance.

but plugins? please no. plugins tend to be confusing. unless you centralize all the plugin developing and handling in a single place, so that people know where to get the latest and supported plugins, typically you end up mixing outdated plugins with more up-to-date ones and things start to misbehave (this is based on my early experiences with psx and n64 emulation in the early 2000s).

Erm... the discussion was about replacing iNES 1.0 with something better (be it 2.0, a separate db, or appending metadata), and I was just stating the obvious (like you did in the reply): nobody will fully replace iNES1.0 because users (and website maintainers) are lazy

hence, we all have to live with supporting iNES1.0 and offer alternative to smart people (in MESS, we also have support for UNIF and iNES2.0 in addition to our split PRG+CHR...)

also, I'm not going to download the full FitzRoy's set just to check this, but do you have the correct manifests for the Nantettatte! Baseball add-on carts (which work more or less like the Sufami Turbo)?

viewtopic.php?p=111013#p111013



some pirate conversions of FDS games. there are a bunch with 2 PRG chips, one multiple of 8K mapped in 0x8000-0xffff and the second with 2K of data mapped separately inside the 0x6000-0x7fff area. have fun supporting this in iNES
it is quite plain to handle this in UNIF (and with xml), as long as the format is properly implemented (so that you don't assume 8K of PRG chunks)

The part that I cannot use parts of the programs in real FOSS programs (although some files are dual-licensed, so those can be used), is my concern. (If the entire program (except for ROM images) is dual-licensed under GPL3 (or AGPL3) then it would work, I suppose.)

Well, I mean mainly the plugins so that when writing a game you can easily add a mapper or input device for your own use; binary compatibility isn't necessary although it would be good for plugins to have source compatibility.

Yes.


To communities centered around software distributed under DFSG-conforming free software licenses, a non-commercial clause itself is "particularly odd."


The definition of mappers 0 and 3 was recently expanded to cover ROMs that cover $4800-$FFFF: NROM-368

> Done

I was referring to 1.0 limitations, as I presumed all of these issues have been solved in 2.0. Is that not the case?

> MAME license it's plain BSD license + non-commercial clause, nothing particularly odd.

Despite MAME being free software and open source, you have people who will claim that it is not Free Software nor Open Source. You can tell by the capitalization that you're dealing with an ideology rather than pragmatism. I've found as much success arguing with these types as religious fundamentalists.

> also, you might want to notice that MAME/MESS transparently handle having zip content uncompressed in a folder since 1999

Interesting, I never even knew it could do that. I require a DLL (included with my emulator) to decompress archives to a folder first, so that my core can use native filesystem APIs on each file, and can write to the folder if need be (flash ROM, SRAM, etc.) Definitely causes more drama, but it seems by making it transparent that nobody even realizes it exists :/

> just two small remarks: separate PRG/CHR is the most natural approach when you come from arcade emulation (chips are separate, not glued together, and their content is often mapped to different CPUs so that it does not make any sense to put pieces together). you might like or dislike it, but it is not strange at all.

This is just another matter where people are passionate about keeping things the same. Let's be honest, regardless of which side you are on, it's really not a big deal whether you load one file or two, and whether you skip 16 bytes at the top of a file or not.

That said, I completely agree. It's just too bad MAME and I still have one key disagreeance.

I prefer to split files based on their actual mappings and usage in practice. The same exact game could have two 16mbit ROM chips on the first printing, and one 32mbit ROM chip on the second printing. It's an interesting note in a database somewhere, but it's useless information to an emulator, unless you want to simulate the 74LS switch that selects each chip, which nobody ever does.

MAME tends to prefer grouping files by ICs, even when this is contradictory to how the data is used by CPUs. Case in point being the SNES DSPs. They have both program and data ROMs inside of them, yet they exist on separate buses inside of the CPU (strict Harvard architecture.)

I know MAME supports alternate loading methods and that's great. It's just a shame we can't agree on how to split files. This is going to be instant anti-splitting fodder. It's part of the reason I came up with that SNES ROM + DSP ROM merged file format. Which is in itself truly ironic: all of the traditionally pro-merged file people (iNES and SNES copier header folks) absolutely abhor it and want the DSP firmware in separate files. Like I said, it's all about keeping things the same to them. There's no logic or consistency, nor agreeance with how things were done in the first place, it's strictly a matter of backward-compatibility.

> also, I'm not going to download the full FitzRoy's set just to check this, but do you have the correct manifests for the Nantettatte! Baseball add-on carts (which work more or less like the Sufami Turbo)?

My NES emulation is primitive and doesn't support wild stuff like that yet.

But for Sufami Turbo, I support that fully. Both the base Sufami Turbo cart, and each slotted sub-cart, get their own folders (or archives) that contain the split files and manifests describing the boards. If you load the Sufami Turbo SFC cart, it asks you to load zero to two ST carts. For BS, the manifests describe writability, size, flash type, etc. For ST, the manifests describe ROM, RAM and linking capabilities (it won't ask you to load a second cart if the first isn't linkable anyway.)

> The part that I cannot use parts of the programs in real FOSS programs

The problem is the GPL, which restricts you from using non-commercial code with it. That's not MAME's fault. BSD is a real FOSS license, and there's no problem with mixing BSD and MAME code. Copyright and copyleft are two sides of the same coin, it's just a matter of who the restrictions are being placed on: the developers or the users.

Regardless, it's much too late for MAME/MESS or Snes9X to ever allow commercial use. Too many long-gone developers have contributed code to it.

It is a matter of definition, not of ideology or pragmatism. Some parts of MAME is dual licensed so that it is FOSS, but most of it isn't. MAME is neither Free Software nor Open Source.

Yes, BSD is a real FOSS license. OK, it is too late. However, at least some files are dual-licensed those files can still be used in other programs. There may be no problem putting BSD code into MAME, but I mean putting MAME code into BSD-licensed programs; the program isn't FOSS if this is done, which means that it isn't a real BSD license.

This is precisely what I meant. I have no intention to sell the EXE and I don't think many people do, but yes you have the point; a non-commercial clause itself is "particularly odd"; I agree.

Might not $4020-$FFFF be a better definition, since the iNES PRG ROM is a multiple of 16K? It would require extra decoding logic, but if a game is written to use NROM-368 then you don't have to use all of it and you can use $4800-$FFFF and avoid the extra decoding logic, but if someone wants to use more than that they can add the extra decoding logic onto their cartridge.

Would it be more honest to say "The license of MAME and MESS does not conform to the Debian Free Software Guidelines as published by the Debian project, the Open Source Definition as published by Open Source Initiative, or the Free Software Definition as published by the Free Software Foundation"?


Or you could wrap file system APIs on all platforms, as Allegro does (packfiles), SDL does (RWops), and Python does (objects implementing the "file-like" concept), and use the wrapper in your core.


I believe this is common among copyleft licenses.

See, etabeta? There's the capitalization I spoke of.



Correct, if a BSD and MAME library were combined, the new combined license would be non-commercial still. All licensing restrictions are viral by nature.



Yes, absolutely. A million times this.

My whole beef is that people take words with well established and popular meanings, like "free" and "open", and say oh yeah, it's not free or open. You point out that it costs no money (hence it's free of charge) and that the source is available (hence it is open to view and to modify), and then you get the capitalization people coming in. Oh but it's not Free or Open.

You know who else capitalizes words that are traditionally not capitalized? Religious people. "And He said His will would be done."

Free and open are taken. The OSI (who are we kidding? Bruce Perens) tried to get a trademark on "Open Source", and the court rejected it, and rightly so.

You also don't have to be as verbose. Just say, "MESS is not DFSG, FSF or OSI approved." Okay, great. Maybe some people care about what those groups like, maybe some don't. I don't care, personally. I have my own opinions. But MAME is free software and it is open source and it is non-commercial.

Let me give you an example. Imagine if I tried to claim that any license which required reproducing the license in binary form (second of the two-clause BSD license) was not Good Software. "Sorry tepples, your application isn't Good Software." It'd be annoying, wouldn't it? A casual person doesn't care about what some loaded capitalized term means. That is the very definition of ideology. They filter out the capital letters entirely, and read it as "your application isn't good software", aka "your application is bad."

I was quoting you.

Yes, I think you are definitely correct; this makes it far more clear what is being meant.

That is partially due to the English language having "free" meaning different things. It isn't my fault. It isn't your fault. So I suppose it is possible both sides are correct but meaning different things. (This also reminds me of something else: If an astrologer says Sun in Cancer, and an astronomer says Sun in Gemini, they are both correct, but they mean different things by the terms "Cancer" and "Gemini".)

Yes, it is true, but in this particularly example, I think it is actually the grammar of English language that pronouns such as "He" are capitalized if it refers to God (I don't really think "He" is the best word, but there aren't better words; sometimes "He/She/It" is used, but I don't know if that is any better; I myself just don't use the pronoun because my own religious points of view seem to work better without it; I do capitalize it when I do use it though, since that is English grammar to do so).

Yes, perhaps "MESS is not DFSG, FSF or OSI approved" is better. It is OK to have your own opinions. However, see above about multiple meanings of words. I say MAME is not free software and it is not open source, which is also correct (sometimes the terms are capitalized to attempt to make it clear that one meaning is meant and not the other one, but not always).

Yeah, I've read that different astrologers use different "time zones" caused by precession of Earth's orbit. But the one I talk to says the Sun is in Cancer every day of the year, and that's why I should cover my skin to keep the UV off.


Not all religious groups consistently use "He" for God. Some religious groups stress the use of modern language and use "he" unless there's potential for confusion, such as God and a man in the same reference window.

Not quite what I meant. Yes there are different reference dates used due to precession (for this example, I am using tropical zodiac, which means the reference date is equal to the date for which you are giving the positions, and the ayanamsha (the Sanskrit word for the amount of precession) is always zero). The term "Cancer" here refers to the astrological sign, not the constellation; and an astrological sign is simply 30 degrees of the ecliptic (Cancer is the fourth sign so its range is 90 to 120 degrees). This is the same "Cancer" as is in the "Tropic of Cancer" on the world map. However, it is also correct that the Sun is in Gemini, but that is if by "Gemini" you mean the constellation and not the astrological sign. There are 88 constellations in total. The article you linked to is making the same mistake; they claim astrologers don't get the position of the Sun correct, but actually they do, they are just using terminology that isn't familiar to most astronomers. Of course this doesn't mean astrology works (I don't believe in astrology either); it only means that the terminology is different. (And note that the two tropics on the world map correspond to astrological signs, not to constellations. A lot of people make these mistakes, and needs to learn that the words can have different meaning in different contexts.)

Maybe their view of "God" makes it sensible. I don't find it sensible to use "he" for God with my own religious points of view. I believe in a non-personal God, although I am also panentheist and ignostic, and believe in some of the concepts of deism.

Of course all of this is getting pretty much off topic.

The English syntax that is appropriate is the capitalization of proper nouns. If you are referring to a god named "God" by name, you would capitalize that name. Capitalizing pronouns like "he" and "his" is not normally correct English syntax, though perhaps "He" is simply another name for your god, in which case it may also be a proper noun? If you don't want to stretch that far, maybe just consider it an improper syntax, but one that religious texts may find useful as a form of highlight or disambiguation. (I.e. a specialized syntax for a specialist field.)

Apart from mentioning the existence of evil deity in the TV series The Powerpuff Girls who goes by the proper name Him (rhymes with Kim or Tim), I'm done with the offtopic discussion. So let me try to rerail the discussion by summarizing what we know so far.

The core problem:

• A game contains a circuit board, ICs, and ROMs.
• The behavior of a board and ICs can be described in plain English, but the behavior of ROMs pretty much depends on their data.
• The behavior of a board can change if different ICs are on it (e.g. normal vs. crazy UNROM), and an IC can produce different behaviors in different boards (e.g. MMC3 on TLSROM vs. TLROM).
• The law has greatly different treatment of board and IC descriptions vs. ROM data.
• There are several different containers for ROM data. These include iNES, PNG-style chunks, Tar, Zip, 7-Zip, and a folder.
• There are several different ways to express a "manifest", or description of the board's behavior: iNES, NES 2.0 (an extension to iNES), XML/JSON/INI/YAML/BML with a schema to be determined, or any of several formats included in a database distributed with an emulator.
• Occasionally a game is made of two cartridges that work together.

This leads to several approaches:

• iNES uses the iNES container with an iNES manifest in the header.
• NES 2.0 uses the iNES container with extended ROM sizes, with an NES 2.0 manifest.
• Occasionally, iNES containers of multiple region versions, revisions, and hacks of a single game are nested in a 7-Zip container to take advantage of 7-Zip's solid compression.
• UNIF used PNG-style chunks with a manifest spread across multiple chunks.
• Pasofami uses a folder with a proprietary manifest.
• A lot of emulators include a database to correct wrong or ambiguous iNES manifests.
• The original ZapFC proposal, intended only for pre-1997 games, used Zip with such a database.
• The revised ZapFC proposal includes a manifest, as does the MESS format and byuu's proposal.
• Super NES and Game Boy games include an internal manifest at a particular address in ROM, in the actual cartridge.
• This appended chunk proposal uses the iNES container with a manifest wrapped in a sync-size-checksum chunk structure similar to PNG and UNIF. It has been suggested that NES emulators also recognize such a manifest in an otherwise unused part of PRG ROM or CHR ROM so that even dumps from a cartridge will include the correct manifest.

Each approach has some limitations:

• Some manifest formats, such as iNES and UNIF, ended up leaving a lot of things ambiguous.
• Some manifest formats are poorly documented.
• Compressed containers work well on emulators and post-GBA handhelds but are prohibitive to parse on flash equipment for the NES.
• The same is true of the more human-readable manifest formats.
• Parsing some manifest formats, such as iNES, requires bit manipulations of roughly the same complexity as the $2005->t logic.
• Some collections of ROM images are distributed with incorrect manifests.
• Separately distributed manifests are inconvenient for people who release new homebrew programs, new Chinese dumps, or ROM hacks.
• Getting people to adopt a new container or manifest for existing collections of ROM dumps is difficult.
• The same is true of emulator developers.
• Developers of new commercial NES games trying to sabotage emulation could include an intentionally misleading internal manifest in the ROM data on a cartridge.
• Users don't want to have to resolve conflicting manifests (internal vs. inner container vs. outer container vs. database) when loading a game.
• Emulator user interfaces occasionally need to consider nested containers with multiple games.
• No container to date includes the ROM of sample playback chips on certain Jaleco games.

> Super NES and Game Boy games include an internal manifest at a particular address in ROM.

Yes, but we have ambiguities here as well. Just not nearly as bad as NES.

For SNES, there are RAM mapping problems with Ys 3 vs Fire Emblem: Thracia 776; ST010 and ST011 use the same special chip ID, etc.

For GB, there's the MMM01 mapper. The most common merged ROM ordering puts the MMM01 boot ROM at the bottom, which means the correct mapper ID is in a different location.

GBA and NDS lack save RAM method (SRAM, EEPROM, Flash) and vendor IDs (that change protocols for Flash.)

Mega Drive lacks SVP, lock-on, EEPROM, and bootleg markup.

Master System doesn't specify mappers.

The pattern is that nearly all systems have either no information, or incomplete information, embedded inside the games themselves. The dream goal would be to have a consistent way, across all platforms, to describe the boards. This will of course never happen, but it'd sure be nice.

> No container to date includes the ROM of sample playback chips on certain Jaleco games.

I have some precedent for that, in that the MSU1 chip accepts a list of PCM recordings.

Has anyone managed to decap and dump those sample ROMs yet?

Incase you were unaware, the reason this is done is because MAME also serves to help preserve actual arcade boards. So if you have a bad ROM chip, you can get the data from the MAME rom set and repair your board. It's not *just* about emulating the games.

OK, I have read the list of what is made so far. Here is a proposal to possibly improve the situation:

• Support iNES and NES 2.0 headers.
• Optionally support UNIF, and you can also optionally include whatever additional formats you find useful.
• Optionally include a database of existing games with the emulator, external to the executable. The format doesn't matter; use whatever is best with the emulator.
• If the header is the old iNES format, then look up the checksums in the database to correct the header.
• If the header is NES 2.0 format, then ignore the database and use the actual contents of the header.
• If you decide to implement .NES.INI, then you may optionally use the Bootgod number to look up in the database if it is the old iNES header. You might also use the Mapper block to disambiguate ambiguous iNES headers, although use of NES 2.0 for this purpose is recommended (the Mapper block is mainly provided in case the cartridge is less sophisticated than the NES 2.0 header implies, or if it is a homebrew designed to work on more than one kind of cartridge).
• Information about input devices, overscan, etc goes in the INI although if it isn't present (or if the emulator doesn't support .NES.INI), the emulator's database can be used.
• If there is more than one cartridge, use a separate file for each cartridge. An emulator might optionally support ZIP archives of multiple cartridges if they are used together.
• The same should be done for disks; each side of each disk should be a separate .QDI file. However, an emulator should also support .FDS (which puts all the disks into one file), or optionally a ZIP archive of multiple .QDI files. (The disk numbers and side numbers are part of the headers, so if you have a bunch of separate .QDI you should easily be able to put them together properly into a .FDS or whatever. Also, supporting .FDS is still recommended since games are generally distributed in this format.)
• I think I have read that some emulators, when loading a .FDS, will create the save files in the user's home directory. It can be extended: Upon loading a .FDS for the first time, create a directory and a manifest file and unpack the .FDS into multiple .QDI files into the directory. One thing contained in this manifest file is then macros to select a disk side.
• ADPCM ROMs could be stored in a separate file to the cartridge (unless it is UNIF, in which case you might add a block for that purpose). (You may combine them into a single ZIP archive if you want to.) The emulator's database can be used to select them, the matching filename of the main ROM image can be used to select them, or the .NES.INI can be used to select them.

I was unaware that is was what was intended, but I did think of that, and yes it can help for that purpose too.

Pretty sure that if it was legal MAME would include the ROMs directly in the source code rather than require users to provide them separately.

I've covered this extensively elsewhere, but I guess I may as well do so here as well.

Let's first consider how many people who play games in MAME actually own the arcade machines. Let's be very generous, and say one in a thousand will actually own the game. Now how many of those people do you think can actually repair those arcade machines? We'll say about one in fifty. Not as extreme here, since the people buying would be more inclined to have the skills to fix them up. Now of those instances, what are the odds that a machine breaks because a ROM chip went bad? How often do you hear of things dying because the ROM chip went bad? It's almost always capacitors, or crystals, or power surges, or voltage regulators, or ... pretty much anything but the ROM chips. So we'll say one in a hundred here. Again, extremely generous.

So stack those odds, and we have about a one in five million chance that a MAME user will have their real arcade machine break, they'll be able to diagnose that the ROM chip went bad, and they'll be able to obtain a new chip to flash and install on the board.

So for a 1:5,000,000 chance, MAME has made their format a pain in the ass for anyone else to support*. Now consider this mythical rich savant that can repair arcade board ROMs. Why can't that person just look in a MAME database, find the entry for their game, and see how the files corresponded to physical chips, and then use a split-file command like dd to make their own burnable images? Do you really think there are people out there that can program and replace ROM chips, yet can't split a file?

* take the case of the Nintendo Super System. The SNES arcade machine. MAME/MESS made a real, well, mess, of that. Say you want to play their nss-actr.zip (Actraiser) game. This is what is inside the archive:



Now say you're anyone but MAME/MESS, and you want to play that. How do you load in the game? Well obviously, you concatenate act-rais.ic3, followed by act-rais.ic2, and ignore act-rais.ic8, duh. /s

As for why the DSP-1 program ROM is there? Well, MAME/MESS required that because they couldn't be bothered to fix their emulation to not require it for non-DSP games. As for spc700.rom? That's because the MAME developers have never heard of Lexmark v Static Control Components. That, and this ROM exists inside the NSS machine (and inside of a sound CPU at that, it's impossible for anyone to replace this ROM), not on the cartridge. Your guess is as good as mine as to what the other files are for. You have .bin, .rom, .ic#, .dat, .02, etc. No consistency. No logic.

So okay, you have to load .ic3+.ic2 as the program ROM. So we do that for all games right? Well, not nss_smw. There, you load nss-mw-0_prg.ic1, obviously.

This is stupid. To make a one in five million chance event slightly more convenient, they make the format a pain for any other emulator to load and support.

In my system, you'd have:

manifest.bml [description of the board, and where each ROM maps to in memory]
program.rom [the game itself]
nss.rom [the NSS boot / menu ROM]

And it would be the same for every NSS game. Had they done this, the games would directly load in all SNES emulators, since they always pick the largest file inside archives to play. As it stands, these games only work in MAME.

the problem is that many former MAME developers (and some of the current ones) work into programming companies and have strict rules in their contracts about usage of their code, even the parts written outside work hours.
this has always made necessary to have a non-commercial rule

also, allowing commercial use makes a bit more legit all those people selling emu packs on e-bay: while in practice it makes no big difference if they are only breaking copyright of the games ROMs they sell or both game copyright *and* the software license, from a moral point of view I see a big difference

that said, MAME code has been legally used in some commercial retro-packs (like Taito Legends): the involved devs were just contacted and their code was dual licensed for the purpose...



I was referring to the fact that 2KB chunks of PRG are not supported by iNES headers, to my knowledge, so I'm not sure how the rom has to be loaded from an iNES file



MAME is compiled with zlib in the source. this allows both to fastly access the checksum info in the zip header and to basically treat any zipfile as it was a folder. hence, storing the roms into pacman.zip or into a pacman/ folder is the same.
the price you pay is that the exe size grows a bit, but on a 60MB exe it's not ~100KB to make the difference...

premise: I'm not trying to convince byuu (I respect people having different opinions) but I'd like to clarify a few bits because they are the base reason why in MESS we pushed separate PRG/CHR files as the main format for our xml software list and why I still believe gluing together the content of all different chips is a bad idea for emulation



the problem is that you are too focused on consoles, so you're too used to have a specific extension identifying where a file goes.
for MAME, extensions are of no usage for end-users, they are only there to help people repairing their boards because they tell repairers where the file goes on the pcb.

for other usages, e.g. for an emu dev, you have to never separate the ROMs from the "mapping manifest" (as would probably be called in byuu-ese ) which is built into the source itself (in nss.c, see the ROM_LOAD bits for the various games). that way you can see where the various .ic* have to be loaded/mapped
some devs have suggested these manifests might be made external to the exe, like the xml softlists, but there have been resistances so it's still built into the exe

personally, I don't think end users would care that much if all the emulators tomorrow would require console ROMs to be split files inside a zip archive like MAME does, or viceversa if arcade ROMs would be glued files with .arc extension, as long as emulators would support the files and there is some automated procedure to convert the files they have in the new format.
and I seriously doubt that there are users who sits staring at their files (single files or archives of separate roms) and taking pleasure from the sight.

so it mostly boils down to what is your purpose. I personally feel important to make people aware that console carts can be internally as complex as some arcade boards, and separate files help to convey this fact. but I see no big deal if other devs feel differently



it's not there anymore in 0.149, just wait for rom sites to update
it's not anymore in the snes bios for MESS either. it is only required for SNES games having a DSP-1, as people would expect, and it can be either in the game zipfile, as a separate bin, or appended to the game .sfc as in higan

similarly, if we ever get ADPCM dumps from Jaleco NES games, these dumps would go as a separate file into the zip archive containing prg and chr...




erm... you are quite off on this subject. spc700.rom is usually stored into nss.zip (we call it BIOS archive), not in the game zipfiles, making it quite clear that it's not into the cart.
as above you seem to assume that everything in the zipfile has to come from the cart, but maybe you are just ignoring the "mapping manifest" in nss.c... the spc700.rom is loaded into a "sound_ipl" rom region which is only accessed by the sound CPU, while the cart data is mapped to a different rom region which is accessed by the S-CPU. and the instruction and security data from the game zip archives are mapped to other rom regions too which are only accessed by the corresponding components which handle the instruction/security...

the problem seems to boil down to the fact that you cannot auto-mount the files into the emulator based on extension and you claim it's due to MAME inconsistencies. but the real hardware had chips in different locations for the different carts, so if anyone should be blamed for inconsistencies is Nintendo itself when they built the original NSS carts.
of course real hardware had wiring prescribing what goes where from cart to the NSS CPUs (and in this aspect NSS hardware is a lot closer to NES carts than to SNES carts, where ROM mappings had just minor variations and hence generic LoROM/HiROM covered 99% of variations).
in emulation MAME relies on the "manifest" built into the source (the rom mapping code in ROM_LOAD) to do the same, and other emulators have to cope with it somehow, be it relying on an external bml manifest prescribing where the things has to be mapped or on an artificial rom format with a iNES-like header...

And it is one thing I hate about MAME... I don't like "dumbed down" and "user friendly" computer interfaces much.

Maybe for arcade machines it is different, although I still think that for NES/Famicom and other console games at least, it makes sense to include the mapper data together with the game, and for each "unit" (a cartridge or a disk side) to be a file.

etabeta, so why do you feel it's appropriate to devise a layout strictly tailored to people "repairing PCBs", at great expense to clarity/consistency/ease of use with emulators, when I've shown that the odds of that actually happening are in the millions?

How does it make sense to split 2x16M vs 1x32M identical ROMs, yet not split DSP internal PROM/DROM simply because it's inside the same chip?

I'd like to hear your thoughts after careful consideration, and not the MAME party line position, if possible please



Despite my critiques of your NSS decision, I still split PRG and CHR as well.

As I said, when you split things by their intended use, things just make sense. PRG and CHR are not connected to the same bus, so it's only natural to split them. Whereas if Nintendo somehow managed to put them both into the same ROM chip, double-clocked with a "selector" pin (ignore the impracticalities and just pretend that happened), MESS would have distributed them as a single file, whereas I still would not.



If preservation is your goal, I should not have to read MAME source code to figure out how to load games.



Oooooooooooooooh trust me. From personal experience, they would.

MESS users might not mind, as higan users didn't. But general/casual users? You can pry their merged ROMs from their cold, dead hands.

Search for the bsnes v091/higan v092 release threads on various other emulation news websites and see how welcome the change was. Many of the topics hit 20+ pages before being locked.



Oh good. So the people who made the original nss_*.zip files just screwed up then.

Okay, it makes a lot of sense to put spc700.rom into "system ZIPs." I do something similar with system folders in my emulator. All of the boot ROMs / BIOS roms / etc go in those.



This is what is principally missing from MAME. How do you expect others to emulate the NSS? By cloning your nss.c and hoping you don't deem that a license violation when used in commercial software?

My manifests allow you to specify names as well, so you are given the option to use junk for filenames and extensions (eg what's written on the ICs.) But I still go for consistency and name every similar type of file the same, so that one doesn't have to parse the manifests to be able to load the games. The filenames are there to tell the user what type of data it is, and the extensions specify the volatility.



Had you used my example, you could have included "wiring.xml" with 100 times the information for all zero people repairing their NSS boards. You could also have pictures of the PCBs (something I always scan in), and label each ROM chip in the photograph. It surely would have been helpful to the handful of people on eBay using nocash's info to create new forgery NSS games that are destroying real NSS boards, however. (Because isn't it so cool to play non-NSS games in your arcade cabinet?)

You mean like multicarts that use CHR RAM (e.g. Nintendo World Championships) or like OneBus?


In terms of what I mentioned earlier, what MAME implements here is the "database hardcoded into the emulator" approach.


Zip is a container, iNES is a container, and gluing is a (much less capable) container. What makes gluing less capable is that without a directory structure, it's hard to tell which version of the manifest the file is supposed to conform to. Zip's directory is labels on the ROM sockets, and iNES's directory is the PRG ROM size CHR ROM size bytes, allowing at least some measure of splitting. Ability to split and reglue becomes important when the manifest changes to add or remove things as more is discovered about a PCB's operation, such as when DSP-1 support went from high-level to low-level emulation.

At least as a first step, I'd recommend zip files with a .mame extension so that the user can double-click one and have it start the game in MAME.

The disagreement I see here is how the filenames relate to the function of the data. Higan prefers filenames that reflect function, while MAME prefers filenames that reflect the ROM sockets.

I've personally used MAME romsets to verify arcade board rom chips while trying to diagnose issues. It certainly would have been annoying if the rom files didn't reflect what was on the actual board.

And that's what I call stupid.

Yes, this would help, even if the file isn't associated. Therefore you can know what the file is, if the extension is ".mame". Maybe it would be better to allow (even if it isn't required) a manifest to be included in the same ZIP archive, in case you are writing your own games and/or slight variations of the arcade machines.

Filename disagreements are fairly petty, just annoying. With higan's manifests, you can give files weird names like nss_mw-prg_0.ic1 if you want.

The real disagreement is the decision on when to split data. MAME splits out data as one file per IC's ROM contents. I prefer one file per purpose, so if data from two buses is inside one chip, it gets two files. If two ICs are supposed to appear in linear order on a bus, it gets one file.



It does add an additional complication onto loading these games, but it's not too bad. You still have to detect the presence of the coprocessor. Once you know it's there, see if the file size & 0x1ffff == DSP size. All of them are weird sizes, 3KB(?), 8KB, 52KB, 160KB. All SFC games are divisible by 256KB.

I don't like this strategy, either. I prefer the split games. This was simply an attempt to stick with what people professed to like (single file per game.) And yet given the reaction, it revealed what I've been saying all along. It's not that people have strong preferences on how things are done: they instead only have strong preferences on keeping things the way they've always been done in the past. It's not okay to split PRG and CHR or program and data ROM, but if new data arrives later on, *then* it's okay to split that off into another file.

Still, if one insists on having a single file for a game, it's the most eloquent way to bypass the copyright issues inherent in distributing this firmware with the emulator itself. These programs exceed Lexmark v SCC's scope.



When I said the odds were astronomical, I didn't say they were non-existent. Do you agree that you are in the extreme minority in being someone who dumps arcade ROM chips and compares them to files in MAME in order to potentially reflash and resolder/resocket your own chips on the boards?

I'm not saying it's never useful. My question is, what is more useful? Making a format that is easier to repair PCBs, or making a format that is easier to emulate? If we chose the latter, would you have been at a detriment to verify your arcade board if instead you had an HTML document complete with images and wiring diagrams instead of filenames in a ZIP archive to go by?

Frankly, MAME's method of storing games is quite arguably the least possible useful way of documenting physical board structure. Absolutely anything would be an improvement, if that were truly such an important goal to them.



Just to let this cat out of the bag ...

MAME has a rule that they can emulate new games the day they come out. These games get posted to a private server where MAME devs can grab them at any time. Although it's okay for MAME devs to play new releases, MAME users need to wait three(?) years before they are allowed to play games (which, is a nice idea, I'm not against helping to protect arcade makers' profits in return for more games made by them in the future.)

Point being, the internal database also serves as a hurdle to prevent loading newly dumped and/or leaked game images into existing MAME builds. The public database gets these new game manifest markers commented out and/or removed. People attempting to play these games anyway usually create and download third-party builds that hack in the entries again. It goes without saying that these third-party builds garner much derision from the devs.

MAME's own moralist model gives impetus to not include manifests with individual games, lest people be able to play new games with current, official MAME builds.

But even if they did, let's be honest. They'd be extremely dependent upon the exact architecture of MAME, anyway. They would do others little good. Still, anything's better than nothing, so I'd support that change.