Updated iNES - NESdev BBS

Updated iNES
by Disch on 2006-07-14 (#15390)

This is a spinoff of what was starting to be discussed in this thread.

The general idea:

To update the iNES format and create an auditing utility to solve various iNES ambiguities, and to correct common iNES header screw-ups (I can't tell you how many games are headered wrong). All while retaining backwards compatibility with prior versions of iNES.

Why not just go with UNIF?:

The obvious question. Since UNIF ROMs are pretty much all labelled properly, and leave nothing to question... why bother with updating iNES? Why not just go with UNIF?

UNIF has some difficulties which prevent it from (ever?) becoming a full replacement to iNES:

- UNIF board name list is incomplete. While it's true that most US releases have a proper UNIF dump, many foreign games, and some rare games do not. Therefore an iNES->UNIF converter won't be able to convert every ROM... making iNES a remaining necessity

- ROM hackers and translators use iNES pretty much exclusively. Even if UNIF were a full replacement, its floating block format is incompatible with IPS (another depricated, yet widely popular format). In addition, every NES ROM editor on the planet is built to work with iNES roms. I'm not aware of any that have UNIF support. So unless every hack, translation, and ROM hacking utility are updated to work with UNIF, there will be a large portion of the emulation community who will refuse to switch.

- Many ROM sites distribute iNES, but not UNIF. Some of the more "major" ROM sites even keep current with GoodNES sets. If we can just take over the next GoodNES release from Cowering, there's a good chance the new ROMs will get injected into the emulation community and begin circulating. UNIF, which has already been around a while, still hasn't really got anywhere.

What do we want out of the updated format:

Blargg brought up a very good point in that other thread:

Quote:

The most important thing is to limit the scope of what is being attempted. As I understand it, the goal is to remove the ambiguity in specifying which mapper is being used (and possibly some extra hardware). Attempting to throw in a kitchen sink would definitely ruin the effort.

I couldn't have said it better myself. UNIF aready does the job of ironing out every tiny thing you can think of. All we're trying to do here is get the ROMs playable without game specific hacks or runtime CRC checks.

What this thread is to accomplish:

I'd really like to outline ideas on what you think will be the best way to approach this. We're stuck with a 16 byte header, of which several bytes are already used.

The things we need to do with this format are:

1) Create something like a "sub-mapper" number which differentiates incompatible mappers which share the same mapper number. For example, MMC3 might be 004:0, while MMC6 might be 004:1 (aaa:b, a = mapper number, b = mapper sub). Most 071 games might be 071:0, and Fire Hawk (1-screen) might be 071:1.

2) Actually get things headered right. Fix ALL the damn mirror mode errors present in 30% of the iNES ROMs floating out there. Remove that DiskDude crap, etc.

3) Actually get a space which tells on-cartridge WRAM and CHR-RAM sizes. None of that "0 defaults to 8k" stuff that's in iNES.

4) Reliable NTSC/PAL/Dual value

5) Perhaps a "Bus Conflicts" flag like the one in the 'unofficial' update listed on the wiki?

That's pretty much all I can think of that the new format should cover. If iNES did all that I'd be perfectly satisfied with it.

But that's not all:

A new format is useless unless we have everything organized! We can shoot the breeze and talk about how awesome this would be all day, but that wouldn't change anything. SO -- in addition to outlining a new format... we need to do the following:

1) Outline DETAILS of the mapper number for every known ROM dump out there, and assign a mapper number:sub for each and every one of them. This is where UNIF falls short in a big way, and if this format update is to have any snowball's chance in hell of working, this is really where we need to have a jump on. If we just slap together something vague, we'll just be back here again in a few months with all the same problems (assuming the format even got anywhere).

2) Get these mapper numbers documented! This probably even has to be done before #1. Register descriptions, differences in sub mapper numbers, etc all has to be layed out clear as day. There HAS to be an official list of mapper numbers, and it HAS to keep current. This is where iNES went to hell, and we have to stay ahead of something like that happening again in the future.

3) All ROMs in the new format should be properly headered. There's no excuse... I don't know how iNES went to hell... but that is really unacceptable.

4) WE NEED AN AUDITOR WHICH WILL CONVERT ROMS AND UPDATE HEADERS. Without one, this project is a pipedream at best. I'd like to say I'd be willing to write one if I could get a list of CRCs for ROM dumps, but my job is keeping me pretty busy so I doubt I'll be able to. But if we can't get someone to write an auditor we might as well just stop now before we waste our time.

5) We need to get in touch with Cowering and tell him to either stop updating GoodNES, or hand off GoodNES to us so that his work doesn't conflict with ours. But of course before we ask him to do either we'll have to have our shit together.

That's about all I have on the subject. I'm too lazy to come up with a propsed new header format right now, but I might later once more ideas are on the table. What do you guys think?

by blargg on 2006-07-14 (#15393)

As Disch said, iNES is established, so improvements to it are much more likely to be successful than replacements. I don't know about all the esoteric mapper issues and how complex they are, but in my mind iNES is close enough to being sufficient that a replacement would have to dead-simple to handle otherwise it wouldn't be able to justify itself.

If this were carried out successfully, the current special case code in emulators would move to the iNES header upgrade tool ("auditor") and the emulator would then rely on the iNES header for unambiguously specifying the cartridge hardware. The additonal bits in the header should be defined in a way that minimizes the changes during updating, in most cases requiring no changes to the header. So in the MMC3/MMC6 case, an extra flag could be added that caused mapper 3 to be treated as MMC6. This seems preferable to defining a new mapper number, because it makes keeps the file usable in current emulators.

Re: Updated iNES
by tepples on 2006-07-14 (#15394)

Disch wrote:
- Many ROM sites distribute iNES, but not UNIF.

By "ROM sites" do you mean pdroms.de? Otherwise, if we work with the pirates, will we get shut down by the fibbies?

Quote:
I'd really like to outline ideas on what you think will be the best way to approach this. We're stuck with a 16 byte header

O RLY? Isn't there a heck of a lot of space after the CHR? See my iNIF proposal.

Quote:
1) Create something like a "sub-mapper" number which differentiates incompatible mappers which share the same mapper number.

As I understand it, the board name is the perfect sub-mapper.

Quote:
2) Actually get things headered right. Fix ALL the damn mirror mode errors present in 30% of the iNES ROMs floating out there. Remove that DiskDude crap, etc.

Won't help. GoodNES is the IE of ROM tools, and it doesn't check the header.

Quote:
1) Outline DETAILS of the mapper number for every known ROM dump out there

Which may require re-finding prototypes. $$$ anyone?

Quote:
and assign a mapper number:sub for each and every one of them. This is where UNIF falls short in a big way

How is the board name an inadequate submapper number? Is it just that people aren't willing to track down rare games and open them?

Quote:
2) Get these mapper numbers documented! This probably even has to be done before #1. Register descriptions, differences in sub mapper numbers, etc all has to be layed out clear as day.

You mean like the effort on the wiki to distinguish all the different S*ROM variants?

Quote:
4) WE NEED AN AUDITOR WHICH WILL CONVERT ROMS AND UPDATE HEADERS. Without one, this project is a pipedream at best.

Who would accept the legal exposure of maintaining this?

Quote:
5) We need to get in touch with Cowering and tell him to either stop updating GoodNES, or hand off GoodNES to us so that his work doesn't conflict with ours. But of course before we ask him to do either we'll have to have our shit together.

And until we have our poo together, we can coexist with GoodTools. For several years, gba-renamer coexisted with GoodGBA, and now Nintenren carries on the gba-renamer tradition.

Re: Updated iNES
by Quietust on 2006-07-14 (#15395)

tepples wrote:
Quote:
1) Create something like a "sub-mapper" number which differentiates incompatible mappers which share the same mapper number.

As I understand it, the board name is the perfect sub-mapper.

Quote:
and assign a mapper number:sub for each and every one of them. This is where UNIF falls short in a big way

How is the board name an inadequate submapper number? Is it just that people aren't willing to track down rare games and open them?

Exactly what board names would you assign for unlicensed games which don't actually have meaningful labels on the cartridge PCBs? Acclaim boards and custom-made Konami PCBs have fun "names" such as 351258, 351298, 351908, 352026, 51555, 53361, 54425, 55741, 56504, and many Famicom cartridge PCBs don't have any board names at all. This is what doomed UNIF to failure.

Re: Updated iNES
by Disch on 2006-07-14 (#15396)

tepples wrote:
Otherwise, if we work with the pirates, will we get shut down by the fibbies?

Who said we would be working with ROM sites? Making a ROM auditor is perfectly legit. That's all we'd be doing. Updated Goodsets would be picked up by ROM sites by them having run the auditor. We would not be distributing ROM sets.

Quote:
O RLY? Isn't there a heck of a lot of space after the CHR?

I suppose. We could go that route. Though personally I'd prefer to stick to the simple 16-byte header. Can't really support that with anything, it's just my personal preference (so much easier)

Quote:
As I understand it, the board name is the perfect sub-mapper.

It would be if we had all the board names. We don't. See my first point in the "Why not UNIF" section above. Unless this new format can cover every single ROM in existence, it's worthless. And guessing or making up board names defeats the whole point of going with board names in the first place.

Quote:
Won't help. GoodNES is the IE of ROM tools, and it doesn't check the header.

Which is why I said we need to make a new auditor. One to replace GoodNES. This one will check and correct/update the header.

Quote:
Which may require re-finding prototypes. $$$ anyone?

I didn't say re-dump the ROMs. I said document mapper details. IE - mapper documentation. The only thing that costs is time.

Quote:
How is the board name an inadequate submapper number?

See above and previous post

Quote:
Is it just that people aren't willing to track down rare games and open them?

Apparently. Like I said, if we had the board names, we could just use UNIF. We don't. So we can't.

Quote:
You mean like the effort on the wiki to distinguish all the different S*ROM variants?

Nothing that specific. Something like the EveryNES mapper section, except wikified and more accurate. Remember, the goal here is not 100% accuracy, it's to retain backwards compatibility and make all games operable without CRC checks or hacks.

You're looking at a different picture. You want something like UNIF. We're not trying to replace UNIF, we're trying to replace iNES.

Quote:
Who would accept the legal exposure of maintaining this?

What legal exposure? Since when is it illegal to make a ROM auditor?

Re: Updated iNES
by BootGod on 2006-07-14 (#15397)

Disch wrote:
ROM hackers and translators use iNES pretty much exclusively. Even if UNIF were a full replacement, its floating block format is incompatible with IPS (another depricated, yet widely popular format).

This really doesn't seem to be a problem in my eyes, it would be a trivial task to make an IPS intended for iNES work on a UNIF ROM.

Disch wrote:
UNIF aready does the job of ironing out every tiny thing you can think of.

Actually I've run into situations where the board name is not enough. Some Camerica boards use the exact same board with a different mapper stuck in. BF9093 is the mapper most Camerica games use, BF9096 is the mapper for Quattro carts, but they sometimes share the same board name.

Also, things like MMC version can not always be derived from the board name. I don't think this is much of a problem, but someone just the other day pointed out that not knowing between MMC3B and MMC3C can cause problems.

Disch wrote:
Actually get things headered right. Fix ALL the damn mirror mode errors present in 30% of the iNES ROMs floating out there. Remove that DiskDude crap, etc.

Believe it or not, the next GoodNES version does have header fixing in it.

Disch wrote:
WE NEED AN AUDITOR WHICH WILL CONVERT ROMS AND UPDATE HEADERS.

I don't mean to be plugging here, but my database/dumping software does all this. Of course, it needs to have the info to be able to convert.

Quietust wrote:
Exactly what board names would you assign for unlicensed games which don't actually have meaningful labels on the cartridge PCBs? Acclaim boards and custom-made Konami PCBs have fun "names" such as 351258, 351298, 351908, 352026, 51555, 53361, 54425, 55741, 56504, and many Famicom cartridge PCBs don't have any board names at all. This is what doomed UNIF to failure.

Yes, this is one of the biggest problems. A lot of the UNIF board names just get "made up", especially pirates and the like.

This is a little OT, but I have been assigning the same names that Kevin Horton made up for non-standard boards. Often he assigned unlicensed boards like "UNL-CompanyAbbrBoardName" like UNL-TEN800032 for a Tengen 800032 board.

Rather than prefixing with UNL, what do you think of prefixing it with a company abbreviation instead? For example:

TEN-800032
AVE-NINA-06
CAM-BF9093 (actually Camerica boards are labelled BIC-xx, but as stated above, that is insufficient)

You could do this for the funky Konami and Acclaim boards too:
KON-351908
ACC-55741

I don't really think it's all that neccessary to always stick to board names. For instance, Color Dreams/Wisdom Tree/AGCI/etc. have many boards, all with odd names or no names at all, but the differences all have to do with lockout-defeat methods. So rather than having to support 10 different Color Dreams boards, you could just use CDR-LS377 instead.

Ideally this sort of thing should be discussed before just winging it and slapping whatever name in there. That's pretty much the same as just going ahead an assigning something to an "unused" mapper number.

by tepples on 2006-07-14 (#15398)

The problem with including anything else in the 16-byte header is that ROMs that use it will likely make the anti-DiskDude! checks in current emulators fail. Barring that, I could see using bytes 12 and 13 for a two-character sub-mapper code. This code could be derived from a board name or from a game title, or it could be set to nul ('\0') bytes for the "default" behavior of the mapper.

Mapper 1:
default: 8 KB PRG RAM
If CHR size >= 8 KB then CHR ROM else 8 KB CHR RAM
If PRG size > 256 KB then a CHR bank line controls PRG ROM A18
If PRG RAM banks > 1 then a CHR bank line controls PRG RAM A13
variant SG: 8 KB CHR RAM, no PRG RAM
variant SL: CHR ROM, no PRG RAM
Mapper 4:
default: MMC3 "generic" (EDIT: where "generic" means 3B)
variant 3A: force MMC3A
variant 3B: force MMC3B
variant 3C: force MMC3C
variant LG: MMC3 "generic" without PRG RAM (for Low G Man)
variant HK: MMC6
Mapper 71:
default: vertical or horizontal mirroring
variant FH: 1-screen mirroring (for Fire Hawk)

by -_pentium5.1_- on 2006-07-14 (#15399)

tepples wrote:
Mapper 4:
default: MMC3 "generic"
variant 3B: MMC3B
variant 3C: MMC3C
variant LG: MMC3 "generic" without PRG RAM (for Low G Man)
variant HK: MMC6

MMC3B is the preferred default now, not MMC3A - see my post here.
(If you're not familiar with FCEU-mm, it's an unofficial FCE Ultra build by CaH4e3.)

by Bregalad on 2006-07-15 (#15401)

The main problem would be backward-compatibility.

Since most people does think iNES work fine, and most people already have iNES roms on their PC, most people will just not care about the format, because most people are pissed about the SRAM sizes, CHR sizes, bus conflicts and whatever since their favourite games works. Most people don't care that some rare Famicom-only games does have glitches on their emulators.

You won't be able to tell to everyone "delete all your ROMs and download new ones, those have a fixed header system". They won't follow you.

Also, to fix every ROM out there, that'd be a lot of work. There is a long official list of licenced NES games in america, but I found none for Europe or Japan, and I think there is hundreds of FC only games that are small, rare and totally unknown, especially if their title is in japaneese.
Pirate games are even more unorganized than licenced games, so the only real purpose of this would be homebrew devloppement ?

by hap on 2006-07-15 (#15402)

tepples wrote:
The problem with including anything else in the 16-byte header is that ROMs that use it will likely make the anti-DiskDude! checks in current emulators fail. Barring that, I could see using bytes 12 and 13 for a two-character sub-mapper code. This code could be derived from a board name or from a game title, or it could be set to nul ('\0') bytes for the "default" behavior of the mapper.
Good point on the anti-DiskDude check, new iNES header bits shouldn't be at the start of the reserved bits. I also like your idea of 2-letter sub-mapper code: it's descriptive, and it would almost certainly be accurate even if the header contains garbage.

Quote:
The main problem would be backward-compatibility.
We'll try to keep it backwards-compatible, so it should still be ok on most older emulators.

Quote:
You won't be able to tell to everyone "delete all your ROMs and download new ones, those have a fixed header system". They won't follow you.
Not if their favourite emulator suddenly 'stops working'. Sure, common users will whine to the author having to implement a checksum database in his emulator, then they'll whine some more, and in the end redownload (or easier: audit).

Quote:
Also, to fix every ROM out there, that'd be a lot of work.
It's not a one-man's job. Yes, it would still be a lot of work but I don't see that as a problem. We can use the wiki for this.

by byuu on 2006-07-15 (#15403)

Fixing hacker memory maps in ROMs so emulators don't have to rely on CRC databases is a respectable goal, but rather unrealistic.
Have you considered ever consolodating an online CRC32+MD5+SHA-1 database for all games that can be downloaded in several formats and with tools and libraries in multiple languages (c, c++, vb, etc) for calculating the checksums and selecting the appropriate game? Since this is predominantly an emulator problem and not a user problem, you're likely to make a lot more progress this way.
Then if all of the top emulators were to start throwing out warning messages when a game doesn't match up in the CRC database, this would encourage fan translators and hackers to start using a new header system that bypasses the need for a CRC lookup.
I'm hesitant to make a database that includes CRCs for translations, as that will only lead to hacks being included pre-patched at ROM sites lacking any and all readmes that came with the original work, a method most ROM hackers do not want their work distributed. The ROM sites can still go out of their way to acquire these unofficial works and include them with the "hacker headers", but at least the indexing project is no longer taking a direct role in promoting the behavior.

Quote:
5) We need to get in touch with Cowering and tell him to either stop updating GoodNES, or hand off GoodNES to us so that his work doesn't conflict with ours. But of course before we ask him to do either we'll have to have our shit together.

Perhaps we're speaking of a different Cowering, but if it's the same one who wouldn't remove a single translation I was responsible for from his piracy database: you stand a snowball's chance in hell of convincing him to step down from an entire system because you have a slightly updated header format. However, I can't say I directly know him so maybe you really can convince him to do this. If so, I wish you luck. One less GoodTool is a GoodDay indeed for emulation.

Quote:
Not if their favourite emulator suddenly 'stops working'. Sure, common users will whine to the author having to implement a checksum database in his emulator, then they'll whine some more, and in the end redownload (or easier: audit).

With how many popular NES emulators there are, do you think you could get at least an 80% majority to outright block an old format from playing at all? If you have less than that, you'll only make another emulator more popular. Remember, end users don't care about accuracy or quality, so long as their favorite ten popular games "look" right and run full speed on their ten year old computers.
If you can get an alliance to do this, then you have your best chance of getting everyone to upgrade their ROM headers right there. But remember, any emulator that pops up and doesn't block these games will suddenly have a desirable end-user feature your emulator doesn't. And I would suspect open source emulators would end up with "patched" binaries floating around that bypass the block.

by hap on 2006-07-15 (#15404)

Quote:
I'm hesitant to make a database that includes CRCs for translations, as that will only lead to hacks being included pre-patched at ROM sites lacking any and all readmes that came with the original work, a method most ROM hackers do not want their work distributed.
I'm with you on that, same goes for 'PD' programs/games.

Quote:
With how many popular NES emulators there are, do you think you could get at least an 80% majority to outright block an old format from playing at all?
The format revisions would be quite compatible with eachother. The current romset (GoodNES 2.01) would work as well as it does now on Nintendulator; an emulator without a built-in database or header corrections. Games with bogus headers, or games like Startropics wouldn't. 80% is too much (people still use the inactive NESticle and FCEU), but if Marty of Nestopia joins in, which I think is currently the most popular active NES emulator, it could succeed.

by tepples on 2006-07-15 (#15408)

byuu wrote:
I'm hesitant to make a database that includes CRCs for translations, as that will only lead to hacks being included pre-patched at ROM sites lacking any and all readmes that came with the original work

New format suggestion: iNES+ZIP. It's built like a self-extracting PKZIP archive, except instead of being an unzipper followed by a zipfile, it's an iNES ROM followed by a zipfile. Store the README in the zipfile and there's no way that the casual user will separate them. Then the auditing program could make sure that the zipfile contents are correct too.

Quote:
Remember, end users don't care about accuracy or quality, so long as their favorite ten popular games "look" right and run full speed on their ten year old computers.

If you replace "ten year old" with "handheld" it's almost correct. People want their ROM collections to work on both PocketNES and PocketNES.

Quote:
If you can get an alliance to do this, then you have your best chance of getting everyone to upgrade their ROM headers right there. But remember, any emulator that pops up and doesn't block these games will suddenly have a desirable end-user feature your emulator doesn't.

Then pop up a dialog box:
Code:
/ \ This ROM is broken. Do you want ScoNES
/_!_\ to fix it for you?

[ Fix ] [ Cancel ]

where pressing OK launches betternes --fix C:\nes\SMB1.nes

hap wrote:
I'm with you on that, same goes for 'PD' programs/games.

Excluding PD programs entirely would mean that you're including only pirated programs, which might get the developers indicted for "inducing" piracy. My policy is include those that work on NES hardware and exclude those that don't. For instance, include Tetramino but exclude GNOME vs. KDE and Mouser. But at least PD programs can display a URL of the manual.

by Bregalad on 2006-07-15 (#15418)

Quote:
Not if their favourite emulator suddenly 'stops working'. Sure, common users will whine to the author having to implement a checksum database in his emulator, then they'll whine some more, and in the end redownload (or easier: audit).

Nope. They'll keep the old version of their emulator on their PC with old ROMs, and say the last version of the emulator suck. A few of them will complain to the author, and he will additionnally recive lot of messages that are annoying to him

by mattmatteh on 2006-07-15 (#15420)

i think if by adding more to the ines file format will just add to the mess that it is now. i am not familiar with the famicom games, but if they dont have a board number, then perhaps bump the unif version and add a section MMC, where that could state the mmc controller used in the event the board name will not work.

or, if you must add to the ines format, put unif chunks at the end of the file. either check to see if there is data at the end or use one of the unused bits in the ines header indicating unif chunks are at the end.

actually both these methods might work.

matt

by dvdmth on 2006-07-16 (#15425)

I'm not sure how anyone will ever succeed in enhancing/modifying/replacing the iNES format when you can't even convince people to stop using Nesticle. The fact is that people are set in their ways and won't change even if a vastly improved alternative (such as the Metric system over the current US measurement system) is provided. If it ain't broke, don't fix it.

If it's true that the next GoodNES version will (finally) resolve bad header issues, would that not reduce the problem substantially? I won't say it will disappear completely as most issues like this never get resolved, but at least the more major ROM sites will finally be providing playable images.

Granted, there are a number of ambiguities in mapper designations, but these only affect a few games (relative to the number of games out there). Similar ambiguities have to be resolved by emulators for other platforms as well. I recently looked at the SNES header spec (Nintendo's own header, built-in to the ROM image), and its coprocessor field does not provide enough information to distinguish between all types of coprocessors (near as I can tell, there's no way to tell from that field whether a game uses the DSP-1, DSP-2, DSP-3, or DSP-4). Emulators resolve this by looking at the game's title and/or checksum. Granted this information is built-in to the header, but what prevents an NES emulator from looking at the name of the ROM file (in today's day and age, how often does the name of a ROM file not contain tthe name of the game itself?) or from calculating a CRC like Nestopia does? Frankly, I fail to see the difference between these approaches.

by tepples on 2006-07-16 (#15426)

dvdmth wrote:
what prevents an NES emulator from looking at the name of the ROM file (in today's day and age, how often does the name of a ROM file not contain tthe name of the game itself?) or from calculating a CRC like Nestopia does?

The existence of homebrew, for one thing.

by -_pentium5.1_- on 2006-07-16 (#15431)

Bregalad wrote:
There is a long official list of licenced NES games in america, but I found none for Europe or Japan...
Japan might be tough, but I know of the following lists for Europe: this one at The Warp Zone and the one at GameFAQs. Neither is totally complete, and the one from GameFAQs seems to be better.
Bregalad wrote:
Pirate games are even more unorganized than licenced games, so the only real purpose of this would be homebrew devloppement ?
Both pirate games and homebrew are very important IMO.

by kyuusaku on 2006-07-17 (#15435)

I have a partial list for Japan, I started transcribing it from a book. I can complete it if there's interest.

by Disch on 2006-07-17 (#15449)

dvdmth wrote:
or from calculating a CRC like Nestopia does?

I fail to see the distinction between CRC checks to correct headers, and CRC checks to correct ROM quirks (like Megaman 3's sloppy enemy select screen), or even CRC checks to correct emulation issues.

All of them should be avoided. In my mind they're all sloppy hacks.

The ROM file all by it's lonesome should contain everything the emulator needs in order to run it successfully. Just like all the NES needs is the cartridge. The emu shouldn't need to have pre-existing knowledge of X or Y obscure games and have to look specifically for them and perform internal tweaks in order to get them to run -- that's bullshit. In fact... emus which did this initially are partially to blame for the bad dumps and improperly headered ROMs out now. If all emulators just emulated and didn't try to be an all-in-one fixitup tool, we wouldn't even have this problem.

I especially dislike CRC checks because they will only work on a specific dump, and will not work on ROM hacks or translations. Also there's the very very slim chance of CRC confliction... but odds of that are probably in the realm of statistical impossibility. At least with runtime logic checks (like a few I layed out in that other thread), the checks are more generic and will run any game unless the game specifically tries to "fool" the emulator.

by tepples on 2006-07-17 (#15450)

Disch wrote:
The ROM file all by it's lonesome should contain everything the emulator needs in order to run it successfully. Just like all the NES needs is the cartridge.

The NES needs more than a cartridge unless you like looking at title screens all day. The emulator needs to know whether to enable a light gun in slot 2 (Duck Hunt), a dance pad in slot 2 (World Class Track Meet), a potentiometer in the expansion port (Arkanoid), etc. That info would normally be on the label of the cartridge, which is not present in an iNES file.

by Disch on 2006-07-17 (#15451)

Well of course if you want to play semantics you can find and pick out technical holes in my analogy, but my point remains clear.

by byuu on 2006-07-17 (#15452)

Quote:
I fail to see the distinction between CRC checks to correct headers, and CRC checks to correct ROM quirks (like Megaman 3's sloppy enemy select screen), or even CRC checks to correct emulation issues.

Then you have a perception problem. You either store the info in an open database that all emulators use and is easily upgradeable to the latest version (possibly even make the emu have an auto update feature ala Firefox), or you stick it into ROM image headers. Either way, it's just extra info that runs the game. You stick it in the ROM, now you get to try and convince 50% of the NES userbase that still thinks NESticle is "good enough" to replace all of their ROMs. Not. Gonna. Happen.

A ROM image is just that, a copy of the ROMs on a PCB. Sticking cart information into a ROM image is as hackish as sticking it into a database is.

Insisting that a database leads to game specific hacks is stupidity. I could just as easily claim all emulation leads to piracy, and school dances lead to pregnancy. If someone wants to add hacks to their emulator, they will do it -with or without- a database. The database won't make it any easier for them at all. AT ALL. I've heard three respectable emulation scene people cite this exact reason and it's completely absurd.

As for ROM hacks, you can simply advocate soft patching, as we should all be doing in the first place, and then check the CRC against the database prior to patching. In the event someone modifies the game so greatly as to require a new mapper to run, then you fall back on the updated header approach.

I don't get why this has to be one way or the other, period end of story. The best thing to do is everything you can to eliminate your current problem: shitty old iNES headers. You do both, and then when one gets to be accepted by the majority, you go with that.

The DB has advantages, too. You don't need to have your userbase upgrade all of their ROMs, ROM sites don't have to update their content, you no longer have the 16-byte size limitation so you can store all the info you want in the DB, and then some. Collision problems are stupid. Store an SHA512 hash if you're worried about that. Nobody will even be able to make a collision in that for the next 3-5 years, at least. You also now don't have to worry about accidentally patching in wrong info into a ROM header, or missing a field you find out you need later on. The database is self-correcting and much easier to update than updating all ROMs everywhere.

And so does the ROM header upgrade idea. It's more flexible for ROM hackers to take advantage of, for one.

Either way, you need a database. Either to use directly in emus, or to make the ROM scanning tool that patches all of the headers. Why not agree to make the database now, and then bicker over which method is better, and let each emu author choose what they want to do?

EDIT: thought this was an amusing point to reference here as well.

I wrote:
pagefault, Nach and Disch oppose the use of a database because it "encourages adding game-specific hacks to correct emulation bugs". And yet, the emulators that do use databases are bsnes, Super Sleuth and Nestopia; written by me, Overload and Marty. The emulators that don't support databases are ZSNES, SNES9x and NESticle. You tell me what's wrong with that.

by blargg on 2006-07-17 (#15454)

Quote:
The emulator needs to know whether to enable a light gun in slot 2 (Duck Hunt), a dance pad in slot 2 (World Class Track Meet), a potentiometer in the expansion port (Arkanoid), etc. That info would normally be on the label of the cartridge, which is not present in an iNES file.

Since it's the user who plugs those controllers into the NES, an accuracy-at-all-costs emulator would require the user to manually command it to plug in a particular controller. You could even plug in the wrong controller for a particular game and get the same effect you'd get on a NES. In essence, the user is left to manually configure the NES hardware for these particular games, so it's definitely not part of the cartridge.

As for where to draw the line between a NES emulator and cartridge files, an important side-effect of the location is the legality of the emulator. I imagine that an emulator with a built-in database of commercial NES games (or built-in functionality for one, like GoodNES) would make it much more susceptible to legal issues. It seems best that the emulator incorporate as little as possible outside the NES control deck hardware. The biggest exception to this is the behavior of mapper hardware, which would be quite hard to specify entirely in the NES cartridge file without imposing a significant performance penalty on the emulator. If we accept that mapper behavior will be stored in the emulator, everything else can be stored in the NES cartridge file.

I think it's unfortunate that people deal with "ROM files" rather than "cartridge files", as the latter is a better idea. Why would you just want the data in the ROM of a cartridge, rather than that plus the wiring information and other behavior? There's nothing hacky about representing this other information in the cartridge file; it's just your standard encoding of information in a machine-readable format.

I can see the use of a database in automatically filling in the cartridge hardware information when someone uses a cartridge dumper so they can play the game on their PC. People experienced with the hardware can ensure that the database entries are correct, and (presumably) distribute this information without infringing on any copyrights, since it'd contain things like "Wizards & Warriors, mapper = AOROM" or whatever. Combine this database with a dump you make of your own cartridge and you have enough information to play it on an emulator, either by building a self-contained cartridge file or letting the emulator do the equivalent of this internally.

I think I have to agree with byuu's central point that a database is the main thing to build. A nice side-effect is that the file format need not be set in stone, since a programmatic interface can be provided with insulates the emulator from the particulars. An emulator could locate an entry based on the name or various checksums, then find what mapper it uses, how big the WRAM is, etc. The user might want to be able to use multiple databases from different sources, in case one is more reliable but doesn't cover as many games.

by tepples on 2006-07-17 (#15456)

byuu wrote:
A ROM image is just that, a copy of the ROMs on a PCB. Sticking cart information into a ROM image is as hackish as sticking it into a database is.

Then what is the "PCB image" or the "mapper image"? The 16-byte header is a (lossy) compressed version of the circuit board and all components on it other than the ROMs.

Quote:
As for ROM hacks, you can simply advocate soft patching, as we should all be doing in the first place

Do most tile editors and (especially) mainstream hex editors support soft patching, or is this an approach for end users and not developers of ROM hacks?

Quote:
And yet, the emulators that do use databases are bsnes, Super Sleuth and Nestopia; written by me, Overload and Marty. The emulators that don't support databases are ZSNES, SNES9x and NESticle. You tell me what's wrong with that.

Unlike ZSNES, bsnes is proprietary software. It's the same reason people still use Sendmail and Postfix: even if qmail is higher quality software, they object to qmail's license, which resembles that of bsnes. To many users, proprietary software has zero quality, solely by virtue of its being proprietary.

by -_pentium5.1_- on 2006-07-17 (#15458)

[OT-1]
tepples wrote:
Unlike ZSNES, bsnes is proprietary software. It's the same reason people still use Sendmail and Postfix: even if qmail is higher quality software, they object to qmail's license, which resembles that of bsnes. To many users, proprietary software has zero quality, solely by virtue of its being proprietary.
IMO/AFAIK, the primary reason for bsnes's unpopularity is not its license; it's the fact that better accuracy results in worse speed. This is just a statement and not a request of any kind. [/OT-1]
[OT-2]
tepples wrote:
Do most tile editors and (especially) mainstream hex editors support soft patching, or is this an approach for end users and not developers of ROM hacks?
One obstacle to soft patching for ROM hacks is that current patch formats don't support ROM expansion in an efficient manner. [/OT-2]

If we are going to seriously consider revising the iNES format, I second the general request for a sub-mapper number. I also support increased use of UNIF and iNIF for tricky games when appropriate.

I would like to add another mapper variant to the list of cases under consideration. It concerns MMC2, specifically the fact that the PC10 version of Mike Tyson's Punch-Out!! uses battery-backed WRAM to save high scores. Some pictures of the PCB were posted in this thread.

by creaothceann on 2006-07-18 (#15460)

tepples wrote:
Do most tile editors and (especially) mainstream hex editors support soft patching, or is this an approach for end users and not developers of ROM hacks?

You could use hardpatching to create a temporary file, and create an IPS when you're done with editing.

Not very nice, but this could probably be automated with batch files.

by kyuusaku on 2006-07-18 (#15465)

Why is this board so concerned about enlightening/supporting "Nesticle" (or any old emulator's) users? If iNES were to be unofficially revamped and Nesticle "broke", wouldn't that be all the more reason for them to move up? If they bitch, why should anyone here care?

Instead of extending the reserved bits, I think it'd be better for authors to agree upon new mapper numbers. Perhaps duplicates and useless mappers could be the first to be replaced.

How about making mapper 6 (currently "FFE F4XXX") MMC6?

by byuu on 2006-07-18 (#15466)

Quote:
I think it's unfortunate that people deal with "ROM files" rather than "cartridge files", as the latter is a better idea.

I agree, putting all info into one file is definitely the best end goal, but this is much harder than it sounds, and a flexible database where you can change the format and not instantly break all old ROMs is a good interim solution. We need to know, down to every last bit, exactly what will be needed in a cartridge header, from now until forever.
Plus, as a nice bonus, this database will be needed by any ROM scanning tools to update the ROM headers when it does reach fruition. Emulators can even begin including an option to patch / auto-patch ROMs with DB info upon load.

I don't believe an emulator using a database of "game a = mapper n" would put the author in legal trouble, but I'm not a lawyer. I also don't think any companies are planning to go after freeware emus. That's just bad press and you won't recover any damages. And people will just start distributing said emulator on piracy web sites anyway, and development will just move underground, harder to stop.
But then, you never know what VPs at Nintendo will plan once the Wii is out and they start relying on that emulation revenue.

So, hopefully in the end we will be able to move all info into the ROMs. Just make sure you have the format finalized for all of time first before you try.

Quote:
Do most tile editors and (especially) mainstream hex editors support soft patching, or is this an approach for end users and not developers of ROM hacks?

Tools adapted to copier and iNES headers. They will adapt to soft patching. In the mean time, any capable rom hacker could easily use automation, or one could always create cart-specific DB entries.
eg : cartname.nes looks for cartname.inf, falling back on cart.db if it does not exist. End users need not worry about this.

Quote:
Unlike ZSNES, bsnes is proprietary software. It's the same reason people still use Sendmail and Postfix: even if qmail is higher quality software, they object to qmail's license, which resembles that of bsnes. To many users, proprietary software has zero quality, solely by virtue of its being proprietary.

<very offtopic, sorry>

Everyone is entitled to their opinions. My code is plenty free. Free to use, free to distribute, free to obtain full source code, free to modify for personal use, free to submit any contributions to the source back to me for inclusion in the main branch.
I do not believe in forking, thusly this allows a unified project working toward one goal: emulation.
End users share the benefits of all contributions in one emulator. They aren't forced to choose between bsnes, bsnes Ultra XD SP, UO bsnes PP SE (unofficial) ++, etc. to get the specific options they want.
I also have no intentions of competing with someone who rips off my code, changes two things, and releases their own "version" claiming it to be superior.
And ultimately, I don't want someone hijacking and ultimately taking lead over my project because they added a few nice features and have more time than me to work on even more features. See Xorg.
You want to compete with me? Make your own emulator, just like I did. You want to collaborate and make the emulator better? Please do, I accept source contributions from anywhere. You want to release your own binary despite all of that, such as Richard Bannister is doing right now? No problem, just ask my permission first, please.
Proprietary my ass.

We can continue this discussion in PM if you prefer.

by hap on 2006-07-18 (#15468)

I agree we could start off with a database.

Quote:
Why is this board so concerned about enlightening/supporting "Nesticle" (or any old emulator's) users? If iNES were to be unofficially revamped and Nesticle "broke", wouldn't that be all the more reason for them to move up? If they bitch, why should anyone here care?
They wouldn't break: older emulators (are supposed to) ignore the reserved bits.

Quote:
Instead of extending the reserved bits, I think it'd be better for authors to agree upon new mapper numbers. Perhaps duplicates and useless mappers could be the first to be replaced.
That would create incompatibilities between iNES header revisions. Besides, 256 possibilities can be too little.

by Disch on 2006-07-18 (#15471)

byuu wrote:
EDIT: thought this was an amusing point to reference here as well.

I wrote:
pagefault, Nach and Disch oppose the use of a database because it "encourages adding game-specific hacks to correct emulation bugs". And yet, the emulators that do use databases are bsnes, Super Sleuth and Nestopia; written by me, Overload and Marty. The emulators that don't support databases are ZSNES, SNES9x and NESticle. You tell me what's wrong with that.

I just saw this now... must have missed it before.

I really, really, REALLY hate being misquoted. Especially in a way which makes me look like a total douche.

If someone can point out where I said that using a database encourages adding game-specific hacks... I'd love to see that. Because to the best of my knowledge I never did.

What I SAID was that I don't see the difference between CRC checking for one thing or CRC checking for another. You could say that I view using a CRC database in itself is a game-specific hack (since, when you think about it, each CRC entry is game specific... and a "hack" would be subbing in-game operation with replacement information --- which such a database lookup does).

If you want to put me on the record like "Disch feels CRC databases are a game specific hack" or something of that sort, go right ahead. That's where I stand.

Saying that I feel it encourages game specific hacks to correct emulation bugs is a total misquote (and the fact that it's in quotes... yet it's something I never said... maybe they just took someone else's quote and put my name to it?).

Person A: "Downloading commercial ROMs is piracy. That's a crime."
Person B: "OMFG did you hear person A? He things downloading ROMs leads to crimes like rape and murder!"

EDIT --

Yeah I overreacted, so what, wanna fight about it?

But seriously, I did kind of overdo it. Was at work, blew it out of perportion. I'm not really that upset. It's all about the love and peace.

by mattmatteh on 2006-07-18 (#15474)

i dont see the point in updating the ines file format. it seems like this was a problem before and unif was created to solve that. it didnt, there are stilll alot of messed up ines formats out there. i think we need to push the unif format and get it over with. and if unif is lacking something then lets talk about adding to that.

matt

by -_pentium5.1_- on 2006-07-18 (#15491)

hap wrote:
Quote:
Instead of extending the reserved bits, I think it'd be better for authors to agree upon new mapper numbers. Perhaps duplicates and useless mappers could be the first to be replaced.
That would create incompatibilities between iNES header revisions. Besides, 256 possibilities can be too little.
IMO, it is possible to do some careful reassigning of mapper numbers to keep the really important mappers on the same numbers and then use UNIF (?) for anything that is left after the 256 mappers run out, which will be a very long time.

This particular suggestion probably won't be implemented but...
tepples wrote:
Mapper 4:
default: MMC3 "generic" (EDIT: where "generic" means 3B)
variant 3A: force MMC3A
variant 3B: force MMC3B
variant 3C: force MMC3C
variant LG: MMC3 "generic" without PRG RAM (for Low G Man)
variant HK: MMC6

What if a game needs Low G Man's MMC3 board (forgot the formal name) with a revision other than MMC3B?

by byuu on 2006-07-19 (#15496)

Quote:
I fail to see the distinction between CRC checks to correct headers, and CRC checks to correct ROM quirks (like Megaman 3's sloppy enemy select screen), or even CRC checks to correct emulation issues.

All of them should be avoided. In my mind they're all sloppy hacks.

Specifically the part where you "fail to see the distinction between..." and "they're all sloppy hacks". But my quote was more of what Nach and pagefault were saying on IRC. Sorry, after arguing with them about "what leads to hacks", seeing your comment got under my skin. It appeared to me you were saying the same thing they were. That using CRCs was akin to use game-specific hacks and/or encouraged using them. Which, so far, has not been the case for any NES or SNES emulator I'm aware of. In fact, it's been the very opposite. Whereas the emulators Nach and pagefault maintain do contain game-specific hacks. N64 and above, sure. Those use DBs for hack information. But that's not what system we're talking about here. Those systems could also just as easily use CRC detection internally to the emulator if they wanted. The fact they're using a DB makes no difference to their use of hacks.

My apologies for misquoting you.

by mattmatteh on 2006-07-19 (#15499)

is there an irc for nes or nesdev chat ? i was looking for one a while aog and couldnt fine one. at least no one was in the channel i tried.

matt