Myask wrote:
Lidnariq's right in that "use chunks" is an odd answer to "how should we extend our binary-blob format"…even if it's not asked that way. (And we have had the holywar argument a fair few times, with myself and… zzo? vs koitsu et al, usually…) And rainwarrior's right in that chunks are more extensible. Should really actually obey IFF constraints if one goes to chunked, though; leading with NES[$1A] isn't compliant, it's not FORM, LIST, nor "CAT ".
I mentioned RIFF as an example of a format that accomplishes the chunk idea, I wasn't making any argument to use IFF standards.
The view I have is just that "if the mapper number is X then the meaning of data appended after PRG + CHR changes" is very fragile. The length of the file provides the minimum amount of information to include
just that, but any further discovered data needs become extremely difficult to accommodate. Lidnariq made a comment about ordering of PC-10 data vs this hypothetical extension, which isn't specifically a problem for anybody, but it was a small representation of the kind of problems that can easily arise: multiple appended blocks are likely to require some data about their size.
Yes, my suggestion was to build that information into the appended data, instead of putting it into the header. Yes, as an answer it was a little bit oblique to the letter of B00daW's question, if you really want to interpret it that way, but I did honestly feel it was relevant to a larger context here. Sorry, if this is not what lidnariq wanted to talk about, but his response felt bafflingly and unnecessarily hostile toward it. I didn't initially want to make a big thing about it either, but he said some things I thought were worth responding to.
Yes also, the particular problem of these jaleco carts doesn't have this case, but I thought it was reasonable to suggest something that might ease the pain of having to renegotiate the format every time something new comes out. In particular the argument to restrict it to commercial era games only...!!! (I'm pretty certain B00daW is interested in being able to emulate new homebrew hardware ideas, and so are a lot of people.)
tepples wrote:
But unless the extensibility is carefully planned out, it isn't going to be easy to support on an NES flash cart such as PowerPak or EverDrive. Is it practical to make a UNIF loader in 1K of code on a 6502? Because that's all you get for each overlay.
I wasn't advocating UNIF, or even thinking about UNIF, though looking at its format now I am reminded that it does use a chunked format.
I don't think PowerPak's lack of support for UNIF has anything to do with the chunks. The hardest thing that would be hard to fit for code size is the insane number of unique board names instead of an iNES mapper number, but even that has feasible solutions, IMO. PowerPak doesn't have a UNIF loader because few people use it and bunnyboy or other capable people just aren't interested in it. Chunks themselves aren't hard or impractical to parse at all.
In this case though I was only suggesting chunks for appended metadata. Not UNIF, not MAME (yes, ZIP compression is a pretty significant problem), just what to do with some extra data which has hitherto been unavailable and thus ignored by emulators. The Jaleco data seems very "optional" to me, and chunks are easy to optionally ignore. (The current emulator solutions seem to be simulation via a collection of WAV files in the same directory.)
tepples wrote:
And as
I mentioned two years ago, chunked NES ROMs are going to be incredibly difficult to patch with things like IPS or xdelta3. They'd need a special-purpose patcher that might be slow to reach minority computing platforms.
Again, I wasn't advocating UNIF. If you want IPS compatibility, the PRG and CHR are still where they always were. We've been discussing how inadequate IPS is for a long time, though.
So... sure IPS wouldn't work on the appended chunks, if there's many versions of the ROM out there with different orderings (though this is not much different from the PRG variation problems we already have due to lack of a CRC check), but as a spitball idea the entire problem might also be solved with a "patch" chunk just appended to the file too; the patching process wouldn't need any special tool, just a simple concatenation!
tepples wrote:
rainwarrior wrote:
Similarly, if you don't want a ROM with MP3s in it, that's no problem for you... you don't have to download that ROM! Someone else would probably care about it though!
That's been true for only a few weeks. Before May 11, 2017, including support for MP3 in an emulator would have incurred patent liability.
MP3 was just a hypothetical example for the purposes of arguing about whether wild homebrew hardware ideas should be accommodated. I don't personally feel there's a reason to decide they should never be, or that they can't necessarily be planned for in advance, and that seemed to be the argument that Lidnariq was making.
The suggestion was really that I think it would be worthwhile to continue the iNES format in a more extensible direction if possible. Instead of having arguments about what the minimum possible amount of data we need to represent is, make it possible for users to add the things they need.
Anyhow, I'm not exactly trying to say that chunks are the only way to go, I really just wanted to float the idea while we're discussing possibilities. I do think
something belongs somewhere in the file to indicate how much extra data follows; even reserving 1 bit in the header for "has
something appended, see mapper definition for how to interpret" would be better than trying to use the length of the file itself, I think. Or if not in the header, some signature just before the appended data to verify that "yes, this is the new stuff".