I've spend the last few months on a format that would make virgin data playable by way of emulator-side mapper database. Nothing will ever become as popular as iNES 1.0 because it was the first to be good enough, which is how all standards are determined in emulation. But unlike iNES 2.0, I think this format actually has a chance to be
used by some people. And that's all we need to get the games preserved, correct boards and all. Supplied in the zip file is a prepared database and an explanation of the format, why it was made, and how it works.
http://www.zapatabase.com/ZapFC_Format_Guide.zip
A full converted library does exist and that's all I'll say about that. This topic actually seems to come up every year or so, the last time by etabeta. But no one seemed to get to the point with why headers are so disliked by some. From the guide:
Code:
This format allows virgin data to function by way of external database checksum lookup. The main problem with rom-side mapping like headers is its inability to effectuate revisions to the format. When an oversight is discovered to affect certain games, as it was with RAM quantity in iNES 1.0, you're offloading mapper update responsibility to users who won't do it with tools that don't exist.
The problem goes deeper than that. Distributed rom files are based on databases like GoodNES or No-Intro, so how they checksum files is a determining factor in whether the rom you download will play properly. GoodNES is 7 years out of date. And if you were to download the latest No-Intro set, you'd find that many of the roms have wrong or missing headers. No-Intro ignores headers in validation, probably because iNES 2.0 files would otherwise go unrecognized. But by doing so, they greenlight files that won't work in emulators. It's a double-edged sword with headers. You can either ensure data integrity or mapping integrity, but never quite both.
The ZapFC format places no such burden on users because mapping is 100% emulator-side. As long as the user has his roms in this format, game issues stemming from missing or inaccurate headers are no longer a possibility.
It's also a great format for dumpers. We would no longer have to use special tools with offset magic to obtain and compare checksums of just the CHR or PRG sections.
I'll also address the "unlicensed" issue since I know it will get brought up.
Code:
My stance on database inclusion is licensed, released games only. I exclude unlicensed games for the same reason that emulator authors don't bother emulating the endless streams of unlicensed add-ons and clone systems: they are a maintenance nightmare and I don't think we should be making inclusion decisions based on subjective quality appraisals. It's like Project Gutenberg trying to catalog everything from self-published books to the notes you took in math class.
We all know that it's not possible to include infinitely creatable material, and licensed games will suffer for it. The database is there if you want to extend it, but that quicksand will sink you like it did Cowering. The solution is to either (a) continue using headered frankenroms for unlicensed games or (b) give option on failed database lookup to add custom db entry with a mapping selector.
Then there's something neither iNES nor an internal db can accommodate, which is people who want to make not only their own games, but their own hardware. Since this is pretty niche, and would be a rom-side format prone to revisions, this should be its own format. Though I doubt if anyone truly cares enough to spend months preparing one.
As for betas of licensed games, I am more sympathetic and could add them in the future. But betas are one-of-a-kind and nearly impossible to verify. How do we know a beta was unmodified and properly dumped?
Anyway, there aren't too many active NES emulators left. Not sure if anyone will support it, let alone see things the way I've come to see them.
iNES successor proposals are like * everyone (here) (probably) has one.
*SMB/DH
A database with all the official games is a great idea, as we can guarantee they will at least have the correct information supplied, however, I don't think the format you're proposing is the way to go. We don't need a format exclusively for licensed games. It would be better to find a universal format that supports both with headers (Unlicensed and homebrew) and without (Licensed games that use a database).
Personally, I don't generally like databases and cheksums (No database = no play), but I support the idea to at least have it available for those who want to use it.
While I like the idea of zip and separate PRG and CHR files because it makes it easier to edit them, I don't see any reason to have them separate just for the sake of having them separate. If your file format doesn't allow for them to be edited, I don't see any reason to keep them separate. Also, you must keep in mind that not every platform may be able to unzip and generate the checksum within a reasonable time due to some crazy restrictions (Like slow processing power or very little memory).
A new file format must also be truly portable for it to be used, thus I would like to see a hybrid of both headers and database (If the database isn't available, it can fall back to the header with the ROM).
However, unlike some people here, I support moving to a new file format, but it must be able to have enough information to be run as a standalone ROM, and it must be portable enough to support slow and memory restricted platforms. The impression I get from your suggested file format it that is lacks some these things I'd like to see in a file format.
Perhaps one should look to other systems that have specific setups for a given ROM/disk image (VIC-20 comes to mind as some programs only run in certain RAM configurations, and fail if you have all possible expansion RAM installed) and draw some inspiration from there? I'm more of a C64-head so I don't really know how the VIC-20 emulators implement this (or if they just fob it off on the user). The NES is unique in this regard for it's widespread use of external memory mappers, so I'm sort of torn in the middle. On the one hand, since most header settings are unique to a ROM (or class of ROMs that use the same board), I can see the argument for keeping everything with the ROM. On the other hand, you make some valid points for not storing the header information with the ROM.
Unfortunately, I truly fear it may be too late to even make a dent regarding this issue. iNES 1.0 is so widespread that Nintendo use it in their own products (same is true for the other VC-emulated systems, AFAIK most if not every one of them uses an emulator "scene" standard to store the actual ROM data).
What's so bad about a 20 byte header that then offers offline play without anything else? How this would be a good idea? Yeah it's interesting and may work okay (What emulator has an internal database of checksums again?) but it won't be any better then the iNES header and will be worse for other situations like homebrew, what is supposed to happen in between compiles and revisions and such?
It's not a very good idea at all. iNES is the standard for a reason, it's the best way to do it. Why are there attempts to even change it?
Mind=BLOWN
These solutions oriented to "software preservation" (i.e. piracy) are a joke. It's been over a decade since the last licensed NES game was released and people are still looking for ways to properly catalog them.
I think there's no point in coming up with solutions that solve only half of the problems. If you are going through the trouble of creating a new way to describe ROMs, please create something that will work for 100% of the cases, including homebrew.
I would very much appreciate a format that allowed for hardware customizations, such as describing the functions of individual chips and specifying how everything is connected on the board, so that we wouldn't be restricted to built-in mappers. This would also solve a very big problem in the homebrew scene, because nobody wants to program games for mappers that don't exist and nobody wants to create a mapper for which no games exist, and maybe we'd finally have interesting and easy to produce homebrew mappers.
tokumaru wrote:
I would very much appreciate a format that allowed for hardware customizations, such as describing the functions of individual chips and specifying how everything is connected on the board, so that we wouldn't be restricted to built-in mappers.
This would require more than a new format, it would require a new way of writing emulators (I'm under the impression that emulators just load the ROM into memory and assigns pointers to the current bank). It also wouldn't be backwards compatible. But this is a very interesting idea. Making it possible to "wire" the chips together as you'd like is possible, but I'm unsure how you would specify the function of the chips. Do you mean making new chips, or selecting from the already existing chips? Care to explain (Possibly in a new thread)?
Quote:
I exclude unlicensed games for the same reason that emulator authors don't bother emulating the endless streams of unlicensed add-ons and clone systems
An emulator that can't run Tetяis, Klax, Spiritual Warfare, and the Battle Kid demo is an emulator that doesn't run notable commercial games for the NES. So emulators would have to support iNES and NES 2.0 anyway.
Quote:
Then there's something neither iNES nor an internal db can accommodate, which is people who want to make not only their own games, but their own hardware. Since this is pretty niche
It happens to be the niche chosen by users of this web site.
Quote:
and would be a rom-side format prone to revisions, this should be its own format. Though I doubt if anyone truly cares enough to spend months preparing one.
Kevin Horton cared enough, and I fully support his NES 2.0 format. It accommodates homebrew fairly well: it expands the "mapper" field to 12 bits and introduces a 4-bit "variant" field. It also expands the range of PRG ROM, CHR ROM, and save sizes that can be represented. Besides, most notable homebrew games appear to be on mapper 0 (NROM), 3 (CNROM), 7 (AOROM), 2 (UOROM), or 1 (S*ROM), which are the mappers supported by retrousb.com ReproPak kits.
Wkter wrote:
While I like the idea of zip and separate PRG and CHR files because it makes it easier to edit them, I don't see any reason to have them separate just for the sake of having them separate.
That's what zip is for. Wrappers using the PKZIP format, such as jar, smzip, odt, and MAME zip files, allow the ROM sections to be simultaneously together and separate.
Quote:
Also, you must keep in mind that not every platform may be able to unzip and generate the checksum within a reasonable time due to some crazy restrictions
Only Game Boy Advance really has a problem with that. Nintendo DS can checksum a whole 2.5 MB DS Download Play client binary in order to check its digital signature.
Wkter wrote:
Making it possible to "wire" the chips together as you'd like is possible
The true way to represent a custom mapper would be to create a netlist in schematic capture or in some sort of HDL, and then simulate that netlist for
every memory access. That probably won't run in real time on a smartphone or an Atom laptop.
tepples wrote:
Only Game Boy Advance really has a problem with that.
Exactly the system I had in mind.
tepples wrote:
The true way to represent a custom mapper would be to create a netlist in schematic capture or in some sort of HDL, and then simulate that netlist for every memory access. That probably won't run in real time on a smartphone or an Atom laptop.
Why stop there? Let's make the custom mappers
like this.
But seriously though, it should be feasible to emulate the wiring of the cartridge by simply adding a few more pointers and emulating the chips standalone. It would of course slow down the emulation to the point where it's unplayable by the really slow systems. But we're getting off-topic here.
Maybe if emulators could instead all make mappers custom mappers, except iNES come defualt, so then people could inject their C code into the mapper file that they decide to make their mapper and then the emulator would compile it when the time comes and then could run it? I don't know enough about C to know if it is possible, but it's an idea, right? That'd allow not-too-much hacking to emulators, correct? And it'd allow for "custom" hardware. Win win!
Going out on a limb here....
Wireshark, tcpdump (linux, bsd), snoop (solaris) all do the same thing. They sniff network packets. You can tell it to filter the traffic and only show (or record to a file) specific packets. This is expressed in a language called "bpf", for "Berkley packet filter". BPF is compiled into a byte-code that is interpreted by libpcap (the library that tcpdump and wireshark use - idk what its called in Solaris). It is possible that some network stacks will handle BPF in kernel mode (to avoid the overhead of kernel/user mode switching). High-end network gear offloads the packet filtering to hardware (asic or fpga or dedicated co-processor on the NIC).
I mention this because the fundamental idea is that a high-level language, like BPF, is turned into an efficient byte code ran inside a VM (or JIT compiled to native code) (btw, bpf is NOT Turing complete).
Maybe someone with a great deal of brains could design a "mapper language". Not as low-level as a net-list. Maybe it implements a logic-gate level abstraction, maybe a higher level abstraction, like "if/then/else" and some integer array for maintaining state. The iNES format can be extended to flag the existence of this "mapper implementation", and it would be appended to the ROM after the (optional) char-rom segment.
I lack the time right now to go into greater detail, but I can envision how to implement MMC1 and NROM this way (in fact, NROM is just a null statement, kind of trivial).
I like this approach (I did my senior "capstone" college credit work on a compiler / language translator), so it appeals to me. Unfortunately, I lack the time to devote to making a reference implementation right now.
Any thoughts?
http://en.wikipedia.org/wiki/Berkeley_Packet_Filter
http://www.tcpdump.org/papers/bpf-usenix93.pdf
(edit, added content)
Wow. I never actually finished typing up my thought. Emulators could be adapted to implement the mapper lang -> byte code conversion, and byte-code VM. The VM would be receive the same input info as a real cart would (addr bus, data bus, ctrl signals). It would be capable of implementing a small state engine that would determine what gets mapped when, and when to fire IRQs. This layer within the emulator would handle all of the reads and writes to the cart. It would need access to the RAM allocated in the EMU that holds the contents of the RAM and ROM chips on the cart.
This might seem like a crazy idea, and indeed the lang -> byte_code "compiler" would be a bit of work (maybe ~1000 lines of C code?), but a fairly efficient VM can be constructed to handle this. A modern PC running an emulator should have NO problems maintaining a proper frame-rate.
I would also like to see an emulator that exposes, via its debugger, the internal state of its current (or future) mapper implementation. For example, when debugging a MMC1 game in FCEUX, it would be nice if I could access to pop-up window (debug -> cart mapper) that would show what pieces of ROM are currently switched in, and the current state of the MMC1 latch
Damn, I wish I had time to work on this. Maybe later this summer.
3gengames wrote:
How this would be a good idea? Yeah it's interesting and may work okay (What emulator has an internal database of checksums again?)
None? Why do you think I'm proposing it? What emulator guarantees correct mappings for downloaded games? None. You offloaded that responsibility to rom site morons who could care less about mapper integrity. They don't even care about rom integrity. If we could supply roms with emulators, we would, but we can't. Because most distributors are morons who know nothing about the files they're distributing. GoodNES has hundreds of bad dumps and mappings. I would know, I converted as much as I could from that set when I started mine.
Quote:
but it won't be any better then the iNES header and will be worse for other situations like homebrew, what is supposed to happen in between compiles and revisions and such?
If that was true, I wouldn't have spent the time that I did. You are as clueless about the integrity of your files as the average user. How many ghost bugs have emulator authors chased down due to bad or missing headers? How many will future ones chase down? Emulator-side mapping is easy to implement and an instantaneous guarantee for everyone, user and author alike. And all I hear is complaining about having to use a different format for unlicensed games. Really? Wallow in it, then.
Wkter wrote:
While I like the idea of zip and separate PRG and CHR files because it makes it easier to edit them, I don't see any reason to have them separate just for the sake of having them separate.
Let's say you try to verify your files against a trusted source, and you get a checksum mismatch. Is it because of a bad mapper, a bad chr, or a bad prg?
Let's say you dump a new game. Once you combine it and compare its checksum to the existing dump, and the checksum is different. Is that because the chr is different, the prg is different, or both?
Combining everything into a headered frankenfile makes it hard to find the reasons for differences, because any one of the portions could be to blame. As an archivist or a user just trying to verify his files, that's what's annoying about it.
FitzRoy wrote:
3gengames wrote:
How this would be a good idea? Yeah it's interesting and may work okay (What emulator has an internal database of checksums again?)
None?
Nope, there's an emulator that does this already. I don't recall which one though. Too late?
I have yet to download a bad ROM from any site, so why is this non-issue being used as the best reason for it? Ughhh...the yearly lets change iNES to something else thread, yay.
3gengames wrote:
Nope, there's an emulator that does this already. I don't recall which one though. Too late?
I believe this to be Nestopia, if I'm not mistaken. jwdonal is also making his FPGA NES with the ability to look up BootGod's database.
FitzRoy wrote:
3gengames wrote:
How this would be a good idea? Yeah it's interesting and may work okay (What emulator has an internal database of checksums again?)
None? Why do you think I'm proposing it? What emulator guarantees correct mappings for downloaded games? None.
NEStopia has support for this. Options -> Database. It supports both internal and external (a file) databases. I imagine all it actually reads is the GoodNES database (I'd have to read the README to verify).
Anyway, to circle back to the original point: for what it's worth (not much), I'm an opponent of replacement file formats for NES ROMs. The simple reason that what came first (the original iNES format) is what will stay; it doesn't matter if your format is superior, has many advantages, solves tons of problems, or even complies with the Right Thing(tm). While a select few care about such, the rest of the world (read: majority) just want to play games. Existing emulators and tools already read the iNES format, and that's pretty much the end of it. I do, however, agree with approaches like the iNES 2.0 format (which retains existing compatibility).
If you want a perfect example of an attempted iNES format replacement that failed, look at UNIF. If you think I'm passing judgement, I can try to dig up a private mail I received from Tennessee Carmel-Veilleux talking about how such an effort (on his part) was truly pointless, and he actually regrets doing so -- and as such, asked that I remove all references to his efforts (yes really). I say this as humbly as possible.
This isn't to say your efforts are worthless, unnoticed, or unappreciated. It's that ultimately at the end of the day nobody is going to bother switching to a new format.
You can try something like
what bSNES is doing (refusing to support all sorts of ROM formats, particularly .SMC which is used by the majority and instead insisting everyone use .BIN), but all that will do is alienate you (or in the case of bSNES, the emulator) from the rest of the world. Again: it doesn't mean the approach is wrong, but is wasted effort at this point in time.
EDIT: And yes, I'm well-aware that the discussion is with regards to a ROM database or equivalent.
FitzRoy wrote:
None?
I'm pretty sure Nestopia does.
Quote:
Why do you think I'm proposing it?
I don't know... But I'm convinced that what this community doesn't need is a half-assed replacement for iNES that only works for old licensed games. Note that by "half-assed" I don't mean that you didn't put any effort into this, because I'm sure you did, what I mean is that your solution is far from ideal, and barely improves the current situation.
Fitzroy, I understand your proposal, and its merits. However, as a homebrew developer, I would find it rather annoying to have to update my database after every build just to test my games. If the emulator supported by iNES and your scheme, then fine.
These are a few issues that I see:
- Who keeps the database up to date?
- Does anyone "own" it? I imagine that since it catalogs mostly ripped commercial games, the creator(s) of database entries are not too concerned with IP ownership rights.
- What happens when the database fragments, forks, branches or whatever?
- How would an emulator detect if it has the "correct" database (ie, the one shipped or supplied by the user might still have incorrect mapper info).
- Would it include checksums for hacks, like "Zelda Outlands" and some metroid edits?
- What about games that I hack myself? I guess that these are no different than uncatalogued 'homebrew' from the point of view of the emulator.
- The problems that lead to the wrong mapper being stored in a ripped iNES format image... How do you prevent the same incorrect data from making it into the database?
I think that it is better to fix the iNES header (like iNES 2.0 is trying to do).
Maybe it is better to publish a program with a built-in database that just analyzes one's ROM collection. It should checksum the prog-rom and char-rom, look up the correct ines header, and just fix the image in-place. For any set of checksums not in the database, leave the file alone, but generate a warning to the user. A user could run the tool whenever they add a rom to their collection and not worry about it any further.
Yeah, not to come off mean too (Sorry I did.) but this crap of a new format always comes up every 6 months or so and is just dumb to replace a perfect format that is also expandable with something else that isn't even compatible with so much of the library won't even work on it is just.....bad. Agreed it's not a bad idea, but there's better things out there and this is already being used.
koitsu wrote:
You can try something like
what bSNES is doing (refusing to support all sorts of ROM formats, particularly .SMC which is used by the majority and instead insisting everyone use .BIN), but all that will do is alienate you (or in the case of bSNES, the emulator) from the rest of the world.
Why do everyone want to use force? A much better approach to a new format would be if we got the popular emulators still in development to open a dialog box asking "This file is using an older format. Would you like to convert it to the more recent .THE_BEST_NES_FORMAT format?" with a "Yes", "No", "Always" and "Don't Ask Me Again" options. Obviously, if we can't get the popular emulators to agree on this there is no reason to even think about a new format.
tokumaru wrote:
I'm pretty sure Nestopia does.
Nestopia's database feature requires roms be in headered frankenfile format. There was never any effort made to make virgin data playable like every other system, only to override the mapping information in the header. That you have to look at the mapper or the database to even know rom sizes is ridiculous. The files and header never needed to be appended because of this thing called containers. So it's just added work adding those values to the header or database. I suppose we're lucky every iNES file wasn't interleaved to boot. You can certainly tangle for the sake of tangling as long as you know how to untangle at load time.
Quote:
I don't know... But I'm convinced that what this community doesn't need is a half-assed replacement for iNES that only works for old licensed games. Note that by "half-assed" I don't mean that you didn't put any effort into this, because I'm sure you did, what I mean is that your solution is far from ideal, and barely improves the current situation.
But it's not a replacement, it's a feature addition, and I made that quite clear at the beginning of this. A finite library can take advantage of an internal db's merits. I'm sorry that your infinitely broad definition of the NES requires external files, but
the NES library sanctioned by Nintendo doesn't.
Quote:
I'm an opponent of replacement file formats for NES ROMs. The simple reason that what came first (the original iNES format) is what will stay; it doesn't matter if your format is superior, has many advantages, solves tons of problems, or even complies with the Right Thing(tm). While a select few care about such, the rest of the world (read: majority) just want to play games. Existing emulators and tools already read the iNES format, and that's pretty much the end of it. I do, however, agree with approaches like the iNES 2.0 format (which retains existing compatibility).
Except it's not a replacement file format, and it's ironic that you would use this logic when game development for dead platforms is just as limited in its ability to reach the masses. The only difference is what I'm doing actually makes some sense and improves the library for history's sake. People here could make a game that looks and sounds just like an NES game in half the time for Windows and get 100 times the audience. But to them, maximum popularity is not as important as tinkering with their childhood system.
koistu wrote:
but all that will do is alienate you
Good, who minds being alienated from sheeple? It's a puzzling extension that stands for a chinese copier. If a letter in your file is what's keeping you from using bsnes, you ought to take the blue bill for life.
Quote:
# Who keeps the database up to date?
Me. If anyone actually uses it, I'll make future revisions a public download on my site.
Quote:
# Does anyone "own" it? I imagine that since it catalogs mostly ripped commercial games, the creator(s) of database entries are not too concerned with IP ownership rights.
No, it's just information. You can do whatever you want with it
Quote:
# What happens when the database fragments, forks, branches or whatever?
I dunno, why would it? To add certain unlicensed games? Fine by me. If they were smart, they'd put all the unlicensed at the bottom so that they could copy my updates over without overriding their own additions.
Quote:
# How would an emulator detect if it has the "correct" database (ie, the one shipped or supplied by the user might still have incorrect mapper info).
Well, if you replace the supplied database with someone else's, you run those risks, don't you?
Quote:
# Would it include checksums for hacks, like "Zelda Outlands" and some metroid edits?
No, because it's unlicensed. But most mods of licensed games can be soft-patched.
Quote:
# What about games that I hack myself? I guess that these are no different than uncatalogued 'homebrew' from the point of view of the emulator.
Any modification generates a new checksum, so you'd have to use iNES until it's finished and then create a patch. Patches don't interfere with checksum lookup.
Quote:
# The problems that lead to the wrong mapper being stored in a ripped iNES format image... How do you prevent the same incorrect data from making it into the database?
Because roms have countless suppliers, but an author can go to my website and know the database there comes from me.
Even if ther 20 byte header is corrupted (DISKDUDED), the purpose of the NESTopia database is to fix it. >.>
And not featuring unlicensed hardware is pretty lacking. Especially since right there it can't support the NES library right there while iNES can.
You can go head and do this, once you write an amazing emulator that uses this and convert every ROM to this format, sure. Until then, good nobody will use this. Sorry, just trying to save you work.
What schema do you propose for your database?
If your project were a tool that "corrected" the ines 2.0 header of ROMs, then I would imagine something like this:
Code:
-- sqlite syntax, http://www.sqlite.org/datatype3.html
create table sometablename (
progrom_xsum text not null, -- md5 checksum, as printable text
charrom_xsum text, -- md5...
title text not null,
ines_hdr blob(16), -- correct ines 2.0 header.
updated text, -- date in yyyymmdd format
author text,
constraint pkeyname primary key (progrom_xsum, charrom_xsum)
);
Or are you suggesting that your ROM images would have NO 16-byte header and just be the raw progrom + charrom (if present), and that the sizes of the ROMs would be determined by the checksum of the entire blob?
Then maybe:
Code:
create table sometablename (
xsum text primary key not null, -- md5 checksum, as printable text
title text not null,
updated text, -- date in yyyymmdd format
author text,
emu_mapper text -- or possibly an integer?
);
I would use a tool that operated like the first (keep ines header, and tool just fixes roms).
What "database" file format would you use? csv, raw binary blob (typically an array of structs), custom text file format? sqlite container (really nice option, imho), or what? Just please don't say OLE container.
If you chose a printable text file format, I would suggest that each ROM be on a line by itself, but that the entire record not span multiple lines. This makes mgmt by grep, sed, awk, etc... much easier.
What kind of implementation were you thinking of?
I think all successor formats fail because emulator authors don't need to thus don't care to implement databases, XML parsing or a universal netlist mapper interpreter (which will probably need dynamic linking and massive reworking of existing code). Some emulators don't even have ZIP support.
Preservation people pedantically fight for straight binary, despite the mapping ambiguity it causes on many platforms, because anything else is an eyesore to them. It's not wrong, but it really sets back creation on the NES since there's no default mapping to fall back on with games greater than 40 KiB. Imagine if every bitmap image required a separate description file or database... Since even preservationists will agree creating new games is good, some board info must be packaged with even the most straight-forwardly programmed NES ROMs. Since the current iNES 1.0 implementation works for 99% of the library, where's the motivation to move on?
All of its shortcomings (bugs in one-off mapper games) are addressed via iNES 2.0 and the few discrete mapper ROMs with broken mirroring (oh my!) in the wild can be fixed with a database powered renaming and header-fixing tool. As of now this has become just another bike shed debate, iNES 1.0 beat out all the competition fair and square and it gets the job done, kind of like TCP/IP. New proposals aren't even BETTER, they're just different in their philosophy following the author's agenda.
As for the netlist mapper, this is something I played with (and the reason I started an emulator years ago). I quickly lost interest because it had no hope of catching on and I was too OCD to follow through on the engineering side. Later I came to the belief that anyone developing their own hardware/mapper can just as easily implement it in a popular open source emulator for testing purposes. Also I now believe that game developers shouldn't go out of their way to make a custom mapper just for the sake of making a custom mapper. I believe if they do, and their game is good, the mapper will be supported by emulators naturally.
Quote:
These solutions oriented to "software preservation" (i.e. piracy) are a joke.
Wow, what a terrible attitude, using your snarky passive-aggressive quotation marks like that.
These games are twenty-plus years old. You cannot buy these actual games from the manufacturer because they are no longer made. You cannot even get them in new condition unless you pay $500+ on eBay per game. These games are going to physically die, forever and ever, in probably 20-50 years, well before even the current copyright laws will allow them to enter the public domain.
People are entitled to their own culture, the public domain. Or should we re-copyright Shakespeare's works so that his great-great-great-great-great-great-great-great-great-great-great-great-great-granddaughter can whore out his work some more in the name of capitalism? It's a sick travesty that our governments are basically for sale to the highest corporate bidders, and evil, greedy pieces of shit like Bono and Disney can extend copyright well beyond the lifespans of the longest-living human beings, such that nothing ever created in our lifetimes will be available to us. All so that Bono can have one more gold-plated swimming pool.
Piracy my ass. I would have no problems paying the original manufacturers for new physical copies of games. The only thing I can spend money on is a game rental from the Wii Virtual Console that only works on a shitty, half-assed emulator that makes NESticle look good, and even then for only 1% of the library.
If the copyright holder is not selling the work anymore, it is not piracy. Just like downloading an MP3 is not theft. Saying it is just makes you a corporate apologist.
I further agree with actual researchers like Rufus Pollock, who indicate the optimal length for copyright is 14 years.
It is
beyond insulting for you to call my work piracy. Saying I just want free games? Look at these pictures:
http://i73.photobucket.com/albums/i221/ ... C00834.jpghttp://i73.photobucket.com/albums/i221/ ... C00835.jpgI spent $8,000 buying every last USA SNES game ever released. Used. I have to open them up and scrub every single PCB board to get decades of filth off of them. You think I'm just looking for free ROMs? Fuck you.
The actual pirates couldn't give a shit less about preservation. They just want free games and they don't care how they get them.
Quote:
You can try something like what bSNES is doing (refusing to support all sorts of ROM formats, particularly .SMC which is used by the majority and instead insisting everyone use .BIN)
.bin? I require .sfc, which stands for Super Famicom. I disallow the Super Magicom extension because I am not emulating a copier.
Quote:
but all that will do is alienate you (or in the case of bSNES, the emulator) from the rest of the world.
What's funny is that it really didn't. I removed fullscreen, ZIP/7-zip multi-archive suppot, rewind, movie recording and playback, integrated software filters (as opposed to plugins), multi-key assignments (shift+N = player 2 B), GUI shortcut key assignments, IPS support, auto and manual turbo fire support, pseudo-widescreen zoom (near aspect-correct scale), video cropping, custom path support for every type of file, OS-native file load dialog, region override, copier header support, 22 different ROM header extensions, drag-and-drop, etc.
Did all that months ago. And you know what? Latest release has 10,000 downloads. That's fairly standard. I include a small, 50KB GUI utility. You point it at your ROM folder, hit scan. It tells you what it will do to your ROMs, hit OK. You are done.
To quote Dr. Seuss, "those that matter don't mind, and those that mind don't matter."
Quote:
Why do everyone want to use force? A much better approach to a new format would be if we got the popular emulators still in development to open a dialog box asking "This file is using an older format. Would you like to convert it to the more recent .THE_BEST_NES_FORMAT format?" with a "Yes", "No", "Always" and "Don't Ask Me Again" options. Obviously, if we can't get the popular emulators to agree on this there is no reason to even think about a new format.
In my case, I didn't want the legacy crap polluting my codebase. I am very philosophically opposed to even keeping it around. My choices will never have any mainstream impact anyway, so why bother? I agree, to do something like this properly would need the popular emulators to offer conversion support right inside the emulator. Start for 1-2 years with a simple "hey do you want me to fix this game? Yes/Always/No/Don't ask again". Next two years, "hey I need to fix this to play it. Yes/Always/Don't play". After that, 1-2 years of auto-fix. And then finally, drop the legacy support.
Still think it's impossible? Most early SNES ROMs were interleaved for copiers. Try and find interleaved games now. Change is possible, you just have to get past the emu authors themselves who believe it is not.
-----
As for the actual new NES format idea ...
The particularly nice thing of my approach is that images that run in bsnes also run everywhere else, but that was not a requirement of mine, just a happy coincidence. We would need something like an nespurify tool, and it would have to make copies rather than convert images, at least by default.
We had something special in the SNES scene. Unlike the NES, where there are more emulators than there are stars in the sky, there are really only two significant SNES emulators. Get two to prefer and auto/manual convert to the new format, and this would be a done deal. It would solve a dozen problems and hurt nobody. ROM sites would update when all of their users asked them to.
But even with two projects, only two, maintaining popularity at the expense of preservation (aka "piracy" amirite guys lolz) is more important.
-----
FitzRoy, I appreciate what you're doing here. I am very much in favor of pure, unadulterated images. I prefer allowing for the possibility of per-game XML mappings of the cartridge hardware, but am all for an internal database for commercially released images for easy fixes/updates.
Unfortunately, I have to agree with everyone else here. With their attitudes, and these are the emulator authors themselves who have suffered from iNES problems personally, you stand no chance of convincing anyone to use this new format. The only thing I think you can do is pay someone (not me :P) to add support for your format into a one-off build of eg Nestopia.
I say, do it anyway. The right thing is the right thing, even if it isn't popular. But be prepared for the entire world treating you like an idiot, an asshole, a zealot trying to force them to change against their will, and to be completely and utterly ungrateful for what you're doing.
byuu wrote:
You think I'm just looking for free ROMs? Fuck you.
No need to be so offended, I didn't attack anyone personally, and certainly not you. The thing is that just like you are offended to be called a pirate, I am offended by people neglecting homebrews as important pieces of software. I mean, it's not like we're gonna lose any NES games, I'm pretty sure they've all been dumped a hundred times over, they are preserved. Now, we live in a time where several people are working on interesting NES games again, some have been finished and hopefully many more will be soon. Yet, some people are still so focused in the past that we keep seeing these shortsighted solutions that don't care about the current developments.
I'm sorry if I perceive this obsession with properly supporting exclusively the old licensed games as a desire to own every bit of commercial software ever rather than actual love by the platform itself, as we could very well be about to witness a big revival of the NES thanks to the new software being made for it, but it looks like the software preservation people just don't care, since they are still coming up with stupid ROM format solutions that do not take new developments into consideration.
Yeah sorry, you kind of implied that all people who preserve software are just into piracy; which is the notion I was saying 'fuck you' to. If that's not what you meant, then that's not what I meant toward you. Software preservation = piracy is the opposite of reality. As an emulator author, that's a fairly major insult.
The pirates I see are the guys wanting the GoodTools ROM sets with the seven thousand header variants and known-bad dumps and juvenile "Super Penis Bros" hacks included. Those are the guys who stuff everything into 7-zip multi-archive packs and seed it around on torrent sites.
Preservation implies all games, so I'm not sure how I can defend against the fact that preservationists do want all games. Yeah, that's kind of the point. But I'd still prefer the guy that has all 2,000 games for the purpose of making sure they are not lost to the sands of time; over the guy who downloads every game he personally wants to play. Both of them are technically committing copyright infringement, but the former is potentially doing something much more important.
But I should also say, I don't think many of us hate homebrew/fan translations/etc right off the bat or anything. Hell I've spent nearly a decade fan translating RPGs. It was a big goal of mine personally to use an XML mapping that could be applied to any homebrew game. I've even prototyped custom chips to help make homebrew even better.
The reason you see less focus is because: a) you cannot dump and verify a software-only ROM image, and b) unlike official games, which is a finite set, homebrew is an infinite set that can theoretically be added to until the end of time. It's disheartening to work on a project that you know can never come to completion.
The unfortunate thing to me is that many homebrew authors don't even seem to care about preservation of their own works. They often only test on the most popular and least accurate emulator, create software that won't run except on said emulator, and some even hack up emulators to support extra features that are impossible on real hardware. What am I supposed to do?
I am all for archiving homebrew, but there are some really tough problems with it. Official games are at least of some degree of quality that they were published and sold. Those juvenile SMB hacks? I really don't care if those are never preserved, sorry. Great SMB hacks? It would be a shame to lose those.
So how do we preserve it? I suggest an XML-based mapping system. Yes, it has the unfortunate problem that everyone will have to support said format, and it's damn-near impossible to get those formats 100% correct. You're going to find a game that pushes the limits, and you have to extend the spec, and now you have competing implementations?
I don't think it's a 'fuck homebrew' attitude, but 'how in the world can I preserve homebrew even if I wanted to?' attitude.
It just seems silly to argue against better, bit-perfect, heuristic-free, hacky made-up-header-free support for official games of a known, finite quantity. If the status quo for fan translations and homebrew is iNES, this effort does not have to negate that. By example, I still have the old manual header parsing stuff myself, for those homebrew images that lack an XML mapping as well.
But I'm not going to let official dumps be mediocre just because I cannot catalog an infinite number of homebrew games.
EDIT: so what we want is clean images without headers, and what others want is homebrew. Why not make the iNES header a separate file, gamename.nes + gamename.ines. Stick it in a ZIP file. Emulators from here can properly checksum games, and use internal database info, and fall back on the iNES headers as needed. Databases are free to add homebrew checksum entries or not. Homebrew authors include the .ines file regardless. Just another theoretical idea that will never happen because nobody will do it, but it seems reasonable to me.
What's wrong with the 20 byte header? This question still hasn't been answered!
Not like adding 20 bytes doesn't destroy the ROM, it's there. Extract it! And with that format you also need two files, which lugging two files on my PC instead of insta-clicking one and is brough up in FCEUX perfectly is annoying. You'd need a linker filer to tell what two ROMs to use anyway.
3gengames wrote:
What's wrong with the 20 byte header? This question still hasn't been answered! :P
For one, it's not twenty-bytes.
Quote:
And with that format you also need two files
Yeah, you certainly wouldn't want to run out of file slots.
Quote:
You'd need a linker filer to tell what two ROMs to use anyway.
filename.zip = { filename.nes + filename.ines }
Eyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy.
Yay Pasofami format! Separate PRG, CHR, and PRM files. Still have no clue what's in the PRM file.
3gengames wrote:
What's wrong with the 20 byte header? This question still hasn't been answered!
It's 16 bytes actually. There are a lot of things wrong with iNES, it can't properly describe a number of existing games because it doesn't allow for things like more than 8KB of CHR-RAM, more than 8KB of SRAM, mix of CHR-RAM and CHR-ROM, less than 16KB of PRG-ROM, things like that. Some of those problems were solved by assigning mapper numbers to these "weird" games, but that's just a hack really. iNES 2.0 solved all of these problems I think, by introducing fields that control those previously overlooked characteristics, but it still only describes existing hardware, which means that creating new mappers is difficult.
For old commercial software preservation I believe iNES 2.0 is OK though, but the problem is that there are already too may ROMs with bad header floating around, and you can't force everyone to update their ROMs, so some people think the solution is to change the emulators. I hardly thing this is a solution, because an incredible number of people (the average pirate Joe) is still using outdated emulators.
For preservation purposes, I say the solution is to manually fix all ROMs with iNES 2.0 headers and make that the "official" preservation package. If people want to download it, fine, if not, screw them. Those who are interested will know that those are the cleanest ROMs possible.
byuu wrote:
koitsu wrote:
You can try something like what bSNES is doing (refusing to support all sorts of ROM formats, particularly .SMC which is used by the majority and instead insisting everyone use .BIN)
.bin? I require .sfc, which stands for Super Famicom. I disallow the Super Magicom extension because I am not emulating a copier.
As I understand it, BIN and SFC are the same thing -- they are raw ROM dumps with no headers or footers/tails (aside from the official Nintendo-sanctioned ROM information that's retained at the end of the first bank; I forget the 24-bit address for it, something like $00FE00-FEFF, or $C0FE00-FEFF).
In turn, this makes me wonder why you don't support (non-split) .SMC files, choosing to ignore the first 512 bytes of the file thus effectively ending up with a BIN/SFC. I mean, is a single lseek() call really that hard? As for the other formats (MGH, etc.), sure, I can understand dropping support for those given that they're truly legacy, but most people honour SMC for the same reason they honour the NES format -- it came first. These are what you'll find across the entire Internet, and even on CD (yes, I had a few organisations send me ROM CDs back in the day, they're all SMC or split-file SMC).
byuu wrote:
koitsu wrote:
but all that will do is alienate you (or in the case of bSNES, the emulator) from the rest of the world.
What's funny is that it really didn't. I removed fullscreen, ZIP/7-zip multi-archive suppot, rewind, movie recording and playback, integrated software filters (as opposed to plugins), multi-key assignments (shift+N = player 2 B), GUI shortcut key assignments, IPS support, auto and manual turbo fire support, pseudo-widescreen zoom (near aspect-correct scale), video cropping, custom path support for every type of file, OS-native file load dialog, region override, copier header support, 22 different ROM header extensions, drag-and-drop, etc.
Did all that months ago. And you know what? Latest release has 10,000 downloads. That's fairly standard. I include a small, 50KB GUI utility. You point it at your ROM folder, hit scan. It tells you what it will do to your ROMs, hit OK. You are done.
The number of downloads doesn't mean crap. Validation: the last time I downloaded and ran bSNES 0.74 (or newer) and pointed it to a directory of ROMs that contained .SMC files, it returned absolutely no results (no files shown). I immediately clicked [X] and deleted the bsnes\ folder, going back to 0.73. So, I downloaded it, but I found the file format tirade you're on to be not worth the effort to deal with. I am not alone in this matter; most of my peers who I've pointed your User Guide to responded with "What the FUCK?" when they read the tirade. So the word "alienate" is accurate here.
Furthermore, like the rest of the world, I am not going to run a ROM conversion utility because a single emulator author is on such a tirade. What makes you think I want to run all of my games in just bsnes and not a multitude of emulators (yes, I do this! I tend to use SNES9x, ZSNES, and bsnes)? Furthermore, I have a SWC DX32 and a working SNES; maybe I want to send some games to my copier via the parallel port transfer utility... oh now look what I have to do, convert all the games from SFC to SMC using UCON or doing it by hand. Then I get to do this again when........ you get the idea. I should point out that the directory is filled with ~400 files.
I say all this with as much sincerity as possible. I don't want to argue or fight, honestly, but this tirade against file formats is silly. The motives are positive, but it's too late and all you're doing is what I described: alienating people (users of your emulator).
Sorry, I didn't intend to start ranting here. It sounds like you didn't read my user guide at all. I should just tell you to re-read it, but I'm bored at the moment.
Quote:
As I understand it, BIN and SFC are the same thing -- they are raw ROM dumps with no headers or footers/tails (aside from the official Nintendo-sanctioned ROM information that's retained at the end of the first bank; I forget the 24-bit address for it, something like $00FE00-FEFF, or $C0FE00-FEFF).
All extensions are the same thing. Idiots rename the extensions all the time. Most images out there are headerless files stored with the SMC extension. There are headered SFC files. The only people insane enough to use the BIN extension are certain MAME dumpers/devs.
I don't know what you're talking about with footers, and of course I keep the internal information at $fc00, that's part of the official ROM data. Copier headers are a floppy disk loading trick to make HiROM/LoROM header parsing a tiny bit quicker.
Quote:
In turn, this makes me wonder why you don't support (non-split) .SMC files, choosing to ignore the first 512 bytes of the file thus effectively ending up with a BIN/SFC.
For the same reason I don't support 2048 byte headers, 6000 byte footers, the WTF extension, the GCA compression file format, native SMB and internal ReiserFS support for loading ROMs, interleaved images, games that are XORed against pictures of Henry Winkler, and so on and so forth. I am not wasting my time with meaningless concepts just because people are ignorant about it. I don't care how popular a stupid idea is.
For an added bonus, try writing a parsing tool that can detect the existence of a header on a certain iNFINITY demo ROM. Hint: a lot of these homebrew authors didn't pad their ROMs to a nice multiple of 32KB.
Oh, and thanks for making fan translation patching hell for authors for the last fifteen years with IPS+SMC. Because a cointoss chance of applying a patch, and then having to hex-edit in or out headers is totally worth the eleven people who still use floppy-based copiers.
Quote:
As for the other formats (MGH, etc.), sure, I can understand dropping support for those given that they're truly legacy
MGH is yet another file that is often just a headerless game image. A lot of these formats did use interleaving initially, and as far as I know, only Super Sleuth supports the SWC interleave format, ZSNES supports a very small number of 5-6MB image interleave formats, and that's it.
Nowadays the extensions are meaningless. Too many extensions wreck a file load window, lots of false positive matches, pain in the ass to use OS file assign when you have to do it for 22+ extensions, etc.
Quote:
These are what you'll find across the entire Internet, and even on CD
So? I do not encourage piracy. I also include a two-click tool to fix this problem.
Quote:
I immediately clicked [X] and deleted the bsnes\ folder
Cool, with you so far.
Quote:
going back to 0.73.
Ah well.
Quote:
Furthermore, like the rest of the world, I am not going to run a ROM conversion utility because a single emulator author is on such a tirade.
Good. If you can't click two buttons to make my life easier after I've spent seven years writing an emulator for you, I'd rather not have you as a user.
If you are that irrational about running a single tool, then you are going to be equally irrational about other little things like certain obscure features and GUI design. You'd just be a pain in the ass who contributes nothing, most likely.
Quote:
What makes you think I want to run all of my games in just bsnes and not a multitude of emulators
I mentioned this in the user guide, and in this thread. And you should know this. Headerless SFC runs in any SNES emulator out there. Running snespurify does not break compatibility with any emulator.
Quote:
Furthermore, I have a SWC DX32 and a working SNES; maybe I want to send some games to my copier via the parallel port transfer utility... oh now look what I have to do, convert all the games from SFC to SMC using UCON or doing it by hand.
Once again covered in my user guide.
Okay, so you're one of eleven people who still use floppy-based copiers in 2011. Great. It'll be completely dead in 5-10 more when the EEPROM BIOS on that POS bitrots. Then you can upgrade to the 21st century where we use flash cartridges with SD/CF support. Near-instant loading and every game ever on one tiny SD card, and they sell for 1/3rd the price of copiers, and you don't have to send an e-mail to a guy on Usenet to track down a lead on a guy he heard may have a copier for sale four years ago to get one. Good for you.
Now let's look at your ROM set. There are over a dozen known copier header formats, and different copiers only support certain ones. No copier supports them all. Most ROMs don't even have headers and all, and Nach subverted them to store NSRT data for some inane reason. The odds that your internet or mail-order CD is going to have SWC-compatible headers is next to none.
So what do you do? You convert your ROMs to have SWC headers! You want to use X, convert to X's format. Wow, that sounds familiar. Why aren't you tracking down the SWC makers and complaining? Hell you actually PAID for your copier. All you've given me was info on Felon's Banana Register.
Or an even smarter idea, that uCON at least should do: uCON send gamename.sfc should add the SWC header in memory and send it, that way it works regardless of header. What about all those user (not plural) with copiers that only support loading interleaved formats? Shouldn't they be complaining that ZSNES and Snes9X won't load those images without conversion, too?
No, I'm not going to miss you, Nightcrawler, and the nine other people with copiers.
Quote:
The motives are positive, but it's too late and all you're doing is what I described: alienating people (users of your emulator).
I used to really care about such things. And then I realized I was making myself miserable, bending over backwards, to support the very users that annoyed me the most.
The same ones complaining that ZSNES was just as accurate and ran faster, the same ones happy to use closed-source emulators, the same ones too lazy to click two buttons to make my life easier, etc.
I don't care. I really, really don't care. Don't use bsnes. I'm not getting royalty checks for it. I just made myself miserable, and pissed off all the other emulator devs engaging these types of people. Kind of like I'm doing now! Oh, the irony of it all.
I have no need to have the most popular emulator for a given 20+ year old game system. I am not out to impress your colleagues. For what? So the two times in my adult professional life I've ran into someone who could name an SNES emulator in person, I could say, "oh hey I wrote that!" Why bother?
Anyway, if the ZSNES and Snes9X team also had a spine, this would already be a non-issue. Unlike the NES scene that is forever cursed with iNES 1.0+DISKDUDE, we had a real chance to fix all of these problems. We would not be having this conversation if they were reasonable.
To be brutally honest, the NES scene could fix the iNES 1.0 situation as well. Yes, even today. But you're more concerned with duking it out for popularity. Yeah, Joe Sixpack can stick with the ROM set and emulator he downloaded in 1998. Who cares, he isn't upgrading to your new emulator anyway. Just as a bsnes v073 user is the same to me as a ZSNES user. In a way, you all get what you deserve. One has no philosophical right to complain about something he actively encourages and passively supports.
Quote:
Nestopia's database feature requires roms be in headered frankenfile format.
Then we could just consider this a feature request for Nestopia to use MAME- or Pasofami-style zipped split ROMs and check them against some database of only those games licensed by Nintendo. Feel free to contribute the patch to Nestopia's maintainer.
Quote:
the NES library sanctioned by Nintendo doesn't.
The NES library sanctioned by Nintendo is Virtual Console. This is a super frankenfile: it consists of an iNES frankenfile and a Wii executable to emulate it, all in one WAD container (unrelated to Id Software's container of the same name).
Quote:
People here could make a game that looks and sounds just like an NES game in half the time for Windows and get 100 times the audience.
Until you get to deal with broken system libraries. For example, PulseAudio on Ubuntu has been interfering with sound in games that use the Allegro library for the past couple years. If the user has a working emulator for one game, the user has a working emulator for every game.
Quote:
It's a puzzling extension that stands for a chinese copier. If a letter in your file is what's keeping you from using bsnes, you ought to take the blue bill for life.
Quote:
this makes me wonder why you don't support (non-split) .SMC files, choosing to ignore the first 512 bytes of the file thus effectively ending up with a BIN/SFC. I mean, is a single lseek() call really that hard?
It's not just the file name. It's also the 512-byte copier header that, unlike the iNES header, provides no information on how to run the game. It's also the fact that people have been renaming files between .smc, .swc, .fig, and .sfc to get Windows's dain-bramaged file extension system to launch the emulator, and an emulator has to try loading the first 64.5 KiB of the ROM four times (LoROM, HiROM, LoROM with header, HiROM with header) to see whether or not a header is present.
Quote:
Quote:
Who keeps the database up to date?
Me. If anyone actually uses it, I'll make future revisions a public download on my site.
If you're willing to keep a database of the hash values of verified good dumps of NES games licensed by Nintendo, I'm all for this effort. "FitzRoy's database of licensed NES games" could be useful for 1. running zipped split ROMs, such as those from your proposal or Vs. Unisystem or PlayChoice 10, and 2. helping emulator authors make a bundled tool (like byuu's suggestion of nespurify) that automatically corrects the header in an iNES or NES 2.0 file based on the PRG and CHR.
Quote:
But most mods of licensed games can be soft-patched.
One example that doesn't fit into "most" is disassemble-patch-reassemble hacks of the first Super Mario Bros. based on doppelganger's disassembly.
byuu wrote:
Unlike the NES, where there are more emulators than there are stars in the sky, there are really only two significant SNES emulators.
[flattery] Did you mean bsnes and Snes9x, or bsnes and zsnes? [/flattery]
Quote:
unlike official games, which is a finite set, homebrew is an infinite set that can theoretically be added to until the end of time.
Where does this leave commercially released unofficial games, such as those by Tengen, Color Dreams, AVE, Panesian, Sachen, Waixing, Sivak Games, and more? So that Tetяis and the like aren't left out, I'd be more willing to include everything distributed on cartridges to the public on or before December 31, 1995. This would include Wario's Woods, Sunday Funday, and everything before them, but not the modern age of HKOs and RetroZone.
Quote:
so what we want is clean images without headers, and what others want is homebrew. Why not make the iNES header a separate file, gamename.nes + gamename.ines. Stick it in a ZIP file.
Or do it Pasofami-style (as Dwedit pointed out) and put Against.prg, Against.chr, and Against.ines (the NES 2.0 header) in a zipfile. The problem with zipfiles is that PocketNES on the Game Boy Advance can't efficiently read them, but PocketNES makes other accuracy sacrifices to run on a 16.78 MHz ARM CPU anyway.
Quote:
Quote:
What's wrong with the 20 byte header? This question still hasn't been answered!
For one, it's not twenty-bytes.
Through this topic, I've been assuming good faith and reading it as octal.
Double posting because some parts of byuu's comments appeared to be heading toward a tangent, and double posts are easier to handle when a moderator splits a topic.
These games are twenty-plus years old. You cannot buy these actual games from the manufacturer because they are no longer made.
You would
so not get along with
clodney:
clodney wrote:
um, maybe accept the fact that what you want isn't available and move on? Just because you want it doesn't mean you are entitled to it.
Quote:
People are entitled to their own culture, the public domain.
Then perhaps they should vote accordingly. No wait, they can't, because
the MPAA controls TV news to keep people unaware of copyright issues and unaware of House and Senate candidates with an MPAA-unfriendly platform.
Quote:
evil, greedy pieces of shit like Bono and Disney can extend copyright well beyond the lifespans of the longest-living human beings, such that nothing ever created in our lifetimes will be available to us. All so that Bono can have one more gold-plated swimming pool.
Are you sure you aren't confusing Sonny Bono of the Copyright Term Extension Act with Bono of U2?
Quote:
If the copyright holder is not selling the work anymore, it is not piracy.
Judges disagree.
Quote:
I further agree with actual researchers like Rufus Pollock, who indicate the optimal length for copyright is 14 years.
Legislators disagree.
Quote:
I removed fullscreen
Then you're not emulating a Super NES connected to an upscaling TV; you're emulating a Super NES connected to the picture-in-picture port.
Quote:
OS-native file load dialog
On a few operating systems I can think of, your application can't even open a file unless the user has chosen it from the OS-native file load dialog. Otherwise you get a privilege violation.
Quote:
drag-and-drop
Then you're not emulating a Super NES, where the user drags cartridges out of a collection and drops them into the Game Pak slot on the Control Deck. You're emulating an arcade system where the ROM board has to be in place before the user gets there.
Quote:
some even hack up emulators to support extra features that are impossible on real hardware.
I assume this doesn't include hacking up emulators to start playing an MP3 file upon a mapper write. Starting Famicom expansion sound effects by number is demonstrated (Moero Pro Yakyuu), and routing an MP3 player's audio out the Famicom expansion sound part is
demonstrated. BGM replacement would just combine these.
3gengames wrote:
Not like adding 20 bytes doesn't destroy the ROM, it's there. Extract it! And with that format you also need two files, which lugging two files on my PC instead of insta-clicking one and is brough up in FCEUX perfectly is annoying. You'd need a linker filer to tell what two ROMs to use anyway.
Because CHR and PRG aren't two separate chips. Why on earth would anyone want separate files for them? It's not like hackers have ever wanted to expand things to have room for more data.
Pasofami format actually got that part right.
It's quite possible to do a mapper hack which, if you were to make an IPS of it, would result in a patch containing the entire CHR ROM since it got shifted to expand the PRG area. Maybe I'm mistaken, but I thought one of the reasons hack-fags liked patches was because it let them distribute only their own modifications rather than the whole damn copyrighted image ...
Of course, the whole concept of "mappers" is bullshit to begin with, but that's another problem ...
byuu wrote:
I used to really care about such things. And then I realized I was making myself miserable, bending over backwards, to support the very users that annoyed me the most.
...
I don't care. I really, really don't care. Don't use bsnes. I'm not getting royalty checks for it. I just made myself miserable, and pissed off all the other emulator devs engaging these types of people. Kind of like I'm doing now! Oh, the irony of it all.
I have no need to have the most popular emulator for a given 20+ year old game system. I am not out to impress your colleagues. For what? So the two times in my adult professional life I've ran into someone who could name an SNES emulator in person, I could say, "oh hey I wrote that!" Why bother?
byuu, your emulator is supposed to be about accuracy and preservation. There is absolutely no point in arguing with old codgers who expect the ass-backward, wrong way of doing things they learned 16 years ago to still work today.
Maybe with some luck the new people trickling into the emulation scene will be more willing to make the change since they aren't carrying around the mental baggage of Super Pasofami and Esnes and they – most likely – won't be trying to run bsnes on handed-down hardware from 1997.
We're talking about users who see nothing wrong with running ClrMame and rebuilding their entire 26GB ROM set to comply with all the inane CRC changes that occur between MAME versions, but who will then bitch about a one-off conversion of their SNES set that not only ensures compatibility in bsnes, but does not break compatibility in other emulators.
You will never get through to people so perfectly illogical. Stop trying and go do something more productive like finishing up bview
D wrote:
It's quite possible to do a mapper hack which, if you were to make an IPS of it, would result in a patch containing the entire CHR ROM since it got shifted to expand the PRG area. Maybe I'm mistaken, but I thought one of the reasons hack-fags liked patches was because it let them distribute only their own modifications rather than the whole damn copyrighted image
And with modern assembly hacks such as those based on SMBDis, an IPS would contain damn near the whole copyrighted PRG even if it doesn't expand CHR.
Quote:
Maybe with some luck the new people trickling into the emulation scene [...] won't be trying to run bsnes on handed-down hardware from 1997.
Even new-in-box hardware manufactured in the past two years might not necessarily be capable of running bsnes at full speed. Case in point: Acer's Aspire Revo, ASUS's Eee Box, and other small, quiet PCs with an Atom CPU and an NVIDIA ION chipset. Now ION has a GeForce 9400 GPU, which can run CUDA- and OpenCL-accelerated decoders, making it ideal for watching AVC video on your HDTV. But then someone gets the bright idea to try playing 20-year-old video games on the PC that happens to be already connected to the TV, and its Atom CPU becomes a liability, as
kyuusaku discovered.
Quote:
Where does this leave commercially released unofficial games, such as those by Tengen, Color Dreams, AVE, Panesian, Sachen, Waixing, Sivak Games, and more?
Outside of the official set, just like ROM hacks and Wisdom Tree games. "Thou shalt not steal, unless it is a Super Nintendo game engine and you are using it for proselytization."
Tengen NES was very unfortunate, some good games there. Would Nintendo really not publish them, or were Tengen just that cheap?
Quote:
The problem with zipfiles is that PocketNES on the Game Boy Advance can't efficiently read them
I don't care about support for a dead handheld, but I agree that ZIP is a sorry choice for a container format. And TAR is full of Unix baggage.
Would be nice to have a mainstream, sane, UTF-8+SHA256 flat-file format that ensures no compression for O(1) updates to fixed-size files (and one variable-length file.)
Quote:
Through this topic, I've been assuming good faith and reading it as octal.
Hah, I was actually going to remark on that, but he did say bytes, and 16*8/3 is an irrational number.
Quote:
You would so not get along with clodney:
clodney wrote:
um, maybe accept the fact that what you want isn't available and move on? Just because you want it doesn't mean you are entitled to it.
No I would not. I wonder if he's a fan of classical music.
Quote:
Are you sure you aren't confusing Sonny Bono of the Copyright Term Extension Act with Bono of U2?
I should have said 'could', not 'can'. Yes, I am aware it was the Bono from Sonny and Cher. And I'm sure he had enough money to not require a 175-year copyright term. A lot of good it's doing him now, huh?
Quote:
Judges disagree. Legislators disagree.
And? With the current laws, every last American citizen is a law breaker. That is by design.
I don't remember choosing to be born in the US, and I am not a free enough human being to be able to emigrate freely to most other countries.
I do not respect, recognize, nor obey unjust laws. It is civil disobedience.
Quote:
Then you're not emulating a Super NES connected to an upscaling TV; you're emulating a Super NES connected to the picture-in-picture port. Then you're not emulating a Super NES, where the user drags cartridges out of a collection and drops them into the Game Pak slot on the Control Deck. You're emulating an arcade system where the ROM board has to be in place before the user gets there.
Man, and here I put so much work into trying to get it just right. I emulate CR2032 battery decay, and as you switch games I add digital 'dirt' to the PCB contacts - so you have to use an alcohol swab every couple of years, I emulate the channel 3/4 switch on the RF box, and I make the spacebar act like the eject button that literally pops the game right out of the emulator. I will get to work on fixing these inaccuracies right away :D
Quote:
I assume this doesn't include hacking up emulators to start playing an MP3 file upon a mapper write. Starting Famicom expansion sound effects by number is demonstrated (Moero Pro Yakyuu), and routing an MP3 player's audio out the Famicom expansion sound part is demonstrated. BGM replacement would just combine these.
No, these idiots were monitoring internal work RAM writes, to send a command from ZSNES to Winamp. Very similar to the Zsnexbox method of adding rumble support to controllers, and heavily influenced by the 'cheat code search' approach to ROM hacking. Aka laziness.
Quote:
Pasofami format actually got that part right.
Agreed. Split a file when it should be split. SNES is tricky because there is no distinction between the data. Indeed for dozens of titles, you will find that older runs would use 2x16mbit chips, and later runs would use 1x32mbit chips. Just like PC RAM today, you pay a premium for the higher capacity chips at first, and once supply catches up they become even cheaper.
There are only a tiny number of games that have truly separate chips that probably should not be packed together, namely SPC7110 data ROMs.
Quote:
Maybe I'm mistaken, but I thought one of the reasons hack-fags liked patches was because it let them distribute only their own modifications rather than the whole damn copyrighted image ...
Of course a delta-patching format would eliminate that issue. But hey, it's oh so much fun relying on Amiga formats from the early '90s!
Quote:
Even new-in-box hardware manufactured in the past two years might not necessarily be capable of running bsnes at full speed.
http://img405.imageshack.us/img405/9608/zelda3v.png
But yeah for the most part, bsnes targets desktops and laptops. Not cell phones and netbooks (cell phones with a keyboard that can't make calls.)
I recently built an HTPC running an
Atom 330 overclocked to 1.91 GHz (it only raises the temperature by 1 C, even with passive cooling). It can handle bsnes, but not the psycho-accuracy core.
The Atom 330 has been around since at least 2008. If Asus is putting lower-end Atom chips in its current offerings, then it is raping its customers.
I believe byuu also said it works on his MSI Wind at full speed, which is using an even less powerful Atom chip.
Perhaps he was running the accuracy core, or he was running a 32-bit binary on a 64-bit OS, or his bsnes binary was compiled by software that was not optimizing for Atom.
I'm wondering how many more times this subject will come up before something actually changes.
MottZilla wrote:
I'm wondering how many more times this subject will come up before something actually changes.
I'm not sure I understand why the *emulator* [the tool or the person writing the tool, you pick] has to be the one worrying about the authenticity/preservation of the ROM it's trying to emulate. That worry is squarely in the *user* domain. If I as a user want to play SMB but I end up downloading and running SMB "penis hack" and offending my kids...my fault. What's next, an emulator spitting up a "this ROM is not authentic and may contain offensive content...continue anyway?" message?
If I as a person care about authenticity/preservation of ROMs, I should feel free to create what I think is a valid database format for storing such information. I should also be willing to accept rejection of adoption of my declared format by either a)fellow preservationists, or b)emulator authors that believe that authenticity/preservation is not their problem [or at least they don't need/want to enforce it and alienate a community of homebrewers].
That being said, my application spits out BootGod information for loaded .NES files, just incase the user cares to validate the authenticity before proceeding.
As an emulator author I don't want more formats to support if there's no benefit to supporting them. But I would be interested in a database that tells me what all the authentic game details should be...which I believe we already have in BootGod [and I think there was mention of one other somewhere in this loooong thread!]
byuu wrote:
Quote:
The problem with zipfiles is that PocketNES on the Game Boy Advance can't efficiently read them
I don't care about support for a dead handheld ...
...yet with bsnes v074 you
added a (Super) GameBoy emulator.
To a SNES emulator that now refuses to support certain SNES/SFC-centric file formats because "bsnes is emulating the SNES, not a copier".
I'll take the rest of this up privately, because actually after reading some of your responses, I'm pretty damn pissed off.
Quote:
Perhaps he was running the accuracy core, or he was running a 32-bit binary on a 64-bit OS, or his bsnes binary was compiled by software that was not optimizing for Atom.
The funny thing is I didn't even do that with my build. It was profiled, though.
Quote:
I'm wondering how many more times this subject will come up before something actually changes.
To be fair, FitzRoy is introducing a new format here. It is different from iNES 2, UNIF, Pasofami and split-XML.
The proper response for those disinterested would be to ignore the thread. Instead we had many iNES supporters try and undermine it immediately.
Quote:
If I as a user want to play SMB but I end up downloading and running SMB "penis hack" and offending my kids...my fault.
Who is telling you that you can't run tasteless hacks? This isn't MAME.
FitzRoy is offering a saner, less error-prone format for officially released games.
Emulators can choose to support only one or the other, or both. Without a way to support homebrew, I can't imagine even an idealist like me only supporting official games.
Quote:
...yet with bsnes v074 you added a (Super) GameBoy emulator. To a SNES emulator that now refuses to support certain SNES/SFC-centric file formats because "bsnes is emulating the SNES, not a copier".
... what? "You said you dislike trucks, so why do you eat chicken?"
The Super Game Boy is an official SNES cartridge, it is not a Chinese bootleg copier.
A comparison would be if I emulated the copier BIOS and actually loaded files off of real floppy disks. Which actually would be kinda fun ... if I had all the time in the world :P
Anyway, I was saying that I'm
personally not concerned with running an NES emulator on a GBA, not that I wasn't interested in the GB itself.
Quote:
I'll take the rest of this up privately, because actually after reading some of your responses, I'm pretty damn pissed off.
I apologize if I've upset you. I am certainly blunt and honest, but even with sugarcoating, our underlying points are the same, eg "no offense, but ..." I'm not the slightest bit upset or angry here, I'm just extremely opinionated as a result of a decade and a half of dealing with these issues.
If you'd rather I not counter your responses, I am willing to comply. It will save me some time anyway.
3gengames wrote:
What's wrong with the 20 byte header? This question still hasn't been answered!
It was answered before anyone even responded. I laid it out cold that rom-side mapping is fundamentally flawed in that it can't effectuate inevitable revisions. "Just get every rom supplier to update to iNES 2.0 and we're good!" Even if you could convince them that it was worth it, with what tool based on whose database? And then, if they succeeded, how would anyone know they were getting it since iNES 2.0 shares the same extension as iNES 1.0? There are too many layers of bullshit for users to understand, you've fucked yourself. No one will ever use your emulator and KNOW that they won't have some random assfuck error from headers 50 years from now.
And you can sit here and lament the fact that unlicensed material can never, attain that level of certainty because no one will ever volunteer to add 50,000 entries to an emulator-side solution, or you can at least get it for licensed games which SHOULD have a higher priority than your basement boards anyway.
Quote:
Not like adding 20 bytes doesn't destroy the ROM, it's there. Extract it!
And I suppose you like to put rocks in your doorways to jump over them.
Quote:
And with that format you also need two files, which lugging two files on my PC instead of insta-clicking one and is brough up in FCEUX perfectly is annoying. You'd need a linker filer to tell what two ROMs to use anyway.
More proof that you can't read. It says right in the official format guide that this is a CONTAINER format with a unique extension. A container is one file to load. Access to the prg or chr data does not require a special tool to offset, split, or recombine. Any commonplace archive tool that you already have will do.
And on the surface, it seems like it wouldn't benefit you. Because you're the kind of user that downloads GoodNES and assumes integrity after playing the most popular games. It's not the popular games it gets wrong, but the obscure games. The type of game you might discover years down the road from the odd recommendation, and when you go to play it, it breaks, because no one made sure that it was good when people who actually gave a shit were still around.
And that's why having data in a format that is easiest to obtain values from matters. Because it helps the people who butter your bread.
So we're putting them in a container with both ROM's? That brings us back to iNES. This is a good idea, it's just the iNES is the best idea, sorry. Prove me wrong and I'll be more then happy to switch, but until 51% of emulators support this as their main extension, I won't change.
A proper container format with popular access tools is light-years ahead of iNES for ease of access. It's extremely unfair to equate it to an obsure, special-purpose binary blob. I don't even know of any tools that split and merge PRG/CHR from iNES files, although I'm sure there are some somewhere.
I only made a splitter and there's alot more on the web >.>....needs fixed up though. And what will the files spaced out have that iNES can't? This offers no advantage as far as I'm concerned.
byuu wrote:
But yeah for the most part, bsnes targets desktops and laptops. Not cell phones and netbooks (cell phones with a keyboard that can't make calls.)
That'd be fine if I could find a 10" laptop. But all the stores appear to have in that size are netbooks.
But back to topic:
FitzRoy wrote:
And then, if they succeeded, how would anyone know they were getting it since iNES 2.0 shares the same extension as iNES 1.0?
Text in UTF-8 encoding shares the same extension as text in CP1252 (extended ISO Latin-1) or CP437 (MS-DOS character set). HTML5 shares the same MIME type as HTML 4.01.
NES ripped music has the same extension as a Lotus Notes database.
Super NES ripped music has the same extension as Authenticode certificates and a bunch of others.
3gengames wrote:
And what will the files spaced out have that iNES can't?
For one thing, patches would be applied to the PRG and CHR separately. So if you expand PRG in a patch, your patch wouldn't include the whole CHR.
3gengames wrote:
I only made a splitter and there's alot more on the web >.>....needs fixed up though. And what will the files spaced out have that iNES can't? This offers no advantage as far as I'm concerned.
Instant access to the chr and prg files, sizes, and checksums without needing to find some assfuck specialty tool. Think about the principle. If we had 10 systems like this, some people would be driven insane by now with all the tools needed to find simple information. Your argument if basically "I've already wasted my time finding or writing the tools necessary to dick with these blobs, so fuck everyone else who's yet to do so, what's in it for me anymore?" You can't see how making it easier for other people causes them to be more efficient at enhancing things you value.
tepples wrote:
Text in UTF-8 encoding shares the same extension as text in CP1252 (extended ISO Latin-1) or CP437 (MS-DOS character set). HTML5 shares the same MIME type as HTML 4.01. NES ripped music has the same extension as a Lotus Notes database. Super NES ripped music has the same extension as Authenticode certificates and a bunch of others.
It wasn't an important point I was trying to make because you'll still have frankenfiles, but you're missing it. No one will go on an NES page and think the files they're downloading might be lotus notes databases. iNES revisions are not likely to be visible on the surface, either in pages or in filenames, especially if GoodNES sticks with iNES 1.0. It won't be as easy to find a romset with proper headers if one is ever constructed.
FitzRoy wrote:
3gengames wrote:
And what will the files spaced out have that iNES can't?
Instant access to the chr and prg files, sizes, and checksums without needing to find some assfuck specialty tool.
IMHO you're looking for "
noddy", not "assfuck". In that case, someone (possibly myself?) should write a 30-line split-and-SHA program for iNES and NES 2.0 in Python and throw it on the wiki.
Quote:
No one will go on an NES page and think the files they're downloading might be lotus notes databases.
But Windows might. If you double click a downloaded SPC sound file, you get "This is not a PKCS #7 certificate" or the like. Unlike Linux file managers, Windows Explorer is incapable of distinguishing files other than by their extension.
Quote:
iNES revisions are not likely to be visible on the surface, either in pages or in filenames, especially if GoodNES sticks with iNES 1.0. It won't be as easy to find a romset with proper headers if one is ever constructed.
An auditing tool could use a database like yours to split, hash, and rewrite the header on each ROM in a collection.
tepples wrote:
Quote:
iNES revisions are not likely to be visible on the surface, either in pages or in filenames, especially if GoodNES sticks with iNES 1.0. It won't be as easy to find a romset with proper headers if one is ever constructed.
An auditing tool could use a database like yours to split, hash, and rewrite the header on each ROM in a collection.
This is my vote. I know that such a tool is not what the OP wanted; he probably considers is a bastardization of his idea.
If I had some spare time, I would write a command-line tool that uses a database (probably a pipe-delimited text file) and "fixes" ROMs to the proper iNES 2.0 header and changes the filename to something canonical. I'd write it in ANSI C, since it would compile and execute fine on Windows and Unix (and unix like OSes) without needing any additional runtime support.
clueless wrote:
I know that [a database-driven header fixer] is not what the OP wanted; he probably considers is a bastardization of his idea.
But then I can't help bastardizing his ideas, being a bastard myself.
As a bastard I'm also for fixing iNES files. I wrote a NES ROM sorter for my private NES/FC CSV table/"database", but that has been vastly superceded by bootgod's. Maybe someone good with XML/XSLT can write a program to sort a dump of his database, sort the ROMs and fix them in one operation. This would make the tool "future-proof" as a new database dump could update the entire ROM set.
Something brought up I don't get is the necessity for split PRG and CHR segments, and why it is more "correct". Do ROM hackers really have such a hard time splitting iNES "blobs"? If someone can learn to ROM "hack" or make reproductions (ARGH!) can't they learn to use/write a processing tool? Most users will never know what a container is. I'm not the biggest fan of iNES, but I don't get why it's SOOO BAD to use binary headers and containers, computers appreciate things in binary after all. Is the idea interweb-age ROM solidarity?
Oh yeah, I'm a bystander but I don't really appreciate people popping out of the woodwork to say the iNES defenders here (it's really more like tolerators) are idiotic and close-minded, it's really uncalled for and frankly says more about a poster's maturity and possibly aptitude. I think the same goes for other disrespect of people and devices mentioned too
tepples wrote:
An auditing tool could use a database like yours to split, hash, and rewrite the header on each ROM in a collection.
It's still a frankenfile and it's still part of a failed mapping delivery concept. 100% of users and 95% of emulators use only 1.0 files even after a better revision was issued years ago. Supporting 2.0 is still supporting a concept that puts mapping in the hands of users and expects them to do all this updating as info comes in for games few if any of them play. Why support a concept that failed where emulator-side mapping wouldn't have in making 100% correct mapping commonplace?
Because 1.0 takes care of 99.99% of games.
I must be missing the whole point. This idea is for a better format than iNES, but only for licensed games. How many licensed games fail with correct iNES headers anyways? 3 or 4?
Game info is looked up with a checksum. That can already be done on an iNES file and the header ignored (or FIXED by the emulator).
FitzRoy wrote:
Quote:
# What about games that I hack myself? I guess that these are no different than uncatalogued 'homebrew' from the point of view of the emulator.
Any modification generates a new checksum, so you'd have to use iNES until it's finished and then create a patch. Patches don't interfere with checksum lookup.
So one of the big benefits of using separate PRG/CHR in a ZIP container is ease of modification to either part without affecting the other. Except you can't actually do modification, or the checksum fails and nothing works at all.
To get around this, you just use iNES while you are developing. This must mean you already have an iNES splitter tool. Also means iNES works fine, so why would you stop using it? Now that development is done you need to make a patch, another frankenfile. Making the patch also requires a tool so the new format doesn't help there.
This all seems to be so narrowly focused to be much less usable than it could be. Nothing extra is preserved in the file (its all in the database), working with the file isn't any easier (checksums fail), and it ignores the majority of the game library (unlicensed + Famicom + VS system + homebrew) which is the place iNES actually has problems with.
http://www.jwz.org/doc/worse-is-better.html
Just to clarify my positions, while I think the iNES 1.0 and iNES 2.0 formats are fucking terrible, I don't consider FitzRoy's proposition to be any better. I really do not like the idea of this large database because it is a serious waste of information for emulator authors to ship. Plus he is not including unlicensed games, which are these days the only things keeping NES emulation interesting.
And then there is the forking argument ...
This is similar to MAME's retarded approach of storing information about every game inside the MAME binary rather than store that information in the ROM sets themselves. The result is something we're already familiar with: 47MB binaries (unless the UPX compress them down to 15MB, which never happens except on Windows) and each revision breaking compatibility with several hundred game sets.
I told FitzRoy on byuu's board that I think it would be far more useful to create a format similar to the one byuu is using for SNES. You have a basic archive with three files: INFO.XML, PRG.BIN, CHR.BIN. The XML contains all the information about how to map these files into memory, what regions are mirrored and so on.
Why is this a good idea?
First off, it shaves 300KB of code out of most emulators. Every emulator has hundreds of files containing source code that specifically hacks the machine state to various mapper numbers. This is pointless. An XML file can tell all this information to the emulator and ensure it is completely correct for the given ROM.
Hackers who wish to change memory map information could simply edit the XML file accordingly and resize the PRG and CHR ROM files as needed.
Patches would be applies against the PRG and CHR separately, removing the problem of diff algorithms including the entire CHR ROM when someone decides to remap and expand a NES ROM.
Moreover, as new cart formats come out in the future (such as NANJING carts), these can be supported through the XML alone. Even abandoned emulators, by supporting such a format, could continue to be relevant and support new dumps and boards even if the author is no longer maintaining his project. We would not longer have to rely on Windows-only hacked emulators distributed by Chinese pirates to run most of these.
For a container, something common like ZIP or TAR would work fine. You could tag the XML with a version number so that if some radically new carts come along that need new fields current emulators don't understand, you just bump the version number and add them without risking backwards compatibility or another dramatic format shift.
As for compatibility with quasi–flash-carts like the PowerPak, I would hope it is extremely trivial for an author to simply write a tool that will convert the XML file into something that can quickly be parsed by the NES: perhaps a plain-text file delimited by new lines where the software already knows that to look for and use from each line. That should be pretty damn snappy.
I cannot see many negatives to doing things this way. It's just better for the emulator authors, for the hackers and for the end users.
Apologies for the OT-nes....
D wrote:
http://www.jwz.org/doc/worse-is-better.html
This opened my eyes
I wonder when UC Berkeley moved back from New Jersey.
D wrote:
Of course, the whole concept of "mappers" is bullshit to begin with, but that's another problem ...
Philosophically right? Because the silicon albeit by other names is pretty real.
D wrote:
Every emulator has hundreds of files containing source code that specifically hacks the machine state to various mapper numbers. This is pointless. An XML file can tell all this information to the emulator and ensure it is completely correct for the given ROM.
Hackers who wish to change memory map information could simply edit the XML file accordingly and resize the PRG and CHR ROM files as needed.
Just checking but so you're a proponent of the netlist method, with XML elements representing nets, functions and registers?
tepples wrote:
But then someone gets the bright idea to try playing 20-year-old video games on the PC that happens to be already connected to the TV, and its Atom CPU becomes a liability, as
kyuusaku discovered.
And these:
http://nesdev.com/bbs/viewtopic.php?t=4896
http://nesdev.com/bbs/viewtopic.php?t=6581
bsnes has become much more usable since then though, unfortunately I can't say the same for Nintendulator.
kyuusaku wrote:
D wrote:
Every emulator has hundreds of files containing source code that specifically hacks the machine state to various mapper numbers. This is pointless. An XML file can tell all this information to the emulator and ensure it is completely correct for the given ROM.
Hackers who wish to change memory map information could simply edit the XML file accordingly and resize the PRG and CHR ROM files as needed.
Just checking but so you're a proponent of the netlist method, with XML elements representing nets, functions and registers?
I realize carts have actual mapper chips in them. I remember Nintendo Power running articles about them back in the day.
Nevertheless, we are pounding square pegs into round holes by the current process of creating new random numbers to collect random board that behave similarly.
The current process goes something like this: Dumper finds a cart with a new memory map. Dumper invents a random numbers to describe the map and prays no one has used it. Dumper hacks on a header with this number, hacks one open source emulator to detect and use it. Dumper releases both into the wild on a piracy site.
And then every emulator, even if it supports all the features of this mapper in other mappers but not in the given combination, gets to complain that it doesn't support "Mapper #243" for the next several years depending on the popularity of the dumped game.
Yeah ... that's not encouraging piracy.
It would be far more beneficial to simply describe the memory map in XML. Within that space define things like mirroring, battery-backed RAM, and WRAM (I think the Nanjing carts made use of it, which may be why they were so slow to get support).
Nes developers today can't just order up MMC5 boards from Nintendo. Forcing them to conform to a board previously manufactured by Nintendo or a third party is kind of pointless and is artificially limiting. A lot of Chinese fab houses have made some pretty weird and interesting boards, and if someone were to commission a run of a homebrew game, their hardware engineer could easily bang out a chip to handle any memory mapping they choose. Being able to test that mapping in emulation first would vastly speed up the development process, much like byuu has done with the MSU1.
I'm a proponent of getting past the idea that what has been made before is all that should ever be made in the future. Freeing emulator authors from the burden of mapper updates and opening the door for developers and hackers are nice side benefits.
But how would a description of all circuits on a board look, including those inside the mapper ASIC, and how would it be emulated efficiently, especially in the case of an MMC5-class mapper? Or must we stick with predefined mappers on any hardware smaller than a full-size laptop computer, such as handheld devices and the PowerPak?
D, NES mappers are something of a hybrid. They are between a Game Boy MBC and an SNES coprocessor in terms of complexity.
Some of these chips have custom sound hardware, custom IRQs, hook into video RAM of the PPU, etc.
As a result, basic XML mapping won't be enough. So what others are suggesting with netlists is some sort of circuit-level simulation, which is, IMO, too slow to be practical. Basing that off of what I've seen in the past, eg Pong circuit emulator, etc. Could be wrong on netlist overhead.
My thoughts on the matter were that the XML should specify what the underlying mapper hardware is, and give that chip all of the configurable parameters it can take. It may be possible to break down chips a bit more, eg "HasIRQType1/2" if MMC{x,y} are the same, but they are different from MMC{z}.
If I had to come up with something wild for this, I would say to go the VM route to write universal mapper emulations. Use the 6502 architecture already in place for the CPU, and write mapper emulations in 6502 (very hard, but only has to be done once.) You'd have to give special MMIO mappings to account for the various hardware specifics and cartridge bus pin accesses. Have this virtual 6502 run at something obscene like 21.47+MHz or something. Possibly give it the ability to run without a clock up to STP instruction or something instead. Other emulators wouldn't need to do much to get up and running. Then your database/XML can just take a tiny XML/flatfile with some configuration parameters (NVRAM size, etc.)
Yeah, you won't be able to run NES emulators on IBM PC Jrs anymore, but I think it'd be worth it.
D wrote:
\Nes developers today can't just order up MMC5 boards from Nintendo. Forcing them to conform to a board previously manufactured by Nintendo or a third party is kind of pointless and is artificially limiting.
Not really, since you can choose any board you want, MMC1,3,5. If the mapper limits you, your not a good programmer I guess. Especially when mappers made today don't work right and are far inferior. (HKOs)
And how would we "descibe" boards like NWC? It'd be just putting the emulator DLL's in one place, and I'm sure nobody will like to use someone elses code for an emulator.
Cloud emulation.....no. Just no.
byuu wrote:
D, NES mappers are something of a hybrid. They are between a Game Boy MBC and an SNES coprocessor in terms of complexity.
Some of these chips have custom sound hardware, custom IRQs, hook into video RAM of the PPU, etc.
OK, I can see how that would be a problem. The Nes has never been a major focus for me, and aside from hacking around with the mapper code in a couple emulators I don't have that much experience. I guess that makes sense though since the point of the chip is to get around the limitations of the base unit.
What I'm getting at is that there should be some way to abstract mapper function from the emulator.
byuu wrote:
My thoughts on the matter were that the XML should specify what the underlying mapper hardware is, and give that chip all of the configurable parameters it can take. It may be possible to break down chips a bit more, eg "HasIRQType1/2" if MMC{x,y} are the same, but they are different from MMC{z}.
If I had to come up with something wild for this, I would say to go the VM route to write universal mapper emulations. Use the 6502 architecture already in place for the CPU, and write mapper emulations in 6502 (very hard, but only has to be done once.) You'd have to give special MMIO mappings to account for the various hardware specifics and cartridge bus pin accesses. Have this virtual 6502 run at something obscene like 21.47+MHz or something. Possibly give it the ability to run without a clock up to STP instruction or something instead. Other emulators wouldn't need to do much to get up and running. Then your database/XML can just take a tiny XML/flatfile with some configuration parameters (NVRAM size, etc.)
That's a pretty wild idea, but I can't imagine anyone being willing to put in that much work for the chips that are out there right now.
3gengames wrote:
Because 1.0 takes care of 99.99% of games.
More like 98%, but 2% is 50 games, including Final Fantasy I+II (J), which is not exactly obscure. And as far as I know, no-intro does not concern themselves with mapping, only rom integrity, so it will have more and better dumps, but the mapping will be as imperfect as GoodNES. Of course, there's really no telling what you're going to get. One rom supplier might have more correct mappers than others. As roms change hands, they get can be messed with or corrected inbetween and most validation tools don't look at the header, especially because there's two kinds now.
We already know you ascribe to the good enough mentality, there's no need to keep reiterating your love for a concept that couldn't achieve more than that.
ibeenew2 wrote:
Game info is looked up with a checksum. That can already be done on an iNES file and the header ignored (or FIXED by the emulator).
You're saying this after someone has already pointed out that frankenfiles complicate patching and verification. I'm not writing offset code to untangle your pointless tangle. If you were to interleave the file, I'm not writing code to deinterleave it. Leave... the data... alone.
ibeenew2 wrote:
So one of the big benefits of using separate PRG/CHR in a ZIP container is ease of modification to either part without affecting the other. Except you can't actually do modification, or the checksum fails and nothing works at all.
How to handle hacks and other unlicensed data has already been discussed. Those games DO need a header or external header format and all the costs that go with that. Licensed games don't. I don't quite believe that Super Mario Bros needs to experience the same format costs as Super KKK Bros.
Why do we keep agreeing the header will be needed anyway? Waste of thread here.
FitzRoy wrote:
3gengames wrote:
Because 1.0 takes care of 99.99% of games.
More like 98%, but 2% is 50 games
Listing those 50 games would go a long way to promoting this idea. Looking at the iNES 2.0 discussions there are far fewer than that in the licensed set which need to be handled.
FitzRoy wrote:
including Final Fantasy I+II (J), which is not exactly obscure.
Famicom games were not licensed. If you are including them you need to change your terminology because that increases the scope of the games you will handle.
FitzRoy wrote:
I don't quite believe that Super Mario Bros needs to experience the same format costs as Super KKK Bros.
But your new idea does not ease the Super Mario Bros format cost, while making Super KKK Bros completely incompatible. This is why listing those games that fail with iNES 1.0 would be helpful, to see if it actually is a benefit given a header would still be needed in so many cases anyways.
The last big discussion in ditching iNES resulted in unif, where you can include any arbitrary block of data into the file (and just ignore blocks you don't need).
But then to make iNES to unif conversion possible, you need a database, but with that same database you're already loading an iNES file, and identifying it exactly, the conversion to unif at that point is just a convenience..
So I guess the question is, what does any new format have to offer, that doesn't exist already in iNES and unif? When a database is used, an iNES header is just a formality pretty much, and something to fall back on if the file hash doesn't match anything. It's easy to parse and I've yet to see anyone try to max out the bank sizes. Maybe we can define a mapper number that means "check the database"? heheh. It could be the "white flag" mapper for iNES.
Final Fantasy I+II is one of the few broken games with >8 KiB of WRAM. It's probably fully playable using save states since neither game has a need to switch banks mid-game. And it can be described via iNES 2.0 or fixed via detection hacks in the emulator. There are only a handful of games like this.
Quote:
Could be wrong on netlist overhead.
It needn't be analog simulation like Pong, it could be implemented through a small interpreted code block with agreed upon interface names. It would be slower, much slower, but the real performance hit will be enforcing emulators to do this every clock cycle, or more if external oscillators are to be emulated.
Quote:
It may be possible to break down chips a bit more, eg "HasIRQType1/2" if MMC{x,y} are the same, but they are different from MMC{z}.
And this is more clean? Few IRQ counters are identical in operation much less mapped the same. In a way this is the same thinking as iNES 1.0, liberties were taken and boards were abstracted into one or two ROMs and "mappers".
Quote:
If I had to come up with something wild for this, I would say to go the VM route to write universal mapper emulations. Use the 6502 architecture already in place for the CPU, and write mapper emulations in 6502
This is what old copiers you seem to despise do, but they timeshare with the game code. A virtual 6502 interpreter which executes a mapper function in 0 clock cycles (why waste resources emulating an actual parallel processor?) sounds like a reasonable idea due to moderately low overhead, but it's very dirty, like dirtier than MAME; everything would need to be hooked to a subroutine and you'd either need a standard sub-mapper so the 6502 could access large memory structures or it'd need to directly manipulate emulator flags and pointers... Oh and it wouldn't be so hot for audio synthesis or MMC5 emulation. I dare say nobody will go for it.
FitzRoy wrote:
You're saying this after someone has already pointed out that frankenfiles complicate patching and verification. I'm not writing offset code to untangle your pointless tangle. If you were to interleave the file, I'm not writing code to deinterleave it. Leave... the data... alone.
Appending CHR to PRG is a "tangle"? Really? Couldn't it be that the patching system in place just sucks?
In short, there is no reason to change anything. Everything works pretty damn well right now. There are a handful of problem games but that is about it. Strange MMC1 setups like FF1&2, StarTropics MMC6, and that's about all I can think of off hand that needs checksums to figure out.
ibeenew2 wrote:
Listing those 50 games would go a long way to promoting this idea. Looking at the iNES 2.0 discussions there are far fewer than that in the licensed set which need to be handled.
That's not the only issue. Missing or incorrect headers are the larger issue. Think about how many new dumps have been released since GoodNES that never got headers added to them. Now think about how files are distributed. If you modify a header for shits and then share it, it gets shared a hundred more times. Eventually, every complete set offered was completed from some outdated or scrupulous sources. You're bound to have some fucked up headers. So 50 years from now, someone goes to play one of these files, it breaks, and they'll have to learn about headers to know it was because of that, then how to update them, find bootgod's database... Where is all that obvious to ignorant people? That stuff takes time to figure out on an individual basis. You can't just update your emu or database because the mapping isn't with the emu.
Quote:
Famicom games were not licensed. If you are including them you need to change your terminology because that increases the scope of the games you will handle.
That's not true. Licensed famicom games didn't include lockout chips, but they were licensed.
Quote:
But your new idea does not ease the Super Mario Bros format cost, while making Super KKK Bros completely incompatible. This is why listing those games that fail with iNES 1.0 would be helpful, to see if it actually is a benefit given a header would still be needed in so many cases anyways.
Read the thread, there was never any intention of dropping iNES support for unlicensed games. You can suggest a better format for unlicensed games any time you want.
What is the difference between Nintendo boards and unlicensed board? Why do they "NEED" a seperate anything even if this happened? This is just one huge derp. >.>
ibeenew2 wrote:
FitzRoy wrote:
3gengames wrote:
Because 1.0 takes care of 99.99% of games.
More like 98%, but 2% is 50 games
Listing those 50 games would go a long way to promoting this idea.
Agreed. I'd like to put a "List of games where iNES fails" on the wiki, linked from the NES 2.0 page. FF1+2 would be a good one, as are any other SOROM, SXROM, and EWROM games.
Quote:
Famicom games were not licensed.
I thought every Famicom game using a Nintendo-made board was "licensed", even if no lockout chip was involved. What's the proper terminology for this?
Memblers wrote:
But then to make iNES to unif conversion possible, you need a database, but with that same database you're already loading an iNES file
But the tool to convert ROMs using a database can be separate from the emulator.
I think the whole "frankenfile" thing is that iNES and NES 2.0 is a container, and nothing else uses this container. FitzRoy wants to change to a more widely used container, namely PKZIP.
FitzRoy wrote:
ibeenew2 wrote:
Listing those 50 games would go a long way to promoting this idea. Looking at the iNES 2.0 discussions there are far fewer than that in the licensed set which need to be handled.
That's not the only issue. Missing or incorrect headers are the larger issue. Think about how many new dumps have been released since GoodNES that never got headers added to them. Now think about how files are distributed. If you modify a header for shits and then share it, it gets shared a hundred more times. Eventually, every complete set offered was completed from some outdated or scrupulous sources. You're bound to have some fucked up headers. So 50 years from now, someone goes to play one of these files, it breaks, and they'll have to learn about headers to know it was because of that, then how to update them, find bootgod's database... Where is all that obvious to ignorant people? That stuff takes time to figure out on an individual basis. You can't just update your emu or database because the mapping isn't with the emu.
You want the emulator to do a checksum and look up mapping information in a database. How is that any different from an emulator doing a checksum and looking up a header in a database? If bad headers are the problem, then an emulator can just always ignore them and use its own. Doesn't Nestopia (among others) already do this? When the checksum isn't in the database then trust the header. If that is the larger issue it has a much easier solution than a whole new format that ignores so much.
tepples wrote:
ibeenew2 wrote:
Famicom games were not licensed.
I thought every Famicom game using a Nintendo-made board was "licensed", even if no lockout chip was involved. What's the proper terminology for this?
And what about companies that used their own boards and fabrication facilities, like Konami? Their Famicom carts have zero Nintendo chip/boards/plastics, no Nintendo product code, no Nintendo logo on the label, etc. Still licensed?
ibeenew2 wrote:
tepples wrote:
I thought every Famicom game using a Nintendo-made board was "licensed", even if no lockout chip was involved. What's the proper terminology for this?
And what about companies that used their own boards and fabrication facilities, like Konami? Their Famicom carts have zero Nintendo chip/boards/plastics, no Nintendo product code, no Nintendo logo on the label, etc. Still licensed?
I guess any Famicom game publisher whose North American subsidiary was "licensed" on the NES would be considered "deutero-licensed" by extension on the Famicom, in the sense of the
deuterocanonical books of the Christian Bible.
ibeenew2 wrote:
You want the emulator to do a checksum and look up mapping information in a database. How is that any different from an emulator doing a checksum and looking up a header in a database? If bad headers are the problem, then an emulator can just always ignore them and use its own. Doesn't Nestopia (among others) already do this?
So you believe the average person can be expected to know:
(a) the difference between a header problem and not an emulation bug
(b) that the option called "database..." which is vague enough to entail anything is actually an option to override their header values with those in a database.
(c) that the database supplied with the emulator is trash and the one you need is located on a website the author doesn't link to.
So how many people get past (a)? That's the difference between theory and practice. This nonsense would never work in practice because it requires too much foreknowledge. With my method, an emulator can just implement the database and say "use a ZapFC romset for less bugs". Think the average person can download roms?
I can't keep arguing with people who can't see the forest for the trees. The people responding aren't even the people I need to convince.
FitzRoy wrote:
So you believe the average person can be expected to know:
(a) the difference between a header problem and not an emulation bug
Would an option on the Help menu "Troubleshoot Game Bugs" work?
Quote:
(b) that the option called "database..." which is vague enough to entail anything is actually an option to override their header values with those in a database.
A header describes a game's circuit board. Would "Board List" be a better name for a list of such headers?
Quote:
With my method, an emulator can just implement the database
Yeah, and implement an old version of
your database.
Quote:
and say "use a ZapFC romset for less bugs".
That or "use FitzRoy's board list with your existing romset for less bugs". Good luck finding individual ZapFC roms; most sites that Google finds have standardized on iNES. Don't get me wrong; I fully support your effort to create a list of the correct headers for North American releases carrying the Official Nintendo Seal. It's just that I'm trying to find the most pragmatic solution that results in the least end user interruption.
tepples wrote:
Quote:
OS-native file load dialog
On a few operating systems I can think of, your application can't even open a file unless the user has chosen it from the OS-native file load dialog. Otherwise you get a privilege violation.
That's completely impractical and, dare I say, impossible.
fopen("log.txt", "w") PRIVILEGE VIOLATION
What operating systems are these, so that I may tell all of my friends, for purposes of pointing and laughing?
kode54 wrote:
tepples wrote:
On a few operating systems I can think of, your application can't even open a file unless the user has chosen it from the OS-native file load dialog. Otherwise you get a privilege violation.
That's completely impractical and, dare I say, impossible.
fopen("log.txt", "w") PRIVILEGE VIOLATION
Instead, you're supposed to log to stderr or another system-provided mechanism.
Quote:
What operating systems are these, so that I may tell all of my friends, for purposes of pointing and laughing?
One of them is JavaScript used with the HTML DOM. The only local file that a script can read through
W3C File API is the one the user chose in an <input type="file"> element.
Another is Sugar, used on the XO-1 laptop. Any program can
read and write files that the user chooses. But a program can't write arbitrary files except to /tmp, so that a rogue program can't overwrite all the user's files. And the permission to read arbitrary files in the user's home directory and the permission to connect to a network cannot both be granted to a program at install time so that a rogue program can't disclose all the user's private files to an attacker; one or the other privilege has to be assigned later.
Okay, I stand corrected. If you don't consider rewriting your programs into weak-sauce toys to be impractical, then you're right.
FitzRoy wrote:
So you believe the average person can be expected to know:
(a) the difference between a header problem and not an emulation bug
No, I expect the EMULATOR to do this. Completely independent from the average person. Just as you expect the emulator to do the work.
FitzRoy wrote:
(b) that the option called "database..." which is vague enough to entail anything is actually an option to override their header values with those in a database.
Not an option, just part of the normal iNES file loading. Check header database, if not in database then use file header. Nestopia already uses checksums to ignore some headers automatically. It could even fix the header so fewer bad roms are spread, again separate from user actions.
FitzRoy wrote:
(c) that the database supplied with the emulator is trash and the one you need is located on a website the author doesn't link to.
How is this solved with your idea? You still need cooperation from the emulator author. They still need to make sure they include a good version of your db instead of an old trash one.
FitzRoy wrote:
an emulator can just implement the database and say "use a ZapFC romset for less bugs".
An emulator can just implement the database (of correct iNES headers) and say "use any romset for less bugs" because it will ignore the incorrect headers.
None of this needs anything from the user, including requiring them to find new files to replace the ones that already work.
ibeenew2 wrote:
FitzRoy wrote:
So you believe the average person can be expected to know:
(a) the difference between a header problem and not an emulation bug
No, I expect the EMULATOR to do this. Completely independent from the average person. Just as you expect the emulator to do the work.
You are so not getting this. The Nestopia feature requires the user to know things he can't possibly know and make conclusions he won't possibly come to. Explain to me how a person having a mapper issue KNOWS its a mapper issue and not an emulator bug to report to the author?
Quote:
Not an option, just part of the normal iNES file loading. Check header database, if not in database then use file header. Nestopia already uses checksums to ignore some headers automatically. It could even fix the header so fewer bad roms are spread, again separate from user actions.
How do they know that? Were they born with that knowledge? What part of "Database..." tells them its a header override feature and not some other thing? The author knows what it's for because he made. We know it's for because we're knee deep in this shit every day.
Quote:
How is this solved with your idea? You still need cooperation from the emulator author. They still need to make sure they include a good version of your db instead of an old trash one.
You're the one telling me that Nestopia is an equivalent idea. I'm telling you what Nestopia does. Are you saying the database should be supplied with the emulator and used by default? I AGREE. THAT IS MY METHOD.
Quote:
An emulator can just implement the database (of correct iNES headers) and say "use any romset for less bugs" because it will ignore the incorrect headers.
But then what the fuck do licensed games need headers for at that point? You may as well not require them so that people whose lives are made easier by having data in its original form can do so.
FitzRoy wrote:
You are so not getting this. The Nestopia feature requires the user to know things he can't possibly know and make conclusions he won't possibly come to. Explain to me how a person having a mapper issue KNOWS its a mapper issue and not an emulator bug to report to the author?
You are so not getting this. I am saying Nestopia only goes half way, but if iNES really was this huge problem it could use a full header database. If bad headers is the problem you are trying to fix then ignoring/replacing the file headers is far easier than a new file format that is incomplete by design.
FitzRoy wrote:
Quote:
Not an option, just part of the normal iNES file loading. Check header database, if not in database then use file header. Nestopia already uses checksums to ignore some headers automatically. It could even fix the header so fewer bad roms are spread, again separate from user actions.
How do they know that? Were they born with that knowledge? What part of "Database..." tells them its a header override feature and not some other thing? The author knows what it's for because he made. We know it's for because we're knee deep in this shit every day.
Thats why it should be AUTOMATIC, not an option. This is an emulator design problem, not an iNES file format problem.
FitzRoy wrote:
Are you saying the database should be supplied with the emulator and used by default? I AGREE. THAT IS MY METHOD.
Your method is based on a whole new file format, instead of a different database that would work with every existing game.
FitzRoy wrote:
You may as well not require them so that people whose lives are made easier by having data in its original form can do so.
How is having the data in its original form but unable to change more useful? As you said, if you change it, you need the header anyways.
Really if using one of the many simple iNES splitters is too hard for users then they wouldn't be able to do anything with the data.
FitzRoy wrote:
Explain to me how a person having a mapper issue KNOWS its a mapper issue and not an emulator bug to report to the author?
The troubleshooter that I mentioned would start to explain the difference. Imagine, if you will, a short video of a good dump of Super Mario Bros. with the wrong mirroring.
Quote:
What part of "Database..." tells them its a header override feature and not some other thing?
That's why I recommended using a more descriptive name than "Database".
Quote:
Quote:
An emulator can just implement the database (of correct iNES headers) and say "use any romset for less bugs" because it will ignore the incorrect headers.
But then what the fuck do licensed games need headers for at that point?
To distinguish an iNES container (16 bytes, first four are "NES\x1A") from any other container. Even your recommended PKZIP container has a header, so to speak. Also so that the ROM, after having its header corrected, will work on emulators that run on platforms too small for an emulator to contain a list of all verified good dumps' headers. Much of your argument appears to boil down to "Microsoft distributes a PKZIP splitter but not an iNES splitter with Windows."
By the way about licensed vs unlicensed etc with this database. Part of the community NEEDS headers because they modify games for whatever their purpose. It is unfeasible for them to get every possible modification added into a database. The header works just fine. Plus who cares if you do or don't want the header. Don't want it? Ignore those 16 bytes. Problem solved. Ofcourse you will have the issue of when does PRG stop and CHR begin unless you just use the checksum/hash of both to identify the game. But then your emulator can't play any modified games that aren't in your database.
MottZilla wrote:
Ofcourse you will have the issue of when does PRG stop and CHR begin unless you just use the checksum/hash of both to identify the game. But then your emulator can't play any modified games that aren't in your database.
That's why he suggested to have the PRG rom and CHR rom separate and zip them. I really don't see the reason to do that when you're not allowed to edit the files, though. Also, as the database will in essence just be a collections of headers, you can store the size of the roms there, so there really is no good reason to have them split in the suggested format.
The only reason I've been presented with for having them split in his format is to make it easier for rom dumpers to spot bad dumps, which is odd, as the dumper can perform the checksums before merging the roms. As for the new game dumps which doesn't have a checksum in the database yet; why bother? They cannot be played in this format anyway... Not in database = no play.
In a new format which allows for all kinds of games, I could understand having them split, though I don't think using a rom splitter can be that much of an hassle.
The term frankenfile keeps getting tossed around. Why? INES files are in no way complex to split or even hex edit yourself.
ibeenew2 wrote:
You are so not getting this. I am saying Nestopia only goes half way, but if iNES really was this huge problem it could use a full header database.
But that's inconsiderate to those who want to work with and store this data in its native form without specialty tools. It's also not an archival format because you're keeping the data in a form that was different from the way it was released by the manufacturer. If these were Virtual Console dumps, I'd include the manufacturer header, but they're not. I can't validate virgin data and get it used/preserved because it's been made unusable in that form... for no reason. And your only excuse to perpetuate that is that it would be easier to keep it that way than download a clean set.
Quote:
If bad headers is the problem you are trying to fix then ignoring/replacing the file headers is far easier than a new file format that is incomplete by design.
That's like saying the NES emulator itself is incomplete by design because it will never emulate clone systems. Define what the NES is as it pertains to preservation. Can you do that for me? Does it include toasters and calculators? What is NES hardware? Everything in the world or something more exclusive?
Quote:
How is having the data in its original form but unable to change more useful? As you said, if you change it, you need the header anyways.
For the last and final time, this does not replace iNES or mitigate the need for a second format. There is no such thing as rom-side mapping that guarantees proper mapping. It is an oxymoron. You cannot put users in charge of mapping and expect a mapping guarantee with traded files! Finite libraries can overcome that with a database and infinite libraries can't! A single format that achieves guaranteed mapping for both libraries is not a rebuttal, it's a unicorn, it's a wish, it's a dream you had one night with no regard for reality. I want CHR and PRG data to be CONTAINED, not COMBINED, hacks to be patches, and external mapping files required for unlicensed games only.
3gengames wrote:
The term frankenfile keeps getting tossed around. Why?
FitzRoy appears to use the terms "frankenfile", "specialty tools", and "contained not combined" to imply that container formats used by more than one application are superior to container formats used by only one. For example, PKZIP format is used by MAME, LibreOffice (as odt and ods), Winamp (as wsz), StepMania (as smzip), Java (as jar), and numerous other applications, and tools for manipulating zipfiles come with every major PC operating system. The iNES format is used by NES emulators and nothing else, and it needs "specialty tools" to pack and unpack.
@FitzRoy: Does the above paragraph correctly represent your point of view?
And as for "hacks to be patches", how would you propose to handle doubling a 128 KiB UNROM to a 256 KiB UOROM? The patch to double the PRG would contain the entire fixed bank, including the copyrighted parts. Or how would you propose to handle a ROM hack created from modifying a disassembly and reassembling it? This would end up moving most bytes in the file.
My main complaint with ZIP'ed ROMs is because the NES is a computer itself, it seems counter-productive to use a format that the system itself can't work with.
Things that could be part of the header/database:
ROM hash
filename
date (if known)
revision
board name/number
mapper name/number
iNES header
Maybe the problem is that iNES header is a ... header.
Would making it a footer solve the problems? How about a margin note?
Memblers wrote:
My main complaint with ZIP'ed ROMs is because the NES is a computer itself, it seems counter-productive to use a format that the system itself can't work with.
The NES itself doesn't understand the iNES header any more than it understands the PKZIP header. So I guess your complaint boils down to the fact that iNES is implemented in the PowerPak and PKZIP is not.
@FitzRoy: Please don't miss the last comment on the previous page. I want to know whether my interpretation of "frankenfile" and related terms is correct.
Layne's Law of Debate states roughly that no progress can be made in a debate until definitions of terms are agreed upon, and as a moderator, it is my duty to help participants understand one another.
@cpow: FitzRoy wants to make it the equivalent of a table in the appendix instead of a chapter header.
FitzRoy wrote:
ibeenew2 wrote:
You are so not getting this. I am saying Nestopia only goes half way, but if iNES really was this huge problem it could use a full header database.
But that's inconsiderate to those who want to work with and store this data in its native form without specialty tools.
Anyone working with the data will already need these "specialty tools" to generate the iNES header or IPS patch. If I have a ZapFC game and change one byte, it is no longer valid, so I would need other tools. So your idea is only considerate to people wanting to play the licensed games, and inconsiderate to much of the nesdev community.
FitzRoy wrote:
Quote:
If bad headers is the problem you are trying to fix then ignoring/replacing the file headers is far easier than a new file format that is incomplete by design.
That's like saying the NES emulator itself is incomplete by design because it will never emulate clone systems. Define what the NES is as it pertains to preservation. Can you do that for me? Does it include toasters and calculators? What is NES hardware? Everything in the world or something more exclusive?
NES emulators are expected to run games from the original NES console. Your idea rejects hundreds (thousands?) of those games that were released during the NES era. It also rejects all homebrew games being created now. That is what makes your idea incomplete by design.
FitzRoy wrote:
You cannot put users in charge of mapping and expect a mapping guarantee with traded files! Finite libraries can overcome that with a database and infinite libraries can't!
For the last and final time, put emulators in charge of mapping, not users. That way the entire known game set can be put in the database instead of just a subset.
FitzRoy wrote:
I want CHR and PRG data to be CONTAINED
The CONTAINER is called iNES, a standard and easy container to work with.
FitzRoy wrote:
hacks to be patches
Does IPS even handle patching multiple files at once?
FitzRoy wrote:
external mapping files required for unlicensed games only.
Why not external mapping files required for unknown games only, like with a good header database?
cpow wrote:
Maybe the problem is that iNES header is a ... header.
Would making it a footer solve the problems? How about a margin note?
:shock:
Making it a footer would actually make things more difficult and emulator ROM load-time slower. One would have to seek to the end of the file + use ftell() (to find out how big the file is), followed by seeking {length-16} then reading 16 bytes, making sure the results match what is considered an iNES header, etc...
The whole point of a header is to allow software to determine what to do with the remaining data in the file.
I do have problems/qualms with having a separate file used for this purpose (e.g. the equivalent of having the iNES header in its own file, followed by the ROM (PRG+CHR) in a separate file). Those qualms might be considered minor to some, but to others are major. The qualm is that it's a waste of an inode (thus wasted space, depending on the filesystem block size, which these days can vary from 4KBytes to 64KBytes), and this plays a huge role on embedded systems or systems with limited space/capacity (such as small CF cards, etc.). This acts as further justification as to why the iNES 2.0 method is superior (in this specific regard).
EDIT: Also for what it's worth, I side with Memblers with regards to using ZIP as a container method. But I also fully agree with byuu (I think?) who said tar and equivalent was a travesty as well. ZIP might be the easiest to implement, but there's all sorts of caveats when it comes to the specification (consider 8.3 format vs. long filenames, etc.). Maybe RAR might work better? I don't know about licensing / complexities of these formats. Let's just stay with iNES 2.0 and be done with it ;-)
tepples wrote:
The troubleshooter that I mentioned would start to explain the difference. Imagine, if you will, a short video of a good dump of Super Mario Bros. with the wrong mirroring.
Take time to read a doc that links to a video to watch about ines headers? Aw jeez. iNES doesn't need to exist solely to be a rite of passage.
Quote:
That's why I recommended using a more descriptive name than "Database".
Like header database? What's a header? You take too much of your knowledge for granted. If we had the choice to do this all over again, users wouldn't need to know somehow that they were responsible for mapping. And they wouldn't have had to update their roms to reflect mapping updates. We have a near complete database TODAY, but back then we didn't, and it made rom-side mapping even more burdensome.
Quote:
To distinguish an iNES container (16 bytes, first four are "NES\x1A") from any other container. Even your recommended PKZIP container has a header, so to speak. Also so that the ROM, after having its header corrected, will work on emulators that run on platforms too small for an emulator to contain a list of all verified good dumps' headers. Much of your argument appears to boil down to "Microsoft distributes a PKZIP splitter but not an iNES splitter with Windows."
ZapFC is a double extension format, it already differentiates itself from a regular zip container. "Frankenfile" is a term I use for distinct parts combined in a unique or arbitrary way.
FitzRoy wrote:
If we had the choice to do this all over again, users wouldn't need to know somehow that they were responsible for mapping.
If we "had it to do all over again" we would use iNES 2.0, but possibly with a larger header that would also include things like:
1) Dumper (person)
2) Dumper device
3) Date of dump
4) Canonical title of game, its region code, language, etc...
It seems silly to me to embed this logic into the emulators, or require yet another data file, your magic database.
That is like hard-coding the directory contents of a PKZIP file into PKZIP and then stripping it out of the PKZIP file.
Would someone please Godwin this damned thread already?
Quote:
Would someone please Godwin this damned thread already?
You know who else had a solution which involved putting things in containers...
seriously though.. if you have a new container format, release a whole set of roms in that format and also make a great emulator which supports those roms
even then i doubt anyone would change from iNes 1.0
Quote:
If we "had it to do all over again" we would use iNES 2.0, but possibly with a larger header that would also include ...
it could be possible to make a iNES 2.1 which adds more meta data to the end of the file, since extra CHR is just ignored usually
But if there can be no arrangement, then we are at an impasse.
(Princess Bride)
clueless wrote:
If we "had it to do all over again" we would use iNES 2.0,
Umm, no. Ever heard the term "hindsight is 20/20"? There is a fundamental risk with rom-side mapping. You know it now and it should have been known then.
Quote:
but possibly with a larger header that would also include things like
1) Dumper (person)
2) Dumper device
3) Date of dump
4) Canonical title of game, its region code, language, etc...
Lawl. No. This information should be referenced on external lists. Games should not be archived this way any more than books should be archived with your notes in or on the pages. Not to mention you could theoretically have thousands of dumpers for a single game, and not everyone trusts the same people. Not to mention that updating such data would change the recorded checksum of the game each time that you did, seeing as how you made the header integral to operation and I can't exclude it as part of the checksum calculation.
Quote:
It seems silly to me to embed this logic into the emulators, or require yet another data file, your magic database.
That is like hard-coding the directory contents of a PKZIP file into PKZIP and then stripping it out of the PKZIP file?
You lost me. What could be sillier than each multi-rom system coming up with its own totally unique way of storing the data? What kind of idiot is so centric to his own system that he would doom an archiver of all systems to battling a bunch of shitty tools just to see why a new dump isn't matching an old one? This is the idea you're defending. It is laughable. It's embarrassingly short-sighted is what it is. Thank Jesus on a pogo stick that MAME was at least smarter than that.
FitzRoy wrote:
clueless wrote:
If we "had it to do all over again" we would use iNES 2.0,
Umm, no. Ever heard the term "hindsight is 20/20"?
Yes, iNES 2.0 was brought up because of this and it fixes pretty much everything.
Fitzroy, your passion for your argument has been exceeded by your rudeness. Its time (imho) to put up or drop the issue. Actions speak louder than words.
Most of the popular NES emulators are open-source. Make patches for them. Create your database (with the checksums of ALL commercial games that you can). Create a tool to strip off the iNES header from various ROMs. Prove that you can make a working prototype. Submit patches and argue your point with the emulator maintainers.
You are highly unlikely to convince most of us to rush out and implement your idea for you. Your beating a dead horse. So prove your point and resurrect it.
Quote:
EDIT: Also for what it's worth, I side with Memblers with regards to using ZIP as a container method. But I also fully agree with byuu (I think?) who said tar and equivalent was a travesty as well.
Yeah.
ZIP supports multiple encryption types, there are compressors (such as 7-zip) that can make ZIP files other tools cannot read, on and on. You can try and enforce no-compression, no-encryption, but the second someone uses zlib or whatever and doesn't bother to check, that program supports compression anyway, and people start relying on it. And now every last emulator and tool has to waste hundreds of KB of source code and third-party libraries to work with these files. Repeat for every compression format used: ZIP, 7-zip, RAR, BZ2, GZ, etc.
ZIP also doesn't define the specs of filename storage, you can't know whether the filename is UTF-8, SJIS, or something else.
TAR is just a mess. Dozens of legacy fields and Unix-specific stuff that makes creating the containers a pain in the ass.
There really is no popular, simplistic container format that end users can easily add and remove files with. Which is why most applications end up using ZIP.
And end users being able to work with the archives easily is the only thing separating iNES from ZIP. And from a permanent preservation standpoint, there's no telling how long ZIP will remain dominant. Imagine finding a bunch of ARC or LHA files today. Or video in Intel Indeo Video. Sure, you can work with them, but they are certainly obscure.
I feel that all containers are simply reinventing the file system. A format that stores multiple independent files is called a folder. And compression can be applied automatically and recursively right through modern file systems like NTFS, completely transparently to individual applications.
The advantage of containers is for distribution, which can be handled better. Give a way to create a download link on a folder, and have the file server send the contents in compressed-form. Allow e-mail clients to attach folders and do the same. Why are end users concerning themselves with what compression format to use, anyway?
Look at Apple OS X. Programs are named such as TextEdit.app, the .app is really a folder, with individual files inside. The OS treats it like a file, and that's how the end user interacts with it. Now say iNES were like that, and inside the header was its own file. You want compatibility? Include .ines1, .ines2, .xml files that each describe the game, AND let an emulator database override those on commercially released games if it so chooses, so the user doesn't have to update them.
But back to the real world, where nothing can change, we have iNES and ZIP.
Yes, people will stick junk in the folders that tools will have to ignore. But then that's just as likely to happen to a common container format like ZIP. Since most people are not idealists, they will ignore extra files, not reject the archives entirely. You'll still get downloaded-from-romsite.txt.
byuu wrote:
The advantage of containers is for distribution, which can be handled better.
That's the biggest reason I dislike using folders as containers. But you know, ".fc" folders are an easy thing to support without having to be distributed as its own set. Because tools like WinRAR can "extract each selected archive to its own folder" en masse. You realize that, right?
You'd have to treat them just like files still, not new places to put things in.
Quote:
Yes, people will stick junk in the folders that tools will have to ignore. But then that's just as likely to happen to a common container format like ZIP. Since most people are not idealists, they will ignore extra files, not reject the archives entirely. You'll still get downloaded-from-romsite.txt.
But tools won't be forced to ignore it. If you stick junk in your container, it won't work in emulators because it's an invalid format according to my spec calling for any container having filenames other than CHR or PRG to be rejected before calculation rejects it for sure. You think distributors will distribute non-working files?
Quote:
And from a permanent preservation standpoint, there's no telling how long ZIP will remain dominant.
Isn't there a law of diminishing returns or something with regards to compressing binary data? Or are you saying that 100 years from now, we'll reach 1% of the original filesize and ZIP will be abandoned for sheer inefficiency?
Quote:
Because tools like WinRAR can "extract each selected archive to its own folder" en masse. You realize that, right?
That is why I support the idea. People will say what about redistribution once they have the files? Compress them back. Hell, solid-archive the whole damn set and get some major space savings. How long can it possibly take to compress a 200MB set? Imagine being able to validate your entire collection by the hash of one solid archive file?
Quote:
But tools won't be forced to ignore it.
Nor would they be for extra files in folders. I'm not saying it's the sure-proof method here. There is too much inertia in getting the OS to treat a folder like a file.
I must say though, the option to use a .fc folder or a .fc.zip container (for emulators that support ZIP), sounds very promising.
Quote:
Isn't there a law of diminishing returns or something with regards to compressing binary data? Or are you saying that 100 years from now, we'll reach 1% of the original filesize and ZIP will be abandoned for sheer inefficiency?
7-zip packs files 10-30% better on average, and solid-archiving provides massive benefits. Especially with barely-changed region variants of game images.
I think in the future techniques like PAQ will catch on. Where specific compression codecs are used for specific files. Compress WAV files with a FLAC-like codec, bitmaps with something like PNG, text files with a burrows-wheeler approach, and so on. Combine with using more CPU power now that it's available and things will keep improving.
I'm sure many thought MPEG-2/XviD was amazing until H.264 came along. So you never know what can happen.
byuu wrote:
Nor would they be for extra files in folders. I'm not saying it's the sure-proof method here. There is too much inertia in getting the OS to treat a folder like a file.
You responded before I could edit, but I was going to give ease of future validation as the reason tools control distributors and not vice versa. But then I realized, according to my spec, emulators wouldn't play the files either. So there is truly no way unverifiable, broken roms would get distributed.
So if I understand what it is you're saying, you'd go out of spec for the folders because you don't think it would affect distribution. And you'd be right because it would be the least used variant. God that's underhanded.
Teal Deer alert: I know this is too long; didn't read. But there were a lot of posts to reply to since I went to bed.
ibeenew2 wrote:
FitzRoy wrote:
ibeenew2 wrote:
I am saying Nestopia only goes half way, but if iNES really was this huge problem it could use a full header database.
But that's inconsiderate to those who want to work with and store this data in its native form without specialty tools.
Anyone working with the data will already need these "specialty tools" to generate the iNES header or IPS patch. If I have a ZapFC game and change one byte, it is no longer valid, so I would need other tools.
But only the
authors of hacks would need such tools: extract the .prg and .chr from the pristine ROM and the .ines from the database, make the hack, and then make and distribute a patch against the .prg and .chr files. I think the idea is that the
end user of a hack obtains the pristine ROM in ZapFC format and then uses in-emulator IPS support to patch the PRG or CHR at runtime.
Quote:
FitzRoy wrote:
Quote:
If bad headers is the problem you are trying to fix then ignoring/replacing the file headers is far easier than a new file format that is incomplete by design.
That's like saying the NES emulator itself is incomplete by design because it will never emulate clone systems. Define what the NES is as it pertains to preservation. Can you do that for me? Does it include toasters and calculators? What is NES hardware? Everything in the world or something more exclusive?
NES emulators are expected to run games from the original NES console. Your idea rejects hundreds (thousands?) of those games that were released during the NES era. It also rejects all homebrew games being created now. That is what makes your idea incomplete by design.
Emulators would continue to support iNES. They would also support zipped .prg+.chr+.ines, and they would support zipped .prg+chr without .ines for games that are on someone's canonical list of good headers. FitzRoy would make the list of North American NES games with the Official Nintendo Seal. Someone else would make the list of Famicom games using Nintendo boards or from publishers that were licensed in North America (e.g. Konami), and someone else would make the list of North American NES games from the well-known unlicensed publishers (Tengen, Color Dreams, Camerica, AVE).
FitzRoy's big problem appears to be that the proposal for switching from the iNES container to split PRG and CHR in a PKZIP container and the proposal for
allowing games with the Official Nintendo Seal to be distributed without the header have been conflated into a single proposal. One step at a time please.
Quote:
For the last and final time, put emulators in charge of mapping, not users.
"Put emulators in charge of mapping" gave me an idea for an interesting research project to make an emulator that automatically guesses what board a game expects. For example, if it routinely writes to ROM locations with the same value, it's probably a discrete mapper of some sort. If a game routinely writes to columns in the middle of the screen as the screen scrolls, it needs its mirroring switched to vertical.
Quote:
That way the entire known game set can be put in the database instead of just a subset.
After the entire known game set is published, someone's going to discover a prototype and the set will become obsolete. FitzRoy wants to start with a subset that won't grow, and then other subsets can be added to it.
Quote:
FitzRoy wrote:
I want CHR and PRG data to be CONTAINED
The CONTAINER is called iNES, a standard and easy container to work with.
But nothing else uses this container. 7-Zip can't open it, nor can File Roller, and libraries to manipulate it don't come with every *n?x distribution. It can't contain a box, label, or manual, or anything else needed to reproduce the original CIB. Nor does iNES even handle the 8 KiB PRG of Galaxian.
Quote:
FitzRoy wrote:
hacks to be patches
Does IPS even handle patching multiple files at once?
No. Nor does it handle expanding the PRG and moving the CHR to compensate. A new binary diff format is needed; I'd recommend a zipped collection of IPS files or (better yet) bsdiff files.
Quote:
FitzRoy wrote:
external mapping files required for unlicensed games only.
Why not external mapping files required for unknown games only, like with a good header database?
"Known games" would start with FitzRoy's set of licensed games. Then you could download someone else's Tengen+Camerica+Color Dreams+AVS set.
koitsu wrote:
cpow wrote:
Would making it a footer solve the problems? How about a margin note?
Making it a footer would actually make things more difficult and emulator ROM load-time slower.
That was a joke son, but PKZIP does indeed use a footer. But on platforms other than the NES and the GBA, an emulator can load the whole file into memory first.
koitsu wrote:
One would have to seek to the end of the file + use ftell() (to find out how big the file is)
By which point the file's extent list or FAT entries are in the operating system's disk cache.
Quote:
The qualm is that it's a waste of an inode
Switch to a PKZIP container with .prg, .chr, and .ines files, and you don't waste an inode.
Quote:
ZIP might be the easiest to implement, but there's all sorts of caveats when it comes to the specification (consider 8.3 format vs. long filenames, etc.).
The zipfile libraries in operating systems and programming languages appear to handle these caveats well.
Quote:
Maybe RAR might work better?
RAR? Hell no. Its license is non-free. If you want to improve on PKZIP, use 7-Zip.
Quote:
Let's just stay with iNES 2.0 and be done with it
Sure, if you're willing to provide the split and join tools. Yes, join
tool, because it would ignore the file sizes in the provided header and properly double-up Galaxian's PRG.
FitzRoy wrote:
tepples wrote:
That's why I recommended using a more descriptive name than "Database".
Like header database? What's a header?
Like "board list", as I recommended earlier. Perhaps users might understand if I write the whole troubleshooter entry:
"All NES Game Paks contain printed circuit boards with ROM chips and other circuitry. A game will work only on the right board. Some ROM files come with board information, which may be correct or wrong; others don't have board information at all. $emulatorname uses a board list to determine the correct board to emulate for any game on the list. A board list for all North American NES games with the Official Nintendo Seal is available from FitzRoy's web site."
Quote:
ZapFC is a double extension format, it already differentiates itself from a regular zip container.
As do .jar, .odt, .smzip, etc. So the big change from .nes to .zapfc is that the container is switched from the ad-hoc iNES to .zip, and an option is available to distribute ROMs without board information, relying on a separately distributed board list. Then you plan to make and distribute a board list for all North American NES games with the Official Nintendo Seal.
Quote:
"Frankenfile" is a term I use for distinct parts combined in a unique or arbitrary way.
In other words, I now understand that "frankenfile" = a container not widely used. Thank you for clarifying.
frantik wrote:
seriously though.. if you have a new container format, release a whole set of roms in that format and also make a great emulator which supports those roms
A tool to convert a whole set of ROMs from iNES to zipped split ROMs would be easy for me to make in Python, and leaving the 16-byte .ines file out of the result would give .zapfc exactly as FitzRoy appears to describe.
FitzRoy wrote:
clueless wrote:
but possibly with a larger header that would also include things like
1) Dumper (person)
2) Dumper device
3) Date of dump
4) Canonical title of game, its region code, language, etc...
Games should not be archived this way any more than books should be archived with your notes in or on the pages.
Books have a cover, and they have a
title page and verso.
I agree with clueless: make your board list, and patch an emulator to support zipped split ROMs.
3gengames wrote:
iNES 2.0 was brought up because of this and it fixes pretty much everything.
Except for the problem of widely used ROM sets with incorrect board information.
byuu wrote:
ZIP supports multiple encryption types, there are compressors (such as 7-zip) that can make ZIP files other tools cannot read
Require .zapfc files to use a baseline ZIP format: either uncompressed or DEFLATE with window no bigger than 32 KiB, and no crypto. What other "etc" were you thinking of that interferes with defining baseline ZIP?
Quote:
ZIP also doesn't define the specs of filename storage, you can't know whether the filename is UTF-8, SJIS, or something else.
That's why zapfc would define a profile of ZIP where all filenames are UTF-8, not shit JIS.
Quote:
And from a permanent preservation standpoint, there's no telling how long ZIP will remain dominant. Imagine finding a bunch of ARC or LHA files today.
The PKZIP 2 format has been around for 18 years. It's an international standard, part of
ISO/IEC 26300:2006 Open Document Format for Office Applications (OpenDocument) v1.0. It's far less obscure than the iNES container and on the same standardization footing as the ISO 9660 file system used on CDs.
Quote:
I feel that all containers are simply reinventing the file system.
Different file systems for different purposes. In fact, I invented my own container for files inside a Game Boy Advance game because I had specific needs.
Quote:
Hell, solid-archive the whole damn set and get some major space savings.
As I understand it, solid archiving saves only when multiple files share data, which is good for region variants but little else. And you need to extract them anyway before playing, as on average half the archive needs to be read to extract one file.
Quote:
I'm sure many thought MPEG-2/XviD was amazing until H.264 came along. So you never know what can happen.
I know what will happen: MPEG is working on two new standards. One is HEVC (aka H.265), which is expected to run at half the bitrate of AVC (aka H.264) for a given quality level. The other is MPEG Royalty Free, which is expected to compete with Xvid without using any patented techniques that require royalties; this
will begin in March 2011.
tepples wrote:
Memblers wrote:
My main complaint with ZIP'ed ROMs is because the NES is a computer itself, it seems counter-productive to use a format that the system itself can't work with.
The NES itself doesn't understand the iNES header any more than it understands the PKZIP header. So I guess your complaint boils down to the fact that iNES is implemented in the PowerPak and PKZIP is not.
Sorta yeah, but there is no source release for the PowerPak, so I would still have to implement file unzipping by myself (if the NES could even do it with all available WRAM). And for absolutely no benefit at all to the user (compression doesn't matter for something this small, these aren't floppy disks), I surely wouldn't waste my time on that.
I'll write a ROM conjoining tool, later. Heck, if I get really bored and mess with some more C, I'll make a program to remove the headers and make two separate .bin files of the ROMs for you. It'll take a couple minutes if you have all the ROM's in a file and just CTRL+A and then drag them all the the windows application. I'll work on these later maybe. I don't feel like doing anything truthfully.
3gengames wrote:
I'll write a ROM conjoining tool, later. Heck, if I get really bored and mess with some more C, I'll make a program to remove the headers and make two separate .bin files of the ROMs for you. It'll take a couple minutes if you have all the ROM's in a file and just CTRL+A and then drag them all the the windows application. I'll work on these later maybe. I don't feel like doing anything truthfully.
I'm just curious. Why write such a one-time use tool as a GUI? This would be ~20 lines or less in perl or python, or ~40 lines of ANSI-C as a command line app that just takes takes filenames in argv[]. Let the shell expand the wildcard glob for you (well, in Windows you still have to do that yourself, but its pretty trivial, 10 lines of code using the win32 api). A gui app w/ drag + drop seems like overkill to me.
I'm not trying to pick a fight / flamewar. I'm just curious about why a GUI app.
clueless wrote:
A gui app w/ drag + drop seems like overkill to me.
I don't think he meant to make a GUI application, since you can drag files and drop them on command line application EXEs, which causes the dropped files to be passed as parameters.
Yep, working on it now. C is alot better when you get the hang of it.
Even if he doesn't use this...I may.
Sorry to sound extremely sarcastic, but just to say I'm laughing loud at you guys doing a 8-page hot flamed debate on a yet another useless proposal for iNES remplacement that nobody will ever use.
MWAHAHAHAHAHA.
(don't tell me that wasn't constructive I know that, I just couldn't resist)
tepples wrote:
No. Nor does it handle expanding the PRG and moving the CHR to compensate. A new binary diff format is needed; I'd recommend a zipped collection of IPS files or (better yet) bsdiff files.
In this instance, patch conversion would be far easier than the SNES because headers on ".nes" files were a certainty.
Quote:
.prg, .chr, and .ines files
When you create an external mapper format for unlicensed games, I would recommend not basing it off iNES, but rather something which allows custom mappers to be created. iNES (any revision) has no mapping guarantee for licensed games and doesn't allow custom mapping for unlicensed. That is the gist of its problems.
Quote:
.nes to .zapfc
The extension is not .zapfc. It's .fc.zip or .fc.7z. I call the format ZapFC because of how I store the data and calculate the checksums.
Quote:
Require .zapfc files to use a baseline ZIP format: either uncompressed or DEFLATE with window no bigger than 32 KiB, and no crypto. What other "etc" were you thinking of that interferes with defining baseline ZIP?
Aren't those requirements implied by the fact that ZapManager will only validate such a zip? If you can't find a verifiable source set, you went terribly wrong somewhere.
Well until I can name a file from char pointer, (ROM+variable+.BIN) in C, it won't happen. Can't get multi file names so whatever. I give up. And the new custom mapper is easy to add to your own dev emulator and header, then get it filed later to the official iNES mapper sheet. Not like there's millions of new mappers being made anyway.
FitzRoy wrote:
When you create an external mapper format for unlicensed games, I would recommend not basing it off iNES, but rather something which allows custom mappers to be created. iNES (any revision) has no mapping guarantee for licensed games and doesn't allow custom mapping for unlicensed. That is the gist of its problems.
How would one represent a custom mapper other than a netlist with the NES on one side and ROMs and RAMs on the other, which would have to be simulated every CPU cycle (1.8 MHz) and every PPU fetch (2.7 MHz)? Feel free to implement your idea in an emulator.
Quote:
The extension is not .zapfc. It's .fc.zip or .fc.7z.
Even worse. Windows Explorer decides what program to start based solely on the characters after the
last dot. You'll have to associate .zip and .7z files with a workaround program that starts your compression program if no ROMs are found inside.
Quote:
Quote:
Require [fc.zip] files to use a baseline ZIP format: [etc]
Aren't those requirements implied by the fact that ZapManager will only validate such a zip?
I was pointing out to others what a "profile of the ZIP format" might mean. And for which platforms will ZapManager be made available?
3gengames wrote:
Well until I can name a file from char pointer, (ROM+variable+.BIN) in C, it won't happen.
I don't understand what you mean by this, other than snprintf.
Bregalad wrote:
Sorry to sound extremely sarcastic, but just to say I'm laughing loud at you guys doing a 8-page hot flamed debate on a yet another useless proposal for iNES remplacement that nobody will ever use.
MWAHAHAHAHAHA.
(don't tell me that wasn't constructive I know that, I just couldn't resist)
yes i keep coming back to this thread just to see how how manages to keep going. Tepples breathed a lot of life into it with that huge post though :-p
I need to make files that say ROM#.bin, and the variable part can't happen I think? Not from what I tell. :/ Oh well....
Do you mean you don't know how to make strings that say "ROM0.bin", "ROM1.bin", etc.? If so:
Code:
char filename[16];
snprintf(filename, sizeof(filename),
"ROM%d.bin", filenumber);
Can that be put into a fopen somehow? I am needing to name all the files in that order, then a master text file will tell what NES rom is equal to what bin files for this, so they're seperated.
3gengames wrote:
Can that be put into a fopen somehow? I am needing to name all the files in that order, then a master text file will tell what NES rom is equal to what bin files for this, so they're seperated.
Of course:
Code:
char filename[16];
FILE *infp;
snprintf(filename, sizeof(filename),
"ROM%d.bin", filenumber);
infp = fopen(filename, "rb");
Equivalent python:
Code:
filename = "ROM%d.bin" % filenumber
infp = open(filename, "rb")
Okay, just got that right before you posted, except mine is like this:
Code:
sprintf(FileName,"Split/ROM%d.bin",FlagNumber);
ROMBinPointer=fopen(FileName,"w");
Okay, now that that is done....just a tad more work and it will be done, but now it's dinner time. I hope the OP even finds this useful to his goal of making this format happen.
Watch out with the "w", as some C library implementations will translate all $0A bytes into $0D $0A. Better "wb".
Wow, I ran into that problem and added the B and it works now, yet again right before you posted. Man, I suck at this stuff. But here's a semi-working dirty version. WAYYY more to be added yet. One mistake is the files need to be in the same directory, I'll fix that next. But I think it works other then that.
http://www.mediafire.com/download.php?seb0nwstbb5184n
Those %d parameters should be %u (shame on you people!), and the 16-character buffer should probably be PATH_LEN or MAXPATHLEN.
Ugh.....my knowledge of C is about a 5 hour crash course....I suck at it. Sorry. I'll fix that next time.
Quote:
Sorry to sound extremely sarcastic, but just to say I'm laughing loud at you guys doing a 8-page hot flamed debate on a yet another useless proposal for iNES remplacement that nobody will ever use.
MWAHAHAHAHAHA.
(don't tell me that wasn't constructive I know that, I just couldn't resist)
And yet, that doesn't stop people from creating NES emulators and homebrews that secure only 1% of users. In this case, 1% is enough. I would be ecstatic with 1%. What if 90% of users still used Nesticle, even today? Would you be comfortable with that? Would you just accept defeat and develop for Nesticle by coding for all its mapper hacks and wrong behaviors, essentially making your homebrew incompatible with the system and emulators more like the system. Being agreeable to the ignorant masses is more important to you than making sure the most archival option exists for all time? I see you've lost your balls, allow me to fetch them for you.
So here are the steps:
- Write a draft specification for ZapFC (Status: done)
- Write a program that reads an iNES or NES 2.0 format binary and writes it to fc.zip
- Make the board database (Status: done)
- Make CLI ZapManager
- Make a GUI front-end to CLI ZapManager
- Write a short Windows program that determines whether a given .zip file is fc.zip or just plain .zip and starts the appropriate program (emulator or archive manager respectively)
- Patch the big three GPL emulators (Nestopia, Nintendulator, and FCEUX) to load fc.zip and use FitzRoy's board database
Who's up to prototype each of the steps?
EDIT: Updated status
tepples wrote:
So here are the steps:
# Write a draft specification for ZapFC
# Write a program that reads an iNES or NES 2.0 format binary and writes it to fc.zip
# Make the board database ?
Has the link on the first post eluded you this whole time? No wonder you're so confused...
Quote:
# Make CLI ZapManager
# Make a GUI front-end to CLI ZapManager
# Write a short Windows program that determines whether a given .zip file is fc.zip or just plain .zip and starts the appropriate program (emulator or archive manager respectively)
ZapManager isn't a conversion tool, it's a GUI rom validation tool on my website. The link to my website is in the format guide.
As for the value of a conversion tool, it would probably do more harm than good. I guarantee you that if a programmer could help me do step 7, you'd be able to find a set within a month on any rom board or filehost search engine. I'm not a programmer, I'm a dumper.
FitzRoy wrote:
Would you just accept defeat and develop for Nesticle by coding for all its mapper hacks and wrong behaviors, essentially making your homebrew incompatible with the system and emulators more like the system..
I am going to agree with Bregalad here, MWAHAHAHAHA! You bring up homebrew but your idea chooses to ban all homebrews! Broken by design, will never be used, end of story.
ibeenew2 wrote:
I am going to agree with Bregalad here, MWAHAHAHAHA! You bring up homebrew but your idea chooses to ban all homebrews! Broken by design, will never be used, end of story.
See, now you're just trolling. This is the 5th time someone has claimed this and it couldn't be more false. iNES will continue to work for unlicensed games. And who knows, maybe someone will create an external format to replace it completely, then licensed games can use a database and unlicensed can come with a mapper file that allows for custom mappers.
tepples wrote:
7. Write a short Windows program that determines whether a given .zip file is fc.zip or just plain .zip and starts the appropriate program (emulator or archive manager respectively)[/list]
Windows only cares about the last extension (.zip in this case), and an extension can only have one associated program. Leave out the .zip part and instead name it .zfc, or else the emulator will open ALL zip files, complain when it isn't a valid ROM, and the users will curse the emulator because of a design flaw in the format.
However, I don't think this format will catch on solely for the fact that it doesn't support homebrew and unlicensed...
To put it this way, my ROM list wont see an update until I actually see a benefit in using it, and so far, I don't see any benefit yet. (Yes, yes, no bad headers.. I know.. But that hasn't been a problem for me.)
Wkter wrote:
the fact that it doesn't support homebrew and unlicensed...
Are you kidding me with this? What grade are you in?
Thank you for reminding me of a link I had missed. I updated the status of two of the steps. Now I'll address specific points from the spec:
FitzRoy wrote:
For now, I am using the same assignments as iNES.
Case in point: StarTropics and StarTropics 2. These have MMC6, whose behavior differs from other mapper 4 ROMs. WRAM Present = "None" might mislead, as MMC6 certainly has battery-backed WRAM albeit internal to the mapper IC. So I guess an emulator would need to fall back on field 10 (mapper IC) has to be used like a submapper field.
Quote:
It should still be possible to soft-patch games in this format because soft-patching does not interfere with checksum lookup.
How would these soft-patches be distributed? As separate patches to prg and chr? As zipfiles containing a patch to prg and chr? As patches to a virtual file containing two or three concatenated files (a 16-byte file, "prg", and "chr" if present, in that order)? And how would soft-patches that change the board type by adding WRAM, a battery, etc. work?
Quote:
I exclude unlicensed games for the same reason that emulator authors don't bother emulating the endless streams of unlicensed add-ons and clone systems: they are a maintenance nightmare
So the main file contains only games with the Official Nintendo Seal. I understand the reasoning behind this. But if we limit it to only unlicensed games commercially released in North America and Europe prior to the release of the Nintendo 64 console (Sunday Funday yes, Battle Kid no), I don't see how an additional ZapFC_North_American_Unlicensed_(20110216).fbsv file containing Tengen, Camerica, Color Dreams, etc. would be a nightmare to maintain. There's a finite set of these companies because there's a finite set of their CIC defeat circuits.
Wkter wrote:
tepples wrote:
7. Write a short Windows program that determines whether a given .zip file is fc.zip or just plain .zip and starts the appropriate program (emulator or archive manager respectively)
Windows only cares about the last extension (.zip in this case), and an extension can only have one associated program. Leave out the .zip part and instead name it .zfc, or else the emulator will open ALL zip files, complain when it isn't a valid ROM, and the users will curse the emulator
That's exactly why I proposed the short Windows program. The emulator wouldn't see all zip files. The .zip extension would be associated with the short Windows program. Any file appearing to meet the spec (filename ends in .fc.zip case insensitive; contents are one file called "prg" or two files called "prg" and "chr") would cause the program to launch the emulator; otherwise, it'd launch 7-Zip.
Quote:
However, I don't think this format will catch on solely for the fact that it doesn't support homebrew and unlicensed
Homebrew can continue to use iNES or NES 2.0 because homebrew developers appear to care more about distributing correct headers than commercial ROM collectors do. Unlicensed I addressed above.
Wkter wrote:
(Yes, yes, no bad headers.. I know.. But that hasn't been a problem for me.)
Me either. I am admittedly clueless when it comes to emulators, but I've never had a problem getting roms to work properly. Is that the reason for this headerless format? (Sorry if that was answered earlier in this thread, but I don't have an hour to burn reading it
)
FitzRoy wrote:
Are you kidding me with this?
No, I'm not kidding... It's my honest opinion. I raised it in a hope that you will see my position on this and support ZapFC files with their own headers. I don't see any good reason to have multiple formats for NES emulators, so, in my opinion, we should instead focus on a universal format that makes everyone happy.
FitzRoy wrote:
What grade are you in?
There is no reason to resort to personal attacks because I raised concern about your suggested format. It IS an issue that it doesn't support these.
tepples wrote:
That's exactly why I proposed the short Windows program. The emulator wouldn't see all zip files. The .zip extension would be associated with the short Windows program. Any file appearing to meet the spec (filename ends in .fc.zip case insensitive; contents are one file called "prg" or two files called "prg" and "chr") would cause the program to launch the emulator; otherwise, it'd launch 7-Zip.
You mean to create a program to associate with the .zip format that will open the correct program? I see your reasoning there, but we'd still have a problem with what program to open for regular zip files. Some people would like to have WinZip, some WinRar, some 7-Zip, and some the standard Windows unzipper. While we certainty can check what the associated program is before overwriting it, some user might want to change this later on.
The safest bet would be to have a (hopefully) unique extension.
tepples wrote:
because homebrew developers appear to care more about distributing correct headers than commercial ROM collectors do.
Isn't the whole point of ZapFC to guarantee correct headers?
tepples wrote:
Case in point: StarTropics and StarTropics 2. These have MMC6, whose behavior differs from other mapper 4 ROMs. WRAM Present = "None" might mislead, as MMC6 certainly has battery-backed WRAM albeit internal to the mapper IC. So I guess an emulator would need to fall back on field 10 (mapper IC) has to be used like a submapper field.
Okay, so that doesn't sound like a deficiency in the iNES assignments, but rather a field deficiency. You're saying that the MMC6 contained RAM that was optionally utilized. That will probably require a new "Mapper WRAM" field is all.
Quote:
How would these soft-patches be distributed? As separate patches to prg and chr? As zipfiles containing a patch to prg and chr? As patches to a virtual file containing two or three concatenated files (a 16-byte file, "prg", and "chr" if present, in that order)? And how would soft-patches that change the board type by adding WRAM, a battery, etc. work?
I would say a patch file for each rom in its own container with an extension of ".xxx.zip" with xxx being whatever the format is. Byuu kinda gave up on UPS after he realized it had some issues with expanding roms, but it's still better than IPS, mainly because it prevents you from applying to a file it wasn't designed to be applied to. That feature is less useful for NES where header ambiguity was never an issue, but it's still useful.
Why even worry about this right now? One step at a time.
Quote:
So the main file contains only games with the Official Nintendo Seal. I understand the reasoning behind this. But if we limit it to only unlicensed games commercially released in North America and Europe prior to the release of the Nintendo 64 console (Sunday Funday yes, Battle Kid no), I don't see how an additional ZapFC_North_American_Unlicensed_(20110216).fbsv file containing Tengen, Camerica, Color Dreams, etc. would be a nightmare to maintain. There's a finite set of these companies because there's a finite set of their CIC defeat circuits.
So your argument is basically "a selected portion of infinity wouldn't be a nightmare to add." I'm sorry, I see exclusion rules based on release timeframes and stuff to be way too wishy washy. Distribute them in iNES until someone makes a customizable mapper file extension for ZapFC containers.
Wkter wrote:
tepples wrote:
because homebrew developers appear to care more about distributing correct headers than commercial ROM collectors do.
Isn't the whole point of ZapFC to guarantee correct headers?
The point of ZapFC is to do so for finite ROM sets, such as all NES games carrying the Official Nintendo Seal. The burden of guaranteeing proper headers for unbounded sets of homemade games is on the developers of such homemade games, and so far they've been doing a far better job than the administrators of some ROMz sitez. To make things perfectly clear:
NES 2.0 isn't going away.
The perceived attack on homebrew here is not unlike the angsting on Wikipedia over
its "notability" standards by people who have a
conflict of interest.
Wkter wrote:
No, I'm not kidding... It's my honest opinion. I raised it in a hope that you will see my position on this and support ZapFC files with their own headers. I don't see any good reason to have multiple formats for NES emulators, so, in my opinion, we should instead focus on a universal format that makes everyone happy.
You want an all-in-one format that works as well as two? I want a unicorn. Let me break this down for you based on merit, not popularity. There are basically 3 ways to deliver mapping:
Emulator-side mapping cons:
-cannot support unlicensed games because it is not humanly feasible to add and maintain an infinite number of entries.
Rom-side mapping (separate, customizable file) cons:
-cannot guarantee mapping integrity of a downloaded game because there are hundreds of different suppliers who may have out of date or messed up files.
-requires users to update files or download new versions of them to effectuate mapping updates and revisions.
Rom-side mapping (header) cons:
-cannot guarantee mapping integrity of a downloaded game because there are hundreds of different suppliers who may have out of date or messed up files.
-requires users to update files or download new versions of them to effectuate mapping updates and revisions.
-does not allow for custom mappers to be created.
-must duplicate data to account for instances where the same data shipped on operationally different boards.
The optimal method is not choosing just one, but emulator-side for licensed games and rom-side (separate, customizable file) for unlicensed. You seeing this now? And all of this is independent of how the rom data is stored. Here's that comparison:
Header-defined container cons:
-requires specialty tools per system to obtain or compare simple information like prg and chr sizes and checksums. dumpers and patchers will hate you.
-requires emulator-side mapping to add prg and chr size fields to each database entry.
ZIP/7z container cons:
-slightly more difficult to support if an author didn't already?
Quote:
There is no reason to resort to personal attacks because I raised concern about your suggested format. It IS an issue that it doesn't support these.
Sure there is. I'm tired of spelling out A-B-C a hundred different ways. Get your mind on track.
Quote:
Isn't the whole point of ZapFC to guarantee correct headers?
For licensed games, yes. Because it's the only type of game that can. ZapFC also changes the way roms are stored, which coincidentally allows for an extensible mapping format someday for unlicensed games.
FitzRoy wrote:
You want an all-in-one format that works as well as two? I want a unicorn.
FitzRoy wrote:
The optimal method is not choosing just one, but emulator-side for licensed games and rom-side (separate, customizable file) for unlicensed.
Hypocrite.
FitzRoy wrote:
You seeing this now?
This is what I've been saying all along! Use both! Ship all ROMs with a header, and in the header make a field "Prioritate databse", if this is set, ignore the header.
FitzRoy wrote:
Sure there is. I'm tired of spelling out A-B-C a hundred different ways. Get your mind on track.
Then I'm not interested in discussing this with you any further. You will not get support with such a bad attitude.
*Wonders if this will reach 20 pages*
tepples wrote:
Official Nintendo Seal
tepples wrote:
Official Nintendo Seal
tepples wrote:
Official Nintendo Seal
Wkter wrote:
Hypocrite.
Oh, what a smart one you are. That's two different mapping methods for two types of games. You want one method for all games. We already have that with iNES and it doesn't come close to being the strategy with the least overall costs. And your method of storing files is also worse.
Quote:
This is what I've been saying all along! Use both! Ship all ROMs with a header, and in the header make a field "Prioritate databse", if this is set, ignore the header.
Goofy and unnecessary to keep storing files in a unique blob while leaving unused headers intact like some decrepit parasite. You're trying to take a dirty shortcut to get a benefit from the database method while still leaving dumpers and patchers pissed off. Fuck that.
byuu wrote:
tepples wrote:
Official Nintendo Seal
tepples wrote:
Official Nintendo Seal
tepples wrote:
Official Nintendo Seal
byuu, you made my day.
I'm generally opposed to censoring people, unless they are being hateful for the sake of being hateful, or spamming. And I oppose the clubbing of seals. But this is just too funny.
FitzRoy wrote:
That's two different mapping methods for two types of games
And the problem being?
Quote:
You want one method for all games. We already have that with iNES and it doesn't come close to being the strategy with the least overall costs.
I have not suggested one MAPPING method for all games, I mearly suggested one CONTAINER. You're putting words in my mouth. Also, what costs? iNES is free. iNES uses less memory than a database can ever achieve. iNES is 16 bytes. No, I'm not in favor for keeping iNES before anything else, but what you're saying here is just ridiculous.
Quote:
And your method of storing files is also worse.
I have not suggested any way of storing the files, only you have. You're again putting words in my mouth.
Quote:
Goofy and unnecessary to keep storing files in a unique blob while
Again, your way of storing the files, not mine.
Quote:
leaving unused headers intact like some decrepit parasite. You're trying to take a dirty shortcut to get a benefit from the database method while still leaving dumpers and patchers pissed off. Fuck that.
What benefits can a database offer me when I'm suggesting to add support for an optional header file
so that not all ROMs are depended on a database?
And how is this supposed to piss dumpers off (Assuming they for whatever reason would want to dump a game that
cannot be played yet)? Maybe something along these lines: "Oh, gee... I'm a dumper and I can add an optional header file so that others can use this ROM without a database... I can't stand this! I'm quitting!"?... Seriously?...
op·tion·al
[op-shuh-nl]
–adjective
1. left to one's choice; not required or mandatory: Formal dress is optional.
2. leaving something to choice.
You, sir, have lost all my support in this file format because you have repeatedly demonstrated your complete lack on interest in the publics opinion and suggestions. I'll say it straight out: this format is useless in its current state, and because nothing we say or do will get through to you (We're on 10 pages already and
no progress has been made!), the format doomed as well. I'm sorry I have to put it this bluntly, but you insisted on bad behavior.
FitzRoy wrote:
Emulator-side mapping cons:
-cannot support unlicensed games because it is not humanly feasible to add and maintain an infinite number of entries.
-cannot support systems without a well-defined concept of "licensed", such as Atari 2600.
Quote:
Header-defined container cons:
-requires specialty tools per system to obtain or compare simple information like prg and chr sizes and checksums. dumpers and patchers will hate you.
By "per system" did you mean per host system (e.g. Windows vs. Linux vs. Mac OS X)? If so, write CLI tools in Java or Python or C+GTK or some other reasonably portable abstraction. Or by "per system" did you mean per emulated system (e.g. NES vs. Atari 2600 vs. ColecoVision vs. SMS)? If so, the file naming and checksumming procedures will vary anyway for those. For example, only NES and Neo Geo would possibly have separate PRG and CHR ROM.
Quote:
ZIP/7z container cons:
-slightly more difficult to support if an author didn't already?
What library for ZIP container do you recommend for tools written in C to manipulate fc.zip? Would you recommend zlib+minizip or something else? Also, what do you recommend for unlicensed games in the ZIP/7z container without storing a frankenfile inside?
I hereby propose an extension to fc.zip to allow distributing unlicensed games in a standard container. It would act as one step (at a time) to transition unlicensed games from ad-hoc descriptions in frankenfiles to richer mapper descriptions in standard containers. Allow a third file with name "fbsv", containing the same data as one line of a board database. It is used as follows:
- If the checksum of "prg" and "chr" matches the checksum in the emulator's board database, ignore the "fbsv" file. Emulators' data overrides possibly outdated data in ROMs.
- If the checksum in the "fbsv" file does not match that of the "prg" and "chr" files or is otherwise unparseable, fail and display a diagnostic.
- Otherwise, treat the "fbsv" file as a one-line extension to the board database.
Wkter wrote:
iNES is free.
specialty tools
Tools to build iNES are not distributed with operating systems.
I get the pun, byuu, but why is something I am trying to preserve being used to smash me? As far as I know, Tepples and I both agree that licensure ensures some level of quality and warranted greater historical significance. If you were truly confident in your game, you could license and find a publisher no problem. This has been done for how long now?
tepples wrote:
Official Nintendo Seal
tepples wrote:
Official Nintendo Seal
tepples wrote:
Official Nintendo Seal
Funny part is Famicom games don't have the Official Nintendo Seal, even ones that are completely Nintendo programmed/fabricated!
FitzRoy wrote:
I get the pun, byuu, but why is something I am trying to preserve being used to smash me?
The preservation has already been done at places like BootGod's amazing database which is designed to include ALL games and far more information. Reformatting what he has already doesn't preserve anything.
FitzRoy wrote:
If you were truly confident in your game, you could license and find a publisher no problem. This has been done for how long now?
Battle Kid is licensed and published by RetroUSB. Somehow I doubt that is enough...
tepples wrote:
specialty tools
Tools to build iNES are not distributed with operating systems.
Homebrew would need specialty tools no matter what (Complier, hex editor and CHR editor if you're not comfortable with hex editors), hacks would also need specialty tools, and to make patches you need specialty tools, those are the only ones I'm suggesting a header file for (excluding patches, of course, which can use the database for official games).
Besides, what's the problem with specialty tools (Def: Tools not distributed with operating systems, i.e. WinRAR)?
FitzRoy wrote:
but why is something I am trying to preserve being used to smash me?
I don't mind databases which can be used with the official games for preservation. I have no objections to that, infact I, as I posted on the very first page, think that is a great idea. However, as I've been trying to tell you in so many posts now, having two formats side by side can cause problems. That's why I suggest having the option to include headers on ROMs that need it (Not official games), so we can have one container for everything.
FitzRoy wrote:
If you were truly confident in your game, you could license and find a publisher no problem. This has been done for how long now?
You won't get a licence from Nintendo.
tepples wrote:
-cannot support systems without a well-defined concept of "licensed", such as Atari 2600.
That's not an additional cost, that falls within the same one I listed. If that means a system can't use a database because it doesn't have a manufacturer-defined library, then all its games end up using the rom-side method.
Quote:
By "per system" did you mean...
Per console system. The more that there are, the more specialty tools you have to use. It's just dumb luck that weren't more systems with multi-rom games. Conceptually, it's a minus.
Quote:
What library for ZIP container do you recommend for tools written in C to manipulate fc.zip? Would you recommend zlib+minizip or something else? Also, what do you recommend for unlicensed games in the ZIP/7z container without storing a frankenfile inside?
I'm assuming you would just pick one? Why does this matter? And no, you can't have combined or interleaved data in a ZapFC container per the spec. And the files have to be named "prg" and "chr" so that the emulator knows what they are.
Quote:
I hereby propose an extension to fc.zip to allow distributing unlicensed games in a standard container. It would act as one step (at a time) to transition unlicensed games from ad-hoc descriptions in frankenfiles to richer mapper descriptions in standard containers. Allow a third file with name "fbsv", containing the same data as one line of a board database. It is used as follows:
- If the checksum of "prg" and "chr" matches the checksum in the emulator's board database, ignore the "fbsv" file. Emulators' data overrides possibly outdated data in ROMs.
- If the checksum in the "fbsv" file does not match that of the "prg" and "chr" files or is otherwise unparseable, fail and display a diagnostic.
- Otherwise, treat the "fbsv" file as a one-line extension to the board database.
I would recommend the third file be called "board" and you go all the way with it so as to allow for custom mappers. Just making it contain a database entry doesn't allow for that. Unfortunately, this is beyond my skill level and I have no idea if we could ever get people to agree on the formatting of such a file. If nobody sounds interested or they can't come to an agreement, then I like just having a database entry in there.
Wkter wrote:
Homebrew would need specialty tools no matter what (Complier, hex editor and CHR editor if you're not comfortable with hex editors), hacks would also need specialty tools, and to make patches you need specialty tools, those are the only ones I'm suggesting a header file for (excluding patches, of course, which can use the database for official games).
So... let's eliminate our need for yet another one?
Quote:
Besides, what's the problem with specialty tools (Def: Tools not distributed with operating systems, i.e. WinRAR)?
Who doesn't have such a tool already for general use? Even my grandmother has WinRAR to open downloaded archives.
Quote:
However, as I've been trying to tell you in so many posts now, having two formats side by side can cause problems. That's why I suggest having the option to include headers on ROMs that need it (Not official games), so we can have one container for everything.
A "header" is metadata
appended to the data. If you want to store your unlicensed data the same way ZapFC stores licensed data, then you need to create a board file format, and you need to be sure you don't screw up like iNES did. Otherwise, use iNES for unlicensed. I ain't taking your iNES anyway.
Quote:
You won't get a licence from Nintendo.
Not anymore, you can't. Because the system is dead.
Fitzroy, you've yet to post what your database schema is. I'm curious what pieces of data you would attach to each rom set checksum (assuming that the checksum of rom sets are your primary key).
Nintendo isn't the authority on "Seal of Quality" any more, it's only a marketing scheme to protect the system back in the day. Now it's internet ratings and demos and such I'd think.
And instead of a header, lets call it a "manifest" that will always over ride the emulator database. Liker how Chrome apps/extensions have a manifest that says what permissions it needs and such. That'd maybe be a good idea to implement a "header" like file.
clueless wrote:
Fitzroy, you've yet to post what your database schema is. I'm curious what pieces of data you would attach to each rom set checksum (assuming that the checksum of rom sets are your primary key).
There's a link in the first post to a completed database and an explanation of its formatting. Tepples never saw it either, he just skimmed like you. It's right there.
I don't quite get it yet, how is this proposal better than looking up the checksum in an xml dump from bootgod, and ignoring the ines header if a match is found?
Perhaps the goal is to remove all metadata from the rom file in order to create a more 'pure' format?
I have the follow little points to make.
- ZapFC seems to be two halves. I'm going to call these two halves by these names: FCZIP for the archive format, and FBSV for the database.
- I do not see what problem FCZIP solves. A valid dump in a zip is no different to the end user than a valid dump with a 'scary frankenheader' .
- Furthermore, because FCZIP uses ZIP, and cannot practically prohibit compression, it precludes solid packing and an FCZIP set can't take advantage of e.g. the European, US, and Japanese releases being very similar but not identical.
- FCZIP doesn't solve the IPS problem, because a new patching tool would be needed. At that point, it is just as easy to modify IPS to support "move this block of data from this offset to this offset" and then this superiority over iNES is moot. Ideally add a checksum to make sure the (patch was applied to/the resultant image was) a clean image.
- What's solved by putting your DB in a FBSV file? There's no information there that's not in e.g. Nestopia's database. Complaining about XML is invalid. even though i hate XML too
- Using SHA256 only makes sense if you're worried about malicious changes to the ROM images. I assume that most issues with "bad dumps" are accidental, and thus even lowly CRC32 is sufficient.
- Similarly, if you're going to require that emulators link against something like lib7z for FCZIP support, there's no good reason for not requiring them to link against something like libsqlite or libxerxes.
- Contrasting (this attempt to partially deprecate iNES with an industry standard (ZIP)) with (making your own halfassed database format) is at best inconsistent.
- iNES2 hasn't caught on because it's unnecessary, not because it's inadequate.
- GoodNES supports fixing headers, but the last release predated iNES2. Getting iNES2 widely deployed is probably as easy as talking to Cowering.
lidnariq wrote:
GoodNES supports fixing headers, but the last release predated iNES2. Getting iNES2 widely deployed is probably as easy as talking to Cowering.
I've been mostly keeping out of the thread, but I will interject here that I have been talking to Cowering about ines2.0 and getting it implemented in the GoodNES set. We're working on getting this done; I am waiting now for the new current set so I can go through and fix the headers that need it.
So, in short, GoodNES WILL support fixing headers to the 2.0 spec if you so desire it to fix them.
This is the main reason I have not released a complete spec- I am waiting on this update so I can make sure I have all the bases covered.
lidnariq wrote:
A valid dump in a zip is no different to the end user than a valid dump with a 'scary frankenheader'.
This was addressed on the
previous page. There is a difference in the labor required to validate dumps so that valid dumps exist. There is a difference in mapping certainty for licensed games given that boards are the only piece of hardware not bundled with the emulator.
Quote:
Furthermore, because FCZIP uses ZIP, and cannot practically prohibit compression, it precludes solid packing and an FCZIP set can't take advantage of e.g. the European, US, and Japanese releases being very similar but not identical.
I have no idea what you're saying. The compression level has no affect on the recorded checksum. Yes, many games shared the same CHR or PRG, what about it?
Quote:
FCZIP doesn't solve the IPS problem, because a new patching tool would be needed. At that point, it is just as easy to modify IPS to support "move this block of data from this offset to this offset" and then this superiority over iNES is moot.
What is "the IPS problem"? Whether we create a suitable replacement for IPS or not, the format will require new patch code. And realize that the licensed iNES file you're patching could have a bad, missing, or outdated mapper, because the people supplying them are dumb, robotic kiddie collectors who don't even realize they're offering people [o] overdumps and [b] bad dumps. That's your vote of confidence for receiving good mappers? Then go back to the blue pill.
Quote:
Ideally add a checksum to make sure the (patch was applied to/the resultant image was) a clean image.
IPS doesn't support that, so if the patch author forgot to tell you what rom region, version, and checksum to apply to, happy guessing! 1 out of 3 odds ain't so bad. But glaring oversights don't matter as long as it has reached Britney Spears groupie status, right?
Quote:
What's solved by putting your DB in a FBSV file? There's no information there that's not in e.g. Nestopia's database.
I hope you mean bootgod's, because Nestopia's stock database is barren. I use FBSV because it achieves the same thing without being 3 times the filesize and having 10 times the number of the lines.
Quote:
Using SHA256 only makes sense if you're worried about malicious changes to the ROM images.
Then I guess it makes sense then, because I am worried about people spending thousands of dollars and manhours preserving data with checksums a 13 year old can spoof? Why WOULDN'T you use an unspoofable hash? Allergic to strong hashes? I can't believe I'm taking shit for my hash choice now.
Quote:
I assume that most issues with "bad dumps" are accidental, and thus even lowly CRC32 is sufficient.
You wouldn't assume jack if you were a dumper.
Quote:
Similarly, if you're going to require that emulators link against something like lib7z for FCZIP support, there's no good reason for not requiring them to link against something like libsqlite or libxerxes.
Um, it's important to choose a container. It's not important to support alternative database formatting that is inferior by multitudes.
Quote:
Contrasting (this attempt to partially deprecate iNES with an industry standard (ZIP)) with (making your own halfassed database format) is at best inconsistent.
Apples to oranges comparison, we don't need "popular" formatting for the database because it's not meant to be accessed by anyone but the maintainer. You're the kind of XML whore I wrote that paragraph for. XML is for things with tons of subattributes. Simple CSV should be used whenever possible, and FBSV is a form of CSV that avoids escaping rules.
Quote:
iNES2 hasn't caught on because it's unnecessary, not because it's inadequate.
iNES 2.0 hasn't caught on because groupies don't care about compatibility assurance above 95%. Nothing will "catch on" at this point, so you can stop concerning yourself with popularity and focus on getting some amount of people preserving a perfect library.
Quote:
GoodNES supports fixing headers, but the last release predated iNES2. Getting iNES2 widely deployed is probably as easy as talking to Cowering.[/list]
Guess you'll have to ask him then! That's 7 years worth of hacks and overdumps to add, plus changing to a format that won't work in most emulators, and some of the most popular emulators are dead. Like I keep saying, we could have had the groupies unwittingly preserving the library, but now it'll just be a couple hundred. The major distribution will always be iNES 1.0 and imperfect. Nice shootin, tex.
FitzRoy wrote:
iNES 2.0 hasn't caught on because groupies don't care about compatibility assurance above 95%.
What I would give if the average emulation user cared about even 80% compatibility.
If the 10-15 games the casual emulation user plays appears to be working, even with inaccurate sound and the occasional minor graphical glitches, then said emulator and ROM format is 100% perfect and accurate. Those 10-15 titles are always the most popular ones possible, so of course everything supports them: SMB1-3; Zelda 1,2; Contra 1,2; Megaman 1-6; Metroid.
Trying to do anything for the ungrateful masses is a lost cause and a waste of time.
byuu wrote:
Trying to do anything for the ungrateful masses is a lost cause and a waste of time.
Unfortunately, the more of them we can get to unwittingly preserve the system, the better. I always believed that you could have done better than 5th page on google search results. Each page you drop back is a huge decline. I am less optimistic about NES because there are so many more emulators, many of which are dead.
FitzRoy wrote:
<lots of sound and fury signifying nothing>
In short:
1- You have a solution in need of a problem. It's a repacked version of an iNES1 image plus a repacked version of BootGod's DB. You have done negligible work and all of that has just been in separating wheat from chaff.
2- You don't persuade people by insulting them. Even if they've already insulted you.
3- You're trying to persuade people whose primary interest is explicitly not supported by your solution. Try somewhere else.
iNES 2.0 was designed mainly to fill in the gaps for games that need CRC checks to determine specific mapper traits (like, MMC1 games that need 16K or 32K of WRAM).
I designed it so that my FPGA NES BIOS does not need to keep such databases in its limited internal ROM. The results have been pretty good for this. Originally I did indeed have a database in it.
It has not "caught on" yet because there hasn't been an official release yet, but slowly I am working on getting this done. I need to go through all the ROMs and apply 2.0 headers to the games that need it. I have done this but only for the older stuff, and all the Vs. system dumps I have made. I suspect that around 50-80 ROMs will be affected and need the new headers.
With the 2.0 headers, all the Vs. system details are right in it making playing them and emulating them easy (No more PPU/etc selection required).
I am not sure if 2.0 will catch on, but for me it's kind of a moot point. It works here on my stuff which is the important bit. If others would like to use it that would be great.
The main driving force behind it was to smooth out the rough edges on iNES and retain several (IMO) important advantages which were backwards compatibility (though some emulators will "clean" the headers) and more important retain the ability to process stuff on a real live NES.
If your new format cannot be read or used easily by a real NES then IMO it's a failure from the start. I would like to see an NES decompress a zip file quickly or at all without extra help like a PIC or other CPU and gallons of RAM. Even with the RAM I suspect decompression will be quite slow.
It's not as if the complete set of ROMs won't fit on a modern CF/SD card many times over.
Also, with this proposed zipped format, what's going to stop people from abusing it? i.e. adding extra info to the zips such as text files proclaiming where it was downloaded from?
What about larger "unlicensed", shall we say, games? (Action 52 comes to mind here). They have multiple PRG/CHR ROMs. Will those be stored separate or unified? If separate basically you will need many "Special case" loading modules and stuff to figure out what to do with three .PRG files.
iNES has the advantage that the CHR/PRG are unified, making loading it and manipulating it relatively simple. On the FPGA NES I just blindly load the entire chunk of PRG and CHR in first and then worry about banking it later. This vastly simplifies the mappers and the loading code when everything has a single, unified structure.
And as a final thought, relying on some format that can be done via human intervention is going to be a load of fail. What happens when people manually make these zip files, and use different conventions like all caps, all lower case, mixed case, unicode filenames, junk files (RIPPED BY DISKDUDE!!!.TXT), and other assorted trash in there? Reading all that stuff on a PC is a headache, but doing it on some embedded system (NES, a PIC or other micro) is incredibly difficult and tedious, if it's even possible.
So I guess in summation: the KISS principle rules all at the end of the day, and "pony"* formats never seem to take off.
(*pony as in, "I want this and that and a pony!")
lidnariq wrote:
1- You have a solution in need of a problem.
Yeah, is that why iNES 2.0 was made? You'd have expected 1.0 to be perfect. It wasn't. You'd have expected people who mindlessly distribute bad roms to update to iNES 2.0. They didn't. And the glorious reasoning behind all of this? No one fucking knows. MAME avoided it, you didn't.
Quote:
It's a repacked version of an iNES1 image plus a repacked version of BootGod's DB.
You say that like you could have been way off. There were only two ways we could have stored the files and three ways we could have delivered mappers. And no, my method is not broken with SOROM like iNES1. And even if it was, I could effectuate a fix via an emulator update instead of asking thousands of users to update all roms with yet another specialty tool. It's a specialty tool bakesale! Two for the price of one. Specialty tool for dumpers, specialty tool for users!
Quote:
You have done negligible work and all of that has just been in separating wheat from chaff.
I never claimed credit for bootgod's dumps, but I do appreciate you admitting that his XML could be simplified for some space savings and a more future-proof hash algo. You're welcome.
Quote:
You don't persuade people by insulting them. Even if they've already insulted you.
You think you're important enough to persuade? I'm just debunking your fud.
Quote:
You're trying to persuade people whose primary interest is explicitly not supported by your solution. Try somewhere else.
Too bad, I thought the custom mapper file was kind of neat. Guess I'll move on to other systems and it's back to threads like this:
http://nesdev.com/bbs/viewtopi ... sc&start=0
FitzRoy wrote:
Guess I'll move on to other systems and it's back to threads like this:
http://nesdev.com/bbs/viewtopi ... sc&start=0Funny thing is that this thread is all about homebrew, which your solution doesn't cover. Regardless of whether your format is accepted or not, that will mean nothing for Neil's problem of not getting emulators to recognize more than 8KB of SRAM for his homebrew program.
byuu wrote:
Imagine finding a bunch of ARC or LHA files today.
Funny thing, isn't LHA still used as widely in Japan as ZIP is used everywhere else?
blargg's File_Extractor library seems like an acceptable solution for reading these archives, except that no Linux or BSD distributions seem to want to supply it. Maybe it has something to do with the fact that it bundles libraries for all supported formats, rather than bothering to use system libraries for anything. Of course, zlib is the only system library it could reasonably use, unless there's some LZMA/2 library that isn't guaranteed to break API compatibility with every single release.
kevtris wrote:
If your new format cannot be read or used easily by a real NES then IMO it's a failure from the start.
iNES couldn't be processed on an NES until PowerPak, and it wasn't a failure.
Quote:
I would like to see an NES decompress a zip file quickly or at all without extra help like a PIC or other CPU and gallons of RAM.
PowerPak has 32 KiB of WRAM. The history in DEFLATE, the primary codec used in zip files, is also 32 KiB.
Quote:
Also, with this proposed zipped format, what's going to stop people from abusing it? i.e. adding extra info to the zips such as text files proclaiming where it was downloaded from?
The spec, linked in the first post of the first page, addresses this. Any files with names other than
prg and
chr cause ZapFC loading to fail, and the emulator offers to launch the ROM in 7-Zip or Compressed Folders instead.
Quote:
What about larger "unlicensed", shall we say, games? (Action 52 comes to mind here). They have multiple PRG/CHR ROMs. Will those be stored separate or unified?
FitzRoy seemed to agree with a proposed extension to the spec (on page 10) that unlicensed games have three files in the zip:
prg,
chr, and
board. The data in Action 52's three PRG ROMs would be concatenated, and the
board file would describe how in some manner yet to be determined.
Quote:
If separate basically you will need many "Special case" loading modules and stuff to figure out what to do with three .PRG files.
There's "Special case" loading modules and stuff to figure out in iNES. Does your ROM have a PRG ROM preload segment before PRG? (Very few do, as I understand it mostly to aid mapper hacks to copier mappers with a
poison mapper symbol on your
mapper matrix, but it's still a possibility in iNES.)
Quote:
What happens when people manually make these zip files, and use different conventions like all caps, all lower case, mixed case, unicode filenames, junk files (RIPPED BY DISKDUDE!!!.TXT), and other assorted trash in there?
Loading fails.
FitzRoy wrote:
You'd have expected people who mindlessly distribute bad roms to update to iNES 2.0. They didn't.
Cowering and kevtris are working on that.
Quote:
And even if it was, I could effectuate a fix via an emulator update instead of asking thousands of users to update all roms with yet another specialty tool. It's a specialty tool bakesale! Two for the price of one. Specialty tool for dumpers, specialty tool for users!
Emulators are a specialty tool. In fact, they're so much of a specialty tool that Apple has banned them from its App Store unless they include ROMs and don't allow the end user to add ROMs.
tokumaru wrote:
Funny thing is that this thread is all about homebrew, which your solution doesn't cover.
See above, where FitzRoy mentions a
board file.
kode54 wrote:
blargg's File_Extractor library
It's not on blargg's site, and Google couldn't find an official page for it. But if it includes .rar, that might be why it's not in Linux or FreeBSD: the unrar code is under a non-free license.
tepples wrote:
But if it includes .rar, that might be why it's not in Linux or FreeBSD: the unrar code is under a non-free license.
That's strange, I wasn't asked to pay anything for my copy of it.
It is bundled with older versions of bsnes, in the snesreader library source. It does bundle the unrar library source, but that can be disabled and excluded.
I don't see what the big deal is about that anyway, other than the whole idea of "you may not use this decompression code to reverse engineer and reimplement the compression algorithms" meaning you're not "totally free" to do "whatever you want" with the code. Oh well, I guess other people can spend all the time they want worrying about how free they are to use whichever software to their liking, and I'll continue to use all the software I want to my liking, regardless of whatever clickthrough EULA I chose to ignore, or if the developer was enough of a jerk, immediately scrolled to the bottom of to enable the "Accept" button. Next thing you know, they'll make a license agreement page in installers which require you to page through the license at a "normal" reading pace to enable the "Accept" button.
kode54 wrote:
Next thing you know, they'll make a license agreement page in installers which require you to page through the license at a "normal" reading pace to enable the "Accept" button.
Nah, they'll make you retype the license, word for word, and they will disable copy + paste. Evil bastards.
I don't understand why people use rar anyway. Does it have some feature set that makes it better than zip?
byuu wrote:
tepples wrote:
But if it includes .rar, that might be why it's not in Linux or FreeBSD: the unrar code is under a non-free license.
That's strange, I wasn't asked to pay anything for my copy of it.
It's not whether you paid anything for it, it's whether you're Free to do whatever you want with the source code, including reinventing the commercial-only Rar archive compression utility, which contains proprietary algorithms that the author doesn't want anyone to know about. (Because they're probably so dead simple that keeping them away from the public is the only incentive he has to get people to pay for his archiver. Not that anyone with a brain does that. There are alternatives. Such as renaming the nag dialog resource so it can't even spawn it. Or more legal alternatives, such as 7-zip.)
kode54 wrote:
It's not whether you paid anything for it, it's whether you're Free to do whatever you want with the source code, including reinventing the commercial-only Rar archive compression utility
Then tepples should have clarified what he meant.
Free as in freedom is reserved for animate objects*, whereas free refers to cost for inanimate objects. Examples:
Makes sense:
The Libyans were finally freed from their captors.
The beer was totally free, I didn't have to pay anything!
Makes no sense:
The Canadians are free, take as many as you like.
After years of torture, the steak knife was finally set free.
* our Supreme Court has not
yet declared software to also be human beings with first amendment rights.
tepples wrote:
kevtris wrote:
If your new format cannot be read or used easily by a real NES then IMO it's a failure from the start.
iNES couldn't be processed on an NES until PowerPak, and it wasn't a failure.
I had Squeedo loading iNES files back in 2004, was the first thing I implemented, years before the PowerPak existed. Just using hyperterminal, no special software needed - just a controller port RS232 adapter. There is no possible way that could have worked if the files were zipped. It would have to load the zip file into flash, uncompress to RAM, and write the unpacked ROM to flashrom again - not really even possible (except with ROMs much smaller than memory - assuming that unzipping on the NES is even possible).
Seems like a waste of effort to make an NES program format that the NES itself can't load without taking heroic efforts.. I like other aspects of the proposal though, but it seems like iNES 2.0 addresses most of the stuff we'd need (none of these formats have everything I'd
want - like a ROM version field and all that crap that ends up getting tacked on the filename).
Quote:
*ENTIRE THREAD*
AHhhh, it's like high school all over again. I love you guys.
tepples wrote:
kevtris wrote:
If your new format cannot be read or used easily by a real NES then IMO it's a failure from the start.
iNES couldn't be processed on an NES until PowerPak, and it wasn't a failure.
Quote:
I would like to see an NES decompress a zip file quickly or at all without extra help like a PIC or other CPU and gallons of RAM.
PowerPak has 32 KiB of WRAM. The history in DEFLATE, the primary codec used in zip files, is also 32 KiB.
I meant in the first instance that ines could be easily used (read: loaded, manipulated, processed) by a stock NES, while zip files cannot since they need an elaborate uncompress program. Barefoot ines files do not- you just load them into memory and go from there.
If you support zip, you must support all the baggage that goes along with it, and essentially reproduce the entire pkunzip.exe in 6502 asm. Sure you can SAY that only compression format XYZ can be used, but that's just a band-aid on the problem. I envision people whining that their games only work in some places and not others due to differences in the way different things zip stuff.
This also doesn't mention the speed (lack thereof) when a 6502 is decompressing anything, even if there is enough RAM for it to do its thing. I really would be interested in seeing someone write a complete decompresser in 6502 asm that has a speed comparable to just directly loading the uncompressed version.
thefox wrote:
http://www.vidarholen.net/~vidar/rage/12986526001782.html
Nice. thefox, very funny. This thread just keeps on giving.
thefox wrote:
http://www.vidarholen.net/~vidar/rage/12986526001782.html
Awesome.
. .
EDIT: off-topic, but I've been looking for that one image, where it reads something like 'Unofficial Nintendo Seal of Quality', only it has an actual animal seal in the picture of it or something. Anyone have that handy? :D
byuu wrote:
tepples wrote:
But if it includes .rar, that might be why it's not in Linux or FreeBSD: the unrar code is under a non-free license.
That's strange, I wasn't asked to pay anything for my copy of it.
See
free software on Wikipedia,
free software on GNU.org, and
free software on Debian.org.
I am aware that you did not pay with money, but instead, you paid with your freedom. The license of UnRAR is "non-free" per the GNU and Debian distributions because it prohibits users from deducing the format of a RAR archive. See
why UnRAR cannot be included in Fedora. The license term in question is as follows, with my emphasis:
Quote:
The UnRAR sources may be used in any software to handle RAR archives without limitations free of charge, but cannot be used to re-create the RAR compression algorithm, which is proprietary. Distribution of modified UnRAR sources in separate form or as a part of other software is permitted, provided that it is clearly stated in the documentation and source comments that the code may not be used to develop a RAR (WinRAR) compatible archiver.
byuu wrote:
Free as in freedom is reserved for animate objects
It's overloading, just as method names and operators can be overloaded in C++ and Python. "Free software" in the GNU sense means anybody who receives the code is free to use it, learn from it, improve it, and share it. And prior to the 1860s, the following was grammatical in U.S. English: "The negroes are free; take as many as you like."
The seal with the animal is shown in the PDF manuals of RetroZone products such as the
ReproPak MMC1.
Wkter wrote:
Obviously this SWF was made just for this thread. What's the best way to make SWFs like this without buying or pirating Adobe Flash?
tepples wrote:
Wkter wrote:
Obviously this SWF was made just for this thread. What's the best way to make SWFs like this without buying or pirating Adobe Flash?
! hehe
tepples wrote:
Wkter wrote:
Obviously this SWF was made just for this thread. What's the best way to make SWFs like this without buying or pirating Adobe Flash?
I don't know about any freeware programs that allows you to make SWFs.
SWiSH Max is a good alternative if you suck at flash, but other than that, I have no idea.
tepples wrote:
What's the best way to make SWFs like this without buying or pirating Adobe Flash?
FlashDevelop is great for creating SWF, but you have to do it all with code, since it doesn't have a stage you can visually edit. An animation like this would be pretty trivial to make, you'd just have to import the graphics (which you can make in GIMP, Inkscape, etc.) and write some simple code to move them around.
I love canvas.
Well I guess since this is coming to an end....frankenfile>frankenfolder.
This one definitely wasn't canvas, as it had the familiar
click-to-play icon that I get for all SWFs from an unfamiliar site.
Now
this is a frankenfile.
I have them disabled on chrome.....eff adobe.
tepples wrote:
This one definitely wasn't canvas.
I know I just like canvas because I don't have to pay hundreds of dollars to develop for it.
tepples wrote:
This one definitely wasn't canvas, as it had the familiar
click-to-play icon that I get for all SWFs from an unfamiliar site.
The one I posted was a 10 minute job in Adobe Flash.
tepples wrote:
Now
this is a frankenfile.
Doesn't it need to be a combination of 7 files in order to be a true frankenfile? So I hereby propose the following format:
File format: .frank
This file format will be made out of a mandatory 7 files. The files must at least have a 60% difference that the emulator must check to ensure byte diversity. It will interlace the data in chunks of 32 bytes to 256 bytes. It will not give you any indication on what file the chunk belongs to, and the files are randomly chosen for each chunk. The header will consist of only 5 bytes instead of 16 to save diskspace, and they will be located at the following offsets in the file:
0x46 (F) - Mapper number (see iFrank mapper list)
0x52 (R) - Horizontal or vertical mirroring (Yes, this needs a whole byte of its own)
0x41 (A) - Bytes of WRAM (Does not support more than 255 bytes of WRAM at the moment)
0x4e (N) - Number of 3KB PRG banks (3 is my favorite number)
0x4b (K) - Reserved for future use
Anyone up for it?
tepples wrote:
I am aware that you did not pay with money, but instead, you paid with your freedom.
What about my freedom to not have my work sold by commercial entities?
Quote:
The license of UnRAR is "non-free" per the GNU and Debian distributions because it prohibits users from deducing the format of a RAR archive.
Ah, I see.
http://dictionary.reference.com/browse/freeQuote:
36.
without cost, payment, or charge.
37.
allowed to deduce the format of the container a software library supports.
–adverb
38.
in a free manner; freely.
39.
Nautical . away from the wind, so that a sailing vessel need not be close-hauled: running free.
Quote:
cannot be used to re-create the RAR compression algorithm, which is proprietary.
Can you name a GPL-licensed software program that does not have a proprietor? If so, who is capable of enforcing said license?
Quote:
It's overloading, just as method names and operators can be overloaded in C++ and Python. "Free software" in the GNU sense means anybody who receives the code is free to use it, learn from it, improve it, and share it. And prior to the 1860s, the following was grammatical in U.S. English: "The negroes are free; take as many as you like."
So you can overload a common, four-letter English word to mean whatever you like; but you can't say "official games" instead of "games that carry the Official Nintendo Seal" to be more terse?
By the way, I consider free software to mean "not joined to or in contact with something else", and will use the term thusly from now on. Using zlib makes your software non-free. It'll be up to everyone else to figure out what I mean by that, or perhaps I will try and convince the world that my definition is the 'right' definition, hmm ...
Quote:
The seal with the animal is shown in the PDF manuals of RetroZone products such as the
ReproPak MMC1.
Excellent, thank you :D
Time did it no favors, I was hoping it was more professional looking ...
I've updated the Format Guide to clarify some things and Tepples' idea is now official. A board file is now a valid option that can be included in the container and it is essentially a single database entry.
QuickNES has pledged to add support. Maybe I can get someone to write a code patch for Nestopia now that c0d3h4x0r is maintaining it.
Tepples, regarding the Star Tropics II WRAM: what do you think should be done about that? Would it make more sense to create a "Mapper WRAM" field or just use the existing WRAM field?
byuu wrote:
What about my freedom to not have my work sold by commercial entities?
Copyright gives you the right to decide whether to give downstream users freedom or restriction. The author of UnRAR chose restriction.
Quote:
Quote:
The license of UnRAR is "non-free" per the GNU and [DFSG] because it prohibits users from deducing the format of a RAR archive.
Ah, I see.
http://dictionary.reference.com/browse/freeQuote:
36.
without cost, payment, or charge.
Here's what I saw when I visited that Dictionary.com link:
Quote:
2.
pertaining to or reserved for those who enjoy personal liberty: They were thankful to be living on free soil.
A work meeting the DFSG is free as in soil.
Quote:
Quote:
cannot be used to re-create the RAR compression algorithm, which is proprietary.
Can you name a GPL-licensed software program that does not have a proprietor? If so, who is capable of enforcing said license?
United States copyright law, 17 USC 102(b), makes a distinction between a work and the process (or algorithm) that it embodies. A computer program distributed under the GPL or another copyleft license has an owner, but the
algorithms it implements do not. UnRAR, on the other hand, has an owned algorithm.
Quote:
So you can overload a common, four-letter English word to mean whatever you like; but you can't say "official games" instead of "games that carry the Official Nintendo Seal" to be more terse?
I didn't invent "free" as in soil; Richard Stallman did. I didn't say "licensed games" because "licensed games" is also ambiguous; it could mean
video game adaptations of an existing story, setting, or sports league. Even "games licensed by Nintendo" is ambiguous because it includes four CD-i games and a few PC games including the game that spawned the WEEGEE meme.
Next post will be on topic:
@FitzRoy: Thank you for including my board idea. There's one little nitpick:
Quote:
Field 1 (File Name): This attribute is purely a reference for database maintainenance. It is not used for lookup.
Could it also be used for displaying the game's title?
Quote:
The board file is a file for unlicensed (undatabased) games that essentially contains one database entry. The only rule difference is that the first three values in each entry MUST be empty.
I guess that shoots down the idea of using the title entry to document the official title of an unlicensed game, much like an ID3 tag.
Quote:
It would be nice to someday create a successor to IPS that uses the checksum of the file it's meant to be patched too.
Would switching from IPS to bsdiff solve that problem?
FitzRoy wrote:
Tepples, regarding the Star Tropics II WRAM: what do you think should be done about that? Would it make more sense to create a "Mapper WRAM" field or just use the existing WRAM field?
Just use the existing WRAM field. Whether it's in the same package as the mapper or not matters little, unless perhaps the mapper has the capacity for two separate WRAMs like the MMC5 does: one on the mapper and one external. To handle these, I'd treat WRAM as "WRAM additional to that which comes with the mapper's base configuration". In this sense, MMC6 has additional WRAM because mapper 4's base configuration is PRG ROM + CHR ROM + no WRAM, but MMC5 does not (unless it's a Koei game or the like) because 1024 bytes of ExRAM at $5C00 are part of the base configuration.
Quote:
A board file is now a valid option that can be included in the container and it is essentially a single database entry.
Neat, so now homebrew devs can have even more flexibility than iNES. I would recommend a tool that can turn an iNES header into a board entry. Or in that case, just a general iNES->ZapFC convertor.
Quote:
Copyright gives you the right to decide whether to give downstream users freedom or restriction. The author of UnRAR chose restriction.
The author of UnRAR chose developer freedom. Why do you hate freedom?
Quote:
A work meeting the DFSG is free as in soil.
7-zip was thankful to be living on free soil.
Quote:
I didn't invent "free" as in soil; Richard Stallman did.
Why is Richard Stallman allowed to invent new meanings for words?
Quote:
I didn't say "licensed games" because "licensed games" is also ambiguous
Exactly!!
Quote:
Would switching from IPS to bsdiff solve that problem?
bsdiff is ridiculously complicated for a patching format. Writing even a patch applier, let alone a patch creator, is a nightmare. The format itself internally uses compression, which was already an issue for .fc.zip for some.
All that's really needed is IPS+checksum+>16MB (for non-NES games.)
And if you want insertion, deletion; then you basically want a sliding-window approach. It need not be any more complicated than LZ.
tepples wrote:
Could it also be used for displaying the game's title?
I think the filename is better for that because it's more accessible to change. I don't believe in such a thing as an "official" name. There is too much crazy stuff you could go either way on. I had to create a giant rulesheet just to get myself to make an objective decision. There are games like Robot Gyro a.k.a. Gyro a.k.a. Gyromite. Disney puts "Disney's" before the title of every game. Should that be included in the beginning, end, or at all? There are all kinds of romaji arguments. Then there's ambiguous title priority: Bootgod puts "Zoda's Revenge" before "StarTropics II" and I don't.
Quote:
I guess that shoots down the idea of using the title entry to document the official title of an unlicensed game, much like an ID3 tag.
You have to remember that rom-side mapping, just like iNES, has the inherent issue where the board must be a part of the checksum calculation if anyone were to try databasing them for future validation (not mapping delivery). You can't exclude it and risk greenlighting files that don't work. So you want the container contents to be as fixed as possible to reduce or eliminate the need for revisions.
Quote:
In this sense, MMC6 has additional WRAM because mapper 4's base configuration is PRG ROM + CHR ROM + no WRAM, but MMC5 does not (unless it's a Koei game or the like) because 1024 bytes of ExRAM at $5C00 are part of the base configuration.
What was the minimum amount of WRAM an MMC3 board could have above 0? What was a valid increment?
I need to research this more, because it seems like the iNES idea is impossibly cookie cutter. It's trying to consolidate the commonalities between boards into numbers, and then what it can't consolidate, it's creating special attributes for. Why did 4-screen mirroring boards get an attribute field and not their own mapper id? Because it's too small of a difference to waste a mapper number on? Why do we want to reserve a field for a technique used by two games?
Why do you want a board file? Could it be a manifest file with board and other info to be added?
Okay, I'm not sure I'm doing this correctly. For example, I have the PRG and CHR blocks of Addams Family, The - Uncle Fester's Quest (NTSC)(Eng)(1.0), as hashed by fsum -sha256:
9530571c8deb8852a370e297caf8c5604036b93dfe3a05ec8fbe937b563247f9 ?SHA256*chr
f979cc2d7650a11ac3dd23d6e7a5262adad26550a2781d44ccac761488de7ff2 ?SHA256*prg
Okay, so I'm supposed to sort those alphabetically, which they already are, then append the two strings together and SHA-256 that:
a0e0c9f62d44b83367d476f467ef12479a3bb5f738893400ecdcd2a4b8f4a2b4 ?SHA256*sum.txt
Yet the sum in the database is:
a31e9f8d1635eaa5986e9139accde7473b787f8230b692fda02941c94bcf133e
Oh, and I found a typo in the filename listing:
Batman - Return of the Joker (PAL)(Eng)(1.0.fc.zip
Oh, never mind. I see the problem. I had assumed that since the database listed hashes in lowercase, that it required the hex strings of the individual file hashes to be lowercase. It requires them to be uppercase.
a31e9f8d1635eaa5986e9139accde7473b787f8230b692fda02941c94bcf133e ?SHA256*sum.txt
I forgot to clarify the case of the string in the guide, that is now corrected, as is the Batman typo. Thanks.
FWIW, info about game pcbs has been collected by various sources. e.g. in the MESS xml list I advertised here last summer [1]
http://nesdev.com/bbs/viewtopic.php?t=6558
(xml file can be found here:
http://git.redump.net/cgit.cgi/mess/tree/hash/nes.xml )
and fwiw split files (PRG+CHR) are floating around in the web since august, so it's not like there would be no way to find the correct dumps...
the main differences between this new proposal and MESS approach is in the choice of separate board files vs. a central db, and in the choice of the checksums, since we only use crc32 + sha1 instead of larger hashes... (there would also be the inclusion of non-official releases, but given FitzRoy background you won't convince him to change his mind... which makes me sorry for the various Nanjing games, which are small gems despite being chinese pirates)
however, I wish all the luck to FitzRoy and I hope someone will get interested in supporting proper dumps one day, no matter which format will be used
[1] for lack of time, I still haven't had time to add homebrew to the list (which was a common objection when I introduced the db and which has been brought up here as well), but users can manually add entries to the database by opening the nes.xml file with a text editor.
I looked at the database and have some proposals for ZapFC++:
-Change (Jap) to (Jpn), the politically correct abbreviation, or better yet use Nintendo's country codes instead of languages
-Remove .fc.zip from the filename field, it's implied
-Use TLROM instead of HVC-TLROM-04 etc, an emulator won't necessarily have board aliases and the preservation of this data is entirely unnecessary for emulation
-Remove (NTSC) and (PAL), they're implied by the region
-Don't include board description strings, they're unofficial, 99% irrelevant and many of them don't even match the board/mapper in your database. The few descriptions with useful info such the Bandai SEEPROM variants should have unique boards names or a board variant column so "DRAGON BALL Z-B/24C01" for example.
-Don't include CIC info (should be optional at most since it's irrelevant to NES emulation)
-Don't include mirroring info, this is also implied by the board variant. Super Mario Bros would be "NROM-256/H" and games with protection diodes or other random jumpers would have their own board/board variant.
-Get rid of iNES mapper numbers or there's no credibility to your movement, if emulators are going to hack on support they can use their own lookup table
-Include a header row (in the board file too) so that the column order is flexible and can be expanded (with multiple hash types for example since people have their preferences) it also will allow unnecessary crap to be bundled in the database (and ignored)
-Allow packing RAM dumps in the container so that any games with suicide data may function, and game saves may be stored/distributed analogous to a cartridge at the user's discretion
-Allow the use of folders as containers in addition to compressed archives, this would allow ROM collections to compress better and make the format less demanding to implement on a NES itself
Some big ideas I'm brewing (maybe I'll fork my own format and implementation):
-Allow for arbitrary ROM filenames, the idea being that they will be named according to the silkscreen or the part/location designator. Reserve PRG and CHR for games without codes. Map board-specific ROM segments to their filenames. Hash each ROM separately. Right now your format requires joined PRG and CHR frankenfile ROMs, and not supporting arbitrary ROM loading and mapping doesn't allow for things like DSP or CIC dumps to be included or nametable ROMs.
-Allow for the board file to contain multiple database rows so that multiple game revisions can packed together and revisions with the same CHR ROM for example can map to the same file.
===========
The format is now flexible enough to not only suit every NES game but it can suit every console and coin-operated game, all through a single auditing and management tool only understanding one format. Theoretically all platforms could use a consolidated database as well, but the DB's flatness would make it ridiculously inefficient and I/O intensive. I would use this same format for each console, only adding columns where absolutely necessary. I also propose use of the same database with CD and DVD images (each track gets its own row) creating a centralized "cuesheet" and TOC repository. For this reason I would rename the board file to <insert console/disc database extension>.db. Since these db fragments are exactly that, the management tool could absorb new dumps easily and dispatch delta fragments to update old databases or just run a game in an emulator which doesn't parse the database at all.
kyuusaku wrote:
The format is now flexible enough to not only suit every NES game but it can suit every console and coin-operated game, all through a single auditing and management tool only understanding one format. Theoretically all platforms could use a consolidated database as well, but the DB's flatness would make it ridiculously inefficient and I/O intensive. I would use this same format for each console, only adding columns where absolutely necessary. I also propose use of the same database with CD and DVD images (each track gets its own row) creating a centralized "cuesheet" and TOC repository. For this reason I would rename the board file to <insert console/disc database extension>.db. Since these db fragments are exactly that, the management tool could absorb new dumps easily and dispatch delta fragments to update old databases or just run a game in an emulator which doesn't parse the database at all.
replace .db with .xml, cd/dvd images with .chd format (used by MAME/MESS for CDs and DVDs) and you will get exactly what MAME/MESS is doing since almost one year. add cmpro as the single tool necessary to handle software for all the databases.
I'm not saying that everyone should use the MESS xml format (
there is always space for alternative approaches, as long as the info remain open so that all projects can gain from progresses done by other projects), but I really find weird seeing devs which consider as "brand new" something which has been around since last spring for most consoles, whose correspondent files are pretty widely available, and that is nothing else than the MAME approach translated into an external database (and the external database is available for MAME too since more than 5 years, if you parse the -listxml output)
talking about reinventing the wheel over and over
etabeta wrote:
the main differences between this new proposal and MESS approach is in the choice of separate board files vs. a central db, and in the choice of the checksums, since we only use crc32 + sha1 instead of larger hashes... (there would also be the inclusion of non-official releases, but given FitzRoy background you won't convince him to change his mind... which makes me sorry for the various Nanjing games, which are small gems despite being chinese pirates)
As I've been trying to imply, the best solution here is to allow use of multiple board databases. Give FitzRoy the job of maintaining the list of NES games licensed by Nintendo, and have someone else make one for all pre-1996 commercial NES games (e.g. Tengen, Camerica, Color Dreams, AVE). Then we could have someone who specializes in the Nanjing games, etc. Possibly completed homebrew projects could get their own database maintained by nesdev and/or pdroms.
kyuusaku wrote:
Remove .fc.zip from the filename field
The filename field is for maintainers' convenience. It isn't necessarily to be displayed to the user.
Quote:
Use TLROM instead of HVC-TLROM-04 etc
I'll answer this one in Python:
Code:
for part in board_name.split('-'):
if part in known_boards:
board = part
elif part == 'HVC':
using_expansion_sound = True
Quote:
Include a header row (in the board file too) so that the column order is flexible
Seconded.
Quote:
Allow packing RAM dumps in the container
You mean "trainers"?
Quote:
Reserve PRG and CHR for games without codes
That'd be easy. Replace file named "prg" with file whose name contains "prg", case-insensitive.]
Quote:
or nametable ROMs
The mappers I've seen such as
68 include nametable ROMs in CHR ROM space.
@etabeta: Yes, I'm aware that this approach is very much like how MESS works without the overhead of XML. I use tab-separated files at work because XML is so bloated.
FitzRoy wrote:
Let's say you try to verify your files against a trusted source, and you get a checksum mismatch. Is it because of a bad mapper, a bad chr, or a bad prg?
The ZapFC checksum is a hash of two hashes, so you still have the same problem.
Anyway (and I'm reluctant to even post in this thread, considering FitzRoy's reaction to any criticism), I just don't get the point. My emulator requires checksum-based mapping for few games. If I'm going to implement a (complete) database, I'd much rather prefer a simpler format that accomplishes the same thing, e.g. prghash,chrhash,board (where board could be iNES mapper number+new mappings to accommodate variations). If there's a match, indicate that it's a verified dump, if not, fall back on the iNES header.
One more thing re: iNES being a custom container. Who cares? The data itself already requires a custom tool (the emulator) to interpret it, so what's the benefit of distributing the files in an 'standard' container?
- Funny. After 14 pages of discussion, the "new" format is being accepted now... instead of rejected. Good job, dude.
tepples wrote:
@etabeta: Yes, I'm aware that this approach is very much like how MESS works without the overhead of XML. I use tab-separated files at work because XML is so bloated.
as I said, there is always space for new approaches, and better info can only improve the overall quality of all the databases (as long as info are kept public)
one of the reasons we used xml in MESS was because MAME already had all the necessary code to handle it (plus the config and cheats are in xml format too, so we reduce to minimum the parsing code this way), but I can understand if other emus decide to use a different approach because e.g. expat is too large or because you consider the syntax 'bloated'
anyway, whatever database approach is going to be used I'd love to see proper dumps finally used by emulators
in addition to iNES2.0 and UNIF.
for this reason, feel free to exploit the info from nes.xml for all the carts which are still unconfirmed by bootgod: I have tested all of them one by one so that the basics are correct (even if you might want to keep in mind that I have always used generic NES-TLROM and NES-SLROM without MMC types for the carts not in bootgod's db). a simple python or perl script should help converting all info, except the checksum...
tepples wrote:
Quote:
Allow packing RAM dumps in the container
You mean "trainers"?
Initial save RAM files.
There is one Game Boy game where the save RAM has program code in it. Battery died? Game is now dead as well. There are also known games where RAM has to be initialized to a certain value for the game to work properly, and sometimes that exact value conflicts between different games.
Idea here would be to use this file to initially populate the RAM contents the first time you load a game. Not sure how useful that'd be on the NES in particular, but it's nice in theory.
etabeta wrote:
as I said, there is always space for new approaches, and better info can only improve the overall quality of all the databases (as long as info are kept public)
There's a good bit of hate for XML. I wasn't fond of it until I gave it a chance. It's really great being able to extend the spec a bit without breaking the old format.
As for expat, it's pretty easy to write a 99% conformant parser in ~3-8KB of code if that is your concern. But you should probably stick with expat for speed and correctness anyway.
But yeah, let people use whatever they want. Conversion isn't that difficult.
James wrote:
The ZapFC checksum is a hash of two hashes, so you still have the same problem.
that's why in my opinion it would be smarter to keep the two files separated and zipped up in a single archive
James wrote:
Anyway (and I'm reluctant to even post in this thread, considering FitzRoy's reaction to any criticism), I just don't get the point. My emulator requires checksum-based mapping for few games. If I'm going to implement a (complete) database, I'd much rather prefer a simpler format that accomplishes the same thing, e.g. prghash,chrhash,board (where board could be iNES mapper number+new mappings to accommodate variations). If there's a match, indicate that it's a verified dump, if not, fall back on the iNES header.
I think the whole problem was to handle files without a iNES header
anyway, you can parse a larger database and simply drop the info you don't want to check... e.g. you can only take in consideration the 1st, the 3rd and the 4th field of each entry in ZapFC, or only the <dataarea name="prg"> & <dataarea name="chr"> of nes.xml from MESS, and ignore the rest taking care of those aspects in the core of your emu
James wrote:
One more thing re: iNES being a custom container. Who cares? The data itself already requires a custom tool (the emulator) to interpret it, so what's the benefit of distributing the files in an 'standard' container?
accuracy and allowing people like bootgod to repair carts... up to you if these are enough
etabeta wrote:
anyway, you can parse a larger database and simply drop the info you don't want to check...
or I can parse the iNES header and drop the info I don't want to check
etabeta wrote:
James wrote:
One more thing re: iNES being a custom container. Who cares? The data itself already requires a custom tool (the emulator) to interpret it, so what's the benefit of distributing the files in an 'standard' container?
accuracy and allowing people like bootgod to repair carts... up to you if these are enough
Ignoring incorrect header info (irrelevant with a verified checksum), I don't see how iNES prevents either of these from happening. Granted, a .zip file is easier to modify than an iNES container. And if the iNES header has an incorrect number of prg/chr banks, all bets are off.
I get it -- iNES is not ideal. For the vast majority of situations, though, it works fine, and the edge cases are easy to handle.
On a side note, why haven't the edge cases been assigned to a new iNES mapper number already? Isn't that a good indicator that the problem has already been adequately addressed?
And now
QuickNES is the first NES emulator to support the ZapFC database. Currently, it supports the database raw or packed into any of the supported archive formats. (zip, gzip, rar, or 7-zip LZMA/LZMA2/PPMd) It searches the configuration folder first, then the program folder, for a file named ZapFC*.fbsv*, favoring the one with the newest modification timestamp. It also correctly supports the board file extension, although it currently allows the first three fields to be populated.
The ROM packages are accepted as *.fc.<archive extension>.
It also supports (rom name).ips.* archived split patches for ZapFC ROM packages, also in any of the supported archive formats.
Oh, and because I'm lazy, and because QuickNES doesn't currently support board differentiation anyway, I use the board type as the mapper number.
I also eliminated my original mapper/mirroring override table, since most of the games it contained were probably licensed games covered by this database. The rest can be fixed with proper ZapFC board files, or if the world really needs them, properly edited iNES headers. (I'm looking at you, Buzz and Waldog, being distributed with a mapper number 255 in the header, but having a CRC32 override to mapper 156 built into various emulators well before the ROM image was even released to the general public.)
I also still need to work out some issues, though, such as >8KB SRAM and handling different banks of SRAM being battery backed or not. At this point, QuickNES only supports 8KB of SRAM, and I'm not exactly sure which mappers support more than that, how they handle swapping it, and whether it even matters as I may not even support any of the relevant mappers.
And while I'm at it, I will mention that I also attempt to match iNES ROM dumps against the database for mapper and mirroring overrides. I already found one dump that was GoodNES verified that failed this match. Maybe my original dump was old or bad, and has since been replaced?
etabeta wrote:
that's why in my opinion it would be smarter to keep the two files separated and zipped up in a single archive
Agreed. So if the primary goal is verifiability/preservation, then how about a zip container containing (only) prg and chr, with a simple database mapping mechanism. No match, no go. Homebrew, etc. stick with iNES (or iNES2 or UNIF).
Wow this all fell behind XD So I don't have to quote etabeta's whole post pretend it's right under.
----------------------
Granted I don't use MESS, but I haven't seen MAME pack XML with ROM sets. Even if it has been done for a year, that's pretty much a blink of the eye; I, like a lot of other people probably, don't really follow either project.
I tried ClrMAMEPro twice, once many years ago, once not so many years ago, quickly removing it each time. From my recollection it's a slightly different concept being strictly an auditing tool where 3rd parties distribute whole databases. It takes hashes and renames (and possibly groups) files into sets, it cannot manipulate anything really, has no concept of ROM-"board" mapping, probably can't redefine it's schema, merge databases or even import/export flat files for editing.
Sure massive XML files have been "done", but as of now they still only implement a static system in a dynamic package, which I feel misses the mark.
On a side note I think big XML (and 50mb executables) are not only computationally unwieldy, but I find them susceptible to "micro-preservation"; people incessantly documenting minutia like a cart's screw color. OK maybe that's...somewhat...interesting, but the "work now, think later" mindset (in No-Intro's case "learn later") results in impressive work in scale, anemic in innovation. An example of this: MAME/MESS put preservation at their forefront, but is there a single platform emulated approaching mediocre NES emulator accuracy? The vast majority of MAME drivers are littered with so-called "kludges" and AFAIK nothing but high-level video emulation. I constantly find myself going to the source for game lists, memory maps and such, and it serves me well, but from past experiences there's not much else I would trust. Most of the time implementations are so convoluted to me I'd sooner reinvent the wheel as to not use MAME's framework.
As for .CHDs, yeah they can pack a disk/disc image, but they cannot be used the same way I have proposed; they can't be used like a scalable cuesheet (I'm not really sure they have proficient metadata capabilities) to auto-organize someone's AccurateRip/FLAC collection for example. That would be cool.
kyuusaku wrote:
Granted I don't use MESS, but I haven't seen MAME pack XML with ROM sets. Even if it has been done for a year, that's pretty much a blink of the eye; I, like a lot of other people probably, don't really follow either project.
we don't pack xml with roms, and I have said that, but you can get an xml database from the exe
bsnes contained support for xml packed with the ROMs since two years at least, though.
where do you think FitzRoy took this idea? he helped byuu too.
kyuusaku wrote:
Sure massive XML files have been "done", but as of now they still only implement a static system in a dynamic package, which I feel misses the mark.
your opinion, and I respect it. leaving the burden of creating the xml to the user, otoh, might be seen as unfriendly and this is what made some emu author to prefer a central xml (see e.g. Nestopia and MESS). to each his own.
btw, 2MB-4MB is far from being massive in my opinion
kyuusaku wrote:
On a side note I think big XML (and 50mb executables) are not only computationally unwieldy,
memory discards unused pages on any CPU from the latest 5 years... check memory usage of MAME for mid-80s games (of course, if you check noami or neogeo, memory usage is heavier since you have to load some hundreds of MB of romdata in the memory)
Quote:
The vast majority of MAME drivers are littered with hacks and AFAIK nothing but high-level video emulation.
can you provide explicit examples or are you referring things you have heard around? 'cause e.g. voodoo code from MAME has suddenly made DOSBox capable of running tons of mid90s titles (
http://vogons.zetafleet.com/viewtopic.php?t=25606 )... despite being probably not good enough in your eyes...
Quote:
As for .CHDs, yeah they can pack a disk/disc image, but they cannot be used the same way I have proposed; they can't be used like a scalable cuesheet (I'm not really sure they have proficient metadata capabilities) to auto-organize someone's AccurateRip/FLAC collection for example. That would be cool.
I don't know what you refer exactly to as scalable cuesheet, sorry... anyway CHD contains metadata of course: you can easily get cue+bin out of a CHD whenever you want, even if it is currently not possible to split the tracks automatically in separate bins
James wrote:
Agreed. So if the primary goal is verifiability/preservation, then how about a zip container containing (only) prg and chr, with a simple database mapping mechanism. No match, no go. Homebrew, etc. stick with iNES (or iNES2 or UNIF).
That's what MESS currently does. Except that I'd like to add homebrew to the database...
I have a NES game that has two variants, MMC1 and a UNROM variant. How will these be listed in the database? That will cause some clutter and confusion.
3gengames wrote:
I have a NES game that has two variants, MMC1 and a UNROM variant. How will these be listed in the database? That will cause some clutter and confusion.
I'm thinking, what if we distribute a bad ROM and it's not until later it's discovered when it's been spread around. Then what? Update the database? That will make the ROM unplayable for the user once the the database is updated, because now the checksum doesn't match with any database entry (Not in database + no header* = no play).
Possible solutions and problems:
Update the entry in the database to match the new dump.
This will cause the old ROM to be unplayable (Remember, it has no header* supplied, so the emulator has no way of telling what to do with the ROM).
Keeping the entry in the database, but add a new one for the good ROM.
Isn't this one of the main complaint about the iNES files? The bad ROM will still be in circulation. Besides, then it kinda defeats the purpose of this format all together.
Auto-update the ROM while keeping the entry in the database.
This might work, but what if they have no internet connection (Don't ask me how they got to update the database... Maybe they just replaced the emulator with a more up-to-date one?)? What if its stored on a read-only format, such as the CD someone here mentioned?
* Header - Definition: Small piece of information that defines how the rest of the information should be used. Not necessarily at the beginning in a file. (Same concept as C header files).
tepples wrote:
I'll answer this one in Python:
Code:
for part in board_name.split('-'):
if part in known_boards:
board = part
elif part == 'HVC':
using_expansion_sound = True
What if the board is unlicensed? There should probably be a more reliable method. Same for expansion audio:
Code:
if territory is 'Jpn' and (not using_nes_with_converter or has_resistor_mod) and board in expansion_sound_boards:
using_expansion_sound = True
Quote:
You mean "trainers"?
Not really (those can be put into any WRAM save), the idea is to preserve the cartridge's true initial state (could contain easter eggs, suicide data, or a game could be shipped with a special save file that that may never be experienced again after saving over it or something). There is little chance of many games being preserved this way, afterall you'd have to dump never played carts which happen to still have sufficient battery charge, but it's a logical feature for applications like the Game Action Replay or homebrew.
Quote:
The mappers I've seen such as
68 include nametable ROMs in CHR ROM space.
Which to my understanding constitutes a "frankenfile". ZapFC's specification isn't very consistent with what I gather to be the philosophy behind the format, which is why I suggested losing all iNES-granted liberties in ZFC++. Funny enough these changes will actually create more work for manual hax0rs.
etabeta wrote:
we don't pack xml with roms, and I have said that ;)
bsnes contained support for xml packed with the ROMs since the past few years, though.
where do you think FitzRoy took this idea?
I don't know, but an XML (or any descriptive file) packed with a ROM by itself is not novel, I think when it becomes more than a glorified .ini file is it interesting. I didn't think this ZapFC thing was interesting at the start of the thread, and I still don't think the arguments for it are significant, more self-indulgent; today (yesterday now) I spent a lot of time on an iNES fixer using NesCartDB's XML which progressed into working on an intermediate CSV database of my own (much like I described) which I realized could serve many organizational purposes, something I'm very excited about.
Quote:
btw, 2MB-4MB is far from being massive in my opinion ;)
Maybe 2-4MiB for licensed NES games with all the redundancy... 2-4MiB of XML isn't so hot on a computer without 2-4MiB of cache. I don't want even a few seconds more delay when I load a game on my netbook with a slow spinning disk. Something I should also point out is that it's not just about parsing,
http://git.redump.net/cgit.cgi/mess/tree/hash/nes.xml nearly brought my computer to a halt. From now on there will be layer upon layer of markup == exponential growth.
Quote:
memory discard unused pages on any CPU from the latest 5 years... check memory usage of MAME for mid-80s games
??? what does ROM loading have to do with in-game resources?
Quote:
can you provide explicit examples or are you referring things you have heard around?
I've looked at lots of drivers, the standard way seems to be to set a very rough periodic un-synchronized NMI/"render_frame"() timer which effectively halts emulation, loops through VRAM rendering pre-optimized tiles to scroll/sprite layer buffers, then prioritizing them and clipping. It's standard fare for sprite evaluation to include speedhacks. Compare this to NES emulators which attempt to emulate the state machine and internal logic of the PPU rendering on a pixel by pixel basis, in sync with the CPU. Sure that level of accuracy isn't necessary to get games running, but it's doable, especially for the older ones.
Quote:
I don't know what you refer to as scalable cuesheet, sorry...
It's a simple text file describing how to compose a CDROM from binary track data, gaps and any metadata. The cuesheet can easily be implemented in only a few DB columns per track (which might be a row). To build an accurate CD image you'd only need to collect the binary/PCM tracks it's comprised of (which are easier to compress, transfer, verify and in the case of PCM listen to, when split) and not worry about generating or finding one to download. Right now it's a big hassle to archive CD games accurately, there are many steps in lots of tools, after which you have to manually check track hashes against sites like redump.org for verification. It can be easier.
3gengames wrote:
I have a NES game that has two variants, MMC1 and a UNROM variant. How will these be listed in the database? That will cause some clutter and confusion.
This seems very unlikely, which game? Typically these are hacks for early copiers with UOROM support but not MMC1.
The MMC1/UNROM is Winter Games. I guess the MMC1 was too expensive and remade the game for UNROM? My version is 5 screw. There is also a 3 screw version, but I don't have one to say if the 3 screw versions are the UNROM.
3gengames wrote:
I have a NES game that has two variants, MMC1 and a UNROM variant. How will these be listed in the database? That will cause some clutter and confusion.
two separate entries, of course.
Wkter wrote:
I'm thinking, what if we distribute a bad ROM and it's not until later it's discovered when it's been spread around. Then what? Update the database?
this is what is going on with arcade dumps in MAME since 14 years, so yes, you should exactly update the database.
Wkter wrote:
That will make the ROM unplayable for the user once the the database is updated, because now the checksum doesn't match with any database entry (Not in database + no header* = no play).
if the original entry was a bad dump, throwing away the file is the best solution to avoid preservation of bad dumps like some website offering GoodNES sets keeps doing
Wkter wrote:
This might work, but what if they have no internet connection (Don't ask me how they got to update the database... Maybe they just replaced the emulator with a more up-to-date one?)?
let me then ask: how did they obtained the new dump? in the same way they will obtain the new database (or they will simply update by hand the xml file.... a text editor is enough for such a task)
things change a bit for homebrew, though. that's why in my opinion, support for per-game xml inside the zipfile should be possible too... this is currently not available in MESS for NES, but support exists for AES and TI99, so it is planned for NES too.
I guess I'm out of this now, it's pointless. It's a downgrade for iNES if there ever was one. You guys can try to make it happen, but don't expect anyone to use this. Ever.
kyuusaku wrote:
Quote:
btw, 2MB-4MB is far from being massive in my opinion
Maybe 2-4MiB for licensed NES games with all the redundancy... 2-4MiB of XML isn't so hot on a computer without 2-4MiB of cache. I don't want even a few seconds more delay when I load a game on my netbook with a slow spinning disk. Something I should also point out is that it's not just about parsing,
http://git.redump.net/cgit.cgi/mess/tree/hash/nes.xml nearly brought my computer to a halt. From now on there will be layer upon layer of markup == exponential growth.
problem is in the website I linked: it regenerates the html code each time someone accesses it. I will upload it separately later, if anyone wants to take a look without using the not-so-optimized page I linked.
actual emulation parses the whole file at loading time in ~4 seconds on my 2-years old EEEPC, and in a heartbeat on my 2GHz macbook
kyuusaku wrote:
Quote:
memory discard unused pages on any CPU from the latest 5 years... check memory usage of MAME for mid-80s games
??? what does ROM loading have to do with in-game resources?
I must have misread your original post, then. sorry. I thought you were saying that using a 50MB exe (like MAME/MESS) was too much for e.g. PacMan. My reply was on the line of: actually only the specific part of the program needed by the game is loaded, not the whole exe. if you emulate pacman, there is no speed drop due by exe dimensions and big speed drop in old games (e.g. donkey kong) has usually been due to the implementation of discrete sound and the removal of wav samples.
I mixed up things when talking about the used resources, though. sorry.
kyuusaku wrote:
I've looked at lots of drivers, the standard way seems to be to set a very rough periodic un-synchronized NMI/"render_frame"() timer which effectively halts emulation, loops through VRAM rendering pre-optimized tiles to scroll/sprite layer buffers, then prioritizing them and clipping. It's standard fare for sprite evaluation to include speedhacks. Compare this to NES emulators which attempt to emulate the state machine and internal logic of the PPU rendering on a pixel by pixel basis, in sync with the CPU. Sure that level of accuracy isn't necessary to get games running, but it's doable, especially for the older ones.
mmm... some arcade games do indeed use large tilemaps and not the same pixel-by-pixel approach of NES PPU. as such, there is no pre-optimization or speedhacks, but only accurate description of the real hardware (see e.g.
http://aarongiles.com/?p=211 and
http://aarongiles.com/?p=212 ). however, you might also have caught some not-so-accurate driver, since a few of them have not been touched for long time (except for macro updates).
anyway, many drivers are very accurate and based directly on the hardware schematics or on tests on the real pcb (see e.g. crystal castles and many old atari systems, or more complicate systems like the kaneko supernova which got accurately emulated by haze last year)
kyuusaku wrote:
It's a simple text file describing how to compose a CDROM from binary track data, gaps and any metadata.
I know about the cuesheet, is the 'scalable' part which was unclear to me. however, as I said, you can already recover the cue from a chd: all necessary metadata is of course included
kode54 wrote:
And now QuickNES is the first NES emulator to support the ZapFC database.
Sweet. Loading works great. And I can even replace the database with a newer one on my own without waiting for an emulator update. Now I need to complete my board reference table so I can make sense of all the variances. Fair warning: this is early in the game and an additional field may get added for this.
James wrote:
The ZapFC checksum is a hash of two hashes, so you still have the same problem.
I can easily find out which it is with this format because I can just open up the container and drag the files over HashCalc. You have to find a specialty tool somewhere to get access to that info with iNES.
Quote:
Anyway (and I'm reluctant to even post in this thread, considering FitzRoy's reaction to any criticism), I just don't get the point.
I'm averse to repeating myself is all. If you don't get the point, don't support it. It bothers me none if you trust distributors to deliver your mappers and you want to wait 10 years before format updates take effect for users if at all.
Quote:
One more thing re: iNES being a custom container. Who cares? The data itself already requires a custom tool (the emulator) to interpret it, so what's the benefit of distributing the files in an 'standard' container?
For dumpers and patchers to easily access simple information that will help them be more efficient at what they do. And the container is no longer used for mapper delivery of the main library.
3gengames wrote:
I guess I'm out of this now, it's pointless. It's a downgrade for iNES if there ever was one. You guys can try to make it happen, but don't expect anyone to use this. Ever.
You've trolled this thread with vacuous comments enough, already. You won't be missed.
What I really want to see is custom mappers for NES files. Need to come up with a good standard for them, and get that feature integrated into FCEUX, Nestopia, and Nintendulator. Throw in the Mapper specification as a trailer of the NES file. Then emulators won't have problems with obscure Chinese pirate games with simple mappers or rewired variations of MMC3.
Lets default mapper 255 to a custom user executable file that the emulator can run and make the way the emulator runs it standard so even closed-soruce emulators can run it without any problems. Either way, you'll have to edit the emulator no matter what file system you use. Lets go back on the old forum post where we'd have a set of bytes that tell what connects to what on each chip. There's so many better solutions then these other file formats that nobody wants, nobody will use, and nobody will ever use because no emulator will support them.
And I will be happy when this thread finally bites the dust, I won't miss it either. This new idea is 100% worthless. NES roms are preserved 110% as it is, I am "trolling" because of how pointless this topic really is. They come up every year, it's only that this ones has gotten more attention because of how ridiculous it is. Nobody will ever support this and it will just become another hurdle for emulators to support another ROM format that only one person will ever use. This is like what was mentioned earlier about the two file types for SNES ROMS. It's only a headache and not worth the time already put into thinking up this idea.
Quote:
bsnes contained support for xml packed with the ROMs since two years at least, though.
I do not support packs, period. I use actual folders. But yes, I have a ROM+XML format. This is the way my XML looks:
Code:
<?xml version="1.0" encoding="UTF-8"?>
<cartridge region="NTSC">
<rom>
<map mode="linear" address="00-2f:8000-ffff"/>
<map mode="linear" address="30-3f:8000-ffff" offset="100000"/>
<map mode="linear" address="80-af:8000-ffff"/>
<map mode="linear" address="b0-bf:8000-ffff" offset="100000"/>
</rom>
<ram size="2000">
<map mode="linear" address="70-77:0000-ffff"/>
</ram>
</cartridge>
The format is meant to describe the literal mapping of the entire image onto the address space. Unlike the NES' 64KB address space, there is a 16MB address range, and there are dozens of mapper configurations for how to actually place ROM data into that space.
One of my primary design goals was to allow homebrew, which I've accomplished. You can easily play neviksti's 96mbit Star Ocean hack, for instance.
The format is also used for mapping in special chips, and you can provide coprocessor frequency speeds, MMIO address ranges, etc. You can even map in multiple special chips onto the same game, which is impossible with internal headers.
Anyway, I don't really think it's directly contrastable to the NES. NES chips are more about being MMCs, whereas SNES chips are more about being MMIO-mapped coprocessors. I think XML works great for SNES, and the extensibility and flexibility of XML is a godsend. For NES, I don't see it being very useful.
Lastly, I do intend to have a hard-coded PCB ID database for official games and their official mappings. I don't really intend to use XML, as it does not need a tree structure, but I may anyway.
The internal DB will basically be:
name, PCB ID (equivalent to mapper# in NES parlance), sha256
<repeat>
And will serve as a falback if an actual XML file is not present.
Quote:
What I really want to see is custom mappers for NES files.
I really want to work on a custom, universal-style NES mapper syntax, it seems like such a fun challenge. But I have less than zero interest (yes, negative interest) in writing an NES emulator. Ah well.
Quote:
actual emulation parses the whole file at loading time in ~4 seconds on my 2-years old EEEPC, and in a heartbeat on my 2GHz macbook
You have to parse the whole XML file? Why?
I have a 2MB cheat database, and it takes a few seconds to parse (I wrote my own 3KB XML parser, so it's not quite as fast), so I just use a speed hack: do a raw string search for the checksum of the game, then do a seek-back to the start of that group's tag, then cut out that single block, and parse it. Takes ~10-20ms tops for the whole operation.
Wkter wrote:
3gengames wrote:
I have a NES game that has two variants, MMC1 and a UNROM variant. How will these be listed in the database? That will cause some clutter and confusion.
I'm thinking, what if we distribute a bad ROM and it's not until later it's discovered when it's been spread around. Then what? Update the database? That will make the ROM unplayable for the user once the the database is updated, because now the checksum doesn't match with any database entry (Not in database + no header* = no play).
Intended. Once the bad ROM is discovered as such, it is no longer the most accurate dump of a licensed game and therefore no longer deserves to be playable from a board database containing the most accurate dumps of licensed games. We
want bad dumps to be unplayable.
kyuusaku wrote:
Quote:
The mappers I've seen such as 68 include nametable ROMs in CHR ROM space.
Which to my understanding constitutes a "frankenfile"
I looked up
After Burner (J) on NesCartDB and it's sick. Not only are there two separate CHR ROMs, but the mapper IC is mounted askew. These would be handled the same way as Action 52's three PRG ROMs.
kyuusaku wrote:
It's standard fare for sprite evaluation to include speedhacks. Compare this to NES emulators which attempt to emulate the state machine and internal logic of the PPU rendering on a pixel by pixel basis, in sync with the CPU.
And a lot of NES emulators include speedhacks for those huge parts of the screen with no PPU port writes and sprite 0 not in range.
kyuusaku wrote:
Sure that level of accuracy isn't necessary to get games running, but it's doable, especially for the older ones.
Then perhaps you need to make a tech demo that runs on the arcade board but fails on MAME. Wrap a simple game around it, make it accept coins, and install it in an arcade somewhere. I might even be willing to help you with putting a Tetris clone or rhythm game around a tech demo if it's a 6502-based board. But still, it's likely that certain kinds of raster effects are less likely on some arcade systems because they need to be able to flip the screen for cocktail mode.
3gengames wrote:
Lets default mapper 255 to a custom user executable file that the emulator can run and make the way the emulator runs it standard so even closed-soruce emulators can run it without any problems.
But in what architecture would this executable be stored? x86? ARM? JVM, CLR, or LLVM? If you have any ideas for how to represent a mapper's logic in a way that is 1. portable across archs and 2. easy to compute, I'd be pleased if you'd offer your proposal in a new topic.
FitzRoy wrote:
I can easily find out which it is with this format because I can just open up the container and drag the files over HashCalc. You have to find a specialty tool somewhere to get access to that info with iNES.
Instead of using your specialty tool, HashCalc (Yes, HashCalc is a specialty tool aswell), why not
use my specialty tool that I just wrote and drag ONE iNES file onto it, then paste your hashes into notepad or your favorite text editing specialty tool. Hell, it even saves you the trouble of generating 3 hashes manually!
Example output of Adventure Island 2:
Code:
PRG - E021CEDF0B7700B3EEE7E5B76B2E23000075E06F4D7C13A4B556452952D3AEC0
CHR - 7F28B46897CA425F840776805E79902E246C450B5A21939E7DF78B2300C8C81B
Combined - 7A675D4C03C7E0CB336083AA1184B43790B015C18A7C38153F16D47DBE4F4634
Seems a lot simpler than opening a ZIP, drag and drop 2 files into a specialty tool, then take the two hash strings into the specialty tool again, and finally copy your final hash string, doesn't it?
You see, your arguments that the need of "specialty tools" for iNES somehow makes your format better is simply invalid, and I just demonstrated it. This also demonstrates that dumpers will not necessarily be happier with your format, as I can just as easily write a iNES file combiner/separator as well. And we're only talking about simple drag and drop programs here...
And keep in mind, I'm still just a beginner in C/C++.
I don't have to even use hashcalc to see if something is different, as CRC32 within ZIP is totally sufficient for that. I was using it to add an entry the other day, not sure why I brought it up with regards to dumping. In any case, it's still a niche specialty tool. Someone 10 years from now who was not involved in any of this would not know where to get it. It's also extra and unnecessary work to add support for custom containers in database programs when ZIP, something they already support, would work perfectly fine for these files.
You know what Fitzroy...write your own emulator. Use this format. Show us we're all wrong. And THEN we'll all adapt to your format.
Jeroen wrote:
You know what Fitzroy...write your own emulator. Use this format. Show us we're all wrong. And THEN we'll all adapt to your format.
Emulator. Already supports this format. Insignificant because it can't play all of the games preserved by the set. Oh well.
Jeroen wrote:
You know what Fitzroy...write your own emulator. Use this format. Show us we're all wrong. And THEN we'll all adapt to your format.
Are you an author? It depends on what you want. If you want a mapping guarantee for users and yourself, you distribute as many boards as you can with the emulator along with all the other hardware. If you want to make it easier on dumpers and authors of archiving tools, you don't cling to custom containers unless there's some sort of unique benefit to doing it. My spending years to create an emulator instead of archiving games isn't going to change your mind if you don't care about that.
FitzRoy wrote:
I don't have to even use hashcalc to see if something is different, as CRC32 within ZIP is totally sufficient for that. And having to open notepad and paste after dragging the file is twice as slow for that. So not only is your method slower, it's still a niche specialty tool. Someone 10 years from now who was not involved in any of this would not know where to get it. It's also extra and unnecessary work to add support for custom containers in database programs when ZIP, something they already support, would work perfectly fine for your files.
.... You're just being unbelievably stubborn and willfully ignorant...
You completely dismiss any sort of argument with some ad-hoc rationalization without justification.
Are you familiar with John Romero's advertising campings? Your format reminds me of Daikatana, and your way of promoting it reminds me of his advertising campaigns.
I'm not saying this to be mean, but you just don't consider other opinions at all. Why don't you?
This isn't a custom container? It's iNES in folder form with more clutter. Your archiving of games is done, iNES is perfect and has 100% accuracy of preserving ROM's, not data is lost at all. It'd be better to write an emulator since there isn't an emulator *yet* that is 100% perfect.
Wkter wrote:
.... You're just being unbelievably stubborn and willfully ignorant...You completely dismiss any sort of argument with some ad-hoc rationalization without justification.
No, that's what you're doing. I'm giving you reasons in plain English, and you can't rebut them, so you resort to pages of personal attacks. Why is it so hard for you to explain why you wish to burden database tool authors into supporting system-specific containers?
3gengames wrote:
This isn't a custom container? It's iNES in folder form with more clutter. Your archiving of games is done, iNES is perfect and has 100% accuracy of preserving ROM's, not data is lost at all. It'd be better to write an emulator since there isn't an emulator *yet* that is 100% perfect.
Why do you still seem so concerned with the matter? Didn't you say you were done with the thread? ZIP is a standard if there ever was one and you get your mappings from people who distribute bad roms. Take the blue bill and stop trolling.
Point to a mapper with a bad header online. This isn't even a problem, maybe 1 or 2 out of 1000 are bad.
Not sure what "Mapper with a bad header" means, but many Namcot 109 / MIMIC-1 games are mistagged as MMC3 games, and have the wrong mirroring mode set. Thanks, goodnes.
And Lagrange Point has the battery backed SRAM bit cleared.
3gengames wrote:
NES roms are preserved 110% as it is, I am "trolling" because of how pointless this topic really is.
no they're not. and if you think they are, you have never looked a picture of an open cart or you have never considered the Galaxian real cart which only has 8K prg making impossible to create an iNES header for it.
3gengames wrote:
They come up every year, it's only that this ones has gotten more attention because of how ridiculous it is. Nobody will ever support this and it will just become another hurdle for emulators to support another ROM format that only one person will ever use.
MESS already support split prg/chr, but using xml and not this ZapFC db format (and I widely prefer xml for possible future extensions, but to each his own)
And MESS isn't for NES. For MESS, Having a small header and a custom file header and ROM(s) separator for each system is unrealistic. Of course they'd have to do this. NES doesn't need this. This is going backwards for iNES and is a downgrade.
And okay, one game. We'll assign NROM-064 later as a board. Then all emulators will assume 8KB PRG.
... NROM-064 as a board. For what, that failure, UNIF?
3gengames wrote:
And MESS isn't for NES.
why not? because we are not a standalone emu focused on that system only? we *fully* emulate a lot of system in par (if not better) than standalone emus (smc777, neo geo pocket, super acan, rx-78, a2600), so why MESS should not be for NES?
we do a discrete job with it, in fact. we only have big issues with MMC5 and mappers like the Tengen ones due to the current way to handle the video code. a rewrite is in progress to overcome this limitation but I'm changing work, so the NES PPU is sort of low in my priorities.
3gengames wrote:
For MESS, Having a small header and a custom file header and ROM(s) separator for each system is unrealistic. Of course they'd have to do this. NES doesn't need this. This is going backwards for iNES and is a downgrade.
you can see it as a downgrade, other people might even like it. MESS supports the available formats for each systems + it tries to introduce a common xml format to describe accurately cart layouts. won't anyone use it? fine with that, the info will still be available next time someone wants to do something similar.
I don't see this as a competition (or as a crusade, as sometimes FitzRoy seems to), I only want the info to be available to anyone and then each emu author can decide freely what he wants to support
3gengames wrote:
And okay, one game. We'll assign NROM-064 later as a board. Then all emulators will assume 8KB PRG.
good. and then you need submappers for carts which have been assigned to wrong mappers, and then you need to properly remap the CHR in a few games like Afterburner, and then... and then...
you can go on fixing mappers over mappers, or simply use an xml layout for the data:
Code:
<description>Galaxian (Jpn)</description>
<part name="cart" interface="nes_cart">
<feature name="pcb" value="NAMCOT-3301" />
<feature name="mirroring" value="horizontal" />
<dataarea name="prg" size="32768">
<rom name="namcot gx prg" size="8192" crc="e48c1ebf" sha1="1976f2da2208405f0fcc2914ba09117980e01aa8" offset="00000" />
<rom size="8192" offset="0x2000" loadflag="reload" />
<rom size="8192" offset="0x4000" loadflag="reload" />
<rom size="8192" offset="0x6000" loadflag="reload" />
</dataarea>
<dataarea name="chr" size="8192">
<rom name="namcot gx chr" size="8192" crc="873d3083" sha1="6a6cd9a2372ef07fbb473bcee81ef976153219f4" offset="00000" />
</dataarea>
</part>
<description>After Burner (Jpn)</description>
<part name="cart" interface="nes_cart">
<feature name="pcb" value="SUNSOFT-4" />
<dataarea name="prg" size="131072">
<rom name="sunsoft-p0" size="131072" crc="88f202f0" sha1="709e2744cf4f7ce43c41ed57ec858128e008f305" offset="00000" />
</dataarea>
<dataarea name="chr" size="262144">
<rom name="sunsoft-00" size="131072" crc="10935d10" sha1="7a3568028d9e1088f4efe8e2ec074b05048c4005" offset="00000" />
<rom name="sunsoft-01" size="131072" crc="0bc56f7a" sha1="b483839a93e4952a4c90906cad4142ed1c8763c2" offset="0x20000" />
</dataarea>
</part>
<description>Action 52 (USA)</description>
<part name="cart" interface="nes_cart">
<feature name="pcb" value="MLT-ACTION52" />
<dataarea name="prg" size="2097152">
<rom name="ati-02 action 52" size="524288" crc="64e40c10" sha1="b5a53f844accb40c9584dc9105985eb5d8c6f604" offset="000000" />
<rom name="ati-03 action 52" size="524288" crc="69fdc534" sha1="0d505ecb691ce688e764affb684fbef596a68d2b" offset="0x080000" />
<rom value="0xff" length="524288" offset="0x180000" loadflag="fill" />
<rom name="ati-04 action 52" size="524288" crc="7e43e9e1" sha1="ee262a09266956a9421aaa14a946bf69627fdf73" offset="0x180000" />
</dataarea>
<dataarea name="chr" size="524288">
<rom name="ati-01 action 52" size="524288" crc="596442ec" sha1="dc31dc0870e393fc46494b5e386ab08e769ac514" offset="000000" />
</dataarea>
</part>
no possible misunderstanding, and you just have to load the data at the prescribed offsets in your PRG and CHR memory regions and you are done...
byuu wrote:
You have to parse the whole XML file? Why?
because we assign shortnames to games like MAME and if you type the wrong one, we offer best match suggestions.
the full parsing might be modified at some point (Aaron is rewriting the rom-handling code, so who knows), but I just wanted to underline that large chunks of data can be parsed pretty fast
Quote:
Not only are there two separate CHR ROMs, but the mapper IC is mounted askew. These would be handled the same way as Action 52's three PRG ROMs.
I probably wouldn't put non-ROM IC in the container. If each mapper contained a unique ROM sure.
Quote:
And a lot of NES emulators include speedhacks for those huge parts of the screen with no PPU port writes and sprite 0 not in range.
Which I'm very much for, they are lossless speedhacks.
Quote:
Then perhaps you need to make a tech demo that runs on the arcade board but fails on MAME.
Fails as in video glitches or full on crashes? MAME's CPU emulators seem robust enough. Most arcade games look like they'll run regardless of audio or video emulation at all (well, besides a Vblank interrupt).
Quote:
But still, it's likely that certain kinds of raster effects are less likely on some arcade systems because they need to be able to flip the screen for cocktail mode.
Flip adds to the hardware cost (typically a lot of XOR gates and adders), but a lot of 16-bit systems still implement line interrupts or rowscroll memory.
Quote:
But in what architecture would this executable be stored? x86? ARM? JVM, CLR, or LLVM? If you have any ideas for how to represent a mapper's logic in a way that is 1. portable across archs and 2. easy to compute, I'd be pleased if you'd offer your proposal in a new topic.
Why not a linked list traversed each event? It's anything but proprietary and relatively quick. Parsing could be easy if a mapper was broken down into a discrete operations for pointer manipulation and choosing the next node.
Zepper wrote:
- Funny. After 14 pages of discussion, the "new" format is being accepted now... instead of rejected. Good job, dude.
Hardly. He's just made it clear that he'll insult you if you disagree with him. What's the point in opening yourself to that kind of abuse?
FitzRoy wrote:
No, that's what you're doing. I'm giving you reasons in plain English, and you can't rebut them, so you resort to pages of personal attacks.
I'm sorry if you've felt I've been attacking you, but that is not the case from my point of view. I attack your poor reasoning and proposed format. I obviously don't think you're stubborn and ignorant across the board, but on this particular subject, you are.
Also, backfiring my arguments without example isn't working. If you read my post about the tool I wrote again, read your response, then read my reply back to you, it should be perfectly clear why I responded as I did.
You effectively set up a
straw man argument of my argument that you beat down, and that is a
logical fallacy. Well done, but it was not my argument you addressed. My argument wasn't at all that this was a faster way of checking if files were different. Besides, you completely ignored the fact that my tool is open source and thus easy to modify to suit your needs perfectly.
Also, your argument that in 10 years no one will know about this tool is completely bogus. If I write a set of dumping and database specific tools for work with iNES, it can be added to the wiki for everyone to see, use and modify.
FitzRoy wrote:
Why is it so hard for you to explain why you wish to burden database tool authors into supporting system-specific containers?
A system-spesific tool (Emulator) is perfectly justified to use a system-spesific format (iNES) so that it can store, read and use the data as effectively as possible when it is known that not much is going to change in the future.
Wkter wrote:
A system-spesific tool (Emulator) is perfectly justified to use a system-spesific format (iNES) so that it can store, read and use the data as effectively as possible when it is known that not much is going to change in the future.
very true in general [1]. except that in this case iNES gives a poor representation of the content of the cart since it glues together things which should stay separate, like 3 prgs of Action 52 or 2 chrs of Afterburner. or since it fails to document properly corner cases of the pcb wirings (like the variants of mapper 3 used by Sansuu games, or the Konami mappers mixed up in mappers 21-22-26), forcing emu authors to rely on crc checks.
those are the basic issues that a new format should be address. the xml database used by MESS gives a unified solution to these problems, and I guess the ZapFC database can give all the necessary info too
[1] e.g. .sfc is a very good system-specific container for SNES: it fails to document the case of carts with multiple chips (and therefore I still prefer our xml), but at least it can cover all the differences a console can detect between carts, due to the smaller number of hw difference inside snes carts
Quantity of chips!=the side of program. Okay, it's a little off, but get over it. You want to change the format completely for about 6 games even when the othher 732 or so work perfectly.
Oh come on 17 pages already ?
Can hardly belive it. Looks like the other guy will win, this usless debate WILL reach 20 pages.
etabeta wrote:
or since it fails to document properly corner cases of the pcb wirings (like the variants of mapper 3 used by Sansuu games, or the Konami mappers mixed up in mappers 21-22-26), forcing emu authors to rely on crc checks.
I agree, the mapper system used in iNES1.0 is quite weak, but iNES2.0 will be able to support for every single mapper variant out there, and Kevtris and Cowering will add iNES2.0 support in the GoodNES set, as he stated:
kevtris wrote:
I have been talking to Cowering about ines2.0 and getting it implemented in the GoodNES set. We're working on getting this done; I am waiting now for the new current set so I can go through and fix the headers that need it.
While I agree that assigning mapper numbers to the boards might not be the ultimate solution for preservation of the hardware information, I find it to work pretty well in terms of emulation. For the preservation of the hardware, we do have BootGod's database and similar. It would be great if the ROM files themselves could contain more information about the hardware than they currently do, but it's not strictly necessarily for the emulation.
About the PCB wiring, I have an idea for a "Custom wired NES cartridge" system, just for shits and giggles, where I basically will have a small netlist emulation. That should hopefully provide accurate emulation of the hardware no matter what wiring you use. I doubt it will be very useful, though.
Bregalad wrote:
Can hardly belive it. Looks like the other guy will win, this usless debate WILL reach 20 pages.
We have basically for the 10 last pages or so been running in circles, what more could possibly be said on the subject?
Who is "the other guy", by the way?
etabeta wrote:
you can go on fixing mappers over mappers, or simply use an xml layout for the data:
*snip*
no possible misunderstanding, and you just have to load the data at the prescribed offsets in your PRG and CHR memory regions and you are done...
Thank you. That's exactly what I wanted to be seeing.
Now if only the MESS team would quit storing all their Super A'can ROMs WRONG (because some genius decided that the result of dumping Big Endian data on a Little Endian system and thereby byteswapping all the data was somehow "more correct") I could say your emulator is awesome.
Bregalad: Sorry, I had to make sure the other guy would win so I added new timber to the fire. No hard feelings.
D wrote:
Now if only the MESS team would quit storing all their Super A'can ROMs WRONG (because some genius decided that the result of dumping Big Endian data on a Little Endian system and thereby byteswapping all the data was somehow "more correct") I could say your emulator is awesome.
but the data *is* stored in that way inside a cart. they have been dumped and verified. similarly, if you open a megadrive cart and you dump the content you get data in the opposite byte order than most of the currently available roms.
it's the CPU that reads them that way, so we cannot do anything about it.
if you need to visually check the content of the files, you can byteswap the files with dd or with an hex editor.
lidnariq wrote:
Zepper wrote:
- Funny. After 14 pages of discussion, the "new" format is being accepted now... instead of rejected. Good job, dude.
Hardly. He's just made it clear that he'll insult you if you disagree with him. What's the point in opening yourself to that kind of abuse?
- I'm out of this discussion. Period.
He's the other guy I was talking about :
kyuusaku wrote:
*Wonders if this will reach 20 pages*
Quote:
[1] e.g. .sfc is a very good system-specific container for SNES: it fails to document the case of carts with multiple chips (and therefore I still prefer our xml), but at least it can cover all the differences a console can detect between carts, due to the smaller number of hw difference inside snes carts
You see? You guys are just nuts about some of these things.
Yes, some SNES boards have more than one ROM chip. And there are two reasons for this. The first is if your game's size does not need to be a nice power of two. For instance, a 12mbit game is going to be 8mbit+4mbit, in most cases.
The second reason is cost. Just like today, the highest capacity memory is often more expensive than two memory sticks that are each half the capacity. Why? People like their free slots and reduced size. And what happens? Costs go down as production ramps up, and even newer, higher capacity chips are made. Eventually, it makes more sense to use bigger chips.
So what you find is that many SNES games will use two 8mbit ROM chips initially, and then the next product run, but with identical code, will use one 16mbit ROM chip. And in fact, in some instances you will find say a 12mbit game that uses a 16mbit ROM chip, and simply mirrors the last 4mbit twice to fill the chip!
Even with as pedantic and accuracy focused as I am, I can clearly see how ridiculously stupid and pointless it is to split SNES games into multiple files. The data was
meant to be linear, and that's how it maps onto the bus.
So what does MAME/MESS do with NSS ROMs? It gives them all
completely fucking random filenames, so that the only possible way to run an NSS game is to keep an internal database of each and every game. And all for nothing. Merge them together and they run just fine in any emulator. Way to go. Now the two people in the entire world who own NSS hardware and can and will repair it on failure, do not have to spend thirty seconds splitting a file. Meanwhile, the rest of the entire world has trouble playing NSS games in anything but MAME.
Quote:
but the data *is* stored in that way inside a cart.
You should store the data as a series of transistor patterns. You know, if you want to be
really accurate.
This decision results in poor compressibility of the images for long-term archival, it breaks graphics in sprite editing tools, it makes trying to edit the scripts for English, etc translations far more difficult, and it accomplishes nothing.
Quote:
- I'm out of this discussion. Period.
If only I had a dollar for every post in this thread saying it would be that person's last post, only that turned out not to be the case ... I'd have about $12 now. THE THINGS I WOULD BUY WITH $12 ...
I said it was my last post, but this new format thing you guys are trying to screw everyone over, with sucks and really pisses me off truthfully. If you want, I'll put together EVERY ROM FOR NES in the EXACT size with separate files with the EXACT binary on the carts if you want, since that is what you want. If you want to preserve ROMS, do it. Just don't try to replace the iNES file which is the way you emulate them! There's a fine line that apparently one other person can see besides me. iNES is for emulation. ZapFC or whatever you make can be for preservation, that's fine. Just don't try to do both with a new format if it isn't what it's intended to do.
That is all I will say now. Last post, although I may edit to add things. But right now, next year, and a decade from now, I will be using an iNES based file system with everything inside one file.
Wkter wrote:
etabeta wrote:
or since it fails to document properly corner cases of the pcb wirings (like the variants of mapper 3 used by Sansuu games, or the Konami mappers mixed up in mappers 21-22-26), forcing emu authors to rely on crc checks.
I agree, the mapper system used in iNES1.0 is quite weak, but iNES2.0 will be able to support for every single mapper variant out there, and Kevtris and Cowering will add iNES2.0 support in the GoodNES set, as he stated:
kevtris wrote:
I have been talking to Cowering about ines2.0 and getting it implemented in the GoodNES set. We're working on getting this done; I am waiting now for the new current set so I can go through and fix the headers that need it.
While I agree that assigning mapper numbers to the boards might not be the ultimate solution for preservation of the hardware information, I find it to work pretty well in terms of emulation.
Actually it was a bit better than assigning a new mapper number per se. It assigns a "sub mapper". Untangling the mess of say, MMC1 turns the special cases of mapper 1 into separate sub mappers. The games are still mapper 1, but now there's a submapper to tell them apart. This was so the ROMs would still be mapper 1 in ines1.0 to older emulators (and there was no reason to assign new numbers when they were all technically MMC1). I denoted 'em kinda like revisions. mapper 1.0 being barefoot stock MMC1, 1.1 being 16K of WRAM, etc. This turned out to be extremely easy to support in the end, I only had to add a few dozen lines of 6502 asm to support them on my FPGA.
Be that as it may, I did add 4 more bits of mapper number just in case it was ever needed. (The sub-mapper number is 4 bits also, so that gives a total of 16 bits worth of mapper state)
Just using the submapper part allowed me to totally eliminate every single checking hack I had to perform in my code. Yes, every single check. This allowed me to drop the CRC code (I was actually CRCing in hardware as the games were loaded), size checking code, and every other form of tests/hacks I had to do during loading to properly select the mapper.
kevtris wrote:
It assigns a "sub mapper". [...] mapper 1.0 being barefoot stock MMC1, 1.1 being 16K of WRAM, etc.
But even that isn't necessary with a WRAM size field, which both the NES 2.0 header and ZapFC FBSV files have. To me, the mapper variant field would be better used for different variants of MMC3 IRQ counter behavior, presence or absence of MMC1 WRAM protection, SL vs. CL configuration on MMC5, the different ways the VRC can be wired, and anything else that can't be reliably determined from the sizes of PRG ROM, CHR ROM, and various RAMs.
byuu wrote:
Even with as pedantic and accuracy focused as I am, I can clearly see how ridiculously stupid and pointless it is to split SNES games into multiple files. The data was meant to be linear, and that's how it maps onto the bus.
and we load it linearly, indeed. but since we already inherited by MAME the code to load separate files from a zip archive, why shouldn't we support split files too in addition to plain .sfc?
byuu wrote:
So what does MAME/MESS do with NSS ROMs? It gives them all completely fucking random filenames, so that the only possible way to run an NSS game is to keep an internal database of each and every game. And all for nothing.
they're not random, they are (or should be, once confirmed) the names present on the chip label, followed by the pcb position as extension. and it's not for nothing: it allows the 0.001% of the userbase to repair original carts and we are happy to make this possible for that tiny niche
byuu wrote:
Merge them together and they run just fine in any emulator. Way to go. Now the two people in the entire world who own NSS hardware and can and will repair it on failure, do not have to spend thirty seconds splitting a file.
except they would not know which part goes where. or how to split files to match the chips. or (for different hw than nss) if a chip contains the lo nibble and the other chip contains the hi nibble, etc.
byuu wrote:
Meanwhile, the rest of the entire world has trouble playing NSS games in anything but MAME.
other arcade emulator have no problem supporting separate files when they emulate NeoGeo or CPS1/CPS2 games. why only console emulators find it that difficult?
byuu wrote:
This decision results in poor compressibility of the images for long-term archival, it breaks graphics in sprite editing tools, it makes trying to edit the scripts for English, etc translations far more difficult, and it accomplishes nothing.
the sprite editing tool should simply be updated to handle byteswapped images. or you can swap the image ("dd conv=swab if=input of=output" in *nix OSes), edit it and then re-swap it...
why should we store the data differently than in the original media just to make happy a (broken) sprite editor?
try burning any nointro megadrive image on a real eeprom and plug it in the real console. result? it does not work. why? wrong byte order... is that pointless? up to you to decide it, but we care about it and we followed that path!
kevtris wrote:
Just using the submapper part allowed me to totally eliminate every single checking hack I had to perform in my code. Yes, every single check. This allowed me to drop the CRC code (I was actually CRCing in hardware as the games were loaded), size checking code, and every other form of tests/hacks I had to do during loading to properly select the mapper.
yup. I'm following iNES2.0 since the beginning and I also have added barebone support for the additional header fields in MESS. would you mind to post a list of the submappers at some point? I'd like to support the variants properly.
> why shouldn't we support split files too in addition to plain .sfc?
Because it makes no sense, and having multiple ways of doing the same exact thing just makes it harder for everyone.
> they're not random, they are (or should be, once confirmed) the names present on the chip label, followed by the pcb position as extension
As I said, random. An emulator cannot predict where chips appeared on a PCB and how to put them together based on the filenames.
Actraiser:
Code:
spc700.rom / 64 bytes / 44bb3a40 / nss / good
dsp1data.bin / 2 KB / 4b02d66d / nss / good
nss-c.dat / 32 KB / a8e202b3 / nss / good
nss-ic14.02 / 32 KB / e06cb58f / nss / good
m50458_char.bin / 4.5 KB / 011cc342 / nss / baddump
m50458_char_mod.bin / 4 KB / 8c4326ef / nss / baddump
act-rais.ic3 / 512 KB / c9f788c2 / nss_actr / good
act-rais.ic2 / 512 KB / 4df9cc63 / nss_actr / good
act-rais.ic8 / 32 KB / 08b38ce6 / nss_actr / good
Super Mario World:
Code:
spc700.rom / 64 bytes / 44bb3a40 / nss / good
dsp1data.bin / 2 KB / 4b02d66d / nss / good
nss-c.dat / 32 KB / a8e202b3 / nss / good
nss-ic14.02 / 32 KB / e06cb58f / nss / good
m50458_char.bin / 4.5 KB / 011cc342 / nss / baddump
m50458_char_mod.bin / 4 KB / 8c4326ef / nss / baddump
nss-mw-0_prg.ic1 / 512 KB / c46766f2 / nss_smw / good
mw.ic3 / 32 KB / f2c5466e / nss_smw / good
Tell me how the fuck I am supposed to load these two games without an internal database telling me what each file is and where it goes?
> and it's not for nothing: it allows the 0.001% of the userbase to repair original carts and we are happy to make this possible for that tiny niche
What about the 99.999% of the userbase that wants their NSS games to work on other emulators? Why aren't you happy to make that possible for that super majority?
> except they would not know which part goes where. or how to split files to match the chips.
Oh come on. Someone who can desolder, flash, and solder new chips onto boards can figure out how to split up files. Stick the info in an XML file on a website and be done with it.
> other arcade emulator have no problem supporting separate files when they emulate NeoGeo or CPS1/CPS2 games. why only console emulators find it that difficult?
It makes sense sometimes, possibly with NES PRG/CHR files for instance (I wouldn't know, I haven't worked with the NES.) It does not make sense for the SNES.
> the sprite editing tool should simply be updated to handle byteswapped images.
I give up.
byuu wrote:
Tell me how the fuck I am supposed to load these two games without an internal database telling me what each file is and where it goes?
pushing a bit more the xml approach you already use for memory maps? just have some xml node prescribing where to load the files and let user add the xml in the same folder as the roms (like you do for mmio related xmls)
tepples wrote:
kevtris wrote:
It assigns a "sub mapper". [...] mapper 1.0 being barefoot stock MMC1, 1.1 being 16K of WRAM, etc.
But even that isn't necessary with a WRAM size field, which both the NES 2.0 header and ZapFC FBSV files have. To me, the mapper variant field would be better used for different variants of MMC3 IRQ counter behavior, presence or absence of MMC1 WRAM protection, SL vs. CL configuration on MMC5, the different ways the VRC can be wired, and anything else that can't be reliably determined from the sizes of PRG ROM, CHR ROM, and various RAMs.
Some cartridges could batteryback the RAM while others may not. An example of this is on MMC5. Carts exist with 16K of WRAM that's half battery backed and half that isn't, while other carts contain 8K or 32K of WRAM which is battery backed. A size field won't tell this apart very easily unless you assume 16K WRAM games always back half only.
I actually do have a version of mapper 1 for the missing WRAM protect bit.
As for the MMC3, is there a documented case where you must use one MMC3 revision over the other? I've been curious about that but haven't been able to definitively find a case that needs it yet.
And on the MMC5, what does SL and CL mode do? Again, I have not been able to tell any difference between the two. I suspect it has something to do with split screen mode, but since only 1 game uses it (and it uses it in the intro) I am not sure it matters for emulation sake. (the game btw is SDF). I actually changed the mode on my CV3 cart a long time ago, and played it but nothing happened. It just played normally.
For emulating VRC2 and VRC4, you can use bit masks to cover all the cases, which is what I do. This way it does not require any special mapper handling. It just works for all cases instead. (Technically though, you COULD use submappers for this but I am not sure if it's required since the current solution results in all the games being playable without any hacks per se).
The notable cases I used submappers were for the BNROM mapper; since someone jammed two games into one mapper (BNROM - Deadly Towers and Impossible Mission 2). ALso, the Subor mapper was two separate mappers jammed into one.
I ended up submappering all the MMC3 variations for completeness sake (The CHRAM+CHROM ones, and the ones with fixed mirroring like Karnov and Holy Diver) but I didn't change the headers because the current solution still works.
Once I get my hands on the latest set, I will go through and 2.0 fix all the required headers and release a complete specification document along with a header pack or similar (and it should be in an upcoming goodNES too).
kevtris wrote:
I ended up submappering all the MMC3 variations for completeness sake (The CHRAM+CHROM ones, and the ones with fixed mirroring like Karnov and Holy Diver)
I don't think Holy Diver should have it's own MMC3 variant since it's a MMC3-less discrete mapper.
kyuusaku wrote:
kevtris wrote:
I ended up submappering all the MMC3 variations for completeness sake (The CHRAM+CHROM ones, and the ones with fixed mirroring like Karnov and Holy Diver)
I don't think Holy Diver should have it's own MMC3 variant since it's a MMC3-less discrete mapper.
Hmm, I must've gotten that confused; I don't have my list here at the moment. It's that game that uses an MMC3 but used the address lines(!) routed to the MMC3's data lines. I assume for copy protection.
I think it's actually "Time Diver Avenger" (mapper 250 or so?). If that's not it, I will check when I get home.
kevtris wrote:
I think it's actually "Time Diver Avenger" (mapper 250 or so?). If that's not it, I will check when I get home.
yeah, it's time diver avenger, not holy diver.
otoh, I'm not sure how would you handle exactly vrc2/vrc4. does iNES2.0 have anything different for these compared to iNES?
EDIT: btw FitzRoy, does your database take into account the different pin configurations of Mapper 185 games? (see
http://nesdev.com/bbs/viewtopic.php?t=2797 and
http://nesdev.com/bbs/viewtopic.php?t=6481 )
Those games cannot be handled as generic 'mapper 185' since they need specific protection values to be read at start... in MESS we added the pin settings as xml attributes
Code:
<feature name="chr-pin26" value="CE" />
<feature name="chr-pin27" value="/CE" />
but I'm not sure if bootgod includes this info in the db profile of the website (it should be in his xml, though, iirc)
etabeta wrote:
kevtris wrote:
I think it's actually "Time Diver Avenger" (mapper 250 or so?). If that's not it, I will check when I get home.
yeah, it's time diver avenger, not holy diver.
otoh, I'm not sure how would you handle exactly vrc2/vrc4. does iNES2.0 have anything different for these compared to iNES?
I don't handle them different from ines 1.0, 'cause it's possible to emulate them all without knowing which game you're running.
Quote:
EDIT: btw FitzRoy, does your database take into account the different pin configurations of Mapper 185 games? (see
http://nesdev.com/bbs/viewtopic.php?t=2797 and
http://nesdev.com/bbs/viewtopic.php?t=6481 )
I handled that by assigning submapper numbers from what I recall.
Related to ZapFC proposal: have you looked at the system that MESS uses for checking NES roms? From what I gather, it is a checksum database which ignores the first 16 bytes of each rom entirely and checksums (CRC32+SHA1) the rest, and then looks up in its own database what the mapper/savesize/mirroring/etc should be. Roms not in the database have the Ines header parsed as usual.
LN
using ZapFC with QuickNES right now. I like the organization of the database, thanks for putting this together Fitz.
Apologies if I've overlooked something, but how is this format proposal better than Nestopia's "NES XML" format? As long as you supply the headerless/split ROM with an XML file that has the cartridge information, you should have the functionality that you want*.
*Unfortunately, I haven't been able to test this myself yet, not being very experienced with XML schemas. Refer to "schemaromset.xsd" in the Nestopia distribution.
etabeta wrote:
they're not random, they are (or should be, once confirmed) the names present on the chip label, followed by the pcb position as extension. and it's not for nothing: it allows the 0.001% of the userbase to repair original carts and we are happy to make this possible for that tiny niche
Although this may not have been true for the NSS, many SNES games that initially shipped between several PRG roms were later manufactured on one. Same data. So that should tell you that the PRG data was whole to begin with. To avoid the distribution of arbitrary variants, that's how it should probably be distributed. There were also a few NES games that did this with the CHR data.
etabeta wrote:
except they would not know which part goes where. or how to split files to match the chips. or (for different hw than nss) if a chip contains the lo nibble and the other chip contains the hi nibble, etc.
And then their hardware will eventually die anyway, so why does emulation need to concern itself with how many ways the data could have been split depending on rom costs?
zamenus wrote:
Apologies if I've overlooked something, but how is this format proposal better than Nestopia's "NES XML" format? As long as you supply the headerless/split ROM with an XML file that has the cartridge information, you should have the functionality that you want*.
I did it to reduce the filesize by 2/3 and use a stronger hash.
kevtris wrote:
Some cartridges could batteryback the RAM while others may not. An example of this is on MMC5. Carts exist with 16K of WRAM that's half battery backed and half that isn't, while other carts contain 8K or 32K of WRAM which is battery backed. A size field won't tell this apart very easily unless you assume 16K WRAM games always back half only.
Yes, and ZapFC lists the WRAM field as an ampersand separated list of chip sizes, with an asterisk after each chip which is battery backed. A notion I have not fully supported in QuickNES yet because I haven't researched how different mappers support WRAM greater than 8KB. It currently searches the whole string for an asterisk and sets the battery backed flag for the whole 8KB it does support. Well, unless any of the mappers I didn't explicitly write myself happen to support more than 8KB of WRAM already. (Since I think QuickNES mostly handles the WRAM block outside of the mapper implementation, and the mappers only set flags which control whether the WRAM is readable and/or writeable.)
FitzRoy wrote:
Although this may not have been true for the NSS, many SNES games that initially shipped between several PRG roms were later manufactured on one. Same data. So that should tell you that the PRG data was whole to begin with. To avoid the distribution of arbitrary variants, that's how it should probably be distributed.
and why does it seem to hurt everyone supporting two variants of a game, even if the content is the same? oh, maybe the waste of HD space... well, I'll go deleting a single one of the tons of .avi I have to make space for all the existing variants of NES and SNES carts...
FitzRoy wrote:
There were also a few NES games that did this with the CHR data.
yes. e.g. Excitebike and Pinball came in two variants. they both are supported by MESS. once you use xml and split prg/chr, it costs 0 lines of codes to support different ways to split prg or chr, as long as the offset you have to load files to is present in the xml.
FitzRoy wrote:
And then their hardware will eventually die anyway, so why does emulation need to concern itself with how many ways the data could have been split depending on rom costs?
and who are you to decide they should not be able to repair their hardware when they still can?
anyway, I would still like to know if ZapFC can handle mapper 185 variants and how...
etabeta wrote:
and who are you to decide they should not be able to repair their hardware when they still can?
All these people really need is documentation. It doesn't require thousands of people storing redundant data to give a few people the knowledge to repair doomed hardware.
Quote:
anyway, I would still like to know if ZapFC can handle mapper 185 variants and how...
It might be a few weeks before I flesh that out.
FitzRoy wrote:
Quote:
anyway, I would still like to know if ZapFC can handle mapper 185 variants and how...
It might be a few weeks before I flesh that out.
ah ok. good luck.
it is a bit tricky: basically the board is always the same (and it's basically a mapper 3 pcb) but there are two pins that can be set to open or close in the factory. the game code at start goes to read the correspondent addresses and if it finds the wrong value, it does not start properly, because it thinks you are pirating the cart or flashing the game on a different pcb.
anyway, bootgod sorted it out (as described in the posts I linked) and we verified the settings for all the games using that pcb type
the same solution you might find in your format for this could also be of use to handle VRC2-VRC4-VRC6 variants which are currently spread across different mappers and require ad hoc address masks to work in emulator.
bootgod found out that the difference between the models are related to different wiring of specific pcb lines, so that different wirings result in different bits being responsible of the prg/chr bankswitch & mirroring mechanism.
hence, you can decide to support 6 mappers with address masks, or you can decide to emulate these as three pcb types only, by setting the proper address lines at start on a game-dependent base.
both NEStopia and MESS follow the latter approach, instead of supporting separately mappers 21, 22, 23, 24, 25, 26.
the only problem is that different games use different lines, so that it can be tricky to cover combinations all in your format (VRC2 has 3 pin connections to be checked, while VRC4 and VRC6 have only two, but they could in principle be connected to any of the 8 prg + 8chr address lines, even if only a few combinations have been found so far).
let me know if you need any test cases for these or for mapper 185 games
[1] actually, if we would like to be uber-picky, mapper 3 and 185 could be merged (given that they are the same except for these pins), and pins should be set to open/open or close/close (I don't remember which one of the two) for mapper 3 games which do not check those addresses... however, not a priority.
FitzRoy wrote:
zamenus wrote:
Apologies if I've overlooked something, but how is this format proposal better than Nestopia's "NES XML" format? As long as you supply the headerless/split ROM with an XML file that has the cartridge information, you should have the functionality that you want*.
I did it to reduce the filesize by 2/3 and use a stronger hash.
Reduce the filesize, why does that matter? Is that also an unspoken reason you want to use PKZIP format? Even though seemingly everyone here who works with the actual NES
hardware is opposed to it? It's not 1997, when I used to store all my ROMs in ZIP files with a front-end to unpack them. Now I've got a 1.5TB hard drive. Why in the world would I, or anyone else in the future, care about filesize for something this small?
Memblers wrote:
Is that also an unspoken reason you want to use PKZIP format?
are you suggesting to keep the files unzipped or to use different compression formats?
FWIW, MAME/MESS treats zips like folders: you can have prg/chr either in gamename.zip or uncompressed in a gamename/ folder. zip is only used to rapidly verify checksums, so that you can store files inside the zip even with the wrong filename and they will be loaded (if you keep file unzipped, their name must match MAME/MESS internal filenames)
that's why we have never cared about .ace, .rar or .7z. zip already gives all we need
etabeta wrote:
FitzRoy wrote:
And then their hardware will eventually die anyway, so why does emulation need to concern itself with how many ways the data could have been split depending on rom costs?
and who are you to decide they should not be able to repair their hardware when they still can?
This is one of the bigger lols to me when it comes to storing data as a single block or split up into chip size chunks. The reason I find it so funny is that most licensed NES games were stored on ROMs that are not pin compatible with EPROMs or any other programmable logic device. Arcade boards on the other hand, often DO have EPROMs or JEDEC pinout ROMs.
Nintendo was fond of using non-JEDEC pinout ROMs for everything NES, so it's impossible to fix broken cartridges without mucho rewiring. In which case, the person doing the rewiring is going to be savvy enough to be able to figure out how to cut out the chunk of data needed.
Not to mention that there's just no way to easily replace those 128K byte 28 pin ROMs that were very popular for awhile. If one of those is bad you might as well just buy another cartridge unless you like to do lots of rewiring.
(As an aside, I noticed that Nintendo wasn't so averse to using JEDEC pinout stuff on the Famicom cartridges though. My cart of Just Breed has a JEDEC pinout 512K ROM on it which I socketed. I am not sure about their earlier releases though, if they were JEDEC or not.)
I've always had an idle curiosity over why Nintendo decided to use two different kinds of ROMs for the PRG and CHR on cartridges that don't have compatible pinouts. Either it was speed issues, or just stubbornness, or it was to try and slow down piracy.
we were talking about SNES, which I thought it uses pretty standard chips. However, I did not know about the pin incompatibility: were they incompatible in PC10 and VS arcade pcbs too? it does not sound a really smart move, unless their repairing service was super-efficient with arcades...
Keep the files unzipped so that they are easier to manipulate with low-level C or ASM code. Flash carts that run on the NES use the NES's CPU to drive all the logic. Unzipping a file in 6502 asm w/ 10K ram is kind a pain in the ass.
Reading a file format like iNES or UNIF is fairly easy in C and ASM. Any compression container adds a lot of overhead for such small systems and even PC based utilities. I know that others have voiced opposition to TAR, but I find TAR really easy to code to. The "unix baggage" only matters if one is using TAR to archive file attributes for future recovery operations.
clueless wrote:
Keep the files unzipped so that they are easier to manipulate with low-level C or ASM code. Flash carts that run on the NES use the NES's CPU to drive all the logic.
If they're based on NAND+RAM like a PowerPak (NES) or SuperCard (GBA). NOR flash-based cards like Blargg's Munchausen cart (NES) or FireLinker (GBA) do the logic in PC-side flash-programming software.
Quote:
Unzipping a file in 6502 asm w/ 10K ram is kind a pain in the ass.
The PowerPak has 512 KiB of PRG RAM dedicated to emulating PRG ROM. Why can't the unzipper use that?
Quote:
I know that others have voiced opposition to TAR, but I find TAR really easy to code to.
I looked at tar early in my GBA era (2001), but I discovered that searching for a file by name involves a linear search through the whole file. At the time, I wanted something where I could binary search through only the directory, and I came up with a simple uncompressed archive format that I named
GBFS. Now GBFS tools are included with every copy of devkitARM.
tepples wrote:
clueless wrote:
Unzipping a file in 6502 asm w/ 10K ram is kind a pain in the ass.
The PowerPak has 512 KiB of PRG RAM dedicated to emulating PRG ROM. Why can't the unzipper use that?
I did not consider that possibility. But what happens when you _WANT_ to decompress a PROG-ROM that big?
tepples wrote:
Quote:
I know that others have voiced opposition to TAR, but I find TAR really easy to code to.
I looked at tar early in my GBA era (2001), but I discovered that searching for a file by name involves a linear search through the whole file. At the time, I wanted something where I could binary search through only the directory, and I came up with a simple uncompressed archive format that I named
GBFS. Now GBFS tools are included with every copy of devkitARM.
But in this case, the "container" only holds at most three objects: prog-rom, char-rom and the "board" file.
A few years ago I wrote a game that stored all of its graphics assets as PNGs inside a single TAR file. My game would mmap() (or the win32 equiv on the windows build) the TAR file read-only, scan through it once making an index (in memory). The code would then directly used the mapped bytes, instead of reading them into a buffer. on huge files, memory mapped IO is more efficient than normal file IO, especially on a Windows NT based kernel (which implements file reading as memory maps under the hood anyway). Since then, I've regarded TAR as a viable container format, depending on the application. TAR headers are fairly easy to parse. The use of octal digit encoding is really odd, but hey, tar was invented in the 70s. They smoked a lot of odd shit back then...
Quote:
HISTORY
A tar command appeared in Seventh Edition Unix, which was released in January, 1979. It replaced the tp program from Fourth Edition Unix which in turn replaced the tap program from First Edition Unix. John Gilmore's pdtar public-domain implementation (circa 1987) was highly influential and formed the basis of GNU tar. Joerg Shilling's star archiver is another open-source (GPL) archiver (originally developed circa 1985) which features complete support for pax interchange format.
from
http://www.freebsd.org/cgi/man.cgi?quer ... +8-current
etabeta wrote:
are you suggesting to keep the files unzipped or to use different compression formats?
Unzipped, absolutely. It seems fine to me that an emulator would load a ROM set from a zip file (in fact, many of them already do for individual zipped iNES files). But it seems totally counter-productive to make the compression enshrined in the file format, because then the NES can't even use it's own ROM format.. Yeah the NES is a "console", but it was designed as computer originally, even the name Famicom stands for 'Family Computer'.
So the compression pretty much seems to make some people's life harder, while lots of users would see no change (and wouldn't care one bit what format it's in). I know if I release a game, I sure wouldn't use this format if I have to tell the users "keep it zipped if you use this device, unzip it if you use this device, sorry you can't rezip it on your own because the algorithm isn't supported on this device, but it is on that one", etc. Much better to just make it an uncompressed format, then just let the emulators load from ZIP files if they want (like they already do).
BTW, PC10 carts, I have 10 of them and most of them do have EPROMs. Some of them have MaskROMs though, but it also seems that some of the boards conveniently have solder jumpers you can change to select either type of chip. Of course that's different from NES carts, where you'd have to cut traces and rewire. I'd bet VS games were all EPROM, because of the code modification needed for coin-op. But I'm not sure. The VS mainboard takes EPROMs, but some games were released on daughterboards, so anything is possible there.
Memblers wrote:
Unzipped, absolutely. It seems fine to me that an emulator would load a ROM set from a zip file (in fact, many of them already do for individual zipped iNES files). But it seems totally counter-productive to make the compression enshrined in the file format, because then the NES can't even use it's own ROM format..
agreed. it's the emulator which should take care of compression/decompression if needed. I had missed this particular aspect of the format (14 pages read in a morning were a bit too much, sorry)
Memblers wrote:
BTW, PC10 carts...
very interesting info, thanks. that's probably where I got the (wrong) idea that also PRG/CHR were using standard ROM chips. now I have a clearer picture.
Decompressing the set to ".fc" folders treated as files by the flash cart is probably better suited for stuff like flash carts. The format only enforces ZIP as the default because folders can't be download links and flash carts are less than 1% of users.
yeah, as I understood it, it's more about using it as a container and a download link than to save space right?
As stated, the Famicom Rom Set is not that big by today's standards.
FitzRoy wrote:
Decompressing the set to ".fc" folders treated as files by the flash cart is probably better suited for stuff like flash carts. The format only enforces ZIP as the default because folders can't be download links and flash carts are less than 1% of users.
but this in my opinion bloats a bit the whole idea: why do you need .fc folders and zip container for download? why not plain zip and extension-less folders?
when you already have a database of hashes to match the files with, what is the advantage of a prescribed extension like .fc? you can open all directories and simply refuse to load files which do not match your database...
or am I missing anything?
etabeta wrote:
but this in my opinion bloats a bit the whole idea: why do you need .fc folders and zip container for download? why not plain zip and extension-less folders?
How does differentiating a game folder from a regular folder bloat the idea? Double clicking a folder in a load window typically assumes navigation. With an extension, it can know you mean to run it without having to do something less efficient like interpret its contents.
Similar logic for archives. How would an emulator know it's meant to be handled differently? How would a user know the difference if the filename was identical to a compressed iNES file? In fact, any game file not using an extension prefix is what prevents emulators from being able to filter out non-game zip files from the load directory.
FitzRoy wrote:
How does differentiating a game folder from a regular folder bloat the idea? Double clicking a folder in a load window typically assumes navigation. With an extension, it can know you mean to run it without having to do something less efficient like interpret its contents.
I meant that you include the compression in the format for something that could be handled by simple .zip without including compression in the format
FitzRoy wrote:
Similar logic for archives. How would an emulator know it's meant to be handled differently? How would a user know the difference if the filename was identical to a compressed iNES file?
by checking the extension: if .nes or .unf it's iNES or UNIF, if anything else is a file requiring a check in the internal database (be it xml or zapfc)
FitzRoy wrote:
In fact, any game file not using an extension prefix is what prevents emulators from being able to filter out non-game zip files from the load directory.
that's what rom manager are for, imho. so that you store gamefile in the way you want and you don't store e.g. a Megadrive game or other zipfile inside your NES folder.
anyway, if other devs find your approach useful, I'm fully supportive!
etabeta wrote:
I meant that you include the compression in the format for something that could be handled by simple .zip without including compression in the format
Every emulator that supports ZIP supports ZIP decompression, so I fail to see how mandating no compression would do anything but confuse people. "Hi, I archived my own file and won't work. Help!"
Quote:
by checking the extension: if .nes or .unf it's iNES or UNIF, if anything else is a file requiring a check in the internal database (be it xml or zapfc)
But that's looking at the contents. One the outside, they look the same.
Quote:
that's what rom manager are for, imho. so that you store gamefile in the way you want and you don't store e.g. a Megadrive game or other zipfile inside your NES folder.
To an extent, I agree. People need to sort their own shit in folders so that the burden isn't on the emulator to filter for you. But let's say you created a multi-patch container to go with multi-rom container. Can't have identical filenames in the same folder.
etabeta wrote:
you don't store e.g. a Megadrive game or other zipfile inside your NES folder.
Unless you're storing, say, all versions of Klax together.
Another case: I don't have an "NES folder" on one of my machines. I have /home/pino/develop, which currently contains SDL projects and NES projects but could expand to Genesis projects and Android projects. Even for someone who plays only licensed NES games, where in the file system would this NES folder be located so that the emulator can find it?
FitzRoy wrote:
But that's looking at the contents. One the outside, they look the same.
got it. I'm not sure using .fc is the best possible solution, but it is indeed a solution to this specific problem.
tepples wrote:
Unless you're storing, say, all versions of Klax together.
yeah, we have added this in MESS (even if it is not the suggested storage method) and it's one of the nice side effects of using an internal database because you can match the crc to see if it is the version for that console, without relying on extensions or anything else... however, there seems to be only a (very small) minority of users interested in such a kind of setup, since most users seem to want to have roms separated by systems.
tepples wrote:
Even for someone who plays only licensed NES games, where in the file system would this NES folder be located so that the emulator can find it?
most emulators let you choose a path to roms, don't them?
YEAH THAT's 20 PAGES !!!!!!!!
HURRAY AND LONG LIVE TO RIDICULOUS USELESS DEBATES ! (that even my tolling can't stop )
Bregalad wrote:
YEAH THAT's 20 PAGES !!!!!!!!
HURRAY AND LONG LIVE TO RIDICULOUS USELESS DEBATES ! (that even my tolling can't stop ) :(
If not for your(s and others') useless posts, we may not have reached 20 pages at all. Thanks for making the dream a reality! :D
Not as useless as the updates on this new thing only the whopping 3 people in the thread will use!
there is a site that I go to that already has 26 hits on ZapFC.
that's actually more people than are pages here
That'll displace the iNES standard! Hell yeah!
I don't think anyone is trying to displace ines so much as provide an alternative to those who want it. It seems apparent that there is some interest or there wouldn't be this much discussion to begin with.
byuu wrote:
Bregalad wrote:
YEAH THAT's 20 PAGES !!!!!!!!
HURRAY AND LONG LIVE TO RIDICULOUS USELESS DEBATES ! (that even my tolling can't stop )
If not for your(s and others') useless posts, we may not have reached 20 pages at all. Thanks for making the dream a reality!
Yeah, perhaps 19 only... but never 20! <>_<> /trollmode
byuu wrote:
Bregalad wrote:
YEAH THAT's 20 PAGES !!!!!!!!
HURRAY AND LONG LIVE TO RIDICULOUS USELESS DEBATES ! (that even my tolling can't stop )
If not for your(s and others') useless posts, we may not have reached 20 pages at all. Thanks for making the dream a reality!
Yay!
Useless formats ftw, coupled with ..........................
I find it amazing no matter how hard I try to register on your forum to do a valid concern, I'm banned.
Just how is that meant to work? Oh yeah, this is how!
"Its not a democracy, its a cheerocracy!
"
Yeah, that makes sense. Everyone 1% as popular as Lagy Gaga should give up music, they suck. Every atheist should find religion immediately. Why fight the crowds?
People trying to change standards are icky. Sorry, can't stop trolling. Hopefully later people with an idea for a new standard will get the message, unlike this thread which is still going.
Let's make more stuff that makes ROM's not useable on real hardware from the virgin files! Despite that being basically the whole idea....
Because the crowds are blind, vapid, insane, shallow, decietful and selfish pricks and you know it
Anyway, nice format. Why no compression like LZMA?
panzeroceania wrote:
I don't think anyone is trying to displace ines so much as provide an alternative to those who want it. It seems apparent that there is some interest or there wouldn't be this much discussion to begin with.
Many people are interested in an alternative. There is this much discussion because this idea is not what those people want.
3gengames wrote:
Let's make more stuff that makes ROM's not useable on real hardware from the virgin files! Despite that being basically the whole idea....
how hard do you think it would be to add support to a flashcart?
ibeenew2 wrote:
Many people are interested in an alternative. There is this much discussion because this idea is not what those people want.
what do people want then? I'm honestly curious. Do I find this the perfect format? no. Do I think it's better than every other NES alternative? Yes.
Most of what I have heard is support for a legacy format, what other alternatives do you speak of? (not ines 2.0)
iNES 2.0, Because nothing else is needed and nothing else will help because nothing else is iNES/iNES 2.0.
3gengames wrote:
iNES 2.0, Because nothing else is needed and nothing else will help because nothing else is iNES/iNES 2.0.
- Agreed.
3gengames wrote:
People trying to change standards are icky.
I never hoped to. But personally, I think we should have stopped at NESticle when that was dominant.
Emulators != ROM formats. ROM formats are standard while Emulators are not.
panzeroceania wrote:
what do people want then? I'm honestly curious. Do I find this the perfect format? no. Do I think it's better than every other NES alternative? Yes.
Something that supports all games, not just an arbitrary subset. Something that is easy for homebrewers to use. Something that is easy to parse for simpler devices. Something that actually preserves information like bootgod's database. Something where you can add extra cart info like label scans or manual text.
UNIF is many of those, ZapFC is none. For all the things that ZapFC fails (like homebrews) the solution is to use iNES. For all the things ZapFC succeeds iNES already does the same job.
ibeenew2 wrote:
Something that supports all games, not just an arbitrary subset. (1) Something that is easy for homebrewers to use.(2) Something that is easy to parse for simpler devices.(3) Something that actually preserves information like bootgod's database.(4) Something where you can add extra cart info like label scans or manual text.(5)
if you use an xml database (like NEStopia's, or the one I created for MESS), you get all of these except (2) and (3).
if you use an xml database + you support a xml file included with the ROMs (and NEStopia already supports this latter addition), you get (2) as well, because creating an xml for an homebrew program is pretty easy (you specify the board, the prg and the chr and you are done)
the hardest part is to handle parsing in a simple system like GBA, but still I haven't seen any other proposal capable of getting 4 out of 5 of your requirements.
3gengames wrote:
People trying to change standards are icky.
I'd agree if the standard was capable to represent all games, which iNES is not... at least iNES2.0 can improve this.
however, it is still limited by the fact that new chinese dumps can use new variants to e.g. MMC3 and you would have to manually add them.
but this might hold true for the xml database as well, depending on the kind of variant.
anyway, I really fail to see why you cannot accept that other people might have different opinions from yours.
do you think iNES is wonderful and that it's the answer to all your needs? ok. fine. just stick to it. I won't lose my sleep.
OTOH, it's a few months that I keep adding split prg/chr to the MESS xml database whenever new chinese/russian dumps appear, or when a new proto is dumped, and I find it very easy and intuitive (much more than verifying if the iNES header chosen by the dumper is correct, when e.g. a new waixing dump appears and the graphics is completely messed up because it had been assigned mapper 4 even if it was not...)
I have another question about this format. Sorry again if it has been addressed already; this thread has gotten pretty large. I did look over the readme, though.
What if someone wants to run a non-split, headerless ROM (i.e. an iNES ROM without the header)? The file could be given the extension .fc, and should work in the same way that the split ROMs work. I tried loading such a file in QuickNES but that extension wasn't supported by the open dialog.
EDIT: I guess this is to prevent ROM sites from bundling "downloadedfrom.txt" with the ROMs. Still, some people may prefer to load their ROMs this way.
xamenus wrote:
What if someone wants to run a non-split, headerless ROM (i.e. an iNES ROM without the header)? The file could be given the extension .fc, and should work in the same way that the split ROMs work. I tried loading such a file in QuickNES but that extension wasn't supported by the open dialog.
you would need to store the checksum of prg+chr too, in addition to those of the split file. I don't think the new format is supposed to work like this. for completeness sake, not even NEStopia or MESS xml support this (even if e.g. bootgod's xml does contain both checksums type...)
3gengames wrote:
Emulators != ROM formats. ROM formats are standard while Emulators are not.
Dude, you are so unbelievably dumb. Rom formats are influenced or even created by emulators. How do you think so many games got mapper hacked? Should we have kept those games in that format because they were mega popular and often played just as well as the original mapper?
ibeenew2 wrote:
Something that supports all games, not just an arbitrary subset. Something that is easy for homebrewers to use. Something that is easy to parse for simpler devices.
You can put a board file in the container. Stop spouting fud to support your indefensible position of using a system-specific container.
Quote:
Something that actually preserves information like bootgod's database. Something where you can add extra cart info like label scans or manual text.
And screw color, and pcb scans, and 3d renders of the plastic molding, and game guides, and genres, and game credits, and, and, and... Most of that stuff is so flippant, complex, or subjective that you will never come to an agreement on how to preserve it or what to exclude. And it's a major distraction from the larger problem which is that no one can download a game and know it has the correct mapping.
mudlord wrote:
Anyway, nice format. Why no compression like LZMA?
I'm pretty sure the guide linked in the first post mentions 7z is allowable.
Cool, specs look reasonable.
Any chance for ZapGBX?
panzeroceania wrote:
3gengames wrote:
Let's make more stuff that makes ROM's not useable on real hardware from the virgin files! Despite that being basically the whole idea....
how hard do you think it would be to add support to a flashcart?
About as hard as porting Info-ZIP UnZip to run on a Commodore 64. To see why, see previous pages.
etabeta wrote:
the hardest part is to handle parsing in a simple system like GBA
That or in an even simpler system like an NES with a CF card attached.
FitzRoy wrote:
How do you think so many games got mapper hacked?
Some games were mapper hacked even in the NES era, such as Contra (hacked to UNROM) and Castlevania (hacked to MMC5). This is because hacking these games to a Nintendo mapper was less expensive than getting a custom mapper approved by NOA and NOE. Other games were mapper hacked because they were originally released on a mapper that was too damn easy to pirate: Metroid, Zelda 1, Kid Icarus, and the game that became SMB2 (U).
@mudlord: Game Boy has an internal header, much like Super NES. So is it in need of extra board information?
tepples wrote:
@mudlord: Game Boy has an internal header, much like Super NES. So is it in need of extra board information?
a couple of mappers can only be detected by crc. I don't remember which ones, sorry, and I still have to clean up our gameboy.xml database, so I don't have the specific info handy
Yeah, same logic for the Super NES version of ZapFC I guess :/
FitzRoy wrote:
You can put a board file in the container.
Thats great, now anyone can put a garbage board file in the container! I thought one of your goals was having no bad ROMs, and you just made up an easy way to create them.
FitzRoy wrote:
And screw color, and pcb scans, and 3d renders of the plastic molding, and game guides, and genres, and game credits, and, and, and... Most of that stuff is so flippant, complex, or subjective that you will never come to an agreement on how to preserve it or what to exclude.
This is why an XML type approach would be better for preservation, that info can be added as needed without breaking all existing ROMs. Things like PCB scans could be added to the same ZIP with the file name in the XML instead of locking down the ZIP to not allow any other files.
Instead you just say its too hard, so ignore it all? Even though bootgod's site is already doing much of it? So much for wanting real preservation. Good job ignoring all the other problems too.
etabeta wrote:
3gengames wrote:
People trying to change standards are icky.
I'd agree if the standard was capable to represent all games, which iNES is not... at least iNES2.0 can eliminate this 100%.
Fixed.
FitzRoy wrote:
3gengames wrote:
Emulators != ROM formats. ROM formats are standard while Emulators are not.
Dude, you are so unbelievably dumb. Rom formats are influenced or even created by emulators.
[/quote]
The first ones are created by emulators, and it's the standard. See the iNES standard set in what seems to be
1998. It's been modified since then slightly for the very few games that did't work and weren't known to exist, but that has since been fixed, sorry.
http://nesdev.com/neshdr20.txt
And also, should ALL PRG-ROM's be at least 32KB big since in games that are only 8K/16K make the whole data range just mirror? Game might be 8K, but that doesn't mean the game program uses the last 8K or the first 8K to run. So it technically is all needed.
It seems to me that there are three main goals, and they don't overlap. All of them are hindered by insane copyright laws adopted by democratic governments where lobbyists prevail.
- Fitzroy is concerned with preserving the exact details of dumped game ROMs. He champions the "ZapFC" format for this.
- Most of #nesdev is concerned with making game images (homebrew, commercial, pirate, anything) playable in emulators and on real hardware. We champion iNES (1 or 2) or sometimes UNIF.
- Arcade emulation, which due to the huge variability in system board, must preserve exact ROMs and uses databases that tell the emulator how each of the CPUs in the arcade use each ROM, where it is mapped, which buses connect to which and so forth. Here emulation requires exact preservation. (Please correct me if I'm wrong on this. The only things that I know about MAME and the like I gleaned from this thread).
Fitzroy stated that his big concern is archiving accurate ROM dumps. ROM dumps that he probably can't openly share with the world for legal reasons. I'm willing to bet that the VAST majority of users of NES ROMs are to play games in emulators, not as digital historians. Those users don't really care how it is packaged, so long as the EMU is able to play the game properly.
I don't know how many "official" NES games there are. I thought that there were 760 or so released in the US market (based on unique prog-roms or titles that use the US region's lockout chip), and a few hundred more that were only released in Japan, Europe or the rest of the world. So I'm going to pull a number out of my ass and guess that not counting hacks or homebrew, there are less than 2000 unique NES titles. There are a handful of solid emulators that are actively maintained and work reasonably well.
If these emulators can successful play the entire corpus of "official" games, then the iNES format is sufficient for emulation.
It doesn't matter if you dump a game to iNES or ZapFC or MAME format. People can't use the ROM until they obtain it from you. If your goal is to archive perfect images of games ot prevent against bit-rot or just history taking its toll, you'll have to store copies in lots of places and ensure that a legacy of maintenance follows behind you.
I just don't see that happening effectively enough to be viable, at least in Japan or any western-civilization type country, due t the insane copyright laws that we tend to have.
If the ZapFC format were agreed upon and adopted, consumers of ROMs archived in that format still need to obtain the ROMs, most likely through technically illegal means. This is no different that using FCEUX and dumps in the iNES format.
It is obvious that ZapFC will not replace iNES. Let those who want to archive ROMs exactly do so, with whatever format they want. Why is it so important to crusade against the established format, when it works well enough? Can't we just have both (or three if MAME does something unique)?
Quote:
Thats great, now anyone can put a garbage board file in the container! I thought one of your goals was having no bad ROMs, and you just made up an easy way to create them.
Huh? A garbage board file will make a licensed rom fail completely to load. And I never said board files for inifinite games wouldn't suffer the same rom-side problems as iNES. That's an inherent problem for that material, you can't improve upon it.
It's an extra feature to support for people who believe in its benefits, it's not going to "replace" the Lady Gaga of rom formats. If you can't stop interjecting with fud statements, it's because you're ridiculously insecure about iNES. What I'm saying isn't rocket science. This is just a way of using a standard container that can be opened by standard programs with licensed mapping delivered by a trusted source. That's all it is. Woe be the emulators that ever support it. Woe be to them. How about letting the worthless, pointless, woeful format be discussed without further interruption from iNES groupies?
This format is the only worthless one here, honestly.
And plus, when the court finally sides with Nintendo (It'll happen eventual....from bribes.) do you think anyone will want the names inside the ROM's? HELL NO.
Sorry you wasted your years on this.
Still insecure enough to troll incessantly?
Please cut the "you're ridiculously insecure" part. Some users have complained to me that it looks like a
personal attack. And the same goes for anyone else replying to this topic: Address the idea, not the person. If it gets even more out of hand than it is, I'll have to call
Secret Bear.
As for the facts: This approach is worth it for the same reason that MAME's handling of Vs. Unisystem and PlayChoice games is worth it.
Are you kidding me? 3gen said he would stop posting 3 pages ago. He posts the same thing. "This format is worthless. This format is worthless." He's not adding anything to the discussion, he's not rebutting points with reasoning, he's just interrupting the thread with provocation. We get his position. We get it already. He'd have been banned from any other forum for doing this. Someone clearly enjoys his prodding.
Well, sure....but not here. And even then, they're still basically just NES carts.
- With respect, but... yes, tepples:
tepples wrote:
panzeroceania wrote:
3gengames wrote:
Let's make more stuff that makes ROM's not useable on real hardware from the virgin files! Despite that being basically the whole idea....
how hard do you think it would be to add support to a flashcart?
About as hard as porting Info-ZIP UnZip to run on a Commodore 64. To see why, see previous pages.
FitzRoy wrote:
Decompressing the set to ".fc" folders treated as files by the flash cart is probably better suited for stuff like flash carts. The format only enforces ZIP as the default because folders can't be download links and flash carts are less than 1% of users.
I think you missed the part where he said not to use .zip for flash carts tepples.
clueless wrote:
ROM dumps that he probably can't openly share with the world for legal reasons. I'm willing to bet that the VAST majority of users of NES ROMs are to play games in emulators, not as digital historians.
I've started to look to the future, I want my kids to get the real mccoy, and I want to be damn sure it's verifiable. I'm being completely serious. I play on a flashcart most the time anyways or real carts, but real carts will die eventually. Some of mine are already failing. And yes that really does matter to me.
clueless wrote:
Those users don't really care how it is packaged, so long as the EMU is able to play the game properly.
while it may be true, that's a pretty complacent attitude, lots of things wouldn't get done with that "good enough" mentality. Of course you can go to far in the opposite direction, but I'd rather do that than be complacent. Just because the majority of people aren't enthusiasts and are complacent, that doesn't mean that's the right attitude. The fact that we are all here on this board because of a console that is over 25 years old, older than me in fact, means we're enthusiasts, not just ROMZ KIDDIES.
clueless wrote:
If these emulators can successful play the entire corpus of "official" games, then the iNES format is sufficient for emulation.
that's a matter of oppinion, but even assuming that's true, that doesn't mean it's enough to reproduce a cart in 100 years.
clueless wrote:
People can't use the ROM until they obtain it from you. ... I just don't see that happening effectively enough to be viable, at lest in Japan or any western-civilization type country, due t the insane copyright laws that we tend to have.
If the ZapFC format were agreed upon and adopted, consumers of ROMs archived in that format still need to obtain the ROMs, most likely through technically illegal means.
people will either dump their own games and verify against the database, or they will do what they've always done, download ROMs from ROM sites, it's not like this format is magically unable to be uploaded to ROM sites.
clueless wrote:
This is no different that using FCEUX and dumps in the iNES format.
What does the emulator have to do with it, and how does being able to download ZapFC ROMs make it the same as iNES? that's like saying jpegs and mp3s are the same, because you can download both of them.
clueless wrote:
It is obvious that ZapFC will not replace iNES.
probably not, just like no-intro hasn't replaced goodtools, but no-intro is popular enough to be self sustaining and is live and being updated.
It's also like bsnes vs zsnes. zsnes is more popular but it's stagnant.
clueless wrote:
Why is it so important to crusade against the established format, when it works well enough? Can't we just have both (or three if MAME does something unique)?
Yeah we can have all of the above. I don't think Fitz honestly expects to "REIGN SUPREME LOLZ" I think he's just got a persistant personality, and wants people to know what he has to offer. Is it perfect? no. is it progress? yes.
Do most average users care? no. Do I care what other people care? no.
This is clearly for the enthusiast, but if the enthusiast does the heavy lifting, it may make it easier for the people who don't care later down the road for random game X.
Finally, there are plenty of things that work well enough but leave plenty of room for improvemnent. What I don't understand is the negative reaction to improvement.
Even if you don't believe it to be improvement, what is the big deal? The only reason I can think people might be upset about it, instead of indifferent, is if they actually think it might take off a little, and become a problem for them.
Who knows.
panzeroceania wrote:
clueless wrote:
If these emulators can successful play the entire corpus of "official" games, then the iNES format is sufficient for emulation.
that's a matter of oppinion, but even assuming that's true, that doesn't mean it's enough to reproduce a cart in 100 years.
My statement was about iNES being good enough for
emulators, not for reproducing carts. To repro a cart, you would _WANT_ something like Fitzroy is proposing. Its not an either this or that mutually exclusive choice.
My main point was that we can have both, as they serve different needs, different audiences.
fair enough, I'm not trying to misrepresent what you're saying.
I agree it doesn't have to be one or the other. I personally tend to lean towards the other side but to each their own I suppose.
I knew you weren't saying anything about reproducing carts. That was kind of my point. It's my inclination to pick something that is robust enough to do many things, and then try to encourage adoption of it so that when it's needed, it's widely available enough to be easy to get and use. Does this mean it needs to dominate? no. Just be prevelant enough that people are familiar with it.
Here is a question. Assuming homebrew developers can use this even if they dont' get added to the database. How does this hurt end users? if an end user can use a more robust format that can be helpful in other arenas, and he's none the wise, how does this hurt him? All that would be needed is support by NES emulator authors and it would be like nothing happened to the end user. No pain, no problem.
ultimately though I don't think this is aimed at the end user primarily, but more at dumpers, devs, etc. and having end users use it would just be a bonus that gives devs an additional reason to work with it.
clueless wrote:
My statement was about iNES being good enough for emulators, not for reproducing carts. To repro a cart, you would _WANT_ something like Fitzroy is proposing. Its not an either this or that mutually exclusive choice.
I'd rather have BootGod's database, as it contains much more useful information. If the game is listed in BootGod's database, you can make a repo with an iNES file in every case that I know of (This includes games like Action 52).
From looking at this thread again, what I see is missing is not a new format, but rather tools that can work with the information already there. ZapFC doesn't offer anything that isn't already there (iNES + BootGod's database).
The NES is a 16-bit system made by Sony in 1994.
BIG TROLL AHEAD.
But only the homebrews in the database should be ones with source released. Or else it'll hurt the homebrew community. And then, who will update the database? If your on vacation and a game source is released, users won't be able to play for a while. And plus, the devs are just going to use the source, their compiler, make a .NES, and then split it and put it on cart. It's super simple. Maybe if you also got a compiler in on this where it could add a compiled game to the database when it's compiled would work better. Then get an emulator to use this with/make an emulator. Then make both good to the point it makes sense to use over FCEUXD/Nestopia/Nintendulator and NESASM/ASM6/CA65/CC65 then it might have a chance. And then you'd save one step in the file-splitting process. It's a big list, and nobody is stopping you.
3gengames wrote:
But only the homebrews in the database should be ones with source released. Or else it'll hurt the homebrew community. And then, who will update the database? If your on vacation and a game source is released, users won't be able to play for a while. And plus, the devs are just going to use the source, their compiler, make a .NES, and then split it and put it on cart. It's super simple. Maybe if you also got a compiler in on this where it could add a compiled game to the database when it's compiled would work better. Then get an emulator to use this with/make an emulator. Then make both good to the point it makes sense to use over FCEUXD/Nestopia/Nintendulator and NESASM/ASM6/CA65/CC65 then it might have a chance. And then you'd save one step in the file-splitting process. It's a big list, and nobody is stopping you.
3gen, and others, one point that the ZapFC pro crowd has been stating all along is that emulators are not going to DROP support for iNES files. As a home-brew creator, I'll probably never create a game that needs some whack-ass mapper. As a developer I will ensure that my game uses a standard mapper, like NROM (no mapper at all), MMC1 or MMC3. My game's iNES headers will be 100% correct. There is no need to place homebrew into the ZapFC database. Maybe once the game is no longer being actively developed and has shipped its "final" version, then fine.
I have yet to hear of an emulator that plans to adopt ZapFC and abandon iNES. I would be suprised if an established** emulator did this.
**- for an arbitrary definition of "established". I guess that means being listed on Zophar's domain in the EMU list, and being capable of properly emulating the majority of non-whack-ass-mapper/board games.
Wkter wrote:
ZapFC doesn't offer anything that isn't already there (iNES + BootGod's database).
That's it, pretty much. Not everyone isn't bitching about it just to complain, we're trying to save anyone the time of implementing (and FitzRoy from creating) yet another NES ROM format for no good reason. We've got Pasofami format, Famtasia format, iNES, UNIF, .DKA/.DKB and .FDS, how many times does the wheel need to be re-invented? At least UNIF allows useful additions.
If the main complaint about iNES is that people in the past have distributed ROMs with wrong info in the headers, how would this seriously change anything? iNES ROMs typically were dumped by very competent people such as kevtris. These complaints seem to boil down to human error, and some new format isn't going to save us from that.
panzeroceania wrote:
I play on a flashcart most the time anyways or real carts, but real carts will die eventually. Some of mine are already failing. And yes that really does matter to me.
I would gladly accept (and maybe buy) "dead" NES carts, if you don't want to fix them. All there is to do is clean the connector and/or change the battery. Unless you're zapping it with thousands of volts of electricity or seeing corrosion, a typical NES and SNES cart will surely outlive all of us here if stored properly (and without a battery).
etabeta wrote:
xamenus wrote:
What if someone wants to run a non-split, headerless ROM (i.e. an iNES ROM without the header)? The file could be given the extension .fc, and should work in the same way that the split ROMs work. I tried loading such a file in QuickNES but that extension wasn't supported by the open dialog.
you would need to store the checksum of prg+chr too, in addition to those of the split file. I don't think the new format is supposed to work like this. for completeness sake, not even NEStopia or MESS xml support this (even if e.g. bootgod's xml does contain both checksums type...)
Doesn't the ZapFC database use prg+chr?
Memblers wrote:
Wkter wrote:
ZapFC doesn't offer anything that isn't already there (iNES + BootGod's database).
That's it, pretty much. Not everyone isn't bitching about it just to complain, we're trying to save anyone the time of implementing (and FitzRoy from creating) yet another NES ROM format for no good reason. We've got Pasofami format, Famtasia format, iNES, UNIF, .DKA/.DKB and .FDS, how many times does the wheel need to be re-invented? At least UNIF allows useful additions.
If the main complaint about iNES is that people in the past have distributed ROMs with wrong info in the headers, how would this seriously change anything? iNES ROMs typically were dumped by very competent people such as kevtris. These complaints seem to boil down to human error, and some new format isn't going to save us from that.
ZapFC is good for people who don't want the cartridge info bundled with their ROMs. Is there any other format that offers that?
Not that I necessarily would use the format myself. The No-Intro team might like it.
xamenus wrote:
ZapFC is good for people who don't want the cartridge info bundled with their ROMs. Is there any other format that offers that?
- UNIF?
sorry for gluing replies to different people, but I'm at the airport and I'm a bit in hurry
panzeroceania wrote:
ultimately though I don't think this is aimed at the end user primarily, but more at dumpers, devs, etc. and having end users use it would just be a bonus that gives devs an additional reason to work with it.
well, most end users do not know what iNES is at all. they would not care if they have to download a .nes file, a .fc.zip file or a zipfile containing separate prg/chr as long as the emulator they use work with that file.
you are right that no one is trying to completely drop iNES, but for sure there seems to be a lot of resistance to the idea of adding to emulator support for a xml or zapfc database in addition to iNES
Wkter wrote:
From looking at this thread again, what I see is missing is not a new format, but rather tools that can work with the information already there. ZapFC doesn't offer anything that isn't already there (iNES + BootGod's database).
except that BootGod's database is not complete.
MESS xml adds info for all the remaining official and pirate dumps (including Dipswitches for pirate carts), but it still lacks some of the items of BG's db (I'm working into adding these as well, but it will take some time)
and both these xmls databases would not be of help if someone does not want to add an xml parser to its emu. in the latter case, ZapFC is a viable solution to offer an easier-to-parse internal database
xamenus wrote:
ZapFC is good for people who don't want the cartridge info bundled with their ROMs. Is there any other format that offers that?
Not that I necessarily would use the format myself. The No-Intro team might like it.
I'm missing something (and the difficulty in reading the db in a text editor does not help at all): is there only the checksum of PRG+CHR? or also the separate ones?
if the former, then I don't see how the format can allow for separate prg/chr handling. and if this is not supported then the format fails to really document the content of a NES cart: at that point, I don't see how it is superior to the current nointro dat which supports the CRC32 + SHA1 of the iNES files without an header.
I thought the whole reason to add a new format was to better document the real carts, not to only use a larger checksum algo.
if the latter, otoh, then we are back to my previous comment: ZapFC can be a useful internal database for whoever does not want to parse xml in its emu
Zepper wrote:
xamenus wrote:
ZapFC is good for people who don't want the cartridge info bundled with their ROMs. Is there any other format that offers that?
- UNIF?
nope. unif contains a global header + a shorter header for each chunk of data.
etabeta wrote:
sorry for gluing replies to different people, but I'm at the airport and I'm a bit in hurry
I love airplanes!
etabeta wrote:
but for sure there seems to be a lot of resistance to the idea of adding to emulator support for a xml or zapfc database in addition to iNES
I'm not opposing to the database, but the format. I successfully generated the SHA256 keys the ZapFC database uses from iNES ROMs, which means that the ZapFC is compatible with iNES ROMs.
etabeta wrote:
except that BootGod's database is not complete.
MESS xml adds info for all the remaining official and pirate dumps (including Dipswitches for pirate carts), but it still lacks some of the items of BG's db (I'm working into adding these as well, but it will take some time)
True, but BootGod's database contains a lot more information that ZapFC does (And probably ever will).
etabeta wrote:
and both these xmls databases would not be of help if someone does not want to add an xml parser to its emu. in the latter case, ZapFC is a viable solution to offer an easier-to-parse internal database
Again, the ZapFC database is iNES compatible, so I'm not opposing to that.
etabeta wrote:
I'm missing something (and the difficulty in reading the db in a text editor does not help at all): is there only the checksum of PRG+CHR? or also the separate ones?
It only contains one SHA256 checksum. It's made by generating a checksum from the CHR and the PRG, combining the two checksums into a single uppercase string, then make another SHA256 checksum out of that. So there is no way of telling what part, CHR or PRG, might be bad in a dump.
xamenus wrote:
ZapFC is good for people who don't want the cartridge info bundled with their ROMs. Is there any other format that offers that?
iNES with all the fields set to 0
xamenus wrote:
ZapFC is good for people who don't want the cartridge info bundled with their ROMs. Is there any other format that offers that?
Pasofami format (delete/ignore the .PRM file), if split PRG/CHR is what's wanted. ucon64 supports it, IIRC the command is "ucon64 -s -paso gamename.nes". I had made a batch file that would split a UNROM iNES file, so I could automatically assemble it to work on my UNROM dev cart (uploaded with a ROM emulator). A similar batch file could convert all of one's iNES files into a split format. But why someone would want to store multiple copies of their ROMs in different formats, I don't know.
Wkter wrote:
It only contains one SHA256 checksum. It's made by generating a checksum from the CHR and the PRG, combining the two checksums into a single uppercase string, then make another SHA256 checksum out of that. So there is no way of telling what part, CHR or PRG, might be bad in a dump.
If you open both containers in WinRAR, you've got separate CRC hashes to compare which will tell you which parts are different. The database has nothing to do with comparing dumps.
FitzRoy wrote:
If you open both containers in WinRAR, you've got separate CRC hashes to compare which will tell you which parts are different. The database has nothing to do with comparing dumps.
You've completely lost me... What are you supposed to compare them to if the database doesn't contain the cheksums? Also, wasn't one of your selling points that your format would make it easier for dumpers to validate what part of their dumb might be bad?
Memblers wrote:
Pasofami format (delete/ignore the .PRM file), if split PRG/CHR is what's wanted. ucon64 supports it, IIRC the command is "ucon64 -s -paso gamename.nes". I had made a batch file that would split a UNROM iNES file, so I could automatically assemble it to work on my UNROM dev cart (uploaded with a ROM emulator). A similar batch file could convert all of one's iNES files into a split format.
the pasofami format is exactly what MESS supports (with no prm file) through its xml database.
and while pasofami sets are impossible to be found, the MESS sets are showing up here and there
Memblers wrote:
But why someone would want to store multiple copies of their ROMs in different formats, I don't know.
given how many users store multiple bad dumps of games just because they do not even know what a [b1] flag is for, I don't think it is really a problem of storing multiple copies.
I think users will simply get the files which are supported by emus and easily available. so far only iNES has been both supported and easy to find, and therefore there has been no real choice.
some MESS users are helping to make split prg/chr (call it MESS format or pasofami, I don't care) a bit easier to find than it currently is, but as long as no other emu supports them, it's a bit difficult that anyone would choose it over iNES
Wkter wrote:
You've completely lost me... What are you supposed to compare them to if the database doesn't contain the cheksums?
You compare it to the already existing dump. An existing dump is a file that any dumper would have, it's not just an entry in a document.
Existing.fc.zip <-open
prg | 8A76BC8A
chr | 1E3DD667
New.fc.zip <-open
prg | 99E2CC7A
chr | 1E3DD667
FitzRoy wrote:
An existing dump is a file that any dumper would have
debatable: a few dumpers might decide to only have the dumps they got from the carts they own, not all the available ones...
but at least it is now clear to me how zapfc treats separate prg/chr
etabeta wrote:
a few dumpers might decide to only have the dumps they got from the carts they own
That's the policy I follow on my GBA, which has "MultiBoot" (essentially
netboot over the serial port) so dumping my own Game Paks to a PC is as easy as hooking up a GBA-to-PC cable. The NES, on the other hand, can't boot over the controller port (yet), and not nearly as many people know how to solder up a CopyNES, so people with a sizable collection of NES Game Paks are more likely to torrent a "GoodNES set" instead of dumping their own. Perhaps a Game Genie with its ROM replaced with
standardized boot code might help.
How do I actually convert a homebrew .nes to .fc ? I've read through 24 pages of trolling, scoured dead link after dead link at zapatabase.com, and can't even find a description of the format, much less a tool to do conversion. My only interest in caring about this is byuu's exclusive use of it in bsnes.
Judging by the thread title alone, I'm guessing you chop off the first 16 bytes with a hex editor.
Dwedit wrote:
Judging by the thread title alone, I'm guessing you chop off the first 16 bytes with a hex editor.
Thanks Dwedit, that was helpful - but what a huge waste of time this was. I have been trying to run Blargg's nes_cpu_test5. With your advice, and hours of digging through the bsnes source code I figured that I need to:
1. create a folder named "blargg_nes_cpu_test5.fc"
2. copy the .nes file into said directory
3. chop the 16 byte header off
4. rename the file to "program.rom"
5. create a "manifest.xml" with the following content:
Code:
<cartridge>
<board>
<type>NES-NROM-256</type>
</board>
</cartridge>
After which, bsnes segfault'd running the program - it fails to load the reset vector from $FFFC. I think I'm done with this topic.
From what I understand, Blargg's test rom wouldn't have worked with this format anyway. Nor any homebrew.
It was a database for licensed games only.
Edit:
Quote:
My only interest in caring about this is byuu's exclusive use of it in bsnes.
I only just noticed this. I just downloaded bsnes_v088 (32bit compatibility) from the bsnes homepage, and I ran one of blargg's CPU tests in the .NES format just fine. Am I missing something (like you need to do it with a different version of bsnes?), or is all you wanted to accomplish just running this rom in bsnes?
v088 and earlier allow loading of iNES 1.0 games as .nes files.
v089 and above will only load images in the FC spec:
Code:
Super Mario Bros.fc/
* program.rom (Super Mario Bros.nes sans first 16-bytes)
* manifest.xml
<cartridge>
<board type="NES-NROM-256"/>
<mirror mode="vertical"/>
</cartridge>
* save.ram (not for SMB, but for other games.)
In v088 there is a tool called purify, that when run on the command line, will turn a collection of NES (or ZIP) files into folders, and generate the manifest files for you.
By v090, we're hoping to have a launcher so you can associate your .nes/.zip file with it to have it launch in bsnes.
I can't call the XML markup spec finalized until I support many more NES mappers, however, so use it now at your own risk.
My format is fundamentally incompatible with what FitzRoy was attempting to do, and I don't have support for his spec. He plans to make a derivative fork of bsnes with support for his spec, which I fully support him doing.
FitzRoy's spec is:
Code:
Super Mario Bros.zip/
* prg.mrom
* chr.mrom
With the mapping information pulled from a database found by hashing both ROMs and finding them.
Although this won't work for homebrew, it does neatly avoid my issue of trying to finalize an XML spec that is still changing between releases.
Save RAM, cheats, states, etc are stored elsewhere.
The last time I read the spec as amended, there were three files: prg, chr (optional), and board. Games licensed by Nintendo would omit the board file; instead, they'd be looked up in the ZapFC database based on hashes of prg and chr. The board file was added in an amendment as a concession to allow unlicensed games, homebrew, and pirate-originals.
