Any interest in an open source GoodNES clone?

This is an archive of a topic from NESdev BBS, taken in mid-October 2019 before a server upgrade.
View original topic

Any interest in an open source GoodNES clone?
by proxy on 2011-03-08 (#75282)

So I've recently decide to revisit some NES related dev and I noticed something. GoodNES is a great ROM organizer, but it has a few faults...

* The database is stored compressed in the .exe itself, so it is difficult for 3rd party tools to use it if they choose.

* It is windows only! I use linux for my primary computer, and I'm sure that OSX has NES emulators too.

* We as a community have little to no control over when and how it gets updated, the last GoodNES was from a few years a go. Since then, iNES 2.0 has been proposed and GoodNES will end up incorrectly "fixing" these headers.

I have already been able to dump the database in plain text from the GoodNES executable, so problem #1 is solved. Now we need some tools which can use this. So, would there be any interest in the community for a GoodNES clone which was open source? I'll probably do some work on this since like I said, I use linux for my day to day computing and do have a need to keep my collection organized.

I think it would be generally beneficial. While the number of official NES ROMs is certainly finite and probably most if not all have been accounted for, certainly home brew and test ROMs can continue to come out as time goes by. It would be nice if there was a way for us as a community to be able to manage these ROM collections in an organized way.

I plan to conform the GoodNES standard as far as naming and all of that. But I also plan to keep the DB as a separate file so it is easy to update it over time.

What do you guys think?

by tepples on 2011-03-08 (#75283)

I would like to see a Free clone of GoodNES, except for the following:
I have qualms about using a database extracted from GoodNES due to compilation copyright and sui generis database rights issues. It might be better to start with BootGod and/or ZapFC databases.
The way (PD) games were managed in the old database, if I remember correctly, encourages keeping outdated versions around.
Putting the ability to make a report analogous to NESMiss.txt into the standard distribution is a bad idea because it encourages hoarding.

Another killer feature would be automatic correction of not only headers but also the various overdumps and CHR underdumps floating around. I have some ideas on how to do this.

Re: Any interest in an open source GoodNES clone?
by lidnariq on 2011-03-08 (#75284)

I have no problem with anything else you've said, but
proxy wrote:
* It is windows only! I use linux for my primary computer, and I'm sure that OSX has NES emulators too.
WINE (and I assume DarWINE) does a beautiful job of running windows console mode applications like GoodNES, so this isn't any worse than any other closed-source software.

by frantik on 2011-03-08 (#75285)

tepples wrote:
[*]Putting the ability to make a report analogous to NESMiss.txt into the standard distribution is a bad idea because it encourages hoarding.

what's wrong with 'hoarding' roms?

by atarimike on 2011-03-08 (#75286)

tepples wrote:
I would like to see a Free clone of GoodNES, except for the following:
I have qualms about using a database extracted from GoodNES due to compilation copyright and sui generis database rights issues. It might be better to start with BootGod and/or ZapFC databases.

You can't legally extract the DB out of the original GoodNES exe and then license it under an open-source license. You'd have to get the owner of GoodNES to relicense the DB or start a new DB.

by tepples on 2011-03-08 (#75288)

frantik wrote:
what's wrong with 'hoarding' roms?

I define ROM hoarders as people who seek out ROMs just to complete their set, not even to play them. A feature useful only to ROM hoarders would likely lead to liability for inducing copyright infringement. I seem to remember one ROM-auditing tool whose license stated that modifying it to make a list of all missing ROMs is forbidden for precisely this reason.

by frantik on 2011-03-08 (#75289)

well im no lawyer but it seems that any complete list of nes games could be construed to say it's encouraging people to infringe copyright

goodnes and any replacement program are primarily going to be used by copyright infringers anyways right?

by Memblers on 2011-03-08 (#75290)

tepples wrote:
The way (PD) games were managed in the old database, if I remember correctly, encourages keeping outdated versions around.

Yes, this is very annoying. I'm pretty sure everyone would release stuff a little more often if it didn't result in every version getting treated the same as the final version.

Often you don't even get to see the full filename when it's listed, which means someone is just as likely (perhaps more likely if it's listed first) to open the v.001alpha instead of whatever is newest. Better option would be notifying there is a newer version, and deleting/relocating older versions.

The only way people download this stuff, is one at a time when they want it, or all in one big file. So supporting the famous missing.txt is irrelevant pretty much. There aren't any old carts being dumped anymore, except the occasional prototype.

by Zepper on 2011-03-08 (#75296)

- One thing: for what special reason would you like to scan homebrew ROMs? What would you like to fix: header? "bad dump"? filename? None of them, right? A way to organize them? Cool, but isn't supposed to the homebrew-guy author to do that when distributing it?

by Memblers on 2011-03-08 (#75297)

Assuming the game is released freely, the author isn't going to have much control over distribution. It will end up in a big zip file with all the other ROMs, and the manual and source will be lost, might as well not leave it totally orphaned.

by tepples on 2011-03-08 (#75298)

Zepper wrote:
- One thing: for what special reason would you like to scan homebrew ROMs? What would you like to fix: header? "bad dump"? filename?

1. Filename, 2. header to take into account further revisions to NES 2.0, and 3. make sure it's not corrupt, hacked, or an outdated version.

by proxy on 2011-03-08 (#75299)

You make a good point about the GoodNES DB and copyright issues.

I think a reasonable compromise would be to have the DB file I do use to be in a compatible format. This would allow me to write the software in a way that lets me test the correctness of the program, I just wouldn't distribute the GoodNES DB. I do plan on extending the DB format. I would love to have a new field which specifies the actual mapper board name. This would allow me to write tools which can automatically convert between iNES and UNIF and back. Along with being a good source for an emulator to do proper checksum based mapper detection. All sorts of goodies.

Once the program is deemed "functional", we could work on constructing a pure/clean Open Source DB. That shouldn't be hard given all the awesome work that people like BootGod have done.

RE using Wine. Yes that will work, but it shouldn't be a requirement. Simply writing it in c or c++ in a portable way and releasing the source is a much more useful solution.

Anyway, it seems that there is indeed interest in this, so I'll definitely try to make this happen and will keep you guys posted.

PS: and please feel free to give suggestions about improvements over the GoodNES tools that I could implement!

by proxy on 2011-03-09 (#75311)

So I've started basic work on this project and things are going well. As far as the DB is concerned, I've looked at the GoodNES license:

Quote:
3. You are NOT authorized to reverse engineer or otherwise extract the
database of names included inside the GoodTools. This includes
generating databases for other renaming programs. All previous licenses
to use the GoodTools databases are REVOKED as of March 13, 2005.
If you wish to continue using GoodTools databases,
please contact the author for a renewal.

So that settles that matter . To be honest, after looking more carefully at the GoodNES DB format, it's slightly insane, (specifically the single variable length field is in the middle of the entry!), so I don't mind creating a new similar one.

My current plan since I have a large amount of ROMs handy already is to use a scan of my collection to bootstrap my DB and then I will supplement it with public/open source information like the BootGod stuff.

by kyuusaku on 2011-03-09 (#75313)

If you'd like some dirty Python code I wrote programs to dump the NesCartDB XML to CSV, and scan iNES files and dump the header fields to CSV. Letting the second program loose on an already renamed GoodNES set would be a slightly more legal way of getting the DB. I plan(ed) to splice and clean them up for ROM fixing, but since NesCartDB isn't complete I'm using CSV as an intermediate format to add more games.

by proxy on 2011-04-11 (#76503)

I've some great progress on my "openGN" project. I have it scanning/confirming (and for test purposes renaming) the contents of my ROM collection and so far it agrees with GoodNES. Currently it is only aware of ordinary iNES files, UNIF and iNES 2.0 will be added once regular iNES support is perfected. I am also going to work on adding the stuff from NesCarts.xml to have a much more complete ROM DB. (I've even found a few minor mistakes in the naming that GoodNES uses ).

So I have a question with regard to iNES 2.0. Has the submapper stuff been hashed out, is there any "authority" out there which has designated them? Currently, the way that I imagine them is something like this:

Mapper 4, Sub 0 (MMC3/MMC6 Generic/Unknown)
Mapper 4, Sub 1 (TBROM)
Mapper 4, Sub 2 (TEROM)
Mapper 4, Sub 3 (TFROM)
Mapper 4, Sub 4 (TGROM)
etc...

Is this the idea of how it's going to be? What about generalizations that are compatible like UxROM? For completness sake I would like this see this:

Mapper 2, Sub 0 (UxROM use 8 bits for PRG swap, supports much larger games)
Mapper 2, Sub 1 (UNROM use 3 bits for PRG swap)
Mapper 2, Sub 2 (UOROM use 4 bits for PRG swap)
etc...

But I can see that UxROM is "good enough"

Thoughts? Concerns? Comments?

I'll keep everyone posted as I continue to figure things out.

by proxy on 2011-04-11 (#76506)

Also, something that I've noticed is that often, ROMs seem to have "junk" after the PRG/CHR data. GoodNES includes this junk as part of its hashing. I counted 3507 ROMs which are longer than the iNES header says that need to be.

Often this junk is a URL or sometimes the title of the game, it never plays a part in the functionality of the ROM and has no net effect (why didn't DiskDude put his tag at the end of the ROM instead of crapping all over the iNES headers!)

This has a couple results:

* GoodNES can detect ROMs even when they have headers that don't make sense.

* You can have two hashes 2 roms which literally identical PRG+CHR.

Obviously point 1 is good and point 2 is bad .

So first things first, I will add to my DB entries which represent the ROMs when no junk is at the end.

I am considering having a feature which would do the following:

First detect if a ROM matches when doing a "to end of file" hash.
Once I've done that, I now know the correct size of the ROM since my DB has the PRG/CHR sizes.
After that, if the file is longer than expected, I could have an option to create a second ROM which is the same but has no junk at the end.

I think this is a reasonable approach since it will appease the collectors since they won't have to acquire new ROMs to keep there collection complete and give us an option to have "junk" free ROMs for those of us who just want accurate digital backups.

What do you guys think?

by tepples on 2011-04-11 (#76519)

proxy wrote:
So I have a question with regard to iNES 2.0. Has the submapper stuff been hashed out, is there any "authority" out there which has designated them? Currently, the way that I imagine them is something like this:

Mapper 4, Sub 0 (MMC3/MMC6 Generic/Unknown)
Mapper 4, Sub 1 (TBROM)
Mapper 4, Sub 2 (TEROM)
Mapper 4, Sub 3 (TFROM)
Mapper 4, Sub 4 (TGROM)
etc...

If the board can be reliably guessed from the size of PRG ROM, CHR ROM, PRG RAM, and CHR RAM, it does not need a submapper. For example, all MMC1 boards with 512 KiB PRG ROM, 8 KiB PRG RAM, and 8 KiB CHR RAM behave the same way as SUROM.

Quote:
Mapper 2, Sub 0 (UxROM use 8 bits for PRG swap, supports much larger games)
Mapper 2, Sub 1 (UNROM use 3 bits for PRG swap)
Mapper 2, Sub 2 (UOROM use 4 bits for PRG swap)

These can be reliably guessed from PRG size.

As for junk at the end: You might want to put in detection for when the appended data looks like a zip file.

by Drag on 2011-04-11 (#76527)

Given that commercial roms don't "change" as frequently as homebrew roms, the open GoodNES utility should have its main focus on commercial, non-public domain roms. As such, I think it shouldn't include any homebrew roms, save for ones that may have actually been "dumped".

Handling homebrew roms that are subject to updates and such would require a different solution, such as checking against an internet database, because that would be the easiest way to propagate updates.

by tepples on 2011-04-11 (#76544)

Drag wrote:
Given that commercial roms don't "change" as frequently as homebrew roms, the open GoodNES utility should have its main focus on commercial, non-public domain roms.

So does that exclude any game that was first sold on a cart but later liberated, such as the NES version of Elite?

Quote:
Handling homebrew roms that are subject to updates and such would require a different solution

I'm glad you're starting to understand FitzRoy's point.

Quote:
such as checking against an internet database, because that would be the easiest way to propagate updates.

I agree as long as the database can be downloaded separately, because some people don't have the ROM collection and continuous Internet access on the same machine.

by Dwedit on 2011-04-11 (#76547)

Since when did the author of Elite get the authority to claim copyright away from the publisher and liberate it? I thought it was still considered a pirated copy, despite the author releasing it.

Edit: Looks like the game itself attributes copyright to the developers, not the publishers. Then it's okay?

by Drag on 2011-04-11 (#76548)

tepples wrote:
So does that exclude any game that was first sold on a cart but later liberated, such as the NES version of Elite?

What's the set going to include? Surely it'll have all dumped rom images (from regular ROMs or prototypes dumped from ((e)e)proms or flash roms or what have you). If the game was complete, but unreleased due the inability to find a publisher, then I think it would qualify for the set, because the game reached a point where it was ready to be frozen to a rom and not changed (otherwise, reaching a "static" form).

Quote:
Quote:
such as checking against an internet database, because that would be the easiest way to propagate updates.

I agree as long as the database can be downloaded separately, because some people don't have the ROM collection and continuous Internet access on the same machine.

Fine with me, just as long as there's some way to easily update the database.

by B00daW on 2011-04-12 (#76555)

I wouldn't mind seeing NESToy resurrected to use the BootGod/GoodNES/no-intro databases that have been fixed. NESToy also did a good job of keeping a compressed archive of iNES headers that could patch the ROM images. Without an iNES header the binaries are effectively useless. Qualms with GoodNES and no-intro databases excluding iNES headers or not fixing them has rendered people collecting giant masses of ROMs while some of them are unplayable unless you add your own headers fueled by personal research and experimentation. Some people without the knowledge of the iNES header format would consider the image to be broken, when in fact the utilities are lacking.

Again we're also on sketchy ground when it comes to Parodius and NESdev board terms. It seems that we allow linking to ROMs on occasion, especially if they are unreleased or of broad interest, but the database preparation and ROM collection efforts of the community to get a functional and accurate NES/Famicom game database once again would be traipsing on a gray area.

I believe as long as people attempt to not directly link copyrighted ROM images in the forum, but link using PMs or other personal communication means to exchange images for databasing purposes, release databases in the public forum, and let it be known without a public link that the image archive is hosted in a torrent on Underground Gamer, that the NESdev community would have a complete and accurate image archive for reverse engineering, hacking, and emulator development.

by kevtris on 2011-04-12 (#76558)

proxy wrote:
So I have a question with regard to iNES 2.0. Has the submapper stuff been hashed out, is there any "authority" out there which has designated them? Currently, the way that I imagine them is something like this:

Yes and no. I have made a "definitive" set of ROMs with 2.0 headers, but it's really outdated. I have been waiting patiently for a new set of ROMs to make the current list but so far have not gotten them.

If there is some tool that can extract just the headers from ROMs, I could run my set of 100 or so 2.0 ROMs through it and produce such a list.

I have a document I haven't released yet which has all the submappers I used defined, also... I guess I should clean that up and release it. I don't anticipate it changing too terribly much in the future... Though the Vs. stuff might change a little (mainly the controller type byte. The PPU and other bytes are OK).

by tepples on 2011-04-12 (#76562)

kevtris wrote:
If there is some tool that can extract just the headers from ROMs, I could run my set of 100 or so 2.0 ROMs through it and produce such a list.

man dd

Can you copy the first 16 bytes out of every file using this?

by lidnariq on 2011-04-12 (#76563)

kevtris wrote:
If there is some tool that can extract just the headers from ROMs, I could run my set of 100 or so 2.0 ROMs through it and produce such a list.

If you're using linux or cygwin, "for i in *.nes; do echo -en "$i"-; hd -n 16 "$i"; done".

by proxy on 2011-04-14 (#76736)

Another update .

So, there is a bit of a flaw in the way that GoodNES handles UNIF dumps. Basically, as far as I can tell, GoodNES will always try two things.

First, skip the first 16 bytes, hash the rest of the file. if that doesn't match anything in the DB, then hash the whole file and see if that matches the DB.

So, as you can imagine, UNIF always falls into the second category. The problem is that any two roms can have different meta-data but have the same PRG/CHR.

Here's my proposed solution:

I load the UNIF file into memory and process the blocks. I hash every PRG[0-9A-F] block in numerical order. I then hash every CHR[0-9A-F] block in numerical order. Non existent blocks have no effect, but you can have discontinuous blocks like PRG0, PRG2, but no PRG1.

I am thinking of making the hash include the UNIF block header so that an image with PRG0 and PRG2 wouldn't match an image with PRG0 and PRG1 where PRG1/2 are the same code. I know this is a corner case, but it's worth addressing if I can.

Then I concatenate the results in the order I collected them and hash that. Now we have a unique hash for UNIF files which will correctly identify that two file are the "same" but may have different meta-data. This now opens the possibility to correct bad meta-data such as the MAPR/MIRR/etc blocks similarly to the fixnes feature. This is necessary since at least one UNIF image in the GoodNES collection has a bad header making correct loading impossible.

Any thoughts? Anyone see a problem with this approach?

by tepples on 2011-04-14 (#76741)

proxy wrote:
Here's my proposed solution:

I load the UNIF file into memory and process the blocks. I hash every PRG[0-9A-F] block in numerical order. I then hash every CHR[0-9A-F] block in numerical order. Non existent blocks have no effect, but you can have discontinuous blocks like PRG0, PRG2, but no PRG1.

Yeah, that matches what one would do when translating UNIF into iNES. The hash would be defined as the hash of the conversion into iNES.

Quote:
I am thinking of making the hash include the UNIF block header so that an image with PRG0 and PRG2 wouldn't match an image with PRG0 and PRG1 where PRG1/2 are the same code. I know this is a corner case, but it's worth addressing if I can.

I'm not entirely sure I follow you. You don't need to hash the metadata because you can store that directly in the database. And you can do that because each individual cartridge's metadata isn't copyrighted (it's a fact). You can just load all the metadata chunks and compare them elementwise to your database.

Quote:
Then I concatenate the results in the order I collected them and hash that.

Reminds me of the hash in ZapFC.

by proxy on 2011-04-14 (#76758)

@tepples: I think perhaps you've misunderstood my plan . But that's ok, I'll explain:

What you describe is basically allocating enough memory and copying the PRG/CHR ROM into that memory and hashing it. This would work, but it requires extra memory. I'm not sure that I want to consider UNIF dumps to be duplicates of iNES dumps (at least not yet, I may be convinced otherwise).

What I am thinking is this:

suppose there are 2 PRG chips (PRG0 and PRG1) and one CHR0. Assuming the following:

SHA1(PRG0) = A
SHA1(PRG1) = B
SHA1(CHR0) = C

I would consider the hash of the UNIF dump to be:

SHA1(ABC)

This allows me to calculate the hashes of the chunks as I load them, and not worry about "making" them contiguous in memory. While still maintaining the following properties:

* unique for each UNIF dump (disregarding meta-data)
* I can process the various ROM dumps in an image in the order which that appear in the file (which can be ANY order), no jumping back and forth.
* no need to allocate memory for all chunks at once and copy data around
* considered different from iNES dumps.

The last two points are not 100% important, and I may be able to be convinced to let them go, but this scheme would work:

Anyway, as for the part you aren't entirely sure on, I'll make it more clear:

Suppose we have 2 dumps the first with the following chips/hashes:

dump #1 has PRG0 (16k)/PRG1 (16k)/CHR0 (8k)

SHA1(PRG0) = A
SHA1(PRG1) = B
SHA1(CHR0) = C

dump #2 has PRG0 (16k)/PRG2 (16k)/CHR0 (8k)

SHA1(PRG0) = A
SHA1(PRG2) = B
SHA1(CHR0) = C

Note that it is PRG2, NOT PRG1. I don't want these two dumps to be considered equal. But if I just do as I originally planned:

SHA1(ABC)

They would be...but, if i include the "PRG0\x00\x00\x40\x00" as part of the byte stream that I hash for the PRG chunk, now which chip it is, is part of the "uniqueness".

Essentially, I would like the hash of identical code found on different chip indexes to be considered different. I know almost all ROMs only have PRG0, but who knows what exotic cart we'll find next

I know this is a corner case, which is likely not happen... But I think since it is easily addressable may as well deal with it. I have a couple of variants on this theme in mind, but that's the general idea. Include the chip#/type as part of the secondary hash and it will be truly unique.

by tepples on 2011-04-15 (#76759)

proxy wrote:
What you describe is basically allocating enough memory and copying the PRG/CHR ROM into that memory and hashing it. This would work, but it requires extra memory.

Don't worry about extra memory. The biggest licensed ROM for NES or Famicom is 1 MB. The system requirements to run Windows 7 alone, let alone any applications, is a thousand times that.

Quote:
suppose there are 2 PRG chips (PRG0 and PRG1) and one CHR0. Assuming the following:

SHA1(PRG0) = A
SHA1(PRG1) = B
SHA1(CHR0) = C

I'd take SHA1(PRG0 + PRG1) and SHA1(CHR0), one hash for each bus.

Quote:
* no need to allocate memory for all chunks at once and copy data around

With files that small, what's the disadvantage of "keep[ing] the entire input file in memory and scan[ning] it there"?

Quote:
dump #1 has PRG0 (16k)/PRG1 (16k)/CHR0 (8k)

SHA1(PRG0) = A
SHA1(PRG1) = B
SHA1(CHR0) = C

dump #2 has PRG0 (16k)/PRG2 (16k)/CHR0 (8k)

SHA1(PRG0) = A
SHA1(PRG2) = B
SHA1(CHR0) = C

Note that it is PRG2, NOT PRG1. I don't want these two dumps to be considered equal.

I don't see why not. The data that the CPU sees consists of the same bytes in the same order, no matter what is silkscreened onto the chips. If PRG1 and PRG2 showed up at different bank addresses, the game wouldn't run anyway. Hence SHA1(PRG0 + PRG2) and SHA1(CHR0).

Quote:
Essentially, I would like the hash of identical code found on different chip indexes to be considered different. I know almost all ROMs only have PRG0, but who knows what exotic cart we'll find next

The multi-chip games I can think of are Action 52 (three PRG ROMs) and the After Burner mapper (two CHR ROMs).

by proxy on 2011-04-15 (#76778)

Fair enough points. I've actually just finished reworking some of my code for loading iNES to prefer memory mapped file. I'll try to think of a scheme similar for UNIF .

You may be right that the chip # doesn't matter. I had it in my mind that it did, but perhaps it shouldn't?

In an unrelated note. There is a single UNIF ROM in my collection which claims to use UNIF revision 8. Tennessee at one point asked me to be the future maintainer of the UNIF standard, as far as I know, 7b was the latest. Was there ever an 8 (official or otherwise?). At the moment, libunif will refuse to consider files with a version > 7 as valid. I have two choices here:

1. bump the official version to 8 to make the ROM valid. If there was a UNIF version 8 and it had a feature 7b didn't, add it to the standard.

2. correct the ROM.

Anyone have any insight into this mystery version 8 UNIF file?

by proxy on 2011-08-10 (#82674)

So I'm back, this project isn't dead, I just took a little break .

Anyway, it seems that that single UNIF file with revision 8 in it was created by a uCon64. Apparently, he decided to add a new TAG to the file which has "converter" information.

There are two things going on here. The ROM is still "loadable" since the spec says to just ignore unknown tags, but I plan on making it so I can "correct" ROMs which do this sort of thing in my tool.

So I need to come up with a better hashing system for UNIF than GoodNES. I've previously discussed things like this before. And I think I have a plan in mind.

Anyway, currently, the code can correctly identify all iNES roms in the GoodNES collection (which is a good start).

Next I just need to implement some basic (and safe with regards to duplicates) renaming capabilities. Once I have some of that done, I plan to put a copy of the code in a repo for all to enjoy .

The DB will be plain text based and open source, probably seeded by NesCartDB from BootGod's awesome work.

proxy