I posted about this over at nesworld the other day, figured I would here as well.
I've been putting off publicly releasing anything because I'm constantly tweaking either the software or the website. But I decided I could just as well open it up for a few, since it seems like I'm never done anyways.
In order to submit data to the site, you currently have to use the software and also have a CopyNES. The software is available on the "submit" page.
I have yet to write a proper readme / guide. It is for the most part self-explantory, but I'm sure there are many things that may need some elaboration. If anyone is having any problems, don't hesitate to ask.
For those of whom would like to just see what's there so far
here is the temp link. Please do not spread this link around for the time being.
Some of you may notice the site layout looks very much like vgrebirth.org because that's what I based it off it. I don't intend to keep it this way, I just needed something to start with, as I'm a programmer, not a web designer.
The software for it is for the most part done, it has been for quite a while I've just been adding small things here and there.
The site still needs plenty of work. Some things do not work and others may not work as expected, and it certainly doesn't have all the features I have planned for it.
The data it is collecting is the main thing I'd like to hear about. Especially if there is anything else you think it should be storing.
That is AWESOME! Heres some suggestions to make it even more awesome:
1. Add search with things like board type and prg/chr memory size.
2. Have some way to compare ines headers. There should be correct info there to produce accurate headers, sort of a new goodnes list that is verified and updated.
3. Have a list of known carts that arent in the database. I can definitely contribute and having a list of needed dumps would make it faster.
Im guessing theres no good legal way to distribute the correct roms so I will still have to dump all my carts sometime...
Thanks for the kudos
More search functions are at the top of the list, specifically hardware based searches.
You can load ROMs into the software, if it recognizes the ROM data, you can re-save it in either INES or UNIF. Basically it functions like goodnes in that sense except it can correct bad headers and what not.
As for known carts that aren't in DB yet, I have no plans for such a thing, but it's a possibility.
You may have noticed there are often many entries for a given game. When you dump a cart, even if you think it's in there, there may be subtle diffs from other entries, so a new one would be added. I found that it's just as quick to dump multiple copies of the same game as it is to check if it's been dumped and there are no differences.
This software currently does not work with my system, since there is no way to specify which parallel port to use (or the ability to manually specify the base I/O address for special parallel I/O PCI cards) - my CopyNES is connected to "LPT2", but it is mapped at the base address 0xC800 instead of the usual 0x278.
As it is, I have 2 unrecognized games (Athletic World [BC5F6C94/51BF1FC5] and Super Team Games [F210E68F/A11F69FF], both on NES-CNROM-256-05 boards) and 2 games on different boards (Ironsword on NES-AOROM-02, Bad Dudes on NES-TL1ROM-01), and I cannot add the correct dumps.
There isn't a proper settings dlg yet. Go into the registry and search for "NesCartDB" you can set the base address there. Hopefully that will do the trick for you.
It would help to know the name of the registry value I'm supposed to add...
Sorry, I thought it was writing the default in there. You need to add a DWORD key name "ParallelPort". Set that to whatever base address you need.
Nice site! I'll be sure to check it out when finding the correct donor carts / chip sizes for EPROM burning. The scans are great, too.
Thanks!
NC
BootGod wrote:
Sorry, I thought it was writing the default in there. You need to add a DWORD key name "ParallelPort". Set that to whatever base address you need.
Thanks, that did the trick (though I'm not sure if it properly set my ECP port to bidirectional or if it was just still in that mode from last running CopyNESW). I was able to successfully submit the 4 new entries I mentioned earlier. Athletic World was a bit of a trick, though, since I had already converted it into a RAM cartridge - using CopyNESW, I was able to reupload the game to the cartridge (and write-protect it) so your program was able to redump it.
Thanks for the contributions
I should point out for others too, that this program is
NOT a replacement for CopyNesW. This software will only dump carts/saves, it does not support writing back nor all the other misc functions that CopyNesW offers.
I'm working on a guide right now that covers how to use it and what everything means, hopefully I'll have it done within a day or two.
Update 3/13:
I'm working on some fairly significant changes to the dumping process, so the guide is on hold until I'm finished.
I just saw your database and I really like it. Dare I ask, is there any way to download a text file summarizing every game's PRG and CHR checksums, or simply all the text fields for each game?
Well...I guess imitation is the sincerest form of flattery.
That being said, and not trying to come off as a complete prick, but we the admin & staff of Video Game Rebirth earnestly hope that the design for your final site is radically changed from its current state for as it stands it is a total and complete swipe of our site design and layout which is not cool from either a technical or legal standpoint. While we understand from the above post that the site design is supposedly "temporary" we will be waiting and watching to see the extent of the redesign.
Aaron "RevQuixo" Arnold
Webmaster Video Game Rebirth
http://vgrebirth.org
I hope BootGod changes the site design ASAP and apologizes, since it's blatant copyright infringement and I'd rather not see the database go down. I noticed that someone started
a thread on Emuversal today and linked to the site, which goes against the fairly private mention and request not to post it elsewhere.
Hello RevQuixo. By no means do I mean to piss you off. It is a blantant rip of your site design as I stated before and it _is_ temporary. I have been taking a break from the site/software for a couple weeks, but I will work on the design next if it bothers you. And by all means if you want me to take it down in the mean time I have no problem with that.
Blargg, the software will support a function to compile an "offline" database, filled with only whatever fields you want in there, so you could easily whip up a small program to do whatever you like with the data. Btw, I love the outstanding work you've done NTSC filter.
Yes, I would prefer that the site comes down until you come up with your own design. As Blargg's last post indicates your URL is spreading outside of this "private" environment and as it stands it is a misrepresentation of "your work" and a fair insult to the hard work we put into our site design.
Just change the page's background color as a one-line fix, and then you can work on making a new design without having to listen to more female dogging.
Website has been put offline for time being. I will look into such things as PHP-Nuke and try to get a functional site up again ASAP. Btw, did you completely build your site from scratch or did you use some sort of system like PHP-Nuke? I know your site is ASP based, something I know nothing about, but I'm just looking for something to start with.
Another temporary "option", if OK with you, would be to post an item on the front page explaining and linking to your site. Everyone visiting the site so far knows this already, but as it has been posted elsewhere now, others may not. Let me know if this works for you for the time being.
Anyways I'm open to any layout/design suggestions, as I said before, I have about as much artistic skill as the keyboard I'm typing on
Well, if you're just looking for something "temporary", then why have it look good? It's temporary. Just take some basig HTML or something, paste the system to it, and put it up until you can HOTDOG it or something. No shame in it looking bad if people visit.
Because honestly noone would have taken interest in it. In its first days, it was just horrendous HTML, white background and black text. It took less than an hour to make it look like vgrebirth, and doing so made it a lot easier for me to concentrate on my software and the functionality of the site.
I've been really pressed for time, and it's going to stay that way for the immediate future it seems. So I haven't been able to work on a redesign, however, I managed to change the existing setup enough where I think it should be suitable to go back online. It still shares some similarities with vgrebirth.org, but I think it's different enough to keep the web admin happy for the time being.
IMO, it's a bit uglier now, but it'll have to do
I found someone willing to work on a new site design not to far down the road, which hopefully should prove to be much better.
how is the database and site comming along?
i had a few request.
1) first is what blargg said, could we download a text file of the database.
2) is it possible to have it accessible like cddb ?
3) and md5sum ?
matt
Well I've been really busy, but things are finally slowing down and i've been able to work on the site a little bit.
To answer your questions:
1) Yes, this is a planned feature. Couldn't give you an ETA, but it will happen.
2) Do you mean have it accessible from other programs like emulators and such? My plan originally was to allow one to download a binary database, containing only the data you choose and to have a simple interface for it. Having it access the database "online" has the obvious advantage of always being up to date, but the disadvantage is that it would be quite slow compared to have a local file like above. For example: from my software, if you scan a folder of files, it takes roughly 1/4 second to process each one.
I will probably just provide both methods and then one can use whichever they wish.
3) I have no plans on using MD5, as it seems unneccesary. If you have a good reason for wanting it though, I would consider it.
BootGod wrote:
3) I have no plans on using MD5, as it seems unneccesary. If you have a good reason for wanting it though, I would consider it.
Any 32-bit hash function is insufficient, as two ROM hacks' hash values are likely to collide. In addition, because CRC of any length is trivial to forge, it's easy for someone to pass off a ROM that displays Goatse (highly not-safe-for-work (link removed, aw c'mon lol -admin)) as any existing ROM. There are several
cryptographic hash functions that you can use. SHA-1 (160 bits) is still good for 126 of those bits as of August 2005.
Out of such a limited data set, the odds running into a collision have got to be virtually zilch unless it was intentional. Considering that all the data comes straight from CopyNES, one would have to go thru some trouble in order to supply their own data. Either one would have to load their data onto a RAM cart and dump it, or hack the program while running. And why anyone would want to do this to begin with, is beyond me
Besides, I was under the impression that MD5 sums could be faked as well...
"Limited data set"? The odds of a collision in a 32-bit hash space increase more rapidly than one might initially think.
Happy birthday.
Intentional
are the problem. Imagine someone who manages to widely disseminate a ROM that displays sick porn (e.g. goatse, bakla, tubgirl, lemonparty) and has the same PRG CRC32 and the same CHR CRC32 as the official ROM. GoodNES (or other tools that use CRC32) won't be able to tell the difference.
True, MD5 is largely broken, but SHA-1 (160 bit) can't be so easily faked as of now. The best collision is 2^63 steps, which means it's roughly as good as a 126 bit hash.
Quote:
Intentional are the problem. Imagine someone who manages to widely disseminate a ROM that displays sick porn (e.g. goatse, bakla, tubgirl, lemonparty) and has the same PRG CRC32 and the same CHR CRC32 as the official ROM. GoodNES (or other tools that use CRC32) won't be able to tell the difference.
And that's only a harmless application of intentional hash collisions. Worse would be hash-preserving modifications that cause glitches in accurate emulators, but otherwise don't significantly change the behavior of the game. Then you'd have people reporting problems that you can't reproduce, even though you are using PRG and CHR data with matching hashes.
By "limited data set", I mean the odds of a collision between valid ROMs, which would be in the order of a few thousand, would be very unlikely.
But if it is felt that intentional corruption is likely to occur, I will go ahead and add it.
I take it you have to run over all the original data again? It seems that it's best to get these hash issues worked out sooner rather than later, because I imagine you will (already?) have submissions from others of just the hash, and not the complete PRG and CHR data. How hard would it be to store all of the popular hashes? CRC-32, MD5, SHA-1 are the ones I know offhand.
CRC32 and MD5 are near useless for file identification nowadays; stick with SHA-1 for now. (SHA-1 is Nintendo approved, as it's used by the Nintendo DS firmware to verify digital signatures on DS Download Play binaries.)
Yes, it would probably be best to go back to the original dumps and SHA-1 them. In fact, it might be better to SHA-1 each 16 KB PRG ROM segment and each 8 KB CHR ROM segment for easier early detection of hacked versions.
blargg wrote:
I take it you have to run over all the original data again? It seems that it's best to get these hash issues worked out sooner rather than later, because I imagine you will (already?) have submissions from others of just the hash, and not the complete PRG and CHR data. How hard would it be to store all of the popular hashes? CRC-32, MD5, SHA-1 are the ones I know offhand.
Yes, it went right to the top of my list. I have already added the functionality to the software/database. Now I just need to write something up to rescan all my (and others) dumps. It shouldn't be to much trouble, a large number of the dumps were from me. I will continue to store the CRC32.
All this talk seems a little extreme, I do 16-bit sumchecks of my PRG and CHR dumps like it's 1985. If you don't have to worry about questionable game integrity, like pirates or porn ROMs, this method works reasonably well and is dead simple to implement on anything. There's a high probability for a repeat sum but I don't have one yet.
Yeah I agree that it seems overkill, I really think CRC32 would be just fine for this project. But since multiple people want it, I've put it in. All ROMs in the DB now have SHA1 hashes generated. For the time being, if you want to view it, hold the pointer over the CRC32 value of the ROM and the SHA1 will display in a tooltip.
But can the SHA-1 value be copied to the Clipboard?
No, I just have to figure out where to shove that string on the page, takes up a lot of space...
Why not put it at the bottom and wrap it?
Is this going to do what NSRT did for the SNES: replace the GoodNES ROM database?
It certainly could do that for North American ROMs. I don't know if the database plans to expand into EU ROMs too, that'd be awesome if it would. Japanese ROMs on the other hand, well you know...
It's set up to work with carts from the rest of the world, just a matter of someone having them and a copynes. I'll be getting another big batch of all new (to DB) US carts soon, which will bring it a lot closer to a complete US set.
It is along the lines of NSRT, with much more extensive info and less "crap". I don't think it will replace GoodNES anymore than NSRT replacing GoodSNES, more of a user preference. In fact, I have been in close contact with Cowering ever since I started this project and have been sharing this data with him. The next GoodNES release will have a lot of [!] in it and corrected headers.
On a side note, I finally got to working on a guide for the software, I may possibly have it up by the end of the day.