I'm guessing such a thing doesn't exist, otherwise there would be an actual iNES->UNIF convertor... But anyways, I have been wanting to make such a utility for ages but just don't have data for it to use. boardtable.txt is the only doc I've seen, but it only has the board name, no mirroring info and the like. Nestopia seems to have a good database packed in, but some of the board names seem ambiguous, for example some are simply called MMC1 rather than the specific board name.
I'm working on a FCEU fork, and I would really, REALLY like to remove iNES support completely and if one were to try and load one, it would bitch and force you to convert. I don't want to go on a rant about it, but the fact that iNES is still around just irritates me to no end.
I'm sure some will not agree with that, but some of the arguments I've heard that keep people sticking to iNES seem ridiculous to me. Like IPS patching not working with UNIF. WTF? Yes it's true that existing tools wouldn't work anymore, but an emu could easily remap the IPS and do a softpatch. Or if you like to do hard patching, creating a program to do so is trivial.
About the UNIF format itself, I of course love it. But I personally would like to see some additions to it. Supplemental info like publisher and year released would be nice. Something else I would really like to do is have an SRAM block and store your save in the same file. I of course would have the option to clear the block out. This wouldn't work if you were running from a read-only source, but I figured if that were the case, it would just create a local copy.
And finally (I know people will hate this idea), I think it would be cool if the UNIF container had blocks for holding images of the cart, box, and manual. Yes this would bloat the hell out of the filesize, but if you don't like that, you could always delete those blocks. I would really like it for manuals to be converted into vector type format. That would make them typically a lot smaller and much more useful. But I know this is unrealistic, wishful thinking.
But back on topic, I'm ready to make a conversion tool but I'm gonna need help on the database. I certainly don't want to be making guesses based off the iNES mapper number. I own ~200 carts so if needed I can have the fun job of opening them all up to get what I need
Kevin Horton seems like he must have (or had access at one time) to damn near every cart. I wonder if he was recording such data for each game.
If I have a PRG file (from cc65's linker), a CHR file (from a bitmap converter or a tile editor), and a board spec (in some format), what is the best tool to make a UNIF file out of them? By contrast, the iNES format needs only 'copy /b'. Such a UNIF building tool would be needed to make homebrews, and it would reduce the scope of your app to creating the board spec.
There exists a "libunif" whose purpose was to simplify the creation and parsing of UNIF files, though it can be somewhat complex to deal with.
For creation of them, I opted for a simpler approach - I wrote a small program, named
MakeUNIF, which takes a small textfile and creates a UNIF file based on the parameters inside. For .NES files, I made a similar utility named
MakeINES.
I've been throwing around a few ideas on the database specifics. I'm planning on using MySQL for the database. I would prefer to write the client interface as a standalone program as opposed to a web-based one. I use Windows, but if I stick with using strict Win32 API, Linux and such should be capable of running it without trouble (actually I would prefer to use MFC, but AFAIK that pretty much limits it to Windows only). Worst case, if one wanted to contribute, but couldn't use the program, they could pass the details along to me to add on their behalf. Also you could technically connect to the DB using any SQL client you please and add the data manually, but that would be a bit of a hassle.
As for the specifics of what the DB would contain, I've come up with this preliminary list. Some of it is not absolutely neccesary or even applicable in some cases but I think is useful nonetheless.
Before each entry, you will have to point to what ROM data your working on. It will allow you to load iNES, UNIF, or raw PRG+CHR. This is so it can calculate CRC data. If you load a UNIF file, it will also pull any other relevant data it can.
Game Name:
The full game name
Board Name:
The complete board name (ex. NES-AOROM-3, NES-NROM-256)
Hardware Onboard:
Chips on board besides the ROM/RAM/CIC (ex. MMCx,LS161)
Copyright holder/year:
Game publisher and year released (ex. Nintendo 1987)
PRG/CHR ROM sizes:
This will be auto-generated
PRG/CHR CRC:
Auto-gen'd
VRAM & WRAM Sizes:
Size of said if present, otherwise leave 0.
Battery:
Flagged if cart contains battery.
System: (See note later)
What system the cart is intended for. (NTSC,PAL,PC10,VS)
Country:
Country of origin.
Mirroring:
Mirroring on cart. (Horz, Vert, Mapper Ctrl'ed, 4 Screen, 1 Screen (there are 2 types of this?))
Catalog #:
The cat number for the cart (ex. NES-XX-USA)
ROM Names:
The ID stamped on the ROM(s) (ex. NES-XX-0 PRG)
Controller Type:
Controller used by game. This one is kinda iffy to me, it doesn't really have anything to do with the cart. Only reasoning for having this one is so an emu could auto-select the proper input device and also as a reference such as ("What games use the zapper?")
Cart Classification:
This is to ID the type of cart, as in "(Un)Licensed, PD, Proto, Pirate"
Any others?
The DB would also store your name/date submitted.
Something else I think would be a rather important item is a "Verified" flag. If all information is complete and you personally dumped the associated ROM data, you would check this off and this would lock the entry from future updates. This would also allow people with carts (and ability to open them) but no dumper, to add entries based on ROMs from elsewhere. Which 99% of the time should be good enough,unless the ROM associated with the entry is a bad/altered dump or a revision.
Back to the note about the system item. It's my understanding that PC10 games have something like an extra 8K? ROM that contains that menu program that runs on the upper screen. Also the VS Unisystem has the various dip switches. Also they have custom palette ROMs. Do you think there should be means to handle this info?
Sorry for the long post. To sum it all up, I'm asking for your insight on the interface idea, any items you think should be added/changed etc.
Game name: would this allow UTF-8, or would it require all titles to be transliterated into US-ASCII or into ISO Latin-1? I'd ask for both.
Hardware on board: NES-EVENT has a DIP switchbank. VS Unisystem has a color ROM. All licensed NES games have a CIC chip; unlicensed games have either a CIC clone (Tengen) or a -5V generator (Camerica, Color Dreams, AVE).
Copyright owner: What happens when a game is subject to multiple copyrights? Example is NBA Jam. Publisher listed on the box is Acclaim; characters and scenario are licensed from NBA Properties; game design is by Midway; program is by Iguana; some low level code libraries are licensed from Nintendo.
Cart classification would have to distinguish programs that have one or more of the following:
- "Pirate" : title screen hacks to remove copyright or change the name of a game, often found on multicarts
- "Graphics Hack" : Mario Baby would be both a pirate and a graphics hack
- "Conversion" : any of the various fighting games based on the TMNT Tournament Fighters engine would be a pirate and a conversion
- "Pirate Original" : game which has an original program but uses copyrighted character likenesses without permission, such as Kart Fighter or Somari or the various ports of Aladdin
Despite the prominence of pdroms.de, I'm rawther not fond of the designation "PD", as its expansion "public domain" implies that all copyright in a work has been abandoned. Some authors give a blanket license to noncommercial distribution of verbatim copies but reserve all other rights under copyright. I'd say "freely redistributable proto" instead.
Well... if you have some patience... go ahead.
Wouldn't a flat file method be better than a MySQL database? It would be much more cross platform friendly and could easily be merged it an app. The number of games and the amount of data is hardly enough to worry about performance in a flat file. Also, if it is in MySQL, doesn't a MySQL instance have to be running on the machine. If it is going to be server based, than a web interface to a MySQL server would work great. If it is a local DB, than a flat file is the only way to go.
Alas, don't count on ANYTHING that has Win32 APIs to run well under Linux. Wine is still lagging in most areas. Standard libs and cross over libs are the only way to keep the fustration to a min.
Also, if you are going to force users to use something different, then you must at least provide a conversion tool.
A great idea though. I am thinking of integrating such a (simpler) DB into my emulator.
tepples wrote:
Game name: would this allow UTF-8, or would it require all titles to be transliterated into US-ASCII or into ISO Latin-1? I'd ask for both.
I was going to use solely UTF-8, but your right, an ASCII version should probably be required too.
tepples wrote:
Copyright owner: What happens when a game is subject to multiple copyrights? Example is NBA Jam. Publisher listed on the box is Acclaim; characters and scenario are licensed from NBA Properties; game design is by Midway; program is by Iguana; some low level code libraries are licensed from Nintendo.
I was considering it to be filled as the publisher only. Or in the case in wasn't a published game, then it would be the creator. I should probably have it labeled as publisher.
tepples wrote:
Cart classification would have to distinguish programs that have one or more of the following:
- "Pirate" : title screen hacks to remove copyright or change the name of a game, often found on multicarts
- "Graphics Hack" : Mario Baby would be both a pirate and a graphics hack
- "Conversion" : any of the various fighting games based on the TMNT Tournament Fighters engine would be a pirate and a conversion
- "Pirate Original" : game which has an original program but uses copyrighted character likenesses without permission, such as Kart Fighter or Somari or the various ports of Aladdin
No problem.
tepples wrote:
Despite the prominence of pdroms.de, I'm rawther not fond of the designation "PD", as its expansion "public domain" implies that all copyright in a work has been abandoned. Some authors give a blanket license to noncommercial distribution of verbatim copies but reserve all other rights under copyright. I'd say "freely redistributable proto" instead.
I see your point, makes no difference to me. We can go by popular vote on that I guess
Omega wrote:
Wouldn't a flat file method be better than a MySQL database? It would be much more cross platform friendly and could easily be merged it an app. The number of games and the amount of data is hardly enough to worry about performance in a flat file. Also, if it is in MySQL, doesn't a MySQL instance have to be running on the machine. If it is going to be server based, than a web interface to a MySQL server would work great. If it is a local DB, than a flat file is the only way to go.
Well I actually started writing this with a flat-file in mind. In fact I pretty much had the basic DB system done. I had initially planned on having this work by each submitter would basically be running this program, and create entries that would have to be sent to me or wherever to be merged into a master DB. But I basically I decided it was going to be easier and much more streamlined to use SQL. It's not a matter of performance, just convience all around. As for the DB server, I was intending on just having it run on a local machine off my own connection, since this isn't something that going to put much load on it. The reasons I'm leary of the web interface: While I want it to be as hassle free as possible, I don't want it to be so easy that any old meathead can come and punch in a bunch of garbage. Not that I suspuct that would be much of a problem. Secondly, I'm not a big VB fan, and the only other way I could go about this would be a VB ActiveX control. I'm sure it's possible with PHP and the like, but I am unfamilar with that. Also this way would require me to be running a webserver as well.
Omega wrote:
Alas, don't count on ANYTHING that has Win32 APIs to run well under Linux. Wine is still lagging in most areas. Standard libs and cross over libs are the only way to keep the fustration to a min.
Yeah I don't use Linux, so I'm not really sure about that. Friends have said they rarely have problems as long as it's Win32 API only. I could of course go to a console based system, but I figured it would be easier for everyone to use a GUI based interface. I'm open to any suggestions.
Omega wrote:
Also, if you are going to force users to use something different, then you must at least provide a conversion tool.
Absolutely. In fact that is the main reason I'm doing this
PHP is VERY easy, and having a webserver (WIN or Linux) is easy and free with Apache. I could provide you with any PHP code you needed to connect to ANY database you want.
I am very interested in this project and I would like to help out. I am very familiar with MySQL, PHP, HTML, javascript, Linux, Apache, Windows, DB design and interfacing. So if you have any questions, let me know.
I do think that you need to make interfacing as easy as possible though, regardless of possible data pollution. DB cleanup is going to happen no matter what. Oh, and don't forget about the Mac and UNIX users out there with that WIN32 API
Omega wrote:
PHP is VERY easy, and having a webserver (WIN or Linux) is easy and free with Apache.
Caution: Most residential Internet access terms of service do not permit running a web server, and even if it does you can't turn off your computer or boot it into a different operating system, so you'll probably have to use some sort of shared hosting plan. This costs per year, as the free web hosting packages generally do not support PHP and MySQL.
I have used PHP and MySQL on several web-server rental sites for pretty cheap ($8.00 - 15.00/mounth). IPowerWeb being one of them. I have also found it to be pretty easy to get web service support from ISPs. As far as not shuting down the computer, well, that's just part of being a server.
Good point though, if you can't run the service, then what do you do.
Yeah, the label "PD" has been misused way too much where "freeware" would be more appropriate. It can be a major annoyance for anyone who releases a game for people to play for free (since people can and do sell collections of "PD" roms).
It also depreciates programs that actually are PD. If you know the saying, when you hear the clock strike 13 times, it also casts doubt on all the strikes that came before, heheh.
Memblers wrote:
Yeah, the label "PD" has been misused way too much where "freeware" would be more appropriate. It can be a major annoyance for anyone who releases a game for people to play for free (since people can and do sell collections of "PD" roms).
I generally don't mind if people sell copies of my games, as long as they include the source code. Heck, if you can port one of my productions from one of the emulators to NES hardware, and you want to stick it on a multicart next to pirated Balloon Fight and all those other pirated NROM-128 classics, go right ahead and put the source code on a CD. It shouldn't make the packaging too much bigger.
Quote:
when you hear the clock strike 13 times, it also casts doubt on all the strikes that came before, heheh.
Unless you're in the
military of course.
Anyway, what's a proper designation for games such as Elite that have been released on a cartridge and then "freed" by the author?
About mirroring, according to the UNIF spec, these are the legal values:
* $00 - Horizontal Mirroring (Hard Wired)
* $01 - Vertical Mirroring (Hard Wired)
* $02 - Mirror All Pages From $2000 (Hard Wired)
* $03 - Mirror All Pages From $2400 (Hard Wired)
* $04 - Four Screens of VRAM (Hard Wired)
* $05 - Mirroring Controlled By Mapper Hardware
I'm wondering though, other than boards that have the H/V select pads, mirroring can always be implied by the board name can't it? If so, it seems unnecessary to make that distinction and the [MIRR] block would only be necessary if the board has the solder pads and only have a value for H and V.
That is correct - the UNIF "MIRR" blocks is ONLY relevant if the board (described in the "MAPR" block) has solder pads on it to select hardwired mirroring. Many [non-Nintendo] cartridges are simply hardwired to horizontal or vertical mirroring (in which case the board name would imply it), while there are also a few Nintendo boards (MMC3, possibly MMC1 as well) which have solder pads to select between Horizontal, Vertical, and Mapper-controlled.
The fact that there are two different "hard-wired single-screen" settings in UNIF is rather stupid, since if it's hard-wired you'll NEVER be able to tell which half you're using.
So as far as the database is concerned, one should only need to be given the options Horizontal, Vertical, and "Board Implied".
As for how the database handles ROMs, the way it is setup right now, each entry has a sub-table for each ROM contained on the board. The sub-table has the following items:
Name (eg. "HVC-EB-0 PRG")
Size
CRC32
Type
The "type" item is a string which tells what the ROM is.
A typical entry would have 2 of these sub-tables, with types "PRG0" and "CHR0" or just "PRG0" in the cases where the board has VRAM instead.
The problem is though, what do you do when you have odd ROMs that don't fit into those categories. Specifically I'm thinking about PC10/VS boards where you run into palette ROMs and those PC10 menu ROMs. Do you set up more predefined types or should the type field be totally user-defined?[/list]
VS boards do not have "palette ROMs" that can be easily read - they have varying revisions of the RP2C0x PPU chip with different on-die palette tables.
However, Kevin Horton (aka kevtris) has a "3-in-1" board which he designed to read the palette out of a PC10/VS Unisystem PPU through an ADC (as well as monitor the PPU data/address bus and monitor the 'chatter' between two CIC chips), so if you have any VS PPUs he doesn't have, let him know so he can have them 'dumped'.
Ok another interesting situation I've run into. What would be the best way to handle a game that uses multiple boards but the same data on the ROMs? I haven't even opened many games, and i've already run into this twice. TMNT and Metal Gear. These are a couple games I happen to have multiple copies off where this has occured.
Taking TMNT for example: One of them has the board NES-SLROM-05 and one of them has a board with just the number 351908 on it. The latter also has an LS7432 onboard, whereas the SLROM does not. Also the SLROM has a 32-pin VRAM chip and the other has a 28-pin chip.
This doesn't really pose a problem for the db at all, I'm just wondering what the best way to handle it for example when a user tries to convert to UNIF. Do you give them a choice of available boards or just automatically pick one? I'm sure functionally of the boards is the same, so it's probably not a big deal. Just figured I'd check
Quote:
Ok another interesting situation I've run into. What would be the best way to handle a game that uses multiple boards but the same data on the ROMs? I haven't even opened many games, and i've already run into this twice. TMNT and Metal Gear. These are a couple games I happen to have multiple copies off where this has occured.
Taking TMNT for example: One of them has the board NES-SLROM-05 and one of them has a board with just the number 351908 on it. The latter also has an LS7432 onboard, whereas the SLROM does not. Also the SLROM has a 32-pin VRAM chip and the other has a 28-pin chip.
This doesn't really pose a problem for the db at all, I'm just wondering what the best way to handle it for example when a user tries to convert to UNIF. Do you give them a choice of available boards or just automatically pick one? I'm sure functionally of the boards is the same, so it's probably not a big deal. Just figured I'd check
First of all, its VROM, not VRAM. Nintendo had some odd 28-pin 128x8 kilobit ROMs floating about and used the above configurations to use them instead of the more common 32-pin 128x8 kilobit ROMs. The TMNT boards are functionally identical, at least to the NES. Always pick the Nintendo board, no need to clutter up UNIF with weird, unnecessary board types.
I disagree, the entire point of UNIF is to *accurately* describe the game that it's a binary representation of.
If you're worried about emulators, then you can do a number of things:
1) Use the 'common' board-type as the UNIF board name, and include in a seperate chunk the true board type (not my preference, but works instantly with current UNIF emulators) or
2) Do it the other way around, listing the true board name in the board name field, which is WHAT IT IS THERE FOR, and have a seperate 'compatible-with' chunk, which would list acceptable board name substitutions.
Hey guest... Calm down about the 'emulators'. Even if you forget them, the NES cannot read UNIF. ^_^;; The storage purpose is to give the binary data information (like you said), but emulation is not prohibited and not something that doesn't matter after all. Well, that's what I think about it. It's a nice format.
Fx3 wrote:
Calm down about the 'emulators'. Even if you forget them, the NES cannot read UNIF. ^_^;;
Some people are under the impression that should we ever get a
Star Trek style matter replicator, you should be able to give the replicator a UNIF file and get a mint-condition Game Pak out.
I have no problem with things being easy to implement for emulators, but I don't see why having extra information lying around is going to hurt anything, and it may help in the future. UNIF isn't just for making emulator authors lives easier, it isn't for making ROM whores lives easier, and it's not just for perfectly preserving everything about a ROM. It's a little bit of all of them, and it lacks something for all of them too. At least it's not iNES, and I think we can all agree that is a good thing.
Combining boards that are either totally identical, or differ only by preventing bus conflicts or inverting ROM select signals for different pinout ROMs is one thing, and I don't mind so much about that (although it would still be nice to list the different board types that this particular CRC of PRG came from, etc...)
However, some people have proposed things like getting rid of any notion of NROM and storing them as CNROM or GNROM or whatever. That's both incorrect, and saving space for no reason (i.e. just silly). Having one extra type doesn't hurt anything, and what about some odd/rare/badly written NROM game that writes to $8000+? The correct behavior is to ignore it, and you'd know that if you had it marked as NROM, rather than latching some bits for no reason (granted, the bits should do nothing if the emulator properly pages only the amount of memory indicated by the file info).
I agree 100% with 'Guest'.
What's the point of keeping an exact format if we're not going to keep it exact. If we start omitting differences that (in the present) we feel are unimportant, we'll be repeatnig the whole iNES fiasco all over again.
Plus things can change. Despite the differences we see between MMC3 and MMC6 now, back in the day they didn't feel it important to distinguish between the two. Are you really suggesting we take that same road? That'll just cause more problems down the line.
Great Hierophant wrote:
First of all, its VROM, not VRAM. Nintendo had some odd 28-pin 128x8 kilobit ROMs floating about and used the above configurations to use them instead of the more common 32-pin 128x8 kilobit ROMs. The TMNT boards are functionally identical, at least to the NES. Always pick the Nintendo board, no need to clutter up UNIF with weird, unnecessary board types.
First of all, excuse the typo
About the number of pins, I was under the impression the PRG ROMs could have up to 128KB in a 28-pin chip whereas the CHR ROM could only have up to 64KB with a 28-pin due to the way it's generally wired. I suppose the board just wires it different than normal, I didn't look too close. Anyways back on topic, I also agree with 'Guest' about the board names and I intended to have the DB take both without hassle. I was more wondering about how deal with it on the other side, when a user (or emulator) is dealing with it.
A little status update on the project for anyone that may care
I haven't had a ton of time to work on it, but it's coming along pretty good regardless. Most of the major functionality is done. It's down to mostly little things/polishing now. Some of the main things that still need to be done:
-Add UNIF read/write support. ATM, it only reads iNES files.
-Database file download. This feature will create a local copy of the DB, containing whatever data you specify. I'll keep the file format simple so it's easy for other programs to use.
-I still have to do the whole user registration/permissions junk.
-It doesn't really have proper support for PC10/VS systems yet, as in there are no means to add info about extra things like dip switches.
If I have time later tonight, I'll post the database table details. I think it's good where it's at, but I'd like some opinions on it.
Well I finally got some time today to work on this and it's pretty much ready to go now. Still have plenty to do with it, but all the important stuff is done. Tonight hopefully I can go online with it and get some testing done. I still have a ton of entries to add, but I wont let that hold it back.
There is one thing however I'm unsure how to go about regarding board names. As it is now, your supposed to enter the entire board name as it appears. Most US boards have a common name plus a revision number. So when it comes to saving it to UNIF, it looks for a revision number and drops it. That's fine and all, but then there is everything else. For unlicensed games it will prepend "UNL-" to whatever was entered. For pirates/multi-carts I really have no idea, because it seems most do not even have board names and are just made up.
So I'm thinking I need to add an additional item for "UNIF board name" along with the real board name. Even for licensed games this may be necessary. Example, I have 2 Pinball carts. One is a normal NES-NROM-128-04 board. The other is the japanese glob-top board HVC-PN with a NES-JOINT CIC pass-thru thing. But if you were to use "HVC-PN" for the UNIF file, I don't think there is an emulator out there that will accept it.
One more thing, are there not any emulators out there that support Tengen,Color Dreams,AVE,Camerica,etc in UNIF form?
Ok I finally got time to do some online testing for this thing with a friend and it seems to be good enough for a public beta. It still lacks some features like a downloadable DB, but that isn't neccessary at this time.
You will need Win2k or WinXP to run this. It hasn't been tested on 2K but it should work. It requires unicode support as well. My friend wasn't able to get unicode chars to display properly, I'm not really sure why, as his browser was able to display them.
And of course to be useful you will need to have some NES carts and the means to open them. I've got about 80 entries in so far, only about 120 to go
Go
here for the download. There is a separate download for a couple dll's you may or may not need. See the readme for more info