DotFami

DotFami is not a emulator. It is a file format I am designing (which could be supported by some emulators in future), which among other things, allows you to specify large amounts of ROM, many kinds of metadata (including descriptions of VS System switches), specify NTSC/PAL/Dendy/RGB, and emulate custom mappers (including audio). DotFami-compliant emulators may still load iNES, UNIF, NSF, and NES 2.0 files; if so, one way to do so is to use the "ines.map" and "unif.map" files to select the mapper. You could also use JIT compiler to compile the mapper codes into native codes to improve the performance.

If you write emulator and/or make custom mappers, then maybe you can help.

(If this is wrong area, please move it. I think this is correct area, though.)

Since you want the format to define everything under the sun, how is this format better than any XML format which is actually extensible, and emulators can support as little or as much of the crap bundled with the ROM as they want? Sure your format encompasses a lot (whether this is good is debatable), but it's very static and rigid. It looks like a nightmare to implement, so have you taken a poll of who will use it before you invest much more time defining it?

I am still in progress of it. And this format is actually extensible. Some of the things in there I have, I intend to remove, and I intend to add some things as well. I do not intend to encompass everything; but it is intended to encompass a lot including custom mappers (one of the most important parts of this specification).

I do not intend it to define absolutely everything under the sun, and not everything is necessary for an emulator to implement; an emulator might completely ignore the symbol tables and that is OK. So some of the things are not necessary for an emulator to support; and even then, a partial implementation is possible. The format can be parsed without too much difficulty in order to ignore parts you do not understand or display error message if not understand; that is, you can easily find out what part is ROM, what part is symbol tables (and to split it into the commands), and what part is mapper codes (and to split it into the commands). Although not recommended, an emulator may even ignore the entire CPU mode flag byte, or may only support some of them. Even then, the user should be allowed to change some of these options such as TV mode within the emulator without having to edit the file. And as the document says, it is also OK if an emulator does not support panning; it is there in case you want it.

So it is not static and rigid; it is extensible: Some bytes of the header are reserved and set to zero (so could be used for later use). New commands can be defined for mapper codes and symbol tables. Parameters can be added to commands in the mapper codes (the length of a command is always specified, so a parser will know what to skip). If there is ROM data that no mapper codes refer to, that part can be safely ignored.

Yes, it could be difficult to implement. It could even be a separate program which compiles the mapper codes into LLVM and then from LLVM into native code, if that helps; the specification does not require the exact way in which the files are used with any given emulator. Some emulator might even compile the mapper codes into some hardware description language (such as Verilog or VHDL); it does not even have to be software.

However, there are some requirements: For example, the DotFami specification specifies that the iNES mapper number -1 (negative one) is assigned to .NSF files. But it is still possible for the emulator to actually use a different number internally or treat it entirely different in some other way, as long as the operation of the program is still working OK.

Of course, emulators do not have to support DotFami at all; it could be an external program which converts it to NES 2.0 or UNIF + a program module (to support the mapper) which is static linked or dynamic linked to the emulator.

I do plan to write some public domain C codes for things related to DotFami format (such as to split into .rom/.cart/.symb and to combine them into a .fami, including specifications of external parameters, as well as some other things), so this could help a lot. Another thing I intend to write is a Haskell library which allows compiling to DotFami .cart files. There is a way to include literate Haskell codes in a MediaWiki page (I have done it on esolang wiki), and a way could also be made to include the codes of files imported by ines.map in a MediaWiki page, so that all mappers can be precisely defined in this way.

But why describe the mapper in a binary format instead of something extensible like JSON or XML? Anything that can compile things to LLVM bitcode can parse JSON or XML.

The ROM data is binary. But I will make text formats such as a Haskell library to create .cart and .symb files; so a full Haskell program using all features of Haskell can be used instead of having a limited preprocessor.

The ROM is binary but everything else is metadata, and since you have SO MUCH metadata, it's time to use a generic container. Using a generic container will mean that controversial areas of your specification can be safely ignored. Personally I don't even understand why your format handles iNES and UNIF (which never caught on), or why the half-width katakana which is little different than romanizing Japanese into ASCII anyways.

I think the mapper idea is inconsistent since you have a few limited 74 series, but can use ASIC and magical sound chip entities. Are people expected to make their own discrete latches, registers etc? Do you know how difficult it is to write a logic simulator that is able to simulate a mapper and sound chip from discrete gates? It'd be essential to simulate propagation. For sound what about all the analog characteristics from the DAC, filters, hybrid chips etc? Few if any emulators in circulation today even emulate at the bus-level.

The iNES format already does what needs to be done for 99% of licensed or non-pirate titles. There is a small minority of titles that currently need hacks or hash checks to identify to run properly such as Startropics (MMC6) or various alternate MMC1 boards. iNES 2.0 should handle this. The main problem is that the ROM sets people have and that are still distributed have the old iNES 1.0 header and so until their ROMs are updated with the new header, iNES 2.0 can't help. And so we are back to hash checks or other detections. But for homebrew and hacks it shouldn't be as much of a problem as they will be newly distributed and can have a iNES 2.0 header and avoid these issues.

Every now and then this format discussion comes up and nothing productive ever comes out other than I guess iNES 2.0.

I mentioned this in B00daW's thread about no-intro thread but it's relevant to DotFami. Just a few thoughts on this:

When you make a spec like this that allows radical customization, it can become succeptible to unexpected abuse/conflicts. There are likely going to be exotic combinations allowed by the spec that will be underdefined or unimplementable, and may expose differences between different emulator implementations, etc. The example I'm reminded of is the NSF spec versus multi-expansion. A lot of people really like this feature, but there are issues like the FDS memory conflicts which are implemented differently in every emulator and create a headache for people that want a widely supported NSF. DotFami seems wider reaching than this, and could open a big can of worms.

As a programmer, I actually like the idea of being able to construct a mapper from generic building blocks, but only thinking of it as an emulator implementation. Like, instead of writing separate code for every mapper, I just have a table of mapper components that I look up and attach to some virtual mapper interface. It might be a sensible design for implementing mappers internally, but I'm not sure I like it as a file format.

If I was a ripper, I would want to choose the correct mapper, and a situation like this is much more error prone than just choosing the right iNES mapper number. Sure it would be nice to be able to fit in edge cases that we currently can't handle with iNES, but for everything else that's already handled you're introducing 100 other settings that could be wrong in the header. (People seem to have enough trouble getting the few settings in iNES correct.)

As a user, I generally prefer single-file formats for my game collection.


Anyhow, as usual I think the practical application and real implementation is the important thing. Get this into an emulator. Create some rips. Show use how it's useful. Maybe it'll be great.

I put a note about the katakana so you should only use ASCII; it is likely to change much. If a generic container i sused, something like ZIP would probably be better since it can contain binary data. However, I think the format as it is now is OK, and like I have said before you can safely skip over much of it, and it is easy to skip over parts of the format which is not understood because a length is included with each command. The DotFami format itself does not handle iNES (including NES 2.0) and UNIF; these are optional features, supported by external files ines.map and unif.map, which is not required for an emulator to use anyways (the ines.map and unif.map files could even be parsed by external programs; the emulator does not need to know anything about them). Another thing that could do, is an emulator might have its own support for many iNES mappers, and if it finds a unknown mapper number to convert to DotFami using ines.map, and could also directly load .fami files.

Note that I have greatly changed over time; at first I listed the 74 series and ASICs but it is different now (look at it if you want to know). Yes you can make your own discrete latches, registers etc, although I intend to add some things to make it easier in some ways. For sound, I did recently add the plain digital-to-analog converter, and I could add filters if it is wanted too. I know they don't emulate at the bus-level, and how it can be difficult. It is why I suggested compiling into native code according to what the emulator requires (LLVM could be used, perhaps, if it helps), and make open source programs, it can reuse code in other programs since it is open source. Using interpreted simulations would be slow, although you could use it if you want if nothing else is available. For computers having CPLD and/or FPGA, that could be used too.

Yes, NES 2.0 is better format than 1.0, and I do also try to work on making a suggested list of submapper numbers for disambiguation (the official workers of NES 2.0 format can reject my suggestions if they want to, of course). While NES 2.0 can help many of these things, it cannot support custom mappers, unless you assign a mapper number. Also with DotFami, it may allow to specify mappers which are too modern for an emulator to support if the new mapper numbers are assigned after the emulator is written. This is what the ines.map file is for. And for experimental purposes, you may not want to assign any mapper numbers at all, so you will use .fami directly.

I have added support for NES 2.0 to Unofficial MagicKit (some people suggested making NES 2.0 the default, but there are some reasons why I refused), and intend to add support for UNIF and DotFami as well (DotFami format is currently written enough that this could be done; the assembler needs only understand $FE and $FF mapper commands, and can write the rest to output directly).

This is the reason why I try to specify explicitly what happens in these cases, and what cases explicitly are defined as undefined behavior.

In case you do not like it as a file format, that is also why ines.map exists and using such things as an emulator implementation you could compile ines.map and the *.cart files into a C code in the emulator, if that is what you prefer.

To choose the correct mapper, one thing you would need to do is to figure out what mapper it uses (which could be partially auto-detected, and I may make something like this in future). I am introducing many settings but most of them are optional anyways, so you do not need to use them. For ripping, I would prefer to make separate files, and then combined into iNES and/or DotFami format.

It works regardless of what format you use.

Agreed. I cannot create any rips, but I may create some files that use custom mappers (since if I wanted to make something using custom mappers, I would certainly do this). For the standard mappers, the .cart (or programs which compile into it) and ines.map/unif.map could be created too.

I mean a container for the metadata. Zip can contain the binary and metadata, but XML (or JSON/whatever) are forever extensible, and can hold all your metadata types that you have set for separate files.


I don't get it. So you expect emulators to ONLY support your format, and the format assists in adapting iNES to DotFami?


Are you going to add one for each bit depth? What about non-linear DAC as seen in the AY-3-8910? If you're going to let people make their own hardware, you might as well include SPICE because anything can go...

Yes it could be, I suppose. I chose not to. We could define the text mode too if it seems important to many people.

Only for emulators which do support DotFami and do not have their own support for iNES; I do not expect all emulators to support DotFami, although those that do, do not necessarily have to support iNES directly (and if they do support both iNES and DotFami, they may use ines.map for conversion, or may use only for unknown mapper numbers, or not at all, depending on the individual emulator).

The bit depth is known by the number of input pins used, similar to how I have the other components defined. For non-linear, I could make it parameters.

Why exactly do we need custom mappers? Aren't there more than enough mappers already?

Hardware hobbyists like to invent new mappers, and currently the only way to emulate them is by modifying existing emulators, which is not exactly trivial, and even if it was, who would want hundreds of variations of different versions of the same emulator floating around, the only difference being whether they can run certain homebrew ROMs?

One of the problems with this community is that a lot of people only care about what already exists, and don't even bother thinking about new possibilities. New mappers may seem like overkill when most homebrews are either NROM or UNROM, but unless the mappers are out there, nobody will use them. There are a few MMC3 homebrews out there, showing that there is a demand for more advanced mapper features (scanline counters, fine CHR bankswitching, WRAM), but using the original Nintendo chips or replicating them perfectly may not always be the best options for various reasons (availability, FPGA/CPLD cost, etc.), hence the need for custom mappers.

There are a lot of talented people in this community, who are as good (if not better, since we've had much more time to study the NES than they did) as people that worked with the NES during its commercial era, and I believe that there's still a lot to be done for the NES, both in hardware and in software, but we have to consider the possibilities now in order to make these future developments easier.

Any mapper as advanced as MMC3 or more is going to require a CPLD if it's going to be commercially sold and thus if you can design the mapper in a CPLD I would think you could handle adding support to an emulator for whatever testing purposes. If not, I'm sure someone could help you and would want to if you can actually make the CPLD code and PCB design for it. I don't think that any "define your own mapper" sort of thing is going to work out very well.

What if in addition to selling cartridges I want to make the ROM available as well? If emulator authors don't feel like officially supporting my mapper will I have to distribute my altered version? I would like people to be able to run my ROMs without having to give them complicated instructions or making them download more stuff, and that would only happen if custom mappers become common to the point where most current (as in still being updated) emulators will support it.


Any particular reason?

What you can do is take an emulator, reskin its UI for things like controller config to look like that of your game, bundle it with a ROM, and call it "the PC version" of your game.

I just don't see anyone ever having the freedom to define cartridge hardware in some sort of file data provided with the ROM in a way that would really work. How exactly would you define the MMC5 in magic mapper file terms? Are we supposed to emulate the NES on a circuit level now or something?

I don't know, especially since I don't know why writing to ExRAM in nametable/attributes mode only works while rendering. But once I write more of the DotFami specification, I intend to try!

At least partially I suppose you would have to if you want your emulator to support DotFami, or have an external program to convert the circuit level description into a native code.

(Unrelated: When I make some computer, I also intend to make the NES/Famicom emulator which can emulate by file (possibly including dynamically programming an FPGA) and by cartridges connected to an adapter to the external memory port. This is not really related except it shows at least one way which it can worked.)

I never wrote an emulator before, but if I were to write one I imagine I would try the approach of emulating everything cycle by cycle and interfacing with the "cart" through the signals that are actually present on the cartridge connector. That way mappers would be able to do anything they wanted with the signals and respond accordingly. I know this is not very efficient, but current computers should be able to handle that, right?

Write an emulator and I think you'll find the idea of adding on some mapper file emulator unappealing. It would be horribly inefficient. There are an infinite number of configurations mappers could have. Look at all the pirate multi-cart mappers. The Mapper #90 used by all sorts of weird pirates. MMC2's CHR behavior. Konami VRC's cpu based scanline counter. Other various scanline counters. Awhile ago someone was proposing mapper hardware to try to detect when DMC interferes with reading joypads. And ofcourse look at MMC5 which still probably isn't fully understand from the hardware level.

Good luck with your format project.

Well, I don't have a format project at all, I was just saying that it would be really cool if the mappers could easily be bundled with the ROMs, much like happens with real cartridges. I know that this wouldn't be easy, though.

I didn't mean you as far as this project. While I agree that it would be cool, I just don't think it is feasible.

Game projects, like yours, don't necessarily need any crazy new mappers. I'm pretty happy to see InfiniteNesLives getting these new MMC3 compatible boards ready. Just look at all the great NES games that just use discrete logic, then put MMC3 games on top of that and you have most of the library. So homebrew creations now have plenty of options open to them.