News:
The proposed NSF2 outlined below is now implemented in NSFPlay 2.4 beta 9, available here:
https://github.com/bbbradsmith/nsfplay/releases
Original proposal below:
So, about 8 years ago kevtris openly speculated about an "NSF 2.0" format that would update the existing NSF format with a few goals in mind:
1. Practical ways of using the IRQ and/or NMI as interrupts.
2. Track times and names, or other non-essential metadata.
I didn't want to continue that thread, as basically nothing ever came of it. Quietust made a Battletoads RIP that relied on the idea of a "non returning INIT that gets interrupted by PLAY", but I think it predated the NSF 2.0 proposal, and wasn't really relying on the whole IRQ idea, just an INIT timeout.
So... now that I'm actively working on some NSF stuff again, I like some of the ideas in that proposal, and I want to make this stuff "real" finally. Here's what I propose to do:
1. Metadata
Kevtris' proposal used the last 3 bytes of the NSF header (previously reserved as 0) to indicate the length of NSF data that follows the header. This means that any metadata appended goes right after the other data. This is good, and I don't think it even requires a version change; it's backward compatible with NSF 1. Any nonzero value in the last 3 bytes should be interpreted as a data length, and indicates the presence of non-essential metadata. (We don't really have the DISKDUDE! problem with NSFs that we did with iNES; the headers are generally very conformant in this respect.)
The other question is what the metadata should look like. Kevtris had some ideas about them, but they weren't very firm. It was some chunked format with basically whatever fields he could think of at the time. My counter to this is: we already have a chunky metadata format for NSFs, called NSFe.
NSFe was Disch's idea, implemented about 15 years ago for his NotSoFatso NSF player. It has always been very extensible (unlike NSF 1), but it never received widespread adoption. Several emulators do support it, though, and I've found a lot of use for it over the last few years. It's existing metadata formats are reasonably parsable by the 6502, IMO. (Emulators can also reuse their NSFe implementation to support both, which is helpful.)
So, in light of this, I see no reason to create a new competing metadata standard. We might as well just merge these two. There's no fundamental incompatibility between the two ideas. So, what I've decided to go with is just putting NSFe chunks (as defined by the NSFe spec) at the metadata location. This does not inclulde the "NSFE" header (unnecessary), and the "INFO"/"DATA"/"BANK" chunks that are normally mandatory in a .NSFe file must be omitted here.
To try this idea out I have implemented exactly this for my most recent beta of NSFPlay:
https://github.com/bbbradsmith/nsfplay/releases/tag/2.4b5
To additionally help test it, I've created this python tool that easily converts NSFe to this NSF + metadata format. Try it out:
https://gist.github.com/bbbradsmith/4bc17ae16b10a9be03e80addfbea5009
Edit: and for the purists, here's a python script to strip that metadata:
https://gist.github.com/bbbradsmith/e225068e750fdb32e59a191a4f6a7e45
I see no reason to increase the version number just for metadata. There is no backward compatibility issue in simply treating a value other than 0 in the last 3 bytes of the NSF header as an indicator for the presence of metadata.
Header change:
(Note that the appended data will become part of the ROM image on old players. NSFs should not be relying on the presence of 0 outside their data areas anyway, though, so this is generally safe, and easily fixed with explicit padding in the file.)
2. Non returning INIT, IRQ / NMI etc.
This part of kevtris' proposal was a bit more solid. I have not implemented any of it yet, but I plan to keep most of it. Here's some plans:
1.1. Non-returning INIT is a problem. We need a way to signal to the player hardware when we're done starting up and ready to receive PLAY / IRQ interruptions. (Quietust's implementaion of this is just to "time out" the INIT routine after some large number of cycles. I think GME ended up borrowing this implementation. This worked, but I think we really need an explicit non-arbitrary moment of synchronization here.)
Instead my thought is: the non-returning INIT flag means that INIT will be called twice. The first call is still required to return,but can set RAM variables to tell itself that it's the second call. (The second call does not have to return, of course, as per the point of this.) Edit: The first and second calls will have the argument Y=$80 and Y=$81 respectively to make it very easy to branch to the non-returning code.
1.2. With non-returning INIT, PLAY becomes an "interrupt" that is intended to be driven by the NMI. A hardware player with no PPU can generate an NMI externally, and the player will likely use some sort of wrapper in between NMI and PLAY but that's up to the implementation. PLAY should still RTS, not RTI. The actual timing of PLAY vs the second call of INIT will necessarily be implementation-defined. (Edit: the NMI wrapping PLAY when this feature is used implies an SEI for it that will be reverted after PLAY returns via the wrapper's RTI. This should be fine, but if using IRQ features at the same time it should be considered. Also: the wrapper should disable its own NMI during execution of PLAY to prevent re-entry.)
2.1. IRQ handling: this I want to keep exactly as kevtris proposed. $FFFE-FFFF becomes a RAM overlay, and the NSF code gets direct IRQ control through it, no intermediary. CLI/SEI is explicitly allowed, of course. The DPCM and APU IRQs are also fair game.
2.2. IRQ timer: again, I think kevtris' proposal for an additional IRQ timer is solid, but with one minor change later proposed by B00daW to change the address of its interface to avoid the "test" registers that were discovered later.
There's more detail about this on the wiki but basically I think the IRQ stuff is good as proposed.
3. NSF2 header change:
This is almost what was in the proposal, *except bit 7 was proposed to indicate the presence or absence of metadata. I think it was redundant for that purpose (non-zero $7D-7F already achieves this), but I think we could use it to indicate instead that there is some essential chunk in the metadata that is critical for correct playback.
This basically works like NSFe, in which any chunk FourCC starting with an uppercase letter means it must be parsed to correctly play the file. This allows the NSFe extensions to work as that format intended, and also gives appropriate places for things like, e.g. sample data chunks for new expansion sound devices that need it.
Basically the presence of the version 2 indicator should only be needed for using the non backward compatible features, which are all encapsulated be $7C. Non-essential metadata can be freely included as version 1.
(FWIW, I couldn't find any players that even checked the version number. )
Anyway, that's what I'm working on for this. Part 1 is already more or less implemented (see above). Part 2 will take more work, but it's on the way. If you think I've made a critical mistake somewhere, let me know before the world turns to mud.
Edit: Replaced Y init spec with $80/$81 for non-returning INIT sequence that's distinguishable from the usual default.
The proposed NSF2 outlined below is now implemented in NSFPlay 2.4 beta 9, available here:
https://github.com/bbbradsmith/nsfplay/releases
Original proposal below:
So, about 8 years ago kevtris openly speculated about an "NSF 2.0" format that would update the existing NSF format with a few goals in mind:
1. Practical ways of using the IRQ and/or NMI as interrupts.
2. Track times and names, or other non-essential metadata.
I didn't want to continue that thread, as basically nothing ever came of it. Quietust made a Battletoads RIP that relied on the idea of a "non returning INIT that gets interrupted by PLAY", but I think it predated the NSF 2.0 proposal, and wasn't really relying on the whole IRQ idea, just an INIT timeout.
So... now that I'm actively working on some NSF stuff again, I like some of the ideas in that proposal, and I want to make this stuff "real" finally. Here's what I propose to do:
1. Metadata
Kevtris' proposal used the last 3 bytes of the NSF header (previously reserved as 0) to indicate the length of NSF data that follows the header. This means that any metadata appended goes right after the other data. This is good, and I don't think it even requires a version change; it's backward compatible with NSF 1. Any nonzero value in the last 3 bytes should be interpreted as a data length, and indicates the presence of non-essential metadata. (We don't really have the DISKDUDE! problem with NSFs that we did with iNES; the headers are generally very conformant in this respect.)
The other question is what the metadata should look like. Kevtris had some ideas about them, but they weren't very firm. It was some chunked format with basically whatever fields he could think of at the time. My counter to this is: we already have a chunky metadata format for NSFs, called NSFe.
NSFe was Disch's idea, implemented about 15 years ago for his NotSoFatso NSF player. It has always been very extensible (unlike NSF 1), but it never received widespread adoption. Several emulators do support it, though, and I've found a lot of use for it over the last few years. It's existing metadata formats are reasonably parsable by the 6502, IMO. (Emulators can also reuse their NSFe implementation to support both, which is helpful.)
So, in light of this, I see no reason to create a new competing metadata standard. We might as well just merge these two. There's no fundamental incompatibility between the two ideas. So, what I've decided to go with is just putting NSFe chunks (as defined by the NSFe spec) at the metadata location. This does not inclulde the "NSFE" header (unnecessary), and the "INFO"/"DATA"/"BANK" chunks that are normally mandatory in a .NSFe file must be omitted here.
To try this idea out I have implemented exactly this for my most recent beta of NSFPlay:
https://github.com/bbbradsmith/nsfplay/releases/tag/2.4b5
To additionally help test it, I've created this python tool that easily converts NSFe to this NSF + metadata format. Try it out:
https://gist.github.com/bbbradsmith/4bc17ae16b10a9be03e80addfbea5009
Edit: and for the purists, here's a python script to strip that metadata:
https://gist.github.com/bbbradsmith/e225068e750fdb32e59a191a4f6a7e45
I see no reason to increase the version number just for metadata. There is no backward compatibility issue in simply treating a value other than 0 in the last 3 bytes of the NSF header as an indicator for the presence of metadata.
Header change:
Code:
header byte $7D-7F:
3 bytes - 24-bit size of data following header (little endian)
3 bytes - 24-bit size of data following header (little endian)
(Note that the appended data will become part of the ROM image on old players. NSFs should not be relying on the presence of 0 outside their data areas anyway, though, so this is generally safe, and easily fixed with explicit padding in the file.)
2. Non returning INIT, IRQ / NMI etc.
This part of kevtris' proposal was a bit more solid. I have not implemented any of it yet, but I plan to keep most of it. Here's some plans:
1.1. Non-returning INIT is a problem. We need a way to signal to the player hardware when we're done starting up and ready to receive PLAY / IRQ interruptions. (Quietust's implementaion of this is just to "time out" the INIT routine after some large number of cycles. I think GME ended up borrowing this implementation. This worked, but I think we really need an explicit non-arbitrary moment of synchronization here.)
Instead my thought is: the non-returning INIT flag means that INIT will be called twice. The first call is still required to return,
1.2. With non-returning INIT, PLAY becomes an "interrupt" that is intended to be driven by the NMI. A hardware player with no PPU can generate an NMI externally, and the player will likely use some sort of wrapper in between NMI and PLAY but that's up to the implementation. PLAY should still RTS, not RTI. The actual timing of PLAY vs the second call of INIT will necessarily be implementation-defined. (Edit: the NMI wrapping PLAY when this feature is used implies an SEI for it that will be reverted after PLAY returns via the wrapper's RTI. This should be fine, but if using IRQ features at the same time it should be considered. Also: the wrapper should disable its own NMI during execution of PLAY to prevent re-entry.)
2.1. IRQ handling: this I want to keep exactly as kevtris proposed. $FFFE-FFFF becomes a RAM overlay, and the NSF code gets direct IRQ control through it, no intermediary. CLI/SEI is explicitly allowed, of course. The DPCM and APU IRQs are also fair game.
2.2. IRQ timer: again, I think kevtris' proposal for an additional IRQ timer is solid, but with one minor change later proposed by B00daW to change the address of its interface to avoid the "test" registers that were discovered later.
Code:
$401B - low 8 bits of 16-bit IRQ timer reload
$401C - low 8 bits of 16-bit IRQ timer reload
$401D - bit 0 controls the IRQ (0 = held in reset, continually reload timer, 1 = enable)
$401C - low 8 bits of 16-bit IRQ timer reload
$401D - bit 0 controls the IRQ (0 = held in reset, continually reload timer, 1 = enable)
There's more detail about this on the wiki but basically I think the IRQ stuff is good as proposed.
3. NSF2 header change:
Code:
header byte $05:
byte = 2 - version 2 - indicates we must interpret byte $7C.
header byte $7C:
bit 0-3 - reserved, 0
bit 4 - IRQ features enabled
bit 5 - Two INIT calls, second not required to return.
bit 6 - Disable PLAY calls (not very useful without bit 5 also set)
bit 7 - metadata contains a critical chunk*
byte = 2 - version 2 - indicates we must interpret byte $7C.
header byte $7C:
bit 0-3 - reserved, 0
bit 4 - IRQ features enabled
bit 5 - Two INIT calls, second not required to return.
bit 6 - Disable PLAY calls (not very useful without bit 5 also set)
bit 7 - metadata contains a critical chunk*
This is almost what was in the proposal, *except bit 7 was proposed to indicate the presence or absence of metadata. I think it was redundant for that purpose (non-zero $7D-7F already achieves this), but I think we could use it to indicate instead that there is some essential chunk in the metadata that is critical for correct playback.
This basically works like NSFe, in which any chunk FourCC starting with an uppercase letter means it must be parsed to correctly play the file. This allows the NSFe extensions to work as that format intended, and also gives appropriate places for things like, e.g. sample data chunks for new expansion sound devices that need it.
Basically the presence of the version 2 indicator should only be needed for using the non backward compatible features, which are all encapsulated be $7C. Non-essential metadata can be freely included as version 1.
(FWIW, I couldn't find any players that even checked the version number. )
Anyway, that's what I'm working on for this. Part 1 is already more or less implemented (see above). Part 2 will take more work, but it's on the way. If you think I've made a critical mistake somewhere, let me know before the world turns to mud.
Edit: Replaced Y init spec with $80/$81 for non-returning INIT sequence that's distinguishable from the usual default.