SPC File Format looks like S***

*From my perspective* - I am adding a new feature to SNES Tracker Debugger (STD) today, and it's SPC Export. Now, some of you might know I'm using my own modified version of Blargg's GameMusicEmu (GME) called libgme_M for this. Blargg has some funny comments in his code for "decoding" SPC fields -- and his frustration with the format is evident. (multiple comments)

Let's refer to some of the "best" SPC documentation I could ever find

• (older) http://vspcplay.raphnet.net/spc_file_format.txt
• (newer) http://translate.google.com/translate?h ... rev=search

I'm assuming this is the end-all be-all for SPC documentation. Someone please comment if you know better.

One of the most obvious issues with the SPC format is an unclear distinction between a "text" SPC and a "binary" SPC -- why there exists both a text and binary form in the first place is beyond me.

Then there's the actual string fields, which are "text" in both the "text" and binary versions of the SPC format -- is this ASCII text only? Do people stick Unicode-16 in there? Should I support that? But since I only plan on supporting ASCII in the first release anyways, that's all I'll do.

Continuing on the text-fields -- I suppose it's unsafe to assume they are null-terminated.. (Duh! *slap* goes the professional). I'm not sure if GME makes this assumption or not, yet!

Another issue is the "emulator" field -- which takes of all things, a number! But, only a few emulator indices were ever documented, and since this format is not at all regularly updated or kept standardized -- it's a crapshoot whether numbers are overlapping or not, or what owns which number. In this kind of non-standardized environment, a text field would have probably been best for this.

---------

If anyone has an SPC they know is distinctly a text one, or a binary one, I'd like your sample please.

Didn't Kevtris make some kind of new format that *might* be related to this?

Can I just go ahead and make a new format that is less difficult to work with?

---------

To the creator of SPC file format - I know how it is. We live and learn. Sometimes, looking back, we see what we've created and go *facepalm*

SPC is complete and utter shit. But it was the first out the gate, and no amount of improvements will ever convince anyone to switch. See: IPS, CHT, SMC, ZIP, WLA-DX, etc.

> Can I just go ahead and make a new format that is less difficult to work with?

I made a vastly superior format named SFM (Super Famicom Music), but we got bogged down because a true state capture has internal variables, and different emulators will want to name those fields differently (they definitely don't have official names.) Some emulators won't even support certain state fields, others may require new ones not yet known for a perfect capture. Forking the format with emulator-specific sections is the worst possible thing you can do.

SPC can't handle streaming audio (Tales of Phantasia opening, etc) because it's merely a snapshot of the SMP+DSP; and SNSF requires ROM hackers to create because they're literally butchered SNES ROMs that try to remove as much as possible except for the music code (and this forces you to emulate the CPU and possibly PPU just to play music.)

So here's the key to a much better format: when ripping the music track, log all the values read from $f4-f7 on the SMP to the end of the file. Remove the data for non-streaming tracks if you want (it'll compress down to nothing anyway), and leave it for streaming tracks. Timing of the accesses doesn't need to be logged, nor does the addresses. Just a linear stream of all bytes read, and then play back those accesses sequentially when playing the music. This can easily result in consistent playback because we have cycle-perfect timing of the SMP and DSP now. We ripped Tales' opening song in like 200KiB, whereas regular songs were the same ~68KiB of regular SPCs.

But good luck with the bike-shedding on how to store the internal state of the SMP/DSP. And enjoy the frustration as you spend the next ten years trying to convince everyone to use your much better format to absolutely no avail because "SPC is good enough."

It's funny you mentioned WLA-DX .. I even switched once to ca65 and then *went back* to WLA DX. I think I am masochistic these days, which could probably be why I'm good at any of this stuff.



Who is "we"?



I love that you considered streaming audio in your format.



I feel your personal pain (ie .sfc).

Now I could get into further details of why I feel your better file format(s) or tool(s) did not gain popularity, or why I want to know why no timing info was needed to log the CPU->SPC writes, but I already have a super-high pile of issues I'm sorting through. If anything, I learned that I am probably *not* the guy to introduce a new SPC format, because you've shown me the 2 crucial changes it should require:

• Streaming Support
• Some sort of *new* internal state issue you've mentioned (which I don't see how the traditional SPC logging of this information doesn't already have this covered). I know some emu's can have *better* state recording but the minimum that SPC already provides seems sufficient and relieves us of additional complexity to the format.

Unfortunately, since I don't give a damn about streaming -- only because the rest of the tracker software isn't written -- I'm already flawed to write a good new format...

But damn, I'm intrigued, and I think I could do it -- so please, byuu, answer these questions:

for Streaming -- Why is it safe to assume no timing info needs to be logged? Isn't that making an assumption that all data being written is at the earliest possible moment? The assumption might suit a lot of test cases, or all of them, but this is definitely an assumption! Unless you school me.

Can you agree with me to leave out emulator-specific state and just leave in traditional SPC state, namely DSP vals, CPU reg vals, and RAM snapshot.

Of course, even if I/we make a new format, SNES Tracker will support SPC as well!

Even if we *can* agree on the basis for a new format, I will probably want to concentrate on SPC support and the rest of my tracker software first just to get it up and running!

This has been a really helpful topic so far! Thanks for getting involved!

Well, if you're making the tracker software, what do you really need to worry about streaming for? Assuming you're streaming to replace instruments mid song or something, when designing the song in the tracker, you'd have it play the incorrect instruments but they're in the right spot to where they're going to be overwritten in the actual game. If the APU can't do anything about it because it's the CPU's job, (like streaming) then why would it have to be in the tracker? It might sound weird in the tracker, but it'll work fine. I guess you could create a "stream" function in the tracker that just changes instrument data for the sake of hearing how it sounds, but it won't affect the final output.

Because it's not logging writes. It's logging reads. They happen when they happen.

I didn't know SNSF required a whole-system emulation. I kinda thought it must be logging timestamped writes, but I didn't think of the above idea because I didn't think it through.

I do wonder if it would take more or less space to log timestamped writes instead of bare reads (one imagines that reads outnumber writes), but the emulation requirements would then be more finicky. Logging reads dispenses with the need to care about timing at all.

Yeah, I planned that.



Creating a new SPC file format transcends SNES Tracker. I would be foolish to create a new SPC file format that does not address these general needs that the first format did not live up to.

SNES Tracker doesn't appear to require streaming support - most general streaming situations will not require direct involvement with the tracker -- but even if a situation were dreamt up, I would not support it in the first release.

The SPC debugger *should* support streaming. Any supporting APU emulator would also require the following additions:

• APU "stream playback" feature - stream the new file-format's "stream content" as SPC reads from $f4-$f7.

Of course, after a format is defined, someone like Byuu would be most fit for writing in the new SPC export feature, while someone like me could write the new SPC Import feature, and then we'd have a working prototype.



Ohhhhhh!



In the read strategy, depending on the regular amount of consecutive repeated values - doing a "value,repetitions" pattern could be efficient. But compared to real compression solutions, this might be a joke. I'm not well educated in compression. Please someone school me.

But, even if there were better compression algorithms, organizing the data as it's being processed in a "value,repetitions" structure could save a lot of memory. Again, depending if the data-set experiences high sequential repetitions.

RLE, basically. Exquisitely suited to this situation (even if an advanced method could do better). Yeah, I guess that makes that entire paragraph of mine look a bit silly...

So, in order to keep this thread well-focused on the new format as a whole and NOT just the streaming component, let's start talking about the overall format.

==== RIFF Chunk Format
I've been well-advised on #botb by blaze_pascal to incorporate a RIFF chunk format for easy parsing and modularity.

ie "you can regard chunks that you aren't interested in as black boxes"
'to get "past" a riff chunk you just need to read the header and then jump <length> bytes'

I've also been told to include version info for obvious reasons, and in the decoding code to "upscale" old versions to newer versions like so:


I can agree with that.

I'd like to further document the different chunks briefly. Let me just summarize what I think a bare-bones necessity would be:

• Dump Info (title, name, etc. -- fields similarly found in orig. SPC format) The "what-dumped-me" AKA emulator field would now be a string for aforementioned reasons. Also considering supporting Unicode, and unicode-16 at that, because I find it easiest / most well-supported.
• SPC State (same fields as found in orig. SPC format)
• Stream data (if present) - in RLE if applicable.

==== Regarding Stream "Export"

now regarding stream data -- let us briefly concern ourselves with the possibility that the "stream log" could contain non-brr-stream data (bad example: "change song" engine command) -- Or perhaps there are other commands being sent on non-brr-streaming songs that had never been recorded! This sounds like a rare but possible exception.

Because of this potential issue, my paranoia, perhaps credibly so - I believe the SPC Exporter should have a "stream-dumping" boolean. It's simply user-configurable -- IMO with a default of "always-off" -- I'm just afraid of seeing auxiliary data "streamed" that's misc. engine commands.. Although this could be at times desired to be streaming.

====

I hope y'all like what I'm cooking!

Very clever indeed! But how do you support infinite playback?

> It's funny you mentioned WLA-DX .. I even switched once to ca65 and then *went back* to WLA DX. I think I am masochistic these days, which could probably be why I'm good at any of this stuff.

You'd kind of have to be, yeah :P

In bass:


98% of mappings described in literally two lines of code in a macro. Yet you have the flexibility to implement vastly more complex mappers if you wanted (64MiB ROMs, MMC-based ROMs, etc.)

I will never understand (nor want to) the complexity of mapping in WLA-DX.

> Who is "we"?

The people on my forum that are heavily into SNES dev.

> Now I could get into further details of why I feel your better file format(s) or tool(s) did not gain popularity

You don't have to, I already know the reason.

> for Streaming -- Why is it safe to assume no timing info needs to be logged? Isn't that making an assumption that all data being written is at the earliest possible moment? The assumption might suit a lot of test cases, or all of them, but this is definitely an assumption! Unless you school me.

When you are emulating the SFM file, you're producing the timing and adressing information again. The bytes written to $f4-f7 that are logged act like an assembly line. It doesn't matter how fast or slow that line moves, so long as the bytes come out the other end in the same order, and they will as long as both the recorder and player have correct timing.

Emulator movie files work on the same principle: they are only a linear stream of 1 = pressed / 0 = clear bits recorded for each time the emulator polls input. So you can record a movie file that can be used to perfectly play back a video game with only 12-bits per frame, or 90-bytes a second, provided you have the original ROM for playback. SFM is the same, SFM is effectively the "original ROM for music playback."

> I don't see how the traditional SPC logging of this information doesn't already have this covered

SPC didn't know about the required state in the beginning. The existing rips are missing the information. Adding a new emulator-specific section to log new fields is not a solution. That's basically making another new format and calling it SPC still. You'll spend the rest of time trying to replace older SPC rips with better, newer ones; and trying to get people to use newer SPC players that support the new state.

> the minimum that SPC already provides seems sufficient and relieves us of additional complexity to the format.
> Can you agree with me to leave out emulator-specific state and just leave in traditional SPC state, namely DSP vals, CPU reg vals, and RAM snapshot.

It's not sufficient. It's not a full state capture. It may not cause any bugs in playback, but it might. It might not result in two different players producing different output (because they assume different values for the undefined state), but it might.

> Of course, even if I/we make a new format, SNES Tracker will support SPC as well!

Right, everyone will support SPC because that's the format that everything's already in.

And that's why everyone will keep using SPC.

And that's why iNES 1.0 will still be the predominant file format in 2116 for NES ROMs.

See? I told you I understood this :)

> I didn't know SNSF required a whole-system emulation.

It's basically the original SNES ROM. They just go in and hack out as much as humanly possible, and patch some of the code. So the reset vector jumps to the song player as soon as possible with a small patch. Then they kill out everything not used.

It's clever, but SFM is a much better way of doing it. Especially because it doesn't require someone with ROM hacking skills to create. And because you don't have to emulate the SNES CPU as well.

> I do wonder if it would take more or less space to log timestamped writes instead of bare reads

No matter the case, timestamps wouldn't save you at all. If the ripper and player don't have the same opcode timing, it will fall apart during playback. But again, it's a non-issue. We know the exact cycle timing of every instruction on the SMP and DSP, thanks to blargg. It has all been verified (although blargg's SMP core has errors, so you'd not want to use that.)

> Yeah, I planned that.

I was going to point out that it seemed a bit immature to associate your software with venereal diseases, but then it dawned on me that the C++ standard library's namespace is "std", so ...

> Of course, after a format is defined, someone like Byuu would be most fit for writing in the new SPC export feature, while someone like me could write the new SPC Import feature, and then we'd have a working prototype.

I gave up on it over the internal state serialization issue.

They say perfection is the enemy of good, but that's what got us iNES, IPS, SPC, etc in the first place.

I don't want to be the guy responsible for over-looking some detail that ends up requiring everyone to replace SFM with some newer, better format ten years down the line. Or much, much worse ... the format everyone says is "good enough" and just groans about the problem(s) it has.

> doing a "value,repetitions" pattern could be efficient. But compared to real compression solutions, this might be a joke.

Correct. The data will be very highly repetitive. Even in the case of ripping Tales' streaming opening song, compression helps immensely. Amateur hour compression like RLE won't hold a candle to even something as simple as ZIP.

And the biggest space savings in SNES music is that the sound player engine and samples are 95% identical between songs. This is why they use solid archivers like RAR for SPCs. It's better to just leave compression to better tools for the job. And when an even better compression comes along, you can switch to it without having to throw out your existing format.

> I hope y'all like what I'm cooking!

I don't mind you playing around with the idea. Maybe you'll come up with some new ideas I like :)

But I want to stress two things. First, my original warning: you will never gain any kind of serious adoption of your format, no matter how amazingly awesome it is. Second, I'm not planning to work on this now, so you would have to patch bsnes yourself to rip into your format, sorry. I am totally swamped with work as always.

> Very clever indeed! But how do you support infinite playback?

I've never seen a streaming game that has infinite playback.

Non-streaming games won't need it. Just like how infinite playback SPCs right now work, you'd return #$00 from reads from $f4-f7 once the stream log runs out (in this case, it wouldn't exist, so that'd be immediately.)

But even if we contrived some example of an infinite streaming song, it would eventually loop, so the format could encode its own "after X samples are decoded, seek back to sample Y" to form a perfect, clean loop just like MSU1 PCM files do.

The only way you could intentionally break this would be to create a truly irrational format (pardon the pun ... you'll see) by doing something like streaming the computed bits of pi on an endless loop. Which, I don't really care about silly cases that don't exist, and won't exist outside of intentionally trying to break the format :P

The overall implication of this statement is simply not true. I know a lot of ROM hackers (myself included) who were early adopters of UPS/BPS, headerless ROM images, game folders etc. I admit those people may still be a minority to this day, but the numbers are clearly increasing.

As for the constant bashing of WLA DX, though I won't even bother defending it, I want to say that for my needs as a SNES developer, there's just no alternative at the current time. Also, if you don't like any of WLA DX's features and/or the current implementation of a given feature, head over here and file an issue, please. Constant bitching about it won't help improve the software, anyway.

Maybe not, but I'm not done assessing the situation.

This is undoubtedly less efficient than standardizing the new state fields -- BUT let's talk about if we didn't -- I can see a major emulator coming out with their own RIFF-chunk (optional) to convey their emulator-specific SPC state information. Any player's that don't recognize it can simply skip over it. That would leave us with "current-spc-playback" functionality or better playback for players that can recognize and import the emulator-specific data. And, since it's completely optional to include - if this turns out to be useless or not well-implemented, people can leave it out of the file anyways! That would still give us a better organized SPC file format.

Now this would be a BAD solution because that means any emulator needs to know about all these different-emulator-specific RIFF chunks if they want to gain the added state-info. Obviously, standardizing the fields would be the REAL solution -- but who's going to get all the SNES APU emulator developoopers together to conform on that level?





How many different major APU emulators in use are there? I expect the answer to be very small, and let's also talk about how many are cycle-accurate.

Could you imagine getting these developers to discuss together the kinds of additional state their emulators would use for a proper SPC save-state? It is possible we could identify similar components and then give them official names to include in a newer format.

But that sounds like so much work, while at the same time - all of these emulators seem to play regular SPC without the additional state just fine.



That's normal. No problem, no harm done.



well if that happened, they wouldn't be able to play any regular SPC file either (or at least not perfectly), so no harm done. -- this new format would at least give them A CHANCE to play perfectly:

so you come up with a new version with the new field. if this "emulator-that-requires-new-field" is used to open a new-style SPC, it would simply be able to perfectly play the version that includes the new field, or play imperfectly (or not play) the older versions of the format (including current SPC format). Again, this is a BETTER compatibility situation than simply using the old SPC-format.


and yet, standardization in the SNES community?!?! haha -- yeah right. I mean, it's do-able. But not on my watch.



Are the errors you talk about in Blargg's core documented? Can you provide reference to an error free core. The thing I like about Blargg's core is that it also has a "Fast" aka "good-for-slower-computers" implementation - I would hope a referenced core does as well.

The reason I want reference(s) is because I DO use Blargg's core in Snes Tracker currently, and it's doing great I have no complaints. I feel it's good enough. But it would nice to know where the next step forward might be in case I need to ditch this emulator for a better one. Ideally, SNES Tracker will have APU emulator abstraction anyways, so that drivers for different APU emulators can be made.

One of the concerns is that I require a lot of read/write access to a lot of emulator internals, and generally modifications to the emulator itself are required to generate the product I'm interested in. For instance, to Blargg's APU emulator, I added code to use libgme_M's new "report" facility to indicate all locations of read/write/execute/echo region/BRR reads -- that's how the memory window is generated. I'm not sure how best to "take in" new APU emulators and add these features as transparently as possible, so that I could keep up with version improvements potentially.

Then there's the fact that SNES Tracker (Debugger) may arbitrarily modify any RAM / DSP etc -- I seem to have been able to pull this off (lol) in a multi-threaded environment, but I'm not sure how that worked out.... I'm using SDL which is calling the APU emulator to produce new samples from an audio thread, and yet I'm somehow able to *I ASSUME, without error* modify freely!! At the very least, I am HAPPY with the results. lol <--> I am sure I spawned death in the eyes of certain engineers at my lack of attention to the cross-thread activity, but I'm HAPPY -- let's see how this pans out.



I don't care if anyone adopts my format. Having created an answer to "SPC is crap" is enough for me.



What is the minimum version of bsnes I can work with?? As you know, I have my fork that I'm curious I can use -- based off v073u1. I'd rather not use if the APU emu is outdated. https://github.com/bazzinotti/bsnes-classic

Having said that, I'm probably *NOT* going to implement stream dumping to an emulator, since that goes way-off-track for my main goal, creating SNES Tracker!! If some other people get involved, I'm more likely to help with a new format.

Yeah, I'm definitely not going to do it. If someone else were to do it, they could get in touch with me and we could put it all together - but my program *does* use Blargg's SMP core which Byuu already stated has errors -- meaning that if it's truly not fit for streaming (which honestly, it MIGHT still be) -- it would mean that I couldn't use STD to test the SPC's that stream anyways, until incorporating a more accurate SPC emu into it. And of course, I mean as a driver!!

EDIT:


I understand why "they" compress whole SPC archives -- but what are you suggesting for a single SPC file. You can't be suggesting to compress single files as zip as well? I'm thinking to use RLE for single files, and RAR for archives like they are already. Don't you agree?

Good on you.

> I know a lot of ROM hackers (myself included) who were early adopters of UPS/BPS, headerless ROM images, game folders etc. I admit those people may still be a minority to this day, but the numbers are clearly increasing.

Not the ones I've seen. I won't name the person, but I even once had a person I helped out probably a half-dozen times with rather significant tasks say to me, "if you help me hack this next game, then I'll use BPS for the patch format instead of IPS!", which I found quite crass. Especially considering I've never asked that person for anything in return before.

I am hopeful that you're right though and that BPS picks up more steam. It's already five years old. RHDN still hosts beat v01 despite that being ancient. And see for example this from two months ago:
http://www.insanedifficulty.com/board/i ... xperience/







And that's after repeated attempts to explain the benefits. People don't want better tools.

And then:



Which is definitely wrong. You can stack BPS patches just like IPS, you just have to ignore the checksums when you do because it's not possible to store a checksum for a file in an unknown state.

> As for the constant bashing of WLA DX, though I won't even bother defending it, I want to say that for my needs as a SNES developer, there's just no alternative at the current time.

I've no interest in trying to convert people to a different assembler. Basically, nine out of every ten people who write large amounts of SNES ASM end up writing their own SNES assemblers anyway.

> Also, if you don't like any of WLA DX's features and/or the current implementation of a given feature, head over here and file an issue, please.

I wrote my own instead. I do that a lot. See: bsnes, BPS, SFM, etc ;)

Hope no one gets mad about a second post; but the first one was too long.

> Any player's that don't recognize it can simply skip over it.

And again, it's not a perfect state capture then for any that are missing this new RIFF section. And it's not a perfect playback for any players that don't support this new RIFF format.

And you may end up with competing emulators implementing competing RIFF sections, and now you're back to "why can metadata be stored in plain-text or binary?? Why have both?"; only for much more important data.

A perfect format needs to be universal.

> Obviously, standardizing the fields would be the REAL solution -- but who's going to get all the SNES APU emulator developers together to conform on that level?

The person that defines the file spec needs to set everything in stone. It is their responsibility.

> How many different major APU emulators in use are there?

There's absolute piss-poor garbage based on things like OpenSPC, older Snes9X, etc. And then there's blargg's DSP which mine is 100% compatible with.

> Could you imagine getting these developers to discuss together the kinds of additional state their emulators would use for a proper SPC save-state?

blargg is all but gone.

> It is possible we could identify similar components and then give them official names to include in a newer format.

Sure. How'd that work out for 'illegal' 6502 instructions? How about for Cx4 instruction names? :P

Something something herding cats. Try and debate standardizing on formats and they'll argue with you for pages and pages over the most asinine edge case minor details, and then block you on Twitter for your trouble :P

> But that sounds like so much work, while at the same time - all of these emulators seem to play regular SPC without the additional state just fine.

If "good enough" is good enough for you, then stick with SPC :P

What's "good enough" for us today may not be for people in the future. I don't want to kick the can down the road, I want to kick it all the way to the finish line.

> and yet, standardization in the SNES community?!?! haha -- yeah right. I mean, it's do-able. But not on my watch.

I know. This is why I gave up on trying to standardize things. I'll just do my own thing, and complain bitterly all the time about it on forums instead :P

I'm very grateful that NESdev is basically the only emulation/ROM hacking forum on the internet I participate on that isn't my own and that doesn't ban people for disagreeing with them.

> Are the errors you talk about in Blargg's core documented? Can you provide reference to an error free core.

I didn't bother to correct the myriad errors in his SMP core. I wrote a new core for Snes9X to replace it. But higan/sfc/smp is what you're looking for. The only limitation of it is that I don't emulate two of the eight TEST register bits (there is no game ever observed that uses this register at all.)

> The thing I like about Blargg's core is that it also has a "Fast" aka "good-for-slower-computers" implementation - I would hope a referenced core does as well.

Even a $5 computer (RPi Zero) can play back SFM files with bsnes' SMP+DSP cores in isolation.

I guess if you're worried about 80 hours of battery life on a Pebble Watch, then okay, go for it. blargg has a fast DSP core, and I gave a fast SMP core to Snes9X (which is in the unreleased v1.54)

> I don't care if anyone adopts my format. Having created an answer to "SPC is crap" is enough for me.

Now that's the spirit! :D

> What is the minimum version of bsnes I can work with??

I don't have any idea when the last SMP/DSP fix went in. I always recommend the latest version in the accuracy profile.

> You can't be suggesting to compress single files as zip as well?

For distribution? Why not? I get what you're saying, people want to store BPS/SFM uncompressed and have them be optimally space efficient, but too bad. Writing an effective compressor/decompressor is very hard work. Writing a toy one like RLE is just going to make the files *bigger* when they are inevitably compressed anyway to distribute online.

> I'm thinking to use RLE for single files, and RAR for archives like they are already. Don't you agree?

I'm not going to say 'absolutely not'. BPS, as a side effect of delta-encoding where you specify a pointer that overlaps not yet decoded data, supports RLE.

It's your format though, do whatever you like :D

I didn't mean to seriously suggest using RLE. I just meant that by its nature it negates my objection to read logging as opposed to write logging, since the major difference is that the SPC700 might read the same value several times before it changes. Any improvement from using a more advanced scheme is thus gravy on top of that.

As far as I can tell, we standardized on the mnemonics used by ca65's 6502X mode.


You're welcome. I've tried my best to apply civility guidelines like those of Wikipedia (as written, not as misinterpreted by rules lawyers).


Ultimately the problem is that Windows Explorer does not support stacked file extensions such as bps.zip or sfm.7z. This is why people have to rename .spc.rar to .rsn to get them to open in a Winamp plug-in (as opposed to), or .s3m.zip to .s3z to get them to open in ModPlug Player (as opposed to Compressed Folders).


Then embed Info-ZIP's effective library. File formats used by Java (.jar), Android (.apk), Winamp (.wsz), StepMania (.smzip), ODF (.odt), and OOXML (.docx) are all built on Zip.


Not necessarily. DEFLATE compression used by the .zip format has a 32K history, and preprocessing with RLE may allow larger matches to fit into 32K. Have you tested this?


Plus RAR is a non-free format. The unrar license forbids understanding the file format, making it incompatible with the Debian Free Software Guidelines, the Open Source Definition, and all copyleft free software licenses.

> As far as I can tell, we standardized on the mnemonics used by ca65's 6502X mode.

Oh, neat! That will be nice when I get back to NES core work again. Been wanting to emulate all the extra opcodes for a while now.

> Ultimately the problem is that Windows Explorer does not support stacked file extensions such as bps.zip or sfm.7z.

Yeah, it's also shit with extensionless files like Makefile and .htaccess (won't even allow you to create the latter, at least from Explorer itself.)

kaijuu gets around this nicely, but I don't expect people to install that :/

> Then embed Info-ZIP's effective library. File formats used by Java (.jar), Android (.apk), Winamp (.wsz), StepMania (.smzip), ODF (.odt), and OOXML (.docx) are all built on Zip.

The point is, I wouldn't mandate the compression in the format. Because then it can't be changed. If you want to use a tiny ZIP library, sure. My library, nall, has an 8KiB inflate implementation that would be perfect for that.

But, look at .tar.gz. Imagine if that was it. Instead, when bz2 came along, compressed sizes went down a lot. And then .xz came out and things went down a lot again. I like the separation of concerns, the Unix model, as it were.

> Not necessarily. DEFLATE compression used by the .zip format has a 32K history, and preprocessing with RLE may allow larger matches to fit into 32K. Have you tested this?

Actually, yes. We tested BPS with a lot of variations. We found BPS raw + 7-zip produced the smallest patch sizes. The BPS files were bigger once decompressed, obviously. The smaller we got the BPS patches with more efficient encodings and through using RLE, the less effective the 7-zip pass was. While the BPS patches were smaller on their own, the bps.7z files were larger this way. And this trend further extended over to the Xdelta format we were testing against (Xdelta3 also has -0 through -9 compression stages of its own. But even at -9, it benefits from a separate 7-zip pass.)

> Plus RAR is a non-free format.

RAR is garbage, yeah. Surpassed by 7-zip a long time ago. But a solid archiver will net you huge gains with all the repetition between tracks in a music album.

I was about to point out the illegal instruction list on Oxyron's 6502 page, but it turns out ca65 already uses it as a reference. Neat!

(That entire series is a pretty useful condensed 65xx series reference, especially the 6502 and 65816 pages.)

Amusingly, I almost pulled off a successful circumnavigation of a sound driver that streams the note data with one of the games (Paperboy 2, to be exact)... by outright implanting the music data from the SNES ROM and attempting to code the SNES portion in SPC700 code. You can find my WIP SPCs here.
The main problem is currently dealing with hiccups with reading the music data with regards to delay, which I copied straight off of the ROM.

Plus, this solution doesn't work for all games due to memory limitations (NBA Jam '96 and '97 are two such games: the memory consumption alone on the sample data is way too much for my taste, even though I have at least partially reverse engineered the music format). For games that have sample swapping (but not streaming in most cases... with the Visual Concepts games, in regards to streaming samples that play pieces of fully digital musical samples, I have circumnavigated that one personally... see NBA Jam '97 for an example), I have been successful in circumnavigating them, especially Lion King. For Lion King, I discovered that there was data that was not properly copied over (and I realized I had to overwrite SFX samples in some cases), and I personally took care of the problem by analyzing every which samples were being swapped out.

Obviously a custom file format that handles command IDs coming from the SNES would work much better than what I'm trying to do, but until that comes along, I'm attempting the impossible with the SPC format.

I would like to see a file that is just the contents of audio ram from $0200-$ffff and has the program counter start at $0200.

Agreed



OK, so then the only APU emulators that have a future are blargg's and yours - and they are compatible (which, could you please expound on what that means? ie Does this mean I could literally replace DSP files.. or what?) -- with yours in 1st place since you are active. Sounds like standardizing an optional RIFF chunk for "perfect state" between 2 of the best cycle-accurate APU emulators shouldn't be too hard. It sounds like a great place to start.

The "piss-poor" emulators can at the very least, rely on the traditional SPC state that will be present in the newer format -- and that's even if there are still active maintainers of those APU emulators or active developers of applications that use those emulators and want to update to support parsing the newer format. Of course, they can also try incorporating the newer "perfect state" attributes if they desire. However, you (byuu) stated that cycle-accurateness will be required for proper streaming-playback of the new format -- so these "piss-poor" emulators may simply skip processing the streaming chunk, declaring inability to stream (for the few spc that do stream). Of course, they could attempt to stream if they want to give it a shot!



Since when? I remember talking to him about a year ago.

Blargg's APU emulator does have a "native state saving" feature -- we could look at the kinds of variables it uses and compare with your emulator and deduce a common-terminology, hopefully .. I couldn't do this without you though, otherwise my work would be exponentially more difficult, since I never designed a DSP emulator, and don't really understand the kind of state/variables involved. I could probably do it if I really tried, but I hope that you would just get involved. I am sure it would be a much simpler task for you.

You might want to look harder.

But never mind. I'm going to keep using your format anyway, and recommend it to fellow ROM hackers like a mantra (with success), even though from your point of view,


(Yeah, right. )

It means my DSP core is basically a plagiarism of blargg's, but with his full permission including the right to license mine as I want.

I removed all of his optimizations with slower but more obvious code, but all of the operations and even variable names are identical. (I hid one in the modulo array, but I'll probably remove that too. I really doubt a few divisions per sample are that big of a deal.)

I've always wanted to restructure it, but I've never found a nicer way to describe it. The inner loop of the SNES DSP is a chaotic, blended mess.

> Since when? I remember talking to him about a year ago.

search.php?author_id=17&sr=posts

The very first page goes all the way back to February 2014.

I used to talk to him on a daily basis.

So many amazing people have come and gone. anomie, neviksti, blargg, TRAC, etc, etc. I miss talking to them all.

> But never mind. I'm going to keep using your format anyway, and recommend it to fellow ROM hackers like a mantra (with success)

Thank you very much :D

It's not even that I want my format to win, it's just that I want IPS to die.

Best we can do right now is deflate the sole false argument that BPS can't handle multi-patching. As you can see, the opponents were looking for any tiny sliver of a regression from IPS to dismiss it entirely, no matter how inconsequential. That's why it was so critical to ensure there were none, and there aren't, any negatives.

> (Yeah, right. :P )

A bit hyperbolic, but that's how I roll :P

<333

*long moment of nostalgia*
(I learned DP addressing from TRAC in #snesdev ~10 years ago <3)
(Neviksti, bright, helpful, and generous, the word "legend" comes to mind, and pcx2snes aha, and his SNES-Starterkit, and all of the other countless projects he was involved in (Tototek SuperFlash reverse engineering, X-Band, WLA-DX, etc. etc.)
anomie's document(s) always had a strong of impression of "THESE ARE THE BEST TECHNICAL DOCS" -- but I never got to meet this person.
blargg, I only recently started talking to -- and I was so obsessed with making quick progress on SNES Tracker that I overwhelmed him lol. For the brief time we worked together it was wonderful.

------

Byuu, I think you realize I expected more response from you based on my previos post. Particularly about creating universal terminology based on these 2 super-similar emulator cores that appear to be the leaders in SNES APU emulation -- which are apparently so similar that it might be as simple as identifying the terms in only 1 emulator! Do you agree?

I'm at my tipping point whether I'm going to even bother with these "universal" perfect state fields -- so please, this is your chance to get involved on that. Again, I'd think this would be easy for you -- but no pressure, we'll let this be. I don't want to see you stressed or overwhelmed because of some stupid fields.

I'm hoping somebody here can help me on this subject, because I gotta say I'm VERY annoyed with there being no good solution to this:

I recently obtained a TOSLINK-modded SNES and I can capture the 32Khz audio from it directly into my computer. I wanted to make direct-hardware ripped soundtracks, and heard there was an SPC-to-ROM tool that could turn SPC files into ROMs to be used on real hardware. Problem is, either the tool or the SPC files SUCK. Most don't even work on real hardware at all when I convert them with the tool, and others only partially work with missing and/or glitched channels. These files exhibit the exact same behavior when loaded into byuu's emulator. This is simply absurd that this is the state of soundtrack ripping on the SNES. I'd like to find out if any of the following is available or being worked on:

1. A SNES sound file format that can actually be run as a ROM CORRECTLY on real hardware.
2. If not, then an SPC player that uses the same sound core as byuu's emulator.

Thanks for your help on this. I don't know much about the subject except that I'm getting very frustrated with SPC thus far.

Which flash cart are you using? My SNES PowerPak (with aftermarket "MUFASA" firmware) has an SPC player.

It could be an issue with the player rather than with the files - SPC files are just an image of the sound processor's state, and should (hopefully) sound exactly as intended on the actual hardware as long as you're able to put the SMP and DSP back into the same state that the SPC file describes (but that's a bit easier said than done.)

Of course the SPC format does have some inherent lack of state that was mentioned previously in the thread but I don't know if it would cause issues as blatant as what you're mentioning.

The sd2snes also has a built-in SPC player that I've had good results with. I can test it with anything you'd like to hear from it.

I've got the SD2SNES. I didn't even know it plays SPC files directly, so I will give that a try next. As I said, I'm new to all this, so my apologies for being ignorant of the capabilities of these flash carts.

Edit: Tried the internal SD2SNES SPC player and it works great! Problem solved!

Thanks!

So I tested out several SPC files with the internal player on the SD2SNES, and it's not perfect. It seems 1 in 10 or 15 tracks will not play properly and has glitched sound. Bummer.

There is no "perfect" SPC uploader/player. There probably can't ever be.


(Source: https://github.com/mrehkopf/sd2snes/issues/26, comment #1)

What I mean is, these files play fine in other PC-based SPC players, so it's not a case of the expected lack of perfection. See the part of my post you edited out (that I have now put back in).

PC-based SPC players can load all of RAM and all DSP registers in one cycle. Hardware cannot. Furthermore, some SPC rips were designed for defective PC-based SPC players that fail to emulate the S-DSP accurately, and they rely on said players' bugs.

What tepples said.

Plus:

vs.

So what? SNES != emulator, as tepples explained.

Apparently I've managed to create the opposite of what was being discussed earlier in this thread: a SPC dump that plays back perfectly on the sd2snes, but fails in every PC-based player I've tried, as well as mic_'s spcplayer. (spc2rom almost works fine, but I think it screws up the echo buffer and clobbers some of the instruments, which is probably an unrelated issue).

Here's a dump of a music track from the Super Wild Card DX firmware:
https://dl.dropboxusercontent.com/u/43107309/swcdx.spc

Here's the spcplayer version:
https://dl.dropboxusercontent.com/u/431 ... x-tune.sfc

Apparently at various points the driver tries to jump to $FFFB in the IPL ROM, except by that point it's not actually enabled, so you can probably guess what happens next. Is the SPC file missing something that causes the IPL to get disabled when loaded by most players?

(I know about the part of the SPC file that's supposed to contain a copy of the IPL ROM, but filling that in vs. leaving it blank didn't seem to make a difference in any players.)

Should be fixed:
https://github.com/mrehkopf/sd2snes/com ... 3fb931bb7a
SPC upload routine had a bug - SPC700 would occasionally grab a byte too early at the beginning of a transfer chunk because the original SPC upload code didn't account for CPU stalls during HDMA.

Oh man, I recognize the one note that did manage to play on my PC! I've been looking for that track, haha. Does it exist as a "working on PC" spc?

I can try to hack one up later, but I'd rather figure out what's wrong with my own dumps first.

(by the way, if anyone who actually owns a SWC DX can give that and the other tune a better description than "unknown song", please do. Getting the menu ROMs to actually run properly under emulation is a pain)

Several things to cover here:

The song in question was composed by The Doctor of Anthrox (ATX). Anthrox did a *lot* of SNES stuff (cracks/intros/demos). But given that it was distributed in a grey-market commercial product owned by Front Fareast Co. of Taiwan *with Anthrox's permission*, I would strongly suggest you use this for the SPC fields:

Song title: SWC DX Menu Music
Game title: (c) Front Fareast Co.
Artist: The Doctor/ATX

The SPC700 music driver used by the copier was written by, and I quote, "His Assistant" (referring tongue-in-cheek to "an assistant" of a doctor). Who that is I do not know; it may be a SPC700 player that was ripped from a commercial title and the sequence format data reverse-engineered; I really don't know.

I don't remember if it was available on the SWC DX2 (64mbit) or not; it's been a very long time.

If you want references for my claims, the easiest place to get them from is the copier itself: the SWC DX had some built-in features that a lot of people didn't use: one was a PCX picture/image viewer, another was a tile game, and finally a demo/intro by mostly Anthrox that had chiptune-like music and provided credits of who did what (for FFE itself -- I know this is the case because JSI of FFE is in fact the guy who wrote the original MS-DOS transfer utility called VGS that let you dump carts + send SMC images to the SWC DX via an LPT port -- I later wrote my own in Pascal + x86 assembly). Someone was kind enough to make a recording of this demo in the copier, so you can see it in the text scroller yourself -- and I assure you it refers to the menu music and not just the intro/demo you see (you'll understand when you read the credits): https://www.youtube.com/watch?v=Qm7qA3sLJYc

Hope this helps. If not, let me know and maybe I can track down some of the Anthrox guys, particularly Pan, who I had talked to a few times back in the 90s. Consider me amazed that nobody's done this in 22+ years. *chuckle*

Also, I don't know if this is helpful but in the Alpha-II SPC player plugin for Winamp, attempting to play "Super Wild Card DX - Unknown Song 2.spc" plays a frame or so of audio (too short to make anything out), then claims the SPC700 has crashed at $7C07.

Edit: P.S. -- If you need someone to actually pull or "do something" with an actual working SWC DX (32mbit) using the latest firmware (from 1996, not 1994), one guy I know who has one -- because I sent mine to him in Germany, and he repaired it (many corroded solder points; I told him he could keep it) -- is Ramsis here on the forum. :-)

Very cool trivia anyway, Koitsu, please share anything else you know.

Wasn't Pan a dude out of New York who espoused some, erhm, unconventional views in an interview?

Yeah, aside from not having a song title, that's pretty much how they are tagged already. The credits page on the menu itself (not the cooler-looking ATX intro) lists The Doctor as doing the sound; plus, their style is pretty unmistakable so it would have definitely been my first guess either way

But, is there normally supposed to be any music playing on the main menu at all? Aside from the little jingle when the Super Wild Card logo appears, there's normally no BGM when I emulate it. But I'm not currently able to actually access all of the menu's features (like the PCX viewer / built-in game), hence wondering if someone who actually owned the unit could clarify that a bit.

(I knew what the ATX intro looked/sounded like from having seen the YouTube capture you mention later in the post, but that's a different/older piece of music that I don't think is ever loaded in APU RAM when the menu itself is running)


I speculated on these facts several months ago in this thread - tl;dr: it's the Argonaut Software sound driver and I strongly suspect The Doctor and Assistant were actually two of Argonaut's actual employees. I tried contacting one of them about it once but he never responded.

(I also had assumed both of their handles were a Doctor Who reference, especially given that they might be British )


The DX2 has totally different audio by someone else (iirc another FFE person whose name I don't remember).


That would be awesome if you could - if nothing else, surely just because he could shred some light on my theories from the aforementioned thread

Really, it'd be cool to see something like a "where are they now" with some of those big names from the 90s SNES scene, but I guess that's a little ahead of the subject


Yeah, thanks to mic_'s SPC->ROM converter and my debugger I was able to figure out why that is (pretty sure it's entirely caused by the IPL ROM being disabled when it shouldn't be) but the mystery is why that's the case to begin with, or why it works on the sd2snes player but not others.


Like I said before, I just need some proper context for where these two bits of music are actually used. I'm only able to half-assedly-emulate it enough to get the logo jingle and a silent menu with sound effects, plus reboots when I try to access any of the "bonus" features (built-in games/the ATX intro/whatever else), which is what I assume the music may be related to.

The SPCs actually were dumped from a 1996 version of the firmware, but they seem to be totally identical audio-wise (except for the SWC DX2).

Anyway, I appreciate the reply. I had a feeling you'd have plenty of knowledge on this specific subject


Yep, according to the interview with him in the first (only) issue of SNES Trainer Charts, if I remember right.

The only real non-SNES-related things I know about him are being the namesake/original author of a certain Game Boy doc, plus his neat C64 work.

Just guessing now but I wonder how many SPC players emulate the $F1 register correctly. Bit 7 controls IPL ROM mapping.
What's the value in your .spc at file offset 0x1f1?

It's 0xC3 in all of them.

In that case the IPL ROM should be mapped (unless the SPC700 code changes it later on)

He was on the east coast (wasn't sure if NY, NJ); I don't remember the interview in question. Doesn't change the fact that asking him for details about something from 20 years ago is still legit; not like I'd be trying to track him down to run for president or something. :P

I'm going off of memory here: I believe it's a toggleable feature in one of the menus somewhere, *OR*, it only played in some of the firmwares versions (1994 version may have it, while 1996 might not). My memory keeps telling me that the SWC DX2 was silent in the menu interface (though later below you state that the DX2 has different music, so I'm probably wrong -- I'm being asked to remember things from over 20 years ago :) ). I imagine if they switched to that method, the reason maybe was that a) the music does get annoying (and it was actually pretty loud) if you let the system sit there for long periods (which you would during development sessions), and/or b) maybe saving EEPROM space for future things.


I'll do what I can. And yes, the latter is something I'd love to see as well, but tracking people down who just used monikers and were really private people is an extreme pain -- and expensive (I've thought of hiring actual agencies to do this for some old snesdev people).

On the flip side, it always makes me laugh when someone says "OMG Y0SHi! you're so hard to find!" I've had several people over the years tell me that -- it's like, are you kidding? There are under a hundred or so Jeremy Chadwicks in the world, and my stuff comes up on the first page of Google. It takes a total of 30 seconds of sleuthing time to find me. But others aren't so public, or didn't put their real name on their works, so they're tricky to track down.

I'll see what I can figure out. I gotta try to remember who all was in the "SNES scene" part of Anthrox and then track down whoever, as maybe they know who went where. Brief story interlude: back in the 90s there was a really obscure demo group from South Africa called Guru Magic -- I loved their Shrine demo (especially the ending credits music -- beautiful piano piece), but several attempts to reach out to them failed. A few years ago, by total chance, I found one of their members on Reddit (talking about old PC demos or music sequencers or something, I forget), and he was able to get me a ton of old music of theirs (stuff that had never been released . He was totally blown away that anyone remembered them from that single demo. I lost it *all* when I fat-fingered a filesystem migration command 2-3 years later -- some of the songs I had were also ripped from their demo (which is encrypted or compressed somehow) by a friend of mine (the music format they used was their own, called Pyrotracker 16, like a 16-channel Amiga MOD), which still bums me out. Maybe I'll try to get those files back today.

Ok, I just fired up the 1996 ROM and you're right - the first song can be made to play on the menu. Still not sure about the second (non-looping) one, but I figure it might be related to the built-in games or something.



I only know for sure that it has different audio (i.e. sound effects and jingles, or whatever), I don't know if it actually has any proper music.



Damn, that's a real shame I'd be interested in getting my hands on that music too, if you're able to get it back.

I can just see it:

"Where are they now?
The Doctor -- Head of Nintendo UK
The Assistant -- MD, Sony of Europe" ulp.

In case anyone cares, I finally fixed up the SWC DX music dumps so they should be non-broken on software SPC players. (edit: had to fix it a second time for foo_gep)

RSN archive

For some reason the driver code was doing some funky shit involving the IPL ROM that was causing software players to choke. Changing a handful of CALLs to NOPs "fixes" it without actually affecting the music at all. Weird stuff.

Haha sweet! Thanks dude!