Sound effects and the Sound Engine

This question is probably more general, with the answer varying from sound engine to sound engine.

In the typical scenario where we have four sound channels, lets suppose that we had music playing which used the noise channel for drums. We also had a certain sound effect that used the noise channel; it has priority over the music.

We know that if we triggered the sound effect while the music is playing, the sound engine will mute the music on channel 3 to play the sound. From the info I gathered, I think I understand what's going on from engine's point of view.

Basically, the sound engine iterates through several "streams" each which, typically, contain different parts or waveforms of a piece of music; i.e. stream 0 might contain the lead square of a piece, stream 1 might contain the backup square, etc.
If a sound needs to be triggered that needs to use a channel that's already in use by the music, then if the sound has priority over the music, the engine will stop writing the note data for that stream to the APU. At the same time, the engine will start writing the note data for the sound effect to the APU. However the all of the music counters are still updated.

So when stream borrowing happens, basically, engine stops writing one stream and starts writing another. When the sound is done playing, the engine stops writing that data and resumes the other stream.

Is this about right?

You'd still have to process the stream to keep it up with the music. Just have it set up if there's a effect using the same channel, process the channel timing/tracking/whatever people call it, and then make sure the sound effect gets output instead.

I would imagine that you would want your sound engine to continue to process the channel normally, but disable all writes to the registers that would be on behalf of that channel's changes, replacing those with those of the sound effect.

My music engine treats drums as sound effects using the normal sound effect priority system (longer effect wins). Sound effects on the pulse and triangle channels are run in parallel with the output from the instrument system, using whichever is louder at the moment.

I use 8 tracks; 4 for music (SqA, SqB, Tri, Nse), and 4 for sound effects (SqA, SqB, Tri, Nse). When a sound effect is triggered, its data is loaded into the appropriate track(s) depending on which channel(s) the sound effect plays on. Then, the equivalent tracks on the music side are told to mute, but they continue to "play" just like normal, they just don't output anything.

When a sound effect track finishes playing, it unmutes the equivalent music track, and even restores the pitch of the current note.

My sound engine just uses a set of shadow registers (RAM copies of the states of the actual APU registers). Music and sound effect handlers both write to the shadow registers, which are written to the APU at the very end of the processing. The sound effects are handled after music, so they get higher priority automatically (they'll overwrite the values produced by music when they sound effect is active.)

Same here, essentially. Though, I actually handle "drums" separately. They are essentially like sound effects, so they can use all of the channels as they choose (they don't only use noise). So first the engine processes single channeled voices, then drums, then sound effects. Priority happens naturally as it is processed in this order.

But then if you want to do something like force a write to $4003/7/b/f to start a new note, or use the sweep registers, you're screwed.

I think a sound engine that properly handles sound effects with multiple layers is the solution. Basically you just have 8 channels instead of 4, you use the first 4 for the music and the last 4 for the sound effects. I do this and many actual games does it something like this. Only very early and simple games did it differently, by hard-coding stuff all the way around, which is a dirty way of coding I wouldn't recommend.

Would it help with the mixing of sound effects if they checked the volume of the sound the music was trying to set for a given channel and only overwrote the data for that channel if the sound effect was louder than the music?

Yeah, that's what my engine does. There are several priority behaviors in play:

1. Sound effect volume compared with the instrument on same channel; the louder on each channel is kept. Triangle has internal volume as well, just for this.
2. A sound effect isn't started if its data length is shorter than the remaining data length of the currently playing effect on the same channel. Longer sound effects are assumed more important.
3. A pulse sound effect is played on pulse 2 if its remaining data length is shorter than that of pulse 1, such as if pulse 1 is playing a sound effect and pulse 2 is free.
4. The next version supports an "attack channel" that replaces a given channel's attack and decay phases with a different note. This allows, say, an ostinato and an arpeggiated chord to share a channel as heard in Super Mario Land NES Mix.NSF. Or it can make single channel echo (as heard in the intro of Sappy and Kilken) simpler, by playing the main part with attacks and the echo with sustains.

FamiTone does this. I haven't heard any comparisons (and they would depend on case by case basis), so I don't know if it's an improvement.

Personally I decided to not implement it, because I felt sound effects should never be obscured by music because they give feedback to the player. But then again, I didn't do any testing.


Did you arrive on these rules based on testing or intuition?

The rules were derived based on testing in a block puzzle game supporting two players. Think like Dr. Mario except on speed, as if both players were as fast as Red Virus from the Super NES version. Having two players pumping rapid fire sound effects into the system without obscuring the music too much requires a fairly robust set of heuristics.

Sure, sound effects give information to the player. But often, most of that information lies in the loud beginning of the effect, not the quiet part at the end as it trails off. For example, the MIRV splitting sound effect in Thwaite has some single channel echo going on in it, but it's quiet at the end so that the attack of the next note covers up the echo.

I recall I had to implement some weird logic to handle $4003/$4007 writes. I'm pretty sure it had to only write when the value would change, but I think there was more to it than that.
But in terms of actually handling each "layer", I have separate virtual channels for each, that in the end merge into the real registers.

For simplicity's sake, I'm pretty sure only player actions make sound in my game. This includes the player getting hit, hitting an enemy, picking up an item, firing a gun, etc. If a sound effect is "requested", any sound effects in progress will cease and the newest sound effect will take over. And they happen no matter what. I haven't noticed any obnoxious mishaps from this logic thus far.

I also don't use the sweep registers for anything, personally. I prefer software sweeps; I have more control that way.

I'm pretty sure ~90% of sound engines made for the NES have an internal volume parameter for the triangle channel. At least mine does. It writes volume OR $80 to $4008, this way if volume is 0 then the channel is silent, otherwise it plays.
What I do is that I have an "OR" mask for the 1st sound reg, which by default is $80. Square channels are supposed to always have a duty cycle change action before playing music, which effectively change the OR into $30, $70 or $b0. Triangle is supposed to never have a duty cycle change action, unless you want to use the timer (but then only 2 lengths, $30 and $70 are available, both offseted by the volume but it changes only the length a little). This way all channels are processed equally from the sound engine, it does it the same no matter if it's a square, triangle or noise channel. The only problem I have is that I have to do a dummy duty cycle select on every song for the noise channel, because it has to be OR-ed with $30 (or $70 or $b0 but I don't use those). I haven't found a way to thread all 3 kinds of channels on a perfectly equal ground alas, but it's pretty close. I also have a special routine for the noise's pitch, because well, it is just to different to be handled by the regular routine.

I also have a system where a new sound effect will always take on another sound effect, but if I have a square channel free and the other one taken by a sound effect, it will use the free square channel. With my current system I could play a different sound effect on every channel at the same time with no problem.


I'd prefer sofware sweeps if I had a way to update my sound engine at 240Hz. Until that, I am stuck to 50 or 60 Hz which is too slow for sweeps so I'm forced to use hardware sweeps.
It's the only really useful hardware features the NES has to offer in my opinion. Hardware decay is useless, and hardware timer even much more useless. When it comes to triangle why does it have TWO timers at the same time when both are useless since it's very easy to do the same thing in software ? I don't get it. Well it's not completely useless because you can have timings shorter than 60 Hz (Blaster master uses this at game over screen), but it's definitely not terribly useful.

Are you using a mapper with a scanline counter IRQ? If so, set it to fire on lines 25, 90, 155, and 220. Are you already using DMC for anything? If not, you can press the DMC IRQ into service as a 522.7 Hz IRQ.

No I'm using mapper 3. And I don't use DMC for anything (as I indirectly stated in my post above...). However I do use poll-for-sprite zero hit loops, and have music engines in IRQ could break them. I know it could be solved with a "inc flag" type of IRQ, but well... I just don't feel like doing it.