Audio rendering?

I'm pretty much a newbie to "advanced" programming on Windows. I was better on the Amiga back in the days.
I'm curious, how are you guys generating the audio in your NES emulators on Windows? The Amiga had 4 soundchannels that could be manipulated in realtime (which made NES audio emulation, well, not that hard anyway).
But things work differently on my PC obviously. Audio has to be rendered to a buffer and then played?
I am no soundexpert at all, but doesn't that seems a bit difficult when generating NES audio? You really have to know what you're doing concerning samplerates/frequencies etc.
Or is there a shortcut?
I've checked out some blip_buf.dll but that was obviously not for mortals like myself.

Yes, you generate the audio yourself as a raw PCM stream, it gets copied to a buffer, usually something like 4096 samples at a time.

There's an inherent delay in the buffered approach, though, but this is to be expected on most modern platforms. The delay is usually small enough not to be a really noticable problem, and you can try decreasing the buffer size to make the delay smaller, at the risk of skipping if the thread ever gets starved for too long. (There is also a special interface called ASIO for very low latency audio on PCs, but it's usually reserved for musical applications.)

Would D-Pad Hero be a "musical application" in this sense?

To me it seems difficult to calculate those waves in realtime (more or less), applying sweeps and such..

Read through Blargg's APU reference if you haven't already (http://nesdev.com/apu_ref.txt). The block diagrams are really helpful.

It's all pretty complex to be honest. To render audio and how the waveforms should look at a certain time, well, it's a alot to think about (AND understand ).
If there's any friendly english/swedish soul in here with skills in this area and good knowledge of C#, feel free to drop me a pm.

Why PM? Why not keep the conversation public so that others can benefit from it?

PCM data is fairly simple to understand. You can think of a waveform as a digital recording of how the cone in the speaker is going to vibrate. Several "samples" at taken...each of which are effectively the cone's position at a particular point in time.

This image may help:
https://upload.wikimedia.org/wikipedia/ ... cm.svg.png

The Y axis here would be the value of the sample (position of the cone), and the X axis would be time. The "taller" the wave, the louder it is. The "wider" the wave, the lower pitch it is.

The NES produces its waveforms by rapidly changing output values for a channel. For example, the pulse channels form a wave by rapidly alternating between '0' output and 'V' output (their volume). Effectively, the channel is clocked every cycle*, and that clock is divided by a down-counter (the "period"). When the period loops, the channel changes its output**. The period then acts are sort of a "delay" which dictates how much of a gap there is between the change in output. Smaller period = smaller delay.

So with a small period, (and with volume=8), the pulse channel might output the following:
00008888000088880000888800008888

But a longer period might output this:
00000000888888880000000088888888

If you were to plot that output out in a method similar to the above image (Y axis=output value, X axis=time), you'd see that it forms a sort of "square" shaped wave. The wave is basically how the speaker cone moves, generating your output.

The interesting thing to note here is that the 'height' of the wave is dictated by V. Higher values for V will produce a taller wave, resulting in a louder sound. And the 'width' of the wave is dictated by the period. So a longer period will create a longer delay which results in a wider wave, and therefore a lower pitch. So the period is effectively the pitch control.



You can think of the NES as outputting one sample every CPU cycle... effectively having a samplerate of 1.79 MHz. The tricky part is scaling that data down... because PCs typically only want to output 44100 samples every second. The easiest way to do this (and the way I recommend starting) is just to do a "nearest neighbor" approach:

1789772.7272 / 44100 = ~40.5

So you can downsample the audio by ignoring APU output for 39 cycles, then using the 40th cycle... then ignoring 40, using 41st... then ignoring 39, using 40th... etc.
This will more or less work, and is VERY easy to implement, but produces low quality sound (it'll sound grainy, particularly with higher tones).

There are higher quality approaches that can be done here -- but I wouldn't concern myself with that yet. Just get it outputting something, and worry about making it sound good later.



_{*Technically I think the squares are clocked every APU cycle, which is every OTHER CPU cycle.
**The period divider actually clocks the duty cycle generator, which is responsible for changing the output.}




I'm happy to help with questions / clarifications.

I understand the theory of waveforms, but the idea of generating them in realtime and dealing with it's effects, well that's a different story.
Ok, so you can render them byte per byte after one clockcycle, I get that. But you still have to keep track of the waveform when it should "shift shape" (sorry If I use bad terminology here, but perhaps you get what I mean anyway).
Assume you start playing a normal squarewave. How do you keep track of how it "should look"? I mean, init the square wave, and if you don't touch any of it's controls, how does the waveform look a few ms later?
I had some hopes this could be done on vertical blank basis instead of after each clockcycle (since most ROMs afaik do their music-stuff at vertical blank anyway), but perhaps that's even more difficult with a buffer..(?)

Doing everything at vblank basis will make Big Bird's Hide & Speak not work, as it writes directly to a DAC at $4011 roughly 8000 times a second while Big Bird is speaking.

That's what the period does. The logic here is actually pretty simple:

Every clock:
- if counter is nonzero, subtract 1 (and do nothing else)
- otherwise (counter is zero):
-- reset counter to period value
-- have channel take another step through its waveform generator (update duty for pulse, tri-step for tri, shift reg for noise, etc)


It's that easy. Channels produce output every CPU cycle (though, again, you want to downscale it) -- and that output is determined by the state of their waveform generator (duty, etc).

In the case of the pulse channels, the duty simply cycles through an 8 step sequence, with some steps being "high" (output current volume) or "low" (output 0)




You render audio samples the same way you render pixels. Just as 1 PPU cycle [roughly] generates 1 pixel.... 1 APU cycle generates 1 audio sample (though again -- scale it down)

You probably have a buffer that your PPU gradually fills up with graphic data as it progresses through the frame, right? So do the same with audio. Give your APU a buffer that it can progressively fill up with samples as it advances through the frame.


As tepples mentioned, you should not emulate the APU on a once-per-frame basis. For the same reasons you wouldn't want to emulate the PPU that way.

With "period", are you refering to some special bits in the audioregisters?

I'm referring to the 11-bit F-value that is written to $4002,4003 to control the pitch.... and the similar values that exist on the other channels.

That value effectively sets the "size" of the clock divider -- which also means it controls the pitch of the generated tone. Ex, if $135 is written to $4002,4003... then Pulse 0's duty cycle generator takes one step every $135+1 APU cycles.


EDIT: Apparently wiki calls it the "timer". The 'T' bits outlined here:

http://wiki.nesdev.com/w/index.php/APU# ... 00-4007.29

I remember blargg calling it the period in his apu doc... I guess that name stuck with me. =P

So, when the frequency value (or "period" as you call it) reaches zero, has an entire waveform "passed" by then?

Thanks for taking the time to answering my newbie-questions btw.

Frequency is how often something happens.

Period is how long something is.

If something is repeating, its frequency and period are very closely related, but the reason we call the values in the registers "period" is because as that value gets larger, the period of the waveform gets larger. They are a direct correlation. The frequency of the waveform, on the other hand, is its inverse (it gets lower as the register's value gets larger).

Perhaps I mixed up the terminology...
The 11-bit "length" describes the length of the squarewave (for example) then? When it reaches zero an entire squarewave has been "rendered"?

Pulse waves in the 2A03 are made up of eight states, four high, and four low. It takes (period + 1) APU cycles, or (2 * period + 2) CPU cycles, to finish one of these states and advance to the next. This means it takes (8 * period + 8) APU cycles, or (16 * period + 16) CPU cycles, to finish one entire cycle of the square wave.

I learn best by example... so let's look at an example.

Using this page for a quick reference:
http://wiki.nesdev.com/w/index.php/APU_Pulse

If you look at the "Duty Cycle Sequences" section, you can see that when the 'D' bits of $4000 are set to %10, the channel is configured to have a 50% duty. This means it will walk through the below sequence:

0 1 1 1 1 0 0 0

With 1s signifying high output and 0s signifying low output.

Let's also say that the channel has an output volume of 7 (low 4 bits of $4000), and let's assume other tricks like envelope and sweep are disabled (don't worry about that stuff for now... the only things actually needed to generate sound are volume, duty, and period/length -- other things are just for effects and can be worked in later)

Lastly, let's say the channel's length/period/timer/whatever you want to call it (11-bit value set via $4002,4003) is set to $002 *


The channel will have two internal counters:
A) keeps track of the timer/period countdown
B) keeps track of its position in the duty cycle

Here's what will happen every APU cycle:


Notice a few things:
- when period = 2, the duty is clocked every 2+1 cycles
- this will produce an alternating 0,7 output pattern, which is the generated pulse/square audio wave.
- volume controls the height of the wave. Taller = louder
- period controls the width of the wave. Higher period = Wider = lower pitch.


_{* a period of $002 would actually be crazy high pitched, and in fact the pulse channel's sweep unit would forcibly silence it, so this example is not really realistic and technically would result in all 0 output on the NES -- but nevermind the sweep unit for now. The above example is pretty much how it works and is conceptually sound. If I wanted to be hypertechnicaly I would have to have chosen a period >= 8, but I wanted to keep the example as short as possible, so whatever.}

Learning by examples are great, I agree.

Thanks Disch for a great example! I think this stuff is really interesting..

EDIT: I made a simple C# prototype based on your example. Sounds really cool. Let's see how I should proceed with this stuff..

I've started to implement this now, it logs to a pcm file at the moment and it seems to be working sort of.
Remains to understand the rest of the register-bits to get it to sound better.
The "envelope decay", is this some sort of volume-fading feature?

"Envelope decay" is an automatic fade-out (or automatic volume change).

It operates on the same 'period' concept... you provide a period P and after P+1 clocks, the output volume will change.

The key thing to note, though, is that the clocks do not come from the APU clock, but rather from the Frame Sequencer, which is almost like an entirely separate subunit within the APU.



The sweep unit operates similarly, but changes the pitch/timer, rather than the volume:

If I understand correctly.. When bit 4 of $4000 is set, the volume is decreased at 240/(bit0-3 of $4000) Hz?

@oRBIT2002

You see. The NTSC refresh rate is 60FPS (or very near that). So, if a unit runs at 240hz, it's updated 4 times per frame. Well, you got it - 120hz means 2 times per frame, and so on. The frequency value is merely the number of CPU cycles to wait.

Hidden 4-bit counter: let's called this 'envvol'.
- envvol = $F after a write to $4003 (but not immediately after... see wiki for details)
- envvol is not directly accessible

$4000.0-3 (low 4 bits of $4000): let's call this 'V'

When $4000.4 is set: Envelope is disabled, V is used as output volume

When $4000.4 is clear: envvol is used as output volume... V is instead used as a period: envvol is decremented roughly 240/(V+1) times per second



Note that the 240 is not really 240 but is a clock signal that comes from the Frame Sequencer. See this page:

http://wiki.nesdev.com/w/index.php/APU_Frame_Counter

Someone should write a "NES Audio for dummies" document.

When reading about the framecounter on the wiki, it only specifies maximum 5 different steps but if the envelope is active there should be 15 (bit 0-3 of $4000)?

The frame counter loops. So after it does those 4 (or 5) steps, it just keeps repeating them.

The thing is there's several APU cycles (~3700) between each step of the frame counter. So there are ~3700*(V+1) cycles between volume changes for the decay.

How do you guys buffer your audio for playback? My initial thoughts was something like this:
Assume 2 buffers of XX ms.
1) Play buffer 1 & fill buffer 2
2) Play buffer 2 & fill buffer 1
Is there a smarter way of doing this? Probably? I'm using SlimDX/DirectSound if that matters...

Yes, double buffering in that manner is how it's usually done.

I'm struggling with playback of my buffers. I don't think it's a performance-issue but something else. I'm using two buffers that I'm switching between at X(X) frames interval, but the playback isn't smooth. The generated audio is OK so obviously something else is wrong.
I've tried playing the buffer at end of each NES-frame (or as I mentioned, at XX frames intervall) but nope. I've also tried setting up a Windows Timer (C#) that runs at 60fps but that doesn't work very good either.
Anyone with hints of what I might have missed?
Thanks in advance.

Audio streaming is tricky because unlike a graphic image, it is a function of time. What I mean by that is that you can generate a graphic image and put it on the screen for as long (or as short) as you want it... but with audio, your chunk of audio has an exact, fixed length of time, and it can't reach further or shorter than that. There are tricks you can do to stretch/compress audio so that it takes a shorter or longer length of time, but they're tricky.

Think of it like an hourglass. You have audio data (sand) that you pour into the top, and it will slowly drain out the bottom (play). If you keep filling the top at roughly the same rate it drains, you'll be fine. But problem come up in two instances:

1) You run out of audio/sand (buffer underrun). In which case audio playback abruptly stops and creates breaks or hiccups in your playback.

2) You have too much audio to fit in your buffer and end up having to discard some of it (buffer overrun). This will cause skips in the playback.


You almost certainly are having one of the above two problems.


Keep in mind the NES does not run at exactly 60 FPS (it's closer to 60.1). I've found that the easiest way to keep audio synced is to have the audio playback drive the framerate, and not the other way around. "Sync to sound" is what it's often referred to. Basically you keep an eye on how fast your audio is draining, and once it drains to a point where you have another frame's worth of free space, then you emulate another frame.


EDIT: note that even that gets tricky if your audio lib takes huge chunks of audio at a time instead of a gradual drain.

I'm not sure about the specifics on Windows, but I think you're confused about something here. You shouldn't have anything time-related in your audio code. The sound hardware keeps its own time, and all you should care about is pumping data into it as fast as it can handle. Buffers should be completely filled and queued for playback as soon as that happens.

I need at least three buffers, preferably four, to get decent sound on Linux, but that's partially because it takes way too long to notice when a buffer is ready for more input. I also like to spin a second thread and pump the audio loop at a higher framerate than the video - it cycles at 100 FPS, but will output as much audio as it can, not the same amount each time. This has the added benefit of not stuttering when the framerate gets choppy.

When you use WaveOut on Windows you can do a little dirty thing and do your buffer commits in the callback which pretty much guarantees problem free playback.
You have to stop committing new buffers and wait for current ones to finish playing prior to closing WaveOut or it'll hang. Works perfectly from Win95 and up

Can you explain this a little bit more perhaps? What do you mean with "buffer commits in the callback".

Currently I'm using SlimDX for DirectSound but /!"(#/ there almost no examples anywhere on how to use it properly.

I cannot help with DirectSound, only WaveOut.

WaveOut provides a callback so you can handle things immediately without having to wait for things to finish etc.

https://msdn.microsoft.com/en-us/library/windows/desktop/dd743866(v=vs.85).aspx?f=255&MSPPError=-2147217396

I've done a simple que-system.. XX ms is buffered and added to the que. The player (which calls itself at XX ms intervall) takes the oldest from the que and plays it.
Still got issues however.. This was alot more annoying/difficult than I thought it should be..