How to properly playback audio when creating an emulator

This is an archive of a topic from NESdev BBS, taken in mid-October 2019 before a server upgrade.
View original topic
How to properly playback audio when creating an emulator
by on (#186697)
I started writing a gba emulator a few weeks ago, and at this point I decided that would be nice to add sound support to it. The emulator is being written is C with the SDL library, so I used SDL audio to play the sound stream. Atm I calculate the amount of samples I need to generate given the gba cpu frequency (16MHz) and my sampling rate, which is 32khz, which gives me 512 cycles per sample ((2 ^ 24) / 32768), so I basically generate a sample after each 512 cycles and write it to the ring buffer, at the position of my write cursor, and then increment the cursor. When SDL calls my callback function, I copy the data starting on the playback cursor position, the length of the data copied being the size of the SDL stream.

So the process is basically this:

- Write samples to the ring buffer periodically (put in on ring buffer, increment write cursor)
- When SDL calls the callback (which means that it needs more samples to play):
-- Copy *length* samples from ring buffer to the SDL stream, playback cursor += *length*

This method kind of works. Actually, it would work perfectly, if the amount of data being written and the amount of data being requested was the same, but unfortunately it isn't. What happens it writes a bit more of data than whats needed. Even through the difference is small, this makes the write cursor get more and more ahead of the playback cursor. After some time (usually a few seconds, depends on the size of the ring buffer) it overwrites data that wasn't even played yet, and that's when the desync occurs. At this point it starts making some weird echoing sounds (or just some weird noise if the buffer is very small) until it gets in "sync" again.

To aliviate the problem, I had a idea to automatically correct the playback cursor over time. The code to do this was really simple, after each copy (ring buffer -> sdl stream), I calculated the difference between the write cursor and the playback cursor (ideally, the playback cursor should be at the position of the write cursor at this point), and divided it by some amount (which was basically an acceptable margin of error), and added the result to the playback cursor. With this code in place, it could automatically correct itself over time, skipping a sample or two, and the loss wasn't even noticeable.

It worked pretty well, for some time, but the problem would still randomly show up. Doing things that could mess up the sync, like moving the window around or even minimizing stills makes the problem go back easily too. So this was no real solution.

So, the question is, how this is usually done? I already played in a bunch of emulators and never saw problems of this kind, so I'm probably doing something wrong here. Thanks in advance. Also sorry if this is the wrong place (I saw the emulation board, but it seems to be about NES emulation only, through this applies to any kind of emulator, hmm...)
Re: How to properly playback audio when creating an emulator
by on (#186809)
Quote:
To aliviate the problem, I had a idea to automatically correct the playback cursor over time. The code to do this was really simple, after each copy (ring buffer -> sdl stream), I calculated the difference between the write cursor and the playback cursor (ideally, the playback cursor should be at the position of the write cursor at this point), and divided it by some amount (which was basically an acceptable margin of error), and added the result to the playback cursor. With this code in place, it could automatically correct itself over time, skipping a sample or two, and the loss wasn't even noticeable.


That seems the logic thing to do. That's how I would have approached it.

Quote:
It worked pretty well, for some time, but the problem would still randomly show up. Doing things that could mess up the sync, like moving the window around or even minimizing stills makes the problem go back easily too. So this was no real solution.

Those sound like different uses though.
Re: How to properly playback audio when creating an emulator
by on (#186925)
gdkchan wrote:
So, the question is, how this is usually done?

Most emulators sync to the audio playback rate and adjust the video rate to accommodate. This results in tearing (no vsync) or stuttering when scrolling (vsync).

You're doing the opposite: adjusting the audio rate to match the video rate. I think this is the better approach and what I do in my emulator.

Your implementation sounds similar to mine, but perhaps needs to be tweaked a bit. The one thing that stands out the most is that you're adjusting your audio rate based off of an instantaneous measurement. Try making adjustments based off of several intervals of measurements. This will avoid problems when you run into random latencies (a missed vsync interval, OS scheduler latency/jitter, etc.). Also, instead of trying to directly change to the new rate, slew towards it by making smaller adjustments.

Quote:
Doing things that could mess up the sync, like moving the window around or even minimizing stills makes the problem go back easily too.

There's not much you can do about this, but your rate adjustment algorithm should be able to quickly recover.
Re: How to properly playback audio when creating an emulator
by on (#186928)
James wrote:
Quote:
Doing things that could mess up the sync, like moving the window around or even minimizing stills makes the problem go back easily too.

There's not much you can do about this, but your rate adjustment algorithm should be able to quickly recover.

Windows sends WM_MOVE (or WM_MOVING or WM_WINDOWPOSCHANGED) and WM_SIZE (or WM_SYSCOMMAND + SC_MINIMIZE) messages -- I'm not sure of all the circumstances that trigger each of these individually -- when they happen. So when you say "there's not much you can do about this" my reaction is "Uh, what?" -- you halt the emulator internals the instant said messages are received, and resume once the situations have ended (mouse button released, etc.).

One thing that really chaps my ass about emulators is that many get these wrong and how badly it affects the overall UX. It would take me hours to make a list of all the problems I've seen across all the emulators, including present-day ones. You might be surprised how a user's impression over the "stability" and "reliability" of an emulator are based on those aspects of the UI and not just the actual emulation bits themselves.

Oh, and I love UIs in Windows that do something (toggle checkbox, run button code, etc.) on WM_LBUTTONDOWN instead of WM_LBUTTONUP. *sarcasm*
Re: How to properly playback audio when creating an emulator
by on (#186930)
koitsu wrote:
So when you say "there's not much you can do about this" my reaction is "Uh, what?" -- you halt the emulator internals the instant said messages are received, and resume once the situations have ended (mouse button released, etc.).

"This" meaning things getting out of sync. Recovery from this scenario should have zero to negligible impact to the user, which I alluded to when I said the algorithm should quickly recover.

Quote:
halt the emulator internals the instant said messages are received, and resume once the situations have ended

Unfortunately, that doesn't really work. Because this method is, in essence, tracking the soundcard buffer's playback position, a delay at the application layer means that the next callback will most likely occur out of sync with the last.
Re: How to properly playback audio when creating an emulator
by on (#186937)
James wrote:
Quote:
halt the emulator internals the instant said messages are received, and resume once the situations have ended

Unfortunately, that doesn't really work. Because this method is, in essence, tracking the soundcard buffer's playback position, a delay at the application layer means that the next callback will most likely occur out of sync with the last.

Yes, there will be some desynchronisation, but per the method you described (adjusting audio playback rate slightly, and using small adjustments rather than large ones), the resynchronisation time is *extremely* short -- I would say a 1/4th a second *at most*? If you want examples of this, all in windowed mode, left-click (and hold) on the title bar of the application window while A/V is actively occurring:

* NestopiaUE -- re-sync time is extremely short (probably 1/16th a second)
* FCEUX 2.2.3 -- re-sync time is extremely short (a la NestopiaUE), often with clicking (heard on multiple sound cards, no clue why that happens)
* Mesen 0.7.0 -- nothing pauses or gets interrupted (A nor V!) -- smooth as silk, including if you move the window around
* Fusion 3.64 -- audio buffer continues to play/loop once left-click is held, for ~1 full seconds, then stops
* snes9x v1.53 (64-bit test builds) -- re-sync time is extremely short (a la NestopiaUE)
* bsnes-plus v073+2 -- audio buffer continues to play/loop once left-click is held, for ~1 full seconds, then *picks up where it left off* (?!?!?!?!?!) -- recovery happens even faster if you actually *move* the window

The variance is remarkable. But you can see clearly that very quick A/V resync/recovery is certainly possible, almost certainly through the method you described. I also agree that adjusting the audio rate is a better overall thing to do -- the variance when this happens is pretty minor, if I remember right (thinking back to the thread you had for nemulator; I still remember it :-) ), meaning the odds of someone noticing is virtually nil (while in comparison, visual desync like what you describe is *very* easily noticed).
Re: How to properly playback audio when creating an emulator
by on (#186943)
We're talking about two different, but related, issues here. I'm referring to synchronization in the context of this particular method of adjusting audio rate. In other words, how to (re-)determine the correct audio playback rate following a desync event. The issue you're referring to is applicable to all emulators, and I agree that window size/move events should be properly handled (I simply pause emulation and stop audio playback while sizing/moving).
Re: How to properly playback audio when creating an emulator
by on (#186944)
Thanks for the help everyone. Was working on cleaning up the mess of the code a bit so I could push it to my git.
The current source can be found here: https://github.com/gdkchan/gdkGBA/blob/master/sound.c

the sound_mix procedure is basically what fills the SDL stream with the samples, and also what increments/adjust the playback cursor. Atm it stills seems to desync randomly during gameplay, and I can also hear some weird frequency changes for a brief moment sometimes. tbh it was working better before, maybe my lastest tweaks made it worse. I will try to follow James advices and maybe try averaging the difference between the cursors over time.
Re: How to properly playback audio when creating an emulator
by on (#187178)
I've posted a thread in the NESemdev forum describing the method used in nemulator, with links to an example implementation: viewtopic.php?f=3&t=15405