Please test nemulator's audio resampling

A recent thread discussed methods for synchronizing audio and video rates. nemulator dynamically resamples audio to match the video frame rate. I think this works well -- video stays locked to vsync so there is no tearing or stuttering, and the changes in audio frequency are not noticeable.

I've received good feedback from people who've tried it out, but I've never really solicited feedback from this group. Please give it a try and let me know what you think. A couple of notes:
- F9 toggles display of stats
- F4 toggles a graph of output frequency (green line) and output buffer position (red line; red area is the target that it's trying to keep the buffer position at)
- If you have a non-60Hz display, I'm especially interested in how well this works as I've been unable to test (make sure app.timer_sync is set to true in nemulator.ini)



Thanks,
James

Yeah, I think you might want to take out the unlimited sprites option. Some games that I don't know of might rely on the limit.

Some games place a bunch of sprites in the same scanline to do sprite masking. Zelda 1 and Gremlins II are two games I can name of the top of my head that do this.

If you want to throw out the sprite limit, but still support this kind of sprite masking, you can detect it easily. Look for 8 early sprites (may or may not include sprite 0) with the same Y coordinate and same tile number, that's a sure sign that it's trying to mask those scanlines.

edit: Sorry, can't test the emulator, I'm on Windows XP.

Can't test the emulator, but about the unlimited sprites feature:

Please make this a toggleable feature. There are lots of games that look better with the sprite limit disabled, but there are also lots of games which behave oddly/look awful unless sprite limit is enabled. Here's a list of games I've found which look/behave oddly when sprite limit is disabled:

* Castlevania 2 -- when Simon is standing a swamp the lower half of his body is visible
* Majou Densetsu 2 - Daimashikyou Galious -- when Populon enters a door, his entire body is visible as the sprite is moved downwards. Just see these two videos I made some time back: #1 #2.

I'm certain there are others, but those are the two I can name off the top of my head. So please, like I said, make this a toggleable feature (default should be sprite limit enabled, just like the real hardware).

It is. F7. Also configurable in nemulator.ini (globally or on a per-game basis).

I think Gimmick! also rely on sprite limit. I noticed this on Nesicide when I put a "breakpoint on sprite overflow", at the beginning of a level; that makes the effect of the hero popping out of the gate.

Super Spy Hunter won't spawn enemies properly without sprite limit emulation.

Works fine for me with 60 hz refresh rate, but with 50 hz I get pops whether the timer sync option is on or off. Looks like the sample rate can't adjust enough. This is important because a lot of laptops, mine included, drop the display refresh rate when running on battery. Here's your statistics and graph.

I got the same results as Grapeshot; audio is not correct at 50hz refresh, no matter what. With timer sync, audio is correct at 75hz refresh, though.

I suppose that's expected, as it doesn't make sense to slow down to 50Hz, though this is the first I've heard of LCDs with that refresh rate. Disabling vsync (nemulator.ini: app.vsync = false) should fix that. Maybe I should automatically disable vsync if the refresh rate is less than ~60Hz...


ok, good. Is video smooth at this refresh rate, or do you see any stuttering when scrolling?

Thanks for testing, guys.

Studdering. However, when checking that, I came across another problem: timer_sync doesn't seem to be working correctly.

With 60hz refresh rate, vsync=off, timer_sync=on gives a framerate of 50fps.

With 75hz refresh rate, vsync=anything, timer_sync=on gives a framerate of 50fps.

With 75hz refresh rate, vsync = on, timer_sync=off, things work correctly (60fps framerate), just with some video studdering as expected.


I wasn't aware refresh rates other than 60hz were so uncommon? My LCD can sync to anything in [50..75]hz.

Is there any trick to getting Nemulator to work on Linux with Wine?

are you using true/false for the settings or on/off? Should be true/false.

re: 50Hz refresh rates. I don't know how common it is/isn't, but all of the LCDs that I have or have access to only run at 60Hz.

hmmm... don't know. Does it even start?

I am using true\false. Sorry for the murky writing; I kind of jotted a bunch of stuff down. This should be much more explicit:

75hz monitor setting

vsync = false, timer_sync = true: 50FPS
vsync = false, timer_sync = false: high (>150) FPS on menu, 60FPS in game
vsync = true, timer_sync = false: 75FPS on menu, 60FPS in game
vsync = true, timer_sync = true: 50FPS

50hz monitor setting

vsync = false, timer_sync = true: 50FPS
vsync = false, timer_sync = false: high (>150) FPS on menu, 60FPS in game NOTE: while there is tearing and studdering, this does sound good
vsync = true, timer_sync = false: 50FPS
vsync = true, timer_sync = true: 50FPS

Most of the internal laptop LCDs I've seen are set to 50Hz.

It's a power saving thing in the latest Intel IGPs and it's very difficult to turn off without registry hacking.

What Grapeshot says is correct, to my knowledge. Some nVidia mobile/laptop chips are known to do this as well -- doesn't require registry hacking in that case though; in the Advanced area of the Power Management panel in Windows 7 there's a toggle for it. I know because on my work laptop, after upgrading my nVidia drivers one day, I noticed the display to be "flickery" intermittently ("the fuck? That looks like 50Hz. I'm NTSC, jackass!") and found the feature they had added + disabled it.

This is just another reason I said what I did (2nd paragraph) and why I disagree with this (last line). Expecting things like EIST, CPU P-states, and GPU power-saving features to be disabled = wrong mentality for software design.

nemulator, if configured to do so (the default setting), will automatically disable CPU throttling upon startup, and restore the previous setting when exiting. So, I don't expect any particular configuration for it to work properly.

Re: 50/75Hz settings - 50Hz with vsync is just never going to work. 75Hz should work, albeit not as smoothly as 60Hz. I'm not sure why the timer sync stuff isn't working for you. With it enabled, and vsync disabled, you should be running at a rock solid 59.97Hz...

I found an LCD that will run at 75Hz. Will test on it when I get a chance.

When you say "CPU throttling" what exactly do you mean? Are you talking about EIST? Are you talking about processor P-states? And how are you disabling it (explain the code)?

If you're doing something like actually disabling a setting (registry, power profile, etc.), this is completely unacceptable. If your emulator crashes, or gets stuck and someone end-tasks the process, that setting is going to remain disabled. This is akin (in my mind) to emulators which used to do things like change screen resolution + bit depth (I can think of quite a few Genesis emulators that did this). Please don't change peoples' system settings. I've also seen programs which to inhibit a screen saver coming on or monitor power-save (standby) while using their software, actually changing the OS/UI settings.

Windows offers a way to inhibit an OS feature by using a WM_* message (probably WM_SYSCOMMAND); that's the way to go. That method is completely fine. In fact, that's the proper way to inhibit the screen saver/monitor standby. Below is a small (4KByte) document I wrote back in 2006-2008 for the folks who maintained the Windower program (which allowed FFXI to run in a window), explaining to them how to inhibit such. Maybe others can benefit from what I've written.



There may be other ways to do this to inhibit P-states but I don't know how at this time. I'd have to dig deep in MSDN.

Edit: added & 0xfff0 per Microsoft requirement.

Yes: http://en.wikipedia.org/wiki/Dynamic_frequency_scaling

I make a copy of the current power scheme, adjust the appropriate settings in the copy, and activate it. Upon exit, I switch back to the scheme in use at startup and delete the copy.


Yep. I'm not 100% happy with it either, but I know of no other way to handle it.

What you're doing (changing the power scheme -- yes I know how to do this, i.e. POWER_POLICY, GetActivePwrScheme(), ReadPwrScheme(), SetActivePwrScheme) is bad for the reasons I listed. I can assure you that in the future you will get hounded and scolded for this. Please do not do this. And yeah, I know about this as well.

The effect is going to be awful if your emulator crashes or someone end-tasks it then wonders why their laptop battery life suddenly drops to crap -- permanently. If someone clever figures it out, they're going to go "what the hell is changing my power profile setting?!?!?!" Nobody is EVER going to consider that an emulator is the cause. Please think about the repercussions of this "solution".

Can you explain, verbosely, why it is you need to disable these CPU features?

Even on *IX systems I've never heard of anyone ever needing to disable these states to get smooth audio playback or to sync to a specific frequency/clock rate. I also don't see why it's necessary either -- if existing emulators don't need to do this (I can list off quite a few), why does yours? This sounds more like a software/application design problem that needs to be solved in a different manner.

I was under the impression that some operating systems determined whether a machine was idle based only on the keyboard and mouse, not any connected joysticks, and used the idle status to determine whether to enter deeper power-saving modes. In such a case, the user would see a problem only if playing the game with a joystick, not a keyboard.

I remember that one of Tepples's own PC programs had a problem where it would alter Windows settings (colors of controls) if it crashed or was killed.

Which program might that have been? I don't remember having inserted any calls to Windows services to change system color settings in any of my own programs. The only system setting that I remember changing is the screen resolution and bit depth when a program is run in full screen. Might Windows have altered the colors itself if it crashes in a 256 color mode and for some reason can't restore the 16-, 24-, or 32-bit mode that your desktop normally uses? Was it running under Windows 9x or under Windows XP?

Status: incomplete

Perhaps. But you're the first since this functionality was introduced in 4/11


The temporary power scheme name is "nemulator" and the description is "Copy of %s scheme. Used by nemulator to disable processor throttling." I do have one change in the works though: if nemulator starts up and detects that the nemulator scheme is already in use, indicating an unclean shutdown, it will revert to the scheme that was in use before the previous scheme change.


I'm unaware of any other emulator that simultaneously provides smooth video output (no tearing, no stuttering) and glitch-free audio. It's a differentiating feature of nemulator and one that I wish that you could try firsthand. I think it's a worthwhile trade-off.


I'll try. To recap, emulators generate video and audio output at slightly different frequencies. To keep everything in sync, most emulators block on audio output, effectively locking the frame rate to the audio output rate. This results in glitch-free audio output, but tearing or stuttering video output.

Alternately, an emulator could lock to 60Hz vsync to provide artifact-free video output, but there will be audio glitches when the audio output buffer over-/under-runs.

To address this issue, nemulator uses vsync for timing (except when it doesn't, but that's a different discussion), and dynamically adjusts the audio frequency to match the current frame rate. Video is output at a consistent 60Hz, audio output is never blocked waiting on the output buffer to drain, and the audio buffer is never under-run.

In order to determine the difference in audio generation and playback rates, I check the playback and write cursor positions of the sound card's primary output buffer. If the distance between the two pointers is consistent -- the goal -- that means that the buffer is being drained at the same rate that it's being filled. If it's not, audio frequency is adjusted to compensate.

So, why does processor throttling interfere with this? When processor throttling is enabled, processor frequency -- and therefore the time between vsync occurring and the audio buffer being filled -- will vary depending on system load. Usually, this 'jitter' isn't too bad, and can be accounted for. But on occasion, there will be large differences that can result in under-runs. Here's an example. The red line represents the number of bytes between the playback and write cursors:



With processor throttling disabled, latency is much more consistent, and large differences generally don't occur:



There's a couple of ways to account for the increased jitter with throttling enabled. I can increase the buffer size, but that can result in noticeable latency between video and audio (e.g., you hear Mario jump later then it happens on-screen). I can make larger/faster adjustments to audio frequency, but that can result in noticeable pitch changes.

So, at the end of the day, temporarily disabling processor throttling provides the best results. I agree that there are downsides, but I think the benefits outweigh those.

I think I understand the concern -- thank you for taking the time to explain it -- but the screenshots in question are quite a bit confusing/difficult to interpret. At this point they mean more to you than they do to me.

All I can discern is that the top screenshot shows two red lines (am I supposed to be looking at the red gradient line, or the slightly-brighter-but-slightly-not line?), and that there is more variance there (on the vertical axis) than in the lower screenshot. This doesn't tell me how much (on a numeric level) delta there is between the audio playback position vs. the audio write position -- what's rendered in the top screenshot could indicate a 10ms delta, or it could indicate a 1000ms delta.

I also assume that the horizontal axis indicates time, i.e. this red-green-mess ( :-) ) is effectively panning/scrolling leftward at all times, with new information drawn on the right-hand side.

At some level I still do not understand why the audio and video pieces have to be "in sync" with one another -- the hardware doesn't operate this way, i.e. there's a separate oscillator/crystal for audio than there is for video/PPU and they operate at two different frequencies, correct? So why can't two separate threads (one for audio, one for video/PPU) be made to run at those rates?

The problem is of two master timebases which are inevitably slightly out of sync, video and audio. The monitor might claim to be 60.00 Hz, and audio 441000 Hz. If an emulator generated exactly 60.00 frames per second and 441000 samples per second, one would inevitably fall behind since their timebases aren't exactly in sync.

Some emulators use sound as the master timebase, periodically doubling/skipping video frames as necessary. Others use video as the master timebase, letting audio drop out periodically or skip forward. As I understand it, this emulator takes a different approach of catering to both video and audio timebases, fine-tuning its audio rate on-the-fly in response to variations. To do this, it needs to accurately measure how much audio is currently buffered, so that it can adjust its sample rate by small increments. It uses some functions that find how much audio is buffered and waiting to go to the DAC in the sound card, and apparently these give unreliable results when processor speed is changed. I believe that the graph shows the results of this audio buffering query, and how it jumps around a lot when processor speed changes. This causes nemulator's audio code to think it needs to make a large adjustment to the audio rate, which I take it noticeable audibly.

By the look of it, the graph has time on the X axis. The green is output sampling rate, higher towards the top (more samples generated per second), and the red is the number of samples buffered waiting for the sound card (higher near the top). I think that the red gradient background might be the desired level of buffering. So where the average number of samples buffered jumps up suddenly and stays there, the emulator lowers the number of samples generated per second so that the sample buffer will become less full on average. At some point, it will increase the number of output samples per second to keep the buffer at this less-full level.

Since it looks like the queries are mostly clean even when processor speed changes, I wonder whether nemulator can ignore these erratic values.

Thanks for the breakdown, blargg.

The audio buffer is circular. The red line represents the number of bytes that the write cursor is behind the playback cursor (the red area depicted below).



If the red area is less than one frame's worth of audio (red line low on the screenshot), then the emulator will block waiting until there is enough room to write pending data. That will lead to a delayed video frame and stuttering. If the green area gets too small (red line high on the screenshot), the playback cursor will pass the write cursor and there will be garbled audio.

On the screenshot, the red gradient represents the target area, and the green line represents the playback frequency. The graph does move right to left and represents about 4.25 seconds worth of data (256 frames).

The jitter in the first screenshot is not a big deal, but what is problematic is the large jump above Mario's head. That represents about a 16 millisecond gap... or one frame. And that got me thinking...

What seems to be happening is that the emulator is missing vblanks, and that problem is exacerbated when processor throttling is enabled. However, that's not the direct cause. I discovered that setting the a/v thread's affinity to one processor leads to performance in line with having processor throttling disabled. So, good news -- I can scrap the power management stuff.

I need to do some more work to discover why vblanks are being missed (it still, infrequently, happens).

koitsu -- thanks for shaming me into finding a better solution

natt -- I discovered why timer_sync wasn't working for you. That fix will be in the next release.

Hmm, I can't even run nemulator on the machine i'm using at the moment. It spits out the error "DX10CreateDeviceAndSwapChain". I'm using Windows 8 and DirectX11. I think the problem is that I am using the motherboard's integrated graphics, and it is only compatible with up to DirectX 9.0c functionality.

This raises a big question, do you REALLY need DirectX10 for a NES emulator? That seems completely unnecessary to me. You're really limiting the number of possible users by doing that. It won't be useable on almost any laptop ever made, on any netbook, and there are still a HUGE number of desktop systems out there without a decent video card like mine here. That's something to think about.

Anyway, this computer is not my main one. It's a spare with a 3.06 GHz Pentium 4 that I've been using to test/screw around with Windows 8. (Which sucks mad goat balls, btw) I'll try nemulator a bit later on my 4.6 GHz eight-core with Geforce 9800GT and Windows 7.

I'm stretching it here, but if by missing a vblank you're referring to an actual hardware vblank (that is, on the host system, not a vblank in the emulated system), there may be an explanation. Possibly. At least on nVidia cards -- no idea about ATI, Intel, or on-die GPUs on present-day Core iX CPUs.

nVidia's semi-recent drivers have enabled a crapload of vblank adjustments on a per-application setting. The one that weirds me out the most is "adaptive", because I have not the slightest idea what that does/how that functions. I took a screenshot of the 3D settings profile page on my XP box, with the pulldown expanded, so you can see what I'm referring to. You can also set this on a per-application (per-exe-file or per-process-name) basis.

Aside from strange emulator bugs or Windows doing really weird crap (my vote is always on the latter, haha), the above is the only thing I can think of that might explain loss of some intervals.

Yes, that's the problem. I should probably have a more helpful error message.


No, of course not. However, when I was switching from DirectDraw to Direct3D, that was the API I chose to learn.


Yeah, I know. And, not to sound complacent, but I'm ok with the decision. FWIW, I've primarily developed nemulator on 2 laptops since ~2009 (Fujitsu Lifebook and MacBook Air).

Anyway, I hope you get a chance to try it out on your desktop and let me know what you think.

Yes, sort of. I only had a few minutes to look into this over the weekend, but I think what's happening is that, on occasion, the scheduling of the process (thread) is delayed/preempted and is unable to complete a frame within a vblank interval. I have a couple of ideas for confirming this and potentially improving, but with a non-realtime OS, there's only so much that can be done...

Might 10level9 help in any way?

Interesting. Yes, I suppose it would help; I'll take a look sometime.

I just tried it. It sounds great! Can't notice any frequency changes at all. Good stuff.

Well, I traced down the occasional stutter to GPU memory paging as a result of a memory leak. Along with some additional tweaks to thread priorities, everything seems to be working well without adjusting processor throttling settings or even thread affinity.

BTW, I used GPUView to help track this down. Very cool stuff; check it out if you're developing on Windows.

I released a new version of nemulator that should resolve this issue.