This post by ap9 claims that a significant cause of aliasing in
this sine sweep is nonlinearity in the DAC driven by $4011 writes, for which the sine table generator in the sweep's build process is not compensating. I seek to implement this compensation, but
this post by rainwarrior has cast doubt on the accuracy of the approximate formulas in the article
"APU Mixer" on the wiki.
So what would be a good way to measure the nonlinearity? Just generating ramps into $4011 and reading off those might not be a good idea because of all the phase distortion caused by band-pass filters in the chain from the 2A03 DAC to the computer's ADC.
I'm thinking a 1000 Hz square wave that changes the top or bottom values by 1 volume unit every 50 ms or so. It might go through ($00, $01) through ($00, $7F) to ($7E, $7F), for example. Nonlinearity would not change the shape or effective volume of a 2-level waveform, and the most prominent frequencies (1000-7000 Hz) would be little affected by filtering. So there'd need to be both an NES program to generate the ping and a PC program to read the volumes out of the wave file so that others can easily repeat the volume measurements on their own systems with slightly different resistor values.
I don't think the filter are a big problem, they just limit how fast you can measure isolated pulses. e.g. wait for signal to reach equilibrium, produce a pulse, repeat. The relative magnitude of all the pulses should be the same as long as you perform them at a consistent rate.
So, here's some stuff I had in mind when I made that comment back then:
Steps of 1 probably are going to be hard to distinguish from noise, esp. 60hz interference. My first instinct would be to just do a pulse at every value, 0 127 0, 0 126 0, 0 125 0, etc... give the pulse a specific width (maybe 200 cycles?) and then for a window around each pulse just integrate (i.e. sum) its difference from the average established baseline value before the pulse (or maybe root-mean-square or abs difference?). Including a few 0 0 0 control pulses might help you measure the noise level variation as well, so you can decide how much you can know about the low volume results.
Something like this for every test:
1. 100000 cycles of 0: integrate and use as baseline value
2. 49900 cycles of 0, 200 of value to test, 49900 of 0 again: integrate this and compare to baseline
3. extra period of waiting to return to baseline before next test
(There's numbers are just made up, I haven't tried to calculate anything ideal here. You can always just make the wait period longer until the filters' effect becomes insignificant)
I'm not sure if it's important to measure all the possible intervals, e.g. 4-5, 3-120, etc. That's a huge matrix of data that could be measured there, but it might not be important. I'd just start with jumps from 0 only... you could do a few relative ones as an additional experiment to see if they deviate from the expectations from the absolute ones, investigate further only if they seem to?
The other thing I wanted to try is noise and triangle. For the triangle, I would just repeat the whole test 32 times, and in between each test let the triangle play 2 or 3 full periods (just so you can inspect/measure its phase) and then halt on the next step in its waveform vs. the previous test. (Is the triangle phase 0 on power on or reset? I know lidnariq determined that noise is deterministic like this.) Similarly for noise, do a test of a short volume 15 burst with the DMC at 0 and at 127. Again there's a billion points of data we could measure here, but I'd just say hit both ends for starters and see if they fit the curve from other expectations before going deeper.
Maybe this goes without saying, but $0D background colour, no NMI, do everything cycle timed, don't use $2002 at startup just wait an appropriate number of cycles, etc.
To clarify, I said the non-genuine sine wave was a problem (sub-harmonics), not that it's a cause of the aliasing.
It would still be good to get a true DAC curve. We are, after all, looking at analog results of
a few filter stages designed with low cost in mind, which includes raw tolerances for the resistors setting the scales for pulse and pcm.
It turns out nonlinearity was the cause of the audible foldover at higher frequencies, at least in FCEUX. Once I baked compensation for FCEUX's curve into the waveform that the sine sweep uses, it sounded as sinusoidal as I expect.
I just discovered that in
a previous topic, I had made a test with triangle, noise, 93-step noise, and a 30-level-tall DMC square on top of base level 0, 48, and 96. That was good for illustrating nonlinearity's existence. The new test I'm planning should be better suited for automatic measurement and will also work in any NSF player that supports a non-returning play routine.