Okay, I've run this through my head a few times now, and I can't find any problems with it. Someone check my logic.
1) According to
Disch's recent tests, DMC DMA always attempts to engage on an odd cycle.
2) It is already well known that the first cycle after an OAM DMA is always an even cycle.
By performing an OAM DMA, then cycle-timing to ensure that all reads from $4016/$4107 occur on even cycles, shouldn't it be possible to ensure that the controller reads are never hijacked? The DMC generates at most one DMA every ~400 cycles, so the alignment after a DMC DMA is irrelevant - the entire routine should be able to complete before a second one can occur.
I don't have the hardware to test it myself (PAL region), but if nobody can find an obvious reason why it won't work I can write a test ROM for others to try.
The problem is that you lose vblank time. You're better off just reading it twice, then reading a third time if they don't agree.
Dwedit wrote:
The problem is that you lose vblank time. You're better off just reading it twice, then reading a third time if they don't agree.
The controller reads don't have to happen in vblank though. You would have sprite DMA as the last part of your vblank code and then the controller reads immediately after.
I think.
... I'm pretty certain you wouldn't lose significant vblank time...? Unless I'm misunderstanding what Rahsennor is saying, it sounds like it should be possible to just put careful odd-number-of-cycle delays in your reading routine to make it such that the reads from $4016/4017 wouldn't be on the relevant even/odd phase to cause the glitch, and so it should only take one read through?
pubby wrote:
sprite DMA as the last part of your vblank code
That'll break on PAL, as sprite DMA needs to happen within the first 20 scanlines (see point K on
http://wiki.nesdev.com/w/index.php/Nestech.txt#PPU).
freem wrote:
pubby wrote:
sprite DMA as the last part of your vblank code
That'll break on PAL, as sprite DMA needs to happen within the first 20 scanlines (see point K on
http://wiki.nesdev.com/w/index.php/Nestech.txt#PPU).
Only if the total amount of vblank code actually takes more than ~20 scanlines (which is usually not the case, to maintain NTSC compatibility). BTW there might be other "windows" in which the OAM DMA can be done (instead of just the first 20 scanlines). All that has been tested is that if the DMA was done slightly after 20 scanlines, the update got corrupted. (This also seemed to cause the strange results in blargg's OAM tests on PAL NES -- after I forced all reads to happen right after start of vblank, the tests passed.)
Then again, the DPCM glitch also doesn't seem to affect PAL NES in the first place.
lidnariq wrote:
... I'm pretty certain you wouldn't lose significant vblank time...? Unless I'm misunderstanding what Rahsennor is saying, it sounds like it should be possible to just put careful odd-number-of-cycle delays in your reading routine to make it such that the reads from $4016/4017 wouldn't be on the relevant even/odd phase to cause the glitch, and so it should only take one read through?
Yeah, assuming one would just be reading the values and storing them for later processing, let's say 8 CPU cycles per read, that's only ~64 cycles wasted (~100 at most).
thefox wrote:
Only if the total amount of vblank code actually takes more than ~20 scanlines (which is usually not the case, to maintain NTSC compatibility).
But 20 PAL scanlines have less CPU cycles than 20 NTSC scanlines. The difference is about 142 cycles, if I'm not mistaken.
freem wrote:
pubby wrote:
sprite DMA as the last part of your vblank code
That'll break on PAL, as sprite DMA needs to happen within the first 20 scanlines (see point K on
http://wiki.nesdev.com/w/index.php/Nestech.txt#PPU).
I think the actual point just was "don't consider joypad reading as part of your vblank code".
freem wrote:
pubby wrote:
sprite DMA as the last part of your vblank code
That'll break on PAL, as sprite DMA needs to happen within the first 20 scanlines
Even if your engine is designed to take advantage of PAL's longer vblank when available, as
RHDE is, I imagine that "If PAL NES then do OAM first else do OAM last" would handle all cases. Incidentally, so would reading the controller in a DMC IRQ handler, if you're using it as a crude raster timer.
Yeah, PAL vblank is so long that reading the controllers during that time shouldn't make any difference... Just take care of the sprite DMA and controller reading right away, while on NTSC you'd do both these things last, with the controller part possibly spilling into the visible frame.
Yes, the idea was to do the DMA last and let the controller reading spill over into the visible frame. Gives you something to do while you wait for sprite 0.
I wasn't aware that there was any special constraint on OAM DMA on PAL systems. I just did it first out of habit.
It only takes 11 cycles to decide whether to do the DMA first or last, assuming you have a flag in the top two bits of a zero page variable. Or if you can make sure your NMI handler always takes an even number of cycles (tricky, but not as bad as making it constant-time) you can just do the DMA first and the controller reading last and it should work fine in either case.
There's also the possibility of using this trick to perform VRAM reads, which you'd want to do inside vblank anyway. In fact, that might be a better way to test it, since you have full control over the contents of the nametables/pattern tables.
Is nametable reads during DMC what caused all the attribute glitches in Castlevania II?
You're trying to be hackish and fancy (½ compliment).
If you're cycle-timing your controller read relative to DMC, then you can just stick it in the time you know there isn't going to be a DMC?
Slapped together the simplest test I could come up with, and to my surprise, it works perfectly in Nestopia... but on odd cycles instead of even. None of the other emulators I tested emulate DMC conflicts at all. Can anyone here try it out and tell me what it does on real hardware?
I made two versions of the test, one that always reads on even cycles, and one that always reads on odd cycles. The test is as simple as it gets: it plays a DMC loop at the highest rate and runs the polling code in a tight loop, with rendering off. If right is pressed on either controller, the screen goes white. Otherwise the screen should stay black. This assumes a standard controller that pads with all 1s, so that glitched reads always result in right being pressed.
Tested on PAL hardware with nes-004 controller for a couple of minutes on each test. Both work 100% of the times tested, assuming that the only thing to test for is the screen to go white. If i raise the tv volume quite a bit, i hear a bright tone that's interrupted for as long as i hold right pressed.
fceux on the other hand, crashes upon rom load.
Post edited for additional findings.
Rahsennor wrote:
Slapped together the simplest test I could come up with, and to my surprise, it works perfectly in Nestopia... but on odd cycles instead of even. None of the other emulators I tested emulate DMC conflicts at all.
Same result in Nintendulator. Definitely should test on real hardware though (I don't have mine hooked up right now).
Just so this is clear to everybody:
- You have to have a standard controller plugged in, and shouldn't press Right.
- If screen stays black => no corruption
- If screen goes white => was corrupted
- (You might want to press Right in the end to force white to make sure that the ROM was properly loaded.)
Just ran it on my NTSC CopyNES, and the "even" test appears to pass (screen only goes white when I press Right on either controller), while the "odd" test fails (goes white immediately).
I saw the same results on my NTSC NES + PowerPak: odd started white, and even started black but turned white when I pressed Right.
I will report this finding to the wiki as a way that the mouse (and other controllers that can't just be reread) can possibly be used with DPCM.
I just tested on a bare flashcart, and saw the same results as both of the above.
Source code for the ROM that works please.
It should be posted on the wiki.
thefox wrote:
Rahsennor wrote:
Slapped together the simplest test I could come up with, and to my surprise, it works perfectly in Nestopia... but on odd cycles instead of even. None of the other emulators I tested emulate DMC conflicts at all.
Same result in Nintendulator.
So if I understand correctly, Nestopia and Nintendulator are handling this behavior incorrectly?
thefox wrote:
Just so this is clear to everybody:
- You have to have a standard controller plugged in, and shouldn't press Right.
- If screen stays black => no corruption
- If screen goes white => was corrupted
- (You might want to press Right in the end to force white to make sure that the ROM was properly loaded.)
oh! in that case, both were black until i pressed right on PAL through powerpak.
tepples wrote:
I will report this finding to the wiki as a way that the mouse (and other controllers that can't just be reread) can possibly be used with DPCM.
Would it be sound to suggest it as an alternative in general, with a list of known benefits/applications following? As to not suggest it is for this only
The 2A07 CPU in the PAL NES is known to have had the relevant bug fixed, so unfortunately tests on PAL will give you the results you found.
Ah, this finally explains how I once accidentally made a DPCM-using ROM that failed to compensate for the glitch, but somehow never had the problem!
I had never understood why the problem never showed up, and it always bothered me; I kept looking through my input logic to find something that would filter out a spurious right press, but couldn't find anything that would explain it. I never thought about OAM DMA vs DPCM even-cycle alignment.
I have to say, this is a very ingenious solution to the dreaded DPCM controller glitch. Definitely better than reading the controller's twice, plus an occasional third time.
pubby wrote:
Source code for the ROM that works please.
It should be posted on the wiki.
Is
my description any good?
The implementation in the NES files is so simple that you can just throw the NES file into da65 to get something correct out.
Okay, here's an updated test with source! Please check that it still works, in case I've done something stupid like post the odd ROM instead of the even one.
The ROM should be run with a standard NES controller in both ports. The expected output is that the screen should remain black until right is pressed on either controller. If it fails, it should do so almost instantly, producing a white screen. PAL NES does not have DPCM conflicts so this test is not meaningful on PAL.
For quick reference, here's the important code:
Code:
lda #OAM
sta $4014 ; ------ DMA ------
ldx #1 ; even odd <- strobe code must take an odd number of cycles total
stx controller1 ; even odd even
stx $4016 ; odd even odd even
dex ; odd even
stx $4016 ; odd even odd even
read_loop:
lda $4017 ; odd even odd EVEN <- loop code must take an even number of cycles total
and #3 ; odd even
cmp #1 ; odd even
rol controller2, x ; odd even odd even odd even (X = 0; waste 1 cycle and 0 bytes for alignment)
lda $4016 ; odd even odd EVEN
and #3 ; odd even
cmp #1 ; odd even
rol controller1 ; odd even odd even odd
bcc read_loop ; even odd [even]
Yes, it really is that simple. (Do note that I read the controller bits in big-endian order; most other implementations I've seen use little-endian. Conversion between the two should be a simple exercise.)
pubby wrote:
thefox wrote:
Rahsennor wrote:
Slapped together the simplest test I could come up with, and to my surprise, it works perfectly in Nestopia... but on odd cycles instead of even. None of the other emulators I tested emulate DMC conflicts at all.
Same result in Nintendulator.
So if I understand correctly, Nestopia and Nintendulator are handling this behavior incorrectly?
Yes.
Previously, Nintendulator didn't implement proper DMC cycle-stealing (it was hardcoded to 4 cycles and didn't care about alignment). It tries to do so now, though I'm pretty sure it isn't 100% correct (specifically for the case of DMC fetch happening one cycle after a $2004 write).
The only caveat is that this gives you less time for PPU updates since sprite DMA should ideally be done first thing, followed by all the PPU bus accessing code. On the other hand, wow, you don't need to reread and compare the controller bytes! That's really spiffy, especially if you need to read more than 8 bits from each port. Nice finding!
Drag wrote:
The only caveat is that this gives you less time for PPU updates since sprite DMA should ideally be done first thing, followed by all the PPU bus accessing code.
On NTSC NES, where avoiding this glitch actually matters, whether you do OAM DMA before or after VRAM updates doesn't matter so long as it's all done before reenabling rendering.
Drag wrote:
On the other hand, wow, you don't need to reread and compare the controller bytes!
That's especially useful if you're using the SNES Mouse, since you
can't reread (it accumulates movement info over the course of a frame, and reading it
resets its counters). Similar code could probably be written for the Arkanoid controller (which uses D3/D4 and also does not allow rereading).
Drag wrote:
The only caveat is that this gives you less time for PPU updates since sprite DMA should ideally be done first thing, followed by all the PPU bus accessing code.
Not really. DMA first or VRAM updates first doesn't matter, added up they'd still take the same amount of time either way. And by doing the DMA last you can let the controller code itself spill into the visible frame, no need to deduce that part from the vblank budget.
You do lose a bit of time on PAL, where the DMA must come first, but is that really a problem with a vblank period that long? That would only be a problem in PAL-only games (that can't possibly be converted to NTSC), something I don't think people should be making.
No, you guys are right.
Sprite DMA being first is only important on PAL machines, where the DMA glitch is fixed. On NTSC machines, sprite DMA can happen after the PPU update routine, and then controller reading can happen immediately afterwards since all the vblank critical stuff is done.
Though I'm going to pout just a little because I came up with that bit-merging method of doing DMA-corrected joypad reading and felt really clever about it, and now it's obsolete.
tokumaru wrote:
You do lose a bit of time on PAL, where the DMA must come first, but is that really a problem with a vblank period that long? That would only be a problem in PAL-only games (that can't possibly be converted to NTSC), something I don't think people should be making.
PAL-only games wouldn't even be doing this in the first place anyway.
So the only problem is one that can be fixed - incompability with current versions of some emulators?
Maybe we could try quickly modifying an existing game to use this technique to see how feasible it is in practice?
Sik wrote:
PAL-only games wouldn't even be doing this in the first place anyway.
True, you lose no vblank time on PAL either.
This is a really awesome find. Can't wait to use this in future projects!
Tested on a Powerpak. At first, neither *_even nor *_odd would turn white without pressing right, until I realized I had my Four Score enabled. Disabled it and then *_odd started turning white immediately. Which makes sense.
Question: is it possible from within the ROM to detect whether the host emulator is doing the wrong thing, by reading the controller on odd cycles while turning DMC on and off? If that were the case, the ROM could switch to using a routine compatible with that host's behavior.
Drag wrote:
The only caveat is that this gives you less time for PPU updates since sprite DMA should ideally be done first thing, followed by all the PPU bus accessing code. On the other hand, wow, you don't need to reread and compare the controller bytes! That's really spiffy, especially if you need to read more than 8 bits from each port. Nice finding!
So…forgive me if this is obvious, but why not just put the DMA after the PPU updates? Or is that only in PAL?
OAM DMA should be done first in PAL because OAM is not writable in lines 261-310.
Recommended pseudocode:
If PAL NES then OAM DMA
Update palettes, nametables, and CHR RAM as needed
If NTSC or Dendy then OAM DMA
Read controller
dustmop wrote:
Tested on a Powerpak. At first, neither *_even nor *_odd would turn white without pressing right, until I realized I had my Four Score enabled. Disabled it and then *_odd started turning white immediately. Which makes sense.
I was going to add a test to make sure you have the right kind of controller hooked up, but I figured simpler code with clear instructions would be better for this.
If I were to write a test ROM for emulator devs, though, I would definitely trick it out with every possible failure mode. I'll probably get on that when I have time.
dustmop wrote:
Question: is it possible from within the ROM to detect whether the host emulator is doing the wrong thing, by reading the controller on odd cycles while turning DMC on and off? If that were the case, the ROM could switch to using a routine compatible with that host's behavior.
You could, but breaking incorrect emulators is a feature in my book.
It only breaks emulators that are already trying to be accurate hard enough to bother implementing the DMC interference in the first place. Most "inaccurate" emulators are already giving you glitch free reads.
What I find amusing is that the broken emulators seem to be actually doing the correct thing, just with the wrong cycle alignment.
Rahsennor wrote:
breaking incorrect emulators is a feature in my book.
It could be, at least until you hear complaints from a user of your game on a platform for which
there is no accurate emulator to recommend. For example, which accurate emulator should I recommend to a user of Android?
I think the idea is "make it break so emulators gets fixed". Especially true since as soon as something capable of detecting emulators is found there will be people who will exploit it just to prevent their stuff from working on them (as an attempt to force you to use cartridges instead - yeah, that kind of people exist).
The kind of people who will patch out your anti-emulator mechanism also exist, though.
Sik wrote:
I think the idea is "make it break so emulators gets fixed".
Which is inconvenient for users if the (hobbyist) developers of emulators for a particular platform have a habit of blowing off compatibility fix requests, or saying "fixed in the next version" when the latest stable release is years old *cough*Nestopia*cough*FCEUX*cough*, rather than implementing and publishing them. It's also inconvenient for game developers if none of the emulators gets fixed until after the hype for a particular game has blown over.
"clearly" the correct thing to do is explicitly elicit a DPCM bit deletion and refuse to run in emulators that don't implement it.
lidnariq wrote:
"clearly" the correct thing to do is explicitly elicit a DPCM bit deletion and refuse to run in emulators that don't implement it.
Sarcasm noted But the problem is, unless you have player cooperation or one of the (uncommon) cases where there's a signature afterward, you can't guarantee detection of a bit deletion.
Myask wrote:
But the problem is, unless you have player cooperation or one of the (uncommon) cases where there's a signature afterward, you can't guarantee detection of a bit deletion.
You can make two quick reads in succession, one uncorrupted, one intentionally corrupted, and just test if they're equal.
A true fail is going to fail this test every time you do it. A human playing on hardware accidentally making a press in the middle of this process probably isn't going to be able to do it two frames in a row. Pretty easy to weed out the false positives just by requiring several failed frames in succession. You don't need to be able to fail all the time, you can just wait until they need to press RIGHT or something and catch it then.
Not that I'd recommend it, anyway, just I don't think this in particular is the problem.
Myask wrote:
unless you have player cooperation or one of the (uncommon) cases where there's a signature afterward, you can't guarantee detection of a bit deletion.
Bit deletions work on reads from $2007, too ... it's just more than a single byte per glitch there.
If you want to detect failure through controller reads, you'll have to autodetect the type of controller in port 1. It could be an authentic NES controller, a Super NES controller with its 16-bit report (especially if played on an FC Twin or if the player also owns a Super NES Mouse-compatible game), the left half of a Four Score with its 24-bit report, or an unlicensed NES controller that returns 0 bits after the 8-bit report instead of 1 bits. It might be easier to test whether $2007 readback skips a byte, though I imagine a broken emulator might implement double reads for $2007 but not $4016 or vice versa.
tepples wrote:
OAM DMA should be done first in PAL because OAM is not writable in lines 261-310.
Recommended pseudocode:
If PAL NES then OAM DMA
Update palettes, nametables, and CHR RAM as needed
If NTSC or Dendy then OAM DMA
Read controller
Is Dendy afflicted by the DMC/controller conflict? If not, then treating Dendy like PAL in this case would make the conditionals a simple pair of
lda with
beq/
bne (In the usual setup of NTSC = 0, PAL = 1, and Dendy = 2).
To be safe, I'd just spend the 4 cycles (2 before, 2 after) on testing bit 0.
Code:
lda tvSystem
lsr a
bcc wherever
Rahsennor wrote:
The expected output is that the screen should remain black until right is pressed on either controller. If it fails, it should do so almost instantly, producing a white screen.
Never noticed this test existed until now (saw a link in another thread). Mesen keeps a black screen the whole time and goes white if I press right - so I guess it passes? I do have the DMC glitch emulated. Just wanted to double-check before I add it to the list of tests it passes.
Offtopic:
Being an emulator author I might be biased, but wouldn't it be helpful if the wiki had a page containing all known test roms? I have a collection of ~300ish of them, but I keep finding test roms posted here and there in both new and old forum threads (and sometimes the download links are not even working anymore). There is a test rom page already, but it is outdated and missing literally hundreds of them.
Sour wrote:
wouldn't it be helpful if the wiki had a page containing all known test roms? [...] There is a test rom page already, but it is outdated and missing literally hundreds of them.
If there are others you know about, feel free to
join and add them to
Emulator tests.
Sour wrote:
Never noticed this test existed until now (saw a link in another thread). Mesen keeps a black screen the whole time and goes white if I press right - so I guess it passes? I do have the DMC glitch emulated. Just wanted to double-check before I add it to the list of tests it passes.
Grab the old version from page 1 and see if the odd test fails instantly. If it doesn't, you either have no DMC glitches or your controller reports aren't padded with 1s.
If it does, congrats, Mesen is the first emulator to pass.
tepples wrote:
If there are others you know about, feel free to
join and add them to
Emulator tests.
I just tried to create a wiki account and get a 500 error with both FF & Chrome.
Rahsennor wrote:
Grab the old version from page 1 and see if the odd test fails instantly.
Yup, the odd test gives me a white screen instantly. Always nice to pass new tests without having to waste a whole day trying to figure out what's wrong :)
Mail from the wiki is broken right now. Try
not providing an email address when signing up. Because mail is broken, you'll need to autoconfirm (2 talk edits and 4 days) or wait for me or another administrator to wake back up and confirm your account.
tepples wrote:
Mail from the wiki is broken right now. Try
not providing an email address when signing up. Because mail is broken, you'll need to autoconfirm (2 talk edits and 4 days) or wait for me or another administrator to wake back up and confirm your account.
Yup, that worked. If someone could confirm the account (same username as here) that'd be nice - no hurry though, I'm not planning on editing this right away :p
I tried to write a test that involved watching for corruption while doing repeated synced joypad reads, with the goal of identifying whether the emulator supported this method of reading. Surprisingly, despite being synced, I was encountering corruption when testing on Mesen.
What I learned was that if your synced region is long enough that two or more DMAs can occur during it (about 430 CPU cycles at minimum, I believe), then you need to ensure that all consecutive write cycles in the routine end on an even cycle; otherwise, DMA landing on that write will take 3 cycles instead of 4. For example, when using ROL on memory, the second write cycle must be even so that DMA that tries to occur on either one is forced onto the following odd cycle, where it will take 4 cycles. Without this, the cycle parity changes, allowing future reads of controller registers to occur on odd cycles where they're subject to bit deletion.
In addition to repeatedly reading the joypad, this seems likely relevant for handling Four Score because of how much time is required to read everything even once.
While I'm reviving this old thread, I'd like to note here on the topic of OAM DMA on PAL that testing done in
this thread indicates that if your OAM DMA fits within NTSC vblank, it should also be fine on PAL, so it does not appear that different behavior needs to be taken depending on the region; ending vblank with OAM DMA and joypad read should work either way.
By testing this on real hardware via Everdrive, I believe I've been able to confirm this write behavior, and also discovered that the only emulator that seems to be getting this totally correct right now is Nintendulator 0.980. Testing this is a bit tricky because it seems that you can have timing alignments where the DMA avoids landing on certain cycles of the loop, which can cause it to miss an odd write cycle (manifesting as tests showing black when they should show white), but I found that by adding variable delay into the loop, it seems to land on the target cycle reliably. I'd appreciate another set of eyes on this to make sure it all makes sense, but I'm fairly confident in the results because my test ROMs show different behavior despite the only meaningful difference being the timing of the ROL write cycles (all other timing, code positioning, and register values remain the same).
I've attached two modified versions of dma_sync_test_v2. Both of these change the loop to no longer include the OAM DMA, so code has to stay synced indefinitely for the controller port reads to work reliably. They also add a variably delay to try to mix up where DMC DMA lands within the function. The difference between these versions is in the second ROL instruction, which ends on an odd cycle in badrol and an even cycle in goodrol. badrol is expected to exhibit bit deletion (screen turns white) because DMA landing on the odd ROL write cycle (cycle 5/5) will be delayed by 1 to an even cycle and thus take only 3 cycles, changing the parity so that future joypad bits can be lost. goodrol is expected to never encounter bit deletion (screen stays black unless right is manually pressed) because DMA landing on the odd ROL write cycle (cycle 5/6) will be delayed by 2 to an odd cycle, where it takes 4 cycles as usual.
Both tests are verified to give the expected result on real hardware and Nintendulator 0.980, and but not on Mesen 0.9.7.37 (goodrol turns white), puNES 0.100 (goodrol turns white), Nestopia UE 1.49 (goodrol turns white), and Nintaco 2018.12.31 (goodrol turns white on some resets). Other emulators I'm aware of don't emulate the bit deletion at all and will show black on both ROMs. There hasn't been much progress in a couple years on emulator support for this, so I'm really hoping to get the ball rolling so hacks and homebrews can use synced reads and still work properly in emulators.
I didn't get it. Please, could someone explain this stuff to me? What "bit deletion" are you talking about?