Following a conversion on the wiki talk page, I have transcribed and translated the Japanese VRC6 data sheet, hopefully without introducing critical errors. I had understood modes 0 to 2 well enough, but was confused about mirroring in mode 3, and the point of clearing the Mode Register's bit 5.
The data sheet made it reasonably clear to me that Mode Register bit 5 being cleared is supposed to be used with 512 KiB (4 kilobit) of CHR-ROM, and that the CHR-ROM must be connected differently. The data sheet does not specify how exactly, but connecting VRC6's A10-A17 to CHR-ROM's A11-A18 and PPU's A10 to CHR-ROM's A10 seems to be the only plausible way, and then the behavior of Mode 1 with bit 5 cleared makes perfect sense: use the same eight bank data bits unchanged for the upper and lower 1 KiB part of the 2 KiB CHR bank as CHR A11-A18, and use PPU A10 as CHR A10, yielding 256 2 KiB banks for a total of 512 KiB.
I still have not the slightest clue about how Mode 3's mirroring really works, and more importantly: what is the point of that complicated nonsense? What application does it have? The data sheet does not provide an explanation, only four examples that are not helpful at all.
Attached Nintendulator source file is written to succeed at Natt's test ROMs. But those (to my knowledge, correct me if I'm wrong) have never been tried on real hardware, and my combinatorial logic of the mirroring behavior in the source file is just me trying to turn the lookup tables from the wiki into something that can be expressed without tables. Also, I tried to implement how I think the 512 KiB CHR-ROM mode works without having any opportunity to test it.
(@rainwarrior: I spotted and corrected some mistakes compared to the previous version of the transcript.)
Edit: v0.2, correcting a typo in a number.
The data sheet made it reasonably clear to me that Mode Register bit 5 being cleared is supposed to be used with 512 KiB (4 kilobit) of CHR-ROM, and that the CHR-ROM must be connected differently. The data sheet does not specify how exactly, but connecting VRC6's A10-A17 to CHR-ROM's A11-A18 and PPU's A10 to CHR-ROM's A10 seems to be the only plausible way, and then the behavior of Mode 1 with bit 5 cleared makes perfect sense: use the same eight bank data bits unchanged for the upper and lower 1 KiB part of the 2 KiB CHR bank as CHR A11-A18, and use PPU A10 as CHR A10, yielding 256 2 KiB banks for a total of 512 KiB.
I still have not the slightest clue about how Mode 3's mirroring really works, and more importantly: what is the point of that complicated nonsense? What application does it have? The data sheet does not provide an explanation, only four examples that are not helpful at all.
Attached Nintendulator source file is written to succeed at Natt's test ROMs. But those (to my knowledge, correct me if I'm wrong) have never been tried on real hardware, and my combinatorial logic of the mirroring behavior in the source file is just me trying to turn the lookup tables from the wiki into something that can be expressed without tables. Also, I tried to implement how I think the 512 KiB CHR-ROM mode works without having any opportunity to test it.
(@rainwarrior: I spotted and corrected some mistakes compared to the previous version of the transcript.)
Edit: v0.2, correcting a typo in a number.