HW Test Harness

I've been looking to do some more hardware projects, and I've always wanted to know EXACTLY how some of the devices work. I'm thinking of creating a test harness something like what Kevtris mentioned about a decade ago. Basically just a 40-pin ZIF connected to a bunch of IO. I was thinking a Arduino Mega, or maybe even a Sparkfun Bit Whacker. The idea being I could control any of the inputs from a PC, and read all the outputs. I could clock the chips as slowly as I need to to take readings, so I wouldn't even need a logic analyzer.
The whole point of this effort would be to come up with a set of test data that is verified on hardware to be accurate. Kind of like the HW version of Blargg's test roms. This test data could be used by others - for example there's a bunch of FPGA related NES projects being discussed throughout the forum. I'm not expecting to break new ground or discover new features (we pretty much know how this stuff works by now), I'm just trying to get a set of repeatable test data.

I was thinking about starting with MMC1. I could progress up to more complicated chips, maybe even the PPU - though to be honest Visual2C02 might be a better way to explore that chip.

My plan right now is:
1. Build the hardware. It's pretty simple, looks like about $20-$50 worth of stuff.
2. Desolder some chips. MMC1 is pretty easy to come by - but some of the other chips I don't have access to. I don't want to destroy too much of my collection, so maybe I'll have to scour ebay. I don't suppose anyone has a collection of MMC chips they'd be willing to sell?
3. Write some software on the PC to drive the IO pins, then extend it to execute/record test vectors.

I think the hardest tasks are acquiring the MMC chips, and coming up with a set of comprehensive tests. The rest is really pretty trivial.
Any thoughts? Would this be useful to anyone?

MMC1 won't be useful for me (although probably will be for some people, so it might be worth it anyways), although if I get some FPGA then these hardware test might be useful to make Verilog Famicom (if I ever write this, it will be public domain, and the resistors/capacitors will be wired externally like in the RF Famicom schematic); the two chips to be tested would be the CPU/APU and PPU. Since they have analog outputs, there would need to be some way to test that too (I suppose this also applies to mappers with expansion audio).

One thing to look out for: the most popular ASIC mappers were in surface mount packages, not DIP, so you'll need various converters to plug them into your harness.

edit: Browsing through nescartdb, there are a significant number of infrequently-used 0.1" DIP ones: RAMBO-1 (m64), MIMC-1 (one of two ICs embodied by m206), many of Taito (X1-005 = m80, TC0350 = m33 or m48), Sunsoft 1,2,4 (m184, m89 or m93, m68), TC-112 (m193), Konami's VRC1,2,3,4,6,7 (m75, m21-m26, m73, m85), Irem's (TAM-S1 = m97, H3001 = m65), Codemasters CME-01and Camerica's BF909x (m71), Acclaim's MC-ACC (m4 mmc3 clone??)

You and I are seeing this in a very similar fashion. I agree that it would be helpful for those developing FPGA famicom clones, as well as those developing mappers for Powerpak/Everdrive. I also agree that it would be very useful to have test data on the 2A03 and 2C02, however I don't know if this test harness is sophisticated enough to do much with them.
I think the Arduino Mega 2560 is the way I'm going to go. It's designed for 5V IO, plus it has some analog inputs - so I may in fact be able to do some testing of the audio on the 2A03. I'm going to start with the easy digital stuff though
I really see this as a enabler for those trying to implement clone HW in programmable logic.

I hadn't even really considered that - I sort of assumed most would be DIP. It looks like MMC5 is a 100-pin PQFP! There's very little chance of me removing that without damage. Even if I did, a socket for it would cost hundreds of dollars. A breakout board would be cheaper - but I doubt I could solder that down accurately. I might be able to get access to all the pins by leaving it on the PCB, but removing all of the other chips. Ugh.
I think I would to have to stick with DIP parts, which as you point out severely limits the value of this effort.

Mappers all work from the 72pin connector, why can't you? Yeah it'll be a little more work and you'll really have to know how the mappers work, but the package issues should be your smallest concern.'

EDIT: to be a little more clear what I mean, generally all the mappers inputs connect to the 72/60 pin connector. And all the outputs are connected to memory or the connector. You can sense these outputs to memory by reading the ROM from the connector. Best benefit: you don't have to actually destroy old (rare) hardware to see how it's behaving. You can see it all from the cart connector the same what the NES/FC does which is really all people are worried about anyways.

EDIT2: Not to mention new mappers will NOT be in the same packaging and certainly not the same pinout of the originals. You really need to be able to connect 'universally' up to the mapper if you want test old an new. The cart connector is really the only way to do this...

This may be the best way to do it (no soldering, no damage to the carts). However, it's not as error-free as testing the chip directly. As you mention, It makes sense that most of the MMC inputs would be connected to the cart connector. Most of the outputs are connected to the memory chips. Those can't be read directly - you have to infer their values by reading from the memory, and comparing to what you expect to be there from a previous ROM dump. Not ideal, but certainly worth a try compared to the alternatives.
On the plus side, this would require the exact same hardware I was planning on putting together anyway for a cart dumper!

Your method of checking things at slow speed isn't error free either. Having reverse engineered and redesigned many of these mapper myself I can think of a few times where SPEED caused or was related to issues/differences I saw between my implementation and the original. In these cases your tester would have told me my re-creation was logically identical to the original, but indeed it's not a perfect replica as noted by the bugs I saw/see.

So while you device would be a great tool for learning things about the original asics, there isn't much substitution for testing the library of games against the device on the end user hardware. I'm not trying to burst your bubble or anything. I'm just trying to give you some insight to the reality of the situation as I'm currently in a position where I would be interested in such a device. (I've even adapted the kazzo to work in the fashion your describing to learn things about the original hardware.) But no matter what you create, you won't be able to tell me things the NES/FC/SNES/original console can when it comes to undetectable accuracy to the end user.

TL;DR: SPEED KILLS!

Additionally there are other nuances such as bus loading, capacitance and other related items. But if you're tricky you can try to sense these as well. The powerpak's issue with using pull-ups on the data bus to achieve 5v outputs which in-turn kills games which rely on bus capacitance is a situation where logic itself isn't enough to be a perfect replica.