Poll: Adding Nametable-to-ROM Address Detection

I'm saying this because there is an old unofficial fork of Nintendulator in rveach's old website... still up to this day,

Here is the special version's source and binary links:

http://rveach.romhack.org/Nintendulator/Nintendulator0967.rar
http://rveach.romhack.org/Nintendulator/Nintendulator0967src.rar

As usual, Use 7Zip to open them.

This version includes a way to detect Nametable addresses in ROM (using the Nametable Viewer), and I think it is useful if used in a updated emulator!

You have 10 days till the tally is final, fellow NES Hackers and Developers, Make it count!

What's the point of this poll? Adding an useful feature is always good isn't it?

I however do not understand the question at all.

Uh... it's a memory search/scan for the nametable data inside the PRG ROM data.
I could do that, Hamtaro... but too bad you don't look at my emu.

Is this a feature request disguised as a poll? How does this even work, considering the many possible ways name table data can be compressed, and that it may very well go through RAM before being written to VRAM?

Between how often this would be likely to work? Don't most games compress nametables? I've heard about uncompressed ones in FDS games, or ports of FDS games since they could load directly to the PPU.

For the times it would work, this problem doesn't sound any different than searching for any particular string in a hex editor. How much do you need to do this that a specialized tool is useful? O_o

I'll see your emulator RockNES to see if anything like this is intact.

Edit- Nope, But it's a good emulator!



I was going to ask the people over at FCEUX team's sourceforge page after the poll, if that's what you're asking. Otherwise, It's not actually a request to NESDEV, rather it's a measure of (re)assurance

Well, I'm not exactly opposed to the inclusion of such a feature, but I can't say I'd find it useful either. If a name table is stored in the ROM in a straightforward enough way that an emulator can find/track, it shouldn't take a person more than 1 minute to find it using the currently available debugging features. However, I imagine that, most of the time, name tables are NOT stored in a straightforward way, due to compression and all kinds of run time manipulations.

FWIW, I have the same questions (and for me: strongly bordering on reservations) that rainwarrior and tokumaru do. And honestly, it sounds painful to implement reliably.

My old "NESExplorer" tool can detect certain Nintendo-compressed nametable-data in PRG-ROM's (well, titlescreens/cutscenes mostly). Works pretty good in many cases:
http://nes.goondocks.se/software.php

Umm... Orbit2002, I should've told you this, that your method does not implent the function to increment by 32 if bit 7 is enabled in the length part of Nintendo's PPU format!

Ah, so you're not testing for uncompressed nametables, but instead for a specific proprietary format? I dunno, this seems like a "one time use" kinda feature. If you found all the nametables in that small set of game and just documented their addresses on some romhacking wiki nobody would really need the tool anymore.

Like, you could make some sort of "konami music editor" extension to an emulator, for example, and it'd be perfectly useful for the few people that want to do that, but it seems like a very specific task, not something that would really need to go in the main branch of a project. Easy enough to share the special fork with the couple of people who can use it.

That's not what I am talking about rainwarrior, All I was saying to Orbit2002 was that his application does not support something properly, which is actually semi-off topic, and is a bit relevant but not enough to warrant being completely on topic.

Please, tell me more. It's becoming interesting... ^_^;; It's not just looking for a nametable buffer inside the PRG ROM data, right? What compression method (or similar) do you want to do?

This is not always about compression, This is about detecting the position of the nametable data via emulator so people can debug or romhack nametable graphics easily...

Obviosly people are not staying on topic, so please don't derail!

Oh, I though you really had some interest in such implementation in an emulator... Sorry.

So it's about detecting substantial literal copies from a consecutive stream in PRG ROM to the VRAM write port within a short time, correct?

What's with the thread creator, the Nintendo Nametable compression format and asking for features all the time.

If you mean ''Oh, I though you need someone to implement nametable compression detection, But I misinterpeted, I apologize.'', That's forgivable.

And Punch: As I said this is a poll, not an actual request for this forum, it's a reassurance:

If it passes, I contact the FCEUX team about it at thier SourceForge page,
If it does not pass, I do nothing about it, Plain and simple

That is the correct answer

Because I don't want to send a poorly defined bug to FCEUX's bug tracker, I held my vote back until the feature became well defined. But with that reply, I vote Bernie yes.

Bernie

Oh, you are in Indiana.

Edit. He's projected to win (Indiana), with 53%.

Sorry for drifting off topic.

I'm going to build a Nintendo metatile map editor and make Hamtaro pay for it.

These off-topic, bad jokes are getting out of hand... AND if this was in any other forum, people can easily get in trouble more for posting shit like this.

Talk about having no sense of humor.

dougeff: I forgive you.

Punch: Apologies for being grumpy, just please don't go overboard next time... The main reason is because of certain people, but that is another story entirely!

Don't get me wrong, though, I think it's great that you made a tool and are sharing it.

The negative part of my response is only about whether it should be a standard feature on either of these two emulators.

I actually think you did not read where the first post states that RVEACH made it, not me,

All things considered, You have a good point, It is just a fork, but a good (and totally outdated) one.

Suggestion for the tool:

I seems to just say "ROM Addresses" in a window called "Detail Window" that opens when I select "Nametables" from the "Debug" menu, this is the feature you're talking about, right?

From what I can tell, I hover over a tile in the nametable, and it's providing me a guess as to where it might appear in the ROM or RAM?

In the couple of games I tried, it seems very noisy, and kinda points to erratic places. I don't know quite how it's trying to follow, is it just telling me the address of the longest matching string in ROM to that location in PPU memory? If that's what it's actually displaying, my suggestion is that it should tell you the number of bytes matched, as well, so you can understand a bit more about what it's matching to.

Oh, sorry, I thought this was your work.

Also this seems to be a rather large change to the debugging stuff, not just an addition of this one feature.

Do you think you could give an example of this feature being used effectively? I tried a few games at random (e.g. SMB) and it was just giving me junk values as far as I can tell (not that I really expected much from it), but what games does it actually work with?

Did you try actually using the correct nametable and not it's respective mirror?

It can work with any game as far as I know, I used it on Super Mario Bros. 2, which worked fine!

Yes, but why would hovering over a "mirror" change the results? It should be looking for the same data...

Oh wait, now I understand what this is. Every time there is a write to memory, it stores the last read address, and this tool just displays that read address for each nametable byte you highlight? OK. (It should still duplicate that for mirrors, BTW, but whatever...)

Like in one game I am seeing that it points to an ASCII table wherever there was text on screen, because that was the last thing fetched before the character is written. I suspect for a lot of games this could find metatiles, rather than nametables. That explains a lot of what I'm seeing, I think. When stuff's compressed, you get RAM addresses, when it's metatiles, you get the metatile address, etc. Hmm. Though, honestly it's really quick to find this kind of stuff with a trace log anyway.

I'm wondering if this could be improved significantly if there was a second layer of indirection for RAM. i.e. any time the last read was from RAM, read back that RAM's last read recursively until you hit something? (Wondering also if it correctly tracks immediate values as the last read?) A lot of cases the resulting value would come from multiple sources though; intermediate arithmetic or binary combinations. (Edit: found it, it's all laid out in Debugger_Update() in src/Debugger.c, see where it calls SetMemFollow. Does seem to track immediates, and do what else it can in this respect.)


I'm thinking a generic memory-follower tool might be pretty useful, actually, like if we could do this for all RAM, not just the PPU and nametables. Though, I dunno what kind of overhead this would incur (probably fine if it can be turned on/off anyway). I change my mind a bit if the scope is expanded; it'd be a pretty decent feature for romhacking. (Easy to do already with a trace though.)

This is a neat idea! I'm doing this manually to learn the ropes (and because there's currently no other way), but a tool like this would certainly be handy.

That's why I suggested adding a filter for increasing ROM addresses: "consecutive stream in PRG ROM".


Then perhaps memory followers could be organized in this way:

1. Keep a trace log.
2. Collect heuristics. In this case, lots of writes to VRAM during a force-blanked frame likely indicate nametable loading.
3. When a heuristic triggers, grep the last hundred thousand instructions for the corresponding access pattern.

Hmm, if you could use the trace log tool and then combine it with the hex memory view. You could click on a memory address and choose "history", and have it follow it back to its roots spitting out a series of instructions and data fetches until it hits a dead end.

The history would be a tree structure. Something like LDA or an immediate operand would be a leaf node. Combining operations like ADC or EOR would be a branch node. A fetch from RAM would be a continuation node, starting a subtree going further back. Perhaps indexed loads could create a branch for the index register or pointer variable used too. Clicking on a node should take you directly to that moment in the trace log and/or directly to that byte in ROM.

I guess I'd invision a UI with 3 panes. Hex view (PRG/PPU/ROM), the history search, and the trace log.

I've written some simple trace parsers in python in the past just to do performance profiling and stuff like that. Something like this could be done as an offline tool as a proof of concept (trace logs can get kinda big anyway).

Anyhow, just dreaming about the kind of tool someone might make if they expanded this concept beyond nametables and last-read history.

Now THAT sounds useful!

Modern GPU debuggers have a "pixel history" feature, which is what this was reminding me of, where each write to a pixel you can drill down into all the associated information.