Reversing compiled code

This is an archive of a topic from NESdev BBS, taken in mid-October 2019 before a server upgrade.
View original topic
Reversing compiled code
by on (#240549)
On my path to ExHiROM I decided to stop by HiROM first, so I've extended my code to handle HiROM. I chose a fun small project Super Piccross to test this HiROM functionality upon, its pic-cross how hard can it be right? Also to learn the Lua extensions so I can modify/augment gameplay.

However the code was weird, once I worked out the bank register changes it started to make a little more sense but then still weird.

Then I noticed things like this..
Code:
jC07A04                          lda #$7000
                                 tax 
                                 lda #$0000
                                 bra bC07A11

bC07A0D                          sta p7F6000,x
bC07A11                          dex 
                                 dex 
                                 bpl bC07A0D

which screams "this was done by a compiler", then the weirdness made sense. What it does is, sets Y to the "struct" pointer, and Bank to 7e and then DP to the other structure. This way all the code is
Code:
lda $0024,y
ora $0036,y
clc
adc $7e08775
sta $0005,x

Which is maddening, as I work out a variable, but even if I change the label for it to it, it doesn't help because the code references it with a lda $XXXX,r since it can shift the X and Y around depending upon what the code needs they don't even always have the same offsets when used in different code.

Anybody got tips for this kind of thing? Do you see it often is worth adding "set offset" and look up into stuct systems to my regenerator?
Re: Reversing compiled code
by on (#240550)
Not necessarily done by a compiler. Humans program like this too.

Tips for what? Figuring out what every RAM address does? That is probably not possible.
Re: Reversing compiled code
by on (#240552)
Sure, but a human that can't see that this
Code:
jC07A04                          lda #$7000
                                 tax
                                 lda #$0000
                                 bra bC07A11

bC07A0D                          sta p7F6000,x
bC07A11                          dex
                                 dex
                                 bpl bC07A0D
should be
Code:
jC07A04                          ldx #$6FFF
                                 lda #$0000
bC07A0D                          sta p7F6000,x
                                 dex
                                 dex
                                 bpl bC07A0D
I doubt is going to go hardcore into Stack, DP, DB relocation, with offset strats.

Tips on how to track/manage/comment lessen the pain.
Re: Reversing compiled code
by on (#240568)
Oziphantom wrote:
Which is maddening, as I work out a variable, but even if I change the label for it to it, it doesn't help because the code references it with a lda $XXXX,r since it can shift the X and Y around depending upon what the code needs they don't even always have the same offsets when used in different code.

I think at this point every reference to a variable could become a two-part expression:
Code:
sta variable-base, X

So you'd have "variable" labels that can be an absolute address, and "base" labels that give a name to a temporary frame of reference. The variable would remain identified and searchable with different base combinations this way.