Reverse-engineering DLDI specs for NDS

This is an archive of a topic from NESdev BBS, taken in mid-October 2019 before a server upgrade.
View original topic
Reverse-engineering DLDI specs for NDS
by on (#228566)
Forward: Don't read unless you want to see me making a fool of myself.
This thread is about rev-engineering other people's open-source code, doing that is a bit foolish, isn't it? More so when doing it on a PC that doesn't support open-source.

---

Things I never could make sense of:

DLDI

As far as I understand that's some sort of driver for using different flashcarts with homebrew NDS titles, and it's extremely popular in most of the homebrew scene, and there's no technical documentation about how it's working. There's some open source code available, including a "template" for the driver structure, but it's all very abstract, not really providing info on which bytes are to be stored at which address for which purpose. Or does anybody know how to make sense of it?

For DSi (and unlaunch), it might be useful to supply a DLDI driver that allows old NDS homebrews to read from the DSi's SD/MMC slot instead from flashcart. Is there already such a DLDI driver?
The two possible problems would be: A few titles might want to access both flashcart (via DLDI) and SD/MMC slot (via 40048xxh), and might get confused if DLDI redirects to 40048xxh, too. And, NDS can access flashcarts via both ARM9 or ARM7, whilst DSi can access SD/MMC via ARM7 only (or would need a way to forward data from ARM7 to ARM9).

tepples wrote:
Does "Writing a DLDI Driver" at Chishm's page answer any of your questions?

Not so much, the page itself doesn't answer anything, and the links on the page only bring up more questions. The only tech specification I could find is the part about "32KiB are allocated for DLDI patches", and, well, I already knew that (it's the standard answer when asking anybody "What the heck is that DLDI thing that you are talking about?").

Looking through some driver source files...

default.zip contains a DLDI header, and some nonfunctional dummy functions. And some comments like ".byte 0x0F @32KiB @ Log [base-2] of the size of this driver in bytes." - I am reading that "use your floating point maths to compute logarithm of value base minus 2 then publish your result on @socialmedia!!!" - or it might try to say "8000h = 32Kbytes = 1 SHL 15".
dlms.zip I can see it containing two files with program code, but can't see any DLDI header in there.
gmtf.zip Contains a DLDI header, and also actual code, that's potentially useful for further research.

Some of the files contain something called "dldi.ld" is it safe to ignore that, or is it important?
The files on the chism webpage are quite old. A while ago I have also seen newer (?) files with Rumble support aside from the sector reading/writing. I don't remember where I have seen that. And, when writing DLDI specs, should that include features like Rumble, too?

Then for the patching tools, there's https://github.com/devkitPro/dstools/tr ... c/dlditool on github - which doesn't allow me to download that source code file(s). Could somebody upload a copy of that package here?
Re: Reverse-engineering DLDI specs for NDS
by on (#228568)
"ld" is the linker in GNU Binutils. My first guess is that the .ld file is a linker config file, analogous to the file specified with -C when calling ld65.

Is there anyone left on http://forum.gbadev.org to answer DLDI questions?

What error do you get when clicking "Raw" on the web view of dlditool.c? And what error do you get when using Git to download the entire devkitPro/dstools repository or download it as a zipfile?
Re: Reverse-engineering DLDI specs for NDS
by on (#228583)
Last time I had posted something on gbadev.org didn't work out too well.

I've managed to preview some of the github installer code on a tablet. Looks as if it contains a function named "quick find" and nobody has realized that it's slow as hell. Maybe I've misread the source code, but it looks as if it were scanning the whole file in 4-steps for finding the DLDI chunk - that would be about the slowest possible "quick find" method that I could think of (except maybe using 1-bit steps for non.byte-aligned addresses).
I guess the original idea was to install the driver once, and then keep the patched file unchanged for the rest of your life. But as far as I know flashcards have adopted the idea as auto-install the driver everytime when booting a game, in worst case that's meaning scanning a whole 128Mbyte file - only to find out that it doesn't contain a slot for DLDI driver at all.
Some possible workarounds might be scanning only the ARM9 and ARM7 sections of the file, hoping that the data isn't elsewhere. Even then, that would mean scanning up to several megabytes. Doing that on an ARM cpu might work via LDMIA r0-r9 registers, and then check r0-r9 for the first word of the ID via one CMP and nine CMPNE opcodes. And even then, the pure idea makes me like having smeared myself with someone else's dirt.

Ideally, the DLDI stuff should have been stored at fixed file location, or at a fixed location within ARM9 or ARM7 areas (eg. in their last 32Kbyte). But I guess the HLL people didn't know how to store data at a fixed location : /
I can't think of a way for fixing the design. One could scan the file once and then store the DLDI address in the carthdr. But that would mean modifying the carthdr - even for retail games (ie. putting a No DLDI flag in there).
Other idea would be scanning files once, and building a database, with info like <filename> was already scanned, and DLDI <does/doesn't> exist, located at <address>, plus a copy of the <TITLE> string to make sure <filename> really contains the expected file. Damn, no, the title is always "HOMEBREW", so maybe some CRC on some small but characteristic cart snippets.
Of course, if the database grows too large, then it might become faster to scan the .nds file than loading the database. Unless the database contains a quick look-up table with sorted entries.

Github is just doing the usual what-you-are-on-win98 stuff
Code:
Unable to complete secure transaction
You tried to access the address https://github.com/devkitPro/dstools/archive/master.zip

Well, I am firmly resisting to update my OS just because they don't want me to download their high-security master.zip file via http.
Anyways, I'll have a look at the gmtf.zip file for now. The master.zip would be also interesting because I had spotted some stuff related to "adjusting addresses" in there.
Re: Reverse-engineering DLDI specs for NDS
by on (#228584)
I have looked through the two source files in gmtf.zip, deleted the tab stops and other irrelevant stuff, and this is what I came up with. Understanding the __addresses would probably require more research via hex editors, but below should give at least some rough idea about the DLDI header.

I don't know where I got that files from, but some files on my SD card (BOOT_FC.NDS, for example) are throwing a "fatInitDefault failed!" message when starting them with unlaunch. I suspect that the error might be DLDI related, and if yes, should I cart about that and try to solve the issue in unlaunch?

Code:
DS Cart DLDI Driver
-------------------

DLDI (Dynamically Linked Device Interface for libfat) is a popular yet
undocumented flashcart driver for homebrew NDS software. Below was
reverse-engineered 11/2018 from "gmtf.zip" source code.

file.dldi --> driver file (can be small like 1.5Kbyte, or max 32Kbyte)
file.nds --> ROM image (must contain 32Kbyte space with DLDI ID for patching)

Driver patch file standard header -- 16 bytes
  00h 4  DLDI ID Word    (BF8DA5EDh)
  04h 8  DLDI ID String  (20h,"Chishm",00h)
  0Ch 1  Version Number  (must be 01h, patching tools will refuse other values)
  0Dh 1  Size of .dldi file (rounded up to 1 SHL N bytes) (max 0Fh=32Kbytes)
  0Eh 1  Sections to fix (see FIX_xxx)
  0Fh 1  Space in .nds file (1 SHL N bytes) (0Fh in .nds, can be 0 in .dldi)
Text identifier - can be anything up to 47 chars + terminating null -- 48 bytes
  10h 48 Name of current driver (ASCII, max 47 chars, plus zero padding)
Offsets to important sections within the data -- 32 bytes
  40h 4  __text_start    @ data start
  44h 4  __data_end      @ data end
  48h 4  __glue_start    @ Interworking glue start    -- Needs address fixing
  4Ch 4  __glue_end      @ Interworking glue end
  50h 4  __got_start     @ GOT start                  -- Needs address fixing
  54h 4  __got_end       @ GOT end
  58h 4  __bss_start     @ bss start                  -- Needs setting to zero
  5Ch 4  __bss_end       @ bss end
IO_INTERFACE data -- 32 bytes
  60h 4  Driver ioType ID (eg. ASCII "GMTF" meaning Datel Games 'n' Music?)
  64h 4  Features (see FEATURE_xxx)
  68h 4  Function Address startup()                           ;0=fail, 1=okay
  6Ch 4  Function Address isInserted()                 ;0=no/fail, 1=yes/okay
  70h 4  Function Address readSectors(sector,numSectors,buf)  ;0=fail, 1=okay
  74h 4  Function Address writeSectors(sector,numSectors,buf) ;0=fail, 1=okay
  78h 4  Function Address clearStatus()                       ;0=fail, 1=okay
  7Ch 4  Function Address shutdown()                          ;0=fail, 1=okay
  80h .. Driver Code (max 7F80h bytes)
Constants:
  FIX_ALL                            01h   ;um, so 01h+02h means... what?
  FIX_GLUE                           02h
  FIX_GOT                            04h
  FIX_BSS                            08h
  FEATURE_MEDIUM_CANREAD             00000001h
  FEATURE_MEDIUM_CANWRITE            00000002h
  FEATURE_SLOT_GBA                   00000010h
  FEATURE_SLOT_NDS                   00000020h


startup, isInserted, clearStatus, shutdown can be dummy functions that nothing (other than returning "true" which is supposedly meaning r0=1).
Alternately startup/shutdown can initialize or power down the hardware (if the hardware requires such things), clearStatus is meant to be some sort of SD card soft reset, and isInserted is allowing to test if the SD card is inserted & working.

read/write sectors are reading/writing one or more sectors. Sector size is 200h bytes, sector numbers is 0=First 200h bytes, 1=Next 200h bytes, and so on.
The driver functions can support SD and SDHC (or the flashcart manufacturer might release driver updates if SDHC wasn't supported).
Higher level FAT functions must be contained in the .nds file (so a driver update won't help if the .nds file lacks FAT32, for example).

FIX_xxx does maybe relate to address adjustments made by the dldi installer. Unknown if/how that's working. Would require https download from github for details.

SLOT_GBA/NDS seem to relate to GBA and NDS slots, the driver can probably have only one of the SLOT bits set (the functions don't allow to select which slot to use).
Purpose is unclear to me, maybe just telling the .nds file that the flashcart is in the given slot (and thereby shouldn't expect other hardware in that slot). Or maybe telling telling the installer which hardware the driver is supposed for.

Some DLDI variants seem to support extra features like Rumble, unknown if/how that's working.

tepples wrote:
"ld" is the linker in GNU Binutils. My first guess is that the .ld file is a linker config file, analogous to the file specified with -C when calling ld65.

Might be so. 7zip opens it with windows notepad on PC then showing an endless ASCII string without linebreaks. My tablet is telling me that I should join googlemail because the Play Store might have an application for viewing the .ld file content.
Well, I guess it's not worth reading the file (or learning how to configure linkers in general). Unless it's containing something important like forcing relocate-able code for DLDI...?
Re: Reverse-engineering DLDI specs for NDS
by on (#228586)
nocash wrote:
Last time I had posted something on gbadev.org didn't work out too well.

When was that?

nocash wrote:
Looks as if it contains a function named "quick find" and nobody has realized that it's slow as hell. Maybe I've misread the source code, but it looks as if it were scanning the whole file in 4-steps for finding the DLDI chunk

I was under the impression that the version in flash card firmware looked only in the ARM9 and ARM7 parts, as you suggest. That's why when I designed GBFS, which searches GBA address space at runtime, I required the space for the first archive to be 256 byte aligned to save on search time.

nocash wrote:
But I guess the HLL people didn't know how to store data at a fixed location : /

Correct. If more than one thing ends up requiring the same fixed location, they end up conflicting.

nocash wrote:
Github is just doing the usual what-you-are-on-win98 stuff
Code:
Unable to complete secure transaction
You tried to access the address https://github.com/devkitPro/dstools/archive/master.zip

Well, I am firmly resisting to update my OS just because they don't want me to download their high-security master.zip file via http.

I guess you could run a browser that uses a TLS library other than the OS's built-in SChannel. Might there be a build of Lynx, w3m, Links, or even Wget or cURL that works?

With KernelEx, Windows 98 users can run Firefox 3.6 stably and kinda-sorta run Firefox 9.
Re: Reverse-engineering DLDI specs for NDS
by on (#228589)
You edited in something about Windows Notepad "showing an endless ASCII string without linebreaks". That happens most often when a text file uses UNIX newlines (0A), not CP/M newlines (0D 0A). From the README file of my NES NROM template:

IF THIS FILE HAS NO LINE BREAKS: View it in a web browser.
Windows Notepad is a very popular text editor that comes with the Windows operating system, but it doesn't recognize line breaks in text files made on Linux or any other UNIX-like operating system.
Text files with UNIX line breaks display correctly in WordPad, Notepad++, Programmer's Notepad, Gedit, or a web browser, just not in Windows Notepad prior to the Windows 10 October 2018 Update.


What does WordPad show? What about versions of Emacs or Vim from the Windows 98 era? Or Notepad++ and Programmers Notepad before they dropped Windows 98 support?
Re: Reverse-engineering DLDI specs for NDS
by on (#228591)
tepples wrote:
What error do you get when clicking "Raw" on the web view of dlditool.c?

That link worked, indirectly, it's in google web cache, not the Raw version, but the one with line numbers is good enough.
Also here is a slightly older clean-text version http://www.smspower.org/maxim/forumstuff/dlditool.c in plain http
Re: Reverse-engineering DLDI specs for NDS
by on (#228597)
Yeah, I have a text editor that can handle 0Dh or 0Ah instead of 0Dh,0Ah. I just wasn't in mood for unpacking the .zip and loading the .ld file into a normal text editor (sorry if I have confused everybody about what I wasn't doing).

---

I've tried to search for DLDI with Rumble. All I could find was the http://www.chishm.com/DLDI/index.html webpage mentioning a "SuperCard Rumble (SD Card)", but it's without source code. However there's source for a "SuperCard Lite (SD Card)"...

The source code contains two rumble functions, "scms_rumble_flash_Erase" and "scms_rumble_flash_write".
Now I am confused, I was thinking that rumble would refer to vibration functions.
But now it does look more as if it were referring to a memory technology called "rumble flash"?
Probably not.

---

Looking through dlditool.c, the stuff about loading the .dldi file is probably not so important, except maybe mentioning the DLDIPATH environment variable. But that's more on the users side, not related to the binary file format.
Re: Reverse-engineering DLDI specs for NDS
by on (#228601)
The main part of the dlditool.c seems to be this
(with the little-endian 32bit readAddr/writeAddr functions abbreviated to rd and wr, and with addrIter abbreviated to i)
(and some blank lines & close brackets & error messages removed, too)
Code:
    patchOffset = quickFind (appFileData, dldiMagicString, appFileSize, sizeof(dldiMagicString)/sizeof(char));

    pDH = dldiFileData;
    pAH = &appFileData[patchOffset];

    // Make sure the DLDI file is valid and usable
    if (strcmp ((char*)dldiMagicString, (char*)&pDH[DO_magicString]) != 0) { err
    if (pDH[DO_version] != DLDI_VERSION) { err
    if (pDH[DO_driverSize] > pAH[DO_allocatedSpace]) { err

    memOffset = rd(pAH, DO_text_start);
    if (memOffset == 0) {                                        <--- ???
                    memOffset = rd(pAH, DO_startup) - DO_code;   <--- ????????

    ddmemOffset = rd(pDH, DO_text_start);
    relocationOffset = memOffset - ddmemOffset;

    printf ("Old driver:          %s\n",     &pAH[DO_friendlyName]);
    printf ("New driver:          %s\n",     &pDH[DO_friendlyName]);
    printf ("Position in file:    0x%08X\n", patchOffset);       eg. 0x000062C0
    printf ("Position in memory:  0x%08X\n", memOffset);         eg. 0x02000000
    printf ("Patch base address:  0x%08X\n", ddmemOffset);       eg. 0xBF800000
    printf ("Relocation offset:   0x%08X\n", relocationOffset);  eg. 0x428060C0

    ddmemStart = rd(pDH, DO_text_start);
    ddmemSize = (1 << pDH[DO_driverSize]);                      <--- !!!
    ddmemEnd = ddmemStart + ddmemSize;                          <--- ?????

    // Remember how much space is actually reserved
    pDH[DO_allocatedSpace] = pAH[DO_allocatedSpace];     <---- IMPORTANT !!!
    // Copy the DLDI patch into the application
    memcpy (pAH, pDH, dldiFileSize);

    // Fix the section pointers in the header
    wr(pAH, DO_text_start,   rd(pAH, DO_text_start)  + relocationOffset);
    wr(pAH, DO_data_end,     rd(pAH, DO_data_end)    + relocationOffset);
    wr(pAH, DO_glue_start,   rd(pAH, DO_glue_start)  + relocationOffset);
    wr(pAH, DO_glue_end,     rd(pAH, DO_glue_end)    + relocationOffset);
    wr(pAH, DO_got_start,    rd(pAH, DO_got_start)   + relocationOffset);
    wr(pAH, DO_got_end,      rd(pAH, DO_got_end)     + relocationOffset);
    wr(pAH, DO_bss_start,    rd(pAH, DO_bss_start)   + relocationOffset);
    wr(pAH, DO_bss_end,      rd(pAH, DO_bss_end)     + relocationOffset);
    // Fix the function pointers in the header
    wr(pAH, DO_startup,      rd(pAH, DO_startup)     + relocationOffset);
    wr(pAH, DO_isInserted,   rd(pAH, DO_isInserted)  + relocationOffset);
    wr(pAH, DO_readSectors,  rd(pAH, DO_readSectors) + relocationOffset);
    wr(pAH, DO_writeSectors, rd(pAH, DO_writeSectors)+ relocationOffset);
    wr(pAH, DO_clearStatus,  rd(pAH, DO_clearStatus) + relocationOffset);
    wr(pAH, DO_shutdown,     rd(pAH, DO_shutdown)    + relocationOffset);

    if (pDH[DO_fixSections] & FIX_ALL) {       <--- who or what is an "ALL" ????
      // Search through and fix pointers within the data section of the file
      for (i = (rd(pDH, DO_text_start) - ddmemStart); i < (rd(pDH, DO_data_end) - ddmemStart); i++) {
        if ((ddmemStart <= rd(pAH, i)) && (rd(pAH, i) < ddmemEnd)) {  <--- ?@#!!
          wr(pAH, i, rd(pAH, i) + relocationOffset);                  <--- !!!!!

    if (pDH[DO_fixSections] & FIX_GLUE) {               <--- is that HLL stuff?
      // Search through and fix pointers within the glue section of the file
      for (i = (rd(pDH, DO_glue_start) - ddmemStart); i < (rd(pDH, DO_glue_end) - ddmemStart); i++) {
        if ((ddmemStart <= rd(pAH, i)) && (rd(pAH, i) < ddmemEnd)) {  <--- ?@#!!
          wr(pAH, i, rd(pAH, i) + relocationOffset);                  <--- !!!!!

    if (pDH[DO_fixSections] & FIX_GOT) {                <--- maybe HLL related?
      // Search through and fix pointers within the Global Offset Table section of the file
      for (i = (rd(pDH, DO_got_start) - ddmemStart); i < (rd(pDH, DO_got_end) - ddmemStart); i++) {
        if ((ddmemStart <= rd(pAH, i)) && (rd(pAH, i) < ddmemEnd)) {  <--- ?@#!!
          wr(pAH, i, rd(pAH, i) + relocationOffset);                  <--- !!!!!

    if (pDH[DO_fixSections] & FIX_BSS) {
      // Initialise the BSS to 0      <--- needed if the .dldi filesize didn't include BSS area?
      memset (&pAH[rd(pDH, DO_bss_start) - ddmemStart] , 0, rd(pDH, DO_bss_end) - rd(pDH, DO_bss_start));

Is that normal coding practice for NDS homebrew code, or is it totally wrecked, or both?
The "ALL" section seems to be "text..data", that is all CODE and DATA? And any constants or opcodes in that area are forcefully modified by "+relocationOffset" if their 32bit value happens to fall within the ddmemStart..End range?
Or is it even applied in 1-byte steps, meaning that the last byte of an opcode, and the first three bytes of the next opcode could be modified by "+relocationOffset"?
I believe that I begin to understand why nobody has documented that functionality yet.
Or did I download the wrong file from google webcache?
Or maybe I am just not too open-minded today, maybe I can make more sense of it tomorrow.
Re: Reverse-engineering DLDI specs for NDS
by on (#228607)
"scms_rumble_flash_write" looks like it stands for "flash write for SuperCard Mini/MicroSD Rumble". The DLDI doesn't support rumble by itself, but it does support block device access on an SD adapter that also happens to have rumble.
Re: Reverse-engineering DLDI specs for NDS
by on (#228631)
I've been wondering if it were possible to use the DLDI driver - without using the bugged official DLDI installer.

Flashcarts with auto-patching might be already able to do that - they could probably repair the rom image in most cases, even if the official installer had damaged it previously.
For additional safety, the flashcarts could be shipped with encrypted DLDI drivers - so the official installer won't know how to install that patches. Hahaha, you didn't expect that, dlditool, didn't you!

Or simply change "Version (Must be 01h, else it won't work with the official installer),
to "Version (Must be anything OTHER than 01h, else it WILL work with the bugged official installer).

But the easiest workaround would be to create only DLDI drivers that are always having the FIX_xxx bits all set to zero - that should disabled the bugs, and then one could probably even use the official installer without running into troubles.
With that workaround, this whole stuff can be removed from the DLDI specs:
Code:
    if (pDH[DO_fixSections] & FIX_ALL) {       <--- who or what is an "ALL" ????
      // Search through and fix pointers within the data section of the file
      for (i = (rd(pDH, DO_text_start) - ddmemStart); i < (rd(pDH, DO_data_end) - ddmemStart); i++) {
        if ((ddmemStart <= rd(pAH, i)) && (rd(pAH, i) < ddmemEnd)) {  <--- ?@#!!
          wr(pAH, i, rd(pAH, i) + relocationOffset);                  <--- !!!!!

    if (pDH[DO_fixSections] & FIX_GLUE) {               <--- is that HLL stuff?
      // Search through and fix pointers within the glue section of the file
      for (i = (rd(pDH, DO_glue_start) - ddmemStart); i < (rd(pDH, DO_glue_end) - ddmemStart); i++) {
        if ((ddmemStart <= rd(pAH, i)) && (rd(pAH, i) < ddmemEnd)) {  <--- ?@#!!
          wr(pAH, i, rd(pAH, i) + relocationOffset);                  <--- !!!!!

    if (pDH[DO_fixSections] & FIX_GOT) {                <--- maybe HLL related?
      // Search through and fix pointers within the Global Offset Table section of the file
      for (i = (rd(pDH, DO_got_start) - ddmemStart); i < (rd(pDH, DO_got_end) - ddmemStart); i++) {
        if ((ddmemStart <= rd(pAH, i)) && (rd(pAH, i) < ddmemEnd)) {  <--- ?@#!!
          wr(pAH, i, rd(pAH, i) + relocationOffset);                  <--- !!!!!

Because there is no way that that code could work for installing DLDI drivers. Or, well, maybe future generations will find a way to make DLDI drivers that are working with that code. As for now, I would treat it as an interesting unsolved mathematical problem...

As far as I understand, ddmemXxx are virtual addresses (usually at BF80xxxxh, which could, say, conflict with zzBF80yyh opcodes), so the compiler could automatically change that virtual addresses to other regions (but take care that they won't conflict with other opcodes - or with things like the "Chishm" string in DLDI header itself). Or the compiler could replace opcodes like "MOV r3,0" by "SUB r3,r9,r9" where needed. A lossy compiler could also try to replace constants like timeout=10000 by timeout=9999, or use two constants like timeout=4000+6000 if that helps on avoiding conflicts. And of course the compiler could insert NOPs or other dummy opcodes/constants where needed.
Also encrypted code might help: Then one would only need to make sure that the header & decryption code are installed without conflicts, and the remaining code could be stored as "code XOR random", and the compiler could brute-force different random values until finding a setting that works without conflicts. Or storing all bytes (xxh) as halfwords (FxFxh), and making sure that ddmem doesn't cover any addresses at FxFxFxFxh.
The interesting part is that "+relocationOffset" is an unknown constant (it depends on the location of the DLDI space in different .nds file(s), and the compiler can't know that location(s) in advance). So, if the installer had patched a word at offset 100h..103h, then the values at 101h,102h,103h could contain unknown new values (with patching applied on random basis). A possible workaround would be inserting dummy words like FFFFFFFFh (or similar values in opcode space) after each and every word, so 101h..103h would become FFxxxxxxh, FFFFxxxxh, FFFFFFxxh (which might be all outside of the selected ddmem space).

With the above three bugged FIX_xxx's removed, one could also remove this from DLDI specs:
Code:
    if (pDH[DO_fixSections] & FIX_BSS) {
      // Initialise the BSS to 0      <--- needed if the .dldi filesize didn't include BSS area?
      memset (&pAH[rd(pDH, DO_bss_start) - ddmemStart] , 0, rd(pDH, DO_bss_end) - rd(pDH, DO_bss_start));

That part isn't strictly bugged, but it's pretty useless (assuming that driver creators would know how to zeropad their files on their own (though I might over-estimate their skills)).
Re: Reverse-engineering DLDI specs for NDS
by on (#228633)
tepples wrote:
"scms_rumble_flash_write" looks like it stands for "flash write for SuperCard Mini/MicroSD Rumble". The DLDI doesn't support rumble by itself, but it does support block device access on an SD adapter that also happens to have rumble.

Yeah, doing an image search, those SuperCards with Rumble seem to be all GBA slot devices (probably requiring a passme adaptor in the NDS slot).
For the GBA slot, there are three different official rumble methods http://problemkaputt.de/gbatek.htm#dscartrumblepak maybe the SuperCards are emulating one (or more) of those methods, or maybe they come up with a fourth method. But, yeah, it's apparently done via direct I/O, not via the DLDI driver.

Speaking of GBA memory. The GBA could map ROM to memory address 8000000h and up, and it could execute code in there.
But DLDI is for NDS, and that requires to transfer code from ROM to RAM before executing it.
At least that's so when reading from NDS slot (and in practice, it's almost always also done that way when reading NDS code from GBA slot).

The RAM addresses for NDS code are stored in NDS cart header - but the DLDI installer doesn't look at the NDS cart header at all. So the installer's buggy address adjustments seem to be yet incomplete...
Maybe it does require further address adjustments to be carried out by the code in the .nds file before using the DLDI functions?
Or maybe DLDI requires the ARM9 code always being located at, say, 2000000h?
Some DLDI drivers are said to work (or don't so) for ARM7, so then ARM7 might be also required to be located at a certain fixed RAM address?

Though the installer adjusts addresses relative to the ROM address, so to come up with fixed RAM addresses, one would also need fixed ROM address in the first place. For ARM9 one could say that it must be always located at ROM offset 4000h or the like, but I don't see how one could do the same for ARM7 (except by always zeropadding the ARM9 area to it's maximum size).
Re: Reverse-engineering DLDI specs for NDS
by on (#228636)
Metapad supports UNIX line endings and Windows 98 still.
http://liquidninja.com/metapad/
Re: Reverse-engineering DLDI specs for NDS
by on (#228642)
nocash wrote:
Speaking of GBA memory. The GBA could map ROM to memory address 8000000h and up, and it could execute code in there.
But DLDI is for NDS, and that requires to transfer code from ROM to RAM before executing it.
At least that's so when reading from NDS slot (and in practice, it's almost always also done that way when reading NDS code from GBA slot).

Which means that if the firmware is waiting for the NAND flash (CF, SD, etc.) to have a sector ready, it can search for the DLDI signature in the previous sector(s) in order to patch the .nds executable as it streams in. This interleaving might be part of why you don't notice a wait when loading an application using DLDI.

In order to help you further, I might need to understand what went wrong last time you asked on forum.gbadev.org.
Re: Reverse-engineering DLDI specs for NDS
by on (#228643)
I have dumped the BOOT_FC.NDS file to see how DLDI would look inside of actual .nds files:
Code:
BOOT_FC.NDS
 000000: 2E 00 00 EA 00 00 00 00  00 00 00 00 23 23 23 23   ............####
 000010: 00 00 00 00 01 00 00 00  00 00 00 00 00 00 00 04   ................
 000020: 00 02 00 00 00 08 00 02  00 00 00 02 78 A2 03 00   ............x...
 000030: 00 A6 03 00 00 00 38 02  00 00 38 02 CC 24 00 00   ......8...8..$..
 ...
 000F70: 3C BC 90 46 99 46 A2 46  AB 46 F0 BD 48 67 02 02   <..F.F.F.F..Hg..
 000F80: EE A5 8D BF 00 76 03 02  44 4C 44 49 F8 B5 DE 46   .....v..DLDI...F
 000F90: 4E 46 45 46 57 46 E0 B5  07 00 01 20 98 46 0A AB   NFEFWF..... .F..
 ...
 001220: 02 20 E3 E7 A0 EF FF FF  E0 65 02 02 18 11 02 02   . .......e......
 001230: FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF   ................
 001240: ED A5 8D BF 20 43 68 69  73 68 6D 00 01 0E 00 0E   .... Chishm.....
 001250: 44 65 66 61 75 6C 74 20  28 4E 6F 20 69 6E 74 65   Default (No inte
 001260: 72 66 61 63 65 29 00 00  00 00 A0 E1 00 00 A0 E1   rface)..........
 001270: 00 00 A0 E1 00 00 A0 E1  00 00 A0 E1 00 00 A0 E1   ................
 001280: 40 10 00 02 40 50 00 02  00 00 00 00 00 00 00 00   @...@P..........
 001290: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
 0012A0: 44 4C 44 49 00 00 00 00  C0 10 00 02 C0 10 00 02   DLDI............
 0012B0: C0 10 00 02 C0 10 00 02  C0 10 00 02 C0 10 00 02   ................
 0012C0: 00 00 A0 E3 1E FF 2F E1  00 00 00 00 00 00 00 00   ....../.........
 0012D0: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
 0012E0: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
 0012F0: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
 ...
 0267B0: E9 52 00 06 00 00 00 00  ED A5 8D BF 20 43 68 69   .R.......... Chi
 0267C0: 73 68 6D 00 ED A5 8D BF  20 43 68 69 73 68 6D 00   shm..... Chishm.
 0267D0: 42 4F 4F 54 2E 4E 44 53  00 FF FF FF B8 54 00 06   BOOT.NDS.....T..
 0267E0: C8 54 00 00 62 6F 6F 74  73 74 75 62 A8 00 00 00   .T..bootstub....
 ...
 026940: 04 FE FF 02 60 01 00 00  EE A5 8D BF 20 43 68 69   ....`....... Chi
 026950: 73 68 6D 00 00 00 00 00  00 00 00 00 00 00 00 00   shm.............
 026960: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
 ...
 03D540: 44 00 53 00 69 00 4D 00  65 00 6E 00 75 00 2B 00   D.S.i.M.e.n.u.+.
 03D550: 2B 00 0A 00 52 00 6F 00  62 00 7A 00 38 00 00 00   +...R.o.b.z.8...
 ...

Ah, okay, the BOOT_FC.NDS is DSiMenu++ by Robz8!
Though the carthdr[012h] unitcode says that it's a NDS title without DSi support... I guess it requires some bootstub hack to load that file in DSi mode.
For some reason there are four DLDI headers, at offset 1240h, 267B8h, 267C4h, and 26948h. The installer will only process one of them (the first one found in memory, ie. at 1240h), so the other three headers won't get patched (and the last one has the EDh byte replaced by EEh so it doesn't fully match the required ID anyways).

Now if you thought that everybody knew that DLDI was supporting 32Kbytes patches, look at 124Dh. That's only max 16Kbytes allocated (1 SHL 14).
I guess there might be also .nds files that have 32Kbyte allocated, but one can't trust on that. Maybe there are even some with less than 16Kbyte out there?

Looking at the carthdr ARM9 area:
- ARM9 ROM offset 200h
- ARM9 RAM address 2000000h
So the stuff at 1240h in ROM will appear at 2001040h in RAM.
I guess DLDI simply requires RAM to be located at ROM+1FFFE00h.
So if the carthdr were 4000h bytes in size and ARM9 code would be at ROM offset 4000h, then RAM load address would have to be 2003E00h, right?
Or alternately one would need the kind of bootstub hack that appears to be used for loading the above file, then one could use files with 200h-byte headers, as they were apparently generated by some old NDS compilers.

Looking at the Function addresses in the DLDI header, for example the init function at ROM 12A8h:
That's already set to 20010C0h (the correct function address in RAM, when using the RAM=ROM+1FFFE00h formula).
So there's apparently no further address adjustment needed to be carried out by the code in the .nds file.
Re: Reverse-engineering DLDI specs for NDS
by on (#228647)
tepples wrote:
In order to help you further, I might need to understand what went wrong last time you asked on forum.gbadev.org.

No, you rather don't want to know that - it just didn't work out well. And, there's almost nobody left having started new threads in the DS development section in past 5 years. Maybe there'll be some bigger NDS/DSi retro scene in some years, but at the moment it's not, at least not at gbadev.org.
Or maybe it'll be in nesdev other retro section - the NDS is really kinda retro now. And the DSi was widely seen as NDS cost-down-retro-revision even when it was new. I only hope nobody feels distracted when I post that kind of stuff here - it's just because I like this forum (and mind that the GBA/NDS video engines aren't so distant from SNES).

tepples wrote:
This interleaving might be part of why you don't notice a wait when loading an application using DLDI.

If the flashcart bootloaders were supporting multithreading for that purpose (I would guess not).
Finding the DLDI header won't take up too much time anyways (if it's in first some Kbytes of the ARM9 area).
The bigger problem would loading NDS programs without DLDI support - then it would need to scan the whole ARM9 area.

I didn't do the maths yet on how slow it could be - maybe it isn't taking up too many milliseconds even on large ARM9 bootcode blocks. But, two ideas for speeding up things:

If the RAM=ROM+1FFFE00h formula does apply to all cart headers, then one could use that to detect files with DLDI (and skip patching normal software without DLDI).

And, there are those sixteen FFh-bytes at 1230h in the above ROM dump, that looks as if the DLDI header at 1240h were 32-byte aligned? If that's the case in all .nds files then one would need to verify only each 8th word. The memory system would still have the same amount of traffic (for loading 32-byte cache lines), but "LDR r0" should be around 7 clks faster than "LDMIA r0-r7", and needing only one CMP instead of eight CMP/CMPNE's should be also 7 clks faster.

Oh, and ARM7 support. My current theory would be that the code is loaded to ARM9 memory space (using the RAM=ROM+1FFFE00h addressing), and ARM7 might simply jump to the same Main RAM locations when using the DLDI functions.
The things that could cause ARM7 issues:
The driver maker might have used ARM9 opcodes (eg. BLX) in DLDI.
And, if the .nds file does NOT align the driver area to 32-byte cache lines, then things could go wrong if it would store variables at the very end of the allocated DLDI memory space (but unlikely that it has squeezed memory that tightly).
Re: Reverse-engineering DLDI specs for NDS
by on (#228651)
Sorry. Not important.
Re: Reverse-engineering DLDI specs for NDS
by on (#228653)
I'd hazard a guess that drivers for SLOT-2 cards are more likely to be ARMv4 compatible in order to run on the ARM7 in GBA mode, as a SLOT-2 card can also be used in an actual GBA.
Re: Reverse-engineering DLDI specs for NDS
by on (#228655)
Yah, the GBA cart makers probably knew how not to use ARM9 opcodes.
Going by the chism page, the problem applies to an older version for some R4 card, and there's a patched bugfixed driver for it.
I haven't compared the two R4 driver versions yet, doing so would probably reveal what had caused the ARM7 issue.

My theory about a fixed "RAM=ROM+1FFFE00h" formula was wrong. The installer is doing that by taking the old and new text_start entries (ie. from the old unpatched .nds file, versus newly installed driver from .dldi file):
Code:
    memOffset = rd(pAH, DO_text_start);       <--- FROM pAH here !!   (.nds)
    if (memOffset == 0) {memOffset = rd(pAH, DO_startup) - DO_code;}
    ddmemOffset = rd(pDH, DO_text_start);     <--- FROM pDH here !!!! (.dldi)
    relocationOffset = memOffset - ddmemOffset;

Hmmmm, I think... that's working for any memory addresses including in ARM7 area, and including in Fast WRAM, or even in VRAM.
The text_start entry in the .nds file tells the installer where the driver finally appear in RAM (regardless of it's location in ROM).
Re: Reverse-engineering DLDI specs for NDS
by on (#228660)
Just making sure that I didn't completely misread the HLL code in dlditool.c. Here comes the proof-of-bugginess.
Attachment:
killer.zip [799 Bytes]
Downloaded 421 times

BEWARE the attachment will DESTROY your NDS file. You can't re-install any working DLDI driver in your NDS file after using killer.dldi.

Code:
BOOT_FC.NDS file before installing killer.dldi:
   001240: ED A5 8D BF 20 43 68 69  73 68 6D 00 01 0E 00 0E   .... Chishm.....

BOOT_FC.NDS file after installing killer.dldi (bytes at 1243h-1246h are destroyed):
   001240: ED A5 8D FF 30 00 02 69  73 68 6D 00 01 0E 01 0E   ....0..ishm.....

Content of the KILLER.DLDI file:
   000000: ED A5 8D BF 20 43 68 69  73 68 6D 00 01 0E 01 0E   .... Chishm.....
   000010: 44 4C 44 49 20 64 72 69  76 65 72 20 66 6F 72 20   DLDI driver for
   000020: 64 65 73 74 72 6F 79 69  6E 67 20 79 6F 75 72 20   destroying your
   000030: 4E 44 53 20 66 69 6C 65  00 00 00 00 00 00 00 00   NDS file........
   000040: 00 00 43 68 F8 7F 43 68  00 00 00 00 00 00 00 00   ..Ch..Ch........
   000050: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
   000060: 4B 49 4C 4C 00 00 00 00  80 00 43 68 80 00 43 68   KILL......Ch..Ch
   000070: 80 00 43 68 80 00 43 68  80 00 43 68 80 00 43 68   ..Ch..Ch..Ch..Ch
   000080: 00 00 A0 E3 1E FF 2F E1  00 00 00 00 00 00 00 00   ....../.........
   000090: 00 00 00 00 00 00 .. ..  .. .. (zeropad to 16Kbyte size)

The killer.dldi file is basically a copy of default.dldi, with FIX_ALL enabled, and with virtual addresses changed from BF80xxxxh to 6843xxxxh (in this case causing the "Ch" in "Chishm" to be seen as matching address MSBs). This is making use of all bugs at once:
- The DLDI header itself can be destroyed by address adjustments.
- Address adjustments can be performed even on non-word-aligned-words.
- FIX_ALL really doesn't work (and the other FIX_features aren't much better).
Re: Reverse-engineering DLDI specs for NDS
by on (#228691)
I've tried to track down a few more programs that use DLDI...

Is there a list with "Games and Tools that do use DLDI" somewhere?

What I've found is https://ezflash.sosuke.com/viewtopic.php?t=4048 (list of 15 dldi games/tools, most links are broken, but 5-6 are still working), and https://pdroms.de/?s=dldi&post_type=pdr_file_post (10 more dldi games/tools). And, http://www.chishm.com/SendSave/index.html (has a GBA (!) tool called "GBA Game Save Backup Utility" using DLDI).

Looking at some of those downloads... they are all having 32Kbytes allocated (so the BOOT_FC.NDS with only 16Kbytes seems to have been a rare exception - and it's probably nonfunctional anyways, as it's a DSi tool that probably doesn't support NDS flashcarts at all).
For the alignment, they seem to be all 64-byte aligned, but the padding space isn't obvious (=FFh filled) in all cases, so it's hard to say if they were padding for 32-byte or 64-byte alignment... except, the GBA tool, that's clearly using padding to 64-byte boundary.
So I guess it might be relative safe to assume that the installer tool needs to check only words at 64-byte boundaries (in "most cases" at least, there's probably somebody who had changed that alignment).

Would there be any interest in DLDI support on DSi? I have no idea how many games & tools are actually needing DLDI support (and if they were really worth adding DLDI support).
The "top ten titles" might already have native support for DSi SD/MMC slot, if that's so, then DLDI would be needed only for some rather obscure old NDS titles, ie. the kind of stuff that wasn't updated (and maybe not even used or downloaded) in the past some years.

About adding DLDI support in unlaunch: It might be possible, but at the moment I don't really want to go through the hazzle...
The how-to idea would be "hoping" that most of the DLDI tools are using some sort of "standard libnds" functions on ARM7, then the installer could hook that code (IRQ handler or so), and the hook could check if ARM9 requests a DLDI function to be executed.
Most of the handshake could be done via the DLDI main RAM buffer (ie. by having the request/param words, and the ARM7 code, and a small buffer for a few sector(s), and the a request/ready flag in that memory space) (so one won't need any DMA or IPC for communicating between ARM9 and ARM7).
The main issue would be to find a good ARM7 hook location (and hope that it's alive & working in all situations, and without crashing or screwing up ARM7 timings too much).

If somebody knows a good hook location in common homebrew ARM7 code, please let me know!!!
Re: Reverse-engineering DLDI specs for NDS
by on (#228694)
I have also disassembled some of the DLDI related code in the boot_fc.nds file (ie. that stuff that uses the .dldi functions).

First of, below is initializing EXMEMCNT using the FEATURE_SLOT_xxx flags in DLDI header.
For ARM7 side, one would need to change the code to set bits in EXMEMCNT to 1=ARM7 instead of 0=ARM9.
EXMEMCNT is write-able only on ARM9, so the code must be always executed on ARM9 side (even when using DLDI on ARM7 side).

Normally only one of the FEATURE_SLOT_xxx bits should be set (depending on where the storage hardware is located).
The code does also permit BOTH bits (in rare case that the memory were wired to BOTH slots, or with having an access LED in the second slot, or whatever weird thing).
The code does also permit NONE of the bits to be set (eg. when using other media) (eg. would work for DSi SD/MMC slot, but, that slot can be accessed by ARM7 only, so it won't work with DLDI code on ARM9 side).
Code:
                 ;------------------
                 init_dldi_features:
0200B028 4B08     ldr     r3,=2001040h       ;DLDI hdr
0200B02A 6E5B     ldr     r3,[r3,64h]   ;hdr[64h] aka FEATURE_xxx flags
0200B02C 06DA     lsl     r2,r3,1Bh  ;bit4 to bit31 (signflag) ;FEATURE_SLOT_GBA
0200B02E D504     bpl     200B03Ah ;@@skip1
0200B030 2080     mov     r0,80h
0200B032 4907     ldr     r1,=4000204h  ;EXMEMCNT
0200B034 880A     ldrh    r2,[r1]
0200B036 4382     bic     r2,r0    ;clear bit7  ;GBA Slot Access (0=ARM9)
0200B038 800A     strh    r2,[r1]
                 @@skip1:
0200B03A 069B     lsl     r3,r3,1Ah  ;bit5 to bit31 (signflag) ;FEATURE_SLOT_NDS
0200B03C D504     bpl     200B048h ;@@skip2
0200B03E 4A04     ldr     r2,=4000204h  ;EXMEMCNT
0200B040 4904     ldr     r1,=0FFFFF7FFh
0200B042 8813     ldrh    r3,[r2]
0200B044 400B     and     r3,r1    ;clear bit11 ;NDS Slot Access (0=ARM9)
0200B046 8013     strh    r3,[r2]
                 @@skip2:
0200B048 4803     ldr     r0,=2001040h+60h   ;return DLDI hdr+60h
0200B04A 4770     bx      r14


And then here's a code snippet for calling the actual DLDI functions:
The BLX would work only on ARM9, but ARM7 could do something similar via some opcode combo.
If the functions are always called via BLX or BX, then they could have THUMB set.
But it might be safer to stick with ARM code for the function entrypoints.
Code:
                 ;------------------
                 dldi_startup_inserted:   ;to be called AFTER init_dldi_features
020061F4 B5F0     push    r4-r7,r14
020061F6 4647     mov     r7,r8      ;save r8
020061F8 46CE     mov     r14,r9     ;save r9
020061FA B580     push    r7,r14
020061FC 0004     lsl     r4,r0,0h   ;r4=r0
020061FE B085     add     sp,-14h
02006200 000D     lsl     r5,r1,0h   ;r5=r1 (aka dldi hdr+60h)
02006202 4690     mov     r8,r2      ;r8=r2
02006204 001E     lsl     r6,r3,0h   ;r6=r3
02006206 2800     cmp     r0,0h
02006208 D011     beq     200622Eh ;@@fail
0200620A F007FE49 bl      200DEA0h              ;---> whatever
0200620E 426A     neg     r2,r5                 ;\
02006210 416A     adc     r2,r5                 ;
02006212 2308     mov     r3,8h                 ; uhm,
02006214 4283     cmp     r3,r0                 ; compiler optimization for not
02006216 4180     sbc     r0,r0                 ; doing CMP r5,0 + CMP r0,8 for
02006218 4240     neg     r0,r0                 ; having only 1 branch opcode?
0200621A 4310     orr     r0,r2                 ;
0200621C D107     bne     200622Eh ;@@fail      ;/
0200621E 68AB     ldr     r3,[r5,8h]            ;\
02006220 4798     blx     r3                    ; call hdr[68h] aka startup()
02006222 2800     cmp     r0,0h                 ;
02006224 D003     beq     200622Eh ;@@fail      ;/
02006226 68EB     ldr     r3,[r5,0Ch]           ;\
02006228 4798     blx     r3                    ; call hdr[6Ch] aka isInserted()
0200622A 1E07     sub     r7,r0,0               ;   ;<-- r7=result (!)
0200622C D106     bne     200623Ch ;@@okay      ;/
                 @@fail:
0200622E 2700     mov     r7,0h
                 @@return_r7:
02006230 0038     lsl     r0,r7,0h
02006232 B005     add     sp,14h
02006234 BC0C     pop     r2,r3
02006236 4690     mov     r8,r2
02006238 4699     mov     r9,r3
0200623A BDF0     pop     r4-r7,r15
                 ;---
                 @@okay:
                  ... more stuff here... then jumping to @@return_7

If startup or isInserted fails, then the program is usually displaying "fatInitDefault failed!".
Some programs are also having more meaningful error messages, directing the user to check if have installed their DLDI driver.
Re: Reverse-engineering DLDI specs for NDS
by on (#228695)
EXMEMCNT? What the crap?
Re: Reverse-engineering DLDI specs for NDS
by on (#228696)
Yes, sorry. I was tired and did spellcheck what I was writing. I meant that one needs to control the excrements, of course.
I'll also need to fix that in http://problemkaputt.de/gbatek.htm#dsme ... andmainram

I've also checked the r4tf v1/v2 drivers.
The older v1 without ARM7 support is using THUMB code trying to return to ARM code with "POP r15".
That won't work on ARM7 (it had to use "POP r1 + BX r1" or the like).
The later v2 doesn't use any THUMB code at all (that new version is also a good bit smaller than the old version).

The v1 source code does also include some comments & options for DMA support (disabled), and support for non-2-byte aligned sector buffers (enabled). I would have assumed that sector buffers were usually/always 4-byte aligned. But going by that comments there might be a few .nds files having completely unaligned buffers.
Re: Reverse-engineering DLDI specs for NDS
by on (#228700)
ndiddy suggested K-Meleon 74 Windows 9x Edition, which runs on a Windows 98 PC with 1 GB of RAM and KernelEx set to Windows 2000 SP4 compatibility.
Re: Reverse-engineering DLDI specs for NDS
by on (#228702)
Hello. I've done some work with DLDI in the past, so if there are questions you would still like to see answered despite your "please disregard this topic" warning at the top of the first post, go ahead.

Basically,
- a DLDI-capable .nds has a location where the DLDI program should get installed and code in that .nds will use the function pointers at that location to access the driver
- the DLDI driver should come from the device itself. The .nds can be patched on-the-fly when loaded into memory.
- some older linker cards did not have DLDI. Instead, they used to expect developer to access SD card through their own, proprietary interface. That was e.g. the case for the supercard. When DLDI came out, people using these cards had to rely on a .dldi file and a batch-patcher tool to embed the driver into the .nds image before loading the game on their system, because nothing in the linker firmware would do that (and the game would not know their device).

That's why you see a DLDI chunk with placeholders in .nds files you download: this is simply the code that is there, waiting to be replaced by the linker firmware. And it likely does nothing because it doesn't know what device it will be running on.

Hope that helps.
Re: Reverse-engineering DLDI specs for NDS
by on (#228712)
tepples wrote:
ndiddy suggested K-Meleon 74 Windows 9x Edition, which runs on a Windows 98 PC with 1 GB of RAM and KernelEx set to Windows 2000 SP4 compatibility.

Interesting, the faked-user-agent-feature might even work for win98 users to get into closed community sites like cpcwiki.eu but I am still feeling badly discriminated because I am not one of the win98 users with whopping 1GB RAM nor with KernelEx (not too mention that they forgot win95 support in their win9x version). But the project as such isn't bad.

I've tried to estimate the chance of DLDI errors to occur with the FIX_xxx features:
Code:
FIX_ALL:
The risk for one word to match ddmemStart base would be:
  1:100000000h  for BF800000h
The risk for one word to match ddmemStart..ddmemEnd would be:
  1:1000000h    for BF800000h..BF8000FFh  ;small .dldi file
  1:20000h      for BF800000h..BF807FFFh  ;large .dldi file
The risk for ANY WORD in the .dldi file to match ddmemStart..ddmemEnd would be:
  1:10000h      for BF800000h..BF8000FFh  ;small .dldi file
  1:4           for BF800000h..BF807FFFh  ;large .dldi file
If there were 16 .dldi drivers, 64 .nds homebrews, each .nds with 4 revisions,
that would give 16x64x4 combinations (=1000h), so the overall risk would be:
  1:10h         for BF800000h..BF8000FFh  ;small .dldi file
  400h:1        for BF800000h..BF807FFFh  ;large .dldi file
Or same if the 64 authors had tested 64 builds with 1 .dldi driver each.
I haven't spotted a .dldi driver actually using FIX_ALL yet, but if it were
used then it would probably cause mysterious troubles once or then.

FIX_GLUE and FIX_GOT:
The GLUE/GOT tables are relative small, so that does reduce the risk.
If all words in the table are WANTED to be adjusted, and if ddmem range is
BF800000h..BF807FFFh then there's no risk of having a word with BF80h in its
LSBs (since BF80BF80h or BFBF80xxh would exceed the BF800000h..BF807FFFh
range).
The LSB of a word could be BFh, but the previous (already adjusted) word
would be likely a NDS memory address (eg. 00xxxxxxh..07xxxxxxh) without
risking to have 80h in it's MSB (which could be seen as BF80h).
No idea if Chishm was doing that intentionally, but using BF800000h as ddmem
base seems to be surprisingly well chosen to overcome the installer bugs.
The only issue would be a .dldi file that uses the whole 32K space with
GOT/GLUE tables located at the very end of the 32K space, then the next bytes
after the 32K area would cause problems if they were 80h,BFh or xxh,80h,BFh.
FIX_GLUE and FIX_GOT are used by most or all .dldi drivers, but it's quite
possible that they will never trigger any bugs due to lucky circumstances.


PypeBros wrote:
if there are questions you would still like to see answered despite your "please disregard this topic" warning at the top of the first post

That warning about making a fool of myself? That's just there because I suspect that most people would say that it's foolish to "reverse engineer open source HLL projects" (or at least, I would say so myself). But if you want to join the club...

I am wondering how many dldi drivers are there out there?

The chishm webpage has about 40 drivers (more than I had estimated above), and they are all from 2006-2008, so there might be even more dldi drivers since then, unless the newer flashcarts are still using the old drivers?

Especially interesting would be if some of them are using FIX_ALL?
Using that would make sense if they have addresses in literal pool, without being aware of the installer bugs.

And is DLDI as error-prone as it looks like in the installer source code?
Eg. people having properly installed the correct driver, and then randomly seeing undefined opcodes during driver execution?

Oh, and is there some reason for the unaligned-word-adjustments?
I mean, ARM CPU can't access unaligned words at all, unless it were using some kind of "unaligned_32bit_read_function".
If some .dldi files (or libfat) are using unaligned words then it would be required, otherwise it's making the address-adjust guessing more unreliable.

Yesterday I have spotted this sentence in internet (discussing ExFAT with DLDI):
Quote:
Oh, and on most linkers, that would require a firmware upgrade because the linker use the SD card to store files of its own, too. again, this might not be available except you have a fresh new linker.

That makes me wonder if there's something going on - that I am not aware of yet.
What is that linker???
It's apparently not the usual "source-->assembler-->linker-->executable" toolchain.
It sounds more as if there were a runtime-linker used by the .dldi or .nds file itself?
Maybe "linker" is referring to the bootmenu in the flashcarts...?
Re: Reverse-engineering DLDI specs for NDS
by on (#228716)
"Flash Advance Linker" was the name for the first GBA cartridge writer from Visoly. I guess the name "Linker" for the writer ended up sticking.
Re: Reverse-engineering DLDI specs for NDS
by on (#228718)
nocash wrote:
Yes, sorry. I was tired and did spellcheck what I was writing. I meant that one needs to control the excrements, of course.
I'll also need to fix that in http://problemkaputt.de/gbatek.htm#dsme ... andmainram

EXMEMCNT for EXternal MEMory CoNTrol, presumably.

Quote:
The v1 source code does also include some comments & options for DMA support (disabled), and support for non-2-byte aligned sector buffers (enabled). I would have assumed that sector buffers were usually/always 4-byte aligned. But going by that comments there might be a few .nds files having completely unaligned buffers.

It would indeed be very welcome to align buffers for performance. However, because the block you want to write might come from something else, with some poor padding before the useful data and so on, it is usually good practice for library-class code to be ready to face the odds of non-aligned input from client code rather than pushing the restriction down to the client code (we see that in memmove, and other similar things, iirc).

Quote:
But as far as I know flashcards have adopted the idea as auto-install the driver everytime when booting a game, in worst case that's meaning scanning a whole 128Mbyte file - only to find out that it doesn't contain a slot for DLDI driver at all.

The flashcard firmware won't load/scan the whole 128MiB. The NDS memory is only 4MiB and there would be no point in loading more than that in memory. The rest of the file (for homebrews) is typically accessed through the DLDI driver. And when you want to run the binary, your flashcard has to load it all in memory anyway, so it can just scan as it loads.

I see that the FIX_ALL thing is puzzling you a lot. And indeed, it feels pretty weird to request that any occurence of a given 32-bit word would require it to be transformed. I can imagine that this was provision for driver writers who would not necessarily generate a "global offset table" (which is a trick C/C++ compiler does to support relocatable code). The technique of relocating code that way is fairly common, and seen in Linux kernel modules and shared libraries. I think FIX_ALL might have been added to that to support hypothetical drivers written e.g. in low-level assembly. Clearly, before doing that (requiring the DLDI patcher to FIX_ALL), you would have to ensure that none of your constants/strings/opcodes accidentally enter the memory range where things are fixed.

Quote:
But DLDI is for NDS, and that requires to transfer code from ROM to RAM before executing it.
At least that's so when reading from NDS slot (and in practice, it's almost always also done that way when reading NDS code from GBA slot).

Even when your linker (like the SuperCard) is a GBA-slot device, only a small part of its internal memory is used for accessing the SD card contents. Usually it will map some registers on the GBA bus, as well as a few sector-sized cache. But the content of the .nds file needs to be streamed from the SD card to the RAM through those on-GBA-slot-"ROM" areas. Which sector of the SD card to fetch and whether the underlying hardware is ready for that is the job of the other memory-mapped registers.
Only the SLOT-1 linker firmware (like a R4) will be streamed into RAM with the "normal" nintendo protocol when homebrew is involved (afaik).
Re: Reverse-engineering DLDI specs for NDS
by on (#228719)
but even with all that, you still have (imho) a valid point with the way the patching loops work. I'm pretty surprised that there is no code advancing to the next word once a word has been patched. After all, having code that just could patch the same 32-bit word twice is a non-sense. I see it survived even in the newer "bootloader" function of 'hbmenu', though.

And no, in 10+ years of NDS development and DLDI patching, I've never encountered an invalid opcode that could be the responsibility of the DLDI process rather than to some silly loop of mine. Of course, if such mis-patching ever occured, it would be caught at the design house of the linkers vendors (as it doesn't patch the .nds, but the .dldi)

NOTE, that when advancing the patcher will keep reading the MSB of the patched pointer while ignoring LSB of the patched pointers and start considering LSB of the next pointer (or non-pointer), because of little-endianness of the NDS platform.
Re: Reverse-engineering DLDI specs for NDS
by on (#228729)
tepples wrote:
"Flash Advance Linker" was the name for the first GBA cartridge writer from Visoly. I guess the name "Linker" for the writer ended up sticking.

Ah! That does explain it. Reminds me that there's also a "DSLink" flashcart with support for "GBA game linking" (whatever that means).
With Bung Game Doctor in mind, I wonder if it had transformed to saying "you use sound if your doctor is too old".

PypeBros wrote:
EXMEMCNT for EXternal MEMory CoNTrol, presumably.

I guess we all thought so. But now, Nintendo's real intention might have been to use waitstates for overcoming problems with solid or semisolid excrements smeared on the cartridge slot contacts.

Yeah, unaligned sector-buffers may or may not make sense.
The driver where I had spotted it did handle two separate cases, alike "IF (addr & 3) THEN slow_transfer ELSE fast_transfer".
The important points are that driver-makers should support that, and that driver-users shouldn't trust on it being supported (unless somebody can confirm that it is working with most or all drivers).

PypeBros wrote:
The flashcard firmware won't load/scan the whole 128MiB. The NDS memory is only 4MiB and there would be no point in loading more than that in memory.

Maybe not. Or maybe yes. I think some flashcarts are first copying the whole program from SD card to faster internal FLASH, and then load the ARM9/ARM7 bootcode to RAM.
But well, if the patching is done during the SD-to-FLASH transfer... that won't make much of a difference because that transfer is slow anyways.

PypeBros wrote:
I see that the FIX_ALL thing is puzzling you a lot.

Yeah, looks so.
I am pretty fascinated about the bugs in there.
And also about nobody ever having noticed that bugs (which, I guess/hope that's because nobody has ever used the FIX_ALL feature).
And wondering about creating a 100% stable .dldi driver with FIX_ALL is fascinating problem (yet impossible, I think).
Or one could wonder about making a .nds file that supports it's own guessing mechanisms for undoing mis-guessed adjustments made by the installer ; )

But, more serious idea: Making a .nds file that detects FIX_ALL would be nice (and then throwing a warning/error message, saying to post about that dldi driver in this forum).

PypeBros wrote:
I think FIX_ALL might have been added to that to support hypothetical drivers written e.g. in low-level assembly.

Probably so, and based on the assumption that ASM coders won't know how to use PC relative addressing (which isn't neccessarily true, the first thing I had learned about ARM was that B and BL are relative jumps, and that ADR and ADRL are for loading relative addresses).

PypeBros wrote:
"global offset table" (which is a trick C/C++ compiler does to support relocatable code). The technique of relocating code that way is fairly common, and seen in Linux kernel modules and shared libraries.

Do you know more about that GOT (and GLUE) tables?
Are they just pointers, used as-is by the code in the .dldi binary (eg. same way as ASM code with pointers in literal pool)?
Or do they need further pre-processing before starting the .dldi code (eg. like relocation info in a DOS .EXE file)?
In that case, as there's no operating system, that pre-processing would have to be done by libfat/libnds?

PypeBros wrote:
I've never encountered an invalid opcode that could be the responsibility of the DLDI process rather than to some silly loop of mine.

Good to know that it had worked. Then, I guess your driver doesn't use FIX_ALL, right?

PypeBros wrote:
I'm pretty surprised that there is no code advancing to the next word once a word has been patched. After all, having code that just could patch the same 32-bit word twice is a non-sense.

Yup, that would fix most of the problems - and it would even allow to maintain the patch-unaligned-word feature (although I can't image why anybody would need to have that feature in the first place).
It would also require to change FIX_ALL not to patch the 80h-byte .dldi header (especially the Function pointers are in danger because they were already adjusted (before FIX_ALL) and FIX_ALL might then apply further patching to them).

PypeBros wrote:
Of course, if such mis-patching ever occured, it would be caught at the design house

But, with FIX_ALL, only if the header-patching and double-patching were fixed.
As it is now, any double-patched ".dldi" bytes do have the ".nds" +relocationOffset added at time of first patch.
And then, neither the creator of the ".dldi" file, nor creator of ".nds" file could predict what would happen :)
Re: Reverse-engineering DLDI specs for NDS
by on (#228735)
Quote:
PypeBros wrote:
"global offset table" (which is a trick C/C++ compiler does to support relocatable code). The technique of relocating code that way is fairly common, and seen in Linux kernel modules and shared libraries.

Do you know more about that GOT (and GLUE) tables?
Are they just pointers, used as-is by the code in the .dldi binary (eg. same way as ASM code with pointers in literal pool)?

I invite you to have a look at ELF file format, which has the GOT thing described at section 2-17. To be honest, it's been a while since I last read it and I never actually coded around that myself. Just designed my own alternative.

I don't know about the 'glue' thing. It seems specific to ARM7, from the .ld file.

PypeBros wrote:
I've never encountered an invalid opcode that could be the responsibility of the DLDI process rather than to some silly loop of mine.

Good to know that it had worked. Then, I guess your driver doesn't use FIX_ALL, right?
[/quote]
most likely, but since it is embedded into the linker firmware, I must admit i've never seen it in a separated file :-/ (the linker happened to be a clone of a famous one, and not what was pretended by the box).

Oh, and
Quote:
But I guess the HLL people didn't know how to store data at a fixed location : /
Some of the files contain something called "dldi.ld" is it safe to ignore that, or is it important?

The
Code:
ddmem ORIGIN = 0xBF800000

line in that .ld file is what tells the linker to make the code start at the "magic" address bf800000 in first place. It will also allow the programmer to reduce the number of "sections" and sometimes make custom binary formats out of .ELF files. (again, long time not used). This is something you may dig into when creating bootloaders and things like that, and that are blissfully ignored by most applicative programming.
Re: Reverse-engineering DLDI specs for NDS
by on (#228775)
I have downloaded all dldi files from the chishm page, and wrote a tool for scanning some interesting entries.
Code:
Fix Feature ddmem    Size  Filesize CRC32    Filename
----------------------------------------------------------
00 00000000 BF800000 0F 0F 00008000 A12DEDB9 default.dldi
0E 00000013 BF800000 0D 00 0000106C 1B0146E0 acek.dldi
0C 00000013 BF800000 0C 00 00000BB8 A2A189EC cyds.dldi
0E 00000023 BF800000 0D 00 000005C4 435CCCFD gmtf.dldi
0E 00000023 BF800000 0D 00 00000794 2AEF560D gmtf2.dldi
0E 00000023 BF800000 0D 00 000011DC BF1CADC2 dlms.dldi
0E 00000013 BF800000 0D 00 0000125C 61852856 dlms_moon.dldi
0E 00000023 BF800000 0D 00 000013FC 814BFBA4 dlms3.dldi
0E 00000023 BF800000 0D 00 000013A0 AD52070D dlms4.dldi
0E 00000023 BF800000 0D 00 00000574 C4760A67 dsx.dldi
0E 00000013 BF800000 0D 00 00001004 7C3E3952 ewsd.dldi
0E 00000013 BF800000 0D 00 00000B90 115B4FE3 ezsd.dldi
0E 00000023 BF800000 0E 00 00001A44 0D2F707B ez5s.dldi
0E 00000023 BF800000 0E 00 00000E84 CA19D819 ez5sdhc.dldi
0E 00000013 BF800000 0F 00 000023BA 5E7F4D4B g6fl.dldi
0E 00000023 BF800000 0F 00 00002448 1C0DB9AB g6real.dldi
0E 00000013 BF800000 0D 00 0000079C 7F2F05DB m3cf.dldi
0C 00000013 BF800000 0C 00 00000DEC F784899F m3sd.dldi
0C 00000013 BF800000 0C 00 00000B70 E113B19F m3sd_alt.dldi
0C 00000013 BF800000 0B 00 00000754 F60C95F8 mmcf.dldi
0C 00000013 BF800000 0B 00 00000754 B8A8B6F6 mpcf.dldi
0E 00000011 BF800000 0F 00 00000594 8AB819BA mpsd.dldi
0E 00000023 BF800000 0C 00 00000FBC B6BD2DA8 nmmc.dldi
0E 00000021 BF800000 0D 00 000004AC 1C0A7124 mk5n.dldi
0E 00000013 BF800000 0D 00 00000ED0 BB6635E2 nsd2.dldi
0E 00000013 BF800000 0D 00 00001028 5A582569 nsd2v2.dldi
0E 00000023 BF800000 0D 00 00001708 BA8E0A84 njsd.dldi
0E 00000023 BF800000 0F 00 00000868 1F0B6682 x9sd6.dldi
0E 00000023 BF800000 0D 00 000008E4 4500CED0 r4tf.dldi
0E 00000023 BF800000 0D 00 000004E4 487F9E66 r4tf_v2.dldi
0C 00000013 BF800000 0B 00 00000784 C37E0574 sccf.dldi
0E 00000023 BF800000 0D 00 000005FC 284E6CCF scds.dldi
0E 00000023 BF800000 0D 00 000005AC A55FC880 scds1.dldi
0E 00000023 BF800000 0D 00 000005F8 FC50EBF0 scds2.dldi
0E 00000023 BF800000 0D 00 000007AC EA8E72BF scdssdhc.dldi
0E 00000023 BF800000 0D 00 00000548 2B4B9CFC scdssdhc1.dldi
0C 00000013 BF800000 0C 00 00000BB8 D41872EA scsd.dldi
0E 00000013 BF800000 0D 00 000007C0 A10A15A7 scsd_moon.dldi
0E 00000013 BF800000 0D 00 000006A8 F27EC518 sclt.dldi
0E 00000013 BF800000 0F 00 00000950 E4140E7C SCRB.dldi

None of that files is using FIX_ALL.
And ddmem is always at BF800000h.
I think that should make them work stable.
And other/older/later drivers are, hopefully, working similar.

When disassembling the code posted earlier above, I had set a memory access breakpoint on the DLDI area.
It did trigger on reading the FEATURE flags (with GBA/NDS flags), and on the Function table reads (with startup/isInserted).
There was nothing reading the FIX flags or the start/end addresses, so the GOT/GLUE stuff is apparently used only internally by the dldi itself, much like a literal-pools or address-arrays in asm.

Some of the above files don't have the FIX_GLUE stuff.
The other files have the FIX_GLUE flag (but not checked yet if there's anything in the glue start/end area).
Hmmm... if GLUE is specifically for ARM7... no idea if/how/when/why for that's supposed to do.
Re: Reverse-engineering DLDI specs for NDS
by on (#228819)
I have expanded the tool to throw warnings on bad dldi headers & table entries etc.
Most of it looks okay.

The GOT table contains ptrs to the ddmem start/end area (up to including end). Or, for some reason, 00000000h pointers in some cases. Anyways, there are no other dangerous values mixed with the GOT pointers, and the pointers in the existing files are all at word-aligned locations. So I guess GOT is safe to use in that form.

However, the GLUE area is a bit risky: It seems to contain a small code section, containing ARM opcodes mixed with .pool ptrs (and followed by whatever in next three bytes) (eg. in nmmc.dldi). It seems to be only a SMALL code section, and it doesn't include the 80h-byte dldi header area - so it isn't as dangerous as FIX_ALL, but it could go wrong, especially if some driver maker isn't aware of not using opcodes/constants like xxBF80xxh, xxxxBF80h in (or directly after) the GLUE area.
Some of the files with FIX_GLUE don't actually have a GLUE area (or they have it, but it's 0 bytes in size, eg. in mpsd.dldi). Alltogether I've found 9 files that do have GLUE and that do contain opcodes or other non-ddmem pointers in that area.
Re: Reverse-engineering DLDI specs for NDS
by on (#228902)
I've finally had a look at those .ld files, too. For ddmem, there's a line saying "ddmem : ORIGIN = 0xBF800000, LENGTH = 16K", with comment:
Code:
  This base address is chosen to be a reserved instruction in THUMB mode,
  and SWILT in ARM mode, with an impossible SWI value. This should allow
  the patcher / linker to find all shifted addresses within each section.
  Changed sections are: glue_7, got

So the DLDI mastermind was apparently aware of possible conflicts with opcodes, and tried to avoid them using the BF80h value. It just doesn't work as reliable as planned because of the unaligned-word-patching.

Then there's some stuff about GLUE:
Code:
  *(.glue_7)
  *(.glue_7t)

I think that's related to "glueing" ARM code sections to THUMB code sections (with the "t" for THUMB). The nmmc.dldi file has this in GLUE area (for THUMB/ARM mode switching):
Code:
  BF800098 E59FC000  ldr     r12,=0BF800BE1h    ;\
  BF80009C E12FFF1C  bx      r12                ;
  BF8000A0 BF800BE1 .pool       ;<-- patch here ;
                    .thumb                      ; GLUE area
  BF8000A4 4778      bx      r15                ;
  BF8000A6 46C0      nop                        ;
                    .arm                        ;
  BF8000A8 EA000387  b       8000ECCh           ;/
  BF8000AC E92D4010  push    r4,r14             ;-after GLUE (no conflict here)

Not too sure what the "7" in "glue_7" and "glue_7t" is meant for. It doesn't seem to be ARM7 related. Or maybe it's meant as "area with ARM7-compatible opcodes", no matter if that area is used on ARM7 or ARM9.

For the GOT table, one corner case are the NULL pointers (I think they are intended as leading alignment padding). Subtracting 0-BF800000h gives 40800000h, then adding the NDS RAM address 0xxxxxxxh gives 4xxxxxxxh. That's stable even if next byte is BFh (because BF4xh won't match the BF80h patch area). EDIT: The NULL pointers aren't patched because they aren't in the BF80xxxxh range.

However, the other corner case is the GOT table usually being located at the END of the .dldi file, and that won't work: The .dldi file content is just copied to the 32Kbyte DLDI area in the .nds file, and if the .dldi file was smaller than 32Kbytes, then it's just leaving the old .nds file data in the remaining area - and does then try to patch the GOT table, up to including the next 3 bytes after the END of GOT table.
If that goes wrong depends on the RAM address used by the .nds file, the content of the new .dldi file, and the content of the previously used .dldi file(s). Maybe that's rare to get wrong, but if it goes wrong then it's probably hard to figure out what had happened : )
Re: Reverse-engineering DLDI specs for NDS
by on (#229083)
I've asked Robz8 what the BOOT_FC.nds file was for, and got some interesting info:
Robz8 wrote:
The 16KBytes is allocated by the latest libnds.
Also, since TWiLight Menu++ has DSi-extended binaries, I made a different "boot.nds" file to boot it, that being "boot_fc.nds", which does not have DSi-extended header and binaries, and arm9 code starting at 0x200, in order for flashcard firmwares to boot it as homebrew correctly.

As for why I have a DLDI driver in a DSi program, it's for TWiLight Menu++ to work on flashcards as well, and to use a flashcard as secondary storage on both DSi and 3DS consoles.
TWiLight Menu++ has DLDI drivers built-in, so a different one is loaded depending on the flashcard's cardhdr.
I don't think there's an option in libnds to disable/remove DLDI drivers.

I was thinking that 16K were from older libraries, but now it turns out to be a new restriction added in newer libraries (libnds versions since January 2017). Don't know why somebody thought it'd be a good idea to do that, maybe for cases when storing DLDI in ARM7 WRAM? Nice idea - except it won't work with DLDI drivers that require 32K allocated (there's at least one, see below).

Also interesting: Going by robz8's comment, some flashcart firmwares refuse to boot homebrew if it doesn't have a homebrew header. Not sure what that means exactly, theoretically there should be no difference between homebrew & retail headers. I guess the flashcart firmwares might use the header to determine if they should apply DLDI patching, and if they should unlock access to direct SD card sector access?
If somebody knows more about that, please say something!
Ie. which firmwares are doing that, what header entries do they look at, what happens if they don't like the header?

Concerning newer flashcarts, I have downloaded three "acekard2" firmwares here http://old.r4wood.com/pages/Acekard-2i-Kernel.html
ak2_V4.23 - contains only 1 official dldi driver (using ARM code) (ak2_sd.dldi)
AKAIO.1.8.1 - contains 3 homebrew dldi drivers (using THUMB code) (ak2_sd.dldi, rpg_sd.dldi, rpg_nand.dldi)
AKAIO.1.9.0 - contains 3 homebrew dldi drivers (using THUMB code) (ak2_sd.dldi, and the same rpg_xxx drivers as in 1.8.1)
The interesting parts are Function Table entries pointing directly to THUMB code in the homebrew(?) drivers.
And, the "rpg_nand.dldi" is disguising itself as needing only 8Kbyte allocated (in hdr[0Dh]=0Dh), but the space used by the BSS area alone is bigger than 19Kbytes (plus yet MORE space needed for code/data), so this driver won't work with the new 16K libnds versions.

PS. Dunno why the official ak2_V4.23 package contains only 1 dldi driver (maybe it supports the "rpg" hardware variants, too? Or maybe the official drivers for "rpg" hardware got lost somehow?)
Re: Reverse-engineering DLDI specs for NDS
by on (#229224)
Here's the dldiscanning tool for win32 (dldiscan.exe, with source code in dldiscan.asm, plus a collection of .dldi files):
Attachment:
dldiscan.zip [95.09 KiB]
Downloaded 388 times

It's scanning all *.dldi files in current folder, and displays some info, plus WARN:Warnings on uncommon/unreliable things.
And, additionally, it's scanning all *.nds files in current folder, and displays similar info (if the file contains a DLDI header).

You can help: Copy all you .dldi files and all your homebrew .nds files to one folder, then run dldiscan.exe in that folder.
Best redirect the output to a log file (eg. via "dldiscan >temp.txt").
I would be particulary interested in any .dldi files throwing "WARN:CRC=UNKNOWN" (ie. files that aren't in the collection yet).
And about any .nds files that display any warnings (eg. for uncommon alignment or the like).
Also, any .nds files that don't display "#### ARM9" would be interesting (except the .sc.nds ones, that are showing garbage because they are having a bogus stub at the begin of the file; I guess I could just remove the stub to fix that issue).

With the files currently tested, the output looks as so:
Code:
DLDISCAN - DLDI Scanning tool 2018 nocash
Fix Feature ddmem    all  glue got  bss  sizes filesize crc32    name
00 00000000 BF800000 7FF8 0000 0000 0000 0F 0F 00008000 A12DEDB9 DLDI  default.dldi
0E 00000013 BF800000 106C 0020 000C 001C 0D 00 0000106C 1B0146E0 ACEK  acek.dldi
0C 00000013 BF800000 0BB8 0000 0014 0020 0C 00 00000BB8 A2A189EC CYDS  cyds.dldi
0E 00000023 BF800000 05C4 0000 0000 0000 0D 00 000005C4 435CCCFD GMTF  gmtf.dldi
0E 00000023 BF800000 0794 0000 0000 0000 0D 00 00000794 2AEF560D GMTF  gmtf2.dldi
0E 00000023 BF800000 11DC 000C 0000 001C 0D 00 000011DC BF1CADC2 DLMS  dlms.dldi
0E 00000013 BF800000 125C 000C 0000 001C 0D 00 0000125C 61852856 DLMS  dlms_moon.dldi
0E 00000023 BF800000 13FC 000C 000C 021C 0D 00 000013FC 814BFBA4 DLMS  dlms3.dldi
0E 00000023 BF800000 13A0 0000 000C 021C 0D 00 000013A0 AD52070D DLMS  dlms4.dldi
0E 00000023 BF800000 0574 0000 000C 0200 0D 00 00000574 C4760A67 DSX   dsx.dldi
0E 00000013 BF800000 1004 000C 001C 0224 0D 00 00001004 7C3E3952 EWSD  ewsd.dldi
0E 00000013 BF800000 0B90 0000 0010 0020 0D 00 00000B90 115B4FE3 EZSD  ezsd.dldi
0E 00000023 BF800000 1A44 0000 0024 0230 0E 00 00001A44 0D2F707B EZ5S  ez5s.dldi
0E 00000023 BF800000 0E84 0000 0014 0280 0E 00 00000E84 CA19D819 EZ5H  ez5sdhc.dldi
0E 00000013 BF800000 23B8 0000 0024 4044 0F 00 000023BA 5E7F4D4B g6fl  g6fl.dldi
0E 00000023 BF800000 2448 0000 0034 4070 0F 00 00002448 1C0DB9AB g6fl  g6real.dldi
0E 00000013 BF800000 079C 0000 0010 0024 0D 00 0000079C 7F2F05DB M3CF WARN:OVERKILLEND  m3cf.dldi
0C 00000013 BF800000 0DEC 0000 0014 0020 0C 00 00000DEC F784899F M3SD  m3sd.dldi
0C 00000013 BF800000 0B70 0000 0000 001C 0C 00 00000B70 E113B19F M3SD  m3sd_alt.dldi
0C 00000013 BF800000 0754 0000 0000 001C 0B 00 00000754 F60C95F8 MMCF  mmcf.dldi
0C 00000013 BF800000 0754 0000 0000 001C 0B 00 00000754 B8A8B6F6 MPCF  mpcf.dldi
0E 00000011 BF800000 0594 0000 0010 0008 0F 00 00000594 8AB819BA MPSD WARN:OVERKILLEND  mpsd.dldi
0E 00000023 BF800000 0FBC 0014 0014 001C 0C 00 00000FBC B6BD2DA8 NMMC  nmmc.dldi
0E 00000021 BF800000 04AC 000C 0000 001C 0D 00 000004AC 1C0A7124 MK5N  mk5n.dldi
0E 00000013 BF800000 0ED0 0000 000C 0020 0D 00 00000ED0 BB6635E2 nsd2  nsd2.dldi
0E 00000013 BF800000 1028 0000 000C 0004 0D 00 00001028 5A582569 nsd2 WARN:OVERKILLEND  nsd2v2.dldi
0E 00000023 BF800000 1708 0014 0014 0024 0D 00 00001708 BA8E0A84 NJSD  njsd.dldi
0E 00000023 BF800000 0868 0000 000C 1A28 0F 00 00000868 1F0B6682 X9SD WARN:OVERKILLEND  x9sd6.dldi
0E 00000023 BF800000 08E4 0020 0000 001C 0D 00 000008E4 4500CED0 R4TF  r4tf.dldi
0E 00000023 BF800000 04E4 0000 0000 0000 0D 00 000004E4 487F9E66 R4TF  r4tf_v2.dldi
0C 00000013 BF800000 0784 0000 0000 001C 0B 00 00000784 C37E0574 SCCF  sccf.dldi
0E 00000023 BF800000 05FC 0000 0000 001C 0D 00 000005FC 284E6CCF SCDS  scds.dldi
0E 00000023 BF800000 05AC 0000 0000 001C 0D 00 000005AC A55FC880 SCDS  scds1.dldi
0E 00000023 BF800000 05F8 0000 0000 001C 0D 00 000005F8 FC50EBF0 SCDS  scds2.dldi
0E 00000023 BF800000 07AC 0000 0000 001C 0D 00 000007AC EA8E72BF SCDS  scdssdhc.dldi
0E 00000023 BF800000 0548 0000 0000 001C 0D 00 00000548 2B4B9CFC SCDS  scdssdhc1.dldi
0C 00000013 BF800000 0BB8 0000 0014 0020 0C 00 00000BB8 D41872EA SCSD  scsd.dldi
0E 00000013 BF800000 07C0 0000 0000 001C 0D 00 000007C0 A10A15A7 SCSD  scsd_moon.dldi
0E 00000013 BF800000 06A8 0000 0000 001C 0D 00 000006A8 F27EC518 SCLT  sclt.dldi
0E 00000013 BF800000 0950 0000 0000 0000 0F 00 00000950 E4140E7C SCRB  SCRB.dldi
01 00000000 68430000 7FF8 0000 0000 0000 0E 0E 00004000 8D3B9260 KILL WARN:UsesBuggedFixAllWARN:OVERKILLEND  killer.dldi
0E 00000023 BF800000 0ED4 0000 000C 002C 0D 00 00000ED4 C1D6DD19 RPGS  ak2_sd_official.dldi
0E 00000023 BF800000 08F4 0000 000C 0008 0C 00 000008F4 69F95B8C RPGS WARN:THUMB WARN:OVERKILLEND  ak2_sd_akaio190.dldi
0E 00000023 BF800000 0958 0000 000C 0020 0C 00 00000958 4B959C56 RPGS WARN:THUMB  rpg_sd_akaio181-190.dldi
0E 00000023 BF800000 1950 0000 000C 4CA8 0D 00 00001950 6E410A07 RPGN WARN:THUMB WARN:BSS_EXCESS  rpg_nand_akaio181-190.dldi
0E 00000023 BF800000 082C 0000 000C 0004 0C 00 0000082C 42BA42AD RPGS WARN:THUMB WARN:OVERKILLEND  ak2_sd_akaio181.dldi

Fix Feature rambase  all  glue got  bss  sizes filesize rombase  name game area
0E 00000021 0202D440 04AC 000C 0000 001C 0D 0F 002D1640 0002D640 MK5N #### ARM9  GeoWars.nds
00 00000000 02001040 4000 0000 0000 0000 0E 0E 0003D640 00001240 DLDI #### ARM9 WARN:NOALLOC32K BOOT_fc.nds
00 00000000 0201BE40 8000 0000 0000 0000 0F 0F 00078840 0001C040 DLDI #### ARM9  ndsmail.nds
00 00000000 02034C80 8000 0000 0000 0000 0F 0F 000CE800 00034E80 DLDI #### ARM9  lmp-ng-1.02.nds
00 00000000 02016240 8000 0000 0000 0000 0F 0F 00051240 00016440 DLDI #### ARM9  000SNEmulDS.nds
00 00000000 02037500 8000 0000 0000 0000 0F 0F 000AC040 00037700 DLDI #### ARM9  ComicBookDS.nds
00 00000000 02037500 8000 0000 0000 0000 0F 0F 000AC240 00037900 DLDI  -??-  ComicBookDS.sc.nds
00 00000000 020301C0 8000 0000 0000 0000 0F 0F 00088440 000303C0 DLDI #### ARM9  pokeyDS.nds
0E 00000013 0202D440 07C0 0000 0000 001C 0D 0F 002D1840 0002D840 SCSD  -??-  GeoWars.sc.nds
00 00000000 02006300 8000 0000 0000 0000 0F 0F 000BCE40 00006500 DLDI #### ARM9  MissileCommand.nds
0C 00000013 02018900 0754 0000 0000 001C 0B 0F 00058E00 00018B00 MMCF #### ARM9  ds81.nds
00 00000000 02024B80 8000 0000 0000 0000 0F 0F 00100840 00024D80 DLDI #### ARM9  GameUP.nds
0E 00000023 02015C40 05C4 0000 0000 0000 0D 0F 00136640 00015E40 GMTF #### ARM9  011-Slots.nds
00 00000000 02013000 8000 0000 0000 0000 0F 0F 00068240 00013200 DLDI #### ARM9  Weckermp3.nds
00 00000000 0200B040 8000 0000 0000 0000 0F 0F 0004A600 0000B240 DLDI #### ARM9  Flickbook_v0.3_(Preview)_(Nintendo_DS).nds
00 00000000 02022780 8000 0000 0000 0000 0F 0F 002E9640 00022980 DLDI #### ARM9  TowerDefense.nds
00 00000000 02022780 8000 0000 0000 0000 0F 0F 002E9840 00022B80 DLDI  -??-  TowerDefense.sc.nds
00 00000000 02011640 8000 0000 0000 0000 0F 0F 00166040 00011840 DLDI #### ARM9  Xmas_gifts.nds
00 00000000 02056880 8000 0000 0000 0000 0F 0F 000BB640 00056A80 DLDI #### ARM9  dsdoom.nds
00 00000000 02056880 8000 0000 0000 0000 0F 0F 000BB640 00056A80 DLDI #### ARM9  dsdoom121.nds
00 00000000 02055380 8000 0000 0000 0000 0F 0F 000A2640 00055580 DLDI #### ARM9  dsdoom112.nds
00 00000000 02055D40 8000 0000 0000 0000 0F 0F 000B7640 00055F40 DLDI #### ARM9  dsdoom120.nds
00 00000000 02054600 8000 0000 0000 0000 0F 0F 000A0040 00054800 DLDI #### ARM9  dsdoom111.nds
Re: Reverse-engineering DLDI specs for NDS
by on (#239292)
oerg866 in Plutiedev (a Sega Genesis-related Discord server) told me there was a build of QtWeb here that works with KernelEx on Windows 98. It used about 46 MB of RAM in tests, and it indeed connects to GitHub (and thus supports TLS 1.2).
https://code.google.com/archive/p/qtweb/issues/182#c2
Re: Reverse-engineering DLDI specs for NDS
by on (#241881)
Since I made Pokémon Emerald to boot and load saves in GBARunner2 (https://bitbucket.org/Coto88/gbarunner2/src/master/) (not saving yet as I need to re-implement the Flash commands), I think this DLDI trick made it possible:

https://bitbucket.org/Coto88/gbarunner2 ... p#lines-56

Which allows us to fetch the DLDI on EWRAM that was given & relocated to us by the cart loader. So I do the same, right after the DLDI code was set up... but to relocate it to either VRAM (DLDI ARM9) or to relocate it to IWRAM (DLDI ARM7)... kinda awesome to discover tricks on that NintendoDS thing... ; - )
Re: Reverse-engineering DLDI specs for NDS
by on (#241883)
I don't get it, sorry. What is the goal of the trick? What is EWRAM? And is that trick GBA, or NDS, or DSi related?
Re: Reverse-engineering DLDI specs for NDS
by on (#241886)
GBA has three main RAMs. From fastest to slowest:

  1. IWRAM (beginning at 03000000h) is next to the CPU and is 32-bit, 0 wait state, and 32 KiB
  2. VRAM (beginning at 06000000h) is next to the PPU and is 16-bit, 0 (or 1?) wait state, and 96 KiB
  3. EWRAM (beginning at 02000000h) is outside the SoC and is 16-bit, 2 wait states, and 256 KiB

These correspond to what GBATEK refers to as "WRAM - On-chip Work RAM", "VRAM - Video RAM", and "WRAM - On-board Work RAM" respectively. "I" is for internal, or on-chip; "E" is for external, or on-board.

(I remember there being some conditions that caused VRAM to have wait states, but I forget exactly what.)
Re: Reverse-engineering DLDI specs for NDS
by on (#241895)
nocash wrote:
I don't get it, sorry.

No hay problema.

nocash wrote:
What is the goal of the trick?


DLDI code is relocatable. That is done by the flashcard loader (if it features auto-DLDI patching, then the DLDI section is relative and relocated depending on the @ data start and @ data end variables), but I think you already knew that. This means the DLDI could be patched anywhere near, say, the start 0x0200xxxx EWRAM (NDS External 4MB WRAM), and then, the upcoming NDS homebrew, which has the DLDI section, might have it near the EWRAM end: 0x023xxxxx.
In this case the DLDI driver must be relative to the new address. (otherwise it cause segfaults).

nocash wrote:
What is EWRAM? And is that trick GBA, or NDS, or DSi related?


EWRAM (NDS External 4MB WRAM), of which belongs to hypervisor code (which requires the DLDI driver so the flashcard works in hypervisor environment), thus, that trick belongs to NDS category. (and DSi as well)

-

From now on, I mean NTR (NintendoDS) entirely for really basic hypervisor explanation.

And what I mean, is that in order to get GBARunner2's DSi compatibility, the DLDI code must work from ARM7. If you are lucky and you try a fixated relative address, the DLDI code might run just fine in there, but, say, if you want to run the DLDI code from... say ARM9 VRAM, you must relocate the code so it actually executes from there. (otherwise data aborts/prefetch aborts, etc will happen).

When GBARunner2 is recompiled to run from DLDI @ ARM9, DLDI code is relocated to VRAM since it is fast, and code can execute from there, also the MPU does not get in the way when dealing with hypervisor side data-aborts (as opposed as running code from EWRAM).

When GBARunner2 is recompiled to run from DLDI @ ARM7, DLDI code is relocated to IWRAM (64K ARM7 Internal Work RAM) so the DLDI layer can run each SD routine natively... from IWRAM, ARM7, through FIFO interrupts (ARM7 <-> ARM9 communications).

How does this trick help? Well, in GBARunner2, actually DLDI at ARM9 code works in some SD cards (and not SDHC), either if I relocate the DLDI to EWRAM or VRAM. Actually it is mandatory in the DLDI specs to put the DLDI stub (if compiling a vanilla NDS homebrew, from sources into binary format), in EWRAM. Because flashcard loaders will look in there and perform the DLDI relocation... as I said before. On the other hand, because of timings and the MPU, the DLDI code must run in fast memory, and MPU-free.. of which makes VRAM the perfect candidate memory. And that break SDHC compatibility because the flashcard loader does weird things and SD is not initialized properly. (I can safely say, based on ToolchainGenericDS development experience, the DLDI code is best suited to run from EWRAM as in there I get flawless SD/SDHC card compatibility in every TGDS project... such as the FTP Server).

And relocating DLDI at ARM7 (something I definitely wanted to have 100% SD/SDHC card compatibility) was a must.


So to sum it up:

1) I take the DLDI that was given to me by the loader, it lives in ARM9's EWRAM.
2) Relocate it to ARM7's IWRAM region
3) Init DLDI code at ARM7
4) run DLDI commands through the struct DLDI_INTERFACE and hook it up to fatfs so we can deal with file handlers (such as POSIX FILE * file handle) in the hypervisor code, instead of hardcoding sectors. In the middle there is the FIFO Interrupts, because DLDI <-> fatfs runs between ARM7 <-> ARM9 FIFO commands.
5) the "sector paging" buffer required by GBARunner2 to actually stream a GBA File from SD Card, through the DLDI protocol, actually lives in the NDS VRAM C block... which only accepts 16 or 32 bit read/writes, and that one is not available from ARM7... unless the VRAM block is set to ARM7. A much better approach is to allocate a chunk of EWRAM memory and let the DLDI code to read/write sectors through it, and then the ARM7/ARM9 can work with it (albeit on the ARM9 side the cache gets in the way, so data invalidation must take place so proper code is retrieved/stored instead of garbage.
Re: Reverse-engineering DLDI specs for NDS
by on (#241898)
coto wrote:
Actually it is mandatory in the DLDI specs to put the DLDI stub (...), in EWRAM. Because flashcard loaders will look in there and perform the DLDI relocation...
Are you sure? My theory was that the loader looks at ARM9 and ARM7 bootcode sections, and the latter can be at 37F8000h..3807E00h, so that should (theoretically) work without problems... unless some loaders don't support 37F8000h..3807E00h?

coto wrote:
When GBARunner2 is recompiled to run from DLDI @ ARM9, DLDI code is relocated to VRAM
When GBARunner2 is recompiled to run from DLDI @ ARM7, DLDI code is relocated to IWRAM
Is that done at compilation time (on the PC, to make one .nds file that uses VRAM, and another .nds file that uses other memory), or does your binary have the capability to recompile itself (so the same .nds file can move the DLDI driver to different locations)?
In the latter case, then, yes, that would require some trickery.

Mmmmh, I didn't try installing KernelEx for gaining access to now modern internet, so no sites like bitbucket or github for me. I don't know if your source code is already recursing this...

The important thing about "relocations" is that it consists of two separate steps:
- adjusting absolute addresses to match up with the target address (unreliable because of DLDI's address-guessing technique)
- copying the code to the target address (stable)
That are separate things, and they can be considered to be done separately, and they can use different target addresses (since the copy address is implied in the cart header, and the adjust address is in the dldi header).
Most or all dldi files contain absolute addresses in the BF80xxxxh address range.

The normal loading procedure would be:
- adjust addresses from BF80xxxxh to 0200xxxh (done by the cart loader or dldi patching tool) (unreliable)
- copy code to 0200xxxxh (done by the cart loader) (stable)

For running in VRAM, the cartloader can't load to that area directly, so you would need "double relocation". But doing it this way...
- adjust addresses from BF80xxxxh to 0200xxxh (done by the cart loader or dldi patching tool) (unreliable)
- copy code to 0200xxxxh (done by the cart loader) (stable)
- copy code from 0200xxxxh to 0600xxxxh in VRAM (by your own code) (stable)
- adjust addresses from 0200xxxxh to 0600xxxxh in VRAM (by your own code) (very unreliable)
that would be unwise because the unreliable address guessing is done twice, increasing chances that things will go wrong. 0200h is also a much more common value than BF80h, so the chances to get things wrong are much higher than usually.

For a fixed load address (eg. VRAM), that does just work about as normal, without any special trickery:
- adjust addresses from BF80xxxxh to 0600xxxh (done by the cart loader or dldi patching tool) (unreliable)
- copy code to 0200xxxxh (done by the cart loader) (stable)
- copy code from 0200xxxxh to 0600xxxxh in VRAM (by your own code) (stable)

For variable load address (eg. allowing to use VRAM or WRAM depending on whatever options):
- adjust addresses from BF80xxxxh to BF80xxxh (done by the cart loader or dldi patching tool) (no change = stable)
- copy code to 0200xxxxh (done by the cart loader) (stable)
- copy code from 0200xxxxh to VRAM or whatever (by your own code) (stable)
- adjust addresses from BF80xxxxh to VRAM or whatever (by your own code) (unreliable)

Or, well, if your source code was already doing that... sorry for writing all that crap.

PS.
the stuff from this thread is meanwhile also summarized in two gbatek chapters:
http://problemkaputt.de/gbatek.htm#dscartdldidriver - dldi format
http://problemkaputt.de/gbatek.htm#dsca ... djustments - dldi problems and workarounds
Re: Reverse-engineering DLDI specs for NDS
by on (#241901)
nocash wrote:
Are you sure? My theory was that the loader looks at ARM9 and ARM7 bootcode sections, and the latter can be at 37F8000h..3807E00h, so that should (theoretically) work without problems... unless some loaders don't support 37F8000h..3807E00h?


1#

I guess that implementation depends on the flashcard vendor so generalizing the implementation would be a mistake (means the ARM9 bootcode's DLDI section is found and gets patched). But usually the "Open Source way" to do it, is to use a DLDI loader(the one that bootstubs a file from an embedded libfat-like small implementation, which requires a cluster start), find the DLDI, if any, in memory source (where @data_start matters, and @data_end doesn't because DLDI size would be @data_end), where EWRAM is in this case, and then put and relocate the DLDI in that loader according to DLDI header sections, then a specific start cluster is added. So then the loader can be ran, effectively hooking a NDS binary + the new DLDI.
This is the Internal Relocation Offset.

Subsequently, another DLDI patcher comes in (such as the one I use), and will read @data_start and @data_end and relocate the DLDI physically.
This allows the DLDI to be runnable from any memory it should live in. This is the Relocation Offset.

(A mistery for me is: why some DLDI code startup break on certain SDHC cards when DLDI section was relocated in (VRAM)0x068xxxxx and not in (EWRAM)0x02xxxxxx), at least in GBARunner2. And that's why I said ...

Quote:
Actually it is mandatory in the DLDI specs to put the DLDI stub (...), in EWRAM. Because flashcard loaders will look in there and perform the DLDI relocation...


*properly*. No matter if SD/SDHC

nocash wrote:
Is that done at compilation time (on the PC, to make one .nds file that uses VRAM, and another .nds file that uses other memory), or does your binary have the capability to recompile itself (so the same .nds file can move the DLDI driver to different locations)?
In the latter case, then, yes, that would require some trickery.


2#

I was just talking about re-compiling / re-building the homebrew through the usual Toolchain depending on the DLDI setting:
DLDI @ ARM9
or
DLDI @ ARM7

But I actually meant the latter, albeit modified to *NOT* to patch the dldi into a loader, but instead, just to relocate the DLDI (regardless of the DLDI setting):

- If DLDI @ ARM9: Then the DLDI section is retrieved from EWRAM and is relocated (sorta what 1# does) onto ARM9's VRAM 0x06800000 (first 32K).

- Else If DLDI @ ARM7: Then the DLDI section is retrieved from EWRAM and is relocated (sorta what 1# does) onto ARM7's IWRAM 0x03800000 (first 32K).



For instance, if I don't relocate (from EWRAM's DLDI source) the DLDI @ ARM9 onto ARM9's VRAM 0x06800000, and otherwise, I just copy -> paste the DLDI in there, I get segfaults just by executing it from that address. Even when updating the DLDI headers.
Re: Reverse-engineering DLDI specs for NDS
by on (#241905)
coto wrote:
and otherwise, I just copy -> paste the DLDI in there, I get segfaults just by executing it from that address. Even when updating the DLDI headers.

Yes, do that, simply memcopy the block from 0200xxxxh to 0680xxxxh, without any address adjustments.
There should be no segfaults if you have correctly specified 0680xxxxh as dldi target address in the "empty dldi area" at time when creating the .nds file.
Re: Reverse-engineering DLDI specs for NDS
by on (#241911)
nocash wrote:
coto wrote:
and otherwise, I just copy -> paste the DLDI in there, I get segfaults just by executing it from that address. Even when updating the DLDI headers.

Yes, do that, simply memcopy the block from 0200xxxxh to 0680xxxxh, without any address adjustments.
There should be no segfaults if you have correctly specified 0680xxxxh as dldi target address in the "empty dldi area" at time when creating the .nds file.


"empty dldi area" ????????????????????????? wtf???

-

Anyway, since I can demonstrate what I am saying (i'm not lying, and I can prove it):

Sources: https://bitbucket.org/Coto88/gbarunner2/src/master/

I enabled DLDI @ ARM9, recompiled GBARunner2, and just dma copied the DLDI from EWRAM -> VRAM (because i cannot use memcpy, as it uses 8 bit writes to VRAM in NintendoDS, which fails). And I get undefined aborts.

Steps to reproduce that:

dldigba.c -> replace this function:

(this one does the relocation perfectly)
Quote:
#ifdef ARM9
PUT_IN_VRAM
#endif
bool dldiPatchLoader(bool clearBSS, u32 DldiRelocatedAddress, u32 dldiSourceInRam)
{
addr_t memOffset; // Offset of DLDI after the file is loaded into memory
addr_t patchOffset; // Position of patch destination in the file
addr_t relocationOffset; // Value added to all offsets within the patch to fix it properly
addr_t ddmemOffset; // Original offset used in the DLDI file
addr_t ddmemStart; // Start of range that offsets can be in the DLDI file
addr_t ddmemEnd; // End of range that offsets can be in the DLDI file
addr_t ddmemSize; // Size of range that offsets can be in the DLDI file

addr_t addrIter;

data_t *pDH;
data_t *pAH;

size_t dldiFileSize = 0;

// Target the DLDI we want to use as stub copy and then relocate it to a DldiRelocatedAddress address
DLDI_INTERFACE* dldiInterface = (DLDI_INTERFACE*)DldiRelocatedAddress;
pDH = (data_t*)dldiInterface;
pAH = (data_t *)dldiSourceInRam;

dldiFileSize = 1 << pAH[DO_driverSize];

// Copy the DLDI patch into the application
dmaCopyWords(0, (void*)pAH, (void*)pDH, dldiFileSize);

if (*((u32*)(pDH + DO_ioType)) == DEVICE_TYPE_DLDI) {
// No DLDI patch
return false;
}

if (pDH[DO_driverSize] > pAH[DO_allocatedSpace]) {
// Not enough space for patch
return false;
}


memOffset = DldiRelocatedAddress; //readAddr (pAH, DO_text_start);
if (memOffset == 0) {
memOffset = readAddr (pAH, DO_startup) - DO_code;
}
ddmemOffset = readAddr (pDH, DO_text_start);
relocationOffset = memOffset - ddmemOffset;

ddmemStart = readAddr (pDH, DO_text_start);
ddmemSize = (1 << pDH[DO_driverSize]);
ddmemEnd = ddmemStart + ddmemSize;

// Remember how much space is actually reserved
pDH[DO_allocatedSpace] = pAH[DO_allocatedSpace];


// Fix the section pointers in the DLDI @ VRAM header
writeAddr (pDH, DO_text_start, readAddr (pAH, DO_text_start) + relocationOffset);
writeAddr (pDH, DO_data_end, readAddr (pAH, DO_data_end) + relocationOffset);
writeAddr (pDH, DO_glue_start, readAddr (pAH, DO_glue_start) + relocationOffset);
writeAddr (pDH, DO_glue_end, readAddr (pAH, DO_glue_end) + relocationOffset);
writeAddr (pDH, DO_got_start, readAddr (pAH, DO_got_start) + relocationOffset);
writeAddr (pDH, DO_got_end, readAddr (pAH, DO_got_end) + relocationOffset);
writeAddr (pDH, DO_bss_start, readAddr (pAH, DO_bss_start) + relocationOffset);
writeAddr (pDH, DO_bss_end, readAddr (pAH, DO_bss_end) + relocationOffset);

// Fix the function pointers in the header
writeAddr (pDH, DO_startup, readAddr (pAH, DO_startup) + relocationOffset);
writeAddr (pDH, DO_isInserted, readAddr (pAH, DO_isInserted) + relocationOffset);
writeAddr (pDH, DO_readSectors, readAddr (pAH, DO_readSectors) + relocationOffset);
writeAddr (pDH, DO_writeSectors, readAddr (pAH, DO_writeSectors) + relocationOffset);
writeAddr (pDH, DO_clearStatus, readAddr (pAH, DO_clearStatus) + relocationOffset);
writeAddr (pDH, DO_shutdown, readAddr (pAH, DO_shutdown) + relocationOffset);

if (pDH[DO_fixSections] & FIX_ALL) {
// Search through and fix pointers within the data section of the file
for (addrIter = (readAddr(pDH, DO_text_start) - ddmemStart); addrIter < (readAddr(pDH, DO_data_end) - ddmemStart); addrIter++) {
if ((ddmemStart <= readAddr(pAH, addrIter)) && (readAddr(pAH, addrIter) < ddmemEnd)) {
writeAddr (pAH, addrIter, readAddr(pAH, addrIter) + relocationOffset);
}
}
}


if (pDH[DO_fixSections] & FIX_GLUE) {
// Search through and fix pointers within the glue section of the file
for (addrIter = (readAddr(pDH, DO_glue_start) - ddmemStart); addrIter < (readAddr(pDH, DO_glue_end) - ddmemStart); addrIter++) {
if ((ddmemStart <= readAddr(pAH, addrIter)) && (readAddr(pAH, addrIter) < ddmemEnd)) {
writeAddr (pAH, addrIter, readAddr(pAH, addrIter) + relocationOffset);
}
}
}

if (pDH[DO_fixSections] & FIX_GOT) {
// Search through and fix pointers within the Global Offset Table section of the file
for (addrIter = (readAddr(pDH, DO_got_start) - ddmemStart); addrIter < (readAddr(pDH, DO_got_end) - ddmemStart); addrIter++) {
if ((ddmemStart <= readAddr(pAH, addrIter)) && (readAddr(pAH, addrIter) < ddmemEnd)) {
writeAddr (pAH, addrIter, readAddr(pAH, addrIter) + relocationOffset);
}
}
}

/*
if (clearBSS && (pDH[DO_fixSections] & FIX_BSS)) {
// Initialise the BSS to 0, only if the disc is being re-inited
for(int i = 0; i < (readAddr(pDH, DO_bss_end) - readAddr(pDH, DO_bss_start)) / 4; i++)
{
((uint32_t*)&pAH[readAddr(pDH, DO_bss_start) - ddmemStart])[i] = 0;
}

}
*/
return true;
}


And instead replace it with:


(just DMA copy DLDI from EWRAM -> VRAM)
Quote:
#ifdef ARM9
PUT_IN_VRAM
#endif
bool dldiPatchLoader(bool clearBSS, u32 DldiRelocatedAddress, u32 dldiSourceInRam)
{
addr_t memOffset; // Offset of DLDI after the file is loaded into memory
addr_t patchOffset; // Position of patch destination in the file
addr_t relocationOffset; // Value added to all offsets within the patch to fix it properly
addr_t ddmemOffset; // Original offset used in the DLDI file
addr_t ddmemStart; // Start of range that offsets can be in the DLDI file
addr_t ddmemEnd; // End of range that offsets can be in the DLDI file
addr_t ddmemSize; // Size of range that offsets can be in the DLDI file

addr_t addrIter;

data_t *pDH;
data_t *pAH;

size_t dldiFileSize = 0;

// Target the DLDI we want to use as stub copy and then relocate it to a DldiRelocatedAddress address
DLDI_INTERFACE* dldiInterface = (DLDI_INTERFACE*)DldiRelocatedAddress;
pDH = (data_t*)dldiInterface;
pAH = (data_t *)dldiSourceInRam;

dldiFileSize = 1 << pAH[DO_driverSize];

// Copy the DLDI patch into the application
dmaCopyWords(0, (void*)pAH, (void*)pDH, dldiFileSize);

/*
if (*((u32*)(pDH + DO_ioType)) == DEVICE_TYPE_DLDI) {
// No DLDI patch
return false;
}

if (pDH[DO_driverSize] > pAH[DO_allocatedSpace]) {
// Not enough space for patch
return false;
}


memOffset = DldiRelocatedAddress; //readAddr (pAH, DO_text_start);
if (memOffset == 0) {
memOffset = readAddr (pAH, DO_startup) - DO_code;
}
ddmemOffset = readAddr (pDH, DO_text_start);
relocationOffset = memOffset - ddmemOffset;

ddmemStart = readAddr (pDH, DO_text_start);
ddmemSize = (1 << pDH[DO_driverSize]);
ddmemEnd = ddmemStart + ddmemSize;

// Remember how much space is actually reserved
pDH[DO_allocatedSpace] = pAH[DO_allocatedSpace];


// Fix the section pointers in the DLDI @ VRAM header
writeAddr (pDH, DO_text_start, readAddr (pAH, DO_text_start) + relocationOffset);
writeAddr (pDH, DO_data_end, readAddr (pAH, DO_data_end) + relocationOffset);
writeAddr (pDH, DO_glue_start, readAddr (pAH, DO_glue_start) + relocationOffset);
writeAddr (pDH, DO_glue_end, readAddr (pAH, DO_glue_end) + relocationOffset);
writeAddr (pDH, DO_got_start, readAddr (pAH, DO_got_start) + relocationOffset);
writeAddr (pDH, DO_got_end, readAddr (pAH, DO_got_end) + relocationOffset);
writeAddr (pDH, DO_bss_start, readAddr (pAH, DO_bss_start) + relocationOffset);
writeAddr (pDH, DO_bss_end, readAddr (pAH, DO_bss_end) + relocationOffset);

// Fix the function pointers in the header
writeAddr (pDH, DO_startup, readAddr (pAH, DO_startup) + relocationOffset);
writeAddr (pDH, DO_isInserted, readAddr (pAH, DO_isInserted) + relocationOffset);
writeAddr (pDH, DO_readSectors, readAddr (pAH, DO_readSectors) + relocationOffset);
writeAddr (pDH, DO_writeSectors, readAddr (pAH, DO_writeSectors) + relocationOffset);
writeAddr (pDH, DO_clearStatus, readAddr (pAH, DO_clearStatus) + relocationOffset);
writeAddr (pDH, DO_shutdown, readAddr (pAH, DO_shutdown) + relocationOffset);

if (pDH[DO_fixSections] & FIX_ALL) {
// Search through and fix pointers within the data section of the file
for (addrIter = (readAddr(pDH, DO_text_start) - ddmemStart); addrIter < (readAddr(pDH, DO_data_end) - ddmemStart); addrIter++) {
if ((ddmemStart <= readAddr(pAH, addrIter)) && (readAddr(pAH, addrIter) < ddmemEnd)) {
writeAddr (pAH, addrIter, readAddr(pAH, addrIter) + relocationOffset);
}
}
}


if (pDH[DO_fixSections] & FIX_GLUE) {
// Search through and fix pointers within the glue section of the file
for (addrIter = (readAddr(pDH, DO_glue_start) - ddmemStart); addrIter < (readAddr(pDH, DO_glue_end) - ddmemStart); addrIter++) {
if ((ddmemStart <= readAddr(pAH, addrIter)) && (readAddr(pAH, addrIter) < ddmemEnd)) {
writeAddr (pAH, addrIter, readAddr(pAH, addrIter) + relocationOffset);
}
}
}

if (pDH[DO_fixSections] & FIX_GOT) {
// Search through and fix pointers within the Global Offset Table section of the file
for (addrIter = (readAddr(pDH, DO_got_start) - ddmemStart); addrIter < (readAddr(pDH, DO_got_end) - ddmemStart); addrIter++) {
if ((ddmemStart <= readAddr(pAH, addrIter)) && (readAddr(pAH, addrIter) < ddmemEnd)) {
writeAddr (pAH, addrIter, readAddr(pAH, addrIter) + relocationOffset);
}
}
}
*/

/*
if (clearBSS && (pDH[DO_fixSections] & FIX_BSS)) {
// Initialise the BSS to 0, only if the disc is being re-inited
for(int i = 0; i < (readAddr(pDH, DO_bss_end) - readAddr(pDH, DO_bss_start)) / 4; i++)
{
((uint32_t*)&pAH[readAddr(pDH, DO_bss_start) - ddmemStart])[i] = 0;
}

}
*/
return true;
}


shared.h -> disable DLDI @ ARM7 (makes DLDI @ ARM9):

Quote:
#ifndef __SHARED_H__
#define __SHARED_H__

//DLDI ARM7 Support (requires to recompile the project)
//#define ARM7_DLDI

#define SOUND_BUFFER_SIZE (8192)
#define SAVE_DATA_SIZE (0x20000) //128K SRAM/EEprom/Flash 8bit/16bit/32bit write compatible RAM memory.
#define ROM_DATA_LENGTH (0x3A0000 - (32*1024) ) //0x400000 - 0x40000 (hypervisor) - 0x20000 (128K) = 0x3A0000
#define ROM_ADDRESS_MAX (0x08000000 + ROM_DATA_LENGTH)
#define MAIN_MEMORY_ADDRESS_SAVE_DATA (0x02400000 - SAVE_DATA_SIZE) //-> (0x023E0000 ~ 0x023FFFFF) -> mirror: 0x01FE0000
#define MAIN_MEMORY_ADDRESS_SDCACHE (MAIN_MEMORY_ADDRESS_SAVE_DATA - (32*1024)) //-> (0x23D8000): -- used for shared DLDI sector memory between ARM7/ARM9

#define SOUND_EMU_QUEUE_LEN 64
#define address_dtcm (0x02C00000) //@0x04F00000 @0x01800000
#define MAIN_MEMORY_ADDRESS_ROM_DATA (0x02040000)

//VRAM Layout
#define sd_cluster_cache_addr (0x06840000)
#define sd_access_driver (0x06860000)

#define FIFO_CNT_EMPTY (1 << 8)
#define REG_FIFO_CNT (*((vu32*)0x04000184))
#define REG_SEND_FIFO (*((vu32*)0x04000188))
#define REG_RECV_FIFO (*((vu32*)0x04100000))

#ifdef __ASSEMBLER__

sd_cluster_cache = sd_cluster_cache_addr
sd_data_base = sd_access_driver
sd_is_cluster_cached_table = (sd_data_base + 32768 + (64*1024) ) @(sd_data_base + (224 * 1024))
sd_cluster_cache_info = (sd_is_cluster_cached_table + (16 * 1024))
sd_sd_info = (sd_cluster_cache_info + (256 * 8 + 4)) @0x0685C404
pu_data_permissions = 0x33600603 @0x33600003 @0x33660003

#endif

#ifndef __ASSEMBLER__

#ifdef ARM9
#define PUT_IN_VRAM __attribute__((section(".vram")))
#define ITCM_CODE __attribute__((section(".itcm"), long_call))
#endif

#define PACKED __attribute__ ((packed))

#endif //__ASSEMBLER__

#endif



now re-build / recompile GBARunner2.

You'll get a build that causes undefined exceptions (registers scope within the original DLDI address, but while the Program Counter is in VRAM).

Thus, I cannot simply copy (or dmacopy) the DLDI from EWRAM to VRAM. Which is why I am actually replying here and calling it "a trick".
Re: Reverse-engineering DLDI specs for NDS
by on (#241921)
The "empty dldi area" is the memory area in your .nds file that is reserved for the flashcart driver. If the driver hasn't been installed yet, then it is initially "empty", or "almost empty" because there are few non-empty bytes in there: The dldi ID bytes/string with EDh,A5h,8Dh,BFh,20h,"Chishm",00h at offset 00h, the size byte at offset 0Fh, and the load address at offset 40h.

The important value here is the load address at offset 40h, I assume you (or your devkit) have configured that value to 02xxxxxxh, and that is wrong because your code will actually load it to VRAM, so the correct value would be 068xx000h (or wherever you have it in VRAM).

If you figure out how to change that value then you won't manually need to adjust the FIX_GLUE and FIX_GOT stuff, and that will make things a good bit less unreliable, and that will probably fix mysterious issues like problems with SD vs SDHC... unless that issue was caused by missing 8bit write support in VRAM.