Question on RLE

My RLE idea is to store 2x2 meta tiles for my backgrounds. The idea I came up with is to have the left nibble
be the repeat number and the right nibble to be the tile number. Instead of the right nibble being 0 - F, it would
be 0-9. A-F in the right nibble would be a bank switch for the 10 tileset.



Of course, I will probably want to draw vertical columns because it will be a side scroller.

The reason for this post is that I'm not sure if I am taking the correct approach here. I have seen links to packBits
elsewhere, but I haven't tested yet.

The goal is to get the most compressed nametables possible because I am running NROM for now.

Thanks,

Using 4 bits for the number of tiles and 4 bits for the tile number isn't that new, Zelda 2 and Dragon Warrior were doing exactly that. But they did it horizontally, not vertically.
But using A-F as tile numbers to select a bank is a waste of encoding space, since the length isn't used anymore.

Suggestions:
0-D = tiles
E = repeat whatever was in the previous column, use the length# for the number of tiles to take.
F = switch bank of 14 tiles, use length as the bank number instead of a length.

Or throw out vertical RLE entirely, and use reusable big blocks of metatiles, such as 4x4 metatiles, or 8x8 metatiles.

You can use the top 2 bits to identify a bank for a tile since they won't be used and then pull the attributes from a table in said bank. Or sacrafice the first few metatiles as operators for the bank changes, so 0-3 would be bank switches and then 4 to 255 would be the attribute-4. I myself would either use the top 2 bits as the attribute and let the LEVEL data contain the bank used for the level of metatiles. If that was not an option, I'd probably choose the "pull attribute from bank" and let the top 2 bits signify the bank.

And as for horizontal compression, I'd say a dictionary-based scheme (With RLE encoded entries?) would provide best size/level size ratio. If you keep standard metatiles (for floor, blank, etc.) you can even re-use lots of the strips on the screen. Just a few ideas, hope they help.

You will certainly not get that with RLE. Throw it a simple checkerboard pattern and RLE won't compress a single bit, it will probably expand the data, actually. If you want better compression you must have a way to repeat patterns longer than 1 symbol.

You should check my CompressTools, which have about 7-8 different compression schemes available. By default it compresses data using all the algorithms, and display info so you can see the ones which works the best for your data.

What you describe is not new, it is what I call "Bitpack RLE". What you describe is a particular case where the data is 4 bit and run length is 4 bit, but the same concept can apply from 1 bit of data + 7 bit of run lenght to 7 bit of data + 1 bit of run length (in this case it's either a single byte or a run of 2).