I wanted to share something I've created that may be potentially marginally useful to some people. I encountered a lot of frustration while originally searching for tools to help with creating SNES sprites, and though I spent a good full day just trying every bit of software I could find nothing seemed to be good for simply editing tiles and converting between a visual and assembly format. So I made a spreadsheet and some macros, which seems to be my solution to everything these days.
Notable features include the ability to import from and export to 16-colour bitmaps (up to 128px wide x 32px tall though could be expanded), exporting as a ".inc" text file of bytes, converting cell colours to two-byte hex values, switching between different palettes and exchanging colours. And honestly, I just find it so much easier to edit in spreadsheet form than any software I've ever used.
The only setup required is to tell it where to save files to. The root path is a global variable at the top of the visual basic file.
If you think it's useful, feel free to suggest other features that could be included. If you think it's redundant then I'm open to suggestions of what to use instead.
EDIT: There have been SEVERAL updates since this first post, I recommend "V2" on this page over the file attached in this post, and the Tiled translation file has been attached here as well
I don't have Excel on my computer so I can't look at it, but what do you mean? does each box (don't remember what they are called, if they even have a name) represent a pixel?
Yup! Each cell is a pixel and you just type in the colour number in hex for it (0-F).
Quote:
SNESImageMacros.xlsm
Image macros, eh? I don't have Excel on my machine either. Does it support Impact? :p
Quote:
HAPPINES IN SLAVERY
What? (Sorry for not getting the joke...)
Yeeeah I figured most people wouldn't have it. I wish I knew of an open source alternative that has access to the same level of functionality you get from their macros like file input/output, the ability to control external programs, etc...
And, uh, impact... I'm guessing you mean the font. I don't think I, uh, do that. Yet.
"Happiness in Slavery" is the title of a Nine Inch Nails song. Its use on a photo of a Super NES Control Deck alludes to the "PC master race" meme. So back to topic:
Khaz wrote:
Yeeeah I figured most people wouldn't have it. I wish I knew of an open source alternative that has access to the same level of functionality you get from their macros like file input/output, the ability to control external programs, etc...
Python.
Or client-side JavaScript. You can make versions of your tool that run on both cscript.exe and Node.
I could theoretically look into learning python someday... Though a large part of what's so efficient about this setup is having the spreadsheet integrated into the program. I'm not sure what would be involved in interacting between something like python and a spreadsheet program. I suppose the practical advice then would be to skip the middleman and not involve a spreadsheet, and just write my own software... But then we get back to how my tool is probably redundant, if I could only figure out how the other dozen I tried are supposed to work.
I'm not sure when I'll have the motivation to put more into this since what I have is working for me quite nicely, but we'll see.
You might look into Open/LibreOffice. I know it supports a lot of the same macro functionality as MSOffice (though it might not use exactly the same syntax), and it's both free and cross-platform.
I've used LibreOffice before but I wasn't aware it supported those kinds of macros.
After doing some experimenting most of what I have works just opening it in LibreOffice Calc without any changes other than removing the "Application" in "Application.InputBox". A couple relied on a Collection for hex conversion (hC) that doesn't work in Calc for no apparent reason, but it's easily cut out and replaced with manually substituting A-F for 10-15. Conditional formatting and cell colouring looks like it'll have to be completely rewritten though and I have no idea how to do it in calc and I can't find any decent documentation of their API (sic) for it and I can't find the record macro feature either because it's not where they say it is in the options menu.
And then I accidentally closed the window only to find that all the macros I just edited for a couple hours are now GONE. I have no idea what the hell happened. I opened and closed it several times between editing before with no problems. And now I don't want to even try without knowing for certain that it's not going to just lose everything all over again. I also don't even know what format I'm supposed to save it as, .xlsm isn't one of the options but if I change it then it seems to drop all the macros and all the formatting.
I keep trying to switch to alternatives like LibreOffice but to be honest it does something stupid like this to me every time I use it. And unlike MSOffice, searching for answers online will almost always get you nowhere...
EDIT: My version of LibreOffice is somehow hilariously out of date. I will try again, after some kind of break.
.xlsm is a MS Office format, the LibreOffice formats are .odX. If you're saving a full spreadsheet, it's .ods. I'm not sure if there's a file format for saving just a macro by itself. Saving as anything other formats will, of course, drop the macros because they don't support them. If you're editing within LibreOffice, it's best to stick with the native formats rather than trying to export/import the MS formats, as sometimes things will get lost in translation. Also, the LibreOffice forums are a great source of information. If you can't find what you're looking for, ask.
This might be relevant to your conditional formatting issues.
Espozo wrote:
does each box (don't remember what they are called, if they even have a name) represent a pixel?
I think they are called "Cells"
(and english is not my language, thus you have no excuse for not knowing this
)
As a bleak update, please don't hold your breath for me to finish the .ods format of this.
Even with a whole lot more searching and reading, a lot of answers I need simply don't seem to be out there without literally contacting the people who wrote the program. What I have ALMOST works exactly as-is but the little finicky differences in what LibreOffice will accept is maddening. As of right now it is having seemingly self-contradicting problems in handling objects. For example, in MS VBA to assign an object to a variable you generally have to type "Set thing = Object" rather than just "thing = Object". But if you use "Set" in Libre it doesn't work, but then you take the "Set" out and it just gives you a different error.
I don't know Libre's macro language and I honestly just plain don't have the patience to learn it right now; not without a full listing of the objects they use and their internal parameters and functions. I mean, lacking that, programming is almost literally impossible. And while I'm at it I will state for the record this is precisely why I LOATHE programming anything at a level higher than assembly: You REQUIRE someone else to tell you in infinite detail exactly how these libraries you're using work, or else you'll lose an entire lifetime debugging in the dark. Microsoft's MSDN for Excel VBA is downright exhaustive and it STILL usually isn't enough for me to figure out these kinds of problems.
Long story short: An unregistered version of Excel 2010 will continue to function exactly the same after it's "trial period" has expired. If you have access to such a thing I recommend it. In the meantime I cannot bring myself to waste any more of my very short life on this conversion until I have some more complete resources to work from.
EDIT: Sorry for my earlier frustration. I'm not giving up. Just don't hold your breath, it might take a while.
Alright, so... I can't say if an open source version of this is ever coming. I looked into converting it to LibreOffice but the short answer is that converting it is impossible and it will have to be entirely rewritten in a different language, probably python. There is support for Excel VBA, but it's not fully implemented and probably never will be.
I wish I could say that I'm going to take the time to redo this open-source, but when I think about how much time that takes out of my actual SNES programming I don't know if I can follow through. What I can do is post a new and improved version of my excel sheet, for those that can use it if any. (It says it was downloaded 15 times, hopefully at least one of those people found it useful.)
To give a better idea of what it is and how it works, I figure I'll just describe it all right here:
------
The basic idea is you have worksheets for tile editing (128 pixels wide by 32 tall, could be made taller) where each cell represents one pixel. You type a colour value (0-F) into a cell and the cell is automatically coloured with conditional formatting. This makes editing very easy to do, and allows you to copy/paste and move blocks of pixels around conveniently.
There is a main "Palettes" worksheet which contains a full set of 16 palettes of colours. Each tile sheet also contains a custom palette of colours in SNES Hex format below the drawing area. You can choose (using macros) whether the conditional formatting will draw from that palette, or one of the 16 palettes on the "Palettes" worksheet.
The macros included are as follows:
CreateBlankTileSheet - Self-explanatory, creates a new blank tile editor worksheet.
DrawTileSheet - Resets the conditional formatting of a tile editor worksheet to display the tiles in any of the 16 palettes on the "Palettes" sheet, or the custom palette on the tile sheet. Can be used in 16 or 4 colour mode: 4 colour mode is for BG palettes only (0-7) and interprets palette numbers as the SNES would (ie/ Palette 1 begins at colour 4).
(You can have multiple palette sheets, but note that the "DrawTileSheet" function will always reference the one titled "Palettes".)
DrawPaletteSheetFromHex - Updates the colours on the active palette sheet according to the hexadecimal values entered.
DrawPaletteSheetFromCellColors - On a palette sheet, you can colour the cell to the left of the colour number (the "Pal #" column) using Excel (Format Cells > Fill, Ctrl+1 is the keyboard shortcut), and then run this macro to update the Hex values to match that colour. This allows you to generate hex values for colours you select visually.
ImportPaletteSheet - Creates a new palette sheet from the specified input file. Must be a text file and generally must be formatted the same as the output of "ExportPaletteSheet" (no indentation, ".dw", one full palette per line).
ExportPaletteSheet - Exports all 16 palettes of the active palette sheet as a .inc (text) file. The file contains two labels, "BGPalettes:" and "SPRPalettes:", each of which are followed by 8 palettes in ".dw" format.
ImportBitmap - Creates a new tile sheet and imports the selected 16-colour (Windows-format) bitmap file into it. Loads the palette inside the bitmap file into the tile sheet's "Custom" palette.
ExportTilesAsBitmap - Exports either a single tile or the entire active tile sheet as a 16-colour (Windows-formatted) bitmap file. User is given the option of white, black or custom background colour.
ExportTilesAsInclude - Exports the active tile sheet in .inc (text) format. Can be run in 16 or 4 colour mode as desired. Gives the user the option of including the custom palette in the same file as well.
SwapColour - On a tile sheet, swaps every cell with a specified colour for another colour. (For example, enter A then B and every cell containing A will be overwritten with B.)
RefreshSheet - Mostly useless. Occasionally the conditional formatting wouldn't take effect for me. This macro just forces it to by re-writing the value of each cell.
------
The only thing you need to do before you get started is set the path to save to. It defaults to "E:\SNESDev\". If you want to change it, you need to edit line 3 of the VBA module, which you can access through Developer > Macros > Edit. (If there's no "Developer" tab at the top you need to enable it under File > Options > Customize Ribbon, on the right side.)
The only thing I can think of to improve this is to possibly add an "ImportTilesFromInclude" function, but I have had no need of it yet since I do all my editing in excel...
Would love feedback if anyone has it.
Wanted to kinda post something new and maybe get some opinions. I'm still doing all my things in Excel and for that I'm sorry, but until I find another intuitively-programmable spreadsheet program I'm afraid it's what I'm sticking with. Anyways to the point:
I'm not an artist by any means. I've somehow ground out some player sprites for my game that I'm pretty happy with, but building an entire world for him to live in still seems an impossible task to me. I've been thinking since the beginning trying to come up with some way I can get some help with that, specifically with designing some backgrounds. My first thought was to try to convert standard images manually into a SNES-friendly format, but I quickly gave up on that when I learned the SNES format. I've recently thought of un-giving-up.
Attached are two files that some might find interesting. ColorDemo is simple and what it sounds like, a few spreadsheets containing every color code supported by the SNES inside cells of that color, so you can just scroll around and pick a color visually if you want. (The different sheets are just different arrangements of the same colours.)
The second project is a bit more ambitious and the point of this post. I've built on my ImportBitmap macro from earlier and created a color-quantizer that will reduce a 24bpp Bitmap image to 15 colors (since color 0 is transparent) using a variation of the "k-means" algorithm. Currently it's limited to images 32x32 pixels in size. Right now the file is very crude. It contains three macros:
"ImportBitmap", basically the same as before, creates a new worksheet and draws the selected bitmap into it. (The difference here is that it's just directly coloring the cells where before I was using cell values and conditional formatting; in other words this isn't set up for output yet.)
"kMeansKwantize", which runs the quantization algorithm. Most of the data it's running through is displayed right on the sheet under the images, you can watch the process. It IS rather slow, as k-Means quantization is an iterative solution basically found through brute force, that may or may not find the optimal solution. You provide it a randomized starting state and it will try to optimize it, but it needs to be run repeatedly to find the best result since the starting state influences the outcome. The way it's set up right now it will go through up to 60 iterations trying to optimize the given starting state before giving up and trying again from the beginning, and it will make a total of 40 attempts before quitting which takes a few minutes. (If you started it and your computer is slow or the file is big and it won't stop I'm sorry, just press Ctrl+Brk.)
"reDrawQuantized" will draw the image in the quantized palettes for you to see. Import bitmap draws two versions of the file on the left, one in original 24bpp and one in SNES-style 15bpp. This macro draws two more on the right side for comparison - third over is current palette, rightmost is best palette. The one sucky part of this that could be fixed in a more legit release is that you have to select the same bitmap file again that you imported originally since it's a separate macro and the raw binary data isn't saved.
ANYWAYS. The big question anyone that read this far is wondering is "Why the hell would you bother when there are programs that quantize for you better and faster". My thought on this is that I can hopefully write a customized program specifically designed to import graphics to the SNES, which will allow you to generate a table of palettes for a much larger image and then assign each 16x16 block of the image a palette such that you're left with a completely assembled image - all you'd have to do is load the tiles, palettes and a tilemap into your game and you've got any arbitrary image file on screen.
In concept I am sure such a thing is possible, but I'm having a hard time right now coming up with a strategy for doing it. If you quantize the entire image all at once you'll end up with every tile containing mixed palettes, which is useless. If you break it down one tile at a time, each tile will generate a unique palette. What I need is a way to calculate a compromise between each piece's ideal palette to come up with a set of 2-to-3 (or up to 8 depending on whether it's just a static image or an actual "background") palettes that will best represent the whole image. I'm not sure where to start.
Does anyone have experience with this kind of project? Has it been done? Anyone know of a reason I should give up?
Khaz wrote:
My thought on this is that I can hopefully write a customized program specifically designed to import graphics to the SNES, which will allow you to generate a table of palettes for a much larger image and then assign each 16x16 block of the image a palette such that you're left with a completely assembled image - all you'd have to do is load the tiles, palettes and a tilemap into your game and you've got any arbitrary image file on screen.
In concept I am sure such a thing is possible, but I'm having a hard time right now coming up with a strategy for doing it. If you quantize the entire image all at once you'll end up with every tile containing mixed palettes, which is useless. If you break it down one tile at a time, each tile will generate a unique palette. What I need is a way to calculate a compromise between each piece's ideal palette to come up with a set of 2-to-3 (or up to 8 depending on whether it's just a static image or an actual "background") palettes that will best represent the whole image. I'm not sure where to start.
Does anyone have experience with this kind of project? Has it been done? Anyone know of a reason I should give up?
I have experience, in the sense that I thought about it for a bit recently.
Perhaps one could quantize each tile (8x8 or 16x16) separately, and then repeatedly pare down the number of palettes by identifying palette pairs that are the most similar in a least-squares sense, and re-quantizing the set of tiles that use them as a single image. For palettes with free entries, you'd obviously want to avoid comparing colours that aren't actually used; perhaps pixel-weighting the least-squares comparison would work...? Though that raises the possibility of botching very small colour regions; a higher error power might be more resistant to such an effect, but it might skew the overall comparison...
Or perhaps you could just compare the tiles up front and pare them down to N sets that have similar colour requirements, and then just quantize once...
You decide whether or not I'm onto something; I haven't actually tried this.
...
It occurs to me that this technique will most likely result in a visible tile structure unless you're very careful to align nearby colours in different palettes...
Okay so I did it. Please find attached the excel spreadsheet that did it all, along with two proof-of-concept ROMs that just display the results.
First thing you do is import a (128x128*, 24bpp) bitmap file (using importLARGEBitmap). This creates two worksheets, one for drawing in and one to do all the calculation. Importing dumps a list of every colour used for each tile, along with how many pixels use that colour.
Next you run the initial quantization (kMeansKwantizeLARGE). This quantizes every 16x16 tile down into its own palette of 15 colours, which takes a bit of time. You need this starting point to run the next step. You can also draw this 64-palette version out if you want (redrawQuantizedLARGE, click "No").
Once you have that, you start eliminating palettes (crunchPalettes). You input a number of palettes to eliminate (up to 20 at a time), and the program will merge that many pairs together. This way you can reduce your image to as many or as few palettes as you want. So far I've been reducing each image to 8 palettes, for maximum quality in a single background layer (without getting fancy). I initially tried palette elimination by running a full quantization of each pair of tiles, then merging the two that resulted in the smallest increase to their average colour displacements. This turned out to be absurdly time-consuming and would have taken days to process one image. So, I took a more direct approach kind of like 93143 suggested:
For each pair of palettes, the program renders their set of tiles in the OPPOSITE palette, and calculates the increase in average colour displacement that would result from switching the two palettes. The palettes with the lowest increase from this swap are then merged and re-quantized together. If a palette has empty slots then the colour displacements for mapping tiles to that palette are all taken to be zero, to encourage the program to merge palettes with empty colour slots first. Colour displacements are also weighted per-pixel, not per-colour (ie/ if you have 83 of colour A and 1 of colour B, colour A is counted 83 times more than colour B. This distinction didn't seem to make much difference that I can see.)
Finally, when you've crunched it down to 8 or fewer palettes, you can export the results (exportInclude). This will generate a 16-color tile set and your palettes in one file, and a corresponding tilemap in a second file. At that point, it's as simple as cramming it into your program.
I have to say I'm very impressed with the results I got from this. I was expecting an ugly grid of barely recognizable garbage. All in all I'm hardly seeing any degradation as compared to every tile having its own palette, I could easily miss the grid pattern if I weren't looking for it, and my cat looks amazing. If only I'd written it in not-excel, so it could finish in less than 4 hours or so.
*For the record, 128x128 seems a convenient size but any bigger and not only will the calculations start to take days again but you start running out of tile space on your BG layer. 128x128 takes I believe half of your available tile space for one BG layer. You could do 256x256 by using 2 BG layers in mode 1 using basically the same technique, to fill the screen.
Alternately, if the processing of each image were sped up considerably such that you could do hundreds of them, I bet you could store frames of a 128x128 colour video and flip through them at at least 30fps. Tiles for a frame would be $2000 bytes, you only use $40 bytes of the tilemap and $100 for palettes so $2140 per frame... So uncompressed you should be able to get a full 16 seconds of video out of a 4MB ROM. Cool! That would be neat to see.
Awesome! It worked!
That squirrel pic has 112 colours, and the cat has 107. I fiddled around in cq to try to get them to look decent in 16 colours (to get a feel for how big the improvement is), and it kinda works on the cat, but the squirrel looks awful no matter what. And the squirrel is the one where it's harder to see the tile boundaries, at least on my monitor. Very impressive.
Khaz wrote:
128x128 takes I believe half of your available tile space for one BG layer. You could do 256x256 by using 2 BG layers in mode 1 using basically the same technique, to fill the screen.
Not half. A quarter. The tile index is 10-bit. One layer is more than enough to fill the screen with unique tiles in any mode except 7.
I notice you seem to be using 16x16 tiles. If the method could be sped up, using 8x8 tiles would probably improve the quality a bit; it's not like you're hurting for tilemap real estate...
I should note that there's a bug in the spreadsheet I just posted I noticed just now. If palettes 0 and 1 are the best ones to merge it screws up and merges the previous pair again. I thought something wasn't quite right.
93143 wrote:
Not half. A quarter. The tile index is 10-bit. One layer is more than enough to fill the screen with unique tiles in any mode except 7.
I notice you seem to be using 16x16 tiles. If the method could be sped up, using 8x8 tiles would probably improve the quality a bit; it's not like you're hurting for tilemap real estate...
Right, of course. I can't think of any reason not to use 8x8 tiles, I'll probably give that a try when I get the motivation to rewrite this again, as well as finding a way to make processing a 256x256 image viable. I think this could actually be useful.
Wow, very impressive work, Khaz. I especially appreciate your color demo. :3
Please keep up the good work (and please, please do make it open source).
Ramsis wrote:
(and please, please do make it open source).
Heh! This is the main problem here. I've already tried looking into converting my existing code into something open source, but the options aren't very promising. LibreOffice has been ruled out completely. I spoke to their community, support for Visual Basic code exists but is incomplete and will never be finished. Their only option is spending several months studying first python, then how python interacts with Libre of which there is little to no documentation, THEN re-writing it all over again, which simply isn't happening. Has anyone tried OpenOffice? I'm not expecting better from them but I haven't tried yet.
(Side note: Proper documentation to work from is the absolute most important factor. EVERY excel problem is fixed by one google or a quick look in the "MSDN". Every other programming language I've ever seen seems to give you absolutely -nothing- to work with.)
What I need is a language completely independent of a spreadsheet program, but I haven't tried to use such a thing in over a decade. I used to know C++ but it was a living hell for me. Java seems slow and useless. Once again I come back to Python, but for how popular it is there sure isn't anything out there to help a person get started from scratch. Excel has the entire user interface naturally built into the program; something like Python I would have to DESIGN that from scratch, which is more months of nothing but studying and no useful progress. It's just too convenient having the spreadsheet available to store and manipulate all your data. I could store everything in arrays and do all the math behind the scenes... Not only is that once again completely rewriting everything from scratch again, but god help me if I have a single bug in the finished product because I will NEVER find it.
If somebody has a suggestion for something that would be ideal for this, let me know. I'm tryin' but at this point I don't think it's going to happen and the words "Open Source" are starting to give me panic attacks.
Something like this (an automatic color reducer) might be done as a command line tool. Python has
import argparse to parse command lines, and it has Python.org for docs and
Stack Overflow for searching similar problems that others have had.
tepples wrote:
Something like this (an automatic color reducer) might be done as a command line tool. Python has
import argparse to parse command lines, and it has Python.org for docs and
Stack Overflow for searching similar problems that others have had.
Okay I see Python actually does have some documents on their website. Fine... I will try this one more time. If anyone here has actually used Python before I would appreciate them just telling me what IDE to use because there are too many choices and I'm tired of wasting hours on irrelevant decisions like that (I mean seriously, their website not only has a list of like two dozen of them but also another 15 articles exclusively about how to choose between them).
If an open source version of this tool is actually coming it won't be anytime soon so please don't hold your breath. This is going to take way more time than I have available at the moment.
Please excuse my less than pleasant mood as I'm not having the best day today...
Khaz wrote:
Okay I see Python actually does have some documents on their website. Fine... I will try this one more time. If anyone here has actually used Python before I would appreciate them just telling me what IDE to use
I just use IDLE, the one that comes with Python for Windows and is an
apt-get away on Debian or Ubuntu. For loading images, you'll probably need either Pillow (Python Imaging Library) or Pygame (Python wrapper around SDL).
I've just spent basically the last day trying to figure out how to write one byte to an output file using python.
I cannot possibly understand what the people who designed this language were thinking when they created the "byte" object. When you read the file in binary, you get these stupid objects that you then have to convert through some needlessly convoluted means into anything useful. That I can at least do. Then when you want to write a file you have to convert your values back into byte objects. I literally have nothing but numbers between 0 and 255. Every programming language ever made should have an expression no greater than ten characters that converts a number into a byte.
Python's only option is apparently "struct.pack". Once you figure out you need to import struct, then you need to figure out what "formatting expression" to use. You have 10 different choices for "integer", NONE OF THEM is a single byte. I somehow get the idea I'm supposed to store it as an unsigned character, since that's basically your only one-byte format. I try using "c" for char, but no, that expects a "byte object" as input which is what I'm trying to pack here in the first place. Try "B" and that doesn't work either because "required argument is not an integer". What does that even mean? Are they saying what it needs isn't an integer, or that what I'm giving it isn't an integer?
I used ord() to convert the byte objects into integers in the first place. Their official documentation states that chr() is the inverse of ord(), but I can't take the output of a chr() and write it as a byte into a file. Why not? Are they aware of the definition of "inverse"?
I can't even just do something like "bytes(n)", because that doesn't create a byte with value n. It creates a list of n bytes with value zero, which is useless. But oh, you can do "bytes([n])", which should give you a single byte of n. (What IS that syntax? I don't understand these "iterators" or whatever is going on in this language.) But then I can't put my number inside the []s because apparently a 'type' object is not subscriptable. And if I ord() it inside there, it says it expects a STRING of length 1. So I try using chr() again and I'm back to the first error.
What in the hell, this is unbelievable. This is why I hate higher level programming languages. In my entire time learning 65816 I never ONCE ran into frustration like this. I don't have enough information here. The
official page is not helpful because I don't understand what the program is looking for and I have no indication of what it is that I've done wrong so far. I understand that Python has "strongly enforced types" but if that's the case it's absolutely unforgivable to not have convenient methods to convert between said types.
Most of the rest of the program is done already, but until I can make it output something I can't even tell if it's working yet. This is stupid.
That whole thing, in a nutshell, is why I'm not trying to learn any high level languages. People always act like they are easier to use, and then you see something like that. I just like to "be in control", I guess.
OKAY. I seem to have figured THAT out. You have to use int(), not ord():
f.write(struct.pack("B", int(byteOfColour)))
Nnngh.
Now I have to diagnose this output:
...Without being able to examine my lovely spreadsheet of RAM to see where something went wrong. This could take a while.
EDIT: I thought I fixed it, but nope:
EDIT: Okay I have no idea what the hell is going on now.
EDIT: So tiles with less than 16 colours weren't being written right. I have a Quantizer again. Now I have to finish the Palette Crusher.
Khaz wrote:
I've just spent basically the last day trying to figure out how to write one byte to an output file using python.
Here's how:
Code:
open( "foo.bin", "wb" ).write( "x" )
or (this one makes sure the file is automatically closed when the "with" block exits):
Code:
with open( "foo.bin", "wb" ) as f: f.write( "x" )
Quote:
Every programming language ever made should have an expression no greater than ten characters that converts a number into a byte.
Code:
b = chr( 123 )
Well, it's not really a byte, but a character (or in fact, a string), but in practice it behaves the same.
Quote:
Python's only option is apparently "struct.pack". Once you figure out you need to import struct, then you need to figure out what "formatting expression" to use. You have 10 different choices for "integer", NONE OF THEM is a single byte. I somehow get the idea I'm supposed to store it as an unsigned character, since that's basically your only one-byte format. I try using "c" for char, but no, that expects a "byte object" as input which is what I'm trying to pack here in the first place. Try "B" and that doesn't work either because "required argument is not an integer". What does that even mean? Are they saying what it needs isn't an integer, or that what I'm giving it isn't an integer?
No need to use use struct.pack/unpack. They can be handy though if you need to store/read values that are longer than a byte and may have varying endiannes.
Quote:
but I can't take the output of a chr() and write it as a byte into a file.
Yes you can (see above).
Quote:
I can't even just do something like "bytes(n)", because that doesn't create a byte with value n.
The "bytes" type is a Python 3 thing. You don't need to use it (I never have, although I'm still on Python 2.x for the most part), but I'd assume there's a reason why it exists.
The problems you have encountered are quite typical in dynamically typed languages. Dynamically typed scripting languages can make it faster to write programs, but that comes at a cost of more runtime errors, often very obscure ones.
thefox wrote:
Code:
b = chr( 123 )
Well, it's not really a byte, but a character (or in fact, a string), but in practice it behaves the same.
I don't know if this is a difference between python 2 and 3 but I'm pretty sure they don't behave the same. ".write" in "binary mode" expects these particular byte objects, I couldn't make anything else work. In retrospect I should probably have been using text-write mode and encoding each integer as a character and doing it that way... In any case, I have that particular problem solved now. Naturally I can go all day without any headway but the second I complain about it it fixes itself...
I'm having palette problems still but at least I have all the bits of python I need working now. Hopefully I can get some results soon. On the negative side of things, python doesn't seem to be executing all THAT much faster than Excel (with screen updating turned off). It's a noticeable improvement, but the added comparisons from 8x8 tiles and a fullscreen image are going to make it very slow... We'll see how much I can do to it.
Khaz wrote:
thefox wrote:
Code:
b = chr( 123 )
Well, it's not really a byte, but a character (or in fact, a string), but in practice it behaves the same.
I don't know if this is a difference between python 2 and 3 but I'm pretty sure they don't behave the same. ".write" in "binary mode" expects these particular byte objects, I couldn't make anything else work.
You're correct it seems, they did change the behavior in Python 3. I'm kind of surprised about that, but then again I have next to no experience with Python 3.
In Python 3, to a file opened for writing in binary mode ("wb"), you can write the bytes type or any other buffer-like type. The array.array type should work as well.
Okay, sooo, I think it might be working in Python now. I need a new palette-merging strategy because it's taking a very long time. The problem is that since I'm doing 8x8 tiles now, a 1/4 screen image gives you 256 different palettes to merge together, which in itself is doable. I re-wrote it to quickly merge all palettes with empty slots into their closest full palette first, which goes by quickly and gets rid of a good 54 palettes right away in my cat test image.
The rest goes by painfully slowly, since I'm comparing every palette to every other palette before doing a single merge. I'm considering strategies to perform multiple merges after each scan, perhaps take the 10 (or so) best merges found per scan and then execute them all at once. Perhaps I should merge a scaling number of best matches per scan, depending on how many palettes still need to be removed. The palette comparisons are DEFINITELY the limiting factor here. I want to get to full-screen images, which is going to be 1024 palettes. I'm not good at execution-time math really, but I know that's going to be a lot more than just 4x the time.
It's running a test as I write this... Removing a palette once every 45 seconds or so. I sincerely hope it's actually working. I'm seeing lots of negative "best merge loss" values coming up, meaning the program thinks it found a pair of palettes whose tiles actually work better in each other's colour tables than their own. I guess with thousands of comparisons it's possible that's happening by chance, considering the initial quantization wasn't terribly exhaustive, but it still doesn't feel right. Guess I'll see if it worked when it's done.
EDIT: Welp. Program ran for two hours only to fail right before it finished with an "IndexError: list index out of range". Perfect example of why I hate trying to work with these languages. WHICH INDEX? THERE ARE EIGHT INDEXES IN THE LINE CITED.
Could you paste the line cited? I'd like to help you troubleshoot it.
Heh, thanks for the offer but I figured it out. Was just a stupid mistake where the list of tiles weren't being copied right as palettes were being merged.
It's functional again. Currently pending processing of the same test cat for comparison with the original program. Once I confirm it's working right I'll post the source code, though I still have some work to do. As it stands processing a full-size (256x256) bitmap will be absurdly time-consuming.
I'm considering a new approach:
-If the image is large (>64 in either dimension), break it down into 64x64 pixel regions (64 palettes) and compare palettes within each region.
-Merge each region together until each region has no more than 8 palettes, for a maximum of 8*4*4 = 128
-Compare and merge the rest as usual or, if it's still too slow, repeat the process breaking the screen into four quadrants.
If you have similar colours far apart it might not work as well, but it's the best option I can think of. Though considering I break down the two-dimensional structure of the palettes into a straight list, perhaps I'd be better off just arbitrarily splitting the list into N chunks of roughly the same size. It would group palettes along approximate horizontal strips first, so if you have some kind of left-to-right gradient, it might not be ideal. Though I think if I keep the total number of palettes for each region at least as much as the final number of palettes, I can't really be hurting the results... right?
I'll do some experimenting
Ha! 248 palettes removed in 13:19! Take that, reality.
Here's some new results. I think it's all a definite improvement over the old version, and it's hard to tell the difference between the fast way (rev2) and the two-hour way (rev1).
Rev3 is a last minute adjustment I tried - lowering the amount of quantization during the merging process, and then running an extra pass at the end to quantize the final palettes a bit harder. It doesn't seem to mess up the palette grouping, and even with an extra minute or so of bonus quantization it comes in at 12:07.
The way it works now during merging is it spins through the list and compares a block of a given number of palettes (default 24) to each other, then merges the best two and does the next 24 rolling over to zero when it reaches the end of the list, which should kind of blur the "regions" as it goes along and hopefully spread the palettes around appropriately.
Code is
now on Github. This is what Github is for, right? I wouldn't know.
Have a different full 256x256 image processing right now, will post results when I have 'em.
I wonder if doing a QR decomposition or similar on the 64 pixels of each tile would help you classify them according to color content as a way of speeding up the comparisons.
Alrighty, 256x256 fullscreen image functionality confimed - took about 1:15:00. Went with something a bit more colourful and less photographic to see how it would handle a different style. It looks just a bit greyed out on my monitor, but when I put it through the SNES onto my TV (with whatever settings it's got going on) it looks bright and fantastic and the little imperfections are hardly noticeable. By far the most impressive part to me is that I can barely see any grid pattern at all on anything I've tried so far.
I could not be much more thrilled with how this program turned out. Thank you for kicking me into finally learning a new language, annoying though it may be.
I will look into QR decomposition and see if I could use it. I'm sure what I have can still be optimized a bit, probably mostly just finding the most appropriate settings. If anyone feels like using it, note that you have command line options to adjust how many trials of quantization to do each time, how many loops for each trial, how many trials of quantization to do after all the merging is finished, and how many comparisons to do between merges. This sample was done on default settings.
This is AWESOME indeed.
Thank you so much for digging through all the difficulties, and posting your Python script, Khaz!
:3
Here's the result of my first test:
http://manuloewe.de/snestuff/violinist.sfcThe crushed palettes version took about 48 minutes to make. I'm equally astounded about how good the output looks.
One small issue I noticed is the tile-sized color/palette "artifacts" round the wolf's legs, as well as on the tree in the background. I wonder if that could be fixed?
Other than that, excellent work, thanks again!!
Ramsis wrote:
One small issue I noticed is the tile-sized color/palette "artifacts" round the wolf's legs, as well as on the tree in the background. I wonder if that could be fixed?
It probably could, but the amount of total colors would probably be decreased. It took me about 5 minutes to even find what you were talking about, so I think it's fine.
Espozo wrote:
It took me about 5 minutes to even find what you were talking about, so I think it's fine.
Says the guy who bothered to express
this not too long ago.
Not meaning to be rude, but seriously dude, what exactly is the point of you even posting in this thread?
Daala: Chroma from Luma is another interesting read and may give you some ideas on how to classify tiles.
Ramsis wrote:
This is AWESOME indeed.
Thank you so much for digging through all the difficulties, and posting your Python script, Khaz!
:3
Glad the work is appreciated!
Ramsis wrote:
One small issue I noticed is the tile-sized color/palette "artifacts" round the wolf's legs, as well as on the tree in the background. I wonder if that could be fixed?
Hmm, yeah, that did turn out pretty bad in those few spots around the legs. It looks to me as though your mergings never really developed a single good palette to cover both the leg colour and the grass together. I imagine it would be extremely difficult ranging to impossible to simply "fix" that. Simpler cases of adjacent tiles having a slight colour disparity such that you see the tile border could be a simple fix - you could try manually averaging the two colours and making them the same between palettes, but you're most likely going to do more harm than good to the overall result.
For the problem in your image, the only recommendation that comes to mind is to try it over again. The quantization process is very random (well, it's as random as python's "random.randint" function which I'm just trusting), meaning that running the program twice should generate a nearly identical image constructed of probably totally different sets and groupings of palettes, which could very likely fix that problem (and just as likely create a new one).
If you really want to grind the palettes down as hard as possible to get an image to come out just-right, no matter how long it takes, feel free to crank up the settings. The number of trials defaults very low (2 where my excel program used 5) just to keep it running in good time. The palette merging comparison chunk size defaults to 24, meaning it compares a block of 24 palettes to each other before choosing a good merge. With a target of 8, I'm assuming that you should never force a bad merge because you should always have several pairs tiles that belong together to choose from. If you want to see if it can make a difference, you can raise it a bit, something like 30 or 40 shouldn't cost TOO much extra time. What I'm curious to see is the result of cranking it all the way down to say 9 - if it scans 9 then merges one, theoretically any losses should be small because again, two of those palettes are going to have to end up merged anyways. But it'll definitely affect the groupings and I can't say how badly.
I can still speed it up a bit too, now that I remember, by removing some unnecessary repetitive square-rooting...
Ramsis wrote:
Espozo wrote:
It took me about 5 minutes to even find what you were talking about, so I think it's fine.
Says the guy who bothered to express
this not too long ago.
And you never used it.
Ramsis wrote:
Not meaning to be rude, but seriously dude, what exactly is the point of you even posting in this thread?
Is this an exclusive club?
Espozo wrote:
And you never used it.
He may not have used it but he DID fix the problem
Espozo wrote:
Is this an exclusive club?
This is my thread and everyone is welcome dammit
EDIT: One thing worth noting: If you feed my program an image that has no tiles with 15 or more colours, it will fail. If you give it one where only, say, one tile has that many colours, the first phase of merging empty palettes into full ones will merge literally everything into those one or two palettes, which could possibly degrade the quality of some images pretty badly. Working on a solution but it's a difficult problem - having nonfull palettes screws up my system. I guess I need to scan through and collect unique colours until a palette is full, merge them, and continue until you get a full set of full palettes... I'll work on it.
Khaz wrote:
For the problem in your image, the only recommendation that comes to mind is to try it over again. The quantization process is very random (well, it's as random as python's "random.randint" function which I'm just trusting), meaning that running the program twice should generate a nearly identical image constructed of probably totally different sets and groupings of palettes, which could very likely fix that problem (and just as likely create a new one).
Thanks for elaborating Khaz, I'll try again then.
(Just wanted to be sure I wasn't doing anything wrong – the wolf pic was just a test, I'm not using it in my game or anything.)
Okay so here's a new demo. It's not quite a full ROM, but it's close, I think you can fit about another second's worth in at best before it's a whole 4MB.
The size isn't quite as big as theoretically possible but it's close at 128 pixels tall. I believe you could also center it vertically, but it would take some fancy work to time the screen turning on right before the picture or else you get garbage on screen. As it stands you get a tiny line of pixels showing below the image when played on a real SNES - I used a V-IRQ to turn off the screen and that's as fast as I could do it. I guess to make it perfect you need to get fancy with your timing starting from the scanline before.
I don't think you can do this with a fullscreen picture. There's not nearly enough DMA bandwidth for the full tile set, let alone the map and palettes. If you split the transfer across multiple vBlanks, you'll just get garbage displayed inbetween good frames. If you want to get really hardcore though, I've considered using sprites as well to draw the picture. It would give you another 8 palettes, but you have a very limited number of tiles... But they can be 16x16 or even 32x32. You could probably split the screen in half, and alternate which set you update so that you get some semblance of smooth motion, but uuuuuughhhhhhhhhh...
Is there any precedent for FMV being done on SNES like this, or has it historically all relied on the MSU chip? I know very little about the MSU generally speaking.
I also don't know anything about how you could possibly build a ROM larger than 4MB. My impression coming from a 6502 background is that there's some kind of mapper register you can use to switch between ROM chips. What would the theoretical limit on total storage space be using something like that?
Finally, how terrible is audio streaming, and how does that even work? I can only assume that you need a ton of storage space if you're actually streaming raw audio data - where is that stored? And how much processor time does it require to maintain per frame? If there's any concieveable way to get both the video and audio playing here I need to see it done.
Anyways. Processing the frames took a couple days running 3/4 cores on my computer. I had to use some elaborate VLC commands to extract the frames, then wrote a quick excel macro to control MS Paint to batch-resize-and-crop them all to the correct dimensions. It runs at 15 FPS, half the original framerate. However, I'm tempted to think that's actually BETTER than the framerate it played at on the original Sega CD - it looked really choppy last time I watched it on the real hardware.
Opinions?
Khaz wrote:
I also don't know anything about how you could possibly build a ROM larger than 4MB. My impression coming from a 6502 background is that there's some kind of mapper register you can use to switch between ROM chips. What would the theoretical limit on total storage space be using something like that?
On an abstract bankable thing? You could memory-map a compact flash card and get DMA-able access to however many GB of data you want. You'll have to negotiate aligning data transfers to align to CF pages/sectors/clusters, which is probably not a problem.
Without any extra hardware, Nintendo officially allowed for a not-quite 8MB memory map, called "Mode 25" or hackers call "ExHiROM". This is HiROM to which you additionally decode SNES A23, allowing for 4MB in banks $C0-$FF and (4MiB-128KiB) in banks $40-$7D and 64KiB between banks $3E and $3F.
The StarOcean decompression hack uses a not-quite 12MB memory map, additionally decoding SNES A22, putting things in the "traditionally LoROM" space.
This is impressive work.
Khaz wrote:
I don't think you can do this with a fullscreen picture. There's not nearly enough DMA bandwidth for the full tile set, let alone the map and palettes. If you split the transfer across multiple vBlanks, you'll just get garbage displayed inbetween good frames.
Double buffer. A whole screen full of 4bpp plus its tilemap is only 30 kB, meaning no matter how big a frame is (unless you insist on using hires mode) you can fit two of them in VRAM and just switch to the new one when it's ready. The palette is tiny and can go last.
Quote:
As it stands you get a tiny line of pixels showing below the image when played on a real SNES - I used a V-IRQ to turn off the screen and that's as fast as I could do it.
That sounds weird. Is there garbage in your tilemap? Are you sure you're not off by one with your scanline count? Does the interrupt code do a bunch of stuff before turning the screen off?
Quote:
If you want to get really hardcore though, I've considered using sprites as well to draw the picture. It would give you another 8 palettes, but you have a very limited number of tiles... But they can be 16x16 or even 32x32.
I wonder if it's possible to have the program check for optimal 2x2 groupings of tiles, so you could use 16x16 sprites... Even if not, you could use BG tiles for the 8 most common palettes and 8x8 sprites for everything else, and just keep merging palettes until the number of sprites drops to 128 or less. This does require you to refresh the lower OAM table, the bottom byte of each tilemap word and the upper half of CGRAM each frame, which could hurt the frame rate a bit...
Quote:
Is there any precedent for FMV being done on SNES like this, or has it historically all relied on the MSU chip? I know very little about the MSU generally speaking.
The only example of FMV in a commercial game that I've personally encountered happens in the title sequence of Wonder Project J; it has a flat blue background, and it doesn't look like it uses more than one 4bpp palette. There are
reportedly other games with FMV, though the only ones I'm familiar with on that list are Out Of This World and Flashback, which I believe rendered polygons based on prerecorded vertex positions.
On MSU1: The MSU1 has two separate functions. One is audio streaming, which puts CD-quality stereo through the analog passthrough pins on the cartridge slot, bypassing the SPC700 entirely (but without disabling it, so you can mix sound effects and whatnot with the streaming audio). The other is data streaming. It doesn't show video as such; it just allows the programmer to DMA a data stream from a 32-bit address space. I believe MSU1 video has historically been 8bpp, which requires heavy letterboxing to achieve 20 fps.
Please note (if you don't already know) that the MSU1 is a very recent addition to the SNES toolbox, introduced in bsnes and implemented in hardware by the SD2SNES flash cart. No official games were released recently enough to use it.
Quote:
Finally, how terrible is audio streaming, and how does that even work? I can only assume that you need a ton of storage space if you're actually streaming raw audio data - where is that stored? And how much processor time does it require to maintain per frame? If there's any concieveable way to get both the video and audio playing here I need to see it done.
Depends what you're doing. The MSU1 just autonomously plays back Red Book audio, with looping if desired; all you have to do is select the track and turn it on. blargg's 32 kHz 16-bit stereo demo uses a custom codec; the CPU has to decompress it and feed it to the SPC700 manually, which takes basically all of the available power. Bad Apple currently does something similar to blargg's demo but with 16 kHz mono BRR, which takes way less power because there's less of it and it doesn't need decompressing; in fact it is probably possible to use HDMA to transfer it, which would take very little CPU time.
The blargg demo is about 4 MB for a minute of really nice audio. Bad Apple uses about 2 MB for 3:40 of passable audio. MSU1 uses up to 4 GB (audio is separate from data, which is also 4 GB) for as much CD-quality audio as you could reasonably want - nearly 7 hours.
[EDIT: Didn't check the run time for Bad Apple, got it slightly wrong. Fixed.]
...
If my calculations and assumptions are correct, you should be able to maintain 12 frames per second full screen (~30 seconds uncompressed in a 12 MB ROM) whether or not you use sprites, though with sprites it's incredibly tight; you'd need to use an HV-IRQ because NMI doesn't fire soon enough. Obviously it would be faster for smaller frames - 16:9 full width is about 256x165, which fits in 256x168, which is 21,504 bytes, plus the tilemap (for the visible portion, so 21 of 32 rows) and OAM/CGRAM updates for a total of either 22,432 or 23,872 bytes. Bandwidth is roughly (262-165)*1324/8 = 16,053 bytes, easily enough for 30 fps. With a 160x120 window like you've got here, there's no reason you shouldn't be able to do 60 fps if you've got the data for it, unless you're doing significant decompression or something.
93143 wrote:
There are
reportedly other games with FMV, though the only ones I'm familiar with on that list are Out Of This World and Flashback, which I believe rendered polygons based on prerecorded vertex positions.
Hmmm... vector animations... Is Flashback where Macromedia got the idea for the name of Flash?
Adobe |
G.dallorto
Very cool, as always, Khaz!
Khaz wrote:
Is there any precedent for FMV being done on SNES like this, or has it historically all relied on the MSU chip? I know very little about the MSU generally speaking.
Apart from what 93143 said, there have been several homebrew projects featuring FMV, e.g. d4s' N-Warp Daisakusen (details in
this thread) or smkd's
video ROMs, the latter of which (using special S-DD1 mappers) were a very impressive achievement back then, and probably sparked the invention of MSU1.
tepples wrote:
Hmmm... vector animations... Is Flashback where Macromedia got the idea for the name of Flash?
Adobe |
G.dallortoThey would've named it Butt, but said butt looked like photoshopped on, so they had no other choice.
lidnariq wrote:
On an abstract bankable thing? You could memory-map a compact flash card and get DMA-able access to however many GB of data you want. You'll have to negotiate aligning data transfers to align to CF pages/sectors/clusters, which is probably not a problem.
Can anyone give more detail about this kind of "abstract bankable thing"? Are large cartridges built routinely, or would I basically have to design my own? Can emulators and something like SD2SNES commonly handle such a program? Would it exist as a bunch of separate ROM files mapped together somehow, or would it all be compiled into a single file, and how? Figuring that out is essential if I want to get the full version of this video playing, but I also have another project I'm probably going to want it for...
93143 wrote:
Double buffer. A whole screen full of 4bpp plus its tilemap is only 30 kB, meaning no matter how big a frame is (unless you insist on using hires mode) you can fit two of them in VRAM and just switch to the new one when it's ready. The palette is tiny and can go last.
OF COURSE! It didn't even occur to me that I wasn't using over half of VRAM.
Quote:
On MSU1: The MSU1 has two separate functions. One is audio streaming, which puts CD-quality stereo through the analog passthrough pins on the cartridge slot, bypassing the SPC700 entirely (but without disabling it, so you can mix sound effects and whatnot with the streaming audio). [...] The MSU1 just autonomously plays back Red Book audio, with looping if desired; all you have to do is select the track and turn it on.
Okay, yes, I definitely need the MSU1 audio streaming then to go with this video. Can anyone direct me somewhere that can teach me to use it?
Quote:
16:9 full width is about 256x165, which fits in 256x168, which is 21,504 bytes, plus the tilemap (for the visible portion, so 21 of 32 rows) and OAM/CGRAM updates for a total of either 22,432 or 23,872 bytes. Bandwidth is roughly (262-165)*1324/8 = 16,053 bytes, easily enough for 30 fps
For now I'm just going to keep things simple and say screw sprites, since quantizing the images is already so painfully time consuming and I also want to maximize the size of the video. The dimensions at full screen width are 256x192, which isn't possible to run at 30fps. (DMA bandwidth/frame ~ 11585 bytes, Tile set = $6000 or 24576 bytes). If I have to choose between 30fps or a bigger picture though, I'll take the bigger picture. 15fps looks perfectly fine to me. What I can do if I build it full-size though is just adjust the BG1 scroll and V-IRQ, trim the top and bottom of the picture and play it in 30fps for a direct comparison. I'm sure I can even put both in the same ROM, just press a button to switch between framerates/sizes.
How big is the original video? Are you respecting the Super NES's 8:7 pixel aspect ratio?
The original video I'm working from is 480x360. I'm not sure what you mean by respecting but when you scale the video down to 256 wide it comes out to 192, which is well within any recommended boundaries, right? I'm not too concerned if half a tile or so is lost on either side, when it can be adjusted for. Though the video itself does have a small black border around it. I was going to remove it all but I could just leave a little bit in. Now would be the time to make up my mind on that I guess.
Square pixels in 240p NTSC are 135/22 = 6.136 MHz. Each pixel lasts 7/12 of the color subcarrier, which is 315/88 = 3.580 MHz. The ColecoVision and SG-1000 use the TMS9918 video chip, whose dot clock is slightly slower: 945/176 = 5.369 MHz, and each pixel lasts 2/3 of the period of the color subcarrier. The Sega Master System, NES, Sega Genesis in 256px mode, and Super NES have the same dot clock as the TMS9918. These pixels are about 14% wider than they are tall, a pixel aspect ratio of 8 to 7. This means the width of everything is multiplied by 8/7. For example, a 192x192 pixel box is not square; a 168x192 pixel box is square. This means if you want a 4:3 picture to be 192 pixels tall, you need to make it 224x192 pixels. For 4:3 at full width, it needs to be 256*3/4*8/7 = 220 pixels tall. Or for 16:9, it's 256*9/16*8/7 = 164 pixels.
Okaay... So if I want this image to appear correctly and fullscreen, it should be 256x224 (though I cut it at the end of the image at 220 to start the DMA), which gives me enough DMA time to transfer about 27804 bytes ($6C9C) split across four frames. Tilesets for that size would be 28672 or $7000... So no, not possible to do 15fps that way. I guess you could do 12, but that's starting to sound pretty low...
216 tall = $6C00 (27648) tile set, $6C0 (1728) tile map, enough DMA time in four frames for $76F4 (30452) and palettes are always $100, so that totals $73C0 to transfer (29632).
So I guess those are the limits then. Fullscreen in 12fps, 216 tall in 15fps, and 165 tall in 30fps. I don't think I mind the 15fps option...
EDIT: To be sure I'm understanding this right, you're basically saying that if you play a SNES game in an emulator, and one on a SNES, the picture dimensions should be different? Or do emulators compensate for the difference to output the correct picture? Because I can't tell for certain but it looks the same dimensions in Bizhawk as it does in the bitmap.
How does a modern, square-pixel-based television handle SNES input? Is that why last time I tried plugging my SNES into one it looked like crap?
12 fps may be acceptable especially for hand-drawn animation
shot on twos. Only the fastest action is shot on ones, and there are ways to work around that using the matrix effect.
Some emulators do compensate for NES and Super NES PAR. Others don't. Still others are switchable. The option may be called "TV Aspect". Open an NES ROM in Nestopia, switch the video filter to "NTSC" (which enables TV aspect as a side effect), and see how much wider it looks than the same ROM running in Nintendulator.
I've been collecting
more dot clock rates and pixel aspect ratios. Here are some selected rates:
- Atari 2600, 7800 160px: 12:7
- Apple II HGR, 7800 320px: 6:7
- Genesis 320px, PlayStation: 32:35
- Commodore 64: 3:4
- DVD: 10:11
A modern TV handles the non-square pixels by resampling the signal to use square pixels, on the assumption that rates slower than 6.136 MHz are wide and rates faster than 6.136 are narrow. For example, DVD rate is 6.75 or 13.5, which is slightly narrow. If you can, try using an S-Video, RGB, or SCART cable with a modern TV. It'll eliminate some of the bleeding between chroma and luma inherent in composite video. The rest may be up to the algorithms that the TV uses to deinterlace 480i, which may produce undesired behavior when applied to 240p signals such as those from the vast majority of Genesis and Super NES games. Also make sure that the "sharpness" option isn't turned up.
tepples wrote:
12 fps may be acceptable especially for hand-drawn animation
shot on twos. Only the fastest action is shot on ones, and there are ways to work around that using the matrix effect.
Good to know! I think the main deciding factor here though is that my source video appears to be in 30fps, so I can get 15 by taking every other frame but there's no good way I can see to get 12fps...
I should also mention I updated the code I posted on github, since I didn't do the command line arguments right. It's obnoxious - all the examples you find for argparse show things like "default=24". But argparse always returns what you type in as a list object, except when you don't type anything in it just returns default which is a plain integer, so there's no clean way to handle it both ways with their examples. What you need to do is specify your defaults like "default=[24]" so it's a list every time. Ugh.
Khaz wrote:
I think the main deciding factor here though is that my source video appears to be in 30fps, so I can get 15 by taking every other frame but there's no good way I can see to get 12fps...
Detect near-duplicate frames and apply what amounts to
inverse telecine. Ask your digital video geek friends "how do I shot IVTC".
Khaz wrote:
But argparse always returns what you type in as a list object, except when you don't type anything in it just returns default which is a plain integer, so there's no clean way to handle it both ways with their examples.
Leave out the "nargs=1", and it should return the passed in value as is.
tepples wrote:
Detect near-duplicate frames and apply what amounts to
inverse telecine. Ask your digital video geek friends "how do I shot IVTC".
Err... I'm just wondering, why would I want to try to do that? If what I have seems to be natively in 30FPS (every frame is unique and seems an even time interval from the last), why would I want to destabilize it to run 12 FPS when I can have it look more natural AND smoother in 15? The only possible incentive I can see is to reach fullscreen size - but the 15FPS size is only about one row of tiles short of a full screen anyways, and I'm only having to cut 4 pixels off the image vertically to make it fit.. Considering the Sega CD played the video inside a tiny box on screen, I'd say I'm already winning by a mile.
Khaz wrote:
I'm just wondering, why would I want to try to do that? If what I have seems to be natively in 30FPS (every frame is unique and seems an even time interval from the last)
I was assuming that the animation was produced for film. Anything produced for film is not natively in 30 fps, and every frame is not unique.
Quote:
Considering the Sega CD played the video inside a tiny box on screen, I'd say I'm already winning by a mile.
The Sega CD also fit the entire audio and video stream into 150 KiB/s, the data rate of single-speed CD-ROM.
tepples wrote:
I was assuming that the animation was produced for film. Anything produced for film is not natively in 30 fps, and every frame is not unique.
Now that I think about it, I wouldn't be surprised if they deliberately animated every frame of that video specifically to try to show off what the system would be capable of. Don't think the effort would have been all that worth it for the Sega CD, but nothing's ever going to beat the original CD-ROM version for PC I had with the DINO libraries and whatnot. It had some nice AVI files.
Quote:
The Sega CD also fit the entire audio and video stream into 150 KiB/s, the data rate of single-speed CD-ROM.
Fair enough. Was 1x really the best they could do at the time? I don't have a great grasp of the timeline but it feels like 1x was around for like, a month before they came out with 4x and 8x...
The original PlayStation's drive was 2x (300 KiB/s). It also had
MDEC hardware that decoded something similar to Motion JPEG.
Mmmkay, so I have the video working now in 256x216 at 15FPS. Still processing the frames. I think I will be processing the frames for the rest of the month.
I only need two things to get this demo working properly: Massive storage space and MSU1 sound.
The MSU sound I've hit another roadblock. I'm trying to convert a .wav file into the correct format for it using wav2msu but it won't accept the input file. I ripped the .wav straight from my original Sonic CD CD with Windows Media Player, but it won't take that saying "Sound is not in PCM format". So I put it in audacity and export it again, now I get "Sample data not where it was expected". Does anyone know a program whose output wav2msu WILL accept?
As for the massive storage space for the video, I've made zero progress investigating that on my own. I understand that it can be done but I haven't the faintest clue how.
P.S. If I finish this demo, do you think I'll be allowed to post the audio file along with it without being sued by someone? Or will people have to build it themselves? I saw there's a Megaman X hack out there with MSU sound, but it doesn't provide the sound files sooo I don't see how anyone is supposed to play it.
EDIT: How come I always figure things out myself literally RIGHT after I complain and ask for help. Just converted it to .flac and it took that just fine. Now I just can't get it to work and I don't know what might be wrong.
Okay. Don't know if anyone here has experience getting MSU1 to work but here's what I have so far...
I have code in my main program as follows:
Code:
.EQU MSU_STATUS $2000
.EQU MSU_READ $2001
.EQU MSU_ID $2002
.EQU MSU_SEEK $2000
.EQU MSU_TRACK $2004
.EQU MSU_VOLUME $2006
.EQU MSU_CONTROL $2007
PlayMSUTrack:
php
sep #$20
rep #$10
; WLA-DX
lda #$FF
sta MSU_VOLUME
ldx #$0001 ; Writing a 16-bit value will automatically
stx MSU_TRACK ; set $2005 as well, so this is easy.
lda #$01 ; Set audio state to play, no repeat.
sta MSU_CONTROL
; The MSU1 will now start playing.
; Use lda #$03 to play a song repeatedly.
plp
rts
...with a "jsr PlayMSUTrack" right after the SNES is initialized.
I have in the same directory a scdtest-1.pcm file as track one, and a blank scdtest.msu file. I also have an .xml file copied right
out of here:
Code:
<?xml version="1.0" encoding="UTF-8"?>
<cartridge region="NTSC">
<!-- This part depends on the ROM -->
<rom>
<map mode="linear" address="00-7f:8000-ffff"/>
<map mode="linear" address="80-ff:8000-ffff"/>
</rom>
<ram size="0x2000">
<map mode="linear" address="20-3f:6000-7fff"/>
<map mode="linear" address="a0-bf:6000-7fff"/>
<map mode="linear" address="70-7f:0000-ffff"/>
<map mode="linear" address="f0-ff:0000-ffff"/>
</ram>
<!-- This is the important bit -->
<msu1>
<map address="00-3f:2000-2007"/>
<map address="80-bf:2000-2007"/>
</msu1>
</cartridge>
And for good measure I also have a manifest.bml file hacked out of the Megaman X example I mentioned earlier:
Code:
unverified
cartridge region=NTSC
rom name=scdtest.sfc size=0x400000
map id=rom address=00-7f,80-ff:8000-ffff mask=0x8000
msu1
rom name=scdtest.msu size=0x0000
map id=io address=00-3f,80-bf:2000-2007
track number=1 name=scdtest-1.pcm
information
title: scdtest
Which oh my god contains formatting now that I paste it here but comes out as one solid line in notepad. Lousy windows.
Does anyone know why WLA outputs a ".smc" file when this seems to want a ".sfc" file? What's the difference there?
Anyways. No sound, not sure what to try next. I'm testing it in recent Bizhawk using bsnes v87
Khaz wrote:
Does anyone know why WLA outputs a ".smc" file when this seems to want a ".sfc" file? What's the difference there?
The difference is
supposed to be that
.sfc is a ROM image, while
.smc is a 512-byte header for floppy disk copiers followed by a ROM image. What is the size in bytes of the file that WLA produces?
According to windows, 4.00 MB (4,194,304 bytes). Does this mean you can't use MSU1 on a ROM with a header?
Khaz wrote:
As for the massive storage space for the video, I've made zero progress investigating that on my own. I understand that it can be done but I haven't the faintest clue how.
...MSU1?
tepples wrote:
Quote:
Considering the Sega CD played the video inside a tiny box on screen, I'd say I'm already winning by a mile.
The Sega CD also fit the entire audio and video stream into 150 KiB/s, the data rate of single-speed CD-ROM.
Not to mention doing all the decompression in software, and early codecs were pretty bad at this (leading to tiny windows). Later games would fill nearly the entire screen though (
case in point).
93143 wrote:
Khaz wrote:
As for the massive storage space for the video, I've made zero progress investigating that on my own. I understand that it can be done but I haven't the faintest clue how.
...MSU1?
Is MSU1 the only way to get a rom of arbitrary size? I also thought it was more of a data streaming thing than a rom mapping thing.
Khaz wrote:
93143 wrote:
Khaz wrote:
As for the massive storage space for the video, I've made zero progress investigating that on my own. I understand that it can be done but I haven't the faintest clue how.
...MSU1?
Is MSU1 the only way to get a rom of arbitrary size? I also thought it was more of a data streaming thing than a rom mapping thing.
What's this "data streaming" thing though? From what I heard, it pretty much just effectively increases the total size a game can be, so there's enough space for the video. I imagine you could even make a game with a Metal Slug amount of tiles. I guess it has hardware to create the music signal though.
I think the idea is that you use MSU1 to stream in the compressed data that your video codec decodes into pixels. In this way, it's like the cart interface of the Nintendo DS.
Tried just cutting off the first 512 bytes of the ROM and renaming it .sfc. Caused an error in bizhawk but I've been there before so I just renamed everything and it fixed that, but now the program is clearly screwed so I guess I didn't just cut a header out.
I could just do MSU1 video streaming, but that's been done already and I'd be starting from scratch again, and really all I want is proof my method works. If I want to build a rom of arbitrarily large size, what are my options? All I need is maybe 10x the space I have right now to fit the entire video. If there's some way I can just map 10 roms and switch between them by writing to some register that would be ideal and it'd be hard to believe there's not several ways to do that by now.
MSU1 video streaming has been done, yes, but not (AFAIK) with multi-palette 4bpp; I believe your method's combination of frame rate, picture size and colour depth is unprecedented on the SNES.
And I don't understand why you're worried about using the MSU1 for the video, if you're willing to use the thing for audio. All it really does is provide a 32-bit address space for data, and you can DMA out of it, almost exactly like you're doing now. The only other option I'm aware of is the S-DD1, which is
supposedly capable of mapping 256 MB to $C0-$FF in 1 MB chunks, but I'm not sure why you would want to use the S-DD1 instead of the MSU1 when you're already using the MSU1... it's not like you need random access... okay, the hardware decompression is nice if you want a small filesize, but that's further from what you're doing now than the MSU1 is...
If you wanted to figure out how to stream BRR audio to the APU, you could eliminate the MSU1 entirely and only use "real" (ie: '90s vintage mass-produced) SNES resources for your video. But it just seems weird to eagerly leap on the MSU1 bandwagon for audio and
then be leery of using the data capability of the chip you've already decided to include.
tepples wrote:
I think the idea is that you use MSU1 to stream in the compressed data that your video codec decodes into pixels.
You don't even need to compress it in this case. The memory space is large enough for more than two and a half hours of the described video format without any compression at all. You could DMA it straight into VRAM.
In fact, if my calculations are correct you could fit Star Wars into 4 GB, in cinematic widescreen at 30 fps, using the slightly heavier BG+sprites scheme, and have enough spare screen space and DMA bandwidth to render subtitles in hires mode below the video.
...
A Super Magicom header is mostly zeroes, and it goes on top of a whole multiple of 65,536 bytes. If your ROM is exactly 4 MB, it doesn't have a copier header. In practice, while .sfc does usually mean no header, .smc doesn't imply a header; it's just a name people are used to. When I got my copy of WLA DX it was set up to produce .fig files, of all things, and I changed it to .smc because I wasn't sure if the SNES header the code was plainly set up to include was a copier header or not (it wasn't)...
93143 wrote:
If you wanted to figure out how to stream BRR audio to the APU, you could eliminate the MSU1 entirely and only use "real" (ie: '90s vintage mass-produced) SNES resources for your video. But it just seems weird to eagerly leap on the MSU1 bandwagon for audio and then be leery of using the data capability of the chip you've already decided to include.
Well, the main reason I definitely want the MSU1 for audio, unless I'm misunderstanding something, is that it should output the raw .wav sound whereas BRR is by definition introducing losses to that sound quality. I guess I hadn't really looked enough at exactly what the MSU1 does for data and I should give it a try though.
Haven't figured out why the sound isn't working. I really have nothing to go on with regards to what should be in the xml or bml files or how you even issue commands, I'm just basically copying that one site.
For MSU1 audio to play, one major prerequisite has to be met (at least on real hardware like the sd2snes), which is to unmute the SPC700, and disable echo. Also, you have to wait for the Audio Busy bit to clear before setting the audio state to play.
Here's a
code snippet I posted on another forum quite some time ago (the posting is in German, but the code itself should be self-explanatory, and has English comments). It's confirmed working on both higan and sd2snes. LMK if you want the ROM and/or full source, which I'd have to dig up from somewhere.
As for the data port, it essentially works the same, except that it's up to you how to process the streamed data, of course. A good reference is e.g. ikari's video player, sources are included in
these (snes/msu1 subfolder).
Mmmkay, so I tried adapting ikari's video player code to my program and nothing at all happened. So I tried using your code and I'm getting somewhere because it's crashing Bizhawk now - which I've learned happens when it hits a WAI command and never gets an interrupt, which due to the way I hacked it out means it's taking the "NoMSU" branch. So, I haven't made it recognize that I'm trying to use MSU1 yet. Bizhawk is just a shell for bsnes v87, so it definitely should work, right...?
I decided to try it in higan and magically, it works. Works on my SD2SNES as well. So I guess somehow Bizhawk lacks MSU1 support despite using bsnes.
I'm grateful for the help trying to get this put together but I can't believe that the actual specification page on byuu's site that all this knowledge seems to come from is just plain gone. Without your help I'd have never figured it out. Did I miss something? I was just reading some heated conversations right on this forum no more than a year ago getting the fine details of it worked out. Seems like an exciting new development like this should have resulted in a bunch of interesting new hacks in the meantime yet MSU1 barely even brings in a handful of google results. Do I dare start relying on it for more than mere demos?
(Speaking of those conversations, did they ever reach some kind of agreement about the audio frequency/clock speed? I was extremely concerned to see the issue of video and audio de-syncing brushed aside as unimportant - a tiny desync becomes absolutely intolerable very quickly.)
Anyways. The audio is hideously out of sync the way I have it written now... It starts playing several seconds before the video begins to load. As far as I can tell from my code, the first frame should appear on screen 4-5 frames after the audio begins playing, so I'm assuming there's some kind of long buffering delay or something from loading the audio data. When looking at ikari's video player it looked like he had it set up to start the audio as the first frame is drawn, but I'm not sure that's any better if you still have a significant delay right after the audio starts.
Was trying to take a closer look in higan but I'm shocked to discover it doesn't even seem to have a frame counter so I can see how much of a delay I'm even dealing with... Why oh why doesn't it just work in Bizhawk. I think I may have to go bother adelikat about that, if he's forgiven me for the last time I wasted his time.
The video is looking amazing though. On my TV it's like 3/4 of an 8x8 tile row away from fullscreen, you can hardly tell it isn't a full picture. Frame processing is at ~200/2682 after maybe a day running full speed. Gives me time to figure out the data stream, though I am worried I won't have a processor left when it's done.
EDIT: Oh, I see:
Code:
lda #$01 ; Set audio state to play, no repeat.
sta MSU_CONTROL
lda #$00
- inc a
wai
sta MSU_VOLUME
cmp #$FF
bne -
; The MSU1 will now start playing.
; Use lda #$03 to play a song repeatedly.
So it's taking a bunch of frames to gradually ramp up the volume. I guess I can build that into my normal frame loop, if I even need it.
EDIT: Audio is nicely synced and I got rid of that annoying little line of pixels at the bottom by blanking both tilemaps first. Literally all I need now is to learn to DMA frames from MSU1 and I've got it.
Frame processing is approaching 50%. I really need to speed things up somehow but I'm already a little disturbed by the quality I'm getting out in places. Certain scenes have really large areas of almost-but-not-quite the same colour and it looks to me like those just become terrible the moment you reduce them to the SNES's maximum colour depth, as the smooth gradient of colour becomes a splotchy broken mess. This also kind of highlights the tilemap grid for those brief moments, which is less than ideal.
I haven't attempted any manual correction for this effect yet. In certain scenes it might be as easy as changing one or two palette colours, I will give it a try later.
98% of the video looks great though. I enjoy how the MSU1 just keeps on reading data after it reaches the end of the file - on emulator it does nothing but on SD2SNES it just starts playing the other ROMs on the card as though they were video, which is very very colourful and poooossibly seizure-inducing I'm not sure. I'll patch it out when I'm done but for now I'm enjoying the show.
...
Serious question: I'm pondering making use of the MSU1 for the other main project I've been working on. I don't dare to dream that it could someday be worthy of burning into a cartridge, but I'd like to keep that possibility open just in case I actually finish something for once. I'm afraid to start planning on it though without knowing how practical the MSU1 is to actually build.
It sounds straightforward enough to me, but it doesn't sound like the intended normal-cartridge implementation of MSU1 has been built yet. I got the impression it is supposed to be simpler than what ikari had to do for SD2SNES. I've recently discovered that I live near people with the means to build high quality reproduction cartridges, out of newly made parts instead of reclaimed used stock (and accurately labelled as reproductions to boot). I'm eager to discuss the possibility of MSU1 support with them, but I'd like to know what I'm getting into first from a hardware perspective.
P.S. I am aware this thread hasn't been about Excel for a long time now and I am sorry
Real-world hardware that can ONLY stream video or audio at once is easy. Real-world hardware that can do both from the same media becomes a royal PITA.
A more practical option might be putting two compactflash cards into a single cart, one for audio, one for the CPU. Or maybe better would be a compactflash card for the CPU and one of those MP3-player-on-a-chips to stream audio.
(P.S. PM tepples a split point)
lidnariq wrote:
Real-world hardware that can ONLY stream video or audio at once is easy. Real-world hardware that can do both from the same media becomes a royal PITA.
Wait, so, the way MSU1 functions in higan and SD2SNES is not how it's intended to work? Or do I misunderstand? Are you saying that you can't DMA data to VRAM using the MSU1 while a track is playing? Or did they deliberately emulate the maximum attainable functionality regardless of how difficult it would be to implement? I mean - how bad is the SD2SNES, when it has to not only stream data and audio simultaneously, but then write SRAM back to the same card too?
I think I'm starting to understand what byuu was talking about now with the "open-ended design allowing for flexibility in deployment" or however it was worded. It makes sense but it's a bit of a pain to not just have a full blown chip design that explicitly lays it out for you. I guess nothing worth doing is easy...
The data and audio are streamed from two separate memory spaces, right?. So in a real system you could well have two different physical storage devices for them - in fact that would seem to make the most sense. The sync issue was dealt with (
I think it was dealt with...) by making the audio timing track the SNES master clock, which implies that the MSU1 itself does no internal A/V synchronization, which implies that the hardware interfaces for data and audio don't have to be tightly coupled. It seems to me that this means you just need 2x"easy" instead of 1x"royal PITA".
At worst you could just copy what the SD2SNES does, with an FPGA addressing a single moderately high-bandwidth memory device. Since save games wouldn't also have to go on said device, the reset-to-save hack wouldn't be necessary any more... mind you, I have no idea what the components would cost...
Ramsis wrote:
disable echo
Why?
Khaz wrote:
Or did they deliberately emulate the maximum attainable functionality regardless of how difficult it would be to implement?
That.
Basically, in order to be able to stream two independent things at the same time from the same medium requires a lot of RAM and a fairly large amount of logic glue. By promising both, it's kind of a pain to implement something similar without either
1- using two separate pieces of media, one just for audio, one just for the CPU
2- or a comparatively large FPGA with a large amount of RAM.
Quote:
I think I'm starting to understand what byuu was talking about now with the "open-ended design allowing for flexibility in deployment" or however it was worded.
"flexibility in deployment" means "difficult to deploy". If the interface were instead just "Here's a CF card/deserialized SD card/whatever" it would, at absolute worst, condemn the hardware instantiation to irrelevance in a decade or two.
That's
why the SegaCD / PC-Engine CD-ROM interface is hairier than the MSU1 one; it reflected the hardware rather than being abstracted away.
93143 wrote:
Ramsis wrote:
disable echo
Why?
Sorry, no clue whatsoever.
So I recently learned about some functionality in Excel I wasn't previously aware of, that inspired me to actually make a more useful tilemap editor. I think what I came up with is going to save me a lot of time, so I thought I'd share it.
There are a few macros to be used in the same way as my previous spreadsheets:
ReadMap: Reads a text file of a tilemap (.dw formatted), either 32x32 or 64x64, and loads it into the MAP EDITOR tab.
WriteMap: Writes the tile map currently in the MAP EDITOR tab to a text file in .dw format.
PaletteShift: Gives three different ways to manipulate the palette numbers of every tile on the sheet. Run it and you'll get a popup window explaining how it works more specifically.
DrawEntireMap: IF you have a tile set in the TILE SET tab (the same way the tile sets are in my original spreadsheets - it reads the cell colours directly), then this macro will render the tile map according to the tile set, and output the results into MS Paint. You must have Paint open for this to work. (The image will also be drawn into another excel worksheet, but you should delete that worksheet as soon as it's done. Not every computer can seem to handle excel when the cells are at minimum size like that.)
UndoLastChange: Undoes the last edit to the map, SEE NEXT SECTION:
The fun part of this is that I discovered you can program macros to be executed on certain events within Excel. In this case, I have macros set to run when you click on cells within the MAP EDITOR worksheet!
The top part of the sheet is the 64x64 map itself. If you left-click on a cell, the value in "LEFT CLICK:" down below will be written to it. If you right-click, the value in "RIGHT CLICK:" gets written. You can drag to select a range of cells, and they will all be assigned the value in LEFT CLICK. You can't drag a range of cells with the right mouse button, but you CAN select a range with the left button (which will change it to LEFT CLICK), and then right-click that selection to change it to RIGHT CLICK.
You can change the values of LEFT and RIGHT CLICK manually using the other fields at the bottom: The box to the bottom left is your tile numbers (right now it's built for 16x16 tiles so there's no odd numbered rows or columns, sorry, can work on that). Click the boxed, numbered cells to set H/V inversion, palette and priority. Alternately, you can copy in a number that's currently on the tilemap by clicking "< COPY TILE INTO LEFT (or RIGHT) CLICK", then clicking the tile on the tilemap. The "COPY" cells will highlight themselves in red to show when they're active.
Finally, since this functionality requires disabling the normal on-click behaviour, there's also a big "DISABLE MACROS" button. When it turns red and says "MACROS DISABLED", everything returns to normal Excel until you click it again. The main thing this allows you to do is select a block of cells on the tilemap, copy it and then paste it elsewhere.
Finally, there's the UNDO STACK. Since Excel can't "undo" a macro action, and since it's very easy with this method for a stray click to screw up a map, I implemented my own undo feature. Every time you click or drag on the tilemap and make a change, it's stored in the UNDO STACK. Whenever you run UndoLastChange, it undoes the last change (the first item in the list). The keyboard shortcut is currently set up as CTRL+U, but you can change that yourself under Macros, Options...
If people have feedback I would welcome it. My continued and sincere apologies to the non-Excel crowd. I figure after a bit more touchup work, I should go back and edit the first post of this thread to put everything in one place...
EDIT: To-do list:
-8x8 tilemap support
-improve drawing (tile set/palette reading/importing, and direct file output because this method is stupid but it was just kinda proof-of-concept)
-Merge with tile editor spreadsheet?
Not that anybody is interested here, but I've realized a few more improvements on this version and I'm currently revising it. The next one should be much better in many ways.
Okay. Massively overhauled the map editor since my last post. Summary of the new state of things:
ImportTileMap, ExportTileMap, PaletteShift, UndoLastChange are unchanged (aside from the names).
DrawMapToBitmap: Does direct-to-bitmap output now, but is curiously somehow slower than my previous stupid method. It's about a 3MB file so maybe excel is just really slow at one-byte-at-a-time binary writes. It takes just under an hour at worst but I can't think of any reasonable way to speed it up further.
DrawTileSet: New macro; Renders the TILE SET sheet according to one of the palettes in the Palettes sheet.
ImportPalettes, ImportTileSet: New macros; Imports palettes or a tile set from a text file. ImportPalettes requires them to be formatted specifically (.dw where each line is one palette). Tile Set just reads in data (either .db or .dw) until the sheet is full or until the end of the input file is reached.
The MAIN overhaul though was how the Map Editor functions. The annoying click-cells-to-control-stuff region at the bottom is gone, replaced by a userform! Double-click any cell in the "MAP EDITOR" worksheet to bring up the "Map Tools" window (or alternately run the "ShowMapTools" macro manually). This has all the same functionality as before but in a separate little window so you can move around the map more freely. Brief summary follows:
Each mouse button has a Tilemap-Word associated with it. LEFT and RIGHT CLICK buttons at the top determine which mouse button's Tilemap-Word you're editing with the other controls on the panel. Left click a cell on the map to assign it the value of LEFT CLICK; right-click a cell to assign it the value of RIGHT CLICK. Drag-select a range of cells to assign them all the value of LEFT CLICK. Right click that selection to change them all to RIGHT CLICK.
When you click the COPY TILE button, then select a cell on the worksheet, the contents of that cell will become the new RIGHT- or LEFT-CLICK word, depending on which button is toggled at the top. UNDO just runs UndoLastChange (as does Ctrl+U).
DISABLE MACROS works the same as before, to disable editing mode so you can copy/paste cells and interact with the sheet normally. Also note that the click-to-edit functionality on the MAP EDITOR will ONLY work while the MapTools window is open. When the window is closed, everything returns to a normal Excel spreadsheet.
I decided against merging the two spreadsheets for now, just for the sake of less work and keeping things more organized. I'm satisfied for my own purposes again, so I'm only going to take on a task like that if other people are interested. As always, let me know if you find it useful and if you have problems with it I'm happy to help.
Khaz wrote:
Code:
if bytelist[0] != b'B' or bytelist[1] != b'M':
print ("This is not a bitmap. Aborting.")
quit()
Is there a reason why you parse Windows bitmaps manually rather than using Pillow (Python Imaging Library)?
tepples wrote:
Khaz wrote:
Code:
if bytelist[0] != b'B' or bytelist[1] != b'M':
print ("This is not a bitmap. Aborting.")
quit()
Is there a reason why you parse Windows bitmaps manually rather than using Pillow (Python Imaging Library)?
Yes, the reason is that this program was originally written in Excel VBA, where I don't think I had a choice but to decode the bitmaps manually. When I made the transition to Python I was basically only concerned with translating what I already had to that language. Also, this is the first time I have heard of Pillow.
Considering that the parsing of the bitmap takes a trivial amount of time in the process I'm not too concerned if it's not done the most efficient way. If anyone happens to find a way to speed up the actual quantization loop without losing quality though, that would be huge and please tell me.
In general if anyone wants to take my program and improve it, please, by all means help yourself. It is my only Python project and I'm sure I did a ton of things the stupid way.
PIL (or Pillow) is simply a widely used image library for python. It's not part of its standard library, you have to download and install it separately.
Writing a simple BMP parser is comparable to the effort required to install PIL, so I think this is a wash.
I imagine that it's a lot easier to
\Python27\scripts\pip install Pillow (
see full instructions) or
sudo apt-get install python-imaging (if on Debian or Ubuntu) than to write your own parser for PNG, JPEG, and compressed BMP.
tepples wrote:
I imagine that it's a lot easier to
\Python27\scripts\pip install Pillow (
see full instructions) or
sudo apt-get install python-imaging (if on Debian or Ubuntu) than to write your own parser for PNG, JPEG, and compressed BMP.
Yeah, I'd imagine you're right. I chose simple uncompressed bitmap as an input format specifically because it's so easy to decode and I already knew how to...
Okay, well I wish someone had told me about pip (so, thanks for telling me about it now). I remember trying to get PIL running on Windows Python 3.4 took me a couple of hours.
The reason you may not have heard of it might be that only recently did PSF start to bundle pip with Python for Windows.
Ramsis wrote:
93143 wrote:
Ramsis wrote:
disable echo
Why?
Sorry, no clue whatsoever.
You only need to disable echo if you don't plan on initializing/using the S-DSP at all.
If you don't disable echo on the uninitialized DSP it might play back the current contents of the echo buffer, resulting in static noise/buzzing.
Thanks; that's good to know. It seemed like a really weird and even unfortunate requirement, but that makes sense.
Hi guys, been a while. I decided my last approach at map editing was impractical, so I tried using Tiled (
http://www.mapeditor.org/ ) and translating its output into something I can use instead. Thought I'd share what I came up with, sorry that it's more VBA macros.
Brief description of what they all do:
* DrawPaletteSheetFromHex - Renders a palette sheet according to the hex values in it
* DrawTileSet - Renders tile set in a different palette
* ExportMapAsBitmap - Creates a 16-color bitmap image of your map, using the tile set and palettes in the excel file (uses the sheet labelled "TILE SET").
* ExportSNESMap - Creates a .inc file containing your map in a WLA-readable format.
* ExportTilesetAsBitmap - Creates a 16-color bitmap image of your tileset (for use in Tiled)
* ExportTMX - Creates a .tmx file for Tiled of the active map sheet. Can add layers as desired. Object layer support pending
* ImportPalettes - Reads data from a SNES-formatted file to populate Palettes worksheet
* ImportSNESMap - Reads a text file containing a map in a WLA-readable format and dumps it to a new worksheet
* ImportSNESTileSet - Reads a text file containing a tile set in a WLA-readable format and dumps it to a new worksheet
* ImportTMX - Creates map sheets and object list sheets as needed to dump data from a Tiled .tmx file into excel
* PaletteShiftMap - For tweaking palette numbers en masse in map sheets. The macro explains how it's used in detail when you run it.
Things to know:
* As before, edit the VBA code and change "rootpath" at the top to somewhere you'd like the program to save files to before first use. Default is "E:\SNES\"
* Tiled supports V/H tile flipping, but also tile rotation. Rotated tiles will cause errors of some sort.
* Tiled has no direct way of handling palettes or priority of tiles
* Tiled requires at least one tile set in a TMX file. When writing a TMX file this program presumes a generic 256x128 file TileSet.bmp to be present for the tmx file to use.
* Supports up to 4 tile sets in 1 file
* Macros designed with 16x16 tile size in mind, if you encounter 8x8 problems let me know
If people find this useful I'm willing to maintain and expand this set of tools to improve it. Let me know what you think, if anyone here tries it out.