Software Recommendations

I'm spending way too much time converting from MSPaint bitmap images to multilayered NES Screen Tool (background + sprites) and to ASM6 code. The result looks great, but there has to be a better way. The image below I did by hand and took a background image with 35 sprites to get right. A smarter algorithm could have done it much more efficiently (fewer total tiles), I'm sure.

Does anyone have any recommendations?

...is there a multilayered NESST? O_O

I've been using photoshop for layered screens, which is sweet in the start but can be a pain once you need to compose the sprite chr page (easiest done with the slice tool cutting out custom 8x8 pieces to paste into your chr page) and type out the OAM data manually once you're satisfied.

Would it be OK to make a tool that requires the user to supply an image, a 32-entry background and sprite palette, and (left, top, right, bottom) coordinates of rectangles along with their sprite subpalette number? I could make that, outputting CHR, nametable, and OAM page. Or is it critical that the software automatically guess the palette and the positions and subpalettes of sprites?

If you know what's supposed to be sprites and what's supposed to be background when you're composing the image, separate them into layers before you export them. (GIMP has a select by colour tool that usually makes this easy, Photoshop has similar.)

That way you can give a converter a much easier job if it doesn't have to try to algorithmically figure out what's supposed to be background layer and what's supposed to be sprites. It's extremely difficult for an algorithm to try and sort out the whole deal of ppicking palettes and choosing what should be sprites and not, but if you can separate that beforehand the tool doesn't have to do that, and it can do a much better job.

Can also help to reduce the background art you're inputting to just 4 colours (i.e. remove the "attributes" from the input) for input and then replace the attributes by hand afterwards in your NES tool. (Again a select by colour tool can make this easy.) This is something that an automated tool should be able to do fairly well, but only if it's designed to assume the input has correct NES style attribute regions to begin with.

Also, I should have said this at the start but use an art program with layers. Don't use MSPaint. Paint.NET or Aseprite might be easy to use. I like GIMP, myself, but it's difficult to learn.

IMO, selecting the best arrangement of sprites is the most time consuming task here, so I do consider it critical that the software is able to this automatically.

If it doesn't have to automate all of those things at once, though, it will do a much better job.

I think writing a program to generate a sprite set from an image that only includes the sprite layer is a billion times easier than trying to sort out what should be sprites vs. BG and assign palettes.

If the tool can make assumptions that it's not supposed to deal with arbitrary images, i.e. the author is supposed to know about NES limitations and compose with that stuff in mind, the tool could do a lot of it much more cleanly and easily.

For example, spitting out an error like "too many colours in the background at tile 25,13" (or better yet visually/interactively highlighting it) would be much more helpful than assuming it should be a sprite and most likely end up completely trashing the palettes/sprite arrangement the author had in mind. Can only do this if the tool is told which is supposed to be background beforehand though.

There's a really good middle ground between a tool that says "give me any image and just trust me" and "you know how NES graphics work, try to meet me half way and I'll tell you if you made a mistake". I think the latter is a very under-served area currently.

Well, no. But the two layers of the image (background and sprites) have to be pulled into into NESST. I then use code to perfectly position the sprites

That's part of the difficulty. In the image above I choose the the sprites to be the red of the woman's dress along with some supporting sprites along the attribute block transition between the two characters.
I took a guess at what should be background and what should be sprites, but I'm confident that it's not even close to optimal.

Best approach from my viewpoint:

-Use GIMP or PS.
-have at least five layers: BG, spr0, spr1, spr2, spr3.
-Up to you if you draw your bg in NESST or in said software. PS is able to prompt an update to your work ram file if you overwrite an asset via export from NESST so that's very efficient.

-each sprite layer naturally corresponds to each subpalette. You could add more layers if you need them to overlap intricately, it would make no difference. Just never mix two subpalettes in the same layer.
-export layer/s as png (individually).


1)have a tool to parse an exported sprite layer png and a reference .pal and a manually entered tile addr offset into a .chr and a .oam.
2)have a tool to piece together .chr blocks into a single one (can already be done in NESST, but it's a little crabby).

Piecing together .oam files is just a matter of .incbining them in your prefered order. All right, you could do that with .chr:s too or if you're on linux, just concatenate.

Oh! Sorry, I misinterpreted your original post. You weren't complaining that a tool cut it up in a sub-optimal way, you were asking how to optimally choose your sprite positions?

My own general technique is to work in rows of sprites. Like if you're just looking at a horizontal row 8 (or 16) pixels high, the optimal way to divide it up into sprites is usually clear, just taking the next 8x8 block at a time until you're through the row.

Vertically, rows will touch each other, normally I wouldn't want vertical overlap between rows because there is that nasty 8 per scanline limit. So really what I need to know is the optimal Y offset to start creating rows from. Usually what I do is put an 8x8 grid over the image, and then move that grid up and down until the tile rows seem to line up well with the shape of the sprite, then I start decomposing it into rows from there.


If I wanted to write a tool to do this, I'd probably identify disconnected "islands" of sprite pixels, and then for each island probably have the tool try all 8 Y offsets and see which one resulted in the fewest tiles. (I would looove an interactive tool for this too, but I've not yet had a high enough volume of that kind of work that doing it manually + current available tools didn't seem unreasonable to me.)

Someone here may have already created such a cool tool that I just don't know about though, and I'd love to know if there's other good methods for cutting up the sprites.

Btw i made a little tutorial on how to overlay sprites optimally* using PS, its grid tool and slice tool.
viewtopic.php?f=21&t=16337&start=21

The slices conveiniently show the sprites' bounding boxes. i have a 1px wide ruler which i move one scanline at a time over the picture to see that everything is where it needs to be and if anything would cancel out (and if that is acceptable). The neat thing about slices is that you can assign file names and metadata while editing.

The slice positions are also moveable independently from the pixel content which means you can paint to your heart's content and then move the bounding boxes to include the most important aspects of your artwork.

Also check the OP.

*meaning i get all the overlay i want in the right places. Not meaning doing this quickly as there's the manual labor of typing .db statements for OAM afterwards.

I've been preparing to write a tool that uses the "algorithm" I developed for Indivisible's sprite overlays.

One problem I ran into is where there's a shared color in the palettes, it can be tricky to decide which sprite palette "layer" should get the pixels of that color.

But a lot of the issues I ran into don't matter if you don't care about specific palettes. The program I'm planning will do worse than I did manually, but I think it's at a point where it won't bother me.

I already have a program that automatically spits out a background map/tiles given a palette and image:
So assuming it was given that, and the colors outside that palette were obviously sprites it's not too much of a stretch to do that too.

Basically, assume that you have to keep track of what your background palettes are. (A 16x1 image with the first palette as the leftmost 4 pixels, and the fourth palette as the rightmost four pixels). Beyond that, you just make your image. You import the image, and the 16x1 palette image and it gives you CHR for the background, a 1024 byte file for the nametable (Up to you to compress it how you want), and CHR for sprites and uncompressed 4 bytes per sprite OAM). No other steps. (Unless you use like 26 colors or >64 sprites or something. Then it will mark the errors.) Is that what you want?

Edit: Actually, yeah, I guess it'd need sprite palettes too. I don't really like the idea of palette guessing.

Do you guys actually have more test images for this? I see this one and FrankenGraphics Doom, but I kinda figure you wouldn't be making this topic if you didn't plan to do it like... a lot more. I make no promises I'll release anything but test images could help anyone else that wants to tackle this.


That's what I'm planning. My idea is you can have it do it automatically, or guide the tool. (Or a mix of both.) Basically it could show which area of the image it wants to "steal" and you could click next or do something different. Honestly just having an image editor that grabs an 8x8 slice (but not grid aligned) from a single click would have helped me tremendously.

It'd also identify parts of the image that matched tiles in the set (based on a given amount of coverage so everything isn't marked just because there are single pixel tiles... OTL). I want to have a nice gif of it grabbing all the sprites one by one in something like Super Mario Bros or Mega Man. I think it'd be satisfying to watch. Indivisible made me think a lot about this problem...

There's a subtle overlay used in a mock level here for the next project with toggle switch. I'll probably produce a couple more like that in exactly a years' time when i can shift focus from one of my main involvements.
And i used PS some more overlay and animations in this mockup for a project with rahsennor.

I'm working on two overlays for two compo entries, but let's keep that behind the curtains for now.

I could supply at least some working materials in due time if you need it.



This problem can be negotiated by keeping your drawn sprites in separate layer groups sorted by subpalette attribute in your drawing environment.
In this example, i have a couple of groups of sprites drawn in various layers belonging to a greenish subpalette, sorted under a folder for that purpose. I can then export that subpalette separately by exporting the layer folder as a whole, or each sprite group separately, or if i had had one layer per sprite - that sprite separately (at this level i used slices instead, though).

If you already know the background palette and have the background separated out to other layers, you can use savtool.py from my NES background graphics editor, which chooses the best fitting attribute for each 16x16 pixel area.

I decide when the art is done. From what I understand, you're suggesting to decide while drawing. But the only difference is when we're deciding. The deciding itself is the problem.

This sprite has black in three palettes.

Any black pixel could be drawn with any of the three. If I have a different layer for every palette, I'm still deciding which palette layer each black pixel should go into.

I think separating it into layers actually makes it harder because it's an extra thing to keep track of while the art is changing. (Every time I draw black, I have to be sure I'm on the right layer.) There's a lot of weird stuff too. Like you can overlap same color pixels to avoid creating a new tile.

One black pixel is needed to complete the foot. Rather than making a single pixel tile, I reused the hair sprite. It's done a lot on the axe handle as well. A single 8x8 handle tile can make any length >= 8. 8 pixels long, is one sprite. 9 pixels long is two sprites, with 7 of their pixels overlapped. 10 pixels long is two sprites with 6 of their pixels overlapped.

I did a lot of small tile saving measures like this throughout the game. But I generally only valued it when it didn't create more sprites. You can see that the first four tiles in that tileset would only need two sprite tiles. (One 1x2 white tile and one 3 pixel black tile) Doing it that way would have added one actual sprite to any metasprite that needed the first three tiles, though.

Anyway, I'll see what magic I can work automatically with images like in the first post, or the Doom title screen.

Oh yeah, for something like a player character, i just use NESST:s metasprite editor, then verify how the animation looks with PS:s frame animator, which helps me get the timing of the cels down.
Only caveat is there's no metasprite to bmp export in NESST so you need to screencap and crop each frame you want sent to PS:ses way.

There's not much need layering metasprites over bg for that sort of work, imo: It's more applicable when you want to do something about the attribute grid or introduce more colours to a scene.