Idea for cell-shaded FMV on the NES

I recently watched some cool anime-style intro sequences on old games, like Astal, Silhouette Mirage and even Sonic CD (one of my favorite intro sequences since forever), and started thinking of a way to do something remotely similar to that on the NES.

Due to the limited memory and PPU bandwidth, the first thing I decided is that the pictures would be created with NT/AT updates only, instead of expensive pattern table manipulations, which are also impossible with CHR-ROM. Keeping things compatible with all the different mappers and types of CHR memory meant using only 512 tiles all the way through. The problem is that packing color combinations in so few tiles usually results in extremely chunky pixels.

Being impressed at how decent rgb121 alternating color components every scanline looked, I started thinking of how to do something similar. Based on the fact that many video codecs encode lightness with more resolution than color, due to the fact that the human eye is more sensitive to lightness, I decided to make that same separation. At 2 levels of brightness, I can pack 8 pixels in a tile, for a total of 256 combinations. If I arrange them in a 4x2 pattern, I can get a monochrome 128x60-pixel picture out of a single name table. As for the colors, 4 colors arranged in a 2x2 pattern (using the remaining 256 tiles) can represent a 64x30-pixel image.

Basically, the name tables would look like this (please don't mind the image I used for testing, it's just some cartoon looking thing I found on Google):


The interesting thing about having the color image use only half of its name table is that at least that part of the image can be double buffered, reducing the amount of bytes to update at a time every time the picture changes. Anyway, since the vertical resolution is not that good and the display will need constant scroll manipulation and pattern table switching, we need borders. The most straightforward thing to do is to squeeze the image to half of its height. Unfortunately that means that half of the screen is occupied by borders, and the video ends up looking very, very wide. This invokes a certain cinematic vibe so I don't think it's so terrible. Anyway, here's the result after combining the two pictures:


Yes, it's stripy and washed out, but it looks much more detailed than the color image alone could. The higher resolution lightness image adds a lot of detail, and allows for light and dark variations of each color in the color image.

Since one palette is used for the lightness image, 3 are free for the color image. Each 4x2-pixel area (area affected by a pair of attribute bits) in that image can have only 4 colors, and the 3 palettes must share 1 color (color 0), because of how the PPU works. Color 0 is not hardcoded to black (or any other color), which means that the palette has to be changed mid-frame because of the border. There is one free color in the lightness palette that could be permanently set to black and pointed to with rendering disabled, but in order to update the name table for the next frame the PPU address register will have to point away from it at some point, so a palette change would be needed anyway.

The frame rate would be only 12fps (which is fairly standard for hand-drawn animation, although they frequently use motion and compositing to make it look more fluid), to save memory and provide more time for processing each image. Here's a little test featuring scaling and rotation (I didn't have time to draw/rip a proper animation):


Obviously, images have to be drawn with this specific format in mind. If you simply take an existing video and try to convert it to this format it will surely look like crap. Ideal images should have thick outlines, few colors in localized areas, and very simplistic backgrounds. Zoomed out scenes should also be avoided, in favor of close-up shots.

I haven't given much thought to the final (and really important) step, which is compression. Each uncompressed frame is nearly 1.5 KB, so there's definitely room for improvement.

Any comments, suggestions, opinions?

P.S.: Bregalad, I won't be offended if you say it looks like crap.

You can remove a few tiles at the sides (they will not be visible anyway due to overscan), this will give you a bit more of room to play with.

Also if you can afford it, consider alternating the position of the scanlines (60FPS blinking, yeah), that will reduce the strippiness (and before somebody comments on seizures, strips also cause them anyway, if anything this may make it less prone to cause them due to being harder to see).

Interesting suggestion, but how would I arrange them vertically in the final picture?


That's something I planned for initially, and is totally doable, but the tests didn't look good (when viewed in a computer screen). It was really jumpy, and I kinda think that the stripy look is less blocky.

BTW, the workflow for creating the animations doesn't have to be much different from what Flash animators are used to. I draw the objects and characters in Illustrator (using a palette containing only the washed out colors) and export high-resolution images with no anti-aliasing. Then I import the images in Flash and disable smoothing for them. Then I just use Flash normally, trying to respect the limitations of the video format I explained, and I export the result as a high-resolution image sequence, ready to be processed by the converter. Avoiding anti-aliasing is important to make sure the encoder will always pick the correct colors.

This could potentially look great, but I feel like your demo using a badly drawn teddy bear doesn't do it justice.

I agree that limiting to 512 tiles and only NT/AT updates is a good idea. I however did not understand how you got arround the colour limitation. Also how'd you store the FMV in a format that is reasonably efficient in ROM ?

Just leave them blank? The idea is to not have to store them in the stream (or possibly rearrange some of them elsewhere if that's affordable).

This looks really promising, and way more advanced than how the Bad Apple demo looked.
If the space becomes a huge issue I think an oversized BNROM comes to mind or something like that...

True, but I wanted to see how the artifacts would look over time and that's the closest thing to the specifications that popped up on Google. I'll try to find the time to draw something myself.


What limitation are you talking about, specifically? In the end, these are just 2 pictures, one occupying a full name table (lightness image) and another occupying half of a name table (color image). The lightness image uses only 1 palette (or rather, 2 entries of that palette), and the remaining 3 palettes are distributed as necessary in the color image. If you spread the objects/characters nicely around the picture, avoiding having to repeat colors, you could get the impression of having 20 colors on screen ((3 colors x 3 palettes + 1 shared color) x 2 shades).


That's the part I haven't given much thought to yet. There should definitely be some temporal delta coding, but I haven't though of the specifics yet. I'm open to suggestions.


Ah, so you're were just suggesting a way to reduce memory usage... I though you suggested that as a way to increase vertical resolution. Yeah, I guess I can cut a few bytes off that way.


I definitely want to think of a good compression scheme for this. I think the goal here is to have a 30 second intro (360 frames) use 128KB or so. That's about 364 bytes per frame, which may sound like a lot coming from 1.5KB, but with good usage of the temporal redundancy I think it might be possible.

I like it.

Just to be clear...one name table is (2 colors) black and white, and one name table is black, white, brown, and pink (in this example)? And you're limiting the first one to (2 colors) black and white to reduce data size?

Well, any FMV that works on a real NES would be great. It would still be nice to add music or real audio samples longer than 1 second long.

It's black and gray, actually. I tried white but it didn't blend as well with the colors. Initially I tried dark gray and light gray, but the NES is really lacking in grays (if you don't use the grays that don't exist in the RGB PPUs), and black improved the contrast a bit.


Yes, but there's a second palette being used for the bracelet that has yellow instead of pink. Normally you'd avoid this kind of thing in order to not waste palettes with lots of repeated colors. Such details would be dropped (yellow would become pink or white) in favor of a more varied palette that could be used elsewhere if necessary.


It's actually to be able to draw 8 pixels per tile and still stay under the limit of 256 tiles. In order to have more colors, the resolution would have to be reduced which is exactly what I tried to avoid.

What I don't understand is how do you merge both colours and black/white layers into the FMV? Do you alternate between them every scanline? As long as there is no flickering I'm happy with that

Exactly. At small sizes, like in the images I posted, the colors blend so well you can hardly tell. Zoom in and you'll see. It will definitely not blend as well in full screen, but the overall result is still acceptable IMO.


Flickering indeed looked bad in my tests, but I'm willing to see how it looks on the real console.

Side tiles are very visible on every recent TV that I've used. True, the official background planning sheets used 224x192 (which corresponds to 10% border per side), but modern video standards are closer to 3-5% border per side. The 4:3 frame in NTSC is 280x240 NES pixels in size, and a (280 - 256)/2 = 12 pixel border on the left and right sides is 4.2%.

Many recent TVs will show even the border area, for that matter.

It was just a suggestion to reduce memory usage some more without it being really noticeable in practice.

How about RLE with predictors? Might work well enough.

You mean something like what's described in this article? I guess the prediction is also helpful for taking advantage of vertical redundancy, which regular RLE doesn't, right? Looks like it'd work well, at least for the color image.

The B/W image, which is responsible for most of the frame's size, looks highly compressible to me. I remember having some success with 1bpp images using quadtrees. I'll have to look into that.

Anyway, I'm hoping that the temporal redundancy will be of great help too. My idea is to keep one frame worth of data in RAM at all times, so I can draw a new frame on top of the previous one, by patching the differences. Since this frame would occupy around 1480 bytes of RAM, there would still be 568 or so bytes left (unless you're using extra RAM, obviously) to hold the state of the game and keep basic things like audio working. Sounds reasonable to me, considering that not much else should be running while videos are playing.

The best part is if you're dealing with monochrome RLE predictors, you don't need to signal what bit you are currently using. You know it will transition from 1 to 0, then 0 to 1. You only need the initial state and the lengths. Then you're just encoding a bunch of numbers (RLE lengths) and nothing else.

True so long as you know you can encode any positive length, not topping out at 127 or 255 or whatever. For this, I'd recommend either a Golomb code or an Exp-Golomb code (such as the gamma code).

Those cases would already compress so well that you may as well add an extension code just for them for a negligible loss.

Yeah, I'd simply have another length for the same color whenever the maximum length is used, even if this extra length is 0.