Cel shading for sprite modeling à la Donkey Kong Country

This is an archive of a topic from NESdev BBS, taken in mid-October 2019 before a server upgrade.
View original topic
Cel shading for sprite modeling à la Donkey Kong Country
by on (#137451)
(Disclaimer : In this post I use "sprite" as a general term for a piece of graphics that could be drawn as BG or as a metasprite)

Fact1 : The Cel Shading technique is to make the computer generate images that looks "hand drawn" with a reduced palette.

Fact2 : The NES has only hardware for very reduced palettes.

Fact3 : Donkey Kong Country had been created by creating 3D models of characters, (rendered with normal lightning, not with cel-shading). Then screen shots of them have been taken and the result was dumbed down to a 15-colour palette. This worked well : Although it's arguable this kind of graphics didn't age so well, the render was seen as more fluid than what is done with classical hand-made sprites.

Fact4 : Dumbing down high-resolution images to a 3-colour palette (or even 7-colour with layering) works extremely poorly because the palette is too small, and graphics generated this way always looks weird and are never on-par with hand-made graphics. In my opinion, around 12 colours seems to be the bare minimum to have an arbitrary piece of high-resolution graphics still look like it's original when dumbing down the number of colours without manual editing.

Now, the combination of all this gives me the idea to create sprites from 3D models, just like Donkey Kong Country did, but using cel-shading instead of normal rendering. This way, the generated sprites will already have a limited colour palette, and there is no need to "dumb down" the rendered image to contain less colours, as it already contains few colours. This applies to all consoles of course, but this would also work on very limited consoles such as the NES, where you can only have black outline and 2 shades of a colour in a palette.

I don't think it would work too well for normal in-game sprite, because usually the NES sprites are too small, but it could make amazing scenes like in Ninja Gaiden (for exemple), with more detail in animation frames due to the original 3D models being exact. Or a game whose graphic style is close to Akagawa Jirou no Yuurei Ressha (i.e. single screen detailed background, large tall realistic human sprites), but with much better graphics.

I unfortunately totally lack skills to do a mock up, but I'd be interested in seeing what someone with better artistic skills could do.
Re: Cel shading for sprite modeling à la Donkey Kong Country
by on (#137452)
Didn't tepples do something like this for the RHDE title screen? (I just checked, it's not cell shading)

Bregalad wrote:
In my opinion, around 12 colours seems to be the bare minimum to have an arbitrary piece of high-resolution graphics still look like it's original when dumbing down the number of colours without manual editing.

As an example of the traditional technique used in an 8-bit platform, we have Sonic Blast on the Game Gear. The Game Gear has better color capabilities than the NES, but it still looks quite bad.

Quote:
I don't think it would work too well for normal in-game sprite, because usually the NES sprites are too small, but it could make amazing scenes like in Ninja Gaiden

I think the exact opposite. 3D is very useful for creating fluid sprite animations, but at bigger sizes, 3D just doesn't have the charm of hand-drawn art. If there was enough memory for a FMV, then maybe I'd agree with you, but not if we're talking about pseudo animations like the cutscenes in Ninja Gaiden or the static screens in that Ressha game.

Personally I don't think this would work very well. The outlines would probably come out very irregular, and would need manual touch-ups anyway. Also, 3D art often lacks the "soul" that hand-drawn art has, unless we're talking about really good Pixar/Dreamworks level 3D artists.
Re: Cel shading for sprite modeling à la Donkey Kong Country
by on (#137456)
Yes, I used something analogous to cel shading for RHDE's title screen. These steps were needed:

  1. Render in Blender.
  2. Shrink to desired size.
  3. Segment the image into regions by hue using channel operations in your paint program.
  4. Convert this color image to indexed color.
  5. Divide the image by the color image, leaving a near-grayscale representation of the shading, and convert the shading image to grayscale.
  6. Posterize the shading image to three levels (dark, medium, light). This is the step that produces the cel shading effect.
  7. Positioned dithering to two levels, so that the medium level becomes a checkerboard.
  8. Lots and lots of hand-tweaking of the shading image.
  9. Select the black pixels in the shading image and paste onto the color image.
  10. Use tools to convert the result to NES format.

For sprites or smaller-scale background objects, you might just want to choose a light, medium, and dark color, as seen in Donkey Kong Land and the GBC port of Donkey Kong Country. Here are some examples that I originally made for this post:

Image
Blender render of character designs


Image
Scaled down and reduced to 3 colors
Re: Cel shading for sprite modeling à la Donkey Kong Country
by on (#137462)
Quote:
I think the exact opposite. 3D is very useful for creating fluid sprite animations, but at bigger sizes, 3D just doesn't have the charm of hand-drawn art.

As long as the artist can touch up the renders by hand, I think starting with a 3D base is a great way to go. Everything stays in proportion and it is easier to animate things fluidly. Not sure why bigger sizes wouldn't look good. Any examples?

This game used 3D models as the starting point for hand-drawn sprite animations: http://www.siliconera.com/2012/02/09/the-art-of-blazblue-part-2-animation-phase/

And this Game Boy Color game seems to use sprites converted from 3D models: https://www.youtube.com/watch?v=3ge6OVwttlA
If that style was applied to a NES game it would look very unique and modern.
Re: Cel shading for sprite modeling à la Donkey Kong Country
by on (#137464)
I've also thought about this, but never tried to do anything like it (I don't have 3D modelling skills either).

In the same vein, using 2D skeletal animation for an NES game might be interesting area to explore as well. Of course the results would have to be "baked" into prerendered sprites. 2D skeletal animations can be made with e.g. Spine (commercial software, http://esotericsoftware.com/). Since Spine and other tools like it aren't designed for pixel graphics, it might be hard to get a good preview of the results. It might be best if something like RotSprite could be coupled with it.
Re: Cel shading for sprite modeling à la Donkey Kong Country
by on (#137465)
Quote:
I think the exact opposite.

Well, pehaps. I am no artist and have no particular skills in this field. I just thought the 3D could be made for more accurate movement and "shapes".

Personally I gave up very early in blender, I managed to design a table with 4 paralepipedic pillars, but then I didn't feel like to continue on because that'd be a huge project by itself, if I wanted to ever model something "alive", human, animal, monster or robot. I think we could also model rooms but that'd be even more complicated.

In all cases the 3D models would be created by a dedicated artists, and then could be used for either in-game or custscenes or both. If something is wrong then manual editing will have to occur.

But the whole point of cell shading is to avoid dumbing down the # of colours (step 6 of tepples reciepe), as well as to get a solid black outline in all frames.

Tepples, your title screen is nice, especially the shadows, but it has flaws. The large area which is a single dithered colour on the left looks especially messed up.

Your sprites are too zoomed out so I can't compare them with their originals. But it seems the black outline arround them is inconsistent, there is missing pixels, and this looks poor (although, the inside looks ok). I guess with cel shading directly in Blender when rendering, you could fix this issue, and remove point 6 in your chart. By rendering with having the camera sufficiently far, you could remove point 2, as shrinking add colours due to anti-aliasing filter. With cel shading you let Blender handle all this mess, and the rendered result is already good.

Quote:
In the same vein, using 2D skeletal animation for an NES game might be interesting area to explore as well.

Yes, but personally I fear it'd have this "201x smartphone look" instead of the good old 90s look we're looking for.

Quote:
This game used 3D models as the starting point for hand-drawn sprite animations: http://www.siliconera.com/2012/02/09/th ... ion-phase/

Sounds pretty close to what I had in mind when I made this thread. Also, I didn't think about it, but while the motion lines looks dumb on a still sprite, in the animation they make it look much much better.

Quote:
For sprites or smaller-scale background objects, you might just want to choose a light, medium, and dark color, as seen in Donkey Kong Land and the GBC port of Donkey Kong Countr

Those are exactly counter-example to how NOT to downscale graphics. They were clearly downscaled form the SNES version (the GBA version too by the way), and, well, it just doesn't look as good as the original.
Re: Cel shading for sprite modeling à la Donkey Kong Country
by on (#137467)
Bregalad wrote:
Your sprites are too zoomed out so I can't compare them with their originals. But it seems the black outline arround them is inconsistent, there is missing pixels

What can I say? I'm a selout.
Re: Cel shading for sprite modeling à la Donkey Kong Country
by on (#137477)
Wow, very interesting reading material, thanks for the link.

However, what applies to 15 colours "16-bit" sprites does not nessesarly applies to 3 colours "8-bit" sprites. The NES does lack enough colours to subtly "prison bar" the outline with two close shades of the same colour, at least for the outside outline. This is perfectly OK for inside outlines though.
Re: Cel shading for sprite modeling à la Donkey Kong Country
by on (#137481)
In defense of tepple's approach, I don't see any major issues with his graphics. The outlines are broken up, yes, but I'm actually rather fond of the look. When you break up your outlines into isolated black pixels like that, it produces a very sharp-looking effect. I've seen some pixel artists intentionally use that technique, so honestly I think it can work. And if you still don't think it could work, you could always go for a sketchy look for the entire game, kind of like how Yoshi's Island used really thick outlines to achieve a hand-drawn art style.

Ah, but I do have one complaint about his technique...When the sprites are rendered with shadows, they produce big black blobs which blend with the character and make them look fat from a side angle. Those shadows should be removed before rendering.