Subpixel rendering - NESdev BBS

Subpixel rendering
by Bisqwit on 2011-11-28 (#86890)

On RGB displays, subpixel rendering can be an interesting topic.

Here is a small sample picture, made small for your convenience (still 111 kilobytes).

No, don't zoom it, for it must be viewed at 1:1 pixel correspondance to the original image.

It contains the same scene twice. Each frame, it scrolls 1:3 pixels towards the left. The top part is rendered at subpixel precision; the bottom part is rendered at whole pixel precision.
At least on my TFT display, the top part (especially the ground) appears to scroll considerably smoother than the bottom part, without noticeable artifacts. (Wait for the file to load completely first!)

I was wondering about applications for this in NES emulator development.
- Subpixel-precise image scaling
- Rendering of NTSC artifacts at subpixel resolution

General information: Subpixel rendering is a tradeoff: It gives 3 x improved horizontal resolution, but because each subpixel is monochromous, it produces color artifacts. It works best when the subpixel details are whole pixel wide, only offseted, and relatively unsaturated (grayscale).
Train of thought: NES can render pixels at three subpixel positions. A pixel may be in its place, or shifted 1/3 pixels to the side, or 2/3 pixels to the side. Coincidentally, TFT has the same architecture. We can shift the entire screen losslessly sideways by 1/3 pixel increments! How about applying it for NTSC rendering?

Your thoughts?

Topic: Ultra-fast NTSC filtering.

Left: No subpixel rendering. Right: Yes subpixel rendering.
(Note: The rightside image appears to jitter more, because it brings more NTSC artifacts forth. The leftside image is 3x more aliased compared to the rightside one.)
EDIT: The pictures above were produced with a buggy algorithm. Below, fixed versions.

Left: No subpixel rendering; Middle: Subpixel interval 2 half-clocks; Right: Subpixel interval 2⅔ half-clocks.
Subpixel components are sampled as follows, depending on the subpixel interval:

Code:

0123456789ab0123456789ab0123456789ab0123 <- halfclock modulo 12
0000000011111111222222223333333344444444 <- pixels as generated by PPU (8 halfclocks per pixel)

No subpixel rendering (8,0,0):
        ............rrrrrrrrrrrr         <- NTSC signal sampled for each rendered pixel
        ............gggggggggggg
        ............bbbbbbbbbbbb
Subpixel interval 2 (4,2,2):
    ............rrrrrrrrrrrr
      ............gggggggggggg
        ............bbbbbbbbbbbb
Subpixel interval 2⅔  (2,3,3):
  ............rrrrrrrrrrrr
     ............gggggggggggg
        ............bbbbbbbbbbbb

Topic: Super Mario Bros. title screen, lanczos-scaled to proper aspect ratio.

Left: At 292x240. Right: At 876x240 (subpixel).
Topic: Same deal, Wizards & Warriors title screen.

Left: original title screen for comparison. Midde & right: The same title screen, shifted 1/3 and 2/3 pixels to the left respectively. Click here to see an animated progression of these three images.

It is easy to observe that the subpixel-scaled version does better job at capturing the sharpness of the original picture, but at the cost of some color artifacts at the edges of details.

Source code of ultra-fast NTSC filtering (old buggy versions: DgDsvnu6)

by tepples on 2011-11-28 (#86892)

Yeah, I've used subpixel to make the title screen of Lockjaw: The Overdose. The GBA's LCD is BGR (blue on the left), which makes the title screen look like crap when played at 1x size in an emulator because most PC LCDs are RGB.

You might want to box filter the red, green, and blue channels horizontally with a width of 1 output pixel (3 subpixels) to cut down on the color fringing that I see in the bottom third of the Wizards and Warriors screenshot. That's what ClearType does.

And then you have to deal with things like PenTile, which has only green at full res and red and blue at half res. ClearType style techniques won't help you there.

by LocalH on 2011-11-28 (#86913)

I like the concept of using LCD subpixels to increase apparent H-res, although as you know there are caveats that apply to greater and lesser extents depending on the individual viewing the output. Here are my observations of your images.

On the first scrolling example using SMB tiles, I can see a small amount of color fringing on the edges of the cloud, although these would be less visible when running at 60fps.

As far as using it for cheap NTSC artifact rendering, I do like the idea, as it could be useful on slower systems that may struggle with the algorithms. It definitely doesn't quite look as nice, especially when looking at the white text on the Ice Climber screens.

SMB title screen: Quite nice, although the color fringing on the SMB logo is extremely visible to me (and being a still graphic for the majority of its appearance onscreen it's slightly jarring but not a dealbreaker).

W&W title screen: Looks pretty nice, although I do notice two visual flaws. The red "PRESS START TO PLAY." text seems a bit chunkier in the subpixel version (for example, the tall vertical stroke on the left of the P and R should be relatively the same size after scaling, on the non-subpixel screen they visually appear to be so, but on the subpixel screen the R's stroke looks to be almost 3/2 the size of the P's). Also, the white text at the bottom seems strangely reminiscent of Apple II NTSC color artifacts when displaying text in split text/gfx mode =P

W&W title shifted across the three subpixel positions: This looks quite good in general, especially when considering that optimal motion will once again occur at 60fps. However, the motion of the aforementioned red text seems almost out of sync, probably due to the red pixels requiring far less use of the G and B subpixels. It almost seems to be merely jumping between two adjacent pixels, with an almost imperceptible color shift between two of the three frames then an apparent one pixel jump. The rest of the image looks fairly good though, even the blue of the sword (although if you look very closely you can see a tiny "gap" appear between the blade and crossguard, and the tip of the blade exhibits a similar appearance to the red text, although a bit more subtle since the blue does appear to contain some green).

My overall impression is positive, Bisqwit. I've long been impressed with your analysis of game engines through your work with TAS, and I've eagerly been following your work with square-based NES palette generation and more fully understanding NTSC PPU output.

by Bisqwit on 2011-11-29 (#86919)

LocalH wrote:
The red "PRESS START TO PLAY." text seems a bit chunkier in the subpixel version

Subpixel rendering won't really help with monochromal details (purely or mostly red, green or blue） because they are confined to one subpixel channel to begin with.

> Also, the white text at the bottom seems strangely reminiscent of
> Apple II NTSC color artifacts when displaying text in split text/gfx mode =P
Now that I have had the chance to look at these images on different monitors, I find that the appearance varies on different monitors. But yes, it has similar consequences.

tepples wrote:
You might want to box filter the red, green, and blue channels horizontally with a width of 1 output pixel (3 subpixels) to cut down on the color fringing that I see in the bottom third of the Wizards and Warriors screenshot. That's what ClearType does.

I tried that, and while it really did reduce the fringing, what it really does is that it brings the outcome back closer to the non-subpixel-scaled version. For instance, the same filter, when applied on the SMB title screen, completely negates the subpixel benefits in the ground pieces on the bottom of the screen: The vertical lines in between of blocks appear blurry and irregular shaped.

Left: Subpixel scaled normally. Right: With block filter of tepples.
Source:
Code:
<?php

define('BLOCKFILTER', true);

/* Input: 876x240 image, created as such:
* take screenshot (256x240) -- a quite normal screenshot from a game.
* convert screenshot.png -filter point -resize 1600% tmp0.png -- use imagemagick.
* convert tmp0.png -filter lanczos -resize 292x240'!' tmp3.png -- for comparison, this is the image scaled without subpixels.
* convert tmp0.png -filter lanczos -resize 876x240'!' tmp1.png -- this file is read by this program.
*/

$im1 = imageCreateFromPng('tmp1.png');
$w = imageSx($im1);
$h = imageSy($im1);
$im = imageCreateTrueColor($w,$h);
imageCopy($im, $im1, 0,0, 0,0, $w,$h);

$im2 = imageCreateTrueColor($w/3, $h);
for($x=0; $x<$w; $x+=3)
for($y=0; $y<$h; $y+=1)
{
if(BLOCKFILTER)
{
$r = (imageColorAt($im, $x+0, $y) >> 16) & 0xFF;
$r += (imageColorAt($im, $x+1, $y) >> 16) & 0xFF;
$r += (imageColorAt($im, $x+2, $y) >> 16) & 0xFF;

$g = (imageColorAt($im, $x+0, $y) >> 8) & 0xFF;
$g += (imageColorAt($im, $x+1, $y) >> 8) & 0xFF;
$g += (imageColorAt($im, $x+2, $y) >> 8) & 0xFF;

$b = (imageColorAt($im, $x+0, $y) >> 0) & 0xFF;
$b += (imageColorAt($im, $x+1, $y) >> 0) & 0xFF;
$b += (imageColorAt($im, $x+2, $y) >> 0) & 0xFF;

$r = (int)($r / 3);
$g = (int)($g / 3);
$b = (int)($b / 3);
}
else
{
$r = imageColorAt($im, $x+0, $y) >> 16) & 0xFF;
$g = imageColorAt($im, $x+1, $y) >> 8) & 0xFF;
$b = imageColorAt($im, $x+2, $y) >> 0) & 0xFF;
}

imageSetPixel($im2, $x/3, $y, 0x010000*$r + 0x000100*$g + 0x000001*$b );
}
ImagePng($im2, 'tmp2.png');
/* Outcome: 292x240 subpixel rendered image. */

Re: Use of subpixel artifacts in NTSC rendering: On an actual CRT television, with analog signal, the horizontal resolution is actually infinite, only defined by the pixel grid on the surface of the CRT. The pixel grid is actually indeed subpixel. Any color transition does not happen at "pixel" boundary; it happens at subpixel boundary. The electron ray gun does not know where pixel edges lie. Therefore I think that subpixel rendering is a way to actually produce a more authentic experience with NTSC filtering. However, you might only want to do it for onscreen rendering. For any material that is presented at non-1:1 pixel ratio, such as a YouTube video, it does not make sense to render with subpixels...

by tepples on 2011-11-29 (#86928)

Bisqwit wrote:
Re: Use of subpixel artifacts in NTSC rendering: On an actual CRT television, with analog signal, the horizontal resolution is actually infinite, only defined by the pixel grid on the surface of the CRT. The pixel grid is actually indeed subpixel. Any color transition does not happen at "pixel" boundary; it happens at subpixel boundary. The electron ray gun does not know where pixel edges lie. Therefore I think that subpixel rendering is a way to actually produce a more authentic experience with NTSC filtering.

Which actually makes sense when you look up close at a Trinitron shadow-mask CRT and any color LCD panel.

by Bisqwit on 2011-11-29 (#86937)

Just a heads-up, I fixed a bug in the NTSC rendering. The subpixel version (in the first post) looks nicer now...

by LocalH on 2011-11-29 (#86952)

Nice, Bisqwit. The fixed algorithm definitely looks much better.

by Zepper on 2011-11-29 (#86953)

Pardon my ignorance
Bisqwit wrote:
Code:
No subpixel rendering (8,0,0):
89ab0123456789ab0123456789ab <- halfclock modulo 12
........rrrrrrrr
........gggggggg
........bbbbbbbb
Subpixel interval 2 (4,2,2):
........rrrrrrrr
........gggggggg
........bbbbbbbb
Subpixel interval 2⅔ (2,3,3):
........rrrrrrrr
........gggggggg
........bbbbbbbb

So, we usually have 3 layers for a 1:1 image - a red, a green and a blue layer. The idea seems to be sliding the green and blue plans to the right by 2 or more pixels...? What happens, for example, in the left side of the image?

by LocalH on 2011-11-29 (#86956)

Based on what I understand, the different alignments basically produce differing blends or intensities for the various subpixels. I would assume it would basically be the subpixel intensities necessary to generate the background color on the scanline?

Re: Subpixel rendering
by koitsu on 2011-11-30 (#86957)

Bisqwit wrote:
Your thoughts?

My thoughts:

I find "subpixel rendering" on any display (CRT, LCD with TFT, etc.) to look like complete and total ass. I go to great lengths to turn off ClearType and "font smoothing" and other crap like that. The technology pisses me off for two reasons:

1. I hate the way it looks visually. Things look either too weighted or too light, looks horizontally phase-shifted (hard to explain; see old CRTs for what I'm talking about), or looks blurry (often "font smoothing" causes this one).

2. It's an absurd waste of CPU time. Even on a Core 2 Quad system running Windows 7 Ultimate, I can tell how much slower text takes to draw on a web page/etc. when ClearType and/or "font smoothing" is enabled. I imagine in an emulator (given the desire to retain 60fps at all times), this is even worse.

I am one of the few (dying?) people who believes strongly that when an artist creates graphics, their intention is for the pixels to appear on-screen as what they intended. Things like ClearType, Eagle (and other "smoothing" or "rounding" technologies), and even (IMO) lit/dead pixels on an LCD, all inhibit that from happening, which pisses me off. It also seems to piss off a lot of pixel artists too; take for example when Firefox implemented hardware scaling on images, so that a 100x100 picture with HTML <img src="blah.png" width="200"> resulted in a pixel-precise picture being linearly scaled, suddenly looking blurry and horrible. Pixel artists bitched enough that Microsoft and Mozilla both had to implement non-W3C-standard CSS tags and values to disable the behaviour (IE = -ms-interpolation-mode:nearest-neighbor; -- Mozilla = image-rendering:-moz-crisp-edges;).

So, in general, I shun this technology. Do I have a problem with people implementing it in their software, etc.? Absolutely not -- implement it, it makes some people happy! Just make sure that when you do, you ABSOLUTELY provide a way to disable it (.ini file, check box, something). I don't even care if it defaults to enabled, I just want a way to turn it off.

...hey, you asked for thoughts. :-)

by Bisqwit on 2011-11-30 (#86962)

Zepper wrote:
So, we usually have 3 layers for a 1:1 image - a red, a green and a blue layer. The idea seems to be sliding the green and blue plans to the right by 2 or more pixels...? What happens, for example, in the left side of the image?

Not by pixels. By half-clocks. I will explain this a bit.

Imagine that the NES screenshot is stretched horizontally by a ratio of 8, like this:

Left: Original picture, right: 8x stretched

Now, let's consider only one scanline of this picture. For the purposes of clear illustration only, I stretched it vertically a bit.

In order to generate NTSC signal, the NES PPU generates 8 half-clocks worth of signal per each pixel. Hence 8x scaling.

However, when the television receives this image, it must take at least 12 half-clocks worth of signal (instead of 8) in order to accurately assess the color to be rendered. Like this (underlined)

Each underline illustrates the region of original image data that is sampled in order to produce the picture rendered on TV screen. Each line is about the width of 1½ of original pixels.
Because they overlap, I put them in two rows, but they really represent one scanline.
Also, the TV actually interprets the input signal in a continuous process; not at intervals, but at any given time 12 most recent samples* are interpreted to produce the color to render. Hence the concept of infinite horizontal resolution.

As a consequence, there will be color artifacts on the television.

However, because of how PPU clock works, the image shifts on each scanline.

And not only that, the shifting varies each frame, too:

This is the reason to both the jagged appearance of straight vertical details, and to why it jitters.

Now, to explain the subpixel part, let's consider a single pixel from the sampled signal:

The pixel really consists of three subpixel components: the red, the green and the blue channel.
What I do in the subpixel rendering is that I sample those values from a slightly different position of the original signal in order to simulate the fact that the rendered color may change at subpixel intervals, not at wholepixel intervals:

Finally, here's how the subpixel components of adjacent pixels interact:

*) Yes, this is a simplification.

by lidnariq on 2011-11-30 (#86979)

Bisqwit wrote:
However, because of how PPU clock works, the image shifts on each scanline.

And not only that, the shifting varies each frame, too:

The image does not shift. The amount of time between hsync and pixel N is always the same -- exactly (N/1.5/3.579545MHz). A purely black-and-white image will not shake like that. Only the angle of the reference color (yellow) is what shifts from scanline to scanline.

by Bisqwit on 2011-11-30 (#86982)

lidnariq wrote:
The image does not shift. The amount of time between hsync and pixel N is always the same -- exactly (N/1.5/3.579545MHz). A purely black-and-white image will not shake like that. Only the angle of the reference color (yellow) is what shifts from scanline to scanline.

You are right. Somehow the fact got away from me while I was making that post: I couldn't convey the right fact, even though I have coded it right. I wanted to avoid going into the detail of how exactly the colors are coded in NTSC signal (I.e. frequency & sine/square wave & phase type details), for the hopes of making an understandable lesson even if it means taking shortcuts at points. I fail.