Sale image and Scale2x

I suck at graphics programming and theory and thus practice.

I have been reading what wikipedia says about Scale2x and i don't get it.
I have downloaded the code for scale2x of http://scale2x.sourceforge.net/ and reading what it says there and still nor explication neither explanation for me.

Anyway here is what at scale2x.sourceforge.net says:



What is it talking about??

Sorry the noob thing.

Totally glad I wrote this last year:
http://6bit.net/shonumi/2013/04/10/in-depth-scale2x/

One of the most annoying things I find about pixel scaling methods is how poorly they are documented, with the exceptions of ScaleX, Nearest Neighbor (and maybe Bilinear?) and xBR. Just about everything else expects you to parse the code in order to understand how the scaling actually works. It is my opinion that source code (except code that is highly commented) for some advanced topics is only ever half of proper documentation. Ideally, you need the application (source code) and the theory behind its ideas (articles, white papers, any well written work) in order for things to be understood. Code is the "how", but detailed explanations are the "why", and both are important. Sadly, many scaling methods are not too good on their detailed explanations.

Um, sorry for the rant Scale2x is pretty easy to implement yourself, because the author took the time to carefully explain how it works. xBR is very well explained to (in words and pictures, not just code) and given its quality you should check that out too.

Thanks much clear now!!

Images of xBR look really good! Very impressive.

BTW do these scaling algorithms work better when applied separately to each layer?

I would guess not. If, for example, you applied it to sprites separately, the edges probably won't blend well overall with the background, but the effect would be interesting to see. I have never tried to apply something like Scale2x on a per-layer basis though. As far as I know, all of the common scaling algorithms are applied to the entire screen in every emulator I've come across. It's somewhat wasteful of CPU cycles to apply the scaling algorithm per-layer for systems that make use of many layers (multiple backgrounds + sprites), but with today's GPUs, a decent shader would make it feasible.

Isn't that the idea, though? Different layers aren't supposed to blend together. I'm pretty sure that the background causing deformities to sprite outlines (and vice-versa) is a good thing to avoid.

To my knowledge, none of these scaling algorithms was designed to work with pixel data as separate layers. As far as I can tell, they were all designed to act on a single frame, treating everything (sprites, background layers, anything rendered as the "final" image) as a very large array of pixels. When I say sprites probably won't blend well with the background, I mean you'll get unintended results when pixel scaling around sprite edges.

Consider what would happen (with Scale2x in this example) if you tried to scale all of the sprites separately. The algorithm works because there is always sufficient data surrounding the pixel we want to scale, E. There needs to always be pixel data immediately available one scanline above it, one scanline below it, one pixel to the left, and one pixel to the right (pixels B, D, F, H. Edges of the frame itself (first scanline, last scanline, and the first and last columns) are the exceptions where input pixel data is "guessed" as in the next closest 3 pixels are used (e.g. the first scanline will act as if pixels A, B, C had the values of D, E, F).

With sprites, there is no guarantee that there will be sufficient data around the edges. Depending on what kind of sprite you have and its X and Y coordinates, Scale2x could easily encounter a situation where the sprite edge only hits pixels E and F. Here's the problem, what do you do for input pixel data that the algorithm needs? If you were normally processing the algorithm, you'd be able to take into account the background pixels, or any other pixels of the final frame. But if you're doing them separately, you'll have to fake or synthesize your own input data for the algorithm, which isn't how it was designed to be run, giving you results that may or may not look good or "correct". Depending on what fake data you feed into Scale2x for the sprite edge, it might revert to Nearest Neighbor scaling when it would not have if BG pixels were part of the equation. Nevertheless, the results would still be interesting as I previously said.

The input data for sprite edges is supposed to contain data for background pixels, because these algorithms were designed not to care or distinguish between any of them. To these algorithms, everything's just a pixel, and the frame is just one big series of them. The different layers on a console don't blend together, but these scaling methods like Scale2x were meant to enlarge the final image, so naturally it's going to blend everything together to some extent. With "fancier" algorithms like HQx (or even Scale4x), overlapping layers get blended more strongly at the edges.

To illustrate what I'm talking about, have a look at the following images using HQ4x and Shantae for the GBC:

http://i1.minus.com/ihcygM6eSKvmr.png - Shantae, HQ4x, scaling applied to entire image
http://i7.minus.com/ijk5gflczAX7Q.png - Shantae, HQ4x, scaling applied separately to BG and sprites

As you can see, where HQ4x has to guess pixel data for the sprites (which is essentially Shantae in this screen) I told it to just assume the pixels were white. This was just to highlight the areas that would be affected if you applied HQ4x separately. A more realistic approach would be to use the color of the nearest sprite pixel, but even then you wouldn't get the blending as intended by the algorithm. If you compare the problematic pixels in the second image with the first, you see that the BG is slightly blended around Shantae's outline (the red roof and the sky are notable areas). If you were to apply HQ4X separately to sprites, and if you were to fake the pixel data (black in the case for her outline), you'd end up with a slightly thicker outline that might not transition well in with respect to the background (essentially making it blockier or bolder than normal) and might give the impression that sprites "stick out".

If the end result is better, what they were originally designed to do shouldn't matter. =)


That certainly is the easy way out, which is probably why most emulators do it like this. Processing the layers separately would significantly change the rendering pipeline in most emulators.


If the filter is designed to smooth out edges, the edge of a sprite against a transparent area should be processed just as well as if it was against a solid color. Unintended results is what you get when processing a full frame that contains sprites in front of busy backgrounds, because the edges of the sprite will keep changing depending on what part of the background it's overlapping.

Consider this example of a green sprite with a black outline in front of an orange background that happens to have a black vertical line:


A filter working on the final image can't possibly tell that the two black lines are not the same object, and will smooth them out together, effectively "welding" the black lines together:


And it will only look like this while the sprite is standing at that same location, because as the sprite moves, the background will be welded to different parts of it, resulting in an inconsistent outline across several frames. That's what I'd call "unintended results".


Yes, and pixels outside of the sprite will be treated as transparency, which can be processed as if it was a flat color, allowing the edges of the sprite to be smoothed out without intervention from the background.


Background data doesn't belong to the sprite in the first place, so it's better to have NO data than have the WRONG data. But that's not even the case, since transparency IS data.


Transparency. If real transparency is not an option, as a last resort you can have a special color double as transparency and remove that later.


I'm not saying that filtering individual layers IS better, because I haven't seen the results yet, but in my head, keeping layers from interfering with each other can only be a good thing, but I'll only be sure after I see some test images. Another important point is that I don't even know how fast this would be.


And that's exactly what the suggestion of filtering the layers separately is trying to improve on! If current filtering technology isn't able to distinguish layers, it will keep blotching together objects that shouldn't be!

But the end result isn't better, or at least no one's devised a good enough result separately applying scaling filters to different layers.



That's how most scaling is done in almost any situation I can think of, even outside of emulation. You scale the resultant image that is a composition of every layer, not the individual layers themselves.



Yet the alternative isn't any better :/ Fwiw, these scaling filters are designed not to drastically produce inconsistencies with sprite art (rasterized 3D images, however, not so much, and Scale4x gets pretty warped). For the most part, BG and sprite blending is kept to a minimum, simply because of how the algorithms act when encountering groups of different colors. If the results were jarringly inconsistent, it would be of interest to few people.





Yeah, that doesn't work as well as you might think. See my edits above. Even if you assumed transparency, the colors are incorrectly blended when you place the scaled sprites over the scaled BG. Slight parts around the BG where it approaches an overlap with the sprite need to blended as well, else you still end up with sprite/BG transitions that look sharper rather than smoother, which is a problem for scaling algorithms that aim for smoothness, not sharpness. Forgoing transparency altogether in place of a special color leaves you with potentially sharp or defined edges when you remove those colors and paste the sprite onto the BG plane. Again that might be a neat technique for scaling methods aiming for overall sharpness, but most prioritize smoothness.

Let me further illustrate the issue. Here is the same Shantae test, scaling the sprite plane with HQ4x and using transparency. Note the ugly lines around her shoulders and her bangs. This is what I was talking about when sprites "stick out":

http://i4.minus.com/ixpbH1QD6ygcr.png

It's like someone just learning Photoshop, which isn't enjoyable at all compared to scaling the entire image via HQ4x. If you look closely, you can see that there's a sort of thin "film" present around Shantae in various parts.



Keeping layers from blending is good when you aren't scaling anything whatsoever. If I were making a game for, say the GBA, I would not want to start blending BG0 with OBJs unless I was aiming for a specific effect. But when you go about scaling an image, you usually don't do the individual layers, you scale the final frame. The final frame will have sprite and BG pixels side-by-side, so if it the scaling algorithm introduces new colors, it's good to blend the two. The overall image looks more natural.



But that's the thing, the layers need to be together See my reasoning above. Again, if you were not doing any scaling, it doesn't really make sense to do blending, but when you are, if you want the entire image to look cohesive, you blend everything to some extent, BG or sprite pixel.

I applied Scale2x by myself (pixel by pixel) and i got this:



Nice, but i still prefer the pixelized output...

Those lines looks odd. How do you interpolate 2 colours with transparency? Do you use the following?
R = (R1 * A1 + R2 * A2) / (A1 + A2)
G = (G1 * A1 + G2 * A2) / (A1 + A2)
B = (B1 * A1 + B2 * A2) / (A1 + A2)
A = (A1 + A2) / 2




Looks pretty good to me!

yeah i think the thing is get used to it. I have been playing my emulator with scale2x and now it feels nice.

@mkwong98 - All I did was take three screenshots of Shantae, one with all sprites disabled, one with the background disabled, and one with both enabled. For the screenshot with the BG disabled (the one only showing sprites) I removed the background in place of completely transparent pixels. For the first two, I ran HQ4x separately on the files, then combined them (background layer on bottom, sprite layer on top); this represents running HQ4x on layers separately. The alpha values you see are generated by HQ4x itself. Those alpha values are weird looking because the algorithm normally blends the BG and sprites at the edges together slightly. So you can't use alpha transparency to mimic how HQ4x normally works when it is applied to the entire screen.

From what I can see from hqx on google code, it doesn't use the alpha channel when calculating RGB values so it won't look correct. That implementation is optimized for solid colours and simply does not work with transparency. I'm not sure if the one you used does the same, but if so than we need to find one that works or test it with another algorithm. Thank you for those images!

Actually, HQx does support transparencies. This was the commit: http://code.google.com/p/hqx/source/detail?r=10 That's what the whole MASK_ALPHA is doing, so alpha transparency can be taken into account.

You can try it for yourself. Here are the two images I generated with HQ4x for Shantae. If you take 2nd picture of Shantae and paste it anywhere on the 1st picture and move her around, the edges around her change their colors, indicating alpha transparency. Or you could look at the RGBA values in the second image with an image editor.

http://i5.minus.com/if52BFCbTPtgX.png
http://i7.minus.com/iHJsOV5haQ2KX.png

That commit doesn't not use the alpha value to calculate the interpolation. For example when the weight of the two colours are the same, it does the following:
A = (A1 + A2) / 2
R = (R1 + R2) / 2
G = (G1 + G2) / 2
B = (B1 + B2) / 2

instead of what I posted:
A = (A1 + A2) / 2
R = (R1 * A1 + R2 * A2) / (A1 + A2)
G = (G1 * A1 + G2 * A2) / (A1 + A2)
B = (B1 * A1 + B2 * A2) / (A1 + A2)

So it is not intended to handle real transparency calculation, just a quick fix for 32bit colors.

It's inaccurate to say it doesn't use the alpha value in calculations during interpolation, since it does just that

http://code.google.com/p/hqx/source/bro ... ommon.h#62
http://code.google.com/p/hqx/source/bro ... ommon.h#73

The alpha value is used alright, but it's more accurate to say that HQx isn't calculating alpha values as you would expect it. Unfortunately, even if we look at your case where the weights are the same (which happens to be only in the Interp5() function when it calls Interpolate_2()) this doesn't seem to change any of the weird lines around Shantae. I've tried it; I went ahead and edited the relevant HQx code to try to accommodate your alpha calculations. I looked carefully however the sprites still stick out This might be due to the fact that the HQ4x is calculating most of its values using Interpolate_3() rather than Interpolate_2() OR the weights of the two colors are not equal to each other all that often.

Here's the code :
http://pastie.org/9579586

Again, I did not observe any changes in the Shantae test, sprites still have the weird outlining, when applying HQ4x on sprites separately (against a completely transparent BG). Perhaps the code is incorrect (likely, I dunno), in which case I encourage anyone to make the necessary changes (it's open-source ). I just jumped into HQx's code last night, but since this discussion has become quite lively, I would really like to see some definitive answers come out of this all.

I post that code to show an example of the problem, I'm not saying the problem only happens to that particular case. I'm not sure how the algorithm uses w1, w2 and s so I ignored them in my example. I can take a look at the full source but it will take a while as I'm busy with work at the moment. Thanks for your work.

See here:
https://code.google.com/p/hqx/issues/detail?id=5

I've seen that issue before, but thanks for bringing it up. I think the problem isn't so much that HQx can't generate transparent pixels (it can) it's just that what transparent pixels are there aren't transparent enough. If we could make the problematic pixels outlining Shantae in my example much more transparent, the weird outline issue might be solved. An easy way to do this would be to tell HQx to arbitrarily reduce the alpha values of any pixel that is even somewhat transparent.

I should be able to code that much. I'll let you see the results later.

This has nothing to do with not transparent enough. The alpha value is correct, the problem has to do with the RGB values. For example red solid 0xFFFF0000 mixs with completely transparent with equal weight, in their code the RGB values of the transparent pixel changes the RGB values of final result, which is incorrect and gives the outline artifact.
Transparent white pixel = 0x00FFFFFF result = 0x7FFF7F7F half transparent pink
Transparent black pixel = 0x00000000 result = 0x7F7F0000 half transparent dark red

Using the correct calculation does not change the RGB values because the alpha value will remove the effect.
Transparent white pixel = 0x00FFFFFF result = 0x7FFF0000 half transparent red
Transparent black pixel = 0x00000000 result = 0x7FFF0000 half transparent red

Try this:
http://pastie.org/9581143

I'm interested in this because if I ever add a filter to my emulator, it must be layer based. But since some games use the sprite layer to add colours to the background colour, I'm not sure if this will break those games.

What happens if you treat all transparent colors as transparent black and then use premultiplied alpha?

In that case it will give the correct result:
A = (A1 + A2) / 2
R = (R1 * A1 + R2 * A2) / (A1 + A2) = (pR1 + pR2) / (A1 + A2)
G = (G1 * A1 + G2 * A2) / (A1 + A2) = (pG1 + pG2) / (A1 + A2)
B = (B1 * A1 + B2 * A2) / (A1 + A2) = (pB1 + pB2) / (A1 + A2)
pR = R * A = (pR1 + pR2) / 2
pG = G * A = (pG1 + pG2) / 2
pB = B * A = (pB1 + pB2) / 2

@mkwong98 - The Pastie you posted doesn't compile (the pointers are new code since it changes some function definitions) so I couldn't copy+paste the entire common.h you linked to, but just taking the code inside Interpolate_2() and Interpolate_3() compiles just fine.

This is what I used: http://pastie.org/9581330

Unfortunately it doesn't work, but not for the reasons you might expect. Rendering the Shantae BG gives me this weird result: http://i4.minus.com/ibqBHPMnS6zqoU.png

I must admit, that's a very cool looking effect, and I think you've discovered how to convert HQx into a neon filter The results when trying to run HQx on Shantae herself (fully transparent BG, source: http://i1.minus.com/ibghiiXJ8tJ1ZX.png) just throws up a Floating Point Exception.

EDIT - Hold on something isn't right, let me try to figure it out... Be back in a few minutes... Okay, you're not using 1.1, I'm using the latest stable release of HQx, let me update the SVN code...
DOUBLE EDIT - Still no good, even using the latest HQx code from Google Code. Results above are the same.



I haven't gone over your code in GitHub yet, but if I may ask, why don't you render everything to a final screen buffer and perform operations on it from there?

Currently my emulator will compose each layer directly to high resolution with parts replaced with custom HD graphics, so there is no final screen with native resolution for the scaling algorithm to work with. But I think I can render each layer with native resolution and scale it up with the scaling algorithm before adding the custom HD graphics. In that case, I have to do it by layer.

I think my code has a div by 0 bug when aSum is 0.

Yeah, it looks like aSum could be zero in some cases, and there's no check to stop the division. I still can't figure out why I get the neon colors I referred to earlier. It's almost like anything that isn't edge detected is filled with black.

Anyway, what I said earlier about the pixels not being transparent enough is that when I render Shantae's sprite layer with a completely transparent BG using HQ4x, the problematic pixels in her outline look as if they're too opaque when placed on top of the BG layer. That is to say, when her transparent outline sits on top of the BG, it looks like the transparent pixels HQx generates have alpha values that are too high. If I manually lower the alpha values in an image editor, it looks fine to my eye, and the weird outline begins to go away. That is why I think it may be worth pursuing this angle.

For fun, I arbitrarily told HQx to halve the alpha values of any output pixel that wasn't fully opaque. The results do away with the lines somewhat, at the cost of sharpening the outlines slightly (contrary to what HQx was meant to achieve) and doing other things like making her left shoulder oddly slimmer compared to HQ4x used on the entire frame. It's still not as good in my opinion in terms of image quality when compared to using HQx on the full frame simply because sprites are still pixelated around the edges

It's a start however. If the alpha values could be tweaked correctly (rather than by dividing by two every time, as in my code) then this might be one way forward. The code is based on HQx 1.1, but the parts that are changed should work (copy+paste) with the last SVN commit.

Results:
http://i1.minus.com/ihcygM6eSKvmr.png - HQ4x on full frame
http://i4.minus.com/ixpbH1QD6ygcr.png - HQ4x on separate layers (Original issue)
http://i2.minus.com/iDPFWdt7bLx0L.png - HQ4x modified on separate layers (Possible progress)

Input (for those wishing to recreate this):
http://i1.minus.com/iTb5x10Jxck6j.png - Shantae BG layer
http://i3.minus.com/iUvqP84v2Zp15.png - Shantae Sprite layer

Code:
http://pastie.org/9581763

The other problem is that Interpolate_2 and Interpolate_3 provide for weighted interpolations (the w1, w2, w3 and s arguments), and mkwong's sample code only does 50% weights.

Speaking of issues with using HQx on separate layers, the slimming of Shantae's shoulders (and the upper-left part of her hair) is annoying, and I don't think it's related to the problem of transparency. If you look at her hair specifically, it keeps shaping it into something like a staircase. The same thing happens with her shoulder. It's quite angular (and looks unintended, or at least unappealing to me). Even accounting for the problematic transparent pixels, HQx just seems to be behaving on Shantae differently when scaling full frame vs. separate layers. I'm not familiar with the innards of HQx, so I can't say why that's happening, code-wise.

Wait a second, I thought we were talking about algorithms that didn't do any sort of antialiasing in the first place? You're supposed to render each plane separately, then merge the result, then stretch on screen (so ultimately the sprite layer still filters against the background, but without the scaler issues).

Hqx family does antialiasing. Scale2x does not; it always uses the existing pixel values.

I think I know what is wrong, the right shift fills the leading bits with 1 and I forgot about it. This should do the trick:
http://pastie.org/9582619

The result of C's >> operator on a signed value is "implementation-defined", which is standards-speak for "if you compile this on a different compiler from the one that the original developer used, you might get screwed". Portable code shifts only unsigned values.

Most major compilers today should interpret a signed right shift as an arithmetic right shift. It's still very much implementation defined, but as far as I know, it's a pretty wide-spread one if you factor in Visual Studio, GCC, ICC (Intel), and MinGW, at least as it concerns x86 platforms. So if we're all running one of those (likely the case) we should be fine (fingers crossed). All other bets are off though unless you can verify it yourself. But that's beside the question, mkwong98, unsigned shifts in C/C++ are not implementation defined (except in one case, read on); they're regular old logical shifts so you'd fill in the bits with 0s, not 1s. However, if the operand of the logical shift is greater than or equal to the length in bits of the number being shifted, that will result in implementation defined behavior too.

Even so, I haven't seen any code posted here using anything but unsigned ints for the relevant HQx code, unless I missed something, I don't think it's an issue with signed shifts. All of your shifts look to be in order though in regards to C/C++ rules; no signed shifts, no shifts greater than or equal to 32 bits. One thing I noted, you declared some additional masks (MASK_1, and MASK 3), so I assume they mask 0x000000FF and 0x00FF0000 respectively, correct? I made these masks myself to get it to compile. If the masks are not correct, let me know. Here are the results.

It looks like the code you posted, mkwong98, completely does away with the alpha values HQx generates. While this solves the outline issue, it makes everything sharper (and more pixelated). On closer inspection, it eliminates all anti-aliasing. No new colors are introduced, and the image essentially works like an improved version of EPX/ScaleX (which is very interesting in and of itself, I'm keeping this for reference ). Anyway, it's still not HQx, and there are still parts of Shantae's outline that slim down (top left border of her hair, left shoulders, other places too). I'm really starting to think that these "angles" are an inherent problem with HQx when rendering things as separate layers, and I don't know how to avoid them at this time. It's probably something to do with HQx's pattern recognition algorithm. Picture time:

Results:
http://i2.minus.com/iCNZokDZ1NP0T.png - Modified HQx, BG+Sprites layers
http://i2.minus.com/isKPFdQI73crO.png - Modified HQx, Sprite layer only (for reference on how this version of HQx handles separate layers).

MASK_1 is 0x00FF0000
MASK_3 is 0x000000FF
Thanks.

Are you it isn't this instead? ->
MASK_1 = 0x000000FF
MASK_3 = 0x00FF0000

If I change my masks to be your masks, I get the neon effects again:

http://i3.minus.com/ibvOwLFKndHpjy.png

Sorry, missed some brackets
http://pastie.org/9582774

This use the MASK_1 and MASK_3 that I gave you. Thanks.

Using the new code you posted + the masks you used : http://i5.minus.com/iuEGTvdXO7gc1.png

Closer but still neon-like, unfortunately.

Sorry for all those bugs. I forgot that take account that dividing by aSum has the hidden effect of right shift by s, so I need to take those out
http://pastie.org/9582827

Much better : http://i2.minus.com/ixSEX35IudDi4.png

So far, the best attempt yet. The weird angling on her hair and shoulders are not nearly as noticeable as before (though I can still see it). The only other thing that comes to mind is that the anti-aliasing when doing it per-layer isn't as strong as doing it per-frame.

Thank you!
Enlarged view of the shoulder area with full frame on the left and layer on the right:

If there were a hybrid method, I think that would produce the best image quality in terms of anti-aliasing and color smoothing. Certain smoothing I like in both methods, but have parts the other seems to improve.

If you do implement this in your emulator, I still suggest using the GPU for HQx scaling or multithreading on the CPU. I have no idea how demanding the per-layer method will be while running in an emulator.

This image shows perfectly how filtering the whole picture at once will have the background and the sprites interfering with each other. There are several examples: the diagonal rope touching Shantae's hair at the top right corner, the ship's rail at the bottom making her hair blockier, and the piece of wood that got bent between her neck and her ear.

These distortions are bad enough in an static image, but it's probably even worse in animations, since they will change over time depending on the alignment between the layers.

@tokumaru - Those distortions aren't applicable to every scaling filter. Scale2x in per-layer and per-frame produce almost identical results. This is most likely an issue with how HQx detects patterns. We haven't even examined other methods like xBR. It's premature to jump to those conclusions, not without data (and by that I mean more data than what we produced here)

Have you ever played with HQx for extended periods of time? I can assure you these "distortions" aren't nearly as big an issue as you can imagine. I just spent the weekend playing Mednafen on my TV and computer. Everything is set to HQ4x, nothing looks horribly mangled or messed up. I'll say it again, but per-layer has its share of distortions I pointed out, which is why I said a hybrid method would be best.

I have an idea! Since NES has a very small palette and emulators use different sets of colours to display it, different emulator will show different things even with the same algorithm. So it will be more consistent to work directly with the 6bit values or values with 2 components than to work with the 32bit colours. It will be easy to decide whether two colours are close to each other by checking whether they are right next to each other in the palette. And for algorithms which interpolate 2 colours, we can partially precalculate them and use lookup tables.