The Quest for the Perfect Palette Generator

This is an archive of a topic from NESdev BBS, taken in mid-October 2019 before a server upgrade.
View original topic
The Quest for the Perfect Palette Generator
by on (#129280)
Update (11/3/2018):
Now available on GitHub.

------
Since I recently had to update my html5 nes palette generator to make newer versions of Firefox accept decimal values in the number fields, I kinda pondered over this problem again.

Right now, I've hit a wall. I cannot go any further, and googling turns up absolutely zero help, and after a year and a half, I've forgotten about any suggestions anyone told me, so here's what I know so far in hopes that maybe some day I might be able to solve this problem either directly or indirectly:

Televisions take a YIQ color signal. Y represents luminance (the black and white portion of the color), and IQ represent the hue and chroma (I'm not keen on how "chroma" is different from "saturation" but that's aside the point), which is then added on top of the black and white.

I made the assumption that the CRT television takes the YIQ information and converts it to RGB. This is not sRGB (what your computer display uses) however, but rather "how much energy to hit the red phosphor, the green phosphor, and the blue phosphor with". I also made the assumption that the television clamps negative R, G, and B values to 0, but doesn't clamp positive values to anything specific, probably just analog voltage limits.

This means that a CRT television taking in an analog signal has the capability of displaying colors that are not displayable on a computer monitor. For the NES specifically, these are many of the bright blue colors. You've probably seen the sky in SMB represented by a myriad of different bluish and purplish colors. The truth is, none of those colors are correct because that blue is actually one of the out-of-range colors I just mentioned.

I tried a bunch of different methods to generate a palette. The one I came up with that almost works is to partially convert the YIQ to RGB. Basically you take the YIQ->RGB matrix and remove Y from it. In my code, I call this tR, tG, and tB for some reason. Then, using CIELuv, I take the NTSC CIE definitions of what color the red, green, and blue phosphors are supposed to be, and treat them as vectors pointing away from the white point (D65 is the white point I'm using). I then multiply these vectors by tR, tG, and tB, and then I add them together, which gives me a point that represents the final color, which I then convert from CIELuv to CIEXYZ (this is where I use the Y that I didn't add to tR, tG, and tB), and from CIEXYZ to sRGB. If select the CIELuv option in my palette generator, you can see the result of this.

It's pretty close, but here's the wall I keep running into. I need to apply gamma correction, and it's not as simple as you'd think. Darker colors mix differently from brighter colors: the colors shift towards primary colors, so they barely mix at all. In my palette generator, the top row is supposed to look like three blues (just straight blue, the only difference is lightness, not hue), then magenta, then three reds (same deal as the blues), then super-dark brown (darkest non-black color in the palette actually), then three greens (again, straight greens, only difference is lightness), and then blue again, and the reason is because that's how it looks on my CRT TV. What you actually see in my palette generator is the full rainbow, which is not what I just described.

I absolutely cannot get this to work, so I figured I should try out a new method: I was going to convert YIQ completely to RGB, then mix R, G, and B in CIELuv space, using NTSC's definitions of red, green, blue, and white, and then convert that to CIEXYZ and then sRGB. When I tried this, the palette was skewed badly, so I probably didn't mix the colors right.

CIELuv is attractive because it's a linearized CIEXYZ. That means, if you pick two colors on the CIELuv map, and draw a straight line between them, the line represents all the colors you can create if you have two colored lights of those two colors. If you have three colored lights, I'm not quite sure what to do, because that makes a triangle, but the center point of the triangle is not necessarily white, which is why you have to use a white point, and a white point just means "what color should be created if all three lights have equal power?"

Though, in this case, my lights are actually phosphors being hit with an electron gun. I don't know if phosphors change color when they get brighter, which would be something to consider. However, this seems to be the way to go because I can gamma correct the R, G, and B signal, which simulates the gamma curve the phosphors have, and by mixing them in CIELuv space, I can simulate how our eyes percieve these colors when they mix on the TV screen. However, I don't know how to properly mix three colors together in CIELuv space, and even if I got it, I don't know if adding the gamma correction would solve the problem with the darker colors not being correct.

And that's why I gave up way back then; I simply do not have enough information to continue, and obtaining said information seems to be more of a chore than I have the will for, since I seem to be the only person on Earth trying to do this. :\
Re: The Quest for the Perfect Palette Generator
by on (#129282)
Two things that might help:

* The vast majority of phosphors don't change wavelength as a function of radiation intensity (especially not the ones standardized in televisions; wikipedia says they use(d?) the P22 trio).

* The gamma correction done by the television was actually an intrinsic property of the amplifier and voltage on the control grid changing the electron current density. (This is basically a triode vacuum tube with extra stuff afterwards). As such, we're doomed to an eternity of gamma correcting things with a gamma of 2.2 or so because including the gamma correction in the end users' sets in 1950 would have made them even more ludicrously expensive.
Re: The Quest for the Perfect Palette Generator
by on (#129307)
Thanks, knowing that the color primaries are constant removes a bit of headache.

Also, I'm pretty sure I have it right, but just to double check, the 2.2 gamma is the curve that the phosphors have, correct? I.e., an actual "gamma-corrected" broadcast uses a gamma of 1/2.2 to compensate, right?
Re: The Quest for the Perfect Palette Generator
by on (#129308)
Drag wrote:
the 2.2 gamma is the curve that the phosphors have, correct?
Not just the phosphors. It's also a function of how CRTs work. ( https://en.wikipedia.org/wiki/File:Trio ... istic1.png )

I'm under the impression that the phosphors themselves are a mostly-linear "N electrons go in, N÷k photons come out".
Quote:
I.e., an actual "gamma-corrected" broadcast uses a gamma of 1/2.2 to compensate, right?
Yes, this is true.
Re: The Quest for the Perfect Palette Generator
by on (#129310)
I feel like you're making this more complicated than it needs to be.

Conversion from YUV/YIQ to the TV-standard RGB is extremely simple and easy to look up. The only catch is that, in the NES, Y, R, G, and B can all be negative. (For your purposes, YUV and YIQ are interchangeable.)

I think all you need to do from there is convert from TV RGB to sRGB using the equations provided in this book.
Re: The Quest for the Perfect Palette Generator
by on (#129316)
That's a cool book that I might actually get at some point, thanks. :D

Well, the reason I wanted to mix R, G, and B in CIELuv space is because the NTSC's definition of RGB is different from sRGB. Therefore, colors combine differently when you're using sRGB vs NTSC. CIELuv space is as low-level as I can get; I literally am emulating 3 colors of phosphors (red, green, and blue as defined by NTSC) whose light is mixing the way our eyes see them, and then I'm converting them back to sRGB. Converting YIQ directly to RGB produces an okay palette, but when I started using CIELuv, the colors looked a LOT closer to what my TV produces. You could argue up and down to me that it doesn't matter, but my eyes disagree. :P

Anyway, I have the color mixing working now (I indeed was mixing incorrectly), but I have one more problem.

R, G, and B are the YIQ -> RGB values, negative numbers clamped to 0, positive numbers unclamped.
Rp, Gp, and Bp are the red, green, and blue primaries as defined by the NTSC.

Here's how I'm mixing:
Code:
  R             G             B
-----Rp   +   -----Gp   +   -----Bp
R+G+B         R+G+B         R+G+B

I'm using that formula for the X coordinate and the Y coordinate.

So, RpGpBp is a triangle, and R, G, and B represent how close to Rp, Gp, or Bp the resulting color is. When R, G, and B are equal in value, the resulting color is in the exact center of the triangle. The problem is, the white point is not in the center of the triangle.

Fortunately, I have the math set up so that the white point is (0,0), which is a point that is off-center. I need to figure out how to add three colors in CIELuv space, while taking the white point into account.

When R, G, and B are equal, R/(R+G+B), G/(R+G+B), and B/(R+G+B) are all equal to 1/3. If I subtract 1/3 from there, I can shift the colors so that unsaturated colors indeed hit the white point, and not the center, and the resulting palette looks good, but I know this isn't the correct way to do white point correction.
Re: The Quest for the Perfect Palette Generator
by on (#129317)
Just as a heads up, I've updated the palette generator to show off what I have so far.

The older version with the junky options I removed is still available here.
Re: The Quest for the Perfect Palette Generator
by on (#129319)
Is there any way this could output a binary file (like FCEUX's palette files), or even just a more easily parsed text version? It would be nice if this was just a little easier to drop into an emulator to try out.
Re: The Quest for the Perfect Palette Generator
by on (#129332)
I will once I can figure out how to do it in HTML5, and once I create a good default palette preset.

I'm still working on the white point correction. Instead of subtracting (1/3) from the normalized R, G, and B, I can apply a gamma curve to them, and if I manually tweak it, I can kinda "bend" the inside of the triangle so the midpoint is where I want it, but I still don't know if this is the correct way to perform an additive mix on 3 colors in CIELuv with respect to a white point.
Re: The Quest for the Perfect Palette Generator
by on (#129338)
Drag wrote:
I will once I can figure out how to do it in HTML5

have a link element with a download attribute containing the file name and the href attribute formated as a data uri for the file data.
Re: The Quest for the Perfect Palette Generator
by on (#129343)
Drag wrote:
Well, the reason I wanted to mix R, G, and B in CIELuv space is because the NTSC's definition of RGB is different from sRGB.
That's why I linked directly to the two pages in the book that describe converting between NTSC's RGB and sRGB, using CIEXYZ (which is one mathematical transform away from CIELuv). Hopefully you can see them okay, it looks like Google has started hiding some pages.

I don't think you need to correct for the white point at all - if NTSC and sRGB specify different white points, then shouldn't "white" in NTSC be different from "white" in sRGB?
Re: The Quest for the Perfect Palette Generator
by on (#129346)
Joe wrote:
Drag wrote:
Well, the reason I wanted to mix R, G, and B in CIELuv space is because the NTSC's definition of RGB is different from sRGB.
That's why I linked directly to the two pages in the book that describe converting between NTSC's RGB and sRGB, using CIEXYZ (which is one mathematical transform away from CIELuv). Hopefully you can see them okay, it looks like Google has started hiding some pages.

I don't think you need to correct for the white point at all - if NTSC and sRGB specify different white points, then shouldn't "white" in NTSC be different from "white" in sRGB?

Yeah, I started thinking about that. I was convinced that mixing the colors manually in CIELuv was the way to go, but after trying to solve this white point problem and seeing how you account for white points in RGB->XYZ matrices, I'm beginning to think that you're right. My new plan of attack later tonight will be to generate the RGB->XYZ matrix (so you can specify different white points and primaries), and run the YIQ->RGB output into that, and then convert CIEXYZ back to sRGB using a fixed matrix. The idea is still the same; generate the color using NTSC's definition of red, green, blue, and white, rather than sRGB's. I'll see if that gives me any better results.

At some point, I'll add the download link now that I know it's easy. :P What's FCEUX's palette file format again? 64 RGB definitions in palette order? Or is it BGR or something?
Re: The Quest for the Perfect Palette Generator
by on (#129360)
http://www.fceux.com/web/help/fceux.html?Palette.html
"Palette files are expected to contain 64 8-bit RGB triplets (each in that order; red comes first in the triplet in the file, then green, then blue)."
Re: The Quest for the Perfect Palette Generator
by on (#129382)
I eventually figured out how to do the white point correction using my CIELuv method. I basically had to multiply R, G, and B by weights before the R/(R+G+B) etc calculations. The weights had to be between [0..1], and if I manually tweaked them, I could get the white point correct.

The only problem is that YIQ->RGB->CIEXYZ(ntsc)->sRGB produces identical results, with a lot less math and manual intervention. :P So you all were right, I was overcomplicating it by manually mixing the colors in CIELuv.

Ok, so what I have now is a palette generator, which generates the color in an RGB space defined by the CIExy points for the red, green, and blue primaries and the white point, and then converts it to sRGB so your computer can display it. If I can dig up the PAL definitions of RGBW and add YUV decoding, I can hypothetically create a PAL palette generator too, but I'd have no way to compare it against a real TV.

Anyway, I'm going to tidy my app's code up, and strip some now-unneeded stuff, and then I'll upload the updated version, in which I'm also planning to add a download button. :P
Re: The Quest for the Perfect Palette Generator
by on (#129383)
There's some weirdness with exactly how the 2C07 generates colors, from my experiment (with thefox's help) over here.

I don't know how we're going to figure out better what's going on without a lot of time with a PAL NES and an oscilloscope, and possibly also changing the clock frequency from 26MHz to something slower.
Re: The Quest for the Perfect Palette Generator
by on (#129389)
Drag wrote:
If I can dig up the PAL definitions of RGBW and add YUV decoding, I can hypothetically create a PAL palette generator too, but I'd have no way to compare it against a real TV.
The PAL definitions (among many others) are helpfully included in the H.264 specification, annex E, section E.2. Table E-3 contains the RGBW coordinates, table E-4 contains the gamma conversion functions, and table E-5 contains the YUV/RGB transformation coefficients. Of course, you have to know which version of NTSC/PAL you're implementing, since many of those values were redefined over the years... (edit: these may also be helpful.)

I'm kind of surprised you don't already have YUV decoding as an option; most analog (and all digital) NTSC televisions use YUV decoders.
Re: The Quest for the Perfect Palette Generator
by on (#129391)
I think for solid colors, YUV and YIQ decoding are equivalent.
Re: The Quest for the Perfect Palette Generator
by on (#129433)
Palette generator updated. Now includes the ability to save your palette to a file, hopefully.

Older version with the stuff I removed in the newer version is still available here for ~historical~ purposes. :D
Re: The Quest for the Perfect Palette Generator
by on (#129456)
After playing around with this a bit, I found that the default settings, but with sat = 0.9, bri = -0.18, and con = 1.18, gives me a palette that's extremely close to my TV, just with less saturation in certain colors (but that's really unavoidable).

I dunno, try it out, this works for me, but may not work for everyone else. :P

You actually don't need to touch the gam setting, it looks like the gamma is somehow correct even without adjusting it.

Edit: Updated again, there was a small bug with the contrast setting.
Re: The Quest for the Perfect Palette Generator
by on (#129476)
Another update.

I found out that one of my older color clipping methods actually matched my TV a lot better than the desaturate setting. I also needed to apply some kind of clipping to the RGB that comes out of the YIQ->RGB equation because having an unbounded positive range is technically incorrect. Either way, with these changes, the setting {darken, -0.25 hue, 0.7 sat, -0.2 bri, 1.2 con, 1.0 gam, fcc} looks nearly identical to my TV, especially since the colors don't wash out now. The best example of this is the SMB title screen, on my TV, the sky is a lot darker than the white text, although much more colorful.

Edit: Just needed one more tweak to the hue setting.
Re: The Quest for the Perfect Palette Generator
by on (#129479)
Real cool.
Re: The Quest for the Perfect Palette Generator
by on (#129491)
Just wanted to say I appreciate the work you're doing on this! :mrgreen:
Re: The Quest for the Perfect Palette Generator
by on (#129499)
Thanks guys!

I added another colorimetry option where it's FCC's color primaries and sRGB's white point. This seems to be a closer match for my TV. I'm torn between darken and desaturate; darken preserves the hue, but the relationship between the colors is wonky. desaturate preserves contrast at the expense of lightening certain colors. I guess without any way to literally make the colors more vibrant, this will have to do.
Re: The Quest for the Perfect Palette Generator
by on (#129502)
One last update for tonight, you can toggle the text on and off, and I changed the default parameters to the ones that work best for me personally, just so people don't blindly open up my palette generator and use the godawful baseline parameters thinking it's a good palette. :P

Do RGB monitors (like ones in coin-ops) use similar primaries to the FCC spec, or something different? I was just thinking that I could use the same math to generate RGB palettes, or LUTs for them anyway. Hell, this is even one tweak away from also generating Atari 2600 palettes, hypothetically. :P
Re: The Quest for the Perfect Palette Generator
by on (#129538)
Just made some UI updates, you can hide the CIE graph if it takes up too much space, and now there are +/- buttons beside the tuning controls.
Re: The Quest for the Perfect Palette Generator
by on (#129539)
Drag wrote:
Do RGB monitors (like ones in coin-ops) use similar primaries to the FCC spec, or something different?
Yeah, it's the same phosphors as an ordinary television, so the X,Y should be the same.
Re: The Quest for the Perfect Palette Generator
by on (#129550)
Perfect. Thank you! :D
Re: The Quest for the Perfect Palette Generator
by on (#129552)
I have a couple of suggestions now:
{-0.25, 0.7, -0.2, 1.2, 1.0} (default settings when you first open the palette generator)
This is very very close to my own TV's output. This is pretty good for just playing games and every day casual use.

{-0.25, 0.8, -0.2, 1.0, 1.0}
This is a slightly darker palette, but represents the colors a lot more faithfully, for instance, I can actually tell what hues the lightest colors are, just like I can with my TV. If you're looking to design some graphics and you really need an accurate representation of the colors and how they contrast against each other, this is what I'd recommend. Even though I said this palette is darker, you don't even notice unless you have something BRIGHT WHITE open next to your emulator.

{0, 0.7, -0.1, 1.1, 1.0}
This is something BMF came up with when he compared against an old 1982 composite monitor. Since said monitor is so old and designed for computers and not for TV signals, you could argue that it follows the FCC standard a bit more strictly, which is evidenced by the fact that a hue adjustment of 0 looked correct.

Finally, I think my TV does some kind of flesh tone boosting. Not only do I need a -0.25 hue tweak to get the colors to look right, a lot of the non-green/blue colors are really warm and reddish.
Re: The Quest for the Perfect Palette Generator
by on (#129787)
You can now specify a custom colorimetry instead of the four that I put in. Changing the color primaries doesn't appear to be as useful as changing the white point is. I think my TV uses a different white point because gray looks different compared to what gray looks like on my screen here. Specifically, gray on my screen looks bluer than gray on my TV, which looks more yellowy. Using D55 seems to be an improvement, but I don't want to say anything until I actually compare some games. I'm pretty sure the white point varies between TVs because I also have a small Sony Trinitron that is extremely yellow, like warmer than D50.

By the way, changing the white point between D50, D55, D65, and D75 is kinda like changing the color temperature on your monitor.

Edit: I just checked, D55 indeed gives me some stronger reds and better grays that aren't as bluish. Also, in case anyone has no clue what I'm talking about, I'm using the coordinates from here for the white points. C is the standard for the original FCC, D65 is what everyone uses nowadays.
Re: The Quest for the Perfect Palette Generator
by on (#130690)
This is all very interesting drag! Thanks for everything you have put in.

One question, i tried your settings of:

{-0.25, 0.8, -0.2, 1.0, 1.0}
This is a slightly darker palette, but represents the colors a lot more faithfully, for instance, I can actually tell what hues the lightest colors are, just like I can with my TV. If you're looking to design some graphics and you really need an accurate representation of the colors and how they contrast against each other, this is what I'd recommend. Even though I said this palette is darker, you don't even notice unless you have something BRIGHT WHITE open next to your emulator.

i tried using this palette on my tv, as i play through a console emulator, and was wondering, the saturation seemed perfect to me with this setting, but it was very very dark on the tv. Anything i can try to have some sort of heavy saturation with the darker colors i remember nes having, but in some way be able to be able to have everything a little brighter, it was just a little too dark and some things on the screen were hard to see.

anyways, great work man! :beer:
Re: The Quest for the Perfect Palette Generator
by on (#130697)
How do the default settings look? The darker palette just has a lower contrast setting (and a slightly higher saturation setting to compensate).
Re: The Quest for the Perfect Palette Generator
by on (#130720)
Drag wrote:
How do the default settings look? The darker palette just has a lower contrast setting (and a slightly higher saturation setting to compensate).



I think they look pretty good. Its close, closer than any other palette i have used. If you are using this palette on a tv instead of a monitor, are there any options you would recommend?

Thanks again for your efforts to the perfect palette! :beer:
Re: The Quest for the Perfect Palette Generator
by on (#130727)
No problem, I'm glad someone else is finally getting use out of it. :P

It really depends on the TV; if it's an LCD TV, then my palette generator with the default settings ought to suffice, though I've discovered that a white point of D55 (0.3324, 0.3474) looks even closer to what my CRT TV displays. To input that into the palette generator, click on "get current preset" and put that coordinate in for Wx and Wy. For you, it might look better or might look worse, this is just what worked best for me.

If you're using a CRT TV with your emulator, I'm not sure what to recommend, because I haven't tested that. I guess it comes down to how warm or cool the grays are.
Re: The Quest for the Perfect Palette Generator
by on (#131096)
sorry for the long delay in my reply. I am using mainly an hd lcd tv. I found that bumping up the brightness on the settings in the emu by about 10 seem to get it pretty good.

Great work on everything sir! :beer: The nes palette has been a long standing discussion (that often) seems to have no true and definitive answer lol.
Re: The Quest for the Perfect Palette Generator
by on (#131967)
A few questions, if I may, from someone who has spent way too much time on the NES palette himself:
  1. How are you normalizing video levels? Are you just treating color 0x0E as black and colors 0x20 and colors 0x30 as white?
  2. Are you using the color burst amplitude as an amplitude reference for the chroma signals?
  3. What does a hue of -0.25 actually mean? How many degrees is that? I notice that 360 degrees (meaning no change) is at about 31.4, so it seems to be related to radians. Better specify it in degrees, because that is what the television literature does.
  4. Are you taking US NTSC's 7.5% setup into account? It seems you do, given that the default brightness and contrast settings are different from zero, though to a larger extent than 7.5% setup would do.

Suggestions:
  1. I assume that "colorimetry" refers to the colorimetry of the emulated ("source") display, and that the target display is always sRGB. Since wide-gamut displays are quite widespread these days, allowing to select the target display colorimetry might be desirable as well.
  2. There are at least two more white points that were (and to some extent are) in widespread use: 9300K+27MPCD (xWhite 0.281, yWhite 0.311) and CIE D93 (xWhite 0.285, yWhite 0.293). Should you decide to add these, you might want to separate the RGB primaries from the white point. Note that when trying to simulate D93 white on sRGB, the blue value will become greater than 1, so everything must be scaled accordingly. Right now that is not done when entering these as custom values.
  3. I understand you are using the default YIQ/YUV to RGB matrix. Using this matrix together with FCC primaries will reflect what an idealized NTSC monitor would show, but not necessarily the actual behavior of an 1980s television set. They would instead use a modified YIQ/YUV-to-RGB matrix with custom phosphors. These can be simulated perfectly if the values are known, which they are to some extent.
    I don't know how deeply you want to dig into this, but given the work you have put into this already, you might want to consider the following two documents:
    Neal, C.B., "Television Colorimetry for Receiver Engineers," IEEE Transactions on Broadcast and Television Receivers, vol.BTR-19, no.3, pp.149,162, Aug. 1973 This documents contains both typical primaries and the matching YUV-to-RGB conversion matrices for 1970s televisions, good enough for NES purposes. The method of creating modified YUV-to-RGB conversion matrices for any set of primaries is described in this document: Parker, N.W., "An Analysis of the Necessary Decoder Corrections for Color Receiver Operation with Non-Standard Receiver Primaries," IEEE Transactions on Broadcast and Television Receivers, vol.12, no.1, pp.23,32, April 1966.
Re: The Quest for the Perfect Palette Generator
by on (#131985)
NewRisingSun wrote:
How are you normalizing video levels? Are you just treating color 0x0E as black and colors 0x20 and colors 0x30 as white?

I'm using the normalized values from here, which indeed normalize as you say.
Quote:
Are you using the color burst amplitude as an amplitude reference for the chroma signals?

I'm not, because I've read documentation somewhere that says the color burst amplitude has absolutely no effect on the resulting picture. Whether or not that's true seems to depend on who you ask, so I don't have a clear answer as to what to do, nor if any kind of DC bias on the colorburst signal makes any difference.
Quote:
What does a hue of -0.25 actually mean? How many degrees is that? I notice that 360 degrees (meaning no change) is at about 31.4, so it seems to be related to radians. Better specify it in degrees, because that is what the television literature does.

Yeah, the hue tweak is in radians, and that's just because that's how most programming languages handle trig functions. I probably should change it to degrees, now that you mention it, if only because radians kinda suck. :P
Quote:
Are you taking US NTSC's 7.5% setup into account? It seems you do, given that the default brightness and contrast settings are different from zero, though to a larger extent than 7.5% setup would do.

I'm not actually sure what that is, to be honest. The reason the brightness is lower is because I noticed the darkest row of colors didn't look right until I lowered it, and when raising the brightness on my TV, it takes a while for 1D to actually start lightening, so I figured the brightness setting is actually supposed to be lowered. The contrast setting is to compensate for the reduced brightness.


Quote:
I assume that "colorimetry" refers to the colorimetry of the emulated ("source") display, and that the target display is always sRGB. Since wide-gamut displays are quite widespread these days, allowing to select the target display colorimetry might be desirable as well.

Your assumption is correct. I can add the target colorimetry at some point, but I'd be doing it without understanding wide gamut displays nor why it's necessary.
Quote:
There are at least two more white points that were (and to some extent are) in widespread use: 9300K+27MPCD (xWhite 0.281, yWhite 0.311) and CIE D93 (xWhite 0.285, yWhite 0.293). Should you decide to add these, you might want to separate the RGB primaries from the white point. Note that when trying to simulate D93 white on sRGB, the blue value will become greater than 1, so everything must be scaled accordingly. Right now that is not done when entering these as custom values.

I started thinking this was the case, given that neither C nor D65 looked correct for me. The custom colorimetry was a quick afterthought, so it doesn't do any scaling or anything like that. I'm not even sure how I'd need to properly "scale" anything.
Quote:
I understand you are using the default YIQ/YUV to RGB matrix. Using this matrix together with FCC primaries will reflect what an idealized NTSC monitor would show, but not necessarily the actual behavior of an 1980s television set. They would instead use a modified YIQ/YUV-to-RGB matrix with custom phosphors. These can be simulated perfectly if the values are known, which they are to some extent.
I don't know how deeply you want to dig into this, but given the work you have put into this already, you might want to consider the following two documents:
Neal, C.B., "Television Colorimetry for Receiver Engineers," IEEE Transactions on Broadcast and Television Receivers, vol.BTR-19, no.3, pp.149,162, Aug. 1973 This documents contains both typical primaries and the matching YUV-to-RGB conversion matrices for 1970s televisions, good enough for NES purposes. The method of creating modified YUV-to-RGB conversion matrices for any set of primaries is described in this document: Parker, N.W., "An Analysis of the Necessary Decoder Corrections for Color Receiver Operation with Non-Standard Receiver Primaries," IEEE Transactions on Broadcast and Television Receivers, vol.12, no.1, pp.23,32, April 1966.

Thanks for the literature! I knew the YIQ->RGB matricies had to be different from what the FCC specified, if only because I couldn't get anything to look exactly right, I could only get it "close". :P (Trying out D55 like I mentioned before helped me, which kinda opened me to the possibility that TVs may not all be using D65)
Re: The Quest for the Perfect Palette Generator
by on (#131988)
Drag wrote:
Quote:
Are you using the color burst amplitude as an amplitude reference for the chroma signals?
I'm not, because I've read documentation somewhere that says the color burst amplitude has absolutely no effect on the resulting picture. Whether or not that's true seems to depend on who you ask, so I don't have a clear answer as to what to do, nor if any kind of DC bias on the colorburst signal makes any difference.
DC during colorburst definitely doesn't matter, but is "supposed" to be 0 IRE.
This is how Macrovision works: VCRs have AGCs, and they sample for "darkest value on a scanline" during colorburst. Macrovision adds a large positive offset during colorburst.

Atari's 2600, as initially released, specifically was designed to take the nominal chroma scaling into account by attenuating the colorburst to the nominal ~40 IRE. Later revisions removed the connection for cost savings, because it was found that basically no televisions cared. Most don't even scale luminance through composite, simply assuming that the sync depth is "good enough"; the only AGC is on OTA because it's amplitude modulated and so has to be adjusted for distance from the broadcaster.

Quote:
Quote:
Are you taking US NTSC's 7.5% setup into account? It seems you do, given that the default brightness and contrast settings are different from zero, though to a larger extent than 7.5% setup would do.

I'm not actually sure what that is, to be honest.
Nominally, US NTSC TV (but not Japanese NTSC TV) defines "black" as 7.5 IRE, and values below that as blacker-than-black. Most early video game consoles don't support this, but the end result is that a console that doesn't compensate for this in the US will have a smidge more contrast and darker than in Japan.
Re: The Quest for the Perfect Palette Generator
by on (#131993)
Oh I see. So this means that the blanking and black levels on the NES are the same?

Reducing the brightness to -0.075 instead of -0.2 doesn't look like my TV though, on which, color $08 is almost black. I guess my TV likes to make the picture even darker?
Re: The Quest for the Perfect Palette Generator
by on (#131995)
Yeah, the NES only emits one voltage—color $1D—for blanking, black, &c.

On the bright side, it's not the hack that was video for the ZX80 (which by default assumed that the TV didn't do any normalizing at all and so just emitted full white during the back porch)
Re: The Quest for the Perfect Palette Generator
by on (#131998)
Drag wrote:
I'm using the normalized values from here, which indeed normalize as you say.
The problem is that no television set will do it this way, because it has no way of knowing at what video level the console wants its black and white to be. Consider this post of mine, which might explain why you need brightness values lower than -0.075 to replicate a particular television set.
lidnariq wrote:
Atari's 2600, as initially released, specifically was designed to take the nominal chroma scaling into account by attenuating the colorburst to the nominal ~40 IRE. Later revisions removed the connection for cost savings, because it was found that basically no televisions cared.
With a baseband composite connection, my multi-standard Sony CRT uses the color burst amplitude as an amplitude reference for chroma with PAL signals and as an amplitude reference for the entire signal with NTSC signals.
lidnariq wrote:
the only AGC is on OTA because it's amplitude modulated and so has to be adjusted for distance from the broadcaster.
This is very important, and implies that the same television with the same console would produce pictures of different brightness between a baseband composite connection and an RF-modulated connection.
Drag wrote:
I can add the target colorimetry at some point, but I'd be doing it without understanding wide gamut displays nor why it's necessary.
Wide gamut displays provide a more saturated picture. Normal sRGB images will appear oversaturated on them. Most web browsers nowadays can be made to color manage these images, that is, convert their values from sRGB to the monitor's native primaries. I have not seen any NES emulator provide that functionality. Therefore, it would be useful to specify the target monitor's primaries to generate the correct colors directly. Another advantage of this is that on these monitors, saturated reds and greens can be seen without them needing to be clipped (as much).
Drag wrote:
The custom colorimetry was a quick afterthought, so it doesn't do any scaling or anything like that. I'm not even sure how I'd need to properly "scale" anything.
Convert 100% white (R=G=B=1.0) to the target colorspace, take the largest value, and divide everything else by that, i.e. if you get R=1.0, G=1.1, B=1.2, then divide everything by 1.2. All this with linear values (the ones you use for color space conversion), not gamma-correct values.

By the way: if you wanted to get really crazy, you could emulate differential phase distortion as well. Basically, your current hue setting is amplitude-independent, so it influences the hue shift at zero amplitude. You could add a second hue setting that is multiplied with Y, resulting in a total hue shift for any pixel of baseHue+Y*diffHue. While it does replicate what is definitely going on in some devices, it might be a bit far out there for emulation purposes.
Re: The Quest for the Perfect Palette Generator
by on (#132001)
NewRisingSun wrote:
This is very important, and implies that the same television with the same console would produce pictures of different brightness between a baseband composite connection and an RF-modulated connection.
Yes, but the usual AGC is on sync depth, normalizing it to -40 IRE. There's no clearly correct thing to do if given contradictory gains needed to normalize both colorburst and sync depth.
Re: The Quest for the Perfect Palette Generator
by on (#132007)
On my Magnavox CRT SDTV, I do get a noticeably brighter picture with composite out than with RF out. I noticed this when I decided to use RF out when recording composite with my DVD recorder to prevent lag during gameplay
Re: The Quest for the Perfect Palette Generator
by on (#135258)
Hey Drag,

Ive been using the settings for :

{-0.25, 0.8, -0.2, 1.0, 1.0}
This is a slightly darker palette, but represents the colors a lot more faithfully, for instance, I can actually tell what hues the lightest colors are, just like I can with my TV. If you're looking to design some graphics and you really need an accurate representation of the colors and how they contrast against each other, this is what I'd recommend. Even though I said this palette is darker, you don't even notice unless you have something BRIGHT WHITE open next to your emulator.

This palette is pretty awesome. One question, is there something i can do to make the palette retain the green color from the border on the second stage of contra for nes? It is supposed to look like this:

Image

but instead it is just one single gray color, hardly any green. When this palette is used in conjunction with the ntsc filter it really shines. Just a few little tweaks and it will be almost perfect. Just figured i would ask since you know more about the settings on the palette generator page than i do. I always remember the nes colors being dark and gritty than most emulators show. Do you know what i can do to retain this green back and possibly just lighten up things just a tiny hair?

anyways this palette generator is awesome, thanks for all you have put into it. Im definitely getting use of it.
Re: The Quest for the Perfect Palette Generator
by on (#135264)
Bumping the brightness up a little might help. The green does look a little dark in the palette, but I haven't looked at this scene on my TV so I don't know if it's dark there too.
Re: The Quest for the Perfect Palette Generator
by on (#138928)
I apologize for bumping this topic. I am sure many are aware of my open source project Retro Graphics Toolkit http://forums.nesdev.com/viewtopic.php?f=21&t=9894. If you don't mind would it be okay if I use your code for palette generation? It was easy to port to C++. I have not committed the code as I have not implemented de-emphasis bits yet which I believe to be a necessary feature. If you have any pointers on how to implement de-emphasis bits I would appreciate that greatly.
Re: The Quest for the Perfect Palette Generator
by on (#138930)
Go for it. :D

I'll need to get back to you on the emphasis bits, because I haven't fully explored that yet.
Re: The Quest for the Perfect Palette Generator
by on (#138931)
Alright I will also be working on this on the weekend hopefully.
Edit: The code for this already existed I merged two palette generators the code to generator the Y,I,Q values is now from viewtopic.php?t=8209
and I use your code to convert the Y,I,Q values to rgb. Here is the code https://github.com/ComputerNerd/Retro-G ... nespal.cpp
Re: The Quest for the Perfect Palette Generator
by on (#215387)
Sorry for the resurrection. I just want to add my two cents. I think the palette becomes more balanced once you specify a 9300k white point per NTSC-J spec:
Image
I say this despire not being a fan of non-D65 white points. There's no denying that the palette was designed around 9300k and might as well be simulated for NES content IMO. Additionally my eyes agree with @lidnariq on a brigtness setting -0.075. Especially since it ends up with a suitable brightness when used with a scan line effect. My IPS display is calibrated to sRGB, gamma 2.3, 120cd/m2.
Re: The Quest for the Perfect Palette Generator
by on (#228409)
Hey all, I'm dropping in because my palette generator is now on GitHub.

I'll keep the version on my website current, but now there's a GitHub project page. As part of publishing it, I've cleaned up the source code, removed some redundant code, fixed variable names, and attempted to DRY some of the WET stuff (which there was lots of). Overall, I really hope this makes things easier for everyone who was interested in using or incorporating palgen. :D

Alright! Going to add a link to the project page to the first post of this thread.