Even though it’s the compositor’s bread and butter, Keying can still be a particularly painful subject.

I’ve been pulling keys for over a decade, yet it is often scarier than comping a full CG realistic shots with projected matte paintings and volumetric smoke. One can never have too many tools on their belt for pulling a good key.

In this post, I will be going over the existing keyers very quickly (they’re not the point here) so that we have a very basic understanding of how they work, and how we can extrapolate their logic to come up with our own Keyer. I could make a gizmo that implements one version of such a new Keyer (let me know if you’d like that), but the point here is to present the concept to you so that you can customize it as per your own needs.

Existing Keyers and how they work

I can’t go in details over the math of every keyer because some of them are not publicly shared and some of them I just don’t know, however we can classify them in 3 broad categories.

Thee categories are my own, they are not an official name or anything, but they help me organize them in my mind:

1D Keyers:

These are the simplest keyers one could thing of. I call 1D Keyer any Keyer that works on a single channel. In Nuke, that would be the Keyer node, or anytime you create a custom matte by extracting a single channel and modifying it to turn it into an alpha. While multiple channels are needed to create some other channel sometimes, for example Luma, I still categorize a Luma Keyer as 1D,  as it’s still a single channel operation, just in a different colorspace.

1d keyer

Algorithmic Keyers:

Algorythmic Keyers are the bulk of the Keyers out there. They often work by comparing the values in some channel with values in other channels. A simplistic algorithm example would be: Set the alpha to black if the green value is greater than the average of the blue and red values, else set the alpha to white (or in Nuke expression: g < (r+b)/2). That is of course a pretty terrible Keyer I wrote right there, though it is one used in quite a few despill algorithms. IBK, Keylight and Flame’s master keyer are examples of such keyers. These algorithms tend to be kept secret, though you can find some of them online (Like the IBK that has been dissected by a few people).

3D Keyers:

The 3D keyers work with the idea that each color can be represented as a 3d position. Whether it uses RGB as XYZ or another mapping depends on implementations. From there, all the pixels of your image can be represented as a 3d point cloud (hey, kind of like when we made custom scopes with higx PointRender). 3D shapes are used to define the keyed areas. For example a sphere. Any point falling within the sphere would be opaque, and a point outside the sphere transparent, or vice versa. Primatte is an example of such a Keyer (See how primatte works). Mad’s Cube Keyer or Flame’s 3D keyer are other examples. Ultimatte, if I understand properly, is an Algorithmic Keyer, but the algorithm could be represented as a 3D keyers as well.

If you’re interested in learning a bit more about Keyer history, I recommend this nice (although nearly 15 years old) fxguide article: Art of Keying
As you can read, little has evolved over the last 15 years. Though it seems like Machine Learning could bring a whole new category of keyers in the near future.

Making a new Keyer

As seen above, there are 1D and 3D keyers, but there isn’t really anything out there using 2D keyers (although both what I call 1D and Algorithmic Keyers are often called 2D keyers by others, I coudn’t call them that otherwise I would be out of a name for what we’re going to look at now).

The idea of this method is to represent the image as a 2D point cloud (vs 1D points or 3D points above) and use some sort of shape to define which area should be transparent or solid, as well as softness. This is convenient because we have some great tools to define areas in 2d: Roto and Rotopaint, as well as a great tool to do 2d mapping: STMap.

Since we’re in 2D, we can now use 2 channels as our 2 axes. Which 2 you would like to use is sort of up to you, and based on your specific footage. I will be using a plate from the Open Source project Tears of Steel, which you’ve probably seem many times, and can download here: https://media.xiph.org/tearsofsteel/tearsofsteel-footage-exr/02_3c/linear_hd/
For chroma keying, two channels that make sense for this approach are the cb and cr channels of the Ycbcr colorspace, though Luma and Hue could work, or Hue and Saturation, etc..

I’m sure I’m not the first person to come up with this approach, but I can’t find it documented anywhere, so if you’ve seen it before get in touch with me I’d be curious to find its origins.

The basic implementation is rather simple: Set 2 channels as the STMap input of an STMap node, and a roto as the source.

At this point we can adjust the shape to adjust the matte, but we are sort of working blind, we can’t really tell which colors fall where in the roto, making it pretty hard to use.

We can use a node to do the reverse operation the STMap would do in order to preview which area we should roto. I adapted a blinkscript  snippet Mads posted on the Nuke forum for that purpose:

BlinkScript {
 recompileCount 1
 ProgramGroup 1
 KernelDescription "2 \"InverseSTKernel\" iterate pixelWise 79737837f0b8f5ef5ba9dd8c86c9e76f184ef0da76c94577b5431d3e3e08b063 3 \"src\" Read Point \"stmap\" Read Point \"dst\" Write Random 1 \"resolution\" Float 2 AAAAAAAAAAA= 1 \"resolution\" 2 1 0"
 kernelSource "// Original Kernel by Mads Hagbarth, modified to use STMaps by Erwan Leroy\n\nkernel InverseSTKernel : ImageComputationKernel<ePixelWise>\n\{\n  Image<eRead, eAccessPoint, eEdgeClamped> src;\n  Image<eRead, eAccessPoint, eEdgeClamped> stmap;\n  Image<eWrite, eAccessRandom> dst;\n\n  param:\n    float2 resolution;\n\n  void process() \{\n    float2 uv;\n    uv.x = stmap().x * resolution.x - 0.5f;\n    uv.y = stmap().y * resolution.y - 0.5f;\n    //always make sure that you don't write outside bounds\n    //as it will usually crash nuke.\n    if (dst.bounds.inside(uv.x,uv.y)) \{ \n      dst(uv.x,uv.y) = src(); \n    \}\n  \}\n\};"
 useGPUIfAvailable false
 rebuild ""
 InverseSTKernel_resolution {{width} {height}}
 rebuild_finalise ""
}

Plugging this to the same STMap input as the previous setup and the plate as a source gives us a 2d point cloud representation of the image in CbCr space.

Point cloud setup

Adding a reverse STMap gives us a cool point cloud to look at.

Point cloud

It becomes more obvious where to draw our roto shape now, doesn’t it?

I could go ahead and roto the green points now, however I find them all a bit too squished together for my taste. I know I’m only interested in the green points in this case, so it would be nice if I could “zoom in” on them more. Since this is all driven by our custom STMap, all we need to do to zoom in on this area is to adjust our black point and white points in red and green. I will let you play with it to see how it behaves, but here is where to put the grade:

Notice how the postage stamp shows that we zoomed in on the green dots

Now all I need to do is create a roto based on these points:

I need to set my mask to be black in the area I roto’d and white outside, so I just inverted my rotoshape. A cool thing is I could come in here and add tiny masks or paint strokes to control the value of each color. The end result, with no tweaking at all, looks like this:

Not bad for a custom key that can be built in a few minutes. The technique can also be used to achieve some rather funky results. For example below I painted colors instead of a black and white matte, to achieve a trippy look:

Arguably, this is not the prettiest image. YCbCr is a decent space to key with 2 channels, discarding the Y, and only using the Cb and Cr channels, but it’s not the most representative of an image, so here we defined colors but the luminance isn’t used at all.

Hopefully this can give ideas to a few of you, or help with keying some of these weird plates we sometimes receive.