In this new chapter of the NDK tutorial, we will look into op engines and pixel engines. These functions process your image data and are used in most plugins.
We will also briefly cover in_channels and _request which are useful secondary functions you use to ensure your op functions properly.
Using your combined knowledge of engines and what we learnt in parts 1 and 2, you should be able to rewrite most of Nuke’s 2D nodes by the end of this post.
For a change in pace, this will also include a video where I’ll go through the entire process of developing a PixelSorting plugin from scratch.
Iop and PixelIop
In part 2, we mostly focused on the NoIop, a subclass of the broader Iop (Image Op) class. The NoIop made things easy for us, as we did not need to modify pixels, and it copied the input pixels to the output for us.
Similarly, PixelIop offers some convenience and has re-implemented some of the functions from Iop so that we can easily make Ops that process a single pixel at the time: Read input pixel, do something to it, write output pixel.
We will generally make a PixelIop if our plugin can process each pixel one by one without dependencies: Grades, Merges, Shuffles, 2D Keyers, etc…
In some situations, the default behaviours of PixelIop may not be what we desire and falling back to a base Iop is also an option. They are not too different.
Introduction to Rows
When Nuke processes an image, it usually does not do so all at once but instead splits the image into Rows.
A row is simply a line of pixels, 1 pixel high, and the same width as the entire image.
The majority of Nuke’s 2D tools use Rows at the base unit unless the tool needs access to pixels from a different row, in which case a different type of access pattern may be preferred. PixelIop is intended to be used with rows, and today’s article is also centred around them. For tiles or render stripes, check the documentation for Iops and PlanarIops.
The pixel_engine function
One of the differences between an Iop and a PixelIop is the presence of the pixel_engine, which is only available on PixelIops.
While Pixel engines are fairly straightforward to write, their advantages over other methods are also limited. Nuke already lets us perform almost all the math we could possibly want using native nodes like merges, and add/multiply. The expression node and BlinkScript can usually bridge the gap for more advanced algorithms.
As a result, it can be difficult to come up with an example that’s both easy to understand and provides some novel algorithm, as many things have been done already.
We will implement a saturation node with custom weights as our example. You could easily make the same tool as a Gizmo or an expression node. In fact, I would highly recommend that if you need such a tool, you don’t use the NDK to make it, due to its inconveniences: per-Nuke-version / per-OS compilation and requiring installation.
Below is the same node we will make, as an expression node:
set cut_paste_input [stack 0]
push $cut_paste_input
Expression {
temp_name0 wsum
temp_expr0 "weights.r + weights.g + weights.b"
temp_name1 bright
temp_expr1 "weights.r * r + weights.g * g + weights.b * b"
temp_name2 norm
temp_expr2 "wsum ? bright / wsum : 0"
expr0 "lerp(norm, r, saturation)"
expr1 "lerp(norm, g, saturation)"
expr2 "lerp(norm, b, saturation)"
name Expression6
selected true
xpos 486
ypos -367
addUserKnob {20 User}
addUserKnob {7 saturation R 0 2}
saturation 1
addUserKnob {18 weights R 0 2}
weights {0.5 0.5 0.5}
addUserKnob {6 weigths_panelDropped l "panel dropped state" -STARTLINE +HIDDEN}
weigths_panelDropped true
}
The only real advantage of C++ over an expression for something this simple is likely the ability to break down the code into many variables and the ability to add comments to improve readability (an advantage also offered by Blinkscript).
The whole code for this tool is available here. It’s derived from the Foundry’s Saturation example. I have modified the comments to provide different information from what Foundry provided in their example. You will recognize a lot of the boilerplate code we covered for NoIop in part 2.
Let’s take a closer look at the pixel_engine signature:
void pixel_engine(const Row& in, int y, int x, int r, ChannelMask channels, Row& out) override
As this is C++, the type and order of the arguments is what defines this signature, rather than the name of the variables, therefore you may run into examples where these variables are named differently.
- const Row& in: The first thing we receive from Nuke is a reference to a Row. It’s a const, so we can’t modify it. This Row is a row from our input. PixelIop has already loaded all the values we need in that row, so we can start reading values with minimal code as long as we only request what was loaded. Here the variable is named “in“.
- int y: Nuke provides us with the Y coordinate of the current row being processed.
- int x: Often also named “l” (for left) in some examples. This is the left-most X coordinate of the pixel being processed in this row. Note that the BBox is intersected with the Region of Interest, so the value of x might be greater than the bbox you set in _validate, however it should never be lower.
- int r: This is the right-most X coordinate of the pixel to process in this row, +1. This means that by iterating over (i = x; i < r; i++) you can iterate over every pixel of the row. Just like x, it may be cropped to the ROI, and the value of r may be smaller than your bbox’s r.
- ChannelMask channels: The ChannelSet of channels to render. This will usually be an intersection of the channels you indicated you are processing in _validate, and the channels requested by your viewer / write node / nodes below. For example, if you indicate in _validate that your node processes RGB, but the next node only requests R, it could be that only R is part of this ChannelMask, depending on what Nuke has decided it needs to process.
- Row& out: A reference to the output row. Memory has already been allocated to store the pixels from x to r, for the channels in channels, you just have to write the pixels in there.
The next step of my pixel_engine is to be able to read the pixel values from the input row:
const float* rIn = in[Chan_Red] + x; const float* gIn = in[Chan_Green] + x; const float* bIn = in[Chan_Blue] + x;
Here, I build a pointer to a float for each channel I’m interested in reading.
Coming from years of Python programming, it still takes me longer than it should to understand how pointers work. The pointer itself (rIn, gIn, or bIn) numerically represents a memory address. In simpler terms, it points to where something is stored in your memory.
Here, we have a pointer to a float number, as returned by in[Chan_Red], but then we +x, what does that mean?
The Row in is an array of Channels. Each channel is an array of pixels (float values) for this row.
When accessing the pointer to the pixels (with in[the_channel]), Nuke gives us a pointer that would theoretically point at pixel 0 of the row. However, the memory address this pointer points at may not actually be allocated to your row, as your Row will only be guaranteed to be allocated between x and r. By plus-ing x to our pointer, we tell it “Move x floats further in the memory”, which makes our rIn point at the first proper pixel we’re interested in.
After that, we make output pointers:
float* rOut = out.writable(Chan_Red) + x; float* gOut = out.writable(Chan_Green) + x; float* bOut = out.writable(Chan_Blue) + x;
This is nearly similar to the input pointers. Row.writable() returns a pointer to a writable memory address. In the previous snippet, we had defined our pointers as const, as we never changed the in values, but here we will be changing the values, so we do not include the const. Again, we need to shift the pointers using + x to point at pixel x.
In the next section, I calculate some values that will be the same for the entire row. It’s not particularly optimized, I could have calculated these values as part of _validate, as these values will be the same for every pixel in the image, but it’s so fast that I didn’t bother. I won’t go in details over that part of the code, as it’s not relevant to the architecture of pixel_engine.
The next section is the juicy bit, where we really iterate over all the pixels in our Row:
// Pointer to when the loop is done:
const float* END = rIn + (r - x);
// Start the loop:
while (rIn < END) {
// Calculate the luma value using the custom weights:
// Note how we do not increment the input pointers here, as we will need the original values for the lerp.
custom_luma = n_weights[0] * *rIn + n_weights[1] * *gIn + n_weights[2] * *bIn;
// Lerp the color channels towards the luma value:
// Here we increment the input pointers, as we are done with the original values, so that next iteration we get the next pixel.
*rOut++ = lerp(custom_luma, *rIn++, saturation);
*gOut++ = lerp(custom_luma, *gIn++, saturation);
*bOut++ = lerp(custom_luma, *bIn++, saturation);
}
There are multiple ways to iterate over our pixels, here, we use a while loop, with a pointer to the pixel to the right of our last pixel to process used as a condition to break out of the loop (END pointer). We say: While you’ve not reached the end, keep iterating.
An alternative, but just as valid, option would be to use a for loop.
Within the loop, to access the pixel’s value, we ask the pointer to dereference the value, using the ‘*’ operator. For example, *rIn will give us the value of the current pixel in the red channel.
Once you’ve read the value of a pixel, you can move on to the next pixel, by incrementing your pointer using gIn++ or gIn+1.
Here we take some shortcuts and increment our pointers at the same time as we dereference them: *rOut++(assign a value then increment pointer) = lerp(custom_luma, *rIn++ (read the value then increment pointer), saturation);
Once we have assigned the output values to all pixels, this is the end of our loop and the end of our pixel engine method. We have now processed pixels!
The in_channels function
The in_channels function is a helper function specific to PixelIops.
You may have noticed that within _validate, we indicate which channels we will be outputting, as well as the BBOX we will be processing. But what if to process your values you need access to different channels from your input? Imagine writing a custom Shuffle node: The channels on the way out may not be the same as the ones from the input.
This is where in_channels comes in.
void in_channels(int input_number, ChannelSet& channels) const override
{
channels = Mask_RGB;
}
In this function, you can change the value of the channels variable to indicate which channels you need. For my saturation tool, I only ever care about RGB, so I set the value to Mask_RGB.
PixelIop will use this result when building the in Row for my pixel_engine, and I know that the channels I set here will be safe to access within my code.
The engine function
Sometimes, PixelIop will not quite do exactly what you want it to do (an example in my video below). In that situation, you can attempt to force PixelIop to behave as desired by re-implementing parts of it, or you can take a step back and use the Iop class as your base instead.
Iop does less of the initial work for you, but isn’t that far away from PixelIop.
One of the key differences is that Iop does not have a pixel_engine function, therefore you need to implement the logic within the engine function instead.
Note that PixelIop also does have an engine function, but it has a default implementation which defers the logic to pixel_engine, which is why we shouldn’t re-implement the PixelIop’s engine, and use the pixel engine instead.
The signature of engine differs slightly from the signature of pixel_engine:
void engine(int y, int x, int r, ChannelMask channels, Row& out) override
Notice that the signature contains the familiar y, x, r, channels and out, but is missing the in Row.
The pixel_engine was preparing the input row for us, but we need to prepare it ourselves here.
It takes the following 2 lines to create the in Row:
Row in(l, r); input0().get(y, l, r, channels, in);
On the first line, we create our Row, and give it the range to initialize. This creates the row object, but that row does not contain any values yet.
On the second line, we load the values from input 0 into our Row object. Note that here, channels should be the channels you require from the input, similar to what we originally had in in_channels.
Once you’ve made your input row, engine and pixel engine are essentially similar.
The _request function
The _request function is used to ensure upstream nodes have the proper pixels calculated before we use them.
For example, if we need the depth channel from the input in order to calculate an image, but the node above has not calculated it, we may obtain invalid values.
Similarly, we need to tell it which pixels to render, as we may not want to render the entire image every time.
By default, in Iop, _request asks all inputs for the same BBOX as requested by downstream nodes, and the channels requested in in_channels.
This is generally fine, but if we need to access a different range of pixels for one reason or another, we can re-implement _request with our custom behaviour.
In the script above, the user is looking at the Viewer1 node.
Viewer1, in its _request() function is calling Grade4’s request() function, which is calling its own _request(), which is calling CheckerBoard1’s request(). As CheckerBoard has no inputs, it stops the chain there. CheckerBoard will run its internal engine() to calculate the requested pixels, which in turn will be processed by the Grade’s pixel_engine() (Grade is a PixelIop), and finally be displayed by the Viewer.
Making a Pixel Sorting Plugin from scratch
I have recorded myself making a Pixel Sorting plugin (somewhat similar to Max Van Leeuwen’s pixel sorting Node) to show the entire process of making such a tool from scratch.
While the video is quite heavily edited, as I recorded it over multiple weeks and with a nearly dying microphone), the intention was to do the whole thing on camera “Live”, to show errors you’re likely going to encounter as beginner, and errors I did not anticipate. I made the plugin before recording but then deleted the code and remade it all live, which helped, as some of the issues I ran into the first time took me multiple hours to solve, and nobody wants to watch a 16h video.
Other editing include taking out a lot of “hummms” and some coughing, but I specifically made sure not to cut any mistake, as I personally find video tutorials taking you straight to the solution to be less educative than the ones breaking down the process. This came out at nearly 2 hours of content, so if you were planning to watch a movie on streaming and can’t find anything interesting, watch this instead! It won’t be entertaining but it won’t be worse than scrolling through titles for 2 hours.
Follow the code along on GitHub here.
Homework: Try to use the DrawIop
The DrawIop is very similar to PixelIop but it specializes in drawing shapes. You only get 1 channel to write into, using the draw_engine, and the DrawIop will handle adding knobs for you to set the color, opacity, etc… You can check rectangle.cpp as an example to guide you. Attempt to draw a different shape, for example, a rounded rectangle.
Here’s one entry from me for the homework: CurveRenderer
This concludes part 3 of the NDK tutorial series. We’ve covered the basics of engines and how to implement pixel engines in your plugins. Stay tuned for part 4, where we’ll delve into more advanced topics and explore additional functionalities of the NDK.
Leave A Comment