Compositing elements with a colored background, or how to invert almost any comp operation.

The operation happening here is using 3 elements to generate a 4th: The Background (1), Foreground (2) and ForegroundAlpha (3) are assembled into the result (4)

Merge nodes are just simple math, and luckily Nuke tells us the math of every single merge mode in the node. By looking at that, it tells us that an Over is doing the following: A+B*(1-a)
Which can also be read as: Result = Foreground + Background * (1 – AlphaOfA)

What that means is that we will be able to un-comp this at the condition that we have the background. In the case of any solid color, it’s easy, just need to pick the color. In the case of something else, it’s about luck, but I had a few agencies giving me elements comped on top of a frame I had given them previously, in which case I did have the background. Without the background there is nothing more this tutorial can do for you.

Just like in middle school, to obtain our proper Foreground, we’ll need to solve a little equation:
Result = Foreground + Background * (1 – AlphaOfA)
Result – Background * (1 – AlphaOfA) = Foreground
(Hey, Middle school was a while ago, if you’re not sure what I did there review it over here: http://www.mathsisfun.com/algebra/equations-solving.html )

Now let’s apply it in Nuke:
(Turns out, we have a merge that does Background * (1 – AlphaOfA): Stencil)
We now have a correct element:

set cut_paste_input [stack 0]
version 9.0 v5
Group {
 inputs 0
 name Group1
 selected true
 xpos 114
 ypos -52
 postage_stamp true
}
 Expression {
 inputs 0
 expr0 1
 expr1 x%20>10?1:0
 expr2 x%20>10?1:0
 name Expression1
 selected true
 xpos -250
 ypos -42
 }
 Transform {
 translate {-260 -232}
 rotate -15
 center {960 540}
 name Transform1
 selected true
 xpos -250
 ypos -16
 }
 Ramp {
 p0 {1564 192}
 p1 {880 184}
 color {1 0.88 1 1}
 color_panelDropped true
 name Ramp1
 selected true
 xpos -250
 ypos 10
 }
 Roto {
 output alpha
 replace true
 curves {{{v x3f99999a}
 {f 0}
 {n
 {layer Root
 {f 2097152}
 {t x44700000 x44070000}
 {a pt1x 0 pt1y 0 pt2x 0 pt2y 0 pt3x 0 pt3y 0 pt4x 0 pt4y 0 ptex00 0 ptex01 0 ptex02 0 ptex03 0 ptex10 0 ptex11 0 ptex12 0 ptex13 0 ptex20 0 ptex21 0 ptex22 0 ptex23 0 ptex30 0 ptex31 0 ptex32 0 ptex33 0 ptof1x 0 ptof1y 0 ptof2x 0 ptof2y 0 ptof3x 0 ptof3y 0 ptof4x 0 ptof4y 0 pterr 0 ptrefset 0 ptmot x40800000 ptref 0}
 {curvegroup RectangleCusped1 512 bezier
 {{cc
 {f 8192}
 {px 1
 {0 0}
 {xc28c0000 x43940000}
 {0 0}
 {0 0}
 {x44368540 x4393fffe}
 {0 0}
 {0 0}
 {x44368540 x4319fffe}
 {0 0}
 {0 0}
 {xc28c0000 x431a0000}
 {0 0}}}
 {cc
 {f 8192}
 {px 1
 {0 0}
 {0 0}
 {0 0}
 {0 0}
 {x445de9d0 x400b2f80}
 {0 0}
 {0 0}
 {x445e4bec xc0569700}
 {0 0}
 {0 0}
 {0 0}
 {0 0}}}}
 {tx 1 x441e8000 x43610000}
 {a osw x41200000 osf 0 str 1 spx x44700000 spy x44070000 sb 1 tt x41100000}}}}}}
 toolbox {selectAll {
 { selectAll str 1 ssx 1 ssy 1 sf 1 }
 { createBezier str 1 ssx 1 ssy 1 sf 1 sb 1 tt 4 }
 { createBezierCusped str 1 ssx 1 ssy 1 sf 1 sb 1 }
 { createBSpline str 1 ssx 1 ssy 1 sf 1 sb 1 }
 { createEllipse str 1 ssx 1 ssy 1 sf 1 sb 1 }
 { createRectangle str 1 ssx 1 ssy 1 sf 1 sb 1 tt 8 }
 { createRectangleCusped str 1 ssx 1 ssy 1 sf 1 sb 1 tt 9 }
 { brush str 1 ssx 1 ssy 1 sf 1 sb 1 }
 { eraser src 2 str 1 ssx 1 ssy 1 sf 1 sb 1 }
 { clone src 1 str 1 ssx 1 ssy 1 sf 1 sb 1 }
 { reveal src 3 str 1 ssx 1 ssy 1 sf 1 sb 1 }
 { dodge src 1 str 1 ssx 1 ssy 1 sf 1 sb 1 }
 { burn src 1 str 1 ssx 1 ssy 1 sf 1 sb 1 }
 { blur src 1 str 1 ssx 1 ssy 1 sf 1 sb 1 }
 { sharpen src 1 str 1 ssx 1 ssy 1 sf 1 sb 1 }
 { smear src 1 str 1 ssx 1 ssy 1 sf 1 sb 1 }
} }
 toolbar_brush_hardness 0.200000003
 toolbar_source_transform_scale {1 1}
 toolbar_source_transform_center {960 540}
 colorOverlay {0 0 0 0}
 lifetime_type "all frames"
 motionblur_shutter_offset_type centred
 source_black_outside true
 name Roto1
 selected true
 xpos -250
 ypos 46
 }
 Premult {
 name Premult1
 selected true
 xpos -250
 ypos 84
 }
 Text2 {
 font_size_toolbar 100
 font_width_toolbar 100
 font_height_toolbar 100
 message "Special News"
 old_message {{83 112 101 99 105 97 108 32 78 101 119 115}
 }
 box {309 169.5 903 270.5}
 transforms {{0 2}
 }
 cursor_position 8
 scale {1 1}
 cursor_initialised true
 initial_cursor_position {{309 270.5}
 }
 group_animations {{0} imported: 0 selected: items: "root transform/"}
 animation_layers {{1 11 960 540 0 0 1 1 0 0 0 0}
 }
 color {0 0 0 1}
 color_panelDropped true
 enable_shadows true
 shadow_opacity 0.835
 name Text1
 selected true
 xpos -250
 ypos 110
 }
 Text2 {
 font_size_toolbar 200
 font_width_toolbar 100
 font_height_toolbar 130
 tracking_toolbar -0.168
 opacity 0.335
 message TV
 old_message {{84 86}
 }
 box {19.6953125 99.5 250.3046875 360.5}
 transforms {{0 2}
 }
 font_size_values {{0 200 1 200 0 101 1 101}
 }
 font_height_values {{0 130 1 130 0 102 1 102}
 }
 kern_values {{0 0 1 0 0 0.002 1 0.002}
 }
 tracking_values {{1 -0.168}
 }
 font {{ Adobe Gothic Std : B : AdobeGothicStd-Bold.otf : 0 }}
 font_size 200
 font_height 130
 tracking -0.168
 scale {1 1}
 cursor_initialised true
 initial_cursor_position {{19.6953125 360.5}
 }
 group_animations {{0} imported: 0 selected: items: "root transform/"}
 animation_layers {{1 11 960 540 0 0 1 1 0 0 0 0}
 }
 color {1 0 0 1}
 color_panelDropped true
 enable_shadows true
 name Text2
 selected true
 xpos -250
 ypos 136
 }
 CheckerBoard2 {
 inputs 0
 format "1920 1080 0 0 1920 1080 1 HD_1080"
 name CheckerBoard1
 xpos 60
 ypos -8
 }
 Shuffle {
 alpha black
 name Shuffle1
 xpos 60
 ypos 64
 }
 Merge2 {
 inputs 2
 name Merge1
 xpos 60
 ypos 136
 }
 Output {
 name Output1
 xpos 60
 ypos 236
 }
end_group
set Nc0008000 [stack 0]
push $Nc0008000
CheckerBoard2 {
 inputs 0
 format "1920 1080 0 0 1920 1080 1 HD_1080"
 name BG
 selected true
 xpos 272
 ypos -52
}
Merge2 {
 inputs 2
 operation stencil
 name Merge1
 selected true
 xpos 193
 ypos 65
 postage_stamp true
}
Merge2 {
 inputs 2
 operation minus
 name Merge2
 selected true
 xpos 114
 ypos 177
 postage_stamp true
}

Using it on other effects

By following the same principle, you can undo almost any comp operation. I have used it to remove a semi-transparent logo from a whole image sequence. (same math as above almost, except that time the Background was the unknown)
Another time I used it to recover the footage in a dissolve. Some video footage was dissolving to a still frame of the commercial’s brand logo. I had to replace that logo frame with the Chinese version of the Logo for versioning, but the production company was unable to provide us with the footage without the dissolve.
Since a dissolve is using the same math as an over, but with an animated solid alpha (from 0% to 100% white), I was able to isolate the background footage and replace the logo seamlessly.

Limitations

As mentioned previously, in order to reverse a comp operation, you need to possess every other elements involved in the operation.
If some data is entirely missing (let’s say a part of the background behind a 100% opaque element) you won’t be able to recover it (99.9% would be fine, in 32bit).
Finally, bit depth, clamping and compression might limit how much of an operation is undoable.

Bonus for Flame Users

The famous Ivar Beer from the Logik Flame Users group on Facebook has built the demonstrated operation into a Matchbox Shader: https://logik-matchbook.org/shader/crok_uncomp