Friday, February 12, 2016

How to manipulate the perceived color temperature of an image with OpenCV and Python

When we look at images, our brain picks up on a number of subtle clues to infer important details about the scene, such as faint color tints that are due to lighting. Warmer colors tend to be perceived as more pleasant, whereas cool colors are associated with night and drabness. This effect is no mystery to photographers, who sometimes purposely manipulate the white balance of an image to convey a certain mood. How can we achieve such an effect using OpenCV?

To manipulate the perceived color temperature of an image, we will implement a curve filter. These filters control how color transitions appear between different regions of an image, allowing us to subtly shift the color spectrum without adding an unnatural-looking overall tint to the image.

Color manipulation via curve shifting

A curve filter is essentially a function, y = f(x), that maps an input pixel value x to an output pixel value y. The curve is parameterized by a set of n+1 anchor points, as follows: {(x_0,y_0), (x_1,y_1), ..., (x_n, y_n)}.

Each anchor point is a pair of numbers that represent the input and output pixel values. For example, the pair (30, 90) means that an input pixel value of 30 is increased to an output value of 90. Values between anchor points are interpolated along a smooth curve (hence the name curve filter).

Such a filter can be applied to any image channel, be it a single grayscale channel or the R, G, and B channels of an RGB color image. Thus, for our purposes, all values of x and y must stay between 0 and 255.

For example, if we wanted to make a grayscale image slightly brighter, we could use a curve filter with the following set of control points: {(0,0), (128, 192), (255,255)}. This would mean that all input pixel values except 0 and 255 would be increased slightly, resulting in an overall brightening effect of the image.

If we want such filters to produce natural-looking images, it is important to respect the following two rules:

• Every set of anchor points should include (0,0) and (255,255). This is important in order to prevent the image from appearing as if it has an overall tint, as black remains black and white remains white.
• The function f(x) should be monotonously increasing. In other words, with increasing x, f(x) either stays the same or increases (that is, it never decreases). This is important for making sure that shadows remain shadows and highlights remain highlights.

Implementing a curve filter by using lookup tables

Curve filters are computationally expensive, because the values of f(x) must be interpolated whenever x does not coincide with one of the prespecified anchor points. Performing this computation for every pixel of every image frame that we encounter would have dramatic effects on performance.

Instead, we make use of a lookup table. Since there are only 256 possible pixel values for our purposes, we need to calculate f(x) only for all the 256 possible values of x. Interpolation is handled by the UnivariateSpline function of the scipy interpolate module, as shown in the following code snippet:

from scipy.interpolate import UnivariateSpline

def create_LUT_8UC1(x, y):
spl = UnivariateSpline(x, y)
return spl(xrange(256))


The return argument of the function is a 256-element list that contains the interpolated f(x) values for every possible value of x.

All we need to do now is come up with a set of anchor points, (x_i, y_i), and we are ready to apply the filter to a grayscale input image (img_gray):

import cv2
import numpy as np

x = [0, 128, 255]
y = [0, 192, 255]
myLUT = create_LUT_8UC1(x, y)
img_curved = cv2.LUT(img_gray, myLUT).astype(np.uint8)


However, in order to warm up or cool down an image, we will need to apply the curve filter to all three RGB channels.

Designing a warming effect

By making use of our create_LUT_8UC1 function developed in the preceding steps, let's define two generic curve filters, one that (by trend) increases all pixel values of a channel, and one that generally decreases them:

incr_ch_lut = create_LUT_8UC1([0, 64, 128, 192, 256],
[0, 70, 140, 210, 256])
decr_ch_lut = create_LUT_8UC1([0, 64, 128, 192, 256],
[0, 30, 80, 120, 192])


The easiest way to make an image appear as if it was taken on a hot, sunny day (maybe close to sunset), is to increase the reds in the image and make the colors appear vivid by increasing the color saturation. We will achieve this in two steps:

1. Increase the pixel values in the R channel and decrease the pixel values in the B channel of an BGR color image using incr_ch_lut and decr_ch_lut, respectively:
img_bgr_in = cv2.imread("img_example.jpg")

c_b, c_g, c_r = cv2.split(img_bgr_in)
c_r = cv2.LUT(c_r, incr_ch_lut).astype(np.uint8)
c_b = cv2.LUT(c_b, decr_ch_lut).astype(np.uint8)
img_bgr_warm = cv2.merge((c_b, c_g, c_r))

2. Transform the image into the HSV color space (H means hue, S means saturation, and V means value), and increase the S channel using incr_ch_lut. This can be achieved with the following function, which expects an RGB color image as input:
c_b = cv2.LUT(c_b, decr_ch_lut).astype(np.uint8)

# increase color saturation
c_h, c_s, c_v = cv2.split(cv2.cvtColor(img_rgb_warm,
cv2.COLOR_BGR2HSV))
c_s = cv2.LUT(c_s, self.incr_ch_lut).astype(np.uint8)

img_bgr_warm = cv2.cvtColor(cv2.merge(
(c_h, c_s, c_v)),
cv2.COLOR_HSV2BGR)


The result looks like this:

Of course, the illusion of a warm sunny day might be limited by the fact that there is snow on the roof... ;-)

Designing a cooling effect

Analogously, we can define a cooling filter that increases the pixel values in the B channel, decreases the pixel values in the R channel of an RGB image, converts the image into the HSV color space, and decreases color saturation via the S channel:

c_b, c_g, c_r = cv2.split(img_bgr_in)
c_r = cv2.LUT(c_r, self.decr_ch_lut).astype(np.uint8)
c_b = cv2.LUT(c_b, self.incr_ch_lut).astype(np.uint8)
img_bgr_cold = cv2.merge((c_b, c_g, c_r))

# decrease color saturation
c_h, c_s, c_v = cv2.split(cv2.cvtColor(img_bgr_cold,
cv2.COLOR_BGR2HSV))
c_s = cv2.LUT(c_s, self.decr_ch_lut).astype(np.uint8)
img_bgr_cold = cv2.cvtColor(cv2.merge(
(c_h, c_s, c_v)),
cv2.COLOR_HSV2BGR)


Now, the result looks like this:

Brrrrr!

More information can be found in the book OpenCV with Python Blueprints. As usual, all source code is available for free on GitHub (refer to the WarmingFilter and CoolingFilter classes in the filters module). Please note that these classes expect an RGB color image, not a BGR.