description.

In this lab, we removed the bluescreen from a set of images, and did some basic manipulation on them. We also implemented iterative algorithms to generate Julia and Mandelbrot sets.

In the first part of lab, we took pictures of people in front of a bluescreen and then tried to replace the blue parts with a background. Our group decided to use a very simple approach based on the color chromaticities. The chromaticity of a particular color (red for example) at a pixel can be calculated using the formula

Chrom_R = R / (R + G + B)

where R, G, and B are the color values at the given pixel (ranging from 0 to 255). The color chromaticity at any pixel tells us how much of the color is present, accounting for the effect of shadows or highlights. This is why chromaticity is a good way to detect the blue in the image.

First off, we started using a simple statistical approach: we picked an area of blue on the image, and wrote a program that would calculate the mean chromaticity for that region and also the standard deviation. Then we would cycle through the entire image and replace any background pixels whose blue chromaticity was within a deviation of the mean. However, this didn't work very well, since a lot of blue spattering remained in the background (Fig. 1). To address this, at first we decided the area of blue we chose didn't offer enough variance, so we needed to examine a larger blue area. After adding two more rectangles of about 490k pixels each, the result didn't improve much, so we decided to change our method. We still calculated a mean, but instead of using the standard deviation as our criterion, we fixed a threshold and adjusted that until we got the best effect (Fig. 2).

After we had taken care of the background, we wrote three additional functions to manipulate the image "geometry" and two more (as extensions) that manipulate the color:

• place - this function takes two images, and inserts one on top of the other, starting at a given point. Mathematically, this wasn't hard, we just had to offset the x and y coords of the top image by the amount specified as input to the routine (Fig. 5-8).
• scale - this would scale down a given image by a factor. In our implementation, the scaling factor must be an integer and both dimensions of the image must be divisible by it. This divided the image into squares with side dimension the scaling factor and then averaged all the pixel colors within to create one new pixel from each square that would go in the new image (Fig 6, 8).
• rotate by 180 degrees - to accomplish this transformation, we moved each pixel currently at index i (our images were indexed as a flat array, from i=0 to i = rows*columns) to index rows*columns - 1 - i. (Fig 6)
• subtract red - this extension takes two images and subtracts the red color of each pixel from one from the other. This effect got interesting since we were allowing the value to overflow during subtraction, so sometimes we actually reinforced the red already there (Fig. 13).
• color average - this extension takes two images and makes an image that averages the colors for each pixel from both images (Fig 14). We used this to create an interesting transparency effect (Fig. 15).

In the second part, we gemerated Julia and Mandelbrot sets to create fractals. Contrary to the causality theory (similar initial conditions yield similar results) a slight difference in starting conditions of a function can create huge differences in the results. In the case of the Julia set we, picked a rectangle in the complex plane, and we iterated a function z(x, y), where x and y are the coords of points within that rectange. Eventually, there were two outcomes of the iteration process: the function could go to either 0 or infinity. Since both of those were hard to detect computationally, we used thresholds. One threshold limited the number of times the equation could be iterated, if that theshold was reached, we assumed the funciton would go down to 0. We set another threshold on the largest value z could get, and if that value was reached, we assumed further iterations would take the function's value to infinity. The function we iterated for the Julia set was

z_n+1 = z_n² - 0.7454054 + j0.1130063 (where j = sqrt(-1)).

Once we knew the number of iterations every pixel took to reach either a 0 or a max, we used that to assign the pixel a color based on

• whether the pixel was solved on an even or odd iteration (Fig. 10)
• the linear relationship between the number of iterations and the max (Fig. 9, 12)
• fixed brackets within the possible range of iteration, where every bracket got its own color (Fig. 11)

The Mandelbrot set was generated very similarly, with the only difference being the equation used:

z_n+1 = z_n² + c (where z_n started at 0 (z₀ = 0), and c = c_x + jc_ywas calculated from the current point)

(For more details, please visit the lab page here)

questions.

1. How big is the picture of yourself (pixel width and height)? How did you determine this?
   Our images were 2560 pixels wide and 1920 pixels high; we used GIMP to determine these values. Also, the readPPM function supplies those when reading a file.
   
2. Where is the origin of an image--pixel (0, 0)--when you manipulate it in your code? How did you determine this?
   The origin is the top left corner for PPM and bottom left corner for TIFF. We wrote some code and made the first 100 pixels white, then looking at the resulting image, it was apparent where the origin was. In our code, we treated the upper left-hand corner of the image as the origin.
   
3. Where is the origin of an image in xv? How did you determine this? Is it consistent with pixel (0, 0) as tested above?
   We used GIMP instead of XV (xv wasn't available) and in GIMP, the origin of the image is located at the upper left hand corner (both for PPM and TIFFs). This was consistent with the representation we used.
   
4. How did you replace the blue pixels in your picture? How well did your replacement method work?
   Was there any "blue-spill" in your image?
   How would you do it better?
   We used two separate methods to replace blue pixels. In both of these techniques, we first picked a few areas on the image where it was all blue (used GIMP to find the coords of these rectangular areas). Then for the first method, we calculated the mean chromaticity and the standard deviation, and cleared all pixels that were in the range. For the second method, we again calculated the mean but set the treshhold ourselves and replaced the blue pixels.  We found out that the second method worked better.
   
   There was some "blue-spill" in our images, but it wasn't nearly as much as we expected. The two places where it was apparent were Adem's watch and Yavor's eyes and hair (Figs. 1-2). In the case of Adem's watch there was little we could do since a large area of color was being removed. With Yavor's hair and eyes, only a few pixels at a time were being removed, and since we were using very high-resolution images, that wasn't a problem.
   
   In general, there are a few things we could have done to improve our bluescreen removal method. First, to make sure blue spill is minimized, we could try to detect edges and make sure we remove only contiguous areas of blue that border the outside of the image. This reminded us of the algorithm you mentioned in class regarding filling polygons. The same idea could be applied here, with the only difference being that a pixel would have to match the average chromaticity before it is "filled" (removed). However, this introduces problems in case someone puts their hands on their waist in the picture (thus creating bounded contiguous areas of blue, not bordering the outside of the image), in which case we would have to tell our algorithm specifically to include these areas. Still, we didn't have much trouble with spill, so we didn't find it necessary to worry about any of these.
   
   In general, one improvement to the general algorithm could be to account for shadows behind people. A possible way to do this is to use two thresholds - one that covers pixels that go all the way to the border of the person's contour, and one that covers the same area, but excludes the shadows. To differentiate between the thresholds, we'd so need to analyze pixels for similarity factoring in shadows/highlights, since generally shadows would just be darker than the background. After we have identified the shadows, we could try to desaturate them from color and then flag the pixels as semi-transparent, so the shadows could blend in with whatever new background we choose to add.
   
   
   

   Fig 1. Blue spill in Yavor's head	Fig 2. Blue spill in Adem's watch
   Fig 3. Replacing blue using standard deviation	Fig. 4. Replacing blue using a fixed threshold of 0.13

   NOTE: Click on an image for a hi-res PPM
   
   
5. How did you creatively insert yourself in an image? For this question, write down the equations you used.
   The images are shown below. Please read the description section to find out about the details of each manipulation.
   
   
   

   Fig 5. Adem fighting off the shark	Fig 6. Adem saving Dori and Marvin (uses offset, scaling, and 180-degree rotation)
   Fig 7. Yavor	Fig 8. Yavor and his older identical twin are about to be eaten (uses offset and scaling)

   NOTE: Click on an image for a hi-res PPM
   
   
6. What part of the Mandelbrot set did you find the most interesting?
   The edges were the most interesting part (Fig. 12) since the set was fairly dull on the inside. For the Jula set we liked both the look of the entire set (Fig. 9) and the look of the interesting protruding colums in the dead center of the set (Fig. 10).
   
   

   Fig 9. Full Julia set (coloring linearly dependent on iteration number)	Fig 10. Selected detail (coloring based on odd/even number of iterations)
   Fig 11. Full Mandelbrot set (coloring based on remainder during division by a constant)	Fig 12. Selected detail (color based on number of iterations)

   NOTE: Click on an image for a hi-res PPM
   
   
7. What coloring schemes did you experiment with, and which did you like the best?
   We started off with a R, G, B color scheme, which made it easy to recognize the sets, but was a horror for the eyes. Then we moved on to other ways to color the set, please see the description section for more details.

EXTENSIONS

Fig 13. Subtraction/addition of red (i.e. Yavor and the shark after a toxic chemical spill)	Fig 14. Color averages (i.e. Yavor is pinned between the shark and a mutated seaweed)
Fig 15. Color averages (i.e Yavor's ghost ate the two pretty fish)

As our extensions, we worked on different ways to overlay images. The two routines that we wrote had to do with using one image to change the color space of another. Both methods are described above in the description section. Some of the images we made showed some surprising features: in Fig. 13 you can see how subtracting the red, but letting the value overflow actually reinforced the color in some places. Also we overlayed (using color averaging) an image with its background by accident (Fig. 15), which created an interesting ghost effect.