When researching further into this topic, one technique that many consumer cameras do today is demosaicking with different algorithms in different areas of the image which are more densely or sparsely populated with features. Specifically, many cameras today use RCD, AMaZe, and Markesteijn 3-pass algorithms in regions of the image with high levels of detail and features, and use the VNG4 algorithm in areas of the image with sparse features (such as a green or blue background color). I found this webpage to be helpful: https://docs.darktable.org/usermanual/3.8/en/module-reference/processing-modules/demosaic/

@kujjwal, after reading your comment I actually did more research into demosaicking and found a new breakthrough called deep demosaicking. Heres a really cool video I think you would find interesting

https://www.youtube.com/watch?v=sjj78KcqYIM

Yeah, this looks quite interesting, I think it could be a good final project idea to try and train a neural network to perform demosaicking since the video described training potentially multiple fully-connected layers to create more optimal "handcrafted" demosaicking filters.

Why is it better not to interpolate across edges? I did some research and it seems that interpolating across edges leads to artifacts and color bleeding when there are sharp edges. Corners lead to less of this. It also generally lessens the quality of the image.

I researched what type of technique consumers cameras use and once very interesting one I found was VNG, or variable number of gradients, which use gradients in the image to estimate colors. It prefers lower gradients because it's smoother and similar parts of the image.

It's cool to see how interpolation is used in many different topics within graphics, such as texture mapping, image scaling, and even in demosaicking algorithm. It's interesting to see that the field of computer graphics tries to "smoothen" colors or pixels by using interpolation but also tries to make images sharper to prevent blurriness.

We've previously had to use bilinear interpolation for position/location, but now in a simple algorithm, we can also use it to average colors for display. My explanation of demosaicking algorithms is that are cameras can't sample every possible pixel of light in a real-life scene. So based off of the sparse samples, demosaicking algorithms use a variety of ways to interpolate RGB pixel image data and thus "make up" the displayed image data. Modern cameras obviously don't use as basic algorithms, but I wonder how much energy and R&D these major camera companies pour into developing better and better techniques and algorithms. Seems like there's a bunch of advanced methods already out there in the works.

Only after reading the slide about mosaicking did I realize what de-mosaicking is. Since mosaicking is the process of combining multiple images together to make a single image, here we're doing the opposite where we take one color image and split it into three monochrome R/G/B images instead. The reason we're demosaicking is because the color sensors are in a mosaic pattern on the sensor, and splitting them up into multiple images demosaics them. (i was wondering for a while what a "demo sake" was)

It's interesting how 2/3 of image data is "made up" due to this phenomenon; since we're averaging values, it seems that we're getting semi-accurate values based on assumptions from nearby values.

I find it interesting how 2/3 image pixels end up being interpolated as color sensors only sample R, G, B colors. What happens in black or white pictures? I guess the key innovation was realizing that a linear combination of RGB values can map to almost all colors and we could interpolate to fill in the gaps. I guess black and white colors don't need made up data.

Just as Lim said, I think the multi-faceted use cases of interpolation are really interesting, especially when it comes to texture mapping like we worked on in HW 4. I think the coolest thing that we can use interpolation for is fluid simulation, though. I wonder how this slide would tie into that type of multi-colored simulation for modeling the flow of water, for example.

I thought that demosaicking algorithms were so interesting and read into it a bit more! It seems like there is a trade-off in these algorithms between computational complexity and image quality, so I wonder what algorithm(s) would achieve the best balance between these two factors.

If we had three cameras -- a red, green, and blue camera -- would this mean we wouldn't need to demosaick? Many modern smartphones have several cameras, so I wonder if this could be used to increase image quality.

I think more clarification can be made on the interpolating process and why it can make images better. When light hits the sensor, each pixel records only the intensity of one of the three primary colors (red, green, or blue), depending on the filter color above it. This means each pixel lacks two-thirds of the color information needed for a full-color representation. The demosaicking algorithm estimates the missing color information at each pixel by interpolating the color values from the surrounding pixels. Different algorithms have different ways of doing this interpolation, aiming to minimize artifacts such as color moiré or zipper effects. The final output after applying the demosaicking process is a full-color image where each pixel has red, green, and blue components, reconstructed from the patterned single-color inputs.

It's interesting to see that this demosaicking algorithm uses pretty much the same techniques as previous anti-aliasing algorithms. It seems like often times, using a previously tried strategy can get the job done.