how does the probability formulation change if the shapes of AAA and BBB are not convex?
I'd imagine for purposes of this class that we wouldn't have to worry about a case of non-convex shapes, as it seems that most of what we do is related to triangles (and other closed surfaces). In fact, I can't imagine a setting in which we would need to do ray tracing on non-convex shapes. Isn't the point of ray tracing to accurately simulate light bouncing off of real-life objects?
I believe that even though the surface of something is modeled by non-convex shapes, such as triangles, it is still possible for the surface itself to be concave.