My understanding is that there are two completely independant clipping systems in app_server (mostly in https://git.haiku-os.org/haiku/tree/src/servers/app/DrawState.cpp )
One is region clipping. This can use a BRegion (only rectangles) and always works in screen space (ignoring transforms). This can be used to tell a window that other windows are overlapping, for example, but it is also possible to clip further (GetClippingRegion/ConstrainClippingRegion).
The other is picture based clipping. The idea is more to use a mask that is stored as a BPicture. This can be used not only for clipping, but also for alpha transparency effects if your picture has partially transparent pixels. As a side effect, your clipping shape can be antialiased. This is used in BView ClipToShape, ClipToPicture, and the corresponding “inverse” functions. Picture based clipping works with transforms, so it can be scaled, rotated, etc.
There is a special case in ClipToRect (called from BView::ClipToRect and ClipToInverseRect): https://git.haiku-os.org/haiku/tree/src/servers/app/DrawState.cpp#n474 if no transform is applied, it will optimize the clipping by using the region-based one (which is faster and uses less memory). But if there is a transform, the rect will be transformed, and if there is rotation or shear, it won’t be a rect anymore. So, in that case, shape clipping will be used. However, I don’t see how this can cause a problem, since the fast path with regions is used only when there is no transform at all? What’s missing here?