Congratulations! Porting Motif through this compatibility layer is not recommended: if there is a way to crash or to behave mysteriously, Motif will find it. It has been well known as a torture test for X for decades, and going through a great deal of trouble to support Motif will not bring Haiku support for many significant applications.
As for Cairo (and by extension GTK), I recommend implementing the X Render Extension. The relevant documentation can be found at https://www.x.org/releases/X11R7.6/doc/renderproto/renderproto.txt, and Cairo is designed to take advantage of it whenever present. Its ubiquitousness throughout X servers has caused the non-XRender code paths in Cairo to rot, so there are certain to be many bugs there.
For GTK to work correctly, you will also need to implement the Extended Window Manager Hints, some non-standard Motif hints (they are not actually related to Motif anymore), and the ICCCM. I recommend starting with the ICCCM: It will introduce you to many concepts used in the other features mentioned here. You can find the documentation at Inter-Client Communication Conventions Manual. The X Session Management Library (www dot x dot org slash releases/X11R7.6/doc/libSM/SMlib.html) is also recommended.
GTK 4 also assumes the presence of the XInput 2 extension, which completely replaces the Core Input API. It is extremely convoluted and ugly, but it is required for GTK 4 to function, and is also recommended for running GTK 3.
Some extensions, especially the shaped window extension, can just be stubbed out, but I see you’ve already done that.