Before I detail more of “what works and what doesn’t”, let’s back up a bit.
Why write an Xlib/X11 compatibility layer at all, instead of native backends?
“Native” backends for toolkits are obviously preferable, allowing for tighter integration with the system; the Qt port takes this approach, and it is proof enough that such things are both feasible and desirable. However, Qt is kind of an exception in many ways; it is well-abstracted internally in a way most other toolkits are not, and it has a wide variety of supported platforms even before it was ported to Haiku.
Not all toolkits are like this; in fact, very few are. GTK has backends only for X11, (now) Wayland, Windows, and macOS. (The macOS backend was pretty poor for a long time; most users of GTK applications on macOS were stuck with using X11 via XQuartz, I seem to recall.) Other applications, like “AzPainter”, support X11 and only X11; or maybe they have support for Wayland, but not even a Windows or macOS port to speak of. Once software has been ported off of X11/Wayland, there is some hope that a native backend for a completely different architecture can be written, but oftentimes before that, the system will just make far too many assumptions about what it is running on to make rearchitecting it internally feasible.
So, even if writing a GTK backend for Haiku were feasible or doable, there is a separate, sizeable amount of Linux/etc. software that can really only be ported by bringing over X11 or Wayland. We could of course port a full X11 server and use that, as you can do on Windows or macOS, but considering the downsides of that approach and the flexibilities of the Haiku APIs, I realized that it was possible to re-implement the X11 APIs directly on top of the Haiku APIs (well, with some minor exceptions, of course), without an X11 server, and in fact provide a lot of the benefits that a “true” native backend would provide.
Why Xlib/X11 and not Wayland?
The Wayland APIs are very spartan and highly protocol-based; that is, there are more implementations of the protocol than just “libwayland”, making it basically impossible to write just one library for Wayland compatibility, but instead probably requiring a server implementation. While there are other X11 interface implementations besides Xlib (notably Xcb, which modern Xlib is based on), they are very rarely used, and in fact implementing an X11 server on top of Xlib is more than possible (Xnest, while very limited compared to Xephyr, is basically this, in fact), while implementing a Wayland server that runs on Wayland is experimental territory.
Further, I’m personally not so sure about Wayland as a system, anyway. It is only now, after over a decade of development, finally being used by default (after many, many years of support for it being theoretically available in toolkits, and tech demos showing it off.) Having looked into its design and implementation, I suspect a lot of its problems are actually design flaws; already it has been noted that the core protocol is so basic as to be functionally unusable (or at least unworkable) without piling on a large array of extensions.
X11, on the other hand, is a relatively stable target. It too has a lot of extensions, and it too has various oddities (what system wouldn’t after almost 40 years?), but it is well documented, well used (GTK+2 does not really support Wayland, for instance, meaning we’d be forever without it with just a Wayland compatibility layer, or be forced to use XWayland which sounds worse – a compatibility layer on top of a compatibility layer?)
While more Linux distributions are switching to Wayland, I don’t expect GTK et al. will drop their X11 code for quite a long time to come. If they eventually do, by then Wayland will be much more mature, and maybe a compatibility layer for it would also make sense. But that’s far off in the future, and not really worth considering at present.
Plus, by implementing Xlib, we have the potential to get xeyes and Motif! You know you want Motif applications, right? (Well, at least @3dEyes told me he wants one or two…)