Make ANSI Common Lisp available on Haiku (again)

alpopa · January 10, 2025, 6:05am

Thank you for your work @cocobean. My low-level programming skills are really very limited.

alpopa · January 13, 2025, 2:01pm

Merged patch to not lose when calling pthread_getattr_np: https://github.com/sbcl/sbcl/commit/511db9240821e0122817a1ac5eb204f7e1350b5e.

alpopa · January 16, 2025, 8:20am

Meanwhile, I am concentrating to something I believe I could do: port SBCL to Haiku x86. I expect it to only work with recent gcc, no BeOS compatibility. So, before build I use:

setarch x86

And I already hit a problem. To detect features provided by OS, which do not need to be emulated, SBCL uses short C programs and watches their exist code. One of these programs compiles and executes without any issue with gcc2 but compiles with warnings and fails during execution with modern gcc13. It makes use of:

#include <netdb.h>
...
getprotobyname_r(...);
getprotobynumber_r(...);

The warnings issued by gcc13 are:

warning: implicit declaration of function 'getprotobyname_r'; did you mean 'getprotobyname'?
warning: implicit declaration of function 'getprotobynumber_r'; did you mean 'getprotobynumber'?

These functions are expected to be defined in header file, not in this program. As result, compiled with gcc2 the program is 16.56 KB compared to just 5.79 KB compiled by gcc13. The situation looks like some devel package is missing for gcc13. However, readelf -d os-provides-getprotoby-r-test only lists:

[libbe.so]
[libnetwork.so]
[libbsd.so]
[libroot.so]

they are part of haiku_devel package AFAIK.

The program is auxiliary and compiled with gcc2 it works exactly the same in x86 and x86_gcc2 environments. It may be unnecessary to actually compile it with gcc13, but I am unsure.

So, I have 2 questions:

Is it possible after setarch x86 to compile something with gcc2?
Is there some devel package I could install to get this program compiled with gcc13?

Begasus · January 16, 2025, 8:25am

Function seems to be provided by libnetwork

~> lgrep getprotoby
/boot/system/lib/x86/libnetwork.so:00014160 T getprotobyname
/boot/system/lib/x86/libnetwork.so:000141b0 T getprotobyname_r
/boot/system/lib/x86/libnetwork.so:00014290 T getprotobynumber
/boot/system/lib/x86/libnetwork.so:000142e0 T getprotobynumber_r

At least one error in the 2.5.0 recipe, cmd:gcc$secondaryArchPrefix should be cmd:gcc$secondaryArchSuffix

Having said that, it expects a value for sbcl_arch? (fixed with an export …)

Another edit, the archive doesn’t seem to contain genesis directory/code? (seems to be triggered with export SBCL_ARCH=x86.

Begasus · January 16, 2025, 9:33am

Try to see if including netdb.h fixes it?

alpopa · January 16, 2025, 10:53am

SBCL contains platform specific source code (Config.x86-haiku, x86-haiku-os.h, x86-haiku-os.c), which is at the moment missing. Without them, it is impossible to build something really helpful for this unsupported platform.

I do not yet use haikuporter and the problems mentioned by me are plain gcc problems.

Is libnetwork_devel contained in haiku_devel package or is it a 3rd party library? How is it possible that it is available for gcc2 and not available for gcc13?

This file is already included. Again, it is visible for gcc2 but not for gcc13.

A more in-depth question is: What exactly does setarch? I imagine it sets some simlinks to proper gcc binaries, headers and libraries. Does it changes something else system-wide (something similar to what chroot does)?

Begasus · January 16, 2025, 2:22pm

setarch switches compiler as you already found out, it also alters the paths to library/headers to look for, like lib/x86 instead of lib or develop/headers/x86 … (most of these are defined by the packages required).

It’s part of Haiku, so should be included in haiku_x86_devel for 32bit.

alpopa · January 17, 2025, 6:43am

Found different packages haiku_devel and haiku_x86_devel, both are installed by default.

After analysis of /boot/system/develop I found common folder headers and separate folders lib and lib/x86. The header netdb.h contain functions getprotobyname and getprotobynumber but does not contain (low level) functions getprotobyname_r and getprotobynumber_r. The former (without _r) probably call the later (with _r suffix). But the difference between gcc2 and gcc13 is these low level functions are exposed to user for gcc2 but are hidden in private API in gcc13.

With assumption of relativity of these functions, I adapted the test code to call functions getprotobyname and getprotobynumber and no more crash occurs.

Still I do not go far forward in porting SBCL to x86 Haiku.

Begasus · January 17, 2025, 6:57am

Looking at the header’s information it’s part of haiku_devel (gcc2), seeing it’s available as /boot/system/develop/headers/posix I assume it would be available cross arch? (not an expert on the matter).
Searching it on haikuports doesn’t really mention any patching for it:

Do you have a seperate branch setup for this?

alpopa · January 17, 2025, 2:12pm

Same for me. But netdb.h does not contain functions getprotobyname_r and getprotobynumber_r. I did not find them in any available header. My assumption is these functions are available elsewhere in private API, which for some reason is visible for gcc2 but not for gcc13.

Anyway, I configured and tested 2 possible solutions (I do not know how reliable are they):

Build SBCL with gcc13 but auxiliary feature tests (features provided by OS) with gcc2
Change this test to call getprotobyname and getprotobynumber instead as they are directly available in the header (assuming these function implementation make use of getprotobyname_r and getprotobynumber_r defined somewhere else).

PulkoMandy · January 19, 2025, 7:58am

How does it fail?

The warning is harmless, it just means the function is not defined in a header and the compiler attempted to guess at its prototype. gcc2 likely has the same issue, but doesn’t bother you with a warning about it, because in old versions of C, this was a normal thing to do.

If the function was private or did not exist, you would get a linker error (“undefined reference” or similar).

So I’m confused about the situation where the program linked correctly (meaning the function is found), but then fails to execute.

That’s possible, but would result in a linker error, not an execution error.

The _r functions are reentrant, which means they are safe to use in threaded environments. The ones without _r may not be. Since you already have problems with threading in SBCL, maybe that is not such a great idea?

alpopa · January 19, 2025, 8:30am

Thank you @PulkoMandy for pointing out the difference between functions with and without _r suffix.

Here is the screenshot of what’s going on:

Here, you can see:

The exact text of the program
The commands to compile it with gcc2 and gcc13
The compile warnings on gcc13
The size difference of binaries compiled with gcc2 and gcc13
The crash running program compiled with gcc13

When compiled with gcc2 there are no compilation warning and program exists with 104 code, i.e. works as intended. So, I guess, the _r variants of functions are visible for gcc2.

Depending on how these functions are used in SBCL, my defending approaches may be relevant or irrelevant. But before I reach SBCL team with this question, I would like to know more about presence / accessibility of _r variants of these functions for gcc13.

PulkoMandy · January 19, 2025, 10:22am

I had a look at it, here are my conclusions (this applies to both functions tested, but I refer to only one for simplicity):

getprotobyname_r is not a standardized function in POSIX
in Haiku, getprotobyname is thread safe (by using a mutex) but slower than getprotobyname_r (which does not need a mutex)
since the function is non-standard, different versions of it exist. The one in Haiku is identical to the one in NetBSD and has a different prototype: struct protoent* getprotobyname_r(const char*, struct protoent*, struct protoent_data*);
However, it is not declared in any public .h files. Which means the compiler can’t detect this problem, and tries to pass the arguments anyways. The linker also cannot check it (it would in C++, but not in C).
The function is called with invalid arguments and crashes.

I think the solution is to make this test behave the same as in NetBSD, by changing the #if __NetBSD__ to #if __NetBSD__ || defined(__HAIKU__). Then these functions will be marked as “not usable” and you should be fine.

alpopa · January 21, 2025, 4:08pm

I doubt I fully understand the way C++ handles functions with identical name but different signature. I just want to make sure that Haiku x86 and Haiku x86-64 has the same signature of getprotobyname_r and getprotobynumber_r functions, and thus neither Haiku x86 nor x86-64 provide them as SBCL expects. Is this understanding right, @PulkoMandy?

PulkoMandy · January 21, 2025, 4:32pm

Yes, it’s the same for both architectures. The code for these functions is imported from NetBSD, and is incompatible with what SBCL expects in the same way as for NetBSD.