Thank you @hoanga. Of course, all sbcl patches available on haikuports and published here, together with any advice, are very welcomed. Any of them is a real help in getting high quality Common Lisp available.
My own work was started in 2019 and shortly after that I was unable to continue. Now, I can resume working, and this time I hope we can do more. It is anyway a teamwork and my gratitude goes to all involved.
Unfortunately, SBCL team stays with outdated version of ASDF with Haiku specific bug. I can understand this decision based on how widespread SBCL is and how dangerous can be any regression due to upgrade. Still, it is possible to use upstream version of this module. For now it is too early to decide what is better: to possible break compatibility with other version of SBCL by shipping a newer ASDF or to try to patch this module shipped with SBCL.
I am testing the recipe for SBCL. Strangely, I can build it in terminal even with a script, which automates all the steps. However when building with haikuporter, the recipe (prepared from the script) arbitrarily fails at some stage with different errors. I cannot reproduce the fail, each time it is different. Sporadically the build finishes. I suspect it may be related to some sort of parallel build in several threads, and because SBCL is supposed to support it, but Haiku port of it does not (yet) support it, this may lead to strange fails. Is there some configuration option for haikuporter to limit the number of threads (similar to -j# option to gcc)? Is there some parameter to use in recipe for it?
You could use âmake -j1â or just âmakeâ without $jobArgs, the last should spread the jobs between the CPUâs but doesnât run multiple jobs at once (iirc).
SBCL does not use autotools or make for building. It has a set of shell scripts to do the job. And gcc part of the build is usually very short and successful. But I can take a look at the scripts to see if some option is defined there.
Locally produced the first SBCL package, installed it and it works!!!
The problem with failure may be due to chroot in haikuporter but not in shell script. When using CLISP to build SBCL, it loads some local files, which are outside the chroot environment, and it fails. I do not fully understand the process or if chroot has something to do with failing. But I changed CLISP usage to not load any local file.
Latest SBCL recipe is building without failure and produces a package that is installable and functional. I was ready to submit this new recipe when observed SBCL is actually available in Haiku Depot and is actually more usable due to Haiku specific patches.
I then tried to build the recipe using older version of SBCL instead of CLISP and it builds. Finally, I tried to rebuild latest SBCL with itself and it also works. This feature was not present last time I played with it.
Having all that, it is probably not a good idea to submit a newer recipe. Its only advantage is no patches, it builds from plain upstream sources.
The next step would be to review available patches (we have 2 different versions in Depot + one provided by @hoanga). I believe many of them are already upstreamed. My goal is to upstream all them. The patches are easier to review and accept when one patch addresses one small problem and actually solves it (and not hides it). So, I will prepare a series of small patches that fill missing bits, enable more contrib modules and resolve failing tests. Of course, I would happily accept some help in this task.
I reviewed all available patches (for now, without actually testing them) and it looks they cover (almost) anything to make SBCL port to Haiku x86-64 a first-class citizen.
I split the patches in the following topics, which I will test individually and then send upstream:
@hoanga, do you think this scheme will work?
hello @alpopa, yes the proposed scheme for submitting the patches looks good as far as i can tell. one thing that could probably be debated is whether to move any patches that fixes tests to step 4 (i do not think itâs necessary). i am assuming step 4 Fix tests means more like fix remaining tests?
and yes i remember your earlier work back in 2019, it was the base that i used to be able to finally get a version of sbcl bootstrapped on haiku. kudos for providing a path to getting there (and of course returning to keep improving and tidying up that created path)
Thank you @hoanga.
Well, step 4 is the last part of existing patches that are not available elsewhere. Step 5 contains the last patch, but it can be also done by using upstream ASDF. And step 6 is not started yet AFAIK. So, yes, step 4 would be a work in progress and can continue after that. I only outlined existing patches.
Merged 2 commits:
https://github.com/sbcl/sbcl/commit/d3af39e34e299edd5fac7968ba48849ac8e12cf6
https://github.com/sbcl/sbcl/commit/b8e701d2098c83c40ce0a585e9e5e91524770213
Is there any source for the constant -4025 in the following patch (contrib/sb-bsd-sockets/sockets.lisp)?
+#-haiku (define-socket-condition sockint::ESOCKTNOSUPPORT socket-type-not-supported-error)
+#+haiku (define-socket-condition -4025 socket-type-not-supported-error)
The first one sockint::ESOCKTNOSUPPORT is defined in contrib/sb-bsd-sockets/constants-unix.lisp as:
(:integer ESOCKTNOSUPPORT "ESOCKTNOSUPPORT")
and in contrib/sb-bsd-sockets/constants-win32.lisp as:
(:integer ESOCKTNOSUPPORT "WSAESOCKTNOSUPPORT")
The former is according to headers: "sys/socket.h" / "errno.h" / "fcntl.h", the later is according to "winsock2.h" "errno.h". Is there some Haiku header, which defines a constant similar to ESOCKTNOSUPPORT / WSAESOCKTNOSUPPORT with the same meaning and value -4025?
Haiku system errors are in the range INT32_MIN to INT32_MIN + 0xFFFF, so -4025 is not an error from Haiku.
It looks like we donât have a dedicated error case corresponding to ESOCKTNOSUPPORT, so that value should never end up being used by any code on Haiku side. If code on clisp side wants to use it, they can use any value as long as it is not in the range of Haiku system errors.
You can check os/support/Errors.h to see all the errors that exist on Haiku.
(we are speaking about SBCL).
I found in FreeBSD sys/errno.h the following declaration:
#define ESOCKTNOSUPPORT 44 /* Socket type not supported */
What error code is used in Haiku in this case? According to existing patch, the value -4025 is used. How can I check it? What is the name of relevant header file?
This is not about what value SBCL wants to use. This is about SBCL uses native OS code and just wants to properly handle specific error codes.
The difference between:
(define-socket-condition sockint::ESOCKTNOSUPPORT socket-type-not-supported-error)
and
(define-socket-condition -4025 socket-type-not-supported-error)
is that the former takes the value from real existing C/C++ header file (in this case it is sys/errno.h), but the later uses some hard-coded number. And this is the reason SBCL declined the patch for BSD Sockets.
Also on the topic of BSD Sockets: Where the function showdown() is defined? It is related to ipv4 support. I have 0 knowledge of network programming, but SBCL tries to call this native function in sb-bsd-sockets contrib module.
EDIT: I see the following declaration in os/supportErrors.h:
#define ENOTSOCK B_TO_POSIX_ERROR(B_POSIX_ERROR_BASE + 44)
Does this Haiku error code correspond to FreeBSD error code No, this is different error, also related to sockets.ESOCKTNOSUPPORT (code number 44)?
EDIT2: it seems the function name in the email to me (explaining why the patch is declined), the function showdown() was misspelled. It maybe wanted to be int shutdown(int, int) declared in sys/socket.h FreeBSD header.
I see similar declaration in Haiku in posix/sys/socket.h:
int shutdown(int socket, int how);
The reason I suspect misspelling is where it is used in Lisp code:
(macrolet ((define-shutdown-test (name who-shuts-down who-reads element-type direction) ... )))
This error does not exist in Haiku. So Iâm not sure what you mean. Haiku will never use this error because it does not exist. There is no hidden numeric value, it just never can happen by calling any function in Haiku. So there is nothing to handle.
I donât know where that -4025 comes from, it does not look like something that would come from Haiku, so I guess someone picked a random value that doesnât mean anything and will never match with any actual error.
No.
ENOTSOCK is a standard error code if you try to do a socket operation on something that is not a socket. For example if you try to do this:
int fd = open("/tmp/test", O_RDWR);
send(fd, ...); // send() works only on socket, but the fd is not a socket
It looks like ESOCKTNOSUPPORT on other OS can happen if you try to create a socket with an invalid combination of socket type (SOCK_DGRAM or SOCK_STREAM) and address family (AF_UNIX, AF_INET, âŚ) in a call to socket(). So, that means the address family exists, but is not compatible with the socket type.
I guess on Haiku you may get a less precise error code such as EAFNOSUPPORT or EPROTONOSUPPORT, but these are likely already handled in sbcl, and you canât distinguish the two cases (an address family that doesnât exist at all, from one that exists, but is incompatible with the requested socket type).
hello @alpopa
as mentioned by @PulkoMandy the error code -4025 does not originate from FreeBSD or Haiku at all. i am trying to remember where i ran across this (i think it was something libuv-related or another haiku port patch where there was a similar problem and in that case used the value -4025. that is if my memory serves right, i will have to double check where exactly i found it but this might take awhile (no promises but i will try).
while that number lets sbcl compile and allowed me to move on with looking at other problems for the sbcl port. i never really came back and found a suitable solution (and kind of forgot all about it). i did not really like this magic number (and still do not). i had been quietly wishing ESOCKTNOSUPPORT might magically show up in Haiku errors one day but did not want to trouble the developers about adding this in if there was not any other compelling use cases for it existing in Haiku.
I discussed the situation also with SBCL. In case ESOCKTNOSUPPORT is something Haiku does not have, we can try to #-haiku (similar #ifndef __HAIKU__) any references to it (in constants-unix.lisp and sockets.lisp). This probably is âthe right wayâ of doing things.
My question is (again, I have zero experience of socket programming): Is it possible that Haiku host to connect some other non-Haiku host (Linux, FreeBSD etc) with a socket and that host to return ESOCKTNOSUPPORT error code. How Haiku is supposed to handle such situation? I mean, the situation when Haiku recieves this error code from some other host (by any mean: connection, virtualization, etc) that supports this error code?
No, error codes donât travel over the network.
Working further on threads support, I gathered the latest patches and followed suggestion from SBCL team to exclude commented out functions and instead of conditionally include <OS.h> in thread.c to include it in haiku-os.h. SBCL compiles, but tests fail, first being arith-slow.pure.lisp.
Using --with-sb-thread, the fail is:
Fatal error encountered in SBCL pid 2589 pthread 0x10cf541519c0:
Unhandles SIGILL at 0x1100c1430e.
EDIT: This seems to be the configuration problem: I have previous SBCL version (2.2.4) installed (the current is git version after 2.4.11), and build scripts find the installed version. So the code is built using one version and run using another. After uninstalling SBCL in Depot, this error is apparently gone.
Or:
;;; Running (LOGAND :COMPLICATED-IDENTITY)
CORRUPTION WARNING in SBCL pid 1656 pthread 0x11e806efcb80:
Memory fault at 0xfca6a302e4 (pc=0x20101076 [code 0x20100521+0xB56 ID 0x0],
The integrity of this image is possibly compromised.
Exiting.
0: fp-0xfbff15ff60 pc=201001076 SB-C::UNWIND
test failed, expected 104 return code, got 1
I see a hack in patches for SBCL recipe that effectively disables threads in arith-slow.pure.lisp, but if applied, the same error is thrown elsewhere. More than that, after application of the patch and compilation of SBCL not using --with-sb-thread I see other errors in tests. Is this misbehavior known in existing SBCL 2.2.4 or is this a regression from that?
The current patch is available here (link is valid 1 week). @hoanga, how can you please review / retest this. Am I missing something?
Found the source of error text âUnhandles SIGILL atâ, it is in src/runtime/x86-64-arch.c:
void
sigill_handler(int __attribute__((unused)) signal,
siginfo_t __attribute__((unused)) *siginfo,
os_context_t *context) {
unsigned char* pc = (void*)OS_CONTEXT_PC(context);
if (UNALIGNED_LOAD16(pc) == UD2_INST) {
OS_CONTEXT_PC(context) += 2;
return sigtrap_handler(signal, siginfo, context);
}
// Interrupt if overflow (INTO) raises SIGILL in 64-bit mode
if (*(unsigned char *)pc == INTO_INST) {
OS_CONTEXT_PC(context) += 1;
return sigtrap_handler(signal, siginfo, context);
}
fake_foreign_function_call(context);
#ifdef LISP_FEATURE_LINUX
extern void sb_dump_mcontext(char*,void*);
sb_dump_mcontext("SIGILL received", context);
#endif
lose("Unhandled SIGILL at %p.", pc);
}
It means, the signal was not successfully compared with either UD2_INST or INTO_INST. This may be due to the fact in Haiku the matching should be done in some other way, e.g. as signed int.
Merged threads support. This commit does not working (yet) as expected and needs further investigation and fixing: https://github.com/sbcl/sbcl/commit/83617b9822ad7e1b541b9613895c073f93242e3e.