Make ANSI Common Lisp available on Haiku (again)

This patch is already included in upstream SBCL. Did you apply the patch again? Or is this some different patch?

I also have seen this problem. It certainly looks as a memory corruption, because it lists symbols, which were present a moment before (and probably all foreign symbols of runtime).

This problem arises much less frequently than previous one about compromised integrity of image, so a new run usually solves the problem.

Yes. Oddly, I did a clean compile again and it was successful with no issues: :partying_face:

creating package sbcl-2.5.0-1-x86_64.hpkg ...
----- Package Info ----------------
header size:                     80
heap size:                 12624831
TOC size:                      3437
package attributes size:        749
total size:                12624911
-----------------------------------
waiting for build package sbcl-2.5.0-1 to be deactivated
cleaning chroot folder
grabbing sbcl-2.5.0-1-x86_64.hpkg and moving it to /boot/home/haikuports/packages/sbcl-2.5.0-1-x86_64.hpkg

real    23m49.896s
user    21m39.618s
sys     1m58.926s
~/haikuports/dev-lisp/sbcl>

This was compiled on Haiku nightly R1B5-hrev58477 x64.

3 Likes

ASDF patch merged: https://github.com/sbcl/sbcl/commit/621f8823520c8f6ff0da876c9b7f1f9cc54d9e9c.

1 Like

SB-BSD-SOCKETS patch is also merged - https://github.com/sbcl/sbcl/commit/da4d0f4fa6d0a7729f0c6ce52be5c375ac6cf232.

This was the last patch available until now. And now, SBCL is “feature complete”. Now, working on stabilizing the port.

1 Like

I am preparing my next patch and need some assistance from Haiku developers. SBCL supports multi-threading via pthread interface. I see there are several pthread headers:

  • <posix/pthread.h> - seems to be the basic pthread header,
  • <bsd/pthread.h> - seems to extend the first one with BSD extensions,
  • <gnu/pthread.h> - seems to extend the first one with GNU extensions.

So, my first question is: When including the 2nd or the 3rd header, there is no need to include also the 1st, right?

Now, SBCL uses the function int pthread_getattr_np(pthread_t thread, pthread_attr_t *attr), not available in POSIX variant, but available in BSD and in GNU variants. My second question is: Including BSD header is probably preferred having linkage against -lbsd.

The third question is how to properly include it:
#include <bsd/pthread.h>
or:
#include <pthread.h>
and maybe predefine something like -D_BSD_SOURSE?

You don’t need to include it at all, the BSD and GNU headers are already in the compiler search paths. And on nightly builds at least you don’t need to define _DEFAULT_SOURCE, it will be defined automatically (unless “strict mode” is enabled in the compiler, e.g. by passing -std=c99 instead of -std=gnu99 or the like.)

2 Likes

Thank you @waddlesplash. Let me recap what I understand:

  1. The different headers are just different interfaces to the same Haiku functionality.
  2. The function implementation int pthread_getattr_np(pthread_t thread, pthread_attr_t *attr) is available by default.
  3. The best way to include it is to not use any #include at all, it is available in compiler search path. Is link option -lpthread even necessary in this case?

the patch looks okay although note the more recent feedback on the include headers. my understanding so far is that while this patch enables more of the threading functionality in sbcl, there are still a/some lingering issue(s) when threads are enabled (as has been observed and also i saw them in personal tests) that might be either runtime related or perhaps something (possibly) needed on the os-dev side to get the threading functionality behave better.

Current snapshot: Jan 3 (9d9492c)
Unit test failed:

  38: fp=0x1279d232de8 pc=0x11007ec2b9 SB-THREAD::RUN
  39: fp=0x1279d232e20 pc=0xdb4e6c5161 Foreign function call_into_lisp_
  40: fp=0x1279d232e40 pc=0xdb4e69618a Foreign function funcall1
test failed, expected 104 return code, got 1

At least, not all sb-thread tests fail, and probably, not majority. I also have an impression that some tests sometimes pass and other times fail. Maybe I am wrong. More rigorous approach is necessary.

Indeed. I would add that in almost all cases Garbage Collection is involved, maybe more constantly than threading. Or maybe the failing is combination of threading + GC. I see SBCL has 3 implementations of GC. Maybe configuring some different one would help (I do not know how).

It may be configuration problem (src/runtime/Config.x86-64-haiku). Or it may be some (missing or wrong) implementation (src/runtime/haiku-os.c, src/runtime/x86-64-haiku-os.c). Or it maybe some Grovel issue.

For now I try to see if failing tests do it consistently. Also, there are several unexpected results (fail, not crash). Maybe analyzing them may give some more hints.

Unfortunately, after applying this patch such issues appear even if SBCL is compiled without threads support, quite strange.

Compiled on Haiku hrev58490 x64.

During this compile, I made note of the “Reference to RESULT-TYPE is implementation-dependent, per ANSI CL 6.1.1.4.” warnings - although no major errors outside of the warnings. Same warnings for SBCL 2.5.0.

Unit test:

// Running /sources/sbcl-master/tests/arith-slow.pure.lisp in COMPILE evaluator mode
::: Running (LOGAND :COMPLICATED-IDENTITY)
CORRUPTION WARNING in SBCL pid 18339 pthread 0x125d6ea11280:
Memory fault at 0xfffffffffffffff9 (pc=0x1100329908 [code 0x1100329590+0x378 ID 0x95a], fp=0x1279d231158, sp=0x1279d231130) pthread 0x125d6ea11280
The integrity of this image is possibly compromised.

The pthread-related unit test that failed - this one stopped the remaining unit testing this time.

On this point, I was doing a clean compile of the latest snapshot without any patching attempts. So, the snapshot still compiles without major issue. The unit testing stopped at the point mentioned. I’ll patch that to continue the unit test review.

1 Like

The compilation of SBCL is clean for me as well. Before proceeding, I always do:

chmod a+x *.sh

Otherwise, the first compilation may fail, but the next attempt succeeds. They use shell scripts for all purposes, and these always have extension *.sh.

1 Like

I made a small change to the receipt:

BUILD()
{
	find . -name *.sh | chmod a+x
	sh make.sh --xc-host='clisp -norc' --with-sb-core-compression --with-sb-simd-pack --without-sb-thread
}

Unit testing - Finished running tests… 109 tests skipped.

Solved the “issue” for now by disabling the SB-THREAD feature.

1 Like

Does the issue really disappear by passing --without-sb-thread? I did not test exactly this argument, I just did not pass --with-sb-thread to make.sh considering default is without. Nice move!

EDIT: Well, it does. It seems sb-thread is enabled by default if the port supports it. Preparing update for recipe without thread support for now. It is much more stable.

New pull request is https://github.com/haikuports/haikuports/pull/11595.

I am in Common Lisp Paradise: latest SBCL works with Emacs+SLIME:

To configure it:

  1. Install Emacs and SBCL from HaikuDepot
  2. Open Emacs and run:
ALT+X package-install <ENTER>
slime
  1. Close Emacs.
  2. Edit ~/.emacs file by appending:
(setq inferior-lisp-program "sbcl")

or, if you have SBCL built from source:

(setq inferior-lisp-program "/path/to/sbcl/run-sbcl.sh")
  1. Start Emacs again and:
ALT+X slime
2 Likes

Congratulations! Nice work too all that helped out. :+1:

My previous attempts to run SBCL with Emacs+SLIME failed. Now, I think the success is due to working sockets interface. SBCL with Emacs+SLY still does not work. But now it loops infinitely during connection. Previously this combination threw error during connection. It seems SBCL needs proper threads support to make Emacs+SLY work with SBCL.

So, having apparent proof of socket interface working in SBCL, I analyzed more in-depth socket interface in CLISP (it also has problems with sockets) in a hope to migrate relevant SBCL changes to CLISP. Recall, SBCL sockets patch removes ESOCKTNOSUPPORT error code and adds EAFNOSUPPORT error handling in tests.

After discussion with CLISP team, I understand that the socket interface is a POSIX standard, which clearly states which error code should exist and should be signaled in which cases. This resulted in the following ticker for Haiku. ESOCKTNOSUPPORT error code is already added to Haiku. When its handling will be also added to OS, the Haiku specific patch for SBCL sockets can be hopefully undone.

Well, it seems Emacs+SLIME works not only with SBCL:

but also with CLIPS:

3 Likes

For SBCL 2.5.x, thoroughly testing SBCL’s SB-THREAD feature for Haiku x64. :fire:

1 Like