As the “de facto Build Guy” of the SQLite project, and along-time member of the Fossil SCM team, i would like to offer my assistance to whomever is maintaining the Haiku packages for those projects. It is my hope that they will see this or that someone who knows them will point out this post to them.
Specifically:
Are there any aspects of those builds which can be tweaked to better support Haiku?
Are you maintaining any code and/or build patches which might be applicable upstream?
i cannot promise to integrate all wishes, in particular source code changes (because of licensing tarpits), but i’ve been known to go out of my way to tweak those builds to improve their portability, including a small handful of Haiku-specific tweaks to date.
SQLite replaced the Autotools with Autosetup early this year (v3.48 for the canonical build and v3.49 for the now-misnamed “autoconf” build) and the new one does not reference -ltinfo.
i’ve been running the canonical builds of both Fossil and SQLite on a Haiku VM on a regular basis for several months, and both of those work as-is.
The Tcl extension included in the “autoconf” bundle also builds and installs as-is on Haiku but it has to do some acrobatics for the installation paths, as the ones reported by Tcl are literally useless for that purpose because they’re read-only:
I wanted to use fossil on Haiku, and thought it might be a good candidate for Haiku itself in the future, barring mimetype support, but that discussion didn’t go over so well
Frankly, i cannot recommend Fossil for OS-scale projects. Fossil has architectural limits which cause O(N) slowdowns, where N is essentially the number of files in a given checkin. That’s fine for a thousand files, or even single-digit-thousands, but tens of thousands or more can bring it to its knees in terms of run-time. (It’ll work, but certain operations are too painfully slow for practical use.) Fossil was literally designed to support SQLite, and it targets small-/medium-scale projects with a single developer or a small team. i.e. it targets the scale of 99%+ of FOSS projects. OSes, LibreOffice, any modern browser, and similarly large projects would find fossil painful.
The Tcl project uses fossil, and that’s one of the larger (tree-size and team-size wise) that i’m personally aware of. AFAIK they have no scaling issues, but BSD’s pkgsrc repo once attempted to use Fossil, highlighting architectural limitations which make it painful to scale to projects that size (with no sensible fix available - these are architectural limitations).
@jprostko used to maintain the Fossil port for Haiku, but we have not seen him around for a while I think.
I know js was also a Fossil user (not sure if he still is) but I’m not sure if he had commit access there.
In any case, @Begasus is our “de facto ports guy” so you could work together on updates if there is a buildsystem change needed.
The normal way to handle this in Haiku would be to use find_directory (available as a shell command or as a C API) to locate the appropriate directories. There are 4 of them, a matrix of user/system and packaged/non-packaged. Well behaved software should check all 4, preferably starting from the user ones, and in there, the non-packaged ones first (this order allows easiest overriding, of system files by the user, and then of packaged files by files outside of packages).
We have patched Python and Perl to do this, but maybe not TCL and other scripting languages and packaging systems yet.
1 - WARNING: Device “/dev/null” is not available for reading or writing.
2 - WARNING: Device “/dev/urandom” is not available for reading. This means that the pseudo-random number generator used by SQLite will be poorly seeded.
These two messages appear on the “Server Administration” page. One of them seemed to be related to “chroot()” and the other was related OS defines for single-user vs multi-user. Sorry for the vagueness, it has been awhile since I looked into these warning messages.
3 - running “fossil ui” hangs
For this one, I am not sure if this is a “me” problem or a Haiku problem. But I think it is related to IPv6 and WebPositive.
Any help or suggestions would be greatly appreciated.
Now that’s a name i’ve not heard in ages. The fossil forum log says we last saw Joe very nearly 7 years ago.
js stepped down a few weeks ago over differences in fossil’s development direction. In short, his ambitions for it surpass the more conservative approaches Richard (the project lead) and myself (a lieutenant, for lack of a better word) both preach and practice, which was understandably frustrating for him, so he left the project. i had hoped to recruit him (and his abundant development energy!) into libfossil, where he’d have more room to experiment than he would in fossil(1), but he chose not to.
(@nephele it may well have been js you were thinking of.)
Great, thank you. It would be premature of me to commit to any activity more involved than that right now, but… stranger things have happened, so who knows?
Those two come from sqlite. The urandom warning is harmless, it just means that the initial random state is less unpredictable (it uses the current time() for the seed).
… poking around…
Ah, the /dev/null warning appears to be part of an old workaround for this issue:
TL;DR: way back then, an assert() in fossil(1) overwrote part of fossil’s own source repo db, overwriting part of the user table (thankfully no SCM’d state) because the db handle was opened on file descriptor 2 (stderr). In response to that, sqlite actively avoids opening file descriptors (1,2). Opening /dev/null looks to be how that aversion was implemented: it’s opened to increase the next available descriptor until sqlite gets a descriptor >2.
(Disclosure/clarification: though i’m on the sqlite team, i’m not a core dev there for the simple reason that my brain isn’t big enough to follow much of what goes on in there. My responsibilities in the project are administrative and in some of the higher-level pieces, most significantly the newly-rewritten build process, the JS/WASM pieces, and the JNI pieces. In the fossil project i’m one of the two current core devs, alongside project lead drh/Richard.)
It may well be an ipv6 thing (not my area of expertise, by any means), but it may also be a locking thing: i’ve had locking issues with fossil on Haiku. IIRC (which is not a guaranty), the main symptom was “fossil patch push/pull” being unable to work because a db lock held by a parent process keeps the child process from modifying the db, but i didn’t poke into it at the time and wouldn’t really know what to do about that (see above my brain ;)).
Ah okay. That does make sense, and is convincing. Still a bit sad about the minetypes though, even for my own projects this would be immensely usefull, and still not available in any current csm it seems. Oh well
By mimetypes i assume you mean something similar to what SVN offers (which i only vaguely recall seeing and never used). Fossil’s “tag” model actually supports adding arbitrary tags (key/value pairs) to arbitrary blobs, so you can even tag a tag of a tag of a tag, as tags themselves are artifacts blob.
There’s hypothetically all sorts of things which “could” be done with that, but nobody’s stepped up to experiment with it beyond the more obvious things that fossil currently does with them (like closing branches and amending checkin metadata, like updating commit messages after the fact or changing the user name associated with a checkin).
Applying a mimetype to a file name would be trickier because file names are a bit ephemeral (for lack of a better word). Tags have to be applied to a record with a persistent hash, whereas filenames aren’t recorded in a way which the architecture can currently tie metadata to. Certainly that’s a case to which the “one additional level of indirection” which proverbially solves all known comp-sci problems could be applied, but it’s never been explored.
Haiku saves mimetypes on most files in the BEOS:TYPE extended attribute instead of relying on file name extensions. Without support in csm for storing this and other xattr data builds often need rsc files to generate the correct extended attributes at build time,
for example the webpositve bookmarks in haikus source are in a zip file, with a wip patch to move them to a ressource file instead.
Getting something like that into fossil would be a very hard sell. Both Richard (project lead) and myself firmly believe that unportable constructs, up to and including symlinks, do not belong in a portable SCM.
We’re proponents of having that sort of OS-specific stuff in OS-specific build scripts.
Fossil does support symlinks, after a fashion, but if we could go back in time and undo that, or remove it without invalidating existing repos, we would. (It was added long ago by a former contributor who absolutely insisted it was necessary for him, but it’s now seen (by the two of us, anyway) as a mistake we have to live with for backwards compatibility’s sake.)
The one obligatory exception is the +x bit, which is ubiquitously used for configure scripts but is of course meaningless on Windows and certain filesystems found elsewhere. Very early on (through sometime in early 2008, IIRC) fossil didn’t support that, and anyone who cloned a repo always had to “chmod +x configure” afterwards, which quickly got annoying, so Richard added that exception for sanity’s sake.
Likewise, if you run into any packaging hurdles or annoyances in SQLite or Fossil, post them here and i’ll be happy to take a look and, if at all possible, adjust the upstream trees.
My apologies for the delayed response, but the forum was upset with me:
Fair enough! (PS: as someone who moderates FOSS forums every day, this forum software is the nicest i’ve ever dealt with.)
I haven’t looked at sqlite for a while, should check repology what our status is on that and thanks for the offer, will surely post here when I (or someone in our haikuports team) come across issues or build errors. +1
EDIT: call me blond seems I already had a recipe for 3.50.2.0, I didn’t push anything yet as I saw the SONAME has changed a lot and adding this as is will probably brake existing packages relying on it.
Having said that, 3.50.3.0 builds and packages fine, I only can’t get it to detect libedit even after adding --editline to configure, config.log isn’t really helpful on that part.
EDIT2: Now I’m confused after switching to libreadline:
Checking for line-editing capability...
NOTE: the local libedit uses <readline/readline.h> so we
will compile with -DHAVE_READLINE=1 but will link with
libedit.
Using editline flags: -I/boot/system/develop/headers -L/boot/system/lib -lreadline
EDIT3: still had --editline with configure, looks ok when using readline:
Checking for line-editing capability...
Using readline flags: -I/boot/system/develop/headers -L/boot/system/lib -lreadline
Line-editing support for the sqlite3 shell: readline
I can kind of understand that, but at the same time stuff like filenames that are longer than 8chars or have any non-ascii in them are also non-portable. Every csm has to have a baseline that it wants to support, there is nothing truely portable about a csm since the entire file metaphore is an abstraction already.
Fossil cannot run on DOS, so that particular one is a non-issue. Fossil does impose some filename character limitations for portability reasons, but it isn’t particularly onerous in that regard.
Fair point. We tend to err on the side of conservative in that respect, but we also expect every supported systems to be able to handle UTF-8-format filenames (possibly via a platform-specific encoding coughwindowscough). We’ve recently had some complaints about fossil not properly supporting emoji in filenames, to which we can just respond with “then don’t do that” and shrug our shoulders.
We removed the soname by default because, frankly, it’s a “platform-ism” which doesn’t involve us at the library level - it’s a detail relevant solely for packagers and people who install sqlite system wide (something literally nobody on our team does - we use our local copies for everything). Despite having coded on Linux since the 90s, i’d literally never heard of SONAME until reworking the sqlite port (late last year) and people started asking us “where’s our SONAME? Why’d you take away our SONAME?”
Through that we discovered that some platforms’ packagers (namely OpenBSD) were going out of there way to shoehorn their own SONAME into the build, so we took that opportunity to support outliers like that: see the --help for the --soname flag, which provides a lot more flexibility that the previous hard-coded one but also requires that the person using it knows they’re using it (which, IMO, they should have to know if they’re going to use it, because it has far-reaching implications (for them, not for the library, so we consider this to be a client-level detail)).
There’s a long “design decision” article about that at:
That summarizes what we, as a team, know, or believe to know, (or strongly suspect (perhaps erroneously!)) about SONAME.
Sidebar: SQLite’s historical SONAME was initially provided by the Autotools and was never once touched by an SQLite dev until the build rewrite. It was voodoo for us, so we removed it. Getting away from the Autotools’ multiple levels of voodoo, and taking back control of our build, was the main driving factor for the build rework. Plus it builds much faster without libtool, which is significant for people who compile it dozens of times a day like we often do.
editline has unfortunately proven challenging to detect reliably because on some systems is masquerades as readline, providing a readline.h but a libeditline.so/a, and that readline.h is sometimes in editline/readline.h and sometimes in readline/readline.h (which makes it looks like readline). Any hints on how reliably detect it would be welcomed.
i’m making a note to try this on Haiku (my machine is turned of because my wife needed that table, but it’ll be back on this evening), but i don’t remember being having tried editline on Haiku.
seems I already had a recipe for 3.50.2.0, I didn’t push anything yet as I saw the SONAME has changed a lot and adding this as is will probably brake existing packages relying on it.