It appears to use CUSE, which is part of FUSE but not really. It doesn’t provide filesystem, but character devices in /dev that are backed by a userland implementation. I don’t think our FUSE compatibility layer neither userlandfs would support that.
Anyway, I’m not even entirely sure this would work. It can only control when new jobs are started, and gcc usually eats memory quite slowly over the course of a minute or so. So if you used the free RAM at the start of the build to decide to start a lot of jobs, when these jobs start growing, there’s nothing you can do about it after they are started.
It can be tested, but given the complications of porting it, maybe think carefully about that before investing so much time in porting it just to be disappointed at how it behaves. Maybe start by building WebKit with it from Linux to see if it really solves the problem first?
Yes, it does use CUSE, but the files do not need to be in /dev, and in userlandfs they cannot be either.
I think the behaviour of when jobs are available or not should be tuneable, for example for my 32GB system, use the 2GB/job rule of thumb madmax mentioned, but also only allow a new job when 4GB are available. Sure, in some cases that may still eclipse the limit, but in these instances webkit usually has severall jobs that behave like this. The first one to finish just doesn’t get another run, and the others can work with some headroom.
Your suggestion of trying it on linux first is a good idea in any case.
I think we could also implement the behaviour ourselves, I’m just not sure where that belongs. Would the roster be a good place to have something like this?
This seems really specific to buildsystems, I don’t see why we should have to add specific code for it in standard OS components.
According to POSIX Jobserver (GNU make) the interface seems quite simple: a named pipe where clients read one character to get a token, and write back the same character to release that token.
It seems not very complicated to implement that as a separate application. Then you just need the logic to be smart about when to release tokens (by watching system free RAM). I don’t see what you gain by trying to integrate that into the roster, since your code will probably not interact with any parts of the roster at all?
Building some new long-lived server for this quite small functionality seems a bit overkill to me, hence the idea to integrate it (as another thread) to an existing server or so.
Atleast for the stevie jobserver the “normal” named pipe was not enough to properly cleanup tokens when for example your compiler or make just dies/gets killed etc, as it then does not release the token. Hence the cuse backed files.
I suppose we could just patch the interface for Haiku to be a bit smarter? Though that would require some design
The effort to integrate it into another unrelated app is going to be higher than just writing your own main() function and maybe a launch_daemon script, I’m pretty sure.
Are you going to change the protocol? If so you lose compatibility with make, ninja, and every other tool that understands the buildserver protocol. I don’t think that’s a smart move.
So that’s why stevie uses a custom thing in /dev that’s not really a named pipe. Then it can know the PID of the process which owns each token, and watch them. Since we don’t have anything like CUSE, if you were to do it with userlandfs or FUSE you’d have to simulate an entire filesystem just for that one file, which seems very much not ideal and complicated.
On the other hand, writing a kernel module that exposes the needed information as a device in /dev shouldn’t be so difficult. And then maybe other parts of stevie can be used, with just the way it interacts with that device modified, for example.
I’m think we already have character device files. Just not the ability to shove them into /dev, which also isn’t required here(?).
If i understood waddlesplash correctly we can do this with userlandfs just aswell, just mounted somewhere else. Or is that what you mean with simulating an entire filesystem?
While a general system-wide solution with process watching and all the stuff would be nice, a simpler thing you start and stop just for this kind of build would still be useful. Do we already have something that assumes everything goes right and just limits the tokens depending on load, RAM and maybe the user telling it to add or remove tokens from the pool in case one notices there are more or less jobs than should be running? If we don’t, that should be easier to write than the full-featured one, I guess.
I don’t know of any for Haiku. As far as I can see the “steve” jobserver from Gentoo basically does what would be required, but it depends on /proc and CUSE as written aboe.
I assumed that porting it would be easier than making a “barebones” one myself, but maybe having a small one for Haiku would be the better solution (in any case, the author hangs around in #gentoo-toolchain on libera, they can probably answer if we have some more questions about the design of steve or jobservers in general)
EDIT: and If i understood correctly they are open to Haiku compatibility patches upstream
GNU Make already provide a jobserver mode, but it’s much simpler than that, just a fixed number of tokens that can be taken and released. The initial idea in Make for this was not to have the jobserver manage system load, it was just to make the -j option behave in a way that makes sense when you have a complex build involving recursive invokations of make. But then people took it into entirely different directions, for which the protocol wasn’t initially designed.
I’m not sure what you mean exactly. The notion of character devices vs block devices doesn’t exist in Haiku, but it also isn’t very relevant.
You need a file somewhere on the filesystem hierarchy, that apps can read and write to, and you need to know what they read and decide what they write. This part is trivial and there are many ways to do it. But then you also need to know the PID of the readers/writers. That’s a bit of a hack, and I’m not sure a userlandfs solution would allow to do it. But a dedicated driâer certainly will, and that seems much simpler to set up to me. Just put your driver in the appropriate kernel/drivers directory, the kernel loads it, the device is published, you’re ready to go. No need to go through starting an userlandfs server, invoking a mount command, and having your filesystem add-on implements a bunch of vnode publishing and reference counting just to publish one single file.
Going that way would mean you’re afraid of writing driâers that run in the kernel, and are willing to take a much more complicated path because of that. I personally would take it as a failure in one of Haiku “side goals”, which is to show that kernel and driver development is not wizardry, and that if a developer can write software, they can naturally write software that runs in the kernel. I am annoyed by projects that pretend otherwise and gatekeep kernel work.
I’m more afraid that a kernel driver will not be upstreamable becuase of it’s perceived “relevance”
(And I’d not distribute a third party kernel driver, because I don’t trust third-party kernel drivers and wouldn’t want to enourage other people to “just install this kernel module”)
If I understand it correctly your suggestion is basically that the kernel publishes some files in /dev for the jobserver, and does some internal accounting for free RAM etc and manages it completely on it’s own?
What I would do is a driver that publishes a device in /dev. The device can be read and written by jobserver clients as specified in gnu make protocol (read one character to get a token, write it back when you’re done).
Then I’d add an extra interface, probably based on ioctl, for the jobserver to control things: get the list of freed and allocated tokens, the pid of processes who own a token, and a write interface to set the target maximum number of jobs.
Then an userspace part of the jobserver can use that interface to control things.
Why do you feel different about this than userspace software?
Process isolation, memory isolation, etc. Atleast for userspace there is the idea or plan to restrict stuff. For example using pledge and unveil from OpenBSD.
Easier to debug, doesn’t bring down the whole system if it crashes
You may need some APIs that are more easily available from userspace
The driver code entirely runs “in the context” of the process that calls it. You could spawn kernel threads to do work in the background, but then you have to coordinate between that thread and the read/write calls from the jobclients anyway. So you may as well move that thread to a dedicated userspace process
I don’t think this needs to be a kernel driver at all. UNIX domain sockets on Linux have a facility to get the PID of the process on the other end of the connection; we don’t have this on Haiku at the moment, but could probably implement it (though, now that I think about it, I’m not sure how this feature works for FDs that are open in multiple processes at once.)
On Haiku, we also have a facility for getting the PID of the process that sent a message to a port. Not as portable as UNIX domain sockets, but it does work.
Yes, if you’re changing the protocol it doesn’t need to be a kernel driver, but then you need to patch Make and Ninja to support your slightly different implementation of it. That’s the framework under which I was thinking here: making it work with something similar to a named pipe, where the client can just call open() with a path, then read() and write(). That is also what Stevie from Gentoo is doing (not with a kernel-side driver, but with a userspace device thanks to CUSE).
So, we could implement some form of recvfrom() for named pipes, that returns the PID of the sender, sure. But I don’t see the point when it can be done with a driver that will be about a hundred lines of code. And allowing this for all named pipes would mean we have to track this data somewhere, which doesn’t really make sense because pipes aren’t datagram-based, they are streams. And so it would make the code more complicated for just one specific use case of working around a quick design choice that has become a de facto protocol.
Well, because then the protocol can be adjusted upstream, and all applications on all *NIX OSes use this setup without needing a kernel driver or a FUSE driver or anything like that.
