Haiku kitchen package builder

Software
1

Just an interesting screenshot of a Haiku x86_gcc2h package builder being controlled remotely by kitchen churning out fresh ports.

builder_1.png1024Ă—768 138 KB

6 Likes
2

So we have a builder for x86_gcc2? great news :smiley:

When can we expect a repo online for testing?

3

Could give a bit more details on how this Haiku is installed? Bare metal, VM?
I’m interested to deploy one or more Haiku VM on a rented server, but I’ve no practice yet on such situation. How to boot a VM (let’s say Qemu running on a Debian) with Haiku, how to remote access graphically, how to setup to start automatically the kitchen client, etc…

4

Can I ask why you’ve switched back to using haiku kitchen?

IIRC it had issues because it had a separate (its own) dependency resolution instead of the one used by haikuporter. Has this been resolved? What was wrong with the previous builder?

(I’m just getting confused about where the project is wrt automated package building, seems to have yoyo’d quite a bit!)

5

@Munchausen, great question!

I ok’ed going back to kitchen (we never actually deployed it) because we kept encountering day-to-day management issues around haikuporter buildmaster. I had to document buildmaster because little to no documentation existed. I already had to wrap it in a docker container because the requirements were pretty complex. I was literally resorting to writing wrappers around wrappers to manage buildmaster buildruns like a service. buildmeister

Problems:

  • Single-shot. Doesn’t run as a service.
    • Somebody has to log in to a VM to start the builds via the command line.
    • You could argue that we could script our git hooks to start builds, etc.
      • Who will maintain the large network of shell and python scripts to make this work reliability?
      • What happens when someone new has to perform OS upgrades?
      • Who is going to document how to re-create the network of shell scripts if there is a large failure?
      • We need a system that is simple and just “works”. Microservices are great things because they are clear, and have an easy to document contract. (REST call here, this happens… etv)
  • There is no client, Every builder has to be accessible over ssh.
    • I don’t feel great about putting a large number of Haiku machines on the internet.
    • Home users running buildslaves have to configure their firewalls, setup NAT ports, static IP’s, etc.
    • Home users can also run reverse ssh tunnels, but those are prone to disconnect.
  • Weak and confusing status information.
    • Clients must parse a few MiB json document served over http (which is literally just Python dumping internal state data) to figure out what what is going on.
      • Weird status information in the json dump. {“Offline builders”: [{builder: XXX, status: “online”}]}
  • Bugs, nobody with advanced python skils + free time to fix them.
    • No maintainers.

None of the original authors for Haikuporter buildmaster stuck around to maintain it. At this point in time, we have grown a bit, and I personally came to the conclusion that I can’t work on deploying and stabilizing a (complex) new solution in a language i don’t love when the original authors are MIA.

Now… About kitchen. I’m not a node.js fan (and definitely not a javascript fan), but:

  • The author (@waddlesplash) is around and helping to deploy the solution.
  • Kitchen runs as a service with features like an IRC bot.
  • Inbound builder SSH not required. A client runs on the haiku machine to get work and perform it.
  • It works, and requires a lot less hand-holding.
  • A lot more people know javascript / can “figure it out” vs Python.

kitchen isn’t perfect and lacks a lot… but I think there is a lot of room for it to grow and is a more acceptable solution.

Now, if someone truly stepped up and did a lot of work to improve buildmaster (without breaking haikuporter which is tightly coupled to it), we could go back… but i’m not expecting that to happen.

6

Deployment steps:

  1. Started a VM on my desktop.
  2. Git clone kitchen
  3. drop in builder.conf generated by the kitchen server
  4. install.sh

System reboots, starts doing builds.

1 Like
7

@kallisti5

Aside technical considerations (which are interesting however):
for the end users when will be possible to get benefit from these packages builder?
I mean by using pkgman and HaikuDepot to rely on these repositories, for updating and installing most of the hpkg from the many recipes which are available.
Anyway: many thanks! :+1:

8

Eventually everyone’s repos will be pointed to packages generated by kitchen. We need to figure out our global package mirroring, etc though before that happens.

9

IMO some bugs you’re complaining exist in haikuporter and in kitchen, the only difference is we’re now relying on waddlesplash as the maintainer of the buildmaster. Now we depend on one python recipe build tool, haikuporter (really, advanced python skills?), and one javascript buildmaster.
The question of munchausen still stands (“it had issues because it had a separate (its own) dependency resolution instead of the one used by haikuporter”). kitchen still doesn’t seem to understand requirements of recipes, which seems to me it’s a rather big weak point (it tries to build recipes for which package versions are missing/disabled). As I see commit emails, the haikuports recipes are now coerced to workaround this, which I find disappointing.
I suppose the “team” has to accept what comes out of this anyway, as you said, nothing will be perfect: a human will have to come and check/patch/improve what comes out of kitchen.
Looking backwards, I’m wondering though if the Haiku team shouldn’t simply release something more manageable, like a release image, and leave the rest to other teams (what Pulkomandy was about to do I think).

10

I had a rather good experience with mmlr’s solution, it was working just fine for me. The problem with the slow web interface was not related to JSON file being large, but to me trying to write javascript (yeah “easy language”… I’d rather invest my time in learning Python). It has been solved by an upgrade to the web frontend code which is now working just fine.

I think mmlr solution building on existing tools and technologies (haikuporter, ssh, json) was just fine. But it seems the current sysadmin team strongly disagrees and I’m not going to fight against them as they are doing the work.

As for the release plan, what I am (or was?) going after is a minimal set of packages in the repos. Ideally just what we need to build Haiku (libpng, libedit, mesa, …). These things have to be managed using jamfiles as we do currently. If the kitchen can provide a complete repo with a lot more software, then that’s all fine. But, I had started working on a much smaller set of packages (this is the “release” branch). When I tested mmlr repo which was built from trunk, I hit too many problems because some packages were updated and Haiku was not compatible. The situation may be much improved thanks to your work with proper naming and versionning of the recipes, now, so we’ll have to try the repos generated by Kitchen and see how it goes.

11

Your using leading phrasing here. We have the author to help fix bugs in kitchen (and is fixing bugs). Over two years of haikuporter buildmaster shows I’m the only one interested in maintaining it.

By helping to switch to kitchen, I’m abandoning my own year of work. I think that says a lot about buildmaster and my frustrations. Everyone is quick to throw stones, nobody is willing to lend a hand (besides pulkomandy who has been awesome in helping set things up and fix some of the horrible JavaScript ui issues)

1 Like
12

Small side note, with the reduced workload using kitchen, I setup the haikuports repositories which provide access to kitchen’s live build packages:

https://eu.hpkg.haiku-os.org

This repo may change, and isn’t official until it’s tested better. Don’t spread it around yet.

13

[quote=“kallisti5, post:11, topic:5897”]
Your using leading phrasing here. We have the author to help fix bugs in kitchen (and is fixing bugs). Over two years of haikuporter buildmaster shows I’m the only one interested in maintaining it.[/quote]

For the perspective, here is your own comparison one year ago:

Well, people help as much as they see fit, and I can’t blame them after some past emails.

I certainly appreciate your contributions and the time invested for the project, thank you. About throwing stones I just prefer to be critical today than in six months when it will be too late to steer the wheel…

14

Thanks for providing that link, it showcases my same complaints in the past :slight_smile:

I think it’s fair to say that anyone with comments about this should have done the following:

Read the code.
Setup, manage, use buildmaster.

I was 80% onboard with it after getting some community input and trying to remove my own biases.

After around 6 months to a year of hacking on it I had to stop and realize I was dumping man hours into something I really don’t agree with design wise (mostly no clients and not running as a service) and writing wrappers to try and make it viable solution.

I have to ask, do you have a few hours a week to do the following work:

  • Make it run as an http rest service
  • Make the clients poll for work from the server

If buildmaster had those things, we would be using it right now and not kitchen. Those are important features so we can grow it long term. Those features also mean buildmaster core functionality ends up not being a bunch of hacked shell scripts.

At this time, nobody has stepped up to do the work, and I don’t have enough faith in buildmaster to do it myself. (And I have trouble finding motivation in writing things in langues I don’t like when I’m not getting paid to :smiley: ).

Package repos have been one of the final few blockers to r1. Our developer base is dwindling. Let’s just get this crap done already. (Which is what kitchen does)

1 Like
15

I did not make these changes because I still don’t understand the need. The json and ssh solution seemed to work just fine for me. We will need some kind of ssh access to the build machines, at least for debug purposes, anyway. And the JSON was in no way a problem, once the javascript to parse it was written the right way and not 1990s style.
As for use of Python, we will still need haikuporter anyway. So I’d rather spend my time learning Python and taking care of buildmaster, than learning both Python for haikuporter, and node.js for kitchen. So, don’t expect any kind of help from me with Kitchen. And do expect complaints about how it can’t get right things that haikuporter is doing fine and has been for years (such as resolving dependencies).
If you can live with that and are taking care of the code, that’s fine.

16

The JSON was in a good format, but the data wasn’t formed properly for parsing directly into a web interface. Using one of the examples I provided:

{"Offline builders": [{builder: XXX, status: "online"}]}

Since the json is a dump of internal status information while the build is running, we see oddities like:

  • Builders in the offline category which are listed as online.
  • Builder status is from “the last run” and doesn’t represent the actual builder status.
  • Older builds showing offline builders when builders could easily be online.

Some of this could be fixed by javascript adjustments, and some could be fixed by python fixes to improve that internal state hash. However, that would require someone actually making improvements to the python code, see complaints above. :slight_smile:

I was actually working on fixing this stuff with my Rust wrapper by introducing a REST endpoint actual live builder information: https://github.com/kallisti5/buildmeister/blob/master/src/builder/mod.rs#L107

17

I got some skills in the last years in Python, Flask (web microframework) and REST API, Swagger-UI included.
Maybe I could help somehow?

18

Go for it! I’d say #1 is make the buildmaster a service with REST endpoints to:

  • Start Builds
  • Stop Builds
  • Return Build status
  • Check Builder statuses

Also nice to haves:

  • A REST hook to spawn scoped builds from things like github triggers on commits.
    (New commit, 1234acbd - 4567abcd)

Ideally, haikuporter buildmaster should run “all of the time” and start + manage build runs.
It also should manage the buildrun directories (right now it’s all shell scripts :-|)

You can get some initial ideas here: (you don’t have to continue using rust though :wink:, the idea is to work in python and do some major rework of hasikuporter buildmaster)

Here is buildmeister (the REST service I was working on):
https://github.com/kallisti5/buildmeister

Example REST routes:

I’d recommend forking haikuporter into your own account, creating a branch and working from there. Then you can do a pull-request to get your changes in.

https://github.com/haikuports/haikuporter/tree/master/buildmaster

I updated the README with some great starting out information. The Dockerfile also documents a lot of documentation on the requirements to get started.

Let me know if you need any information or help.

19

Er… @phoudoin, I don’t think working on the Buildmaster is the best use of your time at this point. The Kitchen is essentially up and running now, and the last few bugs are getting ironed out (nearly all of the build failures at this point are legitimate failures that need to be fixed in HaikuPorts, e.g. Issues · haikuports/haikuports · GitHub)

While there are now more donated builders, I’d still like to try and run builders inside Google Compute Engine, as well as make better use of the ones we already have – perhaps you could resume work on the virtio_net driver?

Yes, the Kitchen at present does not understand different versions of packages that provide the same things. Adding this to the dependency solver would be, probably, another 25-30 lines of code – however I note that the complexity factor of the module it would be added to is already through the roof, so, I was attempting to avoid that if at all possible. So far, I haven’t found any major roadblocks, so tweaking HaikuPorts to not have multiple versions of any one package on any arch seems to be the way to go here.

20

I continue to see that as a blocker. But it’s again only my opinion (for what it’s worth these days).

1 Like