We still quite don’t agree on how this thing should run.
I like the simple approach of the current buildmaster design.
The current system is designed as a script that you just run. This is easy. We can automate it using a cron job, or run it as a git hook everytime someone pushes to the release branch. No need for this REST stuff.
The status is generated as a JSON file while the build is running. It is very easy to render a webpage from it (and the work is done already). Of course the build master has no idea about the slaves status when it is not running - it does not even keep a connection to them. This is a buildsystem, not a hardware monitoring solution. It only connects to the machines when it needs them, which even makes it possible to use machines that are not always online. I ran builds using my development machines, which is possible because the system is designed to not be invasive (for example it does not need you to install any packages on the host haiku - it manages the package it needs by itself).
The build run directories are working just fine with the shell scripts.
So, I think the system is ready to go live. We just set up git hooks to run it and basically that’s it. Each time someone makes a commit, a few minutes later the packages are built and added to the repo.
Why so much REST and Docker above this? It looks like just needlessly complex additions to me.
Yes, there are some strange things with the buildslave status, but it looks like this could easily be fixed without rewriting everything (and even without a lot of python knowledge).
1 Like
FWIW, I agree with that too. We need the tool to use haikuporter to solve dependencies, which is why buildmaster was started in the first place.
1 Like
Look. I was told in 2015 that my solution was just “no good” and that the buildmaster was “ready to go”, after quite a lot of bikeshedding over the technical details like this. Well, it’s now 2017, and since the buildmaster was (at least according to kallisti5) still in need of a bunch of work (and I note that it never produced any usable results, since the most important step – repository generation – it did nothing towards) I resurrected the Kitchen and it’s working, generating repositories, and chugging along towards a release.
So to be honest, all the criticism and remarks of “but buildmaster was better because [blah blah blah]” are pretty annoying at this point. We’ve wasted over a year and a half because people insisted the buildmaster was better. Let’s not start again.
Dependency solving is not some mystical rocket science, you know. Sure, it’s tricky, and there are a lot of edge cases to take care of, but it’s not like the HaikuPorter dependency solver is a mystical black box nobody could ever replicate again.
The problem was already mentionned with dependecy solving: recipes can use bash to do whatever they want. And yes, we will need to make use of that. Is the Kitchen ready to parse recipes using bash yet? I think not, as you have been hacking recipes to workaround Kitchen limitations lately.
As for repo generation, what you say is simply not true. mmlr had buildmaster generating repos (running on an Amazon EC2 instance) when he was working on it. So yes, it was ready to go at that point. It was running automatically, generating the repo, everything. But then when mmlr asked for a Haiku-owned machine to host it on, no one ever granted him access to that.
Back then I tried to use the generated repos and had a lot of apps broken and the like. Which is why I started working on the release branch. This is what took a year, not because it was a difficult task, but because I got zero help from anyone, and because I had many personal events in my life reducing my availability for Haiku. And then when I came back with a release branch ready to use, mmlr had already shut down his EC2 instance, and when kallisti5 finally set up a VM to host the thing, he went into that docker and REST crazyness which I still don’t see the use for. Really, mmlr HAD everything up and running on his EC2 instance and buildmaster IS ready for production - save for a little lack of documentation.
This is a little late since we discussed it all at-length via IRC, but I just wanted to tie up this loose end (this forum thread)
mmlr HAD everything running. Then he disappeared.
The docker environment gave me a reproducible and portable environment to do dev + maintenance in given the complex environmental requirements. After a huge learning curve i was able to piece together how things were supposed to work without any documentation. (at least how I thought things were supposed to work, which mmlr now points out I was wrong about.)
As I mentioned in IRC, kitchen was successfully building repositories, and we got it building repositories in a matter of days vs the few months it took us to figure out buildmaster without any documentation.
mmlr’s back now (again) after a year and a half hiatus. Consensus seems to be to go back to haikuporter buildmaster (again). I’ve been burnt several times trying to get the package building out the door over the years when others disappear.
I’m tired of the bikeshed, and the “we should do it like X, but I don’t plan on doing the work”
At this point i’m stepping away from package building for the long haul. We have enough people pushing for buildmaster to support its day-to-day operation for the years to come. If maintainers of the manual buildmaster buildruns dry-up again, it’ll have to sit until someone else picks it up. I hope buildmaster succeeds and won’t do anything to stand in its way.
Where “no documentation” is several mails mmlr sent to the mailing list when we started working on this, and “months of work” is, in my case, maybe a week to get it running on my own machine. Just to make things clear, things could have been better but they were also not as bad as you continue to imply.
And I certainly don’t plan to keep doing manual buildruns forever, but there are many ways around that. mmlr’s system was using cron jobs, and it should be reasonably easy to set up some git hooks or something. But, that’s not the first emergency, manual triggering of the buildmaster is already a massive improvement on the current situation and would free a lot of time from devs taking care of keeping packages up to date currently (korli is doing a lot of work on that side, for example).