Jamfile frustration

Development
1

Problem

I’ve never been able to build Haiku from its Jamfile to completion. Now that I’ve forked WonderBrush v3 in an effort to create a glyph editor, I finally got my Haiku beta 4 32-bit VM set up to slowly build it. Much to my chagrin, Jam got stuck in an infinite loop. The -q option did nothing. There’s the problem: Jam is buggy.

Solutuion 1 Attempted

Last year in a Summer of Code project a student tried to build a Jam replacement called Ham. It was to be a Haiku-specific version of Jam. The project didn’t succeed.

Update Edit

After reading the final report on Ham it seems it was going to target Ninja build after all! I’ll have to give it a look!

My Tentative Proposal

Generate Ninja is used by Fuchsia and Chromium within Google as a frontend for Ninja build. If I understand correctly, it combines thousands of GN meta build files into a single .ninja file in the root directory of the project to allow multiple program builds to compile and link in parallel. It seems their goal is to take full advantage of big data center systems to cross-build and link all at once.

While Haiku’s systems are modest in comparison, old DDR3 based systems like mine are cheap to max out the RAM. Being able to build and link at the same time seems like an advantage. Also, Ninja’s build system is designed to throttle back if RAM starts to run low.

Potential Snags

There are a few ways to do this project:

  1. Put a Jamfile parser into GN or put GN’s backend into Jam.
  2. Write a Jamfile to GN converter.
  3. Find a more compatible system to target instead. (A last resort.)
  4. Fix Jam.
  5. Complete Ham.

License Compatibility

GN has 3-clause BSD and I don’t remember what Jam’s license is. For method 1 to work, the license must be legally compatible. If the licenses are incompatible, method 2 or later becomes more attractive.

Repairs

Methods 4 and 5 don’t require external code at all so there are no license incompatibilities at all but they are integrated all-in-one solutions. This makes debugging difficult and has failed before in Ham’s case.

Possible Migrations?

Methods 2 and 3 are only possibilities if there are migration paths available. I’d prefer to avoid migrating fully to GN outright because that puts Google software directly into our dependency chain. Adding to that, both a frontend and backend are necessary in the case of method 2. Method 3 depends on 2 also.

Closing Thoughts

I hope nobody is offended at my attempt to circumvent the shortcomings of Jam in an unconventional way. I just want code to work. Jam is pretty Haiku-specific already and offloading onto many hands seems attractive to me. Let me know your thoughts!

Edit

Ham section updated with link from Summer of Code final report. See above.

1 Like
2

If you want to change the build system, maybe cmake would be a better proposal, because you could then generate whatever build system you wanted or direct with cmake. The problem with picking a build system is lock in - which you seem to want to remove?

3

I use it without any problem every day to build Haiku and Wonderbrush and other things.

Jam has its quirks, but, for me, that isn’t one.

What? The project made MASSIVE progress. It was clearly a huge success, and went 90% of the way. But sure, let’s ignore that, pretend it was a failure, and start over?

Just increase the value of the -j option until Jam max out your cores or RAM. With Haiku being made of so many small parts, this is not at all a problem.

People keep suggesting cmake, but it has one big problem: it is not designed to build software with multiple toolchains at the same time. We need at least 3: building tools for the host system (and using them in later steps of the build process), building with gcc2 and modern gcc, and for modern gcc, in the x86_64 version, building both 64 bit (for most of the system) and 32bit (for the BIOS bootloader) code.

Just this one thing is already pushing cmake to its limits. Then there is handling of executable resources, which is possible but quirky, and probably a lot more undiscovered problems.

I think experiments with Meson by x512 lead to a similar conclusion: you can’t build “all at once” like with Jam, you have to invoke Meson multiple times for different configurations. And so you need a meta-build-system on top of it, which limits how much you can parallelize the build.

1 Like
4

So, you encountered what sounds like it could be a bug and instead of investigating it you throw out the baby with the bathwater? It’s kind of strange to me this NIH syndrome to keep wanting to completely replace stuff instead of fixing what we have.

Yes, fixing is boring, but it workd very well.

2 Likes
5

Yeah, I never said it would be painless, but if you use cmake as a build system generator, you can effectively achieve something that will at least build the same code for different architectures from the same cmakelist.txt. We have stuff at work that builds for Linux x64, and Android NDK and will eventually build for iOS also. You do need to probably have one top level cmake config that invokes the individual build’s targets though.

What you probably don’t want to do is use cmake directly to build, as that becomes way more painful.

6

This seems more likely to be a problem on your end. I use it to build haiku over and over and over from scratch, including bootstrapping which is very demanding. Also make sure to always use the jam from buildtools, as jam provided by distros break for building Haiku.

I’d investigate the problem first. Check for overheating and do a full memtest on your machine. That being said I would like to see the ninja based build system completed.

7

i’ve came across two issues with jam

  • too much parallelism makes it unstable, it seems that -j4 is the maximum that i’m able to use reliably
  • boostrap build tends to become unstable with anything more than -j1

having said that, i found jam mostly usable, still looking forward to seeing ham completed

8

After having looked farther into Ham, it seems to use a similar technique to GN. I’ll have to look into finishing Ham or at least comparing it farther to other Ninja build frontends.

9

Please no CMake, it is ugly and hard to understand. Jam is easier to understand.

1 Like
10

I have some problems using -j8 related to build_packages extraction. But:

  • You just have to restart the build and it will work on the second time
  • It is probably an incorrect dependency somewhere, rather than a bug in Jam itself. If I remember correctly, it is because for build packages, the dependency is on the extracted directory, and not on the actual extracted packages. So, as soon as the “zlib” directory is created, things start to run that require the zlib.h file inside that directory, which has not yet been created.
1 Like
11

I get the feeling that it would be a wasted effort. This does seem a lot like NIH and a solution in search of a problem. From what I have read above, Ham and Ninja are unlikely to be adopted.

I guess it depends on how you use it. Using it to do the build is not ideal, but using it to generate the build engine files means you can use that resulting build engine from that point onwards, and really only need to worry about cmake if you need to add in more dependencies to the initial generation step.

But if jam works, I don’t see the need to change anything.

1 Like
12

ham is a drop in replacement for jam and will replace jam in the future, once it can build haiku correctly.

13

The last report that was linked in the parent post seems to contradict it being a “drop in replacement”. I think that was the original intention, but the way I read it - this was going to be something that needed some migration, would have new syntax and would leverage ideas from newer build systems, namely Ninja. I think that is what @SamuraiCrow wants to do - make a ham that is more similar to Ninja, not a drop in replacement with compatible syntax and no need to migrate to the newer system. Feels like ham’s original objective is probably still achievable and would be a lot better than trying to re-envision it in to something different. Maybe I misread his intentions, and I don’t want to put words in to his mouth. But, if someone wants to make a new build system that is not entirely Jam compatible, that seems more like a regression to me.

14

The goal was for Ham v2 to be a 2-stage buildsystem, that is, a first step that parses the jamfiles and generates Ninja files, and a second steps that runs the build (probably reusing ninja).

The reason for this would be to avoid reparsing the whole of the Jamfiles at every build (which takes some time in the case of Haiku).

There were no major changes of syntax planned, but maybe some extensions and cleanups (not requiring a space before ; at end of rules, for example). That means the existing jamfiles would work with it out of the box, still, and no migration would be needed in most cases.

15

Hmmm - any significant changes to the Jamfile would still need to be reparsed and new build engine files generated. Wouldn’t it be better for it to be more tightly coupled and for the whole thing to be one package, rather than depending on an external build engine? It seems like the parsed output could be cached and Ham could just use that unless it decided it needs to be regenerated. Once it needs to be regenerated, you have lost any advantages. If you can make the generation phase parallel to the build phase, the generated files can be passed to the build stage as they are generated on the first pass, and then subsequent passes can just reuse the generated files unless they are marked as dirty. Maybe that is how it was meant to work - I didn’t look as the Ham repo much other than a brief look this morning and I have never looked at jam because it was never very interesting and didn’t solve anything for any of the projects I worked on that were not solved by the build system we used already.

16

Another potential isssue is the extensive use of filesystem attributes, this works fine on Haiku, but Linux filesystems seems to be very bad at extended attributes. I use XFS which seems to be done properly, but others are not so good (IIRC ext can’t handle >4KB attrs) which explains why nobody dares to use extended attrs on Linux.

Haiku has a fallback when they are not available, but if it is bug free I do not know.

17

We actually have two fallbacks:

  • If there is some support for xattrs but with limited size (that’s the case for ext4), we use a system where the attributes are stored in a separate file, and the original file has a native xattr indicating where that file is. This one works quite well, it is slower than real attributes but otherwise ok.
  • If there is no xattrs support at all, the path to the attribute file is derived from the original filepath. This has a risk of things getting out of sync, for example, if you delete a file and recreate it without the attribute overlay knowing about it, the attributes remain on the new file.
18

SInce the links about Ham v2 are broken in the GSoC article blog, the development discussions that have already taken place are in the v2 branch. For future reference, they can be linked from here.

19

Jam works totally fine for me, both on haiku and on Linux. What kind of system are you using to try to build Haiku? I used WSL quite a while ago and it was pretty painful, but since moving to Linux it works perfectly.

20

I’m on multithreaded building on Haiku.