Hello, I just discovered this conference:
Anyone wants to prepare a talk about Haiku package system and/or haikuports?
Applications are open until 31 of August. The conference itself is in November.
3 Likes
I’m interested! Looks like fun, and a great way to tell more people about Haiku.
2 Likes
So I’m thinking of submitting a proposal for a 20 minute talk which highlights:
- history of Haiku & Be lineage,
- packages are mounted and files immutable,
- bootloader allows for going back to prior states, and
- some pros/cons to the approach.
Anyone interested to join in? A 30 minute session has 20 minutes of pre-recorded video, and 10 minute Q&A. Or have a 2 minute lighting talk.
6 Likes
My abstract for PackagingCon was accepted! I’ll be taking about the Haiku packing system, and focusing on:
- how the packages are mounted into the file system as read only, and
- how the boot loader is aware of the packing system to load a prior state of the OS.
Anything else I could highlight? Anyone want to contribute?
17 Likes
Cool!
You might want to look at previous talks for hints… I had one at FOSDEM in 2016 (hmm no video, IIRC the recording was broken), but likely there are more up-to-date things around.
3 Likes
I’ll check it out. I definitely want to get smarter on the topic.
Some cool things to mention:
- The package metadata that is stored in the repository and allows to do things like “pkgman install cmd:foo” without having to know which package contains cmd:foo. This makes implementing
command-not-foundon Haiku trivial (it does not need to check which package the command is in) - Likewise, dependencies for libraries are handled by lib:foo instead of directly on package names, allowing to reorganize (split, merge, …) or replace packages (for example use libjpeg-turbo instead of libjpeg) easily
9 Likes
Draft presentation for review: http://files.richardzak.md/HaikuPackageCon2021_draft.pdf
Any feedback would be appreciated! Thanks!
5 Likes
Maybe add a note on how to create recipes (some quick guide or link on the howto)? .PackageInfo contains almost the same information as a reccipe does if you do it manually.
I know, this is only a draft, but after you added 2 more pages for creating a package in Haiku, you ended with pages 14/13 and 15/13 (still counts total pages as 13).
There are several possible directions to dig further into I can think of, not sure how far you want to go 
A bit more history of how we decided to create this package manager:
- The initial goal was to allow updates to the OS itself (without reinstalling it all). Before this, BeOS “packages” were either a zip file to extract anywhere (sometimes containing some manual instructions to copy files in various places) and sometimes Windows-style installer wizards
- One goal of the package system is to allow applications to be easily redistributed, even offline. As an example, a few years ago I was in a train trip and someone asked me if I happened to have VLC on my laptop so he could copy it and watch a DVD. On Windows, this is no problem, you probably have the installer .exe on your download directory somewhere and can copy it. But try to figure out how to do this on Linux? With Haiku apps, there are usually few dependencies, and so a Windows-like solution should be possible. That’s how we came up with the idea of not extracting the packages at all
On the technical side:
- Mention use of libsolv (also used by SUSE I think) for resolving dependencies
- Maybe give a bit more info about the hpkg file format: how the data is split in fixed size blocks (a few kilobytes) and each block is compressed separately, allowing reasonably cheap random access to the data (unfortunately most compression formats are not designed for this, and operate better on a long stream rather than smaller independant blocks)
More “cool features” (you already gave a good overview of the most important ones I think):
- The ability to identify where any file comes from by looking at the SYS:PACKAGE_FILE filesystem attribute (using Tracker “Get Info” or listattr in Terminal). No need for a separate database to keep track of this
- The possibility to reuse the packages in a separate mount point, to create something similar to a python virtualenv, but in a generalized non-python-specific way (used by haikuports to build packages in a controlled environment)
7 Likes
Nice work.
On the Metadata in Packages page, I found bullet point number three a little awkward and hard to understand.
Other packages, and the command line, can specify dependencies on
by referring to the the library name or command without having to
specify which package exactly.
Great idea, I’ll add some recipe information.
That’s because those slides at the end are intended to be an appendix, to be covered if I don’t run out of time, or if there are questions relating to those items.
- So to regurgitate: because the packages are on the disk, it’s easy to provide said packages to someone else in an off-line environment, and all dependencies are guaranteed to be present on the host machine?
- What should be mentioned about libsolv in regards to Haiku?
- Great info to random access to the archive, I’ve never heard of this before.
- Could you elaborate by what is meant by reusing the packages in a separate mount point? Do you mean like a chroot/jail environment?
- Not part of your post, but how much space to the different versions take up? I imagine someone will ask this, that having an aging Haiku installation will take more and more disk space as the packages are updated.
- How could someone remove old packages, say from the beginning of the Haiku installation, for example, to save some space?
Thanks! I’ll see how I can reword that.
It means you dont need to know the name of the “fancynamedpackage” to install it, it is enough if you know for example the name of the command you need.
So instead of
pkgman install imagemagick7
You can say
pkgman install cmd:convert
Comfy, heh?
5 Likes
- So to regurgitate: because the packages are on the disk, it’s easy to provide said packages to someone else in an off-line environment, and all dependencies are guaranteed to be present on the host machine?
Yes. That is the main point of free/libre software, yet the typical Linux way makes it complicated, and centralizes the software distribution in the hands of distros.
- What should be mentioned about libsolv in regards to Haiku?
Nothing special, just mention that we didn’t reinvent the wheel on everything and the Linux package managers do get some things right 
- Great info to random access to the archive, I’ve never heard of this before.
If you want to dive into details, there’s this: Haiku Package File Format — Haiku internals documentation but I don’t think it provides a good overview of the “why”.
For the history of how we got where we are, there is this page Package Management Ideas — Haiku internals documentation with brainstorming notes from the development team (not everything went in the final implementation).
- Could you elaborate by what is meant by reusing the packages in a separate mount point? Do you mean like a chroot/jail environment?
Yes, in haikuporter this is used with a chroot, but it could also be used without one, just like python does with virtualenv. The idea is to mount a packagefs in some directory, and then adjust some variables (PATH, LIBRARY_PATH and ADDON_PATH mainly) to make the mounted packages “visible” in a shell session. So you could keep the environment for a specific project this way.
You can find the package backups in /system/packages/administrative/state-* (the directory name is a timestamp of when the state was created). A new state directory is created for each operation (install, uninstall, …). The directory contains a list of activated packages in a text file, and the packages necessary to rebuild this state from the previous (or next? I don’t remember which way this works) one. If you only ever add new packages, this takes almost no space (just the text file with the package list). If you remove packages, they are kept there so no space is freed. If you update packages, this is the worst case since both the old (in the state directory) and new (in the active /system/packages) versions are kept.
You can delete the states when you don’t need them anymore. Currently that’s not automated, but some 3rd party app can help with cleaning them up. We may add this to the OS someday if we can decide what the right rules are for automatic deletion. I think you can only safely delete the oldest states, if you leave gaps between the current state and an older one, you may end up missing some data to reactivate that older package.
2 Likes
It will be recorded and hopefully available shortly after. When I have more info I’ll share. I’ll also post the slides.
10 Likes
Edit: There was an interesting talk just before I spoke about Haiku on a project called Distri (video), which is a package manager which mounts files into a read-only space to save installation time and reduce package complexity. Sound familiar?
14 Likes
Well done! 
1 Like
I thought you did an excellent job presenting the packaging system! The Q&A voices at the end sounded robotic! 
Thank you for doing this!
1 Like