Compiling HaikuPorts packages for different architecture

I am currently running into troubles with RISC-V port because something is crashing in 3rd-party libraries (libstdc++.so.6.0.25) that are provided as binary packages. This libraries are compiled with -fomit-frame-pointer that breaks stack trace and makes debugging much harder. Also it may be linked to old brolen glue code.

Is there an instruction how to compile HaikuPorts packages for different architecture?

Cast @pulkomandy, @kallisti5.

4 Likes

libstdc++.so.6.0.25 is provided by gcc, probably when it was bootstrapped? libstdc++.so.6.0.25
see https://github.com/haikuports/haikuports.cross/blob/master/sys-devel/gcc_bootstrap/gcc_bootstrap-8.3.0_2019_05_24.recipe#L47

1 Like

The information for bootstrapping is at https://git.haiku-os.org/haiku/tree/docs/develop/packages/Bootstrapping.rst

Howver the process documented here expects that you already have a working Haiku install for the target architecture. If that’s not the case, the normal process would be to always build in bootstrap mode (everything will be built from source). It takes a lot of time to do this because several steps are re-run everytime.

So I made a script to avoid this problem: https://git.haiku-os.org/haiku/tree/3rdparty/pulkomandy/unbootstrap.sh

The idea of this script is:

  • First you build Haiku in bootstrap mode. As part of the process it will cross-compile various packages, using the haikuports-cross recipes and the compiler flags from your gcc port (buildtools repo) and Jamfile configuration. This has to be done from Linux
  • Then you copy the packages to a Haiku install, and you use the script that will extract and repackage them, removing the “_bootstrap” from the package metadata, making it look like these are the “real”, final packages
  • Finally, these packages have to be put in the “downloads” directory of your generated/ working directory; and ideally also put on Haiku servers so that other people can use them. https://git.haiku-os.org/haiku/tree/build/jam/repositories/HaikuPorts/riscv64 needs to be updated if any package changes versions when you do this (usually not, there are very few updates to the haikuports-cross git repository).

Putting the packages on Haiku server requires root access there currently, so you have to ask kallisti5, waddlesplash or me to do it.

3 Likes

No chances to build from Haiku?

Also is it possible to just build individual packages for target architecture?

For bootstrap packages, not really. It’s all or nothing.

3 Likes

I never managed to get that working. In theory there is nothing that would prevent it, but the build fails in various ways and no one has debugged it.

It should be possible, but not easily accessible currently. Probably it needs adding more specific jam targets to run the builds, or run haikuporter manually.

Bootstrap seems running fine on OpenSUSE.

1 Like

Is it possible to run haikuporter for haikuports.cross directly without using jam -q @bootstrap-raw or specify packages to build?

1 Like

Now I get:

BootstrapRepositoryFetchPackage1 /home/user/Packages/haiku/generated.riscv64-bootstrap/objects/haiku/riscv64/packaging/repositories/HaikuPortsCross-build/packages/icu_bootstrap-57.2-2-riscv64.hpkg 
Checking if any dependency-infos need to be updated ...
Looking for stale dependency-infos ...
======================================================================
dev-libs::icu_bootstrap-57.1
======================================================================
The following build dependencies were found:
----------------------------------------------------------------------
dev-libs::icu_bootstrap-57.1
        /home/user/Packages/haiku/generated.riscv64-bootstrap/../../haikuports.cross/dev-libs/icu_bootstrap/icu_bootstrap-57.1.recipe
----------------------------------------------------------------------
Skipping download of source for icu4c-57_1-src.tgz
Validating checksum of icu4c-57_1-src.tgz
Error: Expected SHA-256: ff8c67cb65949b1e7808f2359f2b80f722697048e90e7cfc382ec1fe229e9581
Error: Found SHA-256:    6e5b3d0218a47698be3314c03863f061ebdad5ca0dc071441af83439a4fc7d5a
Supposedly built package /home/user/Packages/haiku/generated.riscv64-bootstrap/objects/haiku/riscv64/packaging/repositories/HaikuPortsCross-build/packages/icu_bootstrap-57.2-2-riscv64.hpkg does not exist; version mismatch?


package="/home/user/Packages/haiku/generated.riscv64-bootstrap/objects/haiku/riscv64/packaging/repositories/HaikuPortsCross-build/packages/icu_bootstrap-57.2-2-riscv64.hpkg"
if [ -e "$package" ]; then
exit 0
fi


haikuCrossDevelPackage="objects/haiku/riscv64/packaging/packages/haiku_cross_devel_sysroot_stage1_riscv64.hpkg"
if [ "x$haikuCrossDevelPackage" = "x" ]; then
echo "$package does not have a cross-devel package defined!"
exit 1
fi

if [ "x$haikuCrossDevelPackage" = "x${haikuCrossDevelPackage#/}" ]; then
haikuCrossDevelPackage="`pwd`/$haikuCrossDevelPackage"
fi


secondaryCrossDevelPackages=
if [ -n  ]; then
for secondaryCrossDevelPackage in  ; do
if [ "x$secondaryCrossDevelPackage" = "x${secondaryCrossDevelPackage#/}" ]; then
secondaryCrossDevelPackage="`pwd`/$secondaryCrossDevelPackage"
fi
if [ -n "$secondaryCrossDevelPackages" ]; then
secondaryCrossDevelPackages="$secondaryCrossDevelPackages,$secondaryCrossDevelPackage"
else
secondaryCrossDevelPackages="--secondary-cross-devel-package=$secondaryCrossDevelPackage"
fi
done
fi


portSpec=`basename "$package" | sed 's@-.*@@'`

cd /home/user/Packages/haiku/generated.riscv64-bootstrap/objects/haiku/riscv64/packaging/repositories/HaikuPortsCross-build

export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/home/user/Packages/haiku/generated.riscv64-bootstrap/objects/linux/lib"
if [ -n "$secondaryCrossDevelPackages" ]; then
/home/user/Packages/haiku/generated.riscv64-bootstrap/../../haikuporter/haikuporter  \
--all-dependencies  \
--cross-devel-package "$haikuCrossDevelPackage" \
"$secondaryCrossDevelPackages" $portSpec
else
/home/user/Packages/haiku/generated.riscv64-bootstrap/../../haikuporter/haikuporter  \
--all-dependencies  \
--cross-devel-package "$haikuCrossDevelPackage" $portSpec
fi

if [ ! -e "$package" ]; then
echo "Supposedly built package $package does not exist; version mismatch?"
exit 1
fi

...failed BootstrapRepositoryFetchPackage1 /home/user/Packages/haiku/generated.riscv64-bootstrap/objects/haiku/riscv64/packaging/repositories/HaikuPortsCross-build/packages/icu_bootstrap-57.2-2-riscv64.hpkg ...

BUILD FAILURE:
...failed updating 1 target(s)...
...skipped 16 target(s)...
...updated 75 target(s)...

Created ticket #16975.

1 Like

Seems like a failed download or upstream changed the package as you get:
Skipping download of source for icu4c-57_1-src.tgz
Validating checksum of icu4c-57_1-src.tgz
Error: Expected SHA-256: ff8c67cb65949b1e7808f2359f2b80f722697048e90e7cfc382ec1fe229e9581
Error: Found SHA-256: 6e5b3d0218a47698be3314c03863f061ebdad5ca0dc071441af83439a4fc7d5a

1 Like

icu4c-57_1-src.tgz is not available anymore from specified URL. I downloaded it from different place, changed Haiku build system ICU version to 57.1.2 and that stage passed.

Now it creates source packages of a lot of HaikuPorts packages. Why it is needed? Will it pack all HaikuPorts packages? What is expected time to complete?

1 Like

It creates packages for all haikuports.cross packages, it can take some time and it depends on your hardware (speed of CPU, disk, and downloads). It can be from a few minutes to a few hours the first time you run it. The next runs are a little faster because it can reuse existing packages, but it still is not very fast.

1 Like

Why Qt is needed?

The following required ports will be built first:
        dev-qt::qt5-5.15.2
----------------------------------------------------------------------
dev-qt::qt5-5.15.2
        /home/user/Packages/haiku/generated.riscv64-bootstrap/../../haikuports/dev-qt/qt5/qt5-5.15.2.recipe
----------------------------------------------------------------------
Skipping download of source for 5.15.2.tar.gz
Skipping checksum validation of 5.15.2.tar.gz

Downloading: https://invent.kde.org/qt/qt/qt3d/-/archive/72b026a22e6942eca4b70ba948022aecbd8965fc/qt3d-72b026a22e6942eca4b70ba948022aecbd8965fc.tar.gz ...
--2021-05-28 16:00:56--  https://invent.kde.org/qt/qt/qt3d/-/archive/72b026a22e6942eca4b70ba948022aecbd8965fc/qt3d-72b026a22e6942eca4b70ba948022aecbd8965fc.tar.gz
Resolving invent.kde.org (invent.kde.org)... 195.201.182.103
Connecting to invent.kde.org (invent.kde.org)|195.201.182.103|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/octet-stream]
Saving to: ‘/home/user/Packages/haiku/generated.riscv64-bootstrap/../../haikuports/dev-qt/qt5/download/qt3d-72b026a22e6942eca4b70ba948022aecbd8965fc.tar.gz’

haikuports.cross processing seems to be already finished, now it seems processing HaikuPorts.

2 Likes

Depending on what you’re trying to do, maybe you can stop there.

The “normal” bootstrap process is:

  • A lot of source packages will be generated and included in the image
  • You can then boot the bootstrap image
  • From inside the image you can run haikuporter, and compile all these packages from the prepared sources

I am not sure if this complete process has been used recently. Maybe @korli did it when working on the 32bit support for 64bit systems? When this process was designed, haikuports was a lot smaller than it is now.

However, for other architectures we had no way to actually run that image. So we did something else:

  • Build the bootstrap packages only
  • Use the “unbootstrap” script to convert these packages to non-boostrap ones
  • Put these packages in the haiku repository (for this you need the help of @kallisti5, @waddlesplash or myself because it requires root access to one of our servers…)

This appears to be sufficient to get Haiku built normally afterwards.

You can find the script here: unbootstrap.sh « pulkomandy « 3rdparty - haiku - Haiku's main repository

Note that this script uses the “package” command, so it is easier to run it from an Haiku machine (after getting the hpkg files copied there, since the other parts of the bootstrap process work only on Linux).

If this approach works for you, we should probably make it the official one, and see if we can automate it. Unfortunately for other architectures we never got this running far enough to check if it actually results in a working system.

1 Like

After Qt it seems finished correctly:

mimesetting files for package qt5_source_rigged-5.15.2-5-riscv64.hpkg ...
creating package qt5_source_rigged-5.15.2-5-riscv64.hpkg ...
----- Package Info ----------------
header size:                     80
heap size:                336290260
TOC size:                   9359706
package attributes size:       5313
total size:               336290340
-----------------------------------
grabbing qt5_source-5.15.2-5-source.hpkg and moving it to /home/user/Packages/haiku/generated.riscv64-bootstrap/objects/haiku/riscv64/packaging/repositories/HaikuPorts-sources-build/packages/qt5_source-5.15.2-5-source.hpkg
grabbing qt5_source_rigged-5.15.2-5-riscv64.hpkg and moving it to /home/user/Packages/haiku/generated.riscv64-bootstrap/objects/haiku/riscv64/packaging/repositories/HaikuPorts-sources-build/packages/qt5_source_rigged-5.15.2-5-riscv64.hpkg
----------------------------------------------------------------------
meta-ports::meta_portsfile-1
        /home/user/Packages/haiku/generated.riscv64-bootstrap/../../haikuports/meta-ports/meta_portsfile/meta_portsfile-1.recipe
----------------------------------------------------------------------
non-chroot has these packages active:
Setting up sysroot for non-chroot build: /home/user/Packages/haiku/generated.riscv64-bootstrap/objects/haiku/riscv64/packaging/repositories/HaikuPorts-sources-build/meta-ports/meta_portsfile/work-1/boot/cross-sysroot/riscv64
Activating haiku_cross_devel_sysroot package: /home/user/Packages/haiku/generated.riscv64-bootstrap/objects/haiku/riscv64/packaging/packages/haiku_cross_devel_sysroot_stage1_riscv64.hpkg
AddDirectoryToContainerCopyFilesScript <HaikuImage>home/haikuports/input-source-packages/-/<HaikuPorts-repository-source-packages>packages 
MkDir1 objects/haiku/riscv64/packaging/repositories/HaikuPorts-bootstrap 
BuildHaikuPortsRepositoryConfig1 objects/haiku/riscv64/packaging/repositories/HaikuPorts-bootstrap/haikuports.conf 
AppendToContainerCopyFilesScript <HaikuImage>haiku.image-copy-files-dummy-home/haikuports 
AppendToContainerCopyFilesScript <HaikuImage>haiku.image-copy-files-dummy-home/config/settings/boot/launch 
InitScript1 haiku.image-extract-files 
warning: using independent target <HaikuImage>/home/user/Packages/haiku/generated.riscv64-bootstrap/./haiku.image-init-vars
warning: using independent target <HaikuImage>/home/user/Packages/haiku/generated.riscv64-bootstrap/./haiku.image-make-dirs
warning: using independent target <HaikuImage>/home/user/Packages/haiku/generated.riscv64-bootstrap/./haiku.image-copy-files
warning: using independent target <HaikuImage>/home/user/Packages/haiku/generated.riscv64-bootstrap/./haiku.image-extract-files
BuildHaikuImage1 haiku-bootstrap.image 

Creating image ...
Creating filesystem indices...
Populating image ...
Unmounting ...
...updated 549 target(s)...
user@localhost:~/Packages/haiku/generated.riscv64-bootstrap> 
2 Likes

I changed buildPlatform from BuildPlstformHaiku to BuildPlatformUnix in haikuporter and it started working on Haiku. Lets see how far it progress.

6 Likes

I think that the cmake build requires Qt for the GUI. One can removed this locally before bootstrapping.

Yep, this can be the reason. I have extended the port with the GUI pieces, but i see no point in it at all.

I managed to build gcc_bootstrap under Haiku, but some recipe changes were needed. --with-sysroot=/ cause attempting to use Haiku system libraries (mpfr) and fail because of incompatible architecture.

diff --git a/sys-devel/gcc_bootstrap/gcc_bootstrap-8.3.0_2019_05_24.recipe b/sys-devel/gcc_bootstrap/gcc_bootstrap-8.3.0_2019_05_24.recipe
index a44ec01..8d3330a 100644
--- a/sys-devel/gcc_bootstrap/gcc_bootstrap-8.3.0_2019_05_24.recipe
+++ b/sys-devel/gcc_bootstrap/gcc_bootstrap-8.3.0_2019_05_24.recipe
@@ -154,10 +154,10 @@ BUILD()
 		--libexecdir=$installDir/lib --mandir=$manDir --docdir=$docDir \
 		--disable-nls --enable-shared --disable-plugin \
 		--enable-version-specific-runtime-libs \
-		--enable-languages=c,c++ --enable-lto --enable-frame-pointer \
+		--enable-languages=c,c++ --disable-lto --enable-frame-pointer \
 		--enable-threads=posix --enable-tls \
 		--with-pkgversion=$(echo $portVersion | cut -c 7-) \
-		--with-sysroot=/ \
+		--with-sysroot=/boot/data/packages/haiku/generated.riscv64/cross-tools-riscv64 \
 		--with-build-sysroot=$installDestDir/ \
 		--with-threads=posix \
 		--with-default-libstdcxx-abi=gcc4-compatible \
@@ -303,7 +303,7 @@ INSTALL()
 
 	cd $installDestDir$installDir
 	$effectiveTargetMachineTriple-strip --strip-debug bin/*
-	for f in cc1 cc1plus collect2 lto1 lto-wrapper; do
+	for f in cc1 cc1plus collect2 ; do
 		$effectiveTargetMachineTriple-strip --strip-debug \
 			lib/gcc/$effectiveTargetMachineTriple/*/$f
 	done

6 Likes

Never looked into other architectures for haikuports other then x86_64 and x86_gcc2 (and x86 for secondary), how do the packages get build if one would enable another architecture?
The buildbot atm is running the ones for 32bit and 64bit, but I don’t know any other that are able to do this. Is this why you need the gcc_bootstrap for then?
EDIT nvm, should have started reading from the start :wink: