Python and setuptools giving me trouble

Software Ports
1

I really have a bad time with python and its myriad of modules and ways of installing dependencies, not only but in this case on Haiku.

When installing the latest python package, I get python3.13, but it is not aliased to python or python3 and has to be called by the full version, which results in scripts not finding python at all.

Then, when trying to run a setup.py with python manually, it barks on missing setuptools, even when they are installed (got the latest stable package without version suffix from HaikuPorts and installed with pkgman).

I really hate the mess that is Python, besides it being slow and ugly, but I need some tools and libs that are hard to port and sadly only exist in this over hyped scripting glue play tool.

Any idea what’s going wrong here? Can’t imagine I’m the only one with these issues, as I have no special use case and it should just work out of the box (at least for the HPKG’ed python and setuptools packages).

2

I sure don’t know! Used to use it, but in simpler times. But have you thought of installing your own python command, to see if things start working? Like, put this in ~/config/non-packaged/bin/python (and chmod 755 that file):

#!/bin/sh
exec /bin/python3.13 "$@"
1 Like
3

As in every other OS, while you can have different Python interpreter versions installed at the same time, only one of those can be the “default python” (the one you get when calling python3.

In Haiku, that default is Python 3.10 (the only package that provides cmd:python3 / /bin/python3.

The other ones get built and packaged as --alt-install (meaning… they only provide version suffixed commands, like /bin/python3.13, bin/pydoc3.13, etc., and not python3 / pydoc3, to avoid clobbering files from other installed versions.

We don’t provide packaged (.hpkg) files for setuptools anymore, except for the current “default Python version” (3.10). In fact, the aim (to avoid getting overwhelmed by the work needed to “repackage all python things” for multiple versions) currently is to only provide .hpkg files for 3.10, and only where really needed (due to dependencies on other tools/apps we need or want to provide).

Other than that… specially for user of Python interpreter versions other than the “default” one… users are expected to rely on pip (would be pip3.13 in case of @grexe), to manage dependencies they may have.

In fact, we now don’t even package pip for other versions (and I expect to drop the pip .hpkg altogether soon).

Instructions on how to install pip for whatever Python version a user needs (matching the interpreter they intend to use, of course), are given in each Python interpreter package description.

In general, those are:

Note: to install "pip" for this Python version, use the following commands:

> python3.13 -m ensurepip --altinstall
> python3.13 -m pip install --upgrade pip

And then you should be able to use "pip3.13" normally.

I had open a PR for possibly allowing end-users to select their “default Python”, but that needs way more testing (and review by more experienced devs).

My aim (as the guy currently doing most of the python-related updates) is to provide .hpkg files for all the active Python interpreters (so end-users don’t have to do that themselves), but we cannot provide re-packaged versions for every Python module in existence, times X python versions.

So, suggestion is… either stick with “default Python in Haiku” (3.10 ATM), and see if what we provide as .hpkg is enough, or do as Pythonistas do, and rely on the package manager that the language provides: pip.

A lot of things can simply be installed via “pip” (pip3.13 in this case), like setuptools, for example. Others, cannot without patches or tweaks during build time to accommodate to Haiku-related differences to other platforms. When possible we try to upstream things (help welcomed here! I sometimes don’t have the patience or skills to deal with upstreams), sometimes upstreams just do not care about Haiku support.

I understand your frustration @grexe as a user, things ain’t better as a (re)packager.

In any case… always open to any help on improving the current state of things.

4 Likes
4

That sounds like an issue to be solved kn package kit to me, and not one to solve by adding severall “variants” of packages that are manually set to conflict with each other.
Java has the same problem, as would any other versioned language you expect to be co-installable but still have a default accessible version

1 Like
5

What would be the plan at package kit level to do this without any manual work? What would the provides for each package look like? Any additional tools needed?

6

@BiPolar ; hey thanks for the explanation. I totally agree that it’s best not to get into the game of trying to package Python libraries. Debian / apt tries to do that and it creates quite a messy situation to deal with.

I also agree best to package the interpreters alone and for the user to use Python virtual environments (venvs) to ring-fence the Python packages installed with pip3 . This way, the user can setup a number of venvs with different interpreters + sets of dependencies.

Java has the same problem…

In it’s infancy Java had a similar problem in the era when Ant was popular, but Maven effectively solved this problem properly 20+ years ago. It’s strange to me that the approach from Maven is not just copied by other language ecosystems.

2 Likes
7

Useful for users who are writing Python code themselves, but that doesn’t solve the problem of packaging applications that use python at buildtime or at runtime. That’s why we end up repackaging quite a lot of things in Haikuports, in order to provide Python apps the same support as we have for apps written in other languages.

Sometimes, the ability to have patched versions of Python libraries in Haikuports can be convenient as well (when things can’t be upstreamed or aren’t in an upstream release yet).

Similar problems exist with all languages who decided to handle dependencies in their own way, because there isn’t a standard for that (say, in POSIX) and they don’t want to be dependant on OS and distributions to do it. Meanwhile, distributions and OS could do things in a wider way (accross multiple languages, for example projects mixing Python and Rust are more common now). But they don’t have any standardization, and so you’d need to do the work 100 times for 100 different packaging systems.

8

I am wondering, for software distribution, would it not be better to package the dependencies in a self-contained manner in the same payload as the runnable files? I know this consumes more space but would avoid pitfalls for somebody wanting to work on Python code themselves or execute code originating from outside the operating system’s packaging system.

Would it be possible in these cases to distribute in Python wheel format from Haiku’s own PyPi repository instead of HPKG? I know it’s obviously more work but it would presumably avoid polluting the Python-global packages’ namespace with packages at pinned versions.

9

This adds more work for the people doing the packaging and/or the package builders.

It’s nice to be able to fix a bug in just one package, update it, and have all things that depend on it benefit from the bugfix immediately. That’s why we use shared objects and not static linking, and I don’t see why Python should get a different treatment here?

2 Likes
10

exactly, it’s so easy in Java, you have a jvm/jdk-select tool for handling the runtime version problem, and packages are more or less independent of the runtime version (they can still specify min/max Java versions though, and the JVM will complain if the versions are incompatible).

Then you have all the libraries / packages only once in a central local user repository, and the selected JVM will pick them up, just as expected.

Same as for dynamic libs, really.
Only Python didn’t get the memo it seems…:frowning:

11

Thanks for the thorough explanation and sorry for venting my frustration, it’s just so annoying having to deal with Python’s inefficiencies and mismanagement of dependencies on every platform again.

I knew about the manual way to install pip, but not about --alt-install.

Also I see now that f I just alias python and pip with my specific version, it will not work as this will still target the “wrong” local Python package repository.
First it seems to run fine, but then I cannot run setuptools for example.

This works now and I could finally install pdfdataextractor to try out something nice for SEN:)

12

Seems that particular project is not on https://pypi.org (thus you can’t simply pip install pdfdataextractor, but one would need to python3.xx setup.py install from sources (thus the need for setuptools). Also, PDFDataExtractor seems to require a very old version of pdfminer.six (the one from 2020-01-21 specifically).

In turn, that old version of pdfminer.six requires: chardet, pycryptodome, and sortedcontainers. Luckily, all those should be “pip installable” (we even still package pycryptodome as .hpkg, because at some point it didn’t compiled out-of-the box on Haiku).

@grexe, the PDFDateExtractor project doesn’t seems to require 3.13 in any way, so unless you have other reasons for using that particular version, stiking to 3.10 might make your life easier (you’d be targetting the same Python interpreter Haiku considers “default” ATM, and we, at least for now, still package setuptools and pycryptodome).

If needs arise, I don’t think it would be terribly hard to write a .recipe for it (possibly with its depedencies “vendored-in”… given the project seems inactive for the last 3 years… might not be a bad idea if you plan to rely much on it).

Edit: regarding that python3 setup.py install for pdfdataextractor… take into account that you might need to do something like this instead:

export PYTHONPATH=/system/lib/python3.10/vendor-packages/:$PYTHONPATH
python3 setup.py install --root=/ --prefix=/system

(to get it to install on the correct location on Haiku)

1 Like