Python packages in HaikuPorts

Software Ports
1

Seemingly obvious, but probably dumb question of the day: If we know an application can be installed with pip(3) is it feasible to create an hpackage that consists mainly of a post-install script doing that for you? That way, we retain the hpkg system as the one and only way of installing and updating apps on Haiku. IMHO Regular users don’t want to manage one set of apps with hpkgs, another with pip, yet another with gem …

Uninstallation would not work, of course. The hpackage would be gone but the files would be left behind. But that is the same situation as when you uninstall a ported app and it leaves behind a collection of packaged dependencies.

1 Like
2

If you want pip you should use pip. There is nothing wrong with severall package managers doing their own thing in their own space. Trying to force the peg into a different sized hole won’t work, as you said with the uninstallation mysteriously not working… :slight_smile:

3

There has been a general consensus on haikuports that packages that can be installed with pip should be done by the user, a cleanup for some of those python packages have been done some time ago already, having said that, things that won’t work out of the box with pip are still welcome for a PR.

4

Not really. You pretend you do, but installing such package may install uncontrolled dependencies using pip, maybe breaking your other Python packages in the process, and in a way that you can’t revert using package state backups.

I think that’s unfortunate. There surely is no need for packaging every Python library, but we could probably include user-facing applications like Renpy in the repository? (with their dependencies, then)

2 Likes
5

I’m not going to explain the issues that led to this (I think most of the team members followed discussions on IRC/matrix for that and saw the removal of ealier mentioned packages).

In the past I’ve added numerous python packages and I know what a pain it can be to keep dependencies for those, even if it is just for the default python version which is python 3.10* atm.

I’ll probably won’t fight against it, but personally I’m not looking forward on yet another painfull hunt on python dependencies on some packages.

6

I agree that user-facing applications like RenPy should have their own OS-specific packages whenever possible. Developers can use pip if they want, but I don’t see any benefit for the typical user in having to learn a programming language’s package manager. Same goes for Cargo and all the rest.

2 Likes
7

At the moment there are about 220 python packages in haikuports using python3.10, a bit over 50 of them use python3.12, so I guess some missing packages for 3.12 will have to be added, I don’t know how many dependencies RenPy has, but those should be added if not available also.

I know from experience that tracking down python dependencies can be way harder then tracking down regular library dependencies.

So again, I won’t be holding it back, but be aware that this probably won’t be a one recipe adding to the repository. :slight_smile: (if it can, this should be looked into if it also works with python3.10 which is the facto default atm in Haiku)

8

Hello everyone!

As the de-facto “Python guy” (not because of skills, but just because in recent times I’ve been doing most of the janitorial work around the Python packages recipes), a few words:

While I agree that it is nice to have (some) written-in-python user facing apps readly available as .hpkg… there are a few difficulties with re-packaging Python modules/apps:

  • Generally large and complex dependency chains: A given app might have only one or two direct dependencies, but those depend on x other modules, that depend on particular version of an interpreter, or have conflicting version constraints with some other module.

  • It is a large ecosystem, it moves relatively fast, with new ways of doing things popping up all the time, and where several versions of the Python interpreter are in active use at the same time (which means some projects may target a newer Python than we do, or not test on all “active” versions).

  • We cannot have side-by-side installation of multiple versions of the same package. That’s why virtual-envs are a thing.

  • While our Python recipes can target multiple versions of the interpreter… sometimes requirements for a given package vary too much, and we might be better off using different recipes for different Python versions.

  • Re-packaging python projects that do not require patching, or complex or time-consuming building… doesn’t really sounds like a good time investment. We do our best for “popular/well-known/widely-used” deps, of course (and also for user-facing apps, if we have the needed deps already), but we really need to consider future maintenance costs before adding new recipes.

  • While we can package multiple versions of the Python interpreters… we just don’t have enough man-power to maintain and provide .hpkg packages for a whole ecosystem of Python modules, for more than just one Python version at a time (3.10 ATM).

I plan/want to write a post to discuss/RFC about what our long term “strategy” for dealing with the ever-evolving Python interpreters/ecosystem should be. As in… announce EOL of 3.9, choose a next “default”, document intended “support schedules”, etc.

I want to avoid having _python packages for 4 or 5 different python versions, as we had at our worst :-D.

Minor correction: We currently only have two packages that target Python 3.12 (pip_python312 and setuptools_python312), and I’m not sure they’ll stay up in the long term (unless 3.12 gets adopted as the next “default Python on Haiku”).

We do still have many remaining _python39 packages, that we’ll continue dropping, specially with Python 3.9 EOL coming in next October.

9

I’m not sure how it ended up that way, I imagine we need:

  • the one that shipped in the latest beta of haiku
  • one newer version to use in the next beta

that should be enough, right?

as for the rules, yes, surely we shouldn’t be packaging all pyhon things. But things like RenPy look like they would deserve hpkg packaging, with a native icon, a link in deskbar out of the box, etc. I agree it’s not fun to do and a lot of work to maintain. But maybe we don’t have to always chase the latest version of everything, and then it may be easier?

and the same problems or similar ones apply to other languages as well (I think of Rust for example, or nodejs). So we would end up depriving ourselves of a lot of software.

10

This sounds like a serious defect. There are plenty of frameworks out there that have multiple versions. I hope this can be addressed somehow.

11

Yes, one “default Python” + one “next default Python candidate” sounds about right. Issue is…which one to have as “next” (thus my intention to discuss which strategy we should follow, ideally taking into consideration the Python release schedules and EOLs).

That’s why I haven’t been adding _python packages for other versions besides 3.10. And before starting to add packages for a different version (as it seems this app in particular would need), we should decide which Python will be the next default.

I really understand what you’re after, with wanting .hpkg for user-facing apps. My post is not saying “we should not do this”, but rather… “this might need to wait till dependencies are in place” (which for me implies deciding which Python will be “next default”).

but also… consider this:

From Ren’Py’s github:

We strongly suggest installing the Ren’Py modules into a Python virtualenv.

This is not uncommon in Python land.

(edit: not 100% sure about how well virtual-envs work, or not, on Haiku. Did my best at trying to fix a thing or two related to those in newer Python versions, but I’m far from an expert, and I don’t personally use them, so: “here be dragons” :frowning:)

Similar, but far from equal.

We don’t ship multiple versions of Rust, and recipes for rust projects use cargo (praise “cargo.lock”), which seems to take care of all/most of the complexity (we don’t have to create one .recipe per crate, as far as I know). We can’t use pip in our recipes to the same effect (I’ve seen some talk about pip.lock, but that’s hardly current practice).

I do not intend to deprive ourself of a lot of software, just don’t want to add lots of things than then complicate maintenance, or addition of new software in the future (we already had more than one problem trying to update numpy/scipy/cython/etc, trying to accommodate old and newer versions, conflicting versions, having to chase patches from 3rd party sources because upstream is dead, etc…).

We all do our best, and people are encouraged to help up out.

12

This is not something Haiku specific. There’s a reason virtual-environments are a thing in Python land. (I liked local packages directory better, but that was rejected :smiley:)

To hopefully avoid confusion: this is not about having multiple Python interpreters installed at the same time, we can do that just fine in Haiku. What you can’t have is multiple versions of a given python module installed side-by-side for any given Python interpreter.

13

With the incredible complexity you outlined I would just not bother at all. If python deps need fixing those can be submitted upstream, but: don’t ship any python deps in haikuports, people can use pip for that. I would instead only ship full application images. In essence a really fat image where the deps were installed by pip during the package build phase, and then ship that.

Yes this would take more disk space, but it would be lazy and solve the problem for most “I just wanna run this app” usecases… without taking too much toll out of haikuports

2 Likes
14

We can provide a catalog for what python libraries/applications can be installed through Haiku by pip. If those items require deps that MUST be installed from Haikudepot, mention it so that users can install it by themselves. For developers that ports python applications to Haiku, they may find out whether the deps already exists in either pip or Haikuports in order to determine whether to start the porting.

15

A list of things that can be installed via pip can’t ever be complete or up to date enough, to be of any use IMO (new python packages are created every day, new versions get released, etc., who will make sure said list is still valid? or who will volunteer to create such list in the first place, even?).

In true Python fashion… better to ask forgiveness than permission… or in this case… better try pip first and patch (or package) latter :smiley:.

For developers wanting to see what packages we provide for a given Python version… is easy, just call pkgman search _python310, for example, or one can inspect the list of recipes under the dev-python dir on the HaikuPorts repo.

1 Like
16

I don’t really see what we gain. You still have to write recipes that build a lot of things. You lose the possibility to maintain patches for dependencies. And it is annoying to users.

I would suggest:

  • keep only at most two python versions: the one that shipped in the last beta, and the latest one from python, for experimenting and preparing the next release
  • don’t try to have every dependency built with both: run as many things as possible with the python from the latest release. Maybe even mostly stop doing these multi-version packages by default?
  • only a few things which cannot work with the stable python version may use the new one. The dependencies for these would need the dual versioning
  • package only user-facing apps and their dependencies (BiPolar has already done quite a bit of cleanup)

Basically, it is already what we do for other things like removing old ffmpeg versions.

And in any case, the decision will be in the hands of whoever maintains the packages. I am happy to try writing a recipe for RenPy from the instructions shared here. But Begasus’ rule of “if it can be installed with pip, there should be no recipe” prevents me from doing that. I think we should be a bit more relaxed than that, without immediately going to “let’s package everything for every python version”. There should be a reasonable way somewhere in between, and I thought what we were doing was “package user facing apps and their dependencies” and that works good enough.

17

Didn’t mean that as a “hard” rule, it’s something that has been discussed before the cleanup for those mentioned python packages was done, I won’t object a recipe for this as mentioned before. :wink:

EDIT: @BiPolar just mentioned on IRC that I was looking into the wrong renpy, the one I checked came from pypi and not from github. So I stand corrected on this one. :slight_smile:

18

my two cents as someone not in love with python but has to ~deal~ use it constantly: There are too many python packages across too many python versions to maintain distro packages for all of them.

On Linux, every time I go to install a dependency and heed the warnings about using your local package manager over pip, i’m always missing several python packages in the os’s package repo and have to use pip anyway.

Python packages in distro packages is a mess and always has been… it’s a big resource consumption to maintain them.

7 Likes
19

No one is suggesting to package “all of them”.

But some apps happen to be written in Python and still deserve to be provided by the OS.

If you do any Python development, yes, use pip, that works great. But that should not exclude apps from being packaged, just because they happen to use some Python code.

1 Like
20

I definitely agree that some Python apps should be provided by the OS. For anything else, pip is the way to go, we like it or not (I don’t). But be aware, as I’m sure you are, that whatever you do it will still be a mess. The question is which mess is more manageable.
Some GNU/Linux distributions (e.g. Void Linux) tried to include a rather conservative selection of Python packages in the repositories, leaving the rest to pip, and maintained by the user. The result is not very satisfactory, but it is certainly preferable than having a myriad of Python versions installed, complete with a dependency hell for the packagers every… week or so. And that’s not just Python-specific. A similar (but way easier to deal with) situation is happening with Lua packages. I’m sure there are other cases too, I just was lucky enough not to encounter them.

As an example, I have a library I wrote a few years ago and use/improve since then. I recently decided to release it. The library itself has nothing to do with Python (which I never liked btw), but the documentation is created automatically by processing comments in the code, specifically formatted for that purpose. The Makefile itself provides make doc for that purpose, which calls the documeter as appropriate. Unfortunately, the documenter itself is written in Python, and of course it has Python dependencies. It is slow (especially if you want to include graphs), but the result is nice. Rewriting a similar documenter from scratch would take ages, so I had to deal with Python.
I didn’t even think of anything else: If the user wants to build the documentation html files from source code, they should use pip, as I did to create the documentation in the first place. Everybody else will just use the ready-made documentation included in the package.
In my case, Haiku’s pip worked pretty much as everywhere else (that’s its job after all), but the resulting html documentation is slightly different. Oh, and installing the documenter via pip also installed packages of course, some of them already provided by Haiku - just different version. That includes… pip itself, for Devil’s sake. Again, nothing new here, I got the same on GNU/Linux and FreeBSD. Do I like it? Hell no. It’s far from being ideal, but I can’t think of anything better.