HaikuDepot dependencies

Development OS
7

Both should be in sync.
I don’t know how so many ended up non-activated. As nielx said, it shouldn’t happen from an interrupted download.
When you install a package and so also when you run SoftwareUpdater, HaikuDepot makes a transaction-* folder in administrative subdirectory. Only once every package successfully downloaded, HaikuDepot proceeds to the real installation. So normally there shouldn’t be half-downloaded packages in the packages directory. But in administrative subdirectory, there’s probably a few transaction-* folders with one or several packages, last of these packages being not complete and so broken.

To solve your issue, I would find all packages that are not activated comparing lists, move these elsewhere in a new folder. Then, I would clean old transactions folders as they may take a lot of place. At this point, things are back in a clean state and there shouldn’t be errors anymore.
After that, I would examine the packages I removed to see what I need and what I don’t. I would reinstall those I really want through HaikuDepot (Since these packages were in packages directory, they are probably complete and not broken so you could also click on them or drop them back but our goal is to get out of trouble…) then I would delete the folder.

8

Ok, then is there a log I can follow or a way to have verbose output during package management? I guess I’m looking for a transactional trail I can examine to find out what is happening.

9

When a package is installed and activated, it is mentioned in syslog file (/var/log/syslog), you may find it sufficient to check that.
Otherwise, you can use directly pkgman in a terminal. The options are install, remove, update and full-sync. update and full-sync can both be used to update your system, the difference is that full-sync will allow downgrading a package. (This is sometimes needed when projects are changing the way they’re naming versions or if a new version of a package is accidentally pushed. So, usually 3 times a year max)

Also from a terminal, there’s a way to activate debug mode in HaikuDepot but I never tried.
See here for more details https://www.haiku-os.org/docs/userguide/en/applications/haikudepot.html

10

Thanks for this info. I’ll see what I can discover and report back.

11

Mazbrili’s question to Pulko Mandy:
“does filwip help in this situation?”
I can answer with an unequivocal YES!

12

FilWip is very handy but it is not selective. It will wipe all previous states leaving you with no fall back.

13

You don’t have to keep them all. Usually keeping last three or five previous states is sufficient. So, if you need some space in your Haiku partition, you can delete a few without problems. But, states are steps to go back in time and you can’t make a cherry pick of what you want to keep. You have to keep them contiguous. For example, let’s say today is Wednesday. To go back to Friday, Haiku will go back to Tuesday then from there to Monday, from there will go to Friday. It won’t do Sunday or Saturday because it’s the week-end, you didn’t use Haiku and so no updates and no state.

Unfortunately, FilWip and tools like it tend to mask problems. Better look for the cause of the problem like Scott intend to. Once known, it can help to solve it permanently.
Although FilWip has an option to exclude files, last time I checked it wasn’t really helping.
It won’t help you to clean up non-activated packages that ended up already in packages directory either.

14

I envision a service or an extension of package management that deals with this. As it is now, nothing is preventing a program (like FilWip) or a user from deleting states saved under the package structure however they want. This can, as you have stated, render falling back to a previous state potentially useless. As it is now, these states will build up indefinitely unless managed. Something akin to a kernel manager but for states (keep last ‘n’ amount). Perhaps another tool to sniff out mismatches between packages and those listed in activated-packages. Just thinking out loud, I really enjoy using Haiku and want it to be the best it can be.

15

Though, it will take time before Haiku has multi-user support, that’s the kind of thing a superuser password would probably secure enough.
Devs are already thinking about limiting the number of states for R1 but it isn’t easy. There are a lot of cases to consider. I.e when you install a package that has no previous version the state folder only contains an activated-packages file. Not so useful. Repeated few times, it defeats keeping last 3 strategy.
I think that problems like yours are more frequent since SoftwareUpdater was changed to use full-sync option instead of update. It can be that we didn’t noticed them because the tree was hiding the forest. But, there’s certainly something to do also there.

16

For what it’s worth, I usually use the terminal for updating with pkgman full-sync. I use HaikuDepot for installing new software. I think the FilWip type of app is the most appropriate way to clean the system as long as there is fine grain control of which states to clean. It is an interesting mental exercise to think about how to best manage package management. The rollback feature is great and needs to be leveraged to its maximum potential. Perhaps the user could flag stable prior states or the system could flag crashiness of states. I’m an automotive tech, we use things like misfire counters, fault codes, all indicators of performance issues. I’m thinking that a diagnostic counter of sorts could flag states that have regressions or the opposite could exist, a counter of succesful operation could elevate a state as worth keeping. Well, enough rambling.

17

I think an option with pkgman full-sync to take a snapshot would be useful.
Then you could delete all previous statuses and only have one to go back.
pkgman full-sync -snapshot date or name
or when a new beta appears automatically with the name of the beta.
Then you could install the fullsync function in the software updater and the function to manage the snapshot or put function also in haikudepot.

18

This same like to me sir. And i try to pkgman full -sync , and if u have problem with full sync i try to this Screenshot_20201127_162040

19

I don’t understand what you want, this is exactly what the state directories are doing.

You can always keep only the latest one and go back to it. No problem with that. And pkgman creates them whenever you update, full-sync, add, or remove a package.

The SoftwareUpdater already uses the same code as pkgman full-sync and also does this.

The only remaining problems are:

  • We never clean up the states automatically. Deleting old states is not an hard thing to do, we could just add that (keep the 3 or 4 most recent ones or something)
  • What is not possible is deleting newer states while keeping older ones. This is because we need each intermediate state along the way and the system works in a “retracing your steps” way, it can’t jump to an arbitrary state but must go through all intermediate ones. If we want to do this, we need a way to merge states together. I’m not sure if this is needed, because usually you will only need only the most recent states. And I think it adds too much complexity.
20

Hi Pulkomandy, that’s exactly the point.
I can delete the files mamuell, which i will do too. The directory on the hard disk is now 7.5 GB in size and still contains beta1 data. Since the simple user cannot necessarily know that he can and may delete the data there, a cleanup function would be very helpful.
I would be very grateful if I could install this. I think a setting option via spincontrol from 1-5 would be cool. previous status up to the last 5th, as well as a transaction cleanup function would also be very nice.
The name of the app filwip is not exactly obvious to me. I only came across it by chance, so it would be nice if you could install it.

Thank you in advance
Lorglas

1 Like
21

I have been wanting to fix this for a while but have not had the opportunity yet. The general idea for cleanup of old states was to delete them to some set time frame but ensure a certain number of states remain. So, for example, keep the last 10 states, up to 30 days old. There will have to be some testing to find the best default and ensure this idea works correctly.

The reason to look at both time and number of states is to handle weird outliers like if there were no updates or installs for the last 30 days the cleanup could leave no old states if only the time frame was considered. And if you only looked at number of states, a very busy day updating and installing packages could mean you can only go back one day if the time frame was ignored.

By looking at both time frame and number of states the benefits of rollback can be preserved no matter the kind of install pattern used.

We will try to find a good default but may still expose configuration values for this somewhere.

22

Main issue “here” is that when I build a package(s) locally and then (after buildbot finished the build) try to run “pkgman full-sync” and it’s telling me that it can’t finish (package is in use), it would be nice if pkgman would tell which package is conflicting with the update so one can remove the package (installed) so pkgman could finish the update

23

Should opportunity come knocking, please also see if can implement a way to revert to a state. That is, if I have to boot into an older state, I’d like to be able to revert to that state permanently without having to do that manually from the boot options every time.
I suppose that would be done by removing all states newer than the currently running?

3 Likes
24

Good day,

As of lately I can’t listen to anything while driving, I got plenty of time to think, and mostly is Haiku stuff :star_struck:. Regarding the “states”, IMHO I think Haiku needs an improved method to rollback to a “stable” state, and a way to keep certain “states” saved.

Here I was thinking on how to have a tool to do it, though was a wrong approach. Actual Haiku states are text files stored in the /Administrative folder inside /home/config/packages/Administrative IIRC, and they list the installed pieces and their versions. Easy to remove by hand, or with a GUI tool. The replies I got at that post would apply to the GUI, I presume. I didn’t dig any further because lack of knowledge on Haiku :roll_eyes:

These days, I was thinking about having the Core OS isolated from the apps, and saving snapshots of “working-stable” Core OS installs. Not sure but I presume this would break the Haikuports stuff, I mean, softwared installed from Haikuports that would depend on certain OS libs. Packaging software with its dependencies could solve this, is this correct?

By isolated I mean that Haiku, the OS could install itself, and update itself independently from apps, and viceversa. As HD’s are becoming bigger (capacity) and cheaper, it could be possible to keep full OS (core) snapshots without taking much space on the disk. Besides, Haiku itself does not take lots of space, as other OSes do. With a GUI tool, user could lock an HREV and keep it on disk to rollback to a fresh state, or whatever.

I’m starting to learn more about the HPKG format, and recently read here in the forum about the disk image stuff. Also, I’m toying around with HPKGs, to see if I can get more familiar with it and get a better undersanding on how it works. I have the feeling that HPKGs are more powerful than expected in many ways.

Some might thing about AppImages, Snaps or Flatpaks for software deployment, and mostly that is the idea. I would need, once Godot is stable on Haiku, to keep several versions of Godot in order to be able to update old Games with the right version of Godot, using the one the game was developed with to reduce developing time and possible flaws. Importing a project from an old version into a new one has been a big mess for me with Unity, and with Godot moving to v.4 it will be even worse I presume, so I rather keep the version of the software the game was developed with. Also applied to Blender, from 2.79 to 2.90, opening files from 2.79 in 2.90 well… Might be my bad though. This could apply to other software too.

All this is wishful thinking at the moment, though I presume it can be doable in the near future in order to get a “safer” more stable OS. Nonetheless, R1 is the short-term goal, and there are plenty of milestones to reach before release, and this is my specific use case.

Regards,
RR

25

Putting the dependencies inside each package is something we don’t want to do, for the simple reason that it makes deploying bugfixes and security fixes a nightmare (you have to manually rebuild all these packages, when rebuilding just a dependency would work).

For things where versions are incompatible, we usually provide various versions already. This is the case for libpng, libboost, openssl, python, and we can do it for Godot if it’s needed. It’s just a matter of changing the name of the package (for example we have python37 and python38 packages and they can live side by side just fine).

The core vs apps thing is already possible: install your apps in /home/config/packages instead of /system/packages. That’s all. HaikuDepot and pkgman do not easily allow it, yet, but that can be added. And we could extend it to make it easy to create multiple directories like that, in a way similar to Python virtualenv, if desired.

No need for ugly things like AppImage/Snap/Flatpack which only results in lots of duplicated data to achieve this. The ideas and problems were already considered when designing the package system and it can do all that (and more) just fine. We’re just missing some tools to make it a bit easier to work this way, as we covered the basic needs first (that is: being able to update the system from one beta to another) and only planned for all the extra tings.

Now that haikuports is getting bigger and bigger, and there are a lot of apps, the need for all these extra features maybe becomes a little more visible.

3 Likes
26

You just gave me the answer to a test I was going to carry along this weekend. Thanks! :smile:

This is also very good to know. That is part of another test I was planning to carry on this weekend too. Guess I need a better insight on how Haiku does things to get the most out of it, never too late to learn. :wink:
And, now that I re-read that paragraph, it’s clear that I didn’t pay much attention at HaikuDepot packages, precisely Python. True. I should have (or should have had have or had?) realized about that long ago… mmmm… aging :rofl:

I had the right feeling, nonetheless… :smiley:

Thanks for the explanation. Hope I won’t forget that in the near future.

Regards,
RR