LAN "proxy" for Packages Updates?

Feedback Ideas
1

After a long time using basically only 2 Haiku installs (beta4, 32 and 64 bits)… It seems I keep getting more and more Haiku installs everyday.

Either due to using “new” hardware, or testing performance on different VM solutions (also on different hardware), etc… The thing is… I’ve already lost track of how many Haiku installs I have (and were or their hrev :smiley:).

All I know is that eventually I need some of them to be up-to-date (some more regularly than others)… and that much of that could REALLY benefit from having a “local update proxy” of sorts.

It would mean less bandwidth usage on Haiku’s repo servers, and also… it would speed up enormously my updates (after the first one, of course)…

For reference: I rarely have gotten more than 250/300 KiB/s while contacting Haiku’s repos in the last 2 years (while being able to download from anywhere else at ~ 1.5/2 MiB/s when my internet access works well).

Sometimes I get weeks of crappy/moody Internet access, and things get FAR worse, so a working proxy would help even more.

For similar needs on Linux, I had relied, on the past, on tools like apt-cache-ng.

It would be really nice to have something similar for Haiku.

I surely will end up (eventually) creating my own solution (hacking some script that keeps a centralized copy of packages, and setting up a repo on my more “stable” machines, and making sure the rest of my installs contacts that before the Haiku servers)…

But… I would certainly be glad if someone smarter than me either beats me to it, or creates some nice “GUI-first” solution… (or by implementing similar functionality on the OS itself, I guess).

Anyway… just wanted to share this idea/need, and see what other users/devs have to say.

Side note… would be nice to have this working with Zeroconf/Avahi/Bonjour… as I don’t really have much control about most of the IPs on my network.

As always, please excuse both my broken “English”, and my unchecked verbosity :frowning:. Thanks for your time reading this.

3 Likes
2

I thought I’d already made a ticket about that but I cannot find it.

Anyway, currently I do this over ssh for my own machines: just copy the package over and install it.

The long correct way goes like thid

  • Publish local package archive via dns-sd
  • Discover local archives over dns-sd
  • Add to repositories list
  • Get list of updateable packages from upstream
  • Get list of updateable packages from local archives
  • Download packages that are in both lists from local archive
  • verify authenticity with minisign

Anyway, thanks to kallisti5’s work on repo signing the last part is easy. The problem is now that we do not have dns-sd implemented (or mDNS for that matter).
As I understand multicast itself is now supposed to work though.

edit:

It’s just called DNS-SD ; )
https://www.rfc-editor.org/rfc/rfc6763

3

Thanks for your input @nephele.

Yeah… I’ve done that in the past, but… I don’t use the same set of packages on all installs, so that is not always easy. Say… my “daily use” installs are Qt/GTK free, for example, I just can’t install/sync “everything” from the “proxy”.

I certainly “can live” without zeroconf/mDNS/dns-sd/etc (and thus, having to manually update IPs if needed), and as this would be a “local-only” thing… not really worried about minisign and stuff (at least, for now… of course, a proper solution would make use of that!).

Thinking about storage usage (I would like to avoid unnecessary “.hpkg” duplication)… wonder if a really good solution to this “LAN proxy server” couldn’t also handle the storage requirements of the /system/packages/administrative/state-* folders (this of course, assuming OS-level support for such a feature).

Maybe for “v2” of a possible solution? :smiley:

Certainly easier to remember. Thanks for schooling me!

4

Dom’t really understand what you mean.
A package repo is just a http server so.

  • Collect info about all hpkg available in packages/
  • cache this info
  • if any package is requested over http just find this in the packages/ folder (maybe cache path from step 1)

No need to change anything about how they are stored

5

Sorry if I’m not being clear enough.

I mean the actual storage of (possibly out-dated, early-state) .hpkg files.

Currently, they reside in each individual state-[timestamp] folder.

As packages’ names are unique (and we can even check their checksums)… I don’t see why they couldn’t just be stored on a central location/server. In fact, I think I recall a ticket/post where you even suggested using just one folder to have all the .hpkg (hoping I’m not miss-remembering what you wrote).

With that (and each Haiku install having network access and dns-sd working at boot :-D), each Haiku install could, I think, only store its different “activated-packages” files locally on, say, /system/packages/administrative/state-[timestamp], and leave the actual storage location of the needed .hpkg files to the LAN update proxy.

Hope I’m making any sense :slight_smile:

6

They are not. Sorry : (

So this wouldn’t really work

(also, I don’t see any advantage in this aproach. The FS already is a database after all)

7

Not sure if we’re referring to the same thing. On the buldmasters’s repos… you have pretty unique .hpkg filenames. Otherwise… you get the usual (paraphrased) “you forgot to update the revision number, you dummy!” error in the logs).

If those filenames/repo/repo-proxy/repo-signature are the same, why couldn’t they be stored on the same location?

With my HaikuPorts work… I’m painfully aware of the “package name already in use” infamous packagefs error while attempting to sync my installs to the official repos, after submitting a PR that got accepted on Haikuports.

Maybe I need to file some bug reports regarding that, indeed.

Language barrier is either grater than I think, or we are approaching this from different angles. No worries, in either case, of course…

Is the “database” doing de-duplication? Does that better conveys my intention? (avoid storing the same .hpkg file x-amounts of times on the same “local-Haiku-network”)

8

Nothing is stopping me from having severall packages with the exact same name.
For example when working on a patchset, or maybe a friend sends you a fixed version of something.

Now those could also conflict with haikuports. Who is to say what is right? Do you just throw the other version away? or maybe do a “store” with UUID’s like NIX does?

Add to that that the case of “don’t want duplicates in my network” only works in the narrow case of all haiku comouters in your network beeing controlled by you, and not someone else also.
But, if that is the case/ the main concern I do not understand why you would not netboot instead. It sounds much easier than making a finicky network based fs for packages only.
It seems like too big a failure point.

How do I recover if i want to use libreoffice but it is not on my local pc but one that is offline or broken?

9

Seems you and I have pretty different experiences dealing with HaikuPorter/HaikuPorts, and trying to do pkgman full-sync after installing a package for which we created a PR that got merged, I think.

Dumb me kinda expected the repo-priority number have some effect here? like in… Haiku repo has prio=1, beats my silly locally build package with no prio (but only for pkgman full-sync, not for pkgman install/update).

Regarding the previous-states-pxe-boot… ok… forget about it. Was just a side-note, and my guess network is too slow for that to be practical anyway.

By requesting from the “proxy-server” the packages listed on the state-[timestamp]/active-packages file you choosed to boot into, was my expectation.

Any way… Let’s go back to the main point… “Central LAN proxy for local updates when?” :stuck_out_tongue: