AppFile

1

Today with advent of netbooks and high emphasize on portability it occurs to me that perhaps having a slew of highly portable applications a welcome addition to Haiku.

Firstly I’d like to note I’m not too much familiar with Haiku’s current application installation system but from what I gather it will be quite different from my suggestion, so here it goes.

Basically my whole idea is based on klik. What it does is compresses all needed files into a single file which is then mounted and unmounted in order to respectively load and unload it from the memory. Using this approach comes with great advantages:

  1. Highly portable applications - if constructed properly (meaning lots and lots of hours of developer work) these applications could be frozen and redeployed elsewhere in the exact same state as they were before they were frozen.
  2. Intuitive way of handling applications - applications could be deinstalled by deleting and installed by just moving/copying file from one computer to another
  3. Multiple versions of applications running side by side - rename your file and voila you can run programX v2.0 and v3.0 seamlessly without much confusion or fuss
But you may ask why is this any different from apple's AppDir approach except that it is a compressed folder? Well for one this is a single file meaning that when you overwritte an existing file it doesn't merge new and old files into a single entangled mess like AppDir do.

Of course there is also a great deal of disadvantages when dealing with a single file:

  1. Meta data retrieval - it is harder to get meta data such as assosiations and icons from a single file unlike the folder structure
  2. Current application would be hard to transform into an AppFile format - It is pretty much an issue of standardization and order. Some applications might want to store their settings and config files somewhere else on the system leaving behind tails. Possible solution: Internalize configuration files into the application file, which might be a hard task for some apps
  3. Bloat - since bascially all apps would carry their own non-core libraries there is a chance that your apps will simultaneously load same version of certain libraries into the memory. Possible solution: Force AppFiles to check a common library repository for a library they might use before using their internal version.
  4. Security issues - making the instalation this easy could be exploited by a third party to gain access. Possible solution: some form of control when accessing the application for the first time
  5. Updates - Some older versions of the software could use older internal libraries instead of using newer which break compatibility. Possible solution: Design a way for upgrades to replace your current version while preserving settings.

The list of disadvantages does seem a bit bigger but I believe that the pros though less numerous outweigh the cons.

2

There was some talk early this year about application installation systems, some of the ideas in fact, are very much like your own.

software organization/installation
software-management-proposal
Pathrelocatable-software-and-assigns

pity nothing come about from these conversaitions.

<em>
Edit: 
If a dev reads this, the Subscribe bar is looking weird as of late and the new html toolbar when adding comments doesn't work. (well its HTML link code anyway) 
</em>
3

Hmm interesting. That is a huge read and I’ve barely started reading it (more like skimming it actually).

Nothing came of it? Did they voted against it or there wasn’t enough will to implement it?

Either way it is truly a pity. I consider the AppDir/bundle installation a highlight of Apple’s features.

4

Well a new application installation system would be something that I think the devs would not look at implementing before R2, and who knows how long away R1 will be (A1 has nearly taken a year itself).

Although a GSOC project could create something before then, such as
something along the lines of this: GSoC-project-application-updater (ok, not 100% what we are talking about, but you get the point)

5

We Haiku developers are certainly open to the idea of being more innovative in the packaging and installation of applications.

I don’t remember all of the discussions mentioned by Richie, but one of the more interesting ideas was a “package filesystem.” Basically applications would be packaged as they might normally be in singles files (a zip file would be fine), but instead of being extracted and “installed”, they are opened by a special package filesystem module that overlays the contents of the packages on the real filesystem. So it seems like they are installed, but if the original package is removed, the package filesystem would detect it and remove all the associated “overlayed” files.

This sounds pretty similar to what you are talking about, just more transparent to the user since they would be able to see the overlayed files (while your idea seems to be more “behind the scenes.”)

I was certainly pretty intrigued by this idea, but no one has yet had the time or motivation to implement a prototype to try it out. When my browser project is further along I may take another look at this.

6

Sigh It’s not exactly fun for me to see this topic come up again, considering that the idea was first suggested at the first WalterCon five years ago by Michael Phipps himself, and work has been done on it, but it has been almost completely – if not entirely – ignored. Nice to see history repeating itself again. :confused:

Filesystem images work well for most applications, but they don’t work for everything, which is why OS X has .dmg’s and .mpkg’s. The second ones are for more complicated installs. Nathan Whitehorn wrote the original code which used attributes to handle most of the messiness. I built on top of it with ideas of my own and those of Waldemar Kornewald. I’ve already got working code for both kinds of packages. The simple packages aren’t quite stable on R5/Zeta and bugs prevented their use the last time I tried them on Haiku. That was quite some time ago, though, so I should see if they work the next time I get a chance.

You can find most of the ideas that have been discussed and how they could fit in the “big picture” in a proposal that I wrote some time ago at http://darkwyrm.beemulated.net/haiku/R2_Desktop_Refined.pdf .

7

I don’t see why we have to complicate this so much… You know how I “install” most emulators for Windows? I extract the contents of a zip file, to a folder of choice. Settings and DLLs that are not part of the standard system, usually come with the zip file. So, we have a folder with the program and all that it needs: an .exe file, a config file, and a DLL or two. If it’s an advanced emulator it may have subfolders, like “plugins”, “bios”, etc, but it’s all self-contained and does not pollute the system.

How can we make it even better though? Well, we could remove the need to even extract the zip file, that’d be nice. We could name it FILE_NAME.zip.app, making the OS recognize that we want it to be executed directly, and not be opened by an archive manager. A simple mechanism could then look for executables in the .zip.app file (which BTW is just a renamed .zip file). If there’s only one executable, it will load that one automatically. If it finds two or more, it could ask you which one to load. If a single application has several executables for some reason, we could use a simple text file inside, which could look like this:

ExecutableFileToExecuteUsing.appLoader: somefilename

Also, the conflict between libraries installed in the system, and libraries in the application folder/.zip.app file that may arrive in a few occasions, could be solved either with a simple system setting, or through the text file mentioned above, with a string like this:

OverrideSystemLibraries: Yes

Since this would require close to zero extra tinkering compared to Only compressing it, it would be easily done by application developers. Do we need to make it any more complicated than this?

Also, if an update program for the applications (not just the core parts of the OS) would be created for Haiku, it could simply have the option of letting the user show it where your application resides. It can then easily replace the parts of the .zip.app file that are to be updated with new ones.

8

I think Denise Purple is right: why complicate things so much?

I, personally, always hated the installation thing and all this comport: broken registry in Windows, dependency problem in Linux… I love, indeed, application that simple create a folder for them and just works!

So I’m for one application -> one folder that contains the library configuration, localization file and so on that application needs… nothing more!

9

That might even work for native apps… although, if for example, a whole bunch of native apps use libcurl, and a security patch for libcurl is released, every one of them will have to be updated separately!

In any case, at this point, discussion leads nowhere. Someone needs to actually build something (or port something) that we can start with.

I’m OK with some form of package management, and I think for ported apps, it will be a must as the dependencies are already a disaster (see: ScummVM, Gnash, etc.)

10

That might even work for native apps… although, if for example, a whole bunch of native apps use libcurl, and a security patch for libcurl is released, every one of them will have to be updated separately!
[/quote]

Maybe as libcurl is a very used library can be included in the system libraries so all the applications that use it haven’t to include… and if there is an update only the system libcurl is updated… on the other hand I think if some application includes in is “installation folder” libcurl 1.00 and the system version is libcurl 2.00 the application must use its version… to avoid incompatibly problems!

For the GNU ported application I think they must adapts at the Haiku installation philosophy (when we decide what is…) and not the contrary… I think that say GCC or Apache must use a “installation directory” too… if they like to put something in “etc” they simply put in /apps/xxx/etc NOT in a not existing /boot/etc or in another directory created for this scope!
I hate the confusing Linux philosophy: all the files sparse in the filesystem thousand file to change to configure a process… we must boldy avoid this!

And not forget the powerful thing that if the apps are installed in a “folder” they are portable in a easy way: simply copy the appFolder in another Haiku PC… it works and with your settings, too! If you have 1000 files sparse in the filesystem this is IMPOSSIBLE!

11

I don’t remember hearing about this, but if you have something fleshed out already you should definitely try to get it working in Haiku and put it in the wild for others to try out.

I still think the package overlay filesystem idea is worth looking into, and a nice evolutionary step instead of just copying Mac OS X.

[quote=darkwyrm]You can find most of the ideas that have been discussed and how they could fit in the “big picture” in a proposal that I wrote some time ago at http://darkwyrm.beemulated.net/haiku/R2_Desktop_Refined.pdf .
[/quote]

There isn’t anything in the above about packages or installation of applications. I do think it has some nice ideas though.

12

I’m out of town right now, but as soon as I do, I will be. :slight_smile:

[quote=leavengood]
I still think the package overlay filesystem idea is worth looking into, and a nice evolutionary step instead of just copying Mac OS X.[/quote]
I don’t remember hearing about this. Could you elaborate?

Whoops! There was one more pdf to go along with it. You can find the details about the systems at http://darkwyrm.beemulated.net/haiku/PackagingRFC1.2.pdf . It’s out of date – it still uses the name OpenBeOS, for example, :wink: – but it gives you an idea. The category list was updated in the 3D Desktop FAQ. Having looked at it again, methinks I will be updating it in the very near future, seeing how people have interest in it again.

13

@fano: We could have an option in the OS where we can easily choose this:

  1. Always ignore libraries in application directories if the system has them installed system-wide

  2. Ignore libraries in application directories if the system has newer versions of them installed system-wide

  3. Always use the libraries that come with the applications regardless of what is installed system-wide

This way, the user can make sure that every needed library will always be available, and that if a newer one gets installed system-wide, that one can be preferred automatically by the system.

I’m with you about GNU also. If we’re going to have GNU stuff in Haiku, it absolutely must adapt to the system, not the other way around.

14

[quote=Denise Purple]@fano: We could have an option in the OS where we can easily choose this:

  1. Always ignore libraries in application directories if the system has them installed system-wide

  2. Ignore libraries in application directories if the system has newer versions of them installed system-wide

  3. Always use the libraries that come with the applications regardless of what is installed system-wide

This way, the user can make sure that every needed library will always be available, and that if a newer one gets installed system-wide, that one can be preferred automatically by the system.
[/quote]

Yes, but I think this must not a user choice… a “normal” (not nerd) user doesn’t know what option choose… and I think Haiku must be simple as possible so “no choice” -:slight_smile:

IMHO the more safe is 3… but if the library is installed system wide you can use it… but only if is the same version. For the ram-eater concerned if more applications use the same version of a shared library the loader (ld, right?) must be so smart to not load another version of the library in memory but use that that is already running.

That is if the system wide library is updated but not the local version of a program then the program must use its version of the library… it’s application developer fault to say: “version xx.xx of Myapp is compatible with lib.so.2.0. Now is safe to use it!”

[quote=Denise Purple]
I’m with you about GNU also. If we’re going to have GNU stuff in Haiku, it absolutely must adapt to the system, not the other way around.[/quote]

Yes… we are in our house we must dictate the rules -:slight_smile:

DarkWyrm I’ve read your packager proposal very interesting… I think that a field must be added to xml file: the updater site… my idea is that instead of an unique giant repository we can have a lot of small repository ideally we must fetch the new version of the application from the application developer NOT from some Haiku server… obviously we can provide some mirror system, eventually!

You can say what not work with giant Linux repository?
Nothing… when you found your application in them… but if you don’t?
You’ve to go open google found the web site… search where they hided the info… and then open a text file and add the repo for the application.
This succeeded when I tried to install wine in a Fedora some years ago… not really friendly.

So it best to use always this approach: where is the updates?
The installed software knows it (and it’s the developer site!) -:slight_smile:

15

Except that, if the software isn’t yet installed… you still have to search for it somewhere, and tell the system where to find it, don’t you ?

16

Stefano, in the current file system layout, Haiku has a /boot/apps/ directory, which is meant for applications. We should be able to solve your problem by simply recommending users to use this directory for their applications. Actually, as long as you have a default directory for applications that the system knows, whether it’s this one or another one, it should work. An update system should be able to search this directory without too much trouble. We could make it even easier for the system like this: Every program has its own identification file. A simple text file which states the program’s name and essential repository information (So that it can be updated). It doesn’t have to be bigger than a few lines to make it do its work.

17

Denise: I think you misunderstood what Stefano was saying. He was essentially saying that there is still a need for some central location to at least search for software. If you don’t know a certain application even exists, you can’t exactly go to the developer’s homepage.

Another problem I see with having the developer’s site be the main download location is something that could simply be called the “BeBits affect.” As anyone who has browsed BeBits knows, many, many, many files are no longer available because they were kept outside of BeBits. Haikuware on the other hand keeps copies of the files on their site, so there is much less chance of a missing download.

So while I agree that in general the developer site is the main place one might look for updates and downloads, having some centralized backup for downloads might also be useful.

18

Yes where to find the software for the first time is a problem… I’m not a big fan of repository and the synaptic thing… IMHO the repositories are only a trick to hide the big Linux problem: no binary compatibility!

We can have a repository system but at the same time I want to google for let say “Firefox for Haiku” find the Bezilla site and download the file for here!
That is repositories are OK, a commodity, but NOT must not be the only way to download Haiku apps.

Ryan Leavengood I agree that have only the developer site as updater is not “safe”… what do when developer site is down? Indeed I talked of the possibility of mirror site, too…

Another possibility is to use some home-made form of p2p protocol so all the Haiku PCs became mirror themselves -:)… cool but a security threat at same time maybe!

19

This sounds to me very similar to Plan9's features…

20

[quote=darkwyrm][quote=leavengood]
I still think the package overlay filesystem idea is worth looking into, and a nice evolutionary step instead of just copying Mac OS X.[/quote]
I don’t remember hearing about this. Could you elaborate?
[/quote]

The idea is to have a filesystem not backed by a partition but by a collection of packages. (Not meant to replace your /boot disk but to complement a part of it.) You would have a set of packages, whose contents would be -presented- as if they were all stored in the same shared filesystem, mounted at, say /boot/common. Installing Perl would be as simple as dropping its package in that certain backing-store folder, e.g. /boot/packages. Uninstalling Perl would be done by removing the package. (I think the pkgfs would have to live in userland for this to work.) There would be no unarchiving, copying or moving of package contents. Every package would sit there in the backing-store, self-contained.

The pkgfs is so brilliantly simple that its almost boring. One -could- complement it with an actual package manager to handle dependencies and all that jazz. It would use the pkgfs to publish/unpublish packages. Such a package manager could have a more limited scope than the ones in Linux.

We don’t have to give up any of our old ways. They can coexist. The pkgfs would actually help contain the ported unix software. (And help BFS by reducing the amount of small files.)

Some people might think “union mount”. This is not meant to be implemented as one union mount per package. Layering of packages would be internal to the pkgfs. I would make the pkgfs read-only for simplicity, so it would need layering with another filesystem to make it feel like a standard filesystem where you can also build and install thing manually. (Which you could then turn into a package by temporarily unpublishing all packages and zipping what’s left on the writable filesystem.)

See my posts in this thread: