Which Linux/BSD filesystem is most similar to BFS? Especially with regard to extended file attributes.
Also, do you know of any OS, other than Haiku, where extended file attributes are indexed by the operating system itself? Linux has Beagle, which comes close, but that is an application.
In my (limited) knowledge, Haiku is still the best system for indexing/managing extended attributes. Or is there some something else that might compete with it?
XFS is the obvious choice as it’s the biggest inspiration for BFS. It is an extent-based filesystem with extended attributes. XFS has aged fairly well, it remains competitive for normal workloads today. Unlike BFS the XFS project had more than two engineers, and more than a year to go from conception to a working filesystem.
No, other than BeOS and thus Haiku, nobody else tries to bolt this sort of index code inside the kernel. There’s no conceivable reason to do it, you can read Dominic’s rationale in his book but it really doesn’t make a whole heap of sense. And the end-user-visible aspect of the index, Tracker’s search feature, actually works in a way that’s contrary to the whole point of the index, it ends up doing what would in a real database be called a “table scan” in order to deliver the behaviour a normal user would expect with case insensitive filename searches.
Indexing is an ongoing subject of research, so committing to a specific on-disk format, and to code baked into the kernel means you can’t benefit from anything learned in the last ten, now more like twenty years. And it’s not as though the simple B+tree index chosen by Be was the state of the art even back then. You mentioned Beagle: unfortunately Beagle is moth-balled but other projects continue, without being stuck with some ancient data structures.
Dominic’s book (Practical File System Design) is well worth reading, but uh, don’t trust him too far on the technical side. For example Dominic describes the primitive text matching parameters in BFS as “regular expressions”. They’re not, just glob patterns which are far less powerful (the * in a glob is not a Kleene Star).
Do you mean that having the indexing directly supported by the kernel is not a good idea, at the end of the day?!?
I was under the impression that, on the contrary, this had some advantages over application-based indexing, but then I am not an expert of this.
I guess the next question is: will Haiku’s next versions still keep indexing within the kernel?
Sadly the role of extended attributes in Haiku seems to have been downplayed a lot recently. It used to be that extended file attributes were mentioned a lot (especially at the time of Be), but now?
It seems nobody cares much about them today, and they are no longer showcased as they once were. I wonder what the reason might be. It is a pity.
Sure, we still have some applications like mail applications, people contacts applications and the like which use extended attributes, but they are the exception, and not much importance is given to them anymore.
Too bad.
[quote=HaikuForever]
Do you mean that having the indexing directly supported by the kernel is not a good idea, at the end of the day?!?
I was under the impression that, on the contrary, this had some advantages over application-based indexing, but then I am not an expert of this.[/quote]
In my not so humble opinion yes, on balance this costs too much and delivers too little value. Obviously at least some people at Be felt differently, and perhaps some of the Haiku developers likewise.
There is one, very significant upside to having the kernel involved: The kernel can make sure that so long as you’re running BeOS / Haiku the index is kept up-to-date (of course though all bets are off if you run a different OS with BFS write capability which may not maintain the index).
Any non-kernel indexing feature would obviously face the possibility that it wasn’t running when something changed and so it couldn’t update the index. A system like Beagle for example, must identify updated files and re-index them periodically. This was the situation in early BeOS (pre-DR9 I think) with the original BFS now mostly referred to as OFS among BeOS fans.
The “live query” feature is sometimes argued as an upside to the in-kernel index, but this can fairly easily be implemented by monitoring all new file metadata instead, most operating systems provide a way to do this for local files at least.
You mean after Haiku R1? That would be a question for the developers. But even today you will see people recommend formatting auxiliary BFS filesystems with no indexing for certain problems, because the performance of BFS with indexing is really quite bad compared to a typical modern filesystem that doesn’t. Core features of Haiku rely on the index, or at least on fast queries, so a situation where the index never exists, or isn’t provided by default for the root filesystem is unlikely in the foreseeable future.
[quote]
Sure, we still have some applications like mail applications, people contacts applications and the like which use extended attributes, but they are the exception, and not much importance is given to them anymore.
Too bad.[/quote]
Universality is quite important. The Windows (NT) family saw the same trend, initially several Microsoft products used NTFS extended attributes to store metadata but users found it confusing that the metadata was easily (and often silently) lost e.g. if they emailed the file or put it onto a non-NTFS disk. So almost all current Microsoft products avoid using extended attributes, instead stuffing metadata into the file itself.
We have seen this before, record-oriented file storage was very common in the 1970s, but the advent of systems offering simple “files are just an array of bytes” storage dominated, because getting your files correctly from one record-oriented system to another was a pain, while the simple array-of-bytes approach worked well across many systems.
Extended attributes have been replaced in some ways by separate databases of file metadata. The MacOS Spotlight feature is a good example, where the indices are updated by a separate daemon whenever a file changes. It uses plugins to decode the contents of known file types, and generate keywords and other kinds of metadata for the index database from the file’s raw data. http://en.wikipedia.org/wiki/Spotlight_(software)
If you wanted to do this in BeOS/Haiku, you'd need a global keywords attribute that was also in an index. Preferably an index that can handle more than one keyword per file (I did some experiments with that a while ago in BeOS). To keep the indices updated, either have a daemon that updated the keywords (with plugins to handle parsing the different the file types) or require that when saving a file, the related keywords are saved into an attribute.
- Alex
[quote=NoHaikuForMe]Universality is quite important. The Windows (NT) family saw the same trend, initially several Microsoft products used NTFS extended attributes to store metadata but users found it confusing that the metadata was easily (and often silently) lost e.g. if they emailed the file or put it onto a non-NTFS disk. So almost all current Microsoft products avoid using extended attributes, instead stuffing metadata into the file itself.
[/quote]
This observation is very interesting. I had always being wondering why NTFS’s extended attributes are not used, and you have provided an explanation. But this makes me think, how come other (Linux) filesystems support xattrs? If their use was something of a problem, it would be a problem that extends across the board. Why only Microsoft seems to have discarded xattrs while other filesystems actively support them?
As for the use of extended attributes potentially leading to a data loss (if a file is sent across email or to a different filesystem), could not there be a way to preserve them? In fact there is: simply sending the files within an archive that supports extended attributes (such as xar); I believe this is a right step in the direction towards universality. There may also be other solutions, of course.
Moreover, the same problem (loss of the data contained in xattr fields, when sending the file somewhere) would equally occur if the relevant metadata was stored within a separate database, instead of a filesystem attribute. My gut feeling makes me trust more the filesystem-backed metadata managing than an application-based or daemon-based metadata managing; I somehow fear that in the latter case it would be much easier for the data to be out of synch. What do you guys think?
The BeOS MDR mail system has an option for including the attribute information in file attachments, and Zip for BeOS/Haiku also stores the attributes. There’s also the Campus FTP server and NetPenguin FTP client which understand attributes. Though for making multi-gigabyte backups, none of those work.
I think perhaps your misunderstood. Microsoft did not remove extended attribute support from NTFS. Indeed it is still actively used in some specialist applications where it’s appropriate. What they did was stop using it in applications to store metadata of the sort it’s used for in BeOS.
And xattrs are still heavily in use on Unix systems too, but again not for application metadata. For example POSIX ACLs are typically implemented with xattrs, and so are SELinux file labels. Of course BeOS was never intended for such situations, it lacks the namespacing capability needed, altering the contents of an attribute in BeOS or Haiku is as easy as altering the contents of the file. That would never do for ACLs or labels.
To be user friendly it is not enough that it’s possible to preserve the attributes when they contain application metadata. It was of course possible to preserve them in Windows. Everything must silently just work. Sending a file must both preserve the attributes (in case the recipient has BeOS/ Haiku) and yet also work without trouble on systems that know nothing about them. Every available program must be aware of attributes and handle them correctly, even though of course most programs aren’t written for attributes and don’t know how to handle them correctly. This turns out to be hopeless, the inertia involved is too great.
Thanks again for the clarification, NoHaikuForMe.
As you point out, it may not be easy to make something universal.
But what do we care about that? If the way we manage things with Haiku works just fine for us, why should be worry about cross-platform compatibility? As long as what we are doing is meant to stay on our local (Haiku-based) system, e.g. the emails stored locally by the mail program, we should be fine.
[quote=NoHaikuForMe]XFS is the obvious choice as it’s the biggest inspiration for BFS. It is an extent-based filesystem with extended attributes. XFS has aged fairly well, it remains competitive for normal workloads today. Unlike BFS the XFS project had more than two engineers, and more than a year to go from conception to a working filesystem.
No, other than BeOS and thus Haiku, nobody else tries to bolt this sort of index code inside the kernel. There’s no conceivable reason to do it, you can read Dominic’s rationale in his book but it really doesn’t make a whole heap of sense. And the end-user-visible aspect of the index, Tracker’s search feature, actually works in a way that’s contrary to the whole point of the index, it ends up doing what would in a real database be called a “table scan” in order to deliver the behaviour a normal user would expect with case insensitive filename searches.
Indexing is an ongoing subject of research, so committing to a specific on-disk format, and to code baked into the kernel means you can’t benefit from anything learned in the last ten, now more like twenty years. And it’s not as though the simple B+tree index chosen by Be was the state of the art even back then. You mentioned Beagle: unfortunately Beagle is moth-balled but other projects continue, without being stuck with some ancient data structures.
Dominic’s book (Practical File System Design) is well worth reading, but uh, don’t trust him too far on the technical side. For example Dominic describes the primitive text matching parameters in BFS as “regular expressions”. They’re not, just glob patterns which are far less powerful (the * in a glob is not a Kleene Star).[/quote]
Pretty much it can be summed up as such, you don’t like anything about Haiku or BeOS, based on your posts. So why do you keep posting here ? Troll ?
Easy, man. Give him the benefit of the doubt. Anyway, his posts contain useful information too.
[quote=thatguy]
Pretty much it can be summed up as such, you don’t like anything about Haiku or BeOS, based on your posts. So why do you keep posting here ? Troll ?[/quote]
I am interested in the process by which operating systems die and the surrounding community ebbs away. Do you remember the movie “Buffy the Vampire Slayer” ? There is a deliberately extended death scene for one of the villain characters which is most amusing. I have been watching not only BeOS (and so now Haiku) but the Amiga OS, IRIX, OS/2, RISC OS and so on.
Alas the communities associated with any operating system, but particularly dying ones, are awash with misinformation. I see it as a useful expenditure of my time to give better quality information here and there.
Of course I can’t always be there, for example on IRC today PulkoMandy went to some effort to explain about how USB supports devices which incorporate multiple unrelated interfaces such as a mouse and keyboard, but nobody uses this functionality because Windows doesn’t support it. Seemingly a compelling technical argument, except it’s quite wrong - this functionality is used in popular products today and works just fine in Windows.
But this is all off-topic for this thread, which was about BFS and extended attributes so you should probably take any further discussion elsewhere.
Basically, your a troll. Why is it that If your such a gifted developer, why are you troling???, go get a job and find a better way to pass your time. You sound like a paid shill for Microsoft or the Linux foundation. Also, people disgreeing with popular approachs doesn’t make them wrong, if everyone always did everything the same way, Nothing would ever mutate and evolve. To boot, there is a lot of wrong headed thinking in tradition, which is what you regulalry suggest to be the best way, tradition instead of experimentation. Everyone here like whats happening, If you don’t, you don’t need to tear them down, it doesn’t make you a bigger man, even if you are semantically correct on some points. It just makes you annoying and childish, just becuase someone has a different Tonka truck or a love of stamps doesn’t make you morally superior and in a position of authority on the subjec or over others, your opinions are not more valid then anyone elses
t. It just says you are a fucking jackass if you have to act that way.
There a great book you might wanna read called
How to win friends and influence people
Go fix your self.
I don’t know…
when I was 5 my parents gave me baufix instead of legos… Maybe that’s why I’ve been drawn to haiku.
it’s abnormal to me why you think integrating attributes into a file system is a bad thing. Google tried copying BeOS by indexing in windows XP, and generally pissed everyone off, where BeOS made it a pleasant experience. Linux is worse. Apple hired a lot of be engineers to implement spotlight.
so… Probably whatever bastard stepchild of a file system apple has currently would be the closest.
[quote=alphaseinor]
it’s abnormal to me why you think integrating attributes into a file system is a bad thing. Google tried copying BeOS by indexing in windows XP, and generally pissed everyone off, where BeOS made it a pleasant experience. Linux is worse. Apple hired a lot of be engineers to implement spotlight.[/quote]
Attributes themselves are harmless, and so as we saw above NTFS, XFS, ext3 and so on (including HFS+) support attributes, but applications don’t end up being able to make much use of the attributes for metadata in the way Be intended without becoming user hostile because of the problems I described under “Universality”.
Dominic Giampaolo is not “a lot of Be engineers”, just one guy - who else did you have in mind? Spotlight is nothing like the BFS index, it’s a conventional userspace metadata index, along with a bunch of other features. HFS+ named attributes are not used to support the Spotlight feature, and HFS+ does not have a B+tree index for each attribute as BFS does. If Spotlight’s index daemon is not running (typically because it has been stopped by a user who was annoyed by how slow their Mac runs and found out that it goes faster with the daemon disabled) the index becomes gradually stale.
The “bastard stepchild of a file system” you seem to be thinking of is HFS+ which is much the same as its predecessor HFS except enlarged in various ways. It does share with BFS an enthusiasm for B+trees but otherwise the two are very little alike. As I said earlier of all the file systems people might be familiar with, BFS has most resemblance to XFS.
You obviously don’t know who I am. While I never worked directly for Be, I was privy to a lot of the inner workings of the os.
the last time I saw Dom was at a BeDC in 1998. Bruno iirc was also hired by Apple after Dianne left palm for Android inc (I think Google at that point).
you think a fs shouldn’t operate the way we decided back in 1992, 1995, 1996, 1998, and 2001. We made changes to the way the file system worked on a constant basis and never had to change the structure itself. The fs was one of the best ideas to come out of Be, yet you disagree with the intentions of some of the greatest computer science engineers of the 1990s…
so I must ask, who the heck are you? And what does your answer have anything to do with the answer I gave the op, maybe I should clarify… Mac has imho the closest fs to BeOS at this time. Maybe not the same exact technology, but a similar experience to the end user.
What about the second part of my question, which was:
You guys pointed out that XFS and Mac’s HFS(+) are the closest one gets to BFS. In am curious to know, do they have any significant advantage compared to BFS? Again, with special reference to extended attributes (whether indexed by the OS or not).
Thanks again to all knowledgeable guys who share their expertise.
[quote=HaikuForever]
In am curious to know, do they have any significant advantage compared to BFS? Again, with special reference to extended attributes (whether indexed by the OS or not).[/quote]
Only in reference to the extended attribute feature? Not really. XFS offers namespaces for its extended attributes, but that’s only useful in a multi-user OS.
They do have disadvantages in this regard, if you were trying to use them as a “drop in” replacement for BFS. XFS attributes are limited to 64K for example. Few practical uses for attributes larger than this are apparent, but it is a distinct limit compared to BFS where you could shove gigabytes of data into an attribute if for some reason you needed to. If this was a key feature for some reason NTFS is a more attractive option.
Likewise both HFS+ and XFS attributes are untyped while BFS has a type marker with each attribute, so e.g. a MAIL:to attribute also is marked as being text, rather than a 32-bit integer or raw binary data or whatever. The POSIX APIs for extended attributes are untyped too.
For a serious operating system the set of people “privy to a lot of the inner workings of the OS” is the general public, that’s what books like “The Magic Garden Explained” are for.
Bruno … ?
During this period in which you claim “we” (which seems to refer to Be Inc.) never had to change the structure in fact BeOS threw away its entire file system and Dominic Giampaolo developed a new one from scratch.
It’s a serviceable but ordinary 1990s journalling file system that leaves a bunch of really nasty corner cases unsolved. Dominic was one guy with less than a year and he did a good job considering that, but this was about catch-up, not innovation.
Today the emphasis in new systems is on integrity protection and both disk sizes and block sizes have grown (4K drives) so BFS looks a bit antiquated, there’s also been renewed interest (e.g. in ZFS, btrfs) in putting volume management functions into the filesystem itself. Widespread use of non-rotating media (ie SSDs) makes a big difference too.
I presumed that HaikuForever was interested in the filesystems, not the behaviour of Finder / Tracker. Yes, elements of the Spotlight feature exposed in OS X Finder are similar to the Query feature of Tracker, despite the differences in how they’re implemented.
Tracker offers more direct power to the user (e.g. glob matching) but Finder probably better captures what most real users want. I believe Spotlight does not repeat the mistake of using a case-sensitive primary index (also, the OS X insistence on normalising all Unicode means their search does what non-English users would expect, although the OS pays quite a price for that feature)