Just like last month, while we already had an activity report, the work I have been doing thanks to the generous donations of readers like you (thank you!) deserves its own report.
And, while I was at it, I changed I/O scheduling to better account for multi-core lock contention, which might improve performance (though on the already most-performant disk driver by quite a large margin, that may be difficult to notice!)
I like what you did here 
My main PC is nvme based, and Haiku works really well on it thanks to your driver. It’s great to know that compared to it’s contemporary drivers, the driver you wrote is high performance. Well done.
Thank you for your work and dedication as well as for detailed report. I am eager to see USB WiFi support coming soon.
Thanks @waddlesplash for your dedicated work - what a string of achievements for a short work month! And thanks for fixing #12534 - it is a small thing, but it is nice to be able to launch apps like DosBox and LuckyBackup from QuickLaunch. I understand the whole mechanism may be re-engineered vis-Ă -vis multiuser, but still - this is a nice interim fix 
For me NVMe is only used on RISC-V board. Main PC use SATA SSD and speed seems enough. It already cause significant CPU usage on file operations so maybe BFS driver need to be optimized to use full potential of SSD.
BFS2 would be nice. Sadly the ZFS port didnt progressed 
Who will design it? I see no candidates. Even on Linux there are a lot problems with file systems.
BFS it quite good in terms of stability and recoverability. I do Haiku hard shutdown a lot and that don’t cause file system damage.
I am looking with interest to new NTFS driver port. Read only support is already quite good. I use NTFS on my major storage and backups.
Yep, this was always a question. Some years ago there was a discussion to switch to a different FS, like XFS or ZFS. A GSoC student did some work on the ZFS port, but i don’t recall how far it went: https://github.com/s3thi/zfs-haiku
Attribute index should be implemented in some way if switch to non-BFS. Attribute queries are used for running applications and listing applications that can open specified file type.
That is not very nice to the Haiku developer team. We have several people who could do it. But for the moment, BFS is just good enough for our needs and there are other priorities.
The main filesystem on Linux is built on ext2 which is not the best design choice (it was initially a quite simple and traditional filesystem, and extra features were added on top of this simple base design). And then there is btrfs which takes the opposite direction of trying to do everything at once, and is overcomplicated as a result.
And then there are various filesystems that were donated to Linux over the years, but did not get much traction for various reasons: ReiserFS (author went to jail for killing someone), JFS, XFS (both implemented in other UNIX systems and later imported in Linux), ZFS (possible license problems).
In my case I use XFS for my Linux systems, mainly because it has better xattrs support than ext4 so I can use it to compile Haiku without needing attribute overlay tricks. It is also reasonably well documented.
For the future of Haiku and a possible BFS2, there are some notes on what we would like to change: FutureHaiku/BFSIssues – Haiku and FutureHaiku/FileSystem – Haiku
As you can see, it is not a very ambitious or exciting plan: the main feature we would like to add is hardlinks. The other changes are about using more efficient disk organization and data structures, to reduce wasted disk space (do not allocate a whole block for each inode or store parts of the file data directly in the inode in addition to the xattrs, more efficient storage of double indirect runs), improve speed (avoid misaligned data, avoid fragmentation), and make things even more reliable (for example with data journaling).
It would be great if I am wrong, but I see a little activity on file systems from core developers for several years. Work on file systems is mostly done by students. Waddesplash is recently doing new NTFS 3G port to Haiku, but it is not new FS design or reimplementation of existing file system. Some critical BFS problems not present in BeOS are not yet solved (#15585, #12847). If I remember correctly, Waddlesplash have no idea how to fix them. BFS resize code also stay for review for years.
Currently for some operations on BFS volume (usually involving a lot of files) CPU become a bottleneck, not disk speed. I am not sure is it on-disk structures design problem, or FS driver implementation.
Lack of current activity does not mean lack of competence. Korli implemented several filesystems from scratch (Exfat, Ext2/3/4), Axeld is one of the main developers of the original BFS, Waddlesplah is currently working on NTFS. John Scipione also made some improvements to the NTFS and FAT support (for better Unicode handling, mainly). And also mmu_man worked in the past on NFS as well as some creative things like GoogleFS (which exposed Google search results as filesystem queries returning bookmark files).
Also, GSoC students do not work completely on their own, they get mentoring and code review from the existing developers, otherwise the projects wouldn’t get anywhere. I have mentored some of these projects around UFS and XFS, and some of the above developers also did mentoring. The students are generally happy about it and learn a lot in the process, and personally that’s where I find value in doing this. I wish the students would stay around more after GSoC qnd continue contributing, but we have to accept that they are attracted by the payments from Google, and not everyone is willing to contribute to Haiku for free, especially if they don’t use it as their main operating system.
So that’s a total of at least 6 currently active developers in the Haiku team having done some work on filesystems in the past. It is certainly not an area I am worried about.
It is just a matter of available time and priorities/motivation. And since we have a full time developer now, that solves at least a part of the “available time” problem. Let’s see how that goes.
I would love to have error-proof BFS and dump all my files onto a Haiku install…
Will Haiku ever get back to booting as fast as it did with Alpha 4?
There is no such thing as an “error-proof filesystem”, bugs are always going to happen. Keep backups
I have heard some reports of disk corruption now and again, but I don’t know if any are definitively reproducible.
What has that got to do with anything? Am I suddenly the filesystems expert around here? In fact it is one of the areas I have spent the least amount of time with. That is quickly changing with the work on the new NTFS driver, though (already I have some comments about the resize patches after reading through filesystem code.)
I may get to investigating these at some point. But at least the KDL one seems to be a debug assertion and not necessarily a problem (it may be a memory leak where blocks are not put back when they are done being used.)
Disk corruptions, though, are more worrying. Do we have any of those outstanding which are actually reproducible? I’ve heard reports now and again, but not of anything consistently reproducible… (or did you mention something related to KDLs?)
I would bet it is probably the latter. But who knows? None of this is magic, it’s just code like any other.
On Linux I used ReiserFS a lot. I have the best experience using this filesystem than any other, maybe also the best across all OSes (though I didn’t played much with XFS and JFS). The worst damage of the filesystem were always possible to repair with fsck. I am missing this filesystem.
I have a heavy feeling about the whole story of Hans Reiser and Namesys (the company which designed ReiserFS and Reiser4). BTRFS is not a close relative to ReiserFS unfortunately.
On the more positive note, just as Linux and nowadays MacOS, Haiku does not necessarily needs the next generation FS, it may be just one / several more FSes available for Haiku. And each can offer the better experience depending on its usage case.
Not sure if haiku will boot as fast as alpha 4, but I think we can boot faster. Right now I think most developers have bigger things to think about though.
When it comes to filesystems I’m very curious about bcachefs.
That KDL is in the block cache, and it’s more related to memory management and direct usage of the slab allocator. That is indeed an area where less people are knowledgeable, and possibly one to worry about (especially because it’s not as well documented as the filesystem things). But all it needs is someone to dig into the code, understand (and document, ideally) the slab allocator API, and then review its usage to find where the leak is.
Anyway, that is not related at all to the skills needed to improve or redesign the data structures of a filesystem, which is more about pure algorithmics than about knowing Haiku internals.