For a concrete example, suppose I have one driver that says it handles all USB-HID class devices. Then I have another that says it handles all USB-HID class, audio subclass, devices. Which one gets the higher score? Obviously the latter.
But we can’t operate on just number of matched attributes. If I have one driver which matched class/subclass/protocol, and another which did not match any of those but DID match vendor/product, who wins now? Clearly, the driver that matched vendor/product (2 attributes), NOT the one who matched class/subclass/protocol (3 attributes.)
How would a generic system know how to prioritize? Only if we specified “rules.”
It’s not terrible, but the comparison will still take more CPU cycles than the hand-written function, probably by a factor of 10 or more. Even if that’s still not much in terms of wall clock time, I don’t see why a completely generic solution is to be preferred in everything.
Really, let driver developers specify devices in structure arrays. FreeBSD does it, Linux does it, we do it for USB devices and all legacy PCI devices (which is still a lot). It’s a very convenient way of describing what devices you support. The whole-system speedup you want is still possible even with this mechanism.
Furthermore, your generic system will probably wind up being a lot more code. Why write 100-500 lines of code for a “generic” comparison/scoring system, rather than 10-20 lines of code per bus (so, you would need around 10 busses before we would equal the generic one.)
This implies any combination you hive can have a score that is predetermined. This is terribly hard to maintain, overgeneralizing in this case makes no sense.
(and if many of those busses besides USB and PCI just do string comparison, then they will all share a comparison function, and won’t increase the total number of comparison functions.)
Well, so does mine. That’s fine, in the case where more specific testing is required, there’s still a probe function; drivers should set that and not fill out a devices table if they need to have highly specific scoring.
Oh, I see what you mean. Yes, there’s a way in which my mechanism is indeed more flexible: each individual “attribute” can be scored differently, whereas in @X512 proposal, it’s just a hardcoded 0.1 per match, or maybe some other value depending on what the bus set.
But there’s another advantage of my proposal, then. Probably vendor/device each by themselves should be a small amount in the total match, but combined would be worth a lot more. How would a generic system know these attributes go together?
I havn’t read the whole thread yet (will do in the evening). But to me I would like to do drivers exposing their functions in a better way than the big if/else way.
I would propose something like what UEFI does where a struct with functions are returned according to a specific API.
Also to me ELF for drivers probably could be simplified, it seems loading the elf and doing the relocation just to probe maybe isn’t the best?
You can also open the parent device and query anything you want about it (send SCSI commands, call an hook to get an USB descriptor, …).
This flexibility is needed in some cases. While a simple driver can get away with comparing descriptors, you can always find some quirky device that will need more complex probing. You don’t have to look very far: a lot of webcams also include an usb audio part for the microphone, and our existing (but incomplete) webcam and usb_audio drivers already have trouble with fitting this in the old API.
I think we have 2 things to fix here:
Make it easy for an USB device to get the descriptor from its parent node
Add some helper function to easily match that, in the USB device driver, as easily as it was done in the old driver.
This specific part does not require a change to the device_manager itself, it is more a need to build bus-specific helpers on top of it.
Other than that, I don’t have comments on the specific points you make (which are valid), but I can provide a bit more context on my understanding of why this is so complicated.
The big picture goal of the device manager is to leverage our use of devfs to provide faster boot time. One problem with Haiku currently is that it takes a long time during boot to probe all devices. The approach taken to solve this is, mainly, to do lazy device probing as the devices are needed.
The normal way you’d expect device probing to work is (if you looked into it in, say, Linux), when the system boots, you start from a “root” device node and find what is connected to it. For example this can be the root of the FDT, or it can be the root of ACPI. From there, you explore the children of each node and build a device tree. And for each device, you see if you have a matching driver, and you load that. With hotplugging (for example USB), the device tree can also grow and lose branches over time.
In the device manager, the idea is to do only a very shallow enumeration of the tree at first. The driver loading will only happen as userspace starts requesting access to drivers in /dev. For example, opeining “/dev/disk” will ask the device manager to load all disk drivers and get them to publish their devices.
This is somewhat similar to what the old BeOS driver model did. In BeOS, this was achieved in a relatively simple way: drivers would have symlinks to them in a file hierarchy matching the devfs organization. This can be used to locate all “disk” drivers by finding them in a directory.
You can see this difference if you look closely at how Linux does things. On the surface, it has simple and short device names like /dev/sda. But in reality, it internally manages a full device tree that it also publishes in the filesystem (in /sys).
However, on modern hardware it can be a lot more complicated. A typical, relatively simple case would be ACPI->PCI->USB->USB mass storage. How do you know that to enumerate disks, you need to also load PCI and USB drivers? How do you know which branches of the tree to explore? Unfortunately, this part was left on the TODO list, and it is currently all hardcoded in device_manager.cpp.
It seems possible to achieve this by some kind of recursion. You load the usb_disk driver. It tells you “but I need you to load all USB bus drivers”. So you do that. The XHCI driver then tells you “but I need to load all PCI bus drivers”. And so on. The device_manager tries to do a bit more optimization than that, for example in the PCI case it will try to load specifically the PCI devices that are possibly mass storage ones, ignoring, for example, soudncards (until they are probed by some other driver). If we want to realize that initiali idea of the device manager, we have to think about how these relationships can be efficiently described, in a way that makes it possible to load the device tree “from top to bottom”, that is, start at a leaf you don’t know yet (an USB mass storage device), and find your way back up to the tree root. It is not an easy problem.
So, the device manager tries to fit with both of these ideas: do the driver probing based on the /dev file hierarchy, so that, at boot, you will pretty much only load disk device drivers to mount the rootfs, and later on the audi ones as media_server starts, the network ones as net_server enumerates them, and so on. No need to stop userspace from loading things while the kernel is probing all the hardware.
But, this means the device manager drivers have to manage two things: on one side, the device tree (as visible in the Devices app) represents the devices as they are enumerated, with the relationships between parents and children by physical connection. On the other hand, the “driver” hierarchy as published in /dev, which is not organized by physical connection at all, but instead by interface provided to userspace (disk drivers, soundcard drivers, etc). This explains why there are two types of nodes, and also the naming (a “device” node is published in the device tree, a “driver” node is published in /dev).
Another complication of the device tree is, as far as I can see, the strict distinction between “device” and “bus” nodes. I think the initial idea is sound: some things (like USB) provide a standard API to children devices, no matter what the underlying implementation is (XHCI, UHCI, …). For device tree exploration, it seems a good idea to have this materialized in some way in the device tree. But the current solution means a lot of bus and device nodes get published and it would be nice to cut down on it a bit. I am not sure how this would look in real-life.
I hope that helps understanding the top-level view of what the device manager attempts to do, and we can find a way to actually do it. The other option, of course, is to give up on the idea of lazily loading drivers when devfs is explored, and do it the traditional UNIX way. The cost is (probably) a few seconds of boot time.
We never did use structures for PCI devices. We had to scan the whole bus for a matching device.
Also, your example of the ECM code could be greatly simplified; there is no reason to iterate all attributes, and use strcmp() to find the ones you’re interested in. Why not just get the attribute like this (pseudo code):
I’m actually a fan of the attribute mechanism (we would probably use KMessage for that now, though), and a generic bus solution. Having said that, your other points are very valid. I think you can easily invent an API on top of the node/attr principles that greatly reduces the amount of boilerplate code.
The biggest issue with the current approach, IMO, is the device tree discoverability, though, as Adrien perfectly explored in this thread. Hardcoded string lookup is really not a good solution. I guess it would be nice if each driver had a section that contained it’s specs in a declarative way, so that we could just load those, and create the basic device tree from that (ie. with all bus/generic drivers that publish more device nodes). That could even be a text file, or a specific ELF section in a binary format inside the modules.
Also, the closer the new driver API is the current API, the easier it will be to convert all the drivers. We really shouldn’t try to combine three different driver APIs in one system. That will be even harder to manage than the current solution.
Besides, the missing shutdown/sleeping mechanism is not an inherent design decision; it’s just missing, and could be “easily” added to all driver APIs.
In any case, improving upon the device manager would be really great. It’s definitely a bit too much on the complicated side of things, even for simple things.
Linux already boot fast enough, so I see no need to optimize that part. Haiku slow boot problem have other reasons, for example probing a lot drivers again and again, loading and unloading driver add-ons multiple times. Stripping drivers that are not used to boot specific system speed up boot a lot.
Lazy driver loading mechanism based on devfs touching will be likely not work properly because of device relations and drivers that expose non-devfs kernel interface.
It would be great if there were some way to do a multiple-to-multiple hash function but at risk of sounding naïve, how hard would it be to generate such a hash function?
A great way to miss all the birds at once and never get anything done. As said at the start of the topic, please keep this on-topic and not extend the discussion in many tangentials directions. The problem is complex enough.
Also, we are dealing with maybe a dozen attributes, so, I’m pretty sure the naive strcmp based code will be faster than any complex data structure here. If there is one thing we should research, it’s using string interning, a great tool to get cheap comparisons on a set of largely repetitive and mostly static strings.
Just a small suggestion, to make it easier to understand the programming and relationship between modules : keep the “device” word for, well, devices : hard disks, keyboard, serial ports, etc. Things like PCI buses, usb controllers, etc, could use some other name that relates better to their function in the grand scheme of things, but that makes easier to visualize where are something in the tree and how they are related.
Exactly. Linux and NT have had perfectly fine device discovery for longer than Haiku has existed. If the goal is to be effective, those are models you can just outright steal and nobody will care because this wasn’t an innovation twenty years ago, let alone today. That’s not an exciting answer, but this isn’t really an exciting problem.
This is more or less how the “new” driver API functions already, isn’t it? My proposal modifies this somewhat but still keeps this basic design. (The old driver API had device_hooks which was this, partially.)
Oh, of course. You will see in my example that I’ve left in the probe or supports_device method. That will stick around, but for simpler drivers, it won’t be necessary.
Not globally, but there were individual drivers with manually written structs that were then iterated over in for loops in the init_hardware methods. So it did happen anyway.
I think the best-of-both-worlds approach here would be to have name-described structures. For example:
etc. That way, we can use the structs where it makes sense to (i.e. most places) but then fall back on string-descriptors when we want to do that, instead. This has the added bonus of getting backwards compatibility for free: if the driver uses an older version of the structure, just access it using string field name access.
Agreed, we should probably use this even with the above “descriptor” system.
Not sure, for display handling driver I would want to expose functions like, read brightness, read available brightness, write brightness and maybe more. (I’d also want to send events to userland when something changes. I have no clue what the correct way is for that is)
For reading, one of them can use the read, and write can use write, but then where would I expose additional operations. Currently unless I am mistaken, we need to write if (op == available_brightness_op) return available_brightness [1] …
You just ask for block-io protocol with its UUID, and if the manager has any block-io protocols it returns them according to the struct. I think it is quite nice, and we don’t have to use UUID or such, but just having proper interfaces for different kinds of devices instead of the fixed open, close, read, write, select…
I might be outdated on things here, I have not worked on drivers in ages.