[Citation needed.]
As @X512 mentioned, more drivers can potentially slow down the boot, but once you are booted, how will things be slower? The point of all those devices is to make the system faster or support some features, and you can disable them if you don’t need the hardware, right? The same is true on Linux.
As to why Haiku is generally faster, I quote @PulkoMandy from this old post:
A lot of the “feel” of how fast Haiku is, though, is due to input latency, and on that front, we are ahead of the class purely by not doing as much buffering as other OSes.