Check that your drive is USB 3.0 AND that you’re plugging it in a XHCI port.
There are also few things that you can do to reduce memory use. They seem obvious but somehow people tends to forget about it.
Keep the packages count lower as possible. Uninstall what you don’t use, etc.
Clean up states folder once for a while.
Avoid to use dozens of workspaces, usually two are enough to start and you can always add more later.
Don’t set for each workspace a 4k wallpaper just because your screen can handle this resolution. A wallpapper about 5 Mo size in jpeg is more than 30Mo loaded in memory. Most of time the wallpaper is hidden by your running apps, so better reduce the size and use stretching in Background app.
Avoid as much as possible, huge apps like LibreOffice. They have tons of dependencies and as result many packages loaded in memory. If an Haiku apps is available for the same job, it will use less memory.