Unfortunately the VESA driver cannot do anything about it. The only thing it can do, is ask your video card BIOS for a list of modes, and then tell the BIOS which mode to use. There is no way to set a custom mode with VESA. Otherwise, we would not have spent any time developing hardware specific drivers and we would just be using the VESA driver everywhere.
The problem here is that most VESA BIOS implementation just return a fixed list of hardcoded video modes. They will not get info from the plugged display to know which mode it supports. And even worse, the list they use is usually restricted to old 4:3 video modes which are not suitable at all for modern hardware.
As Diver mentionned, the Clover bootloader patches the BIOS (by playing tricks with the MMU and looking for something that looks like a modeline in the BIOS code). We could do something like that in Haiku as well, in theory, but no one has attempted it yet.
You may have better luck by using UEFI, as when booting with UEFI, the VESA BIOS is not used, and instead we use UEFI APIs to set up a frame buffer. It is possible (but not guaranteed) that UEFI in your machine allows to set resolutions which it doesn't offer in BIOS/VESA mode.