I think I can confidently say I’m (probably) the only Mac expert here, as I’ve both owned and worked on Macs since '97 (and still do have a collection of Apples sitting around). The more recent Intel Macs can be quite the pain with certain free software (Haiku not excepted from this), unless you still have the polycarbonate model(s) which could do literally anything, including support Coreboot and Libreboot. One thing to understand is that [Mac] OS X by default works with the GUID partition table (GPT), not MBR.
So, if you tried to create a BFS partition in DriveSetup on Haiku, it most likely would not be recognized at all by the Boot Manager (i.e. this is what appears when you hold Option). Even if it were recognized as “Windows”, Haiku would most likely crash this way. Conversely, if you did a completely clean install, and formatted the whole disk as MBR, then your old partition table (and any partitions assigned to it) are then completely erased, and the Mac will be forced to boot in Legacy mode… and if you’re lucky, Haiku will boot up nicely.
My point (and worry) is that I don’t know which of the two you did, but if it was the first, you may be in luck, assuming you’ve left your partitions alone since the breakage. Your hope in that case would be to boot up with an OS X startup disk (whether this is recovery media or not shouldn’t matter). Open Disk Utility (or diskutil in Terminal) and check your partition map and current partitions. You may be able to delete Haiku (the - button in the Partitions tab if you’re using Disk Utility), repair disk, and use the Startup Disk system preference pane to take back your startup disk and get things working again. If all that fails, “re-bless” the partition with the CoreSystem folder (but I doubt that’d be necessary in this case).
If it was the second scenario, I literally hope you haven’t tried to mess with the disk since this happened, as once your data is overwritten, it’s damaged. This next paragraph is just an idea, but it could work. Try to download and write a Fedora Live USB image, shut down your Mac, and plug it into the USB port. Hold down Option, then either use the arrow keys and return, or the trackpad, to select the startup disk. Once you’ve booted up, connect to the Internet, either with AirPort/wifi (if your card firmware is supported) or with Ethernet and do “sudo -s” then “yum update && yum install parted -y” and wait. Try to run “parted /dev/sda” and type rescue at the prompt. If this does happen to find anything to bring back, answer the prompts to restore your partition table. However, this is a really long hope, as parted usually does this inside itself; I haven’t tested it cold like this and I doubt it’d work, but it’d be worth a try. If the above steps do nothing for you, then reinstall OS X and try using this utility: http://www.recuvaformac.com/ or the like on the Internet and see what you can do. I can only hope you had an external HD, a Time Capsule/Time Machine, iCloud, or anything as at least a partial backup.
I’m sorry about your loss here, but hopefully this should be a hard lesson: ALWAYS back up your data FIRST before any serious changes – this is not just a disclaimer – it’s a step! In any case, I really hope that all this helps you sort this out. If anything, because Haiku is still alpha-level software, why not consider running it inside a VM until it is more stable?
Good luck and regards,