Idea: userland virtual machine to allow running binaries on any platform

X512 · May 20, 2021, 6:40pm

It is possible to create userland virtual machine that will run application binaries on any CPU type even if it not matching host system CPU. So it will be possible to run PowerPC BeOS binaries on x86 and RISC-V Haiku for example.

Userland virtual machine means that it emulates only instructions available from user mode and not implements priveleged instructions and internal data structures like page table and x86 segment table. It just use host OS virtual memory and implements syscalls as callbacks. Syscall callback will translate guest syscalls to host syscalls, performing endian and pointer size convertion if needed.

User mode virtual machine can be started instead of runtime_loader and use runtime_loader protocol. Then it will load target architecture runtime_loader and forward arguments from kernel to it. User mode virtual machine will be the only native module in process, all other modules will be inside emulated environment.

Compiling all system libraries will be needed for each emulated architecture and it can be installed as secondary architecure.

Experiments can be done with running RISC-V binaries on x86 Haiku because it is easy to implement on 64 bit system, even syscall translation will be almost not needed. RISC-V emulation engine can be extracted from TinyEMU.

memsom · May 20, 2021, 6:58pm

If you could make this happen, I for one would help out testing PPC stuff. There was a Java implementation for R3 era BeOS that still more or less works on R5. Everything from AA (Advanced Access, the one before Preview Release 1) potentially runs on R5 on PowerPC. Everything from Preview Release 2 pretty much works. It would be neat, and somewhat nostalgic. Also running 32 bit X86 BeOS software stuff on 64bit Haiku would be pretty cool too. And being able to still run PowerPC BeOS on PowerPC Haiku (if that ever happens to become a thing), would be great.

X512 · May 20, 2021, 7:09pm

Probably hardest part will be make Haiku libraries compiling by PPC compiler. This is even more tricky than GCC2. BeOS system libraries can’t be directly used because of incompatible syscalls and app_server protocol.

This is possible by special CPU mode without emulation. Actually it was achieved once (test image, mailing list, patch), but currently not compiling. Original author seems to lost interest on it.

leavengood · May 20, 2021, 11:39pm

I think you should look at Cosmopolitan Libc. I don’t fully understand it but it does some very interesting things to have a single binary which runs on almost any OS and BIOS (though not yet “ported” to Haiku as far as I know but I am sure it is possible.)

This is a bit different than your idea because it is current only x86 but I think the author has mentioned it might also work across instructions sets as well because as far as I know it embeds QEMU in some way. It also has some sort of either libc or syscall abstraction and rewriting. So pretty similar to what you are thinking.

Maybe it is useful to look at.

Edit: Actually it seems this “APE” format already works across architectures by relying on QEMU as I thought: https://old.reddit.com/r/C_Programming/comments/klge7b/cosmopolitan_libc/gh9fvw6/

It is amazing what computers can do when you put some effort into something.

SamuraiCrow · May 21, 2021, 2:13am

I still think that Wasmer would be a better bytecode using WebAssembly format. It compiles ahead of time for static compilation performance on all supported architectures. LLVM has almost finalized its RISC-V architecture support already. When the coders looked into it there were only a few crates of Rust that hadn’t been ported to Haiku yet. Not to mention that WebAssembly is architecture-neutral enough that 32-bit bytecodes can be generated on 64-bit architectures with little more than a flip of a switch! The only hangup is that it doesn’t support big-endianness so no big-endian PPC or 68k.

win8linux · May 21, 2021, 3:17am

Seems a bit too much to use WebAssembly for running Haiku apps compiled for one architecture on another, no? The idea seems to be closer to something similar to Rosetta for macOS or Box86 for Linux.

SamuraiCrow · May 21, 2021, 6:45pm

Ok. I just hope it isn’t a permanent addition to the platform. I prefer ahead-of-time compilation over just-in-time and either of those above interpreted environments. I hope it is not too late to learn Rust and finish what was started.

X512 · May 21, 2021, 6:47pm

It is just compatibility feature, of course running native binaries without emulation is the best. It don’t introduce any overhead if no emulated applications are running.

X512 · March 24, 2022, 10:11pm

I made some progress with running riscv64 binaries on x86_64. I used RVVM code for RISC-V CPU emulator because it was easy to separate CPU emulator from the rest of code, run emulator without memory virtualization and handle traps by callback function.

Userland virtual machine setup initial state (stack, TLS block etc) loads riscv64 runtime_loader into virtual machine, setup arguments structue and run runtime_loader entry point. rumtime_loader is a program that loads/unloads executables and libraries into process address space. Kernel do not load launched user executable directly, instead it load runtime_loader, pass startup arguments (argv, environ) to it and then runtime_loader do process initialization and executable loading. In Linux world such thing is called “interpreter”.

Guest runtime_loader works fine and it load target application. Currently crash in <libroot.so>__init_time because of improperly implemented TLS block and commpage.

> UserlandVM ../env 
delta: 0x3d5d5fd000
INFO: Hart 0x7fba50309070 started
INFO: Hart 0x7fba50309070 trap at 0x3d5d70fb64, cause b, tval 00000000
syscall 193(_kern_create_area)
INFO: Hart 0x7fba50309070 trap at 0x3d5d70fb64, cause b, tval 00000000
syscall 193(_kern_create_area)
INFO: Hart 0x7fba50309070 trap at 0x3d5d70f604, cause b, tval 00000000
syscall 107(_kern_open)
INFO: Hart 0x7fba50309070 trap at 0x3d5d70f5e4, cause b, tval 00000000
syscall 105(_kern_normalize_path)
INFO: Hart 0x7fba50309070 trap at 0x3d5d70f834, cause b, tval 00000000
syscall 142(_kern_read)
INFO: Hart 0x7fba50309070 trap at 0x3d5d70f834, cause b, tval 00000000
syscall 142(_kern_read)
INFO: Hart 0x7fba50309070 trap at 0x3d5d70fc04, cause b, tval 00000000
syscall 203(_kern_reserve_address_range)
INFO: Hart 0x7fba50309070 trap at 0x3d5d70fc24, cause b, tval 00000000
syscall 205(_kern_map_file)
INFO: Hart 0x7fba50309070 trap at 0x3d5d70fc24, cause b, tval 00000000
syscall 205(_kern_map_file)
INFO: Hart 0x7fba50309070 trap at 0x3d5d70f8a4, cause b, tval 00000000
syscall 149(_kern_read_stat)
INFO: Hart 0x7fba50309070 trap at 0x3d5d70f504, cause b, tval 00000000
syscall 91(_kern_register_image)
INFO: Hart 0x7fba50309070 trap at 0x3d5d70f8c4, cause b, tval 00000000
syscall 151(_kern_close)
INFO: Hart 0x7fba50309070 trap at 0x3d5d70f8a4, cause b, tval 00000000
syscall 149(_kern_read_stat)
INFO: Hart 0x7fba50309070 trap at 0x3d5d70f604, cause b, tval 00000000
syscall 107(_kern_open)
INFO: Hart 0x7fba50309070 trap at 0x3d5d70f8a4, cause b, tval 00000000
syscall 149(_kern_read_stat)
INFO: Hart 0x7fba50309070 trap at 0x3d5d70f5e4, cause b, tval 00000000
syscall 105(_kern_normalize_path)
INFO: Hart 0x7fba50309070 trap at 0x3d5d70f834, cause b, tval 00000000
syscall 142(_kern_read)
INFO: Hart 0x7fba50309070 trap at 0x3d5d70f834, cause b, tval 00000000
syscall 142(_kern_read)
INFO: Hart 0x7fba50309070 trap at 0x3d5d70fc04, cause b, tval 00000000
syscall 203(_kern_reserve_address_range)
INFO: Hart 0x7fba50309070 trap at 0x3d5d70fc24, cause b, tval 00000000
syscall 205(_kern_map_file)
INFO: Hart 0x7fba50309070 trap at 0x3d5d70fc24, cause b, tval 00000000
syscall 205(_kern_map_file)
INFO: Hart 0x7fba50309070 trap at 0x3d5d70fb64, cause b, tval 00000000
syscall 193(_kern_create_area)
INFO: Hart 0x7fba50309070 trap at 0x3d5d70f8a4, cause b, tval 00000000
syscall 149(_kern_read_stat)
INFO: Hart 0x7fba50309070 trap at 0x3d5d70f504, cause b, tval 00000000
syscall 91(_kern_register_image)
INFO: Hart 0x7fba50309070 trap at 0x3d5d70f8c4, cause b, tval 00000000
syscall 151(_kern_close)
INFO: Hart 0x7fba50309070 trap at 0x3d5d70f8a4, cause b, tval 00000000
syscall 149(_kern_read_stat)
INFO: Hart 0x7fba50309070 trap at 0x3d5d70f604, cause b, tval 00000000
syscall 107(_kern_open)
INFO: Hart 0x7fba50309070 trap at 0x3d5d70f8a4, cause b, tval 00000000
syscall 149(_kern_read_stat)
INFO: Hart 0x7fba50309070 trap at 0x3d5d70f5e4, cause b, tval 00000000
syscall 105(_kern_normalize_path)
INFO: Hart 0x7fba50309070 trap at 0x3d5d70f834, cause b, tval 00000000
syscall 142(_kern_read)
INFO: Hart 0x7fba50309070 trap at 0x3d5d70f834, cause b, tval 00000000
syscall 142(_kern_read)
INFO: Hart 0x7fba50309070 trap at 0x3d5d70fc04, cause b, tval 00000000
syscall 203(_kern_reserve_address_range)
INFO: Hart 0x7fba50309070 trap at 0x3d5d70fc24, cause b, tval 00000000
syscall 205(_kern_map_file)
INFO: Hart 0x7fba50309070 trap at 0x3d5d70fc24, cause b, tval 00000000
syscall 205(_kern_map_file)
INFO: Hart 0x7fba50309070 trap at 0x3d5d70f834, cause b, tval 00000000
syscall 142(_kern_read)
INFO: Hart 0x7fba50309070 trap at 0x3d5d70f834, cause b, tval 00000000
syscall 142(_kern_read)
INFO: Hart 0x7fba50309070 trap at 0x3d5d70f834, cause b, tval 00000000
syscall 142(_kern_read)
INFO: Hart 0x7fba50309070 trap at 0x3d5d70f8a4, cause b, tval 00000000
syscall 149(_kern_read_stat)
INFO: Hart 0x7fba50309070 trap at 0x3d5d70f504, cause b, tval 00000000
syscall 91(_kern_register_image)
INFO: Hart 0x7fba50309070 trap at 0x3d5d70f8c4, cause b, tval 00000000
syscall 151(_kern_close)
INFO: Hart 0x7fba50309070 trap at 0x3d5d70f524, cause b, tval 00000000
syscall 93(_kern_image_relocated)
INFO: Hart 0x7fba50309070 trap at 0x3d5d70f524, cause b, tval 00000000
syscall 93(_kern_image_relocated)
INFO: Hart 0x7fba50309070 trap at 0x3d5d70f524, cause b, tval 00000000
syscall 93(_kern_image_relocated)
INFO: Hart 0x7fba50309070 trap at 0x3d5d70fbe4, cause b, tval 00000000
syscall 201(_kern_set_area_protection)
INFO: Hart 0x7fba50309070 trap at 0x3d5d70fbe4, cause b, tval 00000000
syscall 201(_kern_set_area_protection)
INFO: Hart 0x7fba50309070 trap at 0x3d5d70fbe4, cause b, tval 00000000
syscall 201(_kern_set_area_protection)
INFO: Hart 0x7fba50309070 trap at 0x10293d308b4, cause b, tval 00000000
syscall 241(_kern_get_system_info)
INFO: Hart 0x7fba50309070 trap at 0x10293d2f848, cause 5, tval 0x000010
[!] unhandled trap

Crash here (guest libroot.so):

000000000003c842 <__init_time>:
   3c842: 41 11        	addi	sp, sp, -16
   3c844: 22 e4        	sd	s0, 8(sp)
   3c846: 00 08        	addi	s0, sp, 16
   3c848: 1c 69        	ld	a5, 16(a0) // <--

waddlesplash · March 25, 2022, 1:54am

Hmm. How much work would it be to implement, say, a 32-to-64 bit compatibility layer on top of this, so we could run x86-32bit applications without kernel support on x86_64? Or – implement Linux syscalls, for instance?

X512 · March 25, 2022, 3:36am

Running 32 bit code on 64 bit OS will need translation of structures passed in syscall arguments because of different pointer size, types depend on bitness, maybe different alignment. For riscv64 → x86_64 no translation is needed (but some syscalls like thread creation and signal handling are need to be hooked), I just reused kernel syscall dispatcher code generator, but with _kern* functions instead of _user*.

While it is possible to run 32 bit x86 in CPU emulator like riscv64, it is much more efficient to use hardware ability to natively run 32 bit code on x86_64 CPU. It will need some special handling like _kern* functions thunking in guest libroot.so. Syscall/interrupt instruction hooking is also possible, but it will be less efficient ant it will need special kernel interaction.

Yes, it is possible to implement non-Haiku syscalls like Linux by using different syscall dispatcher in virtual machine trap handler callback. libriscv project actually do that, it implements some Linux syscall subset in trap handler with regular POSIX API. First I planned to use libriscv library for CPU emulation, but it is too hardcoded to ELF and Linux syscalls so it will need many changes for Haiku applications userland virtual machine. RVVM can be used with almost no changes.

X512 · March 25, 2022, 9:25pm

GUI multithreaded applications are working:

X512 · March 26, 2022, 12:14pm

More applications running:

Source code: GitHub - X547/UserlandVM.

SCollins · March 27, 2022, 1:03am

Whay exactly am I looking at ? I’m a bit lost

Munchausen · March 27, 2022, 2:19am

@SCollins he’s seamlessly running applications built for RISC V on x86_64!

X512 · March 27, 2022, 3:06am

In theory this approach allow to run Haiku application built for one CPU architecture on any other, not only x86_64 and riscv64. riscv64 was selected first because it is simple to implement and no kernel changes are needed. For running x86 applications on x86_64 natively without software emulation, some kernel changes are needed (exposing 32 bit code segment to userland, proper signal handling in 32 bit mode).

X512 · March 30, 2022, 12:05pm

I made some basic abstraction of CPU emulation API and added alternative RISC-V emulator based on TinyEMU code.

Complex GUI application like WonderBrush are running. I am currently not sure what is needed for proper fork() call operation.

PixelLoop · March 30, 2022, 5:31pm

That’s really cool!

Would this design hypothetically also allow us to, say, run x86 Windows binaries via Wine on the RISC-V and ARM/ARM64 ports at some point?

x68k · March 31, 2022, 6:58pm

I’m curious about this as well. Also BeOS applications on RISC-V.

loll · April 3, 2022, 4:39pm

Very nice !
What about performance of RiscV app running in userlandVM on x86_64 Haiku machine ? Any bench test ?