[GSOC 2024] Proposal: Improving the userland debugging experience

Ok, I got a build of MiniBrowser that you can use to test gdb with

https://0x0.st/XWcI.tar.xz

5 Likes

New stability fixes pushed to gdb-15-haiku at b74dc3ba. Biggest change is GDB is now able to attach to and debug itself. Random hangs should also be much rarer.

An updated recipe will come soon. It would be nice if the version is stress tested, especially with multithreaded programs, before merging the new recipe.

3 Likes

How did you “load the debug symbols”? Is it equivalent to executing the GDB command file /path/to/executable?

Theoretically, the Haiku-specific code should not be related to symbol loading at all.
If I set debug haiku-nat on then run file /path/to/my/big/fat/150MB/executable/with/full/debugging/symbols/of/gdb, absolutely no line is logged. Everything should be read from the disk and processed in the address space of gdb.

This might be related to Haiku’s debugging API’s limitation, which only allows reading 1024 bytes of target process memory at a time. But, like you said, it’s more likely to be a symbol issue.

I tried loading the symbols for the largest library I can find in your build, and it is not that slow…

2 Likes

I guess? I just ran gdb -p $PID_OF_WEBPROCESS and GDB loaded them.

Yeah, when I ran file ./lib/libWebKitLegacy.so.1.9.11, it was similarly fast. It loaded the symbols in less than a minute.


Some updates:

  • Using dwarf-5 didn’t help. It’s still slow at loading symbols. Time to go back to dwarf-4, since at least I can use Debugger by stripping debug symbols from the libraries I don’t care about for the problem at hand.
  • I let gdb try to set a breakpoint overnight. It didn’t finish in that time

Good luck! :slight_smile:

Did anybody tried to setup GDB instead of Debugger as generic program crash handler? Is it even possible?

GDB was used there before Debugger was developped. It should be no problem to switch back if you want to. However, that would use consoled in all cases (similar to Debugger when app_server crashes). A bit more work would be needed to start gdb in a Terminal instead, but there is no dark magic involved: the debug server just passes the process id to attach to on the command line.

The sad thing is, I cannot even launch your test build to try to attach to it:

~/Desktop/work/gdb-haiku-build> LIBRARY_PATH=$LIBRARY_PATH:/boot/home/Desktop/work/MiniBrowser/lib /boot/home/Desktop/work/MiniBrowser/bin/MiniBrowser
runtime_loader: /boot/home/Desktop/work/MiniBrowser/lib/libWebKit.so.1.9.11: Could not resolve symbol '_ZTVN7Nicosia12BackingStoreE'
resolve symbol "_ZTVN7Nicosia12BackingStoreE" returned: -2147478780
runtime_loader: /boot/home/Desktop/work/MiniBrowser/lib/libWebKit.so.1.9.11: Troubles relocating: Symbol not found
~/Desktop/work/gdb-haiku-build>
~/Desktop/work/gdb-haiku-build> echo "_ZTVN7Nicosia12BackingStoreE" | c++filt
vtable for Nicosia::BackingStore
~/Desktop/work/gdb-haiku-build>

Good news, I can reproduce that error :slight_smile: . You need to be in the directory that MiniBrowser was extracted to such that ./bin/MiniBrowser points to the executable. Why? The program and its shared libraries will search in ./lib for the libraries (hee, hee, didn’t spend a lot of time packaging it properly, can you tell?). As to why LIBRARY_PATH doesn’t work, I don’t know. Neither does LD_PRELOAD.

That’s not how this is supposed to work. The programs should not search in ./lib, but in a lib folder next to the executable itself.

Unless the libraries are open with dlopen instead of just being linked, or maybe they are linked with a relative path stored in the ELF executable? readelf on the binary would allow to check that. That would explain why LIBRARY_PATH is not working, if the ELF says it looks for a library named ./lib/libWebKit.so instead of libWebKit.so, you need to put the parent of the lib folder in the LIBRARY_PATH, so the path could be resolved relative to that. Or, a better option is to fix your linker or linker command line so this doesn’t happen, and the binary is linked properly by the library SONAME and not by path.

Ok, so a lot of that is just from some trickery I did to package that. I modified the ELF’s library runpath (using patchelf --set-rpath) from /path/to/build/directory/lib:/boot/system/lib: to ./lib:/boot/system/lib.

Here’s an excerpt from readelf -d, so that we’re all on the same page:

Dynamic section at offset 0x60e8 contains 61 entries:
  Tag        Type                         Name/Value
 0x000000000000001d (RUNPATH)            Library runpath: [./lib:/boot/system/lib:]
 0x0000000000000001 (NEEDED)             Shared library: [libbe.so]
 0x0000000000000001 (NEEDED)             Shared library: [libWebKit.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libWebKitLegacy.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libJavaScriptCore.so.18]

It would be better to remove RUNPATH entirely, and let the runtime loader use the usual paths, I think?

1 Like

Well, the executables in the bin folder will need to find the libraries in the lib folder somehow. I don’t think runtime loader could do that without help?

Confirmed that ./bin/MiniBrowser works. But the browser can’t display my GitHub profile :eyes:

I’ve been debugging this with @waddlesplash.

When attaching to WebKit, GDB spends a lot of time demangling C++ symbols on multiple threads (through the use of std::future). Demangling involves lots of dynamic memory allocation, which, when done on multiple threads, causes heap mutex contention. The threads spend most of the time on _kern_mutex_lock and _kern_mutex_unblock.

A possible hack is to set demangle-style none on GDB before attaching, then use c++filt to lookup your output.

If you don’t have a RUNPATH in the ELF file, the default LIBRARY_PATH environment variable will apply. This contains “%A/lib” which tells runtime_loader to try a “lib” directory next to the binary.

Ah, looks like the OpenSSL mismatch bug. I fixed it in a later version. The only other question is whether the bug that affects discuss.haiku-os.org also affects github.com. I’m still working on a fix to that one.

Hmm, I’ll need to try that.

Ok, good to know.

4 Likes

I managed to use GDB with QEMU to debug boot loader on riscv64. But it is compiled for linux-riscv64 target. Haiku GDB version do not recognize riscv64 target.

7 Likes

You would need to add a few lines of build configuration and a riscv64-haiku-tdep.c file (and a riscv64-haiku-low.cc file if you want gdbserver).

The tough ones (haiku-nat) can (theoretically) be shared among all archs.

2 Likes

We have GDB 15.1 now which (should) supports DAP and can be used to debug C/C++/Rust or other native languages. Are there any initiative to implement DAP in Genio?