You have to build the parts of Haiku (no "OS" btw) you're interested in with debugging enabled. For that you have to modify your "UserBuildConfig", see the UserBuildConfig.Readme.
For example, if you're interested in StyledEdit add:
SetConfigVar DEBUG : HAIKU_TOP src apps stylededit : 1 : global ;
Then just build StyledEdit with
jam -q StyledEdit and run the new binary.
If you want to debug 3rd party apps, it depends on their build system. If it uses Haiku's makefile-engine, a simple "DEBUGGER := TRUE" does the trick.