Considering “stupid” is available in infinite quantities, I would not under estimate those someones…
Does the proposed change mean a poorly configured shell environment could prevent Haiku from booting?
As a test in Haiku, I modified the config/settings/profile to cause a hang when setting a variable. And as expected, opening a Terminal caused it to seem to be hung. Hitting Ctrl-C (a few times) eventually made the prompt appear and all was good. With the proposed approach, I don’t think a Ctrl-C will be available to get Haiku unstuck during boot or a restarting daemon that will just reread the same bad settings.
And thanks for sharing your ideas on possible solutions for Environment Variables Behavior On Haiku. It’s interesting learning how Haiku works and I wish you the best on whatever approach you take.
It turns out launch_daemon already calls the shell to load variables from the Haiku SetupEnvironment shell script, so my idea was not too crazy and in fact is already done. Unfortunately many of us discussing this earlier were not aware of this until recently. I should have done more research before discussing this too much, which is why I have not responded more before your post. I will say in all our defense the various components for this are kind of spread around so it is not obvious what is happening.
The “bug” is simply that some things (like USER, GROUP and various XDG_ prefixed variables) are only created by the profile script which is run by the Terminal and the “quick fix” is indeed to move these to the previously mentioned SetupEnvironment.
But with all that explained, I do think it might make sense to move some of these into the C++ code of launch_daemon as they are unlikely to need to be changed. Using the shell script provides more flexibility and we probably will want to continue to have that so user’s can add their own variables (to $HOME/config/settings/boot/UserSetupEnvironment, see line 51 of the SetupEnvironment script), but it might speed up the boot a bit to do less in the shell script.
To directly address your concern, yes, someone could probably make their boot process be delayed with a bad shell script. I suspect the launch_daemon won’t completely hang though, it forks itself to load the shell script. It might be worth testing this if you want to add your hanging code to that UserSetupEnvironment file I referenced above. I don’t think it will break the boot but it might be good to see what happens. If this totally messes up our Haiku system, don’t worry, there is a safe mode which does not load this file.
Thanks for the info. I modified the UserSetupEnvironment script to hang and Haiku did become unresponsive.
I am running Nightly hrev54979 x86_64 in QEMU emulator version 5.2.0 (openSUSE Tumbleweed) with “-smp cpus=4” and “-m 4G”. When booting with the bad script, Haiku will boot to the blue desktop screen with the “hand” cursor then hangs. Nothing else appears until after a period of time, a dialog window pops up saying the was an error with the application ‘/bin/sh -c . “/system/boot/SetupEnvironment”; export -p’, the options “Terminate”, “Debug”, “Save report”, “Write core file” and a “Oh no!” button. At this point the keyboard and mouse are unresponsive. I don’t know if the lack of input controls is because of QEMU or Haiku itself. Also, I am unable to test with other VMs or on real hardware.
For quite some time a similar issue and I did not fully understand why my PyQt application failed to start when lauched via QuickLaunch [1].
Turns out Python’s ctypes.find_library on Haiku depends on the environment variable LD_LIBRARY_PATH LIBRARY_PATH [2]. Now one could try to update the Python patches to use libbe find_directory instead, but actually the affected code is exactly what is used to find a library like libbe. That probably means the patch should be inside the Python C code to provide a way to access find_directory from Python. But we are also looking at updating the patches for currently 4 different Python versions, and there are currently separate patchsets for 32 and 64 bit.
It really sounds more sensible to me to update launch daemon for at least the basic variables.
I just wonder why Tracker is able to start this, but QuickLaunch fails. Does Tracker do some special handling with environment?
Are you starting QuickLaunch via the Shortcuts preferences? If so, the main problem right now for that is that the code for the Shortcuts preferences which actually launches applications is run from input_server, and that does not inherit the same environment as most things. Therefore the applications it runs also do not inherit the full environment. So when you launch QuickLaunch with Shortcuts, it doesn’t inherit the environment and then neither do the things it launches. At least we think this is what is happening. This is really not necessarily a launch_daemon issue, just an aspect of how Haiku sets up the system.
There is a bit of info in this ticket but I’m leaning toward having the Shortcuts preference input_server add-on send a message to Tracker to launch things and that should resolve these issues. I won’t have time to work on this for a while, but if someone wanted to fix it I would recommend trying the above. The code here already messages Tracker for opening directories, so it might not be hard to have it do the same for programs. The only thing I’m not sure about is the argv command-line arguments and how to ensure those are passed to Tracker when it launches something so it can also send them.
That makes sense, thanks for the details. Indeed the issue happens when launching QuickLaunch via shortcut. So the environment is fine when starting the app via Tracker.
So actually we have multiple levels of this when I get this right? Tracker does provide a basic environment, but inside Terminal additional environment variables are defined (e g. XDG_*). Some ported applications might need those, but it should not affect native applications.
But on a lower level there is even a difference between the basic environment that is available via Tracker and no environment when launched via Shortcuts. And this in some cases even affects core applications (e.g. Terminal starting bash with a different starting directory).
Why should Tracker in particular handle this? Why isn’t it possible to simply use BRoster::Launch? Does that API need to be extended to allow to specify a context in which the apps should be run?
Also, why does input_server gets a different environment in the first place? Isn’t it part of the user session just like app_server and Tracker?
And in any case, a variable like LD_LIBRARY_PATH sounds like it should always be set, no matter what you do. Why and how is it getting not set in this specific case? That will break a lot of things. If it’s missing from input_server environment, how does input_server even manage to load itself? Wouldn’t it fail to find libbe?
According to X512 the input_server is not currently part of the user session but is launched by app_server. Apparently app_server gets a much more minimal environment, and input_server inherits that, and therefore we get the issue with Shortcuts. This appears to be somewhat of a regression from when the launch_daemon was introduced compared to the previous Bootscript.
I don’t know how difficult it would be to have input_server be part of the user session. Thinking about it I don’t think that makes sense because if there was some sort of login screen that would need the input_server. So probably fixing how Shortcuts work is in fact the “correct” fix.
Keep in mind I’m just trying to figure this stuff out, and unfortunately I haven’t had a chance to try to fix it. I was just trying to help since Axel seems too busy to help and apparently no one else is interested in trying to fix this. But I also don’t have much time right now either.
I need to verify whether or not variables defined in the bash profile show up in the launch_daemon environment or not, I am currently unsure. From some of the bugs I’ve seen they may not be set up.
To give a bit of background the environment is just a block of memory in each running program, and when one program launches another the environment of the parent is copied into the child. I think this is handled by the kernel, or maybe it is the runtime_loader. Someone else can chime in, but it doesn’t really matter for this.
But to directly answer your question yes there are currently two ways variables will be undefined when you might want them:
This issue with input_server and Shortcuts.
Things from the shell profile that are defined in Terminal but not Tracker, at least I think so.
For 1 I think we need to adjust show Shortcuts launches things, or find some other fix. For 2 we should try to move any variables needed by applications from the shell profile into the SetupEnvironment script that the launch_daemon runs. You can also define a UserSetupEnvironment if there are particular variables you might need as a user, and maybe in this case that means LD_LIBRARY_PATH: haiku/data/config/boot/UserSetupEnvironment.sample at master · haiku/haiku · GitHub
But even with that fix for now we still have the Shortcuts problem.
Yes, actually it should be LIBRARY_PATH in Python, not sure why LD_LIBRARY_PATH is used there? That is linux specific. In that case, could the fix in Python be as simple as using the correct variable?
Sorry for the confusion, that was my bad, that’s just my Linux wired brain kicking in while typing. Of course it is LIBRARY_PATH, and the problem is LIBRARY_PATH not being set when launching from QuickLaunch. I updated my posting above.
Anyway, back to topic: LIBRARY_PATH really should be set, and I don’t fully understand how this works at all in other cases.
EDIT: And I also just confirmed experimentally what was already suspected: When I launch QuickLaunch from tracker everything works, only from Shortcuts things fail due to missing environment.
I created some crude BeOS app that just displays all environment variables. The result is interesting. The screenshot below shows on the left the app started from Tracker, and on the right the same app started via Shortcuts:
There are exactly two environment variables being available when launched via Shortcuts, SAFEMODE (what’s that?) and LC_TYPE.
SAFEMODE indicates if you are in safe mode (selected from the boot menu). Various system components use it to decide if they should load add-ons, use a default configuration, etc.
So, clearly a large part of the environment is unexpectedly lost. Even if that was the “system” environment from launch_daemon, I expect that most of the variables should still be there? You know, things like PATH, LIBRARY_PATH, ADDON_PATH sound quite essential to running anything.
Ideally the environment would be the same. But at a minimum to my opinion LIBRARY_PATH, ADDON_PATH, HOME, PATH, PWD and all of LC_* should be set.
Also tested what is additionally set when running from bash (on a pretty standard install, I don’t have any custom environment variables). A lot is very terminal specific, such as TERM, HISTFILESIZE and LS_COLORS. But then there are the XDG_* variables to indicate various file locations. I think those don’t need to and probably don’t should be set by default, as they are a Freedesktop thing and Haiku has it’s own way to handle this. But it might be a good idea to have a compatibility mode for ported applications, e.g. by setting an attribute on the application. Haiku could then fill the XDG_* variables for this application with values from find_directory.
Afaik we already use find_directory to set the xdg variables, while haiku native apps do not need these at all they are still much easier to use in ported apps than patching them, i would suspect that a specific compt mode would be more complex than simply having those env vars normally, even if not as pretty.
Would probably be ok as well, I think above had been some concerns that this might give wrong incentives to application developers.
The point is that currently those variables are not set at all (except when running from bash). If an application could just set a flag, e.g. B_XDG_COMPAT or something, and that would trigger the variables being available that would be a solution that can be easily applied to ported applications.
The environment isn’t unexpectedly lost, it just isn’t set up. It doesn’t spring from nothing, clearly whatever is not provided by the kernel or launch_daemon doesn’t exist. I have not tracked it down but SAFEMODE and for some reason LC_TYPE are set up early enough by something that they are in the system launch_daemon and therefore are inherited by input_server and Shortcuts.
As for why anything works I feel like environment variables are a very Unix thing that maybe BeOS and Haiku don’t need quite as badly as Linux and friends. Though it would be interesting to find out the details in this case.
Either way, as I have said many times we now know what the problem is. I feel like maybe @axeld could provide some insight here and suggest what approach should be taken. Either the system launch_daemon also needs to run SetupEnvironment (except it makes less sense to set up USER, GROUP and HOME) or Shortcuts needs to tell Tracker to launch things and not use BRoster. Clearly everything seems to run fine except Shortcuts and things it launches, so I don’t know if it makes sense to re-architect launch_daemon when there is a simpler solution. Shortcuts already delegates opening folders to Tracker so it won’t even complicate the code that much. But I suppose we can argue over the details when I have a change on Gerrit, which won’t be for a few weeks at least as I have a trip coming up.
I don’t understand why you insist on going through Tracker here. Why Tracker? It looks like just picking one random app that is likely to be running. And if, for example, you wanted to kill Tracker for some time, do some experiments, and have a shortcut to bring it back, well, you can’t anymore.
I think it makes more sense to make BRoster::Launch work properly. Currently it inherits the environment of the parent app, and in this case (at least), that is not what we want. So we should fix it. Not just for the specific case of Shortcuts, but in a general way. Maybe it needs to talk to the launch daemon and get the correct environment from there? Or maybe, rather than forwarding launch requests to Tracker, we should forward them to launch_daemon? That makes a bit more sense to me than Tracker, at least.
X512 mentioned it, and when I looked at the Shortcuts input_server add-on it was already messaging Tracker to open directories. Obviously this ties Shortcuts to Tracker but Tracker is already really intertwined in the system. This is a bigger issue but long term if we wanted to make it easier to swap out Tracker we would need some way to override things using some sort of protocol or provider layer, maybe like Android or iOS have.
Tracker not running could be handled when sending the message and it could fallback to BRoster.
I can get behind this. I don’t know enough about launch_daemon yet to know how to specify which launch_daemon to talk to, because this needs to be the user session one, though I think there is some infrastructure already for this. Maybe Axel already had some plans for this, I can recall something he had written about launch_daemon and the roster but I don’t remember the details. One thing to avoid is a weird circular dependency between them. Edit: though in this case you are talking about the BRoster code running in the application, so maybe there are less issues there related to the actual roster and launch_daemon servers. I still think Axel had thoughts about these two…
Anyhow, I get your perspective, we at least understand the issue and I think we can figure out a good solution. I will probably experiment and see what looks good and works.
