My progress on WebKit2 port

SCollins · December 12, 2021, 3:26am

I’ve got a few older amd systems, I’ll dig through my pile and see if i have a complete pc i cam donate

amonpaike · December 12, 2021, 4:50pm

A few years ago, I had sympathy for Google for their openness to open source, but for how they have recently become, for too much invasiveness, and for the totalitarian surveillance policies that suffocate individuals, I am almost glad that chrome/ium does not set foot on HAIKU.

PulkoMandy · December 12, 2021, 7:34pm

Today I investigated the crashes. The first one I had was really easy to fix: I had enabled fullscreen support as a test but the code is not written yet, and returns an uninitialized pointer, resulting in a crash.

After fixing this… I hit another crash

This one is a bit more complicated, it happens in the setup of communications between the different processes used. Currently the code tries to use BLooper and BMessage for communication between the processes. For this to work, creating a connection requires knowing the PID (team_id) of the process to communicate with. But the code for inter process communication in webkit does not work like this. Instead it expects to create a “connection” (for example in linux this is a posix socketpair) and then send a handle to this connection to the other process.

I am trying to find a way to make things work without changing too large parts of the cross-platform code (as long as our code is not upstreamed, it is easier to try to fit the existing architecture). One option is to stop using BMessage for these communications, and use posix socketpairs like Linux. But it creates some extra complexity: at least the process with the UI will still need a normal BApplication loop, and it’s currently not easy to handle both that and the posix sockets in the same thread.

leavengood · December 12, 2021, 8:39pm

Could you send the port_id of the main BApplication to the other process to establish communication? How is the socketpair communicated on other platforms, as part of the command-line when launching the other process? Could that be done with a port_id too?

Edit: Well might as well just send the team_id in that case, though it must not be this simple or you would have just done this. But it feels like something has to be used to pass the communication channel.

leavengood · December 12, 2021, 9:04pm

In my old experiment for multi-process rendering I passed the team_id of the main app in the BMessage used when launching the rendering app.

github.com

leavengood/Haiku-Browser/blob/master/Tranquility.cpp#L56

    
      
          }
          
          

          
void
          Tranquility::ReadyToRun()
          {
          	fBrowserWindow = new BrowserWindow();
          
          
	be_roster->StartWatching(be_app_messenger, B_REQUEST_QUIT);
          
          
	BMessage startMsg(kMsgStartRenderApp);
          	startMsg.AddRect("renderFrame", fBrowserWindow->Bounds());
          	startMsg.AddInt32("teamID", be_app->Team());
          
          
	team_id renderTeam;
          	if (be_roster->Launch(kRenderAppSignature, &startMsg, &renderTeam) != B_OK) {
          		BAlert* alert = new BAlert("Error!",
          			"There was an error trying to launch the render process!", "Damn!");
          		alert->Go();
          	} else {
          		fMessenger = new BMessenger(kRenderAppSignature, renderTeam);

github.com

leavengood/Haiku-Browser/blob/master/renderboy/RenderBoy.cpp#L82

    
      
          		case kMsgStartRenderApp:
          		{
          			BRect renderFrame;
          			if (message->FindRect("renderFrame", &renderFrame) == B_OK) {
          				char str[100];
          				sprintf(str, "RenderBoy: got a renderFrame (%.0f, %.0f, %.0f, %.0f)!",
          					renderFrame.left, renderFrame.top, renderFrame.right, renderFrame.bottom);
          				syslog(LOG_DEBUG, str);
          				fRenderFrame = renderFrame;
          				team_id browserTeam;
          				if (message->FindInt32("teamID", &browserTeam) == B_OK) {
          					sprintf(str, "RenderBoy: got a browser team ID of %d", browserTeam);
          					syslog(LOG_DEBUG, str);
          					status_t error;
          					fMessenger = new BMessenger(kBrowserAppSignature, browserTeam, &error);
          					if (error == B_OK)
          						// Only set this if everything needed was received correctly
          						fStartMsgReceived = true;
          					else {
          						sprintf(str, 
          							"RenderBoy: could not create a valid BMessenger to the browser, error was %d!", error);

PulkoMandy · December 13, 2021, 7:26am

For the main application it works fine already (in one direction by passing the process id as an argument, in the other by sending it in a message after starting). Now my problem is creating the connection between WebProcess and NetworkProcess.

What webkit tries to do is:

main process creates both of these
it sends a message to NetworkProcess, telling it to create a new connection channel
networkprocess replies with an identifier for the channel (in our case this contains networkprocess pid)
main process forwards this identifier to webprocess

In this scenario, NetworkProcess never gets to know the webprocess id so it cannot initiate the communication with it. I think the simplest way to handle this is to modify the message sent from the main Process to include the process id.

It used to be simpler to do this in older versions of webkit because more of the code was platform-specific, but now they have factorized some parts of it which means a little less flexibility (or modifying cross platform code as needed)

steampunklolcat · December 13, 2021, 2:44pm

I would be extremely happy if I could have vertical tabs and firefox sync in WebPositive.

nephele · December 13, 2021, 4:12pm

It seems there is no ticket for this yet, maybe open one at https://dev.haiku-os.org so this doesn’t get lost.

Here is the ticket for firefox sync, feel free to upvote it: https://dev.haiku-os.org/ticket/16589

steampunklolcat · December 13, 2021, 5:12pm

I’ve upvoted the ff-sync one and created a ticket for the vertical tabs.

https://dev.haiku-os.org/ticket/17471

bronzie94 · December 20, 2021, 2:58am

I have to congratulate the involved developers for their progress with Web+ over the last 6 months. Stability, performance and compatibility have moved foward in leaps and bounds - I have gone from mostly using other browsers to mostly using Web+. I think this progress indicates the Haiku dev community’s capacity to achieve their goals - a 1st-class, robust, feature-rich Haiku-first, Haiku-native browser is possible. We have made it a lot of the way there. Getting there will make Haiku much more compelling for everyday users.

PulkoMandy · March 25, 2023, 11:27am

Oh wow, did I not touch this for over a year?

Anyway, I have rebased the haiku-webkit2 branch over the current version of WebKit. It is still as broken as it was last year, that is, it compiles, but the connection between the different processes does not establish.

However, the factorized code I was talking about earlier has now been made a bit more flexible and I think it should be possible to use it now. We’ll have to find a way to pass a BMessenger to another app through a command-line argument, but if nothing else works, we can always serialize the BMessenger into a BMessage, flatten that, and then convert it to base64? That’s the worst case solution, there are probably less stupid ways to do this.

But first, I have two more preparation tasks to do:

I have switched to a 64bit install now, which means I can’t use the old BeDC “developer console” from BeOS for debugging. I have to write a replacement for it that will run on 64bit systems.
The haiku-webkit2 branch has more than a hundred small commits with build fixes I had to make accross multiple git rebase steps, I want to clean that up and extract fixes that can already be merged into the main Haiku branch. I hope to have all these things merged in the main branch but I want to make sure they don’t break the currently in use WebKitLegacy, and also it’s convenient to keep the “work in progress” parts at the top of the branch for future reference.

dsizzle · March 25, 2023, 2:50pm

Can you serialize the BMessenger and store it in an Area, and then pass the area id on the cmdline?

PulkoMandy · April 16, 2023, 7:25am

I rewrote a new implementation of it, named DevConsole, you can get it here for now: devconsole - Gitiles

And I wrote a blogpost about it on my personal blog.

I am now at under 80 commits. Some things got squashed, some things got removed because they were wrong, and some were merged in the main branch already.

To get there, unfortunately I ran into some issues with gitk, which is ported to Haiku currently using undroidwish, a Tk implementation over SDL. Waddlesplash wasn’t able to get Tk to work completely with Xlibe. This would be OK, except it exposed two regressions in our SDL2 port. I fixed one (the handling of the current directory when starting apps was not correct) and wored around the other (resizing the window crashes, but fortunately undroidwish allows to specify a window size on the command line).

The next steps are continuing a bit with this commit cleanup/squashing effort and integrating the “done” parts in the main branch, so I can more clearly find and study the “work in progress” parts and resume the work on getting the interprocess communication going. Of course by the time I’m done, there will be several more months of work in the main WebKit branch that I will have to integrate as well. And I guess at some point I should start on trying to upstream all these things to WebKit, as well…

BiPolar · April 16, 2023, 7:00pm

Oh yeah! I’ll be using your devconsole a lot, so thanks for that one @PulkoMandy!

Edit: for those wondering how to clone that repo anonymously (couldn’t sign-in with oauth):

git clone https://pulkomandy.tk/gerrit/devconsole

worked for me.

BlueSky · April 16, 2023, 7:30pm

Can this repo be cloned via https as well?

Looks like a really useful app, I´m looking forward to test it. Comes in especially handy when debugging an application that redirects stdout/stderr and reads from there.

EDIT: Just saw that my question has been answered in the post above

PulkoMandy · April 21, 2023, 9:21am

I have done enough cleanup and rebasing for now. A few more things were integrated in the main branch and I’m now at 50-60 commits, most of which I am not sure are correct and still quite experimental or outright broken code. I will continue cleaning up after working on it a bit, since there’s no point in cleaning up code that will eventually be rewritten or deleted.

This week I am reworking the log system so that all logs get sent to BeDC/DevConsole. Previously only the “notImplemented()” function would log there, and all the other logs were disabled, not giving me much info about what was happening.

Now the logs are controlled as intended in WebKit, by default they are completely disabled, you can build a version of WebKit with logs enabled and then from there you can configure which logs you want using an environment variable. I have to adjust the process launching to forward the environment variable to the web and network processes which are started in the background using BRoster::Launch (I think the environment is not forwarded in that case).

After all this preparation work I should be able to start debugging the initialization process and make sure all the processes can talk to each other…

PulkoMandy · April 23, 2023, 9:02pm

This weekend progress:

The cleanup of existing commits continue, I am down to less than 40 commits now which is almost manageable. There are still some things in the commit history that are added and then removed again in later commits, so I will continue cleaning this up. It really helps to have a cleaned up history so that I can see what was changed and is still considered work in progress easily.
The log system with BeDC is working well now
The network and web processes (the background apps that will do the actual work of downloading and rendering web pages) are starting but not doing anything. This isn’t surprising as I had to temporarily delete a bit of the code there, because it was very hacky, and the hacks are not needed anymore in newer WebKit versions (or at least, I hope so). So, I did a bit of investigation to why they are not doing anything (I expected they would at least complain that some methods are not implemented). I found and fixed some issues with the “RunLoop” management. In the Haiku port this is basically a BLooper, but the typical all-in-one creation of a BLooper does not work well, because WebKit wants to spawn a thread by itself, and then use that thread as a run loop. I had already patched BLooper to be used in this way, but there are some special case, in particular, for running in the main thread, we should create a BApplication instead of just a BLooper. So I got that working. Now the WebProcess and NetworkProcess are both BApplications. The next step should be that they receive some messages from the UI process (the web browser) asking them to load and render some page. But they don’t seem to receive anything for now. That’s the next thing to investigate (again, it is not surprising because a lot of the code needed for this was removed or commented out during the rebase of the changes on an up to date version of webkit, and needs to be rebuilt).

X512 · April 23, 2023, 10:16pm

It is possible to tell BLooper to run in existing thread.

PulkoMandy · April 24, 2023, 6:36am

Yes, I know, I added BLooper::Loop() to BLooper to allow it a few years ago because it was needed in WebKit.

PulkoMandy · May 18, 2023, 8:22pm

Finally making some progress!

I spent some time re-designing the code for inter process communication and today I managed to get it running again. The WebProcess and NetworkProcess are started and they start sending messages to and from the UIProcess.

One of the first few messages makes things crash, however. Tomorrow I will clean up all debug printf and puts I had tp add everywhere to follow how things worked.

The code is not as clean and simple as I’d like it to be for now, so I may do another pass of re-architecturing it now that I understand how things work.

Things I have done to get it working:

Create a temporary BHandler on the UIProcess
Create a BMessenger targetting it
Serialize that to an hex string I can pass as a command line argument
Rebuild a BMessenger in the Web or NetworkProcess from that argument
Create a BHandler on the other side
Use the BMessenger to send a message containing a BMessenger targetting the BHandler we just created
On the UI process side: “finalize” the connection which creates yet another BHandler, and delete the initial temporary one (this is the step I’d like to delete)
Send a message back to the other process to tell it where the new handler is
Finally, the connection is established and we can start sending messages back and forth

That’s it for today. Tomorrow I’ll fix the crash and see how far I can get things to run