I was playing around with some code on the weekend and wanted to make use of the new netservices API. Found the headers etc. and added netservices2 to my linker flags, which meant the linker had no issue with the classes defined in said headers, but I was getting linker errors from netservices2 itself, things it depends on aren’t being linked in, but I’m not sure where they are.
I don’t have the error to hand right now, but I tried one other library (netapi maybe?) to no avail.
Related, are there any examples of using the new services floating around?
Thanks - I think I tried netapi but network would make sense given what I was saying. Be handy if the docs flagged such things, guess I’ll look to open some PRs on those maybe!
That REST client looks like a great reference, thanks for the heads up! Can see it has both those in the makefile so will give that a whirl tonight
Just to confirm, linking bnetapi and network did the trick. Unfortunately the code blew up at runtime because I’m trying to use a gopher URL and it’s not supported by the newer stack, so I started to work out how to use the older netservices instead (which includes a gopher request class). But then after shutting down my machine I realised perhaps those efforts would be better spent adding gopher URL support to the new stack instead!
Have you considered libcurl? I think using netservices (either v1 or v2) is a waste of time. There’s a perfectly good library for this, just use it, no need to reinvent APIs and make something that will end up similar to curl, but less complete.
I did wonder about that, just wanted to stick “native” but I think I was expecting more support than there is. I’ve started my own class for kicks (this is really all just an excuse to write some code as much as anything) but I suspect libcurl will be the way forward, especially with the amazing amount of pride Daniel takes in keeping it secure!
While I like the libcurl solid implementation, I don’t like the C boilerplate it impose even for the most simple web request. I use more libcpr, a C++ wrapper above libcurl, for that.
But I do think having some basic C++ classes native to parse a URL and some basic classes to do a quick HTTP[S] request make some sense, even if the implementation is not rock solid as libcurl could be.
I’d apreciate if you could stop this hostile talk everytime libnetservices are mentioned.
Offering curl as an alternative is fine, but going “libnetservices sucks” every single time it is mentioned is a huge waste of time and quite frankly disinformation.
Do you think curl is much better suited and required for “playing around with some code on the weekend”?
I don’t think you’d apreciate either if someone came along and said “Just use Nheko, xmpp is a waste of time” in every forum thread about Renga
We should probably provide a native API for basic web requests, but it probably also makes sense to just have Curl be the backing for that API instead of trying to implement all these complex standards ourselves, same as we do with various other things (like the translators.)
I don’t see why I should be quiet about that. I don’t think it is disinformation. The earlier we admit our failure and deprecate this API and tell people to just use curl (or a wrapper over it if really, curl_easy_init is too complicated for you), the better it is.
curl is the reasonable and easy choice here. The libnetservices API does not solve any problem that would be caused by using curl. It does not integrate easier with BMessages (it pretends to, but in the end it doesn’t work). It isn’t faster. It isn’t even simpler (it spawns a thread for each request, forcing you to handle synchronization issues). Doing an http request with curl is 4 lines of code:
int main(void)
{
CURL *curl = curl_easy_init();
if(curl) {
CURLcode result;
curl_easy_setopt(curl, CURLOPT_URL, "https://example.com");
result = curl_easy_perform(curl);
curl_easy_cleanup(curl);
}
}
Here you go. That’s how you do an HTTP request with curl. I’m sure you have plenty of time left in your weekend after you have this running, right?
I will not even try to figure out the corresponding code with libnetapi where you need to set up a BDataIO to store the result, set up a way to wait for the request thread to complete, etc. I have better things to do with my evening. Hey, even figuring out just which libraries to link required a forum post to call for help. This is not good API design, let’s be honest about it.
I think for my use case (making a gopher request) curl definitely makes more sense since netservices2 has been designed for http/https only, and even the class names (like BHTTPSession) don’t lend themselves to adding another protocol.
Right now I’m creating a local classes that are roughly equivalent of some of those, and maybe I"ll open a PR into the main tree at some point, but it’s mostly because I’m just having fun doing the tinkering as much as anything.
I do think @waddlesplash’s approach makes sense - a thin Haiku-native wrapper over the parts of libcurl that’d most commonly be used, perhaps with messaging etc. so things can be used in a Be/Haiku style way would be good. I does make sense to use curl for the backend as it were, purely down to the fact that it’s such a long running, single minded project.
The netservices API was an attempt to use BMessage to handle HTTP request. My personal conclusion (after rewriting it twice, in addition to other rewrites by other people, I think libnetservices2 is the 4th or 5th attempt at this) is that this is not a good design idea.
The way HTTP works is very synchronous. You send a request with headers. You get reply headers. Then you can start sending the data (if you do a POST or similar). Then you receive the response body. This all comes from ang goes to a socket. If you want to use BMessages, you need to copy all the data from the socket into BMessages, and all the replies from BMessages back into the socket. Which you can’t do from the messaging threead, so you need to spin up a worker thread just for this.
So, what was a very simple operation with cURL, before you even started doing anything interesting, involves spinning up a thread, setting up a way to handle messages and sockets together, possible locking issues.
I think the protocol level is just the wrong level to do this. At that level, the cURL API is as good as it gets. Then, you can build some tooling classes, for example if you need cURL to read and write to BDataIO objects. Wire our JSON parser on top of it if you need to. And only then, at the highest level, you can build a nice API on top of this, where you send a BMessage to make a request, and you receive a BMessage with the entire reply (assuming the reply is small). But this will be much better if it’s application specific to whatever REST API you’re interfacing with. Or, at best, it will be application-level-protocol specific (for things like Gopher that are sufficiently specialized, maybe for RSS as well, for REST and JSON, but probably not at the raw HTTP level).
Because you are using the exact same “tactic” as the “Let’s use WASM” or “ is bloated” stuff. There is a big difference between offering a suggestion and telling people to not bother with our apis.
I did not see you -2ing the implementation of libnetservices2 (which is a preview). So why instead go around on the forum telling people to not bother using it or providing feedback?
And quite frankly curl provides major disadvantages over netservices, webpositive now needlesly deviates from other system components (can no longer just log to the console, our build in tool now works differently etc.), and the “obvious” solution, you have mentioned of “just use curl in the backend” was not implemented by you for libnetservices2, and I don’t think proposed either.
Same argument as “Here, i generated this with CoPilot, why bother doing this on your own?”
Provide feedback on the WIP libnetservices2 then?
I find it frankly baffling that you critizise it for beeing hard to link, while curl would be the exact same thing if you don’t know what to link (and netservices2 would be in the same lib, if it was not deliberately excluded because it is a preview!)
Thius is my feedback, I can repeat it again if you want:
Don’t bother with it. Use curl. You don’t need a wrapper. Trying to make a wrapper will waste your time. I have already explained why, but you chose to quote only the parts of my messages where I don’t explain. Should I quote myself for the technical details? I sure can.
I can continue this, since you have more questions. Let’s keep this as a technical dscussion, please. I don’t understand why you think I’m on a crusade or looking for a fight here. I spent considerable time trying to make netservices work, and the conclusion I reached is that it’s a waste of time because the idea is flawed. I see someone in this most making the same design mistake (that network requests should be handled with BMessages at the lowest level). I try to share my experience and guide them to an implementation that will work. There are no tactics here. If you think my opinion of this software architecture is not well funded, you can ignore it.
However also keep in mind that, in this thread, the question was specifically about gopher requests.
These are completely missing from libnetservices2, and in libnetservices 1 it implicitly converts the result to HTML on the fly, making it unusable for a normal gopher client. I don’t see why I would try to direct people towards that: a choice between an API that doesn’t exist, and one that doesn’t do what they need.
I had provided the same feedback to nielx, on the mailing list where he started the discussion. I think it was the right place for that discussion. Gerrit is a tool for code review, it is good to review intricate details of an API or implementation, looking at specific parts of the code. It is not a good tool to talk about software architecture. Which is why this discussion is now taking place here, in an appropriate medium.
Redirecting debug logs to devconsole (a non-standard component) or to syslog is a single call to curl_easy_setopt: CURLOPT_DEBUGFUNCTION, probably also enabling CURLOPT_VERBOSE.
I don’t even know what “our build in tool now works differently” means, I guess this is about proxy settings? Yes, having system wide proxy settings would be nice, and that justifies making a “wrapper” or at least a standardized API to get the proxy settings. This does not justify wrapping the entire request thing under BMessages and enforcing to use it asynchronously.
With libnetservices on-the-fly logging was achieved with a env var, I atleast did not have to recompile the application to get this. I do like DevConsole but I don’t think it alone is enough for proper logging needs. (Indeed, I think live-attaching to a process for this stuff should be possible…)
(and regarding just compiling this in, In Haikuwebkit we are already struggling to set the proxy settings properly. )
And yes, this kind of system integration is important. As is stuff of “where to fetch my certs? what certs should be valid?” If you want a system trust store we need these things fetched in the same way from the same place. With netservices we did have this, even if we did not have the rest of the infrastructure in place.
Now, going around and saying “you don’t need a wrapper” seems wrong to me because of this. Sure, you can use cURL however you want, but for an opinionated usage, as “where is my trust store? Is a self-signed cert OK?” or a UI for “accept self-signed cert this time only, remember this” etc. all requires infrastucture the netservices should provide and not require every damn application to implement on their own.
Ideally this is then a “thin” wrapper with a simple “The default implementation calls this curl function in this way to verify against the system trust store” with nice convenience functions wrt to this functioanlity. With a “simple” way to overwrite functions if your usecase requires something different.
You did not post this in the topic about netservices 2 requiring feedback, but instead choose to post it again when someone tried to use it…
wrt. to this beeing about gopher, I did not see that, I apologize for that. It was not mentioned in the OP however. Gopher in netservices1 is indeed badly implemented, and not at all in netservices2.
I agree that the forum is apropriate, but not every thread about the netservices, and then tone itself is also important..
Anything that downloads files over HTTPs used to be the same as WebPositive, if some HTTP request fails in one app I can recreate in another application; if I am not completely wrong there used to be a tool to download files from the cli via netservices directly… but I can’t find it so I may be wrong about it.
I’m not sure why you are arguing against this, I did not say netservices (or a wrapper) needs to be designed like this. My point is that we do need a wrapper, something like netservices. And I would much prefer we work on that instead of telling users that using our apis is a waste of time.
Since when? As far as I know it is achieved by recompiling the library in debug mode.
You can live attach debugger and it will capture stderr and stdout. So, it’s indeed possible if that’s how you want to handle it. And it could be made with an app simpler than debugger as well.
I built it quickly for my own (indeed limited) needs.
We also had a bug for several years where certificate validation wouldn’t work at all.
So you’re arguiing for centralized settings for proxy (that wasn’t ever fully implemented by libnetservices), certificate store, and other similar system-wide configurable things. Yes, an API for that makes sense. It doesn’t have to be fully wrapping curl. It probably needs some other layers done first (should all apps access a shared settings file directly? should it use environment variable? should there be a way to ask network_roster or some other system component for it?).
There is no conflict here. The current API is bad, and we make our job easier by telling people to not use it so we can deprecate and eventually remove it. I do not want to be introducing a “netservices3” one and having to keep the existing two older APIs in Haiku forever. So, no apps using the APIs make it easier to introduce a new one.
This doesn’t prevent designing a new API, on the contrary. But I think it will be so thin that it will be “just use cURL”, with a few helper functions to setup a cURL context with the correct proxy and certstore settings. And even that, may just be a simple patch to cURL itself, which would be even better, as even ported apps would then use the system settings.
I find this discussion interesting, although I’m not able to follow all the technicalities…
For what it’s worth, NetWorker is using libnetservices2 and it has been working well for what the application currently does. I haven’t tried any kind of certificate validation, and I didn’t find a good way to implement timeout (currently just a BMessageRunner to cancel the request if needed), though I haven’t fully looked through the API for those, so they may well exist and I just haven’t found them. These items might be easier available using curl ( CURLOPT_TIMEOUT, and CURLOPT_SSL_VERIFYPEER).
NetWorker is not in HaikuDepot (yet), and at this point I’m a bit on the fence if I should keep using libnetservices2 or attempt something different if the API has challenges that could eventually get it removed.
FWIW, I am also using libnetservices2 in EmailViewsNeo (a yet to be released native IMAP email client for Haiku) for remote image fetching/pre-fetching, and it has been working well. One corner case required a fallback: some servers send headers with an empty value, which is valid HTTP, but BHttpSession rejects the reply with an exception.
So, reading this thread was interesting (and a bit puzzling) to me too.
