Out of curiousity, what about Beam is needed above the normal mail system?
For me a proper inbox view is one thing, but the seperate Composer window is something I do like.
(currently the “just use a tracker querry” in mail works kind of badly, no button to compose a new email, no thread support, no way to see my imap folders or tags. etc)
Well … maybe I just have never given it a real chance, but my personal view is that the “normal mail system” should be put somewhere where people won’t find it.
In case this was based on outdated impressions, I configured it up and gave it a try.
Nay, it will not do. What I have now is a mail directory, with 7 subdirectories. All empty except the first, which has my email address for a name and one INBOX folder within. Ah ha! this must be my mail, let us then see what’s in INBOX. Hm, another 6 folders, mostly empty except Sent and Trash which have stuff from a year or two ago when I must have been using Thunderbird or something. I feel like this is a positive sign, since these directories were updated this afternoon, but ideally there really should be current mail here? In fact I got a popup alerting me to 6 new mails, but …?
Well, maybe there’s a problem. Let’s ask the email client, which would be … a daemon running in the background. I can see it, but that’s all. Log? None that I see laying around. Good grief. Can I control the activity of this email fetcher in any way, other than just kill it?
OK, it isn’t nothing, I do have some old sent mail, so let’s have a quick look at it. Ah, that’s something - displays the text part, with links to the attachments. And when I select the HTML alternative part - it blasts this WebPositive window I have open on discuss.haiku-os.org. That’s not really very nice, though maybe not entirely mail’s fault.
That’s one half hour wasted on a cool idea that never quite worked out. I mean that - it is a cool idea, and I think the principal author of much of the IMAP protocol would think so too, inasmuch as it’s somewhat parallel to an idea I heard him expound on several times, an IMAP filesystem. (Which this isn’t, but close, given that caching is a common feature of IMAP clients.) But his idea would have been a headache for nothing, and frankly I think the normal mail system is the same.
A network service client is just a whole lot more transparent implemented as an application you can see running in front of you. It’s more flexible (maybe I don’t want to cache my mail.) Some of the problems I just ran into are presumably just minor implementation faults, but the basic premise is what I object to.
It’s probably because I’ve been doing my mail with mail_daemon for 25 years now (with a hiatus between the demise of Be Inc. and Haiku becoming usable…).
I’m just very accustomed to Haiku’s Mail app. Plus I still use email like POP3 (no tags, folders, whatever) even though its over IMAP.
I just have a query open for unread mail. And some more query templates in the Deskbar replicant menu for “Same subject”, “Same sender”, “Last 2 days”.
Emails have a “Thread” attribute that you can sort by (plus secondary by “When”).
Type-ahead filtering is also super useful.
Custom status attributes can be useful as well, like “Important”, “Later”, “Userguide” (when a change impacts the documentation) etc.
A new mail is initiated either by clicking “New” if you have an Email open, or from the Deskbar replicant menu.
Again, I know it’s probably a “you like what you know” situation, but I’d just suggest for people to give it a try. Maybe they come to like it, too. 
I get that, but I still think a “bare” querry is not that nice of a UI. Something that does use the querry is needed. But with a nicer UI of showing the thread relations when selecting an email for example, and also a button for a new email in that folder. etc. Lots of things about discoverability really. : D
@donn
Problem #1, beeing that there is no logs is a problem of Haiku in general. The logging system is deficient.
It does actually log, to stdout… not that nice for users indeed. But users never need this ammount of output either.
(This logs the complete IMap session)
Second thing, html in WebPositive overwriting a tab. That is a bug in WebPositive. But also the mail currently just not supporting html email. I’ll fix that with webkit2 at some point
I agree there is a big ammount of improvement to it that arepossible. But I don’t see how it is “failed” in any way. I’m regularily using it anyhow. as for the storage on the disk, beam does it the same way? as should any sane email client. Storing emails in one single file is just insane.
The difference here is that the emails that are unread are supposed to be querried for example, there is no need that such emails are in a special “inbox” folder, they can be in any imap folder and will be found. Currently this is just a bare querry in tracker with bad UI.
What does that mean? Where is mail_daemon’s output? IMAP mail can encounter problems of many kinds. If users don’t need any output from mail_daemon, it’s because they didn’t really need mail anyway, if something goes wrong.
If you mean every sane IMAP client caches the complete content of every email you receive - I think that isn’t true at all. (“One single file” came out of your head, not mine.)
What does that mean? Do you think email_daemon fetched a thousand or so emails from INBOX, and put them somewhere I can’t find them? They can be in any folder, but in fact they aren’t in any folder, because they were never fetched. If you think I’m wrong, tell me where they went.
(Or do you mean that mail_daemon is waiting for me to issue that query, whereupon it will fetch the request emails? I presume not - even if this ability seemed plausible, I sure never asked for those Trash emails.)
mail_daemon apparently failed somewhere. It may have encountered an error, or it may have just gotten stuck. If it were a normal client application, I’d see that right in front of my eyes, but a daemon doesn’t have that connection to the user. That’s why nowhere else is there a mail client implemented in this manner.
To be clear, I’m sure there’s something to be said for the use of file attributes here. I’ve actually started archiving my mail that way, and no doubt there’s a lot you can do with it. My only real objection is the daemon client. Maybe a normal application could present the queries in a more obvious UI. It wouldn’t have to cache everything, it wouldn’t have to run all the time, etc. Far more flexible, far more transparent.
I think you are misunderstanding something. The email daemon has to run to receive emails. (even if you use imap IDLE).
Saving emails to disk is not caching, it is downloading, If they are not downloaded in your case you encountered a bug. (And there is a setting wether to download the email completely or partially, which you can set)
The mail daemon has an error log that will pop up for errors users can handle. The debug log for IMAP will never pop up, and it shouldn’t.
Shoving this darmon into a thread of a mail app would not solve any problems.
One single file is how older mail servers on linux and freebsd save emails to disk. You complained about the folders, hence my comment on that. The folders are correct and should correspond to your IMAP folders if everything is working correcttly.
Anyhow, none of this answered my question of what UI of Beam makes sense to implememt for Mail.
If you have further bugs with Mail please open a ticket on the bug tracker.
On the contrary, it’s precisely what Mail needs from Beam - an IMAP client application.
It doesn’t need this for the 4 or 5 people for whom the mail_daemon actually works. It needs it so that Haiku will have an email application that can reasonably be presented to new users, whose IMAP problems will be all over the place. When their mail doesn’t work at first, they have a non-working application they can look at and describe to others. There’s no good reason it has to run in the background.
I am not using the mail daemon because it would download something like 15 years worth of emails from my mailbox, that I just don’t need, and create thousands of files. This is just as insane, if not more, as putting them all into a single file.
Eventually I want to try the idea of an imap filesystem. But then I would certainly want an actual client in front of that, and not try to fit it into tracker.
In any case, this isn’t really a problem: all these things can (in theory) be achieved, and can be worked on separately, as long as they all agree on the email file format.
There are certainly a lot of bugs to fix and lots of improvements to make all around the stack. I tend to agree with donn that this may be easier to achieve by replacing all of it
To be clear, I’m just saying, turn mail_daemon into a normal application that does the same thing. Which might be tantamount to replacing all of it, don’t know.
I totally agree that whatever the merits of caching some mail, I’d insist on the ability to examine messages in some folder without downloading all of it. So for me that would be a complicating requirement for mail_daemon.app, but that’s for me and may not be a giant priority for Josephine Average User.
That still makes no sense to me. It is a service now so IMAP can be run whenerver needed. Why would I turn it into an application that cannot retrieve email without having a gui open? What do you mean by “does not have to run in the background”? are you proposing to turn this into something that runs in the window thread?
The design now is fine… This component should remain where it is now, as a service providing imap. And application frontend will have to deal with both usecases, IMAP and POP.
Now, I ask again, what UI of Beam would be usefull for an application to view IMAP folders, offline, online or otherwise?
And Im highlighting that UI part again, because that is what I am asking about, and that is the only part I am interested in implementing at this time.
(and no I will not turn mail_daemon into a thread of this application)
It does not. It downloads what you tell it to. You can ask it to download some ammount of mails from specific imap folders, and it can be told to download email partially.
There is nothing standing in your way to have an application request some view of the IMAP mailbox(es) from mail_daemon. I would argue that that is the main purpose such a daemon serves.
I don’t see the need for an IMAP filesystem, bfs seems to already support everything we need on that front.
I agree that the tracker “it’s just a querry” UI is ill-suited. But that does not mean we should trash the other working components.
Would be great if you created a tutorial on how to manage your email the Haiku way! 
I am very scared to use BFs with a copy of my email. Even worse if it was configured to do partial downloads. Why? Because that is confusing and a risk of losing data. It is never clear if an operation on the local filesystem will be properly translated to the online one. Was the mail daemon connected to the server when you did the operation? Did it pick the filesystem notification? Did it have time to synchronize it with the server before it was disconnected or the machine was shut down? If I move a mail from my inbox to another directory, will it be removed from the IMAP server as well?
And in the other direction: if I delete mails from my inbox, will it delete the copy on my hard disk as well, or can I trust that as a backup? If I mark emails as read, will it propagate?
This asynchronous synchronization happening in the background can’t be reliable. There is no indication that the synchronization is off (I would want something that put the inbox readonly or at least clearly marks that the changes are not synchronized). You wouldn’t want to work that way for other networked filesystems like NFS or SMB, and that’s how I think about my email inbox: a shared network resource that I can access from various computers.
So, if I want to store an email on disk, I’d simply comy it from the mounted IMAP directory to one of my local drives. This happens synchronously (with a Tracker file copy dialog showing progress). Same thing if I decide to delete emails, move them around, etc.
The filesystem may then decide to optimize a few things with a local disk cache. But that tricky part becomes a filesystem problem, something that, as a user, I don’t have to care about. With the mail daemon, this is not at all the case. I have to think constantly about the fact that there is a remote IMAP mailbox, and a local copy, and that they are somewhat synchronized, but not really, and that any operation may or may not replicate between the two.
So, yes, independendtly of the UI, I don’t consider the current approach safe enough to trust my email with it. And that is not a UI problem.
But that’s just me and the way I use my email. I understand that other people are happy with it. As humdinger says, if you think about it in the POP protocol spirit, where you download mails to your machine and then there is no synchronization to the mail server, that approach works great.
However, this is the one nice thing about representing email as single files with attributes: both of these approaches can be implemented. The backend can be different (a server running in the background, or a filesystem), and we could still all use the same email client frontend/UI to manage our emails.
Alright, that makes a bit more sense. Many of these same problems are solved (on other OS) with indicators on the folders/files themselves (like in dropbox) that a file is local/synched etc. But a dedicated filesystem can make sense.
However I don’t think any massive restructuring of the mail_daemon is neccesary for this, it already has most stuff in place. What we want now is basically the frontend application for IMAP folders, have it send a message of “tell me what folders there are”, and it will provide, or ask it about the titles of emails etc. Currently the e-mail preferences must already do this or something similar since it will live retrieve the list of imap folders on the server, I’ve not checked the code yet bbut i don’t see how this could have been implemented without having the groundworks for this already.
For completeness, when I have time (if ever) I plan to experiment with porting imuse: GitHub - rtyler/imuse: Experimental (attempted) FUSE filesystem to represent IMAP accounts
which I think didn’t work because it was attempted in another operating system. But would be a great fit to Haiku’s way of handling mail. We already have the client (could be Beam or the builtin one, or some new GUI), and it would work just fine, and as I want it, with something like that.
But that isn’t incompatible with trying other approaches as well
It’s been part of the User Guide for a long time:
https://www.haiku-os.org/docs/userguide/en/workshop-email.html
Ah, thank you @humdinger !
1+ to humdinger. Well I love the concept and used it for few years on Haiku and before that on BeOS. Until about 2 years ago, when Haiku mail just stopped working for me (reported to bug tracker long ago) - it seems it has an issue, when there are at first installation 40K+ emails in a folder (I have around 1M in several accounts).
There is a catch, no TLS 1.3 or modern ciphers on IMAP seem to work via mail, no certificates / signing / SMIME…
I have my own email client, which does that all but unfortunately only on Win/Mac/Lin - it is in Delphi 
and will not run with Lazarus (which breaks all over in Haiku still)