Getting insight into one’s own email

Standard

Thunderbird knows a lot about your email. After all, it’s got access to large amounts of data, and builds sophisticated databases so that it can be very responsive, even when dealing with large folders containing thousands of messages. Wouldn’t it be nice if we could use this information to extract either faster workflow, or even insight into our email habits?

On Tuesday, as he was slogging through some somewhat antique code in the front-end of Thunderbird, Andrew asked whether Thunderbird’s “start page” was what we wanted it to be, which is Andrew-speak for “this is broken, we should fix it”. We’d actually discussed what to do with that page for a long time, but had focused on other, more infrastructure bits for a while. I decided it was time to dust off those plans, and see what we could do.

First, some background. The “start page”, which makes a lot of sense in Firefox, never made a huge amount of sense to me in Thunderbird. In particular, it’s shown only when a folder is selected, and no message is selected. That’s hardly a logical time to show the (colorful, pretty, but fairly useless) page we show now. Instead, why not show information about the selected folder, and help people who clearly intended to select a folder, so most likely wanted to do something related to that folder!

We could still use a small part of the pane to display interesting news snippets, as a way of keeping users aware of new developments, such as new add-ons that we want to promote, or Mozilla events. In fact, by making that page useful, we’re more likely to get people to read a sentence or two which might change periodically, as opposed to changing a page that is so big, wordy and useless, that I suspect most users are completely blind to it.

A couple of days later, we have an early patch, which feels pretty compelling to me:

  • It summarizes critical data about the folder (name, number of messages, number of unread messages);
  • if you were recently reading that folder, it points out the last message you were reading, so you can quickly find it again;
  • it shows you a few of the messages that are most likely of interest (because they’re unread or starred);
  • and finally it gives some information about the activity in the folder: a histogram showing activity over the last 52 weeks, and most prolific authors and most active threads during that same period.

For example:

Folder summary

The icing on the cake is that if you hover over an author or a thread title, it will highlight when those messages occurred in the histogram (in this case, the “An alternate take on HTML 5″ thread). Readers of m.d.platform won’t be surprised at any of the names that show up above!

There are lots of possible additions: Some of the ones we’ve thought of include:

  • showing how much disk space is used by the folder
  • showing the “largest” messages
  • showing the status of autosync/indexing jobs

Note: this hasn’t been through a visual design phase yet — the look will change, once someone who knows what they’re doing has a chance to make it nicer! There’s also a lot of work to do to make sure that it does the right thing for saved searches, smart folders, etc. And yes, we’ll make it optional, just like the start page is now.

I’ve been playing with it for only a little bit, and it’s fascinating how interesting it is to get some data on one’s email. For example, I more-or-less follow a small-inbox model, where emails don’t stay in my inbox if I can avoid it — they get deleted, or archived. As a result, the sparkline for my inbox is:

inbox sparkline

which isn’t as short as it should be, telling me pretty quickly that I have some old messages that need attention. It’ll be interesting to see what else we learn from these kinds of simple personal analytics.

What other kinds of visualizations, summaries, and analysis would you like to see in Thunderbird, or in add-ons?

[Update]: I liked the idea that a commenter suggested of showing what tags were in use in the folder, so I had some fun, first with a “tagpie”, which proved mostly useless, just like most piecharts; then with a “tagdots” small-multiples visualization:

tagpies, tagdots, sparklines oh my!

I don’t think the tagpie is worth keeping, but the dots are tempting. As an aside, the Protovis library really is as easy to use as I’d hoped.

Getting insight into one's own email

Standard

Thunderbird knows a lot about your email. After all, it’s got access to large amounts of data, and builds sophisticated databases so that it can be very responsive, even when dealing with large folders containing thousands of messages. Wouldn’t it be nice if we could use this information to extract either faster workflow, or even insight into our email habits?

On Tuesday, as he was slogging through some somewhat antique code in the front-end of Thunderbird, Andrew asked whether Thunderbird’s “start page” was what we wanted it to be, which is Andrew-speak for “this is broken, we should fix it”. We’d actually discussed what to do with that page for a long time, but had focused on other, more infrastructure bits for a while. I decided it was time to dust off those plans, and see what we could do.

First, some background. The “start page”, which makes a lot of sense in Firefox, never made a huge amount of sense to me in Thunderbird. In particular, it’s shown only when a folder is selected, and no message is selected. That’s hardly a logical time to show the (colorful, pretty, but fairly useless) page we show now. Instead, why not show information about the selected folder, and help people who clearly intended to select a folder, so most likely wanted to do something related to that folder!

We could still use a small part of the pane to display interesting news snippets, as a way of keeping users aware of new developments, such as new add-ons that we want to promote, or Mozilla events. In fact, by making that page useful, we’re more likely to get people to read a sentence or two which might change periodically, as opposed to changing a page that is so big, wordy and useless, that I suspect most users are completely blind to it.

A couple of days later, we have an early patch, which feels pretty compelling to me:

  • It summarizes critical data about the folder (name, number of messages, number of unread messages);
  • if you were recently reading that folder, it points out the last message you were reading, so you can quickly find it again;
  • it shows you a few of the messages that are most likely of interest (because they’re unread or starred);
  • and finally it gives some information about the activity in the folder: a histogram showing activity over the last 52 weeks, and most prolific authors and most active threads during that same period.

For example:

Folder summary

The icing on the cake is that if you hover over an author or a thread title, it will highlight when those messages occurred in the histogram (in this case, the “An alternate take on HTML 5″ thread). Readers of m.d.platform won’t be surprised at any of the names that show up above!

There are lots of possible additions: Some of the ones we’ve thought of include:

  • showing how much disk space is used by the folder
  • showing the “largest” messages
  • showing the status of autosync/indexing jobs

Note: this hasn’t been through a visual design phase yet — the look will change, once someone who knows what they’re doing has a chance to make it nicer! There’s also a lot of work to do to make sure that it does the right thing for saved searches, smart folders, etc. And yes, we’ll make it optional, just like the start page is now.

I’ve been playing with it for only a little bit, and it’s fascinating how interesting it is to get some data on one’s email. For example, I more-or-less follow a small-inbox model, where emails don’t stay in my inbox if I can avoid it — they get deleted, or archived. As a result, the sparkline for my inbox is:

inbox sparkline

which isn’t as short as it should be, telling me pretty quickly that I have some old messages that need attention. It’ll be interesting to see what else we learn from these kinds of simple personal analytics.

What other kinds of visualizations, summaries, and analysis would you like to see in Thunderbird, or in add-ons?

[Update]: I liked the idea that a commenter suggested of showing what tags were in use in the folder, so I had some fun, first with a “tagpie”, which proved mostly useless, just like most piecharts; then with a “tagdots” small-multiples visualization:

tagpies, tagdots, sparklines oh my!

I don’t think the tagpie is worth keeping, but the dots are tempting. As an aside, the Protovis library really is as easy to use as I’d hoped.

Thunderbird-friendly SOGo reached 1.0

Standard

An open-source project that I pay a bit of attention to is SOGo, aka “scalable OGo”. It’s a project w/ a long history, much of which I don’t know well, but what’s interesting to me is that they’re trying to be the Yin to Thunderbird’s Yang. SOGo includes a web frontend, which is designed to look like Thunderbird, which is a fascinating choice.

The folks at Inverse, who are the primary drivers of SOGo, as far as I know, are also doing Thunderbird & Lightning add-ons (SOGo Connector and SOGo Integrator), including adding support for CardDAV, Free/Busy, etc. They’re also helping with patches to Lightning, for which we’re definitely grateful!

SOGo reached the 1.0 milestone yesterday, which is always worth celebrating. If you’re interested in the groupware space and looking at software that plays well w/ thunderbird, check it out.

Thunderbird 3 beta 2

Standard

On the road to Thunderbird 3, another milestone — this time, Thunderbird 3 beta 2.

illustration

Why do beta releases?

Beta releases are funny things. They serve a few purposes. The first is to make sure that we periodically stabilize the code base, as without periodic ‘cooling’, it’s hard to get a handle on the quality of a piece of software. Betas also serve as deadlines, which are magical motivators for some people. Some of us will spend way too many hours staying up late in the night in order to “make a deadline”.

Betas for open source software are even more odd in that people interested in staying very involved with the project can use nightly builds, which are updated every day. I’ve been using nightly builds of Thunderbird for over a year, as have several thousand other users. As a user of nightlies as much as a project coordinator, by the time the beta is released to a wider audience, all the excitement is historical.

Another fascinating aspect of beta releases is that, because we know there will be another release, and because the purpose of the beta is to get a broader set of testers to shake out edge cases, we try to be conservative about slipping in major features at the last minute, as the odds of those features being polished in time are never what we hope they’ll be. So we routinely delay feature additions until the next cycle, to avoid dragging the beta validation process out. It’s an unpleasant, but unavoidable part of optimizing releases.

If we do our job right there, then by the time the beta ships, the features that have landed are free of major bugs. We of course can’t know that until we get feedback from the beta.

What’s in this release?

The most striking part of the release is the sheer volume of bug fixes. It’s not sexy work, it’s often the hardest work, but it’s very important. This list (of bug fixes and feature work, but mostly bug fixes) is impressive.

Of the features that have landed, I want to talk about two that many users could easily ignore: archiving, and the activity manager.

The archive feature is straightforwardly borrowed from GMail’s archive feature, which we think is great. The idea is that figuring out exactly which folder each message should be filed is a process that can take a lot of time and effort — something that wasn’t a real problem in the early days of email, but which becomes a real time sink with thousands of messages. With a good enough search engine, it’s easier for many users to simply “archive” the message (doesn’t really matter where), get it out of the way, and then rely on the search capability to find the message again.

In this beta, we’re half-way there. The archive feature is there if you want it, but you can also use the standard “file in a folder” method. Thanks to work we did before beta2, the archiving is fast, putting messages in per-month folders at the click of a buttton or a keystroke. The new fast global search hasn’t landed yet, but even our “old” cross-folder search mechanism has gotten a lot better.

I already love the feature — being able to select messages I don’t need to worry about anymore, hit ‘A’ and be done with them, saves me a lot of time and mental effort

The second feature worth highlighting is also not fully deployed, but already useful. The Activity Manager was born out of a recognition that Thunderbird 2 is pretty bad at telling you what it’s doing. It says a lot of things, it says them fairly loudly, but they’re rarely the things you want to know. We’re building infrastructure that will let the various bits of Thunderbird be much more helpful in describing what’s going on (through a log of notable events), what went wrong (non-intrusive but notable alerts), and how it’s progressing at long-running tasks (with more context than just a single progress bar). Teaching software that wasn’t designed with a notification mechanism or philosophy in mind how to be polite and informative is a slow and arduous task, but we’re making good progress. In Thunderbird 3b2, there’s an Activity Manager window, which for now will just report on message moves, copies and deletes, and IMAP auto-syncing. Now that the framework is in place, we should be able to have a lot more informative messages when you need them, and reduce the number of dialog boxes (especially the ones you can’t do anything about!).

One of the fascinating aspects of the activity manager is that it’s giving even those of us who know how the software works on a detailed level a better handle on important global aspects. For example, the activity manager showed me that the autosync function can and should be much more aggressive, so that more of your email is already downloaded before you need it.

Other features you may notice:

  • Much more useful Growl notifications on OS X
  • Keyboard shortcuts for quick tab navigation
  • Better looking forwarded mail
  • Fewer dialog boxes

What’s next?

The next beta release is our last scheduled beta. As such, we’re thinking of it as the last milestone to introduce Big New Features. Furthermore, we’re hoping to be even better behaved this cycle and land features as early in the process as possible. Upcoming features which we hope will be available in a nightly build soon include:

  • the new global search function, leveraging tabs
  • cleaning up the message header area further
  • “pop tarts” to complement the activity manager
  • the beginning of some theming work (prettier icons, etc.)

And then, of course, there will be unplanned bright ideas which show up out of nowhere. Life wouldn’t be fun without those.

Try out the beta, file bugs, send feedback!

PS: the illustration at the top is from a brand spankin’ new website for Mozilla Messaging. We’ve changed the site to make it the primary destination for Thunderbird users, riffing on the look of other Mozilla websites, and yet quite distinct. I find the illustrations in particular a lot of fun, and I’m very proud of the team that built it. Rafael Ebron ran the project with the SpreadThunderbird team, with designs from The Royal Order, and implementation from silver orange. A very nice job, thanks to all who contributed!  The new site also allows us to build localized sites, which will be amazing.

Lightning-in-Thunderbird status update

Standard

For some time, we (the Thunderbird release-drivers) have been exploring how to best integrate calendar functionality into Thunderbird.  Time for an update.

The current plan is to work with the Lightning Add-on community to make a version of it available as an add-on to Thunderbird 3 after we ship later this year.  This is a change from our initial plan of integrating all of Lightning into Thunderbird by default.  Our thinking has evolved based on both technical and product reasons:

  • the calendar team has had a tough time of figuring out how Lightning needed to change to integrate optimally in Thunderbird, in large part because Thunderbird 3 itself has been somewhat of a moving target — Thunderbird’s interface model is still shifting, which is certainly hard on add-on developers
  • partially as a result, there’s a fair amount of work left to do on Lightning before it’s ready to present as part of the core product.  For example, it’s not yet possible to selectively enable the task management or calendaring features; the account configuration is a bit too hard to get right for novice users; error messages are still too cryptic — that sort of thing.
  • looking at Lightning as an addition to the core project, it would represent a lot of new, complex code, with a lot of implied maintenance — we don’t have enough developers on hand to be able to take on that commitment at this stage.  See Philipp’s post on this topic for more information, especially if you’re keen to help out.
  • we try to avoid feature changes in between major releases, and limit minor updates to security fixes.  This would mean that feature changes to Lightning would have to wait for the next major release of Thunderbird.  That would be a real shame.
  • more generally, we recognize that different users need different kinds of calendaring solutions.  Just as there are more and more messaging systems, there is a growing diversity in calendar usage models, such as web calendars, stand alone clients, and calendar and event applications on social networking platforms.

Given all this, we feel the best plan is to take advantage of our add-on ecosystem, to allow a variety of calendaring features to evolve, and to allow Lightning to evolve at whatever pace is best for it, less tied to Thunderbird’s schedule.  Lightning is by far the most popular and important Thunderbird add-on, and we’re going to see what we can do to make it better and more successful, both on the development side, as well as on the promotion side.  Also, thanks to Thunderbird 3′s new add-on manager, it will be easier for users to find and install, which I expect will lead to even greater adoption.

In short: expect a version of Lightning that will work with Thunderbird 3; expect it to have more frequent releases than Thunderbird major release numbers; talk to philipp if you want to help!

FOSDEM 2009 slides

Standard

Just so I don’t forget, here are my slides from FOSDEM. I’m hoping to find some time to write my thoughts about the conference, but that’s not going to happen today.

I had some really good conversations after the talk, so it seemed to at least be interesting to people, which was the main goal.

How should we manage autoconfiguration files?

Standard

One of the features we’re working on for Thunderbird that I’ve mentioned before is the “autoconfiguration” setup, whereby when setting up an account, you type in your email address, and using the domain part (the part after the @ sign), Thunderbird tries to figure out what the right settings are. When it works, this is an amazing feature. This is not a new idea — in fact many mail programs have implemented in the past, including Exchange 2007 and the iPhone’s Mail program.

There are three ways of solving this problem: asking a central server for configuration files per domain, bundling some of those configuration files, and doing local network probing to look for likely domain names (mail.domain.com, imap.domain.com, etc.).

The central repository of configuration files has many advantages if we can figure out how to decentralize the provisioning of these configuration files, while remaining secure. In particular, it allows Thunderbird to “learn” about new domains or changed configurations as soon as those files are vetted, without having to wait for the next version of Thunderbird. Even with only a few configuration files, we should be able to cover the “short, fat, end” of the email domains — the top 200 domains for example probably cover 90% of home email users, which would be an amazing improvement for Thunderbird’s usability. (Gerv has already started a project months ago to collect some of this information)

Local port probing is most useful in corporate environments. Either their small user base or corporate policies may deter them from submitting their configuration files in the central server. In this case, the challenge is to strike the right balance of efficiency vs. completeness. We certainly don’t want to probe hundreds of ports on a local domain in a desperate search for a mail server, but to assume that all mail servers are configured the same way is unrealistic. A hybrid of the Exchange 2007 autodiscover model and the Mail.app configuration hostname-guessing model is likely optimal for this part of the market.

Side note: I could imagine an add-on that would bundle the configuration files and get updated periodically for those users who want to avoid leaking any data.

We welcome input on how we should manage these configuration files. The feature hasn’t landed yet, so it’s still very possible to influence this process. From my perspective there are a few key characteristics that should drive our process:

0) Reusing existing processes when possible makes life easier.

1) User experience is the goal. The point of this feature is to let as many people end up with a well-configured Thunderbird as easily as possible. This is why we’re going after a hybrid approach, trying to cover as many different environments with a single user experience.

2) This problem is not Thunderbird-specific. The data we’re looking for is not about how to configure Thunderbird, but how to configure IMAP/POP/SMTP email clients in general. If Mozilla can help solve the problem of email configuration in a safe, scalable, trusted way, then I’m fine with other email clients using it. I see this as potentially a very powerful public service that Mozilla can provide to email users everywhere.

3) We want to decentralize contributions of configuration files, but control the review processes, to facilitate end-user contributions and cover the long tail of domains, with a consistent policy especially with respect to validation.

In the short term, it feels like we could easily start with a variation on the processes we already use for code, with a security review overlay. In particular, I would suggest as an initial draft that:

– we have something equivalent to a module, which contains the published configuration files and processes around them
– each configuration is reviewed by someone other than the configuration submitter (patch author)
– each configuration needs to be validated by the reviewer against public documents published by the mail provider (ISP, university, etc.)
– Mail administrators can file a bug, or contact the module owner directly if they want to report changes in their email setup or suggested changes.

We will probably want to figure out more sophisticated systems involving cooperation with mail administrators (using MX or DNS records, or signed emails, or …) in a later phase, but I don’t think we should gate being able to serve users of the largest domains (gmail, yahoo, major ISPs, universities). Walk, then run, should be our approach.

Similarly, there are possible futures where users could share configuration files with each other, but I don’t think we’re ready to tackle those securely yet.

We’ll figure out a way for those interested in submitting or reviewing configurations to sign up in a later round. At this point, we’re most keen on process-type feedback. Comment on the bug, or here, thanks!

PS: Whatever process we arrive at, we’ll then apply to the files we’ve been using in testing and the configuration data in Gerv’s spreadsheet, to make sure that all the data has gone through the right hoops.

It’s Friday: Goofy but fascinating Thunderbird Add-ons day

Standard

Two different and equally goofy but interesting add-ons are in my personal news today:

Kent James released ToneQuilla, which I like to call “BiffTones!”, which allows you to set custom notification tones based on Thunderbird rules. Emails from the spouse make one sound, emails from the grandmother make another, etc. Neat!

Andrew Sutherland, on somewhat of a dare that I put in front of him (nothing like waving a red flag of visualization at a canvas bull like him) responded to the pretty but mostly useless Wordle meme which has been going around Mozilla circles, and built a wordle-like visualization of the database-driven queries that I blogged about a couple of days ago. If one can build an add-on to that in a day (well, a night without internet access), what couldn’t one do?

Both of these add-ons have somewhat of a goofy aspect to them, and both could evolve into something really useful. Notification overload is a huge problem in communication clients – it’s useful to know when something important happens, but useless to know when “a message was received” — tools like ToneQuilla can help. Simiarly, visualizations can provide insight into ones’ messaging history. See Themail for interesting research on the topic.

It's Friday: Goofy but fascinating Thunderbird Add-ons day

Standard

Two different and equally goofy but interesting add-ons are in my personal news today:

Kent James released ToneQuilla, which I like to call “BiffTones!”, which allows you to set custom notification tones based on Thunderbird rules. Emails from the spouse make one sound, emails from the grandmother make another, etc. Neat!

Andrew Sutherland, on somewhat of a dare that I put in front of him (nothing like waving a red flag of visualization at a canvas bull like him) responded to the pretty but mostly useless Wordle meme which has been going around Mozilla circles, and built a wordle-like visualization of the database-driven queries that I blogged about a couple of days ago. If one can build an add-on to that in a day (well, a night without internet access), what couldn’t one do?

Both of these add-ons have somewhat of a goofy aspect to them, and both could evolve into something really useful. Notification overload is a huge problem in communication clients – it’s useful to know when something important happens, but useless to know when “a message was received” — tools like ToneQuilla can help. Simiarly, visualizations can provide insight into ones’ messaging history. See Themail for interesting research on the topic.