Thunderbird-friendly SOGo reached 1.0

An open-source project that I pay a bit of attention to is SOGo, aka “scalable OGo”. It’s a project w/ a long history, much of which I don’t know well, but what’s interesting to me is that they’re trying to be the Yin to Thunderbird’s Yang. SOGo includes a web frontend, which is designed to look like Thunderbird, which is a fascinating choice.

The folks at Inverse, who are the primary drivers of SOGo, as far as I know, are also doing Thunderbird & Lightning add-ons (SOGo Connector and SOGo Integrator), including adding support for CardDAV, Free/Busy, etc. They’re also helping with patches to Lightning, for which we’re definitely grateful!

SOGo reached the 1.0 milestone yesterday, which is always worth celebrating. If you’re interested in the groupware space and looking at software that plays well w/ thunderbird, check it out.

Thunderbird 3 beta 2

On the road to Thunderbird 3, another milestone — this time, Thunderbird 3 beta 2.

illustration

Why do beta releases?

Beta releases are funny things. They serve a few purposes. The first is to make sure that we periodically stabilize the code base, as without periodic ‘cooling’, it’s hard to get a handle on the quality of a piece of software. Betas also serve as deadlines, which are magical motivators for some people. Some of us will spend way too many hours staying up late in the night in order to “make a deadline”.

Betas for open source software are even more odd in that people interested in staying very involved with the project can use nightly builds, which are updated every day. I’ve been using nightly builds of Thunderbird for over a year, as have several thousand other users. As a user of nightlies as much as a project coordinator, by the time the beta is released to a wider audience, all the excitement is historical.

Another fascinating aspect of beta releases is that, because we know there will be another release, and because the purpose of the beta is to get a broader set of testers to shake out edge cases, we try to be conservative about slipping in major features at the last minute, as the odds of those features being polished in time are never what we hope they’ll be. So we routinely delay feature additions until the next cycle, to avoid dragging the beta validation process out. It’s an unpleasant, but unavoidable part of optimizing releases.

If we do our job right there, then by the time the beta ships, the features that have landed are free of major bugs. We of course can’t know that until we get feedback from the beta.

What’s in this release?

The most striking part of the release is the sheer volume of bug fixes. It’s not sexy work, it’s often the hardest work, but it’s very important. This list (of bug fixes and feature work, but mostly bug fixes) is impressive.

Of the features that have landed, I want to talk about two that many users could easily ignore: archiving, and the activity manager.

The archive feature is straightforwardly borrowed from GMail’s archive feature, which we think is great. The idea is that figuring out exactly which folder each message should be filed is a process that can take a lot of time and effort — something that wasn’t a real problem in the early days of email, but which becomes a real time sink with thousands of messages. With a good enough search engine, it’s easier for many users to simply “archive” the message (doesn’t really matter where), get it out of the way, and then rely on the search capability to find the message again.

In this beta, we’re half-way there. The archive feature is there if you want it, but you can also use the standard “file in a folder” method. Thanks to work we did before beta2, the archiving is fast, putting messages in per-month folders at the click of a buttton or a keystroke. The new fast global search hasn’t landed yet, but even our “old” cross-folder search mechanism has gotten a lot better.

I already love the feature — being able to select messages I don’t need to worry about anymore, hit ‘A’ and be done with them, saves me a lot of time and mental effort

The second feature worth highlighting is also not fully deployed, but already useful. The Activity Manager was born out of a recognition that Thunderbird 2 is pretty bad at telling you what it’s doing. It says a lot of things, it says them fairly loudly, but they’re rarely the things you want to know. We’re building infrastructure that will let the various bits of Thunderbird be much more helpful in describing what’s going on (through a log of notable events), what went wrong (non-intrusive but notable alerts), and how it’s progressing at long-running tasks (with more context than just a single progress bar). Teaching software that wasn’t designed with a notification mechanism or philosophy in mind how to be polite and informative is a slow and arduous task, but we’re making good progress. In Thunderbird 3b2, there’s an Activity Manager window, which for now will just report on message moves, copies and deletes, and IMAP auto-syncing. Now that the framework is in place, we should be able to have a lot more informative messages when you need them, and reduce the number of dialog boxes (especially the ones you can’t do anything about!).

One of the fascinating aspects of the activity manager is that it’s giving even those of us who know how the software works on a detailed level a better handle on important global aspects. For example, the activity manager showed me that the autosync function can and should be much more aggressive, so that more of your email is already downloaded before you need it.

Other features you may notice:

  • Much more useful Growl notifications on OS X
  • Keyboard shortcuts for quick tab navigation
  • Better looking forwarded mail
  • Fewer dialog boxes

What’s next?

The next beta release is our last scheduled beta. As such, we’re thinking of it as the last milestone to introduce Big New Features. Furthermore, we’re hoping to be even better behaved this cycle and land features as early in the process as possible. Upcoming features which we hope will be available in a nightly build soon include:

  • the new global search function, leveraging tabs
  • cleaning up the message header area further
  • “pop tarts” to complement the activity manager
  • the beginning of some theming work (prettier icons, etc.)

And then, of course, there will be unplanned bright ideas which show up out of nowhere. Life wouldn’t be fun without those.

Try out the beta, file bugs, send feedback!

PS: the illustration at the top is from a brand spankin’ new website for Mozilla Messaging. We’ve changed the site to make it the primary destination for Thunderbird users, riffing on the look of other Mozilla websites, and yet quite distinct. I find the illustrations in particular a lot of fun, and I’m very proud of the team that built it. Rafael Ebron ran the project with the SpreadThunderbird team, with designs from The Royal Order, and implementation from silver orange. A very nice job, thanks to all who contributed!¬† The new site also allows us to build localized sites, which will be amazing.

Lightning-in-Thunderbird status update

For some time, we (the Thunderbird release-drivers) have been exploring how to best integrate calendar functionality into Thunderbird.  Time for an update.

The current plan is to work with the Lightning Add-on community to make a version of it available as an add-on to Thunderbird 3 after we ship later this year.  This is a change from our initial plan of integrating all of Lightning into Thunderbird by default.  Our thinking has evolved based on both technical and product reasons:

  • the calendar team has had a tough time of figuring out how Lightning needed to change to integrate optimally in Thunderbird, in large part because Thunderbird 3 itself has been somewhat of a moving target — Thunderbird’s interface model is still shifting, which is certainly hard on add-on developers
  • partially as a result, there’s a fair amount of work left to do on Lightning before it’s ready to present as part of the core product.¬† For example, it’s not yet possible to selectively enable the task management or calendaring features; the account configuration is a bit too hard to get right for novice users; error messages are still too cryptic — that sort of thing.
  • looking at Lightning as an addition to the core project, it would represent a lot of new, complex code, with a lot of implied maintenance — we don’t have enough developers on hand to be able to take on that commitment at this stage.¬† See Philipp’s post on this topic for more information, especially if you’re keen to help out.
  • we try to avoid feature changes in between major releases, and limit minor updates to security fixes.¬† This would mean that feature changes to Lightning would have to wait for the next major release of Thunderbird.¬† That would be a real shame.
  • more generally, we recognize that different users need different kinds of calendaring solutions.¬† Just as there are more and more messaging systems, there is a growing diversity in calendar usage models, such as web calendars, stand alone clients, and calendar and event applications on social networking platforms.

Given all this, we feel the best plan is to take advantage of our add-on ecosystem, to allow a variety of calendaring features to evolve, and to allow Lightning to evolve at whatever pace is best for it, less tied to Thunderbird’s schedule.¬† Lightning is by far the most popular and important Thunderbird add-on, and we’re going to see what we can do to make it better and more successful, both on the development side, as well as on the promotion side.¬† Also, thanks to Thunderbird 3’s new add-on manager, it will be easier for users to find and install, which I expect will lead to even greater adoption.

In short: expect a version of Lightning that will work with Thunderbird 3; expect it to have more frequent releases than Thunderbird major release numbers; talk to philipp if you want to help!

How should we manage autoconfiguration files?

One of the features we’re working on for Thunderbird that I’ve mentioned before is the “autoconfiguration” setup, whereby when setting up an account, you type in your email address, and using the domain part (the part after the @ sign), Thunderbird tries to figure out what the right settings are. When it works, this is an amazing feature. This is not a new idea — in fact many mail programs have implemented in the past, including Exchange 2007 and the iPhone’s Mail program.

There are three ways of solving this problem: asking a central server for configuration files per domain, bundling some of those configuration files, and doing local network probing to look for likely domain names (mail.domain.com, imap.domain.com, etc.).

The central repository of configuration files has many advantages if we can figure out how to decentralize the provisioning of these configuration files, while remaining secure. In particular, it allows Thunderbird to “learn” about new domains or changed configurations as soon as those files are vetted, without having to wait for the next version of Thunderbird. Even with only a few configuration files, we should be able to cover the “short, fat, end” of the email domains — the top 200 domains for example probably cover 90% of home email users, which would be an amazing improvement for Thunderbird’s usability. (Gerv has already started a project months ago to collect some of this information)

Local port probing is most useful in corporate environments. Either their small user base or corporate policies may deter them from submitting their configuration files in the central server. In this case, the challenge is to strike the right balance of efficiency vs. completeness. We certainly don’t want to probe hundreds of ports on a local domain in a desperate search for a mail server, but to assume that all mail servers are configured the same way is unrealistic. A hybrid of the Exchange 2007 autodiscover model and the Mail.app configuration hostname-guessing model is likely optimal for this part of the market.

Side note: I could imagine an add-on that would bundle the configuration files and get updated periodically for those users who want to avoid leaking any data.

We welcome input on how we should manage these configuration files. The feature hasn’t landed yet, so it’s still very possible to influence this process. From my perspective there are a few key characteristics that should drive our process:

0) Reusing existing processes when possible makes life easier.

1) User experience is the goal. The point of this feature is to let as many people end up with a well-configured Thunderbird as easily as possible. This is why we’re going after a hybrid approach, trying to cover as many different environments with a single user experience.

2) This problem is not Thunderbird-specific. The data we’re looking for is not about how to configure Thunderbird, but how to configure IMAP/POP/SMTP email clients in general. If Mozilla can help solve the problem of email configuration in a safe, scalable, trusted way, then I’m fine with other email clients using it. I see this as potentially a very powerful public service that Mozilla can provide to email users everywhere.

3) We want to decentralize contributions of configuration files, but control the review processes, to facilitate end-user contributions and cover the long tail of domains, with a consistent policy especially with respect to validation.

In the short term, it feels like we could easily start with a variation on the processes we already use for code, with a security review overlay. In particular, I would suggest as an initial draft that:

– we have something equivalent to a module, which contains the published configuration files and processes around them
– each configuration is reviewed by someone other than the configuration submitter (patch author)
– each configuration needs to be validated by the reviewer against public documents published by the mail provider (ISP, university, etc.)
– Mail administrators can file a bug, or contact the module owner directly if they want to report changes in their email setup or suggested changes.

We will probably want to figure out more sophisticated systems involving cooperation with mail administrators (using MX or DNS records, or signed emails, or …) in a later phase, but I don’t think we should gate being able to serve users of the largest domains (gmail, yahoo, major ISPs, universities). Walk, then run, should be our approach.

Similarly, there are possible futures where users could share configuration files with each other, but I don’t think we’re ready to tackle those securely yet.

We’ll figure out a way for those interested in submitting or reviewing configurations to sign up in a later round. At this point, we’re most keen on process-type feedback. Comment on the bug, or here, thanks!

PS: Whatever process we arrive at, we’ll then apply to the files we’ve been using in testing and the configuration data in Gerv’s spreadsheet, to make sure that all the data has gone through the right hoops.

It’s Friday: Goofy but fascinating Thunderbird Add-ons day

Two different and equally goofy but interesting add-ons are in my personal news today:

Kent James released ToneQuilla, which I like to call “BiffTones!”, which allows you to set custom notification tones based on Thunderbird rules. Emails from the spouse make one sound, emails from the grandmother make another, etc. Neat!

Andrew Sutherland, on somewhat of a dare that I put in front of him (nothing like waving a red flag of visualization at a canvas bull like him) responded to the pretty but mostly useless Wordle meme which has been going around Mozilla circles, and built a wordle-like visualization of the database-driven queries that I blogged about a couple of days ago. If one can build an add-on to that in a day (well, a night without internet access), what couldn’t one do?

Both of these add-ons have somewhat of a goofy aspect to them, and both could evolve into something really useful. Notification overload is a huge problem in communication clients – it’s useful to know when something important happens, but useless to know when “a message was received” — tools like ToneQuilla can help. Simiarly, visualizations can provide insight into ones’ messaging history. See Themail for interesting research on the topic.

Thunderbird 3 beta 1 – a platform for innovation shapes up

Today, we’re announcing our first beta-quality release since the Thunderbird project was re-energized about a year ago. It’s exciting to see the first in what will be a series of releases aimed at a broader set of testers make it out the door.

In some ways, this is a typical beta — we’ve changed a lot of code since Thunderbird 2, and we need a lot of people to tell us if we’ve made any boo-boos when fixing bugs. It’s also a good beta in that we’ve moved the product forward, in part thanks to new capabilities in the underlying Mozilla platform, which gives us faster performance all around, an add-on manager which will be even more useful for Thunderbird users than for Firefox users. We also have important new mail-specific capabilities, including a new “autosync” system that gets Thunderbird to download IMAP message bodies early, so they’re already there when you need them, and a much faster implementation for deleting and moving IMAP messages, which I can’t imagine living without at this point. The one-click add-to-addressbook is also an elegant and shameless ripoff of the Firefox bookmarking model, which our alpha users love.

As a result, I feel that even for a first beta, Thunderbird 3 is much better than Thunderbird 2, thanks to a lot of hard work by a motley crew of great contributors worldwide, to whom I’m very grateful. All that and more is described in the release notes, which I encourage beta testers to read.

However, in some other ways it’s far from a typical beta. In particular, unlike the traditional definition of a beta release, we’re definitely not done making feature changes, including some pretty significant feature work that we expect will be integrated in Thunderbird 3 in later beta releases, some features that will live as optional add-ons, and some experiments which may end up in later releases of Thunderbird or not, depending on the result of the experiments.

I’ll talk a bit about some of these upcoming attractions, as I’m quite excited about them (and some more that will have to wait for another post).

First, the autoconfig work, which refers to a complete rethink of the account configuration process in Thunderbird. The account “wizard” in Thunderbird made sense in the early days, but over the years it has acquired complexity and lost relevance, as email systems have gotten more complex. Unfortunately, if you’re lucky enough to have a secure email server, the current Thunderbird user interface unjustly punishes you by making you go through 8 pages of questions and you end up with an account which requires manual tweaks before you can check mail. That’s not good. To deal with this, we have rethought account configuration completely, and came up with a dialog which, when it lands (becomes available by default), should make account configuration really, really easy. It’s been hard to come up with an elegant minimal user interface that hides all of the complexities of email configuration, but it’s worth doing it right.

Next up is tabs. Thunderbird 3 has a great opportunity to be basically coming up with a tabbed interface at a time in history where we’ve learned a lot about how tabs work well or poorly. In Thunderbird 3 beta 1, it’s a fair bit easier to work with tabs than it was in Thunderbird 2 (although many improvements are planned before the final release). For example, it’s much easier to create new kinds of tabs (the calendar Lightning add-on makes great use of those, for example, as I show below). One simple example of this is Bryan Clark’s “glodabook” add-on, which is a starting point for exploring new ways of navigating the address book.

Addressbook prototype

Next up is conversations. Thunderbird’s default mode saves emails you send in a “Sent messages” folder, and emails you receive filed in other folders, typically decided on a per-message basis by the users (more on that below). This is a fine default strategy, but it can make it hard to find related messages if they’re not in the same place (e.g. messages that are replies to emails you sent, or messages that are part of a long conversation, some of which is in your archive folders, and some of which is in your inbox). Thunderbird 3 includes a powerful search engine (“Gloda”) which is designed to let us efficiently find messages that are related, no matter where they are. In particular, it makes it quick to take a message and “show it in a conversation context”. This lets you view the messages you sent interspersed with the messages you received, but also messages from earlier in the conversation which you may have archived. This is still experimental, and not enabled by default in 3.0b1, but early results are very promising:

Conversation view

Next up, search. Part of the Gloda search engine mentioned above is a powerful full-text search engine, which seems to be working quite well so far. Thunderbird search is already better in 3.0b1 than in 2.0 because we’re more aggressively downloading emails, and doing a better job of finding the downloaded copies. With the new search engine, we’ll be able to efficiently do searches like “show me all messages from bryan mentioning ‘conversation’ in the body or the subject”. And we think we can make that easy for users to discover as well:

First we do autocomplete on existing contacts:

autocompleting contacts

and then encapsulate them in graphical objects to simplify the display:

experimental search results view

On that topic, one of the design topics we’re exploring is how to make it easier for users to be smarter about search. Thunderbird has always had very powerful search capabilities, but to use them people have to think like database programmers, which most of us aren’t. We have some plans there to help people build smart searches based on starting with the simple searches people are used to from the web, using suggested sub-searches based on analyzing their search results. Now that we have the search engine in place, we can start to experiment with many different search models, and see what works best.

The last two screenshots are particularly exciting to me because they demonstrate that we can leverage the foundational bits of Thunderbird, and experiment with new ways of working with messages, without disrupting the user experience that Thunderbird 2 that many users are comfortable with. What’s equally exciting is that these new ways can themselves be platforms for experimentation, whether by us, or by others. One such experimentation topics is conversation visualization and interaction models. Andrew Sutherland implemented an add-on that shows thread arcs (here using a view that it out of date by a whole week):

Thunderbird has always been an interesting experimental playground, because of its open source nature and the add-on model. The technology platform in Thunderbird 3 will make it even more so — 1) we have better technology that allows new ways to slice the data, 2) as we’re exploring new features through the use of add-ons ourselves, we find out early what changes we need to make to make the platform more extensible, and 3) because we’re fully leveraging web technologies, something which is a bit new for Thunderbird. In particular, all of the views above build on some of the most compelling advances in web technology, from the canvas widget to Javascript toolkit-based animations (JQuery for now) and modern CSS features.

Finally, last but not least, the Lightning calendaring add-on is moving along great. The Thunderbird+Calendar team has made a lot of progress on tackling the stack of issues that made it hard to integrate into the new Thunderbird codebase. We’re not done yet, but it’s looking great:

calendar tab

There are some other add-ons that some contributors are working on that I’ll talk about as they get polished and ready for screenshots.

As always, we love to get ideas for interesting new capabilities we can bring to the platform. We’re focusing on some of the basic capabilities we think are crucial to solving today’s mail problems, such as search and message management, but it’s a huge field, and email users are desperate for innovative ideas.

We’re identifying way more topics of interest than we have time to tackle, so we’re hoping to reach out to designers to get a broader set of participants helping us with some of the design challenges of a modern approach to messaging, within the context of Mozilla Labs. More on that soon.

Whether you’re a designer or an implementor, if you want to build new features on top of the views we’re building, add new kinds of data to add to our database (twitter, facebook, rss, etc.), or new visualizations, do get in touch.

If you’re interested in the extensions above, and aren’t afraid to try out code that changes daily, my recommendation is to use an IMAP server, Shredder (the nightly builds of Thunderbird, which are already different than the beta 1 build), and the extensions at the following locations:

Visite au Québec visit

(The title is a a cute translating trick I picked up in a Tim Horton’s)

La semaine derni√®re, j’ai eu le plaisir de faire une pr√©sentation sur Thunderbird √† la CLLAP (Version PDF, version Slideshare)– la conference sur les logiciels libres et les administrations publiques.

Last week, I had the pleasure of giving a talk on Thunderbird at the CLLAP conference on free software in public administrations (PDF version, Slideshare version).

It was a memorable event for a couple of reasons: first, it was my first presentation in French, a language I grew up with, but haven’t used much professionally. Second, I had my first experience with “molecular gastronomy”, in the form of a passionfruit sorbet hollow sphere, made in a blown-up balloon swirled in liquid nitrogen.

C’√©tait un √©venement m√©morable pour deux raisons. C’√©tait ma premi√®re pr√©sentation en fran√ßais, quelque chose qui me tracassait un peu. C’√©tait aussi ma premi√®re experience de la gastronomie mol√©culaire, sous la forme d’une sph√®re de sorbet aux fruits de la passion, r√©ussite gr√†ce √† un ballon tourn√© dans un bain d’azote liquide.

Thunderbird est bien connu, √©videmment, dans le monde du logiciel libre, et j’y ai retrouv√© des amis (Inverse, Linagora, Messaging Architects) qui sont tr√®s int√©ress√©s par l’avenir de Thunderbird. Pour beaucoup des 200 personnes dans la salle, c’√©tait clair que l’integration des fonctions d’agenda est vivement attendue.

Conclusions:

  • Qu√©bec √† beaucoup de charme – un m√©lange d’histoire europ√©enne dans un contexte qui me rappelle la nouvelle angleterre. Il faisait un temps magnifiques, et j’y ai retrouv√© les couleurs d’automne sur la cote Est.
  • On y trouve du tr√®s bon pain (les caf√©s, par contre…)
  • J’appr√©cie les d√©riv√©es des patisseries traditionelles fran√ßaises, du genre pain-au-raisins sans raisin mais √† l’orange ou au citron, ou le pain au chocolate avec des bananes.
  • Malgr√© quelques diff√©rences d’accents, et quelques mots qui me laissaient perplexe, je crois avoir compris, et √©t√© compris.

It was nice to get immersed in French again, even if I come out a bit confused at times.