Thunderbird Developer Tools Wrapup

In my earlier two posts I showed you my work on the Google Summer of Code 2013 Project to bring the Developer Tools to Thunderbird. The method for doing so is making use of Firefox’s remote debugging protocol, allowing to use the web developer tools available in Firefox to manipulate Thunderbird. More details are covered in the earlier posts. The Summer of Code has now come to an end, so I would like to tell you about my progress, the goals I’ve reached and those my mentor and I have decided are out of scope.

First of all, let me tell you about the remaining features I have implemented since the last post. One of the features is support for the remote inspector. This was pretty easy to do, although support for it is still preliminary. There are still a few quirks, but it’s mostly usable.  You can see here I’ve changed an attribute value:

Devtools Inspector in Action

Next up is support for scratchpad, which is still work in progress on the client side but is almost complete. Here is a screenshot:

Devtools Scratchpad in Action

Also, there is the app manager. This is, as far as I’ve understood, still in a beta stadium and aims to be a central place for managing remote devices. Thunderbird is one of these “remote devices”. The app manager shows some information about Thunderbird like its resolution and allows making screenshots:

app-manager

Finally, I’ve made progress packaging the glue code required for the debugger server into an extension. This is mostly a build system change that allows packaging the code as a restartless addon which I can distribute on addons.mozilla.org. The extension has an option dialog which allows starting and stopping the remote connection. From within Thunderbird this extension is not needed, but it is helpful for other applications based on the Mozilla Platform, like those based on XULRunner. I will post an update when the extension is available on addons.mozilla.org. Here is a screenshot of the options dialog:

Devtools Server Extension Dialog in Action

In my original milestone planning there were a few features considered a bonus. Some of these were not completed. It turns out those extra features are a substantial amount of effort, possibly even worth their own Summer of Code Project.

The first of these two is adding a way to inspect IMAP connections in the network monitor. This requires providing a specific interface in the IMAP channel implementation which makes it possible to inspect the content even after the request has been sent. Also, it is needed to mimic certain aspects of a http channel, specifically the concept of request and response headers. In Thunderbird, the IMAP channel implementation is heavily cached. Hooking up the channel interface to the network monitor would cause display of cached requests as separate requests. Also, this would only fix it for IMAP connections. A better way would be to add a general mechanism in the Mozilla Platform to be able to inspect TCP connections. This requires some changes very deep down in the networking platform and is probably not easy to carry out. I have filed a bug to solve this, but it won’t be a part of the Summer of Code.

The next feature that is missing is gcli, also known as the “Developer Toolbar”, that small black bar you can open in the Web Developer menu that allows executing text commands. The problem here is that the code has a lot of dependencies to Firefox code. A substantial amount of files need to be moved from the directory containing Firefox code to a directory common to all XUL applications. Some files also need to be split up.  As this feature is a nice to have, but not considered vital functionality for Thunderbird Developer Tools, we have decided to postpone it. If you see a need for this feature, please leave a comment describing what you want to do with it. In the meanwhile you can follow the bug on bugzilla.

With this I have covered all the features I have proposed, I’d say it was a very successful Summer of Code. I have managed to reduce the code needed in Thunderbird and made most of the changes inside the Developer Tools code. This makes sure that support for Thunderbird will work in the future without needing updates. Also, new remote features will automatically work, given there is no Firefox specific code in them.

If you want to jump right in and try it, I have to appeal to your patience. Some of the patches required for functionality are still in review by my mentor and the Mozilla developer tools team. I will let you know once everything is in place. I’m pretty sure we will able to get all code into the tree by the end of the current cycle.

Thunderbird Style Editor, Web Console, Network Monitor and Profiler

As you can see in my earlier post, I have worked on giving the Firefox developer tools access to Thunderbird using the remote protocol. Ultimately this means you will be able to debug and profile scripts, edit styles, view network traffic and view console messages.

Previously, I was able to make the debugger work, which was the most important feature to my mind. Now that the mid-terms are coming up soon, I thought I’d give you an update on what I have achieved. After fixing a few bugs in the developer tools code, I was able to add the remaining actors needed for the profiler, web console and style editor. The network monitor actually worked out of the box.

The code is not yet reviewed or pushed to the tree, so you cannot test it just yet. To bridge the gap, I’d like to present a few screenshots where you can see its working.

This is the web console in action. As you can see, evaluated JavaScript in the web console in Firefox executes in Thunderbird:Devtools Web Console in Action
 

This is the style editor. You can do everything you can do in Firefox: Disable style-sheets, change style rules, or add new ones. Here I’ve changed the #today-pane-panel background color to red:Devtools Style Editor in Action

 

This is the network monitor. You can see the Lightning calendaring extension connecting to a CalDAV server. In this case I have just added the New Event, which sends a PUT request to the server. Pure socket connections like IMAP are not visible yet, but anything that is HTTP will show up. I will be looking into adding socket connections to the network monitor after the midterms:Devtools Network Monitor in Action

 

Finally, the profiler in action. I was able to start the profiler in Firefox, then I did some random actions in Thunderbird. The profiler analyzed which functions were executed how often and how long they took and I could inspect the result in Firefox:Devtools Profiler in Action

 

If you want to track my progress on a more detailed level, head over to this issue on bugzilla and put yourself on the CC list. There you will also see what is left to do. I will add a comment there when the current patches are pushed and usable in the nightly builds.

The Thunderbird (Remote) Debugger is alive!

For quite some time now, I have been forced to use printf-style debugging for any work on the Mozilla Calendar Project. In most cases, its a real pain. Evaluating variables without restarting is so much more comfortable. There used to be Venkman, but due to ongoing “improvements” in the Mozilla Platform and Firefox, Venkman is broken and is no longer doing the job. When support for the first version of the Javascript Debugger interface (JSD1) is removed, that will be the final nail in the coffin of Venkman.

So it looks like we need an alternative. I’ve heard of lots of interest in creating alternatives, but the deal breaker is often the lack of time to actually work on a such project. In the meanwhile, Mozilla is investing time and resources to add native developer tools to Firefox. Maybe there is some way we can make use of these resources? Yes there is! The developer tools team is doing a great job. And by great I mean outstanding. Thanks to Firefox for Android and Firefox OS, the team designed the debugger in a client-server constellation. The Mozilla Platform provides debugger server component that is (almost) free of Firefox-specific code. Then there is the very Firefox specific developer tools client you know from the Firefox Tools Menu.

It became obvious to me that using this debugger server in Thunderbird would be a very future safe method. In contrast to copying the debugger UI into its own extension and make that compatible with Thunderbird, we just need to ensure that the already very general debugger server is kept clean of hardcoded Firefox-isms. For this reason I have applied to the Google Summer of Code as a student to make it happen.

Although the Summer has just started, I am proud to present a first success. With the latest nightly builds of Thunderbird 24.0a1 and a matching Firefox 24.0a1 nightly, its possible to debug Thunderbird code right from in your browser. Here is how:

  1. Download a Firefox nightly build.
  2. Download a Thunderbird nightly build.
  3. Start Thunderbird, select Tools → Allow Remote Debugging
  4. Start Firefox, open about:config, set devtools.debugger.remote-enabled to true and restart Firefox
  5. In Firefox, select Tools → Web Developer → Connect…
  6. Fill in connection details in case you changed anything, otherwise localhost port 6000 should be fine
  7. Now you should get a list with “Main Process”. Click on that

And that’s it! Now switch to the debugger tab in Firefox, and after a short load you will start seeing scripts and can set breakpoints. I will be improving support during the next weeks, so other tools can also be used. Track my progress in bug 876636.

As I’ve used the term “Remote Debugging” more than once in this post and it has already come up on the bugtracker, I will also tell you a little about privacy. It may sound like we are opening doors here so that anyone who might like to connect to your Thunderbird instance can control it. That is not at all true.

First of all, remote debugging is turned off by default. If you don’t do anything about it, then you won’t even notice its there, nor will any attacker. If you do enable remote debugging via the menu, either on purpose or by accident, there is another preference guarding you called devtools.debugger.force-local. The default value for this preference is true, this means that even with “Remote Debugging” enabled, only connections from localhost (i.e your computer) will be accepted. If you decide to circumvent this too by setting that preference to false, there is yet another wall to save you: If a remote debugger attempts to access your computer, you are presented with a dialog to accept, decline or even disable remote debugging. If you decline or disable, no harm is done.

If you have any further concerns regarding privacy, please do comment or contact me.

Executing JS Code when receiving an Email with a specific header set

I spent the last two days figuring out exactly how the Thunderbird new email notifications work. My goal was to listen in new Emails for a specific header and then run a specific task. Before I go into details a general note on this first. Please be careful when doing this and be even more careful when passing that header value to your code. The code processing the new mail is privileged and the header value should be treated as unsafe. In the best case this could cause a DoS type attack on your extension, in the worst case attackers could use it to control the user’s Thunderbird.

When I started on this I though it would be easy, a matter of hours. I started with a simple message filter listener, which is what I found on the internet first:

Components.utils.import("resource://gre/modules/mailServices.js");
Components.utils.import("resource://gre/modules/iteratorUtils.jsm");
Components.utils.import("resource://gre/modules/Services.jsm");

const CUSTOM_HEADER_NAME = "x-my-custom-header";

let listener = {
  QueryInterface: XPCOMUtils.generateQI([Components.interfaces.nsIMsgFolderListener]),
  msgsClassified: function(aMsgs, aJunkProcessed, aTraitProcessed) {
    for each (let msg in fixIterator(aMsgs.enumerate(), Components.interfaces.nsIMsgDBHdr))) {
      if (msg.getProperty(CUSTOM_HEADER_NAME)) {
        // do something
      }
    }
  }
  // other nsIMsgFolderListener methods ommitted for brevity
});

function setupListener(listener) {
  const mfn = Components.interfaces.nsIMsgFolderNotificationService;
  MailServices.mfn.addListener(listener, mfn.msgsClassified);
}

The problem that took me a while to figure out is that the “do something” block is never called. Until I found the actual soltuion, I tried different things like using a pure nsIFolderListener or creating a temporary message filter that takes care. I was convinced that the header values were just not parsed into the nsIMsgDBHdr yet. The reason for this was a very misleading comment in the nsIMsgDBHdr interface. But without some extra code, x-my-custom-header is never added to the properties of the nsIMsgDBHdr. This is done on purpose to keep the database small.

To make it work, we just need to set an extra pref:

let customHeaders = Services.prefs.getCharPref("mailnews.customDBHeaders").split(" ");
if (customHeaders.indexOf(CUSTOM_HEADER_NAME) < 0) {
  customHeaders.push(CUSTOM_HEADER_NAME);
  Services.prefs.setCharPref("mailnews.customDBHeaders", customHeaders.join(" ").trim());
}

I hope this saves you a few hours of digging. I certainly would have hoped to find a guide like this, but then again maybe I was looking at the wrong place with the wrong terms.

Disable Specific Error Messages in the Closure Linter (gjslint, fixjsstyle)

Update: It seems Google has made this possible directly now, the feature request was marked fixed. Usage as follows:

--disable Disable specific error.
Usage Ex.: gjslint --disable 1,0011 foo.js.

--max_line_length Maximum line length allowed without warning.

This makes the following mostly obsolete, but I’m leaving it here anyway.


I have recently been working on ical.js, a library to parse rfc5545 calendar data. James Lal has been helping me outand has suggested to use a Javascript linter on a regular basis. We have agreed on using the Google Closure Linter.

One thing I didn’t like about the Closure Linter is that error messages are not configurable. There are some cases where I think 80 characters per line are just wrong, for example a for() statement that has 81 characters. Would you really want to break lines for just one character? What about URLs in comments that are just longer than 80 characters? Also, I’m not a fan of single quotes.

On the internet I found the Closure Linter’s feature request issue and a few discussion topics. While the feature request links a patch, I wanted a solution that does not modify the Closure Linter so new contributors don’t have to go around patching things just to check and fix the style. To do so I created a wrapper script around gjslint and fixjsstyle, as well as a common file for defining the errors to ignore.

First of all, the common file. Name it something like myerrorrules.py:

from closure_linter import errors
from closure_linter import errorrules

OriginalShouldReportError = None

def InjectErrorReporter():
 global OriginalShouldReportError
 OriginalShouldReportError = errorrules.ShouldReportError
 errorrules.ShouldReportError = MyShouldReportError

def MyShouldReportError(error):
 global OriginalShouldReportError
 return error not in (
   errors.UNNECESSARY_DOUBLE_QUOTED_STRING,
   errors.LINE_TOO_LONG,
 ) and OriginalShouldReportError(error)

Then you need a small wrapper around gjslint.py:

from closure_linter import gjslint
import myerrorrules

if __name__ == '__main__':
  myerrorrules.InjectErrorReporter()
  gjslint.main()

And quite the same for fixjsstyle.py:

from closure_linter import fixjsstyle
import myerrorrules

if __name__ == '__main__':
  myerrorrules.InjectErrorReporter()
  fixjsstyle.main()

That is all! Now you just have to start gjslint and fixjsstyle through these wrappers and you will have all the errors ignored that you like.

Über die Zukunft von Thunderbird

Tristan Nitot hatte Gelegenheit mit JB Piacentino ein Interview zur Zukunft von Mozilla Thunderbird zu führen.
Hier eine Übersetzung dieses Gespräches:
Tristan: Hi Jb, ich habe im Internet gelesen, dass Thunderbird Tot sei. Ist das wahr oder ist das Ende von Thunderbird übertrieben?

Jb: Nein, Thunderbird ist nicht am Ende. Wir ändern die Art und Weise, wie wir neue Funktionen für Thunderbird entwicklen. Nichts wird sich für den einzelnen Privatanwender oder Nutzer in Unternehmen/Organisationen ändern. Mozilla wird Thunderbird weiter unterstützen.

Tristan: Es wird also Thunderbird 14 geben?

Jb: Ja, natürlich! Wir haben einen soliden Plan, um Thunderbird bis Ende 2013 zu unterstützen und sind ebenso in der Planungsphase, was die anschließende Unterstützung angeht.

Tristan: Und was passiert bis dahin?

Jb: Thunderbird 15, 16 und 17 werden demnächst herausgebracht. Für Thunderbird 15 gibt es bereits eine Alpha-Version und auch nach Version 17 werden weiterhin neue Versionen herausgebracht. Wir werden coole, neue Innovationen sehen, wie zum Beispiel Thunderbird Chat (Instant Messaging), weitere Anbieter zum Anmelden neuer E-Mail-Adressen und für Thunderbird Filelink, sowie neue Funktionen, die vollständlig von der Community beigesteuert wurden. Zusätzlich wird Mozilla auch weiterhin die Beiträge aus der Community beachten und mit in das Produkt einfließen lassen.

Tristan: Weisst du, wer die Thunderbird-Benutzer sind und was sie wollen?

Jb: Wir haben eine Mischung aus Privatanwendern und Firmenkunden. Unter den über 20 Millionen Benutzern gibt es viele, die Thunderbird großflächig einsetzen, wie zum Beispiel Firmen, Regierungs- und Bildungseinrichtungen. Wir müssen dabei das Gleichgewicht wahren, um sowohl Privatanwender als auch Firmenkunden zufriedenzustellen.

Eine Gemeinsamkeit unter allen Benutzern ist zu aller erst die Forderung nach einem sehr stabilen und sicheren E-Mail-Produkt. Beispielsweise können in sehr großen Firmen Veränderungen an den vorhandenen Funktionen alle möglichen Probleme hervorrufen, angefangen mit Problemen bei der Zertifizierung, der Verteilung auf allen Systemen bis hin zur Schulung der Benutzer. Ebenso haben unsere Privatanwender kein großes Verlangen nach neuen Innovationen.

Dennoch glauben wir, dass das Erlebnis für den Benutzer verbessert werden kann, zum Beispiel durch die Einbindung zusätzlicher Web-Dienste ähnlich wie wir es für Thunderbird Filelink gemacht haben. Es ist schwer das richtige Gleichgewicht zu finden, aber im Moment glauben wir, dass unsere Benutzer mit den aktuellen Funktionen zufrieden sind und viel mehr Wert auf ein stabiles Produkt legen.

Tristan: Ich sehe eine große Kluft zwischen den Erwartungen der Thunderbird-Benutzer und Mozillas Schwerpunktsetzung auf das Internet und die Mobilgeräte-Sparte mit Firefox für Android und Firefox OS (aka Boot to Gecko). Gibt es einen Zusammenhang mit dieser Entscheidung über Thunderbird?

Jb: Mozilla setzt seinen Schwerpunkt sicherlich auf das Internet als Platform, da wir an dieser Stelle als Gemeinschaft mit der Community einen Unterschied machen können. Mobile Umgebungen sind weitgehend geschlossen. Wir glauben, wenn wir die Werte und Technologien hinter dem Internet auf die mobilen Geräte bringen können, dass wir dadurch einen Mehrwert für die Benutzer mobiler Geräte schaffen können. Thunderbird bleibt ein sehr gutes E-Mail-Programm für den Desktop-Bereich, was sich aber nicht wirklich mit der Strategie von Mozilla vereinbaren lässt.

Tristan: Lass mich einmal den Teufel an die Wand malen: Warum stellt Mozilla die Unterstützung für Thunderbird nicht vollständig ein?

Jb: Ich glaube, dass der Plan, den wir vorschlagen, zeigt, dass Mozilla trotz dieser verzwickten Lage zu seinem Wort steht: Wir werden Thunderbird nicht einstellen. Im Gegenteil, wir schmieden die Vorraussetzungen, damit das Produkt weiterhin sicher und stabil ist und Beiträge aus der Community integriert werden können. Um genau zu sein, ermutigt uns die frühe Resonanz aus der Community und ich bin mir sicher, es werden interessante Entwicklungen von freiwilligen Mitwirkenden und auch Firmen kommen.

Weiterhin gibt es eine Reihe von Open-Source-Beratungsfirmen, die ihre Firmenkunden beim Einsatz von Thunderbird unterstützen, und ich bin mir sicher, sie werden mithelfen, um den Wünschen ihrer Kunden nachzukommen und somit Thunderbird zu verbessern.

Tristan: Werden auch in Zukunft Mitarbeiter von Mozilla an Thunderbird arbeiten? Wie lange wird das noch der Fall sein?

Jb: Ja, Mozilla wird auch weiterhin Thunderbird mit bezahlten Mitarbeitern unterstützen, obwohl diese möglicherweise nur Teilzeit am Projekt arbeiten werden.

Tristan: Wenn du einen Aufruf an unsere Leser richten könntest, was würdest du ihnen mitgeben?
Jb: Das kommt wirklich auf deine Interessen und Fähigkeiten an:

  • Hast du Interesse, über die Zukunft der Verwaltung von Thunderbird als Projekt zu disktuieren, solltest du vielleicht die Wikiseite zur Verwaltung von Thunderbird besuchen und dort den Anweisungen folgen. Wir planen, im Laufe des Sommers Fortschritte zu machen und das neue Verwaltungsmodell anfang September fertigzustellen.
  • Möchtest du hingegen an der Entwickelung von Thunderbird teilnehmen, schau doch einfach mal auf dem Entwicklerportal vorbei . Ein paar Interessante Vorschläge für Innovationen findest du auf der Seite UpForGrabs. Zusätzlich braucht die Thunderbird-Hilfe immer neue Mitarbeiter, so wie die Qualitätssicherung.

Tristan: Jb, vielen Dank für deine Zeit. Da ich selbst auch Thunderbird-Benutzer bin, freue ich mich sehr, diese Information aus erster Hand zu bekommen. Ich freue mich sehr zu sehen, wie sich Thunderbird in Zukunft entwickelt.

Jb: Danke!

Keyword Bookmarks with Multiple Parameters

If you have ever looked at the properties of a bookmark, you might have seen the “Keyword” field. This is a very powerful field and makes your life easier. If you already use this field, you may have been disappointed that you cannot add multiple parameters to a keyword, since there is only one placeholder “%s”.

Inspired by the FileIt page, I wanted to create a bookmark keyword that allows me to pass in product and component for filing a new bug. Javascript to the rescue! All you need to do is to take the following code snippet and create a new bookmark as described on the Bookmark Keywords page. You might want to compress the bookmarklet using a site like jscompress.com.

(function(){
  /* First, split the passed string. You may use any delimiter you want, I chose "/" */
  var pc = "%s".split("/");
  /* Now set the base URL for the target to visit */
  var target = "https://bugzilla.mozilla.org/enter_bug.cgi?rep_platform=All&op_sys=All";
  /* If at least one parameter was passed, then add it as the product */
  if (pc.length) target += "&product=" + pc[0].charAt(0).toUpperCase() + pc[0].substr(1);
  /* If at least two parameters were supplied, then add the second as the component */
  if (pc.length > 1) target += "&component=" + pc[1].charAt(0).toUpperCase() + pc[1].substr(1);
  /* Now go to the target location */
  document.location.href = target;
})();

Afterwards, you can directly use the bookmark using the address bar:

" fileit calendar/alarms "