Monitor all http(s) network requests using the Mozilla Platform

In an xpcshell test, I recently needed a way to monitor all network requests and access both request and response data so I can save them for later use. This required a little bit of digging in Mozilla’s devtools code so I thought I’d write a short blog post about it.

This code will be used in a testcase that ensures that calendar providers in Lightning function properly. In the case of the CalDAV provider, we would need to access a real server for testing. We can’t just set up a few servers and use them for testing, it would end in an unreasonable amount of server maintenance. Given non-local connections are not allowed when running the tests on the Mozilla build infrastructure, it wouldn’t work anyway. The solution is to create a fakeserver, that is able to replay the requests in the same way. Instead of manually making the requests and figuring out how the server replies, we can use this code to quickly collect all the requests we need.

Without further delay, here is the code you have been waiting for:

The Thunderbird (Remote) Debugger is alive!

For quite some time now, I have been forced to use printf-style debugging for any work on the Mozilla Calendar Project. In most cases, its a real pain. Evaluating variables without restarting is so much more comfortable. There used to be Venkman, but due to ongoing “improvements” in the Mozilla Platform and Firefox, Venkman is broken and is no longer doing the job. When support for the first version of the Javascript Debugger interface (JSD1) is removed, that will be the final nail in the coffin of Venkman.

So it looks like we need an alternative. I’ve heard of lots of interest in creating alternatives, but the deal breaker is often the lack of time to actually work on a such project. In the meanwhile, Mozilla is investing time and resources to add native developer tools to Firefox. Maybe there is some way we can make use of these resources? Yes there is! The developer tools team is doing a great job. And by great I mean outstanding. Thanks to Firefox for Android and Firefox OS, the team designed the debugger in a client-server constellation. The Mozilla Platform provides debugger server component that is (almost) free of Firefox-specific code. Then there is the very Firefox specific developer tools client you know from the Firefox Tools Menu.

It became obvious to me that using this debugger server in Thunderbird would be a very future safe method. In contrast to copying the debugger UI into its own extension and make that compatible with Thunderbird, we just need to ensure that the already very general debugger server is kept clean of hardcoded Firefox-isms. For this reason I have applied to the Google Summer of Code as a student to make it happen.

Although the Summer has just started, I am proud to present a first success. With the latest nightly builds of Thunderbird 24.0a1 and a matching Firefox 24.0a1 nightly, its possible to debug Thunderbird code right from in your browser. Here is how:

  1. Download a Firefox nightly build.
  2. Download a Thunderbird nightly build.
  3. Start Thunderbird, select Tools → Allow Remote Debugging
  4. Start Firefox, open about:config, set devtools.debugger.remote-enabled to true and restart Firefox
  5. In Firefox, select Tools → Web Developer → Connect…
  6. Fill in connection details in case you changed anything, otherwise localhost port 6000 should be fine
  7. Now you should get a list with “Main Process”. Click on that

And that’s it! Now switch to the debugger tab in Firefox, and after a short load you will start seeing scripts and can set breakpoints. I will be improving support during the next weeks, so other tools can also be used. Track my progress in bug 876636.

As I’ve used the term “Remote Debugging” more than once in this post and it has already come up on the bugtracker, I will also tell you a little about privacy. It may sound like we are opening doors here so that anyone who might like to connect to your Thunderbird instance can control it. That is not at all true.

First of all, remote debugging is turned off by default. If you don’t do anything about it, then you won’t even notice its there, nor will any attacker. If you do enable remote debugging via the menu, either on purpose or by accident, there is another preference guarding you called devtools.debugger.force-local. The default value for this preference is true, this means that even with “Remote Debugging” enabled, only connections from localhost (i.e your computer) will be accepted. If you decide to circumvent this too by setting that preference to false, there is yet another wall to save you: If a remote debugger attempts to access your computer, you are presented with a dialog to accept, decline or even disable remote debugging. If you decline or disable, no harm is done.

If you have any further concerns regarding privacy, please do comment or contact me.

How to Triage Mozilla Bugs

Once in a while, I get emails from users that are willing to help out with the Calendar Project. Often, these users are not developers but would still like to help. If you believe it or not, there is nothing easier than that! All you need is a cup of coffee (or any mind-soothing drink), a Bugzilla account and a nightly build of the Mozilla product of your choice. I’ll assume you’ll be using Thunderbird and Lightning for now :-)

If you’d like more information on triaging bugs, please also check out the QMO’s Unconfirmed Bug Triage Project.

Getting The Prerequisites

Coffee should be easy. If you don’t have one at home, I’m sure you’ll find one nearby.

Now you need to create a Bugzilla account, unless of course you already have one. Its a very quick signup process, but be sure to pick an email address that you don’t mind sharing with the public, it will be shown on each bug you comment on.

Next, download the latest nightly of Lightning. To use it, you must download a fitting Thunderbird version as mentioned on the Lightning nightly page. Specifically for Lightning  you should use the Lightning version that fits to the next major release of Thunderbird. For example, at the time of writing this post the next major Thunderbird version to be released is Thunderbird 3.1, so you want to download Lightning 1.0b2pre. Make sure you create a new profile for testing, I’ll explain more below.

In the next part of this post, I’ll explain what there is to triage. It makes sense to start small, just asking for more information can be very helpful for us. If you want do dive in deeper, then you may want to test on your own. Later on, I’ll tell you how to stay up to date with new bugs filed so you can quickly assist bug reporters and how to make life easier when processing bugs.

Asking for more Information

This is the easiest step you can take. To do this, you don’t even need a nightly version yourself, although it might be helpful for easy-to-reproduce bugs if you want to check yourself.

It doesn’t work! Help!

In the worst case, people file bugs like “Your application doesn’t work. Its very obvious. Please tell me what to do”. Now what should we do with that! What version is the reporter using? What operating system? Most importantly: what exactly isn’t working? If you see a bug like this, you need to be asking exactly these questions. If the bug is about Lightning, then we need to know:

  • What Thunderbird version is the user using, on which operating system?
  • What Lightning version does he or she have installed?
  • If more extensions are involved (for example, the Provider for Google Calendar), which extension version is being used?

Some reporters turn out to be using older versions of the Product. Users tend to use stable versions of software and if something fails they report a bug. While this might work for Products with many developers, this doesn’t work for smaller projects like Lightning. Others use the latest version in their Linux distribution, which may turn out to be over 2 years old. In any case, it helps to ask: “Does this still happen with the latest Lightning nightly <version> builds? You can find them at <url> and should use Thunderbird version <version> “. This also makes sense if the last time the bug was confirmed has been quite a while back (i.e happened in the previous Lightning version).

Debug, what!? Messages? Where!?

Also, reporters usually don’t know about the extra debugging preferences, or where to look when something is going wrong. For Lightning, there are a few preferences that can be changed to give more debugging output. Its a good idea to tell the user how to find the advanced config editor. Then, you can tell them what to change to get more debugging info. For Lightning, calendar.debug.log and calendar.debug.log.verbose need to be set to true. Although not always required, it might also make sense to have the user restart the application.

Man…I’ve seen this before!

The more bugs you visit, the better picture you get of what bugs are filed. If you see a bug report that you think you’ve seen before, then search for that other one. Pick the one that has less information on it and mark it as a DUPLICATE of the other one.

Are you still out there?

If you or someone else has already asked the reporter to provide additional info and no response has been made for a couple of months, the next thing you can do is mark the bug INCOMPLETE, telling the reporter why: “No response for a while. I’m closing this bug INCOMPLETE for now, please feel free to reopen if you can still reproduce or have an answer to the above questions”. This helps keep the number of open bugs down and makes it easier to find out what issues need to be fixed for the next release.

Testing unconfirmed or new bugs

This step requires you to download a nightly build of Lightning. Assuming you are using Thunderbird/Lightning for your normal Emails and Calendar, you want to create a new profile dedicated to testing, since you might be breaking things with a nightly build. It makes sense to create a desktop shortcut or alias to directly start Thundebird with your testing profile. Adding the  “-no-remote” command line flag to the alias or shortcut allows you to run it alongside your normal Thunderbird. Also, you’ll want to download the Lightning Nightly Updater extension, which allows you to stay up to date with the nightly builds.

Now that you have things set up, you can start testing bugs. In most bugs, there will be a set of “Steps to Reproduce” in the first few comments. Just repeat exactly these steps and see if the error happens to you too. If so, leave a comment on the bug which Lightning Nightly version you are using and that the error is still happening. If not, try to be creative. Think about what the user might have done unknowingly, or change the steps a bit to see if you can make the error occur. To make it easier for the developers, you can document what you’ve changed on the bug.

If you can’t reproduce, go ahead and tell the user too. If its an obvious case, you can mark the bug WORKSFORME, otherwise you might want to ask the user additional questions, see “Asking for more information” above.

Verifying fixed bugs

These are bugs that have been marked RESOLVED and FIXED. Usually a patch has been checked in so the bug was marked FIXED. To identify if the bug is really fixed, you can test again using the original steps to reproduce. If the bug is really fixed, set it to VERIFIED. Otherwise you have two options. If the bug was just fixed recently, you can just set the bug to REOPENED and tell us what fails. This will ensure the developer takes a second look. The other option you have is to file a new bug. Note which bug you were testing and what goes wrong. This is also the better option if you notice that what fails is not directly related to the bug.

Now where do I find those bugs?

There are two different types of bugs that need triaging. The most important are UNCONFIRMED bugs. These are bugs that users have filed, but have not been confirmed by developers or bug triagers like you. We need to find out which of these are really bugs and which ones are support issues. About a week ago, we had almost 500 unconfirmed bugs. These are the bugs that can easily be processed by “asking for more information”, but are more quickly resolved if you test them yourself.

The next category is NEW bugs. These are bugs that have been confirmed before. If this has been a longer while back, then it might be that it works by now, maybe it was fixed by a different bug. These are not so important to triage, but if you see someone asking if the bug still exists, you might want to check if this is the case.

To find the bugs, use the Bugzilla search. For this task it makes sense to use the advanced search interface. There you can set the Product to Calendar, select only NEW or UNCONFIRMED bugs, and make sure nothing is selected in the “Resolution” box (use Ctrl+Click to unselect). You can also use these quick links for the queries: NEW and UNCONFIRMED.

Also, johnath made a very nice video blog post introducing you to Bugzilla, its worth listening to.

But I’m now allowed to!

When creating an account, you don’t directly have all privileges. You will be able to file bugs and comment on them, but actually setting other fields like the bug status needs some extra permission bits. I’d suggest to start out with just adding some comments to existing bugs as noted above. Once you’ve made a few comments and we see that you want to help out, contact me or others from the calendar team, asking for “canconfirm” or “editbugs” priviledges. The first lets you only set bugs from UNCONFIRMED to NEW, the second lets you change any aspect of the bug. Please note that to achieve editbugs, we need to have the feeling we can trust you. Of course we can undo bug changes, but you’ll probably understand that if an evil-doer comes along and just re-opens all calendar bugs, this is quite annoying and causes unneeded work.

I hope this interests you in helping out with a Mozilla product, and personally I hope it even inspires you to help out with the Mozilla Calendar Project. I’m looking forward to hearing from you!

All this typing is getting very annoying!

When triaging bugs, you end up typing the same stuff over and over again. Still happening? What version? Please retest! Your questions get shorter and shorter, which makes it hard for new users to understand what you really want. To help with this, I’ve created a few jetpack extension which makes work much easier. First of all, you need to install the jetpack extension to Firefox. Now you can visit my jetpacks page. You should install “Bugzilla Comments Helper” and “Close on Submit”. The first one gives you a set of predefined comments you can easily select from a dropdown, while the second adds a “Commit and Close” button, that closes the tab after your changes have been made.

Welcome to my world!

Hello reader. You are probably here because you are interested in what I am up to in the Mozilla world, more specifically how the Mozilla Calendar Project is doing. First of all I’d like to tell you a bit about myself so you can get a feeling for what to expect here.

As children often do, I’ve had a number of technical ideas in my childhood. The most related one is my wish to make organizing life as easy as possible. I wanted to have a solution where I can manage my time anywhere I am — be it via cell phone, PC or maybe even something exotic like looking in the mirror. Back then it was very hard to achieve this, but nowadays there are iPhones, laptops (that fit my budget) and mobile internet is probably getting cheaper as we speak.

After having tried Sunbird for a brief moment before, I read about a calendar testday with prizes (yes, that always works ;-). I decided to show up. After the testday was over I was one of the top testers for that day and received a prize. Afterwards I took a little closer look into the application and quickly found things I though I’d like to improve, and so I did.

After working on the Provider for Google Calendar, noticed there were still certain aspects of the Provider that I couldn’t synchronize with the application. This got me into fixing core calendar code, which soon got me into getting to know the team, and later on I was encouraged to apply for a job at Sun.

I’m having a great time at Sun and just recently I was appointed the new lead for the Calendar Project. Since especially in my role it is important to keep in touch with the community and users, I’ve decided to start this blog. I plan to introduce you to some new ideas and vision I’ve been having that might just make your life a bit easier and could bring me closer to my childhood dreams. I’ll leave the more technical posts to the Calendar Weblog, but once in a while you might also hear of an idea for an extension I have (hands on, implementation is up to you ;-) or I’ll introduce a new extension I have created.

Since the success of the product is very dependent on what those who use it think, I’d like to encourage you to tell me whats on your mind regarding Calendar. Tell me about your childhood dreams and what you think is the most important aspect of calendaring. I can’t promise that I’ll sit down right away and add your favorite feature, but your feedback will help me form the future of the Calendar Project.

So go ahead, leave a comment, write me an email, ask me on #calendar. The future of Calendar is waiting for you — get involved!