Firefox Regression Search—What is it?

Many of us hacking on Firefox use bleeding-edge trunk builds as our daily browser. As such, we tend to find regressions, some of which impact our browsing more than others. Pinpointing the cause of a regression is a time-consuming process that typically involves downloading multiple old builds of Firefox and testing each to see if the regression occurs. Daniel Brooks wrote an awesome Perl script to find the regression range (the period of time in which the regression was introduced) by downloading a series of nightly builds from ftp.mozilla.org and binary searching over dates to narrow down the regression. I had thought of this concept earlier, but never got around to implementing it, so I was psyched to see a working implementation. His initial version relied on the user to manually indicate whether the regression appeared or not, but he soon added the ability to use reftests (layout comparison tests) to find regressions in an automated fashion. I took it upon myself to hack support for Mochitest into his script, and then set about driving it via The BuildBot so as to provide a completely automated interface to running the script.

The automated regression search implemented here requires you to write a reftest or a mochitest that detects the presence of the regression bug. A reftest is just a pair of webpages that ought to render the same (or sometimes different), typically by using different markup or script to achieve the same end result. A mochitest is just a webpage that calls JavaScript functions with an expected result, like ok(color == "blue", "color should be blue!");. This little snippet says that the variable color should have the value "blue". How easy is that?

Once you've written a test that can detect your regression, you simply upload it to the form and tell the form what sort of test it is. The server will then proceed to download Firefox builds and run your test on them, eventually producing you a regression range, with a handy link to Bonsai, which will show you what code changes got checked in in that time period, so you can figure out who caused the regression. Also, once you write that automated test, it can be checked into the Mozilla automated test suite so that if the regression ever surfaces again, it will be caught right away! How cool is that?

Yes, let's!

• Go to the regression search form
• Go to the regression search status page