Merge lp://staging/~gary/launchpad/bug844631 into lp://staging/launchpad

[r=bac,lifeless][bug=844631] Add an error page for when the database is down (such as fastdowntime) or has serious issues.

= Overview =

This branch adds an error page for when the database is down (such as fastdowntime) or has serious issues.

It catches both OperationalError and DisconnectionError because, on experimentation, it seems that the database raises OperationalError if it can't connect initially (such as after an app restart when the db is down), and (as expected) DisconnectionError if it was connected but lost the connection.

= UI =

The page design is by Huw, except for the status update section, which I had to rewrite because of technical reasons (see below).

= Implementation =

The registration of the page is straightforward. The branch only has two elements of real interest (and which took up my time).

== Testing ==

The first exciting bit was testing. Stuart suggested that I write integration tests using pgbouncer. This was a bit challenging because it found some edge cases in using the pgbouncer. In particular, you'll see an odd little loop in the test where I get rid of two 500 errors. This was about the best I could do. I talked with him about this for some time and had some alternatives, but we settled on this. I'm not sure what to tell you about it, even though it took a longer time than expected.

== Status updates ==

The second element of this branch that took up my time is the inclusion of the status updates on the page.

=== Rejected ideas ===

(Feel free to skip; I feel compelled to share.)

Huw initially had done this with a Twitter-supplied widget that had a number of problems, such as relying on http. He found another widget, but it relied on jsonp, which would introduce a security hole for our site (http://en.wikipedia.org/wiki/JSONP#Security_concerns). Matthew proposed reusing the same style of memcached RSS loading that we use on the front page for the blog listings, but that was risky for a page that is supposed to be shown when the system is having problems already, and also would have required poking another hole in our firewall. Robert proposed the interesting idea of having a static page hosted in Apache over http which used jsonp, and redirect the appserver error page to that page. That probably would have worked, but I was hoping for a lighter weight solution.

=== Implementation: the story ===

(OK, you can skip this too. This took so long that I feel like talking about it, I guess.)

I next explored the Flash xdr method that YUI provides. Identi.ca does not provide a http://identi.ca/crossdomain.xml , which Flash requires. twitter.com and api.twitter.com explicitly have a crossdomain.xml that only allows their own sites to use it. After a bit of digging, I discovered that search.twitter.com does have a friendly crossdomain.xml. I hooked it up and it was working.

Unfortunately, it used Flash.

Next, I discovered CORS (http://en.wikipedia.org/wiki/Cross-Origin_Resource_Sharing). Yay! This is the answer we want.

=== Implementation: the meat ===

(Read this part, please.)

I try CORS first. That should work on modern Ubuntu installs. If that fails, I fall back to Flash. If that fails, I give up. This seems to work. Yay.

=== Testing the JS out yourself ===

You probably could just shut down PG while you run LP, but I installed a temporary view to look at it.

=== modified file 'lib/lp/app/browser/configure.zcml'
--- lib/lp/app/browser/configure.zcml 2011-09-29 02:32:20 +0000
+++ lib/lp/app/browser/configure.zcml 2011-09-29 18:54:11 +0000
@@ -377,6 +377,15 @@
       class="canonical.launchpad.webapp.error.OperationalErrorView"
       />

+ <!-- XXX Remove -->
+ <browser:page
+ for="*"
+ name="disconnected.html"
+ permission="zope.Public"
+ template="../templates/launchpad-databaseunavailable.pt"
+ class="canonical.launchpad.webapp.error.DisconnectionErrorView"
+ />
+
   <!-- UnsafeFormGetSubmissionError -->
   <browser:page
       for="canonical.launchpad.webapp.interfaces.UnsafeFormGetSubmissionError"

== Lint ==

lint is happy AFAICT except for the usual long line complaints in the lazr config file, but "make lint" is confused and gives me lint on the whole tree, so it's possible I missed something.

That's it.

Thanks!

Gary