Launchpad itself

cronscripts/update-bugzilla-remote-components.py (+41/-0)
cronscripts/update-remote-product.py (+5/-1)
database/schema/security.cfg (+8/-2)
lib/canonical/config/schema-lazr.conf (+8/-0)
lib/canonical/launchpad/scripts/bzremotecomponentfinder.py (+205/-0)
lib/canonical/launchpad/scripts/updateremoteproduct.py (+6/-1)
lib/lp/bugs/configure.zcml (+0/-1)
lib/lp/bugs/model/bugtracker.py (+115/-115)
lib/lp/bugs/tests/test_bugtracker_components.py (+1/-1)
lib/lp/bugs/tests/test_bzremotecomponentfinder.py (+190/-0)
lib/lp/bugs/tests/testfiles/bugzilla-fdo-advanced-query.html (+2629/-0)
lib/lp/bugs/tests/testfiles/bugzilla-wine-advanced-query.html (+1787/-0)

To merge this branch:

bzr merge lp://staging/~bryce/launchpad/lp-617691-retrieve

High

Fix Released

Link a bug report

Reviewer	Review Type	Date Requested	Status
Māris Fogels (community)			Approve on 2010-10-19
Launchpad code reviewers	code	2010-10-19	Pending
Review via email: mp+38804@code.staging.launchpad.net

Commit message

Add cronjob script for updating bugzilla bugtracker component and component group tables.

Description of the change

This adds a cronjob script which gathers lists of components and products (component groups) from remote Bugzilla instances.

Pre-implementation calls were done with Graham and Deryck. It was suggested to model the script off of sfremoteproductfinder.py, which I've attempted to do.

This code is tested via the command:
./bin/test -t bzremotecomponentfinder

I've cleaned up most lint messages; there's a few which I think are spurious. I'm running this through ec2 test presently.

This branch depends on lp-617699-api to be merged first. -api is ready to merge now but I'm having trouble landing it due to unrelated test failures; hopefully this doesn't make it too difficult to review the diff, apologies if it does.

Revision history for this message

Bryce Harrington (bryce) wrote on 2010-10-19:

I've just lp-landed lp-617699-api, which this branch depends on.

Revision history for this message

Māris Fogels (mars) wrote on 2010-10-19:

Hi Bryce,

This code looks good, r=mars. I have a few comments about docstrings and the test suite:

• The constructor arguments to BugzillaRemoteComponentFinder should be documented. The parameter name "txn" does not provide enough information. The parameter object type should be documented at minimum.

• There is a grammar error in the comment on line 293, "need added"

• How long does it take to run your new test suite? We want to try and keep new tests under 2 seconds runtime.

Maris

review: Approve

Revision history for this message

Bryce Harrington (bryce) wrote on 2010-10-20:

Thanks mars.

In looking closer, I don't actually use txn anywhere (I'd inherited it from the sf update script) so I've dropped it entirely.

Grammar error is fixed by restating the comment.

The test suite takes around 11 seconds to run. Most of that is due to it running the full cronjob. Disabling that test brings it down to 2.5 seconds.

There are two tests which parse a copy of the freedesktop.org bugzilla advanced search page, which contains a lengthy number of components. If I replace one of those and have it parse the (much smaller) Wine bugzilla advanced search page, it brings the test down to about 1.8 seconds.