Merge lp://staging/~sinzui/launchpad/revision-karma-1 into lp://staging/launchpad

Proposed by Curtis Hovey
Status: Merged
Approved by: Curtis Hovey
Approved revision: no longer in the source branch.
Merged at revision: 16337
Proposed branch: lp://staging/~sinzui/launchpad/revision-karma-1
Merge into: lp://staging/launchpad
Diff against target: 357 lines (+50/-167)
3 files modified
lib/lp/code/model/revision.py (+6/-19)
lib/lp/code/model/tests/test_revision.py (+11/-59)
lib/lp/code/scripts/tests/test_revisionkarma.py (+33/-89)
To merge this branch: bzr merge lp://staging/~sinzui/launchpad/revision-karma-1
Reviewer Review Type Date Requested Status
Benji York (community) code Approve
Review via email: mp+137660@code.staging.launchpad.net

Commit message

Make allocate-revision-karma fast.

Description of the change

allocate-revision-karma.py is having its DB connection reaped while it's
running, because the query in RevisionSet
getRevisionsNeedingKarmaAllocated is terribly slow.

Right now, we have 21.1 million revisions. 15.9 million of these are
flagged as not having karma allocated. These 15.9 million revisions are
the combination of revisions that have never been processed, or only
exist in +junk branches, or only exist in branches owned by invalid
people. Each run, the scanner needs to check all 15.9 million revisions.

--------------------------------------------------------------------

RULES

    Pre-implementation: stub
    * There are a few parts to a fix, both in lp/code/model/revision.py
      * in getRevisionsNeedingKarmaAllocated should simply return
        revisions where karma_allocated IS FALSE.
      * in allocateKarma, self.karma_allocated should always be set
        to True.
      * This changes the existing behavior, so if a revision first ends
        up on the system in a +junk branch or in a branch owned by an
        invalid user (such as ~vcs-imports), it will never have karma
        allocated to it. Lp will not longer give users karma for
        another user's commit.

QA

    * As a webops to run
      ./cronscripts/allocate-revision-karma.py
      for qastaging's script host (gandwana).
    * Verify there are no errors after a 10 minutes. The proc
      can be killed since we do not need to Verify all 15 million
      revisions.

LINT

    lib/lp/code/model/revision.py
    lib/lp/code/model/tests/test_revision.py
    lib/lp/code/scripts/tests/test_revisionkarma.py

TEST

    ./bin/test -vc -t TestRevisionKarma lp.code.model.tests.test_revision
    ./bin/test -vc lp.code.scripts.tests.test_revisionkarma

IMPLEMENTATION

I modified allocateKarma() to always set self.karma_allocated = True and
getRevisionsNeedingKarmaAllocated() to only look for elf.karma_allocated
is False. This simplification also allowed me to remove the dedupe code
from getRevisionsNeedingKarmaAllocated(). I removed the tests for
allocating karma to retargeted junk branches and claimed profiles with
imported revisions because we don't want to support these cases anymore.
All the tests in test_revisionkarma was invalidated, but I wrote 3 tests
to verify the 1 positive case and 2 negative cases that the script
iterates over.
    lib/lp/code/model/revision.py
    lib/lp/code/model/tests/test_revision.py
    lib/lp/code/scripts/tests/test_revisionkarma.py

To post a comment you must log in.
Revision history for this message
Benji York (benji) wrote :

The branch looks good, especially the code consolodation.

Changing the "fail.." calls to "assert..." calls was a nice touch.

review: Approve (code)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
The diff is not available at this time. You can reload the page or download it.