Merge lp://staging/~stevenk/launchpad/db-add-bprc into lp://staging/launchpad/db-devel

Proposed by Steve Kowalik
Status: Merged
Approved by: Steve Kowalik
Approved revision: no longer in the source branch.
Merged at revision: 10693
Proposed branch: lp://staging/~stevenk/launchpad/db-add-bprc
Merge into: lp://staging/launchpad/db-devel
Diff against target: 416 lines (+341/-0)
10 files modified
database/schema/comments.sql (+10/-0)
database/schema/patch-2208-76-0.sql (+17/-0)
database/schema/security.cfg (+2/-0)
lib/lp/soyuz/configure.zcml (+34/-0)
lib/lp/soyuz/interfaces/binarypackagepath.py (+36/-0)
lib/lp/soyuz/interfaces/binarypackagereleasecontents.py (+43/-0)
lib/lp/soyuz/model/binarypackagepath.py (+37/-0)
lib/lp/soyuz/model/binarypackagereleasecontents.py (+70/-0)
lib/lp/soyuz/tests/test_binarypackagepath.py (+26/-0)
lib/lp/soyuz/tests/test_binarypackagereleasecontents.py (+66/-0)
To merge this branch: bzr merge lp://staging/~stevenk/launchpad/db-add-bprc
Reviewer Review Type Date Requested Status
William Grant code Approve
Gavin Panella (community) Abstain
Stuart Bishop (community) db Approve
Robert Collins db Pending
Review via email: mp+64783@code.staging.launchpad.net

Commit message

[r=stub,wgrant][bug=796997][incr] Add infrastructure to eventually support contents generation from the database.

Description of the change

(Updated to facilitate code review.)

Add a two new tables, BinaryPackagePath and BinaryPackageReleaseContents, and associated interfaces and models.

Background:

Currently, Contents generation (for example, the file at http://archive.ubuntu.com/ubuntu/dists/natty/Contents-amd64.gz) is generated every day, where it stops the publisher from running, and takes an hour or slightly more.

These two tables completes the first step of moving contents generation into the database, and lets it do the heavy lifting, rather than cocoplum's disk. This also gives us the opportunity to perform Contents generation for PPAs.

The next step of the plan is to have a garbo script that slowly populates these tables (but not yet, since they will be large).

The end goal is to move Contents generation into the publisher itself, where it will query the table by a list of BPRs and get back every single filename that the BPRs contain.

IBinaryPackagePath{,Source}:

The idea behind these two interfaces and database table is so that a filename that is in the contents of a Binary Package Release (such as, 'bin/true'), is only mentioned once.

IBinaryPackageReleaseContents{,Set}:

These two interfaces and database table are to link a given Binary Package Release (BPR) to its contents, where the filenames are joined from BinaryPackagePath. Note that a given BPR will have multiple rows in BPRC, one for each file, but each BPR can only link to a given file once.

Tests: All new, all shiny.

To post a comment you must log in.
Revision history for this message
Stuart Bishop (stub) wrote :

BinaryPackagePath.path should be UNIQUE NOT NULL.

Shouldn't this table be called BinaryPackageReleasePath to match BinaryPackageReleaseContents?

Both columns in BinaryPackageReleaseContents should be NOT NULL.

patch-2208-76-0.sql

You likely will want this construct to ensure a path exists in the BinaryPackageReleasePaths table. This will avoid race conditions in scripts (not running in serializable isolation):

insert into binarypackagepaths (path) select 'foo' where not exists (select * from binarypackagepaths where path='foo');

review: Approve (db)
Revision history for this message
Gavin Panella (allenap) :
review: Abstain
Revision history for this message
William Grant (wgrant) wrote :
Download full text (3.3 KiB)

82 + <class
83 + class="lp.soyuz.model.binarypackagepath.BinaryPackagePath">
84 + <allow
85 + interface="lp.soyuz.interfaces.binarypackagepath.IBinaryPackagePathSource"/>
86 + </class>

No need for this. You already grant it to the securedutility.

104 + <class
105 + class="lp.soyuz.model.binarypackagereleasecontents.BinaryPackageReleaseContents">
106 + <allow
107 + interface="lp.soyuz.interfaces.binarypackagereleasecontents.IBinaryPackageReleaseContentsSet"/>
108 + </class>

Same here.

You also have one Source and one Set. Please make them the same.

+ path = TextLine(title=_('Full path name'), required=True, readonly=True)

s/Full path name/Absolute path/, perhaps?

158 + def getOrCreate(path):
159 + """Fetch the ID of the given path name, or create it.
160 +
161 + :param: path: The full path name to query or create.
162 +
163 + :return: An ID.
164 + """

Why an ID? Why not the object itself?

191 + """The contents of a binary package release.
192 +
193 + A binary package release is a representation of a binary package in
194 + one or more releases. This class models the files contained within
195 + the binary package.
196 + """

In one or more releases? I think replacing all this with a one-liner would be good. "A file contained by an `IBinaryPackageRelease`." or so.

250 + def getOrCreate(self, path):
251 + """See `IBinaryPackagePathSource`."""
252 + store = IMasterStore(BinaryPackagePath)
253 + bpp = store.find(BinaryPackagePath, BinaryPackagePath.path == path)
254 + if bpp.count():
255 + return bpp[0]
256 + else:
257 + bpp = BinaryPackagePath()
258 + bpp.path = path
259 + return store.add(bpp)

Try something like this:

    bpp = store.find(BinaryPackagePath, BinaryPackagePath.path == path).one()
    if bpp is None:
        bpp = BinaryPackagePath(path)
        store.add(bpp)
    return bpp

314 + with TempDir() as tmp_dir:

Can you use a TemporaryFile instead? Or even a NamedTemporaryFile, if that doesn't work. Or does DebPackage really want a filename, and care about what it is? Also, might as well break out of the with block as soon as you're done with DebPackage.

324 + bpp = getUtility(IBinaryPackagePathSource).getOrCreate(
325 + unicode(filename))

Objection! unicode() on potentially non-ASCII data == wrong. Probably best to treat it as UTF-8, probably skipping if it's undecodable.

Also, this is going to be pretty slow. You may need to think about optimising it down to a single query eventually.

334 + results = store.find(
335 + BinaryPackageReleaseContents,
336 + BinaryPackageReleaseContents.binarypackagerelease == bpr.id)
337 + for bprc in results:
338 + store.remove(bprc)

You can't just call remove() on the resultset?

362 + def test_get_or_create(self):

Method name style should be preserved here. So, test_getOrCreate.

405 + deb = open(datadir('pmount_0.9.7-2ubuntu2_amd64.deb'), 'r')
406 + lfa = self.factory.makeLibraryFileAlias(...

Read more...

review: Needs Fixing (code)
Revision history for this message
William Grant (wgrant) wrote :

IBPP's docstring is a lie (seems copied from IBPRC).

Also, try using NamedTemporaryFile as a context manager. Bit clearer and shorter.

review: Approve (code)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
The diff is not available at this time. You can reload the page or download it.

Subscribers

People subscribed via source and target branches

to status/vote changes: