Launchpad itself

Merge lp://staging/~julian-edwards/launchpad/publish-copy-archives-bug-520520 into lp://staging/launchpad

  1. publish-copy-archives-bug-520520
  2. Merge into devel
Proposed by Julian Edwards
Status: Merged
Merged at revision: not available
Proposed branch: lp://staging/~julian-edwards/launchpad/publish-copy-archives-bug-520520
Merge into: lp://staging/launchpad
Diff against target: 289 lines (+71/-64)
2 files modified
lib/lp/soyuz/adapters/archivedependencies.py (+35/-28)
lib/lp/soyuz/doc/archive-dependencies.txt (+36/-36)
To merge this branch: bzr merge lp://staging/~julian-edwards/launchpad/publish-copy-archives-bug-520520
Reviewer Review Type Date Requested Status
Paul Hummer (community) code Approve
Review via email: mp+19621@code.staging.launchpad.net

Commit message

Change the ordering of the dependencies for builds to something that makes more sense.

To post a comment you must log in.
Revision history for this message
Julian Edwards (julian-edwards) wrote :

= Summary =
Change the ordering of the dependencies for builds to something that makes
more sense.

== Proposed fix ==
When we send builds to the build slaves, we tell the chroot exactly what
sources.list it needs to install before pulling in build dependencies. The
ordering of that file has a bearing on what packages get installed if two
repos have the same version of something.

This branch makes that ordering sensible and determinate.

== Pre-implementation notes ==
This is the first part of the fix for bug 520520.

== Implementation details ==
The entries are returned in the order that is most useful;
  1. the context archive itself
  2. external dependencies
  3. user-selected archive dependencies
  4. the default primary archive

This is a simple fix in the get_sources_list_for_building() function.

== Tests ==
bin/test -cvvt archive-dependencies.txt

== Demo and Q/A ==

= Launchpad lint =

Checking for conflicts. and issues in doctests and templates.
Running jslint, xmllint, pyflakes, and pylint.
Using normal rules.

Linting changed files:
  lib/lp/soyuz/doc/archive-dependencies.txt
  lib/lp/soyuz/adapters/archivedependencies.py

== Pylint notices ==

lib/lp/soyuz/adapters/archivedependencies.py
    43: [F0401] Unable to import 'lazr.uri' (No module named uri)

Revision history for this message
Paul Hummer (rockstar) wrote :

This all looks good. Thanks for the branch!

review: Approve (code)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'lib/lp/soyuz/adapters/archivedependencies.py'
--- lib/lp/soyuz/adapters/archivedependencies.py 2010-01-15 04:03:26 +0000
+++ lib/lp/soyuz/adapters/archivedependencies.py 2010-02-18 15:35:24 +0000
@@ -119,23 +119,40 @@
119def get_sources_list_for_building(build, distroarchseries, sourcepackagename):119def get_sources_list_for_building(build, distroarchseries, sourcepackagename):
120 """Return the sources_list entries required to build the given item.120 """Return the sources_list entries required to build the given item.
121121
122 The entries are returned in the order that is most useful;
123 1. the context archive itself
124 2. external dependencies
125 3. user-selected archive dependencies
126 4. the default primary archive
127
122 :param build: a context `IBuild`.128 :param build: a context `IBuild`.
123 :param distroarchseries: A `IDistroArchSeries`129 :param distroarchseries: A `IDistroArchSeries`
124 :param sourcepackagename: A source package name (as text)130 :param sourcepackagename: A source package name (as text)
125 :return: a deb sources_list entries (lines).131 :return: a deb sources_list entries (lines).
126 """132 """
127 deps = []133 deps = []
128134 sources_list_lines = []
129 # Consider primary archive dependency override. Add the default135
130 # primary archive dependencies if it's not present.136 # Add implicit self-dependency for non-primary contexts.
131 if build.archive.getArchiveDependency(137 if build.archive.purpose in ALLOW_RELEASE_BUILDS:
132 build.archive.distribution.main_archive) is None:138 self_dep = [(
133 primary_dependencies = _get_default_primary_dependencies(build)139 build.archive, PackagePublishingPocket.RELEASE,
134 deps.extend(primary_dependencies)140 get_components_for_building(build))]
141 sources_list_lines = _get_sources_list_for_dependencies(
142 self_dep, distroarchseries)
143
144 # Append external sources_list lines for this archive if it's
145 # specified in the configuration.
146 dependencies = build.archive.external_dependencies
147 if dependencies is not None:
148 for archive_dep in dependencies.splitlines():
149 line = archive_dep % (
150 {'series': distroarchseries.distroseries.name})
151 sources_list_lines.append(line)
135152
136 # Consider user-selected archive dependencies.153 # Consider user-selected archive dependencies.
137 primary_component = get_primary_current_component(build.archive, 154 primary_component = get_primary_current_component(
138 build.distroseries, sourcepackagename)155 build.archive, build.distroseries, sourcepackagename)
139 for archive_dependency in build.archive.dependencies:156 for archive_dependency in build.archive.dependencies:
140 # When the dependency component is undefined, we should use157 # When the dependency component is undefined, we should use
141 # the component where the source is published in the primary158 # the component where the source is published in the primary
@@ -151,25 +168,15 @@
151 (archive_dependency.dependency, pocket, components)168 (archive_dependency.dependency, pocket, components)
152 )169 )
153170
154 # Add implicit self-dependency for non-primary contexts.171 # Consider primary archive dependency override. Add the default
155 if build.archive.purpose in ALLOW_RELEASE_BUILDS:172 # primary archive dependencies if it's not present.
156 deps.append(173 if build.archive.getArchiveDependency(
157 (build.archive, PackagePublishingPocket.RELEASE,174 build.archive.distribution.main_archive) is None:
158 get_components_for_building(build))175 primary_dependencies = _get_default_primary_dependencies(build)
159 )176 deps.extend(primary_dependencies)
160177
161 sources_list_lines = _get_sources_list_for_dependencies(178 sources_list_lines.extend(
162 deps, distroarchseries)179 _get_sources_list_for_dependencies(deps, distroarchseries))
163
164 # Append external sources_list lines for this archive if it's
165 # specified in the configuration.
166 dependencies = build.archive.external_dependencies
167 if dependencies is not None:
168 for archive_dep in dependencies.splitlines():
169 line = archive_dep % (
170 {'series': distroarchseries.distroseries.name})
171 sources_list_lines.append(line)
172
173 return sources_list_lines180 return sources_list_lines
174181
175def _has_published_binaries(archive, distroarchseries, pocket):182def _has_published_binaries(archive, distroarchseries, pocket):
176183
=== modified file 'lib/lp/soyuz/doc/archive-dependencies.txt'
--- lib/lp/soyuz/doc/archive-dependencies.txt 2010-01-16 09:41:17 +0000
+++ lib/lp/soyuz/doc/archive-dependencies.txt 2010-02-18 15:35:24 +0000
@@ -207,7 +207,7 @@
207 >>> def print_building_sources_list(candidate):207 >>> def print_building_sources_list(candidate):
208 ... sources_list = get_sources_list_for_building(candidate,208 ... sources_list = get_sources_list_for_building(candidate,
209 ... candidate.distroarchseries, candidate.sourcepackagerelease.name)209 ... candidate.distroarchseries, candidate.sourcepackagerelease.name)
210 ... for line in sorted(sources_list):210 ... for line in sources_list:
211 ... print line211 ... print line
212212
213Note that only the default ubuntu dependencies for a public PPA will be213Note that only the default ubuntu dependencies for a public PPA will be
@@ -240,13 +240,13 @@
240 ... status=PackagePublishingStatus.PUBLISHED)240 ... status=PackagePublishingStatus.PUBLISHED)
241241
242 >>> print_building_sources_list(a_build)242 >>> print_building_sources_list(a_build)
243 deb http://ppa.launchpad.dev/cprov/ppa/ubuntu hoary main
243 deb http://ftpmaster.internal/ubuntu hoary244 deb http://ftpmaster.internal/ubuntu hoary
244 main restricted universe multiverse245 main restricted universe multiverse
245 deb http://ftpmaster.internal/ubuntu hoary-security246 deb http://ftpmaster.internal/ubuntu hoary-security
246 main restricted universe multiverse247 main restricted universe multiverse
247 deb http://ftpmaster.internal/ubuntu hoary-updates248 deb http://ftpmaster.internal/ubuntu hoary-updates
248 main restricted universe multiverse249 main restricted universe multiverse
249 deb http://ppa.launchpad.dev/cprov/ppa/ubuntu hoary main
250250
251Similarly, populated PPA dependencies are listed in the building251Similarly, populated PPA dependencies are listed in the building
252'sources_list'.252'sources_list'.
@@ -261,14 +261,14 @@
261 ... getUtility(IComponentSet)['main'])261 ... getUtility(IComponentSet)['main'])
262262
263 >>> print_building_sources_list(a_build)263 >>> print_building_sources_list(a_build)
264 deb http://ftpmaster.internal/ubuntu hoary
265 main restricted universe multiverse
266 deb http://ftpmaster.internal/ubuntu hoary-security
267 main restricted universe multiverse
268 deb http://ftpmaster.internal/ubuntu hoary-updates
269 main restricted universe multiverse
270 deb http://ppa.launchpad.dev/cprov/ppa/ubuntu hoary main264 deb http://ppa.launchpad.dev/cprov/ppa/ubuntu hoary main
271 deb http://ppa.launchpad.dev/mark/ppa/ubuntu hoary main265 deb http://ppa.launchpad.dev/mark/ppa/ubuntu hoary main
266 deb http://ftpmaster.internal/ubuntu hoary
267 main restricted universe multiverse
268 deb http://ftpmaster.internal/ubuntu hoary-security
269 main restricted universe multiverse
270 deb http://ftpmaster.internal/ubuntu hoary-updates
271 main restricted universe multiverse
272272
273The authentication information gets added for private PPA273The authentication information gets added for private PPA
274dependencies.274dependencies.
@@ -277,6 +277,7 @@
277 >>> mark.archive.private = True277 >>> mark.archive.private = True
278278
279 >>> print_building_sources_list(a_build)279 >>> print_building_sources_list(a_build)
280 deb http://ppa.launchpad.dev/cprov/ppa/ubuntu hoary main
280 deb http://buildd:not-so-secret@private-ppa.launchpad.dev/mark/ppa/ubuntu281 deb http://buildd:not-so-secret@private-ppa.launchpad.dev/mark/ppa/ubuntu
281 hoary main282 hoary main
282 deb http://ftpmaster.internal/ubuntu hoary283 deb http://ftpmaster.internal/ubuntu hoary
@@ -285,7 +286,6 @@
285 main restricted universe multiverse286 main restricted universe multiverse
286 deb http://ftpmaster.internal/ubuntu hoary-updates287 deb http://ftpmaster.internal/ubuntu hoary-updates
287 main restricted universe multiverse288 main restricted universe multiverse
288 deb http://ppa.launchpad.dev/cprov/ppa/ubuntu hoary main
289289
290Good enough, let's delete the archive dependency on Mark's PPA.290Good enough, let's delete the archive dependency on Mark's PPA.
291291
@@ -306,13 +306,13 @@
306dependencies.306dependencies.
307307
308 >>> print_building_sources_list(a_build)308 >>> print_building_sources_list(a_build)
309 deb http://ppa.launchpad.dev/cprov/ppa/ubuntu hoary main
309 deb http://ftpmaster.internal/ubuntu hoary310 deb http://ftpmaster.internal/ubuntu hoary
310 main restricted universe multiverse311 main restricted universe multiverse
311 deb http://ftpmaster.internal/ubuntu hoary-security312 deb http://ftpmaster.internal/ubuntu hoary-security
312 main restricted universe multiverse313 main restricted universe multiverse
313 deb http://ftpmaster.internal/ubuntu hoary-updates314 deb http://ftpmaster.internal/ubuntu hoary-updates
314 main restricted universe multiverse315 main restricted universe multiverse
315 deb http://ppa.launchpad.dev/cprov/ppa/ubuntu hoary main
316316
317The default build behaviour will remain unchanged when we override the317The default build behaviour will remain unchanged when we override the
318default primary archive dependencies with exactly the same values.318default primary archive dependencies with exactly the same values.
@@ -322,13 +322,13 @@
322 ... getUtility(IComponentSet)['multiverse'])322 ... getUtility(IComponentSet)['multiverse'])
323323
324 >>> print_building_sources_list(a_build)324 >>> print_building_sources_list(a_build)
325 deb http://ppa.launchpad.dev/cprov/ppa/ubuntu hoary main
325 deb http://ftpmaster.internal/ubuntu hoary326 deb http://ftpmaster.internal/ubuntu hoary
326 main restricted universe multiverse327 main restricted universe multiverse
327 deb http://ftpmaster.internal/ubuntu hoary-security328 deb http://ftpmaster.internal/ubuntu hoary-security
328 main restricted universe multiverse329 main restricted universe multiverse
329 deb http://ftpmaster.internal/ubuntu hoary-updates330 deb http://ftpmaster.internal/ubuntu hoary-updates
330 main restricted universe multiverse331 main restricted universe multiverse
331 deb http://ppa.launchpad.dev/cprov/ppa/ubuntu hoary main
332332
333 >>> cprov.archive.removeArchiveDependency(ubuntu.main_archive)333 >>> cprov.archive.removeArchiveDependency(ubuntu.main_archive)
334334
@@ -349,11 +349,11 @@
349 universe349 universe
350350
351 >>> print_building_sources_list(a_build)351 >>> print_building_sources_list(a_build)
352 deb http://ppa.launchpad.dev/cprov/ppa/ubuntu hoary main
352 deb http://ftpmaster.internal/ubuntu hoary353 deb http://ftpmaster.internal/ubuntu hoary
353 main universe354 main universe
354 deb http://ftpmaster.internal/ubuntu hoary-security355 deb http://ftpmaster.internal/ubuntu hoary-security
355 main universe356 main universe
356 deb http://ppa.launchpad.dev/cprov/ppa/ubuntu hoary main
357357
358 >>> cprov.archive.removeArchiveDependency(ubuntu.main_archive)358 >>> cprov.archive.removeArchiveDependency(ubuntu.main_archive)
359359
@@ -366,8 +366,8 @@
366 ... getUtility(IComponentSet)['restricted'])366 ... getUtility(IComponentSet)['restricted'])
367367
368 >>> print_building_sources_list(a_build)368 >>> print_building_sources_list(a_build)
369 deb http://ppa.launchpad.dev/cprov/ppa/ubuntu hoary main
369 deb http://ftpmaster.internal/ubuntu hoary main restricted370 deb http://ftpmaster.internal/ubuntu hoary main restricted
370 deb http://ppa.launchpad.dev/cprov/ppa/ubuntu hoary main
371371
372 >>> cprov.archive.removeArchiveDependency(ubuntu.main_archive)372 >>> cprov.archive.removeArchiveDependency(ubuntu.main_archive)
373373
@@ -379,15 +379,15 @@
379 ... getUtility(IComponentSet)['multiverse'])379 ... getUtility(IComponentSet)['multiverse'])
380380
381 >>> print_building_sources_list(a_build)381 >>> print_building_sources_list(a_build)
382 deb http://ppa.launchpad.dev/cprov/ppa/ubuntu hoary main
382 deb http://ftpmaster.internal/ubuntu hoary383 deb http://ftpmaster.internal/ubuntu hoary
383 main restricted universe multiverse384 main restricted universe multiverse
385 deb http://ftpmaster.internal/ubuntu hoary-security
386 main restricted universe multiverse
387 deb http://ftpmaster.internal/ubuntu hoary-updates
388 main restricted universe multiverse
384 deb http://ftpmaster.internal/ubuntu hoary-proposed389 deb http://ftpmaster.internal/ubuntu hoary-proposed
385 main restricted universe multiverse390 main restricted universe multiverse
386 deb http://ftpmaster.internal/ubuntu hoary-security
387 main restricted universe multiverse
388 deb http://ftpmaster.internal/ubuntu hoary-updates
389 main restricted universe multiverse
390 deb http://ppa.launchpad.dev/cprov/ppa/ubuntu hoary main
391391
392 >>> cprov.archive.removeArchiveDependency(ubuntu.main_archive)392 >>> cprov.archive.removeArchiveDependency(ubuntu.main_archive)
393393
@@ -398,15 +398,15 @@
398 ... getUtility(IComponentSet)['multiverse'])398 ... getUtility(IComponentSet)['multiverse'])
399399
400 >>> print_building_sources_list(a_build)400 >>> print_building_sources_list(a_build)
401 deb http://ppa.launchpad.dev/cprov/ppa/ubuntu hoary main
401 deb http://ftpmaster.internal/ubuntu hoary402 deb http://ftpmaster.internal/ubuntu hoary
402 main restricted universe multiverse403 main restricted universe multiverse
404 deb http://ftpmaster.internal/ubuntu hoary-security
405 main restricted universe multiverse
406 deb http://ftpmaster.internal/ubuntu hoary-updates
407 main restricted universe multiverse
403 deb http://ftpmaster.internal/ubuntu hoary-backports408 deb http://ftpmaster.internal/ubuntu hoary-backports
404 main restricted universe multiverse409 main restricted universe multiverse
405 deb http://ftpmaster.internal/ubuntu hoary-security
406 main restricted universe multiverse
407 deb http://ftpmaster.internal/ubuntu hoary-updates
408 main restricted universe multiverse
409 deb http://ppa.launchpad.dev/cprov/ppa/ubuntu hoary main
410410
411 >>> cprov.archive.removeArchiveDependency(ubuntu.main_archive)411 >>> cprov.archive.removeArchiveDependency(ubuntu.main_archive)
412412
@@ -432,13 +432,13 @@
432 >>> [partner_build] = pub_source.createMissingBuilds()432 >>> [partner_build] = pub_source.createMissingBuilds()
433433
434 >>> print_building_sources_list(partner_build)434 >>> print_building_sources_list(partner_build)
435 deb http://ftpmaster.internal/ubuntu hoary
436 main restricted universe multiverse
437 deb http://ftpmaster.internal/ubuntu hoary-security
438 main restricted universe multiverse
439 deb http://ftpmaster.internal/ubuntu hoary-updates
440 main restricted universe multiverse
441 deb http://ftpmaster.internal/ubuntu-partner hoary partner435 deb http://ftpmaster.internal/ubuntu-partner hoary partner
436 deb http://ftpmaster.internal/ubuntu hoary
437 main restricted universe multiverse
438 deb http://ftpmaster.internal/ubuntu hoary-security
439 main restricted universe multiverse
440 deb http://ftpmaster.internal/ubuntu hoary-updates
441 main restricted universe multiverse
442442
443443
444== External build dependencies ==444== External build dependencies ==
@@ -466,14 +466,14 @@
466Now builds in Celso's PPA will use the external dependencies.466Now builds in Celso's PPA will use the external dependencies.
467467
468 >>> print_building_sources_list(a_build)468 >>> print_building_sources_list(a_build)
469 deb http://ftpmaster.internal/ubuntu hoary
470 main restricted universe multiverse
471 deb http://ftpmaster.internal/ubuntu hoary-security
472 main restricted universe multiverse
473 deb http://ftpmaster.internal/ubuntu hoary-updates
474 main restricted universe multiverse
475 deb http://ppa.launchpad.dev/cprov/ppa/ubuntu hoary main469 deb http://ppa.launchpad.dev/cprov/ppa/ubuntu hoary main
470 deb http://user:pass@repository zoing everything
476 deb http://user:pass@repository hoary public private471 deb http://user:pass@repository hoary public private
477 deb http://user:pass@repository hoary-extra public472 deb http://user:pass@repository hoary-extra public
478 deb http://user:pass@repository zoing everything473 deb http://ftpmaster.internal/ubuntu hoary
474 main restricted universe multiverse
475 deb http://ftpmaster.internal/ubuntu hoary-security
476 main restricted universe multiverse
477 deb http://ftpmaster.internal/ubuntu hoary-updates
478 main restricted universe multiverse
479479