Launchpad itself

Merge lp://staging/~al-maisan/launchpad/xx-next-builder-502927 into lp://staging/launchpad

  1. xx-next-builder-502927
  2. Merge into devel
Proposed by Muharem Hrnjadovic
Status: Merged
Merged at revision: not available
Proposed branch: lp://staging/~al-maisan/launchpad/xx-next-builder-502927
Merge into: lp://staging/launchpad
Diff against target: 759 lines (+524/-111)
2 files modified
lib/lp/soyuz/model/buildqueue.py (+81/-0)
lib/lp/soyuz/tests/test_buildqueue.py (+443/-111)
To merge this branch: bzr merge lp://staging/~al-maisan/launchpad/xx-next-builder-502927
Reviewer Review Type Date Requested Status
Graham Binns (community) code Approve
Review via email: mp+16896@code.staging.launchpad.net
To post a comment you must log in.
Revision history for this message
Muharem Hrnjadovic (al-maisan) wrote :

Hello there!

This is the next step in getting us to a point where we can estimate build
farm job dispatch times irrespective of job type (binary build, "create source
package from recipe" job, translation job).

The dispatch time estimation requires us to know when the next builder capable
of running the job of interest (JOI) will become available. The branch at hand
adds that logic.

Tests to run:

    bin/test -vv -t test_buildqueue

No "make lint" errors or warnings.

Revision history for this message
Graham Binns (gmb) wrote :
Download full text (27.8 KiB)

Hi Muharem,

Great branch this; I'm really impressed with the whole thing. Couple of
minor nitpicks but otherwise this is r=me.

> === modified file 'lib/lp/soyuz/model/buildqueue.py'
> --- lib/lp/soyuz/model/buildqueue.py 2010-01-04 14:30:52 +0000
> +++ lib/lp/soyuz/model/buildqueue.py 2010-01-06 14:11:14 +0000
> @@ -212,6 +212,83 @@
> free_builders = result_set.get_one()[0]
> return free_builders
>
> + def _estimateTimeToNextBuilder(
> + self, head_job_processor, head_job_virtualized):
> + """Estimate time until next builder becomes available.
> +
> + For the purpose of estimating the dispatch time of the job of interest
> + (JOI) we need to know how long it will take until the job at the head
> + of JOI's queue is dispatched.
> +
> + There are two cases to consider here: the head job is
> +
> + - processor dependent: only builders with the matching
> + processor/virtualization combination should be considered.
> + - *not* processor dependent: all builders should be considered.
> +
> + :param head_job_processor: The processor required by the job at the
> + head of the queue.
> + :param head_job_virtualized: The virtualization setting required by
> + the job at the head of the queue.
> + :return: The estimated number of seconds untils a builder capable of
> + running the head job becomes available or None if no such builder
> + exists.
> + """
> + store = getUtility(IStoreSelector).get(MAIN_STORE, DEFAULT_FLAVOR)
> +
> + # First check whether we have free builders.
> + free_builders = self._freeBuildersCount(
> + head_job_processor, head_job_virtualized)
> +
> + if free_builders > 0:
> + # We have free builders for the given processor/virtualization
> + # combination -> zero delay
> + return 0
> +
> + extra_clauses = ''
> + if head_job_processor is not None:
> + # Only look at builders with specific processor types.
> + extra_clauses += """
> + AND Builder.processor = %s
> + AND Builder.virtualized = %s
> + """ % sqlvalues(head_job_processor, head_job_virtualized)
> +
> + params = sqlvalues(JobStatus.RUNNING) + (extra_clauses,)
> +
> + delay_query = """
> + SELECT MIN(
> + CASE WHEN
> + EXTRACT(EPOCH FROM
> + (BuildQueue.estimated_duration -
> + (((now() AT TIME ZONE 'UTC') - Job.date_started)))) >= 0
> + THEN
> + EXTRACT(EPOCH FROM
> + (BuildQueue.estimated_duration -
> + (((now() AT TIME ZONE 'UTC') - Job.date_started))))
> + ELSE
> + -- Assume that jobs that have overdrawn their estimated
> + -- duration time budget will complete within 2 minutes.
> + -- This is a wild guess but has worked well so far.

As discussed on IRC, this comment should indicate that nothing will
actually break if this wild guess i...

review: Approve (code)
Revision history for this message
Muharem Hrnjadovic (al-maisan) wrote :
Download full text (28.8 KiB)

Graham Binns wrote:
> Review: Approve code
> Hi Muharem,
>
> Great branch this; I'm really impressed with the whole thing. Couple of
> minor nitpicks but otherwise this is r=me.

Hello Graham,

thank you for being so courteous and reviewing this branch on request.
I have revised the branch to accommodate all your suggestions. Please
find the incremental diff enclosed.

>
>> === modified file 'lib/lp/soyuz/model/buildqueue.py'
>> --- lib/lp/soyuz/model/buildqueue.py 2010-01-04 14:30:52 +0000
>> +++ lib/lp/soyuz/model/buildqueue.py 2010-01-06 14:11:14 +0000
>> @@ -212,6 +212,83 @@
>> free_builders = result_set.get_one()[0]
>> return free_builders
>>
>> + def _estimateTimeToNextBuilder(
>> + self, head_job_processor, head_job_virtualized):
>> + """Estimate time until next builder becomes available.
>> +
>> + For the purpose of estimating the dispatch time of the job of interest
>> + (JOI) we need to know how long it will take until the job at the head
>> + of JOI's queue is dispatched.
>> +
>> + There are two cases to consider here: the head job is
>> +
>> + - processor dependent: only builders with the matching
>> + processor/virtualization combination should be considered.
>> + - *not* processor dependent: all builders should be considered.
>> +
>> + :param head_job_processor: The processor required by the job at the
>> + head of the queue.
>> + :param head_job_virtualized: The virtualization setting required by
>> + the job at the head of the queue.
>> + :return: The estimated number of seconds untils a builder capable of
>> + running the head job becomes available or None if no such builder
>> + exists.
>> + """
>> + store = getUtility(IStoreSelector).get(MAIN_STORE, DEFAULT_FLAVOR)
>> +
>> + # First check whether we have free builders.
>> + free_builders = self._freeBuildersCount(
>> + head_job_processor, head_job_virtualized)
>> +
>> + if free_builders > 0:
>> + # We have free builders for the given processor/virtualization
>> + # combination -> zero delay
>> + return 0
>> +
>> + extra_clauses = ''
>> + if head_job_processor is not None:
>> + # Only look at builders with specific processor types.
>> + extra_clauses += """
>> + AND Builder.processor = %s
>> + AND Builder.virtualized = %s
>> + """ % sqlvalues(head_job_processor, head_job_virtualized)
>> +
>> + params = sqlvalues(JobStatus.RUNNING) + (extra_clauses,)
>> +
>> + delay_query = """
>> + SELECT MIN(
>> + CASE WHEN
>> + EXTRACT(EPOCH FROM
>> + (BuildQueue.estimated_duration -
>> + (((now() AT TIME ZONE 'UTC') - Job.date_started)))) >= 0
>> + THEN
>> + EXTRACT(EPOCH FROM
>> + (BuildQueue.estimated_duration -
>> + (((now() AT TIME ZONE 'UTC') - Job.date_started))))
>> + ELSE
>> + ...

xx-next-builder-502927-inc.diff
1=== modified file 'lib/lp/soyuz/model/buildqueue.py'
2--- lib/lp/soyuz/model/buildqueue.py 2010-01-06 14:08:31 +0000
3+++ lib/lp/soyuz/model/buildqueue.py 2010-01-06 15:49:24 +0000
4@@ -269,6 +269,10 @@
5 -- Assume that jobs that have overdrawn their estimated
6 -- duration time budget will complete within 2 minutes.
7 -- This is a wild guess but has worked well so far.
8+ --
9+ -- Please note that this is entirely innocuous i.e. if our
10+ -- guess is off nothing bad will happen but our estimate will
11+ -- not be as good as it could be.
12 120
13 END)
14 FROM
15
16=== modified file 'lib/lp/soyuz/tests/test_buildqueue.py'
17--- lib/lp/soyuz/tests/test_buildqueue.py 2010-01-06 14:08:31 +0000
18+++ lib/lp/soyuz/tests/test_buildqueue.py 2010-01-06 15:51:51 +0000
19@@ -154,21 +154,21 @@
20 processor_fam = ProcessorFamilySet().getByName('x86')
21 x86_proc = processor_fam.processors[0]
22 # x86 native
23- self.builders[(x86_proc.id,False)] = [
24+ self.builders[(x86_proc.id, False)] = [
25 self.i6, self.i7, self.i8, self.i9]
26 # x86 virtual
27- self.builders[(x86_proc.id,True)] = [
28+ self.builders[(x86_proc.id, True)] = [
29 self.i1, self.i2, self.i3, self.i4, self.i5]
30
31 # amd64 native
32- self.builders[(amd_proc.id,False)] = [self.a4, self.a5]
33+ self.builders[(amd_proc.id, False)] = [self.a4, self.a5]
34 # amd64 virtual
35- self.builders[(amd_proc.id,True)] = [self.a1, self.a2, self.a3]
36+ self.builders[(amd_proc.id, True)] = [self.a1, self.a2, self.a3]
37
38 # hppa native
39- self.builders[(hppa_proc.id,False)] = [self.h5, self.h6, self.h7]
40+ self.builders[(hppa_proc.id, False)] = [self.h5, self.h6, self.h7]
41 # hppa virtual
42- self.builders[(hppa_proc.id,True)] = [
43+ self.builders[(hppa_proc.id, True)] = [
44 self.h1, self.h2, self.h3, self.h4]
45
46 # Ensure all builders are operational.
47@@ -183,9 +183,8 @@
48
49
50 class SingleArchBuildsBase(TestBuildQueueBase):
51- """Test the retrieval of builder related data. The latter is required
52- for job dispatch time estimations irrespective of job processor
53- architecture and virtualization setting."""
54+ """Set up a test environment with builds that target a single
55+ processor."""
56 def setUp(self):
57 """Set up some native x86 builds for the test archive."""
58 super(SingleArchBuildsBase, self).setUp()
59@@ -280,29 +279,29 @@
60 processor_fam = ProcessorFamilySet().getByName('x86')
61 x86_proc = processor_fam.processors[0]
62 self.assertEqual(
63- builder_stats[(x86_proc.id,False)], 4,
64+ builder_stats[(x86_proc.id, False)], 4,
65 "The number of native x86 builders is wrong")
66 self.assertEqual(
67- builder_stats[(x86_proc.id,True)], 5,
68+ builder_stats[(x86_proc.id, True)], 5,
69 "The number of virtual x86 builders is wrong")
70 processor_fam = ProcessorFamilySet().getByName('amd64')
71 amd_proc = processor_fam.processors[0]
72 self.assertEqual(
73- builder_stats[(amd_proc.id,False)], 2,
74+ builder_stats[(amd_proc.id, False)], 2,
75 "The number of native amd64 builders is wrong")
76 self.assertEqual(
77- builder_stats[(amd_proc.id,True)], 3,
78+ builder_stats[(amd_proc.id, True)], 3,
79 "The number of virtual amd64 builders is wrong")
80 processor_fam = ProcessorFamilySet().getByName('hppa')
81 hppa_proc = processor_fam.processors[0]
82 self.assertEqual(
83- builder_stats[(hppa_proc.id,False)], 3,
84+ builder_stats[(hppa_proc.id, False)], 3,
85 "The number of native hppa builders is wrong")
86 self.assertEqual(
87- builder_stats[(hppa_proc.id,True)], 4,
88+ builder_stats[(hppa_proc.id, True)], 4,
89 "The number of virtual hppa builders is wrong")
90 # Disable the native x86 builders.
91- for builder in self.builders[(x86_proc.id,False)]:
92+ for builder in self.builders[(x86_proc.id, False)]:
93 builder.builderok = False
94 # Get the builder statistics again.
95 builder_data = bq._getBuilderData()
96@@ -314,7 +313,7 @@
97 "[2] The total number of builders that can build the job in "
98 "question is wrong.")
99 # Re-enable one of them.
100- for builder in self.builders[(x86_proc.id,False)]:
101+ for builder in self.builders[(x86_proc.id, False)]:
102 builder.builderok = True
103 break
104 # Get the builder statistics again.
105@@ -327,7 +326,7 @@
106 "question is wrong.")
107 # Disable the *virtual* x86 builders -- should not make any
108 # difference.
109- for builder in self.builders[(x86_proc.id,True)]:
110+ for builder in self.builders[(x86_proc.id, True)]:
111 builder.builderok = False
112 # Get the builder statistics again.
113 builder_data = bq._getBuilderData()
114@@ -352,7 +351,7 @@
115 builders_in_total, builders_for_job, builder_stats = builder_data
116 # We have 4 x86 native builders.
117 self.assertEqual(
118- builder_stats[(proc_386.id,False)], 4,
119+ builder_stats[(proc_386.id, False)], 4,
120 "The number of native x86 builders is wrong")
121 # Initially all 4 builders are free.
122 free_count = bq._freeBuildersCount(
123@@ -390,9 +389,11 @@
124
125
126 class TestMinTimeToNextBuilder(SingleArchBuildsBase):
127- """When is the next builder capable of running a given job becoming
128- available?"""
129+ """Test estimated time-to-builder with builds targetting a single
130+ processor."""
131 def test_min_time_to_next_builder(self):
132+ """When is the next builder capable of running the job at the head of
133+ the queue becoming available?"""
134 # Test the estimation of the minimum time until a builder becomes
135 # available.
136
137@@ -457,7 +458,7 @@
138 check_mintime_to_builder(self, apg_job, x86_proc, False, 30)
139
140 # Disable the native x86 builders.
141- for builder in self.builders[(1,False)]:
142+ for builder in self.builders[(x86_proc.id, False)]:
143 builder.builderok = False
144
145 # No builders capable of running the job at hand are available now,
146@@ -466,10 +467,7 @@
147
148
149 class MultiArchBuildsBase(TestBuildQueueBase):
150- """Test dispatch time estimates for binary builds (i.e. single build
151- farm job type) targetting a single processor architecture and the primary
152- archive.
153- """
154+ """Set up a test environment with builds and multiple processors."""
155 def setUp(self):
156 """Set up some native x86 builds for the test archive."""
157 super(MultiArchBuildsBase, self).setUp()
158@@ -562,9 +560,10 @@
159
160
161 class TestMinTimeToNextBuilderMulti(MultiArchBuildsBase):
162- """When is the next builder capable of running a given job becoming
163- available?"""
164+ """Test estimated time-to-builder with builds and multiple processors."""
165 def test_min_time_to_next_builder(self):
166+ """When is the next builder capable of running the job at the head of
167+ the queue becoming available?"""
168 processor_fam = ProcessorFamilySet().getByName('hppa')
169 hppa_proc = processor_fam.processors[0]
170
171@@ -610,7 +609,7 @@
172 check_mintime_to_builder(self, apg_job, hppa_proc, False, 30)
173
174 # Disable the native hppa builders.
175- for builder in self.builders[(hppa_proc.id,False)]:
176+ for builder in self.builders[(hppa_proc.id, False)]:
177 builder.builderok = False
178
179 # No builders capable of running the job at hand are available now,
180@@ -636,7 +635,7 @@
181 check_mintime_to_builder(self, apg_job, None, None, None)
182
183 # Re-enable the native hppa builders.
184- for builder in self.builders[(hppa_proc.id,False)]:
185+ for builder in self.builders[(hppa_proc.id, False)]:
186 builder.builderok = True
187
188 # The builder that's becoming available next is the one that's
189@@ -646,7 +645,7 @@
190 # Make sure we'll find an x86 builder as well.
191 processor_fam = ProcessorFamilySet().getByName('x86')
192 x86_proc = processor_fam.processors[0]
193- builder = self.builders[(x86_proc.id,False)][0]
194+ builder = self.builders[(x86_proc.id, False)][0]
195 builder.builderok = True
196
197 # Now this builder is the one that becomes available next (29 minutes
198@@ -658,7 +657,7 @@
199 check_mintime_to_builder(self, apg_job, None, None, 29)
200
201 # Make a second, idle x86 builder available.
202- builder = self.builders[(x86_proc.id,False)][1]
203+ builder = self.builders[(x86_proc.id, False)][1]
204 builder.builderok = True
205
206 # That builder should be available immediately since it's idle.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'lib/lp/soyuz/model/buildqueue.py'
2--- lib/lp/soyuz/model/buildqueue.py 2010-01-04 14:30:52 +0000
3+++ lib/lp/soyuz/model/buildqueue.py 2010-01-06 16:03:24 +0000
4@@ -212,6 +212,87 @@
5 free_builders = result_set.get_one()[0]
6 return free_builders
7
8+ def _estimateTimeToNextBuilder(
9+ self, head_job_processor, head_job_virtualized):
10+ """Estimate time until next builder becomes available.
11+
12+ For the purpose of estimating the dispatch time of the job of interest
13+ (JOI) we need to know how long it will take until the job at the head
14+ of JOI's queue is dispatched.
15+
16+ There are two cases to consider here: the head job is
17+
18+ - processor dependent: only builders with the matching
19+ processor/virtualization combination should be considered.
20+ - *not* processor dependent: all builders should be considered.
21+
22+ :param head_job_processor: The processor required by the job at the
23+ head of the queue.
24+ :param head_job_virtualized: The virtualization setting required by
25+ the job at the head of the queue.
26+ :return: The estimated number of seconds untils a builder capable of
27+ running the head job becomes available or None if no such builder
28+ exists.
29+ """
30+ store = getUtility(IStoreSelector).get(MAIN_STORE, DEFAULT_FLAVOR)
31+
32+ # First check whether we have free builders.
33+ free_builders = self._freeBuildersCount(
34+ head_job_processor, head_job_virtualized)
35+
36+ if free_builders > 0:
37+ # We have free builders for the given processor/virtualization
38+ # combination -> zero delay
39+ return 0
40+
41+ extra_clauses = ''
42+ if head_job_processor is not None:
43+ # Only look at builders with specific processor types.
44+ extra_clauses += """
45+ AND Builder.processor = %s
46+ AND Builder.virtualized = %s
47+ """ % sqlvalues(head_job_processor, head_job_virtualized)
48+
49+ params = sqlvalues(JobStatus.RUNNING) + (extra_clauses,)
50+
51+ delay_query = """
52+ SELECT MIN(
53+ CASE WHEN
54+ EXTRACT(EPOCH FROM
55+ (BuildQueue.estimated_duration -
56+ (((now() AT TIME ZONE 'UTC') - Job.date_started)))) >= 0
57+ THEN
58+ EXTRACT(EPOCH FROM
59+ (BuildQueue.estimated_duration -
60+ (((now() AT TIME ZONE 'UTC') - Job.date_started))))
61+ ELSE
62+ -- Assume that jobs that have overdrawn their estimated
63+ -- duration time budget will complete within 2 minutes.
64+ -- This is a wild guess but has worked well so far.
65+ --
66+ -- Please note that this is entirely innocuous i.e. if our
67+ -- guess is off nothing bad will happen but our estimate will
68+ -- not be as good as it could be.
69+ 120
70+ END)
71+ FROM
72+ BuildQueue, Job, Builder
73+ WHERE
74+ BuildQueue.job = Job.id
75+ AND BuildQueue.builder = Builder.id
76+ AND Builder.manual = False
77+ AND Builder.builderok = True
78+ AND Job.status = %s
79+ %s
80+ """ % params
81+
82+ result_set = store.execute(delay_query)
83+ head_job_delay = result_set.get_one()[0]
84+ if head_job_delay is None:
85+ return None
86+ else:
87+ return int(head_job_delay)
88+
89
90 class BuildQueueSet(object):
91 """Utility to deal with BuildQueue content class."""
92
93=== modified file 'lib/lp/soyuz/tests/test_buildqueue.py'
94--- lib/lp/soyuz/tests/test_buildqueue.py 2010-01-04 10:58:09 +0000
95+++ lib/lp/soyuz/tests/test_buildqueue.py 2010-01-06 16:03:24 +0000
96@@ -4,7 +4,8 @@
97
98 """Test BuildQueue features."""
99
100-from datetime import timedelta
101+from datetime import datetime, timedelta
102+from pytz import utc
103
104 from zope.component import getUtility
105
106@@ -33,8 +34,6 @@
107
108 def nth_builder(test, build, n):
109 """Find nth builder that can execute the given build."""
110- def builder_key(build):
111- return (build.processor.id,build.is_virtualized)
112 builder = None
113 builders = test.builders.get(builder_key(build), [])
114 try:
115@@ -44,9 +43,9 @@
116 return builder
117
118
119-def assign_to_builder(test, job_name, builder_number):
120+def assign_to_builder(test, job_name, builder_number, processor='386'):
121 """Simulate assigning a build to a builder."""
122- build, bq = find_job(test, job_name)
123+ build, bq = find_job(test, job_name, processor)
124 builder = nth_builder(test, build, builder_number)
125 bq.markAsBuilding(builder)
126
127@@ -61,6 +60,43 @@
128 bq.estimated_duration, bq.lastscore)
129
130
131+def builder_key(job):
132+ """Access key for builders capable of running the given job."""
133+ return (job.processor.id, job.is_virtualized)
134+
135+
136+def check_mintime_to_builder(
137+ test, bq, head_job_processor, head_job_virtualized, min_time):
138+ """Test the estimated time until a builder becomes available."""
139+ delay = bq._estimateTimeToNextBuilder(
140+ head_job_processor, head_job_virtualized)
141+ if min_time is not None:
142+ test.assertTrue(
143+ almost_equal(delay, min_time),
144+ "Wrong min time to next available builder (%s != %s)"
145+ % (delay, min_time))
146+ else:
147+ test.assertTrue(delay is None, "No delay to next builder available")
148+
149+
150+def almost_equal(a, b, deviation=1):
151+ """Compare the values tolerating the given deviation.
152+
153+ This used to spurious failures in time based tests.
154+ """
155+ if abs(a - b) <= deviation:
156+ return True
157+ else:
158+ return False
159+
160+
161+def set_remaining_time_for_running_job(bq, remainder):
162+ """Set remaining running time for job."""
163+ offset = bq.estimated_duration.seconds - remainder
164+ bq.setDateStarted(
165+ datetime.utcnow().replace(tzinfo=utc) - timedelta(seconds=offset))
166+
167+
168 class TestBuildQueueBase(TestCaseWithFactory):
169 """Setup the test publisher and some builders."""
170
171@@ -84,45 +120,56 @@
172
173 # Next make seven 'hppa' builders.
174 processor_fam = ProcessorFamilySet().getByName('hppa')
175- proc = processor_fam.processors[0]
176- self.h1 = self.factory.makeBuilder(name='hppa-v-1', processor=proc)
177- self.h2 = self.factory.makeBuilder(name='hppa-v-2', processor=proc)
178- self.h3 = self.factory.makeBuilder(name='hppa-v-3', processor=proc)
179- self.h4 = self.factory.makeBuilder(name='hppa-v-4', processor=proc)
180+ hppa_proc = processor_fam.processors[0]
181+ self.h1 = self.factory.makeBuilder(
182+ name='hppa-v-1', processor=hppa_proc)
183+ self.h2 = self.factory.makeBuilder(
184+ name='hppa-v-2', processor=hppa_proc)
185+ self.h3 = self.factory.makeBuilder(
186+ name='hppa-v-3', processor=hppa_proc)
187+ self.h4 = self.factory.makeBuilder(
188+ name='hppa-v-4', processor=hppa_proc)
189 self.h5 = self.factory.makeBuilder(
190- name='hppa-n-5', processor=proc, virtualized=False)
191+ name='hppa-n-5', processor=hppa_proc, virtualized=False)
192 self.h6 = self.factory.makeBuilder(
193- name='hppa-n-6', processor=proc, virtualized=False)
194+ name='hppa-n-6', processor=hppa_proc, virtualized=False)
195 self.h7 = self.factory.makeBuilder(
196- name='hppa-n-7', processor=proc, virtualized=False)
197+ name='hppa-n-7', processor=hppa_proc, virtualized=False)
198
199 # Finally make five 'amd64' builders.
200 processor_fam = ProcessorFamilySet().getByName('amd64')
201- proc = processor_fam.processors[0]
202- self.a1 = self.factory.makeBuilder(name='amd64-v-1', processor=proc)
203- self.a2 = self.factory.makeBuilder(name='amd64-v-2', processor=proc)
204- self.a3 = self.factory.makeBuilder(name='amd64-v-3', processor=proc)
205+ amd_proc = processor_fam.processors[0]
206+ self.a1 = self.factory.makeBuilder(
207+ name='amd64-v-1', processor=amd_proc)
208+ self.a2 = self.factory.makeBuilder(
209+ name='amd64-v-2', processor=amd_proc)
210+ self.a3 = self.factory.makeBuilder(
211+ name='amd64-v-3', processor=amd_proc)
212 self.a4 = self.factory.makeBuilder(
213- name='amd64-n-4', processor=proc, virtualized=False)
214+ name='amd64-n-4', processor=amd_proc, virtualized=False)
215 self.a5 = self.factory.makeBuilder(
216- name='amd64-n-5', processor=proc, virtualized=False)
217+ name='amd64-n-5', processor=amd_proc, virtualized=False)
218
219 self.builders = dict()
220+ processor_fam = ProcessorFamilySet().getByName('x86')
221+ x86_proc = processor_fam.processors[0]
222 # x86 native
223- self.builders[(1,False)] = [self.i6, self.i7, self.i8, self.i9]
224+ self.builders[(x86_proc.id, False)] = [
225+ self.i6, self.i7, self.i8, self.i9]
226 # x86 virtual
227- self.builders[(1,True)] = [
228+ self.builders[(x86_proc.id, True)] = [
229 self.i1, self.i2, self.i3, self.i4, self.i5]
230
231 # amd64 native
232- self.builders[(2,True)] = [self.a4, self.a5]
233+ self.builders[(amd_proc.id, False)] = [self.a4, self.a5]
234 # amd64 virtual
235- self.builders[(2,False)] = [self.a1, self.a2, self.a3]
236+ self.builders[(amd_proc.id, True)] = [self.a1, self.a2, self.a3]
237
238 # hppa native
239- self.builders[(3,True)] = [self.h5, self.h6, self.h7]
240+ self.builders[(hppa_proc.id, False)] = [self.h5, self.h6, self.h7]
241 # hppa virtual
242- self.builders[(3,False)] = [self.h1, self.h2, self.h3, self.h4]
243+ self.builders[(hppa_proc.id, True)] = [
244+ self.h1, self.h2, self.h3, self.h4]
245
246 # Ensure all builders are operational.
247 for builders in self.builders.values():
248@@ -135,85 +182,88 @@
249 getUtility(IBuilderSet)['frog'].builderok = False
250
251
252-class TestBuilderData(TestBuildQueueBase):
253+class SingleArchBuildsBase(TestBuildQueueBase):
254+ """Set up a test environment with builds that target a single
255+ processor."""
256+ def setUp(self):
257+ """Set up some native x86 builds for the test archive."""
258+ super(SingleArchBuildsBase, self).setUp()
259+ # The builds will be set up as follows:
260+ #
261+ # gedit, p: 386, v:False e:0:01:00 *** s: 1001
262+ # firefox, p: 386, v:False e:0:02:00 *** s: 1002
263+ # apg, p: 386, v:False e:0:03:00 *** s: 1003
264+ # vim, p: 386, v:False e:0:04:00 *** s: 1004
265+ # gcc, p: 386, v:False e:0:05:00 *** s: 1005
266+ # bison, p: 386, v:False e:0:06:00 *** s: 1006
267+ # flex, p: 386, v:False e:0:07:00 *** s: 1007
268+ # postgres, p: 386, v:False e:0:08:00 *** s: 1008
269+ #
270+ # p=processor, v=virtualized, e=estimated_duration, s=score
271+
272+ # First mark all builds in the sample data as already built.
273+ store = getUtility(IStoreSelector).get(MAIN_STORE, DEFAULT_FLAVOR)
274+ sample_data = store.find(Build)
275+ for build in sample_data:
276+ build.buildstate = BuildStatus.FULLYBUILT
277+ store.flush()
278+
279+ # We test builds that target a primary archive.
280+ self.non_ppa = self.factory.makeArchive(
281+ name="primary", purpose=ArchivePurpose.PRIMARY)
282+ self.non_ppa.require_virtualized = False
283+
284+ self.builds = []
285+ self.builds.extend(
286+ self.publisher.getPubSource(
287+ sourcename="gedit", status=PackagePublishingStatus.PUBLISHED,
288+ archive=self.non_ppa).createMissingBuilds())
289+ self.builds.extend(
290+ self.publisher.getPubSource(
291+ sourcename="firefox",
292+ status=PackagePublishingStatus.PUBLISHED,
293+ archive=self.non_ppa).createMissingBuilds())
294+ self.builds.extend(
295+ self.publisher.getPubSource(
296+ sourcename="apg", status=PackagePublishingStatus.PUBLISHED,
297+ archive=self.non_ppa).createMissingBuilds())
298+ self.builds.extend(
299+ self.publisher.getPubSource(
300+ sourcename="vim", status=PackagePublishingStatus.PUBLISHED,
301+ archive=self.non_ppa).createMissingBuilds())
302+ self.builds.extend(
303+ self.publisher.getPubSource(
304+ sourcename="gcc", status=PackagePublishingStatus.PUBLISHED,
305+ archive=self.non_ppa).createMissingBuilds())
306+ self.builds.extend(
307+ self.publisher.getPubSource(
308+ sourcename="bison", status=PackagePublishingStatus.PUBLISHED,
309+ archive=self.non_ppa).createMissingBuilds())
310+ self.builds.extend(
311+ self.publisher.getPubSource(
312+ sourcename="flex", status=PackagePublishingStatus.PUBLISHED,
313+ archive=self.non_ppa).createMissingBuilds())
314+ self.builds.extend(
315+ self.publisher.getPubSource(
316+ sourcename="postgres",
317+ status=PackagePublishingStatus.PUBLISHED,
318+ archive=self.non_ppa).createMissingBuilds())
319+ # Set up the builds for test.
320+ score = 1000
321+ duration = 0
322+ for build in self.builds:
323+ score += 1
324+ duration += 60
325+ bq = build.buildqueue_record
326+ bq.lastscore = score
327+ bq.estimated_duration = timedelta(seconds=duration)
328+ # print_build_setup(self.builds)
329+
330+
331+class TestBuilderData(SingleArchBuildsBase):
332 """Test the retrieval of builder related data. The latter is required
333 for job dispatch time estimations irrespective of job processor
334 architecture and virtualization setting."""
335-
336- def setUp(self):
337- """Set up some native x86 builds for the test archive."""
338- super(TestBuilderData, self).setUp()
339- # The builds will be set up as follows:
340- #
341- # gedit, p: 386, v:False e:0:01:00 *** s: 1001
342- # firefox, p: 386, v:False e:0:02:00 *** s: 1002
343- # apg, p: 386, v:False e:0:03:00 *** s: 1003
344- # vim, p: 386, v:False e:0:04:00 *** s: 1004
345- # gcc, p: 386, v:False e:0:05:00 *** s: 1005
346- # bison, p: 386, v:False e:0:06:00 *** s: 1006
347- # flex, p: 386, v:False e:0:07:00 *** s: 1007
348- # postgres, p: 386, v:False e:0:08:00 *** s: 1008
349- #
350- # p=processor, v=virtualized, e=estimated_duration, s=score
351-
352- # First mark all builds in the sample data as already built.
353- store = getUtility(IStoreSelector).get(MAIN_STORE, DEFAULT_FLAVOR)
354- sample_data = store.find(Build)
355- for build in sample_data:
356- build.buildstate = BuildStatus.FULLYBUILT
357- store.flush()
358-
359- # We test builds that target a primary archive.
360- self.non_ppa = self.factory.makeArchive(
361- name="primary", purpose=ArchivePurpose.PRIMARY)
362- self.non_ppa.require_virtualized = False
363-
364- self.builds = []
365- self.builds.extend(
366- self.publisher.getPubSource(
367- sourcename="gedit", status=PackagePublishingStatus.PUBLISHED,
368- archive=self.non_ppa).createMissingBuilds())
369- self.builds.extend(
370- self.publisher.getPubSource(
371- sourcename="firefox",
372- status=PackagePublishingStatus.PUBLISHED,
373- archive=self.non_ppa).createMissingBuilds())
374- self.builds.extend(
375- self.publisher.getPubSource(
376- sourcename="apg", status=PackagePublishingStatus.PUBLISHED,
377- archive=self.non_ppa).createMissingBuilds())
378- self.builds.extend(
379- self.publisher.getPubSource(
380- sourcename="vim", status=PackagePublishingStatus.PUBLISHED,
381- archive=self.non_ppa).createMissingBuilds())
382- self.builds.extend(
383- self.publisher.getPubSource(
384- sourcename="gcc", status=PackagePublishingStatus.PUBLISHED,
385- archive=self.non_ppa).createMissingBuilds())
386- self.builds.extend(
387- self.publisher.getPubSource(
388- sourcename="bison", status=PackagePublishingStatus.PUBLISHED,
389- archive=self.non_ppa).createMissingBuilds())
390- self.builds.extend(
391- self.publisher.getPubSource(
392- sourcename="flex", status=PackagePublishingStatus.PUBLISHED,
393- archive=self.non_ppa).createMissingBuilds())
394- self.builds.extend(
395- self.publisher.getPubSource(
396- sourcename="postgres",
397- status=PackagePublishingStatus.PUBLISHED,
398- archive=self.non_ppa).createMissingBuilds())
399- # Set up the builds for test.
400- score = 1000
401- duration = 0
402- for build in self.builds:
403- score += 1
404- duration += 60
405- bq = build.buildqueue_record
406- bq.lastscore = score
407- bq.estimated_duration = timedelta(seconds=duration)
408- # print_build_setup(self.builds)
409-
410 def test_builder_data(self):
411 # Make sure the builder numbers are correct. The builder data will
412 # be the same for all of our builds.
413@@ -226,26 +276,32 @@
414 builders_for_job, 4,
415 "[1] The total number of builders that can build the job in "
416 "question is wrong.")
417+ processor_fam = ProcessorFamilySet().getByName('x86')
418+ x86_proc = processor_fam.processors[0]
419 self.assertEqual(
420- builder_stats[(1,False)], 4,
421+ builder_stats[(x86_proc.id, False)], 4,
422 "The number of native x86 builders is wrong")
423 self.assertEqual(
424- builder_stats[(1,True)], 5,
425+ builder_stats[(x86_proc.id, True)], 5,
426 "The number of virtual x86 builders is wrong")
427+ processor_fam = ProcessorFamilySet().getByName('amd64')
428+ amd_proc = processor_fam.processors[0]
429 self.assertEqual(
430- builder_stats[(2,False)], 2,
431+ builder_stats[(amd_proc.id, False)], 2,
432 "The number of native amd64 builders is wrong")
433 self.assertEqual(
434- builder_stats[(2,True)], 3,
435+ builder_stats[(amd_proc.id, True)], 3,
436 "The number of virtual amd64 builders is wrong")
437+ processor_fam = ProcessorFamilySet().getByName('hppa')
438+ hppa_proc = processor_fam.processors[0]
439 self.assertEqual(
440- builder_stats[(3,False)], 3,
441+ builder_stats[(hppa_proc.id, False)], 3,
442 "The number of native hppa builders is wrong")
443 self.assertEqual(
444- builder_stats[(3,True)], 4,
445+ builder_stats[(hppa_proc.id, True)], 4,
446 "The number of virtual hppa builders is wrong")
447 # Disable the native x86 builders.
448- for builder in self.builders[(1,False)]:
449+ for builder in self.builders[(x86_proc.id, False)]:
450 builder.builderok = False
451 # Get the builder statistics again.
452 builder_data = bq._getBuilderData()
453@@ -257,7 +313,7 @@
454 "[2] The total number of builders that can build the job in "
455 "question is wrong.")
456 # Re-enable one of them.
457- for builder in self.builders[(1,False)]:
458+ for builder in self.builders[(x86_proc.id, False)]:
459 builder.builderok = True
460 break
461 # Get the builder statistics again.
462@@ -270,7 +326,7 @@
463 "question is wrong.")
464 # Disable the *virtual* x86 builders -- should not make any
465 # difference.
466- for builder in self.builders[(1,True)]:
467+ for builder in self.builders[(x86_proc.id, True)]:
468 builder.builderok = False
469 # Get the builder statistics again.
470 builder_data = bq._getBuilderData()
471@@ -295,7 +351,7 @@
472 builders_in_total, builders_for_job, builder_stats = builder_data
473 # We have 4 x86 native builders.
474 self.assertEqual(
475- builder_stats[(proc_386.id,False)], 4,
476+ builder_stats[(proc_386.id, False)], 4,
477 "The number of native x86 builders is wrong")
478 # Initially all 4 builders are free.
479 free_count = bq._freeBuildersCount(
480@@ -330,3 +386,279 @@
481 free_count = bq._freeBuildersCount(
482 build.processor, build.is_virtualized)
483 self.assertEqual(free_count, 1)
484+
485+
486+class TestMinTimeToNextBuilder(SingleArchBuildsBase):
487+ """Test estimated time-to-builder with builds targetting a single
488+ processor."""
489+ def test_min_time_to_next_builder(self):
490+ """When is the next builder capable of running the job at the head of
491+ the queue becoming available?"""
492+ # Test the estimation of the minimum time until a builder becomes
493+ # available.
494+
495+ # The builds will be set up as follows:
496+ #
497+ # gedit, p: 386, v:False e:0:01:00 *** s: 1001
498+ # firefox, p: 386, v:False e:0:02:00 *** s: 1002
499+ # apg, p: 386, v:False e:0:03:00 *** s: 1003
500+ # vim, p: 386, v:False e:0:04:00 *** s: 1004
501+ # gcc, p: 386, v:False e:0:05:00 *** s: 1005
502+ # bison, p: 386, v:False e:0:06:00 *** s: 1006
503+ # flex, p: 386, v:False e:0:07:00 *** s: 1007
504+ # postgres, p: 386, v:False e:0:08:00 *** s: 1008
505+ #
506+ # p=processor, v=virtualized, e=estimated_duration, s=score
507+
508+ processor_fam = ProcessorFamilySet().getByName('x86')
509+ x86_proc = processor_fam.processors[0]
510+ # This will be the job of interest.
511+ apg_build, apg_job = find_job(self, 'apg')
512+ # One of four builders for the 'apg' build is immediately available.
513+ check_mintime_to_builder(self, apg_job, x86_proc, False, 0)
514+
515+ # Assign the postgres job to a builder.
516+ assign_to_builder(self, 'postgres', 1)
517+ # Now one builder is gone. But there should still be a builder
518+ # immediately available.
519+ check_mintime_to_builder(self, apg_job, x86_proc, False, 0)
520+
521+ assign_to_builder(self, 'flex', 2)
522+ check_mintime_to_builder(self, apg_job, x86_proc, False, 0)
523+
524+ assign_to_builder(self, 'bison', 3)
525+ check_mintime_to_builder(self, apg_job, x86_proc, False, 0)
526+
527+ assign_to_builder(self, 'gcc', 4)
528+ # Now that no builder is immediately available, the shortest
529+ # remaing build time (based on the estimated duration) is returned:
530+ # 300 seconds
531+ # This is equivalent to the 'gcc' job's estimated duration.
532+ check_mintime_to_builder(self, apg_job, x86_proc, False, 300)
533+
534+ # Now we pretend that the 'postgres' started 6 minutes ago. Its
535+ # remaining execution time should be 2 minutes = 120 seconds and
536+ # it now becomes the job whose builder becomes available next.
537+ build, bq = find_job(self, 'postgres')
538+ set_remaining_time_for_running_job(bq, 120)
539+ check_mintime_to_builder(self, apg_job, x86_proc, False, 120)
540+
541+ # What happens when jobs overdraw the estimated duration? Let's
542+ # pretend the 'flex' job started 8 minutes ago.
543+ build, bq = find_job(self, 'flex')
544+ set_remaining_time_for_running_job(bq, -60)
545+ # In such a case we assume that the job will complete within 2
546+ # minutes, this is a guess that has worked well so far.
547+ check_mintime_to_builder(self, apg_job, x86_proc, False, 120)
548+
549+ # If there's a job that will complete within a shorter time then
550+ # we expect to be given that time frame.
551+ build, bq = find_job(self, 'postgres')
552+ set_remaining_time_for_running_job(bq, 30)
553+ check_mintime_to_builder(self, apg_job, x86_proc, False, 30)
554+
555+ # Disable the native x86 builders.
556+ for builder in self.builders[(x86_proc.id, False)]:
557+ builder.builderok = False
558+
559+ # No builders capable of running the job at hand are available now,
560+ # this is indicated by a None value.
561+ check_mintime_to_builder(self, apg_job, x86_proc, False, None)
562+
563+
564+class MultiArchBuildsBase(TestBuildQueueBase):
565+ """Set up a test environment with builds and multiple processors."""
566+ def setUp(self):
567+ """Set up some native x86 builds for the test archive."""
568+ super(MultiArchBuildsBase, self).setUp()
569+ # The builds will be set up as follows:
570+ #
571+ # gedit, p: hppa, v:False e:0:01:00 *** s: 1001
572+ # gedit, p: 386, v:False e:0:02:00 *** s: 1002
573+ # firefox, p: hppa, v:False e:0:03:00 *** s: 1003
574+ # firefox, p: 386, v:False e:0:04:00 *** s: 1004
575+ # apg, p: hppa, v:False e:0:05:00 *** s: 1005
576+ # apg, p: 386, v:False e:0:06:00 *** s: 1006
577+ # vim, p: hppa, v:False e:0:07:00 *** s: 1007
578+ # vim, p: 386, v:False e:0:08:00 *** s: 1008
579+ # gcc, p: hppa, v:False e:0:09:00 *** s: 1009
580+ # gcc, p: 386, v:False e:0:10:00 *** s: 1010
581+ # bison, p: hppa, v:False e:0:11:00 *** s: 1011
582+ # bison, p: 386, v:False e:0:12:00 *** s: 1012
583+ # flex, p: hppa, v:False e:0:13:00 *** s: 1013
584+ # flex, p: 386, v:False e:0:14:00 *** s: 1014
585+ # postgres, p: hppa, v:False e:0:15:00 *** s: 1015
586+ # postgres, p: 386, v:False e:0:16:00 *** s: 1016
587+ #
588+ # p=processor, v=virtualized, e=estimated_duration, s=score
589+
590+ # First mark all builds in the sample data as already built.
591+ store = getUtility(IStoreSelector).get(MAIN_STORE, DEFAULT_FLAVOR)
592+ sample_data = store.find(Build)
593+ for build in sample_data:
594+ build.buildstate = BuildStatus.FULLYBUILT
595+ store.flush()
596+
597+ # We test builds that target a primary archive.
598+ self.non_ppa = self.factory.makeArchive(
599+ name="primary", purpose=ArchivePurpose.PRIMARY)
600+ self.non_ppa.require_virtualized = False
601+
602+ self.builds = []
603+ self.builds.extend(
604+ self.publisher.getPubSource(
605+ sourcename="gedit", status=PackagePublishingStatus.PUBLISHED,
606+ archive=self.non_ppa,
607+ architecturehintlist='any').createMissingBuilds())
608+ self.builds.extend(
609+ self.publisher.getPubSource(
610+ sourcename="firefox",
611+ status=PackagePublishingStatus.PUBLISHED,
612+ archive=self.non_ppa,
613+ architecturehintlist='any').createMissingBuilds())
614+ self.builds.extend(
615+ self.publisher.getPubSource(
616+ sourcename="apg", status=PackagePublishingStatus.PUBLISHED,
617+ archive=self.non_ppa,
618+ architecturehintlist='any').createMissingBuilds())
619+ self.builds.extend(
620+ self.publisher.getPubSource(
621+ sourcename="vim", status=PackagePublishingStatus.PUBLISHED,
622+ archive=self.non_ppa,
623+ architecturehintlist='any').createMissingBuilds())
624+ self.builds.extend(
625+ self.publisher.getPubSource(
626+ sourcename="gcc", status=PackagePublishingStatus.PUBLISHED,
627+ archive=self.non_ppa,
628+ architecturehintlist='any').createMissingBuilds())
629+ self.builds.extend(
630+ self.publisher.getPubSource(
631+ sourcename="bison", status=PackagePublishingStatus.PUBLISHED,
632+ archive=self.non_ppa,
633+ architecturehintlist='any').createMissingBuilds())
634+ self.builds.extend(
635+ self.publisher.getPubSource(
636+ sourcename="flex", status=PackagePublishingStatus.PUBLISHED,
637+ archive=self.non_ppa,
638+ architecturehintlist='any').createMissingBuilds())
639+ self.builds.extend(
640+ self.publisher.getPubSource(
641+ sourcename="postgres",
642+ status=PackagePublishingStatus.PUBLISHED,
643+ archive=self.non_ppa,
644+ architecturehintlist='any').createMissingBuilds())
645+ # Set up the builds for test.
646+ score = 1000
647+ duration = 0
648+ for build in self.builds:
649+ score += 1
650+ duration += 60
651+ bq = build.buildqueue_record
652+ bq.lastscore = score
653+ bq.estimated_duration = timedelta(seconds=duration)
654+ # print_build_setup(self.builds)
655+
656+
657+class TestMinTimeToNextBuilderMulti(MultiArchBuildsBase):
658+ """Test estimated time-to-builder with builds and multiple processors."""
659+ def test_min_time_to_next_builder(self):
660+ """When is the next builder capable of running the job at the head of
661+ the queue becoming available?"""
662+ processor_fam = ProcessorFamilySet().getByName('hppa')
663+ hppa_proc = processor_fam.processors[0]
664+
665+ # One of four builders for the 'apg' build is immediately available.
666+ apg_build, apg_job = find_job(self, 'apg', 'hppa')
667+ check_mintime_to_builder(self, apg_job, hppa_proc, False, 0)
668+
669+ # Assign the postgres job to a builder.
670+ assign_to_builder(self, 'postgres', 1, 'hppa')
671+ # Now one builder is gone. But there should still be a builder
672+ # immediately available.
673+ check_mintime_to_builder(self, apg_job, hppa_proc, False, 0)
674+
675+ assign_to_builder(self, 'flex', 2, 'hppa')
676+ check_mintime_to_builder(self, apg_job, hppa_proc, False, 0)
677+
678+ assign_to_builder(self, 'bison', 3, 'hppa')
679+ # Now that no builder is immediately available, the shortest
680+ # remaing build time (based on the estimated duration) is returned:
681+ # 660 seconds
682+ # This is equivalent to the 'bison' job's estimated duration.
683+ check_mintime_to_builder(self, apg_job, hppa_proc, False, 660)
684+
685+ # Now we pretend that the 'postgres' started 13 minutes ago. Its
686+ # remaining execution time should be 2 minutes = 120 seconds and
687+ # it now becomes the job whose builder becomes available next.
688+ build, bq = find_job(self, 'postgres', 'hppa')
689+ set_remaining_time_for_running_job(bq, 120)
690+ check_mintime_to_builder(self, apg_job, hppa_proc, False, 120)
691+
692+ # What happens when jobs overdraw the estimated duration? Let's
693+ # pretend the 'flex' job started 14 minutes ago.
694+ build, bq = find_job(self, 'flex', 'hppa')
695+ set_remaining_time_for_running_job(bq, -60)
696+ # In such a case we assume that the job will complete within 2
697+ # minutes, this is a guess that has worked well so far.
698+ check_mintime_to_builder(self, apg_job, hppa_proc, False, 120)
699+
700+ # If there's a job that will complete within a shorter time then
701+ # we expect to be given that time frame.
702+ build, bq = find_job(self, 'postgres', 'hppa')
703+ set_remaining_time_for_running_job(bq, 30)
704+ check_mintime_to_builder(self, apg_job, hppa_proc, False, 30)
705+
706+ # Disable the native hppa builders.
707+ for builder in self.builders[(hppa_proc.id, False)]:
708+ builder.builderok = False
709+
710+ # No builders capable of running the job at hand are available now,
711+ # this is indicated by a None value.
712+ check_mintime_to_builder(self, apg_job, hppa_proc, False, None)
713+
714+ # Let's assume for the moment that the job at the head of the 'apg'
715+ # build queue is processor independent. In that case we'd ask for
716+ # *any* next available builder.
717+ self.assertTrue(
718+ bq._freeBuildersCount(None, None) > 0,
719+ "Builders are immediately available for jobs that don't care "
720+ "about processor architectures or virtualization")
721+ check_mintime_to_builder(self, apg_job, None, None, 0)
722+
723+ # Let's disable all builders.
724+ for builders in self.builders.itervalues():
725+ for builder in builders:
726+ builder.builderok = False
727+
728+ # There are no builders capable of running even the processor
729+ # independent jobs now and that this is indicated by a None value.
730+ check_mintime_to_builder(self, apg_job, None, None, None)
731+
732+ # Re-enable the native hppa builders.
733+ for builder in self.builders[(hppa_proc.id, False)]:
734+ builder.builderok = True
735+
736+ # The builder that's becoming available next is the one that's
737+ # running the 'postgres' build.
738+ check_mintime_to_builder(self, apg_job, None, None, 30)
739+
740+ # Make sure we'll find an x86 builder as well.
741+ processor_fam = ProcessorFamilySet().getByName('x86')
742+ x86_proc = processor_fam.processors[0]
743+ builder = self.builders[(x86_proc.id, False)][0]
744+ builder.builderok = True
745+
746+ # Now this builder is the one that becomes available next (29 minutes
747+ # remaining build time).
748+ assign_to_builder(self, 'gcc', 1, '386')
749+ build, bq = find_job(self, 'gcc', '386')
750+ set_remaining_time_for_running_job(bq, 29)
751+
752+ check_mintime_to_builder(self, apg_job, None, None, 29)
753+
754+ # Make a second, idle x86 builder available.
755+ builder = self.builders[(x86_proc.id, False)][1]
756+ builder.builderok = True
757+
758+ # That builder should be available immediately since it's idle.
759+ check_mintime_to_builder(self, apg_job, None, None, 0)