Linaro Android Build Tools

Merge lp://staging/~danilo/linaro-android-build-tools/use-http-proxy into lp://staging/linaro-android-build-tools

use-http-proxy
Merge into trunk

Proposed by Данило Шеган on 2012-11-29

Status:	Merged
Approved by:	Paul Sokolovsky on 2012-11-29
Approved revision:	539
Merged at revision:	538
Proposed branch:	lp://staging/~danilo/linaro-android-build-tools/use-http-proxy
Merge into:	lp://staging/linaro-android-build-tools
Diff against target:	31 lines (+13/-0) 2 files modified build-scripts/build-android (+1/-0) build-scripts/helpers (+12/-0)
To merge this branch:	bzr merge lp://staging/~danilo/linaro-android-build-tools/use-http-proxy
Related bugs:	Link a bug report
Related blueprints:	Enable lightweight HTTP access for android.git.linaro.org (High)

Reviewer	Review Type	Date Requested	Status
Paul Sokolovsky		2012-11-29	Approve on 2012-11-29
Review via email: mp+136920@code.staging.launchpad.net

Description of the change

Make use of a http_proxy inside Amazon EC2 us-east zone. Instead of setting
up another squid on android-build.linaro.org, I've decided to simply increase
the disk cache storage for Squid on ci.linaro.org to 30GB (up from 10GB) and
point http_proxy at it.

After a few builds, disk cache usage is at about 16GB on ci.linaro.org (it
is already set up on partition of 400GB).

Revision history for this message

Данило Шеган (danilo) wrote on 2012-11-29:

FWIW, without the seed file we got down to 28 mins for "repo sync" (from 44 mins with seed and no proxy, and 41 for initial run with the proxy) on https://android-build.linaro.org/jenkins/job/danilo_galaxynexus-jb-gcc47-aosp-blob/

Revision history for this message

Paul Sokolovsky (pfalcon) wrote on 2012-11-29:

Changes look good, thumbs up for checking proxy availability first.

But I'm not sure about "repo sync" times - do you have a ling for that. "Standard" seed dl + repo sync time is 20min, I took first random build and see 16min: https://android-build.linaro.org/jenkins/job/linaro-android_panda-ics-gcc47-tilt-tracking-blob/176/parsed_console/? , expand "Info" section:

2 TIME: Seed download and uncompress: 8m44.897s

3 TIME: Repo sync (using seed as reference): 7m37.108s
https://android-build.linaro.org/jenkins/job/linaro-android_panda-ics-gcc47-tilt-tracking-blob/176/parsed_console/?

(warning! that page is huge (100mb), and will certainly crash Firefox and make Chromium thrash swap.)

review: Approve

Revision history for this message

Данило Шеган (danilo) wrote on 2012-11-29:

Thanks for the review Paul.

Now on to your questions.

https://android-build.linaro.org/jenkins/job/danilo_galaxynexus-jb-gcc47-aosp-blob/4/consoleText doesn't show the time as "repo sync (using seed as reference)", but it does output:

real 28m46.952s
user 18m19.650s
sys 6m6.020s

Similarly, no-seed build without proxy (https://android-build.linaro.org/jenkins/job/danilo_galaxynexus-jb-gcc47-aosp-blob/3/consoleText) shows:

real 41m18.811s
user 18m36.370s
sys 6m14.210s

And finally, build with the seed and no proxy (https://android-build.linaro.org/jenkins/job/danilo_galaxynexus-jb-gcc47-aosp-blob/1/consoleText) shows:

TIME: Seed download and uncompress: 8m53.905s
...
TIME: Repo sync (using seed as reference): 44m39.969s

This was all done on the same configuration so results are more comparable. It's quite likely that galaxynexus-jb build is not a good reference point, but I wanted something that's relatively stable and this was just an arbitrary pick (and comparing one to the other is good enough).

This is generally very hard to compare, since it will depend on network conditions, time since seed was updated, and similar, but I wanted to ensure we introduce no big slowdowns, and I think the numbers back that. The point of http proxy is to avoid the need to use the network bandwidth more than anything else.