Merge lp://staging/~doanac/lava-android-test/restricted-coremark into lp://staging/lava-android-test

I think this looks fine to me, except we should add some documentation for the test (Yongqin has a branch in progress for this right now to add it to the others... it's pretty simple). I think at least the docstring in the test itself explaining what it is, and definitely mentioning that it does not include the coremark binaries which need to be obtained from somewhere else.

I cc'd the rest of the team and would like to get them to weigh in on this as well. The big issue that I'm concerned about I think is that it looks pretty inconvenient to have to create the binaries for coremark, host them somewhere, make sure we only run this in a private job once the ability to do that exists, and create all the jobs pointing to those binaries. I'd love it if lava already had a build step built into it that would accomodate something like this internally, but it doesn't right now. It might be worth considering if we could somehow handle the build part of this through Jenkins in the same style that we do ci testing. Something for you to think about I guess.

In any case, I wouldn't actually recommend running a job that uses this in lava until we land and install the rest of the privacy changes so that you can:
1. specify a private bundle stream to push the results to
2. make sure that nobody sees your user:password who is not in linaro - this will still be in the job log, and even the job json, but if the job is specifically marked private, it won't be visible to anyone outside of linaro.

On 03/07/2012 10:43 PM, Paul Larson wrote:
> I think this looks fine to me, except we should add some documentation for the test (Yongqin has a branch in progress for this right now to add it to the others... it's pretty simple). I think at least the docstring in the test itself explaining what it is, and definitely mentioning that it does not include the coremark binaries which need to be obtained from somewhere else.

sounds good.

> I cc'd the rest of the team and would like to get them to weigh in on this as well. The big issue that I'm concerned about I think is that it looks pretty inconvenient to have to create the binaries for coremark, host them somewhere, make sure we only run this in a private job once the ability to do that exists, and create all the jobs pointing to those binaries. I'd love it if lava already had a build step built into it that would accomodate something like this internally, but it doesn't right now. It might be worth considering if we could somehow handle the build part of this through Jenkins in the same style that we do ci testing. Something for you to think about I guess.

Its extremely inconvenient but that seems to be the nature of dealing
with these benchmark suites. The changing URL is pretty much a must,
because the actual coremark binary will always be changing.

It could be done from Jenkins. However, I'd have to create special
Android build job just for this, because the code base isn't in a public
git repository which Android requires. So that solution isn't really
trivial either.

I just discovered this won't work because AOSP builds don't include busybox. I have to come up with a new way for this test to be done.