Code review comment for lp://staging/~doanac/lava-android-test/restricted-coremark

On 03/07/2012 10:43 PM, Paul Larson wrote:
> I think this looks fine to me, except we should add some documentation for the test (Yongqin has a branch in progress for this right now to add it to the others... it's pretty simple). I think at least the docstring in the test itself explaining what it is, and definitely mentioning that it does not include the coremark binaries which need to be obtained from somewhere else.

sounds good.

> I cc'd the rest of the team and would like to get them to weigh in on this as well. The big issue that I'm concerned about I think is that it looks pretty inconvenient to have to create the binaries for coremark, host them somewhere, make sure we only run this in a private job once the ability to do that exists, and create all the jobs pointing to those binaries. I'd love it if lava already had a build step built into it that would accomodate something like this internally, but it doesn't right now. It might be worth considering if we could somehow handle the build part of this through Jenkins in the same style that we do ci testing. Something for you to think about I guess.

Its extremely inconvenient but that seems to be the nature of dealing
with these benchmark suites. The changing URL is pretty much a must,
because the actual coremark binary will always be changing.

It could be done from Jenkins. However, I'd have to create special
Android build job just for this, because the code base isn't in a public
git repository which Android requires. So that solution isn't really
trivial either.