Merge lp://staging/~canonical-platform-qa/ubuntu-power-tests/flight-mode-test into lp://staging/ubuntu-power-tests

Is 20 seconds long enough? Is that what the other tests are using?

I'm not sure. I am hoping Max can clarify that requirement.

PASSED: Continuous integration, rev:10
http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-ci/10/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-vivid-amd64-ci/10
    SUCCESS: http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-vivid-armhf-ci/10
    SUCCESS: http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-vivid-i386-ci/10

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-power-tests-ci/10/rebuild

As it is, this won't work. We need to have flight mode left on when the test ends so that we can disconnect, settle the system, then collect the meter data. I'll try it with just the first command.

I guess that opens the question of whether the other tests should be responsible for ensuring that all of the radios are on to be consistent with previous test data. I'm not sure if I should file a bug for that or just discuss making it a card.

So I tried adding a tc-powermeter-06-cleanup test, but if flight mode is on, adt-run fails. Should we report this as a bug against autopkgtest, or try something else to work around it?

It could be possible to do something here using the --setup-commands option, where we enable/disable flight mode using that.

From what you say it sounds like the test needs to be completely disconnected from adt-run? My only other thought was to run a much longer timeout in the test script which would allow the system settle time so that the test can run fully from the script, but it sounds like this may not be possible.

Some more investigation shows adt-run will work in offline mode. It only needs the connection enabled to install the dependencies, in this case fakeroot.

I tried manually installing fakeroot to the image before running the tests offline and this worked fine. But this requires making the image writable, so is not ideal.

I'm not sure why fakeroot is needed for these tests, but if we could remove this dependency at least for the offline tests it might resolve this issue.

How do we enable offline mode? Since we do run the test online initially, is there some way to preserve that installation of fakeroot?

Also, it's been a bit since I checked, can other team members run test.sh to test this without the meter? I think things should still be separated enough for that to work, but I'd like to confirm that.

Sorry for confusion, what I meant by offline mode was just that the device had flight mode enabled.

The dependencies get installed to a unique folder in /tmp for each test run, so there is no preservation of dependencies between test runs, that I know of.

Okay. What commands do you use to run adt-run with flight mode on? If we
can make that work, I think my current cleanup method can work.

I've given this some more thought. I think the best solution is to use a separate script to prepare the device, e.g. to disable flight mode. (using the same dbus command used in the script.)

This script could then be run before a test using --setup-commands option:

adt-run --setup-commands ./debian/tests/prepare-device --testname tc-powermeter-06 -B --unbuilt-tree=. --- ssh -s adb -- -p 0000

This means that the device should always be able to download required dependencies and run a test, regardless of the state the device is left in. You could also expand the script to configure BT + WiFi to the required state too if required.

What do you think?

FWIW I know that pitti has in the past strongly discouraged using setup-commands for doing any part of the actual test. It should only be used to get the test bed ready for testing. I suppose in this case it sounds like that is what it is doing so it's probably ok

Yes, in this case I think it is the best solution.

By keeping the commands in a separate script file rather than adding them directly to adt-run command, they are at least more visible and easier to maintain along with the other scripts.

We would of course need to make sure the documentation is updated as well.

PASSED: Continuous integration, rev:11
http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-ci/11/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-vivid-amd64-ci/11
    SUCCESS: http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-vivid-armhf-ci/11
    SUCCESS: http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-vivid-i386-ci/11

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-power-tests-ci/11/rebuild

PASSED: Continuous integration, rev:12
http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-ci/13/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-vivid-amd64-ci/13
    SUCCESS: http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-vivid-armhf-ci/13
    SUCCESS: http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-vivid-i386-ci/13

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-power-tests-ci/13/rebuild

This works for me. I think we should consider having the preparation script also ensure bluetooth is on, since that's default test behavior, but I don't think that's necessarily needed for this landing. I'll top-approve, and I can merge, or somebody else can. Thanks!

Oh shoot, actually, there is one more thing we need here. I had to change:
TEST_COMMAND="adt-run --setup-commands ./debian/tests/prepare-device --testname ${TEST} -B --unbuilt-tree=${BRANCHDIR} --output-dir ${OUTPUTDIR} --- ssh -s adb -- -p ${DEVICE_PASSWORD}"
to
TEST_COMMAND="adt-run --setup-commands ${BRANCHDIR}/debian/tests/prepare-device --testname ${TEST} -B --unbuilt-tree=${BRANCHDIR} --output-dir ${OUTPUTDIR} --- ssh -s adb -- -p ${DEVICE_PASSWORD}"
to get things working. That branch lives at lp:~nuclearbob/ubuntu-power-tests/flight-mode-test-with-adt-command so we can merge that one instead of that makes more sense, or I can push it to the ~canonical-platform-qa one.

Okay, I see it's already merged. We can look at merging that later. I'm still working on tc-powermeter-07, so maybe those can land together.