Merge lp://staging/~toykeeper/ubuntu-power-tests/python-refactor-tk into lp://staging/ubuntu-power-tests

Proposed by Selene ToyKeeper
Status: Merged
Approved by: Selene ToyKeeper
Approved revision: 76
Merged at revision: 15
Proposed branch: lp://staging/~toykeeper/ubuntu-power-tests/python-refactor-tk
Merge into: lp://staging/ubuntu-power-tests
Diff against target: 3006 lines (+2655/-144)
21 files modified
Makefile (+2/-0)
README (+141/-66)
channels.lst (+4/-0)
debian/tests/power_usage_bt_on_wifi_disconnected (+13/-0)
debian/tests/power_usage_display_on (+10/-6)
debian/tests/power_usage_flight_mode_on (+12/-0)
debian/tests/power_usage_gps_app_backgrounded (+8/-0)
debian/tests/power_usage_gps_app_closed (+9/-0)
debian/tests/power_usage_idle (+12/-0)
debian/tests/prepare-device (+17/-0)
devices.lst (+2/-0)
dut-monitor.sh (+34/-0)
flash/arale/urls (+1/-0)
flash/flash.sh (+305/-0)
flash/krillin/urls (+2/-0)
power-graph.py (+180/-0)
power.cfg.example (+27/-0)
power.py (+1137/-72)
run-in-background.sh (+23/-0)
test-new-builds.sh (+168/-0)
tests/unit/test_power.py (+548/-0)
To merge this branch: bzr merge lp://staging/~toykeeper/ubuntu-power-tests/python-refactor-tk
Reviewer Review Type Date Requested Status
PS Jenkins bot continuous-integration Approve
Selene ToyKeeper (community) Approve
Max Brustkern (community) Approve
Review via email: mp+263325@code.staging.launchpad.net

Commit message

Update the tests to run in python using a class to manage many aspects of the testing.

Description of the change

Early preview of the python rewrite+refactor. Still changing rapidly.

To post a comment you must log in.
Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :

FAILED: Continuous integration, rev:52
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https://code.launchpad.net/~toykeeper/ubuntu-power-tests/python-refactor-tk/+merge/263325/+edit-commit-message

http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-ci/29/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-vivid-amd64-ci/29
    SUCCESS: http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-vivid-armhf-ci/29
    SUCCESS: http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-vivid-i386-ci/29

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-power-tests-ci/29/rebuild

review: Needs Fixing (continuous-integration)
53. By Selene ToyKeeper

Fixed the power graph size / DPI.
Moved the time origin to the test begin tag.

54. By Selene ToyKeeper

Fixed incorrect argument order for a non-refactored function.
Added a wait_after test option, to keep measuring for a bit before rebooting.
Tried to calibrate display_on timing for both arale and krillin,
  but they seem to need different timings.
Added my krillin to the device list.

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :

FAILED: Continuous integration, rev:54
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https://code.launchpad.net/~toykeeper/ubuntu-power-tests/python-refactor-tk/+merge/263325/+edit-commit-message

http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-ci/30/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-vivid-amd64-ci/30
    SUCCESS: http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-vivid-armhf-ci/30
    SUCCESS: http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-vivid-i386-ci/30

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-power-tests-ci/30/rebuild

review: Needs Fixing (continuous-integration)
55. By Selene ToyKeeper

Refactor milestone: Uses TestResult object and keeps adt logs.
Moved a variety of things to TestResult class...
Keep entire result set in a dir, with a subdir per run.
Made partial() arguments explicit to avoid ordering issues.
Renamed run() to avoid name collisions.
  (still need to remove it entirely and merge the two sets of run()s)

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :

FAILED: Continuous integration, rev:55
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https://code.launchpad.net/~toykeeper/ubuntu-power-tests/python-refactor-tk/+merge/263325/+edit-commit-message

http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-ci/31/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-vivid-amd64-ci/31
    SUCCESS: http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-vivid-armhf-ci/31
    SUCCESS: http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-vivid-i386-ci/31

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-power-tests-ci/31/rebuild

review: Needs Fixing (continuous-integration)
56. By Selene ToyKeeper

Added logging to file(s) during test runs.
Fixed code which tried to directly access good_runs.
  (which is intended to be true/false, not a list of run objects)

57. By Selene ToyKeeper

Extended timeout so krillin tests will work. (setup on it is slow!)
Forced a visible axis at power=0 on graphs, in case it goes negative.
Made graph bounds smarter, based mostly on the samples where the
  device wasn't plugged in at the time.

58. By Selene ToyKeeper

Added adt log verification, so it requires adt to pass or it will ignore
  that test run and try again (up to the max tries).

59. By Selene ToyKeeper

Fixed the case where a graph has no "unplugged" data.
Made it easier to switch between devices in new-build test script.

60. By Selene ToyKeeper

Fixed graph layout.

61. By Selene ToyKeeper

Finally got adt-run --keep-screen-active to work.
display_on test is finally working.
Uses value from test file instead of test filename to control behavior.

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :

FAILED: Continuous integration, rev:61
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https://code.launchpad.net/~toykeeper/ubuntu-power-tests/python-refactor-tk/+merge/263325/+edit-commit-message

http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-ci/32/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-vivid-amd64-ci/32
    SUCCESS: http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-vivid-armhf-ci/32
    SUCCESS: http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-vivid-i386-ci/32

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-power-tests-ci/32/rebuild

review: Needs Fixing (continuous-integration)
62. By Selene ToyKeeper

Made grapher work again with legacy graphs.
(let it use autoscale if no USB tags were found)

63. By Selene ToyKeeper

power.py now collects crash dumps and screenshots and the date.
Adjusted test parameters to allow for more failures
  (since the tests now fail a lot more often).
Cleaned up some power test code.

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :

FAILED: Continuous integration, rev:63
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https://code.launchpad.net/~toykeeper/ubuntu-power-tests/python-refactor-tk/+merge/263325/+edit-commit-message

http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-ci/33/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-vivid-amd64-ci/33
    SUCCESS: http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-vivid-armhf-ci/33
    SUCCESS: http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-vivid-i386-ci/33

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-power-tests-ci/33/rebuild

review: Needs Fixing (continuous-integration)
64. By Selene ToyKeeper

Added graphs for bad runs too.
Might help reveal info for fixing the bugs which cause failures.

65. By Selene ToyKeeper

Added my flash script and minimal supporting files.
(includes recovery images)
Cleaned up the flash script a little, but it still has some options
  we don't actually use for power testing.

66. By Selene ToyKeeper

Added date sanity checking. Phone must be within 24 hours of host.
This catches the boots where the phone thinks it's January 1 2010.

67. By Selene ToyKeeper

Added on-device monitoring, at least the basics.
  (took a while to find a way to make it run in the background
   and not die as soon as USB gets unplugged)
Made power.py log full tracebacks instead of just the error string.
Now ensures the clock is correct before starting a test
  (though it might skew again on pre-test reboot, d'oh).
Moved adt verify check lower so it won't prevent collecting logs
  when adt fails.
Added an attempt to collect logs after failure, just in case it
  wasn't done before.
Added a reboot after failure to avoid cases where the error also
  breaks the next test run.
Removed the need to pass anything to usb_connect / usb_disconnect.
Fixed the case where a bad run has no log to graph.
Now collects syslog too.
Refactored a little to allow starting on-device monitoring scripts
  at the appropriate point in the code.
Replaced asserts with exceptions which provide more human-friendly errors.

Made test-new-builds accept a device name/list on the command line.
Also made it not wait after running tests, but instead poll again immediately.

Increased display_on test time to 30 seconds again
  (since the --keep-screen-active thing is working).

68. By Selene ToyKeeper

Er, stop trying to grab a file which never exists. :)
(I forgot I moved it)

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :

FAILED: Continuous integration, rev:68
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https://code.launchpad.net/~toykeeper/ubuntu-power-tests/python-refactor-tk/+merge/263325/+edit-commit-message

http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-ci/34/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-vivid-amd64-ci/34
    SUCCESS: http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-vivid-armhf-ci/34
    SUCCESS: http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-vivid-i386-ci/34

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-power-tests-ci/34/rebuild

review: Needs Fixing (continuous-integration)
69. By Selene ToyKeeper

Moved fix_clock() and date-pre to adt's prepare-device script.
Added pretty-printed stats to the 'stats' result file,
  since this is an important part of the results and should be visible.
Took all the detail out of graph filenames, since the parent dir has it.

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :

FAILED: Continuous integration, rev:69
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https://code.launchpad.net/~toykeeper/ubuntu-power-tests/python-refactor-tk/+merge/263325/+edit-commit-message

http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-ci/35/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-vivid-amd64-ci/35
    SUCCESS: http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-vivid-armhf-ci/35
    SUCCESS: http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-vivid-i386-ci/35

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-power-tests-ci/35/rebuild

review: Needs Fixing (continuous-integration)
70. By Selene ToyKeeper

Hah, make sure the DUT is plugged in before trying to collect logs post-fail.
Also log more detail about post-fail exceptions.
Grab dmesg after each test run.

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :

FAILED: Continuous integration, rev:70
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https://code.launchpad.net/~toykeeper/ubuntu-power-tests/python-refactor-tk/+merge/263325/+edit-commit-message

http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-ci/36/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-vivid-amd64-ci/36
    SUCCESS: http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-vivid-armhf-ci/36
    SUCCESS: http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-vivid-i386-ci/36

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-power-tests-ci/36/rebuild

review: Needs Fixing (continuous-integration)
Revision history for this message
Max Brustkern (nuclearbob) wrote :

I've got a few comments, but mostly this is working when I run jobs with it, so I think we should be on track to land it soon.

71. By Selene ToyKeeper

Check the date later, because if it fails it prevents collecting other logs.

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :

FAILED: Continuous integration, rev:71
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https://code.launchpad.net/~toykeeper/ubuntu-power-tests/python-refactor-tk/+merge/263325/+edit-commit-message

http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-ci/37/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-vivid-amd64-ci/37
    SUCCESS: http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-vivid-armhf-ci/37
    SUCCESS: http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-vivid-i386-ci/37

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-power-tests-ci/37/rebuild

review: Needs Fixing (continuous-integration)
Revision history for this message
Selene ToyKeeper (toykeeper) wrote :

I started to fix each item as it came up, then found some which seemed better to do in a later merge... but then found a bunch more to do now. I think I'll have to finish this in the morning.

72. By Selene ToyKeeper

Oops, tell phablet-screenshot which device to use.
Also, show output from screenshot command for debugging.
Added -h / --help to usage docs.

73. By Selene ToyKeeper

Changed channel order so it'll test rc-proposed last.
This helps reduce how much time the power supply fan spends spinning
between tests, since rc-proposed behaves better and is less likely to
leave the device in a high-power state.

74. By Selene ToyKeeper

Updated the README quite a bit.
Removed outdated info and merged my notes into the main document.

75. By Selene ToyKeeper

Addressed remaining review comments.
Removed a variety of dead code.
Replaced bash with sh.
Pep8'd the graph script.
Renamed the good_runs variables to make the purpose more clear.
Removed good_count.
Switched run_cmd to use .communicate(); should be safer.
Ran a single test to make sure nothing huge broke.

Revision history for this message
Selene ToyKeeper (toykeeper) wrote :

Okay, nearly all comments addressed in the code, one smoke test executed, and a new rev pushed.

review: Needs Resubmitting
Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :

FAILED: Continuous integration, rev:75
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https://code.launchpad.net/~toykeeper/ubuntu-power-tests/python-refactor-tk/+merge/263325/+edit-commit-message

http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-ci/38/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-vivid-amd64-ci/38
    SUCCESS: http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-vivid-armhf-ci/38
    SUCCESS: http://jenkins.qa.ubuntu.com/job/ubuntu-power-tests-vivid-i386-ci/38

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-power-tests-ci/38/rebuild

review: Needs Fixing (continuous-integration)
Revision history for this message
Max Brustkern (nuclearbob) wrote :

My automated jobs have been running with this branch and succeeding last night and this morning, so I think we should be ready to merge.

review: Approve
76. By Selene ToyKeeper

Save a little extra network info before each test.

Revision history for this message
Selene ToyKeeper (toykeeper) wrote :

I pushed a tiny change to make our bots try again.

review: Approve
Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :
review: Approve (continuous-integration)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
The diff is not available at this time. You can reload the page or download it.

Subscribers

People subscribed via source and target branches