Merge lp://staging/~ev/uci-engine/skipped-integration-tests into lp://staging/uci-engine

Proposed by Evan
Status: Needs review
Proposed branch: lp://staging/~ev/uci-engine/skipped-integration-tests
Merge into: lp://staging/uci-engine
Diff against target: 26 lines (+3/-2)
2 files modified
bin/called-by-tarmac.py (+1/-1)
testing/run_tests.py (+2/-1)
To merge this branch: bzr merge lp://staging/~ev/uci-engine/skipped-integration-tests
Reviewer Review Type Date Requested Status
Francis Ginther Approve
PS Jenkins bot (community) continuous-integration Approve
Review via email: mp+242027@code.staging.launchpad.net

Commit message

Raise an error when the integration tests are skipped and --no-skip was provided. Call run-tests with --no-skip under tarmac.

To post a comment you must log in.
Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :

PASSED: Continuous integration, rev:897
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/1720/
Executed test runs:

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/1720/rebuild

review: Approve (continuous-integration)
Revision history for this message
Francis Ginther (fginther) wrote :

Looks good. Let's see what get's skipped!

review: Approve
Revision history for this message
Ubuntu CI Bot (uci-bot) wrote :
Download full text (116.6 KiB)

The attempt to merge lp:~ev/uci-engine/skipped-integration-tests into lp:uci-engine failed. Below is the output from the failed tests.

Running cm...
Updating source dependencies...
Updating source dependencies...
Updating source dependencies...
Updating source dependencies...
Updating source dependencies...
Updating source dependencies...
uploading webui-content.tgz to swift
Updating source dependencies...
Checking juju status
Private PPAs: disabled
Preparing local branch upload...
Uploading local branch, fingerprint 66ade962a9fce5150826b7eb3631cff1a6912f47
Building charm: lander
Building charm: wsgi-app
Building charm: rabbitmq-worker
Building charm: chroot-builder
Building charm: key-secret-subordinate
Building charm: webui
Building charm: system-image-server
Installing keys from bzr+ssh://bazaar.launchpad.net/~ci-engineering-private/+junk/ci-airline-dev-keys/
Running juju-deployer -v -c /tmp/tmpRCdPbG/deployer/branch-source-builder.yaml -c /tmp/tmpRCdPbG/deployer/britney-proxy.yaml -c /tmp/tmpRCdPbG/deployer/coverage-retriever.yaml -c /tmp/tmpRCdPbG/deployer/gatekeeper.yaml -c /tmp/tmpRCdPbG/deployer/image-builder.yaml -c /tmp/tmpRCdPbG/deployer/lander.yaml -c /tmp/tmpRCdPbG/deployer/nf-stats-service.yaml -c /tmp/tmpRCdPbG/deployer/ppa-creator.yaml -c /tmp/tmpRCdPbG/deployer/publisher.yaml -c /tmp/tmpRCdPbG/deployer/relations.yaml -c /tmp/tmpRCdPbG/deployer/test-runner.yaml -c /tmp/tmpRCdPbG/deployer/ticket-system.yaml -c /tmp/tmpRCdPbG/deployer/validator.yaml -c /tmp/tmpRCdPbG/deployer/webui.yaml ci-airline
Tests running...
ci-utils.ci_utils.tests.test_amqp.TestAMQP.testConnectFailed ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp.TestAMQP.testProcessQueue ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_amqp.TestAMQP.testRunForever ... OK (0.102 secs)
ci-utils.ci_utils.tests.test_amqp.TestAMQP.testSent ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp.TestProgressTrigger.testProgress ... OK (0.001 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testCancel ... OK (0.104 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testNoQueue ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testNoTicket ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageCalledProcessError ... OK (0.004 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageFail ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageKilled ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageSimple ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageUnexpected ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testSaveLastRun ... OK (3.046 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestTimer.testCBRuns ... OK (0.021 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestTimer.testCanCancel ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_data_store.TestDataStoreConfig.test_invalid_auth_config ... OK (0.000 secs)
ci-utils.ci_utils.tests.test_data_store.TestDataStoreConfig.test_valid_auth_config ... OK (0.000 secs)
ci-utils.ci_utils.test...

Revision history for this message
Vincent Ladeuil (vila) wrote :
Download full text (3.7 KiB)

======================================================================
FAIL: tests.test_test_runner.TestTestRunner.test_process_ticket
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/tarmac/branch.TrgDBm/bin/../tests/test_test_runner.py", line 94, in test_process_ticket
    assertions.assertLength(self, 6, msgs)
  File "/dev/shm/venv-D1AIEf/local/lib/python2.7/site-packages/ucitests-0.1.5-py2.7.egg/ucitests/assertions.py", line 67, in assertLength
    ' for {!r}'.format(length, len(obj_with_len), obj_with_len))
  File "/usr/lib/python2.7/unittest/case.py", line 412, in fail
    raise self.failureException(msg)
AssertionError: Incorrect length: wanted 6, got 3 for [{u'progress_trigger': u'test-progress-process', u'series': u'precise', u'ppa_list': [], u'package_list': [u'libpng'], u'image_id': u'uci/cloudimg/precise-amd64.img', u'state': u'STATUS', u'ticket_id': u'process-ticket-precise-amd64', u'architecture': u'amd64'}, {u'message': u'Setting up the testbed for ticket process-ticket-precise-amd64', u'state': u'STATUS'}, {u'artifacts': [{u'type': u'LOGS', u'name': u'image_test.libpng-testbed-cloud-init.log', u'reference': u'http://172.20.161.145:8080/v1/AUTH_bcb5bfaa8655475988210d34dab13c34/ticket-process-ticket-precise-amd64/libpng-testbed-cloud-init.log'}, {u'type': u'LOGS', u'name': u'image_test.output.log', u'reference': u'http://172.20.161.145:8080/v1/AUTH_bcb5bfaa8655475988210d34dab13c34/ticket-process-ticket-precise-amd64/image_test.output.log'}], u'request': {u'progress_trigger': u'test-progress-process', u'series': u'precise', u'ppa_list': [], u'package_list': [u'libpng'], u'image_id': u'uci/cloudimg/precise-amd64.img', u'state': u'STATUS', u'ticket_id': u'process-ticket-precise-amd64', u'architecture': u'amd64'}, u'state': u'COMPLETED', u'exit': True, u'result': u'FAILED'}]

Ran 728 tests in 450.880s
FAILED (failures=1)

http://172.20.161.145:8080/v1/AUTH_bcb5bfaa8655475988210d34dab13c34/ticket-process-ticket-precise-amd64/image_test.output.log

[2014-11-19 09:22:14,215] image_test:INFO:Running in /home/ubuntu with HOME: /home/ubuntu
[2014-11-19 09:22:14,216] image_test:INFO:Setting up the testbed for ticket process-ticket-precise-amd64
[2014-11-19 09:22:41,303] image_test:INFO:Got IP 172.19.0.166 for faf4507b-24cd-4eb4-a3f9-fe9c5603b29d
[2014-11-19 09:42:45,349] image_test:ERROR:The testbed setup for process-ticket-precise-amd64 failed
Traceback (most recent call last):
  File "./test_runner/tstrun/run_worker.py", line 181, in setup_testbed
  File "/srv/ci-airline-tr-rabbit-worker/code/r1_d2d1ac9902f6b7ce918a57ab3b79c909f050ca85/test_runner/tstrun/testbed.py", line 306, in setup
    self.wait_for_cloud_init()
  File "/srv/ci-airline-tr-rabbit-worker/code/r1_d2d1ac9902f6b7ce918a57ab3b79c909f050ca85/test_runner/tstrun/testbed.py", line 398, in wait_for_cloud_init
    raise TestBedException('Instance never completed cloud-init')
TestBedException: Instance never completed cloud-init
[2014-11-19 09:42:45,502] image_test:INFO:Saving libpng testbed console

http://172.20.161.145:8080/v1/AUTH_bcb5bfaa8655475988210d34dab13c34/ticket-process-ticket-precise-amd64/libpng-te...

Read more...

Revision history for this message
Vincent Ladeuil (vila) wrote :

Top-approving again hoping for the transient failure to go away so that the --no-skip check can trigger and answer Francis's:

> Looks good. Let's see what get's skipped!

I expect a bunch of test runner tests (at least).

Revision history for this message
Ubuntu CI Bot (uci-bot) wrote :
Download full text (114.7 KiB)

The attempt to merge lp:~ev/uci-engine/skipped-integration-tests into lp:uci-engine failed. Below is the output from the failed tests.

Running cm...
Updating source dependencies...
Updating source dependencies...
Updating source dependencies...
Updating source dependencies...
Updating source dependencies...
Updating source dependencies...
uploading webui-content.tgz to swift
Updating source dependencies...
Checking juju status
Private PPAs: disabled
Preparing local branch upload...
Uploading local branch, fingerprint 7c380ec060f9691fb6e57ae152100bc411d0e5b1
Building charm: lander
Building charm: wsgi-app
Building charm: rabbitmq-worker
Building charm: chroot-builder
Building charm: key-secret-subordinate
Building charm: webui
Building charm: system-image-server
Installing keys from bzr+ssh://bazaar.launchpad.net/~ci-engineering-private/+junk/ci-airline-dev-keys/
Running juju-deployer -v -c /tmp/tmpxMlF7y/deployer/branch-source-builder.yaml -c /tmp/tmpxMlF7y/deployer/britney-proxy.yaml -c /tmp/tmpxMlF7y/deployer/coverage-retriever.yaml -c /tmp/tmpxMlF7y/deployer/gatekeeper.yaml -c /tmp/tmpxMlF7y/deployer/image-builder.yaml -c /tmp/tmpxMlF7y/deployer/lander.yaml -c /tmp/tmpxMlF7y/deployer/nf-stats-service.yaml -c /tmp/tmpxMlF7y/deployer/ppa-creator.yaml -c /tmp/tmpxMlF7y/deployer/publisher.yaml -c /tmp/tmpxMlF7y/deployer/relations.yaml -c /tmp/tmpxMlF7y/deployer/test-runner.yaml -c /tmp/tmpxMlF7y/deployer/ticket-system.yaml -c /tmp/tmpxMlF7y/deployer/validator.yaml -c /tmp/tmpxMlF7y/deployer/webui.yaml ci-airline
Tests running...
ci-utils.ci_utils.tests.test_amqp.TestAMQP.testConnectFailed ... OK (0.001 secs)
ci-utils.ci_utils.tests.test_amqp.TestAMQP.testProcessQueue ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp.TestAMQP.testRunForever ... OK (0.102 secs)
ci-utils.ci_utils.tests.test_amqp.TestAMQP.testSent ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp.TestProgressTrigger.testProgress ... OK (0.001 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testCancel ... OK (0.104 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testNoQueue ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testNoTicket ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageCalledProcessError ... OK (0.005 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageFail ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageKilled ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageSimple ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageUnexpected ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testSaveLastRun ... OK (0.001 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestTimer.testCBRuns ... OK (0.021 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestTimer.testCanCancel ... OK (0.001 secs)
ci-utils.ci_utils.tests.test_data_store.TestDataStoreConfig.test_invalid_auth_config ... OK (0.000 secs)
ci-utils.ci_utils.tests.test_data_store.TestDataStoreConfig.test_valid_auth_config ... OK (0.000 secs)
ci-utils.ci_utils.test...

Revision history for this message
Para Siva (psivaa) wrote :

Where is the failure?.
...
Ran 728 tests in 476.376s
OK
...

Revision history for this message
Vincent Ladeuil (vila) wrote :
Download full text (6.6 KiB)

Ran 728 tests in 476.376s
OK

Epic fail ;)

--no-skip should at least gives some feedback that the test run has failed and better, output the names of the skipped tests.

britney_proxy.britney.tests.test_process_requests.TestBritneyRequestHandler.test_translate_request(amd64) ... SKIP Valid nova credentials file is not available (0.000 secs)
britney_proxy.britney.tests.test_process_requests.TestBritneyRequestHandler.test_translate_request(i386) ... SKIP Valid nova credentials file is not available (0.000 secs)
britney_proxy.britney.tests.test_process_results.TestBritneyResultHandler.test_completed_message ... SKIP Valid nova credentials file is not available (0.000 secs)
britney_proxy.britney.tests.test_process_results.TestBritneyResultHandler.test_status_message ... SKIP Valid nova credentials file is not available (0.000 secs)
test_runner.tstrun.tests.test_data_store.TestDataStore.test_put_file_stores_content ... SKIP Valid nova credentials file is not available (0.000 secs)
test_runner.tstrun.tests.test_data_store.TestDataStore.test_store_created_empty ... SKIP Valid nova credentials file is not available (0.000 secs)
test_runner.tstrun.tests.test_testbed.TestNovaClient.test_known_failure_is_retried ... SKIP Valid nova credentials file is not available (0.000 secs)
test_runner.tstrun.tests.test_testbed.TestNovaClient.test_retry_is_called ... SKIP Valid nova credentials file is not available (0.000 secs)
test_runner.tstrun.tests.test_testbed.TestNovaClient.test_unknown_failure_is_raised ... SKIP Valid nova credentials file is not available (0.000 secs)
test_runner.tstrun.tests.test_testbed.TestTestbed.test_apt_get_update_fails ... SKIP Valid nova credentials file is not available (0.000 secs)
test_runner.tstrun.tests.test_testbed.TestTestbed.test_apt_get_update_retries ... SKIP Valid nova credentials file is not available (0.000 secs)
test_runner.tstrun.tests.test_testbed.TestTestbed.test_create_new_ssh_key ... SKIP Valid nova credentials file is not available (0.000 secs)
test_runner.tstrun.tests.test_testbed.TestTestbed.test_create_no_image ... SKIP Valid nova credentials file is not available (0.000 secs)
test_runner.tstrun.tests.test_testbed.TestTestbed.test_create_unknown_flavor ... SKIP Valid nova credentials file is not available (0.000 secs)
test_runner.tstrun.tests.test_testbed.TestTestbed.test_create_unknown_image ... SKIP Valid nova credentials file is not available (0.000 secs)
test_runner.tstrun.tests.test_testbed.TestTestbed.test_existing_home_ssh ... SKIP Valid nova credentials file is not available (0.000 secs)
test_runner.tstrun.tests.test_testbed.TestTestbed.test_ssh_failure ... SKIP Valid nova credentials file is not available (0.000 secs)
test_runner.tstrun.tests.test_testbed.TestTestbed.test_wait_for_instance_errors ... SKIP Valid nova credentials file is not available (0.000 secs)
test_runner.tstrun.tests.test_testbed.TestTestbed.test_wait_for_instance_fails ... SKIP Valid nova credentials file is not available (0.000 secs)
test_runner.tstrun.tests.test_testbed.TestTestbed.test_wait_for_ip_fails ... SKIP Valid nova credentials file is not available (0.000 secs)
test_runner.tstrun.tests.test_testbed.TestUsableTestbed.t...

Read more...

Unmerged revisions

897. By Evan

Raise an error when the integration tests are skipped and --no-skipped was provided. Call run-tests with --no-skipped under tarmac.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
The diff is not available at this time. You can reload the page or download it.

Subscribers

People subscribed via source and target branches

to all changes: