Merge lp://staging/~ev/ubuntu-ci-services-itself/pythonise-integration-test-runner into lp://staging/ubuntu-ci-services-itself

Note to self: this needs a little bit of work to check the machine still exists before attempting to destroy it:

ERROR no machines were destroyed: machine 11 does not exist
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File /usr/lib/python2.7/atexit.py, line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File ./tests/run.py, line 60, in destroy_services
    _destroy_machine(machine)
  File ./tests/run.py, line 32, in _destroy_machine
    subprocess.check_output(['juju', 'destroy-machine', machine, '--force'])
  File /usr/lib/python2.7/subprocess.py, line 573, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
CalledProcessError: Command '['juju', 'destroy-machine', '11', '--force']' returned non-zero exit status 1

We should find a way to use python-jujuclient for this instead of calling juju directly.

I think this is ready to merge. I sorted out the destroy-machine failing by waiting for the nodes that were dying to die, rather than bringing in jujuclient (which doesn't have a destroy_machine command, as it turns out).

I've now explicitly handled the case where the instance disappears between calling status and terminate-machine.

can you chmod +x ./tests/run.py so its executable by default?

423 + with open(SUDOERS_FILE) as fp:
424 + contents = fp.read()
425 + if not contents.strip('\n').endswith(SUDOERS):
426 + return False

non-root users can't read this file so the script fails.

now that this thing is python, we are missing some of the output the shell version used to cause. its hard to see what the runner is attempting. You think you could add a few status messages for each of the major steps in the process (ie destroying, bootstrapping, installing, etc).

On 23 January 2014 16:36, Andy Doan <email address hidden> wrote:
> can you chmod +x ./tests/run.py so its executable by default?

Fixed as r140.

On 23 January 2014 16:58, Andy Doan <email address hidden> wrote:
> 423 + with open(SUDOERS_FILE) as fp:
> 424 + contents = fp.read()
> 425 + if not contents.strip('\n').endswith(SUDOERS):
> 426 + return False
>
> non-root users can't read this file so the script fails.

Mine is 0644. Fixed as r141.

On 23 January 2014 17:06, Andy Doan <email address hidden> wrote:
> now that this thing is python, we are missing some of the output the shell version used to cause. its hard to see what the runner is attempting. You think you could add a few status messages for each of the major steps in the process (ie destroying, bootstrapping, installing, etc).

Fixed as r143 and r144.

missing the import for "re" now.

D'oh. Fixed as r149.

i also see:

Traceback (most recent call last):
  File "./tests/run.py", line 403, in <module>
    sys.exit(main())
  File "./tests/run.py", line 393, in main
    units = find_failed_units(e.output)
  File "./tests/run.py", line 160, in find_failed_units
    return set(re.findall('unit: (.*): machine: .* error details:', body))
  File "/usr/lib/python2.7/re.py", line 177, in findall
    return _compile(pattern, flags).findall(string)
TypeError: expected string or buffer

On 23 January 2014 20:37, Andy Doan <email address hidden> wrote:
> i also see:
>
> Traceback (most recent call last):
> File "./tests/run.py", line 403, in <module>
> sys.exit(main())
> File "./tests/run.py", line 393, in main
> units = find_failed_units(e.output)
> File "./tests/run.py", line 160, in find_failed_units
> return set(re.findall('unit: (.*): machine: .* error details:', body))
> File "/usr/lib/python2.7/re.py", line 177, in findall
> return _compile(pattern, flags).findall(string)
> TypeError: expected string or buffer

Fixing by using juju status for this instead. I'll ping you when it's ready.

I'm also seeing:

535 + needed = needs_bootstrap()

fail if I have nodes available but no tunnel set up. I have to run "juju destroy-environment" by hand before I can run the script.

On 23 January 2014 22:05, Andy Doan <email address hidden> wrote:
> I'm also seeing:
>
> 535 + needed = needs_bootstrap()
>
> fail if I have nodes available but no tunnel set up. I have to run "juju destroy-environment" by hand before I can run the script.

What's the output of juju status in that instance? I am unable to
reproduce this by running juju bootstrap, not setting up a tunnel, and
then running python tests/run.py.

On 01/23/2014 05:30 PM, Evan Dandrea wrote:
>> fail if I have nodes available but no tunnel set up. I have to run "juju destroy-environment" by hand before I can run the script.
> What's the output of juju status in that instance? I am unable to
> reproduce this by running juju bootstrap, not setting up a tunnel, and
> then running python tests/run.py.

bad wording on my part. "juju status" just hangs. You can't run
juju-staus without a tunnel setup. And we don't have a tunnel set up at
this part of the script.

On 23 January 2014 23:54, Andy Doan <email address hidden> wrote:
> bad wording on my part. "juju status" just hangs. You can't run
> juju-staus without a tunnel setup. And we don't have a tunnel set up at
> this part of the script.

It will eventually time out and succeed. I'll see if I can find a more
expedient check tomorrow.

Okay, I think we're ready to go again. This version should handle whatever crazy broken juju environment you throw at it.

things seem to be working smooth for me now. lets merge it.

Woohoo!