Merge lp://staging/~wesley-wiedenmeier/curtin/partial-testing into lp://staging/~curtin-dev/curtin/trunk

The storagetests are now at a point where they can be fully run and the results collected just using 'make run_storagetests'. The storagetest_runner uses the same general structure as vmtests so it shouldn't require any modifications of the test system other than adding it to the list of things to run.

The storagetest_runner generates a tarball of curtin and issues cloud-config scripts to the target image to retrieve and extract that tarball as it boots. The storagetests are then run, and use curtin.reporter to report success and failure to the test runner, as well as log files generated by block_meta while the test was running

I think that the storagetests and the storagetest_runner are ready for review now, this added a lot of verification for storage configuration and makes it easy to add quite a bit more. The test runner works pretty well, and I have managed to run all of the tests in about 2 hours with 4 proceses. I think that quite a bit of the time there was used waiting for apt-get on a bad internet connection. I haven't tried with a local repo mirror yet though.

There are still some outstanding issues running some of the actual test cases, but these will require bugfixes on curtin to correct. The test cases which cannot be run right now are disabled in the test suite, and can be enabled when the bugs they cause are fixed.

There are some other branches pending merge that this branch depends on, I will go through and make a list of those tomorrow.

I also added a doc to doc/devel/ about storagetests and storagetest_runner that was based on the doc for vmtests.

PASSED: Continuous integration, rev:470
https://server-team-jenkins.canonical.com/job/curtin-ci/303/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/300/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/303/rebuild

Hi Wesley,

I just ran your branch to take a look and found some minor things I'd ask you if you consider them useful to add.

The json files - while being json and not intended for humans - would still be much more readable it they had some line breaks. Like what "python -m json.tool /tmp/foo/storage-events.json" gives me would be great.

Then in the log a few outputs seem not to go through LOG.
I have a few questions about some of them:

1. Parse storage tests reporting data and ensure that all tests passed ... ok
Should get a "2016-07-12 07:15:09,184 - vmtests - INFO -" prefix like the others IMHO.

2. 10.245.168.14 - - [12/Jul/2016 07:15:22] "GET /storagetest-reporting.json HTTP/1.1" 200 -
10.245.168.14 - - [12/Jul/2016 07:15:22] "GET /storagetest-disk-conf.json HTTP/1.1" 200 -
10.245.168.14 - - [12/Jul/2016 07:15:22] "GET /curtin.tar.xz HTTP/1.1" 200 -

What are thos actually doing, it is always the same text - Could be something like:
2016-07-12 07:15:09,184 - vmtests - INFO - fetching json reporting
2016-07-12 07:15:09,184 - vmtests - INFO - fetching json disk config
2016-07-12 07:15:09,184 - vmtests - INFO - fetching foo tarball

3. just after "10.245.168.14 - - [12/Jul/2016 07:15:22] "GET /curtin.tar.xz HTTP/1.1" 200 -" is the longest "wait" duration in the tests. There should be some sort of "doing the test now" or anything else that somebody looking at it knows what is taking the time. This doesn't have to be step 1,2,3,4,../20 - just after the "Booting target image" a short "running test XY now".

Some comments when reading through the code

For a speed check I tried to see if it has issues going concurrently.
I think I didn't find the right way to call it - I tried:
rm -rf output/; CURTIN_VMTEST_KEEP_DATA_FAIL=all ./tools/jenkins-runner -vv --nologcapture --processes=10 --process-timeout=3600 tests/storagetest_runner/

I found that with this call the section following section tries to write an infinitely huge file:
Building tarball of curtin: /mnt/nvme/curtin-wesley/partial-testing

Wanted to let you know just in case that infinite write is a bug.

Later on I found this in the doc:
nosetests3 --processes=-1 tests/storagetest_runner

That gave me a stuck system - maybe too much cpus (6x2threads) and by that too much output?
In any case the output indenting was totally broken - I had to reset my console to scroll again.

That failed me then with errors, not sure if that is bad or just a wrong call - here is the log: http://paste.ubuntu.com/19157752/
I realized that since this first "hanging" processes -1 run all tests were failing this way now.
All on:
Traceback (most recent call last):
  File "/mnt/nvme/curtin-wesley/partial-testing/tests/storagetest_runner/__init__.py", line 385, in test_reported_results
    self.assertTrue(os.path.isfile(self.storage_log))
AssertionError: False is not true

Debugging gave me: "(qemu) qemu-system-x86_64: cannot set up guest memory 'pc.ram': Cannot allocate memory"
That likely also was my first hanging - but that could be fixed by freeing some up :-)
But I wonder if we need some sort of "is enough mem avail" prior to call qemu?

In the following retry it left me again with a good return code, but plenty of running qemu processes up.
That really needs some hardening.

Then I wanted to step down and did only:
nosetests3 --processes=2 tests/storagetest_runner
To check if it works at all.
I got the same console that gets misformatted after a while ending with
Ran 0 tests in 112.494s

Since all fails keep the logs around here the log file: http://paste.ubuntu.com/19158435/

TL;DR: concurrent execution needs some fixes and probably a bit hardening against shooting itself :-)

I ran checkers and found several of the following (no need to mark them all inline):

tests/storagetest_runner/__init__.py|101 col 16 warning| [unused-variable] Unused variable '_err' [python/pylint]

=> if really unused a _ would be even better.

tests/storagetest_runner/__init__.py|208 col 18 warning| [logging-format-interpolation] Use % formatting in logging functions and pass the % parameters as arguments

=> since log would format for you ...

I know pylint it is noisy and sometimes even disagrees with other checkers, but most of these are also only a search and replace away so probably worth to fix.

I'm not so sure about
tests/storagetests/__init__.py|186 col 16 warning| [unidiomatic-typecheck] Using type() instead of isinstance() for a typecheck. [python/pylint]

tests/storagetests/__init__.py|219 col 24 warning| [redefined-builtin] Redefining built-in 'type' [python/pylint]

I already complained about short names before, this is another example (there are more like mp, fp, e, ...)
tests/storagetests/__init__.py|224 col 13 warning| [invalid-name] Invalid variable name "d" [python/pylint]

A totally different one is:
tests/storagetests/verifiers.py|15 col 13 error| [no-member] Instance of 'BasicTests' has no 'assertIsNotNone' member [python/pylint]
That never is an issue so far, as all inheriting classes of e.g. BasicTests also inherit from class BaseStorageTest(unittest.TestCase). But to make sure the methods are in scope would it hurt letting the verifiers also derive from unittest.TestCase?

One that is probably worth for sure to avoid later issues:
tests/storagetests/__init__.py|82 col 31 warning| [redefined-outer-name] Redefining name 'reporter' from outer scope (line 2) [python/pylint]

Hey, thanks for looking through and reviewing, I'm reading through the diff comments atm, I'll reply inline.

> For a speed check I tried to see if it has issues going concurrently.
> I think I didn't find the right way to call it - I tried:
> rm -rf output/; CURTIN_VMTEST_KEEP_DATA_FAIL=all ./tools/jenkins-runner -vv
> --nologcapture --processes=10 --process-timeout=3600 tests/storagetest_runner/
>
> I found that with this call the section following section tries to write an
> infinitely huge file:
> Building tarball of curtin: /mnt/nvme/curtin-wesley/partial-testing
>
> Wanted to let you know just in case that infinite write is a bug.

Thanks for pointing that out, that's definintely a bug in how I generate the curtin tarball. I hadn't run before in the jenkins runner, because sparse files don't work right in my /home partition and I didn't want the tests writing huge files there, but yeah, the jenkins runner puts the curtin tmp dir inside the curtin dir being tested, so it would cause tar to try to include the tarball its generating in itself. I need to make the tests aware of what environment they're running in so they know to omit that directory.

> Later on I found this in the doc:
> nosetests3 --processes=-1 tests/storagetest_runner
>
> That gave me a stuck system - maybe too much cpus (6x2threads) and by that too
> much output?
> In any case the output indenting was totally broken - I had to reset my
> console to scroll again.

Yeah, I've seen the output get messed up too, but I think what is happening is just that the processes are racing to write to stdout and the data they're writing is getting corrupted somehow. I haven't tried with more than 4 processes at once, so there may be some bugs that occur in that case. I'll look through the log and try to see if I can figure out what was going wrong there.

> That failed me then with errors, not sure if that is bad or just a wrong call
> - here is the log: http://paste.ubuntu.com/19157752/
> I realized that since this first "hanging" processes -1 run all tests were
> failing this way now.
> All on:
> Traceback (most recent call last):
> File "/mnt/nvme/curtin-wesley/partial-
> testing/tests/storagetest_runner/__init__.py", line 385, in
> test_reported_results
> self.assertTrue(os.path.isfile(self.storage_log))
> AssertionError: False is not true
>
> Debugging gave me: "(qemu) qemu-system-x86_64: cannot set up guest memory
> 'pc.ram': Cannot allocate memory"
> That likely also was my first hanging - but that could be fixed by freeing
> some up :-)
> But I wonder if we need some sort of "is enough mem avail" prior to call qemu?

Yeah, the test runner eats up memory, not quite as much as the vmtests do while the target system tarball is being extracted in the vm though. A check would definitely be good, I'll look into how to write that.

Watching a vm trying to run from swap isn't fun :)

> In the following retry it left me again with a good return code, but plenty of
> running qemu processes up.
> That really needs some hardening.
>
> Then I wanted to step down and did only:
> nosetests3 --processes=2 tests/storagetest_runner
> To check if it works at all.
> I got the same console that gets misformatted after a while ending with
> Ran 0 tests in 112.494s
>
> Since all fails keep th...

> Hi Wesley,
>
> I just ran your branch to take a look and found some minor things I'd ask you
> if you consider them useful to add.
>
> The json files - while being json and not intended for humans - would still be
> much more readable it they had some line breaks. Like what "python -m
> json.tool /tmp/foo/storage-events.json" gives me would be great.

Yeah, that makes sense. I'll switch it to use something like 'json.dump(... indent=4)'

> Then in the log a few outputs seem not to go through LOG.
> I have a few questions about some of them:
>
> 1. Parse storage tests reporting data and ensure that all tests passed ... ok
> Should get a "2016-07-12 07:15:09,184 - vmtests - INFO -" prefix like the
> others IMHO.

Oh, that's the docstring for the test function, I think that's just added to nosetests's output when running with '-vv'. It may be good to emit a message through LOG though and remove the docstring cause the times could be good for debugging.

> 2. 10.245.168.14 - - [12/Jul/2016 07:15:22] "GET /storagetest-reporting.json
> HTTP/1.1" 200 -
> 10.245.168.14 - - [12/Jul/2016 07:15:22] "GET /storagetest-disk-conf.json
> HTTP/1.1" 200 -
> 10.245.168.14 - - [12/Jul/2016 07:15:22] "GET /curtin.tar.xz HTTP/1.1" 200 -
>
> What are thos actually doing, it is always the same text - Could be something
> like:
> 2016-07-12 07:15:09,184 - vmtests - INFO - fetching json reporting
> 2016-07-12 07:15:09,184 - vmtests - INFO - fetching json disk config
> 2016-07-12 07:15:09,184 - vmtests - INFO - fetching foo tarball

I'm not sure if I can actually change that or not, those messages come from http.server, and I don't know if there is an interface in it to handle custom messages, I'll check in the source for it though.

> 3. just after "10.245.168.14 - - [12/Jul/2016 07:15:22] "GET /curtin.tar.xz
> HTTP/1.1" 200 -" is the longest "wait" duration in the tests. There should be
> some sort of "doing the test now" or anything else that somebody looking at it
> knows what is taking the time. This doesn't have to be step 1,2,3,4,../20 -
> just after the "Booting target image" a short "running test XY now".

Yeah, it would definitely be nice to have a bit of debug there. I can add in optional logging to tools.report_webhook_logger to handle outputting events to stdio as well as the logfile when they're received, that should work okay

> I ran checkers and found several of the following (no need to mark them all
> inline):
>
> tests/storagetest_runner/__init__.py|101 col 16 warning| [unused-variable]
> Unused variable '_err' [python/pylint]
>
> => if really unused a _ would be even better.

Yeah, that makes sense. I just got in the habit of writing _err from block meta, but I should probably clean up some of the variable names there too. For the most part the err part of the util.subp return is never used.

> tests/storagetest_runner/__init__.py|208 col 18 warning| [logging-format-
> interpolation] Use % formatting in logging functions and pass the % parameters
> as arguments
>
> => since log would format for you ...
>
> I know pylint it is noisy and sometimes even disagrees with other checkers,
> but most of these are also only a search and replace away so probably worth to
> fix.

Yeah, I can use the old style format specifiers there, I'll switch that over.

> I'm not so sure about
> tests/storagetests/__init__.py|186 col 16 warning| [unidiomatic-typecheck]
> Using type() instead of isinstance() for a typecheck. [python/pylint]

Right yeah, I'll switch that, isinstance reads nicer.

> tests/storagetests/__init__.py|219 col 24 warning| [redefined-builtin]
> Redefining built-in 'type' [python/pylint]

Oh, yeah, that was for error type in __exit__, I'll rename that to 'etype'

> I already complained about short names before, this is another example (there
> are more like mp, fp, e, ...)
> tests/storagetests/__init__.py|224 col 13 warning| [invalid-name] Invalid
> variable name "d" [python/pylint]

Yeah, I think some of these are just trying to keep the line from going over 80, but some were probably in the list comprehensions, where the scope of the var is so limited it should be okay. I'll switch everything outside of comprehensions over to long names real quick, thanks for bringing that up.

>
> A totally different one is:
> tests/storagetests/verifiers.py|15 col 13 error| [no-member] Instance of
> 'BasicTests' has no 'assertIsNotNone' member [python/pylint]
> That never is an issue so far, as all inheriting classes of e.g. BasicTests
> also inherit from class BaseStorageTest(unittest.TestCase). But to make sure
> the methods are in scope would it hurt letting the verifiers also derive from
> unittest.TestCase?

I guess the verifiers could inherit from TestCase, but I think it may be cleaner for them not to, partly just because that would require '__test__ = False' in all of them to prevent nosetests3 from trying to run them as individual tests. Also, because the BaseStorageTest class overrides the TestCase.run() method, if the verifier classes were to inherit from TestCase and were listed as parents of the individual test classes after the BaseStorageTest, I think that the call to super() in BaseStorageTest.run() would mess up, but I'm not 100% sure there. I should probably do something like define an __init__ for the verifier classes that checks something like 'if not isinstance(self, TestCase)' and throws an error to make the verifiers unusable outside of the test classes.

> One that is probably worth for sure to avoid later issues:
> tests/storagetests/__init__.p...

Replied to some of the inline commments

I just pushed some cleanup for the comments posted. Not everything's been addressed yet, but the rest I'll get to tomorrow.

Just so it doesn't get lost, the diff comments are on revision 470.

PASSED: Continuous integration, rev:476
https://server-team-jenkins.canonical.com/job/curtin-ci-ppc64/8/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/303/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci-ppc64/8/rebuild

FAILED: Continuous integration, rev:483
https://server-team-jenkins.canonical.com/job/curtin-ci-ppc64/10/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/306/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci-ppc64/10/rebuild

PASSED: Continuous integration, rev:483
https://server-team-jenkins.canonical.com/job/curtin-ci/305/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/308/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/305/rebuild

PASSED: Continuous integration, rev:487
https://server-team-jenkins.canonical.com/job/curtin-ci-ppc64/17/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/321/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci-ppc64/17/rebuild

PASSED: Continuous integration, rev:487
https://server-team-jenkins.canonical.com/job/curtin-ci/313/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/322/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/313/rebuild

i really like the goal here.
some comments inline.

PASSED: Continuous integration, rev:488
https://server-team-jenkins.canonical.com/job/curtin-ci/314/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/323/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/314/rebuild

PASSED: Continuous integration, rev:488
https://server-team-jenkins.canonical.com/job/curtin-ci-ppc64/18/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/324/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci-ppc64/18/rebuild

Thanks for reviewing. I went through and replied to the diff comments, I'll get most of them handled soon, one or two may take a bit longer. Just so they aren't lost because of future commits, the diff comments are at: r487

PASSED: Continuous integration, rev:490
https://server-team-jenkins.canonical.com/job/curtin-ci-ppc64/20/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/327/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci-ppc64/20/rebuild

PASSED: Continuous integration, rev:490
https://server-team-jenkins.canonical.com/job/curtin-ci/316/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/328/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/316/rebuild

PASSED: Continuous integration, rev:494
https://server-team-jenkins.canonical.com/job/curtin-ci-ppc64/24/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/334/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci-ppc64/24/rebuild

PASSED: Continuous integration, rev:494
https://server-team-jenkins.canonical.com/job/curtin-ci/319/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/335/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/319/rebuild

PASSED: Continuous integration, rev:495
https://server-team-jenkins.canonical.com/job/curtin-ci/325/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/346/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/325/rebuild

PASSED: Continuous integration, rev:495
https://server-team-jenkins.canonical.com/job/curtin-ci-ppc64/30/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/347/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci-ppc64/30/rebuild

PASSED: Continuous integration, rev:496
https://server-team-jenkins.canonical.com/job/curtin-ci/326/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/348/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/326/rebuild

PASSED: Continuous integration, rev:496
https://server-team-jenkins.canonical.com/job/curtin-ci-ppc64/31/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/349/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci-ppc64/31/rebuild

PASSED: Continuous integration, rev:497
https://server-team-jenkins.canonical.com/job/curtin-ci/327/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/350/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/327/rebuild

PASSED: Continuous integration, rev:497
https://server-team-jenkins.canonical.com/job/curtin-ci-ppc64/32/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/351/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci-ppc64/32/rebuild

There are some branches that have already been merged into
this branch because they were neccessary for getting
storagetests working at all. Most of them are bug fixes,
except trunk.add-web-reporter-to-vmtests, which introduces
report_webhook_logger.CaptureReporting, to retrieve curtin
reporting from a test environment.

In addition, several bugs have already been found using the
current test configs for storagetests. Branches exist to
handle most of these. Those branches have not been merged
into this one, but once they have merged into trunk I can
enable the storagetests that reproduce those bugs.

Already Merged Branches:
lp:~wesley-wiedenmeier/curtin/trunk.add-web-reporter-to-vmtests
    merge into: lp:~raharper/curtin/trunk.add-web-reporter-to-vmtests
    reason: improvements to original logic for
            CaptureReporting and get_lan_ip, makes it
            possible to use tests with reporting in
            parallel by getting port number to bind to
            dynamically and uses ifconfig instead of
            'ip address' to get local lan ip in vmtests
lp:~raharper/curtin/trunk.add-web-reporter-to-vmtests
    merge into: lp:curtin
    reason: during vmtests, capture reporting events from
            curtin and verify that the events are recieved
            and properly formatted
lp:~wesley-wiedenmeier/curtin/curtin-fix-sysfs-partition-data
    merge info: lp:curtin
    reason: the block.sysfs_partition_data function is
            useful, and is used in storagetests, but the
            old implementation was only able to operate on
            the path to whole disks, not to give
            information about a single partition
lp:~wesley-wiedenmeier/curtin/trusty-preserve
    merge into: lp:curtin
    reason: fix handling of disk preservation on
            trusty/precise where blkid may not work
            correctly for virtio disks and some other
            special disks
lp:~wesley-wiedenmeier/curtin/1598310
    merge into: lp:curtin
    reason: (LP: 1598310) The current implementation of
            block.get_blockdev_sector_size fails when
            running on a path for which lsblk may give
            multiple results, such as a disk with
            partitions on it or a dos extended partition.
            This branch fixes this and adds unittests
lp:~wesley-wiedenmeier/curtin/1597522
    merge into: lp:curtin
    reason: (LP: 1597522) A fix that went into trunk a
            while ago as a work around for a bug in
            mkfs.vfat added a '-s' flag to partitioning
            commands, but did so on all filesystem
            commands, not just vfat ones. Some tools,
            like mkfs.ext4 silently ignored it, but it
            caused other tools, such as mkfs.xfs to fail
            and mkfs.btrfs to try to create a filesystem
            assuming the underlying disk had 1 byte blocks.
            This branch fixes that, and adds it to vmtests

Bugs found by storagetests fixed so far:
  - (LP: 1597522) Curtin passes -s flag to all mkfs cmds,
                  not just vfat
    lp:~wesley-wiedenmeier/curtin/1597522
        Already merged into partial-testing, but not
        in trunk
  - (LP: 1592962) U...

PASSED: Continuous integration, rev:498
https://server-team-jenkins.canonical.com/job/curtin-ci/328/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/352/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/328/rebuild

PASSED: Continuous integration, rev:498
https://server-team-jenkins.canonical.com/job/curtin-ci-ppc64/33/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/353/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci-ppc64/33/rebuild

PASSED: Continuous integration, rev:501
https://server-team-jenkins.canonical.com/job/curtin-ci/207/
Executed test runs:
    SUCCESS: https://server-team-jenkins.canonical.com/job/curtin-ci/nodes=amd64/207
    SUCCESS: https://server-team-jenkins.canonical.com/job/curtin-ci/nodes=ppc64el/207

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/207/rebuild

FAILED: Continuous integration, rev:515
https://jenkins.ubuntu.com/server/job/curtin-ci/208/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=amd64/208
    FAILURE: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=ppc64el/208/console

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/208/rebuild

PASSED: Continuous integration, rev:516
https://jenkins.ubuntu.com/server/job/curtin-ci/211/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=amd64/211
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=ppc64el/211

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/211/rebuild

PASSED: Continuous integration, rev:517
https://jenkins.ubuntu.com/server/job/curtin-ci/215/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=amd64/215
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=ppc64el/215

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/215/rebuild

PASSED: Continuous integration, rev:518
https://jenkins.ubuntu.com/server/job/curtin-ci/218/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=amd64/218
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=ppc64el/218

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/218/rebuild

PASSED: Continuous integration, rev:519
https://jenkins.ubuntu.com/server/job/curtin-ci/224/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=amd64/224
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=ppc64el/224

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/224/rebuild

PASSED: Continuous integration, rev:520
https://jenkins.ubuntu.com/server/job/curtin-ci/225/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=amd64/225
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=ppc64el/225

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/225/rebuild

PASSED: Continuous integration, rev:522
https://jenkins.ubuntu.com/server/job/curtin-ci/226/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=amd64/226
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=ppc64el/226

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/226/rebuild

PASSED: Continuous integration, rev:524
https://jenkins.ubuntu.com/server/job/curtin-ci/229/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=amd64/229
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=ppc64el/229

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/229/rebuild

PASSED: Continuous integration, rev:525
https://jenkins.ubuntu.com/server/job/curtin-ci/230/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=amd64/230
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=ppc64el/230

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/230/rebuild

PASSED: Continuous integration, rev:526
https://jenkins.ubuntu.com/server/job/curtin-ci/231/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=amd64/231
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=ppc64el/231

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/231/rebuild

PASSED: Continuous integration, rev:527
https://jenkins.ubuntu.com/server/job/curtin-ci/232/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=amd64/232
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=ppc64el/232

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/232/rebuild

PASSED: Continuous integration, rev:528
https://jenkins.ubuntu.com/server/job/curtin-ci/233/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=amd64/233
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=ppc64el/233

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/233/rebuild

PASSED: Continuous integration, rev:529
https://jenkins.ubuntu.com/server/job/curtin-ci/234/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=amd64/234
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=ppc64el/234

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/234/rebuild

PASSED: Continuous integration, rev:530
https://jenkins.ubuntu.com/server/job/curtin-ci/246/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=amd64/246
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=ppc64el/246

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/246/rebuild

PASSED: Continuous integration, rev:532
https://jenkins.ubuntu.com/server/job/curtin-ci/258/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=amd64/258
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=ppc64el/258

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/258/rebuild

PASSED: Continuous integration, rev:533
https://jenkins.ubuntu.com/server/job/curtin-ci/260/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=amd64/260
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=ppc64el/260
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=s390x/260

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/260/rebuild

I've marked this work in progress.
Ryan and I discussed this some, it is valuable, and we'd like to runt his, but at this point the branch probably needs rework to merge from trunk...
so just categorizing as work-in-progress.