Merge lp://staging/~chris.macnaughton/openstack-mojo-specs/upgrade-spec into lp://staging/openstack-mojo-specs

Reference juju core bug: https://bugs.launchpad.net/juju/+bug/1749201

I understand this is a WIP. Just some notes for the next revision(s).

Most (all?) juju ssh $UNIT sudo $CMD should be juju run --unit $UNIT $CMD. You get root for free and various --format output options more convenient for python.

A number of WIP print statements that either need logging statements or removal before landing. Not also the use of logging.error or logging.warn where appropriate.

The juju series-update commands, when is it safe to set this for an application that may have multiple units?

Individual try/except blocks look good. But it would make sense to have a top level try/except that determines success or failure based on raised exceptions in nested tries.

Larger question. At what point does it make more sense to create a bash script and scp it to the unit to run. Rather, than piecemeal commands over ssh? That might be a cleaner approach and it would be re-usable outside mojo.

Thanks for the suggestions, in addition to the comments inline, changes are incoming!

Several comments inline; regarding making a bash script rather than this approach, at the moment it is fairly isolated, and runnable with the script in helper/scripts/upgrade_all_units.py. One of the goals of this bit of code is that we will be able to migrate it easily into zaza (or just something else) at some point in the future, as a generic set of helper utilities that are consumable outside of mojo.

Need to fix linting errors. I expect this might be challenging as most of them are due to long lines in the bash snippets.

> Most (all?) juju ssh $UNIT sudo $CMD should be juju run --unit $UNIT $CMD. You get root for free and various --format output options more convenient for python.

The do-release-upgrade and the loop check after reboot makes sense to use SSH. The others, reboot and lsb_release check, make more sense as juju run.

Note: it turns out juju run --unit runs as root but juju run --machine runs as ubuntu user. Just for consistency.

> A number of WIP print statements that either need logging statements or removal before landing. Not also the use of logging.error or logging.warn where appropriate.

Still true. Some are commented but they should be removed or made logging statements. Use of logging.error and logging.warn suggestions still in effect.

A couple of things left to discuss:

> The juju series-update commands, when is it safe to set this for an application that may have multiple units?

At the moment, there is no possible config into the actual callable script, although it will only iterate through units, so, depending on the HA model of the hosted application, there shouldn't be service disruption as we will only be upgrading a single unit at a time for now.

> Individual try/except blocks look good. But it would make sense to have a top level try/except that determines success or failure based on raised exceptions in nested tries.

I'm open to working through doing something like this if we decide that it's necessary; however, I'm nervous to make this code perfect, given that the Juju core team has picked up a roadmap item that will reduce our "series_upgrade" function to something along the lines of `juju series-upgrade $machine_id`, at which point we will be removing almost all of the upgrade code. To make the cleanup of that easier, in the last commit I have comment fenced off the upgrade code, although I propose that it could be moved into a new helper alongside the mojo_utils.py for now to ensure isolation.

> Larger question. At what point does it make more sense to create a bash script and scp it to the unit to run. Rather, than piecemeal commands over ssh? That might be a cleaner approach and it would be re-usable outside mojo.

Rather than scp'ing a bash script, I am considering shifting the series of `juju ssh`'d commands into building a bash script that gets rendered (depends entirely on machine numbers, unit names & numbers, etc) and putting that into /tmp to execute this, as it would ease the run time a bit; however, the amount of time spent running these scripts pales in comparison to the runtime of the `do-release-upgrade`, so I'm not sure of the real value, except to improve readability (a good value, but questionable to me given that this code has been written to be deleted).

I think we are good at this point. The previous comment were topics I felt were answered previously. This looks good. I'll be merging. Several specs all back to trunk.