Merge lp://staging/~xfactor973/charms/trusty/ceph-mon/coordinated-upgrade into lp://staging/~openstack-charmers-archive/charms/trusty/ceph-mon/next

Note: I put up the helper functions for review in charmhelpers: https://code.launchpad.net/~xfactor973/charm-helpers/ceph-keystore/+merge/287205

charm_lint_check #1558 ceph-mon-next for xfactor973 mp287375
    LINT FAIL: lint-test failed

LINT Results (max last 2 lines):
make: *** [lint] Error 1
ERROR:root:Make target returned non-zero.

Full lint test output: http://paste.ubuntu.com/15210769/
Build: http://10.245.162.36:8080/job/charm_lint_check/1558/

charm_unit_test #1307 ceph-mon-next for xfactor973 mp287375
    UNIT OK: passed

Build: http://10.245.162.36:8080/job/charm_unit_test/1307/

charm_amulet_test #553 ceph-mon-next for xfactor973 mp287375
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/15210927/
Build: http://10.245.162.36:8080/job/charm_amulet_test/553/

charm_lint_check #1561 ceph-mon-next for xfactor973 mp287375
    LINT FAIL: lint-test failed

LINT Results (max last 2 lines):
make: *** [lint] Error 1
ERROR:root:Make target returned non-zero.

Full lint test output: http://paste.ubuntu.com/15210963/
Build: http://10.245.162.36:8080/job/charm_lint_check/1561/

charm_unit_test #1310 ceph-mon-next for xfactor973 mp287375
    UNIT FAIL: unit-test failed

UNIT Results (max last 2 lines):
make: *** [test] Error 1
ERROR:root:Make target returned non-zero.

Full unit test output: http://paste.ubuntu.com/15210971/
Build: http://10.245.162.36:8080/job/charm_unit_test/1310/

charm_amulet_test #557 ceph-mon-next for xfactor973 mp287375
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/15211079/
Build: http://10.245.162.36:8080/job/charm_amulet_test/557/

Ignoring the numerous failures for now - they need resolving but wanted to have a first pass on principles for upgrading.

This proposal introduces the concept of using packages from ceph.com to drive the upgrade process; in principle I'm happy that we add support for use of packages from ceph.com, but lets do that under separate cover.

I think the initial target is to roll between Ceph versions in Ubuntu Cloud Archive pockets, in much the same way that we do for OpenStack.

This will mean initial support is required for trusty (as that has visibility of firefly, hammer, and infernalis/jewel) with support being required for xenial at some point (when we get a new version of ceph in a future cloud archive pocker); here are the supported combinations:

trusty: firefly
trusty-juno: (firefly from 14.04)
trusty-kilo: (hammer)
trusty-liberty: (hammer)
trusty-mitaka: (jewel)

I think the trigger for an upgrade should be a change in the source which requires that ceph be upgraded.

Re you use of status_set to feedback messages to the end user - this is super-useful, but please rework to actually set the status in assess_status - right now the status_set's you call in the upgrade functions will be overridden by the status_sets from assess_status.

I'd suggest that use use the unitdata charm-helper to manage states for the upgrade, and teach assess_status how to interpret these.

Some inline comments; I'd also say that I'd prefer that general tidying and removal of unused functions as part of the move to ceph-mon be done in a different MP please.

Makes this one harder to review due to other general change noise from this work.

@jamespage - regarding the upstart specific code I did some digging and it looks like charmhelpers takes care of this for me: http://bazaar.launchpad.net/~charm-helpers/charm-helpers/devel/view/head:/charmhelpers/core/host.py#L120

For the status set bits my upgrade function actually blocks forever until the upgrade is complete. That's why I didn't add the messages into assess_status.

@xfactor973 - re the upstart specific code - I was referring to the fact that ceph-mon-all is a upstart specific job - i.e. no systemd equiv exists.

Moved to gerrit review.