Merge into trunk : dup-boot-source-selections-bug-1360280 : Code : MAAS

Status:

Merged

Approved by:

Julian Edwards on 2014-11-12

Approved revision:

no longer in the source branch.

Merged at revision:

3361

Proposed branch:

lp://staging/~julian-edwards/maas/dup-boot-source-selections-bug-1360280

Merge into:

lp://staging/~maas-committers/maas/trunk

Diff against target:

867 lines (+832/-0)

4 files modified

src/maasserver/migrations/0115_unique_boot_source_selections.py (+433/-0)
src/maasserver/migrations/0116_unique_boot_source_selections.py (+373/-0)
src/maasserver/models/bootsourceselection.py (+1/-0)
src/maasserver/tests/test_forms_bootsourceselection.py (+25/-0)

To merge this branch:

bzr merge lp://staging/~julian-edwards/maas/dup-boot-source-selections-bug-1360280

High

Fix Released

Link a bug report

Reviewer	Review Type	Date Requested	Status
Graham Binns (community)		2014-11-10	Approve on 2014-11-11
Review via email: mp+241235@code.staging.launchpad.net

Commit message

Make os, release and boot_source unique for BootSourceSelection to stop duplicate entries.

Description of the change

Add a unique index across boot_source, os and release on BootSourceSelections.

This will obviously potentially fail a migration if someone has dupes already, but this really needs to go in. Since nobody was around to do a pre-imp I'm throwing this up as-is to discuss ideas:

1. Land it as-is and if it fails add a release note telling someone to remove the dupes manually (so they get to choose which to keep)
2. Add code to the migration to force-delete all but the newest entry for each boot source.

Anything else?

Revision history for this message

Graham Binns (gmb) wrote on 2014-11-10:

#

ISTR Blake saying that we had to coalesce the entries for one OS/release rather than just dropping all-but-one, because the entries may all have different arches, subarches and labels — all of which are perfectly valid choices, but which need to be recorded in a single row, not several.

So, if we don't want to have settings go away surprisingly, (1) is a good option.

That said, could we add a persistent, user-clearable error* after doing (2) that says something like "MAAS's records of your boot source settings may have changed; check the Images tab to ensure that your boot image settings are correct."

*Not sure we actually have the facility to do this; I was thinking of the persistent error stuff, but that all seems to be auto-cleared or not at all.

Revision history for this message

Julian Edwards (julian-edwards) wrote on 2014-11-10:

#

On Monday 10 Nov 2014 07:18:40 you wrote:
> ISTR Blake saying that we had to coalesce the entries for one OS/release
> rather than just dropping all-but-one, because the entries may all have
> different arches, subarches and labels — all of which are perfectly valid
> choices, but which need to be recorded in a single row, not several.
>
> So, if we don't want to have settings go away surprisingly, (1) is a good
> option.
>
> That said, could we add a persistent, user-clearable error* after doing (2)
> that says something like "MAAS's records of your boot source settings may
> have changed; check the Images tab to ensure that your boot image settings
> are correct."
>
> *Not sure we actually have the facility to do this; I was thinking of the
> persistent error stuff, but that all seems to be auto-cleared or not at
> all.

Actually I hadn't considered coalescing them - good point! I'll write some
runes to do that later.

Revision history for this message

Julian Edwards (julian-edwards) wrote on 2014-11-11:

#

Right, coalescing migration added, care to take another look?

I have tested this reasonably thoroughly in the dev environment, I used this script to poke values in http://paste.ubuntu.com/8935209/ and the end result looked exactly right (see comments in that script).

Note that there's two migrations as you can't do data and schema changes in the same migration.

Revision history for this message

Graham Binns (gmb) wrote on 2014-11-11:

#

Tip top. Really nice work, Jools!

review: Approve

Revision history for this message

MAAS Lander (maas-lander) wrote on 2014-11-12:

#

Download full text (20.7 KiB)

The attempt to merge lp:~julian-edwards/maas/dup-boot-source-selections-bug-1360280 into lp:maas failed. Below is the output from the failed tests.

Ign http://nova.clouds.archive.ubuntu.com trusty InRelease
Ign http://security.ubuntu.com trusty-security InRelease
Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
Ign http://nova.clouds.archive.ubuntu.com trusty-updates InRelease
Get:2 http://security.ubuntu.com trusty-security Release [62.0 kB]
Hit http://nova.clouds.archive.ubuntu.com trusty Release.gpg
Get:3 http://nova.clouds.archive.ubuntu.com trusty-updates Release.gpg [933 B]
Hit http://nova.clouds.archive.ubuntu.com trusty Release
Get:4 http://nova.clouds.archive.ubuntu.com trusty-updates Release [62.0 kB]
Hit http://nova.clouds.archive.ubuntu.com trusty/main Sources
Hit http://nova.clouds.archive.ubuntu.com trusty/universe Sources
Hit http://nova.clouds.archive.ubuntu.com trusty/main amd64 Packages
Get:5 http://security.ubuntu.com trusty-security/main Sources [49.5 kB]
Hit http://nova.clouds.archive.ubuntu.com trusty/universe amd64 Packages
Hit http://nova.clouds.archive.ubuntu.com trusty/main Translation-en
Hit http://nova.clouds.archive.ubuntu.com trusty/universe Translation-en
Get:6 http://nova.clouds.archive.ubuntu.com trusty-updates/main Sources [136 kB]
Get:7 http://security.ubuntu.com trusty-security/universe Sources [13.1 kB]
Get:8 http://security.ubuntu.com trusty-security/main amd64 Packages [153 kB]
Get:9 http://nova.clouds.archive.ubuntu.com trusty-updates/universe Sources [89.5 kB]
Get:10 http://security.ubuntu.com trusty-security/universe amd64 Packages [60.5 kB]
Get:11 http://nova.clouds.archive.ubuntu.com trusty-updates/main amd64 Packages [356 kB]
Hit http://security.ubuntu.com trusty-security/main Translation-en
Hit http://security.ubuntu.com trusty-security/universe Translation-en
Get:12 http://nova.clouds.archive.ubuntu.com trusty-updates/universe amd64 Packages [217 kB]
Hit http://nova.clouds.archive.ubuntu.com trusty-updates/main Translation-en
Hit http://nova.clouds.archive.ubuntu.com trusty-updates/universe Translation-en
Ign http://nova.clouds.archive.ubuntu.com trusty/main Translation-en_US
Ign http://nova.clouds.archive.ubuntu.com trusty/universe Translation-en_US
Fetched 1,200 kB in 2s (411 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
--no-install-recommends install apache2 authbind bind9 bind9utils build-essential bzr-builddeb curl daemontools debhelper dh-apport distro-info dnsutils firefox freeipmi-tools gjs ipython isc-dhcp-common libjs-raphael libjs-yui3-full libjs-yui3-min libpq-dev make pep8 postgresql pyflakes python-amqplib python-bzrlib python-celery python-convoy python-crochet python-cssselect python-curtin python-dev python-distro-info python-django python-django-piston python-django-south python-djorm-ext-pgarray python-docutils python-extras python-fixtures python-flake8 python-formencode python-hivex python-httplib2 python-jinja2 python-jsonschema python-lockfile python-lxml python-mimeparse python-mock python-netaddr python-netifaces python-nose python-oauth python-oops python-oops-amqp python-oops-datedir-repo python-oops-twisted p...

The attempt to merge lp:~julian-edwards/maas/dup-boot-source-selections-bug-1360280 into lp:maas failed. Below is the output from the failed tests.

Ign http://nova.clouds.archive.ubuntu.com trusty InRelease
Ign http://security.ubuntu.com trusty-security InRelease
Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
Ign http://nova.clouds.archive.ubuntu.com trusty-updates InRelease
Get:2 http://security.ubuntu.com trusty-security Release [62.0 kB]
Hit http://nova.clouds.archive.ubuntu.com trusty Release.gpg
Get:3 http://nova.clouds.archive.ubuntu.com trusty-updates Release.gpg [933 B]
Hit http://nova.clouds.archive.ubuntu.com trusty Release
Get:4 http://nova.clouds.archive.ubuntu.com trusty-updates Release [62.0 kB]
Hit http://nova.clouds.archive.ubuntu.com trusty/main Sources
Hit http://nova.clouds.archive.ubuntu.com trusty/universe Sources
Hit http://nova.clouds.archive.ubuntu.com trusty/main amd64 Packages
Get:5 http://security.ubuntu.com trusty-security/main Sources [49.5 kB]
Hit http://nova.clouds.archive.ubuntu.com trusty/universe amd64 Packages
Hit http://nova.clouds.archive.ubuntu.com trusty/main Translation-en
Hit http://nova.clouds.archive.ubuntu.com trusty/universe Translation-en
Get:6 http://nova.clouds.archive.ubuntu.com trusty-updates/main Sources [136 kB]
Get:7 http://security.ubuntu.com trusty-security/universe Sources [13.1 kB]
Get:8 http://security.ubuntu.com trusty-security/main amd64 Packages [153 kB]
Get:9 http://nova.clouds.archive.ubuntu.com trusty-updates/universe Sources [89.5 kB]
Get:10 http://security.ubuntu.com trusty-security/universe amd64 Packages [60.5 kB]
Get:11 http://nova.clouds.archive.ubuntu.com trusty-updates/main amd64 Packages [356 kB]
Hit http://security.ubuntu.com trusty-security/main Translation-en
Hit http://security.ubuntu.com trusty-security/universe Translation-en
Get:12 http://nova.clouds.archive.ubuntu.com trusty-updates/universe amd64 Packages [217 kB]
Hit http://nova.clouds.archive.ubuntu.com trusty-updates/main Translation-en
Hit http://nova.clouds.archive.ubuntu.com trusty-updates/universe Translation-en
Ign http://nova.clouds.archive.ubuntu.com trusty/main Translation-en_US
Ign http://nova.clouds.archive.ubuntu.com trusty/universe Translation-en_US
Fetched 1,200 kB in 2s (411 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
	    --no-install-recommends install apache2 authbind bind9 bind9utils build-essential bzr-builddeb curl daemontools debhelper dh-apport distro-info dnsutils firefox freeipmi-tools gjs ipython isc-dhcp-common libjs-raphael libjs-yui3-full libjs-yui3-min libpq-dev make pep8 postgresql pyflakes python-amqplib python-bzrlib python-celery python-convoy python-crochet python-cssselect python-curtin python-dev python-distro-info python-django python-django-piston python-django-south python-djorm-ext-pgarray python-docutils python-extras python-fixtures python-flake8 python-formencode python-hivex python-httplib2 python-jinja2 python-jsonschema python-lockfile python-lxml python-mimeparse python-mock python-netaddr python-netifaces python-nose python-oauth python-oops python-oops-amqp python-oops-datedir-repo python-oops-twisted python-oops-wsgi python-openssl python-paramiko python-pexpect python-pip python-pocket-lint python-psycopg2 python-pyinotify python-seamicroclient python-simplejson python-simplestreams python-sphinx python-subunit python-tempita python-testresources python-testscenarios python-testtools python-twisted python-txamqp python-txtftp python-tz python-virtualenv python-yaml python-zope.interface syslinux-common tgt ubuntu-cloudimage-keyring wget xvfb
Reading package lists...
Building dependency tree...
Reading state information...
authbind is already the newest version.
bind9 is already the newest version.
bind9utils is already the newest version.
build-essential is already the newest version.
debhelper is already the newest version.
distro-info is already the newest version.
dnsutils is already the newest version.
isc-dhcp-common is already the newest version.
libjs-raphael is already the newest version.
libjs-yui3-full is already the newest version.
libjs-yui3-min is already the newest version.
make is already the newest version.
pep8 is already the newest version.
postgresql is already the newest version.
pyflakes is already the newest version.
python-amqplib is already the newest version.
python-celery is already the newest version.
python-convoy is already the newest version.
python-crochet is already the newest version.
python-curtin is already the newest version.
python-dev is already the newest version.
python-distro-info is already the newest version.
python-django-piston is already the newest version.
python-django-south is already the newest version.
python-djorm-ext-pgarray is already the newest version.
python-docutils is already the newest version.
python-extras is already the newest version.
python-fixtures is already the newest version.
python-formencode is already the newest version.
python-httplib2 is already the newest version.
python-jinja2 is already the newest version.
python-jsonschema is already the newest version.
python-lockfile is already the newest version.
python-mimeparse is already the newest version.
python-mock is already the newest version.
python-netaddr is already the newest version.
python-netifaces is already the newest version.
python-nose is already the newest version.
python-oauth is already the newest version.
python-oops is already the newest version.
python-oops-amqp is already the newest version.
python-oops-datedir-repo is already the newest version.
python-oops-twisted is already the newest version.
python-oops-wsgi is already the newest version.
python-openssl is already the newest version.
python-paramiko is already the newest version.
python-psycopg2 is already the newest version.
python-pyinotify is already the newest version.
python-simplejson is already the newest version.
python-subunit is already the newest version.
python-tempita is already the newest version.
python-testresources is already the newest version.
python-testscenarios is already the newest version.
python-testtools is already the newest version.
python-twisted is already the newest version.
python-txamqp is already the newest version.
python-txtftp is already the newest version.
python-tz is already the newest version.
python-yaml is already the newest version.
python-zope.interface is already the newest version.
syslinux-common is already the newest version.
tgt is already the newest version.
ubuntu-cloudimage-keyring is already the newest version.
bzr-builddeb is already the newest version.
daemontools is already the newest version.
ipython is already the newest version.
python-cssselect is already the newest version.
python-flake8 is already the newest version.
python-hivex is already the newest version.
python-pip is already the newest version.
python-pocket-lint is already the newest version.
python-virtualenv is already the newest version.
apache2 is already the newest version.
curl is already the newest version.
dh-apport is already the newest version.
firefox is already the newest version.
freeipmi-tools is already the newest version.
libpq-dev is already the newest version.
python-bzrlib is already the newest version.
python-django is already the newest version.
python-lxml is already the newest version.
python-pexpect is already the newest version.
python-seamicroclient is already the newest version.
python-simplestreams is already the newest version.
python-sphinx is already the newest version.
wget is already the newest version.
xvfb is already the newest version.
gjs is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 19 not upgraded.
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
	    purge 
Reading package lists...
Building dependency tree...
Reading state information...
0 upgraded, 0 newly installed, 0 to remove and 19 not upgraded.
make -C acceptance clean
make[1]: Entering directory `/tmp/tarmac/branch.PYXy_l/acceptance'
rm -f -r source build *.build *.changes *.deb
make[1]: Leaving directory `/tmp/tarmac/branch.PYXy_l/acceptance'
find . -type f -name '*.py[co]' -print0 | xargs -r0 rm -f
find . -type f -name '*~' -print0 | xargs -r0 rm -f
find . -type f -name dropin.cache -print0 | xargs -r0 rm -f
rm -f -r media/demo/* media/development
rm -f src/maasserver/static/js/enums.js
rm -f *.log
rm -f docs/api.rst
rm -f -r docs/_autosummary docs/_build
rm -f -r man/.doctrees
rm -f -r bin include lib local
rm -f -r eggs develop-eggs
rm -f -r build dist logs/* parts
rm -f tags TAGS .installed.cfg
rm -f -r *.egg *.egg-info src/*.egg-info
rm -f -r run/* services/*/supervise
virtualenv --python=python2.7 --system-site-packages /tmp/tarmac/branch.PYXy_l
New python executable in /tmp/tarmac/branch.PYXy_l/bin/python2.7
Also creating executable in /tmp/tarmac/branch.PYXy_l/bin/python
Installing setuptools, pip...done.
Running virtualenv with interpreter /usr/bin/python2.7
bin/python -m pip --quiet install --ignore-installed \
	    --no-dependencies bootstrap/zc.buildout-1.5.2.tar.gz
rm -f -f README.txt  # zc.buildout installs an annoying README.txt.
bin/buildout install database
Creating directory '/tmp/tarmac/branch.PYXy_l/parts'.
Creating directory '/tmp/tarmac/branch.PYXy_l/develop-eggs'.
Installing database.
Generated script '/tmp/tarmac/branch.PYXy_l/bin/database'.
bin/buildout install repl
Installing repl.
Generated script '/tmp/tarmac/branch.PYXy_l/bin/ipy'.
Generated interpreter '/tmp/tarmac/branch.PYXy_l/bin/py'.
bin/py -m src/maasserver/utils/jsenums src/maasserver/enum.py src/metadataserver/enum.py > src/maasserver/static/js/enums.js
bin/buildout install maas
Installing maas.
Generated script '/tmp/tarmac/branch.PYXy_l/bin/maas-region-admin'.
bin/buildout install maas-test
Installing maas-test.
Generated script '/tmp/tarmac/branch.PYXy_l/bin/test.maas'.
bin/buildout install maascli
Installing maascli.
Generated script '/tmp/tarmac/branch.PYXy_l/bin/maas'.
bin/buildout install maascli-test
Installing maascli-test.
Generated script '/tmp/tarmac/branch.PYXy_l/bin/test.maascli'.
bin/buildout install maastesting-test
Installing maastesting-test.
Generated script '/tmp/tarmac/branch.PYXy_l/bin/test.maastesting'.
bin/buildout install pserv
Installing pserv.
Generated script '/tmp/tarmac/branch.PYXy_l/bin/maas-provision'.
Generated script '/tmp/tarmac/branch.PYXy_l/bin/twistd.pserv'.
bin/buildout install pserv-test
Installing pserv-test.
Generated script '/tmp/tarmac/branch.PYXy_l/bin/test.pserv'.
bin/buildout install config-test
Installing config-test.
Generated script '/tmp/tarmac/branch.PYXy_l/bin/test.config'.
bin/buildout install maas-probe-dhcp
Installing maas-probe-dhcp.
Generated script '/tmp/tarmac/branch.PYXy_l/bin/maas-probe-dhcp'.
make -C acceptance clean
make[1]: Entering directory `/tmp/tarmac/branch.PYXy_l/acceptance'
rm -f -r source build *.build *.changes *.deb
make[1]: Leaving directory `/tmp/tarmac/branch.PYXy_l/acceptance'
find . -type f -name '*.py[co]' -print0 | xargs -r0 rm -f
find . -type f -name '*~' -print0 | xargs -r0 rm -f
find . -type f -name dropin.cache -print0 | xargs -r0 rm -f
rm -f -r media/demo/* media/development
rm -f src/maasserver/static/js/enums.js
rm -f *.log
rm -f docs/api.rst
rm -f -r docs/_autosummary docs/_build
rm -f -r man/.doctrees
bin/py -m src/maasserver/utils/jsenums src/maasserver/enum.py src/metadataserver/enum.py > src/maasserver/static/js/enums.js
bin/buildout install maas
Updating maas.
bin/buildout install maas-test
Updating maas-test.
echo bin/test.config bin/test.maas bin/test.maascli bin/test.maastesting bin/test.pserv | xargs -n1 env
nosetests --exclude=provisioningserver --exclude=maastesting --exclude=maascli --logging-level=INFO --verbosity=1
Creating test database for alias 'default'...
Destroying test database for alias 'default'...

..............................................
----------------------------------------------------------------------
Ran 46 tests in 0.466s

OK
..S.....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................S...........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................S.............................................................................................................................................................S...................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................SSE.......................................................................................................................................................................................
======================================================================
ERROR: test suite for <class 'maasserver.views.tests.test_nodes.NodeProbedDetailsExpanderTest'>
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/suite.py", line 208, in run
    self.setUp()
  File "/usr/lib/python2.7/dist-packages/nose/suite.py", line 291, in setUp
    self.setupContext(ancestor)
  File "/usr/lib/python2.7/dist-packages/nose/suite.py", line 314, in setupContext
    try_run(context, names)
  File "/usr/lib/python2.7/dist-packages/nose/util.py", line 471, in try_run
    return func()
  File "/tmp/tarmac/branch.PYXy_l/src/maasserver/testing/testcase.py", line 163, in setUpClass
    cls.selenium = WebDriver()
  File "/home/ubuntu/.buildout/eggs/selenium-2.43.0-py2.7.egg/selenium/webdriver/firefox/webdriver.py", line 59, in __init__
    self.binary, timeout),
  File "/home/ubuntu/.buildout/eggs/selenium-2.43.0-py2.7.egg/selenium/webdriver/firefox/extension_connection.py", line 47, in __init__
    self.binary.launch_browser(self.profile)
  File "/home/ubuntu/.buildout/eggs/selenium-2.43.0-py2.7.egg/selenium/webdriver/firefox/firefox_binary.py", line 64, in launch_browser
    self._wait_until_connectable()
  File "/home/ubuntu/.buildout/eggs/selenium-2.43.0-py2.7.egg/selenium/webdriver/firefox/firefox_binary.py", line 103, in _wait_until_connectable
    self._get_firefox_output())
WebDriverException: Message: 'The browser appears to have exited before we could connect. The output was: None'

----------------------------------------------------------------------
Ran 3887 tests in 277.001s

FAILED (SKIP=6, errors=1)
...............................................................................................................................
----------------------------------------------------------------------
Ran 127 tests in 1.897s

OK
.............................................................S..F.....................................................................................
======================================================================
FAIL: maastesting.tests.test_lint.TestLint.test_that_imports_are_formatted
----------------------------------------------------------------------
_StringException: diff: {{{
--- /tmp/tarmac/branch.PYXy_l/src/maasserver/migrations/0115_unique_boot_source_selections.py
+++ /tmp/tarmac/branch.PYXy_l/.export.DkO4Hv/src/maasserver/migrations/0115_unique_boot_source_selections.py
@@ -1,4 +1,5 @@
 from collections import namedtuple
+
 from django.db import models
 from south.db import db
 # -*- coding: utf-8 -*-
}}}

stdout/err from `make --quiet -C /tmp/tarmac/branch.PYXy_l/.export.DkO4Hv format`: {{{src/maasserver/migrations/0115_unique_boot_source_selections.py}}}

Traceback (most recent call last):
  File "/tmp/tarmac/branch.PYXy_l/src/maastesting/tests/test_lint.py", line 114, in test_that_imports_are_formatted
    "Some imports are not formatted; see the diff for the "
  File "/usr/lib/python2.7/unittest/case.py", line 412, in fail
    raise self.failureException(msg)
AssertionError: Some imports are not formatted; see the diff for the missing changes. Use `make format` to address them.

----------------------------------------------------------------------
Ran 150 tests in 28.350s

FAILED (SKIP=1, failures=1)
.....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
----------------------------------------------------------------------
Ran 1717 tests in 15.739s

OK
make: *** [test] Error 123

MAAS

Merge lp://staging/~julian-edwards/maas/dup-boot-source-selections-bug-1360280 into lp://staging/~maas-committers/maas/trunk

Commit message

Description of the change

Preview Diff

Subscribers