maas-images

Merge lp://staging/~ltrager/maas-images/import into lp://staging/maas-images

  1. import
  2. Merge into maas-ephemerals
Proposed by Lee Trager
Status: Merged
Merged at revision: 231
Proposed branch: lp://staging/~ltrager/maas-images/import
Merge into: lp://staging/maas-images
Diff against target: 1261 lines (+1187/-1)
7 files modified
bin/maas-qcow2targz (+105/-0)
conf/centos.yaml (+19/-0)
curtin/centos6/curtin-hooks (+342/-0)
curtin/centos6/finalize (+100/-0)
curtin/centos7/curtin-hooks (+333/-0)
curtin/centos7/finalize (+100/-0)
meph2/commands/meph2_util.py (+188/-1)
To merge this branch: bzr merge lp://staging/~ltrager/maas-images/import
Reviewer Review Type Date Requested Status
maintainers of maas images Pending
Review via email: mp+269684@code.staging.launchpad.net

Description of the change

This is a different approach from the create patch(https://code.launchpad.net/~ltrager/maas-images/create/+merge/268816) to adding external images into the MAAS simplestream

* The import command reads from a specified yaml file which defines the source of the images, the content_id of the stream, a template for the product_id, and the versions to be downloaded
* I found that CentOS provides generic cloud images which use the standard kernel and have cloud-init preinstalled in qcow2 format. Since they don't provide a simplestream containing these images but do provide a sha256sum.txt in a static location I use that as the source of my metadata.
* CentOS uses a date stamp as their version similar to how we use it so I just pull that string out as the simplestream version. Since CentOS published two versions of CentOS 7.0 we import two versions.
* To convert the qcow file into a root-tgz I use a shell script. I had to use a number of shell commands and it seemed easier to subprocess out to one script instead of multiple commands(also wanted to get the nice progress bar wget gives you when downloading the images).
* The script requires a kernel with the nbd module, mount-image-callback, and wget with ca-certificates
* CentOS 7.0 has deprecated python-oauth in favor of python-oauth2. This is despite the fact that the version of cloud-init that they ship with uses python-oauth in /usr/lib/python2.7/site-packages/cloudinit/sources/DataSourceMAAS.py. This forced me to pull a build of python-aouth from EPEL-testing.
* The curtin files were written by Blake which I pulled from http://bazaar.launchpad.net/~blake-rouse/maas-image-builder/trunk/files/head:/contrib/centos/

Example usage:
meph2-util import centos.yaml out

Example com.ubuntu.maas:centos-bases-download.json:
http://paste.ubuntu.com/12241076/

Example index.json:
http://paste.ubuntu.com/12241081/

I also created the merge subcommand which allows you to merge two simplestreams with different content_ids into one index.json file. The existing insert subcommand requires them to be the same.

Example usage:
meph2-util merge /path/to/centos/ /path/to/daily

Example combined index.json:
http://paste.ubuntu.com/12257427/

To post a comment you must log in.
lp://staging/~ltrager/maas-images/import updated
227. By Lee Trager

Use mount-image-callback and add license

228. By Lee Trager

Add merge subcommand to merge two different product streams together

Revision history for this message
Scott Moser (smoser) wrote :

generally, your shell needs lots more quotes. and white spaces are mixed with tabs in bin/maas-qcow2targz .
 - add '# vi: ts=4 expandtab' so that at least vi will do the right thing.

could you please run 'flake8' on the pythonn files and fix those errors?

Revision history for this message
Lee Trager (ltrager) :
lp://staging/~ltrager/maas-images/import updated
229. By Lee Trager

Incorporate smoser's comments

Revision history for this message
Lee Trager (ltrager) wrote :

I've made all the corrections you suggested with two exceptions

* While proc isn't strictly needed when modifying the image, RPM complains about proc missing if it isn't there
* MAAS will only deploy a root tarball if its type is root-tgz

Revision history for this message
Scott Moser (smoser) wrote :

wrt root-tgz, we need to fix maas rather than making *everyone* else accept inconsistency.

Revision history for this message
Andres Rodriguez (andreserl) wrote :

Well the fact that we are using root-tgz instead of root.tar.gz was decision that was made a very long time ago, in fact, I think it was something that was done by you Scott.

Provided that this was a decision made since very early versions of MAAS, and that's the current that we use in MAAS, we won't be changing the name to root.tar.gz and we will continue to use root-tgz. Because, this would not only impact MAAS in every single past release supported (1.2, 1.7, 1.9), it would also require to update all the streams.

We can, however, properly plan an update in the future, but provided it is a quite impacting change, then I think we would need to bump the version of the ephemeral streams to v3, and doing so, won't be supported in MAAS anytime soon.

Furthermore, for any further improvements you'd like to make, then they need to be discussed and planned at sprints and not ad-hoc as it is happening now. That being said, the MAAS team has made the commitment to maintain the ephemerals for the MAAS images, which means that at this time, we won't be doing any more work other than we have already committed to (for CentOS). Otherwise, we can discuss future improvements in the upcoming sprint.

Revision history for this message
Andres Rodriguez (andreserl) wrote :

And by ephemerals I mean the ephemeral streams.

Revision history for this message
Scott Moser (smoser) wrote :

You can fix that very easily now.
You're committing new code to trunk, support either 'root-tgz' or 'root.tar.gz'.
You can then at your leasure enable older versions of maas with an SRU to enable this properly.

WE NEED IT RIGHT NOW is not justification to do something that will hurt you later.
We want to have consistent file types for our images. Inconsistency hurts everywhere.

And 1.2 does not read this. It is only 1.7 and 1.9 that you're calling supported.
So thats exactly 1 release that you have to patch.

This is not 'ad-hoc'. You're wanting to change published data in a way inconsistent with existing published data. I'm saying thats a bad idea.

Revision history for this message
Scott Moser (smoser) wrote :

Can you explain where in maas code that 'root-tgz' is read?

Revision history for this message
Scott Moser (smoser) wrote :

I'm willing to accept this merge proposal with 'ftype' set to 'root-tgz'.

However, I'm bothered that we are creating inconsistent data on an ubuntu.com
website.

In published data on cloud-images.ubuntu.com/streams, we have used
the 'ftype' of 'root.tar.gz' to identify a "root tarball" (as you'd get
if you ran 'tar -C / -zf .'). The maas team has implemented code and has that
code in production that expects this 'ftype' to be 'root-tgz'.

That inconsitency means that in the future, clients that consume multiple
different stream sources (all under .ubuntu.com) and want to find images that
are "root tarball" type, will have to look for either 'root-tgz' or
'root.tar.gz'.

That is quite unfortunate, and in my opinion could almost certainly be avoided.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
The diff is not available at this time. You can reload the page or download it.

Subscribers

People subscribed via source and target branches