Merge lp://staging/~joetalbott/ubuntu-ci-services-itself/data-store into lp://staging/ubuntu-ci-services-itself

On 12/06/2013 08:08 AM, Joe Talbott wrote:
> +put_file(filename, private=False)
> + Add a file to the data store.

Does this need logic to deal with the remote path on disk versus the
relative path in Swift? I'm not sure the mechanics in swift but maybe a
simpler approach would be something like:

  def create_file(filename, private=False):
      # do something to create a Python FD.
      yield fd

Then your logic for creating files might look like:

with open(resource, 'r') inf:
     with ds.create_file('/bad_example.log') as outf:
         outf.write(inf.read())

This might not make any sense in swift though.

> +get_file(filename)
> + Get a file from the data store.

On 6 December 2013 15:39, Andy Doan <email address hidden> wrote:
> On 12/06/2013 08:08 AM, Joe Talbott wrote:
>> +put_file(filename, private=False)
>> + Add a file to the data store.
>
> Does this need logic to deal with the remote path on disk versus the
> relative path in Swift? I'm not sure the mechanics in swift but maybe a
> simpler approach would be something like:
>
> def create_file(filename, private=False):
> # do something to create a Python FD.
> yield fd
>
> Then your logic for creating files might look like:
>
> with open(resource, 'r') inf:
> with ds.create_file('/bad_example.log') as outf:
> outf.write(inf.read())
>
> This might not make any sense in swift though.

Swift doesn't have any concept of paths. It's a key-value blobstore.
So write (put_object) is basically create and write:

http://bazaar.launchpad.net/~daisy-pluckers/daisy/trunk/view/head:/daisy/submit_core.py#L60

(Please forgive me for the terrible code)

we'll need to store things for the image builder obviously, but for cloud images (our first target) the resulting images will need to go to glance rather than the usual data store. Glance would still require cloud credentials from our side, much as I suspect you'll need it for this. Should this data store service also handle glance? I'm inclined to say "no" since I don't think it would normally be needed anywhere else besides the image builder, but I don't want to get into a mess of having to keep the credentials in lots of places either.

At a minimum for phase 0 we need to dump artifacts into the data store and retrieve them via the Web UI. One way to provide this is to return the artifact URL on put_file (does this also assume that everything is public). The components can just pass these URLs up to the project manager when then makes them available through the web UI with no need to go through the library.

We also could use a "list_contents()" method to list the objects in a DataStore object, at least for debugging purposes.

I also recommend swapping the component, identifier container name:
DataStore(identifier, component=None)

My idea is that identifier == the ticket name (i.e. "Request-1234") and the component maps to the service (i.e. "ppa-assigner"). To me, collecting all "Request-1234" artifacts into the same container is more useful then collecting all of the "ppa-assigner" artifact together.

Also this document isn't linked into the docs/components/planned.rst. As a result, it generates this error:

checking consistency... /tmp/data-store/docs/components/data-store.rst:: WARNING: document isn't included in any toctree

Updated to match the implementation and hooked into TOC.

Thanks for the update.

The attempt to merge lp:~joetalbott/ubuntu-ci-services-itself/data-store into lp:ubuntu-ci-services-itself failed. Below is the output from the failed tests.

New python executable in /tmp/tmp.IK5QfZ2I7L/bin/python
Installing distribute.............................................................................................................................................................................................done.
Installing pip...............done.
== Testing ci-utils ....
running develop
running egg_info
creating ci_utils.egg-info
writing requirements to ci_utils.egg-info/requires.txt
writing ci_utils.egg-info/PKG-INFO
writing top-level names to ci_utils.egg-info/top_level.txt
writing dependency_links to ci_utils.egg-info/dependency_links.txt
writing manifest file 'ci_utils.egg-info/SOURCES.txt'
reading manifest file 'ci_utils.egg-info/SOURCES.txt'
writing manifest file 'ci_utils.egg-info/SOURCES.txt'
running build_ext
Creating /tmp/tmp.IK5QfZ2I7L/lib/python2.7/site-packages/ci-utils.egg-link (link to .)
Adding ci-utils 0.1 to easy-install.pth file

Installed /tmp/tarmac/branch.dXIzBL/ci-utils
Processing dependencies for ci-utils==0.1
Searching for testtools
Reading http://pypi.python.org/simple/testtools/
Best match: testtools 0.9.34
Downloading https://pypi.python.org/packages/source/t/testtools/testtools-0.9.34.tar.gz#md5=51d37e7376a70cee40cf17b44889fc88
Processing testtools-0.9.34.tar.gz
Running testtools-0.9.34/setup.py -q bdist_egg --dist-dir /tmp/easy_install-J8sjPO/testtools-0.9.34/egg-dist-tmp-yeUmkz
Adding testtools 0.9.34 to easy-install.pth file

Installed /tmp/tmp.IK5QfZ2I7L/lib/python2.7/site-packages/testtools-0.9.34-py2.7.egg
Searching for restish==0.12.1
Reading http://pypi.python.org/simple/restish/
Best match: restish 0.12.1
Downloading https://pypi.python.org/packages/source/r/restish/restish-0.12.1.tar.gz#md5=c29e0b755c44c21659de8e463093ea47
Processing restish-0.12.1.tar.gz
Running restish-0.12.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-gTQrcx/restish-0.12.1/egg-dist-tmp-3gNrCA
Adding restish 0.12.1 to easy-install.pth file

Installed /tmp/tmp.IK5QfZ2I7L/lib/python2.7/site-packages/restish-0.12.1-py2.7.egg
Searching for python-subunit
Reading http://pypi.python.org/simple/python-subunit/
Reading http://launchpad.net/subunit
Best match: python-subunit 0.0.16
Downloading https://pypi.python.org/packages/source/p/python-subunit/python-subunit-0.0.16.tar.gz#md5=c0ec919f8a1051de4ad89000f95324aa
Processing python-subunit-0.0.16.tar.gz
Running python-subunit-0.0.16/setup.py -q bdist_egg --dist-dir /tmp/easy_install-_PAOeK/python-subunit-0.0.16/egg-dist-tmp-ZjLAE2
Adding python-subunit 0.0.16 to easy-install.pth file
Installing subunit-tags script to /tmp/tmp.IK5QfZ2I7L/bin
Installing subunit-notify script to /tmp/tmp.IK5QfZ2I7L/bin
Installing subunit-1to2 script to /tmp/tmp.IK5QfZ2I7L/bin
Installing tap2subunit script to /tmp/tmp.IK5QfZ2I7L/bin
Installing subunit2gtk script to /tmp/tmp.IK5QfZ2I7L/bin
Installing subunit-ls script to /tmp/tmp.IK5QfZ2I7L/bin
Installing subunit-stats script to /tmp/tmp.IK5QfZ2I7L/bin
Installing subunit2pyunit script to /tmp/tmp.IK5QfZ2I7L/bin
Installing subunit-filter script ...

The attempt to merge lp:~joetalbott/ubuntu-ci-services-itself/data-store into lp:ubuntu-ci-services-itself failed. Below is the output from the failed tests.

New python executable in /tmp/tmp.ypQ9XnFzpV/bin/python
Installing distribute.............................................................................................................................................................................................done.
Installing pip...............done.
== Testing ci-utils ....
running develop
running egg_info
creating ci_utils.egg-info
writing requirements to ci_utils.egg-info/requires.txt
writing ci_utils.egg-info/PKG-INFO
writing top-level names to ci_utils.egg-info/top_level.txt
writing dependency_links to ci_utils.egg-info/dependency_links.txt
writing manifest file 'ci_utils.egg-info/SOURCES.txt'
reading manifest file 'ci_utils.egg-info/SOURCES.txt'
writing manifest file 'ci_utils.egg-info/SOURCES.txt'
running build_ext
Creating /tmp/tmp.ypQ9XnFzpV/lib/python2.7/site-packages/ci-utils.egg-link (link to .)
Adding ci-utils 0.1 to easy-install.pth file

Installed /tmp/tarmac/branch.VOeoKZ/ci-utils
Processing dependencies for ci-utils==0.1
Searching for testtools
Reading http://pypi.python.org/simple/testtools/
Best match: testtools 0.9.34
Downloading https://pypi.python.org/packages/source/t/testtools/testtools-0.9.34.tar.gz#md5=51d37e7376a70cee40cf17b44889fc88
Processing testtools-0.9.34.tar.gz
Running testtools-0.9.34/setup.py -q bdist_egg --dist-dir /tmp/easy_install-AdzFi4/testtools-0.9.34/egg-dist-tmp-ybHzC2
Adding testtools 0.9.34 to easy-install.pth file

Installed /tmp/tmp.ypQ9XnFzpV/lib/python2.7/site-packages/testtools-0.9.34-py2.7.egg
Searching for restish==0.12.1
Reading http://pypi.python.org/simple/restish/
Best match: restish 0.12.1
Downloading https://pypi.python.org/packages/source/r/restish/restish-0.12.1.tar.gz#md5=c29e0b755c44c21659de8e463093ea47
Processing restish-0.12.1.tar.gz
Running restish-0.12.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-tQkvC5/restish-0.12.1/egg-dist-tmp-XdNhwm
Adding restish 0.12.1 to easy-install.pth file

Installed /tmp/tmp.ypQ9XnFzpV/lib/python2.7/site-packages/restish-0.12.1-py2.7.egg
Searching for python-subunit
Reading http://pypi.python.org/simple/python-subunit/
Reading http://launchpad.net/subunit
Best match: python-subunit 0.0.16
Downloading https://pypi.python.org/packages/source/p/python-subunit/python-subunit-0.0.16.tar.gz#md5=c0ec919f8a1051de4ad89000f95324aa
Processing python-subunit-0.0.16.tar.gz
Running python-subunit-0.0.16/setup.py -q bdist_egg --dist-dir /tmp/easy_install-mLNwXf/python-subunit-0.0.16/egg-dist-tmp-i7krRG
Adding python-subunit 0.0.16 to easy-install.pth file
Installing subunit-tags script to /tmp/tmp.ypQ9XnFzpV/bin
Installing subunit-notify script to /tmp/tmp.ypQ9XnFzpV/bin
Installing subunit-1to2 script to /tmp/tmp.ypQ9XnFzpV/bin
Installing tap2subunit script to /tmp/tmp.ypQ9XnFzpV/bin
Installing subunit2gtk script to /tmp/tmp.ypQ9XnFzpV/bin
Installing subunit-ls script to /tmp/tmp.ypQ9XnFzpV/bin
Installing subunit-stats script to /tmp/tmp.ypQ9XnFzpV/bin
Installing subunit2pyunit script to /tmp/tmp.ypQ9XnFzpV/bin
Installing subunit-filter script ...