Merge lp://staging/~jderose/filestore/helpers into lp://staging/filestore

Proposed by Jason Gerard DeRose
Status: Merged
Merged at revision: 190
Proposed branch: lp://staging/~jderose/filestore/helpers
Merge into: lp://staging/filestore
Diff against target: 453 lines (+251/-50)
2 files modified
filestore.py (+95/-42)
test_filestore.py (+156/-8)
To merge this branch: bzr merge lp://staging/~jderose/filestore/helpers
Reviewer Review Type Date Requested Status
David Jordan Approve
Review via email: mp+74932@code.staging.launchpad.net

Description of the change

This merge:

* Adds 3 new highish-level FileStore methods:

  FileStore.content_md5(_id) - compute a known correct MD5 hash (for uploading to S3 using boto)

  FileStore.verify_and_move(tmp_fp, _id) - check content hash of a file in .dmedia/partial/, and if correct, move the file into its canonical location. This replaces an equivalent method of the legacy FileStore that is used by the S3 and BitTorrent backends.

  FileStore.hash_and_move(tmp_fp) - calculate content hash of a file in .dmedia/tmp/, and then move the file into its canonical location. This replaces an equivalent method of the legacy FileStore that is used by the transcoder.

* Makes the use of the `Leaf` namedtuple consistent. Experience has shown the code is simpler and more robust if you keep track of the position in the file (leaf_index) a given chunk of leaf_data corresponds to. As of several revisions ago, when a leaf is read by reader() or batch_reader(), the index and data are bundled together right there in a Leaf(index, data) namedtuple. But this merge fixes some inconsistencies higher up:

  - Consumers of `Leaf` no longer unpack it into (i, data)

  - Consumers of `Leaf` yield/return the exact `Leaf` rather than (i, data)

  - Hasher.update() has been renamed to the clearer Hasher.hash_leaf()

  - Hasher.hash_leaf() now takes a `Leaf` instead of `bytes`

  - Hasher is now more robust as it makes sure the Leaf.index from the readers matches up with its expected leaf_index

To post a comment you must log in.
Revision history for this message
David Jordan (dmj726) wrote :

Looks a lot simpler. Approved.

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
The diff is not available at this time. You can reload the page or download it.

Subscribers

People subscribed via source and target branches