Merge lp://staging/~jderose/filestore/fadvise into lp://staging/filestore
Proposed by
Jason Gerard DeRose
| Status: | Merged |
|---|---|
| Merged at revision: | 268 |
| Proposed branch: | lp://staging/~jderose/filestore/fadvise |
| Merge into: | lp://staging/filestore |
| Diff against target: |
208 lines (+68/-24) 2 files modified
benchmark.py (+25/-22) filestore.py (+43/-2) |
| To merge this branch: | bzr merge lp://staging/~jderose/filestore/fadvise |
| Related bugs: |
| Reviewer | Review Type | Date Requested | Status |
|---|---|---|---|
| James Raymond | Approve | ||
|
Review via email:
|
|||
Description of the change
In the future we should experiment with POSIX_FADV_
Changes:
* When needed, monkey patch `os` module with dummy `os.posix_
* reader() now calls on.posix_fadvise() with os.POSIX_
* batch_reader() now calls on.posix_fadvise() with os.POSIX_
* batch_import_iter() now uses a queue size of 8 instead of 16 (meaning now at most 64 MiB will be in-flight in the queue, instead of 128 MiB)
* reader() and batch_reader() now log.exception() when an exception is caught to make debugging easier
* benchmark.py script now uses batch_import_
To post a comment you must log in.
Revision history for this message
| Jason Gerard DeRose (jderose) wrote | # |
Revision history for this message
| James Raymond (jamesmr) | # |
review:
Approve
Preview Diff
[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
Okay, I decided to undergo said experiment right now... and holy crap, fast.
Using os.POSIX_
31.2 MB/s over USB2, up from 26.5 MB/s.
There are a few things that need further tuning on the Dmedia side of things, but the filestore read performance is now pretty stellar (under Python3.3 anyway).