Juju Charms Collection
ceph-osd package

Merge lp://staging/~xfactor973/charms/trusty/ceph-osd/sysctl-perf into lp://staging/~openstack-charmers-archive/charms/trusty/ceph-osd/trunk

  1. Trusty Tahr (14.04)
  2. sysctl-perf
  3. Merge into trunk
Proposed by Chris Holcombe
Status: Superseded
Proposed branch: lp://staging/~xfactor973/charms/trusty/ceph-osd/sysctl-perf
Merge into: lp://staging/~openstack-charmers-archive/charms/trusty/ceph-osd/trunk
Diff against target: 550 lines (+307/-19)
10 files modified
.bzrignore (+3/-0)
.testr.conf (+8/-0)
config.yaml (+10/-1)
hooks/ceph.py (+203/-7)
hooks/ceph_hooks.py (+28/-10)
hooks/install (+1/-1)
requirements.txt (+11/-0)
templates/ceph.conf (+5/-0)
test-requirements.txt (+9/-0)
tox.ini (+29/-0)
To merge this branch: bzr merge lp://staging/~xfactor973/charms/trusty/ceph-osd/sysctl-perf
Reviewer Review Type Date Requested Status
James Page Needs Fixing
Review via email: mp+284033@code.staging.launchpad.net

This proposal has been superseded by a proposal from 2016-02-18.

Description of the change

This branch adds some HDD and network adapter sysctl tuning. Lots of potential performance has been left on the table with our out of the box ceph setup. This aims to recapture some of that.

For the HDD portion: Linux has set very conservative settings when it comes to HDDs. While this is great for desktop responsiveness it isn't so kind to storage servers. I have exposed a configuration option if an administrator knows they have a RAID card with a lot of cache onboard. In a future patch set I will add some SSD tuning to take advantage of its unique characteristics.

Testing this has proven challenging. While it doesn't break anything on AWS that doesn't really prove anything either. If anyone has access to physical hardware to test this I would really appreciate it.

To post a comment you must log in.
Revision history for this message
Chris MacNaughton (chris.macnaughton) wrote :

See comment on network interfaces. I'd also like to see some validation on varied hardware (SSD / HDD) that this doesn't break things either way.

Revision history for this message
uosci-testing-bot (uosci-testing-bot) wrote :

charm_lint_check #18194 ceph-osd for xfactor973 mp284033
    LINT FAIL: lint-test failed

LINT Results (max last 2 lines):
make: *** [lint] Error 1
ERROR:root:Make target returned non-zero.

Full lint test output: http://paste.ubuntu.com/14678289/
Build: http://10.245.162.77:8080/job/charm_lint_check/18194/

Revision history for this message
uosci-testing-bot (uosci-testing-bot) wrote :

charm_unit_test #16953 ceph-osd for xfactor973 mp284033
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/16953/

Revision history for this message
uosci-testing-bot (uosci-testing-bot) wrote :

charm_amulet_test #9068 ceph-osd for xfactor973 mp284033
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/14678328/
Build: http://10.245.162.77:8080/job/charm_amulet_test/9068/

lp://staging/~xfactor973/charms/trusty/ceph-osd/sysctl-perf updated
51. By Chris Holcombe

Lint spacing

Revision history for this message
uosci-testing-bot (uosci-testing-bot) wrote :

charm_unit_test #16956 ceph-osd for xfactor973 mp284033
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/16956/

Revision history for this message
uosci-testing-bot (uosci-testing-bot) wrote :

charm_lint_check #18198 ceph-osd for xfactor973 mp284033
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/18198/

Revision history for this message
uosci-testing-bot (uosci-testing-bot) wrote :

charm_amulet_test #9071 ceph-osd for xfactor973 mp284033
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/14680501/
Build: http://10.245.162.77:8080/job/charm_amulet_test/9071/

lp://staging/~xfactor973/charms/trusty/ceph-osd/sysctl-perf updated
52. By Chris Holcombe

Change over to using network link speed as the key to sysctl tuning instead of the driver name. This is a more generic approach because most 10Gb links need roughly the same set of sysctls

Revision history for this message
Chris Holcombe (xfactor973) wrote :

@chris.macnaughton Ask and ye shall receive ;). This keys off of link speed now

Revision history for this message
Chris Holcombe (xfactor973) wrote :

Ignore the stupid line nits. That's from me using auto format in pycharms IDE

Revision history for this message
uosci-testing-bot (uosci-testing-bot) wrote :

charm_unit_test #113 ceph-osd for xfactor973 mp284033
    UNIT FAIL: unit-test failed

UNIT Results (max last 2 lines):
make: *** [test] Error 1
ERROR:root:Make target returned non-zero.

Full unit test output: http://paste.ubuntu.com/15002338/
Build: http://10.245.162.36:8080/job/charm_unit_test/113/

Revision history for this message
uosci-testing-bot (uosci-testing-bot) wrote :

charm_lint_check #130 ceph-osd for xfactor973 mp284033
    LINT FAIL: lint-test failed

LINT Results (max last 2 lines):
make: *** [lint] Error 1
ERROR:root:Make target returned non-zero.

Full lint test output: http://paste.ubuntu.com/15002365/
Build: http://10.245.162.36:8080/job/charm_lint_check/130/

Revision history for this message
uosci-testing-bot (uosci-testing-bot) wrote :

charm_amulet_test #12 ceph-osd for xfactor973 mp284033
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/15002518/
Build: http://10.245.162.36:8080/job/charm_amulet_test/12/

lp://staging/~xfactor973/charms/trusty/ceph-osd/sysctl-perf updated
53. By Chris Holcombe

Address the lint issues and try to solve the python-enum34 package missing with the unit tests

Revision history for this message
uosci-testing-bot (uosci-testing-bot) wrote :

charm_unit_test #132 ceph-osd for xfactor973 mp284033
    UNIT FAIL: unit-test failed

UNIT Results (max last 2 lines):
make: *** [test] Error 1
ERROR:root:Make target returned non-zero.

Full unit test output: http://paste.ubuntu.com/15002739/
Build: http://10.245.162.36:8080/job/charm_unit_test/132/

Revision history for this message
uosci-testing-bot (uosci-testing-bot) wrote :

charm_lint_check #140 ceph-osd for xfactor973 mp284033
    LINT OK: passed

Build: http://10.245.162.36:8080/job/charm_lint_check/140/

Revision history for this message
Ryan Beisner (1chb1n) wrote :

@xfactor973 The tests/* dir affects only Amulet test runs, and the 00-setup file is batch-maintained (may be overwritten).

Revision history for this message
uosci-testing-bot (uosci-testing-bot) wrote :

charm_amulet_test #27 ceph-osd for xfactor973 mp284033
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/15004320/
Build: http://10.245.162.36:8080/job/charm_amulet_test/27/

Revision history for this message
James Page (james-page) wrote :

Amulet test failure is genuine; install hook fails on 12.04:

2016-02-09 19:55:55 INFO install After this operation, 1467 kB of additional disk space will be used.
2016-02-09 19:55:55 INFO install Get:1 http://nova.clouds.archive.ubuntu.com/ubuntu/ precise/main python-setuptools all 0.6.24-1ubuntu1 [441 kB]
2016-02-09 19:55:55 INFO install Get:2 http://nova.clouds.archive.ubuntu.com/ubuntu/ precise/universe python-pip all 1.0-1build1 [95.1 kB]
2016-02-09 19:55:55 INFO install Fetched 536 kB in 0s (25.1 MB/s)
2016-02-09 19:55:55 INFO install Selecting previously unselected package python-setuptools.
2016-02-09 19:55:55 INFO install (Reading database ... 47935 files and directories currently installed.)
2016-02-09 19:55:55 INFO install Unpacking python-setuptools (from .../python-setuptools_0.6.24-1ubuntu1_all.deb) ...
2016-02-09 19:55:55 INFO install Selecting previously unselected package python-pip.
2016-02-09 19:55:55 INFO install Unpacking python-pip (from .../python-pip_1.0-1build1_all.deb) ...
2016-02-09 19:55:55 INFO install Processing triggers for man-db ...
2016-02-09 19:55:55 INFO install Setting up python-setuptools (0.6.24-1ubuntu1) ...
2016-02-09 19:55:55 INFO install Setting up python-pip (1.0-1build1) ...
2016-02-09 19:55:56 INFO install Package `python-enum34' is not installed and no info is available.
2016-02-09 19:55:56 INFO install Use dpkg --info (= dpkg-deb --info) to examine archive files,
2016-02-09 19:55:56 INFO install and dpkg --contents (= dpkg-deb --contents) to list their contents.
2016-02-09 19:55:56 INFO install Reading package lists...
2016-02-09 19:55:56 INFO install Building dependency tree...
2016-02-09 19:55:56 INFO install Reading state information...
2016-02-09 19:55:57 INFO install E: Unable to locate package python-enum34
2016-02-09 19:55:57 INFO install Traceback (most recent call last):
2016-02-09 19:55:57 INFO install File "/var/lib/juju/agents/unit-ceph-osd-0/charm/hooks/install.real", line 16, in <module>
2016-02-09 19:55:57 INFO install import ceph
2016-02-09 19:55:57 INFO install File "/var/lib/juju/agents/unit-ceph-osd-0/charm/hooks/ceph.py", line 9, in <module>
2016-02-09 19:55:57 INFO install from enum import Enum
2016-02-09 19:55:57 INFO install ImportError: No module named enum

review: Needs Fixing
Revision history for this message
James Page (james-page) wrote :

I'll have to defer to your knowledge on nic and block device tuning as I can't directly validate that - one question - are the changes make to interface and block device parameters persistent? I don't think that they are, so a server reboot will hit reset on tuned params.

review: Needs Information
lp://staging/~xfactor973/charms/trusty/ceph-osd/sysctl-perf updated
54. By Chris Holcombe

Persist the changes for reboots. Also catch IOError in all the posix operations

55. By Chris Holcombe

Pull in the tox files from next

Revision history for this message
uosci-testing-bot (uosci-testing-bot) wrote :

charm_unit_test #315 ceph-osd for xfactor973 mp284033
    UNIT FAIL: unit-test failed

UNIT Results (max last 2 lines):
make: *** [test] Error 1
ERROR:root:Make target returned non-zero.

Full unit test output: http://paste.ubuntu.com/15031007/
Build: http://10.245.162.36:8080/job/charm_unit_test/315/

Revision history for this message
uosci-testing-bot (uosci-testing-bot) wrote :

charm_lint_check #395 ceph-osd for xfactor973 mp284033
    LINT OK: passed

Build: http://10.245.162.36:8080/job/charm_lint_check/395/

Revision history for this message
uosci-testing-bot (uosci-testing-bot) wrote :

charm_amulet_test #166 ceph-osd for xfactor973 mp284033
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/15031371/
Build: http://10.245.162.36:8080/job/charm_amulet_test/166/

Revision history for this message
uosci-testing-bot (uosci-testing-bot) wrote :

charm_unit_test #373 ceph-osd for xfactor973 mp284033
    UNIT FAIL: unit-test failed

UNIT Results (max last 2 lines):
make: *** [test] Error 1
ERROR:root:Make target returned non-zero.

Full unit test output: http://paste.ubuntu.com/15032005/
Build: http://10.245.162.36:8080/job/charm_unit_test/373/

lp://staging/~xfactor973/charms/trusty/ceph-osd/sysctl-perf updated
56. By Chris Holcombe

Removed enum dependency

Revision history for this message
Chris Holcombe (xfactor973) wrote :

I dumped the enum dependency :)

On 02/11/2016 05:46 AM, James Page wrote:
> Review: Needs Fixing
>
> Amulet test failure is genuine; install hook fails on 12.04:
>
> 2016-02-09 19:55:55 INFO install After this operation, 1467 kB of additional disk space will be used.
> 2016-02-09 19:55:55 INFO install Get:1 http://nova.clouds.archive.ubuntu.com/ubuntu/ precise/main python-setuptools all 0.6.24-1ubuntu1 [441 kB]
> 2016-02-09 19:55:55 INFO install Get:2 http://nova.clouds.archive.ubuntu.com/ubuntu/ precise/universe python-pip all 1.0-1build1 [95.1 kB]
> 2016-02-09 19:55:55 INFO install Fetched 536 kB in 0s (25.1 MB/s)
> 2016-02-09 19:55:55 INFO install Selecting previously unselected package python-setuptools.
> 2016-02-09 19:55:55 INFO install (Reading database ... 47935 files and directories currently installed.)
> 2016-02-09 19:55:55 INFO install Unpacking python-setuptools (from .../python-setuptools_0.6.24-1ubuntu1_all.deb) ...
> 2016-02-09 19:55:55 INFO install Selecting previously unselected package python-pip.
> 2016-02-09 19:55:55 INFO install Unpacking python-pip (from .../python-pip_1.0-1build1_all.deb) ...
> 2016-02-09 19:55:55 INFO install Processing triggers for man-db ...
> 2016-02-09 19:55:55 INFO install Setting up python-setuptools (0.6.24-1ubuntu1) ...
> 2016-02-09 19:55:55 INFO install Setting up python-pip (1.0-1build1) ...
> 2016-02-09 19:55:56 INFO install Package `python-enum34' is not installed and no info is available.
> 2016-02-09 19:55:56 INFO install Use dpkg --info (= dpkg-deb --info) to examine archive files,
> 2016-02-09 19:55:56 INFO install and dpkg --contents (= dpkg-deb --contents) to list their contents.
> 2016-02-09 19:55:56 INFO install Reading package lists...
> 2016-02-09 19:55:56 INFO install Building dependency tree...
> 2016-02-09 19:55:56 INFO install Reading state information...
> 2016-02-09 19:55:57 INFO install E: Unable to locate package python-enum34
> 2016-02-09 19:55:57 INFO install Traceback (most recent call last):
> 2016-02-09 19:55:57 INFO install File "/var/lib/juju/agents/unit-ceph-osd-0/charm/hooks/install.real", line 16, in <module>
> 2016-02-09 19:55:57 INFO install import ceph
> 2016-02-09 19:55:57 INFO install File "/var/lib/juju/agents/unit-ceph-osd-0/charm/hooks/ceph.py", line 9, in <module>
> 2016-02-09 19:55:57 INFO install from enum import Enum
> 2016-02-09 19:55:57 INFO install ImportError: No module named enum
>
>

Revision history for this message
uosci-testing-bot (uosci-testing-bot) wrote :

charm_lint_check #669 ceph-osd for xfactor973 mp284033
    LINT OK: passed

Build: http://10.245.162.36:8080/job/charm_lint_check/669/

Revision history for this message
uosci-testing-bot (uosci-testing-bot) wrote :

charm_unit_test #579 ceph-osd for xfactor973 mp284033
    UNIT OK: passed

Build: http://10.245.162.36:8080/job/charm_unit_test/579/

Revision history for this message
uosci-testing-bot (uosci-testing-bot) wrote :

charm_amulet_test #277 ceph-osd for xfactor973 mp284033
    AMULET OK: passed

Build: http://10.245.162.36:8080/job/charm_amulet_test/277/

Revision history for this message
James Page (james-page) wrote :

Some suggested refactoring to use charmhelpers.core.sysctl to manage the nic settings.

Also I'm not sure that max_sectors_kb is reboot persistent yet

review: Needs Fixing

Unmerged revisions

56. By Chris Holcombe

Removed enum dependency

55. By Chris Holcombe

Pull in the tox files from next

54. By Chris Holcombe

Persist the changes for reboots. Also catch IOError in all the posix operations

53. By Chris Holcombe

Address the lint issues and try to solve the python-enum34 package missing with the unit tests

52. By Chris Holcombe

Change over to using network link speed as the key to sysctl tuning instead of the driver name. This is a more generic approach because most 10Gb links need roughly the same set of sysctls

51. By Chris Holcombe

Lint spacing

50. By Chris Holcombe

Add a status notification to the network adapter tuning section

49. By Chris Holcombe

Added possible network adapter sysctl tuning

48. By Chris Holcombe

Add a config flag and a function to tune the read and write sectors for each osd block device

47. By Chris Holcombe

Sysctl and performance tweaks for ceph

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
The diff is not available at this time. You can reload the page or download it.

Subscribers

People subscribed via source and target branches