Launchpad itself

Merge lp://staging/~stub/launchpad/librarian-report into lp://staging/launchpad

  1. librarian-report
  2. Merge into devel
Proposed by Stuart Bishop
Status: Merged
Approved by: Jeroen T. Vermeulen
Approved revision: not available
Merged at revision: not available
Proposed branch: lp://staging/~stub/launchpad/librarian-report
Merge into: lp://staging/launchpad
Diff against target: 109 lines (+54/-16)
2 files modified
lib/canonical/librarian/tests/librarian-report.txt (+27/-14)
scripts/librarian-report.py (+27/-2)
To merge this branch: bzr merge lp://staging/~stub/launchpad/librarian-report
Reviewer Review Type Date Requested Status
Jeroen T. Vermeulen (community) Approve
Review via email: mp+16522@code.staging.launchpad.net

This proposal supersedes a proposal from 2009-12-22.

Commit message

Add date range selection to librarian-report.py

To post a comment you must log in.
Revision history for this message
Stuart Bishop (stub) wrote : Posted in a previous version of this proposal

Adds a date range selector to the existing librarian-report.py so we can generate deltas in addition to full reports.

Revision history for this message
Stuart Bishop (stub) : Posted in a previous version of this proposal
review: Needs Resubmitting
Revision history for this message
Stuart Bishop (stub) wrote :

Adds a date range selector to the existing librarian-report.py so we can generate deltas in addition to full reports.

Revision history for this message
Jeroen T. Vermeulen (jtv) wrote :

Thanks for making the effort to use run_script... makes for much easier code.

I still hate you for using slashes in dates, but if you really want it this way, all that really matters is that the db accepts it.

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'lib/canonical/librarian/tests/librarian-report.txt'
2--- lib/canonical/librarian/tests/librarian-report.txt 2009-12-15 05:29:47 +0000
3+++ lib/canonical/librarian/tests/librarian-report.txt 2009-12-23 04:36:17 +0000
4@@ -1,18 +1,31 @@
5 We have a report that we run as necessary to see what is using the Librarian
6 storage.
7
8- >>> import os.path
9- >>> from canonical.config import config
10- >>> path = os.path.join(config.root, 'scripts', 'librarian-report.py')
11-
12- >>> import subprocess
13- >>> proc = subprocess.Popen(
14- ... path, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
15- ... stdin=subprocess.PIPE)
16- >>> (out, ignored) = proc.communicate()
17- >>> print '\n' + out
18- <BLANKLINE>
19- ...
20- 0 bytes messageapproval in 0 files
21- ...
22+ >>> from canonical.launchpad.ftests.script import run_script
23+ >>> script = 'scripts/librarian-report.py'
24+
25+ >>> rv, out, err = run_script(script)
26+ >>> print rv
27+ 0
28+ >>> print err
29+ >>> print '\n' + out
30+ <BLANKLINE>
31+ ...
32+ 133 bytes hwsubmission in 2 files
33+ ...
34+
35+
36+We can filter on date to produce deltas.
37+
38+ >>> rv, out, err = run_script(
39+ ... script, ['--from=2005/01/01', '--until=2005/12/31'])
40+ >>> print rv
41+ 0
42+ >>> print err
43+ >>> print '\n' + out
44+ <BLANKLINE>
45+ ...
46+ 0 bytes hwsubmission in 0 files
47+ ...
48+
49
50
51=== modified file 'scripts/librarian-report.py'
52--- scripts/librarian-report.py 2009-12-15 05:29:30 +0000
53+++ scripts/librarian-report.py 2009-12-23 04:36:17 +0000
54@@ -13,17 +13,41 @@
55 from optparse import OptionParser
56 import sys
57
58-from canonical.database.sqlbase import connect, quoteIdentifier
59+from canonical.database.sqlbase import connect, quoteIdentifier, sqlvalues
60 from canonical.database.postgresql import listReferences
61 from canonical.launchpad.scripts import db_options
62
63
64 def main():
65 parser = OptionParser()
66+
67 db_options(parser)
68+ parser.add_option(
69+ "-f", "--from", dest="from_date", default=None,
70+ metavar="DATE", help="Only count new files since DATE (yyyy/mm/dd)")
71+ parser.add_option(
72+ "-u", "--until", dest="until_date", default=None,
73+ metavar="DATE", help="Only count new files until DATE (yyyy/mm/dd)")
74+
75 options, args = parser.parse_args()
76 if len(args) > 0:
77 parser.error("Too many command line arguments.")
78+
79+ # Handle date filters. We use LibraryFileContent.datecreated rather
80+ # than LibraryFileAlias.datecreated as this report is about actual
81+ # disk space usage. A new row in the database linking to a
82+ # previously existing file in the Librarian takes up no new space.
83+ if options.from_date is not None:
84+ from_date = 'AND LFC.datecreated >= %s' % sqlvalues(
85+ options.from_date)
86+ else:
87+ from_date = ''
88+ if options.until_date is not None:
89+ until_date = 'AND LFC.datecreated <= %s' % sqlvalues(
90+ options.until_date)
91+ else:
92+ until_date = ''
93+
94 con = connect(options.dbuser)
95 cur = con.cursor()
96
97@@ -57,11 +81,12 @@
98 AND (
99 LFA.expires IS NULL
100 OR LFA.expires > CURRENT_TIMESTAMP AT TIME ZONE 'UTC')
101+ %s %s
102 ORDER BY LFC.id
103 ) AS Whatever
104 """ % (
105 quoted_referring_table, quoted_referring_table,
106- quoted_referring_column))
107+ quoted_referring_column, from_date, until_date))
108 total_bytes, formatted_size, num_files = cur.fetchone()
109 totals.add((total_bytes, referring_table, formatted_size, num_files))
110