Merge lp://staging/~rvb/maas/nonce-bug-1190986-tasks into lp://staging/~maas-committers/maas/trunk

Looks good.

[1]

+    'cleanup-old-nonces': {
+        'task': 'maasserver.tasks.cleanup_old_nonces',
+        'schedule': timedelta(minutes=5),
+        'options': {'queue': WORKER_QUEUE_REGION},
+    },

Does Celery ensure that jobs don't pile up? If a run of this job takes
more than 5 minutes (I'm thinking of the first run, maybe), will
Celery enqueue the next job, or will it be automatically cancelled?
Or... does schedule mean the time *between* runs, rather than a rigid
from-the-clock schedule?

AFAIK they will pile up (we have seen this happen with all other jobs when something went wrong with Rabbit. I think 5 minutes is too often to run this anyway - how about once a day?

> Or... does schedule mean the time *between* runs, rather than a rigid
> from-the-clock schedule?

Like Julian said, celerybeat works like cron in this respect, it only fires off tasks at regular interval, nothing more intelligent than that.

> AFAIK they will pile up (we have seen this happen with all other jobs when something went wrong with Rabbit.
> I think 5 minutes is too often to run this anyway - how about once a day?

fwiw, the 5 minutes interval comes from the fact that nonces are only valid for 5 minutes (oauth enforces that) and the cleanup_old_nonces utility uses that… so running this utility every 5 minutes is the best we can do (now, your might want to argue doing something different but I'm explaining why I picked up 5 minutes in the first place). Considering that a nonce is created *every time* an authenticated request is made to that API, in the context of hyperscale, I think it's worth running this as often as possible.

Also:
- running cleanup_old_nonces when there is no nonces to clean up is instantaneous (well, obviously).
- running cleanup_old_nonces when there is 6M nonces to clean up takes ~10s on a modest canonistack instance.

Julian's point about jobs piling up is concerning. If we can ensure that jobs don't pile up (is that a per-message, per-queue or per-exchange setting?) then I think that every 5 minutes is fine. If they're going to accumulate, then once a day seems okay. We've not had *anything* for over a year, so once a day seems more than adequate.

All right, https://code.launchpad.net/~rvb/maas/change-freq/+merge/190894

On 14/10/13 18:51, Gavin Panella wrote:
> We've not had *anything* for over a year, so once a day seems more than adequate.

That was precisely my reasoning for once a day.