Merge lp://staging/~parthm/bzr/300062-2.2-bad-pattern-error-part-2 into lp://staging/bzr/2.2

79 - self._add_patterns(ext_patterns,_sub_extension,
80 - prefix=r'(?:.*/)?(?!.*/)(?:.*\.)')
81 - self._add_patterns(base_patterns,_sub_basename,
82 - prefix=r'(?:.*/)?(?!.*/)')
83 - self._add_patterns(path_patterns,_sub_fullpath)
84 + pattern_lists[Globster.identify(pat)].append(pat)
85 + for pattern_type, patterns in pattern_lists.iteritems():
86 + self._add_patterns(patterns,
87 + Globster.translators[pattern_type],
88 + Globster.prefixes[pattern_type])

^- I think factoring the patterns into a dict indirected code is interesting. I'm a bit concerned about:
for pattern_type, patterns in pattern_lists.iteritems():

I'm pretty sure we explicitly intended the order of 'check extensions' then 'check basename' then 'check fullpath'. So that we can do checks on just the little bit of the content before we have to check against the whole content. I don't have any explicit proof of this, but turning the current deterministic ordering with a random one doesn't seem ideal.

otherwise the change seems fine to me. I may be over-concerned, but I do believe that 'is_ignored()' is a fair amount of overhead in the 'initial add of 30k files', and I'd like us to at least be aware of it. I do wonder about changing the internals so that instead of using '?:.*/)?(?!.*/)) to filter out everything but the basename for some patterns, if we wouldn't be better off knowing that this is a basename-only pattern and just supply that to the match.

I'm certainly willing to discuss it, but the easiest way to move forward is to just make the order of _add_patterns deterministic by iterating over a fixed ordering.

> ^- I think factoring the patterns into a dict indirected code is interesting.
> I'm a bit concerned about:
> for pattern_type, patterns in pattern_lists.iteritems():
>
> I'm pretty sure we explicitly intended the order of 'check extensions' then
> 'check basename' then 'check fullpath'. So that we can do checks on just the
> little bit of the content before we have to check against the whole content. I
> don't have any explicit proof of this, but turning the current deterministic
> ordering with a random one doesn't seem ideal.
>

Makes sense. I have updated the patch to make _add_patterns order deterministic.

>
> otherwise the change seems fine to me. I may be over-concerned, but I do
> believe that 'is_ignored()' is a fair amount of overhead in the 'initial add
> of 30k files', and I'd like us to at least be aware of it. I do wonder about
> changing the internals so that instead of using '?:.*/)?(?!.*/)) to filter out
> everything but the basename for some patterns, if we wouldn't be better off
> knowing that this is a basename-only pattern and just supply that to the
> match.
>

Filed bug #607258 to track this.
Thanks for the review.

>
> I'm certainly willing to discuss it, but the easiest way to move forward is to
> just make the order of _add_patterns deterministic by iterating over a fixed
> ordering.

43 + TYPE_FULLPATH = 1
44 + TYPE_BASENAME = 2
45 + TYPE_EXTENSION = 3

This looks a bit C-ish, why don't you just use regular keys ('fullpath', 'basename' and 'extension') ?

From a design point of view it looks like you're defining several set of objects associated with the same key which sounds like you want a dict or even a registry...

Also, while this is certainly out of scope for this proposal, keep in mind that some plugins want to define their own '.xxx-ignore' files and our current API is already a pain to use, it would be nice if we can help them by providing the right set of function/classes to process a set of paths against a given set of patterns coming from a file (or several in bzr case...).

> 43 + TYPE_FULLPATH = 1
> 44 + TYPE_BASENAME = 2
> 45 + TYPE_EXTENSION = 3
>
> This looks a bit C-ish, why don't you just use regular keys ('fullpath',
> 'basename' and 'extension') ?
>
> From a design point of view it looks like you're defining several set of
> objects associated with the same key which sounds like you want a dict or even
> a registry...
>

I have updated the patch to use a dict of dict with strings. Its much cleaner now.

> Also, while this is certainly out of scope for this proposal, keep in mind
> that some plugins want to define their own '.xxx-ignore' files and our current
> API is already a pain to use, it would be nice if we can help them by
> providing the right set of function/classes to process a set of paths against
> a given set of patterns coming from a file (or several in bzr case...).

Makes sense. I think it may be possible to build an API on top of the
current implementation to make it easy. I will look into bzr-upload as
suggested by you on IRC.

Thanks for the review.

Yup, far cleaner.

I think you've also addressed jam's concerns so good to land for me.

sent to pqm by email