Merge lp://staging/~jameinel/bzr/2.0b1-402645-fragmentation into lp://staging/~bzr/bzr/trunk-old

This is a reasonable fix for the fragmentation issue identified in:
  https://bugs.edge.launchpad.net/bzr/+bug/402645

Basically, we were fetching some data in "groupcompress" order, but we didn't actually combine anything to be optimal in the target repository. (So you split the existing groups up to get a new ordering, but then don't combine them to get optimal groups.)

The ideal is to fix the combining issue, but for now, this is far easier and gives us something better in the short term.

I believe there may be some history on this patch with Robert, but it sounds like a reasonable next step to me, so I'll merge it to 2.0.

On Wed, 2009-08-26 at 07:27 +0000, Martin Pool wrote:
> Review: Approve
> I believe there may be some history on this patch with Robert, but it sounds like a
> reasonable next step to me, so I'll merge it to 2.0.

My concern is fallout - it seems risky to change this (we've had bugs
related to e.g. sending unordered to knits before) when we're working on
a different fix anyway. We'll just have to back this out when that lands
anyway.

-Rob

--

(Oh, and I don't have a specific problem, I can spend time looking for
any tomorrow - I just don't know that anyone has).

-Rob

> On Wed, 2009-08-26 at 07:27 +0000, Martin Pool wrote:
> > Review: Approve
> > I believe there may be some history on this patch with Robert, but it sounds
> like a
> > reasonable next step to me, so I'll merge it to 2.0.
>
> My concern is fallout - it seems risky to change this (we've had bugs
> related to e.g. sending unordered to knits before) when we're working on
> a different fix anyway. We'll just have to back this out when that lands
> anyway.

Until John replied, from the discussion we had yesterday, I think the plan is to switch back to gc order once we know how to recombine properly on th receiving side.

But until then using unordered is the only way to avoid fragmentation.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Robert Collins wrote:
> (Oh, and I don't have a specific problem, I can spend time looking for
> any tomorrow - I just don't know that anyone has).
>
> -Rob

So...

1) The code in question is the GroupCHKStreamSource so it is only used
for fetching 2a => 2a (thus can't be a problem for knits, etc.)

2) I certainly have looked at this area of code a lot. And I remembered
that we put in the "groupcompress" fetch order because we believed that
it would eventually "settle out". As everything would maintain order. We
didn't think about group fragmentation, or the fact that 'groupcompress'
order is really hard to get 'stable'. (I can certainly make it stable
for a given graph, getting it stable as the graph evolves is the really
hard part.)

3) We've done 'unordered' pack=>pack fetches for a long time, and have a
pretty good feel that they are safe to do.

I appreciate some level of paranoia to double check changes. I'm pretty
confident on this one, though.

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkqVQEoACgkQJdeBCYSNAANZxgCfaUl8GWv9p0YeeKLSX2axJdIU
/eAAoKTe3tnd+TYt2MYSnwcmMwQxIEG+
=YJpV
-----END PGP SIGNATURE-----