Merge lp://staging/~vanvugt/mir/fix-buffers_ready_for_compositor into lp://staging/mir

FAILED: Continuous integration, rev:2133
http://jenkins.qa.ubuntu.com/job/mir-ci/2773/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/mir-android-vivid-i386-build/1007/console
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-clang-vivid-amd64-build/1006
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-vivid-touch/968
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-vivid-amd64-ci/770
        deb: http://jenkins.qa.ubuntu.com/job/mir-vivid-amd64-ci/770/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-builder-vivid-armhf/968
        deb: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-builder-vivid-armhf/968/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-runner-mako/4055
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/touch-flash-device/17351

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/mir-ci/2773/rebuild

OK. LGTM.

FAILED: Continuous integration, rev:2134
http://jenkins.qa.ubuntu.com/job/mir-ci/2789/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-android-vivid-i386-build/1034
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-clang-vivid-amd64-build/1033
    FAILURE: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-vivid-touch/995/console
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-vivid-amd64-ci/786
        deb: http://jenkins.qa.ubuntu.com/job/mir-vivid-amd64-ci/786/artifact/work/output/*zip*/output.zip
    FAILURE: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-builder-vivid-armhf/995/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/mir-ci/2789/rebuild

Ugg. Same tests failing for all proposals :(

FAILED: Continuous integration, rev:2135
http://jenkins.qa.ubuntu.com/job/mir-ci/2791/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-android-vivid-i386-build/1038
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-clang-vivid-amd64-build/1037
    FAILURE: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-vivid-touch/999/console
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-vivid-amd64-ci/788
        deb: http://jenkins.qa.ubuntu.com/job/mir-vivid-amd64-ci/788/artifact/work/output/*zip*/output.zip
    FAILURE: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-builder-vivid-armhf/999/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/mir-ci/2791/rebuild

PASSED: Continuous integration, rev:2136
http://jenkins.qa.ubuntu.com/job/mir-ci/2816/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-android-vivid-i386-build/1072
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-clang-vivid-amd64-build/1071
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-vivid-touch/1033
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-vivid-amd64-ci/813
        deb: http://jenkins.qa.ubuntu.com/job/mir-vivid-amd64-ci/813/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-builder-vivid-armhf/1033
        deb: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-builder-vivid-armhf/1033/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-runner-mako/4092
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/touch-flash-device/17491

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/mir-ci/2816/rebuild

I need to think through what happens when the compositor drops renderables. But this seem promising.

#1
I don't like mg::Renderables::buffers_ready_for_compositor(), and am happy to see it going away, but it is a bit entrenched in qtmir, so we should have a way to transition the downstream.

http://bazaar.launchpad.net/~mir-team/qtmir/trunk/view/head:/src/modules/Unity/Application/mirsurfaceitem.cpp#L446

#2
This does circumvent the observer architecture by requiring feedback from the BufferQueues. I see how the arch is wrong in the case described, but before we were telling the compositors to composite so many times, but now we're telling them to composite so many times, and then asking if they really meant it. The BufferQueue doesn't have all the information about how many compositions are needed to drain because of things like position-only-updates, and the BasicSurface cant give the information that is needed to drain the queue completely. Maybe we should improve our concept of 'damage' of the stack, and then have every composition reduce the scheduled damage by one.

to elaborate a bit further #1 is the bigger concern, #2 is to point out in the review that mc::Scene now is somewhat strange in that we have two routes to observe the damage

//frames-pending method
virtual int frames_pending(CompositorID id) const = 0;

//observer method
virtual void add_observer(std::shared_ptr<scene::Observer> const& observer) = 0;
virtual void remove_observer(std::weak_ptr<scene::Observer> const& observer) = 0;

and that the observer method (public api) is still broken in that it can't drain the queue properly.

#1
Yeah I knew about QtMir already. It will need fixing. Hopefully without reintroducing the old Renderable method but we'll see. It's a simple matter of code but needs the involvement of Unity8 guys more to test the changes.

#2
The observer architecture obviously doesn't solve all problems. Compositors can and will occasionally skip over some renderable and not consume its buffer. Compositors are free to make their own decisions, so we should support any outcome from that.
  I don't think this is strange or inconsistent though. A compositor's default state is to be asleep. We _need_ the observer stuff to wake it up. And we _need_ a separate counter to cover the case of rendering N frames after N were scheduled might not be enough (due to occlusions whatever). Callback-based observers can not solve this problem, but semaphore-based observers could, so read on...
  I did consider a more unified approach that might work. That is a singular counter (like a semaphore) stored in the scene where each surface adds to the counter when it has something new not consumed. And critically, only the surface/producer can then remove its count from the scene total when it's flushed. But that's an architectural rework, bigger than this proposal and possibly could end up more complicated with some new coupling too. So first a fix, and later maybe redesign.

#2: Fair enough, I think the unified approach is something to work towards, but in the meantime, we may as well use this approach to solve the problem. Keeping track of whether the stack is damaged seems a better concept than what's in trunk or the wakeup and keep-running-until-drained approach.

#1: I guess I'd like to pause this branch to avoid breaking downstream in a way that we don't have a fix for yet. Its mostly just a timing issue, but I'd guess that the Belgium sprinters might want to integrate the tip of mir with the tip of the other components.

So anyways, approve from me, but would appreciate pausing the branch until we have a downstream update available.

#1: I'm sitting next to Gerry and talking to him about it :)

As long as we can work towards a more unified approach, and there's a branch forthcoming downstream, lgtm.

Fortunately we've branched 0.11 already so landing this for 0.12 puts us in no hurry to sync downstreams.

FAILED: Autolanding.
More details in the following jenkins job:
http://jenkins.qa.ubuntu.com/job/mir-autolanding/1096/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/mir-android-vivid-i386-build/1174/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/mir-clang-vivid-amd64-build/1173/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-vivid-touch/1132/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/mir-vivid-amd64-autolanding/276/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-builder-vivid-armhf/1132/console

UNTaed until qtmir branch is available so we can release tomorrow.

QtMir branch available:
https://code.launchpad.net/~vanvugt/qtmir/fix-buffers_ready_for_compositor/+merge/248419

PASSED: Continuous integration, rev:2140
http://jenkins.qa.ubuntu.com/job/mir-ci/2892/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-android-vivid-i386-build/1181
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-clang-vivid-amd64-build/1180
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-vivid-touch/1139
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-vivid-amd64-ci/889
        deb: http://jenkins.qa.ubuntu.com/job/mir-vivid-amd64-ci/889/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-builder-vivid-armhf/1139
        deb: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-builder-vivid-armhf/1139/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-runner-mako/4189
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/touch-flash-device/17725

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/mir-ci/2892/rebuild