ppa-dev-tools

Merge ppa-dev-tools:add_repository_module into ppa-dev-tools:main

  1. Git
  2. lp:ppa-dev-tools
  3. add_repository_module
  4. Merge into main
Proposed by Bryce Harrington
Status: Merged
Merge reported by: Bryce Harrington
Merged at revision: af638afc38955ecbd6d408add476d52dbd26a445
Proposed branch: ppa-dev-tools:add_repository_module
Merge into: ppa-dev-tools:main
Diff against target: 186 lines (+174/-0)
2 files modified
ppa/repository.py (+119/-0)
tests/test_repository.py (+55/-0)
Reviewer Review Type Date Requested Status
Athos Ribeiro (community) Approve
Canonical Server Pending
Canonical Server Reporter Pending
Review via email: mp+438168@code.staging.launchpad.net

Description of the change

This introduces the first class for the soon-to-come rdepends feature. The Repository class is going to be responsible for creating and maintaining a 'lazy local mirror' of the Ubuntu apt repository, downloading information only as needed and caching it for performance.

For now, though, this only implements the bare bones, able to read an already-existing mirror and using it to generate the corresponding sub-objects - Suites. A suite represents what we sometimes term "release pockets", and are present in the directory structure as directory names like 'kinetic-backports' and 'feisty-updates'. A stub of the Suite class is included in this branch but only to give the Repository something to manage; a future branch will move Suite to its own module and flesh it out further. For purposes of this review you can mostly ignore Suite; I haven't made docs or tests for it, intentionally.

The test cases exercise most of the functionality, using temporary directories instead of a proper mirror. However, the module itself includes a built-in smoke test that can be run on a real mirror. (For development purposes, I've generated one using `apt-mirror`; you're welcome to do the same but beware it takes hours to create even the most minimal mirror.)

The class name 'Repository' I wonder if should be renamed, to avoid confusion with a git repository, which is probably what people would assume it referred to. Review advice/opinion welcome here.

---

The tests can be invoked via:

    $ python-3 ./tests/test_repository.py

---
The module's smoketest can be invoked as well, via:

    $ python-3 -m ppa.repository

However, the smoke test expects to run against an apt mirror. Below are the directions on the way I set this up for myself:

    $ sudo apt-get install apt-mirror

Modify /etc/apt/mirror.list to configure all the Ubuntu releases you
wish to mirror. For this validation test we'll only enable the devel
version's release and proposed pockets for the main component:

    ## /etc/apt/mirror.list:
    set nthreads 20
    set _tilde 0

    deb http://archive.ubuntu.com/ubuntu lunar main
    deb http://archive.ubuntu.com/ubuntu lunar-proposed main

    deb-src http://archive.ubuntu.com/ubuntu lunar main
    deb-src http://archive.ubuntu.com/ubuntu lunar-proposed main

    clean http://archive.ubuntu.com/ubuntu

Make sure you have ample space to store everything, at least 25 GiB;
several times that if you want to pull anything more than what's
configured above.

Next, run the sync operation manually:

    $ sudo /usr/bin/apt-mirror

Unfortunately, this takes a long time to run and consumes a lot of bandwidth. The bare minimum needed to test this specific branch, though, is just the directory structure, which you could create by hand:

/var/spool/apt-mirror/skel/archive.ubuntu.com/ubuntu/dists
/var/spool/apt-mirror/skel/archive.ubuntu.com/ubuntu/dists/lunar-proposed
/var/spool/apt-mirror/skel/archive.ubuntu.com/ubuntu/dists/lunar-proposed/main
/var/spool/apt-mirror/skel/archive.ubuntu.com/ubuntu/dists/lunar-proposed/main/source
/var/spool/apt-mirror/skel/archive.ubuntu.com/ubuntu/dists/lunar-proposed/main/dep11
/var/spool/apt-mirror/skel/archive.ubuntu.com/ubuntu/dists/lunar-proposed/main/i18n
/var/spool/apt-mirror/skel/archive.ubuntu.com/ubuntu/dists/lunar-proposed/main/binary-amd64
/var/spool/apt-mirror/skel/archive.ubuntu.com/ubuntu/dists/lunar
/var/spool/apt-mirror/skel/archive.ubuntu.com/ubuntu/dists/lunar/main
/var/spool/apt-mirror/skel/archive.ubuntu.com/ubuntu/dists/lunar/main/source
/var/spool/apt-mirror/skel/archive.ubuntu.com/ubuntu/dists/lunar/main/dep11
/var/spool/apt-mirror/skel/archive.ubuntu.com/ubuntu/dists/lunar/main/i18n
/var/spool/apt-mirror/skel/archive.ubuntu.com/ubuntu/dists/lunar/main/binary-amd64

To post a comment you must log in.
Revision history for this message
Athos Ribeiro (athos-ribeiro) wrote :

Thanks, Bryce!

This looks great. I really like how you covered the new parser with all those test cases :)

I added a few inline comments and questions below.

review: Needs Information
Revision history for this message
Athos Ribeiro (athos-ribeiro) wrote :

On a side note, I am getting the following test failure:

_______________________________________________________________________________________________ test_command_create _______________________________________________________________________________________________

fake_config = {'ppa_name': 'testing', 'team_name': 'me', 'wait_seconds': 0.1}, monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f7aae4c5b90>

    def test_command_create(fake_config, monkeypatch):
        lp = LpServiceMock()
        monkeypatch.setattr("sys.stdin", io.StringIO('test description'))
> assert script.command_create(lp, fake_config) == 0

tests/test_scripts_ppa.py:370:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
scripts/ppa:483: in command_create
    the_ppa.set_architectures(architectures)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = Ppa(ppa_name='testing', team_name='me'), architectures = ['amd64', 'arm64', 'armhf', 'i386', 'powerpc', 'ppc64el', ...]

    def set_architectures(self, architectures):
        """Configures the architectures used to build packages in the PPA.

        Note that some architectures may only be available upon request
        from Launchpad administrators. ppa.constants.ARCHES_PPA is a
        list of standard architectures that don't require permissions.

        :rtype: bool
        :returns: True if architectures could be set, False on error.
        """
        uri_base = "https://api.launchpad.net/devel/+processors/{}"
        procs = [uri_base.format(arch) for arch in architectures]
        try:
> self.archive.setProcessors(processors=procs)
 E AttributeError: 'ArchiveMock' object has no attribute 'setProcessors'

ppa/ppa.py:197: AttributeError
============================================================================================= short test summary info =============================================================================================
FAILED tests/test_scripts_ppa.py::test_command_create - AttributeError: 'ArchiveMock' object has no attribute 'setProcessors'

Revision history for this message
Athos Ribeiro (athos-ribeiro) wrote :

Which is being fixed in https://code.launchpad.net/~bryce/ppa-dev-tools/+git/ppa-dev-tools-1/+merge/438274

Sorry for the noise :)

Revision history for this message
Bryce Harrington (bryce) wrote :

Thanks for reviewing, and I appreciate the good suggestions! I'll finish updating my branch and repost in a bit.

Revision history for this message
Bryce Harrington (bryce) wrote (last edit ):

I've squashed in the suggestions and pushed an updated branch.
The unpack_to_dict() changes and suggestions I've merged into another MP that Lena reviewed, and landed those to main. So what remains here for re-review is just the repository class itself.

Thanks again!

Revision history for this message
Athos Ribeiro (athos-ribeiro) wrote :

LGTM!

Thanks, Bryce :)

review: Approve
Revision history for this message
Bryce Harrington (bryce) wrote :

Thanks, merged:

To git+ssh://git.launchpad.net/ppa-dev-tools
   7e59879..bad3b8b main -> main

There was an error fetching revisions from git servers. Please try again in a few minutes. If the problem persists, contact Launchpad support.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
The diff is not available at this time. You can reload the page or download it.

Subscribers

People subscribed via source and target branches

to all changes: