Merge ppa-dev-tools:add_rdepends_argument into ppa-dev-tools:main

Example output from running the command:

  $ ./scripts/ppa tests --show-rdepends --architectures amd64 https://launchpad.net/~bryce/+archive/ubuntu/apache2-merge-v2.4.54-3
* Triggers:
  - Source apache2/2.4.54-3ubuntu1~lunar1: Published
    + Trigger basic apache2@amd64♻️ Trigger all-proposed apache2@amd64💍
    + Trigger basic libembperl-perl@amd64♻️ Trigger all-proposed libembperl-perl@amd64💍
    + Trigger basic passenger@amd64♻️ Trigger all-proposed passenger@amd64💍
    + Trigger basic libapache2-mod-perl2@amd64♻️ Trigger all-proposed libapache2-mod-perl2@amd64💍
    + Trigger basic libsoup2.4@amd64♻️ Trigger all-proposed libsoup2.4@amd64💍
    + Trigger basic libsoup3@amd64♻️ Trigger all-proposed libsoup3@amd64💍
    + Trigger basic gnome-user-share@amd64♻️ Trigger all-proposed gnome-user-share@amd64💍
    + Trigger basic libapache2-mod-auth-gssapi@amd64♻️ Trigger all-proposed libapache2-mod-auth-gssapi@amd64💍
    + Trigger basic libapache2-mod-authn-sasl@amd64♻️ Trigger all-proposed libapache2-mod-authn-sasl@amd64💍
    + Trigger basic mod-gnutls@amd64♻️ Trigger all-proposed mod-gnutls@amd64💍
    + Trigger basic golang-github-containers-toolbox@amd64♻️ Trigger all-proposed golang-github-containers-toolbox@amd64💍
    + Trigger basic mathjax-siunitx@amd64♻️ Trigger all-proposed mathjax-siunitx@amd64💍
    + Trigger basic pycsw@amd64♻️ Trigger all-proposed pycsw@amd64💍
    + Trigger basic apache-upload-progress-module@amd64♻️ Trigger all-proposed apache-upload-progress-module@amd64💍
    + Trigger basic apache2-mod-xforward@amd64♻️ Trigger all-proposed apache2-mod-xforward@amd64💍
    + Trigger basic dehydrated@amd64♻️ Trigger all-proposed dehydrated@amd64💍
    + Trigger basic dogtag-pki@amd64♻️ Trigger all-proposed dogtag-pki@amd64💍
    + Trigger basic dump1090-mutability@amd64♻️ Trigger all-proposed dump1090-mutability@amd64💍
    + Trigger basic emboss-explorer@amd64♻️ Trigger all-proposed emboss-explorer@amd64💍
    + Trigger basic freeboard@amd64♻️ Trigger all-proposed freeboard@amd64💍
    + Trigger basic gbrowse@amd64♻️ Trigger all-proposed gbrowse@amd64💍
    + Trigger basic gnocchi@amd64♻️ Trigger all-proposed gnocchi@amd64💍
    + Trigger basic gridsite@amd64♻️ Trigger all-proposed gridsite@amd64💍
    + Trigger basic homer-api@amd64♻️ Trigger all-proposed homer-api@amd64💍
    + Trigger basic jsmath@amd64♻️ Trigger all-proposed jsmath@amd64💍
    + Trigger basic libapache-authenhook-perl@amd64♻️ Trigger all-proposed libapache-authenhook-perl@amd64💍
    + Trigger basic libapache-mod-auth-kerb@amd64♻️ Trigger all-proposed libapache-mod-auth-kerb@amd64💍
    + Trigger basic libapache-mod-encoding@amd64♻️ Trigger all-proposed libapache-mod-encoding@amd64💍
    + Trigger basic libapache-mod-evasive@amd64♻️ Trigger all-proposed libapache-mod-evasive@amd64💍
    + Trigger basic libapache-mod-jk@amd64♻️ Trigger all-proposed libapache-mod-jk@amd64💍
    + Trigger basic libapache-mod-log-sql@amd64♻️ Trigger all-proposed libapache-mod-log-sql@amd64💍
    + Trigger basic libapache-mod-musicindex@amd64♻️ Trigger all-proposed libapache-mod-musicindex@amd64💍
    + Trigge...

Again, after tests have finished, and with --show-urls added:

$ ./scripts/ppa tests --show-urls --show-rdepends --architectures amd64 https://launchpad.net/~bryce/+archive/ubuntu/apache2-merge-v2.4.54-3
* Triggers:
  - Source apache2/2.4.54-3ubuntu1~lunar1: Published
    + apache2@amd64: https://autopkgtest.ubuntu.com/request.cgi?release=lunar&package=apache2&arch=amd64&trigger=apache2%2F2.4.54-3ubuntu1~lunar1&ppa=bryce%2Fapache2-merge-v2.4.54-3♻️
    + libembperl-perl@amd64: https://autopkgtest.ubuntu.com/request.cgi?release=lunar&package=libembperl-perl&arch=amd64&trigger=apache2%2F2.4.54-3ubuntu1~lunar1&ppa=bryce%2Fapache2-merge-v2.4.54-3♻️
    + passenger@amd64: https://autopkgtest.ubuntu.com/request.cgi?release=lunar&package=passenger&arch=amd64&trigger=apache2%2F2.4.54-3ubuntu1~lunar1&ppa=bryce%2Fapache2-merge-v2.4.54-3♻️
    + libapache2-mod-perl2@amd64: https://autopkgtest.ubuntu.com/request.cgi?release=lunar&package=libapache2-mod-perl2&arch=amd64&trigger=apache2%2F2.4.54-3ubuntu1~lunar1&ppa=bryce%2Fapache2-merge-v2.4.54-3♻️
    + libsoup2.4@amd64: https://autopkgtest.ubuntu.com/request.cgi?release=lunar&package=libsoup2.4&arch=amd64&trigger=apache2%2F2.4.54-3ubuntu1~lunar1&ppa=bryce%2Fapache2-merge-v2.4.54-3♻️
    + libsoup3@amd64: https://autopkgtest.ubuntu.com/request.cgi?release=lunar&package=libsoup3&arch=amd64&trigger=apache2%2F2.4.54-3ubuntu1~lunar1&ppa=bryce%2Fapache2-merge-v2.4.54-3♻️
    + gnome-user-share@amd64: https://autopkgtest.ubuntu.com/request.cgi?release=lunar&package=gnome-user-share&arch=amd64&trigger=apache2%2F2.4.54-3ubuntu1~lunar1&ppa=bryce%2Fapache2-merge-v2.4.54-3♻️
    + libapache2-mod-auth-gssapi@amd64: https://autopkgtest.ubuntu.com/request.cgi?release=lunar&package=libapache2-mod-auth-gssapi&arch=amd64&trigger=apache2%2F2.4.54-3ubuntu1~lunar1&ppa=bryce%2Fapache2-merge-v2.4.54-3♻️
    + libapache2-mod-authn-sasl@amd64: https://autopkgtest.ubuntu.com/request.cgi?release=lunar&package=libapache2-mod-authn-sasl&arch=amd64&trigger=apache2%2F2.4.54-3ubuntu1~lunar1&ppa=bryce%2Fapache2-merge-v2.4.54-3♻️
    + mod-gnutls@amd64: https://autopkgtest.ubuntu.com/request.cgi?release=lunar&package=mod-gnutls&arch=amd64&trigger=apache2%2F2.4.54-3ubuntu1~lunar1&ppa=bryce%2Fapache2-merge-v2.4.54-3♻️
    + golang-github-containers-toolbox@amd64: https://autopkgtest.ubuntu.com/request.cgi?release=lunar&package=golang-github-containers-toolbox&arch=amd64&trigger=apache2%2F2.4.54-3ubuntu1~lunar1&ppa=bryce%2Fapache2-merge-v2.4.54-3♻️
    + mathjax-siunitx@amd64: https://autopkgtest.ubuntu.com/request.cgi?release=lunar&package=mathjax-siunitx&arch=amd64&trigger=apache2%2F2.4.54-3ubuntu1~lunar1&ppa=bryce%2Fapache2-merge-v2.4.54-3♻️
    + pycsw@amd64: https://autopkgtest.ubuntu.com/request.cgi?release=lunar&package=pycsw&arch=amd64&trigger=apache2%2F2.4.54-3ubuntu1~lunar1&ppa=bryce%2Fapache2-merge-v2.4.54-3♻️
    + apache-upload-progress-module@amd64: https://autopkgtest.ubuntu.com/request.cgi?release=lunar&package=apache-upload-progress-module&arch=amd64&trigger=apache2%2F2.4.54-3ubuntu1~lunar1&ppa=bryce%2Fapache2-merge-v2.4.54-3♻️
    + apache2-mod-xforward@amd64: https://autopkgtest.ubuntu.com/request.cgi?releas...

I looked very briefly and have a few comments, but I'll take a better look tomorrow.

First of all, thank you very much for implementing this feature. It is indeed one of the most useful parts of bileto, and I'd love to be able to use it from inside my Emacs instead of having to open a browser ;-).

Without entering the merit of the implementation itself, I'd like to discuss the approach you've taken to grab a copy of the archive's metadata needed for the rdep processing. I don't mind having to run rsync manually if needed, but I'm wondering if there are better, even simpler options out there.

For example, I know that Ubuntu Wire maintains http://qa.ubuntuwire.org/rdepends/ , which is itself used by the "reverse-depends" tool. The service is good, but unfortunately only outputs the *binary* packages, so there's some massaging needed in order to obtain the source packages (which is what we ultimately need for the autopkgtest URLs).

Another option I've used in the past (and implemented here: https://git.launchpad.net/~ubuntu-server/+git/ubuntu-helpers/tree/sergiodj/list-revdep-autopkgtest-links.sh) is to rely on chdist to pull the archive data for you. In my case, I also relied on chdist's "bin2src" script to do the conversion for me, but it seems like this is something you already have covered in your script. This chdist invocation could even be done inside a TemporaryDirectory so that you guarantee that no metadata will be left behind after the program's execution (although it might be interesting if the user could choose to keep the data for future invocations; anyway, implementation details).

The main point here is that I believe this metadata fetching could be further automated, which would be even more awesome. WDYT?

Hi Sergio, thanks for reviewing.

Indeed, this is one of the next items on the todo list after this MP is landed. This branch should be considered the proof-of-concept for the basic functionality, to be followed with some convenience improvements and optimizations.

I don't know if you remember but the last few weeks I've researched into some caching and mirroring options and tools. Particularly I was looking at caching at urllib/http level python modules, however I did also look at apt-mirror and other Debian tools. I found one option for the former I want to explore more, but the latter options looked too overkill and dependency-heavy. I didn't look at chdist though, so will try to peek just in case. But the basic idea I have as a solution is to integrate "lazy mirroring" where instead of directly loading Sources.xz and Packages.xz from disk, if what's on disk is older than N hours, it automatically re-downloads and re-caches the file (and ONLY the requested file) on the fly. From my testing so far, on a decent network connection that should be quite performant.

I was hoping to slip that feature work in before Prague (and have at least started sketching it in on another branch) but I'm realizing there's some caveats I need to account for, and time is getting short, so decided to just go with the (reliable) rsync for now and make this a next-release enhancement. There's a couple other convenience improvements I plan to make too, and maybe other folks will also have ideas to add in.

I hadn't looked at chdist, but I did look into apt-mirror and some other Debian and python tools for mirroring apt, and for general http client-side cache management. There's one cache manager module I want to look into a bit more, but the tools I played with were a bit too heavyweight for what I need, and added dependencies.

Anyway, yes I agree more automation of the metadata fetching would indeed be awesome, and plan to do so but maybe needs to wait for next release?

Thanks for the detailed explanation, Bryce.

Yes, absolutely no problem to leave this further automation for the next release. I'm glad that we're on the same page on this topic!

I'm leaving a few cosmetic comments below, but otherwise the code looks good to me, so I'm approving the MP. But I have a few other general comments.

- I set up a container in order to fully test the package (including installing it), and I stumbled upon a few problems with missing requirements from the requirements-dev.txt file.

- When I tried running the "tests --show-rdepends" command against a public PPA, it asked me to log into LP. This shouldn't really be necessary, unless dealing with private PPAs and such. It's not something really important to address right now (I understand that you have other things on your plate), but it's something to keep in mind IMHO.

Thanks again for working on this.

> Thanks for the detailed explanation, Bryce.
>
> Yes, absolutely no problem to leave this further automation for the next
> release. I'm glad that we're on the same page on this topic!
>
> I'm leaving a few cosmetic comments below, but otherwise the code looks good
> to me, so I'm approving the MP. But I have a few other general comments.
>
> - I set up a container in order to fully test the package (including
> installing it), and I stumbled upon a few problems with missing requirements
> from the requirements-dev.txt file.

If you can shoot me either the steps you followed, or the changed requirements, it'd be appreciated as I prepare the release. I posted an MP the other day with some packaging work, that includes adding a missing dependency or two, but could be more. I also found that running tox on the codebase scared up some requirements issues that I will try to look at if not for this release than for a future one. Getting tox to pass without issue would be very nice, and probably would take care of the issues you hit here.

> - When I tried running the "tests --show-rdepends" command against a public
> PPA, it asked me to log into LP. This shouldn't really be necessary, unless
> dealing with private PPAs and such. It's not something really important to
> address right now (I understand that you have other things on your plate), but
> it's something to keep in mind IMHO.

Noted, I'll put this on the list to work on later.

I know exactly what the problem is - when constructing the Launchpad service you specify what level of permissions you need, and I just across the board ask for read/write privs regardless of what the individual command actually needs. Fixing it such that commands only request the priv level they need will take a bit of plumbing, however as an outgrowth of some of the experimentation I've done with qastaging I'm realizing I need a more sophisticated system for selecting Launchpad service API versions, permissions, and so on.

> Thanks again for working on this.

Enumerating objects: 86, done.
Counting objects: 100% (86/86), done.
Delta compression using up to 12 threads
Compressing objects: 100% (67/67), done.
Writing objects: 100% (71/71), 13.37 KiB | 4.46 MiB/s, done.
Total 71 (delta 54), reused 0 (delta 0), pack-reused 0
To git+ssh://git.launchpad.net/ppa-dev-tools
   d71ef8d..daf0adc main -> main