Just like the previous article about Elixir, this is another one that has been sitting in the drafts folder for quite a while. I figured I could publish it to keep things rolling until I have the next one for the Software engineering is broken series.
If you use a recent version of Debian, Ubuntu, or one of their derivatives, and you added a PPA or followed instructions found online for adding an external repository to your system, you may have seen a warning like the following:
W: http://ppa.launchpad.net/git-core/ppa/ubuntu: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
TL;DR
If you know what this is, are aware of the implications, and just want the quick fix, here you go.
For each APT repository and key causing the warning message:
-
Export the key to the right location:
sudo apt-key export <key ID> | gpg --dearmor -o /etc/apt/keyrings/<key_name>.gpg -
Set the key in the repository config file by adding
[signed-by=/etc/apt/keyrings/<key_name>.gpg]betweendeband the repository URL. If the file uses the newer DEB822 format, add a line withSigned-By: /etc/apt/keyrings/<key_name>.gpginstead. -
Remove the key from the deprecated location:
sudo apt-key del <key ID> -
Check
/etc/apt/trusted.gpg.d/for any other keys you may want to move to/etc/apt/keyrings/.
If you need more details or want to see how to clean up old keys, check below.
What is the problem?
The “traditional” location for APT keys is /etc/apt/trusted.gpg or /etc/apt/trusted.gpg.d/. The problem is that any key in these locations will be used by APT to authenticate packages from any repository that we have configured. In other words, there will be cross-signing of packages between different repositores, thus weakening the trust in the signing mechanism.
The warning message mentioned at the top of the article comes from apt-key, a tool used manage the keys APT looks at to authenticate packages from repositories. This tool is used automatically when adding PPA’s via the ´add-apt-repository´ command, and a lot of websites also mention it directly on instructions on how to manually add repositories to our system.
The problem is that apt-key installs keys into the aforementioned locations, which is not what we want.
This tool has been deprecated, and according to the manpage, it is marked for removal after Debian 12 and Ubuntu 24.04. But until then, we should probably fix the systems that got the keys in the wrong place.
This is the way
What we should do instead is have separate key files for each repository, store those files in /etc/apt/keyrings/ (technically, they can be stored wherever we want but this is the recommended location), and then specify which key APT should accept for each repository. This way, an attacker would have to not only be able to inject a package into the repository, but they would also have to have that specific key and sign the package with it.
So for any new keys we manually add in our system, we do not want to use apt-key to add them, and we don’t want to place them in either of those two locations, even if the repository documentation instructs us to do so.
The right way to do it for new keys is:
-
Store keys for third-party repositories in
/etc/apt/keyrings/. There’s nothing special about this directory, it’s just a suggestion from APT’s documentation, and APT doesn’t treat it in any special way; we could use any other directory, if we wanted to, as long as it is secure and accessible by APT. -
Explicitly define which key is used to authenticate each repository. This is where we will reference a file in the aforementioned directory.
Fixing the cause of the warning message
Now that we understand the cause of the problem and the general idea of how to prevent it for new keys, let’s see how to fix the problem with the existing keys and get rid of the warning message.
The first step is to move the offending keys from their current location and into /etc/apt/keyrings/.
We should be able to easily tell which keys are causing the problem because they are mentioned in the warning message. Using the example from above:
W: http://ppa.launchpad.net/git-core/ppa/ubuntu: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
It doesn’t explicitly mention the key ID but is says which repository it is used for, and with this information, we can look for the corresponding key by running sudo apt-key list to get a list of keys.
If there are too many keys listed, we can use grep to try to narrow it down to only the key we want. For example, if we want the key for the git package from the git-core PPA, we can try this:
sudo apt-key list | grep -i -A1 -B4 git
This will filter the list for lines containing “git” in them, and will show us the line that matched, the preceeding four lines (-B4), and the line before it (A1). We should get something like this:
1Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)).
2/etc/apt/trusted.gpg
3--------------------
4pub rsa1024 2009-01-22 [SC]
5 E1DD 2702 88B4 E603 0699 E45F A171 5D88 E1DF 1F24
6uid [ unknown] Launchpad PPA for Ubuntu Git Maintainers
From this, we want the key ID, which is the last 8 digits of the key fingerprint, with no whitespace in between. In our Git example, the fingerprint is E1DD 2702 88B4 E603 0699 E45F A171 5D88 E1DF 1F24, which means the key ID is E1DF1F24. We should also be able to use the whole fingerprint but it’s easier to work with the ID, since it’s much shorter and it’s still unique enough that we won’t be referencing another key by mistake.
Now we can export this key and store it in the right place: sudo apt-key export <key ID> | sudo gpg --dearmor -o /etc/apt/keyrings/<key_name>.gpg
<key ID> is the key ID we just got, and <key_name>.gpg can be any descriptive file name we want. Continuing the example for Git, we could run sudo apt-key export E1DF1F24 | sudo gpg --dearmor -o /etc/apt/keyrings/git-core.gpg.
Now we want to tell APT to use this key to authenticate the packages from the Git repository. We do that by adding [signed-by=/etc/apt/keyrings/<key_name>.gpg] between deb and the repository URL in the repository definition file. For our Git example, the file could be something like /etc/apt/sources.list.d/ppa_git_core_ppa_jammy.list and its contents would be something like this:
deb [arch=amd64 signed-by=/etc/apt/keyrings/git-core.gpg] http://ppa.launchpad.net/git-core/ppa/ubuntu jammy main`
It could also be the case that the file uses the newer Deb822 format, in which case it would be names something like ppa_git_core_ppa_jammy.sources (with the .sources extension instead of .list). If that’s the case, we add a line with Signed-By: /etc/apt/keyrings/<key_name>.gpg.
And lastly, we remove the key from the unwanted location, with sudo apt-key del <key ID>. For our example, the command would be sudo apt-key del E1DF1F24.
Cleaning up
After doing the above process for every key that is mentioned in warning messages, we should no longer have any third-party package keys in /etc/apt/trusted.gpg - but we may still have some in /etc/apt/trusted.gpg.d/. We won’t get any error messages about those but we still want to take care of them to prevent the potential security issue mentioned above.
This part may be a little tedious, depending on how many keys we have, because we’ll have to list them all (sudo apt-key list), look at each one individually and decide what we want to do with it.
There are some keys that we want to leave in that directory because that’s where the system expects them to be. For example, take a look at the file list for the debian-archive-keyring package from Debian bullseye, and you’ll see that package stores some key files in that directory. In general, any official Ubuntu or Debian keys should be left there.
Other keys that look like they belong to some third-party package we installed should either be moved into /etc/apt/keyrings/, or deleted.
To determine whether we need the key, we can check the corresponding repository configuration and see if it contains a signed-by directive, and if so, where it points to. If it points to the key file in /etc/apt/trusted.gpg.d/, we want to move it into /etc/apt/keyrings/. If there’s no signed-by directive, we can try moving the key file to a temporary location so that APT can’t find it, and run sudo apt update to see if it complains. If it does, we know we need to move the file into /etc/apt/keyrings/ and update the repository configuration with the corresponding signed-by directive, as mentioned in the previous section.
If the key is not used, we can simply delete the file.
That should do it.
Keeping an eye on things
Unfortunately, even though this deprecation notice has been around for a few years, many websites still have outdated instructions. Even worse, there are .deb packages that you download and install, and they add their own keys in one of the old locations. This means that it’s probably a good idea to have a look once in a while, and see which keys we have and where.
That’s it! I hope this is useful.
Header photo by Gabriel Heinzer on Unsplash
Raúl Santos