[Errno 13] Permission denied: '/var/cache/dnf/metadata_lock.pid'

[marc-hb]

mkosi commit the issue has been seen with

v24.3

Used host distribution

Fedora 40

Used target distribution

Fedora 40

Linux kernel version used

6.12

CPU architectures issue was seen on

x86_64

Unexpected behaviour you saw

Initially reported in #2730, a few more details there.

Since v23.1 commit 17010a2 "Fix invoked_as_root initialization" , Fedora 40 build fails immediately as show below. After a revert on top of v24.3, everything works fine again.

Until v23 things were fine and the problem goes away in (not released yet) v25 commit b3a3e7e ("Introduce mkosi-sandbox and stop using subuids for image builds") which rewrites everything.

Strangely, I think this used to work? I think version v24.3 used to work on my system? Or maybe not. Now the bisect is 100% deterministic.

in directory: kernel/qbuild
running: sudo -E mkosi -i -f --autologin build
‣ kernel/qbuild/mkosi.cache/fedora~40~x86-64.cache does not exist, not reusing cached images
‣ Syncing package manager metadata for default image
No matches found for the following enable plugin patterns: versionlock
[Errno 13] Permission denied: '/var/cache/dnf/metadata_lock.pid'
‣ "dnf --assumeyes --best --releasever=40 --installroot=/buildroot --setopt=keepcache=1 --setopt=logdir=/var/log --setopt=cachedir=/var/cache/dnf --setopt=persistdir=/var/lib/dnf --setopt=install_weak_deps=0 --setopt=check_config_file_age=0 '--disableplugin=*' --enableplugin builddep --enableplugin versionlock --config=/etc/dnf/dnf.conf --setopt=reposdir=/etc/yum.repos.d --setopt=varsdir=/etc/dnf/vars --setopt=proxy=http://proxy-dmz.intel.com:912 --setopt=proxy_sslcacert=/proxy.cacert makecache" returned non-zero exit code 1.

Used mkosi config

[Distribution]

[Output]
Output=root.img

[Content]
Bootable=yes

[Content]
Packages=
  openssh-clients
  openssh-server
  httpd
  ndctl
  daxctl
  cxl-cli
  jq
  sudo
  make
  gcc
  perl
  python
  libcurl-devel
  vim-enhanced
  net-tools
  iproute
  dhclient
  nmap-ncat
  lsof
  dnf
  git-core
  tar
  NetworkManager
  bash-completion
  passwd
  numactl
  numactl-libs
  fio
  fio-engine-dev-dax
  fio-engine-libaio
  fio-engine-libpmem
  strace
  ltrace
  man-db
  xfsprogs
  e2fsprogs
  acpica-tools
  autoconf
  asciidoc
  rubygem-asciidoctor
  xmlto
  automake
  libtool
  pkgconfig
  uuid
  libuuid
  libuuid-devel
  kmod-devel
  kmod-libs
  json-c
  json-c-devel
  keyutils-libs-devel
  systemd-devel
  iniparser
  iniparser-devel
  rsync
  diffutils
  parted
  keyutils
  hostname
  gdb
  tree
  htop
  nvme-cli
  cargo
  meson
  ninja-build
  vbindiff
  trace-cmd
  libtraceevent
  libtraceevent-devel
  libtracefs

  libtracefs-devel
  pciutils
  bind-utils
  systemd-resolved
  systemd-networkd
  perf
  hwloc
  systemd-boot-unsigned

[Content]
RootPassword=root

mkosi output

+ rpm --eval '%{__plugindir}'
‣ kernel/qbuild/mkosi.cache/fedora~40~x86-64.cache does not exist, not reusing cached images
‣ Syncing package manager metadata for default image
‣ Acquiring lock on /var/cache/mkosi/fedora~40~x86-64/cache/dnf
‣ Acquired lock on /var/cache/mkosi/fedora~40~x86-64/cache/dnf
‣ Acquiring lock on /var/cache/mkosi/fedora~40~x86-64/lib/dnf
‣ Acquired lock on /var/cache/mkosi/fedora~40~x86-64/lib/dnf
‣ + dnf --assumeyes --best --releasever=40 --installroot=/buildroot --setopt=keepcache=1 --setopt=logdir=/var/log --setopt=cachedir=/var/cache/dnf --setopt=persistdir=/var/lib/dnf --setopt=install_weak_deps=0 --setopt=check_config_file_age=0 '--disableplugin=*' --enableplugin builddep --enableplugin versionlock --setopt=debuglevel=10 --config=/etc/dnf/dnf.conf --setopt=reposdir=/etc/yum.repos.d --setopt=varsdir=/etc/dnf/vars --setopt=proxy=http://proxy-dmz.intel.com:912 --setopt=proxy_sslcacert=/proxy.cacert makecache
timer: config: 0 ms
No matches found for the following enable plugin patterns: versionlock
Loaded plugins: builddep
DNF version: 4.22.0
Command: dnf --assumeyes --best --releasever=40 --installroot=/buildroot --setopt=keepcache=1 --setopt=logdir=/var/log --setopt=cachedir=/var/cache/dnf --setopt=persistdir=/var/lib/dnf --setopt=install_weak_deps=0 --setopt=check_config_file_age=0 --disableplugin=* --enableplugin builddep --enableplugin versionlock --setopt=debuglevel=10 --config=/etc/dnf/dnf.conf --setopt=reposdir=/etc/yum.repos.d --setopt=varsdir=/etc/dnf/vars --setopt=proxy=http://proxy-dmz.intel.com:912 --setopt=proxy_sslcacert=/proxy.cacert makecache
Installroot: /buildroot
Releasever: 40
cachedir: /buildroot/var/cache/dnf
Base command: makecache
Extra commands: ['--assumeyes', '--best', '--releasever=40', '--installroot=/buildroot', '--setopt=keepcache=1', '--setopt=logdir=/var/log', '--setopt=cachedir=/var/cache/dnf', '--setopt=persistdir=/var/lib/dnf', '--setopt=install_weak_deps=0', '--setopt=check_config_file_age=0', '--disableplugin=*', '--enableplugin', 'builddep', '--enableplugin', 'versionlock', '--setopt=debuglevel=10', '--config=/etc/dnf/dnf.conf', '--setopt=reposdir=/etc/yum.repos.d', '--setopt=varsdir=/etc/dnf/vars', '--setopt=proxy=http://proxy-dmz.intel.com:912', '--setopt=proxy_sslcacert=/proxy.cacert', 'makecache']
Making cache files for all metadata files.
fedora: has expired and will be refreshed.
updates: has expired and will be refreshed.

Traceback (most recent call last):
  File "/usr/lib/python3.12/site-packages/dnf/cli/main.py", line 122, in cli_run
    cli.run()
  File "/usr/lib/python3.12/site-packages/dnf/cli/cli.py", line 1067, in run
    return self.command.run()
           ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/dnf/cli/commands/makecache.py", line 50, in run
    return self.base.update_cache(timer)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/dnf/base.py", line 378, in update_cache
    self.fill_sack(load_system_repo=False, load_available_repos=True)  # performs the md sync
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/dnf/base.py", line 389, in fill_sack
    with lock:
         ^^^^
  File "/usr/lib/python3.12/site-packages/dnf/lock.py", line 131, in __enter__
    pid = self._try_lock(my_pid)
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/dnf/lock.py", line 81, in _try_lock
    fd = os.open(self.target, os.O_CREAT | os.O_RDWR, 0o644)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
PermissionError: [Errno 13] Permission denied: '/var/cache/dnf/metadata_lock.pid'
[Errno 13] Permission denied: '/var/cache/dnf/metadata_lock.pid'

</details>

[DaanDeMeyer]

@marc-hb So I think all other users of mkosi have adapted a "run mkosi from git" workflow as we don't really maintain past releases, the fast paced development of mkosi and the fact that it's trivial to run from git without any build step. I see that you're doing a lot of work to add compat for various versions of mkosi to pmem/run_qemu, but if possible I'd like to suggest pinning a mkosi git commit in CI and suggesting users use that or a newer one. Any bugs in the latest version I can fix immediately whereas with bugs in older versions you're usually stuck with.

[marc-hb]

Thanks for the openness and advice!

I see that you're doing a lot of work to add compat for various versions of mkosi to pmem/run_qemu,

My plan is not to support every mkosi git tag under the sun, I'm just looking for some versions that work for us.

The plan is to first make run_qemu compatible with v15+ which was a massive break of compatibility, while not immediately breaking users stuck with v14- for now. Maintaining a temporary compatibility overlap with both v14- and v15+ was not a big challenge because the configuration files are completely distinct.

The actual challenges have been: 1. Understanding how v15+ configuration works (mostly done) 2. Bugs and quirks in run_qemu itself. For understanding these, temporary compatibility with v14- is critical. I discovered some run_qemu features that never... worked! 3. Making "some" (not "all") mkosi v15+ versions work. Preferably a range close to the versions released by latest releases of popular distros but not necessarily. The mkosi needs of run_qemu are very basic so I'm optimistic.

For 3, it is necessary to know and track mkosi issues, which is the sole reason I filed this. As I wrote in #2730, I opened this bug only for reference; I'm expecting it to be closed as "disappeared in v25" (even though it is not released yet).

it's trivial to run from git without any build step

That's how I've been testing. However run_qemu runs mkosi as root, which requires running pip install --break-system-packages -e . as root which does not look great in user documentation. We'll see.

but if possible I'd like to suggest pinning a mkosi git commit in CI and suggesting users use that...

Yes, just need to find one first ;-)

... or a newer one.

Newer version, newer bugs...