Podman Bug: Extra Malformed Label Created

by Sebastian Müller 42 views

Hey guys!

We've got a tricky issue to dive into today with Podman, specifically when using the podman run command with labels. It seems like there's a bug where adding labels using the key=value format results in an extra, malformed label being created. Let's break down the problem, how to reproduce it, and what the expected behavior should be. Buckle up, it's container time!

Issue Description

So, here's the deal. When you're spinning up a container with podman run and you use the --label flag like this: "key=value", Podman does apply the label you intended. Great! But, it also sneakily adds a second, not-so-great label. This second label is malformed because the entire string "key=value" becomes the label's key, and the value is just empty. It's like Podman is a little too enthusiastic about labels.

This quirky behavior is super consistent, even with the simplest test cases. This suggests the issue lies within the label parsing logic of the podman run command itself. We first stumbled upon this when using Quadlet (which uses .container files) to manage containers via systemd. It's always fun when a small issue leads to a bigger investigation, right?

To reiterate, the main problem is the creation of this additional, malformed label.

Reproducing the Bug: A Step-by-Step Guide

Okay, let's get our hands dirty and see this bug in action. Here's how you can reproduce it:

  1. Run a minimal container with a correctly formatted label. We'll use a clean image like Alpine Linux for this. Fire up your terminal and paste this command:
podman run -d --rm --name minimal-label-bug-test \
  --label com.centurylinklabs.watchtower.enable=true \
  docker.io/library/alpine sleep infinity

Let's break this down:

  • podman run: This is the command to run a container.
  • -d: Runs the container in detached mode (in the background).
  • --rm: Automatically removes the container when it exits.
  • --name minimal-label-bug-test: Gives our container a catchy name.
  • --label com.centurylinklabs.watchtower.enable=true: This is where the magic (or rather, the bug) happens. We're setting a label with the key com.centurylinklabs.watchtower.enable and the value true. This is a common label used by Watchtower, a tool for automatically updating container images.
  • docker.io/library/alpine: Specifies the Alpine Linux image from Docker Hub.
  • sleep infinity: Keeps the container running indefinitely so we can inspect it.

Remember, the core of the issue lies in how the --label flag is parsed when given a key=value pair.

  1. Inspect the container's labels. Now that the container is running, let's peek at its labels using the podman inspect command:
podman inspect minimal-label-bug-test | grep com.centurylinklabs.watchtower.enable

This command will output the container's details, and we're using grep to filter for the label we just added.

The Expected vs. The Actual Results

Describe the results you received

When you run the podman inspect command, you'll see something like this:

"com.centurylinklabs.watchtower.enable": "true",
"com.centurylinklabs.watchtower.enable=true",

Notice anything odd? We've got two entries for the label. The first one is correct – it's the key: value pair we intended. But the second one is…well, a bit of a mess. The entire "com.centurylinklabs.watchtower.enable=true" string is treated as the key, and the value is missing (or rather, an empty string). This is the malformed label we're talking about.

Describe the results you expected

Ideally, podman inspect should only show one entry for the label, and it should be the correctly formatted JSON key-value pair:

"com.centurylinklabs.watchtower.enable": "true",

That's it! Clean, simple, and exactly what we asked for. The fact that we're getting this extra, garbled label indicates a parsing issue within Podman.

Digging Deeper: Podman Info Output

To give you guys a complete picture, here's the output of podman info from the system where this bug was reproduced. This information can be helpful for developers to understand the environment and potentially pinpoint the root cause:

host:
  arch: amd64
  buildahVersion: 1.40.1
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-1:2.1.13-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.13, commit: 82de887596ed8ee6d9b2ee85e4f167f307bb569b'
  cpuUtilization:
    idlePercent: 99.32
    systemPercent: 0.26
    userPercent: 0.42
  cpus: 1
  databaseBackend: sqlite
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  freeLocks: 2045
  hostname: KKKZZZ
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.15.8-arch1-2
  linkmode: dynamic
  logDriver: journald
  memFree: 137682944
  memTotal: 2057535488
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.15.0-1
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.15.0
    package: netavark-1.15.2-1
    path: /usr/lib/podman/netavark
    version: netavark 1.15.2
  ociRuntime:
    name: crun
    package: crun-1.23.1-1
    path: /usr/bin/crun
    version: |-
      crun version 1.23.1
      commit: d20b23dba05e822b93b82f2f34fd5dada433e0c2
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-2025_06_11.0293c6f-1
    version: |
      pasta 2025_06_11.0293c6f
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 0
  swapTotal: 0
  uptime: 97h 55m 19.00s (Approximately 4.04 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
  - ghcr.io
  - quay.io
  - gcr.io
store:
  configFile: /home/seine/.config/containers/storage.conf
  containerStore:
    number: 3
    paused: 0
    running: 3
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/seine/.local/share/containers/storage
  graphRootAllocated: 31525343232
  graphRootUsed: 13861830656
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 5
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/seine/.local/share/containers/storage/volumes
version:
  APIVersion: 5.5.2
  Built: 1751012144
  BuiltTime: Fri Jun 27 16:15:44 2025
  GitCommit: e7d8226745ba07a64b7176a7f128e4ef53225a0e
  GoVersion: go1.24.4
  Os: linux
  OsArch: linux/amd64
  Version: 5.5.2

This output provides details about the host system, Podman's configuration, storage, network settings, and more. Key things to note here are the Podman version (5.5.2), the OS (linux), the architecture (amd64), and the container runtime (crun). The store section gives insights into how container images and layers are stored on the system.

Other Relevant Details

  • Podman in a container: This issue was reproduced outside of a containerized Podman environment (i.e., Podman running directly on the host).
  • Privileged or Rootless: The tests were performed in a rootless environment (running as a non-root user).
  • Upstream Latest Release: This bug was observed on a relatively recent version of Podman, suggesting it's not a long-standing issue.

Conclusion and Next Steps

So, there you have it – a reproducible bug in Podman's label parsing logic. While it might seem minor, this kind of issue can lead to unexpected behavior, especially when relying on labels for container management and automation. Hopefully, this detailed write-up will help the Podman developers track down and fix the bug. If you run into this, too, you now know you're not alone!

Stay tuned for updates, and happy containerizing!