Mender Authorization Failures and Docker Container Restart Issues

[TylonHH]

My Settup

• RPi 5 with 8GB Ram
• LAN Connection
• 2TB NVme SSD HAT
• Plug original RPi with 5.1V and 5A, no other USB devices

Bitcoin Node is fully synced. Electrs is still syncing but after some hours the Web UI is not reachable, then no SSH connection is possible, but I can ping the RPi. I placed while syncing an extra strong extern fan next to the RPi´. Temp went from 70°C to 48°C.

Here is my findings within the journal

Issue Title: Mender Authorization Failures and Docker Container Restart Issues

---

Description:

The system is experiencing multiple issues related to Mender authorization and Docker container restarts. Below is a detailed summary of the observed problems:

---

1. Mender Authorization Failures

• Errors:

  • Mender consistently fails to authorize, citing missing identity data.
  • The file /usr/share/mender/identity/mender-device-identity is not found, leading to repeated "transient error" messages.
  • Inventory submissions and update checks fail due to these authorization issues.
  • Example logs:

    Failed to authorize with "": failed to build authorization request: failed to obtain authorization message data: failed to obtain identity data: failed to call /usr/share/mender/identity/mender-device-identity: fork/exec /usr/share/mender/identity/mender-device-identity: no such file or directory
    

• Impact:

  • Mender is unable to submit inventory data or perform update checks, and enters an error state.

---

2. Docker Container Restart Failures

• Warnings:

  • Docker containers fail to restart with the error: ShouldRestart failed, container will not be restarted.
  • Several containers are manually stopped, preventing automatic restarts.
  • Example logs:

    ShouldRestart failed, container will not be restarted
    container=38cce14ce749cfe6253f66e526bf20471339b267ef58732d88267a26c87504c5
    daemonShuttingDown=false error="restart canceled"
    

• Impact:

  • Critical containers (e.g., Electrs, Mender) do not restart as expected, potentially disrupting system functionality.

---

3. Containerd Plugin Warnings

• Warnings:

  • Containerd plugins for aufs, btrfs, and zfs are skipped due to missing configurations or unsupported filesystems.
  • Example logs:

    skip loading plugin "io.containerd.snapshotter.v1.aufs" error="aufs is not supported"
    skip loading plugin "io.containerd.snapshotter.v1.zfs" error="path must be a zfs filesystem"
    

• Impact:

  • These warnings may indicate misconfigurations in the container runtime environment.

---

4. Mender Update Commit Failure

• Error:

  • Mender fails to commit updates with the error: "Could not commit Artifact: There is nothing to commit".
  • Example log:

    Failed to commit OS partition: Command failed with exit code 2: mender commit
    

• Impact:

  • Updates are not properly applied, leaving the system in an incomplete or inconsistent state.

---

Steps to Reproduce:

1. Start the system and monitor logs for Mender authorization failures.
2. Check Docker logs for container restart failures.
3. Attempt to manually restart critical containers (e.g., Electrs, Mender).
4. Inspect Mender's inventory submission and update processes.

---

Expected Behavior:

• Mender should successfully authorize and submit inventory data.
• Docker containers should restart automatically unless explicitly configured otherwise.
• Containerd should not log unsupported plugins as warnings if they are not required.

---

System Information:

• OS: Raspberry Pi-based system running Umbrel.
• Kernel Version: 6.6.51+rpt-rpi-2712
• Docker Version: Refer to logs.
• Mender Version: Refer to logs.

---

Proposed Solutions:

1. Mender:

   • Verify and restore /usr/share/mender/identity/mender-device-identity.
   • Ensure Mender's configuration (e.g., /etc/mender/mender.conf) is accurate.

2. Docker:

   • Review and correct the restart policies for critical containers.
   • Investigate why containers are being manually stopped.

3. Containerd:

   • Update configurations to suppress warnings for unsupported plugins.
   • Verify runtime compatibility with the current filesystem setup.

4. Update Process:

   • Resolve Mender's "no artifact to commit" error to ensure updates complete successfully.

---

Please advise on further debugging steps or potential fixes for these issues.