v4 upgrade fails to start

[JamesChevalier]

Version:

• listmonk: v4.0.1
• OS: Ubuntu 23.10

Description of the bug and steps to reproduce:
After upgrading to v4 the containers fail to start.

The app container has these logs:

listmonk_app  | Launching listmonk with user=[root] group=[root] PUID=[0] PGID=[0]
listmonk_app  | 2024/10/28 16:14:01.804258 main.go:107: v4.0.1 (f5dfb0c 2024-10-28T07:48:58Z, linux/amd64)
listmonk_app  | 2024/10/28 16:14:01.808478 init.go:302: connecting to db: listmonk_db:5432/listmonk
listmonk_app  | 2024/10/28 16:14:01.832362 init.go:306: error connecting to DB: dial tcp: lookup listmonk_db on 127.0.0.11:53: server misbehaving

The db container has these logs:

listmonk_db  | PostgreSQL Database directory appears to contain a database; Skipping initialization
listmonk_db  | 
listmonk_db  | 2024-10-28 16:13:57.497 UTC [1] FATAL:  database files are incompatible with server
listmonk_db  | 2024-10-28 16:13:57.497 UTC [1] DETAIL:  The data directory was initialized by PostgreSQL version 13, which is not compatible with this version 17.0.

I mainly followed the upgrade nodes in https://listmonk.app/docs/upgrade/#docker with the addition of managing the new docker-compose.yml file in a way that seemed right. My step-by-step upgrade process was:

• curl -LO https://github.com/knadh/listmonk/raw/master/docker-compose.yml
• mv docker-compose.yml listmonk/docker-compose.yml.new
• cd listmonk
• Edit the POSTGRES_PASSWORD and hostname entries in the docker-compose.yml.new file
• docker compose down app
• mv docker-compose.yml docker-compose.yml.old
• mv docker-compose.yml.new docker-compose.yml
• docker compose pull
• docker compose up app -d

I'm unsure how to resolve the situation with the db container in a constant restarting cycle.

Screenshots:
If applicable, add screenshots to help explain your problem.

---

UPDATE
I found a different set of upgrade steps in https://github.com/knadh/listmonk/releases/tag/v4.0.1 which is producing a different error:

$ docker-compose run --rm app ./listmonk --upgrade
Creating network "listmonk_listmonk" with the default driver
Creating listmonk_db ... done
Creating listmonk_app_run ... done
Launching listmonk with user=[root] group=[root] PUID=[0] PGID=[0]
2024/10/28 16:32:04.670925 main.go:107: v4.0.1 (f5dfb0c 2024-10-28T07:48:58Z, linux/amd64)
2024/10/28 16:32:04.677513 init.go:162: reading config: config.toml
2024/10/28 16:32:04.677991 init.go:302: connecting to db: listmonk_db:5432/listmonk
2024/10/28 16:32:04.684539 init.go:306: error connecting to DB: dial tcp 172.21.0.2:5432: connect: connection refused
ERROR: 1

If I immediately run docker compose logs db it displays:

listmonk_db  | 
listmonk_db  | PostgreSQL Database directory appears to contain a database; Skipping initialization
listmonk_db  | 
listmonk_db  | 2024-10-28 16:32:03.577 UTC [1] FATAL:  database files are incompatible with server
listmonk_db  | 2024-10-28 16:32:03.577 UTC [1] DETAIL:  The data directory was initialized by PostgreSQL version 13, which is not compatible with this version 17.0.

[nkosi23]

This is mainly related to your environment (i've run the upgrade earlier today and everything worked as expected).

From the error message, it looks like your new container is running postgres 17 while you were previously using postgres 13, this is root cause of the problem.

I am not a docker user myself but try modifying this line in the docker-compose.yml file:

    image: postgres:17-alpine

By replacing it with the below instead:

    image: postgres:13-alpine

[JamesChevalier]

I've been able to successfully upgrade to v4.0.1 while using the old docker-compose.yml file. In that scenario, the "How to upgrade" section of https://github.com/knadh/listmonk/releases/tag/v4.0.1 works as expected. That set of steps includes the upgrade command, which is missing from the main docs. I'll get a PR in for that addition in a moment.

I haven't been successful in updating to the new docker-compose.yml file, due to the upgrade from PostgreSQL 13 to 17. That process doesn't seem to be documented anywhere.

Thanks @nkosi23

[nkosi23]

Great to hear that!

I think the difficulty is that for new users they have to propose a docker image having the latest stable versions of software such as postgres (as it wouldn't make sense to require new users to start with postgres13 in 2024).

At the same time it would be challenging to provide different images for all combination of legacy software existing users may be running. Therefore I think that you may have to slightly tinker with the docker-compose.yml file from time to time.

[JamesChevalier]

Fair, we do have about a year before v13 is no longer supported. It would be great to have some official docs on how to make the v13->v17 jump, since it's an inevitability.

I was definitely confused by the phrasing in the upgrade docs and the release notes. I took them to mean that the new docker-compose.yml was ready to go for any upgrade to v4.0.1 ... not that further editing would be required to make them compatible.

It seems like several others are confused by the recent doc updates to be so heavily v4-oriented - #2108 & #2107 (and now my #2113 because I didn't notice those)

Thanks again!

[knadh]

Explicitly addressed docker compose instructions for the older files and the new version: https://listmonk.app/docs/upgrade/#docker

It would be great to have some official docs on how to make the v13->v17 jump, since it's an inevitability.

This will involve the standard steps for doing a pg_dump of the old DB and importing to the dump to the new DB. Did a quick search and found these:

• https://thomasbandt.com/postgres-docker-major-version-upgrade
• https://github.com/tianon/docker-postgres-upgrade

[knadh]

At the same time it would be challenging to provide different images for all combination of legacy software existing users may be running. Therefore I think that you may have to slightly tinker with the docker-compose.yml file from time to time.

That is correct. The docker-compose files that repositories have (like the one listmonk has) are examples and starter references, which the end users are expected to customize to work for their specific needs.

[JamesChevalier]

This will involve the standard steps for doing a pg_dump of the old DB and importing to the dump to the new DB.

I was tripped up on the fact that the db container would only go into restart loop. Your links helped me understand the process of isolating the db container outside of the full app context in order to work with the data.

Here's the full process I came up with, in case it's helpful for anyone (or if you want to include it in the docs as a guide to upgrading PostgreSQL):

• Make a folder to hold our data (do this wherever you have your docker-compose.yml file)
  • mkdir data
• Edit docker-compose.yml to add a bind mount for the data folder to the db container

- type: bind
  source: /path/to/data
  target: /data

• Shut down the containers and start just the db container
  • docker compose down
  • docker compose up -d db
• Dump the database from inside the container
  • docker exec -it listmonk_db sh
  • pg_dump -U listmonk listmonk > /data/listmonk.sql
  • exit
• Shut down the db container
  • docker compose down
• Edit docker-compose.yml
  • Change the image for the db container to postgres:17-alpine
  • Change the volume for the db container to use a new name e.g. listmonk-data-pg17
  • Add this new entry to the bottom volumes list as well
• Start just the db container
  • docker-compose up -d db
• Import the database from inside the container
  • docker exec -it listmonk_db sh
  • psql -d listmonk -U listmonk -f /data/listmonk.sql
• Restart the full system
  • docker compose down
  • docker compose up -d
• Verify things work as expected
• Edit docker-compose.yml
  • Remove the data bind mount
  • Remove the old listmonk-data volume
• Restart the containers
  • docker compose down
  • docker compose up -d
• Remove the old volume
  • docker volume rm listmonk-data