Updating Meilisearch on Kubernetes without losing data

[NarHakobyan]

I'm trying to update Meilisearch to the latest version on a Kubernetes cluster, but I'm worried about losing data during the process. I have updated the helm but it's stopped working. My questions are:

• Is there a recommended way to do this?
• What are the steps I should be taking to ensure that my data is not lost during the update process?

Environment:

• Meilisearch version: 0.29.0

[a5r0n]

at least from k8s side,
As long as the PVC is not deleted, the actual data is safe, but backups are always the best practice.
Create a dump and back it up elsewhere before upgrading.

[jankenr]

And what would be the suggested way to restore such a backup then?
Because currently using --import-dump takes a lot of time which causes the Probes to trigger errors and restarts
I was hoping there was a way to check the progress of an import but I could not find any.

Please advice.

Environment:

• GKE v1.22.13
• Meilisearch version: 1.1.0

[oluademola]

Hello @jankenr. Sorry for not getting back to you sooner. The backup method for an update is to create a dump. Importing the dump can take time, depending on your dataset. While the task queue is unavailable during a dump import. To monitor the progress of the import, you can always refer to the logs. Can you please let me know the error you trigger during the import?

[jordanbcooper]

@oluademola , we had a similar issue with upgrading. How can we restore dumps to an updated instance if the probes fail due to version mismatch with existing data on the PVC?

[jordanbcooper]

@NarHakobyan @jankenr did you ever figure out a better solution for this?

[troke12]

Any update for this? i want to migrate to new version because of security patches, but i won't take many times to making dumps and import dump in my cluster, but i can do that even without using kubernetes

[ngdbao]

When working with K8S pods which is stateless service.
It's so good if I we can create/export (download)/import dumbs remotely.

[pfaelzerchen]

Today I tried to update the helm chart from version 0.3.0 to 0.4.0 which includes updating meilisearch from 1.5 to 1.6. The meilisearch-0 pod crashed, and the logs told me that the database is not compatible and gave me a link to an upgrade guide.

However, the guide tells that I have to restore the database dump with the new version. But as the container immediately crashes, and I therefore cannot connect with 'kubectl exec': How should I do that? Any advice?

[svengreb]

We also faced this problem and solved it with a, more or less, way too hacky workaround by using init containers with custom scripts. They check if the database must be migrated, does so if necessary and saves the "state" in simple files. It works, but this is miles away from being the way it should be for a real production environment.
We can share the scripts, or even the whole Kubernetes manifests, but if so it must be clear that these come without any warranty and you're responsible to check and adjust them to fit your needs. At the moment this is all we can give back to this project, but we also hope that the need for a "native" implementation is important to compete on the market when it comes to running Meilisearch in a stable and scalable way.

[pfaelzerchen]

For whom it may help. I found a manual way to do the upgrade. The official guide https://www.meilisearch.com/docs/learn/update_and_migration/updating#update-to-the-latest-meilisearch-version works in principle with some modifications.

1. Just to be on the safe side: Stop the services that actually use meilisearch (gitea in my case)
2. kubectl -n gitea exec -it meilisearch-0 -- sh and dump the database as described.
3. mv /meili_data/data.ms /meili_data/data.ms.old (actually works without terminating meilisearch)
4. deploy the new container
5. kubectl -n gitea exec -it meilisearch-0 -- sh and import the dump as described.
6. kill the container to force the restart of meilisearch
7. bring the services back that depend on meilisearch

[imdmahajankanika]

Hello ! I am trying to take dump using curl but my pod is getting crashed with OOMKilled.

I want to upgrade the meilisearch version from v0.27.1 and to upgrade to a new version, I need dump. But when I am executing dump command using Curl, my pod is getting OOM killed . I tried with 5GBs of RAM but still getting OOM killed. There is nothing under dumps folder too.

In the folder /meili_data, I have approx 10 Gbs of data. I don't know why I have file data.mdb. I don't have data.ms

4.0K ./VERSION
60.0K ./auth
9.4G ./data.mdb
4.0K ./dumps
18.8M ./indexes/c763f298-caeb-4a49-9d33-23160e556eda
18.8M ./indexes
4.0K ./instance-uid
8.0K ./lock.mdb
16.0K ./lost+found
4.0K ./updates/updates_files
8.0K ./updates

[0xMostafa]

Any updates on this?