> ## Documentation Index
> Fetch the complete documentation index at: https://ayakaleaf-pro.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Upgrading on Kubernetes

> Upgrade a Kubernetes deployment of Ayakaleaf Pro, with one replica or several.

An upgrade replaces the image tag. The new version migrates the database when it starts, and these migrations must run **once, from one pod, while no older pod is running**. On Kubernetes this rules out a plain rolling update of a scaled StatefulSet.

> [!CAUTION]
>
> Take a [consistent backup](https://ayakaleaf-pro.mintlify.site/on-premises/maintenance/data-and-backups#performing-a-consistent-backup) before every upgrade, with Ayakaleaf Pro stopped. Read the [release notes](https://ayakaleaf-pro.mintlify.site/on-premises/release-notes) for steps of the version you upgrade to.

Before you start, on every node that runs Ayakaleaf Pro, pull any TeX Live image the new version or your new configuration lists, with `sudo docker pull`.

> [tab] One replica
>
> **1. Stop Ayakaleaf Pro**
>
> ```bash theme={null}
> kubectl -n overleaf scale deployment/overleaf --replicas=0
> kubectl -n overleaf wait --for=delete pod -l app=overleaf --timeout=5m
> ```
>
> **2. Back up**
>
> MongoDB, from any member:
>
> ```bash wrap theme={null}
> kubectl -n overleaf exec mongo-0 -- mongodump --quiet --archive --gzip \
>   --uri 'mongodb://mongo-0.mongo:27017,mongo-1.mongo:27017,mongo-2.mongo:27017/sharelatex?replicaSet=overleaf' \
>   > sharelatex-$(date +%F).archive.gz
> ```
>
> The data volume, through a short-lived pod that mounts it (it lands on the volume's node):
>
> ```bash wrap theme={null}
> kubectl -n overleaf run backup --image=busybox --restart=Never --overrides='
> {"spec":{"containers":[{"name":"backup","image":"busybox","command":["sleep","3600"],
>  "volumeMounts":[{"name":"d","mountPath":"/data"}]}],
>  "volumes":[{"name":"d","persistentVolumeClaim":{"claimName":"overleaf-data"}}]}}'
> kubectl -n overleaf wait --for=condition=Ready pod/backup
> kubectl -n overleaf exec backup -- tar czf - -C /data . > overleaf-data-$(date +%F).tar.gz
> kubectl -n overleaf delete pod backup
> ```
>
> Do the same for git-bridge's volume (`data-git-bridge-0`), and keep a copy of the Secret `overleaf-secrets`. If your storage supports [VolumeSnapshots](https://kubernetes.io/docs/concepts/storage/volume-snapshots/), a snapshot of the volumes does the same.
>
> **3. Change the version and start**
>
> Set the new tag in `40-overleaf.yaml` (and in `45-git-bridge.yaml`, which must use the same tag), then:
>
> ```bash theme={null}
> kubectl apply -f 40-overleaf.yaml -f 45-git-bridge.yaml
> kubectl -n overleaf scale deployment/overleaf --replicas=1
> kubectl -n overleaf rollout status deployment/overleaf --timeout=20m
> kubectl -n overleaf logs deployment/overleaf | grep 'Finished migrations'
> ```
>
> The Deployment uses `strategy: Recreate`, so even a changed tag never runs two versions at once.

> [tab] Several replicas
>
> For the setup in [Horizontal scaling](https://ayakaleaf-pro.mintlify.site/on-premises/configuration/kubernetes/horizontal-scaling.md).
>
> **1. Stop all replicas**
>
> ```bash theme={null}
> kubectl -n overleaf scale statefulset/overleaf --replicas=0
> kubectl -n overleaf wait --for=delete pod -l app=overleaf-ha --timeout=10m
> ```
>
> **2. Back up**
>
> MongoDB as in the one-replica tab, your S3 buckets with your provider's tools, git-bridge's volume (`data-git-bridge-0`) and the Secrets. The replicas' `/srv/overleaf-ha` only hold compile output and need no backup.
>
> **3. Start one replica with the new version**
>
> Set the new tag in `70-overleaf.yaml` and `75-git-bridge.yaml`, and `replicas: 1`:
>
> ```bash theme={null}
> kubectl apply -f 70-overleaf.yaml -f 75-git-bridge.yaml
> kubectl -n overleaf rollout status statefulset/overleaf --timeout=20m
> kubectl -n overleaf logs overleaf-0 | grep 'Finished migrations'
> ```
>
> Check that you can log in, open a project and compile.
>
> **4. Start the other replicas**
>
> Set `replicas: 3` again and apply. `OrderedReady` starts them one after the other.
>
> ```bash theme={null}
> kubectl apply -f 70-overleaf.yaml
> kubectl -n overleaf rollout status statefulset/overleaf --timeout=20m
> ```
>
> > [!WARNING]
> >
> > Don't change the image of a running scaled StatefulSet with `kubectl set image` or `kubectl apply`: the rolling update would start the new version, and its migrations, while older replicas still serve users. Configuration changes without a new version are fine as a rolling restart: `kubectl -n overleaf rollout restart statefulset/overleaf`.

### MongoDB and Redis

Upgrade them separately from Ayakaleaf Pro. For MongoDB, follow [Updating MongoDB](https://ayakaleaf-pro.mintlify.site/on-premises/maintenance/updating-mongodb.md) (one major version at a time, `featureCompatibilityVersion` after each step); a replica set takes the new image one member at a time when you change the image in `10-mongo.yaml`, because the StatefulSet replaces `mongo-2`, `mongo-1` and `mongo-0` in turn.

### Rolling back

Stop Ayakaleaf Pro as above, restore MongoDB from the archive (`mongorestore --drop --archive --gzip`) and the volumes or buckets from the backup, set the old tag and start again. The new version's migrations cannot be undone otherwise.

### Q\&A

> [accordion] A replica stays not ready, and the other replicas never start
>
> **Answer:** With `podManagementPolicy: OrderedReady`, `overleaf-1` only starts once `overleaf-0` is ready, and a rolling update does not replace a pod that never became ready. After you fixed the cause, for example a wrong setting in the ConfigMap, delete the stuck pod so that the StatefulSet creates it again from the current template:
>
> ```bash theme={null}
> kubectl -n overleaf delete pod overleaf-0
> kubectl -n overleaf rollout status statefulset/overleaf --timeout=20m
> ```
>
> We ran into this on our test cluster after a setting kept `overleaf-0` from becoming ready; deleting the pod let all three replicas start.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.