Upgrading on Kubernetes

Upgrade a Kubernetes deployment of Ayakaleaf Pro, with one replica or several.

An upgrade replaces the image tag. The new version migrates the database when it starts, and these migrations must run once, from one pod, while no older pod is running. On Kubernetes this rules out a plain rolling update of a scaled StatefulSet.

Take a consistent backup before every upgrade, with Ayakaleaf Pro stopped. Read the release notes for steps of the version you upgrade to.

Before you start, on every node that runs Ayakaleaf Pro, pull any TeX Live image the new version or your new configuration lists, with sudo docker pull.

One replica

1. Stop Ayakaleaf Pro

   kubectl -n overleaf scale deployment/overleaf --replicas=0
   kubectl -n overleaf wait --for=delete pod -l app=overleaf --timeout=5m
   
2. Back up

   MongoDB, from any member:

   kubectl -n overleaf exec mongo-0 -- mongodump --quiet --archive --gzip \
     --uri 'mongodb://mongo-0.mongo:27017,mongo-1.mongo:27017,mongo-2.mongo:27017/sharelatex?replicaSet=overleaf' \
     > sharelatex-$(date +%F).archive.gz
   

   The data volume, through a short-lived pod that mounts it (it lands on the volume’s node):

   kubectl -n overleaf run backup --image=busybox --restart=Never --overrides='
   {"spec":{"containers":[{"name":"backup","image":"busybox","command":["sleep","3600"],
    "volumeMounts":[{"name":"d","mountPath":"/data"}]}],
    "volumes":[{"name":"d","persistentVolumeClaim":{"claimName":"overleaf-data"}}]}}'
   kubectl -n overleaf wait --for=condition=Ready pod/backup
   kubectl -n overleaf exec backup -- tar czf - -C /data . > overleaf-data-$(date +%F).tar.gz
   kubectl -n overleaf delete pod backup
   

   Do the same for git-bridge’s volume (data-git-bridge-0), and keep a copy of the Secret overleaf-secrets. If your storage supports VolumeSnapshots, a snapshot of the volumes does the same.

3. Change the version and start

   Set the new tag in 40-overleaf.yaml (and in 45-git-bridge.yaml, which must use the same tag), then:

   kubectl apply -f 40-overleaf.yaml -f 45-git-bridge.yaml
   kubectl -n overleaf scale deployment/overleaf --replicas=1
   kubectl -n overleaf rollout status deployment/overleaf --timeout=20m
   kubectl -n overleaf logs deployment/overleaf | grep 'Finished migrations'
   

   The Deployment uses strategy: Recreate, so even a changed tag never runs two versions at once.

Several replicas

For the setup in Horizontal scaling.

1. Stop all replicas

   kubectl -n overleaf scale statefulset/overleaf --replicas=0
   kubectl -n overleaf wait --for=delete pod -l app=overleaf-ha --timeout=10m
   
2. Back up

   MongoDB as in the one-replica tab, your S3 buckets with your provider’s tools, git-bridge’s volume (data-git-bridge-0) and the Secrets. The replicas’ /srv/overleaf-ha only hold compile output and need no backup.

3. Start one replica with the new version

   Set the new tag in 70-overleaf.yaml and 75-git-bridge.yaml, and replicas: 1:

   kubectl apply -f 70-overleaf.yaml -f 75-git-bridge.yaml
   kubectl -n overleaf rollout status statefulset/overleaf --timeout=20m
   kubectl -n overleaf logs overleaf-0 | grep 'Finished migrations'
   

   Check that you can log in, open a project and compile.

4. Start the other replicas

   Set replicas: 3 again and apply. OrderedReady starts them one after the other.

   kubectl apply -f 70-overleaf.yaml
   kubectl -n overleaf rollout status statefulset/overleaf --timeout=20m
   

Don’t change the image of a running scaled StatefulSet with kubectl set image or kubectl apply: the rolling update would start the new version, and its migrations, while older replicas still serve users. Configuration changes without a new version are fine as a rolling restart: kubectl -n overleaf rollout restart statefulset/overleaf.

MongoDB and Redis

Upgrade them separately from Ayakaleaf Pro. For MongoDB, follow Updating MongoDB (one major version at a time, featureCompatibilityVersion after each step); a replica set takes the new image one member at a time when you change the image in 10-mongo.yaml, because the StatefulSet replaces mongo-2, mongo-1 and mongo-0 in turn.

Rolling back

Stop Ayakaleaf Pro as above, restore MongoDB from the archive (mongorestore --drop --archive --gzip) and the volumes or buckets from the backup, set the old tag and start again. The new version’s migrations cannot be undone otherwise.

Q&A

A replica stays not ready, and the other replicas never start

Answer: With podManagementPolicy: OrderedReady, overleaf-1 only starts once overleaf-0 is ready, and a rolling update does not replace a pod that never became ready. After you fixed the cause, for example a wrong setting in the ConfigMap, delete the stuck pod so that the StatefulSet creates it again from the current template:

kubectl -n overleaf delete pod overleaf-0
kubectl -n overleaf rollout status statefulset/overleaf --timeout=20m

We ran into this on our test cluster after a setting kept overleaf-0 from becoming ready; deleting the pod let all three replicas start.