Logging

Read, copy and keep the logs of a Kubernetes deployment.

Every service inside the Ayakaleaf Pro pod writes its own log file under /var/log/overleaf/; kubectl logs shows only the container’s startup output (init scripts, migrations). The Toolkit’s bin/logs reads the same files, see Logging.

Where	What
kubectl -n overleaf logs <pod>	startup: secrets check, migrations (Finished migrations), service starts
/var/log/overleaf/<service>.log in the pod	web.log, web-api.log, clsi.log, real-time.log, document-updater.log, docstore.log, filestore.log, history-v1.log, project-history.log, chat.log, notifications.log, linked-url-proxy.log, and third-party-datastore.log with Dropbox on
/var/log/nginx/ in the pod	access.log, error.log of the nginx in front of the services
kubectl -n overleaf logs mongo-0, redis-0, git-bridge-0	MongoDB, Redis and git-bridge, which run in their own pods

Reading logs

Toolkit	Kubernetes
bin/logs web	kubectl -n overleaf exec deploy/overleaf -- tail -n 20 /var/log/overleaf/web.log
bin/logs -f web clsi	kubectl -n overleaf exec deploy/overleaf -- tail -f /var/log/overleaf/web.log /var/log/overleaf/clsi.log
bin/logs -n all web > web.log	kubectl -n overleaf cp overleaf/<pod>:/var/log/overleaf/web.log web.log
bin/logs mongo	kubectl -n overleaf logs mongo-0

With several replicas, each pod has its own files: use the pod name (overleaf-0, overleaf-1, …) instead of deploy/overleaf. The X-Served-By header that HAProxy adds tells which replica answered a request.

LOG_LEVEL in the ConfigMap (default info, also debug, warn, error) sets the level of every service in the pod at once.

Keeping logs

The files live in the container and are lost when the pod is replaced. To keep them, mount a volume at /var/log/overleaf, for example a hostPath on the node or a PersistentVolumeClaim, in the Deployment or StatefulSet:

          volumeMounts:
            # ...
            - name: logs
              mountPath: /var/log/overleaf
      volumes:
        # ...
        - name: logs
          hostPath:
            path: /srv/overleaf-logs
            type: DirectoryOrCreate

The container rotates the files daily and keeps five, so the volume does not grow without bound. Or collect them with your cluster’s log agent from that path.

Tested on Kubernetes

On Kubernetes 1.37 with three replicas and a hostPath volume at /var/log/overleaf as above, each node’s /srv/overleaf-logs held the twelve service logs of its replica, and the commands in the table worked as shown: tail through kubectl exec, kubectl cp of a log file, kubectl logs of the Overleaf pod (startup and migrations) and of mongo-0.