Dropbox Synchronisation

Let users sync their projects with a folder in their Dropbox on a Kubernetes deployment.

Dropbox Synchronisation uses three parts of every Ayakaleaf Pro pod, and all three start only with DROPBOX_ENABLED: "true":

• web (the dropbox module): linking, the project’s Dropbox dialog, and the webhook that Dropbox calls.
• third-party-datastore: a service of its own inside the pod that works through the queue of changes to send to or fetch from Dropbox.
• document-updater: with Dropbox on, it writes the editor’s changes to MongoDB in the background, a few minutes after everyone has left a project; only then do they go to Dropbox.

Create the Dropbox app and read how syncing works in Dropbox Synchronisation; this page puts it into Kubernetes.

Services	web (dropbox module), third-party-datastore and document-updater, in every Ayakaleaf Pro pod
Dropbox → Overleaf	POST ${OVERLEAF_SITE_URL}/dropbox/webhook from the internet, signed with the app secret
Pod → Dropbox	https://api.dropboxapi.com and https://content.dropboxapi.com
Browser → Dropbox	https://www.dropbox.com/oauth2/authorize, then back to ${OVERLEAF_SITE_URL}/user/dropbox/oauth/callback
State	the queue of pending changes and its locks in Redis; the sync state in MongoDB (dropboxEntities, dropboxProjects)
Background work	third-party-datastore checks its queue every second; document-updater writes projects nobody has had open for 3 minutes

Configuration

1. ConfigMap

   30-config.yaml (additions)

     DROPBOX_ENABLED: "true"
     DROPBOX_APP_KEY: <app key>
     DROPBOX_APP_NAME: Overleaf      # the folder name under Apps/ in Dropbox
     # optional: send the requests to Dropbox through an HTTP proxy
     # DROPBOX_SYNC_PROXY_URL: http://proxy.example.com:3128
   

   GIT_BRIDGE_ENABLED: "true" and the git-bridge StatefulSet from Git integration must be in place: without them the account settings hide the project sync section.

2. Secret

   kubectl -n overleaf patch secret overleaf-secrets --type merge -p '{"stringData":{
     "DROPBOX_APP_SECRET":"<app secret>",
     "DROPBOX_TOKEN_CIPHER_PASSWORD":"'"$(openssl rand -hex 32)"'"}}'
   

   DROPBOX_TOKEN_CIPHER_PASSWORD encrypts the users’ Dropbox tokens in MongoDB. Every replica must use the same value, and changing it later makes every user link Dropbox again.

3. Create the database indexes

   Run the index commands from Create the database indexes once, on the primary of the replica set:

   kubectl -n overleaf exec -i mongo-0 -- mongosh --quiet \
     'mongodb://mongo-0.mongo:27017,mongo-1.mongo:27017,mongo-2.mongo:27017/sharelatex?replicaSet=overleaf' <<'EOF'
   db.dropboxEntities.createIndex(
     { 'overleaf.userId': 1, 'dropbox.id': 1 },
     { name: 'overleaf.userId_1_dropbox.id_1', unique: true,
       partialFilterExpression: { 'dropbox.id': { $exists: true } } })
   db.dropboxEntities.createIndex(
     { 'overleaf.userId': 1, 'overleaf.id': 1 },
     { name: 'overleaf.userId_1_overleaf.id_1', unique: true,
       partialFilterExpression: { 'overleaf.id': { $exists: true } } })
   db.dropboxEntities.createIndex({ 'overleaf.userId': 1, 'dropbox.pathLower': 'hashed' })
   db.dropboxProjects.createIndex({ userId: 1, projectId: 1 }, { name: 'projectId_userId_1', unique: true })
   db.dropboxProjects.createIndex({ userId: 1, dropboxId: 1 }, { name: 'userId_dropboxId_1', unique: true })
   EOF
   
4. Apply and restart

   kubectl apply -f 30-config.yaml
   kubectl -n overleaf rollout restart deployment/overleaf     # or statefulset/overleaf
   kubectl -n overleaf exec deploy/overleaf -- tail -n 5 /var/log/overleaf/third-party-datastore.log
   

   third-party-datastore.log exists only when DROPBOX_ENABLED is "true".

5. Open the webhook to the internet

   Dropbox must reach ${OVERLEAF_SITE_URL}/dropbox/webhook from the internet. If your ingress or load balancer only allows some paths or networks from outside, allow this one; it is the only path Dropbox calls. Check it from outside the cluster:

   curl "https://overleaf.example.com/dropbox/webhook?challenge=test"   # prints: test
   

   Then add the webhook in the Dropbox App Console as described on the Toolkit page.

Several replicas

Nothing extra beyond the shared DROPBOX_TOKEN_CIPHER_PASSWORD:

• The webhook may reach any replica. It only queues the change in Redis.
• Every replica runs third-party-datastore, and each works on the shared queue. A user’s queue is taken with a lock in Redis, so only one replica sends or fetches a given user’s changes at a time.
• Every replica’s document-updater writes idle projects in the background. Each project is taken from a shared list in Redis, so it is written once.

Tested on Kubernetes

Against a mock of the Dropbox API, on Kubernetes 1.37 with three replicas behind HAProxy, our end-to-end tests for Dropbox Synchronisation pass (59 cases): linking and unlinking; sending projects, edits, binary files, renames, moves and deletions to Dropbox, including an editor edit that arrives through the background write after the project was closed, without compiling or “sync now”; applying changes made in Dropbox through the webhook (verification challenge and signature check), new projects from new folders, deleted project folders; projects with the same name; the editor’s Dropbox dialog and “sync now” with its rate limit; retries, rate limits and timeouts of Dropbox; accounts unlinked when Dropbox is full, refuses the token or is suspended; suspended Overleaf accounts; and the webhook’s answer when third-party-datastore is down.