A 60-line deploy script that rolls itself back

Our web app runs on one server. No Kubernetes, no blue-green fleet, no deploy service. A cron job on the box runs a shell script that pulls main, restarts the app if the code changed and goes back to sleep.

For a while that script was ten lines long, and it worked until the day it did not. One push broke startup, the app never came back, and the site was down until a human noticed. This post is about the version we have now: still one Bash file, about sixty lines, but it checks its own work and undoes it.

Why pull and not push

A push deploy needs a CI runner with credentials to the server. We had one, and it broke in a way nobody had time to debug. A pull deploy needs nothing from outside: the server already has read access to the repository, so it can fetch on its own schedule.

The trade-off is that deploys are not instant. A merged commit goes live on the next cron tick. For a small team that is fine, and it removes a whole class of "the pipeline is red, is production broken?" questions.

The naive version

The first script did roughly this:

cd /srv/app
git fetch origin
if [ "$(git rev-parse HEAD)" != "$(git rev-parse origin/main)" ]; then
  git reset --hard origin/main
  systemctl restart app.service
fi

Three problems showed up in practice.

It restarted the app for every commit, including commits that only changed Markdown articles and JSON logs. Most of our pushes are content, so most restarts were pointless.

It had no idea whether the restart worked. systemctl restart returns as soon as the process is started, not when it is ready to serve.

And it quietly corrupted the database. More on that below, because it took the longest to find.

Restart only when code changed

The fix for the first problem is a diff between the two commits and a short allow-list of paths that actually affect the running process:

CHANGED=$(git diff --name-only "$LOCAL" "$REMOTE")
git reset --hard origin/main

if ! echo "$CHANGED" | grep -qE '^(server/|requirements\.txt|deploy/)'; then
  echo "$(now) deployed (content only) -> $REMOTE"
  exit 0
fi
echo "$CHANGED" | grep -q '^requirements\.txt' && pip install -q -r requirements.txt
restart_all

Content files are read from disk on each request, so updating the checkout is enough. Code changes get a restart, and a change to dependencies also gets a pip install first.

We also refuse to move backwards. If the local HEAD is ahead of origin/main (someone shipped a hotfix by hand), git merge-base --is-ancestor tells us so and the script exits without touching anything.

A health check that means something

To know whether a restart worked, you need an endpoint that fails when the app is broken. A static file does not qualify: the proxy can serve it while the application is dead.

Our /healthz is a deliberately boring synchronous handler:

@app.get("/healthz", include_in_schema=False)
def healthz(db: Session = Depends(db_session)):
    db.execute(text("select 1"))
    return PlainTextResponse("ok", headers={"Cache-Control": "no-store"})

Because it is a sync endpoint in FastAPI, it runs in the thread pool. It answers only if the event loop is alive, a worker thread is free and the database hands out a connection. select 1 touches no tables, so a long write lock elsewhere does not make the probe fail.

The deploy script polls it after the restart:

probe()   { curl -s -m 8 -o /dev/null -w '%{http_code}' "$HEALTH" || echo 000; }
healthy() {
  for i in $(seq 1 18); do
    sleep 5
    [ "$(probe)" = "200" ] && return 0
  done
  return 1
}

Eighteen tries five seconds apart gives the app 90 seconds to come up. Our normal startup takes a fraction of that, so 90 seconds without a 200 means something is genuinely wrong.

Rolling back

If the probe never succeeds, the script goes back to the commit it was running before and restarts again:

if healthy; then
  echo "$(now) deployed+restart -> $REMOTE"; rm -f "$BAD"; exit 0
fi

echo "$REMOTE" > "$BAD"                 # quarantine the broken commit
git reset --hard "$LOCAL"
restart_all
sleep 20
echo "$(now) ROLLBACK: $REMOTE failed health check, back on $LOCAL, healthz=$(probe)"
notify "Deploy ${REMOTE:0:8} did not come up. Rolled back to ${LOCAL:0:8}."
exit 1

$LOCAL was captured before the reset, so it is always the last version that was actually serving traffic. The notification goes out through the same helper our process watchdog uses, so a failed deploy lands in the same chat as any other incident.

The quarantine file

Without one more line, the rollback loops. On the next cron tick the script sees that origin/main is ahead again, deploys the same broken commit, waits 90 seconds, rolls back, and repeats forever. The site flaps between up and down.

So the bad SHA goes into a small file, and the script checks it before doing anything:

if [ -f "$BAD" ] && [ "$(cat "$BAD")" = "$REMOTE" ]; then
  exit 0      # same broken commit, wait for a new one
fi

The quarantine is keyed to one exact commit. As soon as someone pushes a fix, origin/main points somewhere new, the check passes and the deploy proceeds normally. A successful deploy deletes the file.

Two runs at once

A cron job that can take two minutes will eventually overlap with itself. Two copies doing git reset and systemctl restart at the same time is a good way to roll back the wrong commit.

flock on a file descriptor solves it in two lines:

exec 9>/run/app-deploy.lock
flock -n 9 || exit 0

-n means "do not wait": if the previous run is still polling the health check, the new one simply exits. The lock is released automatically when the process ends, even if it crashes.

The bug that corrupted the database

The original script also backed up the SQLite database before each deploy, with a plain cp, and copied it back after the reset "just in case". Several deploys in one day were followed by database disk image is malformed errors.

The database runs in WAL mode. Recent writes live in a separate -wal file until a checkpoint moves them into the main file. A cp of the main file alone captures an inconsistent snapshot. Copying that snapshot back over a file the app still had open was worse: it could undo a checkpoint under a live connection.

The fix was to stop treating the database as a file. The database is in .gitignore, so git reset --hard never touches it, and the restore step was simply wrong. The backup now goes through SQLite's own online backup API:

python -c "import sqlite3
src = sqlite3.connect('data/app.db')
dst = sqlite3.connect('/var/backups/app.db.bak')
src.backup(dst); dst.close(); src.close()"
git reset --hard origin/main
[ -s data/app.db ] || cp -f /var/backups/app.db.bak data/app.db

Connection.backup copies pages through SQLite itself, so it sees committed data from both the main file and the WAL and produces a consistent copy while the app keeps writing. The copy is only restored if the live file has somehow disappeared.

Restarts without 502s

The last piece is not in the script but makes the restart cheap. The listening port belongs to a systemd socket unit, and the app server receives the already-open socket as a file descriptor instead of binding the port itself.

While the app restarts, the socket stays open and new connections wait in the kernel's backlog. The reverse proxy never sees "connection refused", so visitors get a slow response instead of a 502.

What we learned

• A deploy is not done when the process starts. It is done when a health check that touches the real dependencies returns 200.
• Remember the last good commit before you change anything. Rollback is then one git reset away.
• Any automatic retry needs a memory of what already failed, or it turns a single bad commit into a flapping site.
• Never copy a live SQLite database in WAL mode with cp. Use the backup API or VACUUM INTO.
• Diff before you restart. Most commits in a content-heavy repository do not need a new process.

We use this at SpinHire, a job board for the iGaming industry (https://spinhire.io).