# A 60-line deploy script that rolls itself back Our web app runs on one server. No Kubernetes, no blue-green fleet, no deploy service. A cron job on the box runs a shell script that pulls `main`, restarts the app if the code changed and goes back to sleep. For a while that script was ten lines long, and it worked until the day it did not. One push broke startup, the app never came back, and the site was down until a human noticed. This post is about the version we have now: still one Bash file, about sixty lines, but it checks its own work and undoes it. ## Why pull and not push A push deploy needs a CI runner with credentials to the server. We had one, and it broke in a way nobody had time to debug. A pull deploy needs nothing from outside: the server already has read access to the repository, so it can fetch on its own schedule. The trade-off is that deploys are not instant. A merged commit goes live on the next cron tick. For a small team that is fine, and it removes a whole class of "the pipeline is red, is production broken?" questions. ## The naive version The first script did roughly this: ```bash cd /srv/app git fetch origin if [ "$(git rev-parse HEAD)" != "$(git rev-parse origin/main)" ]; then git reset --hard origin/main systemctl restart app.service fi ``` Three problems showed up in practice. It restarted the app for every commit, including commits that only changed Markdown articles and JSON logs. Most of our pushes are content, so most restarts were pointless. It had no idea whether the restart worked. `systemctl restart` returns as soon as the process is started, not when it is ready to serve. And it quietly corrupted the database. More on that below, because it took the longest to find. ## Restart only when code changed The fix for the first problem is a diff between the two commits and a short allow-list of paths that actually affect the running process: ```bash CHANGED=$(git diff --name-only "$LOCAL" "$REMOTE") git reset --hard origin/main if ! echo "$CHANGED" | grep -qE '^(server/|requirements\.txt|deploy/)'; then echo "$(now) deployed (content only) -> $REMOTE" exit 0 fi echo "$CHANGED" | grep -q '^requirements\.txt' && pip install -q -r requirements.txt restart_all ``` Content files are read from disk on each request, so updating the checkout is enough. Code changes get a restart, and a change to dependencies also gets a `pip install` first. We also refuse to move backwards. If the local `HEAD` is ahead of `origin/main` (someone shipped a hotfix by hand), `git merge-base --is-ancestor` tells us so and the script exits without touching anything. ## A health check that means something To know whether a restart worked, you need an endpoint that fails when the app is broken. A static file does not qualify: the proxy can serve it while the application is dead. Our `/healthz` is a deliberately boring synchronous handler: ```python @app.get("/healthz", include_in_schema=False) def healthz(db: Session = Depends(db_session)): db.execute(text("select 1")) return PlainTextResponse("ok", headers={"Cache-Control": "no-store"}) ``` Because it is a sync endpoint in FastAPI, it runs in the thread pool. It answers only if the event loop is alive, a worker thread is free and the database hands out a connection. `select 1` touches no tables, so a long write lock elsewhere does not make the probe fail. The deploy script polls it after the restart: ```bash probe() { curl -s -m 8 -o /dev/null -w '%{http_code}' "$HEALTH" || echo 000; } healthy() { for i in $(seq 1 18); do sleep 5 [ "$(probe)" = "200" ] && return 0 done return 1 } ``` Eighteen tries five seconds apart gives the app 90 seconds to come up. Our normal startup takes a fraction of that, so 90 seconds without a 200 means something is genuinely wrong. ## Rolling back If the probe never succeeds, the script goes back to the commit it was running before and restarts again: ```bash if healthy; then echo "$(now) deployed+restart -> $REMOTE"; rm -f "$BAD"; exit 0 fi echo "$REMOTE" > "$BAD" # quarantine the broken commit git reset --hard "$LOCAL" restart_all sleep 20 echo "$(now) ROLLBACK: $REMOTE failed health check, back on $LOCAL, healthz=$(probe)" notify "Deploy ${REMOTE:0:8} did not come up. Rolled back to ${LOCAL:0:8}." exit 1 ``` `$LOCAL` was captured before the reset, so it is always the last version that was actually serving traffic. The notification goes out through the same helper our process watchdog uses, so a failed deploy lands in the same chat as any other incident. ## The quarantine file Without one more line, the rollback loops. On the next cron tick the script sees that `origin/main` is ahead again, deploys the same broken commit, waits 90 seconds, rolls back, and repeats forever. The site flaps between up and down. So the bad SHA goes into a small file, and the script checks it before doing anything: ```bash if [ -f "$BAD" ] && [ "$(cat "$BAD")" = "$REMOTE" ]; then exit 0 # same broken commit, wait for a new one fi ``` The quarantine is keyed to one exact commit. As soon as someone pushes a fix, `origin/main` points somewhere new, the check passes and the deploy proceeds normally. A successful deploy deletes the file. ## Two runs at once A cron job that can take two minutes will eventually overlap with itself. Two copies doing `git reset` and `systemctl restart` at the same time is a good way to roll back the wrong commit. `flock` on a file descriptor solves it in two lines: ```bash exec 9>/run/app-deploy.lock flock -n 9 || exit 0 ``` `-n` means "do not wait": if the previous run is still polling the health check, the new one simply exits. The lock is released automatically when the process ends, even if it crashes. ## The bug that corrupted the database The original script also backed up the SQLite database before each deploy, with a plain `cp`, and copied it back after the reset "just in case". Several deploys in one day were followed by `database disk image is malformed` errors. The database runs in WAL mode. Recent writes live in a separate `-wal` file until a checkpoint moves them into the main file. A `cp` of the main file alone captures an inconsistent snapshot. Copying that snapshot back over a file the app still had open was worse: it could undo a checkpoint under a live connection. The fix was to stop treating the database as a file. The database is in `.gitignore`, so `git reset --hard` never touches it, and the restore step was simply wrong. The backup now goes through SQLite's own online backup API: ```bash python -c "import sqlite3 src = sqlite3.connect('data/app.db') dst = sqlite3.connect('/var/backups/app.db.bak') src.backup(dst); dst.close(); src.close()" git reset --hard origin/main [ -s data/app.db ] || cp -f /var/backups/app.db.bak data/app.db ``` `Connection.backup` copies pages through SQLite itself, so it sees committed data from both the main file and the WAL and produces a consistent copy while the app keeps writing. The copy is only restored if the live file has somehow disappeared. ## Restarts without 502s The last piece is not in the script but makes the restart cheap. The listening port belongs to a systemd socket unit, and the app server receives the already-open socket as a file descriptor instead of binding the port itself. While the app restarts, the socket stays open and new connections wait in the kernel's backlog. The reverse proxy never sees "connection refused", so visitors get a slow response instead of a 502. ## What we learned - A deploy is not done when the process starts. It is done when a health check that touches the real dependencies returns 200. - Remember the last good commit before you change anything. Rollback is then one `git reset` away. - Any automatic retry needs a memory of what already failed, or it turns a single bad commit into a flapping site. - Never copy a live SQLite database in WAL mode with `cp`. Use the backup API or `VACUUM INTO`. - Diff before you restart. Most commits in a content-heavy repository do not need a new process. We use this at SpinHire, a job board for the iGaming industry ().