The backup that said “success” for 14 nights while failing

backups-that-report-failure — The backup that said “success” for 14 nights while failing

A nightly backup reported success while its off-site copy was failing. The fix: a job that fails loudly, so the failure alert fires.

My own lab in the Fraser Valley. A nightly job backs up databases and settings and copies them to a second server. It had reported success every night.

• Found the cause: the script wrote each error to its log and then carried on, so it always ended with “success” and the failure alert never fired.
• The script now records any failure and exits with an error, so the failure alert fires.
• Fixed the off-site copy’s destination path, which the restricted copy account had been silently rewriting.
• Fixed the file exclusions, which were in the wrong order, and replaced a hard-coded database list that missed new databases.
• A failed run sends a push notification to my phone.

• Checked the off-site copy carried the new night’s date after the next run.

• bash
• rsync
• systemd timers
• pg_dump
• push alerts