The RPI5-NAS has twice lost IP connectivity, apparently due to a routing-table issue at restart/boot. Each time, the only fix was a physical restart — not viable without physical access to the cluster.
A bash watchdog script running on a systemd timer that checks connectivity, logs diagnostic evidence on failure, attempts a soft fix, and escalates to a reboot if the soft fix doesn't work. Status: Deployed & running
| Stage | Trigger | Action |
|---|---|---|
| Check | Every 2 min (+ 90s after boot) | Ping gateway / secondary target |
| Diagnose | Any failed check | Log ip route, ip addr, journal, dmesg to a timestamped file |
| Soft fix | 2 consecutive failures | Restart NetworkManager/systemd-networkd, or bounce interface + renew DHCP |
| Hard fix | 4 consecutive failures | Reboot (capped at 2/hour to avoid loops) |
/usr/local/bin/nas-watchdog.sh — the watchdog script/etc/systemd/system/nas-watchdog.service — one-shot service unit/etc/systemd/system/nas-watchdog.timer — timer (2 min interval, 90s after boot)/var/log/nas-watchdog/watchdog.log — event log (OK / failure / soft fix / reboot)/var/log/nas-watchdog/diagnostics/failure_*.log — full diagnostic snapshot per failure/etc/logrotate.d/nas-watchdog — rotation for both log locationssystemctl list-timers nas-watchdog.timer # confirm scheduled sudo systemctl status nas-watchdog.service # last run result tail -f /var/log/nas-watchdog/watchdog.log # live event log sudo /usr/local/bin/nas-watchdog.sh # manual test run