How to automatize your backups with Systemd cronjobs, with logs and health-checks

Hello everyone! Today I wan't to share with you all how I set up my backup scripts/tasks in Linux, with the help of Systemd. As most of you might know if you have more than a few months on using Linux, Systemd provides a system and service manager that runs as PID 1 and starts the rest of the system. Is the standard in most Linux distributions out there, including Debian based ones (the most used on servers worldwide).

We can take advantage of its capabilities to implement robust cron jobs that run our backup tasks or scripts, and we will also get out of the box logs management via journalctl. Let's do it!

A working duo: service and timer

We need two systemd pieces to make this work: a systemd service, that will be the one in charge of executing the task or script, and a systemd timer, that is the one triggering the execution of the service on a schedule. Let's take a quick look to both of them and I'll comment them later. As an example, during the rest of the article, I will use a simple rsync command as a backup tool.

# File under /etc/systemd/system/rsync-backup.service
[Unit]
Description=Rsync backup
After=network.target

[Service]
Type=oneshot
ExecStart=/root/.local/backup.sh

# File under /etc/systemd/system/rsync-backup.timer
[Unit]
Description=Timer to make a rsync backup every Monday

[Timer]
OnCalendar=Mon 02:30:00
Persistent=true

[Install]
WantedBy=timers.target

So we defined our service rsync-backup.service that executes the actual backup script (/root/.local/backup.sh), and a timer rsync-backup.timer that ensures the service is executed every Monday at 2:30 AM. To enable the timer, we must execute the following commands:

systemctl daemon-reload
systemctl enable --now rsync-backup.timer # Chose your real timer name here!

Now, the timer should be visible if we execute systemctl list-timers and the end of the table shown by that command.

If you want to manually run the backup at an arbitrary time, you could just execute the actual backup script under /root/.local/backup.sh (in this particular example, but I recommend you to do systemctl start rsync-backup.service instead, so this way, we get the log management with journalctl out of the box.

Finally, note that the service type is oneshot. This means, the service will execute the script under ExecStart and finish once the script exits. During that time, if you check the service with systemctl status it will show as "activating". You might be tempted to use simple type instead of oneshot. The differences are mostly on parallel executions when multiple ExecStart are added, with doesn't matter for our use case, but here is a nice explanation from Stack Overflow in case you are curious.

Adding health-checks

How do we add health checks then? I use Healthchecks.io for that, which is a simple and free to use (up to 20 jobs) service that works like a gem. I'm in love with it. How it works is simple: they provide an endpoint you must hit with curl or whatever to indicate your job executed. You can even provide more info like how long did the execution took, or even any possible exit error code.

I won't guide on how setup the check in Healthchecks.io itself, as I find the tool very straightforward and they have their own support articles in case you need them. So let's jump directly to an example of a health check configured there for our backup script.

Click on image to enlarge

A quick explanation on the configuration shown on the image:

• We defined the name of the task in Healthchecks.io as "Rsync-backup". The check mark is green because I already did my first health check while testing my script.
• The ping URL is the one we must hit with curl. It's a random UUID they will provide to you.
• Now, in the integrations, is where we setup how we want to be notified on errors. I earlier configured it to notify me via email and telegram.
• Then, we set when the cron is expected to run, and a grace time period. In this example, I indicated the cron must run Mondays at 2:30 AM (30 2 * * 1 in crontab language), and that it should not last more than 30 minutes from that time to ping. Therefore, if 30 minutes past 2:30 AM on Monday a ping has not being received, we'll be notified on our configured channels.
• Finally, there are some quick metric about the last ping (9 hours ago in my case) and how long did the script execution took (7 minutes, more on this next). This will be populated after your first ping.

Nice! So we configured Healthchecks.io and enabled our first health check. How do we integrate it in our backup task? If you remember, we were using the systemd service to execute a backup script /root/.local/backup.sh. Let's take a look to it:

#! /usr/bin/env bash

HEALTHCHECKS_URL=https://hc-ping.com/1a2fdfaf-cbf7-4c37-8d87-e7e013e27e3b

curl -fsS -m 10 --retry 5 -o /dev/null $HEALTHCHECKS_URL/start
rsync -aAHXv --delete /etc /mnt/backup
curl -fsS -m 10 --retry 5 -o /dev/null $HEALTHCHECKS_URL/$?

A simple script as you can see. What this does is hitting our health check endpoint with start path param, which indicates Healthchecks.io our execution started. This is useful for the duration metrics, but also for resetting the grace time period. Then, the actual backup task is executed: in this example, a simple rsync command. Finally, we hit our health check endpoint again, this time sending the exit code of the rsync command as path param. This way, we'll register a successful execution if sending a value of zero, and we'll register (and be notified) of an error if sending something different.

Note: don't forget to provide execution permissions to your script with chmod +x 😉

Checking logs after executions

Did the task failed and you want to know what happened? Easy enough! With systemd we get logs management out of the box. Just run journalctl -u rsync-backup.service (or whatever name your backup service has) and all your logs will be there to be checked!

What's next?

Due to Systemd strong capabilities, we could expand and improve the above service and timer, like adding retries in case of errors. There is so much things we can do with Systemd, so as always on Linux, hack and experiment to find your way!