Even when I do my best to ensure content is correctly rendered on RSS readers, some elements like custom info cards/notes and code style blocks can be shown without proper style or syntax highlighting.

You are free to continue reading as you prefer, but I recommend you to read this article in the website if possible. It's fast, has no ads, and minimal privacy-respecting analytics (browser, OS, country and device type) via Umami.

If you don’t care about the context, you can jump directly to the final solution.

A couple of weeks ago I started noticing something very annoying in my homelab. My mini PC running Proxmox VE, which is connected to the same UPS as my Synology NAS, was force-shutting down every single time there was a brief power blip, even when the battery was still at 99%. To make things worse, once the mini PC went down, it wouldn’t come back up. As soon as I rebooted it, it would shut itself down again within seconds, and the only way to stop the loop was to physically unplug its ethernet cable and boot it in isolation from the network. Not exactly what I want from a headless box in a rack.

In this article I’ll walk you through how I debugged this, what the actual root cause was so you won’t lose your time (spoiler: it’s because the custom UPS server Synology DSM uses), and how I ended up replacing nut-monitor with a tiny custom watchdog script that finally made my homelab behave the way I wanted during power outages. Let’s dive in!

My setup

First let me describe my setup to let you understand my context:

• I have a Synology NAS DS920+ with a CyberPower UPS physically connected to it over USB. Synology DSM runs its own NUT server (upsd) and acts as the primary in the NUT topology.
• Then I have a Beelink Ryzen mini PC running Proxmox VE on the same LAN and plugged into the same UPS. It was running nut-monitor as a slave (or secondary, in newer NUT terminology), configured to read status from the Synology’s UPS server over the network.

The relevant MONITOR line in /etc/nut/upsmon.conf on the mini PC looked like this (the IP of the Synology is 192.168.1.7):

MONITOR ups@192.168.1.7 1 monuser secret slave

This is a very common homelab layout: one machine owns the UPS physically, others watch it over the network. It’s also the officially recommended pattern, so in theory everything should just work.

The symptom

The bug was as follows. Whenever there was a 2 second power blip at home (someone plugging in a high-draw appliance, for example), my Synology would keep running as if nothing happened (its battery threshold for shutdown is set to When battery is low, and we never got close to that), but my mini PC would immediately shut itself off. Checking journalctl on the mini PC afterwards, I’d see entries from nut-monitor saying:

ups@192.168.1.7: forced shutdown in progress

And as I mentioned, the most annoying part was that rebooting didn’t fix it. Every time I brought the mini PC back up, nut-monitor would reconnect to the Synology UPS server, then shut down again within seconds. The only workaround I had was to unplug the ethernet cable, boot the machine, log in, and then plug the ethernet back in. Not a solution for a headless server.

What the forced shutdown actually means

That forced shutdown in progress message is very specific. In NUT, every UPS has a ups.status field that can contain a set of flags. The ones that matter for this article are:

• OL (On Line): UPS is running on mains power.
• OB (On Battery): UPS is running on battery because mains power is out.
• LB (Low Battery): battery is critically low and the UPS is about to die.
• FSD (Forced Shutdown): the UPS server is telling all its slaves to shut down right now, regardless of battery level.

When nut-monitor on a slave sees FSD in the UPS status, it triggers a local shutdown. That’s by design, and it’s meant to coordinate graceful shutdowns between the primary and its slaves during a real power outage.

So the question became: why was the Synology broadcasting FSD on a two-second power blip, when it wasn’t shutting itself down?

Discovering the root cause

Disabled the ups-monitor service on the mini PC (to avoid it shutting down as soon as it connects) and plugged back in the ethernet cable again. Then, I took a look at the UPS status:

upsc ups@192.168.1.7

I could still see ups.status: FSD OL hours after the power outage. OL means the UPS is back on mains already, but FSD was still set. That’s when it clicked: Synology DSM’s NUT implementation is more aggressive than a vanilla upsd. On any power event, it asserts FSD to its slaves almost immediately, before even evaluating its own battery threshold. And it takes a while to clear the flag afterwards, which also explains the post-reboot shutdown loop: my mini PC would come up, see the leftover FSD on the UPS status, and shut itself down again.

This is a known quirk of Synology’s NUT server, discussed in several homelab forums, and it’s not something you can turn off in DSM.

Why NOTIFYFLAG FSD ... does not help

The first thing I tried was to look for a way to tell nut-monitor to just ignore FSD. NUT has a NOTIFYFLAG mechanism in /etc/nut/upsmon.conf that lets you configure what happens on each event, and also a NOTIFYCMD hook to run your own script:

NOTIFYFLAG FSD     SYSLOG+WALL
NOTIFYFLAG ONBATT  SYSLOG+WALL+EXEC
NOTIFYFLAG LOWBATT SYSLOG+WALL+EXEC
NOTIFYFLAG ONLINE  SYSLOG+WALL+EXEC
NOTIFYCMD "/usr/local/bin/nut-notify.sh"

So I wrote a dummy nut-notify.sh that literally did echo "This is a test" and nothing else, just to test the script was actually ran when it should. But it didn’t work. The mini PC still shut down on FSD. Why?

Because upsmon has FSD handling hardcoded. NOTIFYFLAG and NOTIFYCMD are a notification hook, not a shutdown interceptor. Internally, when upsmon sees FSD in the UPS status it runs its own shutdown sequence immediately, completely bypassing any custom command you may have plugged in. There is no way to turn this off, short of patching NUT itself.

At that point it was clear I couldn’t make nut-monitor play nice with my Synology. I needed to replace it with something I fully control.

The solution: a small custom watchdog script

The plan was simple: disable nut-monitor entirely, and write a bash script that polls the UPS status every 30 seconds via upsc, and only triggers a shutdown if the conditions I actually care about are met. Specifically:

• The UPS reports OB and battery charge is at or below 30% (my own threshold, independent of what the UPS thinks).
• Or as a safety net:
  • The UPS reports OB LB (on battery, low battery), which is a legitimate low battery situation.
  • The connection with the Synology UPS server was lost for a while.

The FSD flag is ignored. Only if at least one of the three above conditions is meet the mini PC will shut down. This is the key. And to handle real outages where the Synology itself or my network goes offline, the script also counts consecutive communication failures and shuts down if the UPS server stays unreachable for too long.

So let’s proceed with this solution. First, write a watchdog script to /usr/local/bin/ups-watchdog.sh:

#!/bin/bash

UPS="ups@192.168.1.7"
MIN_BATTERY=30
CHECK_INTERVAL=30
# The Synology NAS takes 5-6 min to reboot, so that's why I set 15 attempts prior to shutdown, which equals
# to 7.5 min at a polling ratio of 30s. I don't want the mini pc to shutdown just because DSM is updating.
COMMS_LOST_THRESHOLD=15

comms_failures=0

while true; do
    UPS_DATA=$(timeout 10 upsc "$UPS" 2>/dev/null)
    STATUS=$(echo "$UPS_DATA" | grep "^ups.status:" | awk '{print $2}')
    CHARGE=$(echo "$UPS_DATA" | grep "^battery.charge:" | awk '{print $2}')

    if [[ -z "$STATUS" ]]; then
        comms_failures=$((comms_failures + 1))
        echo "UPS watchdog: cannot reach UPS server (failure $comms_failures/$COMMS_LOST_THRESHOLD)"

        if [[ "$comms_failures" -ge "$COMMS_LOST_THRESHOLD" ]]; then
            echo "UPS watchdog: UPS server unreachable for too long, shutting down as precaution"
            /sbin/shutdown -h now "UPS server unreachable"
        fi

        sleep "$CHECK_INTERVAL"
        continue
    fi

    comms_failures=0
    echo "UPS watchdog: status=$STATUS charge=$CHARGE%"

    # Shut down on LB flag from UPS
    if [[ "$STATUS" == *"OB"* ]] && [[ "$STATUS" == *"LB"* ]]; then
        echo "UPS watchdog: LB flag set on battery, shutting down"
        /sbin/shutdown -h now "UPS low battery"
    # Shut down on charge threshold while on battery
    elif [[ "$STATUS" == *"OB"* ]] && [[ -n "$CHARGE" ]] && [[ "$CHARGE" -le "$MIN_BATTERY" ]]; then
        echo "UPS watchdog: battery at ${CHARGE}% on battery power, shutting down"
        /sbin/shutdown -h now "UPS battery critical"
    fi

    sleep "$CHECK_INTERVAL"
done

Note that the script uses OB and LB as shutdown triggers but never FSD. That’s intentional. FSD being asserted on its own is not a reason for my mini PC to die, and since we got rid of nut-monitor, nothing on this machine will ever react to that flag again.

Okay, once we have it, make it executable:

sudo chmod +x /usr/local/bin/ups-watchdog.sh

And wrap it in a systemd service at /etc/systemd/system/ups-watchdog.service:

[Unit]
Description=UPS Battery Watchdog
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/ups-watchdog.sh
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

Finally, stop and disable nut-monitor, mask it to prevent it from being started from package updates, and then enable and start the new custom watchdog script:

sudo systemctl stop nut-monitor
sudo systemctl disable nut-monitor
sudo systemctl mask nut-monitor
sudo systemctl daemon-reload
sudo systemctl enable ups-watchdog
sudo systemctl start ups-watchdog

Note: if you also have a nut-client service running, you can also stop and disable it as well since it’s not needed anymore:

sudo systemctl stop nut-client
sudo systemctl disable nut-client
sudo systemctl mask nut-client

Verifying it works

After deploying the script, I did two tests to make sure the new setup behaved correctly.

Test 1: brief power outage. I pulled the UPS plug from the wall for about 40 seconds and then plugged it back in. Before, this would instantly kill the mini PC. Now, the logs from journalctl for the script shows the OB status briefly, then goes back to OL, and the mini PC stays happily up:

UPS watchdog: status=OL charge=100%
UPS watchdog: status=OB charge=99%
UPS watchdog: status=OL charge=99%

Test 2: Synology reboot. I triggered a reboot on the Synology to simulate a DSM update. The mini PC lost contact with the UPS server for about 5 minutes, counted 10 consecutive failures, and then got its connection back, all without shutting itself down:

UPS watchdog: status=OL charge=100%
UPS watchdog: cannot reach UPS server (failure 1/15)
UPS watchdog: cannot reach UPS server (failure 2/15)
...
UPS watchdog: cannot reach UPS server (failure 10/15)
UPS watchdog: status=OL charge=100%

Which was exactly the behavior I wanted.

Wrapping up

And that’s the full story of how I finally stopped my Proxmox mini PC from randomly shutting itself down on every power blip. The lesson I take from this is that nut-monitor is a great tool for well-behaved UPS topologies, but when your primary is a Synology (with its aggressive FSD behavior), the hardcoded slave shutdown logic gets in the way. In that case, replacing it with a small polling script you fully control is, in my opinion, the simplest and most reliable solution. A bit of bash goes a long way.

If you run a similar homelab setup and have been fighting the same battle, I hope this saves you a few hours of head-scratching. Until next time!

Even when I do my best to ensure content is correctly rendered on RSS readers, some elements like custom info cards/notes and code style blocks can be shown without proper style or syntax highlighting.

You are free to continue reading as you prefer, but I recommend you to read this article in the website if possible. It's fast, has no ads, and minimal privacy-respecting analytics (browser, OS, country and device type) via Umami.

I have a Linux gaming PC running Bazzite that I use exclusively for gaming. It sits in a corner of my house with just the power and the ethernet cable, and I connect to it remotely from my TV, my laptop, my iPad, or whatever device is closest to me at the moment. No monitor, no keyboard, no mouse attached to it. Just a box that boots up, waits for me, and streams games over my local network.

Getting to this point was not complicated, but it required a few pieces to come together: a streaming server, a client app, a dummy HDMI plug, and a way to log in without a screen. In this article, I’ll walk you through how I set it all up, so you can do the same with your own PC.

How is this possible?

I use two different pieces of free open-source software to make this magic work. Sunshine and Moonlight.

Sunshine is an open-source, self-hosted game streaming server. It runs on your gaming PC and captures whatever is on the screen (your desktop, a game, etc.) encodes it using your GPU’s hardware encoder, and streams it over the network with very low latency. It supports AMD, Intel, and NVIDIA GPUs, and you can configure it through a web interface.

Moonlight is an open-source game streaming client that connects to Sunshine. It originally started as an implementation of NVIDIA’s GameStream protocol, but with Sunshine as the host, it works with any GPU. Moonlight is available on pretty much every platform you can think of: Windows, macOS, Linux, Android, iOS, Apple TV, and even some smart TVs. It supports up to 4K resolution with HDR and up to 120 FPS on compatible clients.

The idea is simple: Sunshine runs on your gaming PC (the server), Moonlight runs on the device you want to play from (the client), and they talk to each other over your LAN. The result is that you can use your PC and play your PC games from your couch, your bed, or anywhere in your house, with a controller in your hands (or mouse and keyboard if you want) and a screen in front of you that is not your PC monitor.

Setting up Sunshine on Bazzite

Bazzite makes trivially easy to set up sunshine. It comes pre-installed, so you just need to enable it. Bazzite uses ujust as a command runner for common system tasks, and setting up Sunshine is one of them:

ujust setup-sunshine

This will open an interactive menu where you can choose to enable or disable the Sunshine service. Select Enable, and Sunshine will start as a systemd daemon that launches automatically on every boot.

Once enabled, you can access the Sunshine web interface with the same ujust command, but selecting the option Open Portal, or just going to https://localhost:47990 from a browser on the same machine (or via Moonlight later). The first time you open it, you’ll be asked to set up a username and password for the Sunshine admin panel. After that, you can configure your streaming settings from there, like encoder type, bitrate, and resolution. Personally, I didn’t configure anything here and left all defaults untouched.

I also recommend setting a static local IP address in your Bazzite PC. Although it is not necessary, it might simplify things later, like in case your PC doesn’t get auto-discovered by the Moonlight clients. There are multiple ways to setup this, including using the UI like going to System Settings in your desktop environment. All options are valid, and also out of scope for this article, as explaining them would make the article too large. You can always make a quick google search if you want.

Setting up Wake on LAN on Bazzite

I highly recommend also enabling Wake on Lan (WoL) on your Bazzite PC that you will use to stream games. WoL lets you power on your PC remotely by sending a special network packet (aka. “magic packet”) from any device on your LAN. Moonlight clients already has this feature, so you could power on your PC from any moonlight client you are willing to use.

To enable it, your network card needs to support WoL, and it needs to be enabled in the BIOS. Most modern motherboards support it, but the option might be disabled by default. Check your BIOS settings under something like Power Management or Wake Up Event Setup, and enable Wake on LAN or Resume by PCI-E Device.

Once that’s done, Bazzite makes the software side easy with another ujust command:

ujust toggle-wol

This will detect your active Ethernet interface automatically and show you the current status. You’ll get an interactive menu with these options:

• Enable — turns on WoL for the current session and creates a udev rule so it persists across reboots.
• Force-Enable — creates a systemd service that forces WoL on at every boot. Use this if the regular enable doesn’t stick.
• Disable — turns off WoL and removes the persistence rules.

I’d recommend starting with Enable, and if you find that WoL stops working after a reboot, switch to Force-Enable. In my case on a MSI motherboard, it was enough to use just the first option.

Setting up Moonlight on your devices

Moonlight is available on basically every platform. I use it on my iPhone, my iPad, my MacBook, and my XBOX Series X. The setup process is the same everywhere:

1. Install Moonlight go to the official moonlight website to know from where download the client. For phones and tablets, it’s usually the default app store.
2. Open Moonlight, and it will automatically scan your local network for Sunshine hosts. Your Bazzite PC should appear in the list. If it doesn’t appear, introduce the IP of your PC manually.
3. Select your PC, and Moonlight will ask you to enter a PIN. This PIN will also appear on the Sunshine web interface (or on the PC’s screen if one is connected). Enter the PIN in Sunshine to pair the devices.
4. That’s it. Once paired, you can launch your desktop or any game directly from Moonlight.

In Moonlight’s settings, you can adjust the stream resolution, frame rate, bitrate, and video codec. Here are the settings I’d recommend as a starting point:

• Resolution: match your client display or go with 1440p for a good balance.
• Frame rate: 60 FPS is the sweet spot for most setups.
• Video codec: I left this in auto, but if your hardware is fairly old and can only choose between HEVC (H.265) and H.264, chose HEVC (H.265) if your GPU and client support it. It gives better image quality at the same bitrate compared to H.264.
• Bitrate: if your local network is not a bottleneck, push it to 50–80 Mbps for excellent quality. In my case with a good network (ethernet cat. 5e or Wi-fi 6) the default bitrates from Moonlight work nice.

The dummy HDMI plug

Here’s where things get interesting if you want a truly headless setup. When you disconnect the monitor from your PC, the GPU detects that no display is attached and reports no available outputs. On Linux with KDE Wayland (the Desktop Environment I’m using), this means the desktop compositor (KWin) has nothing to render to, so it either fails to start or enters a fallback state. And if there’s no desktop session, Sunshine has nothing to capture and stream.

I found this out the hard way. Everything was working great with the monitor connected, but the moment I unplugged it and tried to connect via Moonlight, it just kept saying the computer was offline. Checking the logs confirmed the issue:

sddm-greeter-qt6: There are no outputs - creating placeholder screen
amdgpu: Cannot find any crtc or sizes

The solution is surprisingly low-tech: a dummy HDMI plug. It’s a small passive dongle (about the size of a USB flash drive) that you plug into your GPU’s HDMI port. Inside, it has a tiny chip with EDID data that tells the GPU “hey, there’s a monitor here.” The GPU believes it, creates a real display output, and everything works as if a monitor were connected.

You can find them on Amazon for about €5–8. Search for “HDMI dummy plug 4K” and pick any well-reviewed one. Make sure it advertises 4K (3840×2160) resolution, so that KDE and Sunshine offer you proper resolution options (some cheaper ones only report 1080p, which would limit your streaming quality).

I went with an HDMI dummy plug specifically because my GPU (an AMD Radeon RX 5700 XT) has HDMI 2.0b, which supports 4K at 60Hz, more than enough for Moonlight streaming. I also wanted to keep the DisplayPort outputs free in case I ever want to connect a real monitor again. But if you prefer DisplayPort, those dummy plugs exist too and work the same way.

Once the dummy plug is in, the headless boot looks exactly like a normal boot with a monitor. KDE starts properly, Sunshine captures the desktop, and Moonlight connects without any issues.

Going fully headless: logging in without a screen

With the dummy plug solving the display problem, there’s one more challenge: how do you get past the login screen without a keyboard and a monitor? Here, you have a few different options, depending on your security concerns, and how much do you want to complicate things.

The easy option: auto-login

The simplest approach is to configure not password for your user, and configure SDDM (KDE’s display manager) to auto-login. You can create a file at /etc/sddm.conf.d/autologin.conf with:

[Autologin]
User=yourusername
Session=plasma
Relogin=true

This makes SDDM log you in automatically at boot. Relogin=true means it will also re-login if the session crashes, which is important for a headless machine that needs to recover on its own.

The downside? As your user has not password, someone with physical access to the machine could access your desktop with just plugin in a monitor and a mouse and keyboard. For many people this is perfectly fine, but I couldn’t leave it like that. Like, what I have the bad luck that someone breaks into my house and steals the whole PC? I have Steam with my credit card there, and also I’m logged into my email there.

The YubiKey option: physical token authentication

I have a YubiKey, a small hardware security key, and it turns out you can use it as a login token for KDE. The idea is that instead of typing a password, you plug the YubiKey into a USB port and SDDM authenticates you automatically via FIDO2/U2F. No typing, no screen needed.

On Bazzite, the pam-u2f package is already installed (you can verify with rpm -q pam-u2f). Setting it up involves two steps:

1. Register your YubiKey:

mkdir -p ~/.config/Yubico
pamu2fcfg -o pam://localhost -i pam://localhost > ~/.config/Yubico/u2f_keys

Touch the key when it blinks. This generates the credential file that PAM will use to verify your identity.

2. Configure PAM for SDDM and the KDE screen locker:

Add the following line before the auth substack password-auth line in both /etc/pam.d/sddm and /etc/pam.d/kde:

auth        sufficient    pam_u2f.so authfile=/home/yourusername/.config/Yubico/u2f_keys cue origin=pam://localhost appid=pam://localhost

The sufficient keyword means: if the YubiKey authenticates successfully, skip the password prompt entirely. If the key isn’t present, fall through to the normal password prompt. You never lose the ability to type your password as a fallback.

With this setup, my boot flow looks like this:

1. Power on the PC (via Wake on LAN or the power button).
2. A wait a minute, as I know it takes 40-50s to my PC to get into login screen.
3. I plug in the YubiKey, I touch it a few times until it blinks → SDDM detects it and logs me in.
4. Sunshine starts capturing the desktop → I connect from Moonlight on any device.

Someone finding the PC would need my YubiKey or my password to get in. And the YubiKey is always with me.

The downside? While the setup is headless, I need to wake up and walk to the computer to plug the YubiKey and touch it to login. With the auto-login option, I can just sit on my sofa or lay on my bed, boot it up remotely with WoL, and start playing once it boots.

Conclusion

And that’s it. With Sunshine on Bazzite, and a Moonlight client on whatever device you want, you can use your gaming PC to stream games anywhere in your LAN. You get a fully silent gaming PC that you can wake up, log into, and play from anywhere in your house. If you want to go a extra step, you could buy a dummy HDMI plug and make the whole setup headless. Do you want more security? Add a YubiKey for secure headless login.

Happy gaming!

Even when I do my best to ensure content is correctly rendered on RSS readers, some elements like custom info cards/notes and code style blocks can be shown without proper style or syntax highlighting.

You are free to continue reading as you prefer, but I recommend you to read this article in the website if possible. It's fast, has no ads, and minimal privacy-respecting analytics (browser, OS, country and device type) via Umami.

This article is a companion to my previous one on how to use Caddy as a reverse proxy for your services. In that article, I set up Caddy so that services are reachable via custom local domains like myservice.home.arpa, but for those domains to actually work, every device on your LAN needs to be able to resolve them — and that’s exactly what we tackle here. In today’s article I’ll explain how I use Pi-hole as a local DNS server to achieve network-wide resolution for those custom domains without touching every device’s hosts file. Let’s dive in!

Why local domains and why .home.arpa?

Before jumping into the setup, let me quickly explain why this is useful and why I use .home.arpa as my local TLD.

When you self-host services, you usually access them by typing their local IP address and port directly in the browser, like http://192.168.1.100:8080. This works, but it’s ugly, hard to remember, and doesn’t play well with HTTPS (certificates are usually tied to domain names, not bare IPs, although I explain how to generate certificates for bare IPs with your own CA in another article). The natural solution is to assign readable domain names to your services, like myservice.home.arpa, which it’s not only easier to remember, but gives you more flexibility if for whatever reason you need to migrate it to a different hardware over a different IP address.

As for the TLD choice: .home.arpa is the officially reserved special-use domain for residential LANs. Using it ensures you will never accidentally conflict with a real public domain now or in the future, which is a real risk if you just invent something like .local or .home. .local in particular is already reserved for mDNS (Bonjour/Avahi), so you must avoid it. Stick with .home.arpa and you’ll be safe.

Installing Pi-hole

Pi-hole is a network-level DNS server and ad-blocker. It’s designed to run on a Raspberry Pi (hence the name), but it will happily run on any Linux machine, including a VM or a Docker container. For this tutorial, I’ll assume you’re installing it on a Debian-based system (Raspberry Pi OS, Debian, Ubuntu…).

Pi-hole provides an official one-liner installer:

curl -sSL https://install.pi-hole.net | bash

The installer is interactive and will guide you through the configuration. A couple of things to keep in mind during the setup:

• When asked for the upstream DNS provider, choose any public resolver you trust (I use Cloudflare’s 1.1.1.1 and 1.0.0.1). Pi-hole will forward queries it can’t answer locally to this provider.
• When asked for the network interface, select the one connected to your LAN.
• Take note of the static IP address your Pi-hole machine will use. The installer will remind you about this, but it’s crucial: if your Pi-hole’s IP changes, everything breaks. Make sure to either set a static IP on the machine itself, or create a DHCP reservation for it on your router.

Once the installer finishes, it will print the URL and credentials to access the Pi-hole web admin panel. Keep them safe!

Adding simple local DNS records

This is the core of what we want to achieve. Pi-hole includes a built-in local DNS resolver that lets you define custom DNS records, mapping domain names to local IPs. Open the Pi-hole web admin panel (by default at http://<pihole-ip>/admin) and log in. Then navigate to Settings → Local DNS Records. You will see a simple form with two fields: Domain and IP Address.

For example, if I have a Caddy reverse proxy running on a machine with IP 192.168.1.100, and I want myservice.home.arpa to point to it, I would fill in:

• Domain: myservice.home.arpa
• IP Address: 192.168.1.100

Click Add and that’s it. The record is immediately active, no restart needed. You can add as many records as you want, one per service or server. In my homelab, for example, I have records for my NAS, my Proxmox nodes, and each of the services I expose through Caddy.

Adding complex local DNS records

But what about wildcard domains? For example, if you want *.home.arpa to resolve to the same IP, so you can have service1.home.arpa, service2.home.arpa, etc without adding a record for each one? Pi-hole’s built-in local DNS doesn’t support wildcards, but there’s a simple workaround: you can use dnsmasq directly, which is the DNS server Pi-hole uses under the hood. Just SSH into your Pi-hole machine and create a new file under /etc/dnsmasq.d/ with a .conf extension, for example /etc/dnsmasq.d/99-homelab.conf, and add the following content:

# Specific subdomain overrides
address=/myservice.home.arpa/192.168.1.100

# Wildcard for everything else
address=/home.arpa/192.168.1.120

With the above configuration, myservice.home.arpa will resolve to 192.168.1.100, while any other subdomain under home.arpa (like service1.home.arpa, nas.home.arpa, etc) will resolve to 192.168.1.120. After saving the file, we need to configure pi-hole to load the custom dnsmasq configuration files, and then restart it so the file is actually loaded. We can do that with the following commands:

sudo pihole-FTL --config misc.etc_dnsmasq_d true
sudo systemctl restart pihole-FTL

Alternatively, you can go to the Pi-hole admin panel, then Settings → Advanced (in the upper right corner) → All settings (back in the lateral menu) → Miscellaneous and checking the checkbox under misc.etc_dnsmasq_d. Then, again, restart Pi-hole to load the new configuration.

Making your devices use Pi-hole as their DNS server

Defining the DNS records is only half the work. The other half is telling your devices to actually query Pi-hole instead of the default DNS server assigned by your ISP. There are two ways to do this.

• Option 1: Configure it on your router (recommended) - This is the best approach because it’s network-wide: every device that connects to your LAN (laptops, phones, smart TVs, IoT devices…) will automatically use Pi-hole as their DNS server without any extra configuration on each device. Every router is different, but the setting you’re looking for is usually under DHCP settings or LAN settings, and is often labeled DNS server or Primary DNS. Set the value to the static IP of your Pi-hole machine.

• Option 2: Configure it per device - If you can’t or don’t want to change your router’s settings, you can configure each device individually to point to Pi-hole. The downside is obvious: you need to repeat this on every device, and any new device that joins your network won’t automatically benefit from it.

  • On macOS, go to System Settings → Network, select your active connection (Wi-Fi or Ethernet), click Details, and under the DNS tab add your Pi-hole’s IP as a DNS server. To avoid breaking internet connection when you are outside of your LAN (for example, if you are using a MacBook on the go), you might want to set up a network location, so that your Mac uses Pi-hole only when connected to your home network.
  • On other systems like Windows and Linux, the process is similar, and very often can be configured directly in the network settings. Just look for the DNS server configuration and set it to your Pi-hole’s IP. An in case you are using a portable device (like a laptop), be sure to configure it to use Pi-hole only when connected to your home network, otherwise you might have DNS resolution issues when outside of your LAN.

Verifying it works

Once you’ve configured at least one device to use Pi-hole as its DNS, it’s time to verify everything works correctly. The quickest way is to use nslookup or dig from a terminal on that device.

For example, to verify that myservice.home.arpa resolves correctly:

nslookup myservice.home.arpa

You should see an output like this:

Server:		192.168.1.10
Address:	192.168.1.10#53

Name:	myservice.home.arpa
Address: 192.168.1.100

Here, 192.168.1.10 is your Pi-hole, and 192.168.1.100 is the IP you configured in the DNS record. If you see that, your local DNS resolution is working! You can also verify it from a browser: open http://myservice.home.arpa and, if the service is running and listening on that IP, it should load. Of course, if you’ve set up TLS with Caddy and your own CA as I described in my previous articles, you can use https://myservice.home.arpa and get a green padlock too.

Bonus: Using your own certificate to secure pi-hole admin panel with HTTPS

If you also want to use your own self-signed certificate from your own CA to secure the Pi-hole admin panel with HTTPS, you can jus follow the instructions on my CA article to generate a certificate for the Pi-hole. Then, once you have your separated server.key and server.crt files, you can just merge them into a pem file with cat server.key server.crt > pihole.pem, and copy that pihole.pem file into the Pi-hole machine under /etc/pihole/pihole.pem. Then, ensure the files permissions are correct:

# In the pihole machine
cd /etc/pihole
sudo chown pihole:pihole pihole.pem
sudo chmod 600 pihole.pem

Then, configure Pi-hole to use that certificate by editing the /etc/pihole/pihole.toml file

  [webserver.tls]
    # Path to the TLS (SSL) certificate file.
    #
    # All directories along the path must be readable and accessible by the user running
    # FTL (typically 'pihole'). This option is only required when at least one of
    # webserver.port is TLS. The file must be in PEM format, and it must have both,
    # private key and certificate (the *.pem file created must contain a 'CERTIFICATE'
    # section as well as a 'RSA PRIVATE KEY' section).
    #
    # The *.pem file can be created using `cp server.crt server.pem && cat server.key >>
    # server.pem` if you have these files instead
    #
    # Allowed values are:
    #     A valid TLS certificate file (*.pem)
-   cert = "/etc/pihole/tls.pem"
+   cert = "/etc/pihole/pihole.pem"

And finally, restart Pi-hole to apply the changes:

sudo systemctl restart pihole-FTL

Bonus: best adblock list to remove ads and protect your LAN against malware

For blocking ads, malware, cookie banners and other annoyances on my LAN I’m using a single adblock list: HaGeZi’s Pro DNS Blocklist. I highly recommend HaGeZi’s lists for a simple and easy single-list installation that will avoid you the hassle and problems of managing multiple lists. The list is updated frequently, and personally, the Multi Pro variant works perfectly for me, with not false positives so far. I recommend you to take your time to fully read the Readme of the GitHub repo to understand what each list contains. Higher tier list already includes the lower tier ones (for example, Multi Pro++ includes Multi Pro, which also includes Multi Normal, etc).

Wrapping up

That’s everything you need to have a clean, network-wide local DNS setup at home; that not only resolves local domain names like *.home.arpa to the correct IPs, but also removes annoying ads from your devices and protect your network against malware. Until next time!

Even when I do my best to ensure content is correctly rendered on RSS readers, some elements like custom info cards/notes and code style blocks can be shown without proper style or syntax highlighting.

You are free to continue reading as you prefer, but I recommend you to read this article in the website if possible. It's fast, has no ads, and minimal privacy-respecting analytics (browser, OS, country and device type) via Umami.

I have a few docker services deployed on a Beelink mini PC at home that I usually access from my LAN. Because all services are deployed in the same machine, they are listening to different ports (say 3000, 5000, 8080, etc), but that’s ugly and not friendly for other non-tech people at home that also use them, so I set up caddy as a reverse proxy. This not only allows me to provide an easy to remember URL like myservice.home.arpa, but also ensures we can use TLS secured connections with HTTPS. In today’s article, I would explain how did I achieve this at home. Let’s dive in!

Installing Caddy

First, we need to install caddy. For that, it’s better that you follow the official instructions for your system, that also will be more up to date than this article in case something change. In my case, I’m using Debian 13 Trixie, so at the time of writing all I did was:

sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https curl gnupg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list
chmod o+r /usr/share/keyrings/caddy-stable-archive-keyring.gpg
chmod o+r /etc/apt/sources.list.d/caddy-stable.list
sudo apt update
sudo apt install caddy

Obtaining and storing the certificates for TLS/HTTPS

Now caddy is installed, but prior to start using it with TLS encryption, we need to plan how to manage the certificates we’ll use for that. First thing we need is to know the domain we’ll use to connect to the service, and once we have that clear, we have two main options:

• You could use Let’s Encrypt and ACME challenges to automatically obtain and renew trusted certificates. For that, the domain you’ll use must be a public one (you own and use a domain bought at a register), and ports 80 and 443 must be open externally, plus other basic requirements explained in Caddy official documentation for automatic HTTPS.

• If you like me, are using local domains such as myservice.home.arpa and only accessing them from your LAN, your best option is to use a self-signed certificate. Caddy already generates one while installing, but to better control it and to simplify the management of multiple services you might have in your homelab, I recommend you to generate your own using your own CA. I have an article about it that covers the topic in better detail. One advantage of using your own CA instead of the default certificate generated by Caddy is that once trusted in your devices, it doesn’t matter what/where else you deploy (a kubernetes cluster, something from a Synology, whatever in another server or a Raspberry PI, etc), they will be always be trusted. For this tutorial, I will use my own self-signed certificated obtained from my own CA in the way I explain in that article.

So continuing with the manual, self-signed generated certificate files. You should have a public certificate (e.g. myservice.crt) and its private key or secret (e.g. myservice.key). Now, let’s install them so we can use them we caddy. We’ll copy them into the /etc/caddy/certs/ folder (create it if it doesn’t exist), then, provide them with the correct file permissions.

sudo chmod 644 /etc/caddy/certs/myservice.crt
sudo chmod 600 /etc/caddy/certs/myservice.key
sudo chown -R caddy:caddy /etc/caddy/certs/*

With the above commands we are indicating those files belong to the caddy user, created automatically as part of the installation process done by caddy. This is the user that will execute our caddy server. Also, for better security, we are indicating that the certificate private key is only accessible by that user caddy. Once this is done, we can proceed with the final configuration for the reverse proxy.

Configuring the reverse proxy setting up TLS/HTTPS

Most, if not all, of caddy configuration is made on its main configuration file under /etc/caddy/Caddyfile. Setting up a reverse proxy with Caddy is fairly simple. For example, imagine I want to configure a reverse proxy for the service myservice under the URL myservice.home.arpa using TLS with the previous certificates to achieve an HTTPS connection. The service is running on that same machine on docker and listening to port 8000 All I need to do is to add the following to the Caddyfile

myservice.home.arpa {
	tls /etc/caddy/certs/myservice.crt /etc/caddy/certs/myservice.key

	reverse_proxy http://localhost:8000 {
		header_down Referrer-Policy "strict-origin-when-cross-origin"
	}
}

And that’s it. Now, the final thing you need to do is to restart caddy to start using the new configuration, which you can do in Debian with sudo systemctl restart caddy. Once caddy restarts, you reverse proxy should be working and you should be able to access your service under https://myservice.home.arpa from your LAN. Things are easy with caddy, right?

# Caddy is running on my Beelink Mini PC with static IP 192.168.1.100
192.168.1.100	myservice.home.arpa

Even when I do my best to ensure content is correctly rendered on RSS readers, some elements like custom info cards/notes and code style blocks can be shown without proper style or syntax highlighting.

You are free to continue reading as you prefer, but I recommend you to read this article in the website if possible. It's fast, has no ads, and minimal privacy-respecting analytics (browser, OS, country and device type) via Umami.

At the beginning of the year, I wrote a similar article on how to achieve this on a regular PC using TPM 2.0, while installing Manjaro (ArchLinux based) on my main desktop. However, what about a VM? When hardware is virtualized, things are a bit different. In this article, I’ll show you what you need to create a Debian VM in ProxmoxVE, with the root partition encrypted with LUKS that automatically unlocks and mounts at boot. Let’s dig in!

Configuring ProxmoxVE

First, we need to provide the virtualized hardware to make this auto-unlock and auto-mount thing to work. For that, like in my previous article, we’ll leave it to TPM 2.0. We can emulate a TPM device with Proxmox. For that, when creating the VM, under the “System” tab, we need to configure two important settings.

• We must define OVMF (UEFI) as the BIOS. This is because TPM will only work with UEFI BIOS.
• We must obviously enable TPM, in its 2.0 version, by checking the Add TPM checkbox and indicating the correct version.

Click on image to enlarge

You can configure where to store the EFI and the TPM. For the sake of this tutorial, I kept things simple and selected the default local-lvm storage on my ProxmoxVE installation.

An additional change from default you might notice: I’m using q35 as the Machine setting, instead of the default i440fx. This is because q35 is modern, has better PCIe support, and works great with Debian. i440fx is there for legacy hardware or legacy device emulation, which is not what I need. It will work anyway, but it’s always better to use the modern and well-supported option if you don’t have an specific reason to stick with the legacy options.

Also, enabling Qemu Agent allows for better communication between the host and the virtualized machine.

Installing Debian

Installing Debian is straight-forward nowadays. After booting up the installation ISO, I choose the default CLI install (plain Install option) just because I’m more familiarized with it and feels quicker to me, but the same can be achieved with the GUI too.

Go to the usual language and locale settings, host, users creation, etc., until you reach the Partition disks step. Here, we’ll select Guided - use entire disk and set up encrypted LVM.

Click on image to enlarge

If asked for a partition scheme, I recommend you to either select All files in one partition or Separate /home partition, but do not separate /var and /tmp or you will suffer from low disk space problems, specially if using docker later. I chose All files in one partition.

Click on image to enlarge

Then, provide the encryption passphrase. This is the passphrase used to encrypt the data on your root partition. It’s (it should be) different from your root and user passwords, and it’ll be later stored in the TPM for the auto-unlock and auto-mount to work. In the next step, accept the proposed partition scheme and continue by writing changes to disk. Note that the installer generated separated /boot and /boot/efi partitions, needed for auto-unlock feature to work as explained on my previous article.

Click on image to enlarge

Next, Debian will start installing. I’m using a net-install ISO, so it will fetch all updated packages from the network and I’ll get an up-to-date system after installation. From here, let’s just wait for the installation to finish and then reboot the VM.

Enabling auto-unlock and auto-mount of root partition

Now that Debian is installed, once the VM reboots, it will load GRUB and start loading Debian. However, due to the disk being LUKS encrypted, we need to manually unlock it this time. Once we configure everything, it will automatically unlock on next restarts.

So open the VM directly from Proxmox web GUI, to land on the VNC interface that allows us to interact with the VM like a regular PC. You will see a prompt asking for the disk encryption passphrase. Something like: Please unlock disk sda3_crypt:. Introduce the passphrase to allow the VM continue booting.

Once the VM fully booted, it’s time to configure the auto-unlock. Log in as root from the VNC, or access via SSH if you enabled it during Debian installation. Then, we’ll install the required packages:

apt install clevis clevis-luks clevis-tpm2 tpm2-tools clevis-initramfs

Now, let’s find our encrypted root partition and configured it to automatically unlock on boot. First, let’s search for it with lsblk:

root@debian:~# lsblk -o NAME,SIZE,FSTYPE,MOUNTPOINT
NAME                     SIZE FSTYPE      MOUNTPOINT
sda                       32G
├─sda1                   976M vfat        /boot/efi
├─sda2                   977M ext4        /boot
└─sda3                  30.1G crypto_LUKS
  └─sda3_crypt          30.1G LVM2_member
    ├─debian--vg-root   28.5G ext4        /
    └─debian--vg-swap_1  1.6G swap        [SWAP]
sr0                      754M iso9660

In my case, I can see that sda3 is the encrypted root partition, as it contains the crypto_LUKS tag. So let’s configure it to auto-unlock with clevis. For that run:

clevis luks bind -d /dev/sda3 tpm2 '{"pcr_bank":"sha256","pcr_ids":"7"}'

A quick explanation on the options I used:

• "pcr_ids":"7" means we want to use PCR ID 7. It will seal the LUKS key against the UEFI settings and the Secure Boot policy using the TPM 2.0 chip. So if the UEFI or Secure Boot settings are modified, the TPM will compute different PCR values and decryption will fail. This gives protection against evil maid attacks.
• "pcr_bank":"sha256" means to use SHA256 as the algorithm to be used for the PCR bank. If not specified, it defaults to SHA1, which is less secure, and in my particular case for my Proxmox VE and host hardware, it’s unsupported, so I would get an error if I don’t specify it.

So execute the clevis command, and when prompted for existing LUKS password, introduce the disk encryption passphrase. And that’s it. Now, you can reboot your VM and it will automatically unlock the root partition and boot normally. I hope it helps!

Even when I do my best to ensure content is correctly rendered on RSS readers, some elements like custom info cards/notes and code style blocks can be shown without proper style or syntax highlighting.

You are free to continue reading as you prefer, but I recommend you to read this article in the website if possible. It's fast, has no ads, and minimal privacy-respecting analytics (browser, OS, country and device type) via Umami.

Hello everyone! This time I just want to share with you how I configured sending email notifications from my Proxmox server. This way, I’ll get an email with different notifications, like backups, jobs and tasks.

I’m using a GMail account I created just for this single purpose, so the steps I’ll share in the following paragraphs are using GMail SMTP servers and settings. However, the steps to use another SMTP server should be the same, just changing some key configuration values, so you can take it as a general guide with some care. Let’s jump into it!

Configuring postfix

Postfix is a popular open-source Mail Transfer Agent (MTA) used to send, receive, and deliver email on Unix-like systems. That’s the tool we’ll use to send our mails from Proxmox. First, let’s ensure we have all the dependencies we need. As root, run:

apt install libsasl2-modules postfix-pcre

Then, let’s modify some config files. First, let’s indicate postfix to use GMail SMTP server.

# Add the following settings to /etc/postfix/main.cf
# Update existing values if keys exist

relayhost = smtp.gmail.com:587
smtp_use_tls = yes
smtp_sasl_auth_enable = yes
smtp_sasl_security_options =
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_tls_CAfile = /etc/ssl/certs/Entrust_Root_Certification_Authority.pem
smtp_header_checks = pcre:/etc/postfix/smtp_header_checks

# Comment the following line out, or just delete it
# mydestination = $myhostname, localhost.$mydomain, localhost

Now, let’s provide postfix the credentials for the server. We do it by generating a /etc/postfix/sasl_passwd file with the email address and the password of the GMail account, in the shape <server>:<port> <email_address>:<password>. The password will likely be your google account password, or an application password if you set up 2FA.

# File /etc/postfix/sasl_passwd
smtp.gmail.com:587 your_email_address@gmail.com:your_password

And the last file we’ll create: provide correct SMTP headers to Postfix, so the sender details will appear correctly on your destination email client. This includes a custom name for the sender (in the example, just My Hostname - Proxmox, but it can be anything), and the sending email address between angle brackets (< >).

# File: /etc/postfix/smtp_header_checks
+ /^From:.*/ REPLACE From: "My Hostname - Proxmox" <your_email_address@gmail.com>

Finally, we need to set the correct permissions to the files, update postfix maps, and restart postfix.

chmod 600 /etc/postfix/sasl_passwd
postmap /etc/postfix/sasl_passwd

chmod 600 /etc/postfix/smtp_header_checks
postmap /etc/postfix/smtp_header_checks

systemctl restart postfix.service

And that’s all for postfix!

Testing it

To test the mail notifications, form Proxmox VE GUI, go to Datacenter > Notifications. There should be an existing “Notification Target” of type sendmail to notify the root user in its configured email address (set up during ProxmoxVE installation). By selecting that notification target and clicking on “Test” button we should be able to test it.

At this point you should get a test email on your inbox. If it’s not there, although it should not happen because we are using GMail SMTP server which have a good reputation, take a look at your SPAM folder, just in case such an automated test email triggered some SPAM rules. Also, if you are using the same email address as the sender and receiver, it might not appear in your inbox.

Alternative approach

I cannot verify it because I didn’t try this way, but it should be possible to make the same configuration we did for Postfix to use GMail SMTP server just from the Proxmox VE web GUI. For that, you should only need to add a new Notification Target of type SMTP. But as said, I didn’t try it, so take it with a pinch of salt.

Even when I do my best to ensure content is correctly rendered on RSS readers, some elements like custom info cards/notes and code style blocks can be shown without proper style or syntax highlighting.

You are free to continue reading as you prefer, but I recommend you to read this article in the website if possible. It's fast, has no ads, and minimal privacy-respecting analytics (browser, OS, country and device type) via Umami.

Hello everyone! Today I wan’t to share with you all how I set up my backup scripts/tasks in Linux, with the help of Systemd. As most of you might know if you have more than a few months on using Linux, Systemd provides a system and service manager that runs as PID 1 and starts the rest of the system. Is the standard in most Linux distributions out there, including Debian based ones (the most used on servers worldwide).

We can take advantage of its capabilities to implement robust cron jobs that run our backup tasks or scripts, and we will also get out of the box logs management via journalctl. Let’s do it!

A working duo: service and timer

We need two systemd pieces to make this work: a systemd service, that will be the one in charge of executing the task or script, and a systemd timer, that is the one triggering the execution of the service on a schedule. Let’s take a quick look to both of them and I’ll comment them later. As an example, during the rest of the article, I will use a simple rsync command as a backup tool.

# File under /etc/systemd/system/rsync-backup.service
[Unit]
Description=Rsync backup
After=network.target

[Service]
Type=oneshot
ExecStart=/root/.local/backup.sh

# File under /etc/systemd/system/rsync-backup.timer
[Unit]
Description=Timer to make a rsync backup every Monday

[Timer]
OnCalendar=Mon 02:30:00
Persistent=true

[Install]
WantedBy=timers.target

So we defined our service rsync-backup.service that executes the actual backup script (/root/.local/backup.sh), and a timer rsync-backup.timer that ensures the service is executed every Monday at 2:30 AM. To enable the timer, we must execute the following commands:

systemctl daemon-reload
systemctl enable --now rsync-backup.timer # Chose your real timer name here!

Now, the timer should be visible if we execute systemctl list-timers and the end of the table shown by that command.

If you want to manually run the backup at an arbitrary time, you could just execute the actual backup script under /root/.local/backup.sh (in this particular example, but I recommend you to do systemctl start rsync-backup.service instead, so this way, we get the log management with journalctl out of the box.

Finally, note that the service type is oneshot. This means, the service will execute the script under ExecStart and finish once the script exits. During that time, if you check the service with systemctl status it will show as “activating”. You might be tempted to use simple type instead of oneshot. The differences are mostly on parallel executions when multiple ExecStart are added, with doesn’t matter for our use case, but here is a nice explanation from Stack Overflow in case you are curious.

Adding health-checks

How do we add health checks then? I use Healthchecks.io for that, which is a simple and free to use (up to 20 jobs) service that works like a gem. I’m in love with it. How it works is simple: they provide an endpoint you must hit with curl or whatever to indicate your job executed. You can even provide more info like how long did the execution took, or even any possible exit error code.

I won’t guide on how setup the check in Healthchecks.io itself, as I find the tool very straightforward and they have their own support articles in case you need them. So let’s jump directly to an example of a health check configured there for our backup script.

Click on image to enlarge

A quick explanation on the configuration shown on the image:

• We defined the name of the task in Healthchecks.io as “Rsync-backup”. The check mark is green because I already did my first health check while testing my script.
• The ping URL is the one we must hit with curl. It’s a random UUID they will provide to you.
• Now, in the integrations, is where we setup how we want to be notified on errors. I earlier configured it to notify me via email and telegram.
• Then, we set when the cron is expected to run, and a grace time period. In this example, I indicated the cron must run Mondays at 2:30 AM (30 2 * * 1 in crontab language), and that it should not last more than 30 minutes from that time to ping. Therefore, if 30 minutes past 2:30 AM on Monday a ping has not being received, we’ll be notified on our configured channels.
• Finally, there are some quick metric about the last ping (9 hours ago in my case) and how long did the script execution took (7 minutes, more on this next). This will be populated after your first ping.

Nice! So we configured Healthchecks.io and enabled our first health check. How do we integrate it in our backup task? If you remember, we were using the systemd service to execute a backup script /root/.local/backup.sh. Let’s take a look to it:

#! /usr/bin/env bash

HEALTHCHECKS_URL=https://hc-ping.com/1a2fdfaf-cbf7-4c37-8d87-e7e013e27e3b

curl -fsS -m 10 --retry 5 -o /dev/null $HEALTHCHECKS_URL/start
rsync -aAHXv --delete /etc /mnt/backup
curl -fsS -m 10 --retry 5 -o /dev/null $HEALTHCHECKS_URL/$?

A simple script as you can see. What this does is hitting our health check endpoint with start path param, which indicates Healthchecks.io our execution started. This is useful for the duration metrics, but also for resetting the grace time period. Then, the actual backup task is executed: in this example, a simple rsync command. Finally, we hit our health check endpoint again, this time sending the exit code of the rsync command as path param. This way, we’ll register a successful execution if sending a value of zero, and we’ll register (and be notified) of an error if sending something different.

Note: don’t forget to provide execution permissions to your script with chmod +x 😉

Checking logs after executions

Did the task failed and you want to know what happened? Easy enough! With systemd we get logs management out of the box. Just run journalctl -u rsync-backup.service (or whatever name your backup service has) and all your logs will be there to be checked!

What’s next?

Due to Systemd strong capabilities, we could expand and improve the above service and timer, like adding retries in case of errors. There is so much things we can do with Systemd, so as always on Linux, hack and experiment to find your way!

Even when I do my best to ensure content is correctly rendered on RSS readers, some elements like custom info cards/notes and code style blocks can be shown without proper style or syntax highlighting.

You are free to continue reading as you prefer, but I recommend you to read this article in the website if possible. It's fast, has no ads, and minimal privacy-respecting analytics (browser, OS, country and device type) via Umami.

In the last article, I wrote how to create your own Certificate Authority to provide certs for your self-hosted services, which is very useful when you have tons of servers, VMs and services deployed in your homelab, as I do. The backbone of my homelab is a Synology NAS server and a Proxmox cluster, along with some docker services here and there. I’m using reverse proxies to access my docker containers with HTTPS: caddy for the ones hosted on Proxmox, and the built-in reverse-proxy from DSM for those hosted on the Synology.

Today, I’ll show you particularities of each platform, and how to upload and renew existing certificates in Synology DSM, Proxmox Ve and Caddy; while maintaining the trust in the new certs thanks to the CA we created in the previous article. Let’s go!

Renewing and updating certs in Synology DSM

As everything with Synology, we can renew the certificate directly from DSM. Following are the instructions for DSM 7.2, and the official instructions from Synology Knowledge Center can be found here.

1. Go to Control Panel > Security > Certificate.
2. Select the desired certificate.
3. Select Renew certificate from the Action drop-down menu, and click Next. A new private key and certificate signing request will be created.
4. Click Renew certificate to retrieve your new private key and certificate signing request (CSR). You can use the new signing request to reapply for another certificate authority signed certificate.

Then, you can use that CSR to renew your certificate using your own CA.

Renewing and updating certs in Proxmox VE

In contrast with Synology, Proxmox VE doesn’t have a built in mechanism to generate a CSR for certificate renewal, and we must do it manually:

1. On a terminal, generate a new private key and a CSR.
2. Then, use that CSR to issue a new certificate.
3. Finally, import the new generated certificate and the private key to Proxmox VE by going in the webgui to Node name > Certificates > Upload Custom Certificate.
4. The previous old certificate should be overwritten. If not, you can manually delete it by selecting it and the clicking on Delete Custom Certificate.

Now, just reload the Proxmox VE webgui and the new cert will be in use.

Renewing and updating certs in a web server like Caddy (and Nginx or Apache)

Caddy requires to be configured manually, usually via an SSH session, but the steps are very similar to what we already know:

1. As with Proxmox VE, on a terminal, generate a new private key and a CSR.
2. Then, use that CSR to issue a new certificate.
3. Now, SSH into your caddy server, and check the configuration at /etc/caddy/Caddyfile to check from where is the server is reading the certs from. You will do the same for Apache or Nginx by looking to their respective configuration files.
4. In my case, in caddy, they are being read from /etc/caddy/certs/ as I can read in the configuration:

tls /etc/caddy/certs/server.crt /etc/caddy/certs/server.key

5. So just replace those files with the key and cert newly generated from the CA.
6. Don’t forget to assign the correct permissions to both files:

chmod 644 /etc/caddy/certs/server.crt
chmod 600 /etc/caddy/certs/server.key
chown -R caddy:caddy /etc/caddy/certs/*

7. Finally, restart or reload your webserver to take the new certificates with systemctl reload caddy or caddy reload depending on how you installed it.

Even when I do my best to ensure content is correctly rendered on RSS readers, some elements like custom info cards/notes and code style blocks can be shown without proper style or syntax highlighting.

You are free to continue reading as you prefer, but I recommend you to read this article in the website if possible. It's fast, has no ads, and minimal privacy-respecting analytics (browser, OS, country and device type) via Umami.

Some context

In the last years, my homelab has grown exponentially. I started with a Raspberry Pi zero back in University, just as a playground with nothing special in there, and nowadays it consists of a Synology NAS server, a four Raspberry PI 4 cluster, and a Proxmox cluster with two nodes backed by some Beelink Mini PCs. And of course, all of this hardware serves multiple self-hosted services that are currently very integrated in my daily workflow and other aspects of my life.

Most of my services are only accessible from my LAN, and I want them to stay like that. I can access them via their local IP 192.168.X.X, or using a local domain name like myservice.home.arpa. And of course, I would like to access them via secure HTTPS connections, just in case. I think nowadays is it possible to use auto-managed Let’s encrypt certificates with those, either using Cloudflare tunneling, Tailscale, or other tools. But for me personally, using your own CA for self-signing certificates is much easier to setup, and also more fun. At the end of the day, those are local services on my homelab, so I don’t need complex solutions, and I want to learn and experiment.

Why creating our own CA?

I also want, for the services I use the most, to keep things stable and avoid service interruptions, failures, and cumbersome manual tasks. When you generate a self-signed certificate, you must manually trust it in your client devices. What happens when you have N services? Without your own CA, you have N different certificates, and you must trust the N different certificates into your M different devices. And each time one of them expires and you generate a new one, you must trust it again.

This problem is already solved, that is one of the reasons why Certificate Authorities (CA) exist! If you create your own CA and use it to generate and sign all of your self-signed certificates, you only need to trust the CA, and that’s it!

• So an existing certificate expires and you renew it? No problem! Your devices already trust it.
• So you need to add a new IP or a new domain name to an existing service certificate? No problem! You won’t need to share and trust the new certificate elsewhere.

So, how do we achieve this?

Creating our own CA

We can create our own CA with openssl from the terminal. The commands I share here works in UNIX like system like Linux and MacOS. I’m not sure if they will work as they are in Windows, but shouldn’t be so much different, and the main concepts are the same.

First what comes first. Let’s generate the private key for our CA. I recommend using a minimum length of 4096 bytes.

openssl genrsa -out rootCA.key 4096

Now, let’s use it to generate the root certificate for our CA. I’ll set an expiration date in ten years from now, but feel free to use whatever value you prefer.

openssl req -x509 -new -key rootCA.key -sha256 -days 3650 -out rootCA.pem

It will ask for the certificate data, I usually only fill the Country Name (ES in my case) and the Organization Name and Common Name with the same value of My own CA.

Congratulations, now you have your own CA! From now on, we will use this newly generated root certificate and key to generate a sign any certificate we need for our homelab. Keep those safe and don’t lose them! I usually store them as a .zip into my 1Password vault, along with the rootCA.srl file (we’ll generate it later). You can use any password manager or store them as an encrypted .zip file or whatever in any public cloud or hosting service you usually use.

Now, as explained earlier, you only need to trust the CA root certificate, and all certificates generated from it will be automatically trusted. Every device differs a little bit, but I’ll show you next how to trust it on Apple devices.

Trusting the CA root certificate in iPhone, iPad and Mac

Trusting the CA root certificate in MacOS is fairly easy. Just double click on the rootCA.pem file and the Keychain app will open. Important! Don’t confuse it with the new Passwords app introduced in (I think) MacOS 15 Sequoia. Once on the Keychain app, look for it (it should appear under the Common Name, in my case My own CA), double click again, and in the window it opens select “Trust” > “When using this certificate” > “Trust always”.

On iPhone and iPad it requires a little bit more of work. Currently tested on iOS and iPadOS versions 18. If in the future we have a new iPadOS more computer like things can differ a little. Anyway, just send the rootCA.pem to your device via Airdrop or whatever you prefer, and open it. It will ask you where do you want to install it. Select “this iPhone” or “this iPad”. Then, go to Settings and you’ll see the new profile ready to install, just bellow your name from your Apple account. Click there, accept the warning about the cert being untrusty (we generated it) and install it. That’s everything? Well, no. The cert is now installed, but Safari and other browsers like Chrome won’t trust it yet. We need a last step. We must go to Settings > General > About > (scroll down to the end) Certificate trust settings. There, you will see our new installed root certificate and a deactivated toggle. Flip the toggle, accept the new warning, and voilà.

Requesting the CA a new certificate for our service

To generate certificates for our services, we need to create a Certificate Signing Request (CSR) for them. Think about it like the CA is the Government, the service is you, and you fill a form (the CSR) to request anything. Like any request, as you will do with your Government, you need to sign it. So first, we need to generate a private key for our service/server. This is the same as we did initially for the CA key.

openssl genrsa -out server.key 4096

Now, we create and sign with the server key the CSR for our CA.

openssl req -new -key server.key -out server.csr

Again, it will ask you some basic data for the future certificate. Like before, I just set the Country Name, Organization Name (in my case to My Homelab) and Common Name (in my case, to the service or server name, like My Service). The rest of fields, including challenge password, are left empty.

Using the CA to generate a new certificate from the CSR

For better security and improved support, we’ll use an X.509 extensions file to define some parameters for the new certificate. Create a file v3.ext and update the values under [alt_names] with your own:

authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names

[alt_names]
DNS.1 = myservice.home.arpa
IP.1 = 192.168.1.123

Is it possible to include more DNS names or IPs. Just add extra entries like DNS.2 or IP.2 and so on. Also, it’s possible to use wildcards to generate a certificate valid for all subdomains, like DNS.2 = *.myserver.home.arpa. If you use wildcards, ensure you are using it under the DNS.2 entry, and leave the DNS.1 for the root domain like DNS.1 = myserver.home.arpa.

Next, let’s use our CA to generate a new certificate from the CSR, using the above extensions file. I will use SHA-256 to sign it and set a validity period of 825 days, as apple won’t trust certificates longer than that, and me and my family use Apple devices. Feel free to set a longer validity period (like another 10 years) if this doesn’t care to you.

openssl x509 -req -in server.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial \
        -out server.crt -days 825 -sha256 -extfile v3.ext

Now you will have two new files in your directory:

• server.crt is the new certificate for your service/server that you can use along with the earlier generated server.key. Just upload them to your server or service and configure it to start using them.
  • I also wrote instructions on how to upload and renew certificates on both Synology DSM and Proxmox VE in another article. Take a look if you use any of them, you might find it useful.
• rootCA.srl stores the next serial number your CA will assign when signing a new certificate. Each time you sign a certificate with your CA, OpenSSL reads and updates this file, ensuring that every certificate has a unique serial number (which is required for security and validity). You must keep this rootCA.srl in the same directory as your rootCA.key and rootCA.pem every time you generate a new certificate with this CA. Deleting or losing this file can cause duplicate serial numbers, which can invalidate your CA or cause certificate management issues.

Renewing server certificates

To renew a server certificates, for example because it’s near its expiry date, you will use a CSR again. You could theoretically reuse the same CSR you used for your initial certificate, and or the same server private key, but it’s better for security to generate a new private key for the server and use it to create a new Certificate Signing Request.

openssl genrsa -out server.key 4096
openssl req -new -key server.key -out server.csr

And then, sign it with the CA. This time, we do not use the -CAcreateserial option and instead relies on the existing rootCA.srl file with -CAserial.

openssl x509 -req -in server.csr -CA rootCA.pem -CAkey rootCA.key \
        -CAserial rootCA.srl -out server.crt -days 825 -sha256 -extfile v3.ext

Renewing CA certificates

And what about renewing the CA certificates without invalidating previous generated certs using it? If we want all existing certs to remain trusted, we just need to generate a new CA root certificate using the same private key we used the first time. In fact, the command is exactly the same:

openssl req -x509 -new -key rootCA.key -sha256 -days 3650 -out rootCA.pem"

And that’s it. Simple, and you keep full control. The only cons if that you need to remember to update the certs before they expiring. It’s a good idea to write you down a reminder somewhere that will notify you in time (I write myself a reminder in Todoist to notify 30 days prior expiration). If you want, you could also automatize it, by writing an small service, CLI or API that will receive a CSR and will provide new certs for you. Kinda like how Let’s Encrypt works, but homemade.

Even when I do my best to ensure content is correctly rendered on RSS readers, some elements like custom info cards/notes and code style blocks can be shown without proper style or syntax highlighting.

You are free to continue reading as you prefer, but I recommend you to read this article in the website if possible. It's fast, has no ads, and minimal privacy-respecting analytics (browser, OS, country and device type) via Umami.

Hello everyone! This month, I want to talk you about the “Ship, Show, Ask” git strategy. I use it with my teammates at Cabify, and I find it to be a wonderful way to maintain a good pace and individual productivity, while at the same time, ensuring a continuos code delivery and transparency and communication with the rest of the team.

Has you ever had a pull request or merge request blocked for days waiting for review? Do you find difficult for you or your team to review every piece of code in time? Then, “Ship, Show, Ask” could be a good fit for you or your team. Only note that, for this git strategy to work, two things must be truth in your team and your code:

• You have a mature team, where everybody knows everybody and trust everybody (how they code, how they work).
• You have a good automatic test suite and checks/regression tests that are automatically run in the pipeline prior to deploying any new code.

without the above, “Ship, Show, Ask” won’t never work. Why? With this strategy, you won’t always open a Pull/Merge Request. And when you do, you won’t always wait for your colleagues to review it prior to merge. So yes, a mature team and good and sufficient automatic checks are a must.

Okay, after this disclaimer, let’s dive into it!

What is “Ship, Show, Ask”?

“Ship, Show, Ask” is a branching strategy that combines the idea of creating Pull Requests with the ability to keep shipping changes quickly. It was introduced by Rousan Wilsenach on Martin Fowler’s blog.

The changes we make in the repository are categorised into three types:

• Ship: Merged into the main branch without review.
• Show: Opens a Pull Request but merges immediately without review.
• Ask: Opens a Pull Request to discuss changes before merging.

Click on image to enlarge

The three possible categories for your changes give the strategy its name. Now, let’s see each of them in detail and why. When should you use each of them?

Ship

Ship means making a change directly to the main branch. We don’t expect code reviews, we go straight to production (though tests or checks will still run automatically by the CI before deployment to prevent errors).

The commit goes directly to production, bypassing the usual Pull Request.

Click on image to enlarge

Intended for:

• Adding a small change using an established pattern.
• Fixing a simple bug caused by an error.
• Documentation updates.
• Small code improvements based on team feedback.
• Adding new tests to prevent errors.

Show

Here, we use a Pull Request, but we don’t expect manual code reviews. Instead, we rely on automated tests, coverage checks, and code validation without requiring another person’s input. We create a Pull Request as a way to document the change and validate that automated tests pass, but don’t wait for team feedback.

This doesn’t mean conversations about the code won’t happen. They’ll just occur after merging. The goal is to keep work flowing forward with minimal blockers while still leaving room to discuss and improve development practices. The team is notified when a Pull Request is created, reviews it later, and provides any necessary feedback. The key is that the Pull Request remains easily identifiable and separate.

Click on image to enlarge

Intended for:

• Necessary bug fixes that leave a record for learning.
• Small code improvements or refactors.
• Adding new features following agreed structures.
• New functionality with automated tests.

Ask

This category is similar to Show, but here we do wait for team feedback before merging. We do this because there’s some uncertainty: maybe the solution is complex, we don’t know how to implement it, or there are doubts. The goal is to keep the branch alive for as little time as possible to avoid blocking others.

With Ask, we expect the team to review the Pull Request before merging. We’re not waiting for an approval, we’re just starting a conversation to resolve uncertainty. This is important to remark: choosing Ask doesn’t mean we’re waiting for colleagues’ approval, but just to open a discussion before merging due to blockers or doubts. We may not need a full review, and if we do, we should clarify that.

Click on image to enlarge

Intended for:

• Large pieces of code requiring assistance.
• Doubts about implementation or code quality.
• Uncertainty about what we’re doing.
• Waiting for external dependencies before merging.

The Rules of “Ship, Show, Ask”

Naturally, as stated at the beginning of this article, following this strategy requires some ground rules:

1. A good, robust, and sufficient automated test suite.
2. A reliable CI/CD system that ensures the main branch is always deployable, and that prevents unwanted errors in production.
3. Branches are as small as possible, short-lived, and always branched directly from main.
4. Trust in the team and strong development practices like pair programming, seniority, and above all, accountability. The developer decides their change’s category. With great power (merging your own Pull Requests) comes great responsibility (not breaking production).
5. The team has moved past individual ego, trusts each other, and accepts that the main branch doesn’t need perfect code (as long as automated tests pass).

Final Thoughts

I want to close this article as I started it, defending all the good about this strategy. I believe the core idea behind it is to empower individuals, give the team autonomy, and treat everyone equally. It also speeds up development and enables continuous delivery. But achieving this requires a lot of trust, which not all teams may have initially. Because of that, being able to apply this inside a team is an achievement in its own, as it’s a proof that the team has reached a mature state, without individual egos and with trust in each others. A truly sane work environment.

I’m proud to be using it from nine to five Mondays through Fridays, and I’m glad for my teammates. Keep it going!

Even when I do my best to ensure content is correctly rendered on RSS readers, some elements like custom info cards/notes and code style blocks can be shown without proper style or syntax highlighting.

You are free to continue reading as you prefer, but I recommend you to read this article in the website if possible. It's fast, has no ads, and minimal privacy-respecting analytics (browser, OS, country and device type) via Umami.

Hello everyone, today I just want to share two code snippets for those that might find it useful. I have always like how the reading progress bar look in the blogs, so I build my own one for my own tech blog (this blog!). Each article in my blog has the same behavior, with a reading bar in the top of the page, the header, that fills as you scroll down. You can check how it works by scrolling down in this page.

My blog was initially built on top of Next.js with React, but later I moved to Nuxt.js and Vue, so I had to rewrite my component, and that’s why I have both versions.

Okay, let’s stop talking, here is the code!

React component

Here is the React component:

// File: src/components/ProgressBar.tsx
import React, { FC, useEffect, useRef } from "react"

type ProgressBarProps = {
  enabled?: boolean
}

const ProgressBar: FC<ProgressBarProps> = ({ enabled = false }) => {
  const progressBar = useRef<HTMLDivElement>(null)

  const updateProgressBar = () => {
    const winScroll = document.body.scrollTop || document.documentElement.scrollTop
    const height = document.documentElement.scrollHeight - document.documentElement.clientHeight
    const scrolled = (winScroll / height) * 100
    if (progressBar && progressBar.current) progressBar.current.style.width = scrolled + "%"
  }

  useEffect(() => {
    if (progressBar && progressBar.current) progressBar.current.style.width = "0"
    if (enabled && typeof window !== typeof undefined) {
      window.addEventListener("scroll", updateProgressBar)
      return () => window.removeEventListener("scroll", updateProgressBar)
    }
  }, [enabled])

  return (
    <div className="w-full h-0.5 bg-transparent">
      <div className="w-0 h-0.5 bg-txt-color-bright" ref={progressBar}></div>
    </div>
  )
}

export default ProgressBar

I was using it back then as a part of the header, but was only visible on the articles, like so:

// File: src/components/Header.tsx
import ProgressBar from "./ProgressBar"

const Header: FC = () => {
  return (
    <header className="fixed top-0 w-full z-20">
      <nav className="border-b border-border-color">{/*...*/}</nav>
      <ProgressBar enabled={router.pathname === ROUTES.article} />
    </header>
  )
}

export default Header

Vue component

And here is the current Vue version as the date of writing this article:

<!-- File: components/article-progress-bar.vue -->
<template>
  <div class="w-full h-0.5 bg-transparent pb-1">
    <div ref="progressBar" class="w-0 h-0.5 bg-primary" />
  </div>
</template>

<script setup lang="ts">
import { ref, onMounted, onUnmounted } from "vue"

const progressBar = ref<HTMLDivElement | null>(null)

const updateProgressBar = () => {
  const winScroll = document.body.scrollTop || document.documentElement.scrollTop
  const height = document.documentElement.scrollHeight - document.documentElement.clientHeight
  const scrolled = (winScroll / height) * 100
  if (progressBar.value) progressBar.value.style.width = scrolled + "%"
}

if (progressBar.value) progressBar.value.style.width = "0"

onMounted(() => {
  if (typeof window !== "undefined") {
    window.addEventListener("scroll", updateProgressBar)
  }
})

onUnmounted(() => {
  if (typeof window !== "undefined") {
    window.removeEventListener("scroll", updateProgressBar)
  }
})
</script>

And again, I’m using it directly in the header:

<!-- File: components/header-component.vue -->
<template>
  <header
    class="fixed inset-x-0 top-0 z-[1] px-4 sm:px-10 backdrop-blur-sm bg-base-100/80 dark:bg-base-100/80 max-w-6xl mx-auto"
  >
    <nav class="py-2 md:py-6">
      <!-- ... -->
    </nav>
    <ArticleProgressBar v-if="route.path.match(/^\/blog\/\d{4}\/[^/]+\/?$/)" />
  </header>
</template>

<script setup lang="ts">
const route = useRoute()
</script>

That’s all. Short and to the point! I hope it helps you make your own reading progress bar for your site!

Even when I do my best to ensure content is correctly rendered on RSS readers, some elements like custom info cards/notes and code style blocks can be shown without proper style or syntax highlighting.

You are free to continue reading as you prefer, but I recommend you to read this article in the website if possible. It's fast, has no ads, and minimal privacy-respecting analytics (browser, OS, country and device type) via Umami.

If you are like me, and you like to play around with hardware and computers, or just want more privacy for your data (all those important documents like tax reports, health documents, family embarrassing photos, etc.), you probably have your own cloud at home, typically with a NAS. Whether you’ve built your own home NAS using TrueNAS, Synology, or a Raspberry Pi with a bunch of external drives, one thing’s certain: you care about your data. You’ve probably invested time in organizing your media library, or you have important files you don’t want to lose.

To really protect your data from any possible incident, you need to follow a backup strategy. And there is a particular backup strategy that fits great either for big corporations like Google or Microsoft as well as small individual users with a server at home like you and me: the 3-2-1 backup strategy.

But I’m already protecting my data with RAID 1/5/6!

You are not! Redundant RAID configurations (like RAID 1, RAID 5 and RAID 6) are not a backup strategy, just a way to avoid disruption in the typical and inevitable case of a disk stop working. RAID works by duplicating your data (RAID 1) between disks, or breaking it into parts and distributing it among multiple disks along with a way to recover missing parts from a parity disk (RAID 5) or disks (RAID 6).

This will avoid you losing any data if one disk (or two, if using RAID 6) breaks at the same time, but is not a proper backup strategy because you could lose your data in any of the following scenarios:

• Multiple disks breaks at the same time. Even if it seems unlikely, this can really happens. For example, you bought all your disks at once, and they might be from the same manufacturing batch. IF that batch had a problem they can fail at the same time. Or you could get a power spike at home that breaks multiple disk at the same time.
• Your home or your NAS burns down. If your disks are ashes you won’t be able to recover your data from them.
• Someone breaks into your home and steals your NAS. If you do not longer have the disk, you won’t be able to recover the data from them either. If someone stealing your NAS and accessing your data concerns you, you might want to encrypt the disk at rest. Synology has Volume Encryption for that, which I set in my Synology.
• You get ransomware and your disk are encrypted and all your data gone for good. You need to start over.

Okay, so now that we are in the same page and we understand why a good backup strategy is important, let’s move on.

What is 3-2-1 backup strategy?

Luckily for all of us, 3-2-1 backup strategy is very easy to understand and follow.

• Keep 3 copies of your data.
• Keep those copies on at least 2 different storage medias.
• Keep 1 copy off-site.

Click on image to enlarge

You could think at first glance that this is difficult and expensive for small users, and that is more oriented to business system, but not at all. Let me show you how I approach 3-2-1 backup strategy a home for my Synology NAS!

Keep 3 copies of your data

You must keep three copies of your data, not necessarily three backups! I have.

• 1 copy on my NAS. The data itself as we are using it day to day
  • Additionally, I have snpashots of most important folders, that are took every night at midnight using Synology Snapshot Replication and taking advantage of my btfs volumes.
• 1 copy on an external NAS grade hard drive. An old drive I was using on the NAS before. Again, every day at 2:00 AM a backup of the data is made to that drive via USB using Hyper backup
• 1 copy on the cloud, specifically on Synology C2, where I create a backup every day at 4:00 AM using also Hyper Backup.

So you see, three copies! Four if you take into account the snapshots, that I particularly don’t consider myself a backup, but more a quick undo for human errors.

Keep those copies on at least 2 different storage medias

I also do!

• 1 copy is in the NAS, along with the snapshots.
• 1 copy is in an external drive. Along it could be debatable if it a different media, as it’s also a mechanical hard drive, and very similar to those I’m using on the NAS. And in my particular case, because the USB is always plugged to the Synology NAS for convenience.
• But if any doubt, having the copy on Synology C2 makes it a totally different storage media.

Keep 1 copy offsite

And bingo! The backup I have in the cloud, using Synology C2, prepares me for any major disaster that could happen. Obviously, this is more like a last option scenario. Likely, I should be able to restore my data form any other of my backups (snapshots or the external drive). The cloud/offsite options is only for major disasters.

Obviously, this is the more expensive to maintain backup of all I have, and it requires for my case, ~250€ every year for 3 TB with data deduplication. I’m storing there 5,5 TB worth of real data, compressed and deduplicated, in several backups versions with a custom retention policy. I’m using for that data around a 80% of the 3 TB.

Follow the 3-2-1 backup strategy your own way

You don’t need to follow my example! The above was just an illustration of how I approach it. I understand having a cloud provider to backup terabytes of data is not cheap, and sometimes this is a stopper for a lot of people. But remember, you should think on what kind of data you have, and what is important to you, and what do you want to protect and what not. Perhaps, you have tons of movies, that you can always download or ask a friend if you lose them. Or you just want to backup some important PDFs and a few photos. You can always upload those, probably a few GBs, to regular cloud provides like Google Drive, One Drive, iCloud or Proton from a synched folder in your server or computer.

The only thing important is tho follow the 3-2-1 rules, as we saw above. Just three rules that should protect your data of any incident you could face. How do you apply them is up to you, as long as you fulfill all of them for the data you never ever want to lose.

Even when I do my best to ensure content is correctly rendered on RSS readers, some elements like custom info cards/notes and code style blocks can be shown without proper style or syntax highlighting.

You are free to continue reading as you prefer, but I recommend you to read this article in the website if possible. It's fast, has no ads, and minimal privacy-respecting analytics (browser, OS, country and device type) via Umami.

After clean installing my PC with both Linux Mint and Windows 11, and after the initial configuration to my taste and installing my essential apps, I decided I wanted to create a disk image, so next time (or I really mess up) won’t take so much time. For that task, I decided to use the old and reliable tool that is the defacto standard for disk imaging and barebones restores among IT techs: Clonezilla.

As you might already now, I also have a Synology NAS server at home. So I wanted to store there the disk images, as I already have my NAS setup with proper redundancies and backups strategies.

This article intends to be very brief and straight forward. Just a reminder of what I need to do the next time I want to image a disk, and I hope it will also help other people with the same problem. First, let’s focus on what we need to setup in the Synology for this to work, then, we’ll take a look on the actual step-by-step instructions on Clonezilla.

Preparing the Synology NAS SFTP server

First, in order to be able to send the cloned image directly from Clonezilla to the Synology NAS over the network, we need to enable both SSH and SFTP server in Synology. This is just checking a checkbox in the DSM Control Panel app. SSH settings are under Control Panel > Terminal & SNMP > Terminal, while SFTP settings are under Control Panel > File Services > FTP.

Apart from that, I personally created a shared folder called Clonezilla (with capital letter ‘C"). I configured it as a compressed folder and disabled recycle bin, as I would store cold data (disk image backups) and I don’t need recycle bin functionality (I already have btrfs snapshots and other backups if I accidentally delete it). I gave my own user (the same I would use to connect to the Synology from Clonezilla) read and writte permissions. The rest of the settings are default, you don’t need to give “NFS permissions” to the folder or any other kind of permissions.

Now, let’s start with the proper disk imaging.

Imaging the disk with Clonezilla

First, we obviously need to prepare a bootable USB with Clonezilla. I won’t enter into too much detail here. Clonezilla have their own instructions on this, and is the same that when installing Linux or Windows.

Initial steps and Synology connection

So we boot up the Clonezilla Live Image (I selected the default option Clonezilla Live (VGA 800x600)) and then we:

Click on image to enlarge

1. Configure the language and keyboard (I’ll obviously use English language options for this guide).
2. Start Clonezilla.
3. Select device-image mode.
4. Select ssh_server as destination of the backup.
5. To configure the network, I just relied on the dhcp auto option.
6. Then, introduce the local IP of your Synology NAS. Probably something like 192.168.1.X
7. Introduce the SSH port of your Synology SSH server. By default it’s 22.
8. Then, introduce your Synology username.
9. Now, indicate the folder where we’ll store the backed up disk image. In my case, I put /Clonezilla
   • IMPORTANT: you don’t need to indicate the volume to your folder or full Linux path in the Synology. Trying to provide a path like /volume1/Clonezilla will end in a “No such file or directory error” when connecting to the server in the next step.
   • Also, ensure you allow to connect via SSH using password to your Synology NAS server. Otherwise, if you only allow access via SSH keys, Clonezilla won’t be able to connect to the server and you will get an error.
10. Finally, introduce your user password for the SSH connection to establish.

Actual backup, disk image options

Once the connection is established, it’s time to start with the disk imaging. I personally stick with the Beginner mode, as I don’t need anything fancy and I want to avoid possible mistakes. Here is how it follows:

Click on image to enlarge

0. As said, select Beginner mode.
1. I want to clone the full disk, so I select savedisk option, but it would also be possible to just image partitions with saveparts option.
   • Note: I will continue this guide with savedisk, and I’m unsure how it changes if only cloning partitions.
2. Select what physical drive you want to image (in case you have more than one).
3. Give a name to the backup, or accept the default one provided by Clonezilla.
4. In my case, the Linux Mint partition was LUKS encrypted, so it gave me an option to introduce the LUKS password and clone the disk in an efficient manner regarding final image size.
5. Regarding compression, I selected z9p that overall, as far as I googled, seems to have better performance, both in final image size and compression speed. z1p seems more like a legacy options for very old hardware.
6. Then, for some systems like Linux, it asks you if you want to automatically check/repair the source filesystem prior to making the image. If you are sure the source filesystem works fine, you can skip it with sfsck as I did. Otherwise, you migh want to use another option.
7. Once the image is created, you can check if it’s restorable. I heavily encourage that, as it’s very quick and will save your ass in the future if you need the image and then you realize it doesn’t work. Checking the image now will prevent problems in the future, as any possible problem will be highlighted and you can take action now and be ready for the future.
8. Clonezilla asks you if you want to encrypt the disk image. In my case, I selected not to (senc), as I already have my Sinology drive encrypted at rest, and I don’t worry about anyone accessing the file when the NAS is up and running (I’m the only one with permissions in the Clonezilla/ folder, and also the only one with SSH and SFTP access).
9. Finally, you need to indicate to Clonezilla what to do after the image is created. I choose -p choose as I want to go back to my computer and see the status of everything, but you can tell it to automatically poweroff or reboot on finish.
10. Now, let it create the disk image, it might take a while. In my case, for a 100-200 GB installation on a 1 TB NVMe it took between 10-20 minutes on my hardware (time differs between systems, Linux and Windows).
    • Note: initially it might say something like “X hours remaining”, but in my case, it dropped after a while.

And that’s it. You should be able to see your disk image in your Synology NAS server. I hope this article helped you. Until next time!

Even when I do my best to ensure content is correctly rendered on RSS readers, some elements like custom info cards/notes and code style blocks can be shown without proper style or syntax highlighting.

You are free to continue reading as you prefer, but I recommend you to read this article in the website if possible. It's fast, has no ads, and minimal privacy-respecting analytics (browser, OS, country and device type) via Umami.

I’m a Linux user at heart, but I also like to play video games. So at the beginning of the year I decided to drop Windows 10 on my desktop, as it was about to reach its end of life for official support, and move to Manjaro Linux. One of the first things I did was to install Steam, and configure my old Xbox One controller to work with it.

I’ve been using an Xbox One controller with a dongle to play on Windows, and I wanted the same experience on Linux. I would just plug the dongle into my computer and start playing. The dongle I use is an old model like the one in the picture below. Microsoft released a new model later, that is less bulky. I guess these instructions will work for the new model as well, but I haven’t tested it.

Click on image to enlarge

Two options to configure the controller

There are two drivers out there to make the dongle work with Linux. One is xow, a mature yet old (not necessary a bad thing!) driver that has been around for a while. The other one is xone, made by the same author, but is newer and also supports Xbox Series X/S controllers and audio/led control with the controller.

I was unable to make the xone driver work with my controller, so I used the xow driver. Following now, I’ll indicate how I configured the controller to work with my system (Manjaro Linux, based on Arch Linux) and play with it on Steam. If you are interested in the xone driver, you can check the installation instructions directly in the repository. I linked both of them above.

Installing and configuring the xow driver

First, we need to install the packages required by xow.

sudo pacman -S curl cabextract libusb

Now, we need to clone the repository and run the installation script.

cd /tmp
git clone https://github.com/medusalix/xow
cd xow
make BUILD=RELEASE
sudo make install

We also need to install the firmware for the controller. Once xow is installed, it also installs a script in our path to get the controller firmware. So we can execute it as follows from anywhere.

sudo xow-get-firmware.sh

Lastly, xow will also install a systemd service to handle the dongle. It should be already started, but we can ensure it by running the following command.

sudo systemctl enable xow
sudo systemctl restart xow

At this point, we should be able to plug the dongle into our computer and recognized, but we still need to pair the controller with the dongle, even if it was paired with the dongle in another computer before (or in a previous Windows installation like my case).

Connecting and pairing the controller

First, let’s connect the dongle to the computer. If everything works you should see under the logs for the daemon that the dongle was initialized.

systemctl status xow
# Or alternatively for the logs only
# journalctl -u xow -f

In my case, it shows the following output:

● xow.service - Xbox One Wireless Dongle Driver
     Loaded: loaded (/etc/systemd/system/xow.service; enabled; preset: disabled)
     Active: active (running) since Sat 2025-02-01 16:05:31 CET; 2h 38min ago
 Invocation: 5704f28070f64b2eb864b645f1f9cdb9
   Main PID: 981 (xow)
      Tasks: 4 (limit: 38251)
     Memory: 1.1M (peak: 2.3M)
        CPU: 28ms
     CGroup: /system.slice/xow.service
             └─981 /usr/local/bin/xow

feb 01 16:05:31 manjaro systemd[1]: Started Xbox One Wireless Dongle Driver.
feb 01 16:05:35 manjaro xow[981]: 2025-02-01 16:05:35 INFO  - xow v0.5-36-gd335d60 ©Severin v. W.
feb 01 16:05:35 manjaro xow[981]: 2025-02-01 16:05:35 INFO  - Waiting for device...
feb 01 18:43:04 manjaro xow[981]: 2025-02-01 18:43:04 INFO  - Wireless address: XX:XX:XX:XX:XX:XX
feb 01 18:43:05 manjaro xow[981]: 2025-02-01 18:43:05 INFO  - Dongle initialized

Also, you can check the device is recognized by running lsusb. This is also a good place to start debugging if you have any issues. In my case, it shows the following (apart from other devices):

Bus 001 Device 005: ID 045e:02e6 Microsoft Corp. Xbox Wireless Adapter for Windows

If everything is working, you should be able to pair the controller with the dongle. To do this, as always, first long-press the pairing button on the dongle, see the white light of the dongle blinking. Then, long-press the pairing button on the controller. The Xbox logo should start blinking on the controller, and once the pairing is successful, the light will turn on steadily.

However, I had some problems when pairing the controller initially, let’s see them, so hopefully I can save you some time if you have the same issue.

Troubleshooting

When pairing the controller, dongle light was blinking, as so the controller light was blinking too. However, the controller light was not turning on steadily after the pairing was successful and the controller was not paired.

Checking the logs for the xow service, I saw the following:

INFO  - Pairing enabled
terminate called after throwing an instance of 'InputException'
what():  Error opening device: Permission denied

The old known permissions issue in Linux. It seemed that xow didn’t have the proper permissions to access the input devices. What I had to do to fix it was:

1. First, stop the xow service.

sudo systemctl stop xow

2. Then, copy the udev rules to the correct location. They are under the install directory of the repository.

sudo cp install/udev.rules /etc/udev/rules.d/99-xow.rules

3. Reload the udev rules.

sudo udevadm control --reload-rules
sudo udevadm trigger

4. Start the xow service again.

sudo systemctl start xow

And that’s it. After these changes, I repeated the pairing process, and it worked like a charm this time. Hope this helps you if you have the same issue.

Even when I do my best to ensure content is correctly rendered on RSS readers, some elements like custom info cards/notes and code style blocks can be shown without proper style or syntax highlighting.

You are free to continue reading as you prefer, but I recommend you to read this article in the website if possible. It's fast, has no ads, and minimal privacy-respecting analytics (browser, OS, country and device type) via Umami.

The two main operating systems for desktop/laptop computers out there, Windows and MacOS, allows you to encrypt your hard drive to protect your data from physical theft. And they do it in a way that is transparent for the end user. That is, you don’t need to do anything special to encrypt your hard drive, you just need to enable it in the system settings or during the installation process. And once it’s enabled, you won’t note it. Everything works as usual. You’ll power on your computer, and you will be presented with the login screen as always.

They achieve this by using a technology called BitLocker for Windows and FileVault for MacOS. Linux distributions are not far behind, and you can encrypt your root partition with LUKS (Linux Unified Key Setup). However, the end user experience is not the same, as, by default, you’ll be asked for the password of the LUKS partition when you power on your computer, even before of showing the login screen (when you will likely be asked for a password again, this time for your user account).

So how do Windows and MacOS achieve this? They store the key to decrypt the disc volume in a secure chip. There is a standard for this called Trusted Platform Module (TPM), that most Intel and AMD CPUs out there support. You can check an article I wrote last month about how to check TPM support and enable it on your computer. If you are curious, Apple implements this in their own T2 security chip.

So can we use the same technology to automatically mount the LUKS encrypted root partition on boot? Yes, we can! Let’s see how to do it.

Prerequisites

You will need to accomplish with the following prerequisites first:

• TPM 2.0 must be supported and enabled on your computer.
• A separated unencrypted /boot partition (apart from the usual separated /boot/efi partition if you are using UEFI).
• A LUKS encrypted root partition.
  • Although this tutorial is focused on automatically mounting a single LUKS encrypted root partition on boot, this will work for any number of LUKS encrypted partitions, like having / and /home on two different LUKS encrypted partitions. You just need to configure all the partitions to be mounted on boot as we’ll see later.

Note that a separated and unencrypted /boot partition is required. This is because your computer will need to be able to access the /boot partition, so it can load the kernel and initramfs, so that it can then load the encryption key from the TPM 2.0 chip. Otherwise, your computer will not be able to boot without asking for the password to even load GRUB or whatever bootloader you are using.

A security note

Some people will argue that having an unencrypted /boot partition is a security risk, and that it opens the door to a lot of potential attacks. For example, if an attacker gains physical access to your computer, they would build a malicious kernel and initramfs, so you will boot your computer with a compromised system that could be used to steal your information or spy on you. Like everything with security, you need to understand what is your threat model. What do I mean?

I’m writing this tutorial for people like me: people that have a computer at home, in a trusted environment, that just want to protect their data from physical theft. Like a burglar breaking into your house, and stealing your computer. I might store some sensitive information on my computer (passwords, private keys, financial information, etc.), that I don’t want to end up in the hands of a burglar. I don’t worry about a malicious actor gaining physical access to my computer, as I trust the people around me.

Again, understand your threat model, and decide if this is something you want to do. If you don’t trust your environment, you might want to fully encrypt your system and manually unlock it on every boot, either with a password or with a FIDO2 key. In that case, this tutorial might not be for you.

Automatically mount the LUKS encrypted root partition on boot

Note: As indicated in the prerequisites section of this article, I’ll assume you already have an encrypted LUKS partition, that you are probably unlocking manually with a password. I won’t explain how to create a LUKS encrypted partition in this article. You can easily encrypt your root partition when installing any Linux distribution in the installation wizard.

First, we need to install the required packages to make all of this work. I’m using Manjaro an european Arch Linux based distribution, so the packages will be slightly different if you are using a non-Arch Linux based distribution like Debian, Fedora or Ubuntu.

We’ll install the packages to interact with the TPM 2.0 chip, and clevis to bind the LUKS encrypted partitions to the TPM 2.0 chip. To make the initramfs automatically mount the LUKS encrypted partitions on boot from the TPM 2.0 chip, we’ll install the mkinitcpio-clevis-hook hook. This is provided as an AUR package, so we’ll need to use a package manager that supports AUR, like yay.

yay -S clevis clevis-luks tpm2-tss mkinitcpio-clevis-hook tpm2-tools

Now, we need to add the clevis hooks to the HOOKS variable in the /etc/mkinitcpio.conf file. The hook must be added before the encrypt hook, as indicated in the Arch Linux wiki. In my case, a fresh installation of Manjaro, it looks like this:

HOOKS=(base udev autodetect microcode kms modconf block keyboard keymap consolefont plymouth clevis encrypt filesystems)

Now, let’s recreate the initramfs.

sudo mkinitcpio -P

Lastly, let’s bind the LUKS encrypted partitions to the TPM 2.0 chip using the clevis tool. First, we need to know what is the encrypted partition that we want to bind with Luks. For that, we can use the lsblk command to list all the block devices and their partitions. This is the output of my system:

NAME                                          MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
nvme0n1                                       259:0    0 931,5G  0 disk
├─nvme0n1p1                                   259:1    0   300M  0 part  /boot/efi
├─nvme0n1p2                                   259:2    0     1G  0 part  /boot
└─nvme0n1p3                                   259:3    0 930,2G  0 part
  └─luks-bc7b9680-6567-45c4-a87a-f73e7ed0245e 254:0    0 930,2G  0 crypt /home
                                                                         /var/log
                                                                         /var/cache
                                                                         /

So in my case, the encrypted partition is /dev/nvme0n1p3. Now, let’s bind it to the TPM 2.0 chip. We’ll indicate to clevis that we want to use the TPM 2.0 chip, and that we want to use the PCR ID 7. PCR ID 7 will seal the LUKS key against the UEFI settings and the Secure Boot policy using the TPM 2.0 chip. So if the UEFI or Secure Boot settings are modified, the TPM will compute different PCR values and decryption will fail. This gives protection against evil maid attacks.

sudo clevis luks bind -d /dev/nvme0n1p3 tmp2 '{"pcr_ids": "7"}'

And that’s it! The partition is now bound to the TPM 2.0 chip, and will be automatically mounted on boot. You don’t need to do anything else, nor modify anything under /etc/fstab nor /etc/crypttab. Just for the records, my /etc/crypttab file looks like this:

# <name>                                  <device>                                  <password> <options>
luks-bc7b9680-6567-45c4-a87a-f73e7ed0245e UUID=bc7b9680-6567-45c4-a87a-f73e7ed0245e            none

You can check the status of the binding with the following command:

sudo clevis luks list -d /dev/nvme0n1p3

You should see something like this:

1: tpm2 '{"hash":"sha256","key":"ecc","pcr_bank":"sha1","pcr_ids":"7"}'

Now, let’s reboot your computer and see if it works. You should see the login screen as usual, and you won’t be asked for the password to decrypt the LUKS partition, just the credentials for your user account. The same user experience as Windows and MacOS!

Even when I do my best to ensure content is correctly rendered on RSS readers, some elements like custom info cards/notes and code style blocks can be shown without proper style or syntax highlighting.

You are free to continue reading as you prefer, but I recommend you to read this article in the website if possible. It's fast, has no ads, and minimal privacy-respecting analytics (browser, OS, country and device type) via Umami.

If you are like me and the ~63% of Windows users in the world as of December 2024, you are probably still using Windows 10 on your computer. And the reason why you have not upgraded to Windows 11 yet is probably because your computer apparently doesn’t support it. You’ll probably have a not too old computer, let’s say 5 years old components (like in my case), but still, Windows Update shows your computer is not compatible with Windows 11. How can this be?

Windows 11 requirements are not a crazy thing: 1 GHz or faster processor with at least 2 cores and 4 GB of RAM, so most of the computers out there should be compatible. The problem is that Windows 11 requires a Trusted Platform Module 2.0 (TPM 2.0) to be present and enabled on your computer. This is a hardware security chip that is used to store cryptographic keys and other security-related data, used mainly to encrypt the hard drive and secure the boot process.

Does that mean that you need to upgrade your hardware, that otherwise meets the requirements by far and it’s just a few years old, just to install Windows 11? Probably not. Big chances are that your motherboard or processor has a TPM 2.0 chip, but it’s disabled by default. In this article, I’ll show you how to enable it on your computer.

Does my motherboard support TPM 2.0?

Microsoft has been requiring hardware manufacturers to support TPM 2.0 since at least 10 years ago, in order to certify the components for Windows. And even before, some manufacturers have been adding it to their products. However, as in my case, it might be disabled by default in your BIOS.

For example, in my particular case, I have a MSI motherboard, and the TPM 2.0 chip is disabled by default, but already supported by all MSI motherboards since the Intel 100 series and AMD 300 series. As an example to illustrate how much hardware manufacturers have been supporting TPM 2.0, here is a table with the series of motherboards and processors that support it specifically for MSI motherboards:

So as you can see, an older hardware doesn’t mean that it doesn’t support TPM 2.0. It just means that you might have to enable it in your BIOS. If your motherboard is not from MSI, you might need to check your manufacturer website or your motherboard manual to find out if it supports TPM 2.0. To continue with this article, let’s see how to enable it on an MSI motherboard as this is the one I have.

Enabling TPM 2.0 on your old hardware

All you have to do, on an MSI motherboard, is to go to the Security tab, and then you will find the TPM settings under Trusted Computing submenu. You have to enable Security Device Support feature, and set one of the following options:

• For Intel motherboards, set TPM Device Selection to PTT.

Click on image to enlarge

• For AMD motherboards, set AMD fTPM Switch to AMD CPU fTPM.

Click on image to enlarge

Then, press F10 key to save the changes and exit. After doing this, you should be able to install Windows 11 on your computer.

Conclusion

As you can see, most of the old computers out there should be compatible with TPM 2.0, even if initially it seems like they don’t support it from Windows. You might just have to enable the TPM 2.0 feature in your BIOS, and you will be ready to go.

Now, you can install Windows 11 on your computer, or go the other way around, and install Linux with a LUKS encrypted root partition that automatically unlocks on boot. The choice is yours, will you take the blue pill or the red pill?

Even when I do my best to ensure content is correctly rendered on RSS readers, some elements like custom info cards/notes and code style blocks can be shown without proper style or syntax highlighting.

You are free to continue reading as you prefer, but I recommend you to read this article in the website if possible. It's fast, has no ads, and minimal privacy-respecting analytics (browser, OS, country and device type) via Umami.

Since UUID v7 came officially out, I’ve always tried to use it for primary keys on all new projects and databases. However, due to it being so new, it’s not supported by default on a lot of databases and libraries.

As I’m working recently, both professionally and personally, with Elixir and PostgreSQL, I’ll show you how do I use UUID v7 on PostgreSQL with Ecto in Elixir. Fully integrated with Ecto library and without the need for any external library. You will forget that you are using UUID v7 under the hood, because once configured,everything will be like working with the good old UUID v4.

Let’s get started!

Defining our UUIDv7 Ecto.Type module

In order to fully integrate UUID v7 with Ecto, we need to define a custom Ecto.Type module. This module will be responsible for converting the UUID v7 to a binary format and vice versa. In my case, I created a file under lib/my_app/uuid.ex as follows:

defmodule MyAPP.UUID do
  @moduledoc """
  A UUID v7 implementation and `Ecto.Type` for Elixir.

  Based on the RFC for the version 7 UUID: [RFC 9562](https://datatracker.ietf.org/doc/rfc9562/).

  Borrowed from https://github.com/martinthenth/uuidv7

  ## Usage

  In the database schema, change primary key attribute from `:binary_id` to `MyApp.UUID`:

  def MyApp.Schemas.User do
  @primary_key {:id, MyApp.UUID, autogenerate: true}
  end
  """

use Ecto.Type

@typedoc """
A hex-encoded UUID string.
"""
@type t :: <<\_::288>>

@typedoc """
A raw binary representation of a UUID.
"""
@type raw :: <<\_::128>>

@doc false
@spec type() :: :uuid
def type, do: :uuid

defdelegate cast(value), to: Ecto.UUID

defdelegate cast!(value),
to: Ecto.UUID

defdelegate dump(value), to: Ecto.UUID

defdelegate dump!(value), to: Ecto.UUID

defdelegate load(value), to: Ecto.UUID

defdelegate load!(value), to: Ecto.UUID

@doc false
@spec autogenerate() :: t()
def autogenerate, do: generate()

@doc """
Generates a version 7 UUID.
"""
@spec generate() :: t()
def generate, do: cast!(bingenerate())

@doc """
Generates a version 7 UUID based on the timestamp (ms).
"""
@spec generate(non_neg_integer()) :: t()
def generate(milliseconds), do: milliseconds |> bingenerate() |> cast!()

@doc """
Generates a version 7 UUID in the binary format.
"""
@spec bingenerate() :: raw()
def bingenerate, do: :millisecond |> System.system_time() |> bingenerate()

@doc """
Generates a version 7 UUID in the binary format based on the timestamp (ms).
"""
@spec bingenerate(non*neg_integer()) :: raw()
def bingenerate(milliseconds) do
  <<u1::12, u2::62, *::6>> = :crypto.strong_rand_bytes(10)
  <<milliseconds::48, 7::4, u1::12, 2::2, u2::62>>
end

@doc """
Extracts the timestamp (ms) from the version 7 UUID.
"""
@spec timestamp(t() | raw()) :: non*neg_integer()
def timestamp(<<milliseconds::48, 7::4, *::76>>), do: milliseconds
def timestamp(<<\_::288>> = uuid), do: uuid |> dump!() |> timestamp()
end

Now that we defined our Ecto.Type module, we can use it on our database schemas.

Using our custom Ecto.Type on our Ecto schemas

Let’s say we have a User schema with a primary key of type UUIDv7. We can define it as follows:

defmodule MyApp.Schemas.User do
  @moduledoc """
  User schema.
  """

  use Ecto.Schema

  @type t() :: %__MODULE__{
          id: MyApp.UUID.t(),
          email: String.t(),
          hashed_password: String.t(),
        }

  @primary_key {:id, MyApp.UUID, autogenerate: true}
  @foreign_key_type MyApp.UUID
  schema "users" do
    field :email, :string
    field :hashed_password, :string, redact: true

    timestamps(type: :utc_datetime)
  end
end

Nice! Now, let’s take a look at how to configure the repository to automatically use our custom UUIDv7 Ecto.Type module on our database migrations.

Configuring the repository and database migrations

We need to update our repository configuration, usually under config/config.exs, to use our custom UUIDv7 Ecto.Type module for autogenerated primary keys and IDs.

config :my_app,
  ecto_repos: [MyApp.Repo],
  generators: [
    timestamp_type: :utc_datetime,
    binary_id: {MyApp.UUID, :generate, []},
    migration_primary_key: {:id, :uuid, autogenerate: true},
    migration_timestamps: [type: :utc_datetime]
  ]

Then, our autogenerated database migrations created with mix ecto.gen.migration should look like this:

defmodule MyApp.Repo.Migrations.CreateUsersTables do
  use Ecto.Migration

  def change do
    execute "CREATE EXTENSION IF NOT EXISTS citext", ""

    create table(:users, primary_key: false) do
      add :id, :uuid, primary_key: true
      add :email, :citext, null: false

      timestamps(type: :utc_datetime)
    end

    create unique_index(:users, [:email])
  end
end

And that’s it! Now, you can use UUID v7 on PostgreSQL with Ecto in Elixir without the need for any external library.

Conclusion

As you can see, it’s very easy to use UUID v7 on PostgreSQL with Ecto in Elixir. You can forget that you are using UUID v7 under the hood, because once configured, everything will be like working with the good old UUID v4. Also, not needing any external library is a huge plus, as you are in control of the implementation and you can easily understand how it works under the hood. You remove potential security risks, potential deps problems, and you can easily integrate it with other libraries and tools that expect UUID v4.

I hope this helps you to start using UUID v7 on PostgreSQL with Ecto in Elixir. Happy coding!

Even when I do my best to ensure content is correctly rendered on RSS readers, some elements like custom info cards/notes and code style blocks can be shown without proper style or syntax highlighting.

You are free to continue reading as you prefer, but I recommend you to read this article in the website if possible. It's fast, has no ads, and minimal privacy-respecting analytics (browser, OS, country and device type) via Umami.

If you, like me, interact with AWS a lot via the CLI, you are probably getting tired of the CLI continuously asking you for your credentials or the OTPs generated by your MFA device. I use 1password to manage all my credentials, and although they provide a shortcut to retrieve credentials if you have the desktop app installed (Cmd+Shift+Space on my Mac), it is still a bit cumbersome to continuously searching for the OTPs.

As an engineer and developer, I like to automate things as much as possible. So I was happy when a coworker told me that the 1password CLI could be used to retrieve AWS credentials. This way, I can use my 1password vault to manage all my credentials, and the CLI to retrieve the temporary credentials when I need them.

This guide explains how to integrate the 1password CLI with the AWS CLI to retrieve your AWS credentials from 1password, including the MFA token, automatically for you. You’ll only need to authorize the operation when prompted, which usually will just require your fingerprint on your computer fingerprint reader, reducing a lot the manual work.

Before you start

Before you configure your local environment, ensure:

• You have a valid AWS account.
• You have installed the 1password CLI.
• You have installed the AWS CLI.

  • Version 2.0 or higher is required. You can check your AWS CLI version by running:

    aws --version
    

Configure main AWS credential retrieval from 1password

First, let’s see how to work normally with AWS CLI without having to store your credentials in your computer.

First, create a new item in 1password with the following fields:

• AWS access key.
• AWS secret access key.

You can use the same item to store your AWS user/password and also the MFA generator (OTP codes). Make sure the name of the item is something unique on your 1password items, and it’s easy to remember (something like aws credentials john.doe@email.com is just perfect).

Click on image to enlarge

Check that you can access your AWS account element from the command line:

# Replace <Your 1password item> with the name of the item you created in the previous step
op item get '<Your 1password item>' --format json | jq '.fields[]|.label'

Following the previous example, the command should be:

op item get 'aws credentials john.doe@email.com' --format json | jq '.fields[]|.label'

1Password will ask you for permission to check your 1password account. You can configure the amount of time you want to be asked for permission on the “developer settings” section of the 1password app.

Click on image to enlarge

Check the command output. It will enumerate the different fields the 1password entry has. It should be like this (it depends on your item layout).

"username"
"password"
"notesPlain"
"one-time password"
"access key"
"secret access key"

Let’s try to obtain the json snippet needed by aws-cli:

op item get '<1password item name>' --format json --fields 'label=access key','label=secret access key' | jq '{ "Version" : 1, "AccessKeyId": "\(.[] | select(.label=="access key") | .value )", "SecretAccessKey": "\(.[] | select(.label=="secret access key") | .value )"}'

The output should be something like this:

{
  "Version": 1,
  "AccessKeyId": "XXXXXXXXXXXXXXXXXXXX",
  "SecretAccessKey": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
}

Now, edit your ~/.aws/credentials file and add the following lines:

[1password]
credential_process = sh -c "op item get '<1password item name>' --format json --fields 'label=access key','label=secret access key' | jq '{ \"Version\" : 1, \"AccessKeyId\": \"\\(.[] | select(.label==\"access key\") | .value )\", \"SecretAccessKey\": \"\\(.[] | select(.label==\"secret access key\") | .value )\"}'"

Make sure you replace <1password item name> with the name of the item you created in the previous step. Also, make sure you comment or remove the aws_access_key_id and aws_secret_access_key lines from your ~/.aws/credentials file. Also, you must take care of using the correct fields in your 1password item. In my example, I have the access key and secret access key in the fields access key and secret access key.

Check you can access your AWS account element from the command line:

aws sts get-caller-identity --profile 1password

the output should be something like this:

{
  "UserId": "xxxxxxxxxxxxxxxxxxxx",
  "Account": "xxxxxxxxxxxx",
  "Arn": "arn:aws:iam::xxxxxxxxxxxx:john/john.doe@email.com"
}

Configure MFA retrieval from 1password

Now, let’s configure AWS CLI to retrieve the OTP for 2FA from 1password. Other tools such as kubectl use AWS CLI under the hood, so this will also configure your MFA retrieval for other tools (like the already mentioned kubectl if you work with kubernetes).

Additionally, the plugin we are going to install will allow you to use two new commands: open-console and set-env that helps a lot to manage your AWS access.

First, check your ~/.aws/config file. You should have something like this:

[default]
region = us-east-1 # Or your default region
output = json

[profile 1password]
region = us-east-1 # Or your default region
output = json

[profile example-profile]
role_arn= # ...
source_profile=1password
mfa_serial = # ...
region = us-east-1 # Or your default region

Test you have access to that profile:

aws sts get-caller-identity --profile example-profile

The output should be something like this:

{
  "UserId": "xxxxxxxxxxxxxxxxxxxx",
  "Account": "xxxxxxxxxxxx",
  "Arn": "arn:aws:sts::xxxxxxxx:assumed-role/IAM_ROLE_NAME/xxxxxxxx"
}

Now, clone the awscli-plugin-op repository from my coworker. In my case, I cloned it directly inside a new plugins/ folder inside ~/.aws/:

cd ~/.aws
mkdir plugins
cd plugins
git clone https://github.com/srlobo/awscli-plugin-op.git

The result directory structure should be like this:

~/.aws/
 |-- plugins/
 |    |-- awscli-plugin-op/
 |    |   |-- README.md
 |    |   |-- awscli_plugin_op/
 |    |   |   |-- *.py

Then, configure the plugin:

1. Add the following lines to the end of your ~/.aws/config file:

[plugins]
op_totp = awscli_plugin_op
# Replace with the absolute path to the cloned repo
cli_legacy_plugin_path = /Users/john.doe/.aws/plugins/awscli-plugin-op

2. Also, in the same file, add the op_identity property on each profile you want to use the plugin with (usually you want to use it on all the profiles):

op_identity = <your 1password item name>
# Without quotes. Following the previous example:
# op_identity = aws credentials john.doe@email.com

Your ~/.aws/config file should look like this:

[default]
region = us-east-1
output = json

[profile 1password]
region = us-east-1
output = json
op_identity = <your 1password item name>

[profile example-profile]
role_arn= # ...
source_profile=1password
mfa_serial = # ...
region = us-east-1
op_identity = <your 1password item name>

[plugins]
op_totp = awscli_plugin_op
cli_legacy_plugin_path = /Users/john.doe/.aws/plugins/awscli-plugin-op

With the [plugins] section, you are telling the AWS CLI to use the plugin. Then, each op_identity attribute is telling the plugin what 1password item hold the MFA data. Note the op_identity field is literal, don’t put quotes around it.

Finally, test the new configuration:

aws sts get-caller-identity --profile example-profile

If you have configured the plugin correctly, the system will ask you for the 1password password or just for the biometric authentication.

Click on image to enlarge

The output should be something like this:

{
  "userid": "xxxxxxxxxxxxxxxxxxxx",
  "account": "xxxxxxxxxxxx",
  "Arn": "arn:aws:sts::xxxxxxxx:assumed-role/IAM_ROLE_NAME/xxxxxxxx"
}

Congratulations! You have successfully configured your AWS CLI to retrieve your credentials from 1password!

Even when I do my best to ensure content is correctly rendered on RSS readers, some elements like custom info cards/notes and code style blocks can be shown without proper style or syntax highlighting.

You are free to continue reading as you prefer, but I recommend you to read this article in the website if possible. It's fast, has no ads, and minimal privacy-respecting analytics (browser, OS, country and device type) via Umami.

As you might already know, Elixir is built on top of the Erlang Virtual Machine (BEAM). This means that Elixir and Erlang share the same underlying runtime system, allowing for seamless interoperability between the two languages. Elixir code compiles to Erlang bytecode, which runs on the BEAM, making it possible to use Erlang libraries and modules directly within Elixir projects.

This interoperability allows Elixir developers to leverage the vast ecosystem of Erlang libraries and applications without the need for translation or rewriting. You can directly use existing Erlang applications in your Elixir code by calling their functions using the atom syntax. Let’s see how it works.

Using Erlang applications directly in Elixir

To call an Erlang function from Elixir, you use the syntax :module.function(args). The colon before the module name tells Elixir that you’re referring to an Erlang module. This works out of the box for modules included in the Erlang standard library.

For example, if you want to use the base64 module from Erlang’s standard library, you can do:

encoded = :base64.encode("Hello, World!")
decoded = :base64.decode(encoded)

Now, let’s see how to use an Erlang application that is not included in the Erlang standard library. Erlang applications are typically started automatically when your Elixir application starts, as long as they are listed in your mix.exs file under the :extra_applications key. For example:

def application do
  [
    extra_applications: [:logger, :crypto]
  ]
end

This ensures that the Erlang crypto application is started along with your Elixir application. After that, you can use its functions in your Elixir code. Here’s an example of using the crypto module to generate a random string:

# Generate 16 random bytes
random_bytes = :crypto.strong_rand_bytes(16)

# Convert the bytes to a hexadecimal string
random_string = Base.encode16(random_bytes, case: :lower)

IO.puts("Random string: #{random_string}")

This code generates a random 16-byte string and encodes it as a lowercase hexadecimal string. The :crypto.strong_rand_bytes/1 function is an Erlang function from the crypto application, called directly from Elixir code.

But what if you want to directly translate Erlang code to Elixir code? Well, Elixir and Erlang are not 100% compatible, but they are very similar, so you can translate Erlang code to Elixir with a few changes. Let’s see how it works.

Translating Erlang code to Elixir code

Translating Erlang code to Elixir is easy as long as you keep in mind the following things when transcoding Erlang to Elixir:

• Erlang atoms start with a lowercase letter, whereas Elixir atoms start with a colon (:). For example, ok in Erlang becomes :ok in Elixir.
• Erlang variables start with an uppercase letter, whereas Elixir variables start with a lowercase letter. For example, Socket in Erlang becomes socket in Elixir.
• Erlang modules are always referenced as atoms. For example, gen_tcp in Erlang becomes :gen_tcp in Elixir.
• Function calls in Erlang use a colon (:) whereas function calls in Elixir always use a dot (.). For example, gen_tcp:listen in Erlang becomes :gen_tcp.listen in Elixir. (Replace the colon with a dot.)
• Last, but no least, it’s important to note that Erlang strings aren’t the same as Elixir strings. In Erlang, a double-quoted string is a list of characters whereas in Elixir a double-quoted string is a sequence of bytes (a binary). Thus, double-quoted Erlang and Elixir strings aren’t compatible. So if an Erlang function takes a string argument, you can’t pass it an Elixir string. Instead, Elixir has a character list which you can create using single-quotes rather than double-quotes. For example, 'hello' is a list of characters that’s compatible with the Erlang string "hello".

Let’s see how it works with an example. Let’s translate the server example from the docs for generic TCP/IP server in Erlang (gen_tcp) to Elixir code.

We start with the following Erlang code, that defines the server function:

server() ->
    {ok, LSock} = gen_tcp:listen(5678, [binary, {packet, 0}, {active, false}]),
    {ok, Sock} = gen_tcp:accept(LSock),
    {ok, Bin} = gen_tcp:recv(Sock, 0),
    ok = gen_tcp:close(Sock),
    ok = gen_tcp:close(LSock),
    Bin.

Now, translated to Elixir code, it looks like this:

def server() do
    {:ok, lsock} = :gen_tcp.listen(5678, [:binary, packet: 0, active: false])
    {:ok, sock} = :gen_tcp.accept(lsock)
    {:ok, bin} = :gen_tcp.recv(sock, 0)
    :ok = :gen_tcp.close(sock)
    :ok = :gen_tcp.close(lsock)
    bin
end

Some notes about the changes made:

• When translating the first function call to :gen_tcp.listen, the arguments received by that function are translated as [:binary, {:packet, 0}, {:active, false}]), which can be simplified to a Keyword list in elixir, as Keyword list are just list of tuples.
• Erlang instructions always ends with a comma. Think of them as the semicolon (;) in other languages such as C or Java. Since Elixir doesn’t use any character to separate instructions in different lines you need to remove them.
• The end of a function in Erlang is marked by a dot (.). In Elixir, it’s marked by the end keyword.

Conclusion

And that’s it! You now know how to use Erlang applications in your Elixir codebase, and also how to translate Erlang code to Elixir code. You can copy paste the above code in your Elixir project and run it.

Even when I do my best to ensure content is correctly rendered on RSS readers, some elements like custom info cards/notes and code style blocks can be shown without proper style or syntax highlighting.

You are free to continue reading as you prefer, but I recommend you to read this article in the website if possible. It's fast, has no ads, and minimal privacy-respecting analytics (browser, OS, country and device type) via Umami.

In the old days of web development, interacting with a web application meant reloading the entire page every time you wanted to update the user interface. This was slow, inefficient, and made for a poor user experience. Any interaction with the website, no matter how small it was, would require a round trip to the server, which would then generate a new HTML page and send it back to the client, usually a web browser. Then, the web browser would load the new page from zero, to show the updated content.

From a few years back to now, interactive web applications have been built using JavaScript frameworks like React, Angular, or Vue. These frameworks allow developers to build dynamic web applications that can update the user interface without reloading the page. The round trip to the server is still there, but only to fetch the data needed to update the user interface. Now, the server, instead of returning HTML, usually returns only the information needed to the frontend as a JSON document. The user interface is then updated using JavaScript, which is executed in the web browser. JavaScript then updates the DOM of the page and only the affected component changes, without requiring a full page load.

As you probably know by your experience as a developer, this approach does the work, but it has its own set of problems. The most common one is that you need to write and maintain two separate codebases: one for the frontend and one for the backend. This can be a problem because you need to keep both codebases in sync, which can be a source of bugs and inconsistencies. Also, you need to know two different programming languages and frameworks, which can be a barrier for some developers. Yes, you can use node for the backend, which will allow to use the same language for both codebases, but still even then some things has very different considerations.

What is Phoenix LiveView?

Phoenix LiveView is an elixir library that provides a new approach to building interactive web applications that aims to solve these problems. It allows you to build interactive web applications using only Elixir and Phoenix, without writing a single line of JavaScript. It achieves that by using server-side rendering and WebSockets.

LiveViews run in a Phoenix server, which can scale to handle millions of WebSocket connections, and it has built-in presence tracking so it knows who’s connected, and a built-in PubSub system for broadcasting real-time updates to LiveViews. And all this rests on a rock-solid foundation of Elixir, OTP, and Erlang.

So it provides an unrivaled stack for building massively scalable, multi-user, interactive, real-time distributed web apps faster and with less code.

How does Phoenix LiveView work?

A LiveView page is initially rendered as a static HTML page via a request-response cycle (the browser sending the habitual GET request to the server to load a page). Then a persistent WebSocket connection is automatically opened between the browser and a stateful LiveView process running on the server. This WebSocket connection is opened by a JavaScript worker loaded on the initial page load and it’s bundled with LiveView. We’ll go back to it later.

And what about the interaction? Any interaction on the web application triggers events that are then pushed down the WebSocket to the LiveView process, which causes its state to change. After those state changes, the LiveView process only re-renders the parts of the page that are affected. Those HTML diffs are then sent back to the browser via the WebSocket as a response. Then, our JavaScript worker patches the DOM.

All this happens out of the box. So as the LiveView process receives events from the GUI, changes state, and re-renders, everything is automatically kept in sync without having to write any JavaScript or manage WebSockets.

LiveView Life Cycle

Let’s walk through the life cycle of a LiveView application. When a user initially browses to the website, a regular HTTP GET request is sent to the server, and we have a live route declared in the router for that.

defmodule MyAppWeb.Router do
  use Phoenix.Router
  import Phoenix.LiveView.Router

  scope "/", MyAppWeb do
    live "/counter", CounterLive
  end
end

That route indicates that the CounterLive module should be used to handle the request. The CounterLive module is a LiveView module that is responsible for rendering the page and handling events.

defmodule MyAppWeb.CounterLive do
  # In Phoenix v1.6+ apps, the line is typically: use MyAppWeb, :live_view
  use Phoenix.LiveView

  def render(assigns) do
    ~H"""
    <div>
      Current value: <%= @counter %>
      <button phx-click="inc_counter">Increment counter</button>
    </div>
    """
  end

  def mount(_params, _session, socket) do
    counter = 0
    {:ok, assign(socket, :counter, counter)}
  end

  def handle_event("inc_counter", _params, socket) do
    {:noreply, update(socket, :counter, &(&1 + 1))}
  end
end

Then, in the CounterLive module, the mount callback is invoked, which assigns the initial state to the socket. In this case, sets its value to zero. Then render is automatically invoked with the state that was assigned to the socket in mount, and a fully-rendered HTML page is sent back to the browser as a regular HTTP response. Handling the initial request in this way has a few important benefits.

• First, the response is super quick.
• Second, you get a fully-rendered, meaningful HTML page, even if JavaScript is disabled in the browser. And so the initial LiveView page is search engine friendly.

Nothing too surprising so far, but here’s where things get interesting. When the initial page is loaded, it also loads a sliver of JavaScript. It’s an app.js, which opens a persistent WebSocket connection to the server (do you remember I already told you about a JavaScript worker?). It’s at this point that a stateful LiveView process is spawned. Mount is then invoked again, this time inside of the stateful process, and initializes the state of that process by assigning values to the socket. Then as you probably already guessed, render is also invoked again to render HTML content for that state. But this time, it’s not the raw HTML that’s pushed back to the browser over the WebSocket, as you might expect. It’s actually something more intriguing and efficient.

Let’s focus on this section of the HEEx template

~H"""
<div>
  Current value: <%= @counter %>
  <button phx-click="inc_counter">+</button>
</div>
"""

Since the assigned counter value is interpolated in this template, we have one dynamic value (@counter), a value that may or may not have changed, and the rest of the template is static. It will never change. So, LiveView splits the template into parts, the stuff that’s dynamic and the stuff that’s static. The dynamic values evaluates to 0 since that’s the initial counter. You can think of this value as being at position or index zero of the template, and all the static and dynamic parts are pushed to the browser over the WebSocket. Then the JavaScript provided by LiveView weaves the static and dynamic parts together.

Splitting the rendered content into static and dynamic parts really pays off when handling events. For example, when we click the button to increment the counter, an inc_counter event is pushed down the WebSocket to the LiveView process and gets handled by a matching handle_event callback. A new value of counter + 1 is assigned to the socket, and whenever a LiveView state changes, the render function is automatically called. Since handling the on event only changed the counter value, only these template expressions need to be evaluated. If the LiveView had other pieces of state rendered by the template, they would only be evaluated if they changed when the counter was incremented.

So what’s sent to the browser this time? Well, the static parts aren’t sent again. They’re already cached in the browser. Only the dynamic value that changed or the diffs get sent. So, as before, all the LiveView JS needs to do is zip the static and dynamic parts together, and then it uses the morphdom library to efficiently patch the DOM to increase the counter.

But how will this scale?

It’s a fair question, especially if you’re new to the Elixir/Phoenix platform. Since each stateful LiveView runs in a separate process, depending on the number of users you could have thousands, hundreds of thousands, or even millions of processes. And all the communication happens over WebSockets.

A lot of systems would buckle under these conditions, but Elixir and Phoenix are uniquely suited for it. And by riding atop this battle-tested platform, LiveView is in a league of its own.

The Road to 2 Million Websocket Connections has a lot of juicy details.

Even when I do my best to ensure content is correctly rendered on RSS readers, some elements like custom info cards/notes and code style blocks can be shown without proper style or syntax highlighting.

You are free to continue reading as you prefer, but I recommend you to read this article in the website if possible. It's fast, has no ads, and minimal privacy-respecting analytics (browser, OS, country and device type) via Umami.

If you, like me, have a Synology NAS and use Synology Photos to manage your photos, you might have faced the problem of adding existing photo files to Synology Photos from another folder in your NAS (different and outside of your Synology Photos folder) without actually coping or duplicating them, which will took more disk space. In this article, I’ll show you exactly that, but you will need a volume with the BTRFS file system.

Context, what was my problem I was trying to solve

I recently returned from a long trip to England and Scotland. I took tons of photos with my camera, as did my girlfriend with her phone. Back at home, I wanted to create a macro album of all the photos we took in Synology Photos, that is what we use at home for our memories together. They were thousands of photos (+5000!), and they were in tons of different folders. I like to separate photos I took with my camera in folders by day and location/activity, all inside my user’s Synology Photos folder. The photos my girlfriend took with her phone are in her MobileBackup folder within her user space. I took a few photos with my phone too, and they are in my own MobileBackup folder.

We could manually select the photos and add them to a shared album, and even when you can select multiple photos from a folder to add to an album at once, you must select all of them folder by folder. You cannot select a folder and add all photos there to an album. So as I had a lot of folders with photos because of how I organize the ones took with the camera, it was a tedious process. And also, I have set Synology Photos to show RAW files from the camera, but I didn’t those to appear in the macro album, so I would need to manually discriminate RAW files from JPEGs manually for thousands of photos, which is inviable.

So I decided to create a conditional album and select the folders in which I have all my camera photos, plus my MobileBackup folder and my girlfriend’s MobileBackup folder, and filter it to do not include RAW files. And here is where the problem arises: I could not access my girlfriend’s MobileBackup folder from my user space. Also, my girlfriend, that has a much simpler setup than me (all photos being .heic and being in the same folder, so could multi-select and add at once), could not manually add her photos to the album, as conditional albums doesn’t allow manually adding photos.

So after a lot of Google searches to know what could I do to fix this problem, I found this Reddit post.

What didn’t work

My initial idea, as a long time Linux nerd and user, was to use symlinks to link her files to my user space, that I should be able to read as we share a linux user group for that. But Synology Photos doesn’t follow symlinks, so that didn’t work.

As I saw in the Reddit post from Arnout, he tried also a bid mount. This achieves something similar, but requires the mount to be set up at boot every time (in /etc/fstab or in some boot scripts such as rc.local). This is not as clean, but can be used as a sort of a hack to fool applications which otherwise would not want to follow symlinks). However, this didn’t work neither.

This leaves us with two possible solutions: hard links and BTRFS CoW (reflink), which are filesystem-specific features.

The solution

I also had a BTRFS volume, so I decided to try the reflink method. What this does is that it “duplicates” the file, but it doesn’t actually duplicate the data, it just creates a new metadata entry for the file, and the data is shared between the two files. Once one of the files get modified, then an actual copy is made and they stop sharing data. This is a feature of BTRFS, and it is called Copy-on-Write (CoW). This is the same feature that allows you to take snapshots of your filesystem without actually duplicating the data.

To achieve this, we can just use the cp command:

cp --reflink=always -R /volume1/teammap/oldbackupfolder /volume1/photo/oldpictures

In my case, what I did was:

cp --reflink=always -R \
  /volume1/homes/girlfriend/Photos/MobileBackup/iPhone/2024/08/ \
  /volume1/homes/myuser/Photos/Girlfriend_UK_Trip

Then, I was able to select Girlfriend_UK_Trip folder to the conditional album, and all photos were there, without actually duplicating the space taken.

Caveats

There a few caveats to know here:

• This only works with BTRFS volumes. If you don’t have a BTRFS volume, you can’t use this method. However, hard links might work, but I didn’t test it.
• Files need to be on the same volume. If you want to link files from different volumes, you will need to use a different method.
• This won’t be useful if files are being modified constantly, or might be modified in the future. If you modify one of the files in the original folder, the CoW in the photos folder will not get the changes, and it will occupy new extra space, as the data is not shared anymore.

If this is something you can live with, then you can use this method to link or import existing photo files to Synology Photos folder without duplicating space on BTRFS. Otherwise, it might be worthy to study other solutions, such as a shared folder between users.

Now what?

That’s all. Thanks for reading this article! If you, like me, are in love with your Synology NAS, you might find interesting other articles I wrote for it. Check them out!

Even when I do my best to ensure content is correctly rendered on RSS readers, some elements like custom info cards/notes and code style blocks can be shown without proper style or syntax highlighting.

You are free to continue reading as you prefer, but I recommend you to read this article in the website if possible. It's fast, has no ads, and minimal privacy-respecting analytics (browser, OS, country and device type) via Umami.

Binary Search Trees (BSTs) are a fundamental data structure in computer science. They provide efficient methods for storing, searching, and managing data. In this article, we’ll explore what BSTs are, how they work, and how to implement them.

What is a Binary Search Tree?

A Binary Search Tree is a binary tree where each node has at most two children, referred to as the left child and the right child. For each node:

• The left subtree contains only nodes with values less than the node’s value.
• The right subtree contains only nodes with values greater than the node’s value.

This property makes BSTs useful for operations like searching, insertion, and deletion.

Okay, now that the basics are covered, let’s see how to code and work with a BST! I’ll use Python for this example, just because it’s very popular and looks like pseudo-code, but the concepts are the same for other programming languages.

Code definition

To create a BST data structure, we just need to model the nodes of the graph. Each node will have a value, and pointers to the left and right children, like so:

class Node:
    def __init__(self, value):
        self.value = value
        self.left = None
        self.right = None

So to create a tree, we just create the root node with a value:

tree = Node(10) # 10 is the root node

And that’s it! Now, let’s see how to implement the basic operations of a BST. We don’t like anemic models, so we’ll implement the insertion, search, and deletion operations as methods of the Node class. Let’s go!

Basic Operations

As I like to learn by example, let’s take a look at how to implement a BST in Python.

Insertion

To insert a value into a BST, we start at the root and compare the value to be inserted with the current node’s value. If the value is less, we move to the left child; if greater, we move to the right child. We repeat this process until we find an empty spot where the new value can be inserted.

def insert(self, root, key):
    # If the root is None, create a new node with the key and return it
    if root is None:
        return Node(key)
    else:
        # If the key is already present in the tree, return the root
        if root.value == key:
            return root
        # If the key is greater than the root's value, insert it in the right subtree
        if root.value < key:
            root.right = self.insert(root.right, key)
        # If the key is less than the root's value, insert it in the left subtree
        else:
            root.left = self.insert(root.left, key)
    # Return the root after insertion
    return root

Let’s see this in action!

tree = Node(10)
tree.insert(tree, 5)
tree.insert(tree, 15)
tree.insert(tree, 3)
tree.insert(tree, 7)
tree.insert(tree, 12)
tree.insert(tree, 18)

Click on image to enlarge

Search

To search for a value in a BST, we start at the root and compare the value with the current node’s value. If the value matches, we return the node. If the value is less, we search the left subtree; if greater, we search the right subtree. We repeat that algorithm until we find the node or the node or we end in a leaf node.

def search(self, root, key):
    # Base case: return the node if it's found or None if it's not
    if root is None or root.value == key:
        return root
    # If the key is greater than the root's value, search in the right subtree
    if root.value < key:
        return self.search(root.right, key)
    # If the key is less than the root's value, search in the left subtree
    return self.search(root.left, key)

This will return the node we’re looking for, or None if the node is not found.

node = tree.search(tree, 12)
print(node and node.value) # 12

Click on image to enlarge

Deletion

Deleting a node from a BST involves three main cases:

1. The node to be deleted is a leaf node.
2. The node to be deleted has one child.
3. The node to be deleted has two children.

If the node to be deleted is a leaf node, we just need to remove the node from the tree. However, if the node has one or two children, we need to handle the deletion differently. For a node with one child, we replace the node with its child. For a node with two children, we find the inorder successor (the smallest node in the right subtree), replace the node’s value with the inorder successor’s value, and then delete the inorder successor.

The inorder successor is the smallest node in the right subtree of the node to be deleted (i.e., the leftmost node in the right subtree, or the following node if we order them from lowest to highest).

def delete(self, root, key):
    # Base case: if the root is None, return None
    if root is None:
        return root
    # If the key to be deleted is smaller than the root's key,
    # then it lies in the left subtree
    if key < root.value:
        root.left = self.delete(root.left, key)
    # If the key to be deleted is greater than the root's key,
    # then it lies in the right subtree
    elif key > root.value:
        root.right = self.delete(root.right, key)
    # If the key is the same as the root's key, then this is the node
    # to be deleted
    else:
        # Node with only one child or no child
        if root.left is None:
            return root.right
        elif root.right is None:
            return root.left
        # Node with two children: Get the inorder successor
        # (smallest in the right subtree)
        temp = self.minValueNode(root.right)
        # Copy the inorder successor's content to this node
        root.value = temp.value
        # Delete the inorder successor
        root.right = self.delete(root.right, temp.value)
    return root

def minValueNode(self, node):
    # Loop to find the leftmost leaf
    current = node
    while current.left is not None:
        current = current.left
    return current

Let’s see this in action, this time, with a bigger tree. This is how our BST initially looks like:

Click on image to enlarge

Now, let’s delete the node with value 5.

tree = tree.delete(tree, 5)

The resulting tree is a follows:

Click on image to enlarge

Caveats

The efficiency of a BST depends on its balance. If the tree is unbalanced, the search, insertion, and deletion operations can take O(n) time, where n is the number of nodes in the tree. Deletions can affect the balance of the tree, but also insert orders. THe resulting tree will differ depending on the order of insertions. For example:

tree = Node(10)
tree.insert(tree, 5)
tree.insert(tree, 15)
tree.insert(tree, 3)
tree.insert(tree, 7)
tree.insert(tree, 12)
tree.insert(tree, 18)
tree.insert(tree, 6)
tree.insert(tree, 8)
tree.insert(tree, 2)
tree.insert(tree, 4)

will produce a tree like this:

Click on image to enlarge

But if we change the order of insertion like so:

tree = Node(10)
tree.insert(tree, 18)
tree.insert(tree, 15)
tree.insert(tree, 12)
tree.insert(tree, 8)
tree.insert(tree, 7)
tree.insert(tree, 6)
tree.insert(tree, 5)
tree.insert(tree, 4)
tree.insert(tree, 3)
tree.insert(tree, 2)

graph TD;
A([10]) --> B([18]);
A --> C([7]);
B --> D([15]);
B --> L([X]);
D --> E([12]);
D --> M([X]);
C --> F([8]);
C --> G([5]);
F --> N([X]);
G --> H([6]);
G --> I([2]);
H --> O([X]);
I --> J([4]);
I --> K([3]);
J --> P([X]);
K --> Q([X]);

We end up with a quite different tree:

Click on image to enlarge

Sometimes, it might be needed to balance the tree when there is a lot of unbalanced nodes, like in the last example. A simple way to improve the balance of a tree is the following:

def balanceBST(self, root):
    # Helper function to store the in-order traversal of the BST
    def storeInOrder(node, inorder_nodes):
        if not node:
            return
        storeInOrder(node.left, inorder_nodes)
        inorder_nodes.append(node.value)
        storeInOrder(node.right, inorder_nodes)

    # Helper function to build a balanced BST from sorted nodes
    def buildBalancedBST(nodes, start, end):
        if start > end:
            return None
        mid = (start + end) // 2
        node = Node(nodes[mid])
        node.left = buildBalancedBST(nodes, start, mid - 1)
        node.right = buildBalancedBST(nodes, mid + 1, end)
        return node

    # Store the in-order traversal of the BST
    inorder_nodes = []
    storeInOrder(root, inorder_nodes)

    # Build and return the balanced BST
    return buildBalancedBST(inorder_nodes, 0, len(inorder_nodes) - 1)

Conclusion

Binary Search Trees are a powerful tool for managing ordered data. They provide efficient algorithms for insertion, search, and deletion, making them a fundamental component in computer science. Understanding BSTs and their operations is crucial for any programmer looking to master data structures and algorithms.

All the code from this article can be found on Github.

Even when I do my best to ensure content is correctly rendered on RSS readers, some elements like custom info cards/notes and code style blocks can be shown without proper style or syntax highlighting.

You are free to continue reading as you prefer, but I recommend you to read this article in the website if possible. It's fast, has no ads, and minimal privacy-respecting analytics (browser, OS, country and device type) via Umami.

In Go, like the majority of programming languages (like C, C++, Rust, etc.), when you assign a value to a variable, that value is stored at a specific memory address in your computer.

You can use the reference operator & to find out this address, as shown below:

package main

import "fmt"

func main() {
    var myVar int = 35

    fmt.Println(&myVar) // Prints a memory address. Ex: 0xc000012028
}

Running this code on my computer, I get an output like 0xc000012028, which represents the memory address of the variable myVar (displayed in hexadecimal). If you run this code on your own machine or in the Go playground, you’ll likely see a different value.

Understanding pointers. What are they?

When you apply the & operator to a variable, it returns a pointer. A pointer is simply a variable that holds the memory address of another variable. You can imagine pointers as “pointing to” a specific location in memory.

Pointers in Go have types, just like the variables they reference. For example, a pointer of type *int can only store the memory address of an int variable, and a pointer of type *string can only store the memory address of a string variable.

This may seem a bit complex, so let’s clarify by modifying the previous example to assign the address of myVar to a new variable instead of printing it. To make it clear, I will use explicit variable declarations without type inference.

package main

import "fmt"

func main() {
    // Declare myVar variable with the type int and assign the value 35.
    var myVar int = 35


    // Declare myVarPtr variable with the type *int. Use the & operator to
    // get a pointer to the myVar variable and assign it as the value.
    var myVarPtr *int = &myVar

    fmt.Println(myVarPtr) // Prints a memory address. Ex: 0xc000012028
}

In this case, the variable myVarPtr is a pointer of type *int, holding the memory address of the myVar variable.

We can rewrite this using the more common := shorthand for variable declarations, like this:

package main

import "fmt"

func main() {
    myVar := 35
    myVarPtr := &myVar

    fmt.Println(myVarPtr)
}

Using a pointer to access a value (dereferencing)

The dereference operator * allows you to read or change the value stored at the address a pointer refers to. This is also called indirection.

Here’s an example of how to use the dereference operator to read a value:

package main

import "fmt"

func main() {
    myVar := 35
    myVarPtr := &myVar

    fmt.Println(myVarPtr)  // Prints a memory address. Ex: 0xc000012028
    fmt.Println(*myVarPtr) // Prints 35
}

In the above example, myVarPtr is a pointer to the myVar variable. By using *myVarPtr, we can access the value of myVar and print it out.

We can also use the dereference operator to modify the value at the memory address. For example:

package main

import "fmt"

func main() {
    myVar := 35
    myVarPtr := &myVar

    *myVarPtr = 88     // Use the dereference operator to assign a new value
    fmt.Println(myVar) // Prints 88
}

Wrapping up

That’s the basics of pointers in Go. They are just variables that store memory addresses of other variables. You can use the reference operator & to get the memory address of a variable and the dereference operator * to access or modify the value stored at that address. This is useful when you need to pass a variable by reference to a function or when you want to modify the value of a variable from another function, or when you want to avoid copying large data structures.

Even when I do my best to ensure content is correctly rendered on RSS readers, some elements like custom info cards/notes and code style blocks can be shown without proper style or syntax highlighting.

You are free to continue reading as you prefer, but I recommend you to read this article in the website if possible. It's fast, has no ads, and minimal privacy-respecting analytics (browser, OS, country and device type) via Umami.

Hello everybody! As someone of you may already know, I’m from the European Union, from Spain, and I’m a big fan of free open-source software. Today, I want to talk about a very interesting initiative from the Free Software Foundation Europe (FSFE) called “Public Money, Public Code”. This initiative advocates that all software developed using public funds (paid from taxes) should be made available as Free and Open Source Software (FOSS).

European citizens will vote next 9th of June for the European Parliament, and I think it’s a good time to talk about this initiative that could benefit all of us, not only in Europe but worldwide. You know that some tech-legal initiatives that start in the EU end coming or affecting to the US and the rest of the world too! Let’s dive into why this is important and how it can benefit everyone.

What is “Public Money, Public Code”?

As said, it’s all about making sure that software funded by the public is available to the public under free and open-source licenses. In other words, code paid by the people should be available to the people!

What are the advantages?

So, what’s at the heart of this initiative? Here are the main points:

• Transparency: With open-source software, anyone can look at the code. This means bugs and security issues get spotted and fixed faster. Plus, it makes government operations way more accountable.

• Cost-Efficiency: Sharing is caring, especially with code. Different government agencies can reuse and share software, cutting down on costs and avoiding repetitive spending.

• Innovation: Open-source encourages collaboration. Developers worldwide can pitch in, improving public software and creating better solutions.

• Sovereignty: Governments can break free from being dependent on big, proprietary software companies. This means more control over their digital tools and fewer worries about vendor lock-in.

Why Should You Care?

Think about all the digital services we rely on daily – e-government, healthcare systems, public transportation apps, online education platforms. These run on software that needs to be secure and efficient. Proprietary software can be limiting and expensive, tying up government funds that could be better spent elsewhere.

Free and open-source software (FOSS) offers a brilliant alternative. It’s all about open access: anyone can see, modify, and propose improvements the code. This openness doesn’t just save money; it creates a more resilient and adaptable tech ecosystem. Plus, it’s a win for democracy, giving citizens more insight into how their governments operate and how their data is handled.

Real-World Wins

Several places have already jumped on the “Public Money, Public Code” bandwagon with great results:

• Spain: During the COVID-19 pandemic, the Spanish government launched and app to track the spread and exposure to the virus by the Spanish citizens. The app used Bluetooth technology to track the proximity of the users to other users, and to link infected people with the people that have been in contact with them. As you can imagine, this raised a lot of privacy concerns, so the app was developed as open-source software, and the code was available for everyone to see it on GitHub

• Spain: The city of Barcelona has massively adopted open-source software. This switch has saved money and boosted local tech communities while giving the city more control over its digital infrastructure. (Source)

• France: The French government has been pushing for more free software in public administration. This move has made things more transparent and opened up more public participation in tech projects. (Source)

• Germany: Munich’s famous “LiMux” project, which switched the city’s systems to open-source software, showed both the potential and the challenges of such a transition. Despite some setbacks, it was a significant step towards digital independence. (Source 1) (Source 2) (Source 3)

The Roadblocks

Switching to open-source isn’t a walk in the park. Governments often face pushback from big software vendors and have to deal with the headache of changing long-standing systems. Plus, there can be a steep learning curve for public sector IT departments to get up to speed with managing open-source solutions.

Also, those FOSS projects need to be maintained and updated, which can be a challenge for governments with limited resources. And in oposition to a normal FOSS project, the public sector has to deal with a lot of bureaucracy and regulations that can slow down the development process. It might be even possible that some projects are not allowed to directly receive contributions from the public, but we should aim for at least the code to be open and available for everyone to see and point out issues and possible improvements. So there is a big chance that those FOSS projects would be not as active as they should be.

But these challenges can be tackled. With the right planning, training, and support, governments can make the transition to open-source software a success. The benefits – in terms of cost savings, security, and innovation – are well worth the effort.

Looking Ahead

The “Public Money, Public Code” initiative isn’t just a cool idea; it’s a blueprint for a more transparent, efficient, and fair digital future. As more governments see the benefits of going open-source, this movement is likely to gain even more momentum.

In short, if public money is funding software, the public should benefit from it. By adopting open-source principles, governments can ensure their digital infrastructure is secure, transparent, and works for everyone. As we continue to move into an increasingly digital world, this approach will be crucial for building trust and fostering innovation in public services. It’s not just about saving money – it’s about creating a better, more open digital world for all of us.

If you want to know more about this initiative, you can visit the Public Money, Public Code website. And if you think a change like this would be positive, I highly encourage you to sign the Open Letter and support this initiative. You don’t need to be a European Citizen to show support on the initiative. Let’s make sure that public money is used for the public good!

Even when I do my best to ensure content is correctly rendered on RSS readers, some elements like custom info cards/notes and code style blocks can be shown without proper style or syntax highlighting.

You are free to continue reading as you prefer, but I recommend you to read this article in the website if possible. It's fast, has no ads, and minimal privacy-respecting analytics (browser, OS, country and device type) via Umami.

Okay, raise your hand if you ever had a bug related to dates and times in your code. 🙋‍♂️

You raised your hand, right? Don’t worry, me too. And I’m pretty sure that almost every developer has already faced this kind of problem at least once in their career. But why is it so difficult to work with dates and times in software development? Why do we have so many problems with them? Today I’ll show you some curiosities about time and dates that you probably didn’t know. So, let’s get started!

Assume that what you know about time is wrong

I’m sure you have had to manipulate dates and times in your code at some point. Something as simple as adding a number of days to a date, to set the expiration of an API token, for example.

Let’s say you want to create a token that expires after one year. Although is not recommended to use such a long expiration time, let ignore the security implications, as I want to focus only in the date manipulation. You might be tempted to do something like this:

from datetime import datetime, timedelta

current_date = datetime.now()
expiration_date = current_date + timedelta(days=365)
print(expiration_date)

I’m sorry to tell you that, if you are doing things that way, you are introducing a bug in your code. Why? Because you are assuming that a year has always 365 days.

If I execute the code above today, April 27, 2024, the expiration date will be April 27, 2025. But if I executed the same code the past year, the expiration date would be April 26, 2024. Why? Because 2024 is a leap year, so February had 29 days. Therefore, if you add 365 days to a date that is in a leap year, you will end up with a date that is one day before the expected date.

This might seem like a silly example, and you might be thinking “This doesn’t happen in the real world. Any programmer will just add 1 year instead of 365 days to ensure the time library being used takes care of that”.

Oh, really? Have you ever heard about leap day bugs? There are many examples of software that had bugs related to leap days. There are even people listing them on the internet. For example, Gergely Orosz did so in The Pragmatic Engineer, and also, Matt Johnson-Pint has been recording leap day bugs for three different leap years.

Just to name a few striking examples this year:

• The biggest airline in Colombia, Avianca, printed tickets for February 29, 2024, with the incorrect date of March 1, 2024.
• Some EA multiplayer videogames could not be played if the player system date was set to February 29, 2024. A workaround to play that day was to set the system date to March 1, 2024.
• Several payment terminals across the globe stopped to work on February 29, 2024.

So did you said that that doesn’t happen in the real world? That is not a valid argument. It happens, and it happens more that we would like to admit.

Not, really, assume that what you know about time is wrong

“Okay, it happens, but time normally behaves in the same way, those are just some minor exceptions I already know.”_ you might be thinking now. Let me ask you: what are those exceptions? You are probably thinking the already mentioned lap years, and also the summer time changes (some countries change the time twice a year, moving the clock one hour forward or backward). Is that all? What if I tell you that the following is true:

• A day can have 30 hours.
• The day after a Thursday is not always a Friday.
• Not all leap years occur every 4 years.
• Timezones shifts can be greater than +12 hours or lower than -12 hours.

And there are a lot more. So please, if you want to become a better programmer, assume that what you know about time is wrong, and take special care when working with dates and times in your code. Always use battle-tested time manipulation libraries, and always test your code with different dates and times.

Now, let me explain the four examples I gave you:

A day can have 30 hours

In Japan, times past midnight can also be counted past the 24 hour mark, usually when an associated activity spans across midnight. For example, bars or clubs may advertise as being open until “30時” (i.e. 6 am). This is a cultural thing in Japan, partly because the closing time is considered part of the previous business day.

Click on image to enlarge

The day after a Thursday is not always a Friday

On the last week of December 2011, the Samoan government decided to skip December 30, 2011, and move directly to December 31, 2011 from December 29, 2011. This was done to align the country’s calendar with its main trading partners, Australia and New Zealand. That means, people at Samoa went to bed on Thursday, and wake up next day on Saturday. They had not Friday that week.

Not all leap years occur every 4 years

Okay, this is a tricky one. The Gregorian calendar has a leap year every 4 years. That effectively means, that every year divisible by 4 will be a leap year, right? Like 2024, 2028… 2100? No, because there are exceptions. For example, the year 2000 was a leap year, but the year 1900 was not. Also, the year 2100 won’t be a lap year neither. This is because the Gregorian calendar has a rule that states that years that are divisible by 100 are not leap years, unless they are also divisible by 400.

That’s because a year is not exactly 365.25 days long, but 365.2425 days long. So, if we don’t correct this, we would have an error of 0.75 days every 100 years (about 3 extra leap days in 400 years). By ensuring only century years divisible by 400 are leap years, we correct this error, as we are removing 3 leap days every 400 years (there are 3 out of 4 century years that are not being defined as leap years).

Timezones shifts can be greater than +12 hours or lower than -12 hours

Okay, we all know that working with timezones is a pain. But if a day has 24 hours (forget about Japan this time), and if we consider Greenwich meridian as UTC+0, then the maximum shift should be +12 hours or -12 hours, right? It doesn’t have sense to have a UTC+14 timezone, right? Shouldn’t that be UTC-10?

Well, let me introduce you to the Line Islands. The Line Islands are a group of atolls in the central Pacific Ocean, south of Hawaii, that are part of Kiribati. The Line Islands are the only place on Earth where the time zone is UTC+14. That means that when it is 14:00 in Line Islands, it is 12:00 in the nearby Baker Island (UTC-12). That is a 26 hours difference! So here is another bonus falsehood about time (take it as a 2 by 1 offer): different places in Earth can have more than 24 hours of difference between them too! (and we are not taking into account summer time changes!).

Also, the Antarctica, been in a pole, has a lot of timezones with shifts greater than 12 hours. For example, the timezone of McMurdo Station, when Daylight Saving Time is in effect, is UTC+13.

Click on image to enlarge

Do you want more about timezones? We might need to differentiate between Earth and Moon timezones soon.

Conclusion

Time is a complex thing. It is not as simple as we might think. There are a lot of exceptions and special cases that we need to take into account when working with dates and times in software development. We, as developers, need to be aware of these exceptions and take special care when working with dates and times in our code. Always ensure that you are using battle-tested libraries to manipulate dates and times.

As a last defensive resource, ensure that you are testing your code with different dates and times. Take into account different time exceptions and special cases, like those I showed you today, when you are building your test scenarios. Until next time!

Even when I do my best to ensure content is correctly rendered on RSS readers, some elements like custom info cards/notes and code style blocks can be shown without proper style or syntax highlighting.

You are free to continue reading as you prefer, but I recommend you to read this article in the website if possible. It's fast, has no ads, and minimal privacy-respecting analytics (browser, OS, country and device type) via Umami.

Background story

I have a Synology DS920+ NAS at home that I really love, and that I use for a tons of things. It holds family files (Synology Drive), family photos (Synology Photos), a Plex server, some Docker containers, and backups from other servers at home. Everything just works, and it has minimal maintenance. I cannot imagine my daily live without it anymore.

When I initially bought it back in 2020, I configured it with 2x 6TB disks in a SHR1 raid configuration. SHR1 is a Synology proprietary RAID configuration with redundancy of one disk (when you have at least two disks). Unlike traditional raid solutions like RAID1 or RAID5, that requires disks of the same size, or otherwise the available storage space is up to the size of the smaller disk, SHR1 is more flexible and allows you to add disks of different sizes to the volume and take advantage of the extra space.

How does SHR work? Under the hood, it uses LVM (Logical Volume Manager) to manage the disks and create different volumes with different sizes and raid configuration (it uses RAID1 when the volume takes two disks, and RAID 5 for 3+). If you are interested, you can read more about it in the Synology Knowledge Center and play with the Synology RAID Calculator.

Recently, I bought two new 8TB disks, and over the years, a few fancy features where released by Synology (like full volume encryption), so I decided to, instead to add the new disk on the empty slots and expand the volume, to reinstall with a new encrypted SHR1 volume with 1x 6TB and 2x 8TB disks, leaving the spare 6TB disk for quick-recovery local backups. Again, under the hood, this means two LVM volumes, one with 6TB and RAID5 (accross the three disks), and another with 2TB and RAID1 (both 8TB disks).

So I did that, and restored system configuration and critical data from my Synology C2 cloud backup with Hyper Backup. It took like a whole day for 2TB of data, but it worked like a charm. I restored data from the cloud backup because I took this opportunity to test my cloud backups setups, and with the assurance that I could restore my files (and other files not covered by the cloud backup) at any moment from the spare 6TB disks if something went wrong.

Until this moment, everything was fine. Then I tried mounting the old 6TB disk on my Linux desktop to access the files, and I had some trouble. I got one error after another, and I had to spend a few days investigating and trying different things to finally mount the disk and access the files. So I decided to write this article to help others that may have the same problem.

Please, if you try to do the same thing, ensure first you are able to access the files on the old disk from a Linux PC before you proceed with the new installation. I didn’t do that, because I wrongly assumed that the disk would be mounted without any problem just out of the box. Luckily, I had a cloud backup of important files as I said, but I would have been freaked out if I didn’t have it and just assumed that the files were safe on the old disk. Also, remember to always do a 3-2-1 backup strategy, with at least 3 copies of your data, 2 local but on different devices, and 1 offsite.

How to mount the Synology SHR1 (RAID1) disk on Linux and access its content

The first problem that I found was that modern Linux kernel (5.4+) doesn’t support the old btrfs version used by Synology. To the date of writing this article (March 2024), Synology uses an old version of the Linux kernel (4.4.302+, under DSM 7.2.1), and includes some custom volume flags that are not supported by the mainline kernel, leading to an error when trying to mount the disk (from dmesg):

BTRFS critical (device sde6): corrupt leaf: root=1 block=31883264 slot=2, invalid root flags, have 0x400000000 expect mask 0x1000000000001

The solution I found was to setup a VM of Ubuntu Bionic Beaver, and install there a 4.15 kernel. I got the idea from a comment in the above linked Reddit post

# Installing the kernel in ubuntu
sudo apt install linux-image-4.15.0-108-generic
# Reboot and select it on the grub menu, inside the advanced options submenu

Then, we can proceed to mount the disk.

sudo mount /dev/sdb5 /mnt/synology

mount: /mnt/synology: unknown filesystem type 'linux_raid_member'.

Here we got the first error. As the disk was part of a RAID1 filesystem (remember, SHR1 with two disks is RAID1 with LVM), I first need to create a software RAID to mount it. Let’s try to assemble it.

sudo mdadm --assemble --run /dev/md0 /dev/sdb5

mdadm: /dev/sdb5 is busy - skipping

Second error. It says the disk is busy, but it’s not even mounted. Let’s see what’s happening.

sudo mdadm --examine /dev/sdb

/dev/sdb:
  MBR Magic : aa55
  Partition[0] :   4294967295 sectors at            1 (type ee)

Okay, let’s see what is using it. I’ll type ls /dev/md on my terminal and then press tab and see what auto-completes. In my case, it was /dev/md127. Let’s stop it and try to create again the software RAID at /dev/md0

sudo mdadm --stop /dev/md127
sudo mdadm --assemble --run /dev/md0 /dev/sdb5

mdadm: /dev/md0 has been started with 1 drive (out of 2).

Our first success! We can ignore the warning about the missing disk, as we are only interested in mounting the disk to access the files. Let’s try to mount it again.

sudo mount /dev/md0 /mnt/synology

mount: /mnt/synology: unknown filesystem type 'LVM2_member'.

Now that things were working we got another error. In that case, we found that our RAID1 disk was using LVM, which is how SHR works as I already explained. Let’s see what LVM volumes are available using vgs

sudo vgs

VG     #PV #LV #SN Attr   VSize VFree
vg1000   1   1   0 wz--n- 5.45t    0

Okay, it seems like we have a volume group named vg1000. Let’s mount the logical volume there.

sudo mount /dev/mapper/vg1000-lv /mnt/synology

Final success! The disk was successfully mounted and I could see all of its data in /mnt/synology. Phew!

Conclusion

From this experience, I learned a few things the hard way

• Don’t expect that mounting a RAID1 disk will be as easy as mounting an external drive formatted with ext4 or nfts. It’s not plug and play.
• Especially, if it’s not RAID1 per se. In my case, it was a SHR1, that also includes a new actor I never dealt with before, LVM.
• Always have a backup of your data. I had a cloud backup of important files, but I would have been freaked out if I didn’t have it and just assumed that the files were safe on the old disk.

I hope this article helps you to mount your Synology disk on Linux and access its content. Hopefully, you have wasted way less time than I did, until my article popped up on your search results. Lastly, today is March 31st (Backup Day!), so let me take this opportunity to remember you the importance of doing a good backup of your data. Have a nice day!

Even when I do my best to ensure content is correctly rendered on RSS readers, some elements like custom info cards/notes and code style blocks can be shown without proper style or syntax highlighting.

You are free to continue reading as you prefer, but I recommend you to read this article in the website if possible. It's fast, has no ads, and minimal privacy-respecting analytics (browser, OS, country and device type) via Umami.

Today, I want to do a brief introduction to the F.I.R.S.T principles of testing, similar to what I did in the article about the S.O.L.I.D principles of software design.

The F.I.R.S.T principles serve as a foundational framework for effective code testing. These principles, which stand for Fast, Isolated, Repeatable, Self-validating, and Thorough, guide developers in creating robust and reliable tests to ensure the quality and integrity of their code.

Let’s take a closer look at each of the F.I.R.S.T principles:

Fast

One key tenet of the F.I.R.S.T principles is speed. Developers are encouraged to run unit tests swiftly and effortlessly throughout the development process. Regardless of the number of unit tests in place, they should execute rapidly, providing prompt feedback on the code’s correctness.

If tests take too long to run, developers may be less inclined to run them frequently, or even adding new ones, leading to a decrease in the overall quality of the codebase. Fast tests are essential for maintaining a productive and efficient development workflow, and a robust and reliable codebase.

Isolated

The principle of isolation emphasizes the independence of each unit test. Tests should not rely on external factors or be influenced by the outcomes of other tests.

It’s a good practice to follow the 3 A’s of testing: Arrange, Act, Assert (also known as Given, When, Then by Martin Fowler), which defines a structure that helps maintain a clear and autonomous testing environment.

Basically, the test should first arrange the necessary preconditions and inputs, then act on the object or method under test, and finally assert that the expected results have occurred.

A unit test should only assert one logical outcome (test one single thing). Multiple asserts can be part of this logical outcome, as long as they all act on the state of the same object.

Repeatable

Reproducibility is crucial in the F.I.R.S.T methodology. Tests should be consistent and deterministic, producing the same results regardless of the environment in which they are executed. Each test should establish its own data context, reducing reliance on external variables.

Sadly, we all know about the flaky tests that sometimes pass and sometimes fail, and how they can be a nightmare to deal with. Sometimes your CI tests jobs fails, blocks your deploy, and just rerunning the same job without changing anything makes it pass. The F.I.R.S.T principles advocate for the elimination of flaky tests, as they can erode confidence in the testing process and the codebase as a whole.

Self-validating

Manual verification of test outcomes should be unnecessary. Tests should inherently indicate success or failure, facilitating quick identification of issues during the development process.

Thorough

Exhaustive testing is a fundamental aspect of the F.I.R.S.T principles. Developers are encouraged to cover all possible scenarios, including edge cases and potential failure points. Testing should extend beyond typical use cases and happy paths to encompass illegal arguments, security considerations, and the impact of large input values. Try to cover all the branches of your code and all possible scenarios.

Summarizing

In conclusion, the F.I.R.S.T principles provide a structured approach to code testing, emphasizing speed, independence, repeatability, self-validation, and thoroughness. By incorporating these principles into the testing process, developers can enhance the readability and maintainability of their tests, and also the reliability and maintainability of their code, ultimately contributing to the delivery of high-quality software.

Even when I do my best to ensure content is correctly rendered on RSS readers, some elements like custom info cards/notes and code style blocks can be shown without proper style or syntax highlighting.

You are free to continue reading as you prefer, but I recommend you to read this article in the website if possible. It's fast, has no ads, and minimal privacy-respecting analytics (browser, OS, country and device type) via Umami.

In August 2021, Docker announced that they would change their licensing model and that Docker Desktop would no longer be free for commercial use. This made a lot of companies and developers look for alternatives to run docker containers on their Mac machines. That’s because, if you are like me and a lot of other professional developers, your are probably coding for a company with a MacBook or another macOS machine, and you only use docker for development purposes, like say, have a local database instance for development. Probably, you don’t even use the Docker Desktop GUI nor any of its “fancy” features, and you only interact with the docker engine from the command line. Ah, those good old fashioned docker and docker-compose commands.

Good news are, docker engine is licensed under Apache license, so you can use it for free, even for commercial purposes. There is not need to migrate to another container solution.

So okay, let’s get rid of Docker Desktop and continue using docker from the CLI then. At the end of the day, Docker Desktop is just a GUI for the docker engine, and you don’t need it, right? Well, not so fast.

The thing is, Docker containers are a linux native technology. You cannot have docker containers, or any other kind of containers, outside of Linux. So, in order to run docker on macOS (or even Windows), you need to have a Linux virtual machine running on your Mac. And that’s exactly what Docker Desktop does. And that’s why it’s so heavy on your machine. It configures and run a full Linux virtual machine for you, out of the box, to use docker on it.

Hopefully, there are other alternatives to run docker on macOS without needed Docker Desktop. We just need a way to run and integrate a Linux VM with docker in the Mac, and one way is to use Colima, a container runtime for macOS that provides a backend for docker using qemu. It’s build on top of Lima, that is like WSL2 in Windows, but for macOS: a lightweight virtual machine that runs a Linux distribution.

In this article, I’ll show you how to use docker on macOS using Colima. It’s very easy to install and use, and it’s free and open source. Let’s get started.

Step by step guide to run docker engine on macOS with Colima

First what is first, get rid of Docker Desktop! You can uninstall it like any other app in macOS, just drag it to the trash bin. But I recommend you to first clean the installation from Docker Desktop itself, as we already saw it does a lot of things under the hook for you (set ups the linux VM, links docker, etc). It’s better to ensure that everything is clean before installing Colima. To do so, open Docker Desktop and go to the Troubleshoot menu (the one with the bug in the upper right corner), and click on the “Uninstall” button. This will remove all the data and configuration files that Docker Desktop created on your machine.

Click on image to enlarge

Then, move the Docker Desktop app to the trash bin if required (the Desktop app will probably ask you to do so).

Next, let’s move to the installation of Colima. You can install it using Homebrew, the package manager for macOS. Just run the following command in your terminal. Note that installing docker engine is necessary, as you deleted it when you uninstalled Docker Desktop.

brew install docker-credential-helper docker colima

Now, let’s create the linux virtual machine that will run the docker engine.

colima start

This will create a new VM with the default config (2 CPUs, 2GiB RAM, 60GiB disk), but you can customize it to your needs. You can check colima start --help for more information. For example, if you want to use 4 CPUs, 4GiB RAM and 100GiB disk, you can run the following command:

colima start -c 4 -m 4 -d 100

On my M2 Pro MacBook Pro, I’m using the following command to start the VM, fine-tuned for Apple Silicon machines:

colima start --arch aarch64 --vm-type=vz --vz-rosetta --cpu 4 --memory 4 --disk 64

What this command specifically does is:

• --arch aarch64: Specifies the CPU architecture as ARM64 (used for Apple Silicon Macs).
• --vm-type=vz: Uses Apple’s Virtualization.framework (VZ) as the virtualization backend.
• --vz-rosetta: Enables Rosetta 2 translation for running x86_64 containers on ARM.
• --cpu 4: Use 4 CPUs.
• --memory 4: Use 4GiB of RAM.
• --disk 64: Use 64GiB of disk space.

Anyway, after you start the VM one way or another, you can run docker ps and see that the docker engine is running as before.

Using docker-compose

If you use docker-compose or the compose command in docker, we need to install it separately and link it as a CLI addon for docker. To do so, run the following command:

brew install docker-compose
mkdir -p ~/.docker/cli-plugins
ln -sfn /opt/homebrew/opt/docker-compose/bin/docker-compose ~/.docker/cli-plugins/docker-compose

Autostart on login

If you want to start the VM automatically when you login to your Mac, you can do so by running the following command:

brew services start colima

Troubleshooting

Colima works with docker out of the box, and this is because it uses docker contexts to correctly set the docker daemon for you. However, some docker applications doesn’t honor the docker contexts, and instead, relies on a hardcoded path for the usual location of the docker daemon.

If after having the above steps you get the following error when running docker commands:

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

You probably need to set the DOCKER_HOST environment variable to the correct value, like this:

export DOCKER_HOST="unix://${HOME}/.colima/default/docker.sock"

Or you can instead link the Colima socket to the default socket path, but note that this may break other Docker servers

sudo ln -sf $HOME/.colima/default/docker.sock /var/run/docker.sock

If you have other problems, take a look at their docs in their repo. I hope not! I didn’t have any problem at all.

Even when I do my best to ensure content is correctly rendered on RSS readers, some elements like custom info cards/notes and code style blocks can be shown without proper style or syntax highlighting.

You are free to continue reading as you prefer, but I recommend you to read this article in the website if possible. It's fast, has no ads, and minimal privacy-respecting analytics (browser, OS, country and device type) via Umami.

The builder pattern is a technique where a developer uses a “builder” to construct an object. The end result could be anything - a string, a struct instance, or even a closure - but it is built using a builder. More often than not a builder is used because the object being created is complex and needs to be constructed in multiple steps, so the builder helps isolate each step and prevent bugs.

Imagine a complex object that requires laborious, step-by-step initialization of many fields and nested objects. Such initialization code is usually buried inside a monstrous constructor with lots of parameters. Or even worse: scattered all over the client code. Here is where the builder pattern comes to the rescue.

To illustrate the idea, I’ll provide an easy example, for what the Builder pattern would be an overkill, but it’ll help us understand this. Obviously, as the title of this article says, I’ll be using Go, but note that this is an universal software design pattern, so it can be applied to any language that supports some kind of Object Oriented Programming.

Naive example

Imagine you have an Car struct:

type Car struct {
  Model string
  Type string
  Traction string
}

Under most circumstances a developer would create an instance of the Car struct in their code and assign a value to each of these fields, but in some circumstances you might want to do more than that. For instance, you might decide that when the Type field is set you can also assign the Traction field based on the car’s type. This could be achieved using a builder.

type Builder struct {
  c Car
}

func (b *Builder) Build() Car {
  return b.c
}

func (b *Builder) Model(model string) *Builder {
  b.c.Model = model
  return b
}

func (b *Builder) Type(carType string) *Builder {
  if carType == "SUV" {
    b.c.Traction = "4x4"
  } else {
    b.c.Traction = "front"
  }
  b.c.Type = carType
  return b
}

Now when we are constructing a Car we can use the builder to ensure that the Traction gets set when we set the Type field. We could even add validation in and verify that the Type is a valid one - though at that point we would need to also return an error.

Below we can see how this builder might be used (Go Playground link):

b := &Builder{}
car := b.
  Model("Toyota Rav4").
  Type("SUV").
  Build()
fmt.Println(car)
// {Toyota Rav4 SUV 4x4}

The magic of the builder pattern is that we can chain the builder methods together, and each method returns the builder itself. This allows us to call the next method on the builder, and so on. The last method called is the Build method, which returns the final object.

As said, this is a really basic example so the builder pattern might feel like overkill, but much more complex scenarios exist where the builder pattern is incredibly helpful. For instance, constructing SQL queries can become complex and we might need to handle conditional queries. In this case a builder can simplify things a bit.

Real-world™ examples

Indeed, most SQL query builder libs out there implement this pattern. One of such libraries is squirrel. Let’s take a look at a few examples from the squirrel docs at the moment of writing this article. First, we start with a basic example of how squirrel can be used to generate the SQL query string we might use with the standard library’s database/sql package.

users := sq.Select("*").From("users").Join("emails USING (email_id)")
sql, args, err := users.ToSql()
// sql == "SELECT * FROM users JOIN emails USING (email_id)"

active := users.Where(sq.Eq{"deleted_at": nil})
sql, args, err := active.ToSql()
// sql == "SELECT * FROM users JOIN emails USING (email_id) WHERE deleted_at IS NULL"

Now, let’s move to a more complex scenario. Let’s say we want to filter by name is a certain variable q has content, which is as simple as adding an if statement and adding the conditional query

if len(q) > 0 {
  users = users.Where("name LIKE ?", fmt.Sprint("%", q, "%"))
}

Notice how we assign the result of the users.Where function call to the users variable, allowing us to retain this new build step. Now whenever we call users.ToSql() (that acts like the Build() function in the Car example) it will have the Where clause here only if the if statement was true.

Builders can even be found in Go’s standard library too. See text/template and html/template packages. Both of the template packages have a Template type that uses the builder pattern:

tmpl, err := template.New("titleTest").
  Funcs(funcMap).
  Delims("[[", "]]").
  Parse(templateText)

Both Funcs and Delims return a *Template, and Parse returns both a *Template and an error kinda like our Role example in the employee builder would have if we validated the role.

Conclusion

To sum up - the builder pattern usually involves chaining functions on a single type as we “build” the end result. It is used to build complex objects, and can be found in quite a few libraries.

If you want to go deeper into the Builder pattern and others, I invite you to take a look at the refactoring.guru page. It’s a great resource to learn about software design patterns. Also, if you want to read more from me, you can check other patterns articles in my blog.

Even when I do my best to ensure content is correctly rendered on RSS readers, some elements like custom info cards/notes and code style blocks can be shown without proper style or syntax highlighting.

You are free to continue reading as you prefer, but I recommend you to read this article in the website if possible. It's fast, has no ads, and minimal privacy-respecting analytics (browser, OS, country and device type) via Umami.

In my previous article about creating an fully functional API in Go under 5 minutes, I showed you how to create a web server using only the standard library. There, we learned that Go standard libs are powerful enough to meet our needs.

In this article, and in a similar way to the aforementioned one, I’ll show you everything you need to know about the database/sql package in Go, to quickly start integrating a SQL database in your application.

Let’s see some snippets! Let’s start by how to connect to a database.

Connecting to a database

Connecting to the database can be done with the sql.Open function. This function receives two parameters: the driver name and the connection string (aka data source name, or DSN). The driver name is the name of the driver that you want to use to connect to the database. The DSN is a string that contains the information needed to connect to the database. The DSN format depends on the database that you are using, but an example for PostgreSQL would be postgres://user:password@host:port/database.

// Configure the connection to the database.
db, err := sql.Open("DRIVER_NAME", "CONNECTION_STRING")
if err != nil {
	log.Fatal(err)
}
defer db.Close()

However, the sql.Open function doesn’t actually open a connection to the database. It just creates it, by returning a new *sql.DB object. The connection to the database is only opened once yo tries to operate on it for the first time. Therefore, if you have an error with your connection configuration (let’s say, you introduced a wrong password or a typo in the database name), you won’t know until you try to insert or query some data. How to avoid that? Is a common good practice to use the Ping method of the returned *sql.DB object just after its creation to check if the connection is working.

// Configure the connection to the database.
db, err := sql.Open("DRIVER_NAME", "CONNECTION_STRING")
if err != nil {
	log.Fatal(err)
}
defer db.Close()

// Use Ping() to actually open the connection and check for any errors.
if err = db.Ping(); err != nil {
	log.Fatal(err)
}

Executing a SQL statement

The obtained *sql.DB object has a generic method to execute SQL statements: the Exec method. This method receives a query string and a list of parameters. The query string is the parametrized SQL query that you want to execute, and the parameters are the values that you want to insert into the query string.

Why like that? Why not just passing an string with the SQL query already build? That’s because the Exec method already escape the parameters to avoid SQL injection attacks.

Inserting data

The Exec method can be used to insert data into the database. Let’s see an example for MySQL:

result, err := db.Exec("INSERT INTO customers (name) VALUES (?)", "Alice")

if err != nil {
	log.Fatal(err)
}

// Note: The LastInsertID() method is not supported by PostgreSQL.
// Check the next snipped for a PostgreSQL example.
id, err := result.LastInsertId()
if err != nil {
	log.Fatal(err)
}

affected, err := result.RowsAffected()
if err != nil {
	log.Fatal(err)
}

About PostgreSQL, it depends if we want to get the ID of the inserted row or not. If we don’t want the ID, it’s the same as MySQL, but with proper PostgreSQL syntax (parameters are $1, $2, instead of interrogations marks). Let’s see it:

result, err := db.Exec("INSERT INTO customers (name) VALUES ($1)", "Alice")

if err != nil {
	log.Fatal(err)
}

affected, err := result.RowsAffected()

if err != nil {
	log.Fatal(err)
}

If we want to get the ID of the inserted row, we need to use the RETURNING clause in the query string, and instead of using the Exec method, we need to use the QueryRow method. This method is used to fetch data from the database, and we will see it in detail in the next section about querying the database. For now, let’s see how to use it to get the ID of the inserted row:

var id int
err := db.QueryRow("INSERT INTO customers (name) VALUES ($1) RETURNING id", "Alice").Scan(&id)

if err != nil {
	log.Fatal(err)
}

Updating and deleting data

Obviously, we can use the Exec method to update and delete data too. We just have to take care to use the correct syntax for our database engine. Let’s see some examples:

// UPDATE in a MySQL syntax.
result, err := db.Exec("UPDATE customers SET name = ? WHERE id = ?", "Alice", 1)

if err != nil {
	log.Fatal(err)
}

affected, err := result.RowsAffected()

if err != nil {
	log.Fatal(err)
}

// DELETE in a PostgreSQL syntax.
result, err := db.Exec("DELETE FROM customers WHERE id = $1", 1)

if err != nil {
	log.Fatal(err)
}

affected, err := result.RowsAffected()

if err != nil {
	log.Fatal(err)
}

Querying the database

The *sql.DB object also have methods for querying the database. The Query method is used to execute a query that returns multiple rows, and the QueryRow method is used to execute a query that returns a single row. Both methods receive a query string and a list of parameters, similar to the Exec method that we already saw, and return a *sql.Rows object and a *sql.Row object, respectively.

Querying a single row

Let’s see an example of how to use the QueryRow method to query a single row from the database. The *sql.Row object returned by the QueryRow method has a Scan method that is used to scan the values of the returned row into variables. It receives a list of pointers to the variables where the values will be stored.

var name, surname string
err := db.QueryRow("SELECT name, surname FROM customers WHERE id = ?", 1).Scan(&name, &surname)
// err := db.QueryRow("SELECT name, surname FROM customers WHERE id = $1", 1).Scan(&name, &surname)

if err == sql.ErrNoRows {
	// There is not a row with the given ID.
	log.Fatal("no rows returned")
} else if err != nil {
	log.Fatal(err)
}

fmt.Printf("Name: %s, Surname: %s\n", name, surname)

Querying multiple rows

The Query method is used to execute a query that returns multiple rows. The *sql.Rows object returned by the Query method has a Next method to iterate over the returned rows. This method returns true if there is a new row to be scanned, and false if there are no more rows to be scanned. The *sql.Rows object also has a Scan method as *sql.Row has, and it’s executed over the current row.

rows, err := db.Query("SELECT name FROM customers LIMIT 10")

if err != nil {
	log.Fatal(err)
}

// Don't forget to close the *sql.Rows when you are done.
defer rows.Close()
names := []string{}

for rows.Next() {
	var name string
	err := rows.Scan(&name)
	if err != nil {
		log.Fatal(err)
	}
	names = append(names, name)
}

if err = rows.Err(); err != nil {
	log.Fatal(err)
}

fmt.Printf("Names: %v\n", names)

Transactions

All statements in a transaction use the same database connection, provided by the *sql.DB object. This object has a Begin method to start a new transaction, that returns a *sql.Tx object. The *sql.Tx object has a Commit and a Rollback methods, used to commit and to rollback the transaction respectively. Let’s see it.

// Start a new transaction.
tx, err := db.Begin()

if err != nil {
	log.Fatal(err)
}

// Execute some SQL statements inside the transaction.
_, err := tx.Exec("INSERT INTO customers ...")
if err != nil {
	// Rollback the transaction if any of the statements fails.
	tx.Rollback()
	log.Fatal(err)
}

// More SQL statements here
// ...

// Commit the transaction.
err = tx.Commit()
if err != nil {
	log.Fatal(err)
}

Wrapping up

And that’s it. As you saw, the database/sql package has everything you need to integrate a SQL database into your golang application. You can connect to the database, insert and query data, and even use transactions, just using the standard library.

With this new knowledge, you are ready to implement a CRUD application in Golang. And if you want to make it a webapp (like an REST API), remember as I said at the beginning of this article that I wrote another one about the net/http package. Happy coding!