"certificate not yet valid" when installing tailscale via cloud-init on Raspberry Pi

Okay, this post is potentially very niche, but the issue was happening consitently for me so I thought I'd drop the fix down here for anyone else running into it.

Before we get going, this is probably only relevant for you if you're running something like Ubuntu Server on a Raspberry Pi. It may apply to other operating systems and devices, but is unlikely to apply to a Cloud VPS.

Scenario

This week I managed to brick an OS on my raspberry pi and figured I'd just flash the SD card and start again. As I'm trying to improve my DevOps skills, I thought this might be a good time to try cloud-init out and see if I could get tailscale with SSH to automatically setup when the pi boots up.

I followed the Tailscale guide for this which was really helpful, but after waiting several minutes there would be no new node on my tailnet. The OS bootstrapped just fine and the standard SSH key I added to my user worked to get me in just fine. When I ran tailscale status to see what was going on, it hadn't been installed at all.

"certificate is not yet valid"

Cloud-init has quite a few different logs associated with it, but eventually I uncovered the logs for the runcmd script and found that while the script had run, it had failed.

The error I was left with was:

Curl: (60) SSL certificate problem: certificate is not yet valid

The only curl command I had was the one to install tailscale so it was clear that the installation was failing and it was due to this SSL issue.

Thankfully, this is a very clear error message from curl, and working on the assumption that the certificate is actually valid on tailscale's side, it means my Pi's clock was wrong.

The fix

In the end I added a bit of bash to trigger a clock sync, and then waited for that to be completed. My full user-data file ended up looking like this:

#cloud-config
# The above header must generally appear on the first line of a cloud config
# file, but all other lines that begin with a # are optional comments.

runcmd:
  # Force the clock to sync and wait for it to do so, or for 60s to pass
  - [ bash, -c, '
      echo "Enabling NTP...";
      timedatectl set-ntp true;
      echo "Waiting for time sync (up to 60 seconds)...";
      for i in {1..60}; do
        if timedatectl show -p NTPSynchronized --value | grep -q yes; then
          echo "Time is synced.";
          break;
        fi;
        sleep 1;
      done'
    ]
  # One-command install, from https://tailscale.com/download/
  - ['sh', '-c', 'curl -fsSL https://tailscale.com/install.sh | sh']
  # Set sysctl settings for IP forwarding (useful when configuring an exit node)
  - ['sh', '-c', "echo 'net.ipv4.ip_forward = 1' | sudo tee -a /etc/sysctl.d/99-tailscale.conf && echo 'net.ipv6.conf.all.forwarding = 1' | sudo tee -a /etc/sysctl.d/99-tailscale.conf && sudo sysctl -p /etc/sysctl.d/99-tailscale.conf" ]
  # Generate an auth key from your Admin console
  # https://login.tailscale.com/admin/settings/keys
  # and replace the placeholder below
  - ['tailscale', 'up', '--auth-key=tskey-abcdef1432341818']
  # (Optional) Include this line to make this node available over Tailscale SSH
  - ['tailscale', 'set', '--ssh']
  # (Optional) Include this line to configure this machine as an exit node
  - ['tailscale', 'set', '--advertise-exit-node']

It's worth nothing, that this solution requires bash and timedatectl are both present and available on the operating system. If you're using something like Alpine, you may need to make changes.

Conclusion

If you're using a Raspberry Pi, or some other small device, with Ubuntu Server, you may need to do this if you want to run curl commands as part of your cloud-init runcmd.