I have been trying to bake cloudflared daemon along with applicable configuration as part of hardened OS image. I am relying on AWS services named EC2 image builder which is in turn dependent on a service named cloud-init service within the Ubuntu OS for orchestrating the entire hardening process.
However , I see that build fails every time when I include cloudflared and install it as a service. After 10 days of debugging along with AWS Support folks, finally found that cloudflared service somehow blocks the remaining services (which includes cloud-init) from starting during boot time , resulting in entire pipeline getting timed out and failing. Apparently it seems , systemd waits on cloudflared service to complete before executing other dependent services , but cloudflared would keep running (which is expected).
cloudflared version used : 2022.5.3 (used in most recent attempt of the troubleshooting exercise)
I have included a small snippet of output of the jobs waiting while cloudflared service running (This is after two hours of the pipeline initialisation).
Once the cloudflared service is cancelled, the cloud-init status is checked again and it outputs done.
[email protected]:~$ cloud-init status --wait
......................................................................................................................................................................^CTraceback (most recent call last):
File "/usr/bin/cloud-init", line 11, in <module>
load_entry_point('cloud-init==20.3', 'console_scripts', 'cloud-init')()
File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 891, in main
retval = util.log_time(
File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2292, in log_time
ret = func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/cloudinit/cmd/status.py", line 61, in handle_status_args
sleep(0.25)
File "/usr/lib/python3/dist-packages/cloudinit/signal_handler.py", line 48, in _handle_exit
util.multi_log(contents.getvalue(),
File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 400, in multi_log
with open(conpath, 'w') as wfh:
PermissionError: [Errno 13] Permission denied: '/dev/console'
[email protected]:~$ sudo systemctl list-jobs
[sudo] password for temp:
JOB UNIT TYPE STATE
157 cloud-final.service start waiting
158 systemd-update-utmp-runlevel.service start waiting
2 multi-user.target start waiting
1 graphical.target start waiting
109 cloudflared.service start running
151 cloud-init.target start waiting
6 jobs listed.
[email protected]:~$ sudo systemctl cancel 109
[email protected]:~$ cloud-init status --wait
status: done
My systemctl configuration looks like this :-
[Unit]
Description=cloudflared
After=network.target
[Service]
TimeoutStartSec=0
Type=notify
ExecStart=/usr/bin/cloudflared --no-autoupdate --config /etc/cloudflared/config.yml tunnel run
Restart=on-failure
RestartSec=5s
[Install]
WantedBy=multi-user.target
I would want to ensure cloudflared does not block other services from running during boot time. Any help is greatly appreciated.