flatcar, cannot finish deployment

[paulj2000ro]

What happened?

Hi!
I have created three instances on GCE with flatcar stable channel (image name flatcar-stable-4152-2-2) and I have created a simple inventory file like this:

flatcar-container-linux-4-vm ansible_host=172.26.232.3
flatcar-container-linux-5-vm ansible_host=172.26.232.4
flatcar-container-linux-6-vm ansible_host=172.26.232.5
[kube_control_plane]
flatcar-container-linux-4-vm
flatcar-container-linux-5-vm
flatcar-container-linux-6-vm
[etcd]
flatcar-container-linux-4-vm
flatcar-container-linux-5-vm
flatcar-container-linux-6-vm
[kube_node]
flatcar-container-linux-4-vm
flatcar-container-linux-5-vm
flatcar-container-linux-6-vm

I also have some customizations in mycluster/group_vars/all/all.yml:

bin_dir: /opt/bin
ansible_python_interpreter: /opt/bin/python
resolvconf_mode: host_resolvconf
ansible_remote_tmp: /tmp/ansible-${USER}
override_system_hostname: false

I got errors at creating tmp directory but i could get past it by using become when deploying:

ansible-playbook -i /inventory/mycluster/inventory.ini --private-key /root/.ssh/id_rsa cluster.yml -u username -b

After playbook is stared it get errors at:

TASK [kubernetes/preinstall : Hosts | create hosts list from inventory] ********************************************************************************************************************************************************************************************************
ok: [pjurco-container-linux-4-vm -> localhost]
Thursday 29 May 2025 10:33:02 +0000 (0:00:00.380) 0:01:54.874 **********

TASK [kubernetes/preinstall : Hosts | populate inventory into hosts file] ******************************************************************************************************************************************************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: OSError: [Errno 30] Read-only file system: b'/usr/share/gce/hosts'
fatal: [flatcar-container-linux-4-vm]: FAILED! => {"changed": false, "msg": "Could not write data to file (b'/usr/share/gce/hosts') from (b'/tmp/ansible-root/ansible-moduletmp-1748514784.8171709-dsvxp7lh/tmp97o27kst'): [Errno 30] Read-only file system: b'/usr/share/gce/hosts'"}
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: OSError: [Errno 30] Read-only file system: b'/usr/share/gce/hosts'
fatal: [flatcar-container-linux-5-vm]: FAILED! => {"changed": false, "msg": "Could not write data to file (b'/usr/share/gce/hosts') from (b'/tmp/ansible-root/ansible-moduletmp-1748514784.9854934-qd955gnz/tmp4679qk1q'): [Errno 30] Read-only file system: b'/usr/share/gce/hosts'"}
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: OSError: [Errno 30] Read-only file system: b'/usr/share/gce/hosts'
fatal: [flatcar-container-linux-6-vm]: FAILED! => {"changed": false, "msg": "Could not write data to file (b'/usr/share/gce/hosts') from (b'/tmp/ansible-root/ansible-moduletmp-1748514784.7516584-c4w59lic/tmplizj_mua'): [Errno 30] Read-only file system: b'/usr/share/gce/hosts'"}

I know /usr is RO, but, how to overcome this situation?
Thank you!

What did you expect to happen?

Succes in kubespray execution, full cluster functional.

How can we reproduce it (as minimally and precisely as possible)?

• deploy 3 instances of flatcar-stable-4152-2-2 in GCE
• create a simple inventory
• create customizations in mycluster/group_vars/all/all.yml
• run kubespray in docker:

docker run --rm -it --mount type=bind,source="$(pwd)"/inventory/sample,dst=/inventory
--mount type=bind,source="${HOME}"/.ssh/id_rsa,dst=/root/.ssh/id_rsa
quay.io/kubespray/kubespray:v2.28.0 bash

• run ansible playbook:

ansible-playbook -i /inventory/mycluster/inventory.ini --private-key /root/.ssh/id_rsa cluster.yml -u user -b

OS

Flatcar Container Linux

Version of Ansible

ansible [core 2.16.14]
config file = /kubespray/ansible.cfg
configured module search path = ['/kubespray/library']
ansible python module location = /usr/local/lib/python3.10/dist-packages/ansible
ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
executable location = /usr/local/bin/ansible
python version = 3.10.12 (main, Feb 4 2025, 14:57:36) [GCC 11.4.0] (/usr/bin/python3)
jinja version = 3.1.6
libyaml = True

Version of Python

Python 3.10.12

Version of Kubespray (commit)

docker image: quay.io/kubespray/kubespray:v2.28.0

Network plugin used

calico

Full inventory with variables

https://gist.github.com/paulj2000ro/f4e1202859f59f0585923e9907cd1e5f

Command used to invoke ansible

ansible-playbook -i /inventory/mycluster/inventory.ini --private-key /root/.ssh/id_rsa cluster.yml -u user -b

Output of ansible run

https://gist.github.com/paulj2000ro/32fef0e0d2217212315a0109734b847e

Anything else we need to know

thank you!

[tico88612]

If populate_inventory_to_hosts_file set to false?

[paulj2000ro]

populate_inventory_to_hosts_file set to false helped to get over that task, but now is stuck at:

TASK [kubernetes/preinstall : Hosts | Update (if necessary) hosts file]

See partial output here:
https://gist.github.com/paulj2000ro/315e21838ae85bbde12b46b53830d73e

[tico88612]

populate_localhost_entries_to_hosts_file: false? I'm not sure if this will affect the final installation.

[paulj2000ro]

It went until etcd checks. Here are found errors along playbook run:

TASK [container-engine/runc : Runc | Remove orphaned binary] *******************************************************************************************************************************************************************************************************************
fatal: [flatcar-container-linux-5-vm]: FAILED! => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "msg": "unlinking failed: [Errno 30] Read-only file system: b'/usr/bin/runc' ", "owner": "root", "path": "/usr/bin/runc", "secontext": "system_u:object_r:unlabeled_t:s0", "size": 13171864, "state": "file", "uid": 0}
...ignoring
fatal: [flatcar-container-linux-4-vm]: FAILED! => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "msg": "unlinking failed: [Errno 30] Read-only file system: b'/usr/bin/runc' ", "owner": "root", "path": "/usr/bin/runc", "secontext": "system_u:object_r:unlabeled_t:s0", "size": 13171864, "state": "file", "uid": 0}
...ignoring
fatal: [flatcar-container-linux-6-vm]: FAILED! => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "msg": "unlinking failed: [Errno 30] Read-only file system: b'/usr/bin/runc' ", "owner": "root", "path": "/usr/bin/runc", "secontext": "system_u:object_r:unlabeled_t:s0", "size": 13171864, "state": "file", "uid": 0}
...ignoring

TASK [etcd : Get currently-deployed etcd version] ******************************************************************************************************************************************************************************************************************************
fatal: [flatcar-container-linux-5-vm]: FAILED! => {"changed": false, "cmd": "/opt/bin/etcd --version", "msg": "[Errno 2] No such file or directory: b'/opt/bin/etcd'", "rc": 2, "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
...ignoring
fatal: [flatcar-container-linux-4-vm]: FAILED! => {"changed": false, "cmd": "/opt/bin/etcd --version", "msg": "[Errno 2] No such file or directory: b'/opt/bin/etcd'", "rc": 2, "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
...ignoring
fatal: [flatcar-container-linux-6-vm]: FAILED! => {"changed": false, "cmd": "/opt/bin/etcd --version", "msg": "[Errno 2] No such file or directory: b'/opt/bin/etcd'", "rc": 2, "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
...ignoring
Friday 30 May 2025 09:07:47 +0000 (0:00:01.243) 0:21:07.360 ************

And it stopped at:

TASK [etcd : Configure | Ensure etcd is running] *******************************************************************************************************************************************************************************************************************************
fatal: [flatcar-container-linux-6-vm]: FAILED! => {"changed": false, "msg": "Unable to start service etcd: Job for etcd.service failed because a timeout was exceeded.\nSee "systemctl status etcd.service" and "journalctl -xeu etcd.service" for details.\n"}
fatal: [flatcar-container-linux-4-vm]: FAILED! => {"changed": false, "msg": "Unable to start service etcd: Job for etcd.service failed because a timeout was exceeded.\nSee "systemctl status etcd.service" and "journalctl -xeu etcd.service" for details.\n"}
fatal: [flatcar-container-linux-5-vm]: FAILED! => {"changed": false, "msg": "Unable to start service etcd: Job for etcd.service failed because a timeout was exceeded.\nSee "systemctl status etcd.service" and "journalctl -xeu etcd.service" for details.\n"}

PLAY RECAP *********************************************************************************************************************************************************************************************************************************************************************
flatcar-container-linux-4-vm : ok=457 changed=64 unreachable=0 failed=1 skipped=534 rescued=0 ignored=2
flatcar-container-linux-5-vm : ok=425 changed=61 unreachable=0 failed=1 skipped=459 rescued=0 ignored=2
flatcar-container-linux-6-vm : ok=425 changed=61 unreachable=0 failed=1 skipped=459 rescued=0 ignored=2

on one instances the service looks up:

user@flatcar-container-linux-4-vm ~ $ systemctl status etcd.service
● etcd.service - etcd
Loaded: loaded (/etc/systemd/system/etcd.service; enabled; preset: disabled)
Active: activating (start) since Fri 2025-05-30 09:13:05 UTC; 40s ago
Main PID: 7782 (etcd)
Tasks: 5 (limit: 63171)
Memory: 10.4M (peak: 10.9M)
CPU: 195ms
CGroup: /system.slice/etcd.service
└─7782 /opt/bin/etcd

thank you!

[tico88612]

kubespray/docs/operating_systems/flatcar.md

Line 10 in c7c3d2b

- Ensure that the bin_dir is set to `/opt/bin`

[tico88612]

Please read the manual first.

[paulj2000ro]

it's set up already like required, here is the latest:

bin_dir: /opt/bin
ansible_python_interpreter: /opt/bin/python
resolvconf_mode: host_resolvconf
ansible_remote_tmp: /tmp/ansible-${USER}
override_system_hostname: false
populate_inventory_to_hosts_file: false
populate_localhost_entries_to_hosts_file: false

[tico88612]

I guess this is GCE problem, because we recently add flatcar 4081 test in #11868. There aren't any other problems here.

[paulj2000ro]

GCE image is provided by Kinvolk and comes in four flavors, none of them is the above, which is LTS.
They provide only stable, beta, alpha and edge :(
I will try to get an image with 4081 version.

[tico88612]

GCE image is provided by Kinvolk and comes in four flavors, none of them is the above, which is LTS.

Yes, but /usr/share/gce/hosts obviously it's from GCE, not from Kinvolk. Perhaps the GCE environment needs to be adjusted.

[paulj2000ro]

/etc/hosts is a symlink of /usr/share/gce/hosts
i will try again with a file instead of a symlink

[tormath1]

Hi @paulj2000ro,

GCE image is provided by Kinvolk and comes in four flavors, none of them is the above, which is LTS.
They provide only stable, beta, alpha and edge :(

I am curious, why do you say this? edge images are not provided since ages now, it's only Alpha, Beta, Stable and LTS. Same on GCE, where edge offer does not even exist:

/ # gcloud compute images describe-from-family flatcar-lts --project kinvolk-public
archiveSizeBytes: '551369600'
creationTimestamp: '2025-05-06T04:28:22.910-07:00'
description: The LTS channel should be used by production clusters. Versions of Flatcar
  Container Linux are battle-tested within the Stable channel before being promoted.,
  4081.3.3, amd64-usr published on 2025-05-06
diskSizeGb: '9'
enableConfidentialCompute: false
family: flatcar-lts
guestOsFeatures:
- type: VIRTIO_SCSI_MULTIQUEUE
- type: UEFI_COMPATIBLE
- type: GVNIC
id: '6127495544936090809'
kind: compute#image
labelFingerprint: 42WmSpB8rSM=
licenseCodes:
- '5895205768306350279'
licenses:
- https://www.googleapis.com/compute/v1/projects/kinvolk-public/global/licenses/flatcar-container-linux
name: flatcar-lts-4081-3-3
rawDisk:
  containerType: TAR
  source: ''
selfLink: https://www.googleapis.com/compute/v1/projects/kinvolk-public/global/images/flatcar-lts-4081-3-3
sourceType: RAW
status: READY
storageLocations:
- us
/ # gcloud compute images describe-from-family flatcar-edge --project kinvolk-public
ERROR: (gcloud.compute.images.describe-from-family) Could not fetch resource:
 - Required 'compute.images.getFromFamily' permission for 'projects/kinvolk-public'

That said, for /usr/share/gce/hosts it's handled by Flatcar team here:
https://github.com/flatcar/scripts/blob/76d9c7580499d6d08bb66c7cab3259962e7870de/sdk_container/src/third_party/coreos-overlay/coreos-base/oem-gce/files/units/setup-oem.service#L9

For etcd to work, what is the Kubespray worflow here? Does it install etcd & friends in the bin_dir or does Kubespray expects to have etcd already on the instance?

[paulj2000ro]

Hi @tormath1,

I am curious, why do you say this? edge images are not provided since ages now, it's only Alpha, Beta, Stable and LTS. Same on GCE, where edge offer does not even exist:

If you go to Marketplace and press Launch, in the menu, there is a list:

To me LTS is not present, but edge exists. However, i do not intend to test flatcar on edge, that's not my scope, based on the description.
I am testing on stable + kubespray for kubernetes tests.

Friday i have tested with LTS on GCE (uploaded tgz file to a bucket, created an image, created 3 instances from it and executed kubespray against it) but i get the same result: stuck at etcd services status after.
Thank you!
Paul