What happened?

Playbook cluster.yml failed on the "Assert that current calico version is enough for upgrade" task
In my understanding task "Get current calico version" return nothing

What did you expect to happen?

playbook cluster.yml reconfigure something (changes in kube-apiserver configuration) in working k8s cluster

How can we reproduce it (as minimally and precisely as possible)?

Run cluster.yml playbook (as user, with --become) or any other playbooks which runs "Check that current calico version is enough for upgrade" task.

OS

Ubuntu 22

Version of Ansible

ansible [core 2.16.14]

Version of Python

Python 3.10.12

Version of Kubespray (commit)

2.27

Network plugin used

calico

Full inventory with variables

without any changes in calico variables

Command used to invoke ansible

ansible-playbook -i inventory/prod/hosts.yaml cluster.yml -u myuser -b --check --diff

Output of ansible run

TASK [network_plugin/calico : Assert that current calico version is enough for upgrade] ****************************************************************************************
fatal: [node1]: FAILED! => {"msg": "The conditional check 'calico_version_on_server.stdout is version(calico_min_version_required, '>=')' failed. The error was: Input version value cannot be empty"}

Anything else we need to know

The problem is in this part of code

- name: Check that current calico version is enough for upgrade
  run_once: true
  delegate_to: "{{ groups['kube_control_plane'][0] }}"
  when: calicoctl_sh_exists.stat.exists and calico_ready.rc == 0
  block:
    - name: Get current calico version
      shell: "set -o pipefail && {{ bin_dir }}/calicoctl.sh version | grep 'Client Version:' | awk '{ print $3}'"
      args:
        executable: /bin/bash
      register: calico_version_on_server
      changed_when: false

    - name: Assert that current calico version is enough for upgrade
      assert:
        that:
          - calico_version_on_server.stdout is version(calico_min_version_required, '>=')
        msg: >
          Your version of calico is not fresh enough for upgrade.
          Minimum version is {{ calico_min_version_required }} supported by the previous kubespray release.
          But current version is {{ calico_version_on_server.stdout }}.

I do test, and this:
shell: "set -o pipefail && {{ bin_dir }}/calicoctl.sh version | grep 'Client Version:' | awk '{ print $3}'"
is not working on my environment.

Even this:

    - name: Get current calico version
      shell: "/usr/local/bin/calicoctl.sh version"
      register: temp
      become: true

     - debug:
        var: temp.stdout

return nothing.

If I run it on my master node I got the answer:
$ sudo /usr/local/bin/calicoctl.sh version
Client Version: v3.29.1
Git commit: ddfc3b1ea
Cluster Version: v3.29.1
Cluster Type: kdd,k8s,bgp,kubeadm

So the problem is somewhere here.