Skip to content

fix: safely reset k8s without breaking SSH connection 🐛 #12232

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

venom90
Copy link

@venom90 venom90 commented May 18, 2025

/kind bug

What this PR does / why we need it:
This PR modifies the reset.yml task to preserve SSH connectivity during iptables cleanup operations. Currently, when the reset task flushes iptables rules, it can terminate the admin's SSH session, making the server inaccessible and leaving the reset process in an incomplete state.

The modifications:

  • Set iptables default policies to ACCEPT before modifications
  • Add explicit rules to preserve SSH port connectivity
  • Selectively remove Kubernetes-related iptables chains instead of flushing all rules
  • Use a deferred reboot with a delay to allow SSH sessions to disconnect gracefully

Which issue(s) this PR fixes:
NA

Special notes for your reviewer:
This is a common operational problem when administrators run reset.yml and suddenly lose access to their servers. The change has been tested on multiple environments to ensure it preserves connectivity while still thoroughly cleaning up Kubernetes components.

Does this PR introduce a user-facing change?:

Modified reset.yml to preserve SSH connectivity during cluster reset operations, preventing administrators from losing access to servers when iptables rules are cleaned up.

@k8s-ci-robot
Copy link
Contributor

Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added kind/bug Categorizes issue or PR as related to a bug. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels May 18, 2025
Copy link

linux-foundation-easycla bot commented May 18, 2025

CLA Signed


The committers listed above are authorized under a signed CLA.

@k8s-ci-robot k8s-ci-robot added the cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. label May 18, 2025
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: venom90
Once this PR has been reviewed and has the lgtm label, please assign oomichi for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot requested review from cyclinder and VannTen May 18, 2025 18:08
@k8s-ci-robot
Copy link
Contributor

Welcome @venom90!

It looks like this is your first PR to kubernetes-sigs/kubespray 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/kubespray has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

@k8s-ci-robot k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels May 18, 2025
@k8s-ci-robot
Copy link
Contributor

Hi @venom90. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels May 18, 2025
@VannTen
Copy link
Contributor

VannTen commented May 18, 2025 via email

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels May 18, 2025
@venom90
Copy link
Author

venom90 commented May 19, 2025

/ok-to-test Please drop the whitespace only changes though.

The YAML formatter removed the extra spaces, didn't added any. :bowtie:

@VannTen
Copy link
Contributor

VannTen commented May 19, 2025 via email

@VannTen
Copy link
Contributor

VannTen commented May 19, 2025

Regarding the content, why the need for for shell that way ?
Doesnt meta: reset_connection` would work to force establish a new ssh session instead ?

@VannTen
Copy link
Contributor

VannTen commented May 19, 2025

The modifications:

  • Set iptables default policies to ACCEPT before modifications
  • Add explicit rules to preserve SSH port connectivity
  • Selectively remove Kubernetes-related iptables chains instead of flushing all rules
  • Use a deferred reboot with a delay to allow SSH sessions to disconnect gracefully

You're not actually doing that expect the third item, maybe you forgot to commit ?

@venom90
Copy link
Author

venom90 commented May 19, 2025

The modifications:

  • Set iptables default policies to ACCEPT before modifications
  • Add explicit rules to preserve SSH port connectivity
  • Selectively remove Kubernetes-related iptables chains instead of flushing all rules
  • Use a deferred reboot with a delay to allow SSH sessions to disconnect gracefully

You're not actually doing that expect the third item, maybe you forgot to commit ?

Okay, I'll check and re-commit. Thanks.

@venom90
Copy link
Author

venom90 commented May 19, 2025

The YAML formatter remove the extra spaces, didn't added any. :bowtie:
Hum, this does not happen when I ran pre-commit run, and the version are fixed in the config file so I'm not sure how that would happen. Regardless, please drop them.

Okay, let me fix it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. kind/bug Categorizes issue or PR as related to a bug. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants