Support taint propagation to the underlying nodes

[Archisman-Mridha]

What would you like to be added (User Story)?

It'll be very helpful if we have this feature support :

Declare what taints should be applied to the underlying node(s) in the Machine / MachineDeployment / MachinePool spec.

Detailed Description

Especially, when I'm creating a MachinePool. I want to specify taints in the MachinePool spec, and those taints will be applied to each node managed by that MachinePool.
Those nodes will then be reserved for specific types of workloads which have tolerations against those node taints.

This is will help manage node taints in a declarative manner using GitOps.

Anything else you would like to add?

I've come up with a Plan Of Action here : Obmondo@ab19c36

Label(s) to be applied

/kind feature
One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels.

[fabriziopandini]

If it is only for machine creation, the current solution is to rely on the capabilities of bootstrap providers, but I'm aware of the fact that not all the MP implementations use a bootstrap providers.

If we are looking for a more generic story, that do not apply only on creation, this requires some design, especially for figuring out the interactions between the new "taint propagation" feature and taints defined by bootstrap providers

In order to define next steps, might be worth taking a look at https://cluster-api.sigs.k8s.io/developer/architecture/controllers/metadata-propagation and underlying proposal:

• https://github.com/kubernetes-sigs/cluster-api/blob/main/docs/proposals/20220927-label-sync-between-machine-and-nodes.md
• 20221003-In-place-propagation-of-Kubernetes-objects-only-changes.md

cc @jackfrancis @mboersma @willie-yao

cc @elmiko this discussion might be useful to revive #7685

[Archisman-Mridha]

@fabriziopandini Thanks for the resources you linked. I wasn't aware of the JoinConfiguration.NodeRegistration.Taints field in the KubeadmConfigTemplate custom resource. I can use that to apply taint to newly minted worker nodes (managed by a MachinePool in my case). Currently this is what my company needs.

However there will be a downside - we can't in-place change (add / remove) taints for those worker nodes. And I'm interested in contributing the solution for this. We want the user to be able to specify the taints in the Machine spec. ClusterAPI will then sync those taints between the Machine resource and the corresponding Kubernetes node.

That's what I'm trying to implement here : Obmondo@ab19c36. Can you please give any opinion on whether I'm going in the right direction or not ?

[JoelSpeed]

However there will be a downside - we can't in-place change (add / remove) taints for those worker nodes

Removing taints becomes problematic if you have the potential for multiple actors to add taints (which is always the case, think about cordoning a node using kubectl).

Typically in cases where we've had fields like this before, you'd omit the removal side of things, and the taints would be added if they are in the spec, and manually removed if removed from spec.

However, now that we have SSA, in theory we can make this work. If we were to use SSA to apply the taints, then whenever we apply the list of taints from our side, any that we applied (and therefore own according to field managers) that we no longer specify would be removed for us. If they are later re-added by something other, then they would be the field owner now and we wouldn't know that we previously owned it, so, this could be safe if we use SSA

Edit: Scratch that, taints are atomic 😢

[fabriziopandini]

@JoelSpeed good catch about taints being atomic/the fact that we cannot remove taint automatically.

@Archisman-Mridha

That's what I'm trying to implement here : Obmondo@ab19c36. Can you please give any opinion on whether I'm going in the right direction or not ?

I don't have bandwidth to research this ATM, but at first sight

• You should take care of the full propagation story, down from CC to Machines and nodes (the change can't be limited to the machine/machine controller)
• You should figure out the interactions between the new "taint propagation" feature and taints defined by bootstrap providers (and consider that not all the bootstrap providers are CABPK)
• If you are interested in getting this to work for MachinePool, this could require some additional considerations. e.g. MachinePool machines are implemented partially and have some caveats, in-place propagation is not yet implemented for MachinePools and probably more (Disclaimer: I'm not using MachinePools, I just help there due to lack of maintainers)

Considering all of this, I strongly suggest to do some design work before implementing, identifying API types changes, controller impacted etc.

It could be a comment on this thread for starting, then we can figure out if it is also needed to amend some of the doc/proposals linked above or else

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[nrb]

/remove-lifecycle stale
/assign

I'm looking into this and #7409. For annotations, most of the propagation amongst Kubernetes objects was handled via #7731, but such plumbing's not yet been done for taints.