separate public and private domains, allow to choose for stacks

[linki]

Alternative for #696 so we can compare what the difference is.

It introduces two more flags and should be fully backwards compatible. The first new flag --cluster-internal-domain works similar to --cluster-domain but tells stackset-controller that it's an internal domain (e.g. ingress.cluster.local). By default external and internal domains are both processed (like today) so there's no change when moving a domain from --cluster-domain to --cluster-internal-domain.

However, the second flag --ignore-public-domains-on-stacks can be used to tell stackset-controller to not use public domains for stack hosts. So, once the domains have been split into two separate groups, this flag can be used to change the behaviour.

Below is the intended usage. The first example is the setup on current clusters, the second one will be the setup on legacy clusters and the third one will be the setup for EKS clusters (only difference is the value of the flag).

Imagine a stackset specifying the following ingress section:

spec:
  ingress:
    hosts:
    - my-app.teapot.zalan.do
    - my-app.ingress.cluster.local

Running stackset-controller with the following configuration will result in the following:

--cluster-domain teapot.zalan.do --cluster-domain ingress.cluster.local (current setting)

main ingress	stack ingress
my-app.teapot.zalan.do my-app.ingress.cluster.local	my-app-v1.teapot.zalan.do my-app-v1.ingress.cluster.local

That's the current result and backwards compatible.

Then, split cluster domains into a public and private group. There's no change and therefore backwards compatible.

--cluster-domain teapot.zalan.do --cluster-internal-domain ingress.cluster.local (future setting for legacy clusters)

main ingress	stack ingress
my-app.teapot.zalan.do my-app.ingress.cluster.local	my-app-v1.teapot.zalan.do my-app-v1.ingress.cluster.local

Finally, switch the flip on EKS clusters to skip public domains on Stacks.

--cluster-domain teapot.zalan.do --cluster-internal-domain ingress.cluster.local --ignore-public-domains-on-stacks (future setting for EKS clusters)

main ingress	stack ingress
my-app.teapot.zalan.do my-app.ingress.cluster.local	my-app-v1.ingress.cluster.local

The idea is to run with the public/private split and without --ignore-public-domains-on-stacks (false) on legacy clusters. On EKS clusters we would run it with --ignore-public-domains-on-stacks.

If the Stackset doesn't make use of the ingress.cluster.local domain in its definition, then the results above are the same but with the internal domain (i.e. just the teapot.zalan.do DNS names and no DNS name at all for the stack on EKS).

[linki]

Looks like "per-stack" is already somewhat supported. I'm getting quite the same (correct) results on the master branch when running with: --cluster-domain ingress.cluster.local. Maybe worth looking into if that already solves our problem.

[demonCoder95]

I think if we define only the --cluster-domain ingress.cluster.local then we will indeed get the correct behavior as you point out. If we have multiple domains, then of course the per-stack domains generated will be multiple, like you also describe in the PR description.