Skip to content

Allow Solr to be run across Availability Zones #53

Closed
@AceHack

Description

@AceHack

I cannot find a way to make this operator spread across AZs in AWS, any suggestions would be welcome.

More info
bitnami/charts#927
https://kubernetes.io/docs/concepts/storage/storage-classes/#volume-binding-mode

Activity

HoustonPutman

HoustonPutman commented on Dec 2, 2019

@HoustonPutman
Contributor

Hey @AceHack , sorry for the delayed response. Things have been pretty crazy, and we have been blocked on the Kubebuilder v2 migration. Finally able to get into these other issues.

In general, Solr is relatively easy to run via multiple kubernetes clusters. What you need to do is make sure that you are using 1 zookeeper cluster for the solr clouds running in all kube clusters, so that they all share the same zookeeper connection information in their SolrCloud Specs. The data volumes don't have to be linked at all, since each Solr node manages it's data independently of other Solr Nodes.

We use different base ingress URLs to differentiate SolrCloud nodes in each kubernetes cluster.

So the solr-operator would be started in Kube cluster A with --base-ingress-domain a.test.com and in Kube cluster B with --base-ingress-domain b.test.com. Then when you create a SolrCloud example in each kube cluster (both connecting to the exact same zookeeper connection information), the nodes will register under their cluster-specific domain names. You just need to make sure that your DNS rules are set up correctly to route a.test.com and b.test.com to the correct ingress controllers.

We are hopefully going to be able to support more addressability options in the near future, such as ExternalDNS, and give you more control over how things are created/named.

AceHack

AceHack commented on Feb 21, 2020

@AceHack
Author

We don't want to run this in multiple clusters, we have 1 kubernetes cluster that spans 3 AZs. Currently, we have 3 different stateful sets (1 per AZ) so that Solr can be AZ aware in its autoscaling policies. It also used to be a nightmare with PVs/PVCs until the WaitForFirstConsumer volumeBindingMode for StorageClasses came out. To really support multi-AZ, I should be able to set a solr autoscaling policy that can reference the different AZs within the cluster.

AceHack

AceHack commented on Feb 21, 2020

@AceHack
Author

I would really like to set a policy that looks something like

{"replica": "#EQUAL", "shard": "#EACH", "sysprop.zone": "#ANY"}
or

{"replica": "#EQUAL", "shard": "#EACH", "sysprop.zone": ["us-east-1a", "us-east-1b", "us-east-1c"]}

HoustonPutman

HoustonPutman commented on Feb 21, 2020

@HoustonPutman
Contributor

I completely understand, and don't want the solr-operator to make any assumptions on how people configure their Solr clouds or Kube clusters.

Hopefully we can find a way to pass Node props to the environment variables of Solr. If that's not going to be an option, we might have to look into creating multiple statefulSets for each SolrCloud with multiple AZs.

Linking discussion on creating Kube options to pass Node labels to Pods: kubernetes/kubernetes#40610

AceHack

AceHack commented on Feb 21, 2020

@AceHack
Author

It's possible in an init container or part of the main container startup you can read what AZ you are in and write it to a file or something. Requires calling kubernetes APIs.

HoustonPutman

HoustonPutman commented on Feb 24, 2020

@HoustonPutman
Contributor

@sepulworld, to follow up on the post you made in the PR about autoscaling. I was doing some research and it looks like Kubernetes 1.16 gives us an additional pod Option: Pod.Spec. topologySpreadConstraints, documented here https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/

This would allow us to generically spread out pods across nodes (and AZs) without having to use autoscaling.

AceHack

AceHack commented on Feb 28, 2020

@AceHack
Author

Without the solr Auto scaling policy being AZ aware, no matter how the pod spread out you can’t be sure the replicas are spread out across AZs.

sepulworld

sepulworld commented on Feb 28, 2020

@sepulworld
Contributor
changed the title [-]AWS EKS Multi AZ[/-] [+]Allow Solr to be run across Availability Zones[/+] on Mar 5, 2020
HoustonPutman

HoustonPutman commented on Mar 5, 2020

@HoustonPutman
Contributor

Got a start on implementing this functionality. Let me know how y'all think it could be improved. This was just the first way I could get it done after tinkering for a while. I've very open to suggestions and improvements.

Also we should be good to go on the availability zone thing once the Labels PR & Collection Policy PR are merged, correct? Do y'all have an issue with me merging the policy PR now, or should it be blocked on this?

alittlec

alittlec commented on Jul 19, 2021

@alittlec

hi - is there any update on this?

HoustonPutman

HoustonPutman commented on Sep 2, 2021

@HoustonPutman
Contributor

Hey @alittlec , sorry didn't see your comment come in. Currently there are no plans to merge the currently linked PR. Instead in the v0.5.0 release we plan on support Pod Topology Spread Constraints.

The idea of autoscaling within Solr is completely changing from 8.x to 9.0, with the autoscaling framework deprecated and not recommended for production use. We will definitely need to investigate how the Solr Operator ties into the future autoscaling features starting with Solr 9.0.

added this to the main (v0.5.0) milestone on Oct 18, 2021
ozlerhakan

ozlerhakan commented on Sep 15, 2024

@ozlerhakan

Hi @HoustonPutman ,

I was wondering if you had a chance to investigate this concept within the operator for Solr 9.x?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

      Participants

      @sepulworld@AceHack@alittlec@ozlerhakan@HoustonPutman

      Issue actions

        Allow Solr to be run across Availability Zones · Issue #53 · apache/solr-operator