Open
Description
The else
condition in SparkDistributedBackend._is_support_stage_scheduling
seems to always print a "Spark version does not support stage-level scheduling." warning whether the context can be parallelize or not.
def _is_support_stage_scheduling(self):
spark_master = self._spark_context.master
is_spark_local_mode = spark_master == "local" or spark_master.startswith("local[")
if is_spark_local_mode:
support_stage_scheduling = False
warnings.warn("Spark local mode doesn't support stage-level scheduling.")
else:
support_stage_scheduling = hasattr(
self._spark_context.parallelize([1]), "withResources"
)
warnings.warn("Spark version does not support stage-level scheduling.")
return support_stage_scheduling
If seems like maybe it should only print the warning if support_state_scheduling
was false?
else:
support_stage_scheduling = hasattr(
self._spark_context.parallelize([1]), "withResources"
)
if not support_stage_scheduling:
warnings.warn("Spark version does not support stage-level scheduling.")
Metadata
Metadata
Assignees
Labels
No labels