Open
Description
Context
InheritableThread is a thread class that pyspark provides to support sync between JVM and python threads.
The ThreadPool
class used in this package is python's original multiprocessing.pool.ThreadPool
.
What goods/bads will it bring? Dunno, I'll start looking around.
Proposal
Without touching this package's source code, a user can already nicely make their own ThreadPool
that uses their own Thread
class. If this package were to adopt the pyspark Thread class, then the steps may be:
- Create a custom
DummyProcess
that inheritspyspark.InheritableThread
. - Create a custom
ThreadPool
that uses the customDummyProcess
. - Use the custom
ThreadPool
in this package'sSparkDistributedBackend._get_pool
method.
Metadata
Metadata
Assignees
Labels
No labels