Containering of processes

[tomkralidis]

Issue by jachym
Thursday Sep 24, 2015 at 08:45 GMT
Originally opened as https://github.com/PyWPS/pywps-4/issues/69

---

Each process execute method should get private safe container before executing. With limited RAM, CPU cores, disk space, limited network communication and other. Possibly using Docker or Vagrant or even something better.

[cehbrecht]

just got a link to "function as a service":

https://dev.to/developius/functions-as-a-service---deploying-functions-to-docker-swarm-via-a-cli

Haven't really looked into it ... but might be useful as an example for using docker/swarm.

[adamsteer]

Just skim read @cehbrecht's linked article - it looks like a good concept to a not-really-developer.

If PyWPS implements containerised services, could we please have options:

a. to not run containers and let processes use the host system resources as they do now
b. to fire up a scalable container set. My near term use case for this is scaling processes beyond the machine where PyWPS sits - either to my organisation's own cloud resources or (insert commercial provider here)'s.

[jorgejesus]

Email from Adam that made a master thesis on the topic, he implemented docker

Hi devs,

during work on my thesis[1] I've added Docker extension to pywps. At my git there is a demo server (pywps-demo with extended config file) [2]. To run it with Docker extension it's necessary to use pywps[3] and OWSLib[4] from my git. Also you need Docker engine to build image and create container (the extension was tested only on Linux).

I will be glad for any comment. Recently, I faced a problem with zombie processes when using multiprocessing python package for deamon that manages created docker containers. Eventually, I used threading[5] instead but not really sure if it's the right solution. So any opinion will be appreciated.

Cheers,
Adam

[1] https://github.com/ctu-geoforall-lab-projects/dp-laza-2018/blob/master/text/adam-laza-dp-2018.pdf
[2] https://github.com/lazaa32/pywps-flask
[3] https://github.com/lazaa32/pywps
[4] https://github.com/lazaa32/OWSLib/tree/sync_request
[5] https://github.com/lazaa32/pywps/blob/develop/pywps/processing/container.py#L73

[huard]

An HPC analyst working with us on deploying PyWPS on a cluster suggested that Singularity would be a better fit than docker for integration into HPC architectures (at least in Canada). I can't answer questions about any of this, but I can redirect them.

[jachym]

Hi I was helping to Adam with this (making it to single commit). But this single commit does not seem to work properly according to Adam's tests. Now he is on New Zealand looking for some job. Not sure, when he get's back to it (if ever). Any volunteer to have a look at my merge and see, if it could work? thanks [1] https://github.com/jachym/pywps/tree/docker čt 28. 6. 2018 v 21:24 odesílatel David Huard <notifications@github.com> napsal:

…

An HPC analyst working with us on deploying PyWPS on a cluster suggested that Singularity would be a better fit than docker for integration into HPC architectures (at least in Canada). I can't answer questions about any of this, but I can redirect them. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#61 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABZKV2I5VRAGXAXBrlC6JwxCOUUQWM65ks5uBS14gaJpZM4HWXj0> .

-- Jachym Cepicky e-mail: jachym.cepicky gmail com URL: http://les-ejk.cz GPG: http://les-ejk.cz/pgp/JachymCepicky.pgp