alt Hacker NewsBut python can fork itself and run multiple processes into one single container. Why would there be a need to run several containers to run several processes?
There's even the multiprocessing module in the stdlib to achieve this.
Replies
kccqzy • today at 4:41 PM
Forking and multi threading do not coexist. Even if one of your transitive dependencies decides to launch a thread that’s 99% idle, it becomes unsafe to fork.
➕ show 4 replies
Threads are cheap, you can do N work simultaneously with N threads in one process, without serialization, IPC or process creation overhead.
With multiprocessing, processes are expensive and work hogs each process. You must serialize data twice for IPC, that's expensive and time consuming.
You shouldn't have to break out multiple processes, for example, to do some simple pure-Python math in parallel. It doesn't make sense to use multiple processes for something like that because the actual work you want to do will be overwhelmed by the IPC overhead.
There are also limitations, only some data can be sent to and from multiple processes. Not all of your objects can be serialized for IPC.