天道酬勤,学无止境

Multiprocess vs Multithread Python time taken

I have 2 simple functions(loops over a range) that can run separately without any dependency.. I'm trying to run this 2 functions both using the Python multiprocessing module as well as multithreading module..

When I compared the output, I see the multiprocess application takes 1 second more than the multi-threading module..

I read multi-threading is not that efficient because of the Global interpreter lock...

Based on the above statements -
1. Is is best to use the multiprocessing if there is no dependency between 2 processes?
2. How to calculate the number of processes/threads that I can run in my machine for maximum efficiency..
3. Also, is there a way to calculate the efficiency of the program by using multithreading...

Multithread module...

from multiprocessing import Process

import thread
import platform

import os
import time
import threading
class Thread1(threading.Thread):
    def __init__(self,threadindicator):
        threading.Thread.__init__(self)
        self.threadind = threadindicator

    def run(self):
        starttime = time.time() 
        if self.threadind == 'A':
            process1()
        else:
            process2()
        endtime = time.time()
        print 'Thread 1 complete : Time Taken = ', endtime - starttime

def process1():
    starttime = time.time() 
    for i in range(100000):
        for j in range(10000):
            pass        
    endtime = time.time() 

def process2():
    for i in range(1000):
        for j in range(1000):
            pass

def main():

    print 'Main Thread'
    starttime = time.time()
    thread1 = Thread1('A')
    thread2 = Thread1('B')
    thread1.start()
    thread2.start()
    threads = []
    threads.append(thread1)
    threads.append(thread2)

    for t in threads:
        t.join()
    endtime = time.time()
    print 'Main Thread Complete , Total Time Taken = ', endtime - starttime


if __name__ == '__main__':
    main()

multiprocess module

from multiprocessing import Process
import platform

import os
import time

def process1():
#     print 'process_1 processor =',platform.processor()
    starttime = time.time() 
    for i in range(100000):
        for j in range(10000):
            pass
    endtime = time.time()
    print 'Process 1 complete : Time Taken = ', endtime - starttime 


def process2():
#     print 'process_2 processor =',platform.processor()
    starttime = time.time()
    for i in range(1000):
        for j in range(1000):
            pass
    endtime = time.time()
    print 'Process 2 complete : Time Taken = ', endtime - starttime

def main():
    print 'Main Process start'
    starttime = time.time()
    processlist = []

    p1 = Process(target=process1)
    p1.start()
    processlist.append(p1)

    p2 = Process(target = process2)
    p2.start()
    processlist.append(p2)

    for i in processlist:
        i.join()
    endtime = time.time()
    print 'Main Process Complete - Total time taken = ', endtime - starttime

if __name__ == '__main__':
    main()

评论

If you have two CPUs available on your machine, you have two processes which don't have to communicate, and you want to use both of them to make your program faster, you should use the multiprocessing module, rather than the threading module.

The Global Interpreter Lock (GIL) prevents the Python interpreter from making efficient use of more than one CPU by using multiple threads, because only one thread can be executing Python bytecode at a time. Therefore, multithreading won't improve the overall runtime of your application unless you have calls that are blocking (e.g. waiting for IO) or that release the GIL (e.g. numpy will do this for some expensive calls) for extended periods of time. However, the multiprocessing library creates separate subprocesses, and therefore several copies of the interpreter, so it can make efficient use of multiple CPUs.

However, in the example you gave, you have one process that finishes very quickly (less than 0.1 seconds on my machine) and one process that takes around 18 seconds to finish on the other. The exact numbers may vary depending on your hardware. In that case, nearly all the work is happening in one process, so you're really only using one CPU regardless. In this case, the increased overhead of spawning processes vs threads is probably causing the process-based version to be slower.

If you make both processes do the 18 second nested loops, you should see that the multiprocessing code goes much faster (assuming your machine actually has more than one CPU). On my machine, I saw the multiprocessing code finish in around 18.5 seconds, and the multithreaded code finish in 71.5 seconds. I'm not sure why the multithreaded one took longer than around 36 seconds, but my guess is the GIL is causing some sort of thread contention issue which is slowing down both threads from executing.

As for your second question, assuming there's no other load on the system, you should use a number of processes equal to the number of CPUs on your system. You can discover this by doing lscpu on a Linux system, sysctl hw.ncpu on a Mac system, or running dxdiag from the Run dialog on Windows (there's probably other ways, but this is how I always do it).

For the third question, the simplest way to figure out how much efficiency you're getting from the extra processes is just to measure the total runtime of your program, using time.time() as you were, or the time utility in Linux (e.g. time python myprog.py). The ideal speedup should be equal to the number of processes you're using, so a 4 process program running on 4 CPUs should be at most 4x faster than the same program with 1 process, assuming you get maximum benefit from the extra processes. If the other processes aren't helping you that much, it will be less than 4x.

受限制的 HTML

  • 允许的HTML标签:<a href hreflang> <em> <strong> <cite> <blockquote cite> <code> <ul type> <ol start type> <li> <dl> <dt> <dd> <h2 id> <h3 id> <h4 id> <h5 id> <h6 id>
  • 自动断行和分段。
  • 网页和电子邮件地址自动转换为链接。

相关推荐
  • Deadlock with logging multiprocess/multithread python script
    I am facing the problem with collecting logs from the following script. Once I set up the SLEEP_TIME to too "small" value, the LoggingThread threads somehow block the logging module. The script freeze on logging request in the action function. If the SLEEP_TIME is about 0.1 the script collect all log messages as I expect. I tried to follow this answer but it does not solve my problem. import multiprocessing import threading import logging import time SLEEP_TIME = 0.000001 logger = logging.getLogger() ch = logging.StreamHandler() ch.setFormatter(logging.Formatter('%(asctime)s %(levelname)s %
  • 使用日志记录多进程/多线程python脚本的死锁(Deadlock with logging multiprocess/multithread python script)
    问题 我从以下脚本收集日志时遇到问题。 一旦将SLEEP_TIME设置为太小的值,LoggingThread线程就会以某种方式阻塞日志记录模块。 该脚本将冻结action功能中的日志记录请求。 如果SLEEP_TIME约为0.1,那么脚本将按我的预期收集所有日志消息。 我尝试遵循此答案,但不能解决我的问题。 import multiprocessing import threading import logging import time SLEEP_TIME = 0.000001 logger = logging.getLogger() ch = logging.StreamHandler() ch.setFormatter(logging.Formatter('%(asctime)s %(levelname)s %(funcName)s(): %(message)s')) ch.setLevel(logging.DEBUG) logger.setLevel(logging.DEBUG) logger.addHandler(ch) class LoggingThread(threading.Thread): def __init__(self): threading.Thread.__init__(self) def run(self): while True: logger
  • How to multiprocess, multithread a big file by dividing into small chunks based on values of a particular column?
    I have written a python program for a biological process https://codereview.stackexchange.com/questions/186396/solve-the-phase-state-between-two-haplotype-blocks-using-markov-transition-proba . If you look into that program you can see that the program takes a lots of time in computing data from two consecutive lines (or keys, vals) at a time. I am not putting the whole code here, but for simplicity I am creating a mock file and mock program (given below) which behaves similarly at simplest level. In this mock program I am calculating, say len(vals) column and writing it back to an output file
  • Python入门(目录全览)-------已更新到第八篇完结
    文章目录 Python入门(目录全览)==已更新到第八篇完结==第一篇 计算机基础第二篇 Python解释器和集成环境第三篇 Python基础第四篇 Python进阶第五篇 文件处理第六篇 函数基础第七篇 函数进阶第八篇 模块基础第九篇 Python常用模块第十篇 面向对象基础第十一篇 面向对象进阶第十二篇 面向对象高阶第十三篇 网络编程第十四篇 并发编程第十五篇 MySQL数据库 Python入门(目录全览)已更新到第八篇完结 第一篇 计算机基础 002 计算机基础之编程 003 计算机组成原理 004 计算机操作系统 005 编程语言分类 006 网络的瓶颈效应 007 计算机基础小结 第二篇 Python解释器和集成环境 008 Python和Python解释器 009 Python解释器安装 010 Anaconada安装 011 Python解释器镜像源修改 012 执行Python程序的两种方式 013 Python的IDE之Pycharm的使用 014 Python的IDE之Jupyter的使用 015 pip的使用 第三篇 Python基础 017 变量 018 常量 019 Python变量内存管理 020 变量的三个特征 021 花式赋值 022 注释 023 数据类型基础 024 数字类型 025 字符串类型 026 列表类型 027 字典类型 028
  • Python从入门到放弃(目录)
    Python基础、网络编程、并发编程、MySQL数据库全教程!!! 计算机基础、计算机组成原理、计算机操作系统、计算机网络; Python数据类型、流程控制、文件处理、函数、面向对象; 网络编程的IOS七层协议、TCP协议的三次握手四次挥手、基于TCP/UDP协议的socket套接字编程; 并发编程的多进程、多线程、协程、GIL全局解释器锁、五个IO模型; MySQL数据库的增删改查、索引优化、高级用法(游标、视图、事务、存储过程)满足你的所有要求。 用轻松愉快的语言+图片让你在学习Python的同时获得乐趣,丰富的代码展示让你知其所以然,说多无益,快上车!!!目录第一篇 markdown编辑器第二篇 计算机基础第三篇 Python解释器和集成环境第三篇 Python基础第四篇 Python进阶第五篇 文件处理第六篇 函数基础第七篇 函数进阶第八篇 模块基础第九篇 Python常用模块第十篇 面向对象基础第十一篇 面向对象进阶第十二篇 面向对象高阶第十三篇 网络编程第十四篇 并发编程第十五篇 MySQL数据库第十六篇 推荐阅读第一篇 markdown编辑器001 markdown基本语法第二篇 计算机基础002 计算机基础之编程003 计算机组成原理004 计算机操作系统005 编程语言分类006 网络的瓶颈效应007 计算机基础小结第三篇 Python解释器和集成环境008
  • How can I process a tarfile with a Python multiprocessing pool?
    I'm trying to process the contents of a tarfile using multiprocessing.Pool. I'm able to successfully use the ThreadPool implementation within the multiprocessing module, but would like to be able to use processes instead of threads as it would possibly be faster and eliminate some changes made for Matplotlib to handle the multithreaded environment. I'm getting an error that I suspect is related to processes not sharing address space, but I'm not sure how to fix it: Traceback (most recent call last): File "test_tarfile.py", line 32, in <module> test_multiproc() File "test_tarfile.py", line 24
  • Python程序中的进程操作-开启多进程(multiprocess.process)
    目录一、multiprocess模块二、multiprocess.process模块三、process模块介绍3.1 方法介绍3.2 属性介绍3.3 在windows中使用process模块的注意事项四、使用process模块创建进程4.1 在Python中启动的第一个子进程4.2 join方法4.3 查看主进程和子进程的进程号4.4 多个进程同时运行4.5 多个进程同时运行,再谈join方法(1)4.6 多个进程同时运行,再谈join方法(2)4.7 通过继承Process类开启进程4.8 进程之间的数据隔离问题五、守护进程5.1 守护进程的启动5.2 主进程代码执行结束守护进程立即结束六、socket聊天并发实例6.1 使用多进程实现socket聊天并发-server端6.2 使用多进程实现socket聊天并发-client端七、多进程中的其他方法7.1 进程对象的其他方法:terminate和is_alive7.2 进程对象的其他属性:pid和name之前我们已经了解了很多进程相关的理论知识,了解进程是什么应该不再困难了,刚刚我们已经了解了,运行中的程序就是一个进程。所有的进程都是通过它的父进程来创建的。因此,运行起来的python程序也是一个进程,那么我们也可以在程序中再创建进程。多个进程可以实现并发效果,也就是说,当我们的程序中存在多个进程的时候,在某些时候
  • python进阶(15)多线程与多进程效率测试
    前言在Python中,计算密集型任务适用于多进程,IO密集型任务适用于多线程 正常来讲,多线程要比多进程效率更高,因为进程间的切换需要的资源和开销更大,而线程相对更小,但是我们使用的Python大多数的解释器是Cpython,众所周知Cpython有个GIL锁,导致执行计算密集型任务时多线程实际只能是单线程,而且由于线程之间切换的开销导致多线程往往比实际的单线程还要慢,所以在 python 中计算密集型任务通常使用多进程,因为各个进程有各自独立的GIL,互不干扰。 而在IO密集型任务中,CPU时常处于等待状态,操作系统需要频繁与外界环境进行交互,如读写文件,在网络间通信等。在这期间GIL会被释放,因而就可以使用真正的多线程。 上面都是理论,接下来实战看看实际效果是否符合理论 练习"""多线程多进程模拟执行效率""" from multiprocessing import Pool from threading import Thread import time, math def simulation_IO(a): """模拟IO操作""" time.sleep(3) def simulation_compute(a): """模拟计算密集型任务""" for i in range(int(1e7)): math.sin(40) + math.cos(40) return def
  • multithreading or multiprocessing
    I am designing a dedicated syslog-processing daemon for Linux that needs to be robust and scalable and I'm debating multithread vs. multiprocess. The obvious objection with multithreading is complexity and nasty bugs. Multi-processes may impact performance because of IPC communications and context switching. "The Art of Unix Programming" discusses this here. Would you recommend a process-based system (like Apache) or a multi-threaded approach?
  • Python多进程multiprocess之多进程返回值获取
    由于python内部GIL(全局解释器锁)的存在,所以python的线程实际上并不能很好的起到任务并行处理的作用,尤其是无法充分利用系统多核的优势,因此想要利用多核处理并行任务,就需要用到多进程——multiprocess。由于多进程任务时,经常需要返回函数的结果,这里主要关注进程返回值的获取。 首先定义一个简单函数: def add(n): s = 0 for i in range(n): s += 1 print(f'子进程{n}') return s 用串行的方法: import time lists = [10000000, 20000000, 5000000, 30000000] results = [] start = time.time() for n in lists: results.append(add(n)) print(f'recurrent time cost: {time.time() - start}s') 打印结果: 子进程10000000 子进程20000000 子进程5000000 子进程30000000 recurrent time cost: 3.95 s 很明显,是按顺序执行函数的。下面用多进程实现,看速度是否会提升。 from multiprocessing import Pool pool = Pool(processes=4) #
  • Django's SuspiciousOperation Invalid HTTP_HOST header
    After upgrading to Django 1.5, I started getting errors like this: Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 92, in get_response response = middleware_method(request) File "/usr/local/lib/python2.7/dist-packages/django/middleware/common.py", line 57, in process_request host = request.get_host() File "/usr/local/lib/python2.7/dist-packages/django/http/request.py", line 72, in get_host "Invalid HTTP_HOST header (you may need to set ALLOWED_HOSTS): %s" % host) SuspiciousOperation: Invalid HTTP_HOST header (you may need to
  • 是否可以对在Python中返回某些内容的函数进行多进程处理?(Is it possible to multiprocess a function that returns something in Python?)
    问题 在Python中,我看到了许多示例,其中调用了多处理,但目标仅打印了一些内容。 我有一种情况,目标返回2个变量,以后需要使用。 例如: def foo(some args): a = someObject b = someObject return a,b p1=multiprocess(target=foo,args(some args)) p2=multiprocess(target=foo,args(some args)) p3=multiprocess(target=foo,args(some args)) 怎么办? 我可以执行.start和.join,但是如何检索单个结果? 我需要捕获我执行的所有作业的返回值a,b,然后对其进行处理。 回答1 是的,可以-您可以使用多种方法。 最简单的方法之一是共享Queue 。 在此处查看示例:http://eli.thegreenplace.net/2012/01/16/python-parallelizing-cpu-bound-tasks-with-multiprocessing/ 回答2 您正在使用多个进程来做一些令人尴尬的并行工作,那么为什么不使用Pool ? Pool将负责启动流程,检索结果并将结果返回给您。 我使用pathos ,它具有multiprocessing的分支,因为它的序列化比标准库提供的版本好得多。 (
  • Python多进程分析(Python multiprocess profiling)
    问题 我正在努力弄清楚如何分析一个简单的多进程python脚本 import multiprocessing import cProfile import time def worker(num): time.sleep(3) print 'Worker:', num if __name__ == '__main__': for i in range(5): p = multiprocessing.Process(target=worker, args=(i,)) cProfile.run('p.start()', 'prof%d.prof' %i) 我正在启动5个进程,因此cProfile会生成5个不同的文件。 在每个方法的内部,我想看到我的方法'worker'大约需要3秒钟才能运行,但是我只看到了'start'方法中正在发生的事情。 如果有人可以向我解释这一点,我将不胜感激。 更新:基于公认答案的工作示例: import multiprocessing import cProfile import time def test(num): time.sleep(3) print 'Worker:', num def worker(num): cProfile.runctx('test(num)', globals(), locals(), 'prof%d.prof' %num)
  • Event Loop vs Multithread blocking IO
    I was reading a comment about server architecture. http://news.ycombinator.com/item?id=520077 In this comment, the person says 3 things: The event loop, time and again, has been shown to truly shine for a high number of low activity connections. In comparison, a blocking IO model with threads or processes has been shown, time and again, to cut down latency on a per-request basis compared to an event loop. On a lightly loaded system the difference is indistinguishable. Under load, most event loops choose to slow down, most blocking models choose to shed load. Are any of these true? And also
  • Python程序中的进程操作-进程间通信(multiprocess.Queue)
    目录一、进程间通信二、队列2.1.1 方法介绍2.1.2 其他方法(了解)2.1 概念介绍——multiprocess.Queue三、代码实例——multiprocess.Queue3.1 单看队列用法3.2 子进程发送数据给父进程3.3 批量生产数据放入队列再批量获取结果四、生产者消费者模型4.1 为什么要使用生产者和消费者模式4.2 什么是生产者消费者模式4.3 基于队列实现生产者消费者模型4.4 改良版——生产者消费者模型4.5 主进程在生产者生产完毕后发送结束信号None4.6 多个消费者的例子:有几个消费者就需要发送几次结束信号五、JoinableQueue([maxsize])5.1 方法介绍5.2 JoinableQueue队列实现消费之生产者模型一、进程间通信IPC(Inter-Process Communication)二、队列2.1 概念介绍——multiprocess.Queue创建共享的进程队列,Queue是多进程安全的队列,可以使用Queue实现多进程之间的数据传递。Queue([maxsize])创建共享的进程队列。参数 :maxsize是队列中允许的最大项数。如果省略此参数,则无大小限制。底层队列使用管道和锁定实现。2.1.1 方法介绍Queue([maxsize]) :创建共享的进程队列。maxsize是队列中允许的最大项数。如果省略此参数
  • python multiprocess 子进程和主进程同时抛出异常时子进程无法退出
    在使用python的multiprocess库时,如果在主进程中的处理子进程的返回函数callback或者处理子进程的错误的函数errorbackerror中抛出异常,则子进程无法退出。 (1)errorcallback中抛出异常 import multiprocessing import os import time import traceback def work(args): try: time.sleep(1) print(os.get_pid()) except Exception as e: # 子进程报错 a += 1 print(traceback.format_exc(3)) def callback(args): print(args) def errorcallback(args): # 处理子进程错误的函数报错 a = 1/0 print(args) def task(): pool = multiprocessing.Pool(processes=2) for i in range(10): res = pool.apply_async(work,args=(i,),callback=callback,error_callback=errorcallback) # for r in res.get(): # print(r) pool.close()
  • Python multiprocess profiling
    I'm struggling to figure out how to profile a simple multiprocess python script import multiprocessing import cProfile import time def worker(num): time.sleep(3) print 'Worker:', num if __name__ == '__main__': for i in range(5): p = multiprocessing.Process(target=worker, args=(i,)) cProfile.run('p.start()', 'prof%d.prof' %i) I'm starting 5 processes and therefore cProfile generates 5 different files. Inside of each I want to see that my method 'worker' takes approximately 3 seconds to run but instead I'm seeing only what's going on inside the 'start'method. I would greatly appreciate if
  • Python、WSGI、多处理和共享数据(Python, WSGI, multiprocessing and shared data)
    问题 我对 mod_wsgi 的 multiproessing 功能以及将在具有多处理能力的 WSGI 服务器上执行的 WSGI 应用程序的一般设计感到有些困惑。 考虑以下指令: WSGIDaemonProcess example processes=5 threads=1 如果我理解正确,mod_wsgi 将产生 5 个 Python(例如 CPython)进程,并且这些进程中的任何一个都可以接收来自用户的请求。 文档说: 如果共享数据需要对所有应用程序实例可见,无论它们在哪个子进程中执行,并且一个应用程序对数据所做的更改立即可供另一个应用程序使用,包括在另一个子进程中执行的任何操作,外部数据存储,例如必须使用数据库或共享内存。 普通 Python 模块中的全局变量不能用于此目的。 但在那种情况下,当人们想要确保应用程序在任何 WSGI 条件(包括多处理条件)下运行时,它就会变得非常繁重。 例如,一个包含当前连接用户数量的简单变量——它应该是进程安全的从/向 memcached 读/写,还是一个 DB 或(如果这种超出标准的库机制可用)共享记忆? 并且代码会喜欢 counter = 0 @app.route('/login') def login(): ... counter += 1 ... @app.route('/logout') def logout():
  • python中的多进程或线程?(multiprocess or threading in python?)
    问题 我有一个python应用程序,它可以捕获数据集合,并为该集合中的每个数据执行任务。 由于涉及到延迟,因此该任务需要一些时间才能完成。 由于这种延迟,我不希望每个数据都随后执行任务,我希望它们全部并行发生。 我应该使用多进程吗? 或执行此操作的线程? 我尝试使用线程,但遇到了一些麻烦,通常某些任务实际上不会执行。 回答1 如果您确实受到计算的限制,那么使用多处理模块可能是最轻量级的解决方案(就内存消耗和实现难度而言)。 如果您受I / O约束,那么使用线程模块通常会给您带来良好的效果。 确保使用线程安全存储(例如Queue)将数据移交给线程。 否则,在生成它们时将它们唯一的数据交给他们。 PyPy专注于性能。 它具有许多功能,可帮助进行计算绑定处理。 他们还支持软件事务存储,尽管这还不是生产质量。 可以保证,您可以使用比多处理更简单的并行或并发机制(有一些笨拙的要求)。 无堆栈Python也是一个好主意。 如上所述,Stackless具有可移植性问题。 空载燕子是有前途的,但现在已不复存在。 Pyston是另一个专注于速度的(未完成的)Python实现。 它采用的方法不同于PyPy,可能会产生更好的(或略有不同)加速。 回答2 任务按顺序运行,但是您有并行运行的错觉。 当用于文件或连接I / O时,任务很好,因为它们是轻量级的。 带池的多进程可能是您的正确解决方案
  • python多进程模块multiprocessing学习笔记(2)之多线程与多进程运行效率对比
    参考链接: 【莫烦Python】Multiprocessing 让你的多核计算机发挥真正潜力 Python 参考链接: 莫烦多进程学习网站 参考链接: multiprocessing — 基于进程的并行 参考链接: class multiprocessing.Queue([maxsize]) 参考链接: queue — 一个同步的队列类 test03_1.py # 第4集 import multiprocessing as mp import multiprocessing import time import threading as td import queue def job(q): res = 0 for i in range(10000000): res += i+i**2+i**3 q.put(res) # queue def multicore(): q = mp.Queue() p1 = mp.Process(target=job, args=(q,)) p2 = mp.Process(target=job, args=(q,)) p1.start() p2.start() p1.join() p2.join() res1 = q.get() res2 = q.get() print('multicore:' , res1+res2) def normal()