天道酬勤,学无止境

concurrent.futures

Understanding concurrent.futures.Executor.map()

问题 回答1 它将允许您同时执行一个函数多次,而不是真正的并行执行。 在性能方面,我最近发现ProcessPoolExecutor.submit()和ProcessPoolExecutor.map()消耗相同数量的计算时间来完成相同的任务。 注意: .submit()返回一个未来对象(我们称之为 f),你需要使用它的f.result选项来查看它的结果。 另一方面, .map()直接返回一个迭代器。 当使用 sorted 方法将它们的结果转换为有序列表时,我发现在某些情况下,整个.map()代码的计算时间可能比整个.submit()代码更快。 当使用 list 方法将它们的结果转换为无序列表时,整个.submit()和.map()代码的计算时间是相同的。 此外,这些代码比使用排序方法的代码执行得更快。 您可以阅读我的答案中的详细信息。 在那里,我还分享了我的代码,您可以在其中看到它们是如何工作的。 我希望他们能对你有所帮助。 我没有使用过ThreadPoolExecutor ,所以无法详细评论。 但是,我读到它们的实现方式与ProcessPoolExecutor相同,它们更适合用于 I/O 绑定任务而不是 CPU 绑定任务。 您确实需要指定max_workers参数,即最大线程数,而在ProcessPoolExecutor max_workers是一个可选参数,默认为os.cpu

2022-05-15 22:51:09    分类:技术分享    python   python-3.x   concurrent.futures

How to break time.sleep() in a python concurrent.futures

问题 我正在玩concurrent.futures。 目前我未来的电话time.sleep(secs) 。 似乎 Future.cancel() 做的比我想象的要少。 如果未来已经在执行,那么time.sleep()不会被它取消。 wait() 的超时参数也是如此。 它不会取消我的time.sleep() 。 如何取消在 concurrent.futures 中执行的time.sleep() ? 对于测试,我使用 ThreadPoolExecutor。 回答1 如果您将函数提交给ThreadPoolExecutor ,则执行器将在线程中运行该函数并将其返回值存储在Future对象中。 由于并发线程的数量是有限的,您可以选择取消未完成的未来执行,但是一旦工作线程中的控制权已传递给可调用对象,就无法停止执行。 考虑这段代码: import concurrent.futures as f import time T = f.ThreadPoolExecutor(1) # Run at most one function concurrently def block5(): time.sleep(5) return 1 q = T.submit(block5) m = T.submit(block5) print q.cancel() # Will fail, because q is

2022-05-14 15:39:04    分类:技术分享    python   concurrency   concurrent.futures

Java Process did not quit after Future task completion

问题 这是我使用 Future 的代码片段。 import java.util.concurrent.*; import java.util.*; public class FutureDemo{ public FutureDemo(){ /* Future */ ExecutorService service = Executors.newFixedThreadPool(2); for ( int i=0; i<10; i++){ MyCallable myCallable = new MyCallable((long)i); Future<Long> futureResult = service.submit(myCallable); Long result = null; try{ result = futureResult.get(5000, TimeUnit.MILLISECONDS); }catch(TimeoutException e){ System.out.println("Time out after 5 seconds"); futureResult.cancel(true); }catch(InterruptedException ie){ System.out.println("Error: Interrupted"); }catch

2022-05-12 07:42:15    分类:技术分享    java   executorservice   concurrent.futures

Python concurrent.futures Error in atexit._run_exitfuncs: OSError: handle is closed only running in Visual studio Debugging Mode

I trying to implement the concurrent.futures have been having an issue, I generated a code that when executed in: Python Terminal Enters in some infinite loop in my last loop and do not finish the code, this code below do finish, however reproduces the error thgat i receive in the Debug Mode below So when I run it in Visual Studio Code Debug Mode Generates the folowing error: Error in atexit._run_exitfuncs: Traceback (most recent call last): File "C:\Users\debor\anaconda3\envs\multimidia_image\lib\concurrent\futures\process.py", line 102, in _python_exit thread_wakeup.wakeup() File "C:\Users

2022-05-02 03:06:00    分类:问答    python   multiprocessing   concurrent.futures

with concurrent.futures.ThreadPoolExecutor() as executor: ... does not wait

I am trying to use the ThreadPoolExecutor() in a method of a class to create a pool of threads that will execute another method within the same class. I have the with concurrent.futures.ThreadPoolExecutor()... however it does not wait, and an error is thrown saying there was no key in the dictionary I query after the "with..." statement. I understand why the error is thrown because the dictionary has not been updated yet because the threads in the pool did not finish executing. I know the threads did not finish executing because I have a print("done") in the method that is called within the

2022-05-01 20:56:10    分类:问答    python   threadpool   wait   threadpoolexecutor   concurrent.futures

Understanding concurrent.futures.Executor.map()

I'm trying to parallelize some Python code using processes and concurrent.futures. It looks like I can execute a function multiple times in parrallel either by submitting calls and then calling Future.result() on the futures, or by using Executor.map(). I'm wondering if the latter is just a syntactic sugar for the former and if there's any difference performance-wise. It doesn't seem immediately clear from the documentation.

2022-04-30 16:37:03    分类:问答    python   python-3.x   concurrent.futures

Script doesn't work when I go for multiple search keywords in the list

I've created a script to fetch different newspaper names derived from a search engine when I initiate search using different keywords, as in CMG제약,DB하이텍 e.t.c. in that pages top right search box. I also used some customized dates within params to get results from those dates. The script is doing fine as long as I use a single keyword in the search list. However, when I use multiple keyword in the search list the script only keeps up with the last keyword. This is the list of keywords I would like to use: keywords = ['CMG제약','DB하이텍','ES큐브','EV첨단소재'] The script is short in size but because of

2022-04-30 01:19:14    分类:问答    python   python-3.x   web-scraping   concurrent.futures

How to break time.sleep() in a python concurrent.futures

I am playing around with concurrent.futures. Currently my future calls time.sleep(secs). It seems that Future.cancel() does less than I thought. If the future is already executing, then time.sleep() does not get cancel by it. The same for the timeout parameter for wait(). It does not cancel my time.sleep(). How to cancel time.sleep() which gets executed in a concurrent.futures? For testing I use the ThreadPoolExecutor.

2022-04-29 04:18:41    分类:问答    python   concurrency   concurrent.futures

Java Process did not quit after Future task completion

This is my code snippet using Future. import java.util.concurrent.*; import java.util.*; public class FutureDemo{ public FutureDemo(){ /* Future */ ExecutorService service = Executors.newFixedThreadPool(2); for ( int i=0; i<10; i++){ MyCallable myCallable = new MyCallable((long)i); Future<Long> futureResult = service.submit(myCallable); Long result = null; try{ result = futureResult.get(5000, TimeUnit.MILLISECONDS); }catch(TimeoutException e){ System.out.println("Time out after 5 seconds"); futureResult.cancel(true); }catch(InterruptedException ie){ System.out.println("Error: Interrupted");

2022-04-27 22:08:14    分类:问答    java   executorservice   concurrent.futures

concurrent.futures not parallelizing write

问题 我有一个列表dataframe_chunk ,其中包含一个非常大的 pandas 数据帧的块。我想将每个块写入不同的 csv,并并行执行。 但是,我看到文件是按顺序写入的,我不确定为什么会这样。 这是代码: import concurrent.futures as cfu def write_chunk_to_file(chunk, fpath): chunk.to_csv(fpath, sep=',', header=False, index=False) pool = cfu.ThreadPoolExecutor(N_CORES) futures = [] for i in range(N_CORES): fpath = '/path_to_files_'+str(i)+'.csv' futures.append(pool.submit( write_chunk_to_file(dataframe_chunk[i], fpath))) for f in cfu.as_completed(futures): print("finished at ",time.time()) 有什么线索吗? 回答1 在 Python 2.7.x 线程文档中但未在 3.x 文档中声明的一件事是 Python 无法使用threading库实现真正的并行性——一次只能执行一个线程。

2022-04-26 11:45:03    分类:技术分享    python   multithreading   python-3.x   concurrent.futures