天道酬勤,学无止境

pool

Python HTTPConnectionPool Failed to establish a new connection: [Errno 11004] getaddrinfo failed

I was wondering if my requests is stopped by the website and I need to set a proxy.I first try to close the http's connection ,bu I failed.I also try to test my code but now it seems no outputs.Mybe I use a proxy everything will be OK? Here is the code. import requests from urllib.parse import urlencode import json from bs4 import BeautifulSoup import re from html.parser import HTMLParser from multiprocessing import Pool from requests.exceptions import RequestException import time def get_page_index(offset, keyword): #headers = {'User-Agent':'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_8

2021-06-03 14:53:34    分类:问答    python   multithreading   python-requests   pool

Why A single Jedis instance is not threadsafe?

https://github.com/xetorthio/jedis/wiki/Getting-started using Jedis in a multithreaded environment You shouldn't use the same instance from different threads because you'll have strange errors. And sometimes creating lots of Jedis instances is not good enough because it means lots of sockets and connections, which leads to strange errors as well. A single Jedis instance is not threadsafe ! To avoid these problems, you should use JedisPool, which is a threadsafe pool of network connections. You can use the pool to reliably create several Jedis instances, given you return the Jedis instance to

2021-06-03 03:16:38    分类:问答    redis   thread-safety   pool   jedis

Optimizing multiprocessing.Pool with expensive initialization

Here is a complete simple working example import multiprocessing as mp import time import random class Foo: def __init__(self): # some expensive set up function in the real code self.x = 2 print('initializing') def run(self, y): time.sleep(random.random() / 10.) return self.x + y def f(y): foo = Foo() return foo.run(y) def main(): pool = mp.Pool(4) for result in pool.map(f, range(10)): print(result) pool.close() pool.join() if __name__ == '__main__': main() How can I modify it so Foo is only initialized once by each worker, not every task? Basically I want the init called 4 times, not 10. I am

2021-06-02 12:40:07    分类:问答    python   python-3.5   python-multiprocessing   pool   object-initializers

Python NotImplementedError:无法在进程之间传递池对象(Python NotImplementedError: pool objects cannot be passed between processes)

问题 当页面附加到页面列表时,我正在尝试交付工作,但是我的代码输出返回NotImplementedError。 这是我正在尝试执行的代码: 代码: from multiprocessing import Pool, current_process import time import random import copy_reg import types import threading class PageControler(object): def __init__(self): self.nProcess = 3 self.pages = [1,2,3,4,5,6,7,8,9,10] self.manageWork() def manageWork(self): self.pool = Pool(processes=self.nProcess) time.sleep(2) work_queue = threading.Thread(target=self.modifyQueue) work_queue.start() #pool.close() #pool.join() def deliverWork(self): if self.pages != []: pag = self.pages.pop() self.pool.apply_async(self.myFun) def

2021-05-31 15:56:16    分类:技术分享    python   multiprocessing   pool

如何防止在由boost :: fast_pool_allocator管理的对象上调用析构函数?(How to prevent destructors from being called on objects managed by boost::fast_pool_allocator?)

问题 我想利用以下宣传的boost::fast_pool_allocator (请参阅Boost Pool的Boost文档): 例如,您可能想在一个位置分配一堆小对象,然后到达程序中不再需要它们的某个位置。 使用池接口,您可以选择运行它们的析构函数,或者只是将它们放到遗忘中... (有关此报价,请参见此处。) 关键是把它们丢掉。 我不希望在这些对象上调用析构函数。 (原因是我有数百万个微小的对象在堆上形成了一个极其复杂的所有权网络,并且当单个最父对象离开堆栈时,我的程序大约需要20分钟才能调用所有析构函数。我不需要调用这些析构函数,因为没有期望的副作用,并且所有内存都包含在boost::pool 。) 不幸的是,尽管有上述文档的承诺以及boost::pool概念的承诺,但我找不到找到防止调用托管对象的析构函数的方法。 在一个小的示例程序中可以很容易地找到问题所在: class Obj { public: ~Obj() { // Placing a breakpoint here indicates that this is *always* reached // (except for the crash scenario discussed below) int m = 0; } }; typedef std::map<int, Obj, std::less<int>

2021-05-25 03:17:24    分类:技术分享    c++   c++11   boost   pool

Node.js和MongoDB,重用数据库对象(Node.js and MongoDB, reusing the DB object)

问题 我是Node.js和MongoDB的新手,但是我设法将SO和mongo文档中的某些部分放在一起。 Mongo文档提供了以下示例: // Retrieve var MongoClient = require('mongodb').MongoClient; // Connect to the db MongoClient.connect("mongodb://localhost:27017/exampleDb", function(err, db) { if(!err) { console.log("We are connected"); } }); 如果我只需要在一个位置的一个函数中使用数据库,那看起来就很好。 在SO上进行搜索和阅读表明,我不应该每次都打开一个新的连接,而应该使用一个池并重用我第一次获得的数据库对象。 这个答案在SO上很丰富,但是我不确定如何首先获得DB对象,然后再如何重用它。 假设我的App.js中有上面的Node.js代码,然后我有不同的路由,需要对db运行不同的操作,例如: app.post('/employee', function(req, res){ //Put req.name in database }); app.post('/car', function(req, res){ //Put req.car in database })

2021-05-19 10:31:12    分类:技术分享    node.js   mongodb   object   reusability   pool

python多处理池终止(python multiprocessing pool terminate)

问题 我正在开发renderfarm,我需要我的客户端能够启动渲染器的多个实例,而不会阻塞,以便客户端可以接收新命令。 我的工作正常,但是在终止创建的进程时遇到了麻烦。 在全局级别,我定义了我的池(以便可以从任何函数访问它): p = Pool(2) 然后,我使用apply_async调用渲染器: for i in range(totalInstances): p.apply_async(render, (allRenderArgs[i],args[2]), callback=renderFinished) p.close() 该功能完成,在后台启动进程,并等待新命令。 我做了一个简单的命令,它将杀死客户端并停止渲染: def close(): ''' close this client instance ''' tn.write ("say "+USER+" is leaving the farm\r\n") try: p.terminate() except Exception,e: print str(e) sys.exit() 它似乎没有给出错误(它将显示错误),python终止了,但是后台进程仍在运行。 谁能推荐一种更好的方法来控制这些已启动的程序? 回答1 我找到了解决方案:在单独的线程中停止池,如下所示: def close_pool(): global pool

2021-05-18 13:26:17    分类:技术分享    python   multiprocessing   pool

java.lang.IllegalMonitorStateException:(m = null)无法获取监视器(java.lang.IllegalMonitorStateException: (m=null) Failed to get monitor for)

问题 为什么会发生这种情况? 事实是,监视器对象肯定不是null,但仍然经常会遇到此异常: java.lang.IllegalMonitorStateException: (m=null) Failed to get monitor for (tIdx=60) at java.lang.Object.wait(Object.java:474) at ... 引起这种情况的代码是一个简单的池解决方案: public Object takeObject() { Object obj = internalTakeObject(); while (obj == null) { try { available.wait(); } catch (InterruptedException e) { throw new RuntimeException(e); } obj = internalTakeObject(); } return obj; } private Object internalTakeObject() { Object obj = null; synchronized (available) { if (available.size() > 0) { obj = available.keySet().iterator().next(); available.remove(obj

2021-05-18 07:34:29    分类:技术分享    java   multithreading   locking   pool

带有嵌套Web请求的Gevent池(Gevent pool with nested web requests)

问题 我尝试组织最多10个并发下载的池。 该功能应下载基本URL,然后解析该页面上的所有URL并下载每个URL,但是并发下载的总数量不应超过10。 from lxml import etree import gevent from gevent import monkey, pool import requests monkey.patch_all() urls = [ 'http://www.google.com', 'http://www.yandex.ru', 'http://www.python.org', 'http://stackoverflow.com', # ... another 100 urls ] LINKS_ON_PAGE=[] POOL = pool.Pool(10) def parse_urls(page): html = etree.HTML(page) if html: links = [link for link in html.xpath("//a/@href") if 'http' in link] # Download each url that appears in the main URL for link in links: data = requests.get(link) LINKS_ON_PAGE.append('%s: %s

2021-05-18 06:32:51    分类:技术分享    python   web   pool   gevent

Python, multithreading too slow, multiprocess

I'm a multiprocessing newbie, I know something about threading but I need to increase the speed of this calculation, hopefully with multiprocessing: Example Description: sends string to a thread, alters string + benchmark test, send result back for printing. from threading import Thread class Alter(Thread): def __init__(self, word): Thread.__init__(self) self.word = word self.word2 = '' def run(self): # Alter string + test processing speed for i in range(80000): self.word2 = self.word2 + self.word # Send a string to be altered thread1 = Alter('foo') thread2 = Alter('bar') thread1.start()

2021-05-18 03:39:06    分类:问答    python   multithreading   pool   multiprocess