� ��fu[c@sfdgZddlZddlZddlZddlZddlZddlmZmZm Z ddl mZmZdZ dZdZej�Zdd�Zd d �ZGdd�de�Zefed d�ZGdd�de�ZGdd�de�ZeZGdd�de�ZGdd�de�ZGdd�de�ZGdd�de�ZdS(uPooliN(uProcessu cpu_countuTimeoutError(uFinalizeudebugiicCstt|��S(N(ulistumap(uargs((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyumapstar'sumapstarcCsttj|d|d��S(Nii(ulistu itertoolsustarmap(uargs((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyustarmapstar*sustarmapstarcsD|EeZdZdZ�fdd�Zdd�Zdd�Z�S(uMaybeEncodingErroruVWraps possible unpickleable errors, so they can be safely sent through the socket.csAt|�|_t|�|_tt|�j|j|j�dS(N(urepruexcuvalueusuperuMaybeEncodingErroru__init__(uselfuexcuvalue(u __class__(u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu__init__5suMaybeEncodingError.__init__cCsd|j|jfS(Nu(Error sending result: '%s'. Reason: '%s'(uvalueuexc(uself((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu__str__:s uMaybeEncodingError.__str__cCsdt|�S(Nu(ustr(uself((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu__repr__>suMaybeEncodingError.__repr__(u__name__u __module__u__qualname__u__doc__u__init__u__str__u__repr__(u __locals__((u __class__u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyuMaybeEncodingError1suMaybeEncodingErrorc)Cs�|j}|j}t|d�r>|jj�|jj�n|dk rW||�nd}xT|dks~|r�||kr�y |�}Wn#ttfk r�t d�PYnX|dkr�t d�Pn|\} } }}} yd||| �f}Wn1tk r,}zd |f}WYdd}~XnXy|| | |f�Wn_tk r�}z?t ||d�}t d|�|| | d |ff�WYdd}~XnX|d7}q`Wt d|�dS( Nu_writeriu)worker got EOFError or IOError -- exitinguworker got sentinel -- exitingiu0Possible encoding error while sending result: %suworker exiting after %d tasksTF(uputugetuhasattru_writerucloseu_readeruNoneuEOFErroruIOErrorudebuguTrueu ExceptionuFalseuMaybeEncodingError(uinqueueuoutqueueuinitializeruinitargsumaxtasksuputugetu completedutaskujobuiufuncuargsukwdsuresultueuwrapped((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyuworkerBs> ! ,uworkercBs�|EeZdZdZeZd7d7fd7dd�Zdd�Zdd�Zdd �Z d d�Z fidd �Zd7dd�Zd7dd�Z d7d7d7dd�Zddd�Zddd�Zfid7d7dd�Zd7d7d7dd�Zd7d7d7dd�Zedd ��Zed!d"��Zed#d$��Zed%d&��Zd'd(�Zd)d*�Zd+d,�Zd-d.�Zed/d0��Zed1d2��Zd3d4�Z d5d6�Z!d7S(8uPooluS Class which supports an async version of applying functions to arguments. cCs-|j�tj�|_i|_t|_||_||_||_ |dkr}y t�}Wq}tk ryd}Yq}Xn|dkr�t d��n|dk r�t|�r�td��n||_g|_|j�tjdtjd|f�|_d|j_t|j_|jj�tjdtjd|j|j|j|j|jf�|_d|j_t|j_|jj�tjdtjd|j|j |jf�|_!d|j!_t|j!_|j!j�t"||j#d|j|j$|j|j|j|j|j!|jfdd�|_%dS( Niu&Number of processes must be at least 1uinitializer must be a callableutargetuargsuexitpriorityiT(&u _setup_queuesuqueueuQueueu _taskqueueu_cacheuRUNu_stateu_maxtasksperchildu_initializeru _initargsuNoneu cpu_countuNotImplementedErroru ValueErrorucallableu TypeErroru _processesu_poolu_repopulate_poolu threadinguThreaduPoolu_handle_workersu_worker_handleruTrueudaemonustartu _handle_tasksu _quick_putu _outqueueu _task_handleru_handle_resultsu _quick_getu_result_handleruFinalizeu_terminate_poolu_inqueueu _terminate(uselfu processesuinitializeruinitargsumaxtasksperchild((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu__init__rsX u Pool.__init__cCswd}xjttt|j��D]M}|j|}|jdk r"td|�|j�d}|j|=q"q"W|S(u�Cleanup after any worker processes which have exited due to reaching their specified lifetime. Returns True if any workers were cleaned up. ucleaning up worker %dFNT( uFalseureversedurangeulenu_pooluexitcodeuNoneudebugujoinuTrue(uselfucleaneduiuworker((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu_join_exited_workers�s" uPool._join_exited_workersc Cs�x�t|jt|j��D]�}|jdtd|j|j|j|j |j f�}|jj|�|jj dd�|_d|_|j�td�qWdS(u�Bring the number of pool processes up to the specified number, for use after reaping workers which have exited. utargetuargsuProcessu PoolWorkeruadded workerNT(urangeu _processesulenu_pooluProcessuworkeru_inqueueu _outqueueu_initializeru _initargsu_maxtasksperchilduappendunameureplaceuTrueudaemonustartudebug(uselfuiuw((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu_repopulate_pool�s# uPool._repopulate_poolcCs|j�r|j�ndS(uEClean up any exited workers and start replacements for them. N(u_join_exited_workersu_repopulate_pool(uself((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu_maintain_pool�suPool._maintain_poolcCsPddlm}|�|_|�|_|jjj|_|jjj|_ dS(Ni(uSimpleQueue( uqueuesuSimpleQueueu_inqueueu _outqueueu_writerusendu _quick_putu_readerurecvu _quick_get(uselfuSimpleQueue((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu _setup_queues�s uPool._setup_queuescCs|j|||�j�S(u6 Equivalent of `func(*args, **kwds)`. (uapply_asyncuget(uselfufuncuargsukwds((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyuapply�su Pool.applycCs|j||t|�j�S(ux Apply `func` to each element in `iterable`, collecting the results in a list that is returned. (u _map_asyncumapstaruget(uselfufuncuiterableu chunksize((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyumap�suPool.mapcCs|j||t|�j�S(u� Like `map()` method but the elements of the `iterable` are expected to be iterables as well and will be unpacked as arguments. Hence `func` and (a, b) becomes func(a, b). (u _map_asyncustarmapstaruget(uselfufuncuiterableu chunksize((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyustarmap�suPool.starmapcCs|j||t|||�S(u= Asynchronous version of `starmap()` method. (u _map_asyncustarmapstar(uselfufuncuiterableu chunksizeucallbackuerror_callback((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu starmap_async�suPool.starmap_asyncics�|jtkrtd��n|dkrrt|j��|jj��fdd�t|�D��jf��St j �||�}t|j��|jj�fdd�t|�D��jf�dd��D�SdS(uP Equivalent of `map()` -- can be MUCH slower than `Pool.map()`. uPool not runningic3s0|]&\}}�j|�|fifVqdS(N(u_job(u.0uiux(ufuncuresult(u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu �suPool.imap..c3s0|]&\}}�j|t|fifVqdS(N(u_jobumapstar(u.0uiux(uresult(u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu scss"|]}|D]}|Vq qdS(N((u.0uchunkuitem((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu sN(u_stateuRUNu ValueErroruIMapIteratoru_cacheu _taskqueueuputu enumerateu_set_lengthuPoolu _get_tasks(uselfufuncuiterableu chunksizeutask_batches((ufuncuresultu9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyuimap�su Pool.imapcs�|jtkrtd��n|dkrrt|j��|jj��fdd�t|�D��jf��St j �||�}t|j��|jj�fdd�t|�D��jf�dd��D�SdS(uL Like `imap()` method but ordering of results is arbitrary. uPool not runningic3s0|]&\}}�j|�|fifVqdS(N(u_job(u.0uiux(ufuncuresult(u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu su&Pool.imap_unordered..c3s0|]&\}}�j|t|fifVqdS(N(u_jobumapstar(u.0uiux(uresult(u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu scss"|]}|D]}|Vq qdS(N((u.0uchunkuitem((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu sN(u_stateuRUNu ValueErroruIMapUnorderedIteratoru_cacheu _taskqueueuputu enumerateu_set_lengthuPoolu _get_tasks(uselfufuncuiterableu chunksizeutask_batches((ufuncuresultu9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyuimap_unordered suPool.imap_unorderedcCsb|jtkrtd��nt|j||�}|jj|jd|||fgdf�|S(u; Asynchronous version of `apply()` method. uPool not runningN( u_stateuRUNu ValueErroruApplyResultu_cacheu _taskqueueuputu_jobuNone(uselfufuncuargsukwdsucallbackuerror_callbackuresult((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyuapply_asyncs +uPool.apply_asynccCs|j||t|||�S(u9 Asynchronous version of `map()` method. (u _map_asyncumapstar(uselfufuncuiterableu chunksizeucallbackuerror_callback((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu map_async(suPool.map_asyncc s|jtkrtd��nt|d�s<t|�}n|d kr�tt|�t|j�d�\}}|r�|d7}q�nt|�dkr�d}nt j |||�}t|j|t|�|d|��|j j��fdd�t|�D�d f��S( uY Helper function to implement map, starmap and their async counterparts. uPool not runningu__len__iiiuerror_callbackc3s0|]&\}}�j|�|fifVqdS(N(u_job(u.0uiux(umapperuresult(u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu Dsu"Pool._map_async..N(u_stateuRUNu ValueErroruhasattrulistuNoneudivmodulenu_pooluPoolu _get_tasksu MapResultu_cacheu _taskqueueuputu enumerate( uselfufuncuiterableumapperu chunksizeucallbackuerror_callbackuextrautask_batches((umapperuresultu9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu _map_async0s ( uPool._map_asynccCsotj�}xB|jtks6|jrP|jtkrP|j�tjd�qW|j j d�td�dS(Ng��?uworker handler exiting( u threadingucurrent_threadu_stateuRUNu_cacheu TERMINATEu_maintain_poolutimeusleepu _taskqueueuputuNoneudebug(upooluthread((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu_handle_workersHs* uPool._handle_workerscCs�tj�}xt|jd�D]�\}}d }x�t|�D]�\}} |jratd�Pny|| �Wq>tk r�} zN| dd�\}}y||j |d| f�Wntk r�YnXWYdd} ~ Xq>Xq>W|rtd�||d�qqPqWtd�y@td�|jd�td�x|D]} |d�qBWWnt k rutd�YnXtd �dS(Niu'task handler found thread._state != RUNiudoing set_length()utask handler got sentinelu/task handler sending sentinel to result handleru(task handler sending sentinel to workersu/task handler got IOError when sending sentinelsutask handler exitingi��F(u threadingucurrent_threaduiterugetuNoneu enumerateu_stateudebugu Exceptionu_setuFalseuKeyErroruputuIOError(u taskqueueuputuoutqueueupoolucacheuthreadutasksequ set_lengthuiutaskueujobuindup((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu _handle_tasksUs< uPool._handle_taskscCs�tj�}x�y |�}Wn&ttfk rDtd�dSYnX|jr\td�Pn|dkrvtd�Pn|\}}}y||j||�Wqtk r�YqXqx�|r^|jt kr^y |�}Wn&ttfk rtd�dSYnX|dkrtd�q�n|\}}}y||j||�Wq�tk rZYq�Xq�Wt |d�r�td�y5x.td�D] }|jj �s�Pn|�q�WWq�ttfk r�Yq�Xntdt|�|j�dS( Nu.result handler got EOFError/IOError -- exitingu,result handler found thread._state=TERMINATEuresult handler got sentinelu&result handler ignoring extra sentinelu_readeru"ensuring that outqueue is not fulli u7result handler exiting: len(cache)=%s, thread._state=%s(u threadingucurrent_threaduIOErroruEOFErrorudebugu_stateuNoneu_setuKeyErroru TERMINATEuhasattrurangeu_readerupollulen(uoutqueueugetucacheuthreadutaskujobuiuobj((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu_handle_resultssV uPool._handle_resultsccsCt|�}x0ttj||��}|s1dS||fVqdS(N(uiterutupleu itertoolsuislice(ufuncuitusizeux((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu _get_tasks�suPool._get_taskscCstd��dS(Nu:pool objects cannot be passed between processes or pickled(uNotImplementedError(uself((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu __reduce__�suPool.__reduce__cCs5td�|jtkr1t|_t|j_ndS(Nuclosing pool(udebugu_stateuRUNuCLOSEu_worker_handler(uself((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyuclose�s u Pool.closecCs-td�t|_t|j_|j�dS(Nuterminating pool(udebugu TERMINATEu_stateu_worker_handleru _terminate(uself((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu terminate�s uPool.terminatecCsStd�|jj�|jj�|jj�x|jD]}|j�q;WdS(Nujoining pool(udebugu_worker_handlerujoinu _task_handleru_result_handleru_pool(uselfup((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyujoin�s u Pool.joincCsWtd�|jj�x9|j�rR|jj�rR|jj�tjd�qWdS(Nu7removing tasks from inqueue until task handler finishedi( udebugu_rlockuacquireuis_aliveu_readerupollurecvutimeusleep(uinqueueutask_handlerusize((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu_help_stuff_finish�s uPool._help_stuff_finishc Cs�td�t|_t|_td�|j||t|��t|_|jd�td�tj�|k r~|j �n|r�t |dd�r�td�x-|D]"} | jdkr�| j�q�q�Wntd�tj�|k r�|j �ntd�tj�|k r#|j �n|r�t |dd�r�td �x;|D]0} | j �rMtd | j�| j �qMqMWndS(Nufinalizing poolu&helping task handler/workers to finishujoining worker handleriu terminateuterminating workersujoining task handlerujoining result handlerujoining pool workersucleaning up worker %d(udebugu TERMINATEu_stateu_help_stuff_finishulenuputuNoneu threadingucurrent_threadujoinuhasattruexitcodeu terminateuis_aliveupid( uclsu taskqueueuinqueueuoutqueueupooluworker_handlerutask_handleruresult_handlerucacheup((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu_terminate_pool�s6 uPool._terminate_poolcCs|S(N((uself((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu __enter__suPool.__enter__cCs|j�dS(N(u terminate(uselfuexc_typeuexc_valuexc_tb((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu__exit__su Pool.__exit__N("u__name__u __module__u__qualname__u__doc__uProcessuNoneu__init__u_join_exited_workersu_repopulate_poolu_maintain_poolu _setup_queuesuapplyumapustarmapu starmap_asyncuimapuimap_unordereduapply_asyncu map_asyncu _map_asyncustaticmethodu_handle_workersu _handle_tasksu_handle_resultsu _get_tasksu __reduce__ucloseu terminateujoinu_help_stuff_finishuclassmethodu_terminate_poolu __enter__u__exit__(u __locals__((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyuPoollsB : *: .cBsb|EeZdZdd�Zdd�Zdd�Zd dd�Zd d d �Zdd�Z d S(uApplyResultcCsJtj�|_tt�|_||_||_||_|||j s. *��&%I