� e f�a�@s�ddgZddlZddlZddlZddlZddlZddlZddlZddlm Z ddlm Z mZdZdZ dZej�Zdd �Zd d�ZGdd �d e�ZGdd�d�Zdd�ZGdd�de�Zdfdddd�ZGdd�de�ZGdd�de�ZeZGdd�de�ZGdd�de�ZGdd�de�ZGd d�de�Z dS)!�Pool� ThreadPool�N�)�util)�get_context�TimeoutError�cCstt|��S)N)�list�map)�args�r�9/opt/alt/python34/lib64/python3.4/multiprocessing/pool.py�mapstar+srcCsttj|d|d��S)Nrr)r � itertools�starmap)rrrr �starmapstar.src@s(eZdZdd�Zdd�ZdS)�RemoteTracebackcCs ||_dS)N)�tb)�selfrrrr �__init__6szRemoteTraceback.__init__cCs|jS)N)r)rrrr �__str__8szRemoteTraceback.__str__N)�__name__� __module__�__qualname__rrrrrr r5src@s(eZdZdd�Zdd�ZdS)�ExceptionWithTracebackcCsDtjt|�||�}dj|�}||_d||_dS)N�z """ %s""")� traceback�format_exception�type�join�excr)rr rrrr r<s zExceptionWithTraceback.__init__cCst|j|jffS)N)�rebuild_excr r)rrrr � __reduce__Asz!ExceptionWithTraceback.__reduce__N)rrrrr"rrrr r;srcCst|�|_|S)N)r� __cause__)r rrrr r!Dsr!cs@eZdZdZ�fdd�Zdd�Zdd�Z�S)�MaybeEncodingErrorzVWraps possible unpickleable errors, so they can be safely sent through the socket.csAt|�|_t|�|_tt|�j|j|j�dS)N)�reprr �value�superr$r)rr r&)� __class__rr rPszMaybeEncodingError.__init__cCsd|j|jfS)Nz(Error sending result: '%s'. Reason: '%s')r&r )rrrr rUs zMaybeEncodingError.__str__cCsdt|�S)Nz)�str)rrrr �__repr__YszMaybeEncodingError.__repr__)rrr�__doc__rrr*rr)r(r r$Lsr$Fc'Cs�|j}|j}t|d�r>|jj�|jj�n|dk rW||�nd}xx|dks~|r�||kr�y |�} Wn&ttfk r�tj d�PYnX| dkr�tj d�Pn| \} }}} }yd|| |�f}WnLt k rM}z,|r/t||j�}nd|f}WYdd}~XnXy|| ||f�Wnbt k r�}zBt ||d�}tj d|�|| |d|ff�WYdd}~XnX|d7}q`Wtj d |�dS) N�_writerrz)worker got EOFError or OSError -- exitingzworker got sentinel -- exitingTFrz0Possible encoding error while sending result: %szworker exiting after %d tasks)�put�get�hasattrr,�close�_reader�EOFError�OSErrorr�debug� Exceptionr� __traceback__r$)�inqueue�outqueue�initializer�initargsZmaxtasksZwrap_exceptionr-r.Z completed�task�job�i�funcr�kwds�result�e�wrappedrrr �worker]sB ! ,rCc@s�eZdZdZdZdd�Zddfdddd�Zdd �Zd d�Zdd �Z dd�Z fidd�Zddd�Zddd�Z ddddd�Zddd�Zddd�Zfidddd�Zddddd �Zdddd!d"�Zed#d$��Zed%d&��Zed'd(��Zed)d*��Zd+d,�Zd-d.�Zd/d0�Zd1d2�Zed3d4��Zed5d6��Zd7d8�Z d9d:�Z!dS);rzS Class which supports an async version of applying functions to arguments. TcOs|jj||�S)N)�_ctx�Process)rrr?rrr rE�szPool.ProcessNcCs,|pt�|_|j�tj�|_i|_t|_||_ ||_ ||_|dkrytj �psd}n|dkr�td��n|dk r�t|�r�td��n||_g|_|j�tjdtjd|f�|_d|j_t|j_|jj�tjdtjd|j|j|j|j|jf�|_d|j_t|j_|jj�tjdtjd|j|j |jf�|_!d|j!_t|j!_|j!j�t"j#||j$d|j|j%|j|j|j|j|j!|jfdd�|_&dS) Nrz&Number of processes must be at least 1zinitializer must be a callable�targetrTZexitpriority�)'rrD� _setup_queues�queue�Queue� _taskqueue�_cache�RUN�_state�_maxtasksperchild�_initializer� _initargs�os� cpu_count� ValueError�callable� TypeError� _processes�_pool�_repopulate_pool� threadingZThreadr�_handle_workers�_worker_handler�daemon�start� _handle_tasks� _quick_put� _outqueue� _task_handler�_handle_results� _quick_get�_result_handlerrZFinalize�_terminate_pool�_inqueue� _terminate)r� processesr9r:Zmaxtasksperchild�contextrrr r�sT z Pool.__init__cCszd}xmttt|j��D]P}|j|}|jdk r"tjd|�|j�d}|j|=q"q"W|S)z�Cleanup after any worker processes which have exited due to reaching their specified lifetime. Returns True if any workers were cleaned up. FNzcleaning up worker %dT)�reversed�range�lenrX�exitcoderr4r)rZcleanedr=rCrrr �_join_exited_workers�s" zPool._join_exited_workerscCs�x�t|jt|j��D]�}|jdtd|j|j|j|j |j |jf�}|jj|�|j jdd�|_ d|_|j�tjd�qWdS)z�Bring the number of pool processes up to the specified number, for use after reaping workers which have exited. rFrrEZ PoolWorkerTzadded workerN)rlrWrmrXrErCrgrarPrQrO�_wrap_exception�append�name�replacer]r^rr4)rr=�wrrr rY�s# zPool._repopulate_poolcCs|j�r|j�ndS)zEClean up any exited workers and start replacements for them. N)rorY)rrrr �_maintain_pool�szPool._maintain_poolcCsL|jj�|_|jj�|_|jjj|_|jjj|_ dS)N) rDZSimpleQueuergrar,�sendr`r1�recvrd)rrrr rH�szPool._setup_queuescCs|j|||�j�S)z6 Equivalent of `func(*args, **kwds)`. )�apply_asyncr.)rr>rr?rrr �apply�sz Pool.applycCs|j||t|�j�S)zx Apply `func` to each element in `iterable`, collecting the results in a list that is returned. )� _map_asyncrr.)rr>�iterable� chunksizerrr r �szPool.mapcCs|j||t|�j�S)z� Like `map()` method but the elements of the `iterable` are expected to be iterables as well and will be unpacked as arguments. Hence `func` and (a, b) becomes func(a, b). )rzrr.)rr>r{r|rrr rszPool.starmapcCs|j||t|||�S)z= Asynchronous version of `starmap()` method. )rzr)rr>r{r|�callback�error_callbackrrr � starmap_asyncszPool.starmap_asyncrcs�|jtkrtd��n|dkrrt|j��|jj��fdd�t|�D��jf��St j �||�}t|j��|jj�fdd�t|�D��jf�dd��D�SdS)zP Equivalent of `map()` -- can be MUCH slower than `Pool.map()`. zPool not runningrc3s0|]&\}}�j|�|fifVqdS)N)�_job)�.0r=�x)r>r@rr � szPool.imap..c3s0|]&\}}�j|t|fifVqdS)N)r�r)r�r=r�)r@rr r�%scss"|]}|D]}|Vq qdS)Nr)r��chunk�itemrrr r�'sN)rNrMrT�IMapIteratorrLrKr-� enumerate�_set_lengthr� _get_tasks)rr>r{r|�task_batchesr)r>r@r �imapsz Pool.imapcs�|jtkrtd��n|dkrrt|j��|jj��fdd�t|�D��jf��St j �||�}t|j��|jj�fdd�t|�D��jf�dd��D�SdS)zL Like `imap()` method but ordering of results is arbitrary. zPool not runningrc3s0|]&\}}�j|�|fifVqdS)N)r�)r�r=r�)r>r@rr r�1sz&Pool.imap_unordered..c3s0|]&\}}�j|t|fifVqdS)N)r�r)r�r=r�)r@rr r�8scss"|]}|D]}|Vq qdS)Nr)r�r�r�rrr r�:sN)rNrMrT�IMapUnorderedIteratorrLrKr-r�r�rr�)rr>r{r|r�r)r>r@r �imap_unordered)szPool.imap_unorderedcCsb|jtkrtd��nt|j||�}|jj|jd|||fgdf�|S)z; Asynchronous version of `apply()` method. zPool not runningN)rNrMrT�ApplyResultrLrKr-r�)rr>rr?r}r~r@rrr rx<s +zPool.apply_asynccCs|j||t|||�S)z9 Asynchronous version of `map()` method. )rzr)rr>r{r|r}r~rrr � map_asyncGszPool.map_asyncc s|jtkrtd��nt|d�s<t|�}n|dkr�tt|�t|j�d�\}}|r�|d7}q�nt|�dkr�d}ntj |||�}t |j|t|�|d|��|jj ��fdd �t|�D�df��S) zY Helper function to implement map, starmap and their async counterparts. zPool not running�__len__N�rrr~c3s0|]&\}}�j|�|fifVqdS)N)r�)r�r=r�)�mapperr@rr r�csz"Pool._map_async..)rNrMrTr/r �divmodrmrXrr�� MapResultrLrKr-r�) rr>r{r�r|r}r~Zextrar�r)r�r@r rzOs ( zPool._map_asynccCsrtj�}xB|jtks6|jrP|jtkrP|j�tjd�qW|j j d�tjd�dS)Ng��?zworker handler exiting) rZ�current_threadrNrMrL� TERMINATEru�time�sleeprKr-rr4)�pool�threadrrr r[gs* zPool._handle_workerscCs>tj�}x�t|jd�D]�\}}d}d} y�x�t|�D]�\} }|jrmtjd�Pny||�WqGtk r�} zN|dd�\}}y||j |d| f�Wnt k r�YnXWYdd} ~ XqGXqGW|rtjd�|| d�nwPWqtk r�} zx|rD|dd�nd \}}||kr}||j |dd| f�n|r�tjd�|| d�nWYdd} ~ XqXqWtjd�yFtjd�|jd�tjd �x|D]}|d�q�WWntk r,tjd �YnXtjd�dS)Nrz'task handler found thread._state != RUNrFzdoing set_length()rztask handler got sentinelz/task handler sending sentinel to result handlerz(task handler sending sentinel to workersz/task handler got OSError when sending sentinelsztask handler exiting��)rr) rZr��iterr.r�rNrr4r5�_set�KeyErrorr-r3)� taskqueuer-r8r��cacher�ZtaskseqZ set_lengthr;r=rAr<ZindZex�prrr r_tsN "! ( zPool._handle_taskscCs�tj�}x�y |�}Wn)ttfk rGtjd�dSYnX|jrbtjd�Pn|dkrtjd�Pn|\}}}y||j||�Wqtk r�YqXqWx�|rn|jt krny |�}Wn)ttfk rtjd�dSYnX|dkr/tjd�q�n|\}}}y||j||�Wq�tk rjYq�Xq�Wt |d�r�tjd�y5x.td�D] }|jj �s�Pn|�q�WWq�ttfk r�Yq�Xntjdt|�|j�dS) Nz.result handler got EOFError/OSError -- exitingz,result handler found thread._state=TERMINATEzresult handler got sentinelz&result handler ignoring extra sentinelr1z"ensuring that outqueue is not full� z7result handler exiting: len(cache)=%s, thread._state=%s)rZr�r3r2rr4rNr�r�r�r/rlr1�pollrm)r8r.r�r�r;r<r=�objrrr rc�sV zPool._handle_resultsccsDt|�}x1ttj||��}|s1dS||fVqWdS)N)r��tupler�islice)r>�it�sizer�rrr r��szPool._get_taskscCstd��dS)Nz:pool objects cannot be passed between processes or pickled)�NotImplementedError)rrrr r"�szPool.__reduce__cCs8tjd�|jtkr4t|_t|j_ndS)Nzclosing pool)rr4rNrM�CLOSEr\)rrrr r0�s z Pool.closecCs0tjd�t|_t|j_|j�dS)Nzterminating pool)rr4r�rNr\rh)rrrr � terminate�s zPool.terminatecCsVtjd�|jj�|jj�|jj�x|jD]}|j�q>WdS)Nzjoining pool)rr4r\rrbrerX)rr�rrr r�s z Pool.joincCsZtjd�|jj�x9|j�rU|jj�rU|jj�tj d�qWdS)Nz7removing tasks from inqueue until task handler finishedr) rr4Z_rlock�acquire�is_aliver1r�rwr�r�)r7�task_handlerr�rrr �_help_stuff_finishs zPool._help_stuff_finishc Cs�tjd�t|_t|_tjd�|j||t|��t|_|jd�tjd�tj�|k r�|j �n|r�t |dd�r�tjd�x-|D]"} | jdkr�| j�q�q�Wntjd�tj�|k r |j �ntjd�tj�|k r5|j �n|r�t |dd�r�tjd �x>|D]3} | j �rbtjd | j�| j �qbqbWndS)Nzfinalizing poolz&helping task handler/workers to finishzjoining worker handlerrr�zterminating workerszjoining task handlerzjoining result handlerzjoining pool workerszcleaning up worker %d)rr4r�rNr�rmr-rZr�rr/rnr�r��pid) �clsr�r7r8r�Zworker_handlerr�Zresult_handlerr�r�rrr rf s6 zPool._terminate_poolcCs|S)Nr)rrrr � __enter__;szPool.__enter__cCs|j�dS)N)r�)r�exc_typeZexc_valZexc_tbrrr �__exit__>sz Pool.__exit__)"rrrr+rprErrorYrurHryr rrr�r�rxr�rz�staticmethodr[r_rcr�r"r0r�rr��classmethodrfr�r�rrrr r�sD 8 3: .c@s^eZdZdd�Zdd�Zdd�Zddd �Zdd d�Zdd �ZdS)r�cCsJtj�|_tt�|_||_||_||_|||j s: ,��&%I