��],U�T@sdZdZddlZddlZddlmZmZddlTddlm Z dd �e e �D�d ddgZ[ yeWne k r�eZYnXd d�Zdd�Zdd�ZdZdZeede�ee�ZdZdZdZdZdZeeeee�ZdZedd�ee�ZdeZeee�Zeded �Zeeee�Z d!Z!d"Z"d#Z#d$Z$ed%d&�Z%ed'd(�Z&ed)d*d+d,d-d.d/d0d1� Z'd2Z(ed3d4�Z)ee'e(e)�Z*ee e*e&e�Z+ee+Z,ed5ed6d�d7ed8d��Z-edee%�Z.eee.e e*e-e�Z/e0e1ej2e,e/e#e$f��\Z3Z4Z5Z6d6ej2e!�d8ej2e"�d9e5d:e6d;e5d<e6d=e5d>e6d?e5d@e6dAe5dBe6dCe5dDe6dEe5dFe6dGe5dHe6dIe5dJe6dKe5dLe6dMe5dNe6dOe5dPe6dQe5dRe6dSe5dTe6dUe5dVe6dWddXddYddZdd[dd\di&Z7iZ8xd�D]Z9e9e8e9d~d�Z?e?d�d �Z@d�d��ZAGd�d��d��ZBej2d�ejC�ZDej2d�ejC�ZEd�d��ZFd�d��ZGd�d�ZHd�d�ZIeJd�krddlKZKeLeKjM�dkr�e@eNeKjMd�jO�ne@eKjPjO�dS)�zKa-Ping Yee z@GvR, ESR, Tim Peters, Thomas Wouters, Fred Drake, Skip Montanaro�N)�BOM_UTF8�lookup)�*�)�tokencCs&g|]}|ddkr|�qS)r�_�)�.0�xrr�-/opt/alt/python35/lib64/python3.5/tokenize.py� %s r�tokenize�generate_tokens� untokenizecGsddj|�dS)N�(�|�))�join)�choicesrrr�group0srcGst|�dS)Nr)r)rrrr�any1srcGst|�dS)N�?)r)rrrr�maybe2srz[ \f\t]*z #[^\r\n]*z\\\r?\nz[a-zA-Z_]\w*z 0[bB][01]*z0[xX][\da-fA-F]*[lL]?z0[oO]?[0-7]*[lL]?z [1-9]\d*[lL]?z[eE][-+]?\d+z\d+\.\d*z\.\d+z\d+z\d+[jJ]z[jJ]z[^'\\]*(?:\\.[^'\\]*)*'z[^"\\]*(?:\\.[^"\\]*)*"z%[^'\\]*(?:(?:\\.|'(?!''))[^'\\]*)*'''z%[^"\\]*(?:(?:\\.|"(?!""))[^"\\]*)*"""z[ubUB]?[rR]?'''z[ubUB]?[rR]?"""z&[uU]?[rR]?'[^\n'\\]*(?:\\.[^\n'\\]*)*'z&[uU]?[rR]?"[^\n"\\]*(?:\\.[^\n"\\]*)*"z\*\*=?z>>=?z<<=?z<>z!=z//=?z->z[+\-*/%&@|^=<>]=?�~z[][(){}]z\r?\nz[:;.,`@]z'[uUbB]?[rR]?'[^\n'\\]*(?:\\.[^\n'\\]*)*�'z'[uUbB]?[rR]?"[^\n"\\]*(?:\\.[^\n"\\]*)*�"�'''�"""�r'''�r"""�u'''�u"""�b'''�b"""�ur'''�ur"""�br'''�br"""�R'''�R"""�U'''�U"""�B'''�B"""�uR'''�uR"""�Ur'''�Ur"""�UR'''�UR"""�bR'''�bR"""�Br'''�Br"""�BR'''�BR"""�r�R�u�U�b�B�r'�r"�R'�R"�u'�u"�U'�U"�b'�b"�B'�B"�ur'�ur"�Ur'�Ur"�uR'�uR"�UR'�UR"�br'�br"�Br'�Br"�bR'�bR"�BR'�BR"�c@seZdZdS)� TokenErrorN)�__name__� __module__�__qualname__rrrrr]�sr]c@seZdZdS)�StopTokenizingN)r^r_r`rrrrra�srac CsF|\}}|\}}td||||t|t|�f�dS)Nz%d,%d-%d,%d: %s %s)�print�tok_name�repr) �typerZxxx_todo_changemeZxxx_todo_changeme1�lineZsrowZscolZerowZecolrrr� printtoken�srgcCs*yt||�Wntk r%YnXdS)N)� tokenize_loopra)�readline� tokeneaterrrrr �s cCs%xt|�D]}||�q WdS)N)r)rirjZ token_inforrrrh�srhc@s@eZdZdd�Zdd�Zdd�Zdd�Zd S) �UntokenizercCsg|_d|_d|_dS)Nrr)�tokens�prev_row�prev_col)�selfrrr�__init__�s zUntokenizer.__init__cCs7|\}}||j}|r3|jjd|�dS)N� )rnrl�append)ro�start�row�col� col_offsetrrr�add_whitespace�s zUntokenizer.add_whitespacecCs�x�|D]�}t|�dkr0|j||�P|\}}}}}|j|�|jj|�|\|_|_|ttfkr|jd7_d|_qWdj |j�S)N�rr�) �len�compatrwrlrrrmrn�NEWLINE�NLr)ro�iterable�t�tok_typerrs�endrfrrrr�s zUntokenizer.untokenizec Cs%d}g}|jj}|\}}|ttfkr@|d7}|ttfkrXd}x�|D]�}|dd�\}}|ttttfkr�|d7}|tkr�|j|�q_nW|t kr�|j �q_n;|ttfkr�d}n |r|r||d�d}||�q_WdS)NFrqTrxr��)rlrr�NAME�NUMBERr|r}�ASYNC�AWAIT�INDENT�DEDENT�pop) rorr~� startline�indents�toks_append�toknum�tokval�tokrrrr{�s0 zUntokenizer.compatN)r^r_r`rprwrr{rrrrrk�srkz&^[ \t\f]*#.*?coding[:=][ \t]*([-\w.]+)s^[ \t\f]*(?:[#\r\n]|$)cCsd|dd�j�jdd�}|dks=|jd�rAdS|dks\|jd �r`dS|S)N�r�-zutf-8zutf-8-�latin-1� iso-8859-1�iso-latin-1�latin-1-�iso-8859-1-�iso-latin-1-)r�r�r�)r�r�r�)�lower�replace� startswith)�orig_enc�encrrr�_get_normal_name�s"r�csd�d}d}�fdd�}�fdd�}|�}|jt�rjd�|dd�}d }|sz|gfS||�}|r�||gfStj|�s�||gfS|�}|s�||gfS||�}|r�|||gfS|||gfS) NFzutf-8cs+y��SWntk r&t�SYnXdS)N)� StopIteration�bytesr)rirr�read_or_stops z%detect_encoding..read_or_stopcs�y|jd�}Wntk r+dSYnXtj|�}|sEdSt|jd��}yt|�}Wn"tk r�td|��YnX�r�|j dkr�td��|d7}|S)N�asciirzunknown encoding: zutf-8zencoding problem: utf-8z-sig) �decode�UnicodeDecodeError� cookie_re�matchr�rr�LookupError�SyntaxError�name)rf�line_stringr��encoding�codec)� bom_foundrr�find_cookies" z$detect_encoding..find_cookieT�z utf-8-sig)r�r�blank_rer�)rir��defaultr�r��first�secondr)r�rir�detect_encoding�s0 r�cCst�}|j|�S)N)rkr)r~�utrrrrFs c!csY d}}}tjdd}}d\}}d}dg} d} d}d}d} x�y |�}Wntk rd}YnX|d}dt|�}}|r�|s�td|��|j|�}|r!|jd�}}t||d|�|||f||fVd \}}d}qv|r�|d!d�d kr�|d"d�dkr�t||||t|�f|fVd}d}qXqv||}||}qXn�|dkrU|rU|s�Pd}xv||kr;||dkr�|d}n?||d kr|t dt }n||dkr-d}nP|d}q�W||krIP| rZ| Vd} ||dkrG||dkr||d�j d�}|t|�}t|||f||t|�f|fVt||d�||f|t|�f|fVqXttf||dk||d�||f|t|�f|fVqX|| d#kr�| j |�t|d|�|df||f|fVx�|| d$kr#|| kr�tdd|||f��| dd%�} |r|| d&krd}d} d}td||f||f|fVq�W|rv| rv|| d'krvd}d} d}n!|sptd|df��d}xl||kr�tj||�}|r�|jd�\}}||f||f|}}}|||�||}}||ks|dkr(|dkr(t||||fVq�|dkr�t}|dkrOt}n|r[d} | rl| Vd} |||||fVq�|dkr�| r�| Vd} t||||fVq�|tkrXt|}|j||�}|r2|jd�}|||�}| r| Vd} t||||f|fVq�||f}||d�}|}Pq�|tks�|dd�tks�|dd �tkr|d(dkr�||f}t|p�t|dp�t|d}||d�d}}|}Pq�| r| Vd} t||||fVq�||kr|d)krb|rb|dkrKtnt||||fVqyt||||f}|dkr�| r�|} qy|dkr�| r�| dtkr�| ddkr�d}| d*}t| d| d| d | dfVd} | r| Vd} |Vq�|dkrX| r5| Vd} t||||f|fVd}q�|dkrq|d}n|dkr�|d}| r�| Vd} t||||fVqyt||||f||df|fV|d}qyWqXW| r�| Vd} x8| dd�D]&} td|df|dfdfVq Wtd|df|dfdfVdS)+Nrr� 0123456789ryFrzEOF in multi-line stringrxz\ r�z\ rq� �z# �#z z3unindent does not match any outer indentation levelz zEOF in multi-line statement�.T� �async�await�def��\z([{z)]})ryr)ryr��r�r�r�r�r�r�)r�r�r�)�string� ascii_lettersr�rzr]r�r��STRING� ERRORTOKEN�tabsize�rstrip�COMMENTr}rrr��IndentationErrorr�� pseudoprog�spanr�r|� triple_quoted�endprogs� single_quotedr�r�r��OP� ENDMARKER)!ri�lnum�parenlev� continuedZ namechars�numchars�contstr�needcont�contliner��stashed� async_def�async_def_indent�async_def_nlrf�pos�max�strstart�endprog�endmatchr��column� comment_token�nl_pos�pseudomatchrs�spos�eposr�initial�newliner��indentrrrr[sp 2 $#$ *$ $�__main__)rrrrr(r)r r!r*r+r"r#r,r-r$r%r0r1r.r/r2r3r&r'r6r7r4r5r8r9)rrr@rArBrCrDrErFrGrHrIrJrKrLrMrNrOrPrQrRrSrTrUrVrWrXrYrZr[)Q� __author__�__credits__r��re�codecsrrZlib2to3.pgen2.tokenryr�dir�__all__r�� NameError�strrrr� Whitespace�Comment�Ignore�Name� Binnumber� Hexnumber� Octnumber� Decnumber� Intnumber�Exponent� Pointfloat�Expfloat�Floatnumber� Imagnumber�Number�Single�Double�Single3�Double3�Triple�String�Operator�Bracket�Special�Funny� PlainToken�Token�ContStr�PseudoExtras�PseudoToken�list�map�compileZ tokenprogr�Zsingle3progZdouble3progr�r�rr�r�� Exceptionr]rargr rhrk�ASCIIr�r�r�r�rrr^�sysrz�argv�openri�stdinrrrr�s� * 8I�