3 ��]��@s�dZddlZddlZddlZddljZddlm Z ddl mZddlm Z mZmZmZmZmZmZmZmZmZmZmZmZdZGd d �d ejj�Zddd �Z dS)z>html2text: Turn HTML into equivalent Markdown-structured text.�N)�wrap)�config) �dumb_css_parser� element_style� escape_md�escape_md_section�google_fixed_width_font�google_has_height�google_list_style�google_text_emphasis�hn�list_numbering_start�pad_tables_in_text�skipwrap�unifiable_n�� cs�eZdZddejf�fdd� Z�fdd�Zdd�Zd d �Z�fdd�Z d d�Z dd�Zdd�Zdd�Z dd�Zdd�Zdd�Zdd�Zdd�Zdd �Zd.d"d#�Zd/d$d%�Zd&d'�Zd(d)�Zd*d+�Zd,d-�Z�ZS)0� HTML2TextN�cs t�jdd�d|_d|_d|_tj|_tj|_ tj |_||_tj |_tj|_tj|_tj|_tj|_tj|_tj|_tj|_tj|_tj|_ tj!|_"tj#|_$d|_%d|_&d|_'d|_(tj)|_*tj+|_,d|_-tj.|_/tj0|_1tj2|_3tj4|_5tj6|_7d|_8tj9|_:tj;|_<|dk�r|j=|_>n||_>g|_?d|_@d|_Ad|_Bd|_Cd|_Dg|_Eg|_Fd|_Gd|_HtIjJd �|_Kd|_Lg|_Md|_Nd|_Od|_Pd|_Qd|_Rd |_Sd|_Td|_Ud|_Vi|_Wg|_Xd|_Yd|_Zd|_[d|_\d|_]i|_^||__d|_`d|_ad|_bd|_cdtjdd<dS) z� Input parameters: out: possible custom replacement for self.outtextf (which appends lines of text). baseurl: base URL of the document we process F)Zconvert_charrefsr�*�_z**NTz^[a-zA-Z+]+://rz _place_holder;�nbsp)e�super�__init__� split_next_td�td_count�table_startrZUNICODE_SNOB�unicode_snobZESCAPE_SNOB�escape_snobZLINKS_EACH_PARAGRAPH�links_each_paragraph� body_widthZSKIP_INTERNAL_LINKS�skip_internal_linksZINLINE_LINKS�inline_linksZ PROTECT_LINKS� protect_linksZGOOGLE_LIST_INDENT�google_list_indentZIGNORE_ANCHORS�ignore_linksZ IGNORE_IMAGES� ignore_imagesZIMAGES_AS_HTML�images_as_htmlZ IMAGES_TO_ALT� images_to_altZIMAGES_WITH_SIZE�images_with_sizeZIGNORE_EMPHASIS�ignore_emphasisZ BYPASS_TABLES� bypass_tablesZ IGNORE_TABLES� ignore_tables� google_doc�ul_item_mark� emphasis_mark�strong_markZSINGLE_LINE_BREAK�single_line_breakZUSE_AUTOMATIC_LINKS�use_automatic_links�hide_strikethroughZ MARK_CODE� mark_codeZWRAP_LIST_ITEMS�wrap_list_itemsZ WRAP_LINKS� wrap_linksZ PAD_TABLES� pad_tablesZDEFAULT_IMAGE_ALT�default_image_alt�tag_callbackZ OPEN_QUOTE� open_quoteZCLOSE_QUOTE�close_quote�outtextf�out�outtextlist�quiet�p_p�outcount�start�space�a�astack�maybe_automatic_link� empty_link�re�compile�absolute_url_matcher�acount�list� blockquote�pre�startpre�code�quote� br_toggle� lastWasNL�lastWasList�style� style_def� tag_stack�emphasis�drop_white_space�inheader� abbr_title� abbr_data� abbr_list�baseurl�stressed�preceding_stressed�preceding_data�current_tag� UNIFIABLE)�selfr>r_� bodywidth)� __class__��/usr/lib/python3.6/__init__.pyr"s� zHTML2Text.__init__cs|jdd�}t�j|�dS)Nzz )�replacer�feed)re�data)rgrhrirk|szHTML2Text.feedcCs8|j|�|jd�|j|j��}|jr0t|�S|SdS)Nr)rk�optwrap�closer8r)rerlZmarkdownrhrhri�handle�s zHTML2Text.handlecCs"|jj|�|r|ddk|_dS)N�� )r?�appendrT)re�srhrhrir=�szHTML2Text.outtextfcsZt�j�|j�|jddd�dj|j�}|jr@tjj d}nd}|j d|�}g|_|S)Nr�end)�forceznbsp;� z _place_holder;)rrn�pbr�o�joinr?r�html�entities�html5rj)reZouttextr)rgrhrirn�s zHTML2Text.closecCs|j|j|�d�dS)NT)�handle_data�charref)re�crhrhri�handle_charref�szHTML2Text.handle_charrefcCs|j|�}|r|j|d�dS)NT)� entityrefr~)rer��refrhrhri�handle_entityref�s zHTML2Text.handle_entityrefcCs|j||dd�dS)NT)rC)� handle_tag)re�tag�attrsrhrhri�handle_starttag�szHTML2Text.handle_starttagcCs|j|ddd�dS)NF)rC)r�)rer�rhrhri� handle_endtag�szHTML2Text.handle_endtagcCs�d|krdSd}xpt|j�D]b\}}d|krv|d|dkrvd|ksLd|krrd|krvd|krv|d|dkrvd}nd}|r|SqWdS)z� :type attrs: dict :returns: The index of certain set of attributes (of a link) in the self.a list. If the set of attributes is not found, returns None :rtype: int �hrefNF�titleT)� enumeraterE)rer��match�irErhrhri� previousIndex�szHTML2Text.previousIndexcCs�t|�}t|�}d|ko|j}d}x$tjD]}||ko<||k}|r*Pq*Wd|koVd|k} t|�opt|�op|j} |�r|s�| s�| r�|jd7_|r�|jd7_| r�|j|j �|j d7_ |r�|j|j�|j d7_ | �r�|jd�|j d7_ d|_n�|�s| �s| �r,|jd8_d|_ | �rZ|j �rJ|j d8_ n |jd�d|_|�r�|j �rx|j d8_ n|j|j�| �r�|j �r�|j d8_ n|j|j �|�s�| �r�|j�r�|jd�|�r�|jd8_dS) z/ Handles various text emphases zline-throughF�italicrp�`TrwN)rr4rZBOLD_TEXT_STYLE_VALUESrrOrYr@ryr0rZr1rQrD)rerC� tag_style�parent_styleZtag_emphasisZparent_emphasisZ strikethroughZboldZbold_markerr�Zfixedrhrhri�handle_emphasis�s^ zHTML2Text.handle_emphasiscCs�||_|dkri}nt|�}|jdk r>|j||||�dkr>dS|rx|jdk rx|d^krx|dksb|jrx|jd�d|_d |_|jr�i}|r�|jr�|jd_d}t ||j |�}|jj|||f�n4|jr�|jj�ndiif\}}}|jr�|jd`d}t |��r0|j�|�r&d|_|jt |�dd �n d |_dS|dak�r�|j�rf|�r\t|��r\|j�n|j�n|j�rz|dk�rzn|j�|dk�r�|�r�|jdk�r�|jd�n |jd�|dk�r�|�r�|j�|jd�|j�|dbk�r|�r�|jd 7_n|jd 8_|dk�r:|�r,|jd 7_n|jd 8_|dck�rJd|_|dk�r�|�r�|j�|jddd�d|_|jd 7_n|jd 8_|j�dd�}|ddk�r�|j�r�|�r�||��r�d |j}n|j}|j|�|�r�d|_|dek�r<|j�r<|�r ||��r d |j} n|j} |j| �|�r|d8|d1<|j1d9��pN|j2}|j3�st|j4�r�d:|k�std;|k�r�|jd<|d8d=�d:|k�r�|jd>|d:d=�d;|k�r�|jd?|d;d=�|�r�|jd@|d=�|jdA�dS|jdk �r\|j}|j0�rFt)|�|k�rF|j5j6|��rF|jd2t)|�d3�d |_dS|jd�d|_d |_|j0�rt|jt)|��n�|jdBt)|�d7�|j(�r�|j1d1��p�d+}|jdCt)t7j8|j9|��dD�nh|j+|�}|dk �r�|j,|}n.|j-d 7_-|j-|d4<|j.|d5<|j,j|�|jdt/|d4�d7�|dk�rD|�rD|j�|dk�r^|�r^|j:�|dEk�rx|�rx|jdF�|dEk�r�|�r�|j:�|dik� r(|j;�r�|j<�r�|j�|�r�|j�r�t=|�}n|}t>|�}|j;j||dI��n0|j;� r |j;j�|j� r |j;� r |jdJ�d|_ z Zhrz* * *�head�script�bodyrNz> )rvcSs|jotjd|jd�S)Nz[^\s]rprr)rbrIr�)rerhrhri�no_preceding_space|sz0HTML2Text.handle_tag..no_preceding_space�emr��u�strong�b�del�strikertz ~~z~~�kbdrQ�ttr��abbrrr��qcSs@tj|j|�}|j�r dj|�nd}|jdjt|�|d��dS)Nz "{}"rz]({url}{title}))�urlr�)�urlparse�urljoinr_�strip�formatryr)re�linkr�r�rhrhri�link_url�sz&HTML2Text.handle_tag..link_urlrEr��<�>�countrBz][�]�src�alt�widthZheightz

)rer�r�rCr�r�Zdummyr�rYr�r�r�rEr�r�r�r�Z list_styleZnumbering_startr�� nest_countrhrhrir�s$ zHTML2Text.handle_tagcCs|jdkrd|_dS)zPretty print has a line breakrrpN)rA)rerhrhrirx�s z HTML2Text.pbrcCs|jr dnd|_dS)z Set pretty print to 1 or 2 linesrpr�N)r2rA)rerhrhrir��szHTML2Text.pcCs|j�d|_dS)zSoft breaksz N)rxrS)rerhrhrir��szHTML2Text.soft_brFcCsF|jdk r|j|7_|j�sB|jrT|j�}|jrF|jp>|jrF|}|dkrTd|_|r�|jr�tjdd|�}|r�|ddkr�d|_ |dd�}|r�|r�dS|j r�|jd�r�|jd �r�d|}|jr�|j d �d|_d|j}|o�|o�|ddk�r|j�r|d7}|j�rX|j�s&|d7}x tt|j��D]}|d7}�q6W|jdd|�}|j �rxd |_ |j�rx|jd�}|j�r�d |_ d|_d |_|dk�r�d|_|j d�d |_ |j�r�|j |jd||j�d |_ d|_|j �r|j�s�|j d�d |_ |j�r�|jdk�r|j�s&|dk�r�|dk�r:|j d�g}x�|jD]x}|j|dk�r�|j dt|d�dtj|j|d��d|k�r�|j d|dd�|j d�n |j|��qFW|j|k�r�|j d�||_|j�r$|dk�r$x2|jj �D]$\} } |j d| d| d��q�Wd|_|j |�|jd7_dS)z6 Deal with indentation and whitespace Nrrz\s+rwTrprqz z [code]r�z Frur�rBz [r�z]: r�r�z (r�z *[)!r]r@r.�lstriprZrOrQrI�subrDrPr�r5r>rArNrM�ranger�rjrCrSrTrEr rBr�r�r�r_rsr^�items)rerl�puredatarvZlstripped_dataZbqr�Znewar�r�Z definitionrhrhriry�s� ( zHTML2Text.ocCs|sdS|jr$|j�}d|_d|_n:|jr^tjd|d�rXt|j�rX|jdkrXd|}d|_|jrt|jj t |��|jdk r�|j}||kr�|jj|�r�|j r�|jd |d �d|_dS|jd�d|_d|_|jr�|jr�|r�t||jd�}||_|j|dd �dS)NFTz[^\s.!?]rrErQrOrwr�r�r�)Zsnob)r�)rErQrO)r`r�rarIr�rrcrVrW�updaterrGrKr3ryrHrQrOrrrb)rerlZentity_charr�rhrhrir~s: zHTML2Text.handle_datacCsb|ddkr t|dd�d�}nt|�}|jr@|tkr@t|Syt|�Stk r\dSXdS)Nr�x�Xrp�r)r�r�)�intrr�chr� ValueError)rer�r�rhrhrir=szHTML2Text.charrefcCsd|jr|tjkrtj|Sytjj|d}Wntk rLd|dSX|dkr`tj|S|S)N�;�&r)rrrdr{r|r}r�)rer�Zchrhrhrir�Ks zHTML2Text.entityrefcCs*d}d|kr&t|ddd��|j}|S)zq Calculate the nesting count of google doc lists :type style: dict :rtype: int rzmargin-leftNr��)r�r%)rerVr�rhrhrir�TszHTML2Text.google_nest_countcCs|js |Sd}d}|jsd|_x�|jd�D]�}t|�dkr�t||j|j�s�d}|jd|j�rdd}n|jd�rrd}t ||jd|d�}|dj |�7}|jd�r�|d 7}d }q�|r�|d7}d }q�|d7}d}ntj j|�s�||d7}d }q*|dkr*|d7}|d 7}q*W|S) zi Wrap all paragraphs in the provided text. :type text: str :rtype: str rrFrqz z z> )Zbreak_long_wordsZsubsequent_indentz rpz r�)r!r7r#�splitr�rr6r�r/rrz�endswithrZRE_SPACEr�)re�text�result�newlinesZpara�indent�wrappedrhrhrirmbsF zHTML2Text.optwrap)FF)F)�__name__� __module__�__qualname__r� BODY_WIDTHrrkror=rnr�r�r�r�r�r�r�rxr�r�ryr~rr�r�rm� __classcell__rhrh)rgrir!s.Z G} o + rrcCs$|dkrtj}t||d�}|j|�S)N)r_rf)rr�rro)r{r_rf�hrhrhri� html2text�sr�)rrr)rN)�__doc__Z html.entitiesr{Zhtml.parserrIZurllib.parse�parser��textwraprr�rZhtml2text.utilsrrrrrr r rrr rrr�__version__�parserZ HTMLParserrrhrhrhri�s" <