Ë
|
nñúhz ã óê d Z ddlmZ ddlmZ ddlZddlZddlmZ ddlm Z m
|
Z
|
m Z ddl Z ddl mZ ddlmZ dd lmZmZ dd
|
lmZ dd lmZ dd lmZ dd lmZ ddl m Z ddl m Z ddl m!Z! ddl"m#Z# ddl$m%Z% ddl&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z- ddl.m/Z/ ddl0m1Z1 e r ddl2m3Z3m4Z4 ddl5m6Z6m7Z7m8Z8m9Z9m:Z:m;Z; ddl m<Z< ejz d« Z>e>fd6dZ?d7dZ@ d8d ZA G d d « ZB G d d!eB« ZCd9d"ZDd#d$iZE G d% d&eB« ZFd:d'ZGd( ZHeFeFeCeCd)ZId;d*ZJd9d+ZKd, ZLd- ZM ee%d ¬.« d/dddddd0d1dd2ddd3d3dej dd4 d<d5« ZOy)=zW
|
:mod:`pandas.io.html` is a module containing functionality for dealing with
|
HTML IO.
|
|
é )Ú annotations)ÚabcN)ÚPattern)Ú TYPE_CHECKINGÚLiteralÚcast)Úlib)Úimport_optional_dependency)ÚAbstractMethodErrorÚEmptyDataError)Údoc)Úfind_stack_level)Úcheck_dtype_backend)Ú is_list_like©Úisna)ÚIndex)Ú
|
MultiIndex)ÚSeries)Ú _shared_docs)Ú file_existsÚ
|
get_handleÚ is_file_likeÚ is_fsspec_urlÚis_urlÚstringify_pathÚvalidate_header_arg)Ú pprint_thing)Ú
|
TextParser)ÚIterableÚSequence)Ú
|
BaseBufferÚ DtypeBackendÚFilePathÚ HTMLFlavorsÚ
|
ReadBufferÚStorageOptions)Ú DataFramez[\r\n]+|\s{2,}c óB |j d| j « « S )a
|
Replace extra whitespace inside of a string with a single space.
|
|
Parameters
|
----------
|
s : str or unicode
|
The string from which to remove extra whitespace.
|
regex : re.Pattern
|
The regular expression to use to remove extra whitespace.
|
|
Returns
|
-------
|
subd : str or unicode
|
`s` with all extra whitespace replaced with a single space.
|
ú )ÚsubÚstrip)ÚsÚregexs úAH:\Change_password\venv_build\Lib\site-packages\pandas/io/html.pyÚ_remove_whitespacer0 I s ð 99S!'')Ó $Ð$ó c óL t | t « r@| j xs d| j xs d}}t t || j |« « S t | t j « s t | « r t d| « S | yt t | « j d« )a
|
Get an iterator given an integer, slice or container.
|
|
Parameters
|
----------
|
skiprows : int, slice, container
|
The iterator to use to skip rows; can also be a slice.
|
|
Raises
|
------
|
TypeError
|
* If `skiprows` is not a slice, integer, or Container
|
|
Returns
|
-------
|
it : iterable
|
A proper iterator to use to skip rows of a DataFrame.
|
r é úint | Sequence[int]z& is not a valid type for skipping rows)Ú
|
isinstanceÚsliceÚstartÚstepÚlistÚrangeÚstopÚnumbersÚIntegralr r Ú TypeErrorÚtypeÚ__name__)Úskiprowsr7 r8 s r/ Ú _get_skiprowsrB \ s ô& (EÔ"Ø nnÒ)¨¨8¯=©=Ò+=¸AtÜE% § ¡ °Ó5Ó6Ð6Ü Hg× .Ñ .Ô /´<ÀÔ3IÜÐ)¨8Ó4Ð4Ø Ð ØÜ
|
tH~×.Ñ.Ð/Ð/UÐVÓ
|
WÐWr1 Ústorage_optionsc ó` t | « s't | d« st | t « r?t | « r4t | d||¬« 5 }|j j « }ddd« |S t | t t f« r| }|S t dt | « j d« # 1 sw Y S xY w)zµ
|
Try to read from a url, file or string.
|
|
Parameters
|
----------
|
obj : str, unicode, path object, or file-like object
|
|
Returns
|
-------
|
raw_text : str
|
ÚreadÚr)ÚencodingrC Nz Cannot read object of type 'ú') r Úhasattrr5 Ústrr r ÚhandlerE Úbytesr> r? r@ )ÚobjrG rC ÚhandlesÚtexts r/ Ú_readrP y s¨ ô$ s Ü 3Ô Ü sCÔ ¤[°Ô%5ä Ø x¸ ô
|
ð )à Ø>>×&Ñ&Ó(D÷ )ð Kô
|
C#u Ô &Øð Kô Ð6´t¸C³y×7IÑ7IÐ6JÈ!ÐLÓMÐM÷ )ð Kús ÁB#Â#B-c ó e Zd ZdZ d ddZd Zd ZddZd Zd Z d Z
|
d
|
Z d Z d Z dd Zd Zd Z ddZddZy)Ú_HtmlFrameParseraZ
|
Base class for parsers that parse HTML into DataFrames.
|
|
Parameters
|
----------
|
io : str or file-like
|
This can be either a string of raw HTML, a valid URL using the HTTP,
|
FTP, or FILE protocols or a file-like object.
|
|
match : str or regex
|
The text to match in the document.
|
|
attrs : dict
|
List of HTML <table> element attributes to match.
|
|
encoding : str
|
Encoding to be used by parser
|
|
displayed_only : bool
|
Whether or not items with "display:none" should be ignored
|
|
extract_links : {None, "all", "header", "body", "footer"}
|
Table elements in the specified section(s) with <a> tags will have their
|
href extracted.
|
|
.. versionadded:: 1.5.0
|
|
Attributes
|
----------
|
io : str or file-like
|
raw HTML, URL, or file-like object
|
|
match : regex
|
The text to match in the raw HTML
|
|
attrs : dict-like
|
A dictionary of valid table attributes to use to search for table
|
elements.
|
|
encoding : str
|
Encoding to be used by parser
|
|
displayed_only : bool
|
Whether or not items with "display:none" should be ignored
|
|
extract_links : {None, "all", "header", "body", "footer"}
|
Table elements in the specified section(s) with <a> tags will have their
|
href extracted.
|
|
.. versionadded:: 1.5.0
|
|
Notes
|
-----
|
To subclass this class effectively you must override the following methods:
|
* :func:`_build_doc`
|
* :func:`_attr_getter`
|
* :func:`_href_getter`
|
* :func:`_text_getter`
|
* :func:`_parse_td`
|
* :func:`_parse_thead_tr`
|
* :func:`_parse_tbody_tr`
|
* :func:`_parse_tfoot_tr`
|
* :func:`_parse_tables`
|
* :func:`_equals_tag`
|
See each method's respective documentation for details on their
|
functionality.
|
Nc óf || _ || _ || _ || _ || _ || _ || _ y ©N)ÚioÚmatchÚattrsrG Údisplayed_onlyÚ extract_linksrC )ÚselfrU rV rW rG rX rY rC s r/ Ú__init__z_HtmlFrameParser.__init__ß s8 ð Ø
|
Ø
|
Ø Ø ,ÔØ *ÔØ .Õ r1 c ó j j « j j « } fd|D « S )z
|
Parse and return all tables from the DOM.
|
|
Returns
|
-------
|
list of parsed (header, body, footer) tuples from tables.
|
c 3 ó@ K | ] }j |« y wrT )Ú_parse_thead_tbody_tfoot)Ú.0ÚtablerZ s r/ ú <genexpr>z0_HtmlFrameParser.parse_tables.<locals>.<genexpr>ú s øè ø ÒI¸×-Ñ-¨e×4ÑIùs )Ú _parse_tablesÚ
|
_build_docrV rW )rZ Útabless ` r/ Ú parse_tablesz _HtmlFrameParser.parse_tablesñ s3 ø ð ×#Ñ# D§O¡OÓ$5°t·z±zÀ4Ç:Á:ÓNÛIÀ&ÔIÐIr1 c ó$ |j |« S )aC
|
Return the attribute value of an individual DOM node.
|
|
Parameters
|
----------
|
obj : node-like
|
A DOM node.
|
|
attr : str or unicode
|
The attribute, such as "colspan"
|
|
Returns
|
-------
|
str or unicode
|
The attribute value.
|
)Úget)rZ rM Úattrs r/ Ú _attr_getterz _HtmlFrameParser._attr_getterü s ð$ wwt}Ð r1 c ó t | « )a
|
Return a href if the DOM node contains a child <a> or None.
|
|
Parameters
|
----------
|
obj : node-like
|
A DOM node.
|
|
Returns
|
-------
|
href : str or unicode
|
The href from the <a> child of the DOM node.
|
©r ©rZ rM s r/ Ú _href_getterz _HtmlFrameParser._href_getter ó ô " $Ó'Ð'r1 c ó t | « )a
|
Return the text of an individual DOM node.
|
|
Parameters
|
----------
|
obj : node-like
|
A DOM node.
|
|
Returns
|
-------
|
text : str or unicode
|
The text from an individual DOM node.
|
rk rl s r/ Ú _text_getterz _HtmlFrameParser._text_getter rn r1 c ó t | « )a
|
Return the td elements from a row element.
|
|
Parameters
|
----------
|
obj : node-like
|
A DOM <tr> node.
|
|
Returns
|
-------
|
list of node-like
|
These are the elements of each row, i.e., the columns.
|
rk rl s r/ Ú _parse_tdz_HtmlFrameParser._parse_td0 rn r1 c ó t | « )a;
|
Return the list of thead row elements from the parsed table element.
|
|
Parameters
|
----------
|
table : a table element that contains zero or more thead elements.
|
|
Returns
|
-------
|
list of node-like
|
These are the <tr> row elements of a table.
|
rk ©rZ r` s r/ Ú_parse_thead_trz _HtmlFrameParser._parse_thead_tr@ ó ô " $Ó'Ð'r1 c ó t | « )aì
|
Return the list of tbody row elements from the parsed table element.
|
|
HTML5 table bodies consist of either 0 or more <tbody> elements (which
|
only contain <tr> elements) or 0 or more <tr> elements. This method
|
checks for both structures.
|
|
Parameters
|
----------
|
table : a table element that contains row elements.
|
|
Returns
|
-------
|
list of node-like
|
These are the <tr> row elements of a table.
|
rk rt s r/ Ú_parse_tbody_trz _HtmlFrameParser._parse_tbody_trO ó ô" " $Ó'Ð'r1 c ó t | « )a,
|
Return the list of tfoot row elements from the parsed table element.
|
|
Parameters
|
----------
|
table : a table element that contains row elements.
|
|
Returns
|
-------
|
list of node-like
|
These are the <tr> row elements of a table.
|
rk rt s r/ Ú_parse_tfoot_trz _HtmlFrameParser._parse_tfoot_trb rv r1 c ó t | « )ai
|
Return all tables from the parsed DOM.
|
|
Parameters
|
----------
|
document : the DOM from which to parse the table element.
|
|
match : str or regular expression
|
The text to search for in the DOM tree.
|
|
attrs : dict
|
A dictionary of table attributes that can be used to disambiguate
|
multiple tables on a page.
|
|
Raises
|
------
|
ValueError : `match` does not match any text in the document.
|
|
Returns
|
-------
|
list of node-like
|
HTML <table> elements to be parsed into raw data.
|
rk )rZ ÚdocumentrV rW s r/ rb z _HtmlFrameParser._parse_tablesq s ô0 " $Ó'Ð'r1 c ó t | « )a?
|
Return whether an individual DOM node matches a tag
|
|
Parameters
|
----------
|
obj : node-like
|
A DOM node.
|
|
tag : str
|
Tag name to be checked for equality.
|
|
Returns
|
-------
|
boolean
|
Whether `obj`'s tag name is `tag`
|
rk ©rZ rM Útags r/ Ú _equals_tagz _HtmlFrameParser._equals_tag ry r1 c ó t | « )zÃ
|
Return a tree-like object that can be used to iterate over the DOM.
|
|
Returns
|
-------
|
node-like
|
The DOM from which to parse the table element.
|
rk )rZ s r/ rc z_HtmlFrameParser._build_doc s ô " $Ó'Ð'r1 c ój j |« } j |« } j |« } fd}|s;|r9 ||d « r.|j |j d« « |r ||d « r. j |d¬« } j |d¬« } j |d¬« }|||fS )a
|
Given a table, return parsed header, body, and foot.
|
|
Parameters
|
----------
|
table_html : node-like
|
|
Returns
|
-------
|
tuple of (header, body, footer), each a list of list-of-text rows.
|
|
Notes
|
-----
|
Header and body are lists-of-lists. Top level list is a list of
|
rows. Each row is a list of str text.
|
|
Logic: Use <thead>, <tbody>, <tfoot> elements to identify
|
header, body, and footer, otherwise:
|
- Put all rows into body
|
- Move rows from top of body to header only if
|
all elements inside row are <th>
|
- Move rows from bottom of body to footer only if
|
all elements inside row are <th>
|
c óJ t fdj | « D « « S )Nc 3 óB K | ] }j |d « yw)ÚthN)r )r_ ÚtrZ s r/ ra zS_HtmlFrameParser._parse_thead_tbody_tfoot.<locals>.row_is_all_th.<locals>.<genexpr>Ç s øè ø ÒN°Qt×'Ñ'¨¨4×0ÑNùs )Úallrr )ÚrowrZ s r/ Ú row_is_all_thz@_HtmlFrameParser._parse_thead_tbody_tfoot.<locals>.row_is_all_thÆ s ø ÜÓN¸$¿.¹.ÈÓ:MÔNÓNÐ Nr1 r Úheader)ÚsectionÚbodyÚfooter)ru rx r{ ÚappendÚpopÚ_expand_colspan_rowspan) rZ Ú
|
table_htmlÚ header_rowsÚ body_rowsÚ footer_rowsr r r r s ` r/ r^ z)_HtmlFrameParser._parse_thead_tbody_tfoot© sÅ ø ð2 ×*Ñ*¨:Ó6 Ø×(Ñ(¨Ó4 Ø×*Ñ*¨:Ó6 ô Oñ ñ ¡ ¨i¸©lÔ ;Ø×"Ñ" 9§=¡=°Ó#3Ô4ñ ¡ ¨i¸©lÕ ;ð ×-Ñ-¨kÀ8Ð-ÓLØ×+Ñ+¨I¸vÐ+ÓFØ×-Ñ-¨kÀ8Ð-ÓLàtVÐ#Ð#r1 c óä g }g }|D ] }g }g }d}| j |« } | D ]$ }
|
|r`|d d |k rU|j d« \ } } } |j | « | dkD r|j | | | dz
|
f« |dz }|r |d d |k rUt | j |
|
« « }| j
|
d|fv r| j |
|
« }||f}t | j |
|
d« xs d« }t | j |
|
d« xs d« }t |« D ]4 }|j |« |dkD r|j |||dz
|
f« |dz }6 ' |D ]4 \ } } } |j | « | dkD s |j | | | dz
|
f« 6 |j |« |} |rSg }g }|D ]4 \ } } } |j | « | dkD s |j | | | dz
|
f« 6 |j |« |}|rS|S )a*
|
Given a list of <tr>s, return a list of text rows.
|
|
Parameters
|
----------
|
rows : list of node-like
|
List of <tr>s
|
section : the section that the rows belong to (header, body or footer).
|
|
Returns
|
-------
|
list of list
|
Each returned row is a list of str text, or tuple (text, link)
|
if extract_links is not None.
|
|
Notes
|
-----
|
Any cell with ``rowspan`` or ``colspan`` will have its contents copied
|
to subsequent cells.
|
r r3 r ÚrowspanÚcolspan)
|
rr r r r0 rp rY rm Úintri r: )rZ Úrowsr Ú all_textsÚ remainderÚtrÚtextsÚnext_remainderÚindexÚtdsÚtdÚprev_iÚ prev_textÚ prev_rowspanrO Úhrefr r Ú_s r/ r z(_HtmlFrameParser._expand_colspan_rowspanÖ s^ ð. ð ð ð ó % 'BØEØ NàEØ.. Ó$CØó ñ I¨a¡L°¡O°uÒ$<Ø6?·m±mÀAÓ6FÑ3FI |ØLL Ô+Ø# aÒ'Ø&×-Ñ-¨v°yÀ,ÐQRÑBRÐ.SÔTØQJEñ I¨a¡L°¡O°uÓ$<ô *¨$×*;Ñ*;¸BÓ*?Ó@Ø×%Ñ%¨%°Ð)9Ñ9Ø ×,Ñ,¨RÓ0DØ $<DÜ d× /Ñ /°°IÓ >Ò CÀ!ÓDÜ d× /Ñ /°°IÓ >Ò CÀ!ÓDä w ò AØLL Ô&Ø {Ø&×-Ñ-¨u°d¸GÀa¹KÐ.HÔIØQJEò ð% ð2 4=ò QÑ/ <Ø YÔ'Ø !Ó#Ø"×)Ñ)¨6°9¸lÈQÑ>NÐ*OÕPð Qð
|
× Ñ UÔ #Ø&IðK% 'ñR Ø NØEØ3<ò QÑ/ <Ø YÔ'Ø !Ó#Ø"×)Ñ)¨6°9¸lÈQÑ>NÐ*OÕPð Qð × Ñ UÔ #Ø&Iò ð Ðr1 c ó¤ | j s|S |D cg c]2 }dt ||« j dd« j dd« vr|4 c}S c c}w )a
|
Return list of tables, potentially removing hidden elements
|
|
Parameters
|
----------
|
tbl_list : list of node-like
|
Type of list elements will vary depending upon parser used
|
attr_name : str
|
Name of the accessor for retrieving HTML attributes
|
|
Returns
|
-------
|
list of node-like
|
Return type matches `tbl_list`
|
ú display:noneÚstyleÚ r* )rX Úgetattrrg Úreplace)rZ Útbl_listÚ attr_nameÚxs r/ Ú_handle_hidden_tablesz&_HtmlFrameParser._handle_hidden_tables( sa ð ×"Ò"ØOð ö
|
àØ Ü1iÓ(×,Ñ,¨W°bÓ9×AÑAÀ#ÀrÓJñKò ò
|
ð
|
ùò
|
s 7A rT )rU z.FilePath | ReadBuffer[str] | ReadBuffer[bytes]rV ú str | PatternrW údict[str, str] | NonerG rJ rX ÚboolrY ú0Literal[None, 'header', 'footer', 'body', 'all']rC r' ÚreturnÚNone©r¶ ú
|
str | None©r¶ r´ )r z#Literal['header', 'footer', 'body'])r¯ rJ )r@ Ú
|
__module__Ú __qualname__Ú__doc__r[ re ri rm rp rr ru rx r{ rb r rc r^ r r± © r1 r/ rR rR sµ ñBðX +/ð/à :ð/ð ð/ð %ð /ð
|
ð /ð ð /ð Hð/ð (ð/ð
|
ó/ò$ Jò ó((ò (ò (ò (ò (ò& (ò (ó4(ò& (ò+$ðZPØ @óPôd
|
r1 rR c óP e Zd ZdZd Zd dZd ZddZd Zd Z d Z
|
d Z d
|
Z d Z y )Ú!_BeautifulSoupHtml5LibFrameParsera/
|
HTML to DataFrame parser that uses BeautifulSoup under the hood.
|
|
See Also
|
--------
|
pandas.io.html._HtmlFrameParser
|
pandas.io.html._LxmlFrameParser
|
|
Notes
|
-----
|
Documentation strings for this class are in the base class
|
:class:`pandas.io.html._HtmlFrameParser`.
|
c ó" d}|j ||¬« }|s t d« g }t « }| j |d« }|D ]¦ }| j r`|j d« D ] } | j « |j t j d« ¬« D ] } | j « ||vr#|j |¬« |j |« |j |« ¨ |s!t d t |j « « |S )
|
Nr` )rW zNo tables foundrW rª zdisplay:\s*none)rª )Ústringz!No tables found matching pattern ) Úfind_allÚ
|
ValueErrorÚsetr± rX Ú decomposeÚreÚcompileÚfindr ÚaddÚreprÚpattern)
|
rZ r} rV rW Ú element_namerd ÚresultÚ unique_tablesr` Úelems
|
r/ rb z/_BeautifulSoupHtml5LibFrameParser._parse_tablesR s Ø Ø×"Ñ" <°uÐ"Ó=ÙÜ Ð .Ó/Ð /àÜ Ø×+Ñ+¨F°GÓ<àò
|
%EØ×"Ò"Ø !NN¨7Ó 3ò %DØNNÕ$ð%ð "NN´·±Ð<NÓ1ONÓ Pò %DØNNÕ$ð%ð MÑ)¨e¯j©jÀ¨jÓ.FÐ.RØ eÔ$Ø × Ñ eÕ $ð
|
%ñ Ü Ð @ÄÀeÇmÁmÓATÐ@UÐ VÓWÐ WØ r1 c ó: |j dd¬« }|sd S |d S )NÚaT)r¦ r¦ )rÉ )rZ rM rÒ s r/ rm z._BeautifulSoupHtml5LibFrameParser._href_getterk s% Ø HHStHÓ $Ù tÐ+ ! F¡)Ð+r1 c ó |j S rT )rO rl s r/ rp z._BeautifulSoupHtml5LibFrameParser._text_gettero s Øxxr1 c ó |j |k( S rT )Únamer s r/ r z-_BeautifulSoupHtml5LibFrameParser._equals_tagr s Øxx3Ð r1 c ó( |j dd¬« S )N)r¢ r F©Ú recursive)rà ©rZ r s r/ rr z+_BeautifulSoupHtml5LibFrameParser._parse_tdu s Ø||L°E|Ó:Ð:r1 c ó$ |j d« S )Nzthead tr©Úselectrt s r/ ru z1_BeautifulSoupHtml5LibFrameParser._parse_thead_trx ó Ø||JÓ'Ð'r1 c óT |j d« }|j dd¬« }||z S )Nztbody trr Fr× )rÜ rà ©rZ r` Ú
|
from_tbodyÚ from_roots r/ rx z1_BeautifulSoupHtml5LibFrameParser._parse_tbody_tr{ s- Ø\\ *Ó-
|
ØNN 4°5NÓ9 àIÑ%Ð%r1 c ó$ |j d« S )Nztfoot trrÛ rt s r/ r{ z1_BeautifulSoupHtml5LibFrameParser._parse_tfoot_tr rÝ r1 c ó t | j | j | j « }|st d| j « |S )Nz No text parsed from document: )rP rU rG rC rÄ )rZ Úraw_texts r/ Ú_setup_build_docz2_BeautifulSoupHtml5LibFrameParser._setup_build_doc s= Ü $§-¡-°×1EÑ1EÓFÙÜ Ð =¸d¿g¹g¸YÐ GÓHÐ HØr1 c ó@ ddl m} | j « }t |t « r*| j
|
|j | j
|
« }d }n|}| j
|
} ||d|¬« }|j d« D ] }|j d|j z « " |S )Nr )Ú BeautifulSoupÚhtml5lib)ÚfeaturesÚ from_encodingÚbrú
|
)
|
Úbs4rç rå r5 rL rG ÚdecoderÃ Ú replace_withrO )rZ rç ÚbdocÚudocrê Úsouprë s r/ rc z,_BeautifulSoupHtml5LibFrameParser._build_doc s Ý%à×$Ñ$Ó&Ü dEÔ " t§}¡}Ð'@Ø;;t}}Ó-DØ MàDØ MMMá T¨JÀmÔTà-- Ó%ò ,BØ OOD 2§7¡7NÕ +ð ,ð r1 Nr¸ rº )r@ r» r¼ r½ rb rm rp r rr ru rx r{ rå rc r¾ r1 r/ rÀ rÀ C s9 ñ ò ó2,òó ò;ò(ò&ò (òó r1 rÀ c óÌ d| v r| j d« | d< dj | j « D cg c] \ }}d| dt |« c}}« }d| dS c c}}w )ar
|
Build an xpath expression to simulate bs4's ability to pass in kwargs to
|
search for attributes when using the lxml parser.
|
|
Parameters
|
----------
|
attrs : dict
|
A dict of HTML attributes. These are NOT checked for validity.
|
|
Returns
|
-------
|
expr : unicode
|
An XPath expression that checks for the given HTML attributes.
|
Úclass_Úclassz and ú@ú=ú[ú])r ÚjoinÚitemsrË )rW ÚkÚvr- s r/ Ú_build_xpath_exprrþ se ð 5ÑØ 8Ó,gà °E·K±K³M×B©D¨A¨q!Ad 1gYÒ'ÓBÓCAØ qc8Oùó Cs ·A
|
rÇ z$http://exslt.org/regular-expressionsc óJ e Zd ZdZd dZd Zd Zd Zd dZd Z d Z
|
d Z d
|
Z y )Ú_LxmlFrameParseraY
|
HTML to DataFrame parser that uses lxml under the hood.
|
|
Warning
|
-------
|
This parser can only handle HTTP, FTP, and FILE urls.
|
|
See Also
|
--------
|
_HtmlFrameParser
|
_BeautifulSoupLxmlFrameParser
|
|
Notes
|
-----
|
Documentation strings for this class are in the base class
|
:class:`_HtmlFrameParser`.
|
c ó6 |j d« }|sd S |d S )Nz
|
.//a/@hrefr ©Úxpath)rZ rM r¦ s r/ rm z _LxmlFrameParser._href_getterÊ s! Øyy Ó&Ù tÐ, T¨!¡WÐ,r1 c ó" |j « S rT )Ú text_contentrl s r/ rp z _LxmlFrameParser._text_getterÎ s Ø× Ñ Ó!Ð!r1 c ó$ |j d« S )Nú ./td|./thr rÙ s r/ rr z_LxmlFrameParser._parse_tdÑ s ð yyÓ%Ð%r1 c óü |j }dt |« d}|r|t |« z }|j |t ¬« }| j |d« }| j r|D ]} }|j d« D ] }|j « |j d« D ]A }d|j j dd « j d
|
d « v s2|j « C |st d t |« « |S ) Nz //table[.//text()[re:test(., z)]])Ú
|
namespacesÚattribz.//stylez .//*[@style]r© rª r« r* z No tables found matching regex ) rÌ rË rþ r Ú _re_namespacer± rX Ú drop_treer
|
rg r rÄ ) rZ r} rV ÚkwargsrÌ Ú
|
xpath_exprrd r` rÐ s r/ rb z _LxmlFrameParser._parse_tablesÖ sü Ø--ð 5´T¸'³]°OÀ3ÐG
|
ñ Ø Ô+¨FÓ3Ñ 3Jà
|
´} ÓEà×+Ñ+¨F°HÓ=Ø × Ò Ø ò )ð "KK¨
|
Ó 3ò %DØNNÕ$ð%à !KK¨Ó 7ò )DØ%¨¯©¯ © ¸À"Ó)E×)MÑ)MÈcÐSUÓ)VÒVØ Õ(ñ)ð )ñ Ü Ð >¼tÀG»}¸oÐ NÓOÐ OØ r1 c ó |j |k( S rT )r r s r/ r z _LxmlFrameParser._equals_tagó s Øww#~Ð r1 c óÔ ddl m} ddlm}m}m} |d| j ¬« } t | j « r@t | j d| j ¬« 5 } ||j |¬« }d d d « n || j |¬« } j « }t d
|
« s |d ddd« |j# d « D ] } d | j$ xs dz | _ |S # 1 sw Y axY w# t $ r Y ^w xY w# t t f$ rV}t | j « s5 || j |¬« } |j « }n# t $ r Y nw xY w|Y d }~«d }~ww xY w)a
|
Raises
|
------
|
ValueError
|
* If a URL that lxml cannot parse is passed.
|
|
Exception
|
* Any other ``Exception`` thrown. For example, trying to parse a
|
URL that is syntactically correct on a machine with no internet
|
connection will fail.
|
|
See Also
|
--------
|
pandas.io.html._HtmlFrameParser._build_doc
|
r )ÚXMLSyntaxError)Ú
|
HTMLParserÚ
|
fromstringÚparseT)ÚrecoverrG rF ©rC )ÚparserNr z no text parsed from documentz*//brrì r« )Ú
|
lxml.etreer Ú lxml.htmlr r r rG r rU r rC rK ÚgetrootÚAttributeErrorrI ÚUnicodeDecodeErrorÚOSErrorr Útail)
|
rZ r r r r r ÚfrF Úerë s
|
r/ rc z_LxmlFrameParser._build_docö sY õ .÷
|
ñ
|
ñ D°4·=±=ÔAð NÜdggÜ ØGGS°$×2FÑ2Fôð 7àÙ ahh¨vÔ6A÷7ð 7ñ $''¨&Ô1ð ØIIKô 1nÔ-Ù$Ð%CÀQÈÈ1ÓMÐMà'''Ó"ò -BØbggm¨Ñ,BGð -ð ÷;7ð 7ûô "ò Ùð ûä"¤GÐ,ò
|
ä$''?Ù tww¨vÔ6ðØ AøÜ%ò Ùðúð ÿøð
|
úsq ¦7D Á C'Á2 D ÂC3 Ã'C0Ã,D Ã3 C?Ã<D Ã>C?Ã?D ÄE'Ä)E"Ä;E Å E"Å EÅE"ÅEÅE"Å"E'c óÀ g }|j d« D ]G }|j |j d« « |j d« }|s7|j |« I |S )Nz.//theadú./trr )r Úextendr )rZ r` r ÚtheadÚelements_at_roots r/ ru z _LxmlFrameParser._parse_thead_tr0 s[ Øà[[ Ó,ò #EØ KK FÓ+Ô ,ð %{{¨;Ó 7Ð Ú Ø EÕ"ð #ð r1 c óP |j d« }|j d« }||z S )Nz .//tbody//trr" r rß s r/ rx z _LxmlFrameParser._parse_tbody_trB s) Ø[[ Ó0
|
ØKK Ó' àIÑ%Ð%r1 c ó$ |j d« S )Nz .//tfoot//trr rt s r/ r{ z _LxmlFrameParser._parse_tfoot_trH s Ø{{>Ó*Ð*r1 Nr¸ rº ) r@ r» r¼ r½ rm rp rr rb r rc ru rx r{ r¾ r1 r/ r r · s5 ñó$-ò"ò&ò
|
ó: ò8òtò$&ó +r1 r c óà | D cg c] }t |« }}t |« }|j « }|||k7 }dg}|j « D ] \ }}| |xx |||z
|
z z cc< y c c}w )Nr« )Úlenr Úmaxrû ) r rÐ ÚdataÚlensÚlens_maxÚnot_maxÚemptyÚindÚlengths r/ Ú_expand_elementsr2 L sy Ø"&Ö '$CIÐ 'DÐ 'Ü $<DØxxzHØ48Ñ#Ñ$Gà DEØ }}ò 1 VØ S Uh¨Ñ /Ñ0Ñ0 ñ1ùò (s
A+c ó | j d« \ }}}| j d« }t | d « | d< |rH||z }|At |« dk( rd}n0t |« D cg c] \ }}t d |D « « s| }}}|r||z }t |« t |fd|i| ¤5 }|j « cd d d « S c c}}w # 1 sw Y y xY w)Nr+ r rA r3 r c 3 ó K | ] }| y wrT r¾ )r_ rO s r/ ra z!_data_to_frame.<locals>.<genexpr>d s è ø Ò@VÈ$ÄÑ@Vùs )r rB r) Ú enumerateÚanyr2 r rE )r Úheadr Úfootr Úir Útps r/ Ú_data_to_framer; W sÚ Ø zz &Ó)ÑD$Ø ZZÓ !FÜ& v¨jÑ'9Ó:F:ÑÙ Ød{ð >Ü4yA~Øô +4°D«/×W¡ 3¼SÑ@VÐRUÔ@VÕ=V!ÐWÑWá Ø ô TÔÜ DÑ 2 Ð 2¨6Ñ 2ð °bØwwy÷ñ ùó X÷ð ús Á B<Á;B<Â"CÃC )ÚlxmlNrè rí c óä t t j « « }| |vrt t | « d| « | dv r t d« t d« t | S t d« t | S )aÜ
|
Choose the parser based on the input flavor.
|
|
Parameters
|
----------
|
flavor : {{"lxml", "html5lib", "bs4"}} or None
|
The type of parser to use. This must be a valid backend.
|
|
Returns
|
-------
|
cls : _HtmlFrameParser subclass
|
The parser class based on the requested input flavor.
|
|
Raises
|
------
|
ValueError
|
* If `flavor` is not a valid backend.
|
ImportError
|
* If you do not have the requested `flavor`
|
z* is not a valid flavor, valid flavors are )rí rè rè rí z
|
lxml.etree)r9 Ú_valid_parsersÚkeysrÄ rË r
|
)ÚflavorÚ valid_parserss r/ Ú_parser_dispatchrB w s| ô* ×,Ñ,Ó.Ó/MØ ]Ñ"ÜÜF|nÐ FÀ}ÀoÐ Vó
|
ð
|
ð Ð$Ñ$Ü" :Ô.Ü" 5Ô)ô &Ñ !Ð!ô # <Ô0Ü &Ñ !Ð!r1 c óh dj | D cg c] }t |« c}« }d| dS c c}w )Nz, ú{ú})rú r )r- ÚelÚargs r/ Ú _print_as_setrH s3 Ø
|
))°Ö2¨"\ "Õ%Ò2Ó
|
3CØ uB<Ðùò 3s /c óê | d} n¡t | t « r| f} nt | t j « r=t d | D « « sat dt t | « j « d« t | t « r t | « n
|
t | « }|dz }t |« t | « } t t « }t | « }||z s"t t |« dt |« « | S )N)r< rí c 3 ó< K | ] }t |t « y wrT )r5 rJ )r_ Úflavs r/ ra z#_validate_flavor.<locals>.<genexpr>¥ s è ø Ò<¨T:d¤C×(Ñ<ùs zObject of type z is not an iterable of stringsz is not a valid flavorz2 is not a valid set of flavors, valid flavors are )r5 rJ r r r r> rË r? r@ rÄ ÚtuplerÅ r> rH )r@ ÚmsgÚ valid_flavorsÚ
|
flavor_sets r/ Ú_validate_flavorrP sî Ø ~Ø Ü FCÔ ØÜ FCLLÔ )ÜÑ<°VÔ<Ô<ÜØ!¤$¤t¨F£|×'<Ñ'<Ó"=Ð!>ð ?0ð 1óð ô
|
)¨´Ô 5d6l¼3¸v»;Ø Ð'Ñ'ÜoÐ ä 6]FÜÓ'MÜVJà Ò %ÜÜ ZÓ(Ð)ð *Ü(¨ Ó7Ð8ð :ó
|
ð
|
ð Mr1 c
|
ó t | « } t j |« } d }
|
| D ]- } t | « } | || |||||« } | j « } n |
|
J |
|
g }|D ]b } t d
|
d|i|¤}|dv r=t |j t « r#t d |j D « d¬ « |_ |j |« d |S # t
|
$ rg}t |d« r"|j « r|j d« n,t |d« r |j « st d| d« ||}
|
Y d }~ d }~ww xY w# t $ r Y åw xY w) NÚseekabler z The flavor z failed to parse your input. Since you passed a non-rewindable file object, we can't rewind it to try another parser. Try read_html() with a different flavor.r+ )r r c 3 óP K | ] }|d t |d « rdn|d f yw)r r3 Nr )r_ Úcols r/ ra z_parse.<locals>.<genexpr>ø s) è ø ÒVÀCc!f¤d¨3¨q©6¤ld¸¸A¹Ô?ÑVùs $&F)Ú tupleize_colsr¾ )rP rÇ rÈ rB re rÄ rI rR Úseekr; r5 Úcolumnsr r r r )r@ rU rV rW rG rX rY rC r Úcompiled_matchÚretainedrK r Úprd ÚcaughtÚretr` Údfs r/ Ú_parser^ » s~ ô fÓ %FÜZZ Ó&NàHØò !Ü! $Ó'Ù Ø Ø Ø Ø Ø Ø Ø ó
|
ð Ø^^Ó%Fñ" ð=!ð@ Ð#Ð#Ð#Øà
|
CØò ð Ü Ñ5 UÐ5¨fÑ5Bð Ð 1Ñ1´jØ
|
|
Jô7ô #ÙVÈ2Ï:É:ÔVØ"'ô
|
ð
|
JJrNð ð" JøôO ò ô r:Ô&¨2¯;©;¬=Ø
|
ÜZÔ(°·±´ ä Ø! $ ð (Oð Oóð
|
ð ð Hûð ûôJ ò Ù ð ús, ÁCÁ#A D9Ã D6ÃA D1Ä1D6Ä9 EÅEr z.+Fú,ú.T)rV r@ r Ú index_colrA rW Ú parse_datesÚ thousandsrG ÚdecimalÚ
|
convertersÚ na_valuesÚkeep_default_narX rY Ú dtype_backendrC c ó
|
t |t j « r|dk r t d« |dvrt d| d« t |« t |« t | « } t | t « rWt t | « t | « t | « t | « g« s$t j dt t! « ¬« t# di d|d | d
|
|d |d |d |d|d|d|d| d|
|
d| d| d| d|d|d|d|S )a´
|
Read HTML tables into a ``list`` of ``DataFrame`` objects.
|
|
Parameters
|
----------
|
io : str, path object, or file-like object
|
String, path object (implementing ``os.PathLike[str]``), or file-like
|
object implementing a string ``read()`` function.
|
The string can represent a URL or the HTML itself. Note that
|
lxml only accepts the http, ftp and file url protocols. If you have a
|
URL that starts with ``'https'`` you might try removing the ``'s'``.
|
|
.. deprecated:: 2.1.0
|
Passing html literal strings is deprecated.
|
Wrap literal string/bytes input in ``io.StringIO``/``io.BytesIO`` instead.
|
|
match : str or compiled regular expression, optional
|
The set of tables containing text matching this regex or string will be
|
returned. Unless the HTML is extremely simple you will probably need to
|
pass a non-empty string here. Defaults to '.+' (match any non-empty
|
string). The default value will return all tables contained on a page.
|
This value is converted to a regular expression so that there is
|
consistent behavior between Beautiful Soup and lxml.
|
|
flavor : {{"lxml", "html5lib", "bs4"}} or list-like, optional
|
The parsing engine (or list of parsing engines) to use. 'bs4' and
|
'html5lib' are synonymous with each other, they are both there for
|
backwards compatibility. The default of ``None`` tries to use ``lxml``
|
to parse and if that fails it falls back on ``bs4`` + ``html5lib``.
|
|
header : int or list-like, optional
|
The row (or list of rows for a :class:`~pandas.MultiIndex`) to use to
|
make the columns headers.
|
|
index_col : int or list-like, optional
|
The column (or list of columns) to use to create the index.
|
|
skiprows : int, list-like or slice, optional
|
Number of rows to skip after parsing the column integer. 0-based. If a
|
sequence of integers or a slice is given, will skip the rows indexed by
|
that sequence. Note that a single element sequence means 'skip the nth
|
row' whereas an integer means 'skip n rows'.
|
|
attrs : dict, optional
|
This is a dictionary of attributes that you can pass to use to identify
|
the table in the HTML. These are not checked for validity before being
|
passed to lxml or Beautiful Soup. However, these attributes must be
|
valid HTML table attributes to work correctly. For example, ::
|
|
attrs = {{'id': 'table'}}
|
|
is a valid attribute dictionary because the 'id' HTML tag attribute is
|
a valid HTML attribute for *any* HTML tag as per `this document
|
<https://html.spec.whatwg.org/multipage/dom.html#global-attributes>`__. ::
|
|
attrs = {{'asdf': 'table'}}
|
|
is *not* a valid attribute dictionary because 'asdf' is not a valid
|
HTML attribute even if it is a valid XML attribute. Valid HTML 4.01
|
table attributes can be found `here
|
<http://www.w3.org/TR/REC-html40/struct/tables.html#h-11.2>`__. A
|
working draft of the HTML 5 spec can be found `here
|
<https://html.spec.whatwg.org/multipage/tables.html>`__. It contains the
|
latest information on table attributes for the modern web.
|
|
parse_dates : bool, optional
|
See :func:`~read_csv` for more details.
|
|
thousands : str, optional
|
Separator to use to parse thousands. Defaults to ``','``.
|
|
encoding : str, optional
|
The encoding used to decode the web page. Defaults to ``None``.``None``
|
preserves the previous encoding behavior, which depends on the
|
underlying parser library (e.g., the parser library will try to use
|
the encoding provided by the document).
|
|
decimal : str, default '.'
|
Character to recognize as decimal point (e.g. use ',' for European
|
data).
|
|
converters : dict, default None
|
Dict of functions for converting values in certain columns. Keys can
|
either be integers or column labels, values are functions that take one
|
input argument, the cell (not column) content, and return the
|
transformed content.
|
|
na_values : iterable, default None
|
Custom NA values.
|
|
keep_default_na : bool, default True
|
If na_values are specified and keep_default_na is False the default NaN
|
values are overridden, otherwise they're appended to.
|
|
displayed_only : bool, default True
|
Whether elements with "display: none" should be parsed.
|
|
extract_links : {{None, "all", "header", "body", "footer"}}
|
Table elements in the specified section(s) with <a> tags will have their
|
href extracted.
|
|
.. versionadded:: 1.5.0
|
|
dtype_backend : {{'numpy_nullable', 'pyarrow'}}, default 'numpy_nullable'
|
Back-end data type applied to the resultant :class:`DataFrame`
|
(still experimental). Behaviour is as follows:
|
|
* ``"numpy_nullable"``: returns nullable-dtype-backed :class:`DataFrame`
|
(default).
|
* ``"pyarrow"``: returns pyarrow-backed nullable :class:`ArrowDtype`
|
DataFrame.
|
|
.. versionadded:: 2.0
|
|
{storage_options}
|
|
.. versionadded:: 2.1.0
|
|
Returns
|
-------
|
dfs
|
A list of DataFrames.
|
|
See Also
|
--------
|
read_csv : Read a comma-separated values (csv) file into DataFrame.
|
|
Notes
|
-----
|
Before using this function you should read the :ref:`gotchas about the
|
HTML parsing libraries <io.html.gotchas>`.
|
|
Expect to do some cleanup after you call this function. For example, you
|
might need to manually assign column names if the column names are
|
converted to NaN when you pass the `header=0` argument. We try to assume as
|
little as possible about the structure of the table and push the
|
idiosyncrasies of the HTML contained in the table to the user.
|
|
This function searches for ``<table>`` elements and only for ``<tr>``
|
and ``<th>`` rows and ``<td>`` elements within each ``<tr>`` or ``<th>``
|
element in the table. ``<td>`` stands for "table data". This function
|
attempts to properly handle ``colspan`` and ``rowspan`` attributes.
|
If the function has a ``<thead>`` argument, it is used to construct
|
the header, otherwise the function attempts to find the header within
|
the body (by putting rows with only ``<th>`` elements into the header).
|
|
Similar to :func:`~read_csv` the `header` argument is applied
|
**after** `skiprows` is applied.
|
|
This function will *always* return a list of :class:`DataFrame` *or*
|
it will fail, e.g., it will *not* return an empty list.
|
|
Examples
|
--------
|
See the :ref:`read_html documentation in the IO section of the docs
|
<io.read_html>` for some examples of reading in HTML tables.
|
r zPcannot skip rows starting from the end of the data (you passed a negative value))Nr r r r zO`extract_links` must be one of {None, "header", "footer", "body", "all"}, got "ú"zPassing literal html to 'read_html' is deprecated and will be removed in a future version. To read from a literal string, wrap it in a 'StringIO' object.)Ú
|
stacklevelr@ rU rV r ra rA rb rc rW rG rd re rf rg rX rY rh rC r¾ )r5 r< r= rÄ r r r rJ r6 r r r r ÚwarningsÚwarnÚ FutureWarningr r^ )rU rV r@ r ra rA rW rb rc rG rd re rf rg rX rY rh rC s r/ Ú read_htmlro s ôj (G× ,Ñ ,Ô-°(¸Q²,Üð 1ó
|
ð
|
ð Ð EÑEÜð à að !ó
|
ð
|
ô Ô Ü Ô&ä Ó Bä"cÔ¤3ä Ó Ü OÜ 2JÜ "Ó ð
|
ô$ô ð >ô Ü'Ó)õ
|
ô ò Ùð á ð ñ ð ñ ð ñ
|
ð ñ ð ñ ð ñ ð ñ ð ñ ð ñ ð ñ ð ñ ð ñ (ð ñ &ð ñ $ð! ñ" $ð# ñ$ (ð% ð r1 )r- rJ r. r r¶ rJ )rA ú"int | Sequence[int] | slice | Noner¶ r4 )rM zFilePath | BaseBufferrG r¹ rC zStorageOptions | Noner¶ z str | bytes)r¶ rJ )r¶ r· )r@ zHTMLFlavors | Noner¶ ztype[_HtmlFrameParser])&rU zFilePath | ReadBuffer[str]rV r² r@ z*HTMLFlavors | Sequence[HTMLFlavors] | Noner úint | Sequence[int] | Nonera rq rA rp rW r³ rb r´ rc r¹ rG r¹ rd rJ re z dict | Nonerf zIterable[object] | Nonerg r´ rX r´ rY rµ rh z DtypeBackend | lib.NoDefaultrC r' r¶ zlist[DataFrame])Pr½ Ú
|
__future__r Ú collectionsr r< rÇ r Útypingr r r rl Ú pandas._libsr Úpandas.compat._optionalr
|
Ú pandas.errorsr r Úpandas.util._decoratorsr Úpandas.util._exceptionsr Úpandas.util._validatorsr Úpandas.core.dtypes.commonr Úpandasr Úpandas.core.indexes.baser Úpandas.core.indexes.multir Úpandas.core.seriesr Úpandas.core.shared_docsr Úpandas.io.commonr r r r r r r Úpandas.io.formats.printingr Úpandas.io.parsersr Úcollections.abcr r! Úpandas._typingr" r# r$ r% r& r' r( rÈ Ú_RE_WHITESPACEr0 rB rP rR rÀ rþ r r r2 r; r> rB rH rP r^ Ú
|
no_defaultro r¾ r1 r/ ú<module>r sl ðñõ #å Û Û Ý ÷ñ ó
|
å Ý >÷õ (Ý 4Ý 7å 2å Ý *Ý 0Ý %Ý 0÷÷ ñ õ 4Ý (á÷÷
|
÷ õ !ð
|
Ð -Ó.ð 1?ô %ó&Xð: Ø ð àð ð +ð ð ó ÷Bf
|
ñ f
|
ôR WÐ(8ô Wótð. Ð=Ð> ôR+Ð'ô R+ój1òð2 Ø
|
Ø1Ø ,ñ ó "óFò
|
ò8DñN \Ð"3Ñ4Ô5ð Ø9=Ø)-Ø,0Ø37Ø#'Ø Ø Ø ØØ "Ø)-Ø Ø ØFJØ25·.±.Ø&*ñ'hØ"ðhð ðhð 7ð hð
|
'ð hð *ð hð 1ðhð !ðhð ðhð ðhð ðhð ðhð ðhð 'ð hð ð hð ð!hð" Dð#hð$ 0ð%hð& $ð'hð( ò)hó 6ñhr1
|
