Ë
|
oñúhÛ ã óN d Z ddlmZ ddlmZ ddlmZ ddlZddl Z ddl
|
Z ddl
|
m Z ddl mZ ddlmZ e j$ j' d« Ze j$ j* e j$ j, gZe j$ j1 d « d
|
« Zd Zed « Zy) zd
|
Tests multithreading behaviour for reading and
|
parsing files for each parser defined in parsers.py
|
é )Ú ExitStack)ÚBytesIO)Ú
|
ThreadPoolN)Ú DataFrame)ÚVersionÚ pyarrow_xfailz0ignore:Passing a BlockManager:DeprecationWarningc ój | }|j dk( ret j d« }t |j « t d« k r/|j t j j d¬« « d d} fdt |« D « }t « 5 }|D cg c] }|j t |« « }}|j t d« « } | j |j |« }
|
|
|
d } |
|
D ] } t j | | « d d d « y c c}w # 1 sw Y y xY w)
|
NÚpyarrowz16.0z+# ValueError: Found non-unique column index)Úreasonéd é
|
c 3 óª K | ]E }d j t « D cg c] }|dd|dd|d c}« j « G yc c}w w)ú
|
Údú,N)ÚjoinÚrangeÚencode)Ú.0Ú_ÚiÚ max_row_ranges ú[H:\Change_password\venv_build\Lib\site-packages\pandas/tests/io/parser/test_multi_thread.pyú <genexpr>z7test_multi_thread_string_io_read_csv.<locals>.<genexpr>( sU øè ø ò à ð ´°}Ó1EÖF¨AaU!Aa5 ! A Ò'ÒFÓG×NÑN×PñùÚFùs A A
|
´ Aé r )ÚengineÚpytestÚ importorskipr Ú __version__Ú applymarkerÚmarkÚxfailr r Ú enter_contextr r ÚmapÚread_csvÚtmÚassert_frame_equal)Ú all_parsersÚrequestÚparserÚpaÚ num_filesÚ bytes_to_dfÚstackÚbÚfilesÚpoolÚresultsÚ first_resultÚresultr s @r Ú$test_multi_thread_string_io_read_csvr5 s ø ð FØ }} Ò!Ü × Ñ Ó +Ü 2>>Ó "¤W¨V£_Ò 4Ø × Ñ Ü ×!Ñ!Ð)VÐ!ÓWô ð MØIóäyÓ!ôKô
|
ð 8Ø:EÖF°Q×$Ñ$¤W¨Q£ZÕ0ÐFÐFà×"Ñ"¤:¨a£=Ó1à((6??¨EÓ2Ø qz à ò 8FÜ × !Ñ ! ,°Õ 7ñ 8÷ 8ð 8ùÚF÷ 8ð 8ús ÂD) !D$à AD)Ä$D)Ä)D2c ó6 fd}t |« D cg c] }||z |z ||z f }}t |¬« 5 }|j ||« }ddd« d j } |dd D ] }
|
| |
|
_ t j
|
|« } | S c c}w # 1 sw Y ExY w)a
|
Generate a DataFrame via multi-thread.
|
|
Parameters
|
----------
|
parser : BaseParser
|
The parser object to use for reading the data.
|
path : str
|
The location of the CSV file to read.
|
num_rows : int
|
The number of rows to read per task.
|
num_tasks : int
|
The number of tasks to use for reading this DataFrame.
|
|
Returns
|
-------
|
df : DataFrame
|
c ó | \ }}|sj dd|dg¬« S j ddt |« dz |dg¬« S )aj
|
Create a reader for part of the CSV.
|
|
Parameters
|
----------
|
arg : tuple
|
A tuple of the following:
|
|
* start : int
|
The starting row to start for parsing CSV
|
* nrows : int
|
The number of rows to read.
|
|
Returns
|
-------
|
df : DataFrame
|
r Údate)Ú index_colÚheaderÚnrowsÚ parse_datesNé é )r9 r: Úskiprowsr; r< )r% Úint)ÚargÚstartr; r* Úpaths r Úreaderz0_generate_multi_thread_dataframe.<locals>.readerN sh ø ð$ uáØ??Ø ¨!°5ÀvÀhð #ó ð ð Ø ØØÜZ !^ØØð ó
|
ð
|
ó )Ú processesNr r= )r r r$ ÚcolumnsÚpdÚconcat) r* rC Únum_rowsÚ num_tasksrD r Útasksr1 r2 r: ÚrÚfinal_dataframes `` r Ú _generate_multi_thread_dataframerO : s¸ ù õ(
|
ôF EJÈ)ÓDTö Ø?@A Ñ " H° Ñ$9Ò:ð Eð ô
|
iÔ (ð *¨DØ((6 5Ó)÷*ð QZ× Ñ Fà QR[ò Ø ðô ii Ó(OØ Ðùò ÷*ð *ús B
|
¸BÂBc ó$ d}d}| }d}t t j j d« j |« t j j d« j |« t j j d« j |« t j j d« j |« t j j d« j |« dg|z dg|z dg|z t j
|
d|d ¬
|
« t j |d ¬ « d
|
« }t j |« 5 }|j |« t ||||« }t j ||« d d d « y # 1 sw Y y xY w)Né é0 z__thread_pool_reader__.csvé ÚfooÚbarÚbazz20000101 09:00:00Ús)ÚperiodsÚfreqÚint64)Údtype)
|
Úar/ Úcr ÚerT rU rV r8 r@ ) r ÚnpÚrandomÚ default_rngrH Ú
|
date_rangeÚaranger& Ú ensure_cleanÚto_csvrO r' )r( rK rJ r* Ú file_nameÚdfrC rN s r Ú)test_multi_thread_path_multipart_read_csvrh sJ ð IØHà FØ,IÜ ä×&Ñ& qÓ)×0Ñ0°Ó:Ü×&Ñ& qÓ)×0Ñ0°Ó:Ü×&Ñ& qÓ)×0Ñ0°Ó:Ü×&Ñ& qÓ)×0Ñ0°Ó:Ü×&Ñ& qÓ)×0Ñ0°Ó:Ø7XÑ%Ø7XÑ%Ø7XÑ%ÜMMÐ"5¸xÈcÔRÜ99X¨WÔ5ñ
|
ó
|
Bô
|
Ó #ð 3 tØ
|
$ä:Ø D( Ió
|
ô × Ñ b /Ô2÷ 3÷ 3ñ 3ús Å6FÆF) Ú__doc__Ú
|
contextlibr Úior Úmultiprocessing.poolr Únumpyr_ r ÚpandasrH r Úpandas._testingÚ_testingr& Úpandas.util.versionr r! Ú usefixturesÚ xfail_pyarrowÚ
|
single_cpuÚslowÚ
|
pytestmarkÚfilterwarningsr5 rO rh © rE r ú<module>ry s ðñõ !Ý Ý +ã Û ã Ý Ý Ý 'à ×'Ñ'¨Ó8 ð
|
KK×ÑØ
|
KK×Ñð
|
ð ×ÑÐ NÓOñ8ó Pð8ò<CðL ñ 3ó ñ 3rE
|
