Ë
|
nñúh ; ã óH d Z ddlmZ ddlmZ ddlmZ ddlZddlmZ ddl Z ddl
|
Z ddl m Z ddlmZ ddlZdd lmZ dd
|
lmZ er dd lmZmZmZmZ d Zd Z dZ dZ g d¢Z dZ dZ!dZ"dZ#de de" de! de# d Z$de de! dZ%dZ&d!dZ'd"d Z(d Z)d Z* G d d eejV « Z,y)#a-
|
Read a SAS XPort format file into a Pandas DataFrame.
|
|
Based on code from Jack Cushman (github.com/jcushman/xport).
|
|
The file format is defined here:
|
|
https://support.sas.com/content/dam/SAS/support/en/technical-papers/record-layout-of-a-sas-version-5-or-6-data-set-in-sas-transport-xport-format.pdf
|
é )Ú annotations)Úabc)ÚdatetimeN)Ú TYPE_CHECKING)ÚAppender)Úfind_stack_level)Ú
|
get_handle)Ú
|
ReaderBase)ÚCompressionOptionsÚDatetimeNaTTypeÚFilePathÚ
|
ReadBufferzPHEADER RECORD*******LIBRARY HEADER RECORD!!!!!!!000000000000000000000000000000 zKHEADER RECORD*******MEMBER HEADER RECORD!!!!!!!000000000000000001600000000zPHEADER RECORD*******DSCRPTR HEADER RECORD!!!!!!!000000000000000000000000000000 zPHEADER RECORD*******OBS HEADER RECORD!!!!!!!000000000000000000000000000000 )ÚntypeÚnhfunÚ field_lengthÚnvar0ÚnameÚlabelÚnformÚnflÚ num_decimalsÚnfjÚnfillÚniformÚniflÚnifdÚnposÚ_zParameters
|
----------
|
filepath_or_buffer : str or file-like object
|
Path to SAS file or object implementing binary read method.zâindex : identifier of index column
|
Identifier of column that should be used as index of the DataFrame.
|
encoding : str
|
Encoding for text data.
|
chunksize : int
|
Read file `chunksize` lines at a time, returns iterator.zBformat : str
|
File format, only `xport` is currently supported.z\iterator : bool, default False
|
Return XportReader object for reading file incrementally.z#Read a SAS file into a DataFrame.
|
|
ú
|
a
|
|
Returns
|
-------
|
DataFrame or XportReader
|
|
Examples
|
--------
|
Read a SAS Xport file:
|
|
>>> df = pd.read_sas('filename.XPT')
|
|
Read a Xport file in 10,000 line chunks:
|
|
>>> itr = pd.read_sas('filename.XPT', chunksize=10000)
|
>>> for chunk in itr:
|
>>> do_something(chunk)
|
|
z$Class for reading SAS Xport files.
|
|
z
|
|
Attributes
|
----------
|
member_info : list
|
Contains information about the file
|
fields : list
|
Contains information about the variables in the file
|
zÊRead observations from SAS Xport file, returning as data frame.
|
|
Parameters
|
----------
|
nrows : int
|
Number of rows to read from data file; if None, read whole
|
file.
|
|
Returns
|
-------
|
A DataFrame.
|
c ón t j | d« S # t $ r t j cY S w xY w)z1Given a date in xport format, return Python date.z%d%b%y:%H:%M:%S)r ÚstrptimeÚ
|
ValueErrorÚpdÚNaT)Údatestrs úJH:\Change_password\venv_build\Lib\site-packages\pandas/io/sas/sas_xport.pyÚ _parse_dater' s3 ðä× Ñ Ð*;Ó<Ð<øÜ ò Üvv ðús 4³4c ód i }d}|D ]# \ }}| |||z j « ||< ||z }% |d= |S )a
|
Parameters
|
----------
|
s: str
|
Fixed-length string to split
|
parts: list of (name, length) pairs
|
Used to break up string, name '_' will be filtered from output.
|
|
Returns
|
-------
|
Dict of name:contents of string at given location.
|
r r )Ústrip)ÚsÚpartsÚoutÚstartr Úlengths r& Ú _split_liner/ sU ð CØ EØ ò fØee fnÐ-×3Ñ3Ó5D Ø ðð CØ Jó c óâ |dk7 rit j t | « t j d« « }t j d| dd|z
|
« }|j |¬« }| |d< |S | S )Né ÚS8ÚSz,S©ÚdtypeÚf0)ÚnpÚzerosÚlenr6 Úview)ÚvecÚnbytesÚvec1r6 Úvec2s r& Ú_handle_truncated_float_vecr@ ² si ð {ÜxxC¤"§(¡(¨4£.Ó1Ü1VH B q¨6¡z lÐ3Ó4ØyyuyÓ%ØT
|
Ø à Jr0 c ó t j d« }| j |¬« }|d }|d }|dz }t j t | « t j
|
¬« }d|t j |dz « < d|t j |d z « < d
|
|t j |d z « < ||z }||z |d z d d
|
|z
|
z z z }|dz }||dz dz dz
|
dz |z dz dz |dz z z }t j t |« fd¬« }||d< ||d< |j d¬« }|j d« }|S )zf
|
Parse a vector of float values representing IBM 8 byte floats into
|
native 8 byte floats.
|
z>u4,>u4r5 r7 Úf1iÿÿÿ é i é i @ é i é é l ÿß é é éA iÿ é l z>f8Úf8) r8 r6 r; r9 r: Úuint8ÚwhereÚemptyÚastype) r< r6 r> Úxport1Úxport2Úieee1ÚshiftÚieee2Úieees r& Ú_parse_float_vecrW Å sd ô
|
HHYÓ EØ 88%8Ó DØ $ZFØ $ZFð ZÑ Eô HHSX¤R§X¡XÔ .EØ+,E"((6JÑ&Ó
|
'Ñ(Ø+,E"((6JÑ&Ó
|
'Ñ(Ø+,E"((6JÑ&Ó
|
'Ñ(ð
|
eOEØ u_ &¨:Ñ"5¸2ÀÀUÁÑ;KÑ!LÑ MEð
|
ZÑEð
|
6R< 4Ñ'¨2Ñ-°!Ñ3°uÑ<¸tÑCÈÑJØÑññ Eô 88SZM¨Ô 3DØDJØDJØ 9959Ó !DØ ;;tÓ Dà Kr0 c ó e Zd ZeZ d d dZd dZd Zd dZddZ ddZ
|
dddZ d Z e e« ddd
|
« Zy)Ú XportReaderNc óú || _ d| _ || _ || _ t |d|d|¬« | _ | j
|
j | _ | j « y # t $ r | j « w xY w)Nr ÚrbF)ÚencodingÚis_textÚ compression) Ú _encodingÚ _lines_readÚ_indexÚ
|
_chunksizer ÚhandlesÚhandleÚfilepath_or_bufferÚ _read_headerÚ ExceptionÚclose)Úselfre Úindexr\ Ú chunksizer^ s r& Ú__init__zXportReader.__init__ s| ð "Ø ÔØ Ø#ä!Ø Ø Ø ØØ#ô
|
ð #'§,¡,×"5Ñ"5Ô ð Ø × Ñ Õ øÜò Ø JJLØ ð ús ÁA Á A:c ó8 | j j « y ©N)rc rh ©ri s r& rh zXportReader.close s Ø ×ÑÕ r0 c óT | j j d« j « S )NéP )re ÚreadÚdecodero s r& Ú_get_rowzXportReader._get_row s" Ø×&Ñ&×+Ñ+¨BÓ/×6Ñ6Ó8Ð8r0 c
|
óä | j j d« | j « }|t k7 rd|v r t d« t d« | j « }ddgddgd dgd
|
dgd d gg}t ||« }|d d k7 r t d« t |d « |d < || _ | j « }t |d d « |d< | j « }| j « }|j t « }|t k( } |r| s t d« t |dd « }
|
ddgddgddgddgd dgd
|
dgd d gg} t | j « | « } dd gd
|
d gddgddgg} | j t | j « | « « t | d « | d< t | d « | d < | | _ ddd} t | j « dd « }|
|
|z }|d z r |d |d z z
|
z }| j j |« }g }d}t |« |
|
k\ rÉ|d |
|
||
|
d }}|j! d « }t# j$ d |« }t' t) t* |« « }|d
|
= | |d |d < |d! }|d dk( r|d"k s|dkD rd#| d$}t- |« |j/ « D ] \ }} |j1 « ||< ||d! z }||gz }t |« |
|
k\ rÉ| j « }|t4 k( s t d%« || _ || _ | j j; « | _ | j? « | _ | j6 D cg c] }|d& jC « c}| _" tG | j6 « D cg c]$ \ } }d'tI | « z d(tI |d! « z f& } } }tK jL | « } | | _' y # t2 $ r Y <w xY wc c}w c c}} w ))Nr z**COMPRESSED**z<Header record indicates a CPORT file, which is not readable.z#Header record is not an XPORT file.ÚprefixrH Úversionr2 ÚOSr Úcreatedé zSAS SAS SASLIBz!Header record has invalid prefix.ÚmodifiedzMember header not foundéûÿÿÿéþÿÿÿÚset_nameÚsasdatar é( ÚtypeÚnumericÚchar)rC rD é6 é: rq é z>hhhh8s40s8shhh2s8shhl52sr r rD zFloating field width z is not between 2 and 8.z Observation header not found.r r* r4 )(re Úseekrt Ú_correct_line1r" r/ r' Ú file_infoÚ
|
startswithÚ_correct_header1Ú_correct_header2ÚintÚupdateÚ member_inforr r: ÚljustÚstructÚunpackÚdictÚzipÚ
|
_fieldkeysÚ TypeErrorÚitemsr) ÚAttributeErrorÚ_correct_obs_headerÚfieldsÚ record_lengthÚtellÚ record_startÚ _record_countÚnobsrs ÚcolumnsÚ enumerateÚstrr8 r6 Ú_dtype) ri Úline1Úline2Úfifr Úline3Úheader1Úheader2Ú headflag1Ú headflag2ÚfieldnamelengthÚmemr ÚtypesÚ
|
fieldcountÚ
|
datalengthÚ fielddatar Ú
|
obs_lengthÚ
|
fieldbytesÚ fieldstructÚfieldÚflÚmsgÚkÚvÚheaderÚxÚiÚdtypelr6 s r& rf zXportReader._read_header# s[ Ø × Ñ ×$Ñ$ QÔ'ð Ø NÒ "Ø 5Ñ(ô !ØRóð ô Ð BÓCÐ Cà Ø"~ ¨1~°°a¨y¸3À¸)ÀiÐQSÀ_ÐUÜ sÓ+ Ø XÑ Ð":Ò :Ü Ð @ÓAÐ AÜ *¨9°YÑ+?Ó @ )Ñ Ø"à Ü +¨E°#°2¨JÓ 7 *Ñ ð --/Ø--/Ø×&Ñ&Ô'7Ó8 ØÔ /Ñ/ ÙiÜ Ð 6Ó7Ð 7ä g b¨nÓ-ð qMØ OØ NØ NØ 1IØ "IØ Oð
|
ô " $§-¡-£/°3Ó7 ØBÐ # r ¨W°b¨M¸FÀA¸;ÐGØ×Ñ; t§}¡}£¸Ó<Ô=Ü"-¨k¸*Ñ.EÓ"F JÑ Ü!,¨[¸Ñ-CÓ!D IÑ Ø&Ôð &Ñ)Ü ¨¨BÐ/Ó0
|
Ø$ zÑ1
|
à ?Ø "z¨BÑ.Ñ .JØ×+Ñ+×0Ñ0°Ó< ØØ
|
Ü)n Ò/ð Ð*?Ð+Ø/Ð*Ð+ð "Jð $×)Ñ)¨#Ó.Jä --Ð(CÀZÓPKÜZ¨Ó5Ó6EØc
|
Ø " 5¨¡>Ñ 2E'NØ~Ñ&BØW~ Ò*°°a²¸RÀ!ºVØ-¨b¨TÐ1IÐJÜ nÐ$à ò 1ðØ wwyE!Hð ð % Ñ/Ñ /JØ ugÑ Fô7 )n Ó/ð: ØÔ,Ò,Ü Ð <Ó=Ð =à Ø 'ÔØ × 3Ñ 3× 8Ñ 8Ó :Ôà×&Ñ&Ó( Ø48·K±KÖ@¨q& ×(Ñ(Õ*Ò@ ô
|
& d§k¡kÓ 2÷
|
á5ð 3q6\3¤ U¨>Ñ%:Ó!;Ñ;Ò <ð
|
ñ
|
ô Ó Ø øô/ &ò Úðüò Aùó
|
s Ê.OÍO'Î)O,Ï O$Ï#O$c óB | j | j xs d¬« S )NrC ©Únrows)rr rb ro s r& Ú__next__zXportReader.__next__ s ØyytÒ 3°!yÓ4Ð4r0 c óð | j j dd« | j j « | j z
|
}|dz dk7 r t j
|
dt « ¬« | j dkD r4| j j | j « || j z S | j j dd« | j j d« }t j |t j ¬« }t j |dk( « }t |« dk( rd}nd t |« z }| j j | j « ||z
|
| j z S )
|
zÇ
|
Get number of records in file.
|
|
This is maybe suboptimal because we have to seek to the end of
|
the file.
|
|
Side effect: returns file position to record_start.
|
r rD rq z xport file may be corrupted.)Ú
|
stackleveli°ÿÿÿr5 l @@ r2 )re r r r ÚwarningsÚwarnr r rr r8 Ú
|
frombufferÚuint64Ú flatnonzeror: )ri Útotal_records_lengthÚlast_card_bytesÚ last_cardÚixÚtail_pads r& r zXportReader._record_count s: ð × Ñ ×$Ñ$ Q¨Ô*Ø #× 6Ñ 6× ;Ñ ;Ó =À×@QÑ@QÑ QÐ à "Ñ $¨Ò )Ü MMØ.Ü+Ó-õ ð
|
× Ñ Ò "Ø × #Ñ #× (Ñ (¨×):Ñ):Ô ;Ø'¨4×+=Ñ+=Ñ=Ð =à × Ñ ×$Ñ$ S¨!Ô,Ø ×1Ñ1×6Ñ6°rÓ:ÜMM /¼¿¹ÔC ô ^^IÐ)<Ñ <Ó =ä r7a<ØHà3r7{Hà × Ñ ×$Ñ$ T×%6Ñ%6Ô7à$ xÑ/°D×4FÑ4FÑFÐFr0 c óB | | j }| j |¬« S )a
|
Reads lines from Xport file and returns as dataframe
|
|
Parameters
|
----------
|
size : int, defaults to None
|
Number of lines to read. If None, reads whole file.
|
|
Returns
|
-------
|
DataFrame
|
r¿ )rb rr )ri Úsizes r& Ú get_chunkzXportReader.get_chunkº s# ð <Ø??DØyytyÓ$Ð$r0 c ó® |j d¬« }|d dk( |d dk( z |d dk( z }|d dk\ |d d k z |d d
|
k( z |d d k( z }||z }|S ) Nz u1,u1,u2,u4r5 rB r Úf2Úf3r7 rJ éZ é_ é. )r; )ri r< r¹ ÚmissÚmiss1s r& Ú_missing_doublezXportReader._missing_doubleË s Ø HH=HÓ )Ø$1 4¡¨A¡ Ñ.°!°D±'¸Q±,Ñ?àgo ! D¡'¨T¡/Ñ 2Øw$ñ àw$ñ ð ð
|
Ø r0 c ó| | | j }t || j | j z
|
« }|| j z }|dk r| j « t
|
| j j |« }t j || j |¬« }i }t | j « D ]ò \ }}|dt |« z } | j | d }
|
|
|
dk( rLt | | j | d « } | j! | « } t# | « } t j$ | | < nf| j | d dk( rQ| D cg c] } | j' « } } | j( (| D cg c] } | j+ | j( « } } |j- | i« ô t/ j0 |« }| j2 <t/ j4 t7 | j | j |z « « |_ n|j; | j2 « }| xj |z c_ |S c c} w c c} w )Nr )r6 Úcountr* r r r r ) r Úminr` r rh Ú StopIterationre rr r8 rÆ r£ r¡ r r¢ r r@ rÙ rW ÚnanÚrstripr_ rs r r# Ú DataFramera ÚIndexÚrangerj Ú set_index)ri rÀ Ú
|
read_linesÚread_lenÚrawÚdataÚdf_dataÚjr» r< r r× r¹ ÚyÚdfs r& rr zXportReader.readÖ sæ à =ØIIEä § ¡ ¨D×,<Ñ,<Ñ <Ó=
|
Ø × 2Ñ 2Ñ2Ø q=Ø JJLÜ Ð Ø×%Ñ%×*Ñ*¨8Ó4Ü}}S¨¯ © ¸:ÔFàÜ dllÓ+ò #DAqØsS V|Ñ$CØKK N 7Ñ+EØ Ò!Ü1°#°t·{±{À1±~ÀnÑ7UÓVØ×+Ñ+¨CÓ0Ü$ SÓ)Ü &&$ØQ Ñ(¨FÒ2Ø),Ö- AQXXZÐ-Ð-à>>Ð-Ø;<Ö=°a $§.¡.Õ1Ð=AÐ=à NNAq6Õ "ð #ô \\'Ó "à ;;Ð Üxx¤ d×&6Ñ&6¸×8HÑ8HÈ:Ñ8UÓ VÓWBHà dkkÓ*Bà ×ÒJÑ&Õà ùò .ùò >s ÅH4Å,"H9)Nz
|
ISO-8859-1NÚinfer)
|
re z FilePath | ReadBuffer[bytes]r\ z
|
str | Nonerk ú
|
int | Noner^ r ÚreturnÚNone)rî rï )rî ú pd.DataFrame)rî r rn )rÏ rí rî rð )rÀ rí rî rð )Ú__name__Ú
|
__module__Ú __qualname__Ú_xport_reader_docÚ__doc__rl rh rt rf rÁ r rÐ rÙ r Ú_read_method_docrr © r0 r& rY rY þ s Ø Gð
|
Ø +Ø $Ø*1ð à 8ðð ð ð
|
ð ð (ð ð
|
óó8 ò9ól ó\5ó$GôL%ò" ñ Ð Ó ó%ó ñ%r0 rY )r% r¢ rî r )r* r¢ )-rõ Ú
|
__future__r Ú collectionsr r r Útypingr rÄ Únumpyr8 Úpandas.util._decoratorsr Úpandas.util._exceptionsr Úpandasr# Úpandas.io.commonr Úpandas.io.sas.sasreaderr
|
Úpandas._typingr r r r r r r r r Ú_base_params_docÚ _params2_docÚ_format_params_docÚ _iterator_docÚ _read_sas_docrô rö r' r/ r@ rW ÚIteratorrY r÷ r0 r& ú<module>r s ðñõ #å Ý Û Ý Û ã å ,Ý 4ã å 'Ý .á÷ó ð'ð ð
|
Rð ð'ð ð
|
'ð ò
|
ð(CÐ ð @ ð9Ð ðA ð
|
àÐ ð ØÐ ð Ø ð Øð ð ð2ð Ð ð Ø ð ð Ð ð Ð ó óò,ò&6ôr~*cllõ ~r0
|
