Ë
|
ê^i)[ ã ó@ d dl mZ d dlZd dlmZ d dlmZ ddlmZm Z m
|
Z
|
m Z ddl m Z mZmZmZ ddlmZ dd lmZmZ dd
|
lmZmZmZmZmZm Z m Z ej< d « Z ej@ « Z!e!jE ejF d « « d dd Z$ d ddZ% d ddZ& d ddZ'y)é )Ú annotationsN)ÚPathLike)ÚBinaryIOé )Úcoherence_ratioÚencoding_languagesÚmb_encoding_languagesÚmerge_coherence_ratios)ÚIANA_SUPPORTEDÚTOO_BIG_SEQUENCEÚTOO_SMALL_SEQUENCEÚTRACE)Ú
|
mess_ratio)Ú CharsetMatchÚCharsetMatches)Úany_specified_encodingÚcut_sequence_chunksÚ iana_nameÚidentify_sig_or_bomÚ is_cp_similarÚis_multi_byte_encodingÚshould_strip_sig_or_bomÚcharset_normalizerz)%(asctime)s | %(levelname)s | %(message)sc
|
ó$ t | t t f« s#t dj t | « « « |rBt j }
|
t j t « t j t « t | « } | dk( rqt j d« |r@t j t « t j
|
xs t j « t# t% | dddg d« g« S |Dt j' t d d
|
j) |« « |D cg c] } t+ | d« }} ng }|Dt j' t d d
|
j) |« « |D cg c] } t+ | d« }} ng }| ||z k r!t j' t d ||| « d }| }|d kD r| |z |k rt- | |z « }t | « t. k } t | « t0 k\ }| r*t j' t dj | « « n+|r)t j' t dj | « « g }|r t3 | « nd}|,|j5 |« t j' t d|« t7 « }g }g }d}d}d}t# « }t# « }t9 | « \ }}|6|j5 |« t j' t dt |« |« |j5 d« d|vr|j5 d« |t: z D ]` }|r||vr |r||v r||v r|j= |« d} ||k( } | xr t? |« } |dv r | s t j' t d|« `|dv r | s t j' t d|« tA |« } |r9| du r5tG | du r| dt- d« n| t |« t- d« |¬« n tG | du r| n | t |« d |¬« } d}!|D ] }"tM ||"« sd}! n |!r t j' t d |"« "tO | sdn
|
t |« | t- | |z « « }#| xr | duxr t | « | k }$|$rt j' t d |« t- t |#« d z « }%tQ |%d « }%d}&d}'g }(g }) tS | ||#|| | || | « D ]g }*|(j5 |*« |)j5 tU |*||du xr d t |« cxk xr d k nc « « |)d |k\ r|&d z }&|&|%k\ s| sb| du sg n |'s$|r"| s | t- d"« d jW |d#¬$« |)rtY |)« t |)« z nd}+|+|k\ s|&|%k\ ro|j5 |« t j' t d&||&t[ |+d'z d(¬)« « | r/|dd|d*d+fv r&|'s$t% | ||| g | |¬,« },||k( r|,}n
|
|dk( r|,}n|,}àt j' t d-|t[ |+d'z d(¬)« « | s t] |« }-n t_ |« }-|-r3t j' t d.j |tG |-« « « g }.|dk7 r8|(D ]3 }*ta |*||-rd/j) |-« nd« }/|.j5 |/« 5 tc |.« }0|0r*t j' t d0j |0|« « t% | ||+| |0|du s||ddfv r| nd|¬,« }1|j5 |1« ||ddfv ry|+d1k rt|+dk( r^t j d2|1jd « |r.t j t « t j
|
« t# |1g« c S |j5 |1« t |« r||||v rvd|v rrd|v rn|jg « }2t j d2|2jd « |r.t j t « t j
|
« t# |2g« c S ||k( s t j d3|« |r.t j t « t j
|
« t# || g« c S t |« dk( rÆ|s|s|rt j' t d4« |r2t j d5|jd « |j5 |« nr|r| |r|r|jh |jh k7 s|'t j d6« |j5 |« n(|r&t j d7« |j5 |« |r<t j d8|jg « jd t |« d z
|
« nt j d9« |r.t j t « t j
|
« |S c c} w c c} w # tB tD f$ r t j' t d|« Y ôw xY w# tH tJ f$ rQ} t | tJ « s%t j' t d|tG | « « |j5 |« Y d} ~ Sd} ~ ww xY w# tH $ r4} t j' t d!|tG | « « |%}&d}'Y d} ~ md} ~ ww xY w# tH $ rA} t j' t d%|tG | « « |j5 |« Y d} ~ àd} ~ ww xY w):af
|
Given a raw bytes sequence, return the best possibles charset usable to render str objects.
|
If there is no results, it is a strong indicator that the source is binary/not text.
|
By default, the process will extract 5 blocks of 512o each to assess the mess and coherence of a given sequence.
|
And will give up a particular code page after 20% of measured mess. Those criteria are customizable at will.
|
|
The preemptive behavior DOES NOT replace the traditional detection workflow, it prioritize a particular code page
|
but never take it for granted. Can improve the performance.
|
|
You may want to focus your attention to some code page or/and not others, use cp_isolation and cp_exclusion for that
|
purpose.
|
|
This function will strip the SIG in the payload/sequence every time except on UTF-16, UTF-32.
|
By default the library does not setup any handler other than the NullHandler, if you choose to set the 'explain'
|
toggle to True it will alter the logger configuration to add a StreamHandler that is suitable for debugging.
|
Custom logging format and handler can be set manually.
|
z3Expected object of type bytes or bytearray, got: {}r z<Encoding detection on empty bytes, assuming utf_8 intention.Úutf_8g FÚ Nz`cp_isolation is set. use this flag for debugging purpose. limited list of encoding allowed : %s.z, zacp_exclusion is set. use this flag for debugging purpose. limited list of encoding excluded : %s.z^override steps (%i) and chunk_size (%i) as content does not fit (%i byte(s) given) parameters.r z>Trying to detect encoding from a tiny portion of ({}) byte(s).zIUsing lazy str decoding because the payload is quite large, ({}) byte(s).z@Detected declarative mark in sequence. Priority +1 given for %s.zIDetected a SIG or BOM mark on first %i byte(s). Priority +1 given for %s.Úascii> Úutf_16Úutf_32z\Encoding %s won't be tested as-is because it require a BOM. Will try some sub-encoder LE/BE.> Úutf_7zREncoding %s won't be tested as-is because detection is unreliable without BOM/SIG.z2Encoding %s does not provide an IncrementalDecoderg A)Úencodingz9Code page %s does not fit given bytes sequence at ALL. %sTzW%s is deemed too similar to code page %s and was consider unsuited already. Continuing!zpCode page %s is a multi byte encoding table and it appear that at least one character was encoded using n-bytes.é é éÿÿÿÿzaLazyStr Loading: After MD chunk decode, code page %s does not fit given bytes sequence at ALL. %sg jè@Ústrict)Úerrorsz^LazyStr Loading: After final lookup, code page %s does not fit given bytes sequence at ALL. %szc%s was excluded because of initial chaos probing. Gave up %i time(s). Computed mean chaos is %f %%.éd é )Úndigitsr r )Úpreemptive_declarationz=%s passed initial chaos probing. Mean measured chaos is %f %%z&{} should target any language(s) of {}ú,z We detected language {} using {}ç¹?z.Encoding detection: %s is most likely the one.zoEncoding detection: %s is most likely the one as we detected a BOM or SIG within the beginning of the sequence.zONothing got out of the detection process. Using ASCII/UTF-8/Specified fallback.z7Encoding detection: %s will be used as a fallback matchz:Encoding detection: utf_8 will be used as a fallback matchz:Encoding detection: ascii will be used as a fallback matchz]Encoding detection: Found %s as plausible (best-candidate) for content. With %i alternatives.z=Encoding detection: Unable to determine any suitable charset.)5Ú
|
isinstanceÚ bytearrayÚbytesÚ TypeErrorÚformatÚtypeÚloggerÚlevelÚ
|
addHandlerÚexplain_handlerÚsetLevelr ÚlenÚdebugÚ removeHandlerÚloggingÚWARNINGr r ÚlogÚjoinr Úintr r r ÚappendÚsetr r Úaddr r ÚModuleNotFoundErrorÚ ImportErrorÚstrÚUnicodeDecodeErrorÚ LookupErrorr ÚrangeÚmaxr r ÚdecodeÚsumÚroundr r r r
|
r! ÚbestÚ fingerprint)3Ú sequencesÚstepsÚ
|
chunk_sizeÚ thresholdÚ cp_isolationÚ cp_exclusionÚpreemptive_behaviourÚexplainÚlanguage_thresholdÚenable_fallbackÚprevious_logger_levelÚlengthÚcpÚis_too_small_sequenceÚis_too_large_sequenceÚprioritized_encodingsÚspecified_encodingÚtestedÚtested_but_hard_failureÚtested_but_soft_failureÚfallback_asciiÚ fallback_u8Úfallback_specifiedÚresultsÚearly_stop_resultsÚ sig_encodingÚ sig_payloadÚ encoding_ianaÚdecoded_payloadÚbom_or_sig_availableÚstrip_sig_or_bomÚis_multi_byte_decoderÚeÚsimilar_soft_failure_testÚencoding_soft_failedÚr_Úmulti_byte_bonusÚmax_chunk_gave_upÚearly_stop_countÚlazy_str_hard_failureÚ md_chunksÚ md_ratiosÚchunkÚmean_mess_ratioÚfallback_entryÚtarget_languagesÚ cd_ratiosÚchunk_languagesÚcd_ratios_mergedÚ current_matchÚprobable_results3 õi H:\项ç®\archive\æµè¯ç»\èæ¬\Change_password\venv_build\Lib\site-packages\charset_normalizer/api.pyÚ
|
from_bytesr ! sb ô< i¤)¬UÐ!3Ô 4ÜØ A× HÑ HÜYó ó
|
ð
|
ñ Ü%+§\¡\Ð Ü×Ñ/Ô*ÜÔ äi.Fà {Ü ÐSÔTÙ Ü × Ñ ¤ Ô 1Ü OOÐ 1Ò D´W·_±_Ô EÜ |¨I°wÀÀUÈBÐPRÓ SÐ TÓUÐUàÐ Ü
|
|
Ü ð 5à IIlÓ #ô
|
ð 8DÖD° " eÕ,ÐD ÑDà àÐ Ü
|
|
Ü ð 6à IIlÓ #ô
|
ð 8DÖD° " eÕ,ÐD ÑDà à *uÑ$Ò%Ü
|
|
Ü Ø lØ Ø Ø ô
|
ð Ø
|
à qyVe^ jÒ0Ü % Ó(
|
ä"% i£.Ô3EÑ"EÐÜ"% i£.Ô4DÑ"DÐá Ü
|
|
Ü Ø L× SÑ SØó õ
|
ñ
|
Ü
|
|
Ü Ø W× ^Ñ ^Øó ô
|
ð (*Ðñ .BÔ yÔ)Àtð ð Ð%Ø ×$Ñ$Ð%7Ô8Ü
|
|
Ü Ø NØ ô
|
ô uFØ)+ÐØ)+Ðà*.NØ'+KØ.2Ðä ,Ó .Gä)7Ó)9Ðä 3°IÓ >Ñ L+àÐ Ø ×$Ñ$ \Ô2Ü
|
|
Ü Ø WÜ Ó Ø ô
|
ð × Ñ Ô)àÐ+Ñ+Ø ×$Ñ$ WÔ-à.´ Ñ?ó < Ù M° Ñ =Ø á M¨\Ñ 9Ø à FÑ "Ø à
|
|
=Ô!à&*Ø%1°]Ñ%BÐ Ø!5ò "
|
Ô:QØ ó;
|
Ðð Ð 0Ñ 0Ñ9MÜ JJÜØnØ ô ð
|
Ø IÑ %Ñ.BÜ JJÜØdØ ô ð
|
ð Ü*@À Ó*OÐ !ð Ù$Ð)>À%Ñ)GÜð ,¨uÑ4ð " +¤C¨£IÑ.à &¤s¨;Ó'7¼#¸d»)Ð Dà *ö ô #&ð ,¨uÑ4ñ "à &¤s¨;Ó'7Ð'9Ð :à *ô #ð& +0Ð!à$;ò Ð Ü ]Ð,@ÕAØ,0Ð)Ùð ñ
|
%Ü JJÜØiØ Ø$ô ñ ä Ù)A¬s°;Ó/?Ø Ü Ó ó
|
ð "ò .Ø tÐ+ò .äOÓ$ vÑ-ð ñ Ü JJÜð-à ô ô "%¤S¨£W¨q¡[Ó!1Ðä Ð 1°1Ó 5ÐØ !ÐØ %Ð à ! Ø ð' )Ü,ØØ ØØØ$Ø ØØ%Ø ó
|
ò ð × Ñ Ô'à× Ñ Ü Ø Ø!Ø 4ÒG¨A´°\Ó1BÖ,GÀaÔ,Góôð R= IÒ-Ø$¨Ñ)Ð$à$Ð(9Ò9Ú(Ð-=ÀÒ-Fáð7 ñV &Ù%Ù)ð
|
Ø#d)+Ð&×-Ñ-¨mÀHÐ-ÔMñ EN¤ Y£ ´#°i³.Ò!@ÐSVØ iÒ 'Ð+;Ð?PÒ+PØ #× *Ñ *¨=Ô 9Ü JJÜð0à Ø Üo¨Ñ+°QÔ7ô ñ Ø!ØWÐ&8¸(ÀHÐMñNá-ä!-Ø Ø!Ø Ø(ØØ#Ø+=ô"ð !Ð$6Ò6Ø)7Ñ&Ø" gÒ-Ø%3Nà"0KÙ ä
|
|
Ü Ø KØ Ü / CÑ'°Ô 3ô
|
ñ %Ü*<¸]Ó*KÑ ä 4°]Ó CÐ á Ü JJÜØ8×?Ñ?Ø!¤3Ð'7Ó#8óô ð ð GÒ #Ø"ò 2Ü"1ØØ&Ù2BCHHÐ -Ô.Èó#ð × Ñ Õ1ð 2ô 2°)Ó<Ðá Ü JJÜØ2×9Ñ9Ø$ móô ô %Ø Ø Ø Ø Ø ð *¨UÑ2Ø$Ð);¸WÀgÐ(NÑNñ ð
|
à#5ô
|
ð" }Ô%ð Ð 0°'¸7Ð CÑ CØ #Ò%ð #Ò%Ü ØDØ!×*Ñ*ôñ Ü×(Ñ(¬ Ô9ÜOOÐ$9Ô:Ü% } oÓ6Ò6à × %Ñ % mÔ 4ô Ð"Ô #Ø#Ð+Ð/AÀVÑ/KØ6Ñ!Ø6Ñ!à,>×,CÑ,CÓ,EOÜ LLØ@Ø ×(Ñ(ô ñ Ü×$Ñ$¤_Ô5ÜÐ 5Ô6ä! ?Ð"3Ó4Ò 4à LÔ (Ü LLð1à ô ñ
|
Ü×$Ñ$¤_Ô5ÜÐ 5Ô6Ü! 7¨=Ñ#9Ð":Ó;Ò ;ð <ôB
|
7|qÒÙ .Ñ,>Ü JJÜØaô ñ
|
Ü LLØIØ"×+Ñ+ô ð NNÐ-Õ .á ^Ð 3áÙ"Ø ×+Ñ+¨~×/IÑ/IÒIàÐ'ä LLÐUÔ VØ NN;Õ 'Ù Ü LLÐUÔ VØ NN>Ô *áÜ Ø kØ LLN× #Ñ #Ü L1Ñ õ
|
ô ÐTÔUáÜ× Ñ _Ô-ÜÐ-Ô.à NùòG Eùò Eøôb $¤[Ð1ò Ü JJÜØDØ ô ò
|
ð ûô6 #¤KÐ0ò Ü a¤Ô-Ü
|
|
ÜØOØ!ÜFô ð $× *Ñ *¨=Ô 9Ý ûð ûôl ò
|
)ô JJÜØsØ ÜAô ð 1Ð Ø$(Ö !ûð
|
)ûô* &ò Ü
|
|
ÜØtØ!ÜFô ð (×.Ñ.¨}Ô=Ýûð úsu Ä/g'Å7g,Î- g1Î9Ah"ÓA4jÕjÕ jÕ kç1*h è h è"jè1Ai=é=jê kê)j=ê=kë lë6l
|
ì
|
lc
|
óF t | j « ||||||||| «
|
S )z
|
Same thing than the function from_bytes but using a file pointer that is already ready.
|
Will not close the file pointer.
|
)r Úread)
|
ÚfprP rQ rR rS rT rU rV rW rX s
|
r Úfrom_fpr ! s5 ô Ø
|
Ø ØØØØØ ØØØó ð ó c
|
ón t | d« 5 }
|
t |
|
||||||||| «
|
cddd« S # 1 sw Y yxY w)z
|
Same thing than the function from_bytes but with one extra step. Opening and reading given file path in binary mode.
|
Can raise IOError.
|
ÚrbN)Úopenr ) ÚpathrP rQ rR rS rT rU rV rW rX r s r Ú from_pathr ? sK ô
|
dDÓ ð
|
RÜØ Ø Ø Ø Ø Ø Ø Ø Ø Ø ó
|
÷
|
÷
|
ò
|
ús +«4c
|
óê t | t t f« rt | ||||||||| ¬«
|
}
|
|
|
S t | t t
|
f« rt | ||||||||| ¬«
|
}
|
|
|
S t | ||||||||| ¬«
|
}
|
|
|
S )a)
|
Detect if the given input (file, bytes, or path) points to a binary file. aka. not a string.
|
Based on the same main heuristic algorithms and default kwargs at the sole exception that fallbacks match
|
are disabled to be stricter around ASCII-compatible but unlikely to be a string.
|
) rP rQ rR rS rT rU rV rW rX )r- rE r r r/ r. r r ) Úfp_or_path_or_payloadrP rQ rR rS rT rU rV rW rX Úguessess r Ú is_binaryr ^ sÇ ô" Ð'¬#¬x¨ Ô9ÜØ !ØØ!Ø Ø%Ø%Ø!5ØØ 1Ø +ô
|
ðZ ;ÐôC
|
Ø ä Ü ð
|
ô
|
ô Ø !ØØ!Ø Ø%Ø%Ø!5ØØ 1Ø +ô
|
ð4 ;Ðô Ø !ØØ!Ø Ø%Ø%Ø!5ØØ 1Ø +ô
|
ð ;Ðr ) é é çÉ?NNTFr, T)rO zbytes | bytearrayrP r? rQ r? rR ÚfloatrS úlist[str] | NonerT r rU ÚboolrV r rW r rX r Úreturnr )r r rP r? rQ r? rR r rS r rT r rU r rV r rW r rX r r r )r zstr | bytes | PathLikerP r? rQ r? rR r rS r rT r rU r rV r rW r rX r r r ) r r r NNTFr, F)r z!PathLike | str | BinaryIO | bytesrP r? rQ r? rR r rS r rT r rU r rV r rW r rX r r r )(Ú
|
__future__r r; Úosr Útypingr Úcdr r r r
|
Úconstantr r r r Úmdr Úmodelsr r Úutilsr r r r r r r Ú getLoggerr3 Ú StreamHandlerr6 Ú setFormatterÚ Formatterr r r r © r r ú<module>r¦ s¼ ðÝ "ã Ý Ý ÷ó ÷ RÓ QÝ ß 0÷÷ ñ ð
|
× Ñ Ð/Ó 0Ø''×'Ñ'Ó)Ø × Ñ ØG×ÑÐAÓBôð ØØØ%)Ø%)Ø!%ØØ #Ø ð}Ø ð}à ð}ð ð}ð ð }ð
|
#ð }ð #ð }ð ð}ð ð}ð ð}ð ð}ð ó}ðD ØØØ%)Ø%)Ø!%ØØ #Ø ðØðà ðð ðð ð ð
|
#ð ð #ð ð ðð ðð ðð ðð óð@ ØØØ%)Ø%)Ø!%ØØ #Ø ð
|
Ø
|
ð
|
à ð
|
ð ð
|
ð ð
|
ð
|
#ð
|
ð #ð
|
ð ð
|
ð ð
|
ð ð
|
ð ð
|
ð ó
|
ðB ØØØ%)Ø%)Ø!%ØØ #Ø !ð?Ø<ð?à ð?ð ð?ð ð ?ð
|
#ð ?ð #ð ?ð ð?ð ð?ð ð?ð ð?ð
|
ô?r
|
