Ë
|
ê^iu2 ã ót d dl mZ d dlZd dlmZ d dlmZ d dlmZ d dl mZ
|
ddl m Z m Z mZmZmZ ddlmZ dd lmZ dd
|
lmZmZmZmZmZ dd Zdd Z e« dd « Z e« dd« Z ee¬« dd« Z d d dZ d dZ!d dZ"d dZ#d dZ$ ed¬« d! d"d« Z%y)#é )Ú annotationsN)ÚIncrementalDecoder)ÚCounter)Ú lru_cacheé )Ú FREQUENCIESÚKO_NAMESÚLANGUAGE_SUPPORTED_COUNTÚTOO_SMALL_SEQUENCEÚZH_NAMES)Ú is_suspiciously_successive_range)ÚCoherenceMatches)Úis_accentuatedÚis_latinÚis_multi_byte_encodingÚis_unicode_range_secondaryÚ unicode_rangec óª t | « r t d« t j d| « j } |d¬« }i }d}t dd« D ]W }|j t |g« « }|s!t |« }|/t |« du s=||vrd||< ||xx d z cc< |d z }Y t |D cg c] }|| |z d
|
k\ r| c}« S c c}w ) zF
|
Return associated unicode ranges in a single byte code page.
|
z.Function not supported on multi-byte code pagez
|
encodings.Úignore)Úerrorsr é@ éÿ Fr g333333Ã?) r ÚOSErrorÚ importlibÚ import_moduler ÚrangeÚdecodeÚbytesr r Úsorted)Ú iana_nameÚdecoderÚpÚ seen_rangesÚcharacter_countÚiÚchunkÚcharacter_ranges õh H:\项ç®\archive\æµè¯ç»\èæ¬\Change_password\venv_build\Lib\site-packages\charset_normalizer/cd.pyÚencoding_unicode_ranger) s ô iÔ(ÜÐFÓGÐGä×%Ñ%¨
|
°9°+Ð&>Ó?×RÑRGá #¨8Ô 4AØ"$KØ Oä 4Ó ò %ØXXe Q CjÓ)â Ü*7¸Ó*>Oà Ð&Øä)¨/Ó:¸eÒCØ"¨+Ñ5Ø34K Ñ0ØOÓ,°Ñ1Ó,Ø 1Ñ$ð %ô ð $/ö
|
à Ø?Ñ+¨oÑ=ÀÒEò ò
|
ó ð ùò
|
s Â5Cc ó g }t j « D ]- \ }}|D ]# }t |« | k( s|j |« - / |S )z>
|
Return inferred languages used with a unicode range.
|
)r Úitemsr Úappend)Ú primary_rangeÚ languagesÚlanguageÚ
|
charactersÚ characters r( Úunicode_range_languagesr2 @ s[ ð Iä +× 1Ñ 1Ó 3ò Ñ *Ø#ò IÜ YÓ'¨=Ó8Ø× Ñ Ô*Ùñ ðð Ðó c óZ t | « }d}|D ]
|
}d|vs|} n |dgS t |« S )z
|
Single-byte encoding language association. Some code page are heavily linked to particular language(s).
|
This function does the correspondence.
|
NÚLatinú Latin Based)r) r2 )r Úunicode_rangesr- Úspecified_ranges r( Úencoding_languagesr9 O sM ô !7°yÓ ANØ $Mà)ò Ø /Ò )Ø +MÙ ðð
|
Ð Ø Ð ä " =Ó 1Ð1r3 c óì | j d« s'| j d« s| j d« s| dk( rdgS | j d« s| t v rdgS | j d« s| t v rd gS g S )
|
z
|
Multi-byte encoding language association. Some code page are heavily linked to particular language(s).
|
This function does the correspondence.
|
Úshift_Ú
|
iso2022_jpÚeuc_jÚcp932ÚJapaneseÚgbÚChineseÚ
|
iso2022_krÚKorean)Ú
|
startswithr r )r s r( Úmb_encoding_languagesrE c s} ð × Ñ XÔ&Ø × Ñ Ô -Ø × Ñ Ô (Ø Ò à|ÐØ×ÑDÔ! Y´(Ñ%:Ø{ÐØ×ÑLÔ)¨Y¼(Ñ-BØzÐà Ir3 )Úmaxsizec ór d}d}t | D ]$ }|s t |« rd}|st |« du s#d}& ||fS )zg
|
Determine main aspects from a supported language if it contains accents and if is pure Latin.
|
FT)r r r )r/ Útarget_have_accentsÚtarget_pure_latinr1 s r( Úget_target_featuresrJ x sW ð
|
!&ÐØ "Ðä Ñ*ò & Ù"¤~°iÔ'@Ø"&Ð Ú ¤¨)Ó!4¸Ò!=Ø %Ñ ð &ð Ð 1Ð 1Ð1r3 c ó g }t d | D « « }t j « D ]h \ }}t |« \ }}|r|du r|du r|r"t |« }t |D cg c] } | | v s| c} « }
|
|
|
|z } | dk\ sV|j || f« j t |d d¬« }|D cg c] } | d c} S c c} w c c} w )zE
|
Return associated languages associated to given characters.
|
c 3 ó2 K | ] }t |« y w©N)r )Ú.0r1 s r( ú <genexpr>z%alphabet_languages.<locals>.<genexpr> s è ø Ò T¸In¨Y× 7Ñ Tùs FgÉ?c ó | d S ©Nr © ©Úxs r( ú<lambda>z$alphabet_languages.<locals>.<lambda>§ s
|
°°!± r3 T©ÚkeyÚreverser )Úanyr r+ rJ Úlenr, r ) r0 Úignore_non_latinr. Úsource_have_accentsr/ Úlanguage_charactersrH rI r$ ÚcÚcharacter_match_countÚratioÚcompatible_languages r( Úalphabet_languagesrb sé ð *,Iä Ñ TÈÔ TÓTÐä)4×):Ñ):Ó)<ò 0Ñ%Ð%Ü1DÀXÓ1NÑ.ÐÐ .á Ð 1°UÑ :Ø à %Ñ 'Ñ,?Ø ä "Ð#6Ó 7ä%(Ø+Ö ?1¨q°JªQÒ ?ó&
|
Ð ð -¨Ñ>à C<Ø × Ñ h¨Ð .Õ /ð%0ô( y¡n¸dÔCIà>GÖ HÐ':Ð Ó "Ò HÐHùò @ùò Is Á B6
|
Á)B6
|
Â' B;c ó8 | t vrt | d« d}t t | « }t |« }t t | « }|dkD }t |t d|« « D ]' \ }}||vr t | j |« } ||z }
|
t ||
|
z « } |du rt | | z
|
« dkD rM|du rt | | z
|
« |dz k r|dz }kt | d| } t | | d } |d| }||d }t t |« t | « z « }t t |« t | « z « }t | « dk( r |dk r|dz }æt | « dk( r |dk r|dz }ÿ|t | « z d
|
k\ s|t | « z d
|
k\ s#|dz }* |t |« z S ) aN
|
Determine if a ordered characters list (by occurrence from most appearance to rarest) match a particular language.
|
The result is a ratio between 0. (absolutely no correspondence) and 1. (near perfect fit).
|
Beware that is function is not strict on the match in order to ease the detection. (Meaning close match is 1.)
|
z not availabler é Fé Té r NgÙ?) r Ú
|
ValueErrorÚsetrZ Úzipr ÚindexÚintÚabs)r/ Úordered_charactersÚcharacter_approved_countÚFREQUENCIES_language_setÚordered_characters_countÚ target_language_characters_countÚlarge_alphabetr1 Úcharacter_rankÚcharacter_rank_in_languageÚexpected_projection_ratioÚcharacter_rank_projectionÚcharacters_before_sourceÚcharacters_after_sourceÚcharacters_beforeÚcharacters_afterÚbefore_match_countÚafter_match_counts r( Ú characters_popularity_comparer} ¬ s; ð {Ñ"ÜH: ^Ð4Ó5Ð5à$%Ð Ü "¤;¨xÑ#8Ó 9Ð ä$'Ð(:Ó$;Ð Ü,/´ ¸HÑ0EÓ,FÐ$à;¸bÑ@Nä%(ØE !Ð%=Ó >ó&ó 8Ñ! >ð Ð 4Ñ 4Ø ä*5°hÑ*?×*EÑ*EÀiÓ*PÐ"à ,Ð/GÑ Gð "ô *-¨^Ð>WÑ-WÓ)XÐ!ð eÑ #ÜÐ-Ð0JÑJÓKÈaÒOà ð dÑ "ÜÐ-Ð0JÑJÓKØ.°Ñ2ò3ð %¨Ñ )Ð $Ø ä.9¸(Ñ.CØ Ð(ð/
|
Ð ô .9¸Ñ-BØ &Ð 'ð.
|
Ð ð (:¸!¸NÐ'KÐØ&8¸ ¸ Ð&IÐä"%Ü Ð!Ó "¤SÐ)AÓ%BÑ Bó#
|
Ðô "%Ü Ð Ó !¤CÐ(?Ó$@Ñ @ó"
|
Ðô Ð'Ó (¨AÒ -Ð2DÈÒ2IØ $¨Ñ )Ð $Ø ä Ð&Ó '¨1Ò ,Ð1BÀaÒ1GØ $¨Ñ )Ð $Ø ð ¤Ð%=Ó!>Ñ >À#Ò EØ ¤3Ð'>Ó#?Ñ?À3ÔFà $¨Ñ )Ð $Ù ðq8ðt $¤cÐ*<Ó&=Ñ =Ð=r3 c ó, i }| D ]u }|j « du rt |« }|$d}|D ] }t ||« du s|} n ||}||vr|j « ||< [||xx |j « z cc< w t |j « « S )a
|
Given a decoded text sequence, return a list of str. Unicode range / alphabet separation.
|
Ex. a text containing English/Latin with a bit a Hebrew will return two items in the resulting list;
|
One containing the latin letters and the other hebrew.
|
FN)Úisalphar r ÚlowerÚlistÚvalues)Údecoded_sequenceÚlayersr1 r' Úlayer_target_rangeÚdiscovered_ranges r( Úalpha_unicode_splitr ü sÉ ð Fà%ò 8 Ø × Ñ Ó %Ñ 'Ø ä&3°IÓ&>à Ð "Ø à)-Ðà &ò Ð ä0Ð1AÀ?ÓSØòð &6Ð"Ùð ð Ð %Ø!0Ð à VÑ +Ø)2¯ © Ó):FÐ%Ñ &Ø àÐ!Ó" i§o¡oÓ&7Ñ7Ô"ð58ô8 Ó Ð r3 c óþ i }| D ]- }|D ]& }|\ }}||vr|g||< || j |« ( / |D cg c]+ }|t t || « t || « z d« f- }}t |d d¬« S c c}w )z
|
This function merge results previously given by the function coherence_ratio.
|
The return type is the same as coherence_ratio.
|
re c ó | d S rQ rR rS s r( rU z(merge_coherence_ratios.<locals>.<lambda>< s
|
q¨¡t r3 TrV )r, ÚroundÚsumrZ r )ÚresultsÚper_language_ratiosÚresultÚ
|
sub_resultr/ r` Úmerges r( Úmerge_coherence_ratiosr # s ð
|
35ÐØò 8Ø ò 8JØ (OHeØÐ 2Ñ2Ø16°Ð# HÑ-ØØ Ñ )× 0Ñ 0°Õ 7ñ 8ð8ð ,ö ð ð Ü ÜÐ'¨Ñ1Ó2´SÐ9LÈXÑ9VÓ5WÑWØó ò
|
ð Eð ô %^°TÔ :Ð:ùò s ¹0A:c ó t « | D ]6 }|\ }}|j dd« }|vrg |< | j |« 8 t fdD « « r*g }D ]! }|j |t | « f« # |S | S )u³
|
We shall NOT return "Englishâ" in CoherenceMatches because it is an alternative
|
of "English". This function only keeps the best match and remove the em-dash in it.
|
u âÚ c 3 ó@ K | ] }t | « d kD yw)r N)rZ )rN ÚeÚ index_resultss r( rO z/filter_alt_coherence_matches.<locals>.<genexpr>O s øè ø Ò
|
<¨3}QÑ Ó 1Õ $Ñ
|
<ùs )ÚdictÚreplacer, rY Úmax)r r r/ r` Ú
|
no_em_nameÚfiltered_resultsr s @r( Ú filter_alt_coherence_matchesr ? s¬ ø ô
|
-1«FMàò 0Ø %Ø"×*Ñ*¨5°"Ó5
|
à ]Ñ *Ø(*M*Ñ %àjÑ!×(Ñ(¨Õ/ð0ô Ó
|
<¨mÔ
|
<Ô<Ø-/Ðà%ò NHØ × #Ñ # X¬s°=ÀÑ3JÓ/KÐ$LÕ Mð Nð Ð à Nr3 i c ó g }d}d}||j d« ng }d|v rd}|j d« t | « D ]¢ }t |« }|j « } t d | D « « }
|
|
|
t k r:| D cg c] \ } } | } } } |xs t | |« D ]B }t || « }||k r|dk\ r|dz }|j |t |d « f« |d
|
k\ sB ¢ ¤ t t |« d d¬ « S c c} } w ) z¨
|
Detect ANY language that can be identified in given sequence. The sequence will be analysed by layers.
|
A layer = Character extraction by alphabets/ranges.
|
Fr ú,r6 Tc 3 ó& K | ] \ }}| y wrM rR )rN r^ Úos r( rO z"coherence_ratio.<locals>.<genexpr>q s è ø Ò"=©¨¨A¤1Ñ"=ùs gé?r re rf c ó | d S rQ rR rS s r( rU z!coherence_ratio.<locals>.<lambda> s
|
¸Q¸q¹T r3 rV ) ÚsplitÚremover r Ú most_commonr r rb r} r, r r r )r Ú thresholdÚ lg_inclusionr r[ Úsufficient_match_countÚlg_inclusion_listÚlayerÚsequence_frequenciesr¤ r$ r^ r Úpopular_character_orderedr/ r` s r( Úcoherence_ratior¬ Z sB ð (*GØ "Ðà"#Ðà3?Ð3K ×*Ñ*¨3Ô/ÐQSÐØÐ)Ñ)Ø ÐØ× Ñ Ô/ä$Ð%5Ó6ò Ü18¸³ Ð Ø*×6Ñ6Ó8 ä "Ñ"=°Ô"=Ó =à Ô 0Ò 0Ø à>I×/J±d°a¸²Ð/JÐ!Ñ/Jà)ò
|
Ô-?Ø %Ð'7ó.
|
ò Hô 9ØÐ3ó Eð yÒ ØØ# Ø&¨!Ñ+Ð&à NNH¤e¨E°1£oÐ6Ô 7à%¨Ó*Ùñ ðô8 Ü$ WÓ-±>È4ô ð ùó' 0Ks Á= C;)r ÚstrÚreturnú list[str])r- r r® r¯ )r/ r r® ztuple[bool, bool])F)r0 r¯ r[ Úboolr® r¯ )r/ r rm r¯ r® Úfloat)r r r® r¯ )r zlist[CoherenceMatches]r® r )r r r® r )g¹?N)r r r¥ r± r¦ z
|
str | Noner® r )&Ú
|
__future__r r Úcodecsr Ú collectionsr Ú functoolsr ÚtypingÚ TypeCounterÚconstantr r r
|
r r Úmdr Úmodelsr Úutilsr r r r r r) r2 r9 rE rJ rb r} r r r r¬ rR r3 r( ú<module>r¼ s ðÝ "ã Ý %Ý Ý Ý )÷õ õ 1Ý $÷õ ó"óJ ñ ò2ó ð2ñ& òó ðñ( Ð+Ô,ò 2ó -ð 2ð" 5:ð IØð IØ-1ð Iàó IðFM>ØðM>Ø'0ðM>à
|
óM>ó`$!óN;ó8ñ6 4ÔàNRð0Øð0Ø&+ð0ØAKð0àò0ó ñ0r3
|
