Ë
|
ñúh ã óX d Z ddlZddlZddlZddlmZmZ ddlmZ ddl m
|
Z
|
m Z m Z m Z mZ ddlmZmZmZmZmZmZ ddlmZ g d¢Z ej2 d « Zg d
|
¢Z e d « Z G d d e « Z G d de ¬« Z G d de « Z! G d de"« Z# G d d« Z$ e$« Z% G d de&« Z' G d d« Z(d Z) G d d « Z* e*« Z+d Z, G d d « Z- G d! d"e« Z. G d# d$e « Z/ G d% d&e e/¬« Z0 G d' d(« Z1 G d) d*e0« Z2d+ Z3 G d, d-e/« Z4 G d. d/e0e4¬« Z5y)0z´
|
pygments.lexer
|
~~~~~~~~~~~~~~
|
|
Base lexer classes.
|
|
:copyright: Copyright 2006-2025 by the Pygments team, see AUTHORS.
|
:license: BSD, see LICENSE for details.
|
é N)Ú apply_filtersÚFilter)Úget_filter_by_name)ÚErrorÚTextÚOtherÚ
|
WhitespaceÚ
|
_TokenType)Ú get_bool_optÚ get_int_optÚ get_list_optÚmake_analysatorÚFutureÚ guess_decode)Ú regex_opt) ÚLexerÚ
|
RegexLexerÚExtendedRegexLexerÚDelegatingLexerÚ LexerContextÚincludeÚinheritÚbygroupsÚusingÚthisÚdefaultÚwordsÚline_rez.*?
|
))s zutf-8)s ÿþ zutf-32)s þÿzutf-32be)s ÿþzutf-16)s þÿzutf-16bec ó y)Nç © )Úxs úMH:\Change_password\venv_build\Lib\site-packages\pip/_vendor/pygments/lexer.pyú<lambda>r$ " ó ó c ó e Zd ZdZd Zy)Ú LexerMetaz
|
This metaclass automagically converts ``analyse_text`` methods into
|
static methods which always return float values.
|
c ó\ d|v rt |d « |d< t j | |||« S )NÚ analyse_text)r ÚtypeÚ__new__)ÚmcsÚnameÚbasesÚds r# r, zLexerMeta.__new__+ s3 Ø QÑ Ü /°°.Ñ0AÓ BAnÑ Ü||C u¨aÓ0Ð0r& N)Ú__name__Ú
|
__module__Ú __qualname__Ú__doc__r, r! r& r# r( r( % s ñó
|
1r& r( c ó` e Zd ZdZdZg Zg Zg Zg ZdZ dZ
|
dZ dZ d Z d Zd Zd Zd Zd d Zd
|
Zy) r a"
|
Lexer for a specific language.
|
|
See also :doc:`lexerdevelopment`, a high-level guide to writing
|
lexers.
|
|
Lexer classes have attributes used for choosing the most appropriate
|
lexer based on various criteria.
|
|
.. autoattribute:: name
|
:no-value:
|
.. autoattribute:: aliases
|
:no-value:
|
.. autoattribute:: filenames
|
:no-value:
|
.. autoattribute:: alias_filenames
|
.. autoattribute:: mimetypes
|
:no-value:
|
.. autoattribute:: priority
|
|
Lexers included in Pygments should have two additional attributes:
|
|
.. autoattribute:: url
|
:no-value:
|
.. autoattribute:: version_added
|
:no-value:
|
|
Lexers included in Pygments may have additional attributes:
|
|
.. autoattribute:: _example
|
:no-value:
|
|
You can pass options to the constructor. The basic options recognized
|
by all lexers and processed by the base `Lexer` class are:
|
|
``stripnl``
|
Strip leading and trailing newlines from the input (default: True).
|
``stripall``
|
Strip all leading and trailing whitespace from the input
|
(default: False).
|
``ensurenl``
|
Make sure that the input ends with a newline (default: True). This
|
is required for some lexers that consume input linewise.
|
|
.. versionadded:: 1.3
|
|
``tabsize``
|
If given and greater than 0, expand tabs in the input (default: 0).
|
``encoding``
|
If given, must be an encoding name. This encoding will be used to
|
convert the input string to Unicode, if it is not already a Unicode
|
string (default: ``'guess'``, which uses a simple UTF-8 / Locale /
|
Latin1 detection. Can also be ``'chardet'`` to use the chardet
|
library, if it is installed.
|
``inencoding``
|
Overrides the ``encoding`` if given.
|
Nr c ól || _ t |dd« | _ t |dd« | _ t |dd« | _ t |dd« | _ |j dd « | _ |j d
|
« xs | j | _ g | _ t |d d « D ] }| j |« y )a
|
This constructor takes arbitrary options as keyword arguments.
|
Every subclass must first process its own options and then call
|
the `Lexer` constructor, since it processes the basic
|
options like `stripnl`.
|
|
An example looks like this:
|
|
.. sourcecode:: python
|
|
def __init__(self, **options):
|
self.compress = options.get('compress', '')
|
Lexer.__init__(self, **options)
|
|
As these options must all be specifiable as strings (due to the
|
command line usage), there are various utility functions
|
available to help with that, see `Utilities`_.
|
ÚstripnlTÚstripallFÚensurenlÚtabsizer ÚencodingÚguessÚ
|
inencodingÚfiltersr! N) Úoptionsr r7 r8 r9 r r: Úgetr; r> r Ú
|
add_filter)Úselfr? Úfilter_s r# Ú__init__zLexer.__init__ s¡ ð& Ü# G¨Y¸Ó= Ü$ W¨j¸%Ó@ Ü$ W¨j¸$Ó? Ü" 7¨I°qÓ9 Ø J°Ó8 Ø LÓ1ÒB°T·]±] Ø Ü# G¨Y¸Ó;ò %GØ OOGÕ $ñ %r& c ó | j r'd| j j d| j dS d| j j dS )Nz<pygments.lexers.z with ú>)r? Ú __class__r1 ©rB s r# Ú__repr__zLexer.__repr__¯ sI Ø <<Ø& t§~¡~×'>Ñ'>Ð&?¸vÀdÇlÁlÐEUÐUVÐWÐ Wà& t§~¡~×'>Ñ'>Ð&?¸qÐAÐ Ar& c ór t |t « s t |fi |¤}| j j |« y)z8
|
Add a new stream filter to this lexer.
|
N)Ú
|
isinstancer r r> Úappend)rB rC r? s r# rA zLexer.add_filterµ s/ ô '¤6Ô*Ü(¨Ñ<°GÑ<GØ ×ÑGÕ$r& c ó y)aÁ
|
A static method which is called for lexer guessing.
|
|
It should analyse the text and return a float in the range
|
from ``0.0`` to ``1.0``. If it returns ``0.0``, the lexer
|
will not be selected as the most probable one, if it returns
|
``1.0``, it will be selected immediately. This is used by
|
`guess_lexer`.
|
|
The `LexerMeta` metaclass automatically wraps this function so
|
that it works like a static method (no ``self`` or ``cls``
|
parameter) and the return value is automatically converted to
|
`float`. If the return value is an object that is boolean `False`
|
it's the same as if the return values was ``0.0``.
|
Nr! )Útexts r# r* zLexer.analyse_text½ r% r& c óÚ t |t « st| j dk( rt |« \ }}nu| j dk( r t d« |j | j « }|j d« r.|t d« d }n |j d« r|t d« d }|j dd« }|j d d« }| j r|j « }n | j r|j d« }| j d
|
kD r|j# | j « }| j$ r|j' d« s|dz }|S # t $ r}t d« |d}~ww xY w) zVApply preprocessing such as decoding the input, removing BOM and normalizing newlines.r< Úchardetz chardet is not vendored by pipzkTo enable chardet encoding guessing, please install the chardet library from http://chardet.feedparser.org/Nu z
|
ú
|
ú r )rK Ústrr; r Ú ImportErrorÚ _encoding_mapÚ
|
startswithÚlenÚdecoderP Údetectr@ Úreplacer8 Ústripr7 r: Ú
|
expandtabsr9 Úendswith)rB rN Ú_ÚeÚdecodedÚbomr; Úencs r# Ú_preprocess_lexer_inputz Lexer._preprocess_lexer_inputÎ sH ô $¤Ô$Ø}} Ò'Ü& tÓ,aØ )Ò+ðTô &Ð&FÓGÐGð$ {{ 4§=¡=Ó1Ø?? 8Ô,Ø ¤ H£ Ð/DàxÔ(ØC MNÐ+ð ||F DÓ)Ø||D $Ó'Ø ==Ø::<DØ \\Ø::dÓ#DØ <<!Ò Ø?? 4§<¡<Ó0DØ == § ¡ ¨tÔ!4Ø DLDà øôI #ò TÜ%ð 'Ló MàRSðTûðTús ¿ E Å E*Å E%Å%E*c óx j « fd} |« }|st | j « }|S )ae
|
This method is the basic interface of a lexer. It is called by
|
the `highlight()` function. It must process the text and return an
|
iterable of ``(tokentype, value)`` pairs from `text`.
|
|
Normally, you don't need to override this method. The default
|
implementation processes the options recognized by all lexers
|
(`stripnl`, `stripall` and so on), and then yields all tokens
|
from `get_tokens_unprocessed()`, with the ``index`` dropped.
|
|
If `unfiltered` is set to `True`, the filtering mechanism is
|
bypassed even if filters are defined.
|
c 3 óN K j « D ] \ } }}||f y w©N)Úget_tokens_unprocessed)r^ ÚtÚvrB rN s r# Ústreamerz"Lexer.get_tokens.<locals>.streamer s0 øè ø Ø ×6Ñ6°tÓ<ò 1aØd
|
ñ ùs "%)rc r r> )rB rN Ú
|
unfilteredrj Ústreams `` r# Ú
|
get_tokenszLexer.get_tokensÿ s= ù ð ×+Ñ+¨DÓ1õ ñ ÙÜ" 6¨4¯<©<¸Ó>FØ r& c ó t )aS
|
This method should process the text and return an iterable of
|
``(index, tokentype, value)`` tuples where ``index`` is the starting
|
position of the token within the input text.
|
|
It must be overridden by subclasses. It is recommended to
|
implement it as a generator to maximize effectiveness.
|
)ÚNotImplementedError)rB rN s r# rg z Lexer.get_tokens_unprocessed s
|
ô "Ð!r& )F)r1 r2 r3 r4 r. ÚaliasesÚ filenamesÚalias_filenamesÚ mimetypesÚpriorityÚurlÚ version_addedÚ_examplerD rI rA r* rc rm rg r! r& r# r r 1 sl ñ8ðv Dð Gð
|
Ið Oð Ið Hð Cð Mð Hò %ò<Bò %ò ò"/óbó0 "r& r )Ú metaclassc ó e Zd ZdZefdZd Zy)r a
|
This lexer takes two lexer as arguments. A root lexer and
|
a language lexer. First everything is scanned using the language
|
lexer, afterwards all ``Other`` tokens are lexed using the root
|
lexer.
|
|
The lexers from the ``template`` lexer package use this base lexer.
|
c ór |di |¤| _ |di |¤| _ || _ t j | fi |¤ y ©Nr! )Ú
|
root_lexerÚlanguage_lexerÚneedler rD )rB Ú _root_lexerÚ_language_lexerÚ_needler? s r# rD zDelegatingLexer.__init__- s9 Ù%Ñ0¨Ñ0Ù -Ñ 8°Ñ 8ÔØ Ü tÑ'wÓ'r& c ól d}g }g }| j j |« D ]N \ }}}|| j u r&|r |j t |« |f« g }||z };|j |||f« P |r |j t |« |f« t || j j |« « S )NÚ )r} rg r~ rL rW Ú do_insertionsr| )rB rN ÚbufferedÚ
|
insertionsÚ
|
lng_bufferÚirh ri s r# rg z&DelegatingLexer.get_tokens_unprocessed3 s¾ ØØ
|
Ø
|
Ø×*Ñ*×AÑAÀ$ÓGò -GAq!ØDKKÑ Ù Ø ×%Ñ%¤s¨8£}°jÐ&AÔBØ!#JØA à×!Ñ! 1 a¨ )Õ,ð -ñ Ø × Ñ s 8}¨jÐ 9Ô :Ü ZØ !__× CÑ CÀHÓ MóOð Or& N)r1 r2 r3 r4 r rD rg r! r& r# r r # s ñð >Có (ó Or& r c ó e Zd ZdZy)r zI
|
Indicates that a state should include rules from another state.
|
N©r1 r2 r3 r4 r! r& r# r r J s ñð r& r c ó e Zd ZdZd Zy)Ú_inheritzC
|
Indicates the a state should inherit from its superclass.
|
c ó y)Nr r! rH s r# rI z_inherit.__repr__U s Ør& N)r1 r2 r3 r4 rI r! r& r# r r Q s ñór& r c ó e Zd ZdZd Zd Zy)Úcombinedz:
|
Indicates a state combined from multiple states.
|
c ó. t j | |« S rf )Útupler, )ÚclsÚargss r# r, zcombined.__new__` s Ü}}S $Ó'Ð'r& c ó y rf r! )rB r s r# rD zcombined.__init__c s à r& N)r1 r2 r3 r4 r, rD r! r& r# r r [ s ñò(ó r& r c ó: e Zd ZdZd Zd dZd dZd dZd Zd Z y)
|
Ú _PseudoMatchz:
|
A pseudo match object constructed from a string.
|
c ó || _ || _ y rf )Ú_textÚ_start)rB ÚstartrN s r# rD z_PseudoMatch.__init__m s Ø
|
Ø r& Nc ó | j S rf )r ©rB Úargs r# r z_PseudoMatch.startq s Ø{{Ðr& c óF | j t | j « z S rf )r rW r r s r# Úendz_PseudoMatch.endt s Ø{{S §¡_Ñ,Ð,r& c ó4 |r t d« | j S )Nz No such group)Ú
|
IndexErrorr r s r# Úgroupz_PseudoMatch.groupw s Ù Ü _Ó-Ð -ØzzÐr& c ó | j fS rf )r rH s r# Úgroupsz_PseudoMatch.groups| s Ø
|
|
}Ð r& c ó i S rf r! rH s r# Ú groupdictz_PseudoMatch.groupdict s Ø r& rf )
|
r1 r2 r3 r4 rD r r r¢ r¤ r¦ r! r& r# r r h s% ñò óó-óò
|
ór& r c ó d fd }|S )zL
|
Callback that yields multiple actions for each group in the match.
|
c
|
3 ó¶ K t « D ]® \ }}| t |« t u r1|j |dz « }|s1|j |dz « ||f K|j |dz « }|b|r|j |dz « |_ || t |j |dz « |« |« D ] }|s| ° |r|j « |_ y y w)Né )Ú enumerater+ r
|
r¢ r Úposr r )ÚlexerÚmatchÚctxr ÚactionÚdataÚitemr s r# Úcallbackzbygroups.<locals>.callback sß øè ø Ü" 4 ò 'IAvØ~ØÜf ¤Ñ+Ø {{ 1 q¡5Ó)ÚØ ++ a¨!¡eÓ,¨f°dÐ:Ó:à {{ 1 q¡5Ó)ØÑ#ÙØ"'§+¡+¨a°!©eÓ"4Ù & uÜ'3°E·K±KÀÀAÁÓ4FÈÓ'MÈsó!Tò 'â Ø"&Jñ'ð 'ñ ØiikCGð ùs <CÁ 0CÁ1ACÂ8!Crf r! )r r² s ` r# r r s ø õ"ð& Or& c ó e Zd ZdZy)Ú_ThiszX
|
Special singleton used for indicating the caller class.
|
Used by ``using``.
|
Nr r! r& r# r´ r´ s òr& r´ c ó² i dv r4j d« }t |t t f« r|d< nd|fd< t u r dfd }|S d fd }|S )aÂ
|
Callback that processes the match with a different lexer.
|
|
The keyword arguments are forwarded to the lexer, except `state` which
|
is handled separately.
|
|
`state` specifies the state that the new lexer will start in, and can
|
be an enumerable such as ('root', 'inline', 'string') or a simple
|
string which is assumed to be on top of the root state.
|
|
Note: For that to work, `_other` must not be an `ExtendedRegexLexer`.
|
ÚstateÚstackÚrootc 3 ó* K r. j | j « | j di ¤}n| }|j « } |j |j « fi ¤D ] \ }}}||z ||f |r|j « |_ y y wr{ )Úupdater? rG r rg r¢ r r« )
|
r¬ r r® ÚlxÚsr rh ri Ú gt_kwargsÚkwargss
|
r# r² zusing.<locals>.callback¼ s øè ø ñ à emmÔ,Ø$U__Ñ. vÑ.àØ AØ42×4Ñ4°U·[±[³]ÑPÀiÑPò "1aØ!eQ kÓ!ð "áØ ))+ð ùs BBc 3 ó K
|
j | j « di
|
¤}|j « } |j |j « fi ¤D ] \ }}}||z ||f |r|j « |_ y y wr{ )rº r? r rg r¢ r r« ) r¬ r r® r» r¼ r rh ri Ú_otherr½ r¾ s r# r² zusing.<locals>.callbackË s øè ø à MM%--Ô (ÙÑ!&Ñ!Bà AØ42×4Ñ4°U·[±[³]ÑPÀiÑPò "1aØ!eQ kÓ!ð "áØ ))+ð ùs BBrf )ÚpoprK Úlistr r )rÀ r¾ r¼ r² r½ s `` @r# r r ¦ se ú ð IØ&ÑØ JJwÓ Ü a$¤ Ô 'Ø!"IgÒ à"(¨! IgÑ à ~ö &ð2 O÷ &ð Or& c ó e Zd ZdZd Zy)r zÑ
|
Indicates a state or state action (e.g. #pop) to apply.
|
For example default('#pop') is equivalent to ('', Token, '#pop')
|
Note that state tuples may be used as well.
|
|
.. versionadded:: 2.0
|
c ó || _ y rf )r¶ )rB r¶ s r# rD zdefault.__init__à s Ø
|
r& N)r1 r2 r3 r4 rD r! r& r# r r Ø s ñór& r c ó e Zd ZdZddZd Zy)r z
|
Indicates a list of literal words that is transformed into an optimized
|
regex that matches any of the words.
|
|
.. versionadded:: 2.0
|
c ó. || _ || _ || _ y rf )r ÚprefixÚsuffix)rB r rÇ rÈ s r# rD zwords.__init__ë s Ø
|
Ø Ø r& c óZ t | j | j | j ¬« S )N©rÇ rÈ )r r rÇ rÈ rH s r# r@ z words.getð s ܨD¯K©KÀÇ Á ÔLÐLr& N)r r )r1 r2 r3 r4 rD r@ r! r& r# r r ä s ñó ó
|
Mr& r c ó< e Zd ZdZd Zd Zd Zd Zd
|
dZd Z d Z
|
y) ÚRegexLexerMetazw
|
Metaclass for RegexLexer, creates the self._tokens attribute from
|
self.tokens on the first instantiation.
|
c ó t |t « r|j « }t j ||« j
|
S )zBPreprocess the regular expression component of a token definition.)rK r r@ ÚreÚcompiler )r ÚregexÚrflagsr¶ s r# Ú_process_regexz RegexLexerMeta._process_regexú s. ä eVÔ $ØIIKEÜzz% Ó(×.Ñ.Ð.r& c óR t |« t u st |« s
|
J d|« |S )z5Preprocess the token component of a token definition.z0token type must be simple type or callable, not )r+ r
|
Úcallable)r Útokens r# Ú_process_tokenz RegexLexerMeta._process_token s2 äE{jÑ(¬H°U¬Oð IØ>¸u¸iÐ Hó IÐ;à r& c óä t |t « r5|dk( ry||v r|fS |dk( r|S |dd dk( rt |dd « S J d|« t |t « rfd| j z }| xj d z c_ g }|D ]3 }||k7 s
|
J d
|
|« |j | j |||« « 5 |||< |fS t |t « r |D ] }||v r|d v r J d|z « |S J d |« ) z=Preprocess the state transition action of a token definition.ú#popéÿÿÿÿú#pushNé z#pop:zunknown new state z_tmp_%dr© zcircular state ref )rØ rÚ zunknown new state def )rK rS Úintr Ú_tmpnameÚextendÚ_process_stater )r Ú new_stateÚ unprocessedÚ processedÚ tmp_stateÚitokensÚistates r# Ú_process_new_statez!RegexLexerMeta._process_new_state sQ ä i¤Ô %àFÒ"ØØkÑ)Ø!|Ð#ØgÒ%Ø Ð Ø2A 'Ò)ÜI a bMÓ*Ð*Ð*à@Ð 2°9°-Ð @Ó@uÜ ¤8Ô ,à! C§L¡LÑ0IØ LLAÑ LØGØ#ò FØ Ò*ÐLÐ.AÀ&ÀÐ,LÓLÐ*Øs× 1Ñ 1°+Ø2;¸Vó Eõ Fð Fð $+IiÑ Ø <Ð Ü ¤5Ô )à#ò 2Ø +Ò-Ø Ð"3Ò3ð2à(¨6Ñ1ó2ð 4ð 2ð Ð à @Ð 2°9°-Ð@Ó @5r& c ó~ t |t « s
|
J d|« |d dk7 s
|
J d|« ||v r|| S g x}||< | j }|| D ]? }t |t « r;||k7 s
|
J d|« |j | j ||t |« « « Ot |t « r`t |t « rO| j |j ||« }|j t j d« j d|f« ¿t |« t u s
|
J d|« | j! |d ||« }| j' |d « }
|
t) |« dk( rd}n| j |d ||« }|j ||
|
|f« B |S # t" $ r } t% d |d d
|
|d | d | « | d} ~ ww xY w)z%Preprocess a single state definition.zwrong state name r ú#zinvalid state name zcircular state reference r Nzwrong rule def zuncompilable regex z
|
in state z of z: r© é )rK rS Úflagsr rÞ rß r r ræ r¶ rL rÎ rÏ r r+ r rÒ Ú ExceptionÚ
|
ValueErrorrÖ rW ) r rá râ r¶ ÚtokensrÑ Útdefrà ÚrexÚerrrÕ s r# rß z RegexLexerMeta._process_state) sü ä%¤Ô%ÐDÐ):¸5¸)Ð'DÓDÐ%ØQx3Ð?Ð"5°e°YÐ ?Ó?Ø IÑ Ø UÑ#Ð #Ø$&Ð&5Ñ!ØØ Ñ&ó 3DÜ$¤Ô(àu}ÐKÐ(AÀ%ÀÐ&KÓK}Ø c× 0Ñ 0°¸iÜ14°T³ó <ô =àÜ$¤Ô)ð Ü$¤Ô(Ø × 2Ñ 2°4·:±:¸{ÈIÓ V Ø rzz¨"~× 3Ñ 3°T¸9Ð EÔFØä:¤Ñ&Ð B¨/¸$¸Ð(BÓ BÐ&ð rØ×(Ñ(¨¨a©°&¸%Ó@ð ×&Ñ& t¨A¡wÓ/Eä4yA~Ø à × 2Ñ 2°4¸±7Ø3>À ó K ð MM3 yÐ1Ö 2ðA 3ðB øô ò rÜ Ð#6°t¸A±w°kÀÈEÈ9ÐTXÐY\ÐX_Ð_aÐbeÐafÐ!gÓhÐnqÐqûð rús Ä)FÆ F<Æ F7Æ7F<Nc ó i x}| j |< |xs | j | }t |« D ] }| j |||« |S )z-Preprocess a dictionary of token definitions.)Ú _all_tokensrí r rß )r r. Ú tokendefsrâ r¶ s r# Úprocess_tokendefz RegexLexerMeta.process_tokendefT sS à,.Ð. COO DÑ)Ø Ò1 §¡¨DÑ!1 Ü)_ò <EØ × Ñ y¨)°UÕ ;ð <àÐr& c ó² i }i }| j D ]¥ }|j j di « }|j « D ]t \ }}|j |« }|!|||< |j t
|
« }|||< :|j |d« }|O||||dz |j t
|
« } || z ||< v § |S # t $ r Y w xY w# t $ r Y w xY w)a
|
Merge tokens from superclasses in MRO order, returning a single tokendef
|
dictionary.
|
|
Any state that is not defined by a subclass will be inherited
|
automatically. States that *are* defined by subclasses will, by
|
default, override that state in the superclass. If a subclass wishes to
|
inherit definitions from a superclass, it can use the special value
|
"inherit", which will cause the superclass' state definition to be
|
included at that point in the state.
|
rí Nr© )Ú__mro__Ú__dict__r@ ÚitemsÚindexr rì rÁ )
|
r rí Ú inheritableÚcÚtoksr¶ rø ÚcuritemsÚ inherit_ndxÚ new_inh_ndxs
|
r# Ú get_tokendefsz RegexLexerMeta.get_tokendefs\ s ð Ø Øò CAØ::>> (¨BÓ/Dà $§
|
¡
|
£ ò C uØ!:: eÓ,ØÐ#ð
|
%*F5Mð!Ø&+§k¡k´'Ó&: ð *5K Ñ&Ø à )oo¨e°TÓ : Ø Ð&Ø ð 7< [°¡]Ð3ðCð #(§+¡+¬gÓ"6Kð *5°{Ñ)BK Ò&ñ9 Cð CðB øô) &ò !Ù ð!ûô "ò Ùðús$ Á B;ÂC
|
Â; CÃCÃ
|
CÃCc óè d| j vrLi | _ d| _ t | d« r | j rn%| j d| j « « | _ t j | g|¢i |¤S )z:Instantiate cls after preprocessing its token definitions.Ú_tokensr Útoken_variantsr )
|
r÷ rò rÝ Úhasattrr rô r r r+ Ú__call__)r r Úkwdss r# r zRegexLexerMeta.__call__ sh à CLLÑ (Ø COØ CLÜsÐ ,Ô-°#×2DÒ2Dàà !× 2Ñ 2°2°s×7HÑ7HÓ7JÓ K ä}}SÐ0 4Ò0¨4Ñ0Ð0r& rf ) r1 r2 r3 r4 rÒ rÖ ræ rß rô r r r! r& r# rÌ rÌ ô s. ñò
|
/ò ò !AòF)óVò/ób 1r& rÌ c ó4 e Zd ZdZej
|
Zi ZddZy)r z±
|
Base for simple stateful regular expression-based lexers.
|
Simplifies the lexing process so that you need only
|
provide a list of states and regular expressions.
|
c # ó: K d}| j }t |« }||d } |D ]& \ }}} |||« }
|
|
|
s|8t |« t u r|||
|
j « f n || |
|
« E d{ |
|
j « }| Çt | t « rX| D ]R } | dk( r t |« dkD s|j « (| dk( r|j |d « B|j | « T nWt | t « r#t | « t |« k\ r|dd
= n*|| d
= n$| dk( r|j |d « n
|
J d| « ||d } n7 || dk( rd g}|d }|t df |dz }P|t || f |dz }d7 # t $ r Y yw xY ww)
|
z~
|
Split ``text`` into (tokentype, text) pairs.
|
|
``stack`` is the initial stack (default: ``['root']``)
|
r rÙ r© NrØ rÚ úwrong state def: rQ r¸ )r r r+ r
|
r¢ r rK r rW rÁ rL rÜ Úabsr r r¡ ) rB rN r· r« ró Ú
|
statestackÚ statetokensÚrexmatchr¯ rà Úmr¶ s r# rg z!RegexLexer.get_tokens_unprocessed¾ sÜ è ø ð ØLL Ü%[
|
Ø
|
¨2¡Ñ/ ØØ/:ó 0 Ñ+& )Ù T 3Ó'ÚØ Ð)Ü <¬:Ñ5Ø"% v¨q¯w©w«yÐ"8Ó 8á'-¨d°A£× 6Ð 6Ø%%'CØ Ð,ä% i´Ô7Ø)2ò = Ø#(¨F¢?Ü'*¨:£¸Ó':Ø(2¯©Õ(8Ø%*¨gÒ%5Ø$.×$5Ñ$5°jÀ±nÕ$Eà$.×$5Ñ$5°eÕ$<ñ =ô (¨ ´3Ô 7ô # 9~´°Z³ Ò @Ø$.¨q©r¡Nà$.¨y©zÑ$:Ø &¨'Ò 1Ø &× -Ñ -¨j¸©nÕ =à KÐ,=¸i¸]Ð*KÓ K 5Ø&/°
|
¸2±Ñ&? ÙðC0 ðJ ØCy DÒ(à&, X
|
Ø&/°Ñ&7 Ø !¤:¨tÐ 3Ò3ØqÙ Ø u d¨3¡iÐ/Ò/Ø1HCñ_ ð 7úôP "ò ÙðüsM 8F»5FÁ0F Á1>FÂ0B!FÅ F Å2FÅ4F ÆFÆ FÆFÆFÆFN©)r¸ ) r1 r2 r3 r4 rÎ Ú MULTILINErê rí rg r! r& r# r r s ñð LLEð0 Fô;r& r c ó e Zd ZdZddZd Zy)r z9
|
A helper object that holds lexer position data.
|
Nc ó` || _ || _ |xs t |« | _ |xs dg| _ y )Nr¸ )rN r« rW r r· )rB rN r« r· r s r# rD zLexerContext.__init__ s. Ø ØØÒ##d)ØÒ&vh
|
r& c óV d| j d| j d| j dS )Nz LexerContext(z, ú))rN r« r· rH s r# rI zLexerContext.__repr__ s) Ø tyym¨2¨d¯h©h¨\¸¸D¿J¹J¸>ÈÐKÐKr& ©NN)r1 r2 r3 r4 rD rI r! r& r# r r ü s ñó'ó Lr& r c ó e Zd ZdZddZy)r zE
|
A RegexLexer that uses a context object to store its state.
|
Nc # ó K | j }|st |d« }|d }n |}||j d }|j } |D ]Î \ }}} |||j |j
|
« } | s)|lt |« t u r5|j || j « f | j « |_ n& || | |« E d{ |s||j d }|5t |t « r|D ] }
|
|
|
dk( r4t |j « dkD s!|j j « <|
|
dk( r)|j j |j d « j|j j |
|
« nt |t « rAt |« t |j « k\ r|j dd
= nH|j |d
= n8|dk( r)|j j |j d « n
|
J d|« ||j d } n |j |j
|
k\ ry||j d k( r9dg|_ |d }|j t d f |xj dz c_ ;|j t" ||j f |xj dz c_ s7 ó# t$ $ r Y yw xY ww)
|
z
|
Split ``text`` into (tokentype, text) pairs.
|
If ``context`` is given, use this lexer context instead.
|
r r¸ rÙ r© NrØ rÚ r rQ )r r r· rN r« r r+ r
|
r¢ rK r rW rÁ rL rÜ r
|
r r r¡ ) rB rN Úcontextró r® r r r¯ rà r r¶ s r# rg z)ExtendedRegexLexer.get_tokens_unprocessed sw è ø ð
|
LL ÙÜ t QÓ'CØ# FÑ+KàCØ# C§I¡I¨b¡MÑ2KØ88DØØ/:ó 2 Ñ+& )Ù T 3§7¡7¨C¯G©GÓ4ÚØ Ð)Ü <¬:Ñ5Ø"%§'¡'¨6°1·7±7³9Ð"<Ò <Ø&'§e¡e£gCGá'-¨d°A°sÓ';× ;Ð ;Ù#,à.7¸¿ ¹ À"¹ Ñ.F à Ñ,ä% i´Ô7Ø)2ò < Ø#(¨F¢?Ü'*¨3¯9©9£~¸Ó'9Ø(+¯ © ¯ © Ø%*¨gÒ%5Ø$'§I¡I×$4Ñ$4°S·Y±Y¸r±]Õ$Cà$'§I¡I×$4Ñ$4°UÕ$;ñ <ô (¨ ´3Ô 7ä " 9~´°S·Y±Y³ Ò ?Ø$'§I¡I¨a©b¡Mà$'§I¡I¨i©jÑ$9Ø &¨'Ò 1Ø II× ,Ñ ,¨S¯Y©Y°r©]Õ ;à KÐ,=¸i¸]Ð*KÓ K 5Ø&/°· ± ¸"± Ñ&> ÙðG2 ðJ Øww #§'¡'Ò)Ø ØCGG}¨Ò,à%+ H Ø&/°Ñ&7 Ø !gg¤t¨TÐ 1Ò1Ø 1 Ù Ø ''¤5¨$¨s¯w©w©-Ð7Ò7ØGGqLGñc ð <úôR "ò Ùðüs^ A,K Á/AK ÃJ7ÃA K ÄDK ÈJ: È2K È3A J: É<K É>7J: Ê5K Ê: KËK ËKËK r )r1 r2 r3 r4 rg r! r& r# r r s ñô@r& r c # ó K t | « } t | « \ }}d}d}|D ]° \ }}}||}d} |rx|t |« z |k\ rg|| ||z
|
}
|
|
|
r|||
|
f |t |
|
« z }|D ] \ } } } || | f |t | « z } ||z
|
} t | « \ }}|r|t |« z |k\ rg| t |« k s|||| d f |t |« | z
|
z }² |r9|xs d}|D ] \ }}}|||f |t |« z } t | « \ }}|r8yy# t $ r |E d{ 7 Y yw xY w# t $ r d}Y w xY w# t $ r d}Y yw xY ww)ag
|
Helper for lexers which must combine the results of several
|
sublexers.
|
|
``insertions`` is a list of ``(index, itokens)`` pairs.
|
Each ``itokens`` iterable should be inserted at position
|
``index`` into the token stream given by the ``tokens``
|
argument.
|
|
The result is a combined token stream.
|
|
TODO: clean up the code here.
|
NTr F)ÚiterÚnextÚ StopIterationrW )r rí rù rä ÚrealposÚinsleftr rh ri ÚoldiÚtmpvalÚit_indexÚit_tokenÚit_valueÚps r# r r S sÎ è ø ô jÓ!JðÜ jÓ)wð GØGð ò %1aà ?ØGØÙ!c !f*¨Ò-ØtE AIÐ&FÙØ q &Ð(Ò(Ø3v;Ñ&Ø07ò )Ñ,( HØ x¨Ð1Ò1Ø3x=Ñ(ð )ð 19Dð Ü!% jÓ!1wñ !c !f*¨Ó-ð #a&=Ø1a hÐ&Ò &Ø s1v }Ñ $Gð+%ñ0 à,QØ ò GAq!Ø1a-Ò Ø s1vÑ Gð ð Ü ! *Ó -NE7ô øôE ò à×ÑÙðûô4 !ò Ø Ùð ûô ò ØGÙ ð üs ED A*EÂD,ÂEÂ*EÂ9AEÃ?D= Ä EÄEÄD)Ä D#Ä!D)Ä&EÄ(D)Ä)EÄ, D:Ä7EÄ9D:Ä:EÄ= E ÅEÅ
|
E Å Ec ó e Zd ZdZd Zy)ÚProfilingRegexLexerMetaz>Metaclass for ProfilingRegexLexer, collects regex timing info.c óè t |t « r-t |j |j |j ¬« n|t j |« t j f fd }|S )NrÊ c óþ j d j
|
fddg« }t j « }j | ||« }t j « }|dxx dz cc< |dxx ||z
|
z cc< |S )NrÙ r r r© )Ú
|
_prof_dataÚ
|
setdefaultÚtimer ) rN r« ÚendposÚinfoÚt0ÚresÚt1r Úcompiledrï r¶ s r# Ú
|
match_funcz:ProfilingRegexLexerMeta._process_regex.<locals>.match_func sr ø Ø>> "Ñ%×0Ñ0°%¸° ÀÀ3¸xÓHDÜBØ.. s¨FÓ3CÜBØ GqLGØ GrBwÑ GØJr& ) rK r r rÇ rÈ rÎ rÏ ÚsysÚmaxsize)r rÐ rÑ r¶ r2 r1 rï s ` ` @@r# rÒ z&ProfilingRegexLexerMeta._process_regex sZ û Ü eUÔ #ÜEKK°· ± Ø#(§<¡<ô1Cð CÜ::c 6Ó*ä),¯©÷ ð ð Ðr& N)r1 r2 r3 r4 rÒ r! r& r# r& r& s
|
ÙHór& r& c ó e Zd ZdZg ZdZddZy)ÚProfilingRegexLexerzFDrop-in replacement for RegexLexer that does profiling of its regexes.é c # óJ K j j j i « t j ||« E d { j j j « }t d |j « D « fdd¬« }t d |D « « }t « t d j j t |« |fz « t d« t dd z « t d
|
« |D ] }t d |z « t d« y 7 Ýw) Nc 3 ó¤ K | ]H \ \ }}\ }}|t |« j d « j dd« dd |d|z d|z |z f J yw)zu'z\\ú\NéA iè )Úreprr[ rZ )Ú.0r¼ ÚrÚnrh s r# ú <genexpr>z=ProfilingRegexLexer.get_tokens_unprocessed.<locals>.<genexpr>´ sa è ø ò @á+FQ ¡F Q¨ð 4 7==¨Ó/×7Ñ7¸ÀÓEÀcÀrÐJØ4 !8 T¨A¡X°¡\ô3ñ @ùs AAc ó" | j S rf )Ú_prof_sort_index)r" rB s r# r$ z<ProfilingRegexLexer.get_tokens_unprocessed.<locals>.<lambda>· s ø A d×&;Ñ&;Ñ$< r& T)ÚkeyÚreversec 3 ó& K | ] }|d yw)é Nr! )r= r" s r# r@ z=ProfilingRegexLexer.get_tokens_unprocessed.<locals>.<genexpr>¹ s è ø Ò+ !Ñ+ùs z2Profiling result for %s lexing %d chars in %.3f mszn==============================================================================================================z$%-20s %-64s ncalls tottime percall)r¶ rÐ zn--------------------------------------------------------------------------------------------------------------z%-20s %-65s %5d %8.4f %8.4f) rG r) rL r rg rÁ Úsortedrø ÚsumÚprintr1 rW )rB rN r· Úrawdatar° Ú sum_totalr0 s ` r# rg z*ProfilingRegexLexer.get_tokens_unprocessed¯ sÿ øè ø à ×!Ñ!×(Ñ(¨Ô,Ü ×4Ñ4°T¸4ÀÓG×GÐGØ..×+Ñ+×/Ñ/Ó1Üñ @à/6¯}©}«ô@ó =Ø "ô $ô
|
Ñ+ dÔ+Ó+ ä Ü ÐBØ~~×&Ñ&¬¨D« °9Ð=ñ>ô ?ä iÔÜ Ð4Ð7IÑIÔJÜ iÔØò 5AÜ Ð/°!Ñ3Õ 4ð 5ä iÕð# Hús A D#ÁD!ÁC D#Nr )r1 r2 r3 r4 r) rB rg r! r& r# r6 r6 © s ÙPàJØÐôr& r6 )6r4 rÎ r3 r+ Úpip._vendor.pygments.filterr r Ú pip._vendor.pygments.filtersr Úpip._vendor.pygments.tokenr r r r r
|
Úpip._vendor.pygments.utilr r r r r r Ú pip._vendor.pygments.regexoptr Ú__all__rÏ r rU Ú staticmethodÚ_default_analyser+ r( r r rS r r r r r r r r´ r r r r rÌ r r r r r& r6 r! r& r# ú<module>rT sL ðñó
|
Û
|
Û ç =Ý ;ß QÕ Q÷*÷ *å 3ò *ð "**WÓ
|
ò, ñ ¡ Ó.Ð ô 1ô 1ôo"iõ o"ôd Oeô OôN cô ÷ñ ñ *ô
|
uô
|
÷ñ ò6÷4ñ ñ wò/÷d ñ ô MFô Mô d1Yô d1ôN^ .õ ^÷B Lñ Lô Eô EòP=ô@nô ô,*Ð0Gö r&
|
