stouputils.applications.automatic_docs.sphinx.highlighting.semantics module#

Recovery of the token distinctions Pygments’ Python lexer does not make.

An editor colours task: str = "all" in three different ways because its grammar knows that task is a variable, str a type and "all" a string. Pygments only knows the third: it tags an identifier as Name.Function or Name.Class when a literal def or class introduces it, and emits a bare Name for every call, annotation, attribute and argument.

This module restores the missing distinctions as a stream filter rather than as a lexer subclass, because a filter also reaches the Python nested inside a doctest block, which PythonConsoleLexer lexes with its own instance. Being purely lexical, it follows the same conventions an editor’s grammar uses before a type checker weighs in: a name followed by a parenthesis is a call, and a CamelCase name is a type.

TYPE_BUILTINS: frozenset[str] = frozenset({'bool', 'bytearray', 'bytes', 'classmethod', 'complex', 'dict', 'enumerate', 'filter', 'float', 'frozenset', 'int', 'list', 'map', 'memoryview', 'object', 'property', 'range', 'reversed', 'set', 'slice', 'staticmethod', 'str', 'super', 'tuple', 'type', 'zip'})[source]#

Builtins Python implements as classes, which an editor colours as types while print or len stay functions.

DECLARATION_KEYWORDS: frozenset[str] = frozenset({'class', 'def', 'lambda'})[source]#

Keywords that introduce a binding, coloured apart from control flow the way def differs from return.

CAMEL_CASE: Pattern[str] = re.compile('_{0,2}[A-Z][A-Za-z0-9_]*[a-z][A-Za-z0-9_]*')[source]#

A leading capital plus a lowercase somewhere after it, which keeps DataContext apart from TINY_DEBUG.

class VSCodeSemanticFilter(**options)[source]#

Bases: Filter

Refine Name and Keyword tokens so a palette can colour calls, types and declarations apart.

Examples

>>> from pygments.lexers.python import PythonLexer
>>> lexer = PythonLexer()
>>> lexer.add_filter(VSCodeSemanticFilter())
>>> for token, text in lexer.get_tokens("ctx = DataContext(load(x))"):
...     if text.strip():
...         print(f"{text:<12} {token}")
ctx          Token.Name
=            Token.Operator
DataContext  Token.Name.Class
(            Token.Punctuation
load         Token.Name.Function
(            Token.Punctuation
x            Token.Name
)            Token.Punctuation
)            Token.Punctuation
retype(
ttype: _TokenType,
value: str,
) → tuple[_TokenType, str][source]#

Refine a token that needs no lookahead to classify.

Parameters:

  • ttype (TokenType) – Token type the lexer produced

  • value (str) – Text the token covers

Returns:

The token, refined when it deserves it

Return type:

tuple[TokenType, str]

Examples

>>> from pygments.token import Keyword, Name
>>> VSCodeSemanticFilter().retype(Name.Builtin, "str")
(Token.Name.Class, 'str')
>>> VSCodeSemanticFilter().retype(Name.Builtin, "print")
(Token.Name.Builtin, 'print')
>>> VSCodeSemanticFilter().retype(Keyword, "def")
(Token.Keyword.Declaration, 'def')
classify_name(
value: str,
following: str,
) → _TokenType[source]#

Decide what a bare Name really is, given the next significant text.

A type wins over a call so that DataContext(ctx) reads as a constructor rather than as a function.

Parameters:

  • value (str) – The identifier itself

  • following (str) – Text of the next non-whitespace token, empty at the end of the stream

Returns:

The refined token type

Return type:

TokenType

Examples

>>> VSCodeSemanticFilter().classify_name("DataContext", "(")
Token.Name.Class
>>> VSCodeSemanticFilter().classify_name("load_split", "(")
Token.Name.Function
>>> VSCodeSemanticFilter().classify_name("task", "=")
Token.Name
filter(
lexer: Lexer | None,
stream: Iterable[tuple[_TokenType, str]],
) → Iterator[tuple[_TokenType, str]][source]#

Rewrite the token stream, holding each Name back until the next significant token is known.

Parameters:

  • lexer (Lexer | None) – Lexer that produced the stream, unused

  • stream (Iterable) – Pairs of token type and text

Returns:

The rewritten pairs

Return type:

Iterator[tuple[TokenType, str]]