Copy page

• Copy page
• View source

🖍️ highlighting

Editor-grade syntax highlighting for the Python code blocks of the generated documentation.

Two pieces are needed, and neither works without the other: styles supplies the VS Code palettes, and semantics supplies the token distinctions those palettes expect but that Pygments’ Python lexer does not make on its own.

register() wires both into Sphinx and must run before the first page is highlighted, which is why the generated conf.py calls it from its setup hook.

register_styles() → None

Make the two palettes resolvable by name, the way pygments.styles.get_style_by_name() expects.

The pygments.styles entry points declared in pyproject.toml are the real mechanism, and the only one a consumer other than this generator will see. Filling the lookup tables directly costs two lines on top, and keeps a build working from a checkout whose installed metadata predates those entry points.

register_lexers() → None

Install Python lexers carrying VSCodeSemanticFilter into Sphinx’s lexer table.

Sphinx narrows every Python language alias down to python, or to pycon when the block opens on >>>, then hands back any lexer registered under that name untouched. Filtering both therefore covers plain code blocks and doctests alike, and building them from Sphinx’s own lexer_classes keeps the options it would have applied, stripnl included.

register() → None

Register both the palettes and the filtered lexers.

semantics

Recovery of the token distinctions Pygments’ Python lexer does not make.

An editor colours task: str = "all" in three different ways because its grammar knows that task is a variable, str a type and "all" a string. Pygments only knows the third: it tags an identifier as Name.Function or Name.Class when a literal def or class introduces it, and emits a bare Name for every call, annotation, attribute and argument.

This module restores the missing distinctions as a stream filter rather than as a lexer subclass, because a filter also reaches the Python nested inside a doctest block, which PythonConsoleLexer lexes with its own instance. Being purely lexical, it follows the same conventions an editor’s grammar uses before a type checker weighs in: a name followed by a parenthesis is a call, and a CamelCase name is a type.

TYPE_BUILTINS: frozenset[str] = frozenset({'bool', 'bytearray', 'bytes', 'classmethod', 'complex', 'dict', 'enumerate', 'filter', 'float', 'frozenset', 'int', 'list', 'map', 'memoryview', 'object', 'property', 'range', 'reversed', 'set', 'slice', 'staticmethod', 'str', 'super', 'tuple', 'type', 'zip'})

Builtins Python implements as classes, which an editor colours as types while print or len stay functions.

DECLARATION_KEYWORDS: frozenset[str] = frozenset({'class', 'def', 'lambda'})

Keywords that introduce a binding, coloured apart from control flow the way def differs from return.

CAMEL_CASE: Pattern[str] = re.compile('_{0,2}[A-Z][A-Za-z0-9_]*[a-z][A-Za-z0-9_]*')

A leading capital plus a lowercase somewhere after it, which keeps DataContext apart from TINY_DEBUG.

class VSCodeSemanticFilter(**options)

Bases: Filter

Refine Name and Keyword tokens so a palette can colour calls, types and declarations apart.

>>> from pygments.lexers.python import PythonLexer
>>> lexer = PythonLexer()
>>> lexer.add_filter(VSCodeSemanticFilter())
>>> for token, text in lexer.get_tokens("ctx = DataContext(load(x))"):
...     if text.strip():
...         print(f"{text:<12} {token}")
ctx          Token.Name
=            Token.Operator
DataContext  Token.Name.Class
(            Token.Punctuation
load         Token.Name.Function
(            Token.Punctuation
x            Token.Name
)            Token.Punctuation
)            Token.Punctuation

retype(

ttype: _TokenType,

value: str,

) → tuple[_TokenType, str]

Refine a token that needs no lookahead to classify.

Parameters:

• ttype – Token type the lexer produced

• value – Text the token covers

Returns:

The token, refined when it deserves it

>>> from pygments.token import Keyword, Name
>>> VSCodeSemanticFilter().retype(Name.Builtin, "str")
(Token.Name.Class, 'str')
>>> VSCodeSemanticFilter().retype(Name.Builtin, "print")
(Token.Name.Builtin, 'print')
>>> VSCodeSemanticFilter().retype(Keyword, "def")
(Token.Keyword.Declaration, 'def')

classify_name(

value: str,

following: str,

) → _TokenType

Decide what a bare Name really is, given the next significant text.

A type wins over a call so that DataContext(ctx) reads as a constructor rather than as a function.

Parameters:

• value – The identifier itself

• following – Text of the next non-whitespace token, empty at the end of the stream

Returns:

The refined token type

>>> VSCodeSemanticFilter().classify_name("DataContext", "(")
Token.Name.Class
>>> VSCodeSemanticFilter().classify_name("load_split", "(")
Token.Name.Function
>>> VSCodeSemanticFilter().classify_name("task", "=")
Token.Name

filter(

lexer: Lexer | None,

stream: Iterable[tuple[_TokenType, str]],

) → Iterator[tuple[_TokenType, str]]

Rewrite the token stream, holding each Name back until the next significant token is known.

Parameters:

• lexer – Lexer that produced the stream, unused

• stream – Pairs of token type and text

Returns:

The rewritten pairs

styles

Pygments transcriptions of the two default VS Code themes, Dark+ and Light+.

The colours are the ones the editor ships, so a snippet in the documentation matches the same snippet in the reader’s editor: types teal, functions yellow, variables light blue, strings orange, control flow purple. They only pay off together with VSCodeSemanticFilter, which is what tells a call apart from a variable.

class VSCodeDarkPlusStyle

Bases: Style

The VS Code “Dark Modern” / Dark+ palette.

name = 'vscode-dark-plus'

background_color = '#1F1F1F'

overall background color (None means transparent)

highlight_color = '#264F78'

highlight background color

line_number_color = '#6E7681'

line number font color

line_number_background_color = '#1F1F1F'

line number background color

styles = {(): '#D4D4D4', ('Comment',): 'italic #6A9955', ('Comment', 'Hashbang'): '', ('Comment', 'Multiline'): '', ('Comment', 'Preproc'): '#569CD6', ('Comment', 'PreprocFile'): '', ('Comment', 'Single'): '', ('Comment', 'Special'): '', ('Error',): '#F14C4C', ('Escape',): '', ('Generic',): '', ('Generic', 'Deleted'): '#F14C4C', ('Generic', 'Emph'): 'italic', ('Generic', 'EmphStrong'): '', ('Generic', 'Error'): '#F14C4C', ('Generic', 'Heading'): 'bold #569CD6', ('Generic', 'Inserted'): '#6A9955', ('Generic', 'Output'): '#CCCCCC', ('Generic', 'Prompt'): '#6E7681', ('Generic', 'Strong'): 'bold', ('Generic', 'Subheading'): 'bold #569CD6', ('Generic', 'Traceback'): '#F14C4C', ('Keyword',): '#C586C0', ('Keyword', 'Constant'): '#569CD6', ('Keyword', 'Declaration'): '#569CD6', ('Keyword', 'Namespace'): '#C586C0', ('Keyword', 'Pseudo'): '', ('Keyword', 'Reserved'): '', ('Keyword', 'Type'): '#4EC9B0', ('Literal',): '', ('Literal', 'Date'): '', ('Literal', 'Number'): '#B5CEA8', ('Literal', 'Number', 'Bin'): '', ('Literal', 'Number', 'Float'): '', ('Literal', 'Number', 'Hex'): '', ('Literal', 'Number', 'Integer'): '', ('Literal', 'Number', 'Integer', 'Long'): '', ('Literal', 'Number', 'Oct'): '', ('Literal', 'String'): '#CE9178', ('Literal', 'String', 'Affix'): '#569CD6', ('Literal', 'String', 'Backtick'): '', ('Literal', 'String', 'Char'): '', ('Literal', 'String', 'Delimiter'): '', ('Literal', 'String', 'Doc'): '#CE9178', ('Literal', 'String', 'Double'): '', ('Literal', 'String', 'Escape'): '#D7BA7D', ('Literal', 'String', 'Heredoc'): '', ('Literal', 'String', 'Interpol'): '#569CD6', ('Literal', 'String', 'Other'): '', ('Literal', 'String', 'Regex'): '#D16969', ('Literal', 'String', 'Single'): '', ('Literal', 'String', 'Symbol'): '', ('Name',): '#9CDCFE', ('Name', 'Attribute'): '#9CDCFE', ('Name', 'Builtin'): '#DCDCAA', ('Name', 'Builtin', 'Pseudo'): '#9CDCFE', ('Name', 'Class'): '#4EC9B0', ('Name', 'Constant'): '#9CDCFE', ('Name', 'Decorator'): '#DCDCAA', ('Name', 'Entity'): '#9CDCFE', ('Name', 'Exception'): '#4EC9B0', ('Name', 'Function'): '#DCDCAA', ('Name', 'Function', 'Magic'): '#DCDCAA', ('Name', 'Label'): '#C8C8C8', ('Name', 'Namespace'): '#4EC9B0', ('Name', 'Other'): '', ('Name', 'Property'): '', ('Name', 'Tag'): '#569CD6', ('Name', 'Variable'): '#9CDCFE', ('Name', 'Variable', 'Class'): '', ('Name', 'Variable', 'Global'): '', ('Name', 'Variable', 'Instance'): '', ('Name', 'Variable', 'Magic'): '#9CDCFE', ('Operator',): '#D4D4D4', ('Operator', 'Word'): '#569CD6', ('Other',): '', ('Punctuation',): '#D4D4D4', ('Punctuation', 'Marker'): '', ('Text',): '#D4D4D4', ('Text', 'Whitespace'): '#D4D4D4'}

Style definitions for individual token types.

class VSCodeLightPlusStyle

Bases: Style

The VS Code “Light Modern” / Light+ palette, mapped token for token onto its dark sibling.

name = 'vscode-light-plus'

background_color = '#FFFFFF'

overall background color (None means transparent)

highlight_color = '#ADD6FF'

highlight background color

line_number_color = '#6E7681'

line number font color

line_number_background_color = '#FFFFFF'

line number background color

styles = {(): '#000000', ('Comment',): 'italic #008000', ('Comment', 'Hashbang'): '', ('Comment', 'Multiline'): '', ('Comment', 'Preproc'): '#0000FF', ('Comment', 'PreprocFile'): '', ('Comment', 'Single'): '', ('Comment', 'Special'): '', ('Error',): '#CD3131', ('Escape',): '', ('Generic',): '', ('Generic', 'Deleted'): '#CD3131', ('Generic', 'Emph'): 'italic', ('Generic', 'EmphStrong'): '', ('Generic', 'Error'): '#CD3131', ('Generic', 'Heading'): 'bold #0000FF', ('Generic', 'Inserted'): '#008000', ('Generic', 'Output'): '#3B3B3B', ('Generic', 'Prompt'): '#6E7681', ('Generic', 'Strong'): 'bold', ('Generic', 'Subheading'): 'bold #0000FF', ('Generic', 'Traceback'): '#CD3131', ('Keyword',): '#AF00DB', ('Keyword', 'Constant'): '#0000FF', ('Keyword', 'Declaration'): '#0000FF', ('Keyword', 'Namespace'): '#AF00DB', ('Keyword', 'Pseudo'): '', ('Keyword', 'Reserved'): '', ('Keyword', 'Type'): '#267F99', ('Literal',): '', ('Literal', 'Date'): '', ('Literal', 'Number'): '#098658', ('Literal', 'Number', 'Bin'): '', ('Literal', 'Number', 'Float'): '', ('Literal', 'Number', 'Hex'): '', ('Literal', 'Number', 'Integer'): '', ('Literal', 'Number', 'Integer', 'Long'): '', ('Literal', 'Number', 'Oct'): '', ('Literal', 'String'): '#A31515', ('Literal', 'String', 'Affix'): '#0000FF', ('Literal', 'String', 'Backtick'): '', ('Literal', 'String', 'Char'): '', ('Literal', 'String', 'Delimiter'): '', ('Literal', 'String', 'Doc'): '#A31515', ('Literal', 'String', 'Double'): '', ('Literal', 'String', 'Escape'): '#EE0000', ('Literal', 'String', 'Heredoc'): '', ('Literal', 'String', 'Interpol'): '#0000FF', ('Literal', 'String', 'Other'): '', ('Literal', 'String', 'Regex'): '#811F3F', ('Literal', 'String', 'Single'): '', ('Literal', 'String', 'Symbol'): '', ('Name',): '#001080', ('Name', 'Attribute'): '#001080', ('Name', 'Builtin'): '#795E26', ('Name', 'Builtin', 'Pseudo'): '#001080', ('Name', 'Class'): '#267F99', ('Name', 'Constant'): '#001080', ('Name', 'Decorator'): '#795E26', ('Name', 'Entity'): '#001080', ('Name', 'Exception'): '#267F99', ('Name', 'Function'): '#795E26', ('Name', 'Function', 'Magic'): '#795E26', ('Name', 'Label'): '#3B3B3B', ('Name', 'Namespace'): '#267F99', ('Name', 'Other'): '', ('Name', 'Property'): '', ('Name', 'Tag'): '#800000', ('Name', 'Variable'): '#001080', ('Name', 'Variable', 'Class'): '', ('Name', 'Variable', 'Global'): '', ('Name', 'Variable', 'Instance'): '', ('Name', 'Variable', 'Magic'): '#001080', ('Operator',): '#000000', ('Operator', 'Word'): '#0000FF', ('Other',): '', ('Punctuation',): '#000000', ('Punctuation', 'Marker'): '', ('Text',): '#000000', ('Text', 'Whitespace'): '#000000'}

Style definitions for individual token types.