Custom Pygment Lexers

Update: 2025-08-07

//# was changed to //# + #//, flanking the critic markup so that it doesn't consume the whole line like the regex in screenshots does. For the latest version look up: https://github.com/vectorspacexyz/pymdown-lexers

  • The Problem

    1754425876.png

    1754426527.png

  • The Solution

    1754424658.png

    1754424696.png

Here in the markdown tab note that I'm trying to highlight widget.count++ by surrounding it in critic markup . The issue I encounter is highlighted in the result tab. I also noted while trying to debug the problem that if the critic markup is inside a comment, it renders correctly. This gave me a high degree of certainty that it was something that was happening with pygment syntax highlighting that was interfering with critic.

The solution i went with is creating a new lexer dart-critic, where a new syntax is added //# is used. This dart-critic is built on top of the existing dart lexer in pygments.

The solution implementation can be found here: https://github.com/vectorspacexyz/pymdown-lexers

To develop the solution I first played around with the file that contained the original Dart lexer directly, I found out that the dart lexer was in javascript.py by searching for "dart" in repo in https://github.com/pygments/pygments. Since I used arch:

Get all lexers

You can get all the pygment lexers installed on your system with: 

python -c "
from pygments.lexers import get_all_lexers
for name, aliases, filenames, mimetypes in get_all_lexers():
    print(f'{name}: {aliases}')
"


vector@ThinkPadP14s ~/admin/pymdown-lexers $ pacman -Ql python-pygments | grep javascript
python-pygments /usr/lib/python3.13/site-packages/pygments/lexers/__pycache__/javascript.cpython-313.opt-1.pyc
python-pygments /usr/lib/python3.13/site-packages/pygments/lexers/__pycache__/javascript.cpython-313.pyc
python-pygments /usr/lib/python3.13/site-packages/pygments/lexers/javascript.py

Just had to add a single loc at the javascript.py file (this is just for testing, please note any pacman upgrade might rewrite this file back to its old state):

/usr/lib/python3.13/site-packages/pygments/lexers/javascript.py
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
class DartLexer(RegexLexer):
    """
    For Dart source code.
    """

    name = 'Dart'
    url = 'http://dart.dev/'
    aliases = ['dart']
    filenames = ['*.dart']
    mimetypes = ['text/x-dart']
    version_added = '1.5'

    flags = re.MULTILINE | re.DOTALL

    tokens = {
        'root': [
            include('string_literal'),
            (r'#!(.*?)$', Comment.Preproc),
            (r'\b(import|export)\b', Keyword, 'import_decl'),
            (r'\b(library|source|part of|part)\b', Keyword),
            (r'[^\S\n]+', Whitespace),
            (r'(//.*?)(\n)', bygroups(Comment.Single, Whitespace)),
            (r'//#\s*(.*?)(\n)', bygroups(Text, Whitespace)),
            (r'/\*.*?\*/', Comment.Multiline),
            (r'\b(class|extension|mixin)\b(\s+)',
             bygroups(Keyword.Declaration, Whitespace), 'class'),
            (r'\b(as|assert|break|case|catch|const|continue|default|do|else|finally|'
             r'for|if|in|is|new|rethrow|return|super|switch|this|throw|try|while)\b',
             Keyword),
            (r'\b(abstract|async|await|const|covariant|extends|external|factory|final|'
             r'get|implements|late|native|on|operator|required|set|static|sync|typedef|'
             r'var|with|yield)\b', Keyword.Declaration),
            (r'\b(bool|double|dynamic|int|num|Function|Never|Null|Object|String|void)\b',
             Keyword.Type),
            (r'\b(false|null|true)\b', Keyword.Constant),
            (r'[~!%^&*+=|?:<>/-]|as\b', Operator),
            (r'@[a-zA-Z_$]\w*', Name.Decorator),
            (r'[a-zA-Z_$]\w*:', Name.Label),
            (r'[a-zA-Z_$]\w*', Name),
            (r'[(){}\[\],.;]', Punctuation),
            (r'0[xX][0-9a-fA-F]+', Number.Hex),
            # DIGIT+ (‘.’ DIGIT*)? EXPONENT?
            (r'\d+(\.\d*)?([eE][+-]?\d+)?', Number),
            (r'\.\d+([eE][+-]?\d+)?', Number),  # ‘.’ DIGIT+ EXPONENT?
            (r'\n', Whitespace)
            # pseudo-keyword negate intentionally left out
        ],
    ...

This solution isn't perfect as there are more lexers with this same issue, or maybe lexers are not the real issue at all the critic implementation is.

I also need to find some way to edit system python install on the fly: pacman the SOURCE in PKGBUILDs to be a directory. As such, I was committing and pushing to the repo and redoing makepkg -Si over and over again before reaching a state that works. This also made me reflect on whether mkdocs and material-theme should be a system install at all, should I have used venv instead?


Comments