Valid Tag Names

The tag name can be any undotted name that isn’t already an existing valid string or bytes prefix, as seen in the lexical analysis specification. Therefore these prefixes can’t be used as a tag:

stringprefix: "r" | "u" | "R" | "U" | "f" | "F"
            : | "fr" | "Fr" | "fR" | "FR" | "rf" | "rF" | "Rf" | "RF"

bytesprefix: "b" | "B" | "br" | "Br" | "bR" | "BR" | "rb" | "rB" | "Rb" | "RB"

Python restricts certain keywords from being used as identifiers. This restriction also applies to tag names. Usage of keywords should trigger a helpful error, as done in recent CPython releases.

Tags Must Immediately Precede the Quote Mark

As with other string literal prefixes, no whitespace can be between the tag and the quote mark.

PEP 701

Tag strings support the full syntax of PEP 701 in that any string literal, with any quote mark, can be nested in the interpolation. This nesting includes of course tag strings.

Evaluating Tag Strings

When the tag string is evaluated, the tag must have a binding, or a NameError is raised; and it must be a callable, or a TypeError is raised. The callable must accept a sequence of positional arguments. This behavior follows from the de-sugaring of:

trade = 'shrubberies'
mytag'Did you say "{trade}"?'

to:

mytag(DecodedConcrete(r'Did you say "'), InterpolationConcrete(lambda: trade, 'trade', None, None), DecodedConcrete(r'"?'))

Decoded Strings

In the mytag'Did you say "{trade}"?' example, there are two strings: r'Did you say "' and r'"?'.

Strings are internally stored as objects with a Decoded structure, meaning: conforming to a protocol Decoded:

@runtime_checkable
class Decoded(Protocol):
    def __str__(self) -> str:
        ...

    raw: str

These Decoded objects have access to raw strings. Raw strings are used because tag strings are meant to target a variety of DSLs, such as the shell and regexes. Such DSLs have their own specific treatment of metacharacters, namely the backslash.

However, often the “cooked” string is what is needed, by decoding the string as if it were a standard Python string. In the proposed implementation, the decoded object’s __new__ will store the raw string and store and return the “cooked” string.

The protocol is marked as @runtime_checkable to allow structural pattern matching to test against the protocol instead of a type. This can incur a small performance penalty. Since the case tests are in user-code tag functions, authors can choose to optimize by testing for the implementation type discussed next.

The Decoded protocol will be available from typing. In CPython, Decoded will be implemented in C, but for discussion of this PEP, the following is a compatible implementation:

class DecodedConcrete(str):
    _raw: str

    def __new__(cls, raw: str):
        decoded = raw.encode("utf-8").decode("unicode-escape")
        if decoded == raw:
            decoded = raw
        chunk = super().__new__(cls, decoded)
        chunk._raw = raw
        return chunk

    @property
    def raw(self):
        return self._raw

Interpolation

An Interpolation is the data structure representing an expression inside the tag string. Interpolations enable a delayed evaluation model, where the interpolation expression is computed, transformed, memoized, or processed in any way.

In addition, the original text of the interpolation expression is made available to the tag function. This can be useful for debugging or metaprogramming.

Interpolation is a Protocol which will be made available from typing. It has the following definition:

@runtime_checkable
class Interpolation(Protocol):
    def __len__(self):
        ...

    def __getitem__(self, index: int):
        ...

    def getvalue(self) -> Callable[[], Any]:
        ...

    expr: str
    conv: Literal["a", "r", "s"] | None
    format_spec: str | None

Given this example interpolation:

mytag'{trade!r:some-formatspec}'

these attributes are as follows:

• getvalue is a zero argument closure for the interpolation. In this case, lambda: trade.
• expr is the expression text of the interpolation. Example: 'trade'.
• conv is the optional conversion to be used by the tag function, one of r, s, and a, corresponding to repr, str, and ascii conversions. Note that as with f-strings, no other conversions are supported. Example: 'r'.
• format_spec is the optional format_spec string. A format_spec is eagerly evaluated if it contains any expressions before being passed to the tag function. Example: 'some-formatspec'.

In all cases, the tag function determines what to do with valid Interpolation attributes.

In the CPython reference implementation, implementing Interpolation in C would use the equivalent Struct Sequence Objects (see such code as os.stat_result). For purposes of this PEP, here is an example of a pure Python implementation:

class InterpolationConcrete(NamedTuple):
    getvalue: Callable[[], Any]
    expr: str
    conv: Literal['a', 'r', 's'] | None = None
    format_spec: str | None = None

Interpolation Expression Evaluation

Expression evaluation for interpolations is the same as in PEP 498, except that all expressions are always implicitly wrapped with a lambda:

The expressions that are extracted from the string are evaluated in the context where the tag string appeared. This means the expression has full access to its lexical scope, including local and global variables. Any valid Python expression can be used, including function and method calls.

However, there’s one additional nuance to consider, function scope versus annotation scope. Consider this somewhat contrived example to configure captions:

class CaptionConfig:
    tag = 'b'
    figure = f'<{tag}>Figure</{tag}>'

Let’s now attempt to rewrite the above example to use tag strings:

class CaptionConfig:
    tag = 'b'
    figure = html'<{tag}>Figure</{tag}>'

Unfortunately, this rewrite doesn’t work if using the usual lambda wrapping to implement interpolations, namely lambda: tag. When the interpolations are evaluated by the tag function, it will result in NameError: name 'tag' is not defined. The root cause of this name error is that lambda: tag uses function scope, and it’s therefore not able to use the class definition where tag is defined.

Desugaring how the tag string could be evaluated will result in the same NameError even using f-strings; the lambda wrapping here also uses function scoping:

class CaptionConfig:
    tag = 'b'
    figure = f'<{(lambda: tag)()}>Figure</{(lambda: tag)()}>'

For tag strings, getting such a NameError would be surprising. It would also be a rough edge in using tag strings in this specific case of working with class variables. After all, tag strings are supposed to support a superset of the capabilities of f-strings.

The solution is to use annotation scope for tag string interpolations. While the name “annotation scope” suggests it’s only about annotations, it solves this problem by lexically resolving names in the class definition, such as tag, unlike function scope.

This PEP and its reference implementation therefore use the support for annotation scope. Note that this usage is a separable part from the implementation of PEP 649 and PEP 695 which provides a somewhat similar deferred execution model for annotations. Instead it’s up to the tag function to evaluate any interpolations.

With annotation scope in place, lambda-wrapped expressions in interpolations then provide the usual lexical scoping seen with f-strings. So there’s no need to use locals(), globals(), or frame introspection with sys._getframe to evaluate the interpolation. In addition, the code of each expression is available and does not have to be looked up with inspect.getsource or some other means.

Format Specification

The format_spec is by default None if it is not specified in the tag string’s corresponding interpolation.

Because the tag function is completely responsible for processing Decoded and Interpolation values, there is no required interpretation for the format spec and conversion in an interpolation. For example, this is a valid usage:

html'<div id={id:int}>{content:HTML|str}</div>'

In this case the format_spec for the second interpolation is the string 'HTML|str'; it is up to the html tag to do something with the “format spec” here, if anything.

f-string-style = Evaluation

mytag'{expr=}' is parsed to being the same as mytag'expr={expr}’, as implemented in the issue Add = to f-strings for easier debugging.

Tag Function Arguments

The tag function has the following signature:

def mytag(*args: Decoded | Interpolation) -> Any:
    ...

This corresponds to the following protocol:

class TagFunction(Protocol):
    def __call__(self, *args: Decoded | Interpolation) -> Any:
        ...

Because of subclassing, the signature for mytag can of course be widened to the following, at the cost of losing some type specificity:

def mytag(*args: str | tuple) -> Any:
    ...

A user might write a tag string as follows:

def tag(*args):
    return args

tag"\N{{GRINNING FACE}}"

Tag strings will represent this as exactly one Decoded argument. In this case, Decoded.raw would be '\\N{GRINNING FACE}'. The “cooked” representation via encode and decode would be:

'\\N{GRINNING FACE}'.encode('utf-8').decode('unicode-escape')
'😀'

Named unicode characters immediately followed by more text will still produce just one Decoded argument:

def tag(*args):
    return args

assert tag"\N{{GRINNING FACE}}sometext" == (DecodedConcrete("😀sometext"),)

Return Value

Tag functions can return any type. Often they will return a string, but richer systems can be built by returning richer objects. See below for a motivating example.

Function Application

Tag strings desugar as follows:

mytag'Hi, {name!s:format_spec}!'

This is equivalent to:

mytag(DecodedConcrete(r'Hi, '), InterpolationConcrete(lambda: name, 'name',
's', 'format_spec'), DecodedConcrete(r'!'))

No Empty Decoded String

Alternation between decodeds and interpolations is commonly seen, but it depends on the tag string. Decoded strings will never have a value that is the empty string:

mytag'{a}{b}{c}'

…which results in this desugaring:

mytag(InterpolationConcrete(lambda: a, 'a', None, None), InterpolationConcrete(lambda: b, 'b', None, None), InterpolationConcrete(lambda: c, 'c', None, None))

Likewise:

mytag''

…results in this desugaring:

mytag()

Python Semantics in Tag Strings

Python template languages and other DSLs have semantics quite apart from Python. Different scope rules, different calling semantics e.g. for macros, their own grammar for loops, and the like.

This means all tools need to write special support for each language. Even then, it is usually difficult to find all the possible scopes, for example to autocomplete values.

However, f-strings do not have this issue. An f-string is considered part of Python. Expressions in curly braces behave as expected and values should resolve based on regular scoping rules. Tools such as mypy can see inside f-string expressions, but will likely never look inside a Jinja2 template.

DSLs written with tag strings will inherit much of this value. While we can’t expect standard tooling to understand the “domain” in the DSL, they can still inspect anything expressible in an f-string.

Structural Pattern Matching

Iterating over the arguments with structural pattern matching is the expected best practice for many tag function implementations:

def tag(*args: Decoded | Interpolation) -> Any:
    for arg in args:
        match arg:
            case Decoded() as decoded:
                ... # handle each decoded string
            case Interpolation() as interpolation:
                ... # handle each interpolation

Lazy Evaluation

The example tag functions above each call the interpolation’s getvalue lambda immediately. Python developers have frequently wished that f-strings could be deferred, or lazily evaluated. It would be straightforward to write a wrapper that, for example, defers calling the lambda until an __str__ was invoked.

Memoizing

Tag function authors have control of processing the static string parts and the dynamic interpolation parts. For higher performance, they can deploy approaches for memoizing processing, for example by generating keys.

Order of Evaluation

Imagine a tag that generates a number of sections in HTML. The tag needs inputs for each section. But what if the last input argument takes a while? You can’t return the HTML for the first section until all the arguments are available.

You’d prefer to emit markup as the inputs are available. Some templating tools support this approach, as does tag strings.

Enable Exact Round-Tripping of conv and format_spec

There are two limitations with respect to exactly round-tripping to the original source text.

First, the format_spec can be arbitrarily nested:

mytag'{x:{a{b{c}}}}'

In this PEP and corresponding reference implementation, the format_spec is eagerly evaluated to set the format_spec in the interpolation, thereby losing the original expressions.

While it would be feasible to preserve round-tripping in every usage, this would require an extra flag equals to support, for example, {x=}, and a recursive Interpolation definition for format_spec. The following is roughly the pure Python equivalent of this type, including preserving the sequence unpacking (as used in case statements):

class InterpolationConcrete(NamedTuple):
    getvalue: Callable[[], Any]
    raw: str
    conv: str | None = None
    format_spec: str | None | tuple[Decoded | Interpolation, ...] = None
    equals: bool = False

    def __len__(self):
        return 4

    def __iter__(self):
        return iter((self.getvalue, self.raw, self.conv, self.format_spec))

However, the additional complexity to support exact round-tripping seems unnecessary and is thus rejected.

No Implicit String Concatenation

Implicit tag string concatenation isn’t supported, which is unlike other string literals.

The expectation is that triple quoting is sufficient. If implicit string concatenation is supported, results from tag evaluations would need to support the + operator with __add__ and __radd__.

Because tag strings target embedded DSLs, this complexity introduces other issues, such as determining appropriate separators. This seems unnecessarily complicated and is thus rejected.

Arbitrary Conversion Values

Python allows only r, s, or a as possible conversion type values. Trying to assign a different value results in SyntaxError.

In theory, tag functions could choose to handle other conversion types. But this PEP adheres closely to PEP 701. Any changes to allowed values should be in a separate PEP.