Syntax

The grammar should be changed to allow the expression in list/set comprehensions and generator expressions to be preceded by a *, and allowing an alternative form of dictionary comprehension in which a double-starred expression can be used in place of a key: value pair.

This can be accomplished by updating the listcomp and setcomp rules to use star_named_expression instead of named_expression:

listcomp[expr_ty]:
    | '[' a=star_named_expression b=for_if_clauses ']'

setcomp[expr_ty]:
    | '{' a=star_named_expression b=for_if_clauses '}'

The rule for genexp would similarly need to be modified to allow a starred_expression:

genexp[expr_ty]:
    | '(' a=(assignment_expression | expression !':=' | starred_expression) b=for_if_clauses ')'

The rule for dictionary comprehensions would need to be adjusted as well, to allow for this new form:

dictcomp[expr_ty]:
    | '{' a=double_starred_kvpair b=for_if_clauses '}'

No change should be made to the way that argument unpacking is handled in function calls, i.e., the general rule that generator expressions provided as the sole argument to functions do not require additional redundant parentheses should be retained. Note that this implies that, for example, f(*x for x in it) is equivalent to f((*x for x in it)) (see Starred Generators as Function Arguments for more discussion).

* and ** should only be allowed at the top-most level of the expression in the comprehension (see Further Generalizing Unpacking Operators for more discussion).

Semantics: List/Set/Dict Comprehensions

The meaning of a starred expression in a list comprehension [*expr for x in it] is to treat each expression as an iterable, and concatenate them, in the same way as if they were explicitly listed via [*expr1, *expr2, ...]. Similarly, {*expr for x in it} forms a set union, as if the expressions were explicitly listed via {*expr1, *expr2, ...}; and {**expr for x in it} combines dictionaries, as if the expressions were explicitly listed via {**expr1, **expr2, ...}. These operations should retain all of the equivalent semantics for combining collections in this way (including, for example, later values replacing earlier ones in the case of a duplicated key when combining dictionaries).

Said another way, the objects created by the following comprehensions:

new_list = [*expr for x in its]
new_set = {*expr for x in its}
new_dict = {**expr for d in dicts}

should be equivalent to the objects created by the following pieces of code, respectively:

new_list = []
for x in its:
    new_list.extend(expr)

new_set = set()
for x in its:
    new_set.update(expr)

new_dict = {}
for x in dicts:
    new_dict.update(expr)

Semantics: Generator Expressions

Generator expressions using the unpacking syntax should form new generators producing values from the concatenation of the iterables given by the expressions. Specifically, the behavior is defined to be equivalent to the following:

# equivalent to g = (*expr for x in it)
def generator():
    for x in it:
        yield from expr

g = generator()

Since yield from is not allowed inside of async generators (see the section of PEP 525 on Asynchronous yield from), the equivalent for (*expr async for x in ait()) is more like the following (though of course this new form should not define or reference the looping variable i):

# equivalent to g = (*expr async for x in ait())
async def generator():
    async for x in ait():
        for i in expr:
            yield i

g = generator()

The specifics of these semantics should be revisited in the future, particularly if async generators receive support for yield from (in which case the async variant may wish to be changed to make use of yield from instead of an explicit loop). See Alternative Generator Expression Semantics for more discussion.

def generator():
    for i in (0, 2, 4):
        yield from (y := [i, i+1])

>>> g = (*(y := [i, i+1]) for i in (0, 2, 4))
>>> y
Traceback (most recent call last):
  File "<python-input-1>", line 1, in <module>
    y
NameError: name 'y' is not defined
>>> next(g)
0
>>> y
[0, 1]
>>> next(g)
1
>>> y
[0, 1]
>>> next(g)
2
>>> y
[2, 3]

Error Reporting

Currently, the proposed syntax generates a SyntaxError. Allowing these forms to be recognized as syntactically valid requires adjusting the grammar rules for invalid_comprehension and invalid_dict_comprehension to allow the use of * and **, respectively.

Additional specific error messages should be provided in at least the following cases:

• Attempting to use ** in a list comprehension or generator expression should report that dictionary unpacking cannot be used in those structures, for example:

  >>> [**x for x in y]
    File "<stdin>", line 1
      [**x for x in y]
       ^^^
  SyntaxError: cannot use dict unpacking in list comprehension
  
  >>> (**x for x in y)
    File "<stdin>", line 1
      (**x for x in y)
       ^^^
  SyntaxError: cannot use dict unpacking in generator expression
  

• The existing error message for attempting to use * in a dictionary key/value should be retained, but similar messages should be reported when attempting to use ** unpacking on a dictionary key or value, for example:

  >>> {*k: v for k,v in items}
    File "<stdin>", line 1
      {*k: v for k,v in items}
       ^^
  SyntaxError: cannot use a starred expression in a dictionary key
  
  >>> {k: *v for k,v in items}
    File "<stdin>", line 1
      {k: *v for k,v in items}
          ^^
  SyntaxError: cannot use a starred expression in a dictionary value
  
  >>> {**k: v for k,v in items}
    File "<stdin>", line 1
      {**k: v for k,v in items}
       ^^^
  SyntaxError: cannot use dict unpacking in a dictionary key
  
  >>> {k: **v for k,v in items}
    File "<stdin>", line 1
      {k: **v for k,v in items}
          ^^^
  SyntaxError: cannot use dict unpacking in a dictionary value
  

• The phrasing of some other existing error messages should similarly be adjusted to account for the presence of the new syntax, and/or to clarify ambiguous or confusing cases relating to unpacking more generally (particularly the cases mentioned in Further Generalizing Unpacking Operators), for example:

  >>> [*x if x else y]
    File "<stdin>", line 1
      [*x if x else y]
       ^^^^^^^^^^^^^^
  SyntaxError: invalid starred expression. Did you forget to wrap the conditional expression in parentheses?
  
   >>> {**x if x else y}
    File "<stdin>", line 1
      {**x if x else y}
       ^^^^^^^^^^^^^^^
  SyntaxError: invalid double starred expression. Did you forget to wrap the conditional expression in parentheses?
  
  >>> [x if x else *y]
    File "<stdin>", line 1
      [x if x else *y]
                   ^
  SyntaxError: cannot unpack only part of a conditional expression
  
  >>> {x if x else **y}
    File "<stdin>", line 1
      {x if x else **y}
                   ^^
  SyntaxError: cannot use dict unpacking on only part of a conditional expression
  

Replacing Explicit Loops

Replacing explicit loops compresses multiple lines into one, and avoids the need for defining and referencing an auxiliary variable.

• From email/_header_value_parser.py:

  # current:
  comments = []
  for token in self:
      comments.extend(token.comments)
  return comments
  
  # proposed:
  return [*token.comments for token in self]
  

• From shutil.py:

  # current:
  ignored_names = []
  for pattern in patterns:
      ignored_names.extend(fnmatch.filter(names, pattern))
  return set(ignored_names)
  
  # proposed:
  return {*fnmatch.filter(names, pattern) for pattern in patterns}
  

• From http/cookiejar.py:

  # current:
  cookies = []
  for domain in self._cookies.keys():
      cookies.extend(self._cookies_for_domain(domain, request))
  return cookies
  
  # proposed:
  return [
      *self._cookies_for_domain(domain, request)
      for domain in self._cookies.keys()
  ]
  

Replacing from_iterable and Friends

While not always the right choice, replacing itertools.chain.from_iterable and map can avoid an extra level of redirection, resulting in code that follows conventional wisdom that comprehensions are generally more readable than map/filter.

• From dataclasses.py:

  # current:
  inherited_slots = set(
      itertools.chain.from_iterable(map(_get_slots, cls.__mro__[1:-1]))
  )
  
  # proposed:
  inherited_slots = {*_get_slots(c) for c in cls.__mro__[1:-1]}
  

• From importlib/metadata/__init__.py:

  # current:
  return itertools.chain.from_iterable(
      path.search(prepared) for path in map(FastPath, paths)
  )
  
  # proposed:
  return (*FastPath(path).search(prepared) for path in paths)
  

• From collections/__init__.py (Counter class):

  # current:
  return _chain.from_iterable(_starmap(_repeat, self.items()))
  
  # proposed:
  return (*_repeat(elt, num) for elt, num in self.items())
  

• From zipfile/_path/__init__.py:

  # current:
  parents = itertools.chain.from_iterable(map(_parents, names))
  
  # proposed:
  parents = (*_parents(name) for name in names)
  

• From _pyrepl/_module_completer.py:

  # current:
  search_locations = set(chain.from_iterable(
      getattr(spec, 'submodule_search_locations', [])
      for spec in specs if spec
  ))
  
  # proposed:
  search_locations = {
      *getattr(spec, 'submodule_search_locations', [])
      for spec in specs if spec
  }
  

Replacing Double Loops in Comprehensions

Replacing double loops in comprehensions avoids the need for defining and referencing an auxiliary variable.

• From importlib/resources/readers.py:

  # current:
  children = (child for path in self._paths for child in path.iterdir())
  
  # proposed:
  children = (*path.iterdir() for path in self._paths)
  

• From asyncio/base_events.py:

  # current:
  exceptions = [exc for sub in exceptions for exc in sub]
  
  # proposed:
  exceptions = [*sub for sub in exceptions]
  

• From _weakrefset.py:

  # current:
  return self.__class__(e for s in (self, other) for e in s)
  
  # proposed:
  return self.__class__(*s for s in (self, other))
  

Starred Generators as Function Arguments

One common concern that has arisen multiple times (not only in the discussion threads linked above but also in previous discussions around this same idea) is a possible syntactical ambiguity when passing a starred generator as the sole argument to f(*x for x in y). In the original PEP 448, this ambiguity was cited as a reason for not including a similar generalization as part of the proposal.

This proposal suggests that f(*x for x in y) should be interpreted as f((*x for x in y)) and should not attempt further unpacking of the resulting generator, but several alternatives were suggested in our discussion (and/or have been suggested in the past), including:

• interpreting f(*x for x in y) as f(*(x for x in y),
• interpreting f(*x for x in y) as f(*(*x for x in y)), or
• continuing to raise a SyntaxError for f(*x for x in y) even if the other aspects of this proposal are accepted.

The reason to prefer this proposal over these alternatives is the preservation of existing conventions for punctuation around generator expressions. Currently, the general rule is that generator expressions must be wrapped in parentheses except when provided as the sole argument to a function, and this proposal suggests maintaining that rule even as we allow more kinds of generator expressions. This option maintains a full symmetry between comprehensions and generator expressions that use unpacking and those that don’t.

Currently, we have the following conventions:

f([x for x in y])  # pass in a single list
f({x for x in y})  # pass in a single set
f(x for x in y)  # pass in a single generator (no additional parentheses required around genexp)

f(*[x for x in y])  # pass in elements from the list separately
f(*{x for x in y})  # pass in elements from the set separately
f(*(x for x in y))  # pass in elements from the generator separately (parentheses required)

This proposal opts to maintain those conventions even when the comprehensions make use of unpacking:

f([*x for x in y])  # pass in a single list
f({*x for x in y})  # pass in a single set
f(*x for x in y)  # pass in a single generator (no additional parentheses required around genexp)

f(*[*x for x in y])  # pass in elements from the list separately
f(*{*x for x in y})  # pass in elements from the set separately
f(*(*x for x in y))  # pass in elements from the generator separately (parentheses required)

Further Generalizing Unpacking Operators

Another suggestion that came out of the discussion involved further generalizing the * beyond simply allowing it to be used to unpack the expression in a comprehension. Two main flavors of this extension were considered:

• making * and ** true unary operators that create a new kind of Unpackable object (or similar), which comprehensions could treat by unpacking it but which could also be used in other contexts; or
• continuing to allow * and ** only in the places they are allowed elsewhere in this proposal (expression lists, comprehensions, generator expressions, and argument lists), but also allow them to be used in subexpressions within a comprehension, allowing, for example, the following as a way to flatten a list that contains some iterables but some non-iterable objects:

  [*x if isinstance(x, Iterable) else x for x in [[1,2,3], 4]]
  

These variants were considered substantially more complex (both to understand and to implement) and of only marginal utility, so neither is included in this PEP. As such, these forms should continue to raise a SyntaxError, but with a new error message as described above, though it should not be ruled out as a consideration for future proposals.

Alternative Generator Expression Semantics

Another point of discussion centered around the semantics of unpacking in generator expressions, particularly the relationship between the semantics of synchronous and asynchronous generator expressions given that async generators do not support yield from (see the section of PEP 525 on Asynchronous yield from).

The core question centered around whether sync and async generator expressions should use yield from (or an equivalent) when unpacking, as opposed to an explicit loop. The main difference between these options is whether the resulting generator delegates to the objects being unpacked, which would affect the behavior of these generator expressions when used with .send()/.asend(), .throw()/.athrow(), and .close()/.aclose() in the case where the objects being unpacked are themselves generators. The differences between these options are summarized in Appendix: Semantics of Generator Delegation.

Several reasonable options were considered, none of which was a clear winner in a poll in the Discourse thread. Beyond the proposal outlined above, the following were also considered:

1. Using explicit loops for both synchronous and asynchronous generator expressions.

   This strategy would have resulted in a symmetry between synchronous and asynchronous generator expressions but would have prevented a potentially-useful tool by disallowing delegation in the case of synchronous generator expressions. One specific concern with this approach is the introduction of an asymmetry between synchronous and asynchronous generators, but this concern is mitigated by the fact that these asymmetries already exist between synchronous and asynchronous generators more generally.

2. Using yield from for unpacking in synchronous generator expressions and mimicking the behavior of yield from for unpacking in async generator expressions.

   This strategy would also make unpacking in synchronous and asynchronous generators behave symmetrically, but it would also be more complex, enough so that the cost may not be worth the benefit. As such, this PEP proposes that asynchronous generator expressions using the unpacking operator should not adopt semantics similar to yield from until yield from is supported in asynchronous generators more generally.

3. Using yield from for unpacking in synchronous generator expressions, and disallowing unpacking in asynchronous generator expressions until they support yield from.

   This strategy could possibly reduce friction if asynchronous generator expressions do gain support for yield from in the future by making sure that any decision made at that point would be fully backwards-compatible. But the utility of unpacking in that context seems to outweigh the potential downside of a minimally-invasive backwards-incompatible change in the future if async generator expressions do receive support for yield from.

4. Disallowing unpacking in all generator expressions.

   This would retain symmetry between the two cases, but with the downside of losing a very expressive form.

Each of these options (including the one presented in this PEP) has its benefits and drawbacks, with no option being clearly superior on all fronts. The semantics proposed in Semantics: Generator Expressions represent a reasonable compromise where unpacking in both synchronous and asynchronous generator expressions mirrors common ways of writing equivalent generators currently. Moreover, these subtle differences are unlikely to be impactful for common use cases (for example, there is no difference for the likely most-common use case of combining simple collections).

As suggested above, this decision should be revisited in the event that asynchronous generators receive support for yield from in the future, in which case adjusting the semantics of unpacking in async generator expressions to use yield from should be considered.

Basic Behavior

For simple iteration over values, which we expect to be by far the most-common use of unpacking in generator expressions, both approaches produce identical results:

def yield_from(iterables):
    for iterable in iterables:
        yield from iterable

def explicit_loop(iterables):
    for iterable in iterables:
        for item in iterable:
            yield item

# Both produce the same sequence of values
x = list(yield_from([[1, 2], [3, 4]]))
y = list(explicit_loop([[1, 2], [3, 4]]))
print(x == y)  # prints True

Advanced Generator Protocol Differences

The differences become apparent when using the advanced generator protocol methods .send(), .throw(), and .close(), and when the sub-iterables are themselves generators rather than simple sequences. In these cases, the yield from version results in the associated signal reaching the subgenerator, but the version with the explicit loop does not.

def sub_generator():
    x = yield "first"
    yield f"received: {x}"
    yield "last"

def yield_from():
    yield from sub_generator()

def explicit_loop():
    for item in sub_generator():
        yield item

# With yield from, values are passed through to sub-generator
gen1 = yield_from()
print(next(gen1))  # prints "first"
print(gen1.send("hello"))  # prints "received: hello"
print(next(gen1))  # prints "last"

# With explicit loop, .send() affects the outer generator; values don't reach the sub-generator
gen2 = explicit_loop()
print(next(gen2))  # prints "first"
print(gen2.send("hello"))  # prints "received: None" (sub-generator receives None instead of "hello")
print(next(gen2))  # prints "last"

def sub_generator_with_exception_handling():
    try:
        yield "first"
        yield "second"
    except ValueError as e:
        yield f"caught: {e}"

def yield_from():
    yield from sub_generator_with_exception_handling()

def explicit_loop():
    for item in sub_generator_with_exception_handling():
        yield item

# With yield from, exceptions are passed to sub-generator
gen1 = yield_from()
print(next(gen1))  # prints "first"
print(gen1.throw(ValueError("test")))  # prints "caught: test"

# With explicit loop, exceptions affect the outer generator only
gen2 = explicit_loop()
print(next(gen2))  # prints "first"
print(gen2.throw(ValueError("test")))  # ValueError is raised; sub-generator doesn't see it

# hold references to sub-generators so GC doesn't close the explicit loop version
references = []

def sub_generator_with_cleanup():
    try:
        yield "first"
        yield "second"
    finally:
        print("sub-generator received GeneratorExit")

def yield_from():
    try:
        g = sub_generator_with_cleanup()
        references.append(g)
        yield from g
    finally:
        print("outer generator received GeneratorExit")

def explicit_loop():
    try:
        g = sub_generator_with_cleanup()
        references.append(g)
        for item in g:
            yield item
    finally:
        print("outer generator received GeneratorExit")

# With yield from, GeneratorExit is passed through to sub-generator
gen1 = yield_from()
print(next(gen1))  # prints "first"
gen1.close()  # closes sub-generator and then outer generator

# With explicit loop, GeneratorExit goes to outer generator only
gen2 = explicit_loop()
print(next(gen2))  # prints "first"
gen2.close()  # only closes outer generator

print('program finished; GC will close the explicit loop subgenerator')
# second inner generator closes when GC closes it at the end