Generators & Iterators

Overview

Generators are functions that can pause their execution and yield values one at a time, resuming from where they left off when the consumer requests the next value. They bridge synchronous and asynchronous programming and allow infinite or on-demand sequences without blocking.

Origin

Generator functions appeared in CLU (1975) and Icon. Python introduced them in version 2.2 (2001) via PEP 255. JavaScript added them in ES6 (2015). Ruby's Enumerator and Fiber serve the same purpose.

Examples

async function* fetchAllPages(url) {
  let nextUrl = url
  while (nextUrl) {
    const response = await fetch(nextUrl)
    const data = await response.json()
    yield* data.items           // yield each item individually
    nextUrl = data.next_page_url
  }
}

// Consumer processes items without caring about pagination
for await (const item of fetchAllPages('/api/orders')) {
  await processOrder(item)
}

fib = Enumerator.new do |yielder|
  a, b = 0, 1
  loop do
    yielder.yield a
    a, b = b, a + b
  end
end

fib.take(8)          # => [0, 1, 1, 2, 3, 5, 8, 13]
fib.next             # resumes from last position
fib.lazy.select(&:even?).first(4)  # => [0, 2, 8, 34]

Use Cases

• 01Paginating through large result sets without loading all pages first
• 02Streaming file or network data line-by-line
• 03Generating test fixtures or IDs on demand
• 04Implementing cooperative multitasking (coroutines)
• 05Async iterators that pull data from message queues

When Not to Use

• //When a simple array or collection is sufficient, generators add indirection
• //When the consumer always needs all values, eager evaluation is simpler
• //When sharing a generator across threads without synchronisation, generators are stateful and not thread-safe

Technical Notes

• JavaScript async generators (async function*) enable streaming over async sources with for await...of
• Ruby Fibers underpin Enumerator and also enable the async/non-blocking style in the Async gem
• Generators in Python are implemented by adding __iter__ and __next__ protocol, any object implementing these is iterable
• Generators compose with lazy evaluation: a pipeline of generators processes one element at a time through all stages