Generating Parsers With pego gen

pego gen turns a grammar into Go source files: a parser package you commit to your repository, build with your program, and call like any other package. By default the generated Go package depends only on the standard library: it carries a copy of the PEGO runtime in its internal/runtime directory, and it does not load or compile the grammar at run time.

This guide shows how to generate a parser, what the generator supports compared with the engine (the pego.Parser of Running Parsers), how to keep generated code up to date, and when it is the right choice. The API of the generated Go package is in The Generated Go API, and the options that add typed values, documents and streams have guides of their own, linked below. pego gen -lang ts generates TypeScript modules with the same behavior instead; it has a guide of its own, TypeScript parsers.

• Why generate
• Quick start
• The pego gen command
• What is supported
• Keeping generated code up to date
• Trade-offs
• Recipes

#Why Generate

Generate a parser when the grammar is fixed, the parser is on a hot path or in a small binary, or you want no dependency on PEGO at run time. Use the engine when the grammar is data. For streams of Nodes, generate a Go ParseStream with -stream; for incremental Node parsing, generate a Go Document with -document. The engine offers the broader stream and Document APIs. A good deal of the time you can use both: the engine in tools and tests, and the generated code in production (see Recipes).

The generated parser returns the same tree and the same errors as the engine, so switching between them does not change what your code sees. The test suite checks this for every grammar in examples/ (design record 008).

#Quick Start

A complete, runnable setup. The grammar, pairs.pego, is the one used in the Running Parsers:

type Pair = { Key node[string]; Value node[string] }

@start(default)
rule main = ws ps:pair+ $$ => $ps
rule pair Pair = k:key "=" v:value ";"? ws => Pair{Key: $k, Value: $v}
@start
rule key = [a-zあ-ん]+ #atomic
rule value = [0-9]+ #atomic
rule ws = [ \t\n]*

The layout is a module with the grammar next to the code that generates from it, and the generated parser in its own package (see why its own package):

example.com/pairs
├── go.mod
├── pairs.pego
├── generate.go
├── main.go
└── pairsparser/        # created by go generate
    ├── parser.go
    ├── api.go
    └── internal/runtime/   # and, with -types, types/types.go

Record pego as a tool dependency of the module, so that go generate runs the pinned version and nobody needs a separate install:

go get -tool github.com/ornew/pego/cmd/pego

This adds a tool github.com/ornew/pego/cmd/pego line to go.mod. Then declare the generation step with a go:generate directive; pego gen creates the output directory:

// generate.go
package main

//go:generate go tool pego gen -g pairs.pego -module example.com/pairs/pairsparser -o pairsparser

go generate ./...

-module is the import path of the generated package, which the copy of the runtime in pairsparser/internal/runtime is imported by; -o is the output directory, which pego gen creates. The result is pairsparser/parser.go, with the code generated from the grammar, pairsparser/api.go, which re-exports the runtime names the users of the parser need (Node, SyntaxError and so on), the runtime copy and, with -types, pairsparser/types/types.go, the package of the Go types of the grammar (the -module path is then required). parser.go starts with:

// Code generated by pego. DO NOT EDIT.

// Package pairsparser is a parser generated from a PEGO grammar.
package pairsparser

Now use it:

// main.go
package main

import (
	"encoding/json"
	"errors"
	"fmt"
	"log"

	"example.com/pairs/pairsparser"
)

func main() {
	node, err := pairsparser.Parse("abc=12; あい=3")
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(node)
	for _, pair := range node.Elems() {
		key := pair.Field("Key").(*pairsparser.Node)
		fmt.Printf("%s %q at %d-%d\n", pair.Type(), key.Text(), key.Start(), key.End())
	}

	// Positions in UTF-8 bytes.
	node, _ = pairsparser.Parse("あい=3", pairsparser.WithUnit(pairsparser.Bytes))
	fmt.Println(node.Elem(0).End())

	// Any rule marked @start can be the start rule.
	node, err = pairsparser.ParseRule("key", "abc")
	fmt.Println(node, err)
	_, err = pairsparser.ParseRule("nope", "abc")
	fmt.Println(err)

	// Errors have the same fields as the engine's, but are types of the generated package.
	_, err = pairsparser.Parse("abc=x")
	var se *pairsparser.SyntaxError
	fmt.Println(errors.As(err, &se), se.Line, se.Col, se.Expected, err)

	out, _ := json.Marshal(node)
	fmt.Println(string(out))
}

[Pair{Key: "abc", Value: "12"} Pair{Key: "あい", Value: "3"}]
Pair "abc" at 0-3
Pair "あい" at 8-10
8
"abc" <nil>
rule nope is not defined
true 1 5 [[0-9]] 1:5: syntax error: expected [0-9]
{"start":0,"end":3,"value":"abc"}

The output is exactly what the engine prints for the same grammar and input.

#The pego gen Command

pego gen -g <grammar> -module <import path> [-pkg <package>] [-runtime embed|import|only] [-s <rule>] [-o <directory>] [-types] [-document] [-stream] [-recognize] [-observability] [-work-controls] [-default-value] [-nodoc] [-max-depth <n>]

Details:

• Both languages need -o, because the output is several files. TypeScript writes parser.ts, runtime.ts and, with -types, types.ts.

• The output is formatted with gofmt, passes go vet, and is deterministic: generating twice gives identical files. That is what makes the up-to-date check possible.

• Grammar errors are reported with positions, before any file is written. A rule with no expression is a syntax error:

  // broken.pego
  @start
  rule main =
  

  pego gen -g broken.pego -module example.com/pairs -o /dev/null
  

  pego: broken.pego:4:1: expected an expression, found end of file
  

  An action that reads a capture the rule does not have is a compile error:

  // typeerr.pego
  @start
  rule main = x:[a-z]+ => $y
  

  pego gen -g typeerr.pego -module example.com/pairs -o /dev/null
  

  pego: typeerr.pego:3:25: undefined capture $y
  

  pego gen -g pairs.pego -module example.com/pairs -s nope -o /dev/null
  

  pego: pairs.pego: rule nope is not defined
  

  pego gen -g pairs.pego -module example.com/pairs -pkg foo-bar -o /dev/null
  

  pego: pairs.pego: invalid package name "foo-bar"
  

• A .pegoc cannot be used, because code generation needs the grammar and a compiled grammar holds only bytecode: pego: pairs.pegoc: a compiled grammar holds only the bytecode of one mode; gen needs the grammar source (.pego or .json).

• Without -s, the grammar's default start rule is used.

#From Go

The same generator is available from Go, in package github.com/ornew/pego/gen, for build tools that produce parsers programmatically:

g, err := pego.ParseGrammar(src)
if err != nil {
	log.Fatal(err)
}
files, err := gen.Go{Module: "example.com/pairs/pairsparser"}.Generate(g, "main") // grammar, start rule
if err != nil {
	log.Fatal(err)
}
for _, f := range files { // parser.go, api.go, types/types.go with Types, and internal/runtime/*.go
	path := filepath.Join("pairsparser", filepath.FromSlash(f.Path))
	if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
		log.Fatal(err)
	}
	if err := os.WriteFile(path, f.Content, 0o644); err != nil {
		log.Fatal(err)
	}
}

The fields of gen.Go are the flags of pego gen: Module is -module; Types: true is -types; Document is -document; Stream is -stream; Recognize is -recognize; TypedOnly is -typed-only; WorkControls is -work-controls; Observability is -observability; SourceInfo is -source-info; DefaultValue is -default-value; OmitPackageDoc is -nodoc; Runtime: gen.RuntimeImport is -runtime import; PackageName: "p" is -pkg p; and MaxDepth: 600_000 is -max-depth 600000. Omitted, zero and explicit 100,000 defaults generate identical source. Observability supports standalone Go Node and typed batch parsers, including ParseAST and ParseRuleAST, Go Documents and streams, and TypeScript batch parsers with or without -types. Generated Go Node, typed and Document packages expose WithProfile; TypeScript batch packages expose Profile through per-call options. Typed direct execution and Node-conversion fallback report their actual rule calls; conversion itself has no separate event. Invalid defaults return an error before generated output is written.

pego gen is a thin wrapper around gen.Go (and gen.TypeScript with -lang ts, whose Generate returns the files too); both implement gen.Generator. Because a grammar built with the grammar package (see the Running Parsers) is just an AST, it can be generated from, too.

When -observability is enabled, the generated Go or TypeScript package includes opt-in rule tracing and profiling. See generated Go Node and typed traces, generated TypeScript traces, and generated TypeScript profiling for the contracts and examples.

The ordinary generated source is unchanged when this option is off. Enabling it adds generated code and some parse overhead even when no callback is passed; delivering events costs more. Focused Node measurements showed no-callback parse ratios around 1.01–1.06× and recognition around 1.02–1.20×. Typed no-callback ratios ranged from 0.995–1.075×; scalar-store callbacks ranged from 1.169–3.164× depending on workload and execution path. These are synthetic measurements, not workload-wide estimates. Focused WithProfile runs were 3.8–26.9× ordinary time and added about 45–395 KB/op and 25–50 allocations/op across the measured cases; see design 043 for scope and generation costs. Focused TypeScript profiles measured 1.58–4.02× ordinary time, depending on API and fixture; see the TypeScript profile measurements in design 043 for conditions and limits.

#What Is Supported

The generator supports the whole grammar language: Rule Expressions, Operators Rules, left recursion, Predicates, variables, actions, types, built-in functions, @error and #recover. What differs from the engine is the runtime machinery, which the generated code only has when you ask for it:

• -stream, -document and -recognize add ParseStream, Document and Recognize (and -types their typed forms). Without -stream, #stream is an ordinary repetition. TypeScript has neither streams nor documents.
• ParseRule replaces WithStart for another start rule, and WithUnit and WithMaxDepth work as in the engine; -max-depth sets the generated default.
• Cancellation and work limits are WithContext and WithWorkLimits in the engine and in generated Go with -work-controls, and the signal, maxInputWork and maxSteps options in TypeScript with -work-controls; see Cancellation and Work Limits and the specification.
• There is no choice of backend, because the generated code is a backend of its own, and no Parser.Grammar(), saving or loading, because the grammar is gone at run time.
• #missing works only in the engine's Closure backend for whole-input parsing and recognition, and the generator rejects grammars that use it.

The specification has the complete table.

Notes:

• Depth. Generated code uses the Go stack for rule calls and fails with nesting too deep: more than 100000 rule calls at the default limit. WithMaxDepth changes the limit for one invocation, and -max-depth changes the generated default. With the nested-parentheses grammar rule nested = "(" nested ")" / "x", 50,000 levels parse and 200,000 levels fail with that error. If you must accept deeper nesting beyond the host stack, the engine's BytecodeIterative backend with WithMaxDepth is the tool (Choosing a Backend).
• Memory. Generated parsers memoize in the same way as the engine, so memory still grows with the input. For input larger than memory, generate ParseStream with -stream or use the engine's ParseStream.
• Using a stream grammar with the generator is fine: Parse treats the #stream pragma as an ordinary repetition, so the same grammar file serves both streaming and whole-input parsing.

#Keeping Generated Code Up to Date

Generated code is source: commit it, review its diffs, and regenerate it whenever the grammar changes or you upgrade PEGO (the embedded runtime comes from the version of pego that generated it). Three habits keep it honest.

1. Generate with go generate. Put the directive next to the grammar, as in Quick start, and run go generate ./.... Pin the generator with go get -tool as shown above. (go install github.com/ornew/pego/cmd/pego@latest and //go:generate pego gen ... works, too, but every developer then has to install a matching version.) With several grammars, add one directive per grammar.

2. Test that it is up to date. A test regenerates in memory and compares with the committed file; it fails with a reminder when somebody changed the grammar and forgot to regenerate:

//go:embed pairs.pego
var grammarSrc string

// The files of pairsparser are up to date with pairs.pego.
func TestGeneratedIsUpToDate(t *testing.T) {
	g, err := pego.ParseGrammar(grammarSrc)
	if err != nil {
		t.Fatal(err)
	}
	want, err := gen.Go{Module: "example.com/pairs/pairsparser"}.Generate(g, "main")
	if err != nil {
		t.Fatal(err)
	}
	for _, f := range want {
		got, err := os.ReadFile(filepath.Join("pairsparser", filepath.FromSlash(f.Path)))
		if err != nil {
			t.Fatal(err)
		}
		if string(got) != string(f.Content) {
			t.Errorf("pairsparser/%s is out of date; run go generate ./...", f.Path)
		}
	}
}

If you change pairs.pego (here: rule value = [0-9a-f]+ #atomic ) and skip regeneration, the test fails. Among the output:

--- FAIL: TestGeneratedIsUpToDate (0.00s)
    generated_test.go:49: pairsparser/parser.go is out of date; run go generate ./...
FAIL

Run go generate ./... in your project and check that the generated files have no diff (for example, with git diff --exit-code). This makes stale output fail in your own CI.

3. Test that it behaves like the engine. The engine is the reference implementation, and a one-time parity test protects you from the two diverging (for example after an upgrade):

func TestGeneratedMatchesEngine(t *testing.T) {
	p, err := pego.CompileSource(grammarSrc, "main")
	if err != nil {
		t.Fatal(err)
	}
	for _, in := range []string{"abc=12", "abc=12; あい=3", "", "abc=", "abc=1 x"} {
		want, werr := p.Parse(in)
		got, gerr := pairsparser.Parse(in)
		wj, _ := json.Marshal(want)
		gj, _ := json.Marshal(got)
		if string(wj) != string(gj) || fmt.Sprint(werr) != fmt.Sprint(gerr) {
			t.Errorf("%q:\n engine    %s %v\n generated %s %v", in, wj, werr, gj, gerr)
		}
	}
}

Comparing JSON and error text covers the whole tree (with positions) and all error details. Include inputs that fail, and inputs that exercise each feature of your grammar.

#Trade-Offs

• Speed. In the current benchmark results, generated Go takes 0.44–0.61× closure time in CodePoints and 0.50–0.66× in Bytes across eight workloads. Direct ParseAST values take 0.69–0.98× generated Parse time; conversion workloads that include generic nodes can take longer. Standard-library comparisons produce different outputs and remain faster where they apply. These measurements are from one machine and commit; run go run ./bench/report to measure yours.
• Binary and repository size. Each generated parser adds source that you commit and that compiles into your binary: parser.go and api.go, with -types types/types.go, and with -runtime embed a copy of the runtime in internal/runtime (the same for every grammar; -runtime import shares one copy through github.com/ornew/pego/runtime instead). It does not pull in PEGO's compiler, analyzer or VMs.
• Start-up. There is none: no grammar to parse or compile. If start-up time is the only concern and a source file is acceptable, loading a .pegoc is the cheaper alternative.
• Flexibility. The grammar is frozen into the binary. Changing it means regenerating, rebuilding and redeploying; if users supply grammars at run time, use the engine.
• Features. No backend choice; streaming requires -stream and incremental Node parsing -document (see what is supported).
• Review noise. Large diffs on regeneration (particularly after upgrading PEGO) are expected. Mark the files as generated for your review tool: the first line, // Code generated by pego. DO NOT EDIT., follows the Go convention that tools recognize.
• Typed AST. Parse returns generic *Node trees like the engine, which is what makes the two interchangeable. For Go types matching the grammar's types, generate with -types and call ParseAST; for types of your own, write a small conversion from *Node or from the typed values, as parsers/json does (ToAny).

#Recipes

#One Grammar, One Package

Each generated package defines Node, SyntaxError, Parse and the rest at package level, so two generated parsers in the same package collide:

both/b.go:17:6: Node redeclared in this block
	both/a.go:17:6: other declaration of Node

Give every grammar its own package (and directory), as parsers/* and internal/bench/gen/* do. If you also want several entry points of one grammar, use ParseRule rather than generating it twice.

#Engine in Development, Generated in Production

Keep one .pego file. Use pego parse and the engine (CompileSource) in tools, experiments and tests, where start-up and features matter more than speed; generate the production parser from the same file; and let the parity test prove they agree.

#Choosing the Start Rule

Mark every rule where parsing may begin with @start, and the usual entry rule with @start(default), so that Parse is the common case; -s (for example pego gen -g lang.pego -module example.com/lang -s program) selects another start rule for Parse. Use ParseRule("expr", input) for the other start rules, for example for an expression evaluator embedded in a larger tool.

#Wrapping the Result in Your Own Types

If the grammar declares the types you want, -types generates them. Otherwise: the generated Node is the same loose shape for every grammar. A thin layer keeps that out of the rest of your program:

type Pair struct{ Key, Value string }

func Pairs(input string) ([]Pair, error) {
	node, err := pairsparser.Parse(input)
	if err != nil {
		return nil, err
	}
	var out []Pair
	for _, n := range node.Elems() {
		out = append(out, Pair{
			Key:   n.Field("Key").(*pairsparser.Node).Text(),
			Value: n.Field("Value").(*pairsparser.Node).Text(),
		})
	}
	return out, nil
}

Design the grammar's types and actions (Declaring Types, Actions) so that this layer is thin, and unit-test it with the engine and the generated parser alike.

#Related Topics

• The Generated Go API: Parse, recognition, call options, cancellation and work limits.
• Typed Values in Generated Go, Incremental Documents in Generated Go and Streams in Generated Go: the packages that -types, -document and -stream add.
• Generated TypeScript Parsers: the same generator for TypeScript.
• Compiled Grammars: the other way to skip compiling at run time.
• Using the Ready-Made Parsers: parsers made with this generator.