# A dead simple parser package for Go
[](https://pkg.go.dev/github.com/alecthomas/participle/v2) [](https://github.com/alecthomas/participle/actions)
[](https://goreportcard.com/report/github.com/alecthomas/participle/v2) [](https://gophers.slack.com/messages/CN9DS8YF3)
- [V2](#v2)
- [Introduction](#introduction)
- [Tutorial](#tutorial)
- [Tag syntax](#tag-syntax)
- [Overview](#overview)
- [Grammar syntax](#grammar-syntax)
- [Capturing](#capturing)
- [Capturing boolean value](#capturing-boolean-value)
- [Streaming](#streaming)
- [Lexing](#lexing)
- [Stateful lexer](#stateful-lexer)
- [Example stateful lexer](#example-stateful-lexer)
- [Example simple/non-stateful lexer](#example-simplenon-stateful-lexer)
- [Experimental - code generation](#experimental---code-generation)
- [Options](#options)
- [Examples](#examples)
- [Performance](#performance)
- [Concurrency](#concurrency)
- [Error reporting](#error-reporting)
- [Limitations](#limitations)
- [EBNF](#ebnf)
- [Syntax/Railroad Diagrams](#syntaxrailroad-diagrams)
## V2
This is an alpha of version 2 of Participle. It is still subject to change but should be mostly stable at this point.
See the [Change Log](CHANGES.md) for details.
> **Note:** semantic versioning API guarantees do not apply to the [experimental](https://pkg.go.dev/github.com/alecthomas/participle/v2/experimental) packages - the API may break between minor point releases.
It can be installed with:
```shell
$ go get github.com/alecthomas/participle/v2@latest
```
The latest version from v0 can be installed via:
```shell
$ go get github.com/alecthomas/participle@latest
```
## Introduction
The goal of this package is to provide a simple, idiomatic and elegant way of
defining parsers in Go.
Participle's method of defining grammars should be familiar to any Go
programmer who has used the `encoding/json` package: struct field tags define
what and how input is mapped to those same fields. This is not unusual for Go
encoders, but is unusual for a parser.
## Tutorial
A [tutorial](TUTORIAL.md) is available, walking through the creation of an .ini parser.
## Tag syntax
Participle supports two forms of struct tag grammar syntax.
The easiest to read is when the grammar uses the entire struct tag content, eg.
```go
Field string `@Ident @("," Ident)*`
```
However, this does not coexist well with other tags such as JSON, etc. and
may cause issues with linters. If this is an issue then you can use the
`parser:""` tag format. In this case single quotes can be used to quote
literals making the tags somewhat easier to write, eg.
```go
Field string `parser:"@ident (',' Ident)*" json:"field"`
```
## Overview
A grammar is an annotated Go structure used to both define the parser grammar,
and be the AST output by the parser. As an example, following is the final INI
parser from the tutorial.
```go
type INI struct {
Properties []*Property `@@*`
Sections []*Section `@@*`
}
type Section struct {
Identifier string `"[" @Ident "]"`
Properties []*Property `@@*`
}
type Property struct {
Key string `@Ident "="`
Value *Value `@@`
}
type Value struct {
String *string ` @String`
Number *float64 `| @Float`
}
```
> **Note:** Participle also supports named struct tags (eg. Hello string `parser:"@Ident"`).
A parser is constructed from a grammar and a lexer:
```go
parser, err := participle.Build(&INI{})
```
Once constructed, the parser is applied to input to produce an AST:
```go
ast := &INI{}
err := parser.ParseString("", "size = 10", ast)
// ast == &INI{
// Properties: []*Property{
// {Key: "size", Value: &Value{Number: &10}},
// },
// }
```
## Grammar syntax
Participle grammars are defined as tagged Go structures. Participle will
first look for tags in the form `parser:"..."`. It will then fall back to
using the entire tag body.
The grammar format is:
- `@` Capture expression into the field.
- `@@` Recursively capture using the fields own type.
- `` Match named lexer token.
- `( ... )` Group.
- `"..."` or `'...'` Match the literal (note that the lexer must emit tokens matching this literal exactly).
- `"...":` Match the literal, specifying the exact lexer token type to match.
- ` ...` Match expressions.
- ` | | ...` Match one of the alternatives. Each alternative is tried in order, with backtracking.
- `~` Match any token that is _not_ the start of the expression (eg: `@~";"` matches anything but the `;` character into the field).
- `(?= ... )` Positive lookahead group - requires the contents to match further input, without consuming it.
- `(?! ... )` Negative lookahead group - requires the contents not to match further input, without consuming it.
The following modifiers can be used after any expression:
- `*` Expression can match zero or more times.
- `+` Expression must match one or more times.
- `?` Expression can match zero or once.
- `!` Require a non-empty match (this is useful with a sequence of optional matches eg. `("a"? "b"? "c"?)!`).
Notes:
- Each struct is a single production, with each field applied in sequence.
- `@` is the mechanism for capturing matches into the field.
- if a struct field is not keyed with "parser", the entire struct tag
will be used as the grammar fragment. This allows the grammar syntax to remain
clear and simple to maintain.
## Capturing
Prefixing any expression in the grammar with `@` will capture matching values
for that expression into the corresponding field.
For example:
```go
// The grammar definition.
type Grammar struct {
Hello string `@Ident`
}
// The source text to parse.
source := "world"
// After parsing, the resulting AST.
result == &Grammar{
Hello: "world",
}
```
For slice and string fields, each instance of `@` will accumulate into the
field (including repeated patterns). Accumulation into other types is not
supported.
For integer and floating point types, a successful capture will be parsed
with `strconv.ParseInt()` and `strconv.ParseFloat()` respectively.
A successful capture match into a `bool` field will set the field to true.
Tokens can also be captured directly into fields of type `lexer.Token` and
`[]lexer.Token`.
Custom control of how values are captured into fields can be achieved by a
field type implementing the `Capture` interface (`Capture(values []string)
error`).
Additionally, any field implementing the `encoding.TextUnmarshaler` interface
will be capturable too. One caveat is that `UnmarshalText()` will be called once
for each captured token, so eg. `@(Ident Ident Ident)` will be called three times.
### Capturing boolean value
By default a boolean field is used to indicate that a match occurred, which
turns out to be much more useful and common in Participle than parsing true
or false literals. For example, parsing a variable declaration with a
trailing optional syntax:
```go
type Var struct {
Name string `"var" @Ident`
Type string `":" @Ident`
Optional bool `@"?"?`
}
```
In practice this gives more useful AST's. If bool were to be parsed literally
then you'd need to have some alternate type for Optional such as string or a
custom type.
To capture literal boolean values such as `true` or `false`, implement the
Capture interface like so:
```go
type Boolean bool
func (b *Boolean) Capture(values []string) error {
*b = values[0] == "true"
return nil
}
type Value struct {
Float *float64 ` @Float`
Int *int `| @Int`
String *string `| @String`
Bool *Boolean `| @("true" | "false")`
}
```
## Streaming
Participle supports streaming parsing. Simply pass a channel of your grammar into
`Parse*()`. The grammar will be repeatedly parsed and sent to the channel. Note that
the `Parse*()` call will not return until parsing completes, so it should generally be
started in a goroutine.
```go
type token struct {
Str string ` @Ident`
Num int `| @Int`
}
parser, err := participle.Build(&token{})
tokens := make(chan *token, 128)
err := parser.ParseString("", `hello 10 11 12 world`, tokens)
for token := range tokens {
fmt.Printf("%#v\n", token)
}
```
## Lexing
Participle relies on distinct lexing and parsing phases. The lexer takes raw
bytes and produces tokens which the parser consumes. The parser transforms
these tokens into Go values.
The default lexer, if one is not explicitly configured, is based on the Go
`text/scanner` package and thus produces tokens for C/Go-like source code. This
is surprisingly useful, but if you do require more control over lexing the
builtin [`participle/lexer/stateful`](#markdown-stateful-lexer) lexer should
cover most other cases. If that in turn is not flexible enough, you can
implement your own lexer.
Configure your parser with a lexer using the `participle.Lexer()` option.
To use your own Lexer you will need to implement two interfaces:
[Definition](https://pkg.go.dev/github.com/alecthomas/participle/v2/lexer#Definition)
(and optionally [StringsDefinition](https://pkg.go.dev/github.com/alecthomas/participle/v2/lexer#StringDefinition) and [BytesDefinition](https://pkg.go.dev/github.com/alecthomas/participle/v2/lexer#BytesDefinition)) and [Lexer](https://pkg.go.dev/github.com/alecthomas/participle/v2/lexer#Lexer).
### Stateful lexer
Participle's included stateful/modal lexer provides powerful yet convenient
construction of most lexers (notably, indentation based lexers cannot be
expressed).
It is sometimes the case that a simple lexer cannot fully express the tokens
required by a parser. The canonical example of this is interpolated strings
within a larger language. eg.
```go
let a = "hello ${name + ", ${last + "!"}"}"
```
This is impossible to tokenise with a normal lexer due to the arbitrarily
deep nesting of expressions.
To support this case Participle's lexer is now stateful by default.
The lexer is a state machine defined by a map of rules keyed by the state
name. Each rule within the state includes the name of the produced token, the
regex to match, and an optional operation to apply when the rule matches.
As a convenience, any `Rule` starting with a lowercase letter will be elided from output.
Lexing starts in the `Root` group. Each rule is matched in order, with the first
successful match producing a lexeme. If the matching rule has an associated Action
it will be executed. The name of each non-root rule is prefixed with the name
of its group to yield the token identifier used during matching.
A state change can be introduced with the Action `Push(state)`. `Pop()` will
return to the previous state.
To reuse rules from another state, use `Include(state)`.
A special named rule `Return()` can also be used as the final rule in a state
to always return to the previous state.
As a special case, regexes containing backrefs in the form `\N` (where `N` is
a digit) will match the corresponding capture group from the immediate parent
group. This can be used to parse, among other things, heredocs. See the
[tests](https://github.com/alecthomas/participle/blob/master/lexer/stateful/stateful_test.go#L59)
for an example of this, among others.
### Example stateful lexer
Here's a cut down example of the string interpolation described above. Refer to
the [stateful example](https://github.com/alecthomas/participle/tree/master/_examples/stateful)
for the corresponding parser.
```go
var lexer = stateful.Must(Rules{
"Root": {
{`String`, `"`, Push("String")},
},
"String": {
{"Escaped", `\\.`, nil},
{"StringEnd", `"`, Pop()},
{"Expr", `\${`, Push("Expr")},
{"Char", `[^$"\\]+`, nil},
},
"Expr": {
Include("Root"),
{`whitespace`, `\s+`, nil},
{`Oper`, `[-+/*%]`, nil},
{"Ident", `\w+`, nil},
{"ExprEnd", `}`, Pop()},
},
})
```
### Example simple/non-stateful lexer
The Stateful lexer is now the only custom lexer supported by Participle, but
most parsers won't need this level of flexibility. To support this common
case, which replaces the old `Regex` and `EBNF` lexers, you can use
`stateful.MustSimple()` and `stateful.NewSimple()`.
eg. The lexer for a form of BASIC:
```go
var basicLexer = stateful.MustSimple([]stateful.Rule{
{"Comment", `(?i)rem[^\n]*`, nil},
{"String", `"(\\"|[^"])*"`, nil},
{"Number", `[-+]?(\d*\.)?\d+`, nil},
{"Ident", `[a-zA-Z_]\w*`, nil},
{"Punct", `[-[!@#$%^&*()+_={}\|:;"'<,>.?/]|]`, nil},
{"EOL", `[\n\r]+`, nil},
{"whitespace", `[ \t]+`, nil},
})
```
### Experimental - code generation
Participle v2 now has experimental support for generating code to perform
lexing. Use `participle/experimental/codegen.GenerateLexer()` to compile a
`stateful` lexer to Go code.
This will generally provide around a 10x improvement in lexing performance
while producing O(1) garbage.
## Options
The Parser's behaviour can be configured via [Options](https://pkg.go.dev/github.com/alecthomas/participle/v2#Option).
## Examples
There are several [examples](https://github.com/alecthomas/participle/tree/master/_examples) included:
Example | Description
--------|---------------
[BASIC](https://github.com/alecthomas/participle/tree/master/_examples/basic) | A lexer, parser and interpreter for a [rudimentary dialect](https://caml.inria.fr/pub/docs/oreilly-book/html/book-ora058.html) of BASIC.
[EBNF](https://github.com/alecthomas/participle/tree/master/_examples/ebnf) | Parser for the form of EBNF used by Go.
[Expr](https://github.com/alecthomas/participle/tree/master/_examples/expr) | A basic mathematical expression parser and evaluator.
[GraphQL](https://github.com/alecthomas/participle/tree/master/_examples/graphql) | Lexer+parser for GraphQL schemas
[HCL](https://github.com/alecthomas/participle/tree/master/_examples/hcl) | A parser for the [HashiCorp Configuration Language](https://github.com/hashicorp/hcl).
[INI](https://github.com/alecthomas/participle/tree/master/_examples/ini) | An INI file parser.
[Protobuf](https://github.com/alecthomas/participle/tree/master/_examples/protobuf) | A full [Protobuf](https://developers.google.com/protocol-buffers/) version 2 and 3 parser.
[SQL](https://github.com/alecthomas/participle/tree/master/_examples/sql) | A *very* rudimentary SQL SELECT parser.
[Stateful](https://github.com/alecthomas/participle/tree/master/_examples/stateful) | A basic example of a stateful lexer and corresponding parser.
[Thrift](https://github.com/alecthomas/participle/tree/master/_examples/thrift) | A full [Thrift](https://thrift.apache.org/docs/idl) parser.
[TOML](https://github.com/alecthomas/participle/tree/master/_examples/toml) | A [TOML](https://github.com/toml-lang/toml) parser.
Included below is a full GraphQL lexer and parser:
```go
package main
import (
"fmt"
"os"
"github.com/alecthomas/kong"
"github.com/alecthomas/repr"
"github.com/alecthomas/participle/v2"
"github.com/alecthomas/participle/v2/lexer"
"github.com/alecthomas/participle/v2/lexer/stateful"
)
type File struct {
Entries []*Entry `@@*`
}
type Entry struct {
Type *Type ` @@`
Schema *Schema `| @@`
Enum *Enum `| @@`
Scalar string `| "scalar" @Ident`
}
type Enum struct {
Name string `"enum" @Ident`
Cases []string `"{" @Ident* "}"`
}
type Schema struct {
Fields []*Field `"schema" "{" @@* "}"`
}
type Type struct {
Name string `"type" @Ident`
Implements string `( "implements" @Ident )?`
Fields []*Field `"{" @@* "}"`
}
type Field struct {
Name string `@Ident`
Arguments []*Argument `( "(" ( @@ ( "," @@ )* )? ")" )?`
Type *TypeRef `":" @@`
Annotation string `( "@" @Ident )?`
}
type Argument struct {
Name string `@Ident`
Type *TypeRef `":" @@`
Default *Value `( "=" @@ )`
}
type TypeRef struct {
Array *TypeRef `( "[" @@ "]"`
Type string ` | @Ident )`
NonNullable bool `( @"!" )?`
}
type Value struct {
Symbol string `@Ident`
}
var (
graphQLLexer = stateful.MustSimple([]stateful.Rule{
{"Comment", `(?:#|//)[^\n]*\n?`, nil},
{"Ident", `[a-zA-Z]\w*`, nil},
{"Number", `(?:\d*\.)?\d+`, nil},
{"Punct", `[-[!@#$%^&*()+_={}\|:;"'<,>.?/]|]`, nil},
{"Whitespace", `[ \t\n\r]+`, nil},
})
parser = participle.MustBuild(&File{},
participle.Lexer(graphQLLexer),
participle.Elide("Comment", "Whitespace"),
participle.UseLookahead(2),
)
)
var cli struct {
EBNF bool `help"Dump EBNF."`
Files []string `arg:"" optional:"" type:"existingfile" help:"GraphQL schema files to parse."`
}
func main() {
ctx := kong.Parse(&cli)
if cli.EBNF {
fmt.Println(parser.String())
ctx.Exit(0)
}
for _, file := range cli.Files {
ast := &File{}
r, err := os.Open(file)
ctx.FatalIfErrorf(err)
err = parser.Parse(file, r, ast)
r.Close()
repr.Println(ast)
ctx.FatalIfErrorf(err)
}
}
```
## Performance
One of the included examples is a complete Thrift parser
(shell-style comments are not supported). This gives
a convenient baseline for comparing to the PEG based
[pigeon](https://github.com/PuerkitoBio/pigeon), which is the parser used by
[go-thrift](https://github.com/samuel/go-thrift). Additionally, the pigeon
parser is utilising a generated parser, while the participle parser is built at
run time.
You can run the benchmarks yourself, but here's the output on my machine:
BenchmarkParticipleThrift-12 5941 201242 ns/op 178088 B/op 2390 allocs/op
BenchmarkGoThriftParser-12 3196 379226 ns/op 157560 B/op 2644 allocs/op
On a real life codebase of 47K lines of Thrift, Participle takes 200ms and go-
thrift takes 630ms, which aligns quite closely with the benchmarks.
## Concurrency
A compiled `Parser` instance can be used concurrently. A `LexerDefinition` can be used concurrently. A `Lexer` instance cannot be used concurrently.
## Error reporting
There are a few areas where Participle can provide useful feedback to users of your parser.
1. Errors returned by [Parser.Parse*()](https://pkg.go.dev/github.com/alecthomas/participle/v2#Parser.Parse) will be of type [Error](https://pkg.go.dev/github.com/alecthomas/participle/v2#Error). This will contain positional information where available.
2. Participle will make a best effort to return as much of the AST up to the error location as possible.
3. Any node in the AST containing a field `Pos lexer.Position` will be automatically
populated from the nearest matching token.
4. Any node in the AST containing a field `EndPos lexer.Position` will be
automatically populated from the token at the end of the node.
5. Any node in the AST containing a field `Tokens []lexer.Token` will be automatically
populated with _all_ tokens captured by the node, _including_ elided tokens.
These related pieces of information can be combined to provide fairly comprehensive error reporting.
## Limitations
Internally, Participle is a recursive descent parser with backtracking (see
`UseLookahead(K)`).
Among other things, this means that they do not support left recursion. Left
recursion must be eliminated by restructuring your grammar.
## EBNF
Participle supports outputting an EBNF grammar from a Participle parser. Once
the parser is constructed simply call `String()`.
Participle also [includes a parser](https://pkg.go.dev/github.com/alecthomas/participle/v2/ebnf) for this form of EBNF (naturally).
eg. The [GraphQL example](https://github.com/alecthomas/participle/blob/master/_examples/graphql/main.go#L15-L62)
gives in the following EBNF:
```ebnf
File = Entry* .
Entry = Type | Schema | Enum | "scalar" ident .
Type = "type" ident ("implements" ident)? "{" Field* "}" .
Field = ident ("(" (Argument ("," Argument)*)? ")")? ":" TypeRef ("@" ident)? .
Argument = ident ":" TypeRef ("=" Value)? .
TypeRef = "[" TypeRef "]" | ident "!"? .
Value = ident .
Schema = "schema" "{" Field* "}" .
Enum = "enum" ident "{" ident* "}" .
```
## Syntax/Railroad Diagrams
Participle includes a [command-line utility]() to take an EBNF representation of a Participle grammar
(as returned by `Parser.String()`) and produce a Railroad Diagram using
[tabatkins/railroad-diagrams](https://github.com/tabatkins/railroad-diagrams).
Here's what the GraphQL grammar looks like:
