First, thank you to my employer Rob Mee of Pivotal Labs for funding a substantial portion of Treetop's development. He gets it.
I'd also like to thank:
* Damon McCormick for several hours of pair programming.
* Nick Kallen for lots of well-considered feedback and a few afternoons of programming.
* Brian Takita for a night of pair programming.
* Eliot Miranda for urging me rewrite as a compiler right away rather than putting it off.
* Ryan Davis and Eric Hodel for hurting my code.
* Dav Yaginuma for kicking me into action on my idea.
* Bryan Ford for his seminal work on Packrat Parsers.
* The editors of Lambda the Ultimate, where I discovered parsing expression grammars.
treetop-1.6.12/doc/pitfalls_and_advanced_techniques.markdown 0000664 0000000 0000000 00000007346 14337774710 0024340 0 ustar 00root root 0000000 0000000 #Pitfalls
##Left Recursion
An weakness shared by all recursive descent parsers is the inability to parse left-recursive rules. Consider the following rule:
rule left_recursive
left_recursive 'a' / 'a'
end
Logically it should match a list of 'a' characters. But it never consumes anything, because attempting to recognize `left_recursive` begins by attempting to recognize `left_recursive`, and so goes an infinite recursion. There's always a way to eliminate these types of structures from your grammar. There's a mechanistic transformation called _left factorization_ that can eliminate it, but it isn't always pretty, especially in combination with automatically constructed syntax trees. So far, I have found more thoughtful ways around the problem. For instance, in the interpreter example I interpret inherently left-recursive function application right recursively in syntax, then correct the directionality in my semantic interpretation. You may have to be clever.
#Advanced Techniques
Here are a few interesting problems I've encountered. I figure sharing them may give you insight into how these types of issues are addressed with the tools of parsing expressions.
##Matching a String
rule string
'"' ('\"' / !'"' .)* '"'
end
This expression says: Match a quote, then zero or more of, an escaped quote or any character but a quote, followed by a quote. Lookahead assertions are essential for these types of problems.
##Matching Nested Structures With Non-Unique Delimeters
Say I want to parse a diabolical wiki syntax in which the following interpretations apply.
** *hello* ** --> hello
* **hello** * --> hello
rule strong
'**' (em / !'*' . / '\*')+ '**'
end
rule em
'*' (strong / !'*' . / '\*')+ '*'
end
Emphasized text is allowed within strong text by virtue of `em` being the first alternative. Since `em` will only successfully parse if a matching `*` is found, it is permitted, but other than that, no `*` characters are allowed unless they are escaped.
##Matching a Keyword But Not Words Prefixed Therewith
Say I want to consider a given string a characters only when it occurs in isolation. Lets use the `end` keyword as an example. We don't want the prefix of `'enders_game'` to be considered a keyword. A naiive implementation might be the following.
rule end_keyword
'end' &space
end
This says that `'end'` must be followed by a space, but this space is not consumed as part of the matching of `keyword`. This works in most cases, but is actually incorrect. What if `end` occurs at the end of the buffer? In that case, it occurs in isolation but will not match the above expression. What we really mean is that `'end'` cannot be followed by a _non-space_ character.
rule end_keyword
'end' !(!' ' .)
end
In general, when the syntax gets tough, it helps to focus on what you really mean. A keyword is a character not followed by another character that isn't a space.
## Poor Performance with Large Unicode Strings
Treetop may perform poorly when parsing very large (more than 100KB) unicode strings. This is due to the fact that substring lookups on Ruby unicode strings are linear-time operations, and not constant-time operations like they are on ASCII encoded strings. This means that parse times for larger strings can be exponentially worse than for smaller strings.
If your input and grammar only expect ASCII strings, you can achieve significant performance improvements for large strings by re-encoding them to ASCII using `input.encode(Encoding::US_ASCII)`. See [this issue on GitHub](https://github.com/cjheath/treetop/issues/31) for more information and other possible workarounds for unicode strings.
treetop-1.6.12/doc/semantic_interpretation.markdown 0000664 0000000 0000000 00000016375 14337774710 0022557 0 ustar 00root root 0000000 0000000 #Semantic Interpretation
Lets use the below grammar as an example. It describes parentheses wrapping a single character to an arbitrary depth.
grammar ParenLanguage
rule parenthesized_letter
'(' parenthesized_letter ')'
/
[a-z]
end
end
Matches:
* `'a'`
* `'(a)'`
* `'((a))'`
* etc.
Output from a parser for this grammar looks like this:
This is a parse tree whose nodes are instances of `Treetop::Runtime::SyntaxNode`. What if we could define methods on these node objects? We would then have an object-oriented program whose structure corresponded to the structure of our language. Treetop provides two techniques for doing just this.
##Associating Methods with Node-Instantiating Expressions
Sequences and all types of terminals are node-instantiating expressions. When they match, they create instances of `Treetop::Runtime::SyntaxNode`. Methods can be added to these nodes in the following ways:
###Inline Method Definition
Methods can be added to the nodes instantiated by the successful match of an expression
grammar ParenLanguage
rule parenthesized_letter
'(' parenthesized_letter ')' {
def depth
parenthesized_letter.depth + 1
end
}
/
[a-z] {
def depth
0
end
}
end
end
Note that each alternative expression is followed by a block containing a method definition. A `depth` method is defined on both expressions. The recursive `depth` method defined in the block following the first expression determines the depth of the nested parentheses and adds one to it. The base case is implemented in the block following the second expression; a single character has a depth of 0.
###Custom `SyntaxNode` Subclass Declarations
You can instruct the parser to instantiate a custom subclass of Treetop::Runtime::SyntaxNode for an expression by following it by the name of that class enclosed in angle brackets (`<>`). The above inline method definitions could have been moved out into a single class like so.
# in .treetop file
grammar ParenLanguage
rule parenthesized_letter
'(' parenthesized_letter ')'
terminal?
|
Was this node produced by the matching of a terminal symbol? |
nonterminal?
|
Was this node produced by the matching of a nonterminal symbol? |
text_value
|
The substring of the input represented by this node. |
elements
|
Available only on nonterminal nodes, returns the nodes parsed by the elements of the matched sequence. |
input
|
The entire input string, which is useful mainly in conjunction with interval
|
interval
|
The Range of characters in input matched by this rule
|
empty?
|
returns true if this rule matched no characters of input |
inspect
|
Handy-dandy method that returns an indented subtree dump of the syntax tree starting here. This dump includes, for each node, the offset and a snippet of the text this rule matched, and the names of mixin modules and the accessor and extension methods. |
rule alts
( foo bar / baz '' )
{
def value
elements.map{|e| e.text_value }
end
}
end
##Nonterminal Symbols
Nonterminal symbols are unquoted references to other named rules. They are equivalent to an inline substitution of the named expression.
rule foo
"the dog " bar
end
rule bar
"jumped"
end
The above grammar is equivalent to:
rule foo
"the dog jumped"
end
##Ordered Choice
Parsers attempt to match ordered choices in left-to-right order, and stop after the first successful match.
"foobar" / "foo" / "bar"
Note that if `"foo"` in the above expression came first, `"foobar"` would never be matched.
Note also that the above rule will match `"bar"` as a prefix of `"barbie"`.
Care is required when it's desired to match language keywords exactly.
##Sequences
Sequences are a space-separated list of parsing expressions. They have higher precedence than choices, so choices must be parenthesized to be used as the elements of a sequence.
"foo" "bar" ("baz" / "bop")
##Zero or More
Parsers will greedily match an expression zero or more times if it is followed by the star (`*`) symbol.
* `'foo'*` matches the empty string, `"foo"`, `"foofoo"`, etc.
##One or More
Parsers will greedily match an expression one or more times if it is followed by the plus (`+`) symbol.
* `'foo'+` does not match the empty string, but matches `"foo"`, `"foofoo"`, etc.
##Optional Expressions
An expression can be declared optional by following it with a question mark (`?`).
* `'foo'?` matches `"foo"` or the empty string.
##Repetition count
A generalised repetition count (minimum, maximum) is also available.
* `'foo' 2..` matches `'foo'` two or more times
* `'foo' 3..5` matches `'foo'` from three to five times
* `'foo' ..4` matches `'foo'` from zero to four times
##Lookahead Assertions
Lookahead assertions can be used to make parsing expressions context-sensitive.
The parser will look ahead into the buffer and attempt to match an expression without consuming input.
###Positive Lookahead Assertion
Preceding an expression with an ampersand `(&)` indicates that it must match, but no input will be consumed in the process of determining whether this is true.
* `"foo" &"bar"` matches `"foobar"` but only consumes up to the end `"foo"`. It will not match `"foobaz"`.
###Negative Lookahead Assertion
Preceding an expression with a bang `(!)` indicates that the expression must not match, but no input will be consumed in the process of determining whether this is true.
* `"foo" !"bar"` matches `"foobaz"` but only consumes up to the end `"foo"`. It will not match `"foobar"`.
Note that a lookahead assertion may be used on any rule, not just a string terminal.
rule things
thing (!(disallowed / ',') following)*
end
Here's a common use case:
rule word
[a-zA-Z]+
end
rule conjunction
primitive ('and' ' '+ primitive)*
end
rule primitive
(!'and' word ' '+)*
end
Here's the easiest way to handle C-style comments:
rule c_comment
'/*'
(
!'*/'
(. / "\n")
)*
'*/'
end
##Semantic predicates (positive and negative)
Sometimes you must execute Ruby code during parsing in order to decide how to proceed.
This is an advanced feature, and must be used with great care, because it can change the
way a Treetop parser backtracks in a way that breaks the parsing algorithm. See the
notes below on how to use this feature safely.
The code block is the body of a Ruby lambda block, and should return true or false, to cause this
parse rule to continue or fail (for positive sempreds), fail or continue (for negative sempreds).
* `&{ ... }` Evaluate the block and fail this rule if the result is false or nil
* `!{ ... }` Evaluate the block and fail this rule if the result is not false or nil
The lambda is passed a single argument which is the array of syntax nodes matched so far in the
current sequence. Note that because the current rule has not yet succeeded, no syntax node has
yet been constructed, and so the lambda block is being run in a context where the `names` of the
preceding rules (or as assigned by labels) are not available to access the sub-rules.
rule id
[a-zA-Z][a-zA-Z0-9]*
{
def is_reserved
ReservedSymbols.include? text_value
end
}
end
rule foo_rule
foo id &{|seq| seq[1].is_reserved } baz
end
Match "foo id baz" only if `id.is_reserved`. Note that `id` cannot be referenced by name from `foo_rule`,
since that rule has not yet succeeded, but `id` has completed and so its added methods are available.
rule test_it
foo bar &{|s| debugger; true } baz
end
Match `foo` then `bar`, stop to enter the debugger (make sure you have said `require "ruby-debug"` somewhere),
then continue by trying to match `baz`.
Treetop, like other PEG parsers, achieves its performance guarantee by remembering which rules it has
tried at which locations in the input, and what the result was. This process, called memoization,
requires that the rule would produce the same result (if run again) as it produced the first time when
the result was remembered. If you violate this principle in your semantic predicates, be prepared to
fight Cerberus before you're allowed out of Hades again.
There's an example of how to use semantic predicates to parse a language with white-space indented blocks
in the examples directory.
treetop-1.6.12/doc/tt.1 0000664 0000000 0000000 00000004613 14337774710 0014562 0 ustar 00root root 0000000 0000000 .\" treetop - Bringing the simplicity of Ruby to syntactic analysis
.\"
.\" Copyright (c) 2007 Nathan Sobo.
.\"
.\" Permission is hereby granted, free of charge, to any person obtaining a copy
.\" of this software and associated documentation files (the "Software"), to deal
.\" in the Software without restriction, including without limitation the rights
.\" to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
.\" copies of the Software, and to permit persons to whom the Software is
.\" furnished to do so, subject to the following conditions:
.\"
.\" The above copyright notice and this permission notice shall be included in
.\" all copies or substantial portions of the Software.
.\"
.\" THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
.\" IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
.\" FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
.\" AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
.\" LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
.\" OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
.\" THE SOFTWARE.
.TH tt 1 2013-06-19 Treetop "Treetop v1.4.14"
.SH NAME
tt \- Compile a treetop grammar file to ruby source code
.SH SYNOPSIS
.B tt
.RI [ options "] " grammar_file "[.treetop|.tt] ..."
.SH DESCRIPTION
The
.B tt
program is a command-line script to compile .treetop files into Ruby
source code.
The
.B tt
program takes a list of files with a .treetop extension and compiles
them into .rb files of the same name. You can then require these files
like any other Ruby script.
Alternately, you can supply just one .treetop file and a \-o flag to
specify the name of the output file.
Note: while treetop grammar files
.B must
have a supported filename extensions, (.treetop or .tt), the extension
name is not required when calling the compiler with grammar file
names.
.SH OPTIONS
.TP 4
.BI "\-o, \-\-output" " FILENAME"
Write parser source to
.I FILENAME.
.TP 4
.B \-f, \-\-force
Overwrite existing output file(s)
.TP 4
.B \-v, \-\-version
Show Treetop version
.TP 4
.B \-h, \-\-help
.SH EXAMPLES
.TP 4
1 grammar -> 1 parser source
tt foo.tt
.TP 4
2 grammars -> 2 separate parsers
tt foo bar.treetop
.TP 4
Alternately named output file
tt \-o alterate_name.rb foo
.SH SEE ALSO
The treetop website:
.B http://cjheath.github.io/treetop/
treetop-1.6.12/doc/using_in_ruby.markdown 0000664 0000000 0000000 00000010743 14337774710 0020472 0 ustar 00root root 0000000 0000000 #Using Treetop Grammars in Ruby
##Using the Command Line Compiler
You can compile `.treetop` files into Ruby source code with the `tt` command line script. `tt` takes an list of files with a `.treetop` extension and compiles them into `.rb` files of the same name. You can then `require` these files like any other Ruby script. Alternately, you can supply just one `.treetop` file and a `-o` flag to name specify the name of the output file. Improvements to this compilation script are welcome.
tt foo.treetop bar.treetop
tt foo.treetop -o foogrammar.rb
##Loading A Grammar Directly
The Polyglot gem makes it possible to load `.treetop` or `.tt` files directly with `require`. This will invoke `Treetop.load`, which automatically compiles the grammar to Ruby and then evaluates the Ruby source. If you are getting errors in methods you define on the syntax tree, try using the command line compiler for better stack trace feedback. A better solution to this issue is in the works.
In order to use Polyglot dynamic loading of `.treetop` or `.tt` files though, you need to require the Polyglot gem before you require the Treetop gem as Treetop will only create hooks into Polyglot for the treetop files if Polyglot is already loaded. So you need to use:
require 'polyglot'
require 'treetop'
in order to use Polyglot auto loading with Treetop in Ruby.
##Instantiating and Using Parsers
If a grammar by the name of `Foo` is defined, the compiled Ruby source will define a `FooParser` class.
To parse input, create an instance and call its `parse` method with a string.
The parser will return the syntax tree of the match or `nil` if there is a failure.
Note that by default, the parser will fail unless *all* input is consumed.
Treetop.load "arithmetic"
parser = ArithmeticParser.new
if parser.parse('1+1')
puts 'success'
else
puts 'failure'
end
##Parser Options
A Treetop parser has several options you may set.
Some are settable permanently by methods on the parser, but all may be passed in as options to the `parse` method.
parser = ArithmeticParser.new
input = 'x = 2; y = x+3;'
# Temporarily override an option:
result1 = parser.parse(input, :consume_all_input => false)
puts "consumed #{parser.index} characters"
parser.consume_all_input = false
result1 = parser.parse(input)
puts "consumed #{parser.index} characters"
# Continue the parse with the next character:
result2 = parser.parse(input, :index => parser.index)
# Parse, but match rule `variable` instead of the normal root rule:
parser.parse(input, :root => :variable)
parser.root = :variable # Permanent setting
If you have a statement-oriented language, you can save memory by parsing just one statement at a time,
and discarding the parse tree after each statement.
##Learning From Failure
If a parse fails, it returns nil. In this case, you can ask the parser for an explanation.
The failure reasons include the terminal nodes which were tried at the furthermost point the parse reached.
parser = ArithmeticParser.new
result = parser.parse('4+=3')
if !result
puts parser.failure_reason
puts parser.failure_line
puts parser.failure_column
end
=>
Expected one of (, - at line 1, column 3 (byte 3) after +
1
3
##Using Parse Results
Please don't try to walk down the syntax tree yourself, and please don't use the tree as your own convenient data structure.
It contains many more nodes than your application needs, often even more than one for every character of input.
parser = ArithmeticParser.new
p parser.parse('2+3')
=>
SyntaxNode+Additive1 offset=0, "2+3" (multitive):
SyntaxNode+Multitive1 offset=0, "2" (primary):
SyntaxNode+Number0 offset=0, "2":
SyntaxNode offset=0, ""
SyntaxNode offset=0, "2"
SyntaxNode offset=1, ""
SyntaxNode offset=1, ""
SyntaxNode offset=1, "+3":
SyntaxNode+Additive0 offset=1, "+3" (multitive):
SyntaxNode offset=1, "+"
SyntaxNode+Multitive1 offset=2, "3" (primary):
SyntaxNode+Number0 offset=2, "3":
SyntaxNode offset=2, ""
SyntaxNode offset=2, "3"
SyntaxNode offset=3, ""
SyntaxNode offset=3, ""
Instead, add methods to the root rule which return the information you require in a sensible form.
Each rule can call its sub-rules, and this method of walking the syntax tree is much preferable to
attempting to walk it from the outside.
treetop-1.6.12/examples/ 0000775 0000000 0000000 00000000000 14337774710 0015116 5 ustar 00root root 0000000 0000000 treetop-1.6.12/examples/indented_blocks/ 0000775 0000000 0000000 00000000000 14337774710 0020245 5 ustar 00root root 0000000 0000000 treetop-1.6.12/examples/indented_blocks/indented_blocks.tt 0000664 0000000 0000000 00000003064 14337774710 0023750 0 ustar 00root root 0000000 0000000 grammar IndentedBlocks
rule top
# Initialise the indent stack with a sentinel:
&{|s| @indents = [-1] }
foo:('foo'?)
nested_blocks
{
def inspect
nested_blocks.inspect
end
}
end
rule nested_blocks
(
# Do not try to extract this semantic predicate into a new rule.
# It will be memo-ized incorrectly because @indents.last will change.
!{|s|
# Peek at the following indentation:
save = index; i = _nt_indentation; index = save
# We're closing if the indentation is less or the same as our enclosing block's:
closing = i.text_value.length <= @indents.last
}
block
)*
{
def inspect
elements.map{|e| e.block.inspect}*"\n"
end
}
end
rule block
indented_line # The block's opening line
&{|s| # Push the indent level to the stack
level = s[0].indentation.text_value.length
@indents << level
true
}
nested_blocks # Parse any nested blocks
&{|s| # Pop the indent stack
# Note that under no circumstances should "nested_blocks" fail, or the stack will be mis-aligned
@indents.pop
true
}
{
def inspect
indented_line.inspect +
(nested_blocks.elements.size > 0 ? (
"\n{\n" +
nested_blocks.elements.map { |content|
content.block.inspect+"\n"
}*'' +
"}"
)
: "")
end
}
end
rule indented_line
indentation text:((!"\n" .)*) "\n"
{
def inspect
text.text_value
end
}
end
rule indentation
' '*
end
end
treetop-1.6.12/examples/indented_blocks/indented_blocks_test.rb 0000664 0000000 0000000 00000000722 14337774710 0024761 0 ustar 00root root 0000000 0000000 require 'polyglot'
require 'byebug'
require 'treetop'
require 'indented_blocks'
parser = IndentedBlocksParser.new
input = <