Yuku
AST
A parse returns an ast.Tree, a flat tree whose nodes, child lists, and strings live in arrays in one arena. From JavaScript, the tree decodes to ESTree.
| Type | Is |
|---|---|
NodeIndex | A node’s position in tree.nodes, .null for an absent child |
IndexRange | A child list, a start and len in tree.extras |
String | Text, a zero-copy slice of the source or an entry in the pool |
Span | A source range in bytes, start and end |
Reading
tree.root // the program node
tree.data(index) // its payload, a NodeData union
tree.span(index)
tree.extra(range) // the nodes of a child list
tree.string(text) // the bytes of a String
tree.source // also lang and source_type
tree.isTs() // also isJsx and isModule
switch (tree.data(index)) {
.variable_declaration => |decl| {
for (tree.extra(decl.declarators)) |declarator| {
_ = declarator;
}
},
.identifier_reference => |id| std.debug.print("{s}\n", .{tree.string(id.name)}),
else => {},
}
NodeData has a tag per node kind, each the snake_case name of its payload, as in ts_as_expression: TSAsExpression. ast.zig defines every one with its fields. To walk the whole tree, use the traverser.
data.isExpression() // also isStatement, isDeclaration, isLiteral, isPattern,
// isCallable, isIteration, and isTypeContext
literal.value(&tree) // a numeric literal's f64
ast.BinaryOperator.fromToken(t) // also toToken and toString, on LogicalOperator,
// UnaryOperator, UpdateOperator, and AssignmentOperator
kind.toString() // on VariableKind, PropertyKind, Accessibility, and the other kinds
Building and editing
var tree = ast.Tree.init(allocator, ""); // or the source to edit
defer tree.deinit();
const name = try tree.addString("total"); // interned
const id = try tree.addNode(.{ .identifier_reference = .{ .name = name } }, span);
const list = try tree.addExtra(&.{id}); // a child list
tree.setData(index, data);
tree.setSpan(index, span);
tree.setName(index, name); // any node a declaration or reference can be
tree.sourceSlice(start, end); // a String over the source, no copy
tree.allocator(); // the arena
A built tree sets tree.root itself. The transform traverser makes these changes during a walk.
Tokens
.tokens = true keeps every token in tree.tokens, as the parser resolved it, ending with eof.
for (tree.tokens) |token| std.debug.print("{t} {s}\n", .{ token.tag, token.text(tree.source) });
token.span
token.tag // an ast.TokenTag
token.len()
token.hasLineTerminatorBefore() // what ASI reads
token.isEscaped() // also hasInvalidEscape and hasLoneSurrogates
token.tag.toString() // its spelling, null for literals and names
token.tag.precedence() // an ast.Precedence, 0 when none
token.tag.isKeyword() // also isReserved, isUnconditionallyReserved, isStrictModeReserved,
// isIdentifierLike, isNumericLiteral, isBinaryOperator,
// isLogicalOperator, isUnaryOperator, and isAssignmentOperator
tree.tokensIn(tree.span(node)) // a node's tokens, a contiguous run
Every node but program, template_element, and jsx_empty_expression starts and ends on a token boundary.
Lexical grammar
parser.lexical answers as the lexer does, on UTF-8 names with escapes decoded.
lexical.isValidIdentifier("class"); // false, reserved, as is await
lexical.isEvalOrArguments("eval"); // true, which strict code cannot bind
lexical.isIdentifierName("class"); // true, reserved words included
lexical.isIdentifierStart('$'); // also isIdentifierChar and isJSXIdentifierChar
lexical.keyword("yield"); // ?TokenTag, contextual keywords included
lexical.lineBreakLen(source, i); // 0, or the byte length of the line terminator at i
try lexical.decodeStringEscapes(gpa, raw, &out); // appends WTF-8
try lexical.decodeJSXEntities(gpa, raw, &out);
Comments
.comments = .collectkeeps every comment intree.comments, in source order..attachalso gives each to the node it prints with, so it moves with its node and codegen prints it in place.