Yuku

AST

A parse returns an ast.Tree, a flat tree whose nodes, child lists, and strings live in arrays in one arena. From JavaScript, the tree decodes to ESTree.

TypeIs
NodeIndexA node’s position in tree.nodes, .null for an absent child
IndexRangeA child list, a start and len in tree.extras
StringText, a zero-copy slice of the source or an entry in the pool
SpanA source range in bytes, start and end

Reading

tree.root          // the program node
tree.data(index)   // its payload, a NodeData union
tree.span(index)
tree.extra(range)  // the nodes of a child list
tree.string(text)  // the bytes of a String

tree.source        // also lang and source_type
tree.isTs()        // also isJsx and isModule
switch (tree.data(index)) {
    .variable_declaration => |decl| {
        for (tree.extra(decl.declarators)) |declarator| {
            _ = declarator;
        }
    },
    .identifier_reference => |id| std.debug.print("{s}\n", .{tree.string(id.name)}),
    else => {},
}

NodeData has a tag per node kind, each the snake_case name of its payload, as in ts_as_expression: TSAsExpression. ast.zig defines every one with its fields. To walk the whole tree, use the traverser.

data.isExpression()             // also isStatement, isDeclaration, isLiteral, isPattern,
                                // isCallable, isIteration, and isTypeContext

literal.value(&tree)            // a numeric literal's f64
ast.BinaryOperator.fromToken(t) // also toToken and toString, on LogicalOperator,
                                // UnaryOperator, UpdateOperator, and AssignmentOperator
kind.toString()                 // on VariableKind, PropertyKind, Accessibility, and the other kinds

Building and editing

var tree = ast.Tree.init(allocator, ""); // or the source to edit
defer tree.deinit();

const name = try tree.addString("total"); // interned
const id = try tree.addNode(.{ .identifier_reference = .{ .name = name } }, span);
const list = try tree.addExtra(&.{id});   // a child list

tree.setData(index, data);
tree.setSpan(index, span);
tree.setName(index, name);                // any node a declaration or reference can be
tree.sourceSlice(start, end);             // a String over the source, no copy
tree.allocator();                         // the arena

A built tree sets tree.root itself. The transform traverser makes these changes during a walk.

Comments

.comments = .collect keeps every comment in tree.comments, in source order. .attach also gives each to the node it prints with, so it moves with its node and codegen prints it in place.

pub const Comment = struct {
    type: Type,         // .line or .block
    value: String,      // without delimiters
    span: Span,         // empty for an added comment
    position: Position, // .before, .after, or .inside an otherwise empty node
    same_line: bool,
};
var comments = tree.commentsOf(node);
while (comments.next()) |comment| std.debug.print("{s}\n", .{tree.string(comment.value)});

tree.hasComments(node);
tree.commentsIn(tree.span(node)); // the source comments within a span
try tree.addComment(node, .{ .type = .block, .value = try tree.addString("#__PURE__") });
try tree.moveComments(from, to);  // to .null drops them

Tokens

.tokens = true keeps every token in tree.tokens, as the parser resolved it, ending with eof.

for (tree.tokens) |token| std.debug.print("{t} {s}\n", .{ token.tag, token.text(tree.source) });
token.span
token.tag                       // an ast.TokenTag
token.len()
token.hasLineTerminatorBefore() // what ASI reads
token.isEscaped()               // also hasInvalidEscape and hasLoneSurrogates

token.tag.toString()            // its spelling, null for literals and names
token.tag.precedence()          // an ast.Precedence, 0 when none
token.tag.isKeyword()           // also isReserved, isUnconditionallyReserved, isStrictModeReserved,
                                // isIdentifierLike, isNumericLiteral, isBinaryOperator,
                                // isLogicalOperator, isUnaryOperator, and isAssignmentOperator

tree.tokensIn(tree.span(node))  // a node's tokens, a contiguous run

Every node but program, template_element, and jsx_empty_expression starts and ends on a token boundary.

Lexical grammar

parser.lexical answers as the lexer does, on UTF-8 names with escapes decoded.

lexical.isValidIdentifier("class"); // false, reserved, as is await
lexical.isEvalOrArguments("eval");  // true, which strict code cannot bind
lexical.isIdentifierName("class");  // true, reserved words included
lexical.isIdentifierStart('$');     // also isIdentifierChar and isJSXIdentifierChar
lexical.keyword("yield");           // ?TokenTag, contextual keywords included
lexical.lineBreakLen(source, i);    // 0, or the byte length of the line terminator at i

try lexical.decodeStringEscapes(gpa, raw, &out); // appends WTF-8
try lexical.decodeJSXEntities(gpa, raw, &out);