picking up where I left off

last time I wrote about validating my PEG parser against real-world JavaScript, I was at 54 out of 69 files passing. the grammar had gaps — template literals, regex ambiguity, destructuring depth. I fixed those, added more test files, and got to a new target: 76 production JS files totaling about 2.5 MB. Three.js, lodash, chalk, Express, Node internals.

the grammar fixes alone got me to 76/76 on small-to-medium files. then I tried lodash.js — 545 KB of JavaScript — and the parser got OOM-killed.

the allocation problem

the issue was death by a thousand allocations. every time the VM popped a stack entry, it was calling slice.to_dynamic which allocates a new backing array. on a file with thousands of nested expressions, that's millions of small allocations that the default allocator has to track individually. the memory profile looked like confetti — tiny fragments everywhere, nothing getting freed until the parse was done.

the fix was two things. first, I wrapped the entire match() call in a growing virtual arena:

arena := new(virtual.Arena)
virtual.arena_init_growing(arena)
context.allocator = virtual.arena_allocator(arena)

bump allocation means every alloc is just incrementing a pointer. no fragmentation, no per-allocation bookkeeping. when the parse is done, you free the whole arena in one shot.

second, I fixed the stack operations to stop allocating on every pop. instead of creating a new dynamic array from a slice, I just write directly to the Raw_Dynamic_Array.len field to shrink in place. same for adding captures — pre-allocate with the right capacity instead of converting from a slice.

the results were dramatic. three-webgl.js (106 KB) went from 3.4 seconds to 63 milliseconds — a 57× speedup. lodash.js went from OOM-killed to parsing in 2.2 seconds at 67 MB peak memory. I also capped the memoization table at 100K entries so it can't grow unbounded on huge inputs. entries past the cap just get dropped — the parse stays correct, just potentially slower for those portions.

ten grammar fixes in one commit

with performance sorted, I ran the full 76-file suite and found 10 more grammar bugs. each one was a real ES2025 feature I'd missed:

accessor keyword — ES2025 decorators include accessor for auto-accessor class fields. I needed a new AutoAccessorDefinition rule.

legacy octals — files still using 0755 style octal literals. I'd only handled the modern 0o755 syntax.

for-using — ES2025 explicit resource management lets you write for (using x of iterable). I had using in declarations but not in for-loop heads.

nested new — new new Foo() is valid JavaScript. my NewExpression rule wasn't self-recursive so it only handled one level of new.

import() with two args — import attributes allow import("./mod", { with: { type: "json" } }). my ImportCall rule only accepted one argument.

unicode identifiers — identifiers can contain characters from the Latin-1 supplement range (U+0080–U+00FF). I had to extend the IdentStart character class to [\200-\377] in octal, which also meant fixing the meta-grammar to support the full octal range up to \377 instead of stopping at \277.

NBSP and BOM — non-breaking space (\u00A0) and byte-order mark (\uFEFF) are valid whitespace in JavaScript. two characters I'd never thought about.

the other three were an assert clause variant for import attributes (assert vs with), labeled function declarations in sloppy mode, and hashbang comments at the start of scripts.

after all ten fixes: 76 out of 76 passing. the grammar is now 802 lines with about 304 rules.

the ASI problem

JavaScript has Automatic Semicolon Insertion — the parser is supposed to insert a semicolon when it encounters a newline in certain contexts. return\nvalue should parse as return; followed by value, not return value. this affects return, throw, break, continue, yield, postfix ++/--, and arrow functions.

my original approach was !LineTerminator — a negative lookahead that checks if the next character is a newline. but that's semantically wrong. ASI doesn't care what the next character is. it cares whether a newline appeared between the previous token and the current position. !LineTerminator is a lookahead. ASI needs a lookbehind.

PEG grammars don't have lookbehind. they only have lookahead (& and !). so I had to extend the engine.

I added a zero-width assertion called %begin_line. it succeeds at the start of input or immediately after a newline character. in the grammar, !%begin_line means "fail if we're at the beginning of a line" — which is exactly "fail if a newline just happened." the grammar change is clean:

# before — wrong semantics
ReturnStatement <- RETURN (!LineTerminator Expression)? EOS

# after — correct ASI behavior
ReturnStatement <- RETURN (!%begin_line Expression)? EOS

nine ASI-sensitive locations got this treatment. three locations that check for actual line terminators (regex literals, single-line comments, hashbang) kept !LineTerminator because they genuinely care about the next character.

implementing this in the VM meant changing the EmptyOp enum values from sequential integers to powers of 2 so they work as proper bit flags. the old values could produce false positives when combined. the % prefix gets intercepted during compilation and turned into an EmptyOpNode that the VM evaluates as a position check rather than a character match.

benchmarks

I wrote a proper benchmark comparing Pegasus against acorn, babel, oxc, and tree-sitter on three files. the results tell a nuanced story.

on small files like chalk.js (6 KB), Pegasus actually wins end-to-end — 31 ms vs acorn's 46 ms. but that's misleading because Node.js has 35–45 ms of startup overhead. in pure parse time, tree-sitter does chalk in 8.5 ms.

on medium files like three-webgl.js (106 KB), Pegasus takes 64 ms. tree-sitter does it in 13.5 ms. oxc does it in 1 ms. on lodash.js (545 KB), Pegasus takes 2.2 seconds. tree-sitter takes 37 ms. oxc takes 5 ms.

so Pegasus is roughly 5× slower than tree-sitter on medium files and 60× slower on large files. oxc is in another universe entirely — it's a hand-written Rust parser doing 100,000 KB/s. Pegasus does about 240–2,300 KB/s depending on file size.

but here's the thing: Pegasus is a general-purpose PEG parser running a grammar file. it's not hand-optimized for JavaScript. the grammar is 802 lines of PEG that you can read and modify. try reading oxc's parser — it's 15,000 lines of specialized Rust. different tools for different purposes.

what's next

the parser now handles all 76 test files, has a CI pipeline that builds Odin from source at a pinned commit (because monthly releases lag behind the nightly API), runs 188 unit tests, and validates against the full file corpus. the grammar covers ES2025 including decorators, explicit resource management, import attributes, and private fields.

next I want to try test262 — the official JavaScript test suite with ~50,000 test cases. that's going to break things in ways I haven't imagined yet. but 76 production files was enough to find the patterns that matter, and now the parser is fast enough to actually run that many tests without waiting all day.