objectionary/eo-lsp-server

Semantic token `length = tk.stop - tk.start + 1` miscounts non-BMP characters under LSP's default UTF-16 encoding

Open

#322 opened on May 11, 2026

View on GitHub
 (2 comments) (0 reactions) (0 assignees)TypeScript (9 forks)github user discovery
bughelp wanted

Repository metrics

Stars
 (6 stars)
PR merge metrics
 (PR metrics pending)

Description

What happens

In src/semantics.ts, semantic tokens are emitted with a length computed from raw ANTLR character offsets:

start: tk.column,
length: tk.stop - tk.start + 1,

LSP 3.17 specifies that, unless the server negotiates a different positionEncoding, positions and lengths are measured in UTF-16 code units. This server's InitializeResult (server.ts:55-60) does not advertise a positionEncoding, so the default UTF-16 contract applies.

ANTLR4-js stores Token.start / Token.stop as codepoint indices into the input CharStream. For characters within the Basic Multilingual Plane (every code point < U+10000), one codepoint equals one UTF-16 code unit and the math accidentally works. For characters outside the BMP — emoji, some CJK extensions, mathematical symbols (U+1D400-…), and many other astral-plane code points — one codepoint equals two UTF-16 code units (a surrogate pair). The reported token length is therefore off by one for every astral character inside the token.

VSCode and other LSP clients position semantic highlighting using UTF-16 offsets supplied by the server. A miscount manifests as misaligned syntax highlighting that drifts further the more astral characters appear earlier in the line.

What should happen

Convert the codepoint offsets to UTF-16 code units using the document text, e.g.:

const text = doc.getText();
const utf16Start = text.substring(0, tk.start).length;  // already UTF-16-counted
const utf16Length = text.substring(tk.start, tk.stop + 1).length;

Or negotiate positionEncoding: 'utf-32' in InitializeResult.capabilities and emit codepoint offsets explicitly (only some clients support that). Result: tokens with astral-plane characters are highlighted at their actual positions instead of drifting.

Contributor guide