01Overview

Lexical boundaries
for code.

Can a tokenizer that respects the lexical boundaries of programming-languages help a language model generalize syntax, semantics, and structure better?

working hypothesis

Make the units of code look like the units of grammar.

characters→lexemes→meaning

Programming languages have a built-in grammar, identifiers name values, operators follow clear rules, and delimiters define the structure. Standard subword tokenizers learn useful statistical patterns, but their token boundaries are determined by character frequency in the training data rather than the syntax of the code.

02Tokenization

How code gets tokenized

Prompt / code fragment128 / 180 chars

checkpoint

Qwen3.5-SWE (extended)

SamLegion888/Qwen3.5-SWE / extended-vocab

The experiment's extended-vocabulary tokenizer with code-aware boundary cues.

token count

lexical units

selected id

—

Tokenizer assets are loaded through @huggingface/transformers.

03What counts as a token?

Subword tokenizers can split code in awkward places.

A lexical token is a smallest meaningful unit emitted by a language lexer. Such as an identifier, keyword, literal, operator, or delimiter. The parser then combines these units into an abstract syntax tree. A tokenizer designed around those same atoms gives the model a representation whose transitions already carry hints about the next legal move.

Subword algorithms such as BPE begin with a compact base vocabulary and learn frequent adjacent merges. That is efficient and expressive; the learned merge table can also cut across an identifier, attach whitespace to a token, or bundle a delimiter with a nearby symbol. The result is a useful statistical vocabulary with a less explicit grammar.

Python

def transform(value: int, exponent: int) -> int:
    result = value
    result **= exponent
    result >>= 2
    return result

lexical tokens

defkeywordtransformidentifier(delimitervalueidentifier:delimiterinttype,delimiterexponentidentifier:delimiterinttype)delimiter->operatorinttype:delimiterresultidentifier=operatorvalueidentifierresultidentifier**=operatorexponentidentifierresultidentifier>>=operator2literalreturnkeywordresultidentifier

selected tokenizer · Qwen3.5-SWE

Press Tokenize to render the selected tokenizer pieces.

Rust

fn count_bits(mut value: u32) -> u32 {
    let mut count = 0;

    while value != 0 {
        count += value & 1;
        value >>= 1;
    }

    count
}

lexical tokens

fnkeywordcount_bitsidentifier(delimitermutkeywordvalueidentifier:delimiteru32type)delimiter->operatoru32type{delimiterletkeywordmutkeywordcountidentifier=operator0literal;delimiterwhilekeywordvalueidentifier!=operator0literal{delimitercountidentifier+=operatorvalueidentifier&operator1literal;delimitervalueidentifier>>=operator1literal;delimiter}delimitercountidentifier}delimiter

selected tokenizer · Qwen3.5-SWE

Press Tokenize to render the selected tokenizer pieces.

TypeScript

function normalize(value: number, fallback?: number): number {
  let result = value;
  result >>>= 1;
  fallback ??= 10;
  return result || fallback;
}

lexical tokens

functionkeywordnormalizeidentifier(delimitervalueidentifier:delimiternumbertype,delimiterfallbackidentifier?operator:delimiternumbertype)delimiter:delimiternumbertype{delimiterletkeywordresultidentifier=operatorvalueidentifier;delimiterresultidentifier>>>=operator1literal;delimiterfallbackidentifier??=operator10literal;delimiterreturnkeywordresultidentifier||operatorfallbackidentifier;delimiter}delimiter

selected tokenizer · Qwen3.5-SWE

Press Tokenize to render the selected tokenizer pieces.

04Causal attention & lexical boundaries

How attention builds code meaning

Causal self-attention builds the representation at each position from that token and the available prefix. Each position is projected into a query, key, and value; query–key dot products produce scores, the causal mask assigns -Infinity score to future positions before softmax, and the resulting weights mix the value vectors into the position’s output. This lets an autoregressive model predict the next token from the sequence it has read so far. ^{[04, 05]}

Token boundaries determine the units that participate in these information paths. Preserving lexical boundaries gives an keyword, operator, literal, or delimiter its own position in the sequence, so a query token can attend directly to the unit that carries the relevant grammatical or semantic role. A model can then connect an identifier to its declaration and uses, an operator to its operands, a delimiter to the region it opens or closes, and a type annotation to the value or expression it constrains. These direct token-to-token paths give the model a representation that follows the structure of the programming language. Lexical roles are visible at the token level, syntax is expressed through relationships among those roles, and semantics is built from their contextual composition.

Consider the operator <<=. With a subword split such as <, <= or <<, =, the compound operator occupies several key positions. A later query that uses the operator can distribute its attention across those fragments, and their value vectors must be combined to reconstruct that the sequence forms one grammatical unit with the left-shift-assignment meaning. With <<= preserved as one lexical token, the same query has a single explicit key position for the complete operator while attending separately to its left-hand operand, right-hand operand, and surrounding statement context. Subword splits therefore scatter attention across fragments; clean lexical boundaries create clear attention paths for the operator’s syntax and semantics. This is the representation hypothesis tested by the comparison below.

attention layer

·result / 9

token / idclick to focus

attention weights

softmax(QKᵀ / √d)

·result

·=

·value

···

·result

**=

·exponent

···

·result

>>=

·result

·=

·value

···

·result

**=

·exponent

···

·result

>>=

low activationhigh activation

05Experiment

Training, Evaluation, and Results

Qwen3.5-SWE compares two supervised fine-tunes of Qwen3.5-4B-Base. Both variants use NVIDIA’s Nemotron-SFT-SWE-v3.5, a SWE instruction-tuning dataset containing agentic trajectories with code, text, structured data, patches, and configuration artifacts. ^[06]
The Standard variant keeps the base tokenizer. The Extended-vocab variant adds 278 new lexical code tokens across

various programming languages which includes the keywords, operators, delimiters, but does not consider the identifier such as variable names, and resizes the model vocabulary, exposes the added embedding rows as trainable token indices while applying the same LoRA supervised-fine-tuning recipe. The recipe uses the dataset’s train split and creates a seeded 90/10 train-test split. ^[05]

model cardQwen3.5-SWE variants datasetNemotron-SFT-SWE-v3.5 fine-tuning notebookOpen in Molab evaluation notebookOpen in Molab

Benchmark Setup and Evaluation Criteria

I used EvalPlus to evaluate the code-generation capability on HumanEval and MBPP. HumanEval focuses on function-level programming tasks, while MBPP uses natural-language problem statements paired with Python solutions. Their base suites measure agreement with the original task tests; HumanEval+ and MBPP+ add substantially larger extra-test suites that probe behavior across more inputs; EvalPlus describes these extensions as roughly 80× the original HumanEval tests and 35× the original MBPP tests. ^[07]For each tokenizer variant, the evaluation notebook uses the corresponding model directory, generates one greedy completion per task with

results · pass@1 (%)

Higher is better (more tests passed)

StandardExtended-vocab

HumanEvalBase tests

standard

45.7%

extended

52.4%

HumanEval+Base + extra tests

standard

39.0%

extended

47.0%

MBPPBase tests

standard

65.3%

extended

64.6%

MBPP+Base + extra tests

standard

54.5%

extended

53.7%

vLLM, and runs EvalPlus against the base and extended tests using the forced base prompt. The reported metric is pass@1: the percentage of tasks whose first generated solution passes the selected executable tests. ^[07]The Extended-vocab variant leads on HumanEval by 6.7 percentage points and on HumanEval+ by 8.0 points. Its HumanEval score falls by 5.4 points when the extra tests are added, compared with a 6.7-point fall for standard, so the extended variant retains more of its base-test

performance on this evaluation.Standard leads MBPP by 0.7 points and MBPP+ by 0.8 points. Both variants experience a similar MBPP-to-MBPP+ reduction of roughly eleven points, showing the additional pressure created by broader test coverage. Across this matched run, the results support a benchmark-specific effect: lexical vocabulary expansion helps the HumanEval family, while the original performance remains slightly stronger on MBPP. ^[07]

06Trade-offs

Benefits and costs of lexical token boundaries.

Token merging and sequence length. BPE learns frequent adjacent merges, so a standard tokenizer can represent spaces, brackets, and punctuation together with nearby code in units such as return or ):. These merges reduce the number of sequence positions. Strict lexical boundaries separate the delimiter, operator, and keyword units, which can increase token count for the same source text and increase the number of key/value states held during autoregressive generation. ^[01]

Where it helps, where the effect is small, and where it costs. Explicit lexical boundaries are most relevant when inputs contain recurring syntax, stable semantic roles, and structural relationships, such as programming languages, mathematical notation, markup, configuration files, and symbolic expressions. They can be less effective for corpora with many rare or open-ended identifiers, highly variable conventions, mixed domains, or tasks that benefit from reusable subword pieces across many types of text. In those settings, a larger specialized vocabulary can require more learned token embeddings while providing less reusable coverage for unseen forms.

Practical limitation. The extended tokenizer is most useful when the corpus and model are designed around the same lexical units. It may not be worth the additional tokenizer, vocabulary, and deployment requirements for short or mostly unstructured inputs, or when broad compatibility and compact model artifacts are the primary goals. These conditions define the situations that future comparisons should measure.

07References

Reading list

01Hugging Face TokenizersTokenization algorithms.
02Qwen Team (2026)Qwen3.5: Towards Native Multimodal Agents.
03SamLegion888 (2026)Qwen3.5-SWE — extended vocabulary checkpoint.
04Vaswani et al. (2017)Attention Is All You Need.
05Hugging Face TransformersCausal language modeling.
06NVIDIA (2026)Nemotron-SFT-SWE-v3.5 dataset.
07EvalPlus teamEvalPlus — rigorous evaluation of LLM-generated code with HumanEval(+) and MBPP(+).

citation

@misc{qwen35_swe,
  title        = {Qwen3.5-SWE},
  author       = {SamLegion888},
  year         = {2026},
  url          = {https://huggingface.co/SamLegion888/Qwen3.5-SWE}
}

Lexical boundaries
for code.

Can a tokenizer that respects the lexical boundaries of programming-languages help a language model generalize syntax, semantics, and structure better?

working hypothesis

Make the units of code look like the units of grammar.

characters→lexemes→meaning

Subword tokenizers can split code in awkward places.

Python

def transform(value: int, exponent: int) -> int:
    result = value
    result **= exponent
    result >>= 2
    return result

lexical tokens

selected tokenizer · Qwen3.5-SWE

Press Tokenize to render the selected tokenizer pieces.

Rust

fn count_bits(mut value: u32) -> u32 {
    let mut count = 0;

    while value != 0 {
        count += value & 1;
        value >>= 1;
    }

    count
}

lexical tokens

selected tokenizer · Qwen3.5-SWE

Press Tokenize to render the selected tokenizer pieces.

TypeScript

function normalize(value: number, fallback?: number): number {
  let result = value;
  result >>>= 1;
  fallback ??= 10;
  return result || fallback;
}

lexical tokens

selected tokenizer · Qwen3.5-SWE

Press Tokenize to render the selected tokenizer pieces.

Training, Evaluation, and Results

Benchmark Setup and Evaluation Criteria

results · pass@1 (%)

Higher is better (more tests passed)

StandardExtended-vocab

HumanEvalBase tests

standard

45.7%

extended

52.4%

HumanEval+Base + extra tests

standard

39.0%

extended

47.0%

MBPPBase tests

standard

65.3%

extended

64.6%

MBPP+Base + extra tests

standard

54.5%

extended

53.7%

Benefits and costs of lexical token boundaries.

Lexical boundariesfor code.

How code gets tokenized

Subword tokenizers can split code in awkward places.

How attention builds code meaning

Training, Evaluation, and Results

Benchmark Setup and Evaluation Criteria

Benefits and costs of lexical token boundaries.

Reading list

Lexical boundariesfor code.

How code gets tokenized

Subword tokenizers can split code in awkward places.

How attention builds code meaning

Training, Evaluation, and Results

Benchmark Setup and Evaluation Criteria

Benefits and costs of lexical token boundaries.

Reading list

Lexical boundaries
for code.

Lexical boundaries
for code.