FossilCoder

NSJSONSerialization silently deletes U+FEFF from JSON strings

NSJSONSerialization strips U+FEFF (the BOM / zero-width no-break space) from inside JSON strings, so distinct keys merge and tokenizer.json vocabularies load wrong. Swift's JSONDecoder doesn't. Repro, scope, and three fixes.

· 6 min read · foundation, json, objective-c, swift, unicode, llm

The problem

ObjCTokenizer is my Objective-C port of Hugging Face’s tokenizers, built for a local LLM inference engine on macOS. With Gemma 4’s tokenizer.json, one test kept failing: encoding a line of Python ending in # comment produced the wrong token ID for #. It came back as 208867 instead of 236865. Not a crash, not an error: just a wrong number.

The cause: Gemma 4’s vocabulary has 8 keys that contain U+FEFF, for example "\uFEFF#" (ID 208867). NSJSONSerialization deletes U+FEFF from anywhere inside a JSON string, not just a byte-order mark at the start of the document. So "\uFEFF#" turns into "#", collides with the real "#" key, and its ID overwrites the real one. The same happened to keys like "\uFEFF//" and "\u2581\uFEFF", and the 262,144-entry vocabulary loaded three entries short.

Minimal repro:

// Object keys collapse:
NSData *d1 = [@"{\"\\uFEFF#\":1,\"#\":2}" dataUsingEncoding:NSUTF8StringEncoding];
NSDictionary *o1 = [NSJSONSerialization JSONObjectWithData:d1 options:0 error:nil];
// Expected: 2 keys ("\uFEFF#" and "#").  Actual: 1 key ("#").

// String content disappears:
NSData *d2 = [@"[\"\\uFEFF\"]" dataUsingEncoding:NSUTF8StringEncoding];
NSArray *o2 = [NSJSONSerialization JSONObjectWithData:d2 options:0 error:nil];
// Expected: a one-character string.  Actual: "" (empty).

It makes no difference whether U+FEFF arrives as raw EF BB BF bytes or as the \uFEFF escape: the deletion happens after unescaping. And it’s the same from Swift, since JSONSerialization is the same parser.

Why it’s hard to spot

U+FEFF is zero-width. It doesn’t show up in an editor, a diff, or a debugger’s string preview, so the corrupted dictionary looks exactly like the correct one, just with a few fewer entries. Nothing fails until a token ID is wrong.

What didn’t work

Blaming the tokenizer. The bug report pointed at the byte-level BPE code: a stray BOM in the byte-to-token map seemed the obvious suspect. It was innocent. Splitting the input into pieces returned the correct "#" (U+0023, one character); only the step from piece to ID was wrong. Printing both ends side by side, the pieces with their code points and the IDs they mapped to, moved the fault from the tokenizer to the vocabulary dictionary. Parsing tokenizer.json directly then settled it: vocab[@"#"] was 208867, vocab[@"#"] was nil, and the count was 262,141.

Fixing only the raw bytes. My first fix swapped out raw U+FEFF characters before parsing. Two days later it turned out that the  escape form is stripped too, because the deletion happens after unescaping. The workaround below handles both.

Scope: one character, exhaustively

To find out whether this was general Unicode mangling, I round-tripped all 1,112,064 valid Unicode scalars (U+0000–U+10FFFF minus surrogates) through NSJSONSerialization, in both the escaped and the raw UTF-8 form.

  • U+FEFF is the only scalar altered. Everything else round-trips byte for byte, including the other zero-width and invisible characters: U+200B, U+2060, U+00A0.
  • No normalization happens. Decomposed sequences stay decomposed; compatibility characters are preserved.

So it’s a byte-order-mark heuristic applied far too broadly. RFC 8259 §7 defines a JSON string as a sequence of Unicode code points, and U+FEFF is ordinary content there. Tolerating a BOM at the start of the document is fine; deleting U+FEFF from string content is a bug.

Still present on macOS 27.3 (Swift 6.4.2).

Fix 1: use Swift’s JSONDecoder

This one surprised me. JSONDecoder uses Swift Foundation’s own JSON parser, not NSJSONSerialization, and it gets this right: both the escaped and the raw-byte form survive, even with a leading document BOM.

import Foundation

struct TokenizerFile: Decodable {
    struct Model: Decodable {
        let vocab: [String: Int]
    }
    let model: Model
}

let vocab = try JSONDecoder().decode(TokenizerFile.self, from: data).model.vocab
// "\u{FEFF}#" and "#" stay two distinct keys.

If you’re in Swift and the structure is known, this is the simplest fix. It doesn’t help if you need an untyped Any tree, or if you’re in Objective-C.

Fix 2: stay on NSJSONSerialization with a placeholder swap

Replace every U+FEFF with a private-use character before parsing, then put it back in the parsed tree. You have to catch both forms: the raw character, and the \uFEFF escape.

// Put the real U+FEFF back everywhere in the parsed tree.
static id RestoreSentinel(id o, NSString *s, NSString *bom) {
    if ([o isKindOfClass:NSString.class])
        return [o rangeOfString:s].location == NSNotFound ? o
             : [o stringByReplacingOccurrencesOfString:s withString:bom];
    if ([o isKindOfClass:NSArray.class]) {
        NSMutableArray *a = [NSMutableArray arrayWithCapacity:[o count]];
        for (id e in o) [a addObject:RestoreSentinel(e, s, bom)];
        return a;
    }
    if ([o isKindOfClass:NSDictionary.class]) {
        NSMutableDictionary *d = [NSMutableDictionary dictionary];
        [o enumerateKeysAndObjectsUsingBlock:^(id k, id v, BOOL *stop) {
            d[RestoreSentinel(k, s, bom)] = RestoreSentinel(v, s, bom);
        }];
        return d;
    }
    return o;
}

// A private-use scalar that must not already occur in the file.
NSString *sentinel = @"\U000F0000";
NSAssert([json rangeOfString:sentinel].location == NSNotFound, @"sentinel occurs in input");

// 1. The raw character.
NSString *t = [json stringByReplacingOccurrencesOfString:@"\uFEFF" withString:sentinel];

// 2. The \uFEFF escape. Count the backslashes before it, so an escaped
//    backslash followed by a literal "uFEFF" is left alone.
NSRegularExpression *re = [NSRegularExpression
    regularExpressionWithPattern:@"(?<!\\\\)((?:\\\\\\\\)*)\\\\u[Ff][Ee][Ff][Ff]"
                         options:0 error:nil];
t = [re stringByReplacingMatchesInString:t options:0 range:NSMakeRange(0, t.length)
                            withTemplate:[@"$1" stringByAppendingString:sentinel]];

// 3. Parse, then restore.
id parsed = [NSJSONSerialization JSONObjectWithData:[t dataUsingEncoding:NSUTF8StringEncoding]
                                            options:0 error:nil];
id fixed = RestoreSentinel(parsed, sentinel, @"\uFEFF");

The regex, without Objective-C’s string escaping, is (?<!\\)((?:\\\\)*)\\u[Ff][Ee][Ff][Ff].

Tested against a raw U+FEFF key, an escaped one, U+FEFF in the middle of a key, and a literal \\uFEFF that must stay as text: all four come back correctly.

Two refinements from the production version in ObjCTokenizer:

  • Strip a genuine byte-order mark at the very start of the document first, and only do the swap if U+FEFF still occurs, so ordinary files skip it.
  • Pick the placeholder by scanning the input for a free one rather than hard-coding it. ObjCTokenizer searched the BMP private-use range, U+E000–U+F8FF. And have RestoreSentinel return containers unchanged when nothing inside them changed, so only the few affected branches of a 262,144-entry vocabulary get copied.

With the full Gemma 4 vocabulary, this version matched Hugging Face’s own tokenizer byte for byte across an 83-input multilingual test set.

Fix 3: don’t use Foundation for this file

A spec-compliant C parser like yyjson preserves U+FEFF. It only strips a leading document BOM, and only if you ask it to. It’s also faster on large files like tokenizer vocabularies. A week after the placeholder fix, ObjCTokenizer switched to it: MIT-licensed, one header and one source file, and easy to test.

Hugging Face’s swift-transformers reads tokenizer.json through yyjson too, and I checked: its current version keeps all U+FEFF keys with the right IDs. Older versions that parsed it with JSONSerialization would have had the same corruption, so if you’re pinned to an old release, check.

The pattern

This wasn’t the first time. Building ObjCTokenizer, the same kind of bug kept turning up: a Foundation convenience that’s helpful for text people read, and destructive for text that has to match byte for byte. componentsSeparatedByString:@"\n" misses CRLF and Unicode line separators, which silently dropped 3 of 30,522 lines from BERT’s vocab.txt and shifted every later token ID. And Cocoa’s grapheme-cluster splitting merges characters that a tokenizer has to treat separately. The rule I work by now: on tokenizer data, treat every NSString and NSJSONSerialization convenience as a suspect, and print both ends of a transformation before blaming the component that looks guilty.

Status

Reported as FB23271905, with a thread on the Apple Developer Forums. Quinn “The Eskimo!” from Apple DTS replied that a change is unlikely, because altering long-standing parser behavior carries compatibility risk, and pointed to the New Codable prototype on the Swift Forums as the place where JSON handling is moving.

If this has bitten you too, file a duplicate referencing FB23271905. Duplicates are what get bugs triaged.