Skip to content

Commit

Permalink
Fixed wrong regex in the token vocab parser, which lead to invalid to…
Browse files Browse the repository at this point in the history
…ken names on import

Fixes #22
  • Loading branch information
mike-lischke committed Dec 28, 2024
1 parent 189863a commit 7e910cf
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion src/parse/TokenVocabParser.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ import { Constants } from "../Constants.js";
import { ErrorType } from "../tool/ErrorType.js";
import type { IGrammar } from "../types.js";

const linePattern = /(?<tokenID>[^\n]+?)[ \\t]*?=[ \\t]*?(?<tokenTypeS>[0-9]+)/;
const linePattern = /(?<tokenID>[^\n]+)[ \\t]*=[ \\t]*?(?<tokenTypeS>[0-9]+)/;

export class TokenVocabParser {
protected readonly g: IGrammar;
Expand Down

0 comments on commit 7e910cf

Please sign in to comment.