ggml-org/llama.cpp

Include `eos_token` and `bos_token` from `tokenizer_config.json` for chat templating

Open

#5,040 opened on Jan 19, 2024

View on GitHub
 (12 comments) (4 reactions) (0 assignees)C++ (18,202 forks)batch import
enhancementhelp wanted

Repository metrics

Stars
 (110,169 stars)
PR merge metrics
 (Avg merge 6d 8h) (389 merged PRs in 30d)

Description

Feature Description

Include (at minimum) eos_token and bos_token keys the huggingface tokenizer_config.json as gguf metadata keys.

Motivation

For chat models these differ from the normal eos and bos tokens and are required to stop the model generating user message tokens.

For example to correctly stop generation of chatml chat models you need to watch for the <|im_end|> token specified in the tokenizer_config.json below.

https://huggingface.co/mlabonne/NeuralBeagle14-7B/blob/main/tokenizer_config.json

however the gguf metadata doesn't include this token

image

using the chat_format alone without the eos_token causes generation to continue incorrectly past the end of the assistant respone.

image

Possible Implementation

I'm not too familiar with the gguf-py package but I think adding these to the SpecialVocab class and updating the _try_load_from_tokenizer_json would be a start. Not sure what the correct gguf key should be for these so as to not create confusion.

Contributor guide

Research direction
Inspect the gguf py package's SpecialVocab class and try load from tokenizer json method to understand how tokenizer config.json keys are mapped to gguf metadata. Add support for eos token and bos token keys.
Tech stack
python
Domain
backendapi
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
2
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
PythonGit
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
65