SCOOP: Source Codes of the Past

Back to the programme

Which Languages Are In-Vocabulary?

Andrew Janco ・ Princeton University

Monday, 7th September ・ 13:30 - 15:00 ・ WG4-1 ・ Language Challenges I ・ Room 3

Transformer-based language models have a fixed vocabulary of tokens used to represent words and word parts. Which languages can and cannot be represented with current model vocabularies? When native support does not exist, what are the possibilities and drawbacks of byte pair encoding? This short talk presents a tool to check how well current model vocabularies support the language(s) you’re working with. It joins data for more than 8,000 languages from the Unicode CLDR and Glottolog, with common vocabularies in tiktoken and HuggingFace AutoTokenizer.