Tokenizer Illustration
Input Text
GPT-2 (gpt2)
Number of Tokens:
Tokens:
Token IDs:
GPT-3.5/GPT-4 (cl100k_base)
Number of Tokens:
Tokens:
Token IDs:
GPT-4o (o200k_base)
Number of Tokens:
Tokens:
Token IDs: