Preview v0.2.0

Tiktoken Ex

Pure-Elixir TikToken-style byte-level BPE tokenizer, with helpers for Kimi K2 encodings.

Stats & Info

Add tiktoken_ex to your list of dependencies in mix.exs:

def deps do
  [
    {:tiktoken_ex, "~> 0.2.0"}
  ]
end

Then run:

mix deps.get