This commit is contained in:
Nathan Evans 2025-12-22 12:48:25 -08:00
parent 88af7f8dc2
commit ee20153d8c

View File

@ -21,12 +21,12 @@ print(chunks) # ["This is a", " random test fragment", " of some text"]
Using the factory via helper util
```python
tokenizer = tiktoken.get_encoding("o200k_base")
config = ChunkingConfig(
strategy="tokens",
size=3,
overlap=0
tokenizer = tiktoken.get_encoding("o200k_base")
config = ChunkingConfig(
strategy="tokens",
size=3,
overlap=0
)
chunker = create_chunker(config, tokenizer.encode, tokenizer.decode)
...
`
chunker = create_chunker(config, tokenizer.encode, tokenizer.decode)
...
```