Commit Graph

6 Commits

Author SHA1 Message Date
Nathan Evans
9aa94dfd86 Streamline config 2025-12-18 17:15:04 -08:00
Nathan Evans
896a48ce1e Move pre-pending into chunkers 2025-12-18 16:38:34 -08:00
Nathan Evans
b7c06730d7 Move Tokenizer base class to common package 2025-12-18 13:56:32 -08:00
Nathan Evans
e5c1aa7d52 Restore create_base_text_units parameterization 2025-12-18 13:46:22 -08:00
Nathan Evans
a20dbdb795 Collapse token splitting functionality into one class/function 2025-12-18 13:22:18 -08:00
Nathan Evans
b63f747d44 Co-locate chunking/splitting 2025-12-18 11:58:53 -08:00