446 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
gaudyb	a4ffc3d34c	Remove embeddings optional new (#2128) ... * remove optional embeddings * fix test * fix tests * fix pipeline * fix test * fix test * fix test * fix tests --------- Co-authored-by: Gaudy Blanco <gaudy-microsoft@MacBook-Pro-m4-Gaudy-For-Work.local>	2025-11-17 13:10:54 -06:00
Nathan Evans	4512ce027f	Empty graph guards (#2126) ... * Remove networkx from graph_extractor and clean out redundancy * Bubble pipeline error to console	2025-11-11 10:11:08 -08:00
Derek Worthen	e0cce31f54	Graphrag config (#2119) ... * Add load_config to graphrag-common package.	2025-11-10 07:57:03 -08:00
Nathan Evans	ae1f5e1811	Nov 2025 housekeeping (#2120) ... * Remove gensim sideload * Split CI build/type checks from unit tests * Thorough review of docs to align with v3 * Format * Fix version * Fix type	2025-11-06 10:03:22 -08:00
Nathan Evans	6033e4ffa2	Storage fixes and cleanup (#2118) ... * Fix pipeline recursion * Remove base_dir from storage.find * Remove max_count from storage.find * Remove prefix on storage integ test * Add base_dir in creation_date test * Wrap base_dir in Path * Use constants for input/update directories	2025-11-05 13:06:09 -08:00
Nathan Evans	6b03af6277	Fix formatting	2025-11-04 10:48:09 -08:00
Derek Worthen	619269243d	Restructure project as monorepo. (#2111) ... * Restructure project as monorepo.	2025-11-04 09:51:56 -08:00
Gaudy Blanco	c43a58c353	fix for container name	2025-10-23 10:40:21 -06:00
gaudyb	542d3db2a9	Prefix vector store (#2106) ... * add prefix to vector store configuration and removal of container name * docs updated * change prefix property name * change prefix property name * feedback implemented --------- Co-authored-by: Gaudy Blanco <gaudy-microsoft@MacBook-Pro-m4-Gaudy-For-Work.local>	2025-10-21 15:30:50 -06:00
Nathan Evans	1bb9fa8e13	Unified factory (#2105) ... * Simplify Factory interface * Migrate CacheFactory to standard base class * Migrate LoggerFactory to standard base class * Migrate StorageFactory to standard base class * Migrate VectorStoreFactory to standard base class * Update vector store example notebook * Delete notebook outputs * Move default providers into factories * Move retry/limit tests into integ * Split language model factories * Set smoke test tpm/rpm * Fix factory integ tests * Add method to smoke test, switch text to 'fast' * Fix text smoke config for fast workflow * Add new workflows to text smoke test * Convert input readers to a proper factory * Remove covariates from fast smoke test * Update docs for input factory * Bump smoke runtime * Even longer runtime * min-csv timeout * Remove unnecessary lambdas	2025-10-20 12:05:27 -07:00
gaudyb	0436405962	Remove document overwrite (#2101) ... * remove document overwrite from vector store configuration * remove document overwrite and refactor load documents method * fix test * fix test * fix test --------- Co-authored-by: Gaudy Blanco <gaudy-microsoft@MacBook-Pro-m4-Gaudy-For-Work.local>	2025-10-16 07:56:54 -06:00
Nathan Evans	5ec49fd39c	V3 docs and cleanup (#2100) ... * Remove community contrib notebooks * Add migration notebook and breaking changes page edits * Update/polish docs * Make model instance name configurable * Add vector schema updates to v3 migration notebook * Spellcheck * Bump smoke test runtimes	2025-10-15 13:47:19 -07:00
Nathan Evans	b732445535	Remove multi search (#2093) ... * Remove multi-search from CLI * Remove multi-search from API * Flatten vector_store config * Push hydrated vector store down to embed_text * Remove outputs from config * Remove multi-search notebook/docs * Add missing response_type in basic search API * Fix basic search context and id mapping * Fix v1 migration notebook * Fix query entity search tests	2025-10-10 17:20:53 -07:00
Nathan Evans	715be61404	Sort deps alpha	2025-10-10 17:10:42 -07:00
Nathan Evans	f7a8a0888d	Merge branch 'main' into v3/main	2025-10-10 17:03:52 -07:00
Nathan Evans	6284cdd110	Remove fnllm (#2095)	2025-10-10 16:59:25 -07:00
Nathan Evans	eb0dfe376b	Remove strategy dicts (#2090) ... * Remove "strategy" from community reports config/workflow * Remove extraction strategy from extract_graph * Remove summarization strategy from extract_graph * Remove strategy from claim extraction * Strongly type prompt templates * Remove strategy from embed_text * Push hydrated params into community report workflows * Push hyrdated params into extract covariates * Push hydrated params into extract graph NLP * Push hydrated params into extract graph * Push hydrated params into text embeddings * Remove a few more low-level defaults * Semver * Remove configurable prompt delimiters * Update smoke tests	2025-10-10 12:15:23 -07:00
gaudyb	79ad9b96f3	reduce schema fields (#2089) ... * reduce schema fields * fix launch.json --------- Co-authored-by: Gaudy Blanco <gaudy-microsoft@MacBook-Pro-m4-Gaudy-For-Work.local>	2025-10-09 13:41:31 -06:00
Alonso Guevara	fdb7e3835b	Release v2.7.0 (#2087)	2025-10-08 21:33:34 -07:00
Nathan Evans	2b5284ca1b	Merge branch 'main' into v3/main	2025-10-07 16:24:15 -07:00
Nathan Evans	ac8a7f5eef	Housekeeping (#2086) ... * Add deprecation warnings for fnllm and multi-search * Fix dangling token_encoder refs * Fix local_search notebook * Fix global search dynamic notebook * Fix global search notebook * Fix drift notebook * Switch example notebooks to use LiteLLM config * Properly annotate dev deps as a group * Semver * Remove --extra dev * Remove llm_model variable * Ignore ruff ASYNC240 * Add note about expected broken notebook in docs * Fix custom vector store notebook * Push tokenizer throughout	2025-10-07 16:21:24 -07:00
Nathan Evans	4364d678dd	Merge branch 'main' into v3/main	2025-10-06 12:41:12 -07:00
Nathan Evans	6c86b0a7bb	Init config cleanup (#2084) ... * Spruce up init_config output, including LiteLLM default * Remove deployment_name requirement for Azure * Semver * Add model_provider * Add default model_provider * Remove OBE test * Update minimal config for tests * Add model_provider to verb tests	2025-10-06 12:06:41 -07:00
Nathan Evans	2bd3922d8d	Litellm auth fix (#2083) ... * Fix scope for Azure auth with LiteLLM * Change internal language on max_attempts to max_retries * Rework model config connectivity validation * Semver * Swtich smoke tests to LiteLLM * Take out temporary retry_strategy = none since it is not fnllm compatible * Bump smoke test timeout * Bump smoke timeout further * Tune smoke params * Update smoke test bounds * Remove covariates from min-csv smoke * Smoke: adjust communities, remove drift * Remove secrets where they aren't necessary * Clean out old env var references	2025-10-06 10:54:21 -07:00
Andres Morales	ebe959a9db	Update v3/main missing config + functions (#2082)	2025-09-30 15:48:40 -06:00
gaudyb	d7773bd15c	Clean vector store (#2077) ... * clean vector store code * fix * fix launch.json --------- Co-authored-by: Gaudy Blanco <gaudy-microsoft@MacBook-Pro-m4-Gaudy-For-Work.local>	2025-09-25 21:17:10 -06:00
Nathan Evans	d7516825e2	Update comment	2025-09-24 14:51:07 -07:00
Nathan Evans	de767cc881	Use 4.1 and text-embedding-3-large as defaults	2025-09-24 14:49:55 -07:00
Nathan Evans	7f996cf584	Docs/2.6.0 (#2070) ... * Add basic search to overview * Add info on input documents DataFrame * Add info on factories to docs * Add consumption warning and switch to "christmas" for folder name * Add logger to factories list * Add litellm docs. (#2058) * Fix version for input docs * Spelling --------- Co-authored-by: Derek Worthen <worthend.derek@gmail.com>	2025-09-23 14:48:28 -07:00
Nathan Evans	b73053010e	Merge branch 'main' into v3/main	2025-09-23 11:07:42 -07:00
Alonso Guevara	9bc899fe95	Release v2.6.0 (#2068)	2025-09-22 16:16:54 -06:00
Derek Worthen	2b70e4a4f3	Tokenizer (#2051) ... * Add LiteLLM chat and embedding model providers. * Fix code review findings. * Add litellm. * Fix formatting. * Update dictionary. * Update litellm. * Fix embedding. * Remove manual use of tiktoken and replace with Tokenizer interface. Adds support for encoding and decoding the models supported by litellm. * Update litellm. * Configure litellm to drop unsupported params. * Cleanup semversioner release notes. * Add num_tokens util to Tokenizer interface. * Update litellm service factories. * Cleanup litellm chat/embedding model argument assignment. * Update chat and embedding type field for litellm use and future migration away from fnllm. * Flatten litellm service organization. * Update litellm. * Update litellm factory validation. * Flatten litellm rate limit service organization. * Update rate limiter - disable with None/null instead of 0. * Fix usage of get_tokenizer. * Update litellm service registrations. * Add jitter to exponential retry. * Update validation. * Update validation. * Add litellm request logging layer. * Update cache key. * Update defaults. --------- Co-authored-by: Alonso Guevara <alonsog@microsoft.com>	2025-09-22 13:55:14 -06:00
gaudyb	82cd3b7df2	Custom vector store schema implementation (#2062) ... * progress on vector customization * fix for lancedb vectors * cosmosdb implementation * uv run poe format * clean test for vector store * semversioner update * test_factory.py integration test fixes * fixes for cosmosdb test * integration test fix for lancedb * uv fix for format * test fixes * fixes for tests * fix cosmosdb bug * print statement * test * test * fix cosmosdb bug * test validation * validation cosmosdb * validate cosmosdb * fix cosmosdb * fix small feedback from PR --------- Co-authored-by: Gaudy Blanco <gaudy-microsoft@MacBook-Pro-m4-Gaudy-For-Work.local>	2025-09-19 10:11:34 -07:00
Nathan Evans	075cadd59a	Clarify managed auth setup in Azure documentation (#2064) ... Updated instructions for using managed auth on Azure.	2025-09-18 14:58:09 -07:00
Nathan Evans	6d7a50b7f0	Remove community reports rate limiter (#2056) ... * Remove hard-coded community reports rate limiter * Semver * Format * Add memory cache factory	2025-09-18 13:40:24 -07:00
Nathan Evans	2bf7e7c018	Fix multi-index search (#2063)	2025-09-18 12:49:56 -07:00
Nathan Evans	84e600871e	Merge branch 'main' into v3/main	2025-09-17 16:05:28 -07:00
Nathan Evans	6c66b7c30f	Configure async for NLP extraction (#2059) ... * Make async mode configurable for NLP extraction * Semver	2025-09-16 11:52:18 -07:00
Nathan Evans	8720accf0b	Merge branch 'main' into v3/main	2025-09-15 12:09:08 -07:00
Nathan Evans	04d9f585c8	Re-implement hierarchical Leiden (#2049) ... * Use graspologic-native hierarchical leiden * Re-implement largest_connected_component * Copy in modularity * Use graspologic-native directly in pyproject * Remove directed graph tests (we don't use this) * Semver * Remove graspologic dep	2025-09-09 16:26:07 -07:00
Nathan Evans	97704ab933	Remove text unit grouping (#2052) ... * Remove text unit group_by_columns * Semver * Fix default token split test * Fix models in config test samples * Fix token length in context sort test * Fix document sort	2025-09-09 16:04:04 -07:00
Chenghua Duan	a398cc38bb	Update command to use no-discover-entity-types (#2038) ... "no-entity-types" is an incorrect configuration parameter. Co-authored-by: Alonso Guevara <alonsog@microsoft.com>	2025-09-09 16:46:06 -06:00
Nathan Evans	978e79875e	Remove file filtering (#2050) ... * Remove document filtering * Semver * Fix integ tests * Fix file find tuple * Fix another dangling find tuple	2025-09-09 15:36:25 -07:00
Nathan Evans	429e1b1f9e	Remove graph embedding and UMAP (#2048) ... * Remove umap/layout operation * Remove graph embedding * Bump unified-search to GR 2.5.0 * Remove graph vis from unified-search	2025-09-09 15:35:43 -07:00
Derek Worthen	ac95c917d3	Update fnllm. (#2043)	2025-09-05 08:52:05 -07:00
Nathan Evans	1cb20b66f5	Input docs API parameter (#2034) ... * Add optional input_documents to index API * Semver * Add input dataframe example notebook * Format * Fix docs and notebook	2025-09-02 16:15:50 -07:00
Copilot	2030f94eb4	Refactor CacheFactory, StorageFactory, and VectorStoreFactory to use consistent registration patterns and add custom vector store documentation (#2006) ... * Initial plan * Refactor VectorStoreFactory to use registration functionality like StorageFactory Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com> * Fix linting issues in VectorStoreFactory refactoring Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com> * Remove backward compatibility support from VectorStoreFactory and StorageFactory Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com> * Run ruff check --fix and ruff format, add semversioner file Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com> * ruff formatting fixes * Fix pytest errors in storage factory tests by updating PipelineStorage interface implementation Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com> * ruff formatting fixes * update storage factory design * Refactor CacheFactory to use registration functionality like StorageFactory Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com> * revert copilot changes * fix copilot changes * update comments * Fix failing pytest compatibility for factory tests Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com> * update class instantiation issue * ruff fixes * fix pytest * add default value * ruff formatting changes * ruff fixes * revert minor changes * cleanup cache factory * Update CacheFactory tests to match consistent factory pattern Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com> * update pytest thresholds * adjust threshold levels * Add custom vector store implementation notebook Create comprehensive notebook demonstrating how to implement and register custom vector stores with GraphRAG as a plug-and-play framework. Includes: - Complete implementation of SimpleInMemoryVectorStore - Registration with VectorStoreFactory - Testing and validation examples - Configuration examples for GraphRAG settings - Advanced features and best practices - Production considerations checklist The notebook provides a complete walkthrough for developers to understand and implement their own vector store backends. Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com> * remove sample notebook for now * update tests * fix cache pytests * add pandas-stub to dev dependencies * disable warning check for well known key * skip tests when running on ubuntu * add documentation for custom vector store implementations * ignore ruff findings in notebooks * fix merge breakages * speedup CLI import statements * remove unnecessary import statements in init file * Add str type option on storage/cache type * Fix store name * Add LoggerFactory * Fix up logging setup across CLI/API * Add LoggerFactory test * Fix err message * Semver * Remove enums from factory methods --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com> Co-authored-by: Josh Bradley <joshbradley@microsoft.com> Co-authored-by: Nathan Evans <github@talkswithnumbers.com>	2025-08-28 13:53:07 -07:00
Nathan Evans	69ad36e735	Fix id baseline (#2036) ... * Fix all human_readable_id columns to start at 0 * Semver	2025-08-27 11:15:21 -07:00
Nathan Evans	30bdb35cc8	Selective embeddings loading (#2035) ... * Invert embedding table loading logic * Semver	2025-08-27 11:12:01 -07:00
Nathan Evans	77fb7d9d7d	Logging improvements (#2030) ... * Turn down blob/cosmos exception reporting to match file storage * Restore indexing-engine.log * Restore some basic console logging and progress for index CLI * Semver * Ignore small ruff complaints * Fix CLI console printing	2025-08-25 14:56:43 -07:00