* Fix scope for Azure auth with LiteLLM
* Change internal language on max_attempts to max_retries
* Rework model config connectivity validation
* Semver
* Swtich smoke tests to LiteLLM
* Take out temporary retry_strategy = none since it is not fnllm compatible
* Bump smoke test timeout
* Bump smoke timeout further
* Tune smoke params
* Update smoke test bounds
* Remove covariates from min-csv smoke
* Smoke: adjust communities, remove drift
* Remove secrets where they aren't necessary
* Clean out old env var references
* progress on vector customization
* fix for lancedb vectors
* cosmosdb implementation
* uv run poe format
* clean test for vector store
* semversioner update
* test_factory.py integration test fixes
* fixes for cosmosdb test
* integration test fix for lancedb
* uv fix for format
* test fixes
* fixes for tests
* fix cosmosdb bug
* print statement
* test
* test
* fix cosmosdb bug
* test validation
* validation cosmosdb
* validate cosmosdb
* fix cosmosdb
* fix small feedback from PR
---------
Co-authored-by: Gaudy Blanco <gaudy-microsoft@MacBook-Pro-m4-Gaudy-For-Work.local>
* Turn down blob/cosmos exception reporting to match file storage
* Restore indexing-engine.log
* Restore some basic console logging and progress for index CLI
* Semver
* Ignore small ruff complaints
* Fix CLI console printing
* Move covariate run conditional
* All pipeline registration
* Fix method name construction
* Rename context storage -> output_storage
* Rename OutputConfig as generic StorageConfig
* Reuse Storage model under InputConfig
* Move input storage creation out of document loading
* Move document loading into workflows
* Semver
* Fix smoke test config for new workflows
* Fix unit tests
---------
Co-authored-by: Alonso Guevara <alonsog@microsoft.com>
* Update tiktoken
* Add max_completion_tokens to model config
* Update/remove outdated comments
* Remove max_tokens from report generation
* Remove max_tokens from entity summarization
* Remove logit_bias from graph extraction
* Remove logit_bias from claim extraction
* Swap params if reasoning model
* Add reasoning model support to basic search
* Add reasoning model support for local and global search
* Support reasoning models with dynamic community selection
* Support reasoning models in DRIFT search
* Remove unused num_threads entry
* Semver
* Update openai
* Add reasoning_effort param
* Rework update output structure
* Semver
* Fix unit test
* Update frequency in incremental
---------
Co-authored-by: Alonso Guevara <alonsog@microsoft.com>
* Add vector store id reference to embeddings config.
* changed structure of output config section
* added cli integration for multi index global
* added cli integration for multi index local
* added cli integration for multi index drift and basic
* finished local testing of multi-index cli
* ruff fixes
* partially refactored test code to align with new output section
* more test changes for new output structure
* semversioner
* refactored to align with new multi index config proposal
* locally tested new multi-index output proposal
* cleaned up tests to align with new structure
---------
Co-authored-by: Derek Worthen <worthend.derek@gmail.com>
* remove unused columns and change property document_attribute_columns to metadata
* format file
* fix 'metadata' column on output
* run check
* fix test on nltk
* remove docs changes
* Require explicit azure auth settings when using AOI.
- Must set LanguageModel.azure_auth_type to either
"api_key" or "managed_identity" when using AOI.
* Fix smoke tests
* Use general auth_type property instead of azure_auth_type
* Remove unused error type
* Update validation
* Update validation comment
* added multi-global-query function header
* ported over code for merging dataframes
* added connection to global streaming api function
* added function header for update context helper
* implemented and incorperated update_context function
* Updated to make sure 'parent' column in final_communities gets incremented for multi index.
* first cut at multi_local_seach function
* several minor changes and fixes
* Updated multi index local search.
* Cleaned up code.
* fixed lambda function ruff errors
* fixed more ruff errors
* moved query api helpers to util file
* moved index api helpers to util file
* merged in code left out of conflict
* changed GraphRagConfig object to support lists of vector stores
* Updated with fixes for multi_local_search.
* Minor updates.
* Minor updates.
* Updates for ruff check.
* Minor updates.
* removed redundant vector_store_configs arg
* ruff formatting changes
* semversioner
* Minor fix.
* spellcheck fixes
* ruff
* test fix for cicd errors
* another test fix
* added explicit typing for ci tests
* added dict type check for vector_store during indexing
* more ruff fixes
* moved type check
* Removed streaming. Added multi drift and basic searches.
* Formatting changes.
* Updates for pyright.
* Update for ruff.
* Ruff formatted.
* first cut at fixing vector store typing errors
* got multi local search working with new config
* ruff and test fixes
* added fix for embeddings type error
* renamed multi index api functions
* ruff
* convert config model to dict[VectorStoreConfig]
* modified tests to support new vector_store model
* ruff fixes
* changed some test setups to match new model
* changed ci/cd settings files to match new structure
* Fix stderror check
* fixed bug in vector_store_config validation
* ruff
* add database_name field to vectorstoreconfig
* removed print statements
* small refactoring for PR comments
* modified default config in test
* modified vector store config unit test
---------
Co-authored-by: dorbaker <dorbaker@microsoft.com>
Co-authored-by: Alonso Guevara <alonsog@microsoft.com>
* Refactor config
- Add new ModelConfig to represent LLM settings
- Combines LLMParameters, ParallelizationParameters, encoding_model, and async_mode
- Add top level models config that is a list of available LLM ModelConfigs
- Remove LLMConfig inheritance and delete LLMConfig
- Replace the inheritance with a model_id reference to the ModelConfig listed in the top level models config
- Remove all fallbacks and hydration logic from create_graphrag_config
- This removes the automatic env variable overrides
- Support env variables within config files using Templating
- This requires "$" to be escaped with extra "$" so ".*\\.txt$" becomes ".*\\.txt$$"
- Update init content to initialize new config file with the ModelConfig structure
* Use dict of ModelConfig instead of list
* Add model validations and unit tests
* Fix ruff checks
* Add semversioner change
* Fix unit tests
* validate root_dir in pydantic model
* Rename ModelConfig to LanguageModelConfig
* Rename ModelConfigMissingError to LanguageModelConfigMissingError
* Add validationg for unexpected API keys
* Allow skipping pydantic validation for testing/mocking purposes.
* Add default lm configs to verb tests
* smoke test
* remove config from flows to fix llm arg mapping
* Fix embedding llm arg mapping
* Remove timestamp from smoke test outputs
* Remove unused "subworkflows" smoke test properties
* Add models to smoke test configs
* Update smoke test output path
* Send logs to logs folder
* Fix output path
* Fix csv test file pattern
* Update placeholder
* Format
* Instantiate default model configs
* Fix unit tests for config defaults
* Fix migration notebook
* Remove create_pipeline_config
* Remove several unused config models
* Remove indexing embedding and input configs
* Move embeddings function to config
* Remove skip_workflows
* Remove skip embeddings in favor of explicit naming
* fix unit test spelling mistake
* self.models[model_id] is already a language model. Remove redundant casting.
* update validation errors to instruct users to rerun graphrag init
* instantiate LanguageModelConfigs with validation
* skip validation in unit tests
* update verb tests to use default model settings instead of skipping validation
* test using llm settings
* cleanup verb tests
* remove unsafe default model config
* remove the ability to skip pydantic validation
* remove None union types when default values are set
* move vector_store from embeddings to top level of config and delete resolve_paths
* update vector store settings
* fix vector store and smoke tests
* fix serializing vector_store settings
* fix vector_store usage
* fix vector_store type
* support cli overrides for loading graphrag config
* rename storage to output
* Add --force flag to init
* Remove run_id and resume, fix Drift config assignment
* Ruff
---------
Co-authored-by: Nathan Evans <github@talkswithnumbers.com>
Co-authored-by: Alonso Guevara <alonsog@microsoft.com>
* add text_splitting unit test
* change folder test text splitting
* fix chunk fn
* test new function
* run formatter
* run spell check
* run semver
* remove tiktoken mocked from tests
* change progress ticker
* fix ruff check
* added cosmosdb constructor and database methods
* added rest of abstract method headers
* added cosmos db container methods
* implemented has and delete methods
* finished implementing abstract class methods
* integrated class into storage factory
* integrated cosmosdb class into cache factory
* added support for new config file fields
* replaced primary key cosmosdb initialization with connection strings
* modified cosmosdb setter to require json
* Fix non-default emitters
* Format
* Ruff
* ruff
* first successful run of cosmosdb indexing
* removed extraneous container_name setting
* require base_dir to be typed as str
* reverted merged changed from closed branch
* removed nested try statement
* readded initial non-parquet emitter fix
* added basic support for parquet emitter using internal conversions
* merged with main and resolved conflicts
* fixed more merge conflicts
* added cosmosdb functionality to query pipeline
* tested query for cosmosdb
* collapsed cosmosdb schema to use minimal containers and databases
* simplified create_database and create_container functions
* ruff fixes and semversioner
* spellcheck and ci fixes
* updated pyproject toml and lock file
* apply fixes after merge from main
* add temporary comments
* refactor cache factory
* refactored storage factory
* minor formatting
* update dictionary
* fix spellcheck typo
* fix default value
* fix pydantic model defaults
* update pydantic models
* fix init_content
* cleanup how factory passes parameters to file storage
* remove unnecessary output file type
* update pydantic model
* cleanup code
* implemented clear method
* fix merge from main
* add test stub for cosmosdb
* regenerate lock file
* modified set method to collapse parquet rows
* modified get method to collapse parquet rows
* updated has and delete methods and docstrings to adhere to new schema
* added prefix helper function
* replaced delimiter for prefixed id
* verified empty tests are passing
* fix merges from main
* add find test
* update cicd step name
* tested querying for new schema
* resolved errors from merge conflicts
* refactored set method to handle cache in new schema
* refactored get method to handle cache in new schema
* force unique ids to be written to cosmos for nodes
* found bug with has and delete methods
* modified has and delete to work with cache in new schema
* fix the merge from main
* minor typo fixes
* update lock file
* spellcheck fix
* fix init function signature
* minor formatting updates
* remove https protocol
* change localhost to 127.0.0.1 address
* update pytest to use bacj engine
* verified cache tests
* improved speed of has function
* resolved pytest error with find function
* added test for child method
* make container_name variable private as _container_name
* minor variable name fix
* cleanup cosmos pytest and make the cosmosdb storage class operations more efficient
* update cicd to use different cosmosdb emulator
* test with http protocol
* added pytest for clear()
* add longer timeout for cosmosdb emulator startup
* revert http connection back to https
* add comments to cicd code for future dev usage
* set to container and database clients to none upon deletion
* ruff changes
* add comments to cicd code
* removed unneeded None statements and ruff fixes
* more ruff fixes
* Update test_run.py
* remove unnecessary call to delete container
* ruff format updates
* Reverted test_run.py
* fix ruff formatter errors
* cleanup variable names to be more consistent
* remove extra semversioner file
* revert pydantic model changes
* revert pydantic model change
* revert pydantic model change
* re-enable inline formatting rule
* update documentation in dev guide
---------
Co-authored-by: Alonso Guevara <alonsog@microsoft.com>
Co-authored-by: Josh Bradley <joshbradley@microsoft.com>
* Firsst cut at config cleanup
* Reorder top nav
* Add query prompts to tuning page
* Remove dynamic notebook from nav
* Add more thorough yml config descriptions in docs
* Further clean out the config
* Semver
* Add new blog post
* Emphasize yaml
* Clarify output
* Fix unit test
* Fix bullet nesting
* Move indexing prompts to root
* Move query prompts to root
* Export query prompts during init
* Extract general knowledge prompt
* Load query prompts from disk
* Semver
* Fix unit tests
* Add source documents for verb tests
* Remove entity_type erroneous column
* Add new test data
* Remove source/target degree columns
* Remove top_level_node_id
* Remove chunk column configs
* Rename "chunk" to "text"
* Rename "chunk" to "text" in base
* Re-map document input to use base text units
* Revert base text units as final documents dep
* Update test data
* Split/rename node source_id
* Drop node size (dup of degree)
* Drop document_ids from covariates
* Remove unused document_ids from models
* Remove n_tokens from covariate table
* Fix missed document_ids delete
* Wire base text units to final documents
* Rename relationship rank as combined_degree
* Add rank as first-class property to Relationship
* Remove split_text operation
* Fix relationships test parquet
* Update test parquets
* Add entity ids to community table
* Remove stored graph embedding columns
* Format
* Semver
* Fix JSON typo
* Spelling
* Rename lancedb
* Sort lancedb
* Fix unit test
* Fix test to account for changing period
* Update tests for separate embeddings
* Format
* Better assertion printing
* Fix unit test for windows
* Rename document.raw_content -> document.text
* Remove read_documents function
* Remove unused document summary from model
* Remove unused imports
* Format
* Add new snapshots to default init
* Use util to construct embeddings collection name
* Align inc index model with branch changes
* Update data and tests for int ids
* Clean up embedding locs
* Switch entity "name" to "title" for consistency
* Fix short_id -> human_readable_id defaults
* Format
* Rework community IDs
* Fix community size compute
* Fix unit tests
* Fix report read
* Pare down nodes table output
* Fix unit test
* Fix merge
* Fix community loading
* Format
* Fix community id report extraction
* Update tests
* Consistent short IDs and ordering
* Update ordering and tests
* Update incremental for new nodes model
* Guard document columns loc
* Match column ordering
* Fix document guard
* Update smoke tests
* Fill NA on community extract
* Logging for smoke test debug
* Add parquet schema details doc
* Fix community hierarchy guard
* Use better empty hierarchy guard
* Back-compat shims
* Semver
* Fix warning
* Format
* Remove default fallback
* Reuse key
