* progress on vector customization
* fix for lancedb vectors
* cosmosdb implementation
* uv run poe format
* clean test for vector store
* semversioner update
* test_factory.py integration test fixes
* fixes for cosmosdb test
* integration test fix for lancedb
* uv fix for format
* test fixes
* fixes for tests
* fix cosmosdb bug
* print statement
* test
* test
* fix cosmosdb bug
* test validation
* validation cosmosdb
* validate cosmosdb
* fix cosmosdb
* fix small feedback from PR
---------
Co-authored-by: Gaudy Blanco <gaudy-microsoft@MacBook-Pro-m4-Gaudy-For-Work.local>
* Move covariate run conditional
* All pipeline registration
* Fix method name construction
* Rename context storage -> output_storage
* Rename OutputConfig as generic StorageConfig
* Reuse Storage model under InputConfig
* Move input storage creation out of document loading
* Move document loading into workflows
* Semver
* Fix smoke test config for new workflows
* Fix unit tests
---------
Co-authored-by: Alonso Guevara <alonsog@microsoft.com>
* Update tiktoken
* Add max_completion_tokens to model config
* Update/remove outdated comments
* Remove max_tokens from report generation
* Remove max_tokens from entity summarization
* Remove logit_bias from graph extraction
* Remove logit_bias from claim extraction
* Swap params if reasoning model
* Add reasoning model support to basic search
* Add reasoning model support for local and global search
* Support reasoning models with dynamic community selection
* Support reasoning models in DRIFT search
* Remove unused num_threads entry
* Semver
* Update openai
* Add reasoning_effort param
* Add vector store id reference to embeddings config.
* generated initial vector store pytests
* cleaned up cosmosdb vector store test
* fixed class name typo and debugged cosmosdb vector store test
* reset emulator connection string
* remove unneccessary comments
* removed extra comments from azure ai search test
* ruff
* semversioner
* fix cicd issues
* bypass diskANN policy for test env
* handle floating point inprecisions
---------
Co-authored-by: Derek Worthen <worthend.derek@gmail.com>
* Added support for verbose logging and csv-metadata to the prompt tune client.
* Updated community report summarization file name and prompt template
* updated semversioner
* ran ruff linter
* Ran poe format
* Fix Ruff complains
* Fix a new ruff complain :P
* Pyright
* Fix tests
---------
Co-authored-by: Gabriel Nieves <gnievesponce@microsoft.com>
Co-authored-by: Alonso Guevara <alonsog@microsoft.com>
* Refactor config
- Add new ModelConfig to represent LLM settings
- Combines LLMParameters, ParallelizationParameters, encoding_model, and async_mode
- Add top level models config that is a list of available LLM ModelConfigs
- Remove LLMConfig inheritance and delete LLMConfig
- Replace the inheritance with a model_id reference to the ModelConfig listed in the top level models config
- Remove all fallbacks and hydration logic from create_graphrag_config
- This removes the automatic env variable overrides
- Support env variables within config files using Templating
- This requires "$" to be escaped with extra "$" so ".*\\.txt$" becomes ".*\\.txt$$"
- Update init content to initialize new config file with the ModelConfig structure
* Use dict of ModelConfig instead of list
* Add model validations and unit tests
* Fix ruff checks
* Add semversioner change
* Fix unit tests
* validate root_dir in pydantic model
* Rename ModelConfig to LanguageModelConfig
* Rename ModelConfigMissingError to LanguageModelConfigMissingError
* Add validationg for unexpected API keys
* Allow skipping pydantic validation for testing/mocking purposes.
* Add default lm configs to verb tests
* smoke test
* remove config from flows to fix llm arg mapping
* Fix embedding llm arg mapping
* Remove timestamp from smoke test outputs
* Remove unused "subworkflows" smoke test properties
* Add models to smoke test configs
* Update smoke test output path
* Send logs to logs folder
* Fix output path
* Fix csv test file pattern
* Update placeholder
* Format
* Instantiate default model configs
* Fix unit tests for config defaults
* Fix migration notebook
* Remove create_pipeline_config
* Remove several unused config models
* Remove indexing embedding and input configs
* Move embeddings function to config
* Remove skip_workflows
* Remove skip embeddings in favor of explicit naming
* fix unit test spelling mistake
* self.models[model_id] is already a language model. Remove redundant casting.
* update validation errors to instruct users to rerun graphrag init
* instantiate LanguageModelConfigs with validation
* skip validation in unit tests
* update verb tests to use default model settings instead of skipping validation
* test using llm settings
* cleanup verb tests
* remove unsafe default model config
* remove the ability to skip pydantic validation
* remove None union types when default values are set
* move vector_store from embeddings to top level of config and delete resolve_paths
* update vector store settings
* fix vector store and smoke tests
* fix serializing vector_store settings
* fix vector_store usage
* fix vector_store type
* support cli overrides for loading graphrag config
* rename storage to output
* Add --force flag to init
* Remove run_id and resume, fix Drift config assignment
* Ruff
---------
Co-authored-by: Nathan Evans <github@talkswithnumbers.com>
Co-authored-by: Alonso Guevara <alonsog@microsoft.com>
* added cosmosdb constructor and database methods
* added rest of abstract method headers
* added cosmos db container methods
* implemented has and delete methods
* finished implementing abstract class methods
* integrated class into storage factory
* integrated cosmosdb class into cache factory
* added support for new config file fields
* replaced primary key cosmosdb initialization with connection strings
* modified cosmosdb setter to require json
* Fix non-default emitters
* Format
* Ruff
* ruff
* first successful run of cosmosdb indexing
* removed extraneous container_name setting
* require base_dir to be typed as str
* reverted merged changed from closed branch
* removed nested try statement
* readded initial non-parquet emitter fix
* added basic support for parquet emitter using internal conversions
* merged with main and resolved conflicts
* fixed more merge conflicts
* added cosmosdb functionality to query pipeline
* tested query for cosmosdb
* collapsed cosmosdb schema to use minimal containers and databases
* simplified create_database and create_container functions
* ruff fixes and semversioner
* spellcheck and ci fixes
* updated pyproject toml and lock file
* apply fixes after merge from main
* add temporary comments
* refactor cache factory
* refactored storage factory
* minor formatting
* update dictionary
* fix spellcheck typo
* fix default value
* fix pydantic model defaults
* update pydantic models
* fix init_content
* cleanup how factory passes parameters to file storage
* remove unnecessary output file type
* update pydantic model
* cleanup code
* implemented clear method
* fix merge from main
* add test stub for cosmosdb
* regenerate lock file
* modified set method to collapse parquet rows
* modified get method to collapse parquet rows
* updated has and delete methods and docstrings to adhere to new schema
* added prefix helper function
* replaced delimiter for prefixed id
* verified empty tests are passing
* fix merges from main
* add find test
* update cicd step name
* tested querying for new schema
* resolved errors from merge conflicts
* refactored set method to handle cache in new schema
* refactored get method to handle cache in new schema
* force unique ids to be written to cosmos for nodes
* found bug with has and delete methods
* modified has and delete to work with cache in new schema
* fix the merge from main
* minor typo fixes
* update lock file
* spellcheck fix
* fix init function signature
* minor formatting updates
* remove https protocol
* change localhost to 127.0.0.1 address
* update pytest to use bacj engine
* verified cache tests
* improved speed of has function
* resolved pytest error with find function
* added test for child method
* make container_name variable private as _container_name
* minor variable name fix
* cleanup cosmos pytest and make the cosmosdb storage class operations more efficient
* update cicd to use different cosmosdb emulator
* test with http protocol
* added pytest for clear()
* add longer timeout for cosmosdb emulator startup
* revert http connection back to https
* add comments to cicd code for future dev usage
* set to container and database clients to none upon deletion
* ruff changes
* add comments to cicd code
* removed unneeded None statements and ruff fixes
* more ruff fixes
* Update test_run.py
* remove unnecessary call to delete container
* ruff format updates
* Reverted test_run.py
* fix ruff formatter errors
* cleanup variable names to be more consistent
* remove extra semversioner file
* revert pydantic model changes
* revert pydantic model change
* revert pydantic model change
* re-enable inline formatting rule
* update documentation in dev guide
---------
Co-authored-by: Alonso Guevara <alonsog@microsoft.com>
Co-authored-by: Josh Bradley <joshbradley@microsoft.com>
* Remove excess vars from gh-pages build
* Delete redundant javascript ci
* Pull apart testing CI
* Clean up integration tests build
* Move storage tests to integration CI
* Take py 3.10 out of smoke tests matrix
* Use minimum supported python version for most tests
* Re-run main CI on any test change
* Add Josh and Kenny to author list
* Update auto-resolve perms
