diff --git a/examples_notebooks/drift_search/index.html b/examples_notebooks/drift_search/index.html index 36ff8cc0..1fc62a10 100644 --- a/examples_notebooks/drift_search/index.html +++ b/examples_notebooks/drift_search/index.html @@ -1998,7 +1998,7 @@ Text unit records: 12
-
[2024-11-05T18:06:35Z WARN  lance::dataset] No existing dataset at /home/runner/work/graphrag/graphrag/docs/examples_notebooks/inputs/operation dulce/lancedb/entity_description_embeddings.lance, it will be created
+
[2024-11-05T20:15:29Z WARN  lance::dataset] No existing dataset at /home/runner/work/graphrag/graphrag/docs/examples_notebooks/inputs/operation dulce/lancedb/entity_description_embeddings.lance, it will be created
@@ -2375,31 +2375,31 @@ search = DRIFTSearch(
-
 20%|██        | 1/5 [00:06<00:27,  6.87s/it]
+
 20%|██        | 1/5 [00:07<00:30,  7.67s/it]
-
 40%|████      | 2/5 [00:07<00:09,  3.08s/it]
+
 40%|████      | 2/5 [00:10<00:14,  4.91s/it]
-
 60%|██████    | 3/5 [00:07<00:03,  1.75s/it]
+
 60%|██████    | 3/5 [00:11<00:06,  3.10s/it]
-
100%|██████████| 5/5 [00:08<00:00,  1.09s/it]
+
 80%|████████  | 4/5 [00:11<00:01,  1.99s/it]
-
100%|██████████| 5/5 [00:08<00:00,  1.72s/it]
+
100%|██████████| 5/5 [00:11<00:00,  2.39s/it]
@@ -2418,121 +2418,91 @@ search = DRIFTSearch(
-
-
- - -
-
- - -
-
- - -
-
- - -
-
- -
@@ -2551,85 +2521,97 @@ search = DRIFTSearch( + + @@ -5519,7 +5464,7 @@ search = DRIFTSearch(
diff --git a/examples_notebooks/global_search/index.html b/examples_notebooks/global_search/index.html index c5e4f629..da781c7e 100644 --- a/examples_notebooks/global_search/index.html +++ b/examples_notebooks/global_search/index.html @@ -2543,15 +2543,15 @@ print(result.response)
### Major Conflict
 
-The central conflict in the story revolves around the Paranormal Military Squad's mission to establish contact with extraterrestrial intelligence. This involves deciphering alien signals and managing the potential implications of first contact. The mission is characterized by its secrecy and high stakes, as well as the challenges posed by the unknown nature of the extraterrestrial entities [Data: Reports (4, 5, 2, 3)].
+The central conflict in the story revolves around the Paranormal Military Squad's mission to establish contact with extraterrestrial intelligence. This involves deciphering alien signals and managing the potential implications of first contact. The mission is characterized by its secrecy and high stakes, as well as the challenges posed by the unknown nature of the extraterrestrial entities. The squad must navigate these uncertainties and the potential risks associated with engaging with an alien presence [Data: Reports (4, 5, 2, 3, 0)].
 
 ### Protagonists
 
-The protagonists are the key members of the Paranormal Military Squad, including Taylor Cruz, Dr. Jordan Hayes, Alex Mercer, and Sam Rivera. Each of these individuals plays a crucial role in the mission, contributing their expertise in leadership, signal decryption, diplomatic engagement, and technical innovation [Data: Reports (4, 5, 2, 3)].
+The protagonists of the story are the key members of the Paranormal Military Squad. This includes Taylor Cruz, Dr. Jordan Hayes, Alex Mercer, and Sam Rivera. Each of these individuals plays a pivotal role in the mission, contributing their unique expertise to the squad's efforts in extraterrestrial communication and engagement. Their collaboration and problem-solving skills are central to the narrative as they work together to achieve their mission objectives [Data: Reports (4, 5, 2, 3, 0)].
 
 ### Antagonist
 
-There is no clear antagonist identified in the data provided. The conflict appears to be more about the challenges and uncertainties of engaging with extraterrestrial intelligence rather than a direct opposition from a specific antagonist [Data: Reports (4, 5, 2, 3)].
+Interestingly, the story does not feature a clear antagonist in the traditional sense. Instead, the conflict is driven by the challenges and uncertainties associated with extraterrestrial contact rather than a specific opposing force. The narrative focuses on the internal and external challenges faced by the Paranormal Military Squad as they navigate the complexities of their mission [Data: Reports (4, 5, 2, 3, 0)].
@@ -2702,7 +2702,7 @@ print(f"LLM calls: {result.llm_calls}. LLM tokens: {result.prompt_tokens}")
-
LLM calls: 2. LLM tokens: 5251
+
LLM calls: 2. LLM tokens: 5274
diff --git a/examples_notebooks/local_search/index.html b/examples_notebooks/local_search/index.html index 8d6f4e65..0be42ab3 100644 --- a/examples_notebooks/local_search/index.html +++ b/examples_notebooks/local_search/index.html @@ -2359,7 +2359,7 @@ entity_df.head()
-
[2024-11-05T18:11:22Z WARN  lance::dataset] No existing dataset at /home/runner/work/graphrag/graphrag/docs/examples_notebooks/inputs/operation dulce/lancedb/entity.description.lance, it will be created
+
[2024-11-05T20:16:58Z WARN  lance::dataset] No existing dataset at /home/runner/work/graphrag/graphrag/docs/examples_notebooks/inputs/operation dulce/lancedb/entity.description.lance, it will be created
@@ -3347,21 +3347,21 @@ print(result.response) 
### Overview of Agent Alex Mercer
 
-Agent Alex Mercer is a central figure within the Paranormal Military Squad Team at Dulce Base, where he plays a pivotal role in the team's operations and mission objectives. His responsibilities are multifaceted, encompassing leadership, strategic oversight, and direct involvement in the analysis and interpretation of extraterrestrial signals. Mercer's military background and experience are crucial to his role, as he guides the team through complex scenarios involving potential first contact with alien intelligence [Data: Entities (0, 209); Relationships (5, 8, 65)].
+Agent Alex Mercer is a key figure within the Paranormal Military Squad Team at Dulce Base, where he plays a significant leadership role. His responsibilities include overseeing operations and making critical decisions during potential first contact scenarios with extraterrestrial intelligence. Mercer's military background equips him with the skills necessary to guide his team through complex interspecies communication efforts and the decryption of alien signals [Data: Entities (0, 209); Relationships (5, 8, 65)].
 
-### Leadership and Responsibilities
+### Role and Responsibilities
 
-As a leader, Alex Mercer is instrumental in overseeing the team's efforts to engage with extraterrestrial intelligence. His role involves ensuring a cautious and strategic approach to interspecies communication, which is vital for the success of their mission. Mercer is responsible for guiding the team's response to extraterrestrial contact, emphasizing the importance of understanding and engaging with alien signals. His leadership is characterized by a mix of concern and anticipation, reflecting the gravity of the mission at hand [Data: Entities (0); Relationships (5, 6, 8)].
+Mercer is deeply involved in the analysis of paranormal activities and the operational aspects of Dulce Base's command center. His role extends to ensuring a cautious approach to interspecies communication, unraveling galactic mysteries, and engaging with alien signals. As a leader, he is responsible for guiding the team's response to extraterrestrial contact, emphasizing the importance of understanding and engaging with the signals [Data: Entities (0, 209); Claims (73, 82)].
 
 ### Collaboration and Team Dynamics
 
-Agent Mercer works closely with other key members of the Paranormal Military Squad, such as Dr. Jordan Hayes and Sam Rivera. His collaboration with Dr. Hayes is particularly significant, as they jointly focus on decrypting and communicating with extraterrestrial intelligence. This partnership is built on mutual respect and recognition of each other's analytical skills, which are essential for the team's success. Mercer's interactions with Sam Rivera highlight the importance of intuition and trust in addition to technology for the mission's success [Data: Reports (0); Relationships (1, 2, 4, 26, 67)].
+Agent Mercer works closely with other key members of the Paranormal Military Squad, such as Dr. Jordan Hayes. Their collaboration focuses on decrypting and communicating with extraterrestrial intelligence, highlighting a mutual respect for each other's analytical skills. This partnership is crucial for the success of their mission, as they navigate the complexities of alien communication [Data: Relationships (1, 4, 26, 67)].
 
-### Involvement in Extraterrestrial Communication
+### Challenges and Strategic Approach
 
-Mercer's involvement in the decryption and analysis of alien signals is a critical aspect of his role. He is seen as a key figure in establishing contact and dialogue with alien intelligence, acting as a representative of humanity. This responsibility underscores the significance of his work, as the team navigates the complexities of communicating with beings from other worlds. Mercer's efforts are not only focused on overseeing the team but also on unraveling galactic mysteries and engaging with alien signals [Data: Claims (73, 82); Reports (0)].
+Mercer faces the challenge of balancing the need for technological solutions with the importance of intuition and trust within his team. He discusses these aspects with colleagues like Agent Sam Rivera, emphasizing the psychological complexities of their mission. Mercer's leadership is characterized by a diplomatic approach to potential extraterrestrial contact, ensuring that the engagement is handled with strategic foresight [Data: Claims (8, 50); Sources (9)].
 
-In summary, Agent Alex Mercer is a vital member of the Paranormal Military Squad Team, whose leadership and expertise are crucial in guiding the team through the challenges of extraterrestrial communication and potential first contact scenarios. His role is marked by a strategic approach, collaboration with key team members, and a deep involvement in the analysis of alien signals.
+In summary, Agent Alex Mercer is a pivotal leader within the Paranormal Military Squad, guiding his team through the intricacies of extraterrestrial communication and ensuring a strategic approach to potential first contact scenarios. His collaboration with team members and his balanced strategy underscore his critical role in the mission at Dulce Base.
@@ -3405,27 +3405,23 @@ print(result.response)
-
### Overview of Dr. Jordan Hayes
+
## Overview of Dr. Jordan Hayes
 
-Dr. Jordan Hayes is a prominent member of the Paranormal Military Squad, a specialized team based at Dulce Base, dedicated to the study and communication with extraterrestrial entities. Dr. Hayes plays a crucial role in the team's mission, focusing on deciphering alien code and interpreting extraterrestrial patterns. This expertise is vital to the squad's efforts in understanding and interacting with alien intelligence [Data: Entities (104, 2); Reports (0)].
+Dr. Jordan Hayes is a prominent member of the Paranormal Military Squad, a specialized team based at Dulce Base, dedicated to the study and communication with extraterrestrial entities. Dr. Hayes plays a crucial role in the team's mission, focusing on deciphering alien code and interpreting extraterrestrial patterns. This expertise is vital to the squad's efforts to understand and interact with alien intelligence [Data: Entities (104, 2); Reports (0)].
 
-### Role and Responsibilities
+## Role and Responsibilities
 
-Dr. Hayes is deeply involved in the analysis and interpretation of alien signals, which includes isolating signal harmonics, decrypting alien messages, and interpreting these signals for further analysis. This work is central to the team's operations at Dulce Base, where Dr. Hayes collaborates closely with other team members, such as Agent Alex Mercer, to manage interspecies communication [Data: Entities (2, 166, 192); Relationships (1, 4, 26, 67)].
+Dr. Hayes is deeply involved in the analysis and interpretation of alien signals, which includes isolating signal harmonics and decrypting alien messages. This work is central to the team's operations at Dulce Base, where they aim to establish interspecies communication. Dr. Hayes' analytical mindset and expertise in decryption algorithms are pivotal in deciphering extraterrestrial signals, which is a cornerstone of the team's mission [Data: Entities (2, 166, 180); Claims (61, 68, 79)].
 
-### Expertise and Approach
+## Collaboration and Team Dynamics
 
-Known for an analytical and skeptical mindset, Dr. Hayes emphasizes empirical evidence and adaptability in the face of the unknown. This approach is particularly evident in their work on decryption algorithms and signal analysis, which are crucial for deciphering extraterrestrial communications. Dr. Hayes' role is pivotal in the team's focus on analyzing and interpreting alien signals, which is a cornerstone of their operation at Dulce Base [Data: Entities (2, 180, 124); Claims (12, 54, 68)].
+Dr. Hayes works closely with other key members of the Paranormal Military Squad, such as Agent Alex Mercer. Their collaboration is essential for managing interspecies communication and interpreting alien signals. The mutual respect and understanding between Dr. Hayes and Agent Mercer highlight the importance of teamwork in achieving the squad's objectives. Additionally, Dr. Hayes collaborates with colleagues like Sam Rivera and Taylor Cruz, further emphasizing the collective effort required to decipher alien code and prepare for first contact [Data: Reports (0); Relationships (1, 4, 21, 9)].
 
-### Collaboration and Impact
+## Scientific Contributions and Challenges
 
-Dr. Hayes works closely with colleagues such as Sam Rivera and Taylor Cruz, contributing to the team's collective effort in deciphering alien code and preparing for first contact. The collaboration with Agent Alex Mercer is particularly significant, as they share a mutual respect and understanding of each other's strengths and the mission's significance. This teamwork is essential for the success of the Paranormal Military Squad's objectives [Data: Reports (0); Relationships (21, 9, 1, 4, 67)].
+Dr. Hayes is known for their analytical and skeptical approach, emphasizing empirical evidence and adaptability in the face of the unknown. This skepticism is evident in their contemplation of potential data and evidence that could be discovered during the mission. Dr. Hayes' work on deciphering alien patterns is not only about understanding potential threats but also about uncovering untapped wisdom, which could lead to significant scientific breakthroughs [Data: Claims (12, 49, 84); Sources (5)].
 
-### Scientific Breakthroughs and Challenges
-
-Dr. Hayes is on the verge of a scientific breakthrough, as their work involves deciphering alien code that could reveal extraterrestrial thought patterns. This endeavor is not without its challenges, as Dr. Hayes must navigate the complexities of evolving alien signals and the potential implications of these communications. The discovery of hidden technology and the analysis of extraterrestrial patterns suggest that Dr. Hayes' contributions are critical to the team's success and the broader understanding of alien intelligence [Data: Reports (0); Claims (18, 49, 74, 84)].
-
-In summary, Dr. Jordan Hayes is a key figure in the Paranormal Military Squad, bringing a critical and methodical approach to the team's extraordinary endeavors in extraterrestrial communication and analysis. Their work at Dulce Base is instrumental in advancing humanity's understanding of alien entities and preparing for potential interstellar interactions.
+In summary, Dr. Jordan Hayes is a central figure in the Paranormal Military Squad's efforts to engage with extraterrestrial intelligence. Their expertise in alien code deciphering and signal analysis is crucial to the team's mission at Dulce Base, and their collaborative efforts with other team members underscore the importance of teamwork in this groundbreaking endeavor.
@@ -3987,7 +3983,7 @@ print(candidate_questions.response)
-
['- What is the role of Alex Mercer in Operation: Dulce?', '- How does the Paranormal Military Squad interact with extraterrestrial intelligence at Dulce Base?', '- What are the main objectives of Operation: Dulce?', '- How does the environment of Dulce Military Base affect the team members?', '- What challenges does the Paranormal Military Squad face during their mission at Dulce Base?']
+
['- What is the role of Alex Mercer in Operation: Dulce?', '- How does the Paranormal Military Squad engage with extraterrestrial intelligence at Dulce Base?', '- What are the main objectives of Operation: Dulce?', '- How does the environment of Dulce Military Base affect the team members?', '- What challenges does the Paranormal Military Squad face during their mission at Dulce Base?']
diff --git a/query/drift_search/index.html b/query/drift_search/index.html index 564448e4..d717d03d 100644 --- a/query/drift_search/index.html +++ b/query/drift_search/index.html @@ -1447,8 +1447,10 @@

DRIFT search (Dynamic Reasoning and Inference with Flexible Traversal) builds upon Microsoft’s GraphRAG technique, combining characteristics of both global and local search to generate detailed responses in a method that balances computational costs with quality outcomes using our drift search method.

Methodology

-Figure 1. An entire DRIFT search hierarchy highlighting the three core phases of the DRIFT search process. A (Primer): DRIFT compares the user’s query with the top K most semantically relevant community reports, generating a broad initial answer and follow-up questions to steer further exploration. B (Follow-Up): DRIFT uses local search to refine queries, producing additional intermediate answers and follow-up questions that enhance specificity, guiding the engine towards context-rich information. A glyph on each node in the diagram shows the confidence the algorithm has to continue the query expansion step. C (Output Hierarchy): The final output is a hierarchical structure of questions and answers ranked by relevance, reflecting a balanced mix of global insights and local refinements, making the results adaptable and comprehensive. +Figure 1. An entire DRIFT search hierarchy highlighting the three core phases of the DRIFT search process.

+

+Figure 1. An entire DRIFT search hierarchy highlighting the three core phases of the DRIFT search process. A (Primer): DRIFT compares the user’s query with the top K most semantically relevant community reports, generating a broad initial answer and follow-up questions to steer further exploration. B (Follow-Up): DRIFT uses local search to refine queries, producing additional intermediate answers and follow-up questions that enhance specificity, guiding the engine towards context-rich information. A glyph on each node in the diagram shows the confidence the algorithm has to continue the query expansion step. C (Output Hierarchy): The final output is a hierarchical structure of questions and answers ranked by relevance, reflecting a balanced mix of global insights and local refinements, making the results adaptable and comprehensive.

DRIFT Search introduces a new approach to local search queries by including community information in the search process. This greatly expands the breadth of the query’s starting point and leads to retrieval and usage of a far higher variety of facts in the final answer. This addition expands the GraphRAG query engine by providing a more comprehensive option for local search, which uses community insights to refine a query into detailed follow-up questions. diff --git a/search/search_index.json b/search/search_index.json index 2113a501..8f87d376 100644 --- a/search/search_index.json +++ b/search/search_index.json @@ -1 +1 @@ -{"config": {"lang": ["en"], "separator": "[\\s\\-]+", "pipeline": ["stopWordFilter"]}, "docs": [{"location": "", "title": "Welcome to GraphRAG", "text": "

\ud83d\udc49 Microsoft Research Blog Post \ud83d\udc49 GraphRAG Accelerator \ud83d\udc49 GraphRAG Arxiv

Figure 1: An LLM-generated knowledge graph built using GPT-4 Turbo.

GraphRAG is a structured, hierarchical approach to Retrieval Augmented Generation (RAG), as opposed to naive semantic-search approaches using plain text snippets. The GraphRAG process involves extracting a knowledge graph out of raw text, building a community hierarchy, generating summaries for these communities, and then leveraging these structures when perform RAG-based tasks.

To learn more about GraphRAG and how it can be used to enhance your LLMs ability to reason about your private data, please visit the Microsoft Research Blog Post.

"}, {"location": "#solution-accelerator", "title": "Solution Accelerator \ud83d\ude80", "text": "

To quickstart the GraphRAG system we recommend trying the Solution Accelerator package. This provides a user-friendly end-to-end experience with Azure resources.

"}, {"location": "#get-started-with-graphrag", "title": "Get Started with GraphRAG \ud83d\ude80", "text": "

To start using GraphRAG, check out the Get Started guide. For a deeper dive into the main sub-systems, please visit the docpages for the Indexer and Query packages.

"}, {"location": "#graphrag-vs-baseline-rag", "title": "GraphRAG vs Baseline RAG \ud83d\udd0d", "text": "

Retrieval-Augmented Generation (RAG) is a technique to improve LLM outputs using real-world information. This technique is an important part of most LLM-based tools and the majority of RAG approaches use vector similarity as the search technique, which we call Baseline RAG. GraphRAG uses knowledge graphs to provide substantial improvements in question-and-answer performance when reasoning about complex information. RAG techniques have shown promise in helping LLMs to reason about private datasets - data that the LLM is not trained on and has never seen before, such as an enterprise\u2019s proprietary research, business documents, or communications. Baseline RAG was created to help solve this problem, but we observe situations where baseline RAG performs very poorly. For example:

To address this, the tech community is working to develop methods that extend and enhance RAG. Microsoft Research\u2019s new approach, GraphRAG, uses LLMs to create a knowledge graph based on an input corpus. This graph, along with community summaries and graph machine learning outputs, are used to augment prompts at query time. GraphRAG shows substantial improvement in answering the two classes of questions described above, demonstrating intelligence or mastery that outperforms other approaches previously applied to private datasets.

"}, {"location": "#the-graphrag-process", "title": "The GraphRAG Process \ud83e\udd16", "text": "

GraphRAG builds upon our prior research and tooling using graph machine learning. The basic steps of the GraphRAG process are as follows:

"}, {"location": "#index", "title": "Index", "text": ""}, {"location": "#query", "title": "Query", "text": "

At query time, these structures are used to provide materials for the LLM context window when answering a question. The primary query modes are:

"}, {"location": "#prompt-tuning", "title": "Prompt Tuning", "text": "

Using GraphRAG with your data out of the box may not yield the best possible results. We strongly recommend to fine-tune your prompts following the Prompt Tuning Guide in our documentation.

"}, {"location": "blog_posts/", "title": "Microsoft Research Blog", "text": ""}, {"location": "blog_posts/#published-february-13-2024-by-jonathan-larsonhttpswwwmicrosoftcomen-usresearchpeoplejolarso-senior-principal-data-architect-steven-truitthttpswwwmicrosoftcomen-usresearchpeoplesteventruitt-principal-program-manager", "title": "Published February 13, 2024 By [Jonathan Larson](https://www.microsoft.com/en-us/research/people/jolarso/), Senior Principal Data Architect; [Steven Truitt](https://www.microsoft.com/en-us/research/people/steventruitt/), Principal Program Manager", "text": ""}, {"location": "blog_posts/#published-july-2-2024-by-darren-edgehttpswwwmicrosoftcomen-usresearchpeopledaedge-senior-director-ha-trinhhttpswwwmicrosoftcomen-usresearchpeopletrinhha-senior-data-scientist-steven-truitthttpswwwmicrosoftcomen-usresearchpeoplesteventruitt-principal-program-manager-jonathan-larsonhttpswwwmicrosoftcomen-usresearchpeoplejolarso-senior-principal-data-architect", "title": "Published July 2, 2024 By [Darren Edge](https://www.microsoft.com/en-us/research/people/daedge/), Senior Director; [Ha Trinh](https://www.microsoft.com/en-us/research/people/trinhha/), Senior Data Scientist; [Steven Truitt](https://www.microsoft.com/en-us/research/people/steventruitt/), Principal Program Manager; [Jonathan Larson](https://www.microsoft.com/en-us/research/people/jolarso/), Senior Principal Data Architect", "text": ""}, {"location": "blog_posts/#published-september-9-2024-by-alonso-guevara-fernandezhttpswwwmicrosoftcomen-usresearchpeoplealonsog-sr-software-engineer-katy-smith-data-scientist-ii-joshua-bradleyhttpswwwmicrosoftcomen-usresearchpeoplejoshbradley-senior-data-scientist-darren-edgehttpswwwmicrosoftcomen-usresearchpeopledaedge-senior-director-ha-trinhhttpswwwmicrosoftcomen-usresearchpeopletrinhha-senior-data-scientist-sarah-smithhttpswwwmicrosoftcomen-usresearchpeoplesmithsarah-senior-program-manager-ben-cutlerhttpswwwmicrosoftcomen-usresearchpeoplebcutler-senior-director-steven-truitthttpswwwmicrosoftcomen-usresearchpeoplesteventruitt-principal-program-manager-jonathan-larsonhttpswwwmicrosoftcomen-usresearchpeoplejolarso-senior-principal-data-architect-octicons-arrow-right-24-introducing-drift-search-combining-global-and-local-search-methods-to-improve-quality-and-efficiencyhttpswwwmicrosoftcomen-usresearchblogintroducing-drift-search-combining-global-and-local-search-methods-to-improve-quality-and-efficiency-published-october-31-2024-by-julian-whiting-senior-machine-learning-engineer-zachary-hills-senior-software-engineer-alonso-guevara-fernandezhttpswwwmicrosoftcomen-usresearchpeoplealonsog-sr-software-engineer-ha-trinhhttpswwwmicrosoftcomen-usresearchpeopletrinhha-senior-data-scientist-adam-bradley-managing-partner-strategic-research-jonathan-larsonhttpswwwmicrosoftcomen-usresearchpeoplejolarso-senior-principal-data-architect", "title": "Published September 9, 2024 By [Alonso Guevara Fern\u00e1ndez](https://www.microsoft.com/en-us/research/people/alonsog/), Sr. Software Engineer; Katy Smith, Data Scientist II; [Joshua Bradley](https://www.microsoft.com/en-us/research/people/joshbradley/), Senior Data Scientist; [Darren Edge](https://www.microsoft.com/en-us/research/people/daedge/), Senior Director; [Ha Trinh](https://www.microsoft.com/en-us/research/people/trinhha/), Senior Data Scientist; [Sarah Smith](https://www.microsoft.com/en-us/research/people/smithsarah/), Senior Program Manager; [Ben Cutler](https://www.microsoft.com/en-us/research/people/bcutler/), Senior Director; [Steven Truitt](https://www.microsoft.com/en-us/research/people/steventruitt/), Principal Program Manager; [Jonathan Larson](https://www.microsoft.com/en-us/research/people/jolarso/), Senior Principal Data Architect - [:octicons-arrow-right-24: **Introducing DRIFT Search: Combining global and local search methods to improve quality and efficiency**](https://www.microsoft.com/en-us/research/blog/introducing-drift-search-combining-global-and-local-search-methods-to-improve-quality-and-efficiency/) ***", "text": ""}, {"location": "blog_posts/#published-october-31-2024-by-julian-whiting-senior-machine-learning-engineer-zachary-hills-senior-software-engineer-alonso-guevara-fernandezhttpswwwmicrosoftcomen-usresearchpeoplealonsog-sr-software-engineer-ha-trinhhttpswwwmicrosoftcomen-usresearchpeopletrinhha-senior-data-scientist-adam-bradley-managing-partner-strategic-research-jonathan-larsonhttpswwwmicrosoftcomen-usresearchpeoplejolarso-senior-principal-data-architect", "title": "Published October 31, 2024 By Julian Whiting , Senior Machine Learning Engineer; Zachary Hills , Senior Software Engineer; [Alonso Guevara Fern\u00e1ndez](https://www.microsoft.com/en-us/research/people/alonsog/), Sr. Software Engineer; [Ha Trinh](https://www.microsoft.com/en-us/research/people/trinhha/), Senior Data Scientist; Adam Bradley , Managing Partner, Strategic Research; [Jonathan Larson](https://www.microsoft.com/en-us/research/people/jolarso/), Senior Principal Data Architect", "text": ""}, {"location": "cli/", "title": "CLI Reference", "text": "

This page documents the command-line interface of the graphrag library.

"}, {"location": "cli/#graphrag", "title": "graphrag", "text": "

GraphRAG: A graph-based retrieval-augmented generation (RAG) system.

Usage:

 [OPTIONS] COMMAND [ARGS]...\n

Options:

  --install-completion  Install completion for the current shell.\n  --show-completion     Show completion for the current shell, to copy it or\n                        customize the installation.\n
"}, {"location": "cli/#index", "title": "index", "text": "

Build a knowledge graph index.

Usage:

 index [OPTIONS]\n

Options:

  --config PATH                   The configuration to use.\n  --root PATH                     The project root directory.  [default: .]\n  --verbose / --no-verbose        Run the indexing pipeline with verbose\n                                  logging  [default: no-verbose]\n  --memprofile / --no-memprofile  Run the indexing pipeline with memory\n                                  profiling  [default: no-memprofile]\n  --resume TEXT                   Resume a given indexing run\n  --reporter [rich|print|none]    The progress reporter to use.  [default:\n                                  rich]\n  --emit TEXT                     The data formats to emit, comma-separated.\n                                  [default: parquet]\n  --dry-run / --no-dry-run        Run the indexing pipeline without executing\n                                  any steps to inspect and validate the\n                                  configuration.  [default: no-dry-run]\n  --cache / --no-cache            Use LLM cache.  [default: cache]\n  --skip-validation / --no-skip-validation\n                                  Skip any preflight validation. Useful when\n                                  running no LLM steps.  [default: no-skip-\n                                  validation]\n  --output PATH                   Indexing pipeline output directory.\n                                  Overrides storage.base_dir in the\n                                  configuration file.\n
"}, {"location": "cli/#init", "title": "init", "text": "

Generate a default configuration file.

Usage:

 init [OPTIONS]\n

Options:

  --root PATH  The project root directory.  [required]\n
"}, {"location": "cli/#prompt-tune", "title": "prompt-tune", "text": "

Generate custom graphrag prompts with your own data (i.e. auto templating).

Usage:

 prompt-tune [OPTIONS]\n

Options:

  --root PATH                     The project root directory.  [default: .]\n  --config PATH                   The configuration to use.\n  --domain TEXT                   The domain your input data is related to.\n                                  For example 'space science', 'microbiology',\n                                  'environmental news'. If not defined, a\n                                  domain will be inferred from the input data.\n  --selection-method [all|random|top|auto]\n                                  The text chunk selection method.  [default:\n                                  random]\n  --n-subset-max INTEGER          The number of text chunks to embed when\n                                  --selection-method=auto.  [default: 300]\n  --k INTEGER                     The maximum number of documents to select\n                                  from each centroid when --selection-\n                                  method=auto.  [default: 15]\n  --limit INTEGER                 The number of documents to load when\n                                  --selection-method={random,top}.  [default:\n                                  15]\n  --max-tokens INTEGER            The max token count for prompt generation.\n                                  [default: 2000]\n  --min-examples-required INTEGER\n                                  The minimum number of examples to\n                                  generate/include in the entity extraction\n                                  prompt.  [default: 2]\n  --chunk-size INTEGER            The max token count for prompt generation.\n                                  [default: 200]\n  --language TEXT                 The primary language used for inputs and\n                                  outputs in graphrag prompts.\n  --discover-entity-types / --no-discover-entity-types\n                                  Discover and extract unspecified entity\n                                  types.  [default: discover-entity-types]\n  --output PATH                   The directory to save prompts to, relative\n                                  to the project root directory.  [default:\n                                  prompts]\n
"}, {"location": "cli/#query", "title": "query", "text": "

Query a knowledge graph index.

Usage:

 query [OPTIONS]\n

Options:

  --method [local|global|drift]  The query algorithm to use.  [required]\n  --query TEXT                   The query to execute.  [required]\n  --config PATH                  The configuration to use.\n  --data PATH                    Indexing pipeline output directory (i.e.\n                                 contains the parquet files).\n  --root PATH                    The project root directory.  [default: .]\n  --community-level INTEGER      The community level in the Leiden community\n                                 hierarchy from which to load community\n                                 reports. Higher values represent reports from\n                                 smaller communities.  [default: 2]\n  --response-type TEXT           Free form text describing the response type\n                                 and format, can be anything, e.g. Multiple\n                                 Paragraphs, Single Paragraph, Single\n                                 Sentence, List of 3-7 Points, Single Page,\n                                 Multi-Page Report. Default: Multiple\n                                 Paragraphs  [default: Multiple Paragraphs]\n  --streaming / --no-streaming   Print response in a streaming manner.\n                                 [default: no-streaming]\n
"}, {"location": "developing/", "title": "Development Guide", "text": ""}, {"location": "developing/#requirements", "title": "Requirements", "text": "Name Installation Purpose Python 3.10-3.12 Download The library is Python-based. Poetry Instructions Poetry is used for package management and virtualenv management in Python codebases"}, {"location": "developing/#getting-started", "title": "Getting Started", "text": ""}, {"location": "developing/#install-dependencies", "title": "Install Dependencies", "text": "
# Install Python dependencies.\npoetry install\n
"}, {"location": "developing/#execute-the-indexing-engine", "title": "Execute the Indexing Engine", "text": "
poetry run poe index <...args>\n
"}, {"location": "developing/#executing-queries", "title": "Executing Queries", "text": "
poetry run poe query <...args>\n
"}, {"location": "developing/#azurite", "title": "Azurite", "text": "

Some unit and smoke tests use Azurite to emulate Azure resources. This can be started by running:

./scripts/start-azurite.sh\n

or by simply running azurite in the terminal if already installed globally. See the Azurite documentation for more information about how to install and use Azurite.

"}, {"location": "developing/#lifecycle-scripts", "title": "Lifecycle Scripts", "text": "

Our Python package utilizes Poetry to manage dependencies and poethepoet to manage build scripts.

Available scripts are:

"}, {"location": "developing/#troubleshooting", "title": "Troubleshooting", "text": ""}, {"location": "developing/#runtimeerror-llvm-config-failed-executing-please-point-llvm_config-to-the-path-for-llvm-config-when-running-poetry-install", "title": "\"RuntimeError: llvm-config failed executing, please point LLVM_CONFIG to the path for llvm-config\" when running poetry install", "text": "

Make sure llvm-9 and llvm-9-dev are installed:

sudo apt-get install llvm-9 llvm-9-dev

and then in your bashrc, add

export LLVM_CONFIG=/usr/bin/llvm-config-9

"}, {"location": "developing/#numba_pymoduleh610-fatal-error-pythonh-no-such-file-or-directory-when-running-poetry-install", "title": "\"numba/_pymodule.h:6:10: fatal error: Python.h: No such file or directory\" when running poetry install", "text": "

Make sure you have python3.10-dev installed or more generally python<version>-dev

sudo apt-get install python3.10-dev

"}, {"location": "developing/#llm-call-constantly-exceeds-tpm-rpm-or-time-limits", "title": "LLM call constantly exceeds TPM, RPM or time limits", "text": "

GRAPHRAG_LLM_THREAD_COUNT and GRAPHRAG_EMBEDDING_THREAD_COUNT are both set to 50 by default. You can modify this values to reduce concurrency. Please refer to the Configuration Documents

"}, {"location": "get_started/", "title": "Getting Started", "text": ""}, {"location": "get_started/#requirements", "title": "Requirements", "text": "

Python 3.10-3.12

To get started with the GraphRAG system, you have a few options:

\ud83d\udc49 Use the GraphRAG Accelerator solution \ud83d\udc49 Install from pypi. \ud83d\udc49 Use it from source

"}, {"location": "get_started/#quickstart", "title": "Quickstart", "text": "

To get started with the GraphRAG system we recommend trying the Solution Accelerator package. This provides a user-friendly end-to-end experience with Azure resources.

"}, {"location": "get_started/#top-level-modules", "title": "Top-Level Modules", "text": ""}, {"location": "get_started/#overview", "title": "Overview", "text": "

The following is a simple end-to-end example for using the GraphRAG system. It shows how to use the system to index some text, and then use the indexed data to answer questions about the documents.

"}, {"location": "get_started/#install-graphrag", "title": "Install GraphRAG", "text": "
pip install graphrag\n

The graphrag library includes a CLI for a no-code approach to getting started. Please review the full CLI documentation for further detail.

"}, {"location": "get_started/#running-the-indexer", "title": "Running the Indexer", "text": "

Now we need to set up a data project and some initial configuration. Let's set that up. We're using the default configuration mode, which you can customize as needed using a config file, which we recommend, or environment variables.

First let's get a sample dataset ready:

mkdir -p ./ragtest/input\n

Now let's get a copy of A Christmas Carol by Charles Dickens from a trusted source

curl https://www.gutenberg.org/cache/epub/24022/pg24022.txt -o ./ragtest/input/book.txt\n

Next we'll inject some required config variables:

"}, {"location": "get_started/#set-up-your-workspace-variables", "title": "Set Up Your Workspace Variables", "text": "

First let's make sure to setup the required environment variables. For details on these environment variables, and what environment variables are available, see the variables documentation.

To initialize your workspace, first run the graphrag init command. Since we have already configured a directory named ./ragtest in the previous step, run the following command:

graphrag init --root ./ragtest\n

This will create two files: .env and settings.yaml in the ./ragtest directory.

"}, {"location": "get_started/#openai-and-azure-openai", "title": "OpenAI and Azure OpenAI", "text": "

If running in OpenAI mode, update the value of GRAPHRAG_API_KEY in the .env file with your OpenAI API key.

"}, {"location": "get_started/#azure-openai", "title": "Azure OpenAI", "text": "

In addition, Azure OpenAI users should set the following variables in the settings.yaml file. To find the appropriate sections, just search for the llm: configuration, you should see two sections, one for the chat endpoint and one for the embeddings endpoint. Here is an example of how to configure the chat endpoint:

type: azure_openai_chat # Or azure_openai_embedding for embeddings\napi_base: https://<instance>.openai.azure.com\napi_version: 2024-02-15-preview # You can customize this for other versions\ndeployment_name: <azure_model_deployment_name>\n
"}, {"location": "get_started/#running-the-indexing-pipeline", "title": "Running the Indexing pipeline", "text": "

Finally we'll run the pipeline!

graphrag index --root ./ragtest\n

This process will take some time to run. This depends on the size of your input data, what model you're using, and the text chunk size being used (these can be configured in your settings.yml file). Once the pipeline is complete, you should see a new folder called ./ragtest/output with a series of parquet files.

"}, {"location": "get_started/#using-the-query-engine", "title": "Using the Query Engine", "text": ""}, {"location": "get_started/#running-the-query-engine", "title": "Running the Query Engine", "text": "

Now let's ask some questions using this dataset.

Here is an example using Global search to ask a high-level question:

graphrag query \\\n--root ./ragtest \\\n--method global \\\n--query \"What are the top themes in this story?\"\n

Here is an example using Local search to ask a more specific question about a particular character:

graphrag query \\\n--root ./ragtest \\\n--method local \\\n--query \"Who is Scrooge and what are his main relationships?\"\n

Please refer to Query Engine docs for detailed information about how to leverage our Local and Global search mechanisms for extracting meaningful insights from data after the Indexer has wrapped up execution.

"}, {"location": "config/custom/", "title": "Fully Custom Config", "text": "

The primary configuration sections for Indexing Engine pipelines are described below. Each configuration section can be expressed in Python (for use in Python API mode) as well as YAML, but YAML is show here for brevity.

Using custom configuration is an advanced use-case. Most users will want to use the Default Configuration instead.

"}, {"location": "config/custom/#indexing-engine-examples", "title": "Indexing Engine Examples", "text": "

The examples directory contains several examples of how to use the indexing engine with custom configuration.

Most examples include two different forms of running the pipeline, both are contained in the examples run.py

  1. Using mostly the Python API
  2. Using mostly the a pipeline configuration file

To run an example:

For example to run the single_verb example, you would run the following commands:

poetry shell\n
PYTHONPATH=\"$(pwd)\" python examples/single_verb/run.py\n
"}, {"location": "config/custom/#configuration-sections", "title": "Configuration Sections", "text": ""}, {"location": "config/custom/#extends", "title": "> extends", "text": "

This configuration allows you to extend a base configuration file or files.

# single base\nextends: ../base_config.yml\n
# multiple bases\nextends:\n  - ../base_config.yml\n  - ../base_config2.yml\n
"}, {"location": "config/custom/#root_dir", "title": "> root_dir", "text": "

This configuration allows you to set the root directory for the pipeline. All data inputs and outputs are assumed to be relative to this path.

root_dir: /workspace/data_project\n
"}, {"location": "config/custom/#storage", "title": "> storage", "text": "

This configuration allows you define the output strategy for the pipeline.

"}, {"location": "config/custom/#cache", "title": "> cache", "text": "

This configuration allows you define the cache strategy for the pipeline.

"}, {"location": "config/custom/#reporting", "title": "> reporting", "text": "

This configuration allows you define the reporting strategy for the pipeline. Report files are generated artifacts that summarize the performance metrics of the pipeline and emit any error messages.

"}, {"location": "config/custom/#workflows", "title": "> workflows", "text": "

This configuration section defines the workflow DAG for the pipeline. Here we define an array of workflows and express their inter-dependencies in steps:

workflows:\n  - name: workflow1\n    steps:\n      - verb: derive\n        args:\n          column1: \"col1\"\n          column2: \"col2\"\n  - name: workflow2\n    steps:\n      - verb: derive\n        args:\n          column1: \"col1\"\n          column2: \"col2\"\n        input:\n          # dependency established here\n          source: workflow:workflow1\n
"}, {"location": "config/custom/#input", "title": "> input", "text": " 
input:\n  type: file\n  file_type: csv\n  base_dir: ../data/csv # the directory containing the CSV files, this is relative to the config file\n  file_pattern: '.*[\\/](?P<source>[^\\/]+)[\\/](?P<year>\\d{4})-(?P<month>\\d{2})-(?P<day>\\d{2})_(?P<author>[^_]+)_\\d+\\.csv$' # a regex to match the CSV files\n  # An additional file filter which uses the named groups from the file_pattern to further filter the files\n  # file_filter:\n  #   # source: (source_filter)\n  #   year: (2023)\n  #   month: (06)\n  #   # day: (22)\n  source_column: \"author\" # the column containing the source/author of the data\n  text_column: \"message\" # the column containing the text of the data\n  timestamp_column: \"date(yyyyMMddHHmmss)\" # optional, the column containing the timestamp of the data\n  timestamp_format: \"%Y%m%d%H%M%S\" # optional,  the format of the timestamp\n  post_process: # Optional, set of steps to process the data before going into the workflow\n    - verb: filter\n      args:\n        column: \"title\",\n        value: \"My document\"\n
input:\n  type: file\n  file_type: csv\n  base_dir: ../data/csv # the directory containing the CSV files, this is relative to the config file\n  file_pattern: '.*[\\/](?P<source>[^\\/]+)[\\/](?P<year>\\d{4})-(?P<month>\\d{2})-(?P<day>\\d{2})_(?P<author>[^_]+)_\\d+\\.csv$' # a regex to match the CSV files\n  # An additional file filter which uses the named groups from the file_pattern to further filter the files\n  # file_filter:\n  #   # source: (source_filter)\n  #   year: (2023)\n  #   month: (06)\n  #   # day: (22)\n  post_process: # Optional, set of steps to process the data before going into the workflow\n    - verb: filter\n      args:\n        column: \"title\",\n        value: \"My document\"\n
"}, {"location": "config/env_vars/", "title": "Default Configuration Mode (using Env Vars)", "text": ""}, {"location": "config/env_vars/#text-embeddings-customization", "title": "Text-Embeddings Customization", "text": "

By default, the GraphRAG indexer will only emit embeddings required for our query methods. However, the model has embeddings defined for all plaintext fields, and these can be generated by setting the GRAPHRAG_EMBEDDING_TARGET environment variable to all.

If the embedding target is all, and you want to only embed a subset of these fields, you may specify which embeddings to skip using the GRAPHRAG_EMBEDDING_SKIP argument described below.

"}, {"location": "config/env_vars/#embedded-fields", "title": "Embedded Fields", "text": ""}, {"location": "config/env_vars/#input-data", "title": "Input Data", "text": "

Our pipeline can ingest .csv or .txt data from an input folder. These files can be nested within subfolders. To configure how input data is handled, what fields are mapped over, and how timestamps are parsed, look for configuration values starting with GRAPHRAG_INPUT_ below. In general, CSV-based data provides the most customizability. Each CSV should at least contain a text field (which can be mapped with environment variables), but it's helpful if they also have title, timestamp, and source fields. Additional fields can be included as well, which will land as extra fields on the Document table.

"}, {"location": "config/env_vars/#base-llm-settings", "title": "Base LLM Settings", "text": "

These are the primary settings for configuring LLM connectivity.

Parameter Required? Description Type Default Value GRAPHRAG_API_KEY Yes for OpenAI. Optional for AOAI The API key. (Note: OPENAI_API_KEY is also used as a fallback). If not defined when using AOAI, managed identity will be used. |str|None` GRAPHRAG_API_BASE For AOAI The API Base URL str None GRAPHRAG_API_VERSION For AOAI The AOAI API version. str None GRAPHRAG_API_ORGANIZATION The AOAI organization. str None GRAPHRAG_API_PROXY The AOAI proxy. str None"}, {"location": "config/env_vars/#text-generation-settings", "title": "Text Generation Settings", "text": "

These settings control the text generation model used by the pipeline. Any settings with a fallback will use the base LLM settings, if available.

Parameter Required? Description Type Default Value GRAPHRAG_LLM_TYPE For AOAI The LLM operation type. Either openai_chat or azure_openai_chat str openai_chat GRAPHRAG_LLM_DEPLOYMENT_NAME For AOAI The AOAI model deployment name. str None GRAPHRAG_LLM_API_KEY Yes (uses fallback) The API key. If not defined when using AOAI, managed identity will be used. str None GRAPHRAG_LLM_API_BASE For AOAI (uses fallback) The API Base URL str None GRAPHRAG_LLM_API_VERSION For AOAI (uses fallback) The AOAI API version. str None GRAPHRAG_LLM_API_ORGANIZATION For AOAI (uses fallback) The AOAI organization. str None GRAPHRAG_LLM_API_PROXY The AOAI proxy. str None GRAPHRAG_LLM_MODEL The LLM model. str gpt-4-turbo-preview GRAPHRAG_LLM_MAX_TOKENS The maximum number of tokens. int 4000 GRAPHRAG_LLM_REQUEST_TIMEOUT The maximum number of seconds to wait for a response from the chat client. int 180 GRAPHRAG_LLM_MODEL_SUPPORTS_JSON Indicates whether the given model supports JSON output mode. True to enable. str None GRAPHRAG_LLM_THREAD_COUNT The number of threads to use for LLM parallelization. int 50 GRAPHRAG_LLM_THREAD_STAGGER The time to wait (in seconds) between starting each thread. float 0.3 GRAPHRAG_LLM_CONCURRENT_REQUESTS The number of concurrent requests to allow for the embedding client. int 25 GRAPHRAG_LLM_TOKENS_PER_MINUTE The number of tokens per minute to allow for the LLM client. 0 = Bypass int 0 GRAPHRAG_LLM_REQUESTS_PER_MINUTE The number of requests per minute to allow for the LLM client. 0 = Bypass int 0 GRAPHRAG_LLM_MAX_RETRIES The maximum number of retries to attempt when a request fails. int 10 GRAPHRAG_LLM_MAX_RETRY_WAIT The maximum number of seconds to wait between retries. int 10 GRAPHRAG_LLM_SLEEP_ON_RATE_LIMIT_RECOMMENDATION Whether to sleep on rate limit recommendation. (Azure Only) bool True GRAPHRAG_LLM_TEMPERATURE The temperature to use generation. float 0 GRAPHRAG_LLM_TOP_P The top_p to use for sampling. float 1 GRAPHRAG_LLM_N The number of responses to generate. int 1"}, {"location": "config/env_vars/#text-embedding-settings", "title": "Text Embedding Settings", "text": "

These settings control the text embedding model used by the pipeline. Any settings with a fallback will use the base LLM settings, if available.

Parameter Required ? Description Type Default GRAPHRAG_EMBEDDING_TYPE For AOAI The embedding client to use. Either openai_embedding or azure_openai_embedding str openai_embedding GRAPHRAG_EMBEDDING_DEPLOYMENT_NAME For AOAI The AOAI deployment name. str None GRAPHRAG_EMBEDDING_API_KEY Yes (uses fallback) The API key to use for the embedding client. If not defined when using AOAI, managed identity will be used. str None GRAPHRAG_EMBEDDING_API_BASE For AOAI (uses fallback) The API base URL. str None GRAPHRAG_EMBEDDING_API_VERSION For AOAI (uses fallback) The AOAI API version to use for the embedding client. str None GRAPHRAG_EMBEDDING_API_ORGANIZATION For AOAI (uses fallback) The AOAI organization to use for the embedding client. str None GRAPHRAG_EMBEDDING_API_PROXY The AOAI proxy to use for the embedding client. str None GRAPHRAG_EMBEDDING_MODEL The model to use for the embedding client. str text-embedding-3-small GRAPHRAG_EMBEDDING_BATCH_SIZE The number of texts to embed at once. (Azure limit is 16) int 16 GRAPHRAG_EMBEDDING_BATCH_MAX_TOKENS The maximum tokens per batch (Azure limit is 8191) int 8191 GRAPHRAG_EMBEDDING_TARGET The target fields to embed. Either required or all. str required GRAPHRAG_EMBEDDING_SKIP A comma-separated list of fields to skip embeddings for . (e.g. 'relationship.description') str None GRAPHRAG_EMBEDDING_THREAD_COUNT The number of threads to use for parallelization for embeddings. int GRAPHRAG_EMBEDDING_THREAD_STAGGER The time to wait (in seconds) between starting each thread for embeddings. float 50 GRAPHRAG_EMBEDDING_CONCURRENT_REQUESTS The number of concurrent requests to allow for the embedding client. int 25 GRAPHRAG_EMBEDDING_TOKENS_PER_MINUTE The number of tokens per minute to allow for the embedding client. 0 = Bypass int 0 GRAPHRAG_EMBEDDING_REQUESTS_PER_MINUTE The number of requests per minute to allow for the embedding client. 0 = Bypass int 0 GRAPHRAG_EMBEDDING_MAX_RETRIES The maximum number of retries to attempt when a request fails. int 10 GRAPHRAG_EMBEDDING_MAX_RETRY_WAIT The maximum number of seconds to wait between retries. int 10 GRAPHRAG_EMBEDDING_SLEEP_ON_RATE_LIMIT_RECOMMENDATION Whether to sleep on rate limit recommendation. (Azure Only) bool True"}, {"location": "config/env_vars/#input-settings", "title": "Input Settings", "text": "

These settings control the data input used by the pipeline. Any settings with a fallback will use the base LLM settings, if available.

"}, {"location": "config/env_vars/#plaintext-input-data-graphrag_input_file_typetext", "title": "Plaintext Input Data (GRAPHRAG_INPUT_FILE_TYPE=text)", "text": "Parameter Description Type Required or Optional Default GRAPHRAG_INPUT_FILE_PATTERN The file pattern regexp to use when reading input files from the input directory. str optional .*\\.txt$"}, {"location": "config/env_vars/#csv-input-data-graphrag_input_file_typecsv", "title": "CSV Input Data (GRAPHRAG_INPUT_FILE_TYPE=csv)", "text": "Parameter Description Type Required or Optional Default GRAPHRAG_INPUT_TYPE The input storage type to use when reading files. (file or blob) str optional file GRAPHRAG_INPUT_FILE_PATTERN The file pattern regexp to use when reading input files from the input directory. str optional .*\\.txt$ GRAPHRAG_INPUT_SOURCE_COLUMN The 'source' column to use when reading CSV input files. str optional source GRAPHRAG_INPUT_TIMESTAMP_COLUMN The 'timestamp' column to use when reading CSV input files. str optional None GRAPHRAG_INPUT_TIMESTAMP_FORMAT The timestamp format to use when parsing timestamps in the timestamp column. str optional None GRAPHRAG_INPUT_TEXT_COLUMN The 'text' column to use when reading CSV input files. str optional text GRAPHRAG_INPUT_DOCUMENT_ATTRIBUTE_COLUMNS A list of CSV columns, comma-separated, to incorporate as document fields. str optional id GRAPHRAG_INPUT_TITLE_COLUMN The 'title' column to use when reading CSV input files. str optional title GRAPHRAG_INPUT_STORAGE_ACCOUNT_BLOB_URL The Azure Storage blob endpoint to use when in blob mode and using managed identity. Will have the format https://<storage_account_name>.blob.core.windows.net str optional None GRAPHRAG_INPUT_CONNECTION_STRING The connection string to use when reading CSV input files from Azure Blob Storage. str optional None GRAPHRAG_INPUT_CONTAINER_NAME The container name to use when reading CSV input files from Azure Blob Storage. str optional None GRAPHRAG_INPUT_BASE_DIR The base directory to read input files from. str optional None"}, {"location": "config/env_vars/#data-mapping-settings", "title": "Data Mapping Settings", "text": "Parameter Description Type Required or Optional Default GRAPHRAG_INPUT_FILE_TYPE The type of input data, csv or text str optional text GRAPHRAG_INPUT_ENCODING The encoding to apply when reading CSV/text input files. str optional utf-8"}, {"location": "config/env_vars/#data-chunking", "title": "Data Chunking", "text": "Parameter Description Type Required or Optional Default GRAPHRAG_CHUNK_SIZE The chunk size in tokens for text-chunk analysis windows. str optional 1200 GRAPHRAG_CHUNK_OVERLAP The chunk overlap in tokens for text-chunk analysis windows. str optional 100 GRAPHRAG_CHUNK_BY_COLUMNS A comma-separated list of document attributes to groupby when performing TextUnit chunking. str optional id GRAPHRAG_CHUNK_ENCODING_MODEL The encoding model to use for chunking. str optional The top-level encoding model."}, {"location": "config/env_vars/#prompting-overrides", "title": "Prompting Overrides", "text": "Parameter Description Type Required or Optional Default GRAPHRAG_ENTITY_EXTRACTION_PROMPT_FILE The path (relative to the root) of an entity extraction prompt template text file. str optional None GRAPHRAG_ENTITY_EXTRACTION_MAX_GLEANINGS The maximum number of redrives (gleanings) to invoke when extracting entities in a loop. int optional 1 GRAPHRAG_ENTITY_EXTRACTION_ENTITY_TYPES A comma-separated list of entity types to extract. str optional organization,person,event,geo GRAPHRAG_ENTITY_EXTRACTION_ENCODING_MODEL The encoding model to use for entity extraction. str optional The top-level encoding model. GRAPHRAG_SUMMARIZE_DESCRIPTIONS_PROMPT_FILE The path (relative to the root) of an description summarization prompt template text file. str optional None GRAPHRAG_SUMMARIZE_DESCRIPTIONS_MAX_LENGTH The maximum number of tokens to generate per description summarization. int optional 500 GRAPHRAG_CLAIM_EXTRACTION_ENABLED Whether claim extraction is enabled for this pipeline. bool optional False GRAPHRAG_CLAIM_EXTRACTION_DESCRIPTION The claim_description prompting argument to utilize. string optional \"Any claims or facts that could be relevant to threat analysis.\" GRAPHRAG_CLAIM_EXTRACTION_PROMPT_FILE The claim extraction prompt to utilize. string optional None GRAPHRAG_CLAIM_EXTRACTION_MAX_GLEANINGS The maximum number of redrives (gleanings) to invoke when extracting claims in a loop. int optional 1 GRAPHRAG_CLAIM_EXTRACTION_ENCODING_MODEL The encoding model to use for claim extraction. str optional The top-level encoding model GRAPHRAG_COMMUNITY_REPORTS_PROMPT_FILE The community reports extraction prompt to utilize. string optional None GRAPHRAG_COMMUNITY_REPORTS_MAX_LENGTH The maximum number of tokens to generate per community reports. int optional 1500"}, {"location": "config/env_vars/#storage", "title": "Storage", "text": "

This section controls the storage mechanism used by the pipeline used for emitting output tables.

Parameter Description Type Required or Optional Default GRAPHRAG_STORAGE_TYPE The type of reporter to use. Options are file, memory, or blob str optional file GRAPHRAG_STORAGE_STORAGE_ACCOUNT_BLOB_URL The Azure Storage blob endpoint to use when in blob mode and using managed identity. Will have the format https://<storage_account_name>.blob.core.windows.net str optional None GRAPHRAG_STORAGE_CONNECTION_STRING The Azure Storage connection string to use when in blob mode. str optional None GRAPHRAG_STORAGE_CONTAINER_NAME The Azure Storage container name to use when in blob mode. str optional None GRAPHRAG_STORAGE_BASE_DIR The base path to data outputs outputs. str optional None"}, {"location": "config/env_vars/#cache", "title": "Cache", "text": "

This section controls the cache mechanism used by the pipeline. This is used to cache LLM invocation results.

Parameter Description Type Required or Optional Default GRAPHRAG_CACHE_TYPE The type of cache to use. Options are file, memory, none or blob str optional file GRAPHRAG_CACHE_STORAGE_ACCOUNT_BLOB_URL The Azure Storage blob endpoint to use when in blob mode and using managed identity. Will have the format https://<storage_account_name>.blob.core.windows.net str optional None GRAPHRAG_CACHE_CONNECTION_STRING The Azure Storage connection string to use when in blob mode. str optional None GRAPHRAG_CACHE_CONTAINER_NAME The Azure Storage container name to use when in blob mode. str optional None GRAPHRAG_CACHE_BASE_DIR The base path to the cache files. str optional None"}, {"location": "config/env_vars/#reporting", "title": "Reporting", "text": "

This section controls the reporting mechanism used by the pipeline, for common events and error messages. The default is to write reports to a file in the output directory. However, you can also choose to write reports to the console or to an Azure Blob Storage container.

Parameter Description Type Required or Optional Default GRAPHRAG_REPORTING_TYPE The type of reporter to use. Options are file, console, or blob str optional file GRAPHRAG_REPORTING_STORAGE_ACCOUNT_BLOB_URL The Azure Storage blob endpoint to use when in blob mode and using managed identity. Will have the format https://<storage_account_name>.blob.core.windows.net str optional None GRAPHRAG_REPORTING_CONNECTION_STRING The Azure Storage connection string to use when in blob mode. str optional None GRAPHRAG_REPORTING_CONTAINER_NAME The Azure Storage container name to use when in blob mode. str optional None GRAPHRAG_REPORTING_BASE_DIR The base path to the reporting outputs. str optional None"}, {"location": "config/env_vars/#node2vec-parameters", "title": "Node2Vec Parameters", "text": "Parameter Description Type Required or Optional Default GRAPHRAG_NODE2VEC_ENABLED Whether to enable Node2Vec bool optional False GRAPHRAG_NODE2VEC_NUM_WALKS The Node2Vec number of walks to perform int optional 10 GRAPHRAG_NODE2VEC_WALK_LENGTH The Node2Vec walk length int optional 40 GRAPHRAG_NODE2VEC_WINDOW_SIZE The Node2Vec window size int optional 2 GRAPHRAG_NODE2VEC_ITERATIONS The number of iterations to run node2vec int optional 3 GRAPHRAG_NODE2VEC_RANDOM_SEED The random seed to use for node2vec int optional 597832"}, {"location": "config/env_vars/#data-snapshotting", "title": "Data Snapshotting", "text": "Parameter Description Type Required or Optional Default GRAPHRAG_SNAPSHOT_EMBEDDINGS Whether to enable embeddings snapshots. bool optional False GRAPHRAG_SNAPSHOT_GRAPHML Whether to enable GraphML snapshots. bool optional False GRAPHRAG_SNAPSHOT_RAW_ENTITIES Whether to enable raw entity snapshots. bool optional False GRAPHRAG_SNAPSHOT_TOP_LEVEL_NODES Whether to enable top-level node snapshots. bool optional False GRAPHRAG_SNAPSHOT_TRANSIENT Whether to enable transient table snapshots. bool optional False"}, {"location": "config/env_vars/#miscellaneous-settings", "title": "Miscellaneous Settings", "text": "Parameter Description Type Required or Optional Default GRAPHRAG_ASYNC_MODE Which async mode to use. Either asyncio or threaded. str optional asyncio GRAPHRAG_ENCODING_MODEL The text encoding model, used in tiktoken, to encode text. str optional cl100k_base GRAPHRAG_MAX_CLUSTER_SIZE The maximum number of entities to include in a single Leiden cluster. int optional 10 GRAPHRAG_SKIP_WORKFLOWS A comma-separated list of workflow names to skip. str optional None GRAPHRAG_UMAP_ENABLED Whether to enable UMAP layouts bool optional False"}, {"location": "config/init/", "title": "Configuring GraphRAG Indexing", "text": "

To start using GraphRAG, you must generate a configuration file. The init command is the easiest way to get started. It will create a .env and settings.yaml files in the specified directory with the necessary configuration settings. It will also output the default LLM prompts used by GraphRAG.

"}, {"location": "config/init/#usage", "title": "Usage", "text": "
graphrag init [--root PATH]\n
"}, {"location": "config/init/#options", "title": "Options", "text": ""}, {"location": "config/init/#example", "title": "Example", "text": "
graphrag init --root ./ragtest\n
"}, {"location": "config/init/#output", "title": "Output", "text": "

The init command will create the following files in the specified directory:

"}, {"location": "config/init/#next-steps", "title": "Next Steps", "text": "

After initializing your workspace, you can either run the Prompt Tuning command to adapt the prompts to your data or even start running the Indexing Pipeline to index your data. For more information on configuring GraphRAG, see the Configuration documentation.

"}, {"location": "config/json_yaml/", "title": "Default Configuration Mode (using JSON/YAML)", "text": "

The default configuration mode may be configured by using a settings.json or settings.yml file in the data project root. If a .env file is present along with this config file, then it will be loaded, and the environment variables defined therein will be available for token replacements in your configuration document using ${ENV_VAR} syntax.

For example:

# .env\nAPI_KEY=some_api_key\n\n# settings.json\n{\n    \"llm\": {\n        \"api_key\": \"${API_KEY}\"\n    }\n}\n
"}, {"location": "config/json_yaml/#config-sections", "title": "Config Sections", "text": ""}, {"location": "config/json_yaml/#input", "title": "input", "text": ""}, {"location": "config/json_yaml/#fields", "title": "Fields", "text": ""}, {"location": "config/json_yaml/#llm", "title": "llm", "text": "

This is the base LLM configuration section. Other steps may override this configuration with their own LLM configuration.

"}, {"location": "config/json_yaml/#fields_1", "title": "Fields", "text": ""}, {"location": "config/json_yaml/#parallelization", "title": "parallelization", "text": ""}, {"location": "config/json_yaml/#fields_2", "title": "Fields", "text": ""}, {"location": "config/json_yaml/#async_mode", "title": "async_mode", "text": "

asyncio|threaded The async mode to use. Either asyncio or `threaded.

"}, {"location": "config/json_yaml/#embeddings", "title": "embeddings", "text": ""}, {"location": "config/json_yaml/#fields_3", "title": "Fields", "text": ""}, {"location": "config/json_yaml/#chunks", "title": "chunks", "text": ""}, {"location": "config/json_yaml/#fields_4", "title": "Fields", "text": ""}, {"location": "config/json_yaml/#cache", "title": "cache", "text": ""}, {"location": "config/json_yaml/#fields_5", "title": "Fields", "text": ""}, {"location": "config/json_yaml/#storage", "title": "storage", "text": ""}, {"location": "config/json_yaml/#fields_6", "title": "Fields", "text": ""}, {"location": "config/json_yaml/#reporting", "title": "reporting", "text": ""}, {"location": "config/json_yaml/#fields_7", "title": "Fields", "text": ""}, {"location": "config/json_yaml/#entity_extraction", "title": "entity_extraction", "text": ""}, {"location": "config/json_yaml/#fields_8", "title": "Fields", "text": ""}, {"location": "config/json_yaml/#summarize_descriptions", "title": "summarize_descriptions", "text": ""}, {"location": "config/json_yaml/#fields_9", "title": "Fields", "text": ""}, {"location": "config/json_yaml/#claim_extraction", "title": "claim_extraction", "text": ""}, {"location": "config/json_yaml/#fields_10", "title": "Fields", "text": ""}, {"location": "config/json_yaml/#community_reports", "title": "community_reports", "text": ""}, {"location": "config/json_yaml/#fields_11", "title": "Fields", "text": ""}, {"location": "config/json_yaml/#cluster_graph", "title": "cluster_graph", "text": ""}, {"location": "config/json_yaml/#fields_12", "title": "Fields", "text": ""}, {"location": "config/json_yaml/#embed_graph", "title": "embed_graph", "text": ""}, {"location": "config/json_yaml/#fields_13", "title": "Fields", "text": ""}, {"location": "config/json_yaml/#umap", "title": "umap", "text": ""}, {"location": "config/json_yaml/#fields_14", "title": "Fields", "text": ""}, {"location": "config/json_yaml/#snapshots", "title": "snapshots", "text": ""}, {"location": "config/json_yaml/#fields_15", "title": "Fields", "text": ""}, {"location": "config/json_yaml/#encoding_model", "title": "encoding_model", "text": "

str - The text encoding model to use. Default=cl100k_base.

"}, {"location": "config/json_yaml/#skip_workflows", "title": "skip_workflows", "text": "

list[str] - Which workflow names to skip.

"}, {"location": "config/overview/", "title": "Configuring GraphRAG Indexing", "text": "

The GraphRAG system is highly configurable. This page provides an overview of the configuration options available for the GraphRAG indexing engine.

"}, {"location": "config/overview/#default-configuration-mode", "title": "Default Configuration Mode", "text": "

The default configuration mode is the simplest way to get started with the GraphRAG system. It is designed to work out-of-the-box with minimal configuration. The primary configuration sections for the Indexing Engine pipelines are described below. The main ways to set up GraphRAG in Default Configuration mode are via:

"}, {"location": "config/overview/#custom-configuration-mode", "title": "Custom Configuration Mode", "text": "

Custom configuration mode is an advanced use-case. Most users will want to use the Default Configuration instead. The primary configuration sections for Indexing Engine pipelines are described below. Details about how to use custom configuration are available in the Custom Configuration Mode documentation.

"}, {"location": "config/template/", "title": "Configuration Template", "text": "

The following template can be used and stored as a .env in the the directory where you're are pointing the --root parameter on your Indexing Pipeline execution.

For details about how to run the Indexing Pipeline, refer to the Index CLI documentation.

"}, {"location": "config/template/#env-file-template", "title": ".env File Template", "text": "

Required variables are uncommented. All the optional configuration can be turned on or off as needed.

"}, {"location": "config/template/#minimal-configuration", "title": "Minimal Configuration", "text": "
# Base LLM Settings\nGRAPHRAG_API_KEY=\"your_api_key\"\nGRAPHRAG_API_BASE=\"http://<domain>.openai.azure.com\" # For Azure OpenAI Users\nGRAPHRAG_API_VERSION=\"api_version\" # For Azure OpenAI Users\n\n# Text Generation Settings\nGRAPHRAG_LLM_TYPE=\"azure_openai_chat\" # or openai_chat\nGRAPHRAG_LLM_DEPLOYMENT_NAME=\"gpt-4-turbo-preview\"\nGRAPHRAG_LLM_MODEL_SUPPORTS_JSON=True\n\n# Text Embedding Settings\nGRAPHRAG_EMBEDDING_TYPE=\"azure_openai_embedding\" # or openai_embedding\nGRAPHRAG_LLM_DEPLOYMENT_NAME=\"text-embedding-3-small\"\n\n# Data Mapping Settings\nGRAPHRAG_INPUT_TYPE=\"text\"\n
"}, {"location": "config/template/#full-configuration", "title": "Full Configuration", "text": "
# Required LLM Config\n\n# Input Data Configuration\nGRAPHRAG_INPUT_TYPE=\"file\"\n\n# Plaintext Input Data Configuration\n# GRAPHRAG_INPUT_FILE_PATTERN=.*\\.txt\n\n# Text Input Data Configuration\nGRAPHRAG_INPUT_FILE_TYPE=\"text\"\nGRAPHRAG_INPUT_FILE_PATTERN=\".*\\.txt$\"\nGRAPHRAG_INPUT_SOURCE_COLUMN=source\n# GRAPHRAG_INPUT_TIMESTAMP_COLUMN=None\n# GRAPHRAG_INPUT_TIMESTAMP_FORMAT=None\n# GRAPHRAG_INPUT_TEXT_COLUMN=\"text\"\n# GRAPHRAG_INPUT_ATTRIBUTE_COLUMNS=id\n# GRAPHRAG_INPUT_TITLE_COLUMN=\"title\"\n# GRAPHRAG_INPUT_TYPE=\"file\"\n# GRAPHRAG_INPUT_CONNECTION_STRING=None\n# GRAPHRAG_INPUT_CONTAINER_NAME=None\n# GRAPHRAG_INPUT_BASE_DIR=None\n\n# Base LLM Settings\nGRAPHRAG_API_KEY=\"your_api_key\"\nGRAPHRAG_API_BASE=\"http://<domain>.openai.azure.com\" # For Azure OpenAI Users\nGRAPHRAG_API_VERSION=\"api_version\" # For Azure OpenAI Users\n# GRAPHRAG_API_ORGANIZATION=None\n# GRAPHRAG_API_PROXY=None\n\n# Text Generation Settings\n# GRAPHRAG_LLM_TYPE=openai_chat\nGRAPHRAG_LLM_API_KEY=\"your_api_key\" # If GRAPHRAG_API_KEY is not set\nGRAPHRAG_LLM_API_BASE=\"http://<domain>.openai.azure.com\" # For Azure OpenAI Users and if GRAPHRAG_API_BASE is not set\nGRAPHRAG_LLM_API_VERSION=\"api_version\" # For Azure OpenAI Users and if GRAPHRAG_API_VERSION is not set\nGRAPHRAG_LLM_MODEL_SUPPORTS_JSON=True # Suggested by default\n# GRAPHRAG_LLM_API_ORGANIZATION=None\n# GRAPHRAG_LLM_API_PROXY=None\n# GRAPHRAG_LLM_DEPLOYMENT_NAME=None\n# GRAPHRAG_LLM_MODEL=gpt-4-turbo-preview\n# GRAPHRAG_LLM_MAX_TOKENS=4000\n# GRAPHRAG_LLM_REQUEST_TIMEOUT=180\n# GRAPHRAG_LLM_THREAD_COUNT=50\n# GRAPHRAG_LLM_THREAD_STAGGER=0.3\n# GRAPHRAG_LLM_CONCURRENT_REQUESTS=25\n# GRAPHRAG_LLM_TPM=0\n# GRAPHRAG_LLM_RPM=0\n# GRAPHRAG_LLM_MAX_RETRIES=10\n# GRAPHRAG_LLM_MAX_RETRY_WAIT=10\n# GRAPHRAG_LLM_SLEEP_ON_RATE_LIMIT_RECOMMENDATION=True\n\n# Text Embedding Settings\n# GRAPHRAG_EMBEDDING_TYPE=openai_embedding\nGRAPHRAG_EMBEDDING_API_KEY=\"your_api_key\" # If GRAPHRAG_API_KEY is not set\nGRAPHRAG_EMBEDDING_API_BASE=\"http://<domain>.openai.azure.com\"  # For Azure OpenAI Users and if GRAPHRAG_API_BASE is not set\nGRAPHRAG_EMBEDDING_API_VERSION=\"api_version\" # For Azure OpenAI Users and if GRAPHRAG_API_VERSION is not set\n# GRAPHRAG_EMBEDDING_API_ORGANIZATION=None\n# GRAPHRAG_EMBEDDING_API_PROXY=None\n# GRAPHRAG_EMBEDDING_DEPLOYMENT_NAME=None\n# GRAPHRAG_EMBEDDING_MODEL=text-embedding-3-small\n# GRAPHRAG_EMBEDDING_BATCH_SIZE=16\n# GRAPHRAG_EMBEDDING_BATCH_MAX_TOKENS=8191\n# GRAPHRAG_EMBEDDING_TARGET=required\n# GRAPHRAG_EMBEDDING_SKIP=None\n# GRAPHRAG_EMBEDDING_THREAD_COUNT=None\n# GRAPHRAG_EMBEDDING_THREAD_STAGGER=50\n# GRAPHRAG_EMBEDDING_CONCURRENT_REQUESTS=25\n# GRAPHRAG_EMBEDDING_TPM=0\n# GRAPHRAG_EMBEDDING_RPM=0\n# GRAPHRAG_EMBEDDING_MAX_RETRIES=10\n# GRAPHRAG_EMBEDDING_MAX_RETRY_WAIT=10\n# GRAPHRAG_EMBEDDING_SLEEP_ON_RATE_LIMIT_RECOMMENDATION=True\n\n# Data Mapping Settings\n# GRAPHRAG_INPUT_ENCODING=utf-8\n\n# Data Chunking\n# GRAPHRAG_CHUNK_SIZE=1200\n# GRAPHRAG_CHUNK_OVERLAP=100\n# GRAPHRAG_CHUNK_BY_COLUMNS=id\n\n# Prompting Overrides\n# GRAPHRAG_ENTITY_EXTRACTION_PROMPT_FILE=None\n# GRAPHRAG_ENTITY_EXTRACTION_MAX_GLEANINGS=1\n# GRAPHRAG_ENTITY_EXTRACTION_ENTITY_TYPES=organization,person,event,geo\n# GRAPHRAG_SUMMARIZE_DESCRIPTIONS_PROMPT_FILE=None\n# GRAPHRAG_SUMMARIZE_DESCRIPTIONS_MAX_LENGTH=500\n# GRAPHRAG_CLAIM_EXTRACTION_DESCRIPTION=\"Any claims or facts that could be relevant to threat analysis.\"\n# GRAPHRAG_CLAIM_EXTRACTION_PROMPT_FILE=None\n# GRAPHRAG_CLAIM_EXTRACTION_MAX_GLEANINGS=1\n# GRAPHRAG_COMMUNITY_REPORT_PROMPT_FILE=None\n# GRAPHRAG_COMMUNITY_REPORT_MAX_LENGTH=1500\n\n# Storage\n# GRAPHRAG_STORAGE_TYPE=file\n# GRAPHRAG_STORAGE_CONNECTION_STRING=None\n# GRAPHRAG_STORAGE_CONTAINER_NAME=None\n# GRAPHRAG_STORAGE_BASE_DIR=None\n\n# Cache\n# GRAPHRAG_CACHE_TYPE=file\n# GRAPHRAG_CACHE_CONNECTION_STRING=None\n# GRAPHRAG_CACHE_CONTAINER_NAME=None\n# GRAPHRAG_CACHE_BASE_DIR=None\n\n# Reporting\n# GRAPHRAG_REPORTING_TYPE=file\n# GRAPHRAG_REPORTING_CONNECTION_STRING=None\n# GRAPHRAG_REPORTING_CONTAINER_NAME=None\n# GRAPHRAG_REPORTING_BASE_DIR=None\n\n# Node2Vec Parameters\n# GRAPHRAG_NODE2VEC_ENABLED=False\n# GRAPHRAG_NODE2VEC_NUM_WALKS=10\n# GRAPHRAG_NODE2VEC_WALK_LENGTH=40\n# GRAPHRAG_NODE2VEC_WINDOW_SIZE=2\n# GRAPHRAG_NODE2VEC_ITERATIONS=3\n# GRAPHRAG_NODE2VEC_RANDOM_SEED=597832\n\n# Data Snapshotting\n# GRAPHRAG_SNAPSHOT_GRAPHML=False\n# GRAPHRAG_SNAPSHOT_RAW_ENTITIES=False\n# GRAPHRAG_SNAPSHOT_TOP_LEVEL_NODES=False\n\n# Miscellaneous Settings\n# GRAPHRAG_ASYNC_MODE=asyncio\n# GRAPHRAG_ENCODING_MODEL=cl100k_base\n# GRAPHRAG_MAX_CLUSTER_SIZE=10\n# GRAPHRAG_SKIP_WORKFLOWS=None\n# GRAPHRAG_UMAP_ENABLED=False\n
"}, {"location": "data/operation_dulce/ABOUT/", "title": "About", "text": "

This document (Operation Dulce) is an AI-generated science fiction novella, included here for the purposes of integration testing.

"}, {"location": "index/architecture/", "title": "Indexing Architecture", "text": ""}, {"location": "index/architecture/#key-concepts", "title": "Key Concepts", "text": ""}, {"location": "index/architecture/#knowledge-model", "title": "Knowledge Model", "text": "

In order to support the GraphRAG system, the outputs of the indexing engine (in the Default Configuration Mode) are aligned to a knowledge model we call the GraphRAG Knowledge Model. This model is designed to be an abstraction over the underlying data storage technology, and to provide a common interface for the GraphRAG system to interact with. In normal use-cases the outputs of the GraphRAG Indexer would be loaded into a database system, and the GraphRAG's Query Engine would interact with the database using the knowledge model data-store types.

"}, {"location": "index/architecture/#datashaper-workflows", "title": "DataShaper Workflows", "text": "

GraphRAG's Indexing Pipeline is built on top of our open-source library, DataShaper. DataShaper is a data processing library that allows users to declaratively express data pipelines, schemas, and related assets using well-defined schemas. DataShaper has implementations in JavaScript and Python, and is designed to be extensible to other languages.

One of the core resource types within DataShaper is a Workflow. Workflows are expressed as sequences of steps, which we call verbs. Each step has a verb name and a configuration object. In DataShaper, these verbs model relational concepts such as SELECT, DROP, JOIN, etc.. Each verb transforms an input data table, and that table is passed down the pipeline.

---\ntitle: Sample Workflow\n---\nflowchart LR\n    input[Input Table] --> select[SELECT] --> join[JOIN] --> binarize[BINARIZE] --> output[Output Table]
"}, {"location": "index/architecture/#llm-based-workflow-steps", "title": "LLM-based Workflow Steps", "text": "

GraphRAG's Indexing Pipeline implements a handful of custom verbs on top of the standard, relational verbs that our DataShaper library provides. These verbs give us the ability to augment text documents with rich, structured data using the power of LLMs such as GPT-4. We utilize these verbs in our standard workflow to extract entities, relationships, claims, community structures, and community reports and summaries. This behavior is customizable and can be extended to support many kinds of AI-based data enrichment and extraction tasks.

"}, {"location": "index/architecture/#workflow-graphs", "title": "Workflow Graphs", "text": "

Because of the complexity of our data indexing tasks, we needed to be able to express our data pipeline as series of multiple, interdependent workflows. In the GraphRAG Indexing Pipeline, each workflow may define dependencies on other workflows, effectively forming a directed acyclic graph (DAG) of workflows, which is then used to schedule processing.

---\ntitle: Sample Workflow DAG\n---\nstateDiagram-v2\n    [*] --> Prepare\n    Prepare --> Chunk\n    Chunk --> ExtractGraph\n    Chunk --> EmbedDocuments\n    ExtractGraph --> GenerateReports\n    ExtractGraph --> EmbedEntities\n    ExtractGraph --> EmbedGraph
"}, {"location": "index/architecture/#dataframe-message-format", "title": "Dataframe Message Format", "text": "

The primary unit of communication between workflows, and between workflow steps is an instance of pandas.DataFrame. Although side-effects are possible, our goal is to be data-centric and table-centric in our approach to data processing. This allows us to easily reason about our data, and to leverage the power of dataframe-based ecosystems. Our underlying dataframe technology may change over time, but our primary goal is to support the DataShaper workflow schema while retaining single-machine ease of use and developer ergonomics.

"}, {"location": "index/architecture/#llm-caching", "title": "LLM Caching", "text": "

The GraphRAG library was designed with LLM interactions in mind, and a common setback when working with LLM APIs is various errors due to network latency, throttling, etc.. Because of these potential error cases, we've added a cache layer around LLM interactions. When completion requests are made using the same input set (prompt and tuning parameters), we return a cached result if one exists. This allows our indexer to be more resilient to network issues, to act idempotently, and to provide a more efficient end-user experience.

"}, {"location": "index/default_dataflow/", "title": "Indexing Dataflow", "text": ""}, {"location": "index/default_dataflow/#the-graphrag-knowledge-model", "title": "The GraphRAG Knowledge Model", "text": "

The knowledge model is a specification for data outputs that conform to our data-model definition. You can find these definitions in the python/graphrag/graphrag/model folder within the GraphRAG repository. The following entity types are provided. The fields here represent the fields that are text-embedded by default.

"}, {"location": "index/default_dataflow/#the-default-configuration-workflow", "title": "The Default Configuration Workflow", "text": "

Let's take a look at how the default-configuration workflow transforms text documents into the GraphRAG Knowledge Model. This page gives a general overview of the major steps in this process. To fully configure this workflow, check out the configuration documentation.

---\ntitle: Dataflow Overview\n---\nflowchart TB\n    subgraph phase1[Phase 1: Compose TextUnits]\n    documents[Documents] --> chunk[Chunk]\n    chunk --> embed[Embed] --> textUnits[Text Units]\n    end\n    subgraph phase2[Phase 2: Graph Extraction]\n    textUnits --> graph_extract[Entity & Relationship Extraction]\n    graph_extract --> graph_summarize[Entity & Relationship Summarization]\n    graph_summarize --> claim_extraction[Claim Extraction]\n    claim_extraction --> graph_outputs[Graph Tables]\n    end\n    subgraph phase3[Phase 3: Graph Augmentation]\n    graph_outputs --> community_detect[Community Detection]\n    community_detect --> graph_embed[Graph Embedding]\n    graph_embed --> augmented_graph[Augmented Graph Tables]\n    end\n    subgraph phase4[Phase 4: Community Summarization]\n    augmented_graph --> summarized_communities[Community Summarization]\n    summarized_communities --> embed_communities[Community Embedding]\n    embed_communities --> community_outputs[Community Tables]\n    end\n    subgraph phase5[Phase 5: Document Processing]\n    documents --> link_to_text_units[Link to TextUnits]\n    textUnits --> link_to_text_units\n    link_to_text_units --> embed_documents[Document Embedding]\n    embed_documents --> document_graph[Document Graph Creation]\n    document_graph --> document_outputs[Document Tables]\n    end\n    subgraph phase6[Phase 6: Network Visualization]\n    document_outputs --> umap_docs[Umap Documents]\n    augmented_graph --> umap_entities[Umap Entities]\n    umap_docs --> combine_nodes[Nodes Table]\n    umap_entities --> combine_nodes\n    end
"}, {"location": "index/default_dataflow/#phase-1-compose-textunits", "title": "Phase 1: Compose TextUnits", "text": "

The first phase of the default-configuration workflow is to transform input documents into TextUnits. A TextUnit is a chunk of text that is used for our graph extraction techniques. They are also used as source-references by extracted knowledge items in order to empower breadcrumbs and provenance by concepts back to their original source tex.

The chunk size (counted in tokens), is user-configurable. By default this is set to 300 tokens, although we've had positive experience with 1200-token chunks using a single \"glean\" step. (A \"glean\" step is a follow-on extraction). Larger chunks result in lower-fidelity output and less meaningful reference texts; however, using larger chunks can result in much faster processing time.

The group-by configuration is also user-configurable. By default, we align our chunks to document boundaries, meaning that there is a strict 1-to-many relationship between Documents and TextUnits. In rare cases, this can be turned into a many-to-many relationship. This is useful when the documents are very short and we need several of them to compose a meaningful analysis unit (e.g. Tweets or a chat log)

Each of these text-units are text-embedded and passed into the next phase of the pipeline.

---\ntitle: Documents into Text Chunks\n---\nflowchart LR\n    doc1[Document 1] --> tu1[TextUnit 1]\n    doc1 --> tu2[TextUnit 2]\n    doc2[Document 2] --> tu3[TextUnit 3]\n    doc2 --> tu4[TextUnit 4]\n
"}, {"location": "index/default_dataflow/#phase-2-graph-extraction", "title": "Phase 2: Graph Extraction", "text": "

In this phase, we analyze each text unit and extract our graph primitives: Entities, Relationships, and Claims. Entities and Relationships are extracted at once in our entity_extract verb, and claims are extracted in our claim_extract verb. Results are then combined and passed into following phases of the pipeline.

---\ntitle: Graph Extraction\n---\nflowchart LR\n    tu[TextUnit] --> ge[Graph Extraction] --> gs[Graph Summarization]\n    tu --> ce[Claim Extraction]
"}, {"location": "index/default_dataflow/#entity-relationship-extraction", "title": "Entity & Relationship Extraction", "text": "

In this first step of graph extraction, we process each text-unit in order to extract entities and relationships out of the raw text using the LLM. The output of this step is a subgraph-per-TextUnit containing a list of entities with a name, type, and description, and a list of relationships with a source, target, and description.

These subgraphs are merged together - any entities with the same name and type are merged by creating an array of their descriptions. Similarly, any relationships with the same source and target are merged by creating an array of their descriptions.

"}, {"location": "index/default_dataflow/#entity-relationship-summarization", "title": "Entity & Relationship Summarization", "text": "

Now that we have a graph of entities and relationships, each with a list of descriptions, we can summarize these lists into a single description per entity and relationship. This is done by asking the LLM for a short summary that captures all of the distinct information from each description. This allows all of our entities and relationships to have a single concise description.

"}, {"location": "index/default_dataflow/#claim-extraction-emission", "title": "Claim Extraction & Emission", "text": "

Finally, as an independent workflow, we extract claims from the source TextUnits. These claims represent positive factual statements with an evaluated status and time-bounds. These are emitted as a primary artifact called Covariates.

Note: claim extraction is optional and turned off by default. This is because claim extraction generally needs prompt tuning to be useful.

"}, {"location": "index/default_dataflow/#phase-3-graph-augmentation", "title": "Phase 3: Graph Augmentation", "text": "

Now that we have a usable graph of entities and relationships, we want to understand their community structure and augment the graph with additional information. This is done in two steps: Community Detection and Graph Embedding. These give us explicit (communities) and implicit (embeddings) ways of understanding the topological structure of our graph.

---\ntitle: Graph Augmentation\n---\nflowchart LR\n    cd[Leiden Hierarchical Community Detection] --> ge[Node2Vec Graph Embedding] --> ag[Graph Table Emission]
"}, {"location": "index/default_dataflow/#community-detection", "title": "Community Detection", "text": "

In this step, we generate a hierarchy of entity communities using the Hierarchical Leiden Algorithm. This method will apply a recursive community-clustering to our graph until we reach a community-size threshold. This will allow us to understand the community structure of our graph and provide a way to navigate and summarize the graph at different levels of granularity.

"}, {"location": "index/default_dataflow/#graph-embedding", "title": "Graph Embedding", "text": "

In this step, we generate a vector representation of our graph using the Node2Vec algorithm. This will allow us to understand the implicit structure of our graph and provide an additional vector-space in which to search for related concepts during our query phase.

"}, {"location": "index/default_dataflow/#graph-tables-emission", "title": "Graph Tables Emission", "text": "

Once our graph augmentation steps are complete, the final Entities and Relationships tables are emitted after their text fields are text-embedded.

"}, {"location": "index/default_dataflow/#phase-4-community-summarization", "title": "Phase 4: Community Summarization", "text": "
---\ntitle: Community Summarization\n---\nflowchart LR\n    sc[Generate Community Reports] --> ss[Summarize Community Reports] --> ce[Community Embedding] --> co[Community Tables Emission]

At this point, we have a functional graph of entities and relationships, a hierarchy of communities for the entities, as well as node2vec embeddings.

Now we want to build on the communities data and generate reports for each community. This gives us a high-level understanding of the graph at several points of graph granularity. For example, if community A is the top-level community, we'll get a report about the entire graph. If the community is lower-level, we'll get a report about a local cluster.

"}, {"location": "index/default_dataflow/#generate-community-reports", "title": "Generate Community Reports", "text": "

In this step, we generate a summary of each community using the LLM. This will allow us to understand the distinct information contained within each community and provide a scoped understanding of the graph, from either a high-level or a low-level perspective. These reports contain an executive overview and reference the key entities, relationships, and claims within the community sub-structure.

"}, {"location": "index/default_dataflow/#summarize-community-reports", "title": "Summarize Community Reports", "text": "

In this step, each community report is then summarized via the LLM for shorthand use.

"}, {"location": "index/default_dataflow/#community-embedding", "title": "Community Embedding", "text": "

In this step, we generate a vector representation of our communities by generating text embeddings of the community report, the community report summary, and the title of the community report.

"}, {"location": "index/default_dataflow/#community-tables-emission", "title": "Community Tables Emission", "text": "

At this point, some bookkeeping work is performed and we emit the Communities and CommunityReports tables.

"}, {"location": "index/default_dataflow/#phase-5-document-processing", "title": "Phase 5: Document Processing", "text": "

In this phase of the workflow, we create the Documents table for the knowledge model.

---\ntitle: Document Processing\n---\nflowchart LR\n    aug[Augment] --> dp[Link to TextUnits] --> de[Avg. Embedding] --> dg[Document Table Emission]
"}, {"location": "index/default_dataflow/#augment-with-columns-csv-only", "title": "Augment with Columns (CSV Only)", "text": "

If the workflow is operating on CSV data, you may configure your workflow to add additional fields to Documents output. These fields should exist on the incoming CSV tables. Details about configuring this can be found in the configuration documentation.

"}, {"location": "index/default_dataflow/#link-to-textunits", "title": "Link to TextUnits", "text": "

In this step, we link each document to the text-units that were created in the first phase. This allows us to understand which documents are related to which text-units and vice-versa.

"}, {"location": "index/default_dataflow/#document-embedding", "title": "Document Embedding", "text": "

In this step, we generate a vector representation of our documents using an average embedding of document slices. We re-chunk documents without overlapping chunks, and then generate an embedding for each chunk. We create an average of these chunks weighted by token-count and use this as the document embedding. This will allow us to understand the implicit relationship between documents, and will help us generate a network representation of our documents.

"}, {"location": "index/default_dataflow/#documents-table-emission", "title": "Documents Table Emission", "text": "

At this point, we can emit the Documents table into the knowledge Model.

"}, {"location": "index/default_dataflow/#phase-6-network-visualization", "title": "Phase 6: Network Visualization", "text": "

In this phase of the workflow, we perform some steps to support network visualization of our high-dimensional vector spaces within our existing graphs. At this point there are two logical graphs at play: the Entity-Relationship graph and the Document graph.

---\ntitle: Network Visualization Workflows\n---\nflowchart LR\n    nv[Umap Documents] --> ne[Umap Entities] --> ng[Nodes Table Emission]

For each of the logical graphs, we perform a UMAP dimensionality reduction to generate a 2D representation of the graph. This will allow us to visualize the graph in a 2D space and understand the relationships between the nodes in the graph. The UMAP embeddings are then emitted as a table of Nodes. The rows of this table include a discriminator indicating whether the node is a document or an entity, and the UMAP coordinates.

"}, {"location": "index/overview/", "title": "GraphRAG Indexing \ud83e\udd16", "text": "

The GraphRAG indexing package is a data pipeline and transformation suite that is designed to extract meaningful, structured data from unstructured text using LLMs.

Indexing Pipelines are configurable. They are composed of workflows, standard and custom steps, prompt templates, and input/output adapters. Our standard pipeline is designed to:

The outputs of the pipeline can be stored in a variety of formats, including JSON and Parquet - or they can be handled manually via the Python API.

"}, {"location": "index/overview/#getting-started", "title": "Getting Started", "text": ""}, {"location": "index/overview/#requirements", "title": "Requirements", "text": "

See the requirements section in Get Started for details on setting up a development environment.

The Indexing Engine can be used in either a default configuration mode or with a custom pipeline. To configure GraphRAG, see the configuration documentation. After you have a config file you can run the pipeline using the CLI or the Python API.

"}, {"location": "index/overview/#usage", "title": "Usage", "text": ""}, {"location": "index/overview/#cli", "title": "CLI", "text": "
# Via Poetry\npoetry run poe cli --root <data_root> # default config mode\npoetry run poe cli --config your_pipeline.yml # custom config mode\n\n# Via Node\nyarn run:index --root <data_root> # default config mode\nyarn run:index --config your_pipeline.yml # custom config mode\n
"}, {"location": "index/overview/#python-api", "title": "Python API", "text": "
from graphrag.index import run_pipeline\nfrom graphrag.index.config import PipelineWorkflowReference\n\nworkflows: list[PipelineWorkflowReference] = [\n    PipelineWorkflowReference(\n        steps=[\n            {\n                # built-in verb\n                \"verb\": \"derive\",  # https://github.com/microsoft/datashaper/blob/main/python/datashaper/datashaper/verbs/derive.py\n                \"args\": {\n                    \"column1\": \"col1\",  # from above\n                    \"column2\": \"col2\",  # from above\n                    \"to\": \"col_multiplied\",  # new column name\n                    \"operator\": \"*\",  # multiply the two columns\n                },\n                # Since we're trying to act on the default input, we don't need explicitly to specify an input\n            }\n        ]\n    ),\n]\n\ndataset = pd.DataFrame([{\"col1\": 2, \"col2\": 4}, {\"col1\": 5, \"col2\": 10}])\noutputs = []\nasync for output in await run_pipeline(dataset=dataset, workflows=workflows):\n    outputs.append(output)\npipeline_result = outputs[-1]\nprint(pipeline_result)\n
"}, {"location": "index/overview/#further-reading", "title": "Further Reading", "text": ""}, {"location": "prompt_tuning/auto_prompt_tuning/", "title": "Auto Prompt Tuning \u2699\ufe0f", "text": "

GraphRAG provides the ability to create domain adapted prompts for the generation of the knowledge graph. This step is optional, though it is highly encouraged to run it as it will yield better results when executing an Index Run.

These are generated by loading the inputs, splitting them into chunks (text units) and then running a series of LLM invocations and template substitutions to generate the final prompts. We suggest using the default values provided by the script, but in this page you'll find the detail of each in case you want to further explore and tweak the prompt tuning algorithm.

Figure 1: Auto Tuning Conceptual Diagram.

"}, {"location": "prompt_tuning/auto_prompt_tuning/#prerequisites", "title": "Prerequisites", "text": "

Before running auto tuning, ensure you have already initialized your workspace with the graphrag init command. This will create the necessary configuration files and the default prompts. Refer to the Init Documentation for more information about the initialization process.

"}, {"location": "prompt_tuning/auto_prompt_tuning/#usage", "title": "Usage", "text": "

You can run the main script from the command line with various options:

graphrag prompt-tune [--root ROOT] [--domain DOMAIN]  [--method METHOD] [--limit LIMIT] [--language LANGUAGE] \\\n[--max-tokens MAX_TOKENS] [--chunk-size CHUNK_SIZE] [--n-subset-max N_SUBSET_MAX] [--k K] \\\n[--min-examples-required MIN_EXAMPLES_REQUIRED] [--no-entity-types] [--output OUTPUT]\n
"}, {"location": "prompt_tuning/auto_prompt_tuning/#command-line-options", "title": "Command-Line Options", "text": ""}, {"location": "prompt_tuning/auto_prompt_tuning/#example-usage", "title": "Example Usage", "text": "
python -m graphrag prompt-tune --root /path/to/project --config /path/to/settings.yaml --domain \"environmental news\" \\\n--method random --limit 10 --language English --max-tokens 2048 --chunk-size 256 --min-examples-required 3 \\\n--no-entity-types --output /path/to/output\n

or, with minimal configuration (suggested):

python -m graphrag prompt-tune --root /path/to/project --config /path/to/settings.yaml --no-entity-types\n
"}, {"location": "prompt_tuning/auto_prompt_tuning/#document-selection-methods", "title": "Document Selection Methods", "text": "

The auto tuning feature ingests the input data and then divides it into text units the size of the chunk size parameter. After that, it uses one of the following selection methods to pick a sample to work with for prompt generation:

"}, {"location": "prompt_tuning/auto_prompt_tuning/#modify-env-vars", "title": "Modify Env Vars", "text": "

After running auto tuning, you should modify the following environment variables (or config variables) to pick up the new prompts on your index run. Note: Please make sure to update the correct path to the generated prompts, in this example we are using the default \"prompts\" path.

or in your yaml config file:

entity_extraction:\n  prompt: \"prompts/entity_extraction.txt\"\n\nsummarize_descriptions:\n  prompt: \"prompts/summarize_descriptions.txt\"\n\ncommunity_reports:\n  prompt: \"prompts/community_report.txt\"\n
"}, {"location": "prompt_tuning/manual_prompt_tuning/", "title": "Manual Prompt Tuning \u2699\ufe0f", "text": "

The GraphRAG indexer, by default, will run with a handful of prompts that are designed to work well in the broad context of knowledge discovery. However, it is quite common to want to tune the prompts to better suit your specific use case. We provide a means for you to do this by allowing you to specify a custom prompt file, which will each use a series of token-replacements internally.

Each of these prompts may be overridden by writing a custom prompt file in plaintext. We use token-replacements in the form of {token_name}, and the descriptions for the available tokens can be found below.

"}, {"location": "prompt_tuning/manual_prompt_tuning/#entityrelationship-extraction", "title": "Entity/Relationship Extraction", "text": "

Prompt Source

"}, {"location": "prompt_tuning/manual_prompt_tuning/#tokens-values-provided-by-extractor", "title": "Tokens (values provided by extractor)", "text": ""}, {"location": "prompt_tuning/manual_prompt_tuning/#summarize-entityrelationship-descriptions", "title": "Summarize Entity/Relationship Descriptions", "text": "

Prompt Source

"}, {"location": "prompt_tuning/manual_prompt_tuning/#tokens-values-provided-by-extractor_1", "title": "Tokens (values provided by extractor)", "text": ""}, {"location": "prompt_tuning/manual_prompt_tuning/#claim-extraction", "title": "Claim Extraction", "text": "

Prompt Source

"}, {"location": "prompt_tuning/manual_prompt_tuning/#tokens-values-provided-by-extractor_2", "title": "Tokens (values provided by extractor)", "text": "

Note: there is additional parameter for the Claim Description that is used in claim extraction. The default value is

\"Any claims or facts that could be relevant to information discovery.\"

See the configuration documentation for details on how to change this.

"}, {"location": "prompt_tuning/manual_prompt_tuning/#generate-community-reports", "title": "Generate Community Reports", "text": "

Prompt Source

"}, {"location": "prompt_tuning/manual_prompt_tuning/#tokens-values-provided-by-extractor_3", "title": "Tokens (values provided by extractor)", "text": ""}, {"location": "prompt_tuning/overview/", "title": "Prompt Tuning \u2699\ufe0f", "text": "

This page provides an overview of the prompt tuning options available for the GraphRAG indexing engine.

"}, {"location": "prompt_tuning/overview/#default-prompts", "title": "Default Prompts", "text": "

The default prompts are the simplest way to get started with the GraphRAG system. It is designed to work out-of-the-box with minimal configuration. You can find more detail about these prompts in the following links:

"}, {"location": "prompt_tuning/overview/#auto-tuning", "title": "Auto Tuning", "text": "

Auto Tuning leverages your input data and LLM interactions to create domain adapted prompts for the generation of the knowledge graph. It is highly encouraged to run it as it will yield better results when executing an Index Run. For more details about how to use it, please refer to the Auto Tuning documentation.

"}, {"location": "prompt_tuning/overview/#manual-tuning", "title": "Manual Tuning", "text": "

Manual tuning is an advanced use-case. Most users will want to use the Auto Tuning feature instead. Details about how to use manual configuration are available in the Manual Tuning documentation.

"}, {"location": "query/drift_search/", "title": "DRIFT Search \ud83d\udd0e", "text": ""}, {"location": "query/drift_search/#combining-local-and-global-search", "title": "Combining Local and Global Search", "text": "

GraphRAG is a technique that uses large language models (LLMs) to create knowledge graphs and summaries from unstructured text documents and leverages them to improve retrieval-augmented generation (RAG) operations on private datasets. It offers comprehensive global overviews of large, private troves of unstructured text documents while also enabling exploration of detailed, localized information. By using LLMs to create comprehensive knowledge graphs that connect and describe entities and relationships contained in those documents, GraphRAG leverages semantic structuring of the data to generate responses to a wide variety of complex user queries.

DRIFT search (Dynamic Reasoning and Inference with Flexible Traversal) builds upon Microsoft\u2019s GraphRAG technique, combining characteristics of both global and local search to generate detailed responses in a method that balances computational costs with quality outcomes using our drift search method.

"}, {"location": "query/drift_search/#methodology", "title": "Methodology", "text": "

DRIFT Search introduces a new approach to local search queries by including community information in the search process. This greatly expands the breadth of the query\u2019s starting point and leads to retrieval and usage of a far higher variety of facts in the final answer. This addition expands the GraphRAG query engine by providing a more comprehensive option for local search, which uses community insights to refine a query into detailed follow-up questions. ## Configuration Below are the key parameters of the [DRIFTSearch class](https://github.com/microsoft/graphrag/blob/main//graphrag/query/structured_search/drift_search/search.py): - `llm`: OpenAI model object to be used for response generation - `context_builder`: [context builder](https://github.com/microsoft/graphrag/blob/main/graphrag/query/structured_search/drift_search/drift_context.py) object to be used for preparing context data from community reports and query information - `config`: model to define the DRIFT Search hyperparameters. [DRIFT Config model](https://github.com/microsoft/graphrag/blob/main/graphrag/config/models/drift_config.py) - `token_encoder`: token encoder for tracking the budget for the algorithm. - `query_state`: a state object as defined in [Query State](https://github.com/microsoft/graphrag/blob/main/graphrag/query/structured_search/drift_search/state.py) that allows to track execution of a DRIFT Search instance, alongside follow ups and [DRIFT actions](https://github.com/microsoft/graphrag/blob/main/graphrag/query/structured_search/drift_search/action.py). ## How to Use An example of a global search scenario can be found in the following [notebook](../examples_notebooks/drift_search.ipynb). ## Learn More For a more in-depth look at the DRIFT search method, please refer to our [DRIFT Search blog post](https://www.microsoft.com/en-us/research/blog/introducing-drift-search-combining-global-and-local-search-methods-to-improve-quality-and-efficiency/)"}, {"location": "query/global_search/", "title": "Global Search \ud83d\udd0e", "text": ""}, {"location": "query/global_search/#whole-dataset-reasoning", "title": "Whole Dataset Reasoning", "text": "

Baseline RAG struggles with queries that require aggregation of information across the dataset to compose an answer. Queries such as \u201cWhat are the top 5 themes in the data?\u201d perform terribly because baseline RAG relies on a vector search of semantically similar text content within the dataset. There is nothing in the query to direct it to the correct information.

However, with GraphRAG we can answer such questions, because the structure of the LLM-generated knowledge graph tells us about the structure (and thus themes) of the dataset as a whole. This allows the private dataset to be organized into meaningful semantic clusters that are pre-summarized. Using our global search method, the LLM uses these clusters to summarize these themes when responding to a user query.

"}, {"location": "query/global_search/#methodology", "title": "Methodology", "text": "
---\ntitle: Global Search Dataflow\n---\n%%{ init: { 'flowchart': { 'curve': 'step' } } }%%\nflowchart LR\n\n    uq[User Query] --- .1\n    ch1[Conversation History] --- .1\n\n    subgraph RIR\n        direction TB\n        ri1[Rated Intermediate<br/>Response 1]~~~ri2[Rated Intermediate<br/>Response 2] -.\"{1..N}\".-rin[Rated Intermediate<br/>Response N]\n    end\n\n    .1--Shuffled Community<br/>Report Batch 1-->RIR\n    .1--Shuffled Community<br/>Report Batch 2-->RIR---.2\n    .1--Shuffled Community<br/>Report Batch N-->RIR\n\n    .2--Ranking +<br/>Filtering-->agr[Aggregated Intermediate<br/>Responses]-->res[Response]\n\n\n\n     classDef green fill:#26B653,stroke:#333,stroke-width:2px,color:#fff;\n     classDef turquoise fill:#19CCD3,stroke:#333,stroke-width:2px,color:#fff;\n     classDef rose fill:#DD8694,stroke:#333,stroke-width:2px,color:#fff;\n     classDef orange fill:#F19914,stroke:#333,stroke-width:2px,color:#fff;\n     classDef purple fill:#B356CD,stroke:#333,stroke-width:2px,color:#fff;\n     classDef invisible fill:#fff,stroke:#fff,stroke-width:0px,color:#fff, width:0px;\n     class uq,ch1 turquoise;\n     class ri1,ri2,rin rose;\n     class agr orange;\n     class res purple;\n     class .1,.2 invisible;\n

Given a user query and, optionally, the conversation history, the global search method uses a collection of LLM-generated community reports from a specified level of the graph's community hierarchy as context data to generate response in a map-reduce manner. At the map step, community reports are segmented into text chunks of pre-defined size. Each text chunk is then used to produce an intermediate response containing a list of point, each of which is accompanied by a numerical rating indicating the importance of the point. At the reduce step, a filtered set of the most important points from the intermediate responses are aggregated and used as the context to generate the final response.

The quality of the global search\u2019s response can be heavily influenced by the level of the community hierarchy chosen for sourcing community reports. Lower hierarchy levels, with their detailed reports, tend to yield more thorough responses, but may also increase the time and LLM resources needed to generate the final response due to the volume of reports.

"}, {"location": "query/global_search/#configuration", "title": "Configuration", "text": "

Below are the key parameters of the GlobalSearch class:

"}, {"location": "query/global_search/#how-to-use", "title": "How to Use", "text": "

An example of a global search scenario can be found in the following notebook.

"}, {"location": "query/local_search/", "title": "Local Search \ud83d\udd0e", "text": ""}, {"location": "query/local_search/#entity-based-reasoning", "title": "Entity-based Reasoning", "text": "

The local search method combines structured data from the knowledge graph with unstructured data from the input documents to augment the LLM context with relevant entity information at query time. It is well-suited for answering questions that require an understanding of specific entities mentioned in the input documents (e.g., \u201cWhat are the healing properties of chamomile?\u201d).

"}, {"location": "query/local_search/#methodology", "title": "Methodology", "text": "
---\ntitle: Local Search Dataflow\n---\n%%{ init: { 'flowchart': { 'curve': 'step' } } }%%\nflowchart LR\n\n    uq[User Query] ---.1\n    ch1[Conversation<br/>History]---.1\n\n    .1--Entity<br/>Description<br/>Embedding--> ee[Extracted Entities]\n\n    ee[Extracted Entities] ---.2--Entity-Text<br/>Unit Mapping--> ctu[Candidate<br/>Text Units]--Ranking + <br/>Filtering -->ptu[Prioritized<br/>Text Units]---.3\n    .2--Entity-Report<br/>Mapping--> ccr[Candidate<br/>Community Reports]--Ranking + <br/>Filtering -->pcr[Prioritized<br/>Community Reports]---.3\n    .2--Entity-Entity<br/>Relationships--> ce[Candidate<br/>Entities]--Ranking + <br/>Filtering -->pe[Prioritized<br/>Entities]---.3\n    .2--Entity-Entity<br/>Relationships--> cr[Candidate<br/>Relationships]--Ranking + <br/>Filtering -->pr[Prioritized<br/>Relationships]---.3\n    .2--Entity-Covariate<br/>Mappings--> cc[Candidate<br/>Covariates]--Ranking + <br/>Filtering -->pc[Prioritized<br/>Covariates]---.3\n    ch1 -->ch2[Conversation History]---.3\n    .3-->res[Response]\n\n     classDef green fill:#26B653,stroke:#333,stroke-width:2px,color:#fff;\n     classDef turquoise fill:#19CCD3,stroke:#333,stroke-width:2px,color:#fff;\n     classDef rose fill:#DD8694,stroke:#333,stroke-width:2px,color:#fff;\n     classDef orange fill:#F19914,stroke:#333,stroke-width:2px,color:#fff;\n     classDef purple fill:#B356CD,stroke:#333,stroke-width:2px,color:#fff;\n     classDef invisible fill:#fff,stroke:#fff,stroke-width:0px,color:#fff, width:0px;\n     class uq,ch1 turquoise\n     class ee green\n     class ctu,ccr,ce,cr,cc rose\n     class ptu,pcr,pe,pr,pc,ch2 orange\n     class res purple\n     class .1,.2,.3 invisible\n\n

Given a user query and, optionally, the conversation history, the local search method identifies a set of entities from the knowledge graph that are semantically-related to the user input. These entities serve as access points into the knowledge graph, enabling the extraction of further relevant details such as connected entities, relationships, entity covariates, and community reports. Additionally, it also extracts relevant text chunks from the raw input documents that are associated with the identified entities. These candidate data sources are then prioritized and filtered to fit within a single context window of pre-defined size, which is used to generate a response to the user query.

"}, {"location": "query/local_search/#configuration", "title": "Configuration", "text": "

Below are the key parameters of the LocalSearch class:

"}, {"location": "query/local_search/#how-to-use", "title": "How to Use", "text": "

An example of a local search scenario can be found in the following notebook.

"}, {"location": "query/overview/", "title": "Query Engine \ud83d\udd0e", "text": "

The Query Engine is the retrieval module of the Graph RAG Library. It is one of the two main components of the Graph RAG library, the other being the Indexing Pipeline (see Indexing Pipeline). It is responsible for the following tasks:

"}, {"location": "query/overview/#local-search", "title": "Local Search", "text": "

Local search method generates answers by combining relevant data from the AI-extracted knowledge-graph with text chunks of the raw documents. This method is suitable for questions that require an understanding of specific entities mentioned in the documents (e.g. What are the healing properties of chamomile?).

For more details about how Local Search works please refer to the Local Search documentation.

"}, {"location": "query/overview/#global-search", "title": "Global Search", "text": "

Global search method generates answers by searching over all AI-generated community reports in a map-reduce fashion. This is a resource-intensive method, but often gives good responses for questions that require an understanding of the dataset as a whole (e.g. What are the most significant values of the herbs mentioned in this notebook?).

More about this can be checked at the Global Search documentation.

"}, {"location": "query/overview/#drift-search", "title": "DRIFT Search", "text": "

DRIFT Search introduces a new approach to local search queries by including community information in the search process. This greatly expands the breadth of the query\u2019s starting point and leads to retrieval and usage of a far higher variety of facts in the final answer. This addition expands the GraphRAG query engine by providing a more comprehensive option for local search, which uses community insights to refine a query into detailed follow-up questions.

To learn more about DRIFT Search, please refer to the DRIFT Search documentation.

"}, {"location": "query/overview/#question-generation", "title": "Question Generation", "text": "

This functionality takes a list of user queries and generates the next candidate questions. This is useful for generating follow-up questions in a conversation or for generating a list of questions for the investigator to dive deeper into the dataset.

Information about how question generation works can be found at the Question Generation documentation page.

"}, {"location": "query/question_generation/", "title": "Question Generation \u2754", "text": ""}, {"location": "query/question_generation/#entity-based-question-generation", "title": "Entity-based Question Generation", "text": "

The question generation method combines structured data from the knowledge graph with unstructured data from the input documents to generate candidate questions related to specific entities.

"}, {"location": "query/question_generation/#methodology", "title": "Methodology", "text": "

Given a list of prior user questions, the question generation method uses the same context-building approach employed in local search to extract and prioritize relevant structured and unstructured data, including entities, relationships, covariates, community reports and raw text chunks. These data records are then fitted into a single LLM prompt to generate candidate follow-up questions that represent the most important or urgent information content or themes in the data.

"}, {"location": "query/question_generation/#configuration", "title": "Configuration", "text": "

Below are the key parameters of the Question Generation class:

"}, {"location": "query/question_generation/#how-to-use", "title": "How to Use", "text": "

An example of the question generation function can be found in the following notebook.

"}, {"location": "query/notebooks/overview/", "title": "Query Engine Notebooks", "text": "

For examples about running Query please refer to the following notebooks:

The test dataset for these notebooks can be found in dataset.zip.

"}]} \ No newline at end of file +{"config": {"lang": ["en"], "separator": "[\\s\\-]+", "pipeline": ["stopWordFilter"]}, "docs": [{"location": "", "title": "Welcome to GraphRAG", "text": "

\ud83d\udc49 Microsoft Research Blog Post \ud83d\udc49 GraphRAG Accelerator \ud83d\udc49 GraphRAG Arxiv

Figure 1: An LLM-generated knowledge graph built using GPT-4 Turbo.

GraphRAG is a structured, hierarchical approach to Retrieval Augmented Generation (RAG), as opposed to naive semantic-search approaches using plain text snippets. The GraphRAG process involves extracting a knowledge graph out of raw text, building a community hierarchy, generating summaries for these communities, and then leveraging these structures when perform RAG-based tasks.

To learn more about GraphRAG and how it can be used to enhance your LLMs ability to reason about your private data, please visit the Microsoft Research Blog Post.

"}, {"location": "#solution-accelerator", "title": "Solution Accelerator \ud83d\ude80", "text": "

To quickstart the GraphRAG system we recommend trying the Solution Accelerator package. This provides a user-friendly end-to-end experience with Azure resources.

"}, {"location": "#get-started-with-graphrag", "title": "Get Started with GraphRAG \ud83d\ude80", "text": "

To start using GraphRAG, check out the Get Started guide. For a deeper dive into the main sub-systems, please visit the docpages for the Indexer and Query packages.

"}, {"location": "#graphrag-vs-baseline-rag", "title": "GraphRAG vs Baseline RAG \ud83d\udd0d", "text": "

Retrieval-Augmented Generation (RAG) is a technique to improve LLM outputs using real-world information. This technique is an important part of most LLM-based tools and the majority of RAG approaches use vector similarity as the search technique, which we call Baseline RAG. GraphRAG uses knowledge graphs to provide substantial improvements in question-and-answer performance when reasoning about complex information. RAG techniques have shown promise in helping LLMs to reason about private datasets - data that the LLM is not trained on and has never seen before, such as an enterprise\u2019s proprietary research, business documents, or communications. Baseline RAG was created to help solve this problem, but we observe situations where baseline RAG performs very poorly. For example:

To address this, the tech community is working to develop methods that extend and enhance RAG. Microsoft Research\u2019s new approach, GraphRAG, uses LLMs to create a knowledge graph based on an input corpus. This graph, along with community summaries and graph machine learning outputs, are used to augment prompts at query time. GraphRAG shows substantial improvement in answering the two classes of questions described above, demonstrating intelligence or mastery that outperforms other approaches previously applied to private datasets.

"}, {"location": "#the-graphrag-process", "title": "The GraphRAG Process \ud83e\udd16", "text": "

GraphRAG builds upon our prior research and tooling using graph machine learning. The basic steps of the GraphRAG process are as follows:

"}, {"location": "#index", "title": "Index", "text": ""}, {"location": "#query", "title": "Query", "text": "

At query time, these structures are used to provide materials for the LLM context window when answering a question. The primary query modes are:

"}, {"location": "#prompt-tuning", "title": "Prompt Tuning", "text": "

Using GraphRAG with your data out of the box may not yield the best possible results. We strongly recommend to fine-tune your prompts following the Prompt Tuning Guide in our documentation.

"}, {"location": "blog_posts/", "title": "Microsoft Research Blog", "text": ""}, {"location": "blog_posts/#published-february-13-2024-by-jonathan-larsonhttpswwwmicrosoftcomen-usresearchpeoplejolarso-senior-principal-data-architect-steven-truitthttpswwwmicrosoftcomen-usresearchpeoplesteventruitt-principal-program-manager", "title": "Published February 13, 2024 By [Jonathan Larson](https://www.microsoft.com/en-us/research/people/jolarso/), Senior Principal Data Architect; [Steven Truitt](https://www.microsoft.com/en-us/research/people/steventruitt/), Principal Program Manager", "text": ""}, {"location": "blog_posts/#published-july-2-2024-by-darren-edgehttpswwwmicrosoftcomen-usresearchpeopledaedge-senior-director-ha-trinhhttpswwwmicrosoftcomen-usresearchpeopletrinhha-senior-data-scientist-steven-truitthttpswwwmicrosoftcomen-usresearchpeoplesteventruitt-principal-program-manager-jonathan-larsonhttpswwwmicrosoftcomen-usresearchpeoplejolarso-senior-principal-data-architect", "title": "Published July 2, 2024 By [Darren Edge](https://www.microsoft.com/en-us/research/people/daedge/), Senior Director; [Ha Trinh](https://www.microsoft.com/en-us/research/people/trinhha/), Senior Data Scientist; [Steven Truitt](https://www.microsoft.com/en-us/research/people/steventruitt/), Principal Program Manager; [Jonathan Larson](https://www.microsoft.com/en-us/research/people/jolarso/), Senior Principal Data Architect", "text": ""}, {"location": "blog_posts/#published-september-9-2024-by-alonso-guevara-fernandezhttpswwwmicrosoftcomen-usresearchpeoplealonsog-sr-software-engineer-katy-smith-data-scientist-ii-joshua-bradleyhttpswwwmicrosoftcomen-usresearchpeoplejoshbradley-senior-data-scientist-darren-edgehttpswwwmicrosoftcomen-usresearchpeopledaedge-senior-director-ha-trinhhttpswwwmicrosoftcomen-usresearchpeopletrinhha-senior-data-scientist-sarah-smithhttpswwwmicrosoftcomen-usresearchpeoplesmithsarah-senior-program-manager-ben-cutlerhttpswwwmicrosoftcomen-usresearchpeoplebcutler-senior-director-steven-truitthttpswwwmicrosoftcomen-usresearchpeoplesteventruitt-principal-program-manager-jonathan-larsonhttpswwwmicrosoftcomen-usresearchpeoplejolarso-senior-principal-data-architect-octicons-arrow-right-24-introducing-drift-search-combining-global-and-local-search-methods-to-improve-quality-and-efficiencyhttpswwwmicrosoftcomen-usresearchblogintroducing-drift-search-combining-global-and-local-search-methods-to-improve-quality-and-efficiency-published-october-31-2024-by-julian-whiting-senior-machine-learning-engineer-zachary-hills-senior-software-engineer-alonso-guevara-fernandezhttpswwwmicrosoftcomen-usresearchpeoplealonsog-sr-software-engineer-ha-trinhhttpswwwmicrosoftcomen-usresearchpeopletrinhha-senior-data-scientist-adam-bradley-managing-partner-strategic-research-jonathan-larsonhttpswwwmicrosoftcomen-usresearchpeoplejolarso-senior-principal-data-architect", "title": "Published September 9, 2024 By [Alonso Guevara Fern\u00e1ndez](https://www.microsoft.com/en-us/research/people/alonsog/), Sr. Software Engineer; Katy Smith, Data Scientist II; [Joshua Bradley](https://www.microsoft.com/en-us/research/people/joshbradley/), Senior Data Scientist; [Darren Edge](https://www.microsoft.com/en-us/research/people/daedge/), Senior Director; [Ha Trinh](https://www.microsoft.com/en-us/research/people/trinhha/), Senior Data Scientist; [Sarah Smith](https://www.microsoft.com/en-us/research/people/smithsarah/), Senior Program Manager; [Ben Cutler](https://www.microsoft.com/en-us/research/people/bcutler/), Senior Director; [Steven Truitt](https://www.microsoft.com/en-us/research/people/steventruitt/), Principal Program Manager; [Jonathan Larson](https://www.microsoft.com/en-us/research/people/jolarso/), Senior Principal Data Architect - [:octicons-arrow-right-24: **Introducing DRIFT Search: Combining global and local search methods to improve quality and efficiency**](https://www.microsoft.com/en-us/research/blog/introducing-drift-search-combining-global-and-local-search-methods-to-improve-quality-and-efficiency/) ***", "text": ""}, {"location": "blog_posts/#published-october-31-2024-by-julian-whiting-senior-machine-learning-engineer-zachary-hills-senior-software-engineer-alonso-guevara-fernandezhttpswwwmicrosoftcomen-usresearchpeoplealonsog-sr-software-engineer-ha-trinhhttpswwwmicrosoftcomen-usresearchpeopletrinhha-senior-data-scientist-adam-bradley-managing-partner-strategic-research-jonathan-larsonhttpswwwmicrosoftcomen-usresearchpeoplejolarso-senior-principal-data-architect", "title": "Published October 31, 2024 By Julian Whiting , Senior Machine Learning Engineer; Zachary Hills , Senior Software Engineer; [Alonso Guevara Fern\u00e1ndez](https://www.microsoft.com/en-us/research/people/alonsog/), Sr. Software Engineer; [Ha Trinh](https://www.microsoft.com/en-us/research/people/trinhha/), Senior Data Scientist; Adam Bradley , Managing Partner, Strategic Research; [Jonathan Larson](https://www.microsoft.com/en-us/research/people/jolarso/), Senior Principal Data Architect", "text": ""}, {"location": "cli/", "title": "CLI Reference", "text": "

This page documents the command-line interface of the graphrag library.

"}, {"location": "cli/#graphrag", "title": "graphrag", "text": "

GraphRAG: A graph-based retrieval-augmented generation (RAG) system.

Usage:

 [OPTIONS] COMMAND [ARGS]...\n

Options:

  --install-completion  Install completion for the current shell.\n  --show-completion     Show completion for the current shell, to copy it or\n                        customize the installation.\n
"}, {"location": "cli/#index", "title": "index", "text": "

Build a knowledge graph index.

Usage:

 index [OPTIONS]\n

Options:

  --config PATH                   The configuration to use.\n  --root PATH                     The project root directory.  [default: .]\n  --verbose / --no-verbose        Run the indexing pipeline with verbose\n                                  logging  [default: no-verbose]\n  --memprofile / --no-memprofile  Run the indexing pipeline with memory\n                                  profiling  [default: no-memprofile]\n  --resume TEXT                   Resume a given indexing run\n  --reporter [rich|print|none]    The progress reporter to use.  [default:\n                                  rich]\n  --emit TEXT                     The data formats to emit, comma-separated.\n                                  [default: parquet]\n  --dry-run / --no-dry-run        Run the indexing pipeline without executing\n                                  any steps to inspect and validate the\n                                  configuration.  [default: no-dry-run]\n  --cache / --no-cache            Use LLM cache.  [default: cache]\n  --skip-validation / --no-skip-validation\n                                  Skip any preflight validation. Useful when\n                                  running no LLM steps.  [default: no-skip-\n                                  validation]\n  --output PATH                   Indexing pipeline output directory.\n                                  Overrides storage.base_dir in the\n                                  configuration file.\n
"}, {"location": "cli/#init", "title": "init", "text": "

Generate a default configuration file.

Usage:

 init [OPTIONS]\n

Options:

  --root PATH  The project root directory.  [required]\n
"}, {"location": "cli/#prompt-tune", "title": "prompt-tune", "text": "

Generate custom graphrag prompts with your own data (i.e. auto templating).

Usage:

 prompt-tune [OPTIONS]\n

Options:

  --root PATH                     The project root directory.  [default: .]\n  --config PATH                   The configuration to use.\n  --domain TEXT                   The domain your input data is related to.\n                                  For example 'space science', 'microbiology',\n                                  'environmental news'. If not defined, a\n                                  domain will be inferred from the input data.\n  --selection-method [all|random|top|auto]\n                                  The text chunk selection method.  [default:\n                                  random]\n  --n-subset-max INTEGER          The number of text chunks to embed when\n                                  --selection-method=auto.  [default: 300]\n  --k INTEGER                     The maximum number of documents to select\n                                  from each centroid when --selection-\n                                  method=auto.  [default: 15]\n  --limit INTEGER                 The number of documents to load when\n                                  --selection-method={random,top}.  [default:\n                                  15]\n  --max-tokens INTEGER            The max token count for prompt generation.\n                                  [default: 2000]\n  --min-examples-required INTEGER\n                                  The minimum number of examples to\n                                  generate/include in the entity extraction\n                                  prompt.  [default: 2]\n  --chunk-size INTEGER            The max token count for prompt generation.\n                                  [default: 200]\n  --language TEXT                 The primary language used for inputs and\n                                  outputs in graphrag prompts.\n  --discover-entity-types / --no-discover-entity-types\n                                  Discover and extract unspecified entity\n                                  types.  [default: discover-entity-types]\n  --output PATH                   The directory to save prompts to, relative\n                                  to the project root directory.  [default:\n                                  prompts]\n
"}, {"location": "cli/#query", "title": "query", "text": "

Query a knowledge graph index.

Usage:

 query [OPTIONS]\n

Options:

  --method [local|global|drift]  The query algorithm to use.  [required]\n  --query TEXT                   The query to execute.  [required]\n  --config PATH                  The configuration to use.\n  --data PATH                    Indexing pipeline output directory (i.e.\n                                 contains the parquet files).\n  --root PATH                    The project root directory.  [default: .]\n  --community-level INTEGER      The community level in the Leiden community\n                                 hierarchy from which to load community\n                                 reports. Higher values represent reports from\n                                 smaller communities.  [default: 2]\n  --response-type TEXT           Free form text describing the response type\n                                 and format, can be anything, e.g. Multiple\n                                 Paragraphs, Single Paragraph, Single\n                                 Sentence, List of 3-7 Points, Single Page,\n                                 Multi-Page Report. Default: Multiple\n                                 Paragraphs  [default: Multiple Paragraphs]\n  --streaming / --no-streaming   Print response in a streaming manner.\n                                 [default: no-streaming]\n
"}, {"location": "developing/", "title": "Development Guide", "text": ""}, {"location": "developing/#requirements", "title": "Requirements", "text": "Name Installation Purpose Python 3.10-3.12 Download The library is Python-based. Poetry Instructions Poetry is used for package management and virtualenv management in Python codebases"}, {"location": "developing/#getting-started", "title": "Getting Started", "text": ""}, {"location": "developing/#install-dependencies", "title": "Install Dependencies", "text": "
# Install Python dependencies.\npoetry install\n
"}, {"location": "developing/#execute-the-indexing-engine", "title": "Execute the Indexing Engine", "text": "
poetry run poe index <...args>\n
"}, {"location": "developing/#executing-queries", "title": "Executing Queries", "text": "
poetry run poe query <...args>\n
"}, {"location": "developing/#azurite", "title": "Azurite", "text": "

Some unit and smoke tests use Azurite to emulate Azure resources. This can be started by running:

./scripts/start-azurite.sh\n

or by simply running azurite in the terminal if already installed globally. See the Azurite documentation for more information about how to install and use Azurite.

"}, {"location": "developing/#lifecycle-scripts", "title": "Lifecycle Scripts", "text": "

Our Python package utilizes Poetry to manage dependencies and poethepoet to manage build scripts.

Available scripts are:

"}, {"location": "developing/#troubleshooting", "title": "Troubleshooting", "text": ""}, {"location": "developing/#runtimeerror-llvm-config-failed-executing-please-point-llvm_config-to-the-path-for-llvm-config-when-running-poetry-install", "title": "\"RuntimeError: llvm-config failed executing, please point LLVM_CONFIG to the path for llvm-config\" when running poetry install", "text": "

Make sure llvm-9 and llvm-9-dev are installed:

sudo apt-get install llvm-9 llvm-9-dev

and then in your bashrc, add

export LLVM_CONFIG=/usr/bin/llvm-config-9

"}, {"location": "developing/#numba_pymoduleh610-fatal-error-pythonh-no-such-file-or-directory-when-running-poetry-install", "title": "\"numba/_pymodule.h:6:10: fatal error: Python.h: No such file or directory\" when running poetry install", "text": "

Make sure you have python3.10-dev installed or more generally python<version>-dev

sudo apt-get install python3.10-dev

"}, {"location": "developing/#llm-call-constantly-exceeds-tpm-rpm-or-time-limits", "title": "LLM call constantly exceeds TPM, RPM or time limits", "text": "

GRAPHRAG_LLM_THREAD_COUNT and GRAPHRAG_EMBEDDING_THREAD_COUNT are both set to 50 by default. You can modify this values to reduce concurrency. Please refer to the Configuration Documents

"}, {"location": "get_started/", "title": "Getting Started", "text": ""}, {"location": "get_started/#requirements", "title": "Requirements", "text": "

Python 3.10-3.12

To get started with the GraphRAG system, you have a few options:

\ud83d\udc49 Use the GraphRAG Accelerator solution \ud83d\udc49 Install from pypi. \ud83d\udc49 Use it from source

"}, {"location": "get_started/#quickstart", "title": "Quickstart", "text": "

To get started with the GraphRAG system we recommend trying the Solution Accelerator package. This provides a user-friendly end-to-end experience with Azure resources.

"}, {"location": "get_started/#top-level-modules", "title": "Top-Level Modules", "text": ""}, {"location": "get_started/#overview", "title": "Overview", "text": "

The following is a simple end-to-end example for using the GraphRAG system. It shows how to use the system to index some text, and then use the indexed data to answer questions about the documents.

"}, {"location": "get_started/#install-graphrag", "title": "Install GraphRAG", "text": "
pip install graphrag\n

The graphrag library includes a CLI for a no-code approach to getting started. Please review the full CLI documentation for further detail.

"}, {"location": "get_started/#running-the-indexer", "title": "Running the Indexer", "text": "

Now we need to set up a data project and some initial configuration. Let's set that up. We're using the default configuration mode, which you can customize as needed using a config file, which we recommend, or environment variables.

First let's get a sample dataset ready:

mkdir -p ./ragtest/input\n

Now let's get a copy of A Christmas Carol by Charles Dickens from a trusted source

curl https://www.gutenberg.org/cache/epub/24022/pg24022.txt -o ./ragtest/input/book.txt\n

Next we'll inject some required config variables:

"}, {"location": "get_started/#set-up-your-workspace-variables", "title": "Set Up Your Workspace Variables", "text": "

First let's make sure to setup the required environment variables. For details on these environment variables, and what environment variables are available, see the variables documentation.

To initialize your workspace, first run the graphrag init command. Since we have already configured a directory named ./ragtest in the previous step, run the following command:

graphrag init --root ./ragtest\n

This will create two files: .env and settings.yaml in the ./ragtest directory.

"}, {"location": "get_started/#openai-and-azure-openai", "title": "OpenAI and Azure OpenAI", "text": "

If running in OpenAI mode, update the value of GRAPHRAG_API_KEY in the .env file with your OpenAI API key.

"}, {"location": "get_started/#azure-openai", "title": "Azure OpenAI", "text": "

In addition, Azure OpenAI users should set the following variables in the settings.yaml file. To find the appropriate sections, just search for the llm: configuration, you should see two sections, one for the chat endpoint and one for the embeddings endpoint. Here is an example of how to configure the chat endpoint:

type: azure_openai_chat # Or azure_openai_embedding for embeddings\napi_base: https://<instance>.openai.azure.com\napi_version: 2024-02-15-preview # You can customize this for other versions\ndeployment_name: <azure_model_deployment_name>\n
"}, {"location": "get_started/#running-the-indexing-pipeline", "title": "Running the Indexing pipeline", "text": "

Finally we'll run the pipeline!

graphrag index --root ./ragtest\n

This process will take some time to run. This depends on the size of your input data, what model you're using, and the text chunk size being used (these can be configured in your settings.yml file). Once the pipeline is complete, you should see a new folder called ./ragtest/output with a series of parquet files.

"}, {"location": "get_started/#using-the-query-engine", "title": "Using the Query Engine", "text": ""}, {"location": "get_started/#running-the-query-engine", "title": "Running the Query Engine", "text": "

Now let's ask some questions using this dataset.

Here is an example using Global search to ask a high-level question:

graphrag query \\\n--root ./ragtest \\\n--method global \\\n--query \"What are the top themes in this story?\"\n

Here is an example using Local search to ask a more specific question about a particular character:

graphrag query \\\n--root ./ragtest \\\n--method local \\\n--query \"Who is Scrooge and what are his main relationships?\"\n

Please refer to Query Engine docs for detailed information about how to leverage our Local and Global search mechanisms for extracting meaningful insights from data after the Indexer has wrapped up execution.

"}, {"location": "config/custom/", "title": "Fully Custom Config", "text": "

The primary configuration sections for Indexing Engine pipelines are described below. Each configuration section can be expressed in Python (for use in Python API mode) as well as YAML, but YAML is show here for brevity.

Using custom configuration is an advanced use-case. Most users will want to use the Default Configuration instead.

"}, {"location": "config/custom/#indexing-engine-examples", "title": "Indexing Engine Examples", "text": "

The examples directory contains several examples of how to use the indexing engine with custom configuration.

Most examples include two different forms of running the pipeline, both are contained in the examples run.py

  1. Using mostly the Python API
  2. Using mostly the a pipeline configuration file

To run an example:

For example to run the single_verb example, you would run the following commands:

poetry shell\n
PYTHONPATH=\"$(pwd)\" python examples/single_verb/run.py\n
"}, {"location": "config/custom/#configuration-sections", "title": "Configuration Sections", "text": ""}, {"location": "config/custom/#extends", "title": "> extends", "text": "

This configuration allows you to extend a base configuration file or files.

# single base\nextends: ../base_config.yml\n
# multiple bases\nextends:\n  - ../base_config.yml\n  - ../base_config2.yml\n
"}, {"location": "config/custom/#root_dir", "title": "> root_dir", "text": "

This configuration allows you to set the root directory for the pipeline. All data inputs and outputs are assumed to be relative to this path.

root_dir: /workspace/data_project\n
"}, {"location": "config/custom/#storage", "title": "> storage", "text": "

This configuration allows you define the output strategy for the pipeline.

"}, {"location": "config/custom/#cache", "title": "> cache", "text": "

This configuration allows you define the cache strategy for the pipeline.

"}, {"location": "config/custom/#reporting", "title": "> reporting", "text": "

This configuration allows you define the reporting strategy for the pipeline. Report files are generated artifacts that summarize the performance metrics of the pipeline and emit any error messages.

"}, {"location": "config/custom/#workflows", "title": "> workflows", "text": "

This configuration section defines the workflow DAG for the pipeline. Here we define an array of workflows and express their inter-dependencies in steps:

workflows:\n  - name: workflow1\n    steps:\n      - verb: derive\n        args:\n          column1: \"col1\"\n          column2: \"col2\"\n  - name: workflow2\n    steps:\n      - verb: derive\n        args:\n          column1: \"col1\"\n          column2: \"col2\"\n        input:\n          # dependency established here\n          source: workflow:workflow1\n
"}, {"location": "config/custom/#input", "title": "> input", "text": " 
input:\n  type: file\n  file_type: csv\n  base_dir: ../data/csv # the directory containing the CSV files, this is relative to the config file\n  file_pattern: '.*[\\/](?P<source>[^\\/]+)[\\/](?P<year>\\d{4})-(?P<month>\\d{2})-(?P<day>\\d{2})_(?P<author>[^_]+)_\\d+\\.csv$' # a regex to match the CSV files\n  # An additional file filter which uses the named groups from the file_pattern to further filter the files\n  # file_filter:\n  #   # source: (source_filter)\n  #   year: (2023)\n  #   month: (06)\n  #   # day: (22)\n  source_column: \"author\" # the column containing the source/author of the data\n  text_column: \"message\" # the column containing the text of the data\n  timestamp_column: \"date(yyyyMMddHHmmss)\" # optional, the column containing the timestamp of the data\n  timestamp_format: \"%Y%m%d%H%M%S\" # optional,  the format of the timestamp\n  post_process: # Optional, set of steps to process the data before going into the workflow\n    - verb: filter\n      args:\n        column: \"title\",\n        value: \"My document\"\n
input:\n  type: file\n  file_type: csv\n  base_dir: ../data/csv # the directory containing the CSV files, this is relative to the config file\n  file_pattern: '.*[\\/](?P<source>[^\\/]+)[\\/](?P<year>\\d{4})-(?P<month>\\d{2})-(?P<day>\\d{2})_(?P<author>[^_]+)_\\d+\\.csv$' # a regex to match the CSV files\n  # An additional file filter which uses the named groups from the file_pattern to further filter the files\n  # file_filter:\n  #   # source: (source_filter)\n  #   year: (2023)\n  #   month: (06)\n  #   # day: (22)\n  post_process: # Optional, set of steps to process the data before going into the workflow\n    - verb: filter\n      args:\n        column: \"title\",\n        value: \"My document\"\n
"}, {"location": "config/env_vars/", "title": "Default Configuration Mode (using Env Vars)", "text": ""}, {"location": "config/env_vars/#text-embeddings-customization", "title": "Text-Embeddings Customization", "text": "

By default, the GraphRAG indexer will only emit embeddings required for our query methods. However, the model has embeddings defined for all plaintext fields, and these can be generated by setting the GRAPHRAG_EMBEDDING_TARGET environment variable to all.

If the embedding target is all, and you want to only embed a subset of these fields, you may specify which embeddings to skip using the GRAPHRAG_EMBEDDING_SKIP argument described below.

"}, {"location": "config/env_vars/#embedded-fields", "title": "Embedded Fields", "text": ""}, {"location": "config/env_vars/#input-data", "title": "Input Data", "text": "

Our pipeline can ingest .csv or .txt data from an input folder. These files can be nested within subfolders. To configure how input data is handled, what fields are mapped over, and how timestamps are parsed, look for configuration values starting with GRAPHRAG_INPUT_ below. In general, CSV-based data provides the most customizability. Each CSV should at least contain a text field (which can be mapped with environment variables), but it's helpful if they also have title, timestamp, and source fields. Additional fields can be included as well, which will land as extra fields on the Document table.

"}, {"location": "config/env_vars/#base-llm-settings", "title": "Base LLM Settings", "text": "

These are the primary settings for configuring LLM connectivity.

Parameter Required? Description Type Default Value GRAPHRAG_API_KEY Yes for OpenAI. Optional for AOAI The API key. (Note: OPENAI_API_KEY is also used as a fallback). If not defined when using AOAI, managed identity will be used. |str|None` GRAPHRAG_API_BASE For AOAI The API Base URL str None GRAPHRAG_API_VERSION For AOAI The AOAI API version. str None GRAPHRAG_API_ORGANIZATION The AOAI organization. str None GRAPHRAG_API_PROXY The AOAI proxy. str None"}, {"location": "config/env_vars/#text-generation-settings", "title": "Text Generation Settings", "text": "

These settings control the text generation model used by the pipeline. Any settings with a fallback will use the base LLM settings, if available.

Parameter Required? Description Type Default Value GRAPHRAG_LLM_TYPE For AOAI The LLM operation type. Either openai_chat or azure_openai_chat str openai_chat GRAPHRAG_LLM_DEPLOYMENT_NAME For AOAI The AOAI model deployment name. str None GRAPHRAG_LLM_API_KEY Yes (uses fallback) The API key. If not defined when using AOAI, managed identity will be used. str None GRAPHRAG_LLM_API_BASE For AOAI (uses fallback) The API Base URL str None GRAPHRAG_LLM_API_VERSION For AOAI (uses fallback) The AOAI API version. str None GRAPHRAG_LLM_API_ORGANIZATION For AOAI (uses fallback) The AOAI organization. str None GRAPHRAG_LLM_API_PROXY The AOAI proxy. str None GRAPHRAG_LLM_MODEL The LLM model. str gpt-4-turbo-preview GRAPHRAG_LLM_MAX_TOKENS The maximum number of tokens. int 4000 GRAPHRAG_LLM_REQUEST_TIMEOUT The maximum number of seconds to wait for a response from the chat client. int 180 GRAPHRAG_LLM_MODEL_SUPPORTS_JSON Indicates whether the given model supports JSON output mode. True to enable. str None GRAPHRAG_LLM_THREAD_COUNT The number of threads to use for LLM parallelization. int 50 GRAPHRAG_LLM_THREAD_STAGGER The time to wait (in seconds) between starting each thread. float 0.3 GRAPHRAG_LLM_CONCURRENT_REQUESTS The number of concurrent requests to allow for the embedding client. int 25 GRAPHRAG_LLM_TOKENS_PER_MINUTE The number of tokens per minute to allow for the LLM client. 0 = Bypass int 0 GRAPHRAG_LLM_REQUESTS_PER_MINUTE The number of requests per minute to allow for the LLM client. 0 = Bypass int 0 GRAPHRAG_LLM_MAX_RETRIES The maximum number of retries to attempt when a request fails. int 10 GRAPHRAG_LLM_MAX_RETRY_WAIT The maximum number of seconds to wait between retries. int 10 GRAPHRAG_LLM_SLEEP_ON_RATE_LIMIT_RECOMMENDATION Whether to sleep on rate limit recommendation. (Azure Only) bool True GRAPHRAG_LLM_TEMPERATURE The temperature to use generation. float 0 GRAPHRAG_LLM_TOP_P The top_p to use for sampling. float 1 GRAPHRAG_LLM_N The number of responses to generate. int 1"}, {"location": "config/env_vars/#text-embedding-settings", "title": "Text Embedding Settings", "text": "

These settings control the text embedding model used by the pipeline. Any settings with a fallback will use the base LLM settings, if available.

Parameter Required ? Description Type Default GRAPHRAG_EMBEDDING_TYPE For AOAI The embedding client to use. Either openai_embedding or azure_openai_embedding str openai_embedding GRAPHRAG_EMBEDDING_DEPLOYMENT_NAME For AOAI The AOAI deployment name. str None GRAPHRAG_EMBEDDING_API_KEY Yes (uses fallback) The API key to use for the embedding client. If not defined when using AOAI, managed identity will be used. str None GRAPHRAG_EMBEDDING_API_BASE For AOAI (uses fallback) The API base URL. str None GRAPHRAG_EMBEDDING_API_VERSION For AOAI (uses fallback) The AOAI API version to use for the embedding client. str None GRAPHRAG_EMBEDDING_API_ORGANIZATION For AOAI (uses fallback) The AOAI organization to use for the embedding client. str None GRAPHRAG_EMBEDDING_API_PROXY The AOAI proxy to use for the embedding client. str None GRAPHRAG_EMBEDDING_MODEL The model to use for the embedding client. str text-embedding-3-small GRAPHRAG_EMBEDDING_BATCH_SIZE The number of texts to embed at once. (Azure limit is 16) int 16 GRAPHRAG_EMBEDDING_BATCH_MAX_TOKENS The maximum tokens per batch (Azure limit is 8191) int 8191 GRAPHRAG_EMBEDDING_TARGET The target fields to embed. Either required or all. str required GRAPHRAG_EMBEDDING_SKIP A comma-separated list of fields to skip embeddings for . (e.g. 'relationship.description') str None GRAPHRAG_EMBEDDING_THREAD_COUNT The number of threads to use for parallelization for embeddings. int GRAPHRAG_EMBEDDING_THREAD_STAGGER The time to wait (in seconds) between starting each thread for embeddings. float 50 GRAPHRAG_EMBEDDING_CONCURRENT_REQUESTS The number of concurrent requests to allow for the embedding client. int 25 GRAPHRAG_EMBEDDING_TOKENS_PER_MINUTE The number of tokens per minute to allow for the embedding client. 0 = Bypass int 0 GRAPHRAG_EMBEDDING_REQUESTS_PER_MINUTE The number of requests per minute to allow for the embedding client. 0 = Bypass int 0 GRAPHRAG_EMBEDDING_MAX_RETRIES The maximum number of retries to attempt when a request fails. int 10 GRAPHRAG_EMBEDDING_MAX_RETRY_WAIT The maximum number of seconds to wait between retries. int 10 GRAPHRAG_EMBEDDING_SLEEP_ON_RATE_LIMIT_RECOMMENDATION Whether to sleep on rate limit recommendation. (Azure Only) bool True"}, {"location": "config/env_vars/#input-settings", "title": "Input Settings", "text": "

These settings control the data input used by the pipeline. Any settings with a fallback will use the base LLM settings, if available.

"}, {"location": "config/env_vars/#plaintext-input-data-graphrag_input_file_typetext", "title": "Plaintext Input Data (GRAPHRAG_INPUT_FILE_TYPE=text)", "text": "Parameter Description Type Required or Optional Default GRAPHRAG_INPUT_FILE_PATTERN The file pattern regexp to use when reading input files from the input directory. str optional .*\\.txt$"}, {"location": "config/env_vars/#csv-input-data-graphrag_input_file_typecsv", "title": "CSV Input Data (GRAPHRAG_INPUT_FILE_TYPE=csv)", "text": "Parameter Description Type Required or Optional Default GRAPHRAG_INPUT_TYPE The input storage type to use when reading files. (file or blob) str optional file GRAPHRAG_INPUT_FILE_PATTERN The file pattern regexp to use when reading input files from the input directory. str optional .*\\.txt$ GRAPHRAG_INPUT_SOURCE_COLUMN The 'source' column to use when reading CSV input files. str optional source GRAPHRAG_INPUT_TIMESTAMP_COLUMN The 'timestamp' column to use when reading CSV input files. str optional None GRAPHRAG_INPUT_TIMESTAMP_FORMAT The timestamp format to use when parsing timestamps in the timestamp column. str optional None GRAPHRAG_INPUT_TEXT_COLUMN The 'text' column to use when reading CSV input files. str optional text GRAPHRAG_INPUT_DOCUMENT_ATTRIBUTE_COLUMNS A list of CSV columns, comma-separated, to incorporate as document fields. str optional id GRAPHRAG_INPUT_TITLE_COLUMN The 'title' column to use when reading CSV input files. str optional title GRAPHRAG_INPUT_STORAGE_ACCOUNT_BLOB_URL The Azure Storage blob endpoint to use when in blob mode and using managed identity. Will have the format https://<storage_account_name>.blob.core.windows.net str optional None GRAPHRAG_INPUT_CONNECTION_STRING The connection string to use when reading CSV input files from Azure Blob Storage. str optional None GRAPHRAG_INPUT_CONTAINER_NAME The container name to use when reading CSV input files from Azure Blob Storage. str optional None GRAPHRAG_INPUT_BASE_DIR The base directory to read input files from. str optional None"}, {"location": "config/env_vars/#data-mapping-settings", "title": "Data Mapping Settings", "text": "Parameter Description Type Required or Optional Default GRAPHRAG_INPUT_FILE_TYPE The type of input data, csv or text str optional text GRAPHRAG_INPUT_ENCODING The encoding to apply when reading CSV/text input files. str optional utf-8"}, {"location": "config/env_vars/#data-chunking", "title": "Data Chunking", "text": "Parameter Description Type Required or Optional Default GRAPHRAG_CHUNK_SIZE The chunk size in tokens for text-chunk analysis windows. str optional 1200 GRAPHRAG_CHUNK_OVERLAP The chunk overlap in tokens for text-chunk analysis windows. str optional 100 GRAPHRAG_CHUNK_BY_COLUMNS A comma-separated list of document attributes to groupby when performing TextUnit chunking. str optional id GRAPHRAG_CHUNK_ENCODING_MODEL The encoding model to use for chunking. str optional The top-level encoding model."}, {"location": "config/env_vars/#prompting-overrides", "title": "Prompting Overrides", "text": "Parameter Description Type Required or Optional Default GRAPHRAG_ENTITY_EXTRACTION_PROMPT_FILE The path (relative to the root) of an entity extraction prompt template text file. str optional None GRAPHRAG_ENTITY_EXTRACTION_MAX_GLEANINGS The maximum number of redrives (gleanings) to invoke when extracting entities in a loop. int optional 1 GRAPHRAG_ENTITY_EXTRACTION_ENTITY_TYPES A comma-separated list of entity types to extract. str optional organization,person,event,geo GRAPHRAG_ENTITY_EXTRACTION_ENCODING_MODEL The encoding model to use for entity extraction. str optional The top-level encoding model. GRAPHRAG_SUMMARIZE_DESCRIPTIONS_PROMPT_FILE The path (relative to the root) of an description summarization prompt template text file. str optional None GRAPHRAG_SUMMARIZE_DESCRIPTIONS_MAX_LENGTH The maximum number of tokens to generate per description summarization. int optional 500 GRAPHRAG_CLAIM_EXTRACTION_ENABLED Whether claim extraction is enabled for this pipeline. bool optional False GRAPHRAG_CLAIM_EXTRACTION_DESCRIPTION The claim_description prompting argument to utilize. string optional \"Any claims or facts that could be relevant to threat analysis.\" GRAPHRAG_CLAIM_EXTRACTION_PROMPT_FILE The claim extraction prompt to utilize. string optional None GRAPHRAG_CLAIM_EXTRACTION_MAX_GLEANINGS The maximum number of redrives (gleanings) to invoke when extracting claims in a loop. int optional 1 GRAPHRAG_CLAIM_EXTRACTION_ENCODING_MODEL The encoding model to use for claim extraction. str optional The top-level encoding model GRAPHRAG_COMMUNITY_REPORTS_PROMPT_FILE The community reports extraction prompt to utilize. string optional None GRAPHRAG_COMMUNITY_REPORTS_MAX_LENGTH The maximum number of tokens to generate per community reports. int optional 1500"}, {"location": "config/env_vars/#storage", "title": "Storage", "text": "

This section controls the storage mechanism used by the pipeline used for emitting output tables.

Parameter Description Type Required or Optional Default GRAPHRAG_STORAGE_TYPE The type of reporter to use. Options are file, memory, or blob str optional file GRAPHRAG_STORAGE_STORAGE_ACCOUNT_BLOB_URL The Azure Storage blob endpoint to use when in blob mode and using managed identity. Will have the format https://<storage_account_name>.blob.core.windows.net str optional None GRAPHRAG_STORAGE_CONNECTION_STRING The Azure Storage connection string to use when in blob mode. str optional None GRAPHRAG_STORAGE_CONTAINER_NAME The Azure Storage container name to use when in blob mode. str optional None GRAPHRAG_STORAGE_BASE_DIR The base path to data outputs outputs. str optional None"}, {"location": "config/env_vars/#cache", "title": "Cache", "text": "

This section controls the cache mechanism used by the pipeline. This is used to cache LLM invocation results.

Parameter Description Type Required or Optional Default GRAPHRAG_CACHE_TYPE The type of cache to use. Options are file, memory, none or blob str optional file GRAPHRAG_CACHE_STORAGE_ACCOUNT_BLOB_URL The Azure Storage blob endpoint to use when in blob mode and using managed identity. Will have the format https://<storage_account_name>.blob.core.windows.net str optional None GRAPHRAG_CACHE_CONNECTION_STRING The Azure Storage connection string to use when in blob mode. str optional None GRAPHRAG_CACHE_CONTAINER_NAME The Azure Storage container name to use when in blob mode. str optional None GRAPHRAG_CACHE_BASE_DIR The base path to the cache files. str optional None"}, {"location": "config/env_vars/#reporting", "title": "Reporting", "text": "

This section controls the reporting mechanism used by the pipeline, for common events and error messages. The default is to write reports to a file in the output directory. However, you can also choose to write reports to the console or to an Azure Blob Storage container.

Parameter Description Type Required or Optional Default GRAPHRAG_REPORTING_TYPE The type of reporter to use. Options are file, console, or blob str optional file GRAPHRAG_REPORTING_STORAGE_ACCOUNT_BLOB_URL The Azure Storage blob endpoint to use when in blob mode and using managed identity. Will have the format https://<storage_account_name>.blob.core.windows.net str optional None GRAPHRAG_REPORTING_CONNECTION_STRING The Azure Storage connection string to use when in blob mode. str optional None GRAPHRAG_REPORTING_CONTAINER_NAME The Azure Storage container name to use when in blob mode. str optional None GRAPHRAG_REPORTING_BASE_DIR The base path to the reporting outputs. str optional None"}, {"location": "config/env_vars/#node2vec-parameters", "title": "Node2Vec Parameters", "text": "Parameter Description Type Required or Optional Default GRAPHRAG_NODE2VEC_ENABLED Whether to enable Node2Vec bool optional False GRAPHRAG_NODE2VEC_NUM_WALKS The Node2Vec number of walks to perform int optional 10 GRAPHRAG_NODE2VEC_WALK_LENGTH The Node2Vec walk length int optional 40 GRAPHRAG_NODE2VEC_WINDOW_SIZE The Node2Vec window size int optional 2 GRAPHRAG_NODE2VEC_ITERATIONS The number of iterations to run node2vec int optional 3 GRAPHRAG_NODE2VEC_RANDOM_SEED The random seed to use for node2vec int optional 597832"}, {"location": "config/env_vars/#data-snapshotting", "title": "Data Snapshotting", "text": "Parameter Description Type Required or Optional Default GRAPHRAG_SNAPSHOT_EMBEDDINGS Whether to enable embeddings snapshots. bool optional False GRAPHRAG_SNAPSHOT_GRAPHML Whether to enable GraphML snapshots. bool optional False GRAPHRAG_SNAPSHOT_RAW_ENTITIES Whether to enable raw entity snapshots. bool optional False GRAPHRAG_SNAPSHOT_TOP_LEVEL_NODES Whether to enable top-level node snapshots. bool optional False GRAPHRAG_SNAPSHOT_TRANSIENT Whether to enable transient table snapshots. bool optional False"}, {"location": "config/env_vars/#miscellaneous-settings", "title": "Miscellaneous Settings", "text": "Parameter Description Type Required or Optional Default GRAPHRAG_ASYNC_MODE Which async mode to use. Either asyncio or threaded. str optional asyncio GRAPHRAG_ENCODING_MODEL The text encoding model, used in tiktoken, to encode text. str optional cl100k_base GRAPHRAG_MAX_CLUSTER_SIZE The maximum number of entities to include in a single Leiden cluster. int optional 10 GRAPHRAG_SKIP_WORKFLOWS A comma-separated list of workflow names to skip. str optional None GRAPHRAG_UMAP_ENABLED Whether to enable UMAP layouts bool optional False"}, {"location": "config/init/", "title": "Configuring GraphRAG Indexing", "text": "

To start using GraphRAG, you must generate a configuration file. The init command is the easiest way to get started. It will create a .env and settings.yaml files in the specified directory with the necessary configuration settings. It will also output the default LLM prompts used by GraphRAG.

"}, {"location": "config/init/#usage", "title": "Usage", "text": "
graphrag init [--root PATH]\n
"}, {"location": "config/init/#options", "title": "Options", "text": ""}, {"location": "config/init/#example", "title": "Example", "text": "
graphrag init --root ./ragtest\n
"}, {"location": "config/init/#output", "title": "Output", "text": "

The init command will create the following files in the specified directory:

"}, {"location": "config/init/#next-steps", "title": "Next Steps", "text": "

After initializing your workspace, you can either run the Prompt Tuning command to adapt the prompts to your data or even start running the Indexing Pipeline to index your data. For more information on configuring GraphRAG, see the Configuration documentation.

"}, {"location": "config/json_yaml/", "title": "Default Configuration Mode (using JSON/YAML)", "text": "

The default configuration mode may be configured by using a settings.json or settings.yml file in the data project root. If a .env file is present along with this config file, then it will be loaded, and the environment variables defined therein will be available for token replacements in your configuration document using ${ENV_VAR} syntax.

For example:

# .env\nAPI_KEY=some_api_key\n\n# settings.json\n{\n    \"llm\": {\n        \"api_key\": \"${API_KEY}\"\n    }\n}\n
"}, {"location": "config/json_yaml/#config-sections", "title": "Config Sections", "text": ""}, {"location": "config/json_yaml/#input", "title": "input", "text": ""}, {"location": "config/json_yaml/#fields", "title": "Fields", "text": ""}, {"location": "config/json_yaml/#llm", "title": "llm", "text": "

This is the base LLM configuration section. Other steps may override this configuration with their own LLM configuration.

"}, {"location": "config/json_yaml/#fields_1", "title": "Fields", "text": ""}, {"location": "config/json_yaml/#parallelization", "title": "parallelization", "text": ""}, {"location": "config/json_yaml/#fields_2", "title": "Fields", "text": ""}, {"location": "config/json_yaml/#async_mode", "title": "async_mode", "text": "

asyncio|threaded The async mode to use. Either asyncio or `threaded.

"}, {"location": "config/json_yaml/#embeddings", "title": "embeddings", "text": ""}, {"location": "config/json_yaml/#fields_3", "title": "Fields", "text": ""}, {"location": "config/json_yaml/#chunks", "title": "chunks", "text": ""}, {"location": "config/json_yaml/#fields_4", "title": "Fields", "text": ""}, {"location": "config/json_yaml/#cache", "title": "cache", "text": ""}, {"location": "config/json_yaml/#fields_5", "title": "Fields", "text": ""}, {"location": "config/json_yaml/#storage", "title": "storage", "text": ""}, {"location": "config/json_yaml/#fields_6", "title": "Fields", "text": ""}, {"location": "config/json_yaml/#reporting", "title": "reporting", "text": ""}, {"location": "config/json_yaml/#fields_7", "title": "Fields", "text": ""}, {"location": "config/json_yaml/#entity_extraction", "title": "entity_extraction", "text": ""}, {"location": "config/json_yaml/#fields_8", "title": "Fields", "text": ""}, {"location": "config/json_yaml/#summarize_descriptions", "title": "summarize_descriptions", "text": ""}, {"location": "config/json_yaml/#fields_9", "title": "Fields", "text": ""}, {"location": "config/json_yaml/#claim_extraction", "title": "claim_extraction", "text": ""}, {"location": "config/json_yaml/#fields_10", "title": "Fields", "text": ""}, {"location": "config/json_yaml/#community_reports", "title": "community_reports", "text": ""}, {"location": "config/json_yaml/#fields_11", "title": "Fields", "text": ""}, {"location": "config/json_yaml/#cluster_graph", "title": "cluster_graph", "text": ""}, {"location": "config/json_yaml/#fields_12", "title": "Fields", "text": ""}, {"location": "config/json_yaml/#embed_graph", "title": "embed_graph", "text": ""}, {"location": "config/json_yaml/#fields_13", "title": "Fields", "text": ""}, {"location": "config/json_yaml/#umap", "title": "umap", "text": ""}, {"location": "config/json_yaml/#fields_14", "title": "Fields", "text": ""}, {"location": "config/json_yaml/#snapshots", "title": "snapshots", "text": ""}, {"location": "config/json_yaml/#fields_15", "title": "Fields", "text": ""}, {"location": "config/json_yaml/#encoding_model", "title": "encoding_model", "text": "

str - The text encoding model to use. Default=cl100k_base.

"}, {"location": "config/json_yaml/#skip_workflows", "title": "skip_workflows", "text": "

list[str] - Which workflow names to skip.

"}, {"location": "config/overview/", "title": "Configuring GraphRAG Indexing", "text": "

The GraphRAG system is highly configurable. This page provides an overview of the configuration options available for the GraphRAG indexing engine.

"}, {"location": "config/overview/#default-configuration-mode", "title": "Default Configuration Mode", "text": "

The default configuration mode is the simplest way to get started with the GraphRAG system. It is designed to work out-of-the-box with minimal configuration. The primary configuration sections for the Indexing Engine pipelines are described below. The main ways to set up GraphRAG in Default Configuration mode are via:

"}, {"location": "config/overview/#custom-configuration-mode", "title": "Custom Configuration Mode", "text": "

Custom configuration mode is an advanced use-case. Most users will want to use the Default Configuration instead. The primary configuration sections for Indexing Engine pipelines are described below. Details about how to use custom configuration are available in the Custom Configuration Mode documentation.

"}, {"location": "config/template/", "title": "Configuration Template", "text": "

The following template can be used and stored as a .env in the the directory where you're are pointing the --root parameter on your Indexing Pipeline execution.

For details about how to run the Indexing Pipeline, refer to the Index CLI documentation.

"}, {"location": "config/template/#env-file-template", "title": ".env File Template", "text": "

Required variables are uncommented. All the optional configuration can be turned on or off as needed.

"}, {"location": "config/template/#minimal-configuration", "title": "Minimal Configuration", "text": "
# Base LLM Settings\nGRAPHRAG_API_KEY=\"your_api_key\"\nGRAPHRAG_API_BASE=\"http://<domain>.openai.azure.com\" # For Azure OpenAI Users\nGRAPHRAG_API_VERSION=\"api_version\" # For Azure OpenAI Users\n\n# Text Generation Settings\nGRAPHRAG_LLM_TYPE=\"azure_openai_chat\" # or openai_chat\nGRAPHRAG_LLM_DEPLOYMENT_NAME=\"gpt-4-turbo-preview\"\nGRAPHRAG_LLM_MODEL_SUPPORTS_JSON=True\n\n# Text Embedding Settings\nGRAPHRAG_EMBEDDING_TYPE=\"azure_openai_embedding\" # or openai_embedding\nGRAPHRAG_LLM_DEPLOYMENT_NAME=\"text-embedding-3-small\"\n\n# Data Mapping Settings\nGRAPHRAG_INPUT_TYPE=\"text\"\n
"}, {"location": "config/template/#full-configuration", "title": "Full Configuration", "text": "
# Required LLM Config\n\n# Input Data Configuration\nGRAPHRAG_INPUT_TYPE=\"file\"\n\n# Plaintext Input Data Configuration\n# GRAPHRAG_INPUT_FILE_PATTERN=.*\\.txt\n\n# Text Input Data Configuration\nGRAPHRAG_INPUT_FILE_TYPE=\"text\"\nGRAPHRAG_INPUT_FILE_PATTERN=\".*\\.txt$\"\nGRAPHRAG_INPUT_SOURCE_COLUMN=source\n# GRAPHRAG_INPUT_TIMESTAMP_COLUMN=None\n# GRAPHRAG_INPUT_TIMESTAMP_FORMAT=None\n# GRAPHRAG_INPUT_TEXT_COLUMN=\"text\"\n# GRAPHRAG_INPUT_ATTRIBUTE_COLUMNS=id\n# GRAPHRAG_INPUT_TITLE_COLUMN=\"title\"\n# GRAPHRAG_INPUT_TYPE=\"file\"\n# GRAPHRAG_INPUT_CONNECTION_STRING=None\n# GRAPHRAG_INPUT_CONTAINER_NAME=None\n# GRAPHRAG_INPUT_BASE_DIR=None\n\n# Base LLM Settings\nGRAPHRAG_API_KEY=\"your_api_key\"\nGRAPHRAG_API_BASE=\"http://<domain>.openai.azure.com\" # For Azure OpenAI Users\nGRAPHRAG_API_VERSION=\"api_version\" # For Azure OpenAI Users\n# GRAPHRAG_API_ORGANIZATION=None\n# GRAPHRAG_API_PROXY=None\n\n# Text Generation Settings\n# GRAPHRAG_LLM_TYPE=openai_chat\nGRAPHRAG_LLM_API_KEY=\"your_api_key\" # If GRAPHRAG_API_KEY is not set\nGRAPHRAG_LLM_API_BASE=\"http://<domain>.openai.azure.com\" # For Azure OpenAI Users and if GRAPHRAG_API_BASE is not set\nGRAPHRAG_LLM_API_VERSION=\"api_version\" # For Azure OpenAI Users and if GRAPHRAG_API_VERSION is not set\nGRAPHRAG_LLM_MODEL_SUPPORTS_JSON=True # Suggested by default\n# GRAPHRAG_LLM_API_ORGANIZATION=None\n# GRAPHRAG_LLM_API_PROXY=None\n# GRAPHRAG_LLM_DEPLOYMENT_NAME=None\n# GRAPHRAG_LLM_MODEL=gpt-4-turbo-preview\n# GRAPHRAG_LLM_MAX_TOKENS=4000\n# GRAPHRAG_LLM_REQUEST_TIMEOUT=180\n# GRAPHRAG_LLM_THREAD_COUNT=50\n# GRAPHRAG_LLM_THREAD_STAGGER=0.3\n# GRAPHRAG_LLM_CONCURRENT_REQUESTS=25\n# GRAPHRAG_LLM_TPM=0\n# GRAPHRAG_LLM_RPM=0\n# GRAPHRAG_LLM_MAX_RETRIES=10\n# GRAPHRAG_LLM_MAX_RETRY_WAIT=10\n# GRAPHRAG_LLM_SLEEP_ON_RATE_LIMIT_RECOMMENDATION=True\n\n# Text Embedding Settings\n# GRAPHRAG_EMBEDDING_TYPE=openai_embedding\nGRAPHRAG_EMBEDDING_API_KEY=\"your_api_key\" # If GRAPHRAG_API_KEY is not set\nGRAPHRAG_EMBEDDING_API_BASE=\"http://<domain>.openai.azure.com\"  # For Azure OpenAI Users and if GRAPHRAG_API_BASE is not set\nGRAPHRAG_EMBEDDING_API_VERSION=\"api_version\" # For Azure OpenAI Users and if GRAPHRAG_API_VERSION is not set\n# GRAPHRAG_EMBEDDING_API_ORGANIZATION=None\n# GRAPHRAG_EMBEDDING_API_PROXY=None\n# GRAPHRAG_EMBEDDING_DEPLOYMENT_NAME=None\n# GRAPHRAG_EMBEDDING_MODEL=text-embedding-3-small\n# GRAPHRAG_EMBEDDING_BATCH_SIZE=16\n# GRAPHRAG_EMBEDDING_BATCH_MAX_TOKENS=8191\n# GRAPHRAG_EMBEDDING_TARGET=required\n# GRAPHRAG_EMBEDDING_SKIP=None\n# GRAPHRAG_EMBEDDING_THREAD_COUNT=None\n# GRAPHRAG_EMBEDDING_THREAD_STAGGER=50\n# GRAPHRAG_EMBEDDING_CONCURRENT_REQUESTS=25\n# GRAPHRAG_EMBEDDING_TPM=0\n# GRAPHRAG_EMBEDDING_RPM=0\n# GRAPHRAG_EMBEDDING_MAX_RETRIES=10\n# GRAPHRAG_EMBEDDING_MAX_RETRY_WAIT=10\n# GRAPHRAG_EMBEDDING_SLEEP_ON_RATE_LIMIT_RECOMMENDATION=True\n\n# Data Mapping Settings\n# GRAPHRAG_INPUT_ENCODING=utf-8\n\n# Data Chunking\n# GRAPHRAG_CHUNK_SIZE=1200\n# GRAPHRAG_CHUNK_OVERLAP=100\n# GRAPHRAG_CHUNK_BY_COLUMNS=id\n\n# Prompting Overrides\n# GRAPHRAG_ENTITY_EXTRACTION_PROMPT_FILE=None\n# GRAPHRAG_ENTITY_EXTRACTION_MAX_GLEANINGS=1\n# GRAPHRAG_ENTITY_EXTRACTION_ENTITY_TYPES=organization,person,event,geo\n# GRAPHRAG_SUMMARIZE_DESCRIPTIONS_PROMPT_FILE=None\n# GRAPHRAG_SUMMARIZE_DESCRIPTIONS_MAX_LENGTH=500\n# GRAPHRAG_CLAIM_EXTRACTION_DESCRIPTION=\"Any claims or facts that could be relevant to threat analysis.\"\n# GRAPHRAG_CLAIM_EXTRACTION_PROMPT_FILE=None\n# GRAPHRAG_CLAIM_EXTRACTION_MAX_GLEANINGS=1\n# GRAPHRAG_COMMUNITY_REPORT_PROMPT_FILE=None\n# GRAPHRAG_COMMUNITY_REPORT_MAX_LENGTH=1500\n\n# Storage\n# GRAPHRAG_STORAGE_TYPE=file\n# GRAPHRAG_STORAGE_CONNECTION_STRING=None\n# GRAPHRAG_STORAGE_CONTAINER_NAME=None\n# GRAPHRAG_STORAGE_BASE_DIR=None\n\n# Cache\n# GRAPHRAG_CACHE_TYPE=file\n# GRAPHRAG_CACHE_CONNECTION_STRING=None\n# GRAPHRAG_CACHE_CONTAINER_NAME=None\n# GRAPHRAG_CACHE_BASE_DIR=None\n\n# Reporting\n# GRAPHRAG_REPORTING_TYPE=file\n# GRAPHRAG_REPORTING_CONNECTION_STRING=None\n# GRAPHRAG_REPORTING_CONTAINER_NAME=None\n# GRAPHRAG_REPORTING_BASE_DIR=None\n\n# Node2Vec Parameters\n# GRAPHRAG_NODE2VEC_ENABLED=False\n# GRAPHRAG_NODE2VEC_NUM_WALKS=10\n# GRAPHRAG_NODE2VEC_WALK_LENGTH=40\n# GRAPHRAG_NODE2VEC_WINDOW_SIZE=2\n# GRAPHRAG_NODE2VEC_ITERATIONS=3\n# GRAPHRAG_NODE2VEC_RANDOM_SEED=597832\n\n# Data Snapshotting\n# GRAPHRAG_SNAPSHOT_GRAPHML=False\n# GRAPHRAG_SNAPSHOT_RAW_ENTITIES=False\n# GRAPHRAG_SNAPSHOT_TOP_LEVEL_NODES=False\n\n# Miscellaneous Settings\n# GRAPHRAG_ASYNC_MODE=asyncio\n# GRAPHRAG_ENCODING_MODEL=cl100k_base\n# GRAPHRAG_MAX_CLUSTER_SIZE=10\n# GRAPHRAG_SKIP_WORKFLOWS=None\n# GRAPHRAG_UMAP_ENABLED=False\n
"}, {"location": "data/operation_dulce/ABOUT/", "title": "About", "text": "

This document (Operation Dulce) is an AI-generated science fiction novella, included here for the purposes of integration testing.

"}, {"location": "index/architecture/", "title": "Indexing Architecture", "text": ""}, {"location": "index/architecture/#key-concepts", "title": "Key Concepts", "text": ""}, {"location": "index/architecture/#knowledge-model", "title": "Knowledge Model", "text": "

In order to support the GraphRAG system, the outputs of the indexing engine (in the Default Configuration Mode) are aligned to a knowledge model we call the GraphRAG Knowledge Model. This model is designed to be an abstraction over the underlying data storage technology, and to provide a common interface for the GraphRAG system to interact with. In normal use-cases the outputs of the GraphRAG Indexer would be loaded into a database system, and the GraphRAG's Query Engine would interact with the database using the knowledge model data-store types.

"}, {"location": "index/architecture/#datashaper-workflows", "title": "DataShaper Workflows", "text": "

GraphRAG's Indexing Pipeline is built on top of our open-source library, DataShaper. DataShaper is a data processing library that allows users to declaratively express data pipelines, schemas, and related assets using well-defined schemas. DataShaper has implementations in JavaScript and Python, and is designed to be extensible to other languages.

One of the core resource types within DataShaper is a Workflow. Workflows are expressed as sequences of steps, which we call verbs. Each step has a verb name and a configuration object. In DataShaper, these verbs model relational concepts such as SELECT, DROP, JOIN, etc.. Each verb transforms an input data table, and that table is passed down the pipeline.

---\ntitle: Sample Workflow\n---\nflowchart LR\n    input[Input Table] --> select[SELECT] --> join[JOIN] --> binarize[BINARIZE] --> output[Output Table]
"}, {"location": "index/architecture/#llm-based-workflow-steps", "title": "LLM-based Workflow Steps", "text": "

GraphRAG's Indexing Pipeline implements a handful of custom verbs on top of the standard, relational verbs that our DataShaper library provides. These verbs give us the ability to augment text documents with rich, structured data using the power of LLMs such as GPT-4. We utilize these verbs in our standard workflow to extract entities, relationships, claims, community structures, and community reports and summaries. This behavior is customizable and can be extended to support many kinds of AI-based data enrichment and extraction tasks.

"}, {"location": "index/architecture/#workflow-graphs", "title": "Workflow Graphs", "text": "

Because of the complexity of our data indexing tasks, we needed to be able to express our data pipeline as series of multiple, interdependent workflows. In the GraphRAG Indexing Pipeline, each workflow may define dependencies on other workflows, effectively forming a directed acyclic graph (DAG) of workflows, which is then used to schedule processing.

---\ntitle: Sample Workflow DAG\n---\nstateDiagram-v2\n    [*] --> Prepare\n    Prepare --> Chunk\n    Chunk --> ExtractGraph\n    Chunk --> EmbedDocuments\n    ExtractGraph --> GenerateReports\n    ExtractGraph --> EmbedEntities\n    ExtractGraph --> EmbedGraph
"}, {"location": "index/architecture/#dataframe-message-format", "title": "Dataframe Message Format", "text": "

The primary unit of communication between workflows, and between workflow steps is an instance of pandas.DataFrame. Although side-effects are possible, our goal is to be data-centric and table-centric in our approach to data processing. This allows us to easily reason about our data, and to leverage the power of dataframe-based ecosystems. Our underlying dataframe technology may change over time, but our primary goal is to support the DataShaper workflow schema while retaining single-machine ease of use and developer ergonomics.

"}, {"location": "index/architecture/#llm-caching", "title": "LLM Caching", "text": "

The GraphRAG library was designed with LLM interactions in mind, and a common setback when working with LLM APIs is various errors due to network latency, throttling, etc.. Because of these potential error cases, we've added a cache layer around LLM interactions. When completion requests are made using the same input set (prompt and tuning parameters), we return a cached result if one exists. This allows our indexer to be more resilient to network issues, to act idempotently, and to provide a more efficient end-user experience.

"}, {"location": "index/default_dataflow/", "title": "Indexing Dataflow", "text": ""}, {"location": "index/default_dataflow/#the-graphrag-knowledge-model", "title": "The GraphRAG Knowledge Model", "text": "

The knowledge model is a specification for data outputs that conform to our data-model definition. You can find these definitions in the python/graphrag/graphrag/model folder within the GraphRAG repository. The following entity types are provided. The fields here represent the fields that are text-embedded by default.

"}, {"location": "index/default_dataflow/#the-default-configuration-workflow", "title": "The Default Configuration Workflow", "text": "

Let's take a look at how the default-configuration workflow transforms text documents into the GraphRAG Knowledge Model. This page gives a general overview of the major steps in this process. To fully configure this workflow, check out the configuration documentation.

---\ntitle: Dataflow Overview\n---\nflowchart TB\n    subgraph phase1[Phase 1: Compose TextUnits]\n    documents[Documents] --> chunk[Chunk]\n    chunk --> embed[Embed] --> textUnits[Text Units]\n    end\n    subgraph phase2[Phase 2: Graph Extraction]\n    textUnits --> graph_extract[Entity & Relationship Extraction]\n    graph_extract --> graph_summarize[Entity & Relationship Summarization]\n    graph_summarize --> claim_extraction[Claim Extraction]\n    claim_extraction --> graph_outputs[Graph Tables]\n    end\n    subgraph phase3[Phase 3: Graph Augmentation]\n    graph_outputs --> community_detect[Community Detection]\n    community_detect --> graph_embed[Graph Embedding]\n    graph_embed --> augmented_graph[Augmented Graph Tables]\n    end\n    subgraph phase4[Phase 4: Community Summarization]\n    augmented_graph --> summarized_communities[Community Summarization]\n    summarized_communities --> embed_communities[Community Embedding]\n    embed_communities --> community_outputs[Community Tables]\n    end\n    subgraph phase5[Phase 5: Document Processing]\n    documents --> link_to_text_units[Link to TextUnits]\n    textUnits --> link_to_text_units\n    link_to_text_units --> embed_documents[Document Embedding]\n    embed_documents --> document_graph[Document Graph Creation]\n    document_graph --> document_outputs[Document Tables]\n    end\n    subgraph phase6[Phase 6: Network Visualization]\n    document_outputs --> umap_docs[Umap Documents]\n    augmented_graph --> umap_entities[Umap Entities]\n    umap_docs --> combine_nodes[Nodes Table]\n    umap_entities --> combine_nodes\n    end
"}, {"location": "index/default_dataflow/#phase-1-compose-textunits", "title": "Phase 1: Compose TextUnits", "text": "

The first phase of the default-configuration workflow is to transform input documents into TextUnits. A TextUnit is a chunk of text that is used for our graph extraction techniques. They are also used as source-references by extracted knowledge items in order to empower breadcrumbs and provenance by concepts back to their original source tex.

The chunk size (counted in tokens), is user-configurable. By default this is set to 300 tokens, although we've had positive experience with 1200-token chunks using a single \"glean\" step. (A \"glean\" step is a follow-on extraction). Larger chunks result in lower-fidelity output and less meaningful reference texts; however, using larger chunks can result in much faster processing time.

The group-by configuration is also user-configurable. By default, we align our chunks to document boundaries, meaning that there is a strict 1-to-many relationship between Documents and TextUnits. In rare cases, this can be turned into a many-to-many relationship. This is useful when the documents are very short and we need several of them to compose a meaningful analysis unit (e.g. Tweets or a chat log)

Each of these text-units are text-embedded and passed into the next phase of the pipeline.

---\ntitle: Documents into Text Chunks\n---\nflowchart LR\n    doc1[Document 1] --> tu1[TextUnit 1]\n    doc1 --> tu2[TextUnit 2]\n    doc2[Document 2] --> tu3[TextUnit 3]\n    doc2 --> tu4[TextUnit 4]\n
"}, {"location": "index/default_dataflow/#phase-2-graph-extraction", "title": "Phase 2: Graph Extraction", "text": "

In this phase, we analyze each text unit and extract our graph primitives: Entities, Relationships, and Claims. Entities and Relationships are extracted at once in our entity_extract verb, and claims are extracted in our claim_extract verb. Results are then combined and passed into following phases of the pipeline.

---\ntitle: Graph Extraction\n---\nflowchart LR\n    tu[TextUnit] --> ge[Graph Extraction] --> gs[Graph Summarization]\n    tu --> ce[Claim Extraction]
"}, {"location": "index/default_dataflow/#entity-relationship-extraction", "title": "Entity & Relationship Extraction", "text": "

In this first step of graph extraction, we process each text-unit in order to extract entities and relationships out of the raw text using the LLM. The output of this step is a subgraph-per-TextUnit containing a list of entities with a name, type, and description, and a list of relationships with a source, target, and description.

These subgraphs are merged together - any entities with the same name and type are merged by creating an array of their descriptions. Similarly, any relationships with the same source and target are merged by creating an array of their descriptions.

"}, {"location": "index/default_dataflow/#entity-relationship-summarization", "title": "Entity & Relationship Summarization", "text": "

Now that we have a graph of entities and relationships, each with a list of descriptions, we can summarize these lists into a single description per entity and relationship. This is done by asking the LLM for a short summary that captures all of the distinct information from each description. This allows all of our entities and relationships to have a single concise description.

"}, {"location": "index/default_dataflow/#claim-extraction-emission", "title": "Claim Extraction & Emission", "text": "

Finally, as an independent workflow, we extract claims from the source TextUnits. These claims represent positive factual statements with an evaluated status and time-bounds. These are emitted as a primary artifact called Covariates.

Note: claim extraction is optional and turned off by default. This is because claim extraction generally needs prompt tuning to be useful.

"}, {"location": "index/default_dataflow/#phase-3-graph-augmentation", "title": "Phase 3: Graph Augmentation", "text": "

Now that we have a usable graph of entities and relationships, we want to understand their community structure and augment the graph with additional information. This is done in two steps: Community Detection and Graph Embedding. These give us explicit (communities) and implicit (embeddings) ways of understanding the topological structure of our graph.

---\ntitle: Graph Augmentation\n---\nflowchart LR\n    cd[Leiden Hierarchical Community Detection] --> ge[Node2Vec Graph Embedding] --> ag[Graph Table Emission]
"}, {"location": "index/default_dataflow/#community-detection", "title": "Community Detection", "text": "

In this step, we generate a hierarchy of entity communities using the Hierarchical Leiden Algorithm. This method will apply a recursive community-clustering to our graph until we reach a community-size threshold. This will allow us to understand the community structure of our graph and provide a way to navigate and summarize the graph at different levels of granularity.

"}, {"location": "index/default_dataflow/#graph-embedding", "title": "Graph Embedding", "text": "

In this step, we generate a vector representation of our graph using the Node2Vec algorithm. This will allow us to understand the implicit structure of our graph and provide an additional vector-space in which to search for related concepts during our query phase.

"}, {"location": "index/default_dataflow/#graph-tables-emission", "title": "Graph Tables Emission", "text": "

Once our graph augmentation steps are complete, the final Entities and Relationships tables are emitted after their text fields are text-embedded.

"}, {"location": "index/default_dataflow/#phase-4-community-summarization", "title": "Phase 4: Community Summarization", "text": "
---\ntitle: Community Summarization\n---\nflowchart LR\n    sc[Generate Community Reports] --> ss[Summarize Community Reports] --> ce[Community Embedding] --> co[Community Tables Emission]

At this point, we have a functional graph of entities and relationships, a hierarchy of communities for the entities, as well as node2vec embeddings.

Now we want to build on the communities data and generate reports for each community. This gives us a high-level understanding of the graph at several points of graph granularity. For example, if community A is the top-level community, we'll get a report about the entire graph. If the community is lower-level, we'll get a report about a local cluster.

"}, {"location": "index/default_dataflow/#generate-community-reports", "title": "Generate Community Reports", "text": "

In this step, we generate a summary of each community using the LLM. This will allow us to understand the distinct information contained within each community and provide a scoped understanding of the graph, from either a high-level or a low-level perspective. These reports contain an executive overview and reference the key entities, relationships, and claims within the community sub-structure.

"}, {"location": "index/default_dataflow/#summarize-community-reports", "title": "Summarize Community Reports", "text": "

In this step, each community report is then summarized via the LLM for shorthand use.

"}, {"location": "index/default_dataflow/#community-embedding", "title": "Community Embedding", "text": "

In this step, we generate a vector representation of our communities by generating text embeddings of the community report, the community report summary, and the title of the community report.

"}, {"location": "index/default_dataflow/#community-tables-emission", "title": "Community Tables Emission", "text": "

At this point, some bookkeeping work is performed and we emit the Communities and CommunityReports tables.

"}, {"location": "index/default_dataflow/#phase-5-document-processing", "title": "Phase 5: Document Processing", "text": "

In this phase of the workflow, we create the Documents table for the knowledge model.

---\ntitle: Document Processing\n---\nflowchart LR\n    aug[Augment] --> dp[Link to TextUnits] --> de[Avg. Embedding] --> dg[Document Table Emission]
"}, {"location": "index/default_dataflow/#augment-with-columns-csv-only", "title": "Augment with Columns (CSV Only)", "text": "

If the workflow is operating on CSV data, you may configure your workflow to add additional fields to Documents output. These fields should exist on the incoming CSV tables. Details about configuring this can be found in the configuration documentation.

"}, {"location": "index/default_dataflow/#link-to-textunits", "title": "Link to TextUnits", "text": "

In this step, we link each document to the text-units that were created in the first phase. This allows us to understand which documents are related to which text-units and vice-versa.

"}, {"location": "index/default_dataflow/#document-embedding", "title": "Document Embedding", "text": "

In this step, we generate a vector representation of our documents using an average embedding of document slices. We re-chunk documents without overlapping chunks, and then generate an embedding for each chunk. We create an average of these chunks weighted by token-count and use this as the document embedding. This will allow us to understand the implicit relationship between documents, and will help us generate a network representation of our documents.

"}, {"location": "index/default_dataflow/#documents-table-emission", "title": "Documents Table Emission", "text": "

At this point, we can emit the Documents table into the knowledge Model.

"}, {"location": "index/default_dataflow/#phase-6-network-visualization", "title": "Phase 6: Network Visualization", "text": "

In this phase of the workflow, we perform some steps to support network visualization of our high-dimensional vector spaces within our existing graphs. At this point there are two logical graphs at play: the Entity-Relationship graph and the Document graph.

---\ntitle: Network Visualization Workflows\n---\nflowchart LR\n    nv[Umap Documents] --> ne[Umap Entities] --> ng[Nodes Table Emission]

For each of the logical graphs, we perform a UMAP dimensionality reduction to generate a 2D representation of the graph. This will allow us to visualize the graph in a 2D space and understand the relationships between the nodes in the graph. The UMAP embeddings are then emitted as a table of Nodes. The rows of this table include a discriminator indicating whether the node is a document or an entity, and the UMAP coordinates.

"}, {"location": "index/overview/", "title": "GraphRAG Indexing \ud83e\udd16", "text": "

The GraphRAG indexing package is a data pipeline and transformation suite that is designed to extract meaningful, structured data from unstructured text using LLMs.

Indexing Pipelines are configurable. They are composed of workflows, standard and custom steps, prompt templates, and input/output adapters. Our standard pipeline is designed to:

The outputs of the pipeline can be stored in a variety of formats, including JSON and Parquet - or they can be handled manually via the Python API.

"}, {"location": "index/overview/#getting-started", "title": "Getting Started", "text": ""}, {"location": "index/overview/#requirements", "title": "Requirements", "text": "

See the requirements section in Get Started for details on setting up a development environment.

The Indexing Engine can be used in either a default configuration mode or with a custom pipeline. To configure GraphRAG, see the configuration documentation. After you have a config file you can run the pipeline using the CLI or the Python API.

"}, {"location": "index/overview/#usage", "title": "Usage", "text": ""}, {"location": "index/overview/#cli", "title": "CLI", "text": "
# Via Poetry\npoetry run poe cli --root <data_root> # default config mode\npoetry run poe cli --config your_pipeline.yml # custom config mode\n\n# Via Node\nyarn run:index --root <data_root> # default config mode\nyarn run:index --config your_pipeline.yml # custom config mode\n
"}, {"location": "index/overview/#python-api", "title": "Python API", "text": "
from graphrag.index import run_pipeline\nfrom graphrag.index.config import PipelineWorkflowReference\n\nworkflows: list[PipelineWorkflowReference] = [\n    PipelineWorkflowReference(\n        steps=[\n            {\n                # built-in verb\n                \"verb\": \"derive\",  # https://github.com/microsoft/datashaper/blob/main/python/datashaper/datashaper/verbs/derive.py\n                \"args\": {\n                    \"column1\": \"col1\",  # from above\n                    \"column2\": \"col2\",  # from above\n                    \"to\": \"col_multiplied\",  # new column name\n                    \"operator\": \"*\",  # multiply the two columns\n                },\n                # Since we're trying to act on the default input, we don't need explicitly to specify an input\n            }\n        ]\n    ),\n]\n\ndataset = pd.DataFrame([{\"col1\": 2, \"col2\": 4}, {\"col1\": 5, \"col2\": 10}])\noutputs = []\nasync for output in await run_pipeline(dataset=dataset, workflows=workflows):\n    outputs.append(output)\npipeline_result = outputs[-1]\nprint(pipeline_result)\n
"}, {"location": "index/overview/#further-reading", "title": "Further Reading", "text": ""}, {"location": "prompt_tuning/auto_prompt_tuning/", "title": "Auto Prompt Tuning \u2699\ufe0f", "text": "

GraphRAG provides the ability to create domain adapted prompts for the generation of the knowledge graph. This step is optional, though it is highly encouraged to run it as it will yield better results when executing an Index Run.

These are generated by loading the inputs, splitting them into chunks (text units) and then running a series of LLM invocations and template substitutions to generate the final prompts. We suggest using the default values provided by the script, but in this page you'll find the detail of each in case you want to further explore and tweak the prompt tuning algorithm.

Figure 1: Auto Tuning Conceptual Diagram.

"}, {"location": "prompt_tuning/auto_prompt_tuning/#prerequisites", "title": "Prerequisites", "text": "

Before running auto tuning, ensure you have already initialized your workspace with the graphrag init command. This will create the necessary configuration files and the default prompts. Refer to the Init Documentation for more information about the initialization process.

"}, {"location": "prompt_tuning/auto_prompt_tuning/#usage", "title": "Usage", "text": "

You can run the main script from the command line with various options:

graphrag prompt-tune [--root ROOT] [--domain DOMAIN]  [--method METHOD] [--limit LIMIT] [--language LANGUAGE] \\\n[--max-tokens MAX_TOKENS] [--chunk-size CHUNK_SIZE] [--n-subset-max N_SUBSET_MAX] [--k K] \\\n[--min-examples-required MIN_EXAMPLES_REQUIRED] [--no-entity-types] [--output OUTPUT]\n
"}, {"location": "prompt_tuning/auto_prompt_tuning/#command-line-options", "title": "Command-Line Options", "text": ""}, {"location": "prompt_tuning/auto_prompt_tuning/#example-usage", "title": "Example Usage", "text": "
python -m graphrag prompt-tune --root /path/to/project --config /path/to/settings.yaml --domain \"environmental news\" \\\n--method random --limit 10 --language English --max-tokens 2048 --chunk-size 256 --min-examples-required 3 \\\n--no-entity-types --output /path/to/output\n

or, with minimal configuration (suggested):

python -m graphrag prompt-tune --root /path/to/project --config /path/to/settings.yaml --no-entity-types\n
"}, {"location": "prompt_tuning/auto_prompt_tuning/#document-selection-methods", "title": "Document Selection Methods", "text": "

The auto tuning feature ingests the input data and then divides it into text units the size of the chunk size parameter. After that, it uses one of the following selection methods to pick a sample to work with for prompt generation:

"}, {"location": "prompt_tuning/auto_prompt_tuning/#modify-env-vars", "title": "Modify Env Vars", "text": "

After running auto tuning, you should modify the following environment variables (or config variables) to pick up the new prompts on your index run. Note: Please make sure to update the correct path to the generated prompts, in this example we are using the default \"prompts\" path.

or in your yaml config file:

entity_extraction:\n  prompt: \"prompts/entity_extraction.txt\"\n\nsummarize_descriptions:\n  prompt: \"prompts/summarize_descriptions.txt\"\n\ncommunity_reports:\n  prompt: \"prompts/community_report.txt\"\n
"}, {"location": "prompt_tuning/manual_prompt_tuning/", "title": "Manual Prompt Tuning \u2699\ufe0f", "text": "

The GraphRAG indexer, by default, will run with a handful of prompts that are designed to work well in the broad context of knowledge discovery. However, it is quite common to want to tune the prompts to better suit your specific use case. We provide a means for you to do this by allowing you to specify a custom prompt file, which will each use a series of token-replacements internally.

Each of these prompts may be overridden by writing a custom prompt file in plaintext. We use token-replacements in the form of {token_name}, and the descriptions for the available tokens can be found below.

"}, {"location": "prompt_tuning/manual_prompt_tuning/#entityrelationship-extraction", "title": "Entity/Relationship Extraction", "text": "

Prompt Source

"}, {"location": "prompt_tuning/manual_prompt_tuning/#tokens-values-provided-by-extractor", "title": "Tokens (values provided by extractor)", "text": ""}, {"location": "prompt_tuning/manual_prompt_tuning/#summarize-entityrelationship-descriptions", "title": "Summarize Entity/Relationship Descriptions", "text": "

Prompt Source

"}, {"location": "prompt_tuning/manual_prompt_tuning/#tokens-values-provided-by-extractor_1", "title": "Tokens (values provided by extractor)", "text": ""}, {"location": "prompt_tuning/manual_prompt_tuning/#claim-extraction", "title": "Claim Extraction", "text": "

Prompt Source

"}, {"location": "prompt_tuning/manual_prompt_tuning/#tokens-values-provided-by-extractor_2", "title": "Tokens (values provided by extractor)", "text": "

Note: there is additional parameter for the Claim Description that is used in claim extraction. The default value is

\"Any claims or facts that could be relevant to information discovery.\"

See the configuration documentation for details on how to change this.

"}, {"location": "prompt_tuning/manual_prompt_tuning/#generate-community-reports", "title": "Generate Community Reports", "text": "

Prompt Source

"}, {"location": "prompt_tuning/manual_prompt_tuning/#tokens-values-provided-by-extractor_3", "title": "Tokens (values provided by extractor)", "text": ""}, {"location": "prompt_tuning/overview/", "title": "Prompt Tuning \u2699\ufe0f", "text": "

This page provides an overview of the prompt tuning options available for the GraphRAG indexing engine.

"}, {"location": "prompt_tuning/overview/#default-prompts", "title": "Default Prompts", "text": "

The default prompts are the simplest way to get started with the GraphRAG system. It is designed to work out-of-the-box with minimal configuration. You can find more detail about these prompts in the following links:

"}, {"location": "prompt_tuning/overview/#auto-tuning", "title": "Auto Tuning", "text": "

Auto Tuning leverages your input data and LLM interactions to create domain adapted prompts for the generation of the knowledge graph. It is highly encouraged to run it as it will yield better results when executing an Index Run. For more details about how to use it, please refer to the Auto Tuning documentation.

"}, {"location": "prompt_tuning/overview/#manual-tuning", "title": "Manual Tuning", "text": "

Manual tuning is an advanced use-case. Most users will want to use the Auto Tuning feature instead. Details about how to use manual configuration are available in the Manual Tuning documentation.

"}, {"location": "query/drift_search/", "title": "DRIFT Search \ud83d\udd0e", "text": ""}, {"location": "query/drift_search/#combining-local-and-global-search", "title": "Combining Local and Global Search", "text": "

GraphRAG is a technique that uses large language models (LLMs) to create knowledge graphs and summaries from unstructured text documents and leverages them to improve retrieval-augmented generation (RAG) operations on private datasets. It offers comprehensive global overviews of large, private troves of unstructured text documents while also enabling exploration of detailed, localized information. By using LLMs to create comprehensive knowledge graphs that connect and describe entities and relationships contained in those documents, GraphRAG leverages semantic structuring of the data to generate responses to a wide variety of complex user queries.

DRIFT search (Dynamic Reasoning and Inference with Flexible Traversal) builds upon Microsoft\u2019s GraphRAG technique, combining characteristics of both global and local search to generate detailed responses in a method that balances computational costs with quality outcomes using our drift search method.

"}, {"location": "query/drift_search/#methodology", "title": "Methodology", "text": "

Figure 1. An entire DRIFT search hierarchy highlighting the three core phases of the DRIFT search process. A (Primer): DRIFT compares the user\u2019s query with the top K most semantically relevant community reports, generating a broad initial answer and follow-up questions to steer further exploration. B (Follow-Up): DRIFT uses local search to refine queries, producing additional intermediate answers and follow-up questions that enhance specificity, guiding the engine towards context-rich information. A glyph on each node in the diagram shows the confidence the algorithm has to continue the query expansion step. C (Output Hierarchy): The final output is a hierarchical structure of questions and answers ranked by relevance, reflecting a balanced mix of global insights and local refinements, making the results adaptable and comprehensive.

DRIFT Search introduces a new approach to local search queries by including community information in the search process. This greatly expands the breadth of the query\u2019s starting point and leads to retrieval and usage of a far higher variety of facts in the final answer. This addition expands the GraphRAG query engine by providing a more comprehensive option for local search, which uses community insights to refine a query into detailed follow-up questions. ## Configuration Below are the key parameters of the [DRIFTSearch class](https://github.com/microsoft/graphrag/blob/main//graphrag/query/structured_search/drift_search/search.py): - `llm`: OpenAI model object to be used for response generation - `context_builder`: [context builder](https://github.com/microsoft/graphrag/blob/main/graphrag/query/structured_search/drift_search/drift_context.py) object to be used for preparing context data from community reports and query information - `config`: model to define the DRIFT Search hyperparameters. [DRIFT Config model](https://github.com/microsoft/graphrag/blob/main/graphrag/config/models/drift_config.py) - `token_encoder`: token encoder for tracking the budget for the algorithm. - `query_state`: a state object as defined in [Query State](https://github.com/microsoft/graphrag/blob/main/graphrag/query/structured_search/drift_search/state.py) that allows to track execution of a DRIFT Search instance, alongside follow ups and [DRIFT actions](https://github.com/microsoft/graphrag/blob/main/graphrag/query/structured_search/drift_search/action.py). ## How to Use An example of a global search scenario can be found in the following [notebook](../examples_notebooks/drift_search.ipynb). ## Learn More For a more in-depth look at the DRIFT search method, please refer to our [DRIFT Search blog post](https://www.microsoft.com/en-us/research/blog/introducing-drift-search-combining-global-and-local-search-methods-to-improve-quality-and-efficiency/)"}, {"location": "query/global_search/", "title": "Global Search \ud83d\udd0e", "text": ""}, {"location": "query/global_search/#whole-dataset-reasoning", "title": "Whole Dataset Reasoning", "text": "

Baseline RAG struggles with queries that require aggregation of information across the dataset to compose an answer. Queries such as \u201cWhat are the top 5 themes in the data?\u201d perform terribly because baseline RAG relies on a vector search of semantically similar text content within the dataset. There is nothing in the query to direct it to the correct information.

However, with GraphRAG we can answer such questions, because the structure of the LLM-generated knowledge graph tells us about the structure (and thus themes) of the dataset as a whole. This allows the private dataset to be organized into meaningful semantic clusters that are pre-summarized. Using our global search method, the LLM uses these clusters to summarize these themes when responding to a user query.

"}, {"location": "query/global_search/#methodology", "title": "Methodology", "text": "
---\ntitle: Global Search Dataflow\n---\n%%{ init: { 'flowchart': { 'curve': 'step' } } }%%\nflowchart LR\n\n    uq[User Query] --- .1\n    ch1[Conversation History] --- .1\n\n    subgraph RIR\n        direction TB\n        ri1[Rated Intermediate<br/>Response 1]~~~ri2[Rated Intermediate<br/>Response 2] -.\"{1..N}\".-rin[Rated Intermediate<br/>Response N]\n    end\n\n    .1--Shuffled Community<br/>Report Batch 1-->RIR\n    .1--Shuffled Community<br/>Report Batch 2-->RIR---.2\n    .1--Shuffled Community<br/>Report Batch N-->RIR\n\n    .2--Ranking +<br/>Filtering-->agr[Aggregated Intermediate<br/>Responses]-->res[Response]\n\n\n\n     classDef green fill:#26B653,stroke:#333,stroke-width:2px,color:#fff;\n     classDef turquoise fill:#19CCD3,stroke:#333,stroke-width:2px,color:#fff;\n     classDef rose fill:#DD8694,stroke:#333,stroke-width:2px,color:#fff;\n     classDef orange fill:#F19914,stroke:#333,stroke-width:2px,color:#fff;\n     classDef purple fill:#B356CD,stroke:#333,stroke-width:2px,color:#fff;\n     classDef invisible fill:#fff,stroke:#fff,stroke-width:0px,color:#fff, width:0px;\n     class uq,ch1 turquoise;\n     class ri1,ri2,rin rose;\n     class agr orange;\n     class res purple;\n     class .1,.2 invisible;\n

Given a user query and, optionally, the conversation history, the global search method uses a collection of LLM-generated community reports from a specified level of the graph's community hierarchy as context data to generate response in a map-reduce manner. At the map step, community reports are segmented into text chunks of pre-defined size. Each text chunk is then used to produce an intermediate response containing a list of point, each of which is accompanied by a numerical rating indicating the importance of the point. At the reduce step, a filtered set of the most important points from the intermediate responses are aggregated and used as the context to generate the final response.

The quality of the global search\u2019s response can be heavily influenced by the level of the community hierarchy chosen for sourcing community reports. Lower hierarchy levels, with their detailed reports, tend to yield more thorough responses, but may also increase the time and LLM resources needed to generate the final response due to the volume of reports.

"}, {"location": "query/global_search/#configuration", "title": "Configuration", "text": "

Below are the key parameters of the GlobalSearch class:

"}, {"location": "query/global_search/#how-to-use", "title": "How to Use", "text": "

An example of a global search scenario can be found in the following notebook.

"}, {"location": "query/local_search/", "title": "Local Search \ud83d\udd0e", "text": ""}, {"location": "query/local_search/#entity-based-reasoning", "title": "Entity-based Reasoning", "text": "

The local search method combines structured data from the knowledge graph with unstructured data from the input documents to augment the LLM context with relevant entity information at query time. It is well-suited for answering questions that require an understanding of specific entities mentioned in the input documents (e.g., \u201cWhat are the healing properties of chamomile?\u201d).

"}, {"location": "query/local_search/#methodology", "title": "Methodology", "text": "
---\ntitle: Local Search Dataflow\n---\n%%{ init: { 'flowchart': { 'curve': 'step' } } }%%\nflowchart LR\n\n    uq[User Query] ---.1\n    ch1[Conversation<br/>History]---.1\n\n    .1--Entity<br/>Description<br/>Embedding--> ee[Extracted Entities]\n\n    ee[Extracted Entities] ---.2--Entity-Text<br/>Unit Mapping--> ctu[Candidate<br/>Text Units]--Ranking + <br/>Filtering -->ptu[Prioritized<br/>Text Units]---.3\n    .2--Entity-Report<br/>Mapping--> ccr[Candidate<br/>Community Reports]--Ranking + <br/>Filtering -->pcr[Prioritized<br/>Community Reports]---.3\n    .2--Entity-Entity<br/>Relationships--> ce[Candidate<br/>Entities]--Ranking + <br/>Filtering -->pe[Prioritized<br/>Entities]---.3\n    .2--Entity-Entity<br/>Relationships--> cr[Candidate<br/>Relationships]--Ranking + <br/>Filtering -->pr[Prioritized<br/>Relationships]---.3\n    .2--Entity-Covariate<br/>Mappings--> cc[Candidate<br/>Covariates]--Ranking + <br/>Filtering -->pc[Prioritized<br/>Covariates]---.3\n    ch1 -->ch2[Conversation History]---.3\n    .3-->res[Response]\n\n     classDef green fill:#26B653,stroke:#333,stroke-width:2px,color:#fff;\n     classDef turquoise fill:#19CCD3,stroke:#333,stroke-width:2px,color:#fff;\n     classDef rose fill:#DD8694,stroke:#333,stroke-width:2px,color:#fff;\n     classDef orange fill:#F19914,stroke:#333,stroke-width:2px,color:#fff;\n     classDef purple fill:#B356CD,stroke:#333,stroke-width:2px,color:#fff;\n     classDef invisible fill:#fff,stroke:#fff,stroke-width:0px,color:#fff, width:0px;\n     class uq,ch1 turquoise\n     class ee green\n     class ctu,ccr,ce,cr,cc rose\n     class ptu,pcr,pe,pr,pc,ch2 orange\n     class res purple\n     class .1,.2,.3 invisible\n\n

Given a user query and, optionally, the conversation history, the local search method identifies a set of entities from the knowledge graph that are semantically-related to the user input. These entities serve as access points into the knowledge graph, enabling the extraction of further relevant details such as connected entities, relationships, entity covariates, and community reports. Additionally, it also extracts relevant text chunks from the raw input documents that are associated with the identified entities. These candidate data sources are then prioritized and filtered to fit within a single context window of pre-defined size, which is used to generate a response to the user query.

"}, {"location": "query/local_search/#configuration", "title": "Configuration", "text": "

Below are the key parameters of the LocalSearch class:

"}, {"location": "query/local_search/#how-to-use", "title": "How to Use", "text": "

An example of a local search scenario can be found in the following notebook.

"}, {"location": "query/overview/", "title": "Query Engine \ud83d\udd0e", "text": "

The Query Engine is the retrieval module of the Graph RAG Library. It is one of the two main components of the Graph RAG library, the other being the Indexing Pipeline (see Indexing Pipeline). It is responsible for the following tasks:

"}, {"location": "query/overview/#local-search", "title": "Local Search", "text": "

Local search method generates answers by combining relevant data from the AI-extracted knowledge-graph with text chunks of the raw documents. This method is suitable for questions that require an understanding of specific entities mentioned in the documents (e.g. What are the healing properties of chamomile?).

For more details about how Local Search works please refer to the Local Search documentation.

"}, {"location": "query/overview/#global-search", "title": "Global Search", "text": "

Global search method generates answers by searching over all AI-generated community reports in a map-reduce fashion. This is a resource-intensive method, but often gives good responses for questions that require an understanding of the dataset as a whole (e.g. What are the most significant values of the herbs mentioned in this notebook?).

More about this can be checked at the Global Search documentation.

"}, {"location": "query/overview/#drift-search", "title": "DRIFT Search", "text": "

DRIFT Search introduces a new approach to local search queries by including community information in the search process. This greatly expands the breadth of the query\u2019s starting point and leads to retrieval and usage of a far higher variety of facts in the final answer. This addition expands the GraphRAG query engine by providing a more comprehensive option for local search, which uses community insights to refine a query into detailed follow-up questions.

To learn more about DRIFT Search, please refer to the DRIFT Search documentation.

"}, {"location": "query/overview/#question-generation", "title": "Question Generation", "text": "

This functionality takes a list of user queries and generates the next candidate questions. This is useful for generating follow-up questions in a conversation or for generating a list of questions for the investigator to dive deeper into the dataset.

Information about how question generation works can be found at the Question Generation documentation page.

"}, {"location": "query/question_generation/", "title": "Question Generation \u2754", "text": ""}, {"location": "query/question_generation/#entity-based-question-generation", "title": "Entity-based Question Generation", "text": "

The question generation method combines structured data from the knowledge graph with unstructured data from the input documents to generate candidate questions related to specific entities.

"}, {"location": "query/question_generation/#methodology", "title": "Methodology", "text": "

Given a list of prior user questions, the question generation method uses the same context-building approach employed in local search to extract and prioritize relevant structured and unstructured data, including entities, relationships, covariates, community reports and raw text chunks. These data records are then fitted into a single LLM prompt to generate candidate follow-up questions that represent the most important or urgent information content or themes in the data.

"}, {"location": "query/question_generation/#configuration", "title": "Configuration", "text": "

Below are the key parameters of the Question Generation class:

"}, {"location": "query/question_generation/#how-to-use", "title": "How to Use", "text": "

An example of the question generation function can be found in the following notebook.

"}, {"location": "query/notebooks/overview/", "title": "Query Engine Notebooks", "text": "

For examples about running Query please refer to the following notebooks:

The test dataset for these notebooks can be found in dataset.zip.

"}]} \ No newline at end of file