From 7e334c1b310f3a9dc3086bcb1ca0001eedab8f4a Mon Sep 17 00:00:00 2001 From: AlonsoGuevara Date: Thu, 7 Nov 2024 13:02:23 +0000 Subject: [PATCH] =?UTF-8?q?Deploying=20to=20gh-pages=20from=20@=20microsof?= =?UTF-8?q?t/graphrag@20c120288b42135bf0849a2db4c2dd48d09ea345=20?= =?UTF-8?q?=F0=9F=9A=80?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- cli/index.html | 26 + examples_notebooks/drift_search/index.html | 4024 +++++++++---------- examples_notebooks/global_search/index.html | 8 +- examples_notebooks/local_search/index.html | 34 +- search/search_index.json | 2 +- sitemap.xml.gz | Bin 127 -> 127 bytes 6 files changed, 1992 insertions(+), 2102 deletions(-) diff --git a/cli/index.html b/cli/index.html index 1c1e3889..2ad855b4 100644 --- a/cli/index.html +++ b/cli/index.html @@ -1502,6 +1502,32 @@ --streaming / --no-streaming Print response in a streaming manner. [default: no-streaming] +

update

+

Update an existing knowledge graph index.

+

Applies a default storage configuration (if not provided by config), saving the new index to the local file system in the update_output folder.

+

Usage:

+
 update [OPTIONS]
+
+

Options:

+
  --config PATH                   The configuration to use.
+  --root PATH                     The project root directory.  [default: .]
+  --verbose / --no-verbose        Run the indexing pipeline with verbose
+                                  logging  [default: no-verbose]
+  --memprofile / --no-memprofile  Run the indexing pipeline with memory
+                                  profiling  [default: no-memprofile]
+  --reporter [rich|print|none]    The progress reporter to use.  [default:
+                                  rich]
+  --emit TEXT                     The data formats to emit, comma-separated.
+                                  [default: parquet]
+  --cache / --no-cache            Use LLM cache.  [default: cache]
+  --skip-validation / --no-skip-validation
+                                  Skip any preflight validation. Useful when
+                                  running no LLM steps.  [default: no-skip-
+                                  validation]
+  --output PATH                   Indexing pipeline output directory.
+                                  Overrides storage.base_dir in the
+                                  configuration file.
+
diff --git a/examples_notebooks/drift_search/index.html b/examples_notebooks/drift_search/index.html index fbc84dc3..4005bd17 100644 --- a/examples_notebooks/drift_search/index.html +++ b/examples_notebooks/drift_search/index.html @@ -2019,7 +2019,7 @@ Text unit records: 12
@@ -2396,25 +2396,37 @@ search = DRIFTSearch(
+
+
+ + +
+
+ +
@@ -2433,109 +2445,91 @@ search = DRIFTSearch( - - - diff --git a/examples_notebooks/global_search/index.html b/examples_notebooks/global_search/index.html index d9a9b5c5..7615881c 100644 --- a/examples_notebooks/global_search/index.html +++ b/examples_notebooks/global_search/index.html @@ -2564,15 +2564,15 @@ print(result.response)
### Major Conflict
 
-The central conflict in the story revolves around the Paranormal Military Squad's mission to establish contact with extraterrestrial intelligence. This mission involves deciphering alien signals and managing the potential implications of first contact. The conflict is marked by the secrecy and high stakes of the mission, as well as the challenges posed by the unknown nature of the extraterrestrial entities. The squad must navigate these uncertainties and the potential risks associated with communicating with an unknown intelligence [Data: Reports (4, 5, 2, 3, 0)].
+The central conflict in the story revolves around the Paranormal Military Squad's mission to establish contact with extraterrestrial intelligence. This mission involves deciphering alien signals and managing the potential implications of first contact. The conflict is marked by the secrecy and high stakes associated with the mission, as well as the challenges posed by the unknown nature of the extraterrestrial entities [Data: Reports (4, 5, 2, 3, 0)].
 
 ### Protagonists
 
-The protagonists of the story are the key members of the Paranormal Military Squad. This includes Taylor Cruz, Dr. Jordan Hayes, Alex Mercer, and Sam Rivera. Each of these individuals plays a crucial role in the mission, contributing their unique expertise to the squad's efforts in extraterrestrial communication and engagement. Their collaboration and skills are essential to the success of the mission [Data: Reports (4, 5, 2, 3, 0)].
+The protagonists are the key members of the Paranormal Military Squad, including Taylor Cruz, Dr. Jordan Hayes, Alex Mercer, and Sam Rivera. Each of these individuals plays a crucial role in the mission, contributing their expertise in leadership, signal decryption, diplomatic engagement, and technical innovation [Data: Reports (4, 5, 2, 3, 0)].
 
 ### Antagonist
 
-The antagonist in the story is not a single entity or character. Instead, it is represented by the challenges and uncertainties associated with the extraterrestrial signals and the potential risks of first contact. The unknown nature of the extraterrestrial entities and the implications of their communication pose significant obstacles to the squad's mission. These elements create a formidable opposition that the protagonists must overcome to achieve their objectives [Data: Reports (4, 5, 2, 3, 0)].
+The antagonist in the story is not a single entity or character. Instead, it may be considered the unknown and potentially hostile nature of the extraterrestrial entities and the challenges they present to the Paranormal Military Squad's mission [Data: Reports (4, 5, 2, 3, 0)].
@@ -2723,7 +2723,7 @@ print(f"LLM calls: {result.llm_calls}. LLM tokens: {result.prompt_tokens}")
-
LLM calls: 2. LLM tokens: 5288
+
LLM calls: 2. LLM tokens: 5270
diff --git a/examples_notebooks/local_search/index.html b/examples_notebooks/local_search/index.html index c73187ab..27eecef1 100644 --- a/examples_notebooks/local_search/index.html +++ b/examples_notebooks/local_search/index.html @@ -2380,7 +2380,7 @@ entity_df.head()
@@ -3368,21 +3368,21 @@ print(result.response) 
### Overview of Agent Alex Mercer
 
-Agent Alex Mercer is a central figure within the Paranormal Military Squad at Dulce Base, where he plays a pivotal role in overseeing operations and making critical decisions during potential first contact scenarios. His military background and leadership skills are instrumental in guiding the team through complex missions involving extraterrestrial intelligence [Data: Entities (0, 209); Relationships (5, 8, 7)].
+Agent Alex Mercer is a central figure within the Paranormal Military Squad at Dulce Base, where he plays a pivotal role in overseeing operations related to potential extraterrestrial contact. His military background equips him with the leadership skills necessary to guide his team through complex and high-stakes missions. Mercer is deeply involved in the strategic aspects of Operation: Dulce, which focuses on interspecies communication and the decryption of alien signals [Data: Entities (0, 209); Relationships (5, 8, 7)].
 
 ### Role and Responsibilities
 
-Mercer is deeply involved in the analysis and interpretation of alien signals, a task that requires both technical expertise and strategic foresight. He leads efforts in interspecies communication, ensuring that the engagement with extraterrestrial intelligence is handled with care and precision. His responsibilities also include unraveling galactic mysteries and overseeing the team's response to extraterrestrial contact [Data: Entities (0); Relationships (6, 1, 4)].
+As the leader of the Paranormal Military Squad, Mercer is responsible for ensuring a cautious and strategic approach to engaging with extraterrestrial intelligence. His duties include overseeing the team's response to alien communications and unraveling galactic mysteries. Mercer's leadership is characterized by a mix of concern and anticipation, reflecting the gravity of the mission at hand. He collaborates closely with his team, particularly with Dr. Jordan Hayes, to interpret alien signals and develop protocols for responding to them [Data: Entities (0); Relationships (1, 4, 6)].
 
-### Leadership and Team Dynamics
+### Interpersonal Dynamics
 
-As a leader, Mercer is known for his cautious approach and respect for his team's abilities. He collaborates closely with colleagues like Dr. Jordan Hayes, with whom he shares a mutual understanding and recognition of each other's analytical skills. This partnership is crucial for the success of their mission at Dulce Base. Additionally, Mercer acts as a mentor to team members such as Sam Rivera, emphasizing the importance of intuition and trust in their operations [Data: Entities (0); Relationships (1, 2, 5); Claims (8, 9)].
+Mercer is known for his mentorship and guidance within the team, fostering a sense of trust and intuition among his colleagues. His relationship with Sam Rivera, for instance, is built on mutual respect and collaboration, with Mercer providing guidance in their specialized roles. Similarly, his partnership with Jordan Hayes is marked by a shared understanding of the mission's significance and a mutual recognition of each other's analytical skills [Data: Relationships (2, 1, 4); Claims (8, 9)].
 
 ### Challenges and Complexities
 
-Agent Mercer faces several challenges, including the need to balance the strategic and diplomatic aspects of potential extraterrestrial contact. He is involved in the decryption and analysis of alien signals, contributing to the understanding of an extraterrestrial society. Despite his leadership role, there are complexities and reservations about Operation: Dulce, suggesting that Mercer might have unspoken concerns about the mission's implications [Data: Claims (1, 73, 85, 22, 50)].
+Despite his leadership role, Mercer faces challenges and complexities, particularly in navigating the unknowns of extraterrestrial contact. He is involved in the decryption and analysis of alien signals, contributing to the understanding of an extraterrestrial society. Mercer's approach to the mission is both diplomatic and strategic, as he contemplates the transformation of their task from a simple recovery mission to a broader cosmic engagement [Data: Claims (73, 85, 57, 60, 67)].
 
-In summary, Agent Alex Mercer is a key figure in the Paranormal Military Squad, whose leadership and expertise are vital to the team's efforts in engaging with extraterrestrial intelligence. His role is marked by a blend of strategic oversight, technical skill, and a deep commitment to the mission's success.
+In summary, Agent Alex Mercer is a key figure in the Paranormal Military Squad, leading efforts to establish contact with extraterrestrial intelligence while navigating the complexities and uncertainties of such a mission. His leadership, strategic foresight, and collaborative approach are crucial to the success of Operation: Dulce.
@@ -3428,25 +3428,21 @@ print(result.response) 
### Overview of Dr. Jordan Hayes
 
-Dr. Jordan Hayes is a pivotal figure within the Paranormal Military Squad, a specialized team operating at the Dulce Base. Dr. Hayes's primary role involves deciphering alien code and interpreting extraterrestrial patterns, which are crucial for the team's mission of understanding and interacting with alien entities [Data: Entities (104, 2); Relationships (22, 40, 64)].
+Dr. Jordan Hayes is a prominent figure within the Paranormal Military Squad, a specialized team operating at the Dulce Base. Dr. Hayes plays a crucial role in the squad's mission to understand and communicate with extraterrestrial entities. Their primary responsibilities include deciphering alien code, interpreting extraterrestrial patterns, and analyzing data related to star alignments. This expertise is vital for the squad's efforts in interspecies communication and understanding alien intelligence [Data: Entities (104, 2); Relationships (22, 40, 64)].
 
 ### Expertise and Contributions
 
-Dr. Hayes is renowned for their expertise in decryption algorithms and signal analysis, which are essential for the team's efforts in interstellar communication. This involves isolating signal harmonics, decrypting alien messages, and interpreting these signals for further analysis. Dr. Hayes's work is central to the team's operations at Dulce Base, where they focus on analyzing and interpreting alien signals [Data: Entities (2, 180, 192, 166); Relationships (25, 53)].
+Dr. Hayes is known for their analytical mindset and skepticism, which they apply rigorously to their work. They focus on empirical evidence and adaptability, especially when dealing with the unknown aspects of extraterrestrial phenomena. Their expertise in decryption algorithms and signal analysis is pivotal to the team's mission, as they work on isolating signal harmonics and interpreting alien messages. This role is essential for the team's operations at Dulce Base, where they aim to uncover the mysteries of extraterrestrial intelligence [Data: Entities (2, 148, 166, 180); Claims (12, 36, 39)].
 
-### Collaboration and Team Dynamics
+### Collaboration and Relationships
 
-Dr. Hayes works closely with Alex Mercer, another key member of the Paranormal Military Squad. Their collaboration is marked by mutual respect and understanding, as they both contribute their analytical skills to the mission. This partnership is crucial for managing interspecies communication and achieving the team's objectives [Data: Relationships (1, 4, 67); Claims (18, 42)].
+Dr. Hayes collaborates closely with other members of the Paranormal Military Squad, including Alex Mercer and Sam Rivera. Their relationship with Alex Mercer is characterized by mutual respect and understanding, as they work together on deciphering alien signals and managing interspecies communication. This partnership is crucial for the success of their mission, as both bring complementary skills to the table. Additionally, Dr. Hayes works alongside Sam Rivera, whose technical expertise complements their efforts in interpreting alien signals [Data: Relationships (1, 4, 21, 34); Claims (18, 42)].
 
-### Analytical Mindset and Skepticism
+### Challenges and Achievements
 
-Known for their analytical and skeptical approach, Dr. Hayes emphasizes empirical evidence and adaptability in the face of the unknown. This mindset is particularly evident when the team explores underground facilities, where Dr. Hayes remains cautious about non-empirical possibilities while reflecting on potential data discoveries [Data: Entities (2, 124); Claims (12, 13, 26)].
+Throughout their mission, Dr. Hayes has faced numerous challenges, including the need to adapt to evolving alien signals and the potential implications of their discoveries. They have been involved in analyzing extraterrestrial patterns that could represent both threats and untapped wisdom. Dr. Hayes's work has led to significant breakthroughs, such as discovering warnings within alien messages and contemplating the potential for interspecies communication as a form of cosmic diplomacy [Data: Claims (49, 54, 61, 68, 79)].
 
-### Challenges and Discoveries
-
-Throughout their mission, Dr. Hayes has encountered various challenges, including deciphering alien messages that could reveal extraterrestrial thought patterns. These efforts have led to significant discoveries, such as identifying potential threats and untapped wisdom within extraterrestrial patterns [Data: Claims (49, 84, 39)].
-
-In summary, Dr. Jordan Hayes is a central figure in the Paranormal Military Squad, contributing their expertise in alien code deciphering and signal analysis to the team's mission at Dulce Base. Their collaboration with Alex Mercer and their analytical mindset are key to navigating the complexities of interstellar communication and understanding extraterrestrial intelligence.
+In summary, Dr. Jordan Hayes is a key figure in the Paranormal Military Squad, contributing their expertise in alien code deciphering and signal analysis to the team's mission at Dulce Base. Their analytical approach and collaborative efforts with colleagues like Alex Mercer and Sam Rivera are instrumental in advancing the understanding of extraterrestrial intelligence.
@@ -3982,7 +3978,7 @@ print(candidate_questions.response)
diff --git a/search/search_index.json b/search/search_index.json index 5313804b..2ab5bcbc 100644 --- a/search/search_index.json +++ b/search/search_index.json @@ -1 +1 @@ -{"config": {"lang": ["en"], "separator": "[\\s\\-]+", "pipeline": ["stopWordFilter"]}, "docs": [{"location": "", "title": "Welcome to GraphRAG", "text": "

\ud83d\udc49 Microsoft Research Blog Post \ud83d\udc49 GraphRAG Accelerator \ud83d\udc49 GraphRAG Arxiv

Figure 1: An LLM-generated knowledge graph built using GPT-4 Turbo.

GraphRAG is a structured, hierarchical approach to Retrieval Augmented Generation (RAG), as opposed to naive semantic-search approaches using plain text snippets. The GraphRAG process involves extracting a knowledge graph out of raw text, building a community hierarchy, generating summaries for these communities, and then leveraging these structures when perform RAG-based tasks.

To learn more about GraphRAG and how it can be used to enhance your LLMs ability to reason about your private data, please visit the Microsoft Research Blog Post.

"}, {"location": "#solution-accelerator", "title": "Solution Accelerator \ud83d\ude80", "text": "

To quickstart the GraphRAG system we recommend trying the Solution Accelerator package. This provides a user-friendly end-to-end experience with Azure resources.

"}, {"location": "#get-started-with-graphrag", "title": "Get Started with GraphRAG \ud83d\ude80", "text": "

To start using GraphRAG, check out the Get Started guide. For a deeper dive into the main sub-systems, please visit the docpages for the Indexer and Query packages.

"}, {"location": "#graphrag-vs-baseline-rag", "title": "GraphRAG vs Baseline RAG \ud83d\udd0d", "text": "

Retrieval-Augmented Generation (RAG) is a technique to improve LLM outputs using real-world information. This technique is an important part of most LLM-based tools and the majority of RAG approaches use vector similarity as the search technique, which we call Baseline RAG. GraphRAG uses knowledge graphs to provide substantial improvements in question-and-answer performance when reasoning about complex information. RAG techniques have shown promise in helping LLMs to reason about private datasets - data that the LLM is not trained on and has never seen before, such as an enterprise\u2019s proprietary research, business documents, or communications. Baseline RAG was created to help solve this problem, but we observe situations where baseline RAG performs very poorly. For example:

To address this, the tech community is working to develop methods that extend and enhance RAG. Microsoft Research\u2019s new approach, GraphRAG, uses LLMs to create a knowledge graph based on an input corpus. This graph, along with community summaries and graph machine learning outputs, are used to augment prompts at query time. GraphRAG shows substantial improvement in answering the two classes of questions described above, demonstrating intelligence or mastery that outperforms other approaches previously applied to private datasets.

"}, {"location": "#the-graphrag-process", "title": "The GraphRAG Process \ud83e\udd16", "text": "

GraphRAG builds upon our prior research and tooling using graph machine learning. The basic steps of the GraphRAG process are as follows:

"}, {"location": "#index", "title": "Index", "text": ""}, {"location": "#query", "title": "Query", "text": "

At query time, these structures are used to provide materials for the LLM context window when answering a question. The primary query modes are:

"}, {"location": "#prompt-tuning", "title": "Prompt Tuning", "text": "

Using GraphRAG with your data out of the box may not yield the best possible results. We strongly recommend to fine-tune your prompts following the Prompt Tuning Guide in our documentation.

"}, {"location": "blog_posts/", "title": "Microsoft Research Blog", "text": "