Add new basic pipeline runner

.semversioner/next-release/minor-20241227205339264730.json

@@ -0,0 +1,4 @@
+{
+  "type": "minor",
+  "description": "new search implemented as a new option for the api"
+}

.semversioner/next-release/minor-20241231213627966329.json

@@ -0,0 +1,4 @@
+{
+  "type": "minor",
+  "description": "Remove old pipeline runner."
+}

.semversioner/next-release/minor-20241231214323349946.json

@@ -0,0 +1,4 @@
+{
+  "type": "minor",
+  "description": "Remove DataShaper (first steps)."
+}

.semversioner/next-release/patch-20241230224307150194.json

@@ -0,0 +1,4 @@
+{
+  "type": "minor",
+  "description": "Make gleanings independent of encoding"
+}

.semversioner/next-release/patch-20250102170720512799.json

@@ -0,0 +1,4 @@
+{
+  "type": "patch",
+  "description": "Remove config input models."
+}

.semversioner/next-release/patch-20250102232542899735.json

@@ -0,0 +1,4 @@
+{
+  "type": "patch",
+  "description": "Ruff update"
+}

dictionary.txt

@@ -148,10 +148,6 @@ codebases
 # Microsoft
 MSRC
 
-# Broken Upstream
-# TODO FIX IN DATASHAPER
-Arrary
-
 # Prompt Inputs
 ABILA
 Abila

docs/examples_notebooks/index_migration.ipynb

@@ -206,9 +206,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from datashaper import NoopVerbCallbacks\n",
-    "\n",
     "from graphrag.cache.factory import create_cache\n",
+    "from graphrag.callbacks.noop_verb_callbacks import NoopVerbCallbacks\n",
     "from graphrag.index.flows.generate_text_embeddings import generate_text_embeddings\n",
     "\n",
     "# We only need to re-run the embeddings workflow, to ensure that embeddings for all required search fields are in place\n",

docs/index/architecture.md

@@ -8,33 +8,9 @@ In order to support the GraphRAG system, the outputs of the indexing engine (in
 This model is designed to be an abstraction over the underlying data storage technology, and to provide a common interface for the GraphRAG system to interact with.
 In normal use-cases the outputs of the GraphRAG Indexer would be loaded into a database system, and the GraphRAG's Query Engine would interact with the database using the knowledge model data-store types.
 
-### DataShaper Workflows
-
-GraphRAG's Indexing Pipeline is built on top of our open-source library, [DataShaper](https://github.com/microsoft/datashaper).
-DataShaper is a data processing library that allows users to declaratively express data pipelines, schemas, and related assets using well-defined schemas.
-DataShaper has implementations in JavaScript and Python, and is designed to be extensible to other languages.
-
-One of the core resource types within DataShaper is a [Workflow](https://github.com/microsoft/datashaper/blob/main/javascript/schema/src/workflow/WorkflowSchema.ts).
-Workflows are expressed as sequences of steps, which we call [verbs](https://github.com/microsoft/datashaper/blob/main/javascript/schema/src/workflow/verbs.ts).
-Each step has a verb name and a configuration object.
-In DataShaper, these verbs model relational concepts such as SELECT, DROP, JOIN, etc.. Each verb transforms an input data table, and that table is passed down the pipeline.
-
-```mermaid
----
-title: Sample Workflow
----
-flowchart LR
-    input[Input Table] --> select[SELECT] --> join[JOIN] --> binarize[BINARIZE] --> output[Output Table]
-```
-
-### LLM-based Workflow Steps
-
-GraphRAG's Indexing Pipeline implements a handful of custom verbs on top of the standard, relational verbs that our DataShaper library provides. These verbs give us the ability to augment text documents with rich, structured data using the power of LLMs such as GPT-4. We utilize these verbs in our standard workflow to extract entities, relationships, claims, community structures, and community reports and summaries. This behavior is customizable and can be extended to support many kinds of AI-based data enrichment and extraction tasks.
-
-### Workflow Graphs
+### Workflows
 
 Because of the complexity of our data indexing tasks, we needed to be able to express our data pipeline as series of multiple, interdependent workflows.
-In the GraphRAG Indexing Pipeline, each workflow may define dependencies on other workflows, effectively forming a directed acyclic graph (DAG) of workflows, which is then used to schedule processing.
 
 ```mermaid
 ---
@@ -55,7 +31,7 @@ stateDiagram-v2
 The primary unit of communication between workflows, and between workflow steps is an instance of `pandas.DataFrame`.
 Although side-effects are possible, our goal is to be _data-centric_ and _table-centric_ in our approach to data processing.
 This allows us to easily reason about our data, and to leverage the power of dataframe-based ecosystems.
-Our underlying dataframe technology may change over time, but our primary goal is to support the DataShaper workflow schema while retaining single-machine ease of use and developer ergonomics.
+Our underlying dataframe technology may change over time, but our primary goal is to support the workflow schema while retaining single-machine ease of use and developer ergonomics.
 
 ### LLM Caching