Add integration testing w/external NextJS app (Edge runtime) (#304)

## Problem

We do not currently have a good way to test how our client behaves end
to end in different environments. One of the chief problems our users
have brought to our attention in the past is that some functionalities
do not work in certain runtimes, e.g. Edge or Bun, and/or with certain
frameworks, e.g. [NextJS](https://nextjs.org/docs).

## Solution

Build an external application in a runtime and framework known to have
caused problems in the past and test our client from the end-user's
perspective.

This PR introduces and end-to-end test suite that interacts with an
external application written using the NextJS framework and the
[Edge](https://vercel.com/docs/functions/edge-middleware/edge-runtime#edge-runtime)
runtime.

Note: the app is automagically run in `Edge` by way of it using
[middleware](https://nextjs.org/docs/app/building-your-application/routing/middleware)
(`middleware.ts`), and the [`Headers()`
API](https://developer.mozilla.org/en-US/docs/Web/API/Headers).

## Overview of changes
- We now have [a public
repo](https://github.com/pinecone-io/ts-client-e2e-tests) that contains
a super simple application that builds a Pinecone serverless index,
seeds it with data, and queries that data. We plan to add more
operations in the near future; this is just a start.
- This sample application creates an API endpoint that our client repo
can send a POST request to and assert on the response. If the endpoint
fails to deliver the expected response, we know that something is wrong
with the most recent changes we are proposing to introduce in the
client.
- This PR enables 2 types of interaction with the sample application:
- Local end-to-end tests: for local runs, there is a new bash script
that devs can execute. This will spin up the application on their
`localhost:3000`.
- CI end-to-end tests: for CI runs, we will now run this test
automatically by way of adding it the `testing.yml` file that is run for
each push to an open PR. In CI, the Github workflow and action hit the
application's Vercel endpoint and assert on its response.
- There is a new CONTRIBUTING.md file with some info on this new test
suite + the other existing ones, as well as a new README in the
`external-app` dir itself with more in depth information.

## Type of Change

- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- [x] This change requires a documentation update
- [x] Infrastructure change (CI configs, etc)
- [x] Non-code change (docs, etc)
- [ ] None of the above: (explain here)

## Test Plan

If reviewers can pull down the code in this branch and try to run the
tests locally, that'd be great.

## To Dos
I'll add a README and a CONTRIBUTING file to [the external
app](https://github.com/pinecone-io/ts-client-e2e-tests)

---
- To see the specific tasks where the Asana app for GitHub is being
used, see below:
  - https://app.asana.com/0/0/1208129226784956

.github/actions/e2e-testing/edge/action.yml

@@ -3,6 +3,10 @@ name: Spin up Vercel App
 inputs:
   vercel-token:
     required: true
+    description: 'Vercel token to deploy the app'
+  PINECONE_API_KEY:
+    required: true
+    description: 'Pinecone API key to send requests to the Vercel app'
 
 runs:
   using: 'composite'
@@ -27,27 +31,55 @@ runs:
       uses: GuillaumeFalourd/clone-github-repo-action@main
       with:
         owner: 'pinecone-io'
-        repository: 'pinecone-rag-demo'
+        repository: 'ts-client-test-external-app'
         branch: 'main'
 
-    - name: Move package to pinecone-rag-demo
-      run: cd pinecone-rag-demo && mv ../pinecone-database-pinecone-*.tgz .
+    - name: Install Vercel app dependencies
+      run: cd ts-client-test-external-app && npm install
+      shell: bash
+
+    - name: Move packed ts-client code into the Vercel app dir
+      run: cd ts-client-test-external-app && mv ../pinecone-database-pinecone-*.tgz .
       shell: bash
 
-    - name: Install rag-demo deps
-      run: cd pinecone-rag-demo && npm install
+    - name: Install ts-client code into the Vercel app's package.json file
+      run: |
+        cd ts-client-test-external-app
+        npm install pinecone-database-pinecone-*.tgz
       shell: bash
 
-    - name: Install pinecone-ts-client "main" branch code into the Vercel app
-      run: cd pinecone-rag-demo && npm install pinecone-database-pinecone-*.tgz
+    - name: Install Vercel CLI and (re)deploy the app
+      run: |
+        cd ts-client-test-external-app
+        npm install vercel
+        vercel --token ${{ inputs.vercel-token }}
+        vercel pull --yes --token ${{ inputs.vercel-token }}
+        vercel build --token ${{ inputs.vercel-token }}
+        vercel --token ${{ inputs.vercel-token }} --prod
       shell: bash
 
-    - name: Install Vercel CLI
-      run: cd pinecone-rag-demo && npm install vercel
+    - name: Hit Vercel app endpoint(s) via assertResponse.ts file
+      run: |
+        npm install -g typescript
+        npm install -g ts-node
+        export PINECONE_API_KEY="${{ inputs.PINECONE_API_KEY }}"
+        ciUrl='http://ts-client-test-external-app.vercel.app/api/createSeedQuery'
+        indexName=$(ts-node src/external-app/assertResponse.ts "$ciUrl"| grep -v "Test passed!")
+        echo "indexName=$indexName" >> $GITHUB_ENV
       shell: bash
 
-    - name: Deploy Project Artifacts to Vercel
+    - name: Clean up test index(es)
       run: |
-        cd pinecone-rag-demo
-        vercel --scope=pinecone-io --token=${{ inputs.vercel-token }} --yes
+        PINECONE_API_KEY="${{ inputs.PINECONE_API_KEY }}"
+        matching_index="${{ env.indexName }}"
+
+        delete_response=$(curl -s -o /dev/null -w "%{http_code}" -X DELETE "https://api.pinecone.io/indexes/$matching_index" \
+        -H "Api-Key: $PINECONE_API_KEY")
+
+        if [ "$delete_response" -eq 202 ]; then
+          echo "Successfully deleted index: $matching_index"
+        else
+          echo "Failed to delete index: $matching_index. HTTP status code: $delete_response"
+          exit 1
+        fi
       shell: bash

.github/workflows/e2e-testing.yml

@@ -4,17 +4,16 @@ on:
   workflow_dispatch: {}
 
 jobs:
-  e2e-tests:
-    name: Run end-to-end tests
+  external-app:
+    name: external-app
     runs-on: ubuntu-latest
+    if: always() # Run teardown even if the tests fail
     steps:
-      - name: Checkout
+      - name: Checkout code
         uses: actions/checkout@v4
 
-      - name: Setup
-        uses: ./.github/actions/setup
-
-      - name: Run e2e tests for edge runtime
+      - name: Run external app tests
         uses: ./.github/actions/e2e-testing/edge
         with:
           vercel-token: ${{ secrets.VERCEL_TOKEN }}
+          PINECONE_API_KEY: ${{ secrets.PINECONE_API_KEY }}

.github/workflows/testing.yml

@@ -210,3 +210,17 @@ jobs:
         run: |
           cd semantic-search-example
           npm run test
+
+  external-app:
+    name: external-app
+    runs-on: ubuntu-latest
+    if: always() # Run teardown even if the tests fail
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Run external app tests
+        uses: ./.github/actions/e2e-testing/edge
+        with:
+          vercel-token: ${{ secrets.VERCEL_TOKEN }}
+          PINECONE_API_KEY: ${{ secrets.PINECONE_API_KEY }}

.gitignore

@@ -19,3 +19,7 @@ scratch
 # Jetbrains
 .idea
 *.iml
+
+# External app testing dir
+ts-client-test-external-app/
+.next/

CONTRIBUTING.md

@@ -0,0 +1,37 @@
+# Contributing
+
+We welcome contributions to this project.
+
+## Repl
+
+For quick troubleshooting, there is a repl available by running `npm run repl`. This will start a Node.js repl with the
+`@pinecone-io/client` package preloaded. The `npm` command runs the file `utils/replInit.ts`.
+
+## Local testing
+
+To run all tests locally, excluding the unit tests, you will need to set your Pinecone API key to an environment
+variable (or hard-code it in a .env file in this repo).
+
+You can retrieve your API key from [app.pinecone.io](https://app.pinecone.io).
+
+```bash
+export PINECONE_API_KEY=your_api_key
+```
+
+To see the exact commands run by the aliases mentioned in this doc, see the `scripts` section in the `package.json`
+file.
+
+### Unit tests
+
+Simply run `npm run test:unit` to run all unit tests.
+
+### Integration tests
+
+Simply run `npm run test:integration-local:<runtime>` to run all integration tests, substituting `<runtime>` with
+either `node` or `edge` as you see fit.
+
+### External app tests
+
+Simply run `npm run test:external-app-local` to run all tests that integrate with the `ts-client-test-external-app` repo:
+
+The `npm` command runs the bash file located in the `src/external-app` directory.

README.md

@@ -1181,3 +1181,10 @@ console.log(response);
 // usage: { rerankUnits: 1 }
 //}
 ```
+
+## Testing
+
+All testing takes place automatically in CI and is configured using Github actions
+and workflows, located in the `.github` directory of this repo.
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) for more information.

package.json

@@ -26,6 +26,7 @@
     "test:integration:edge": "TEST_ENV=edge jest src/integration/ -c jest.config.integration-edge.js --runInBand --bail",
     "test:integration-local:edge": "TEST_ENV=edge node src/integration/integrationRunner.js",
     "test:integration:cleanup": "npm run build && node utils/cleanupResources.ts",
+    "test:external-app-local": "chmod +x src/external-app/local-external-app.sh && src/external-app/local-external-app.sh",
     "test:unit": "jest src/"
   },
   "engines": {

src/external-app/README.md

@@ -0,0 +1,15 @@
+# External app tests
+
+The test(s) in this directory test the initialization and some basic functionality of our client, as used in an
+external app. The goal of this type of testing is to ensure that the Typescript client can be used in real-world
+scenarios that have been incompatible with our client in the past, such as Vercel apps running on Edge.
+
+These tests differ from integration tests in that they are aimed at testing the client's interaction with various
+runtimes and frameworks, rather than the client's interaction with Pinecone's APIs directly.
+
+## Local runs
+
+To run the external app tests locally, execute `npm run test:external-app-local` from the root of this repository.
+
+You will need set a `PINECONE_API_KEY` environment variable for the tests to succeed. Additionally, ensure your local
+port `3000` is available, as that is the port the external app will run on.

src/external-app/assertResponse.ts

@@ -0,0 +1,47 @@
+/* Send requests to an external application and assert the response matches expectations. */
+
+class EdgeExternalAppTest {
+  url: string;
+  constructor(public readonly apiKey: string, url: string) {
+    this.apiKey = apiKey;
+    this.url = url;
+  }
+
+  hitEndpoint = async (url: string) => {
+    const response = await fetch(url, {
+      method: 'POST',
+      headers: {
+        'Content-Type': 'application/json',
+        pinecone_api_key: this.apiKey,
+      },
+    });
+    if (!response || response.status !== 200) {
+      throw new Error(`Failed to hit endpoint: ${response.status}`);
+    }
+    return response.json();
+  };
+
+  assertOnResponse = async () => {
+    const queryResponse = await this.hitEndpoint(this.url);
+    if (!(queryResponse['queryResult']['matches'].length >= 1)) {
+      throw new Error(
+        `Test failure, query response is empty: ${queryResponse}`
+      );
+    } else {
+      return queryResponse['indexName'];
+    }
+  };
+}
+
+const apiKey = process.env['PINECONE_API_KEY'];
+if (!apiKey) {
+  throw new Error('PINECONE_API_KEY key is required');
+}
+
+const url = process.argv[2]; // Get local URL from the command line arg
+
+const { assertOnResponse } = new EdgeExternalAppTest(apiKey, url);
+
+assertOnResponse().then((indexName) => {
+  console.log(indexName);
+});

src/external-app/local-external-app.sh

@@ -0,0 +1,58 @@
+#!/bin/bash
+
+set -eu -o pipefail
+
+# Must have API set
+if [ -z "$PINECONE_API_KEY" ]; then
+  echo "Please set the PINECONE_API_KEY environment variable."
+  exit 1
+fi
+
+# If ts-client-test-external-app exists, remove it
+if [ -d "ts-client-test-external-app" ]; then
+  echo "Removing existing ts-client-test-external-app directory..."
+  rm -rf ts-client-test-external-app
+fi
+
+# Clone ts-client-test-external-app repo
+echo "Cloning ts-client-test-external-app repo..."
+pushd .
+  git clone [email protected]:pinecone-io/ts-client-test-external-app.git
+  cd ts-client-test-external-app
+  git pull origin main
+popd
+
+# Compile ts-client and make a local link
+npm run build
+npm link
+
+# Temporarily cd into ts-client-e2e-tests repo; install deps; link and overwrite its ts-client dep w/local version of
+# ts-client; start the Next.js server
+pushd "ts-client-test-external-app"
+  git pull origin main
+  npm install
+  npm link @pinecone-database/pinecone
+  npm install -D next@latest
+  next dev & # `&` runs the command in the background
+popd
+
+# Run test file
+echo "Running tests..."
+localUrl='http://localhost:3000/api/createSeedQuery'  # TODO: parameterize later for different endpoints
+indexName=$(ts-node src/external-app/assertResponse.ts "$localUrl")
+
+# Delete test index
+echo "Deleting index '$indexName'..."
+delete_response=$(curl -s -o /dev/null -w "%{http_code}" -X DELETE "https://api.pinecone.io/indexes/${indexName}" -H \
+  "Api-Key: $PINECONE_API_KEY")
+
+if [ "$delete_response" -eq 202 ]; then
+  echo "Successfully deleted index: $indexName"
+else
+  echo "Failed to delete index: $indexName. HTTP status code: $delete_response"
+  exit 1
+fi
+
+# Kill Next.js server
+echo "Killing Next.js server..."
+kill $(lsof -t -i:3000)