mirror of
https://github.com/ToolJet/ToolJet
synced 2026-05-23 08:58:26 +00:00
Integrate Pinecone Plugin with Core Operations for Advanced Vector Management (#11188)
* Initialize ToolJet plugin with default files and installed dependencies Created the folder structure and default files for the pinecone plugin the server/src/assets/marketplace/plugins.json file also got updated automatically and added an entry for the Pinecone plugin. Installed the specified versions of @grpc/grpc-js@^1.12.0 and @pinecone-database/pinecone@^3.0.3 packages into the project. * Update Pinecone plugin manifest with API key authentication Updated Pinecone plugin manifest to include API key authentication as an encrypted string. Added API key field with description and help text for generating the key via the Pinecone Console. Marked apiKey as a required field in the schema. * Enhance Pinecone operations.json with vector management options Amended operations.json to support various vector management operations for the Pinecone plugin. Added new fields for index, operation, and specific properties for operations like get_index_stats, list_vector_ids, fetch_vectors, upsert_vectors, update_vector, and delete_vectors. Introduced support for vector ID management, metadata, namespace, and filter options in JSON format. Updated the schema type from api to database to better reflect the plugin's functionality. * Add query operations and enhance types for Pinecone plugin Created query_operations.ts to handle Pinecone operations: getIndexStats, listVectorIds, fetchVectors, upsertVectors, updateVector, and deleteVectors. Added error handling, namespace support, and consolidated vector deletion logic. Updated types.ts with detailed SourceOptions and QueryOptions, including fields like index, ids, vectors, and pagination. Introduced types for Vector, SparseValues, and an enum for available operations, ensuring type safety. Functions for getIndexStats and listVectorIds work correctly, but other operations are not functioning as expected. * Implement Pinecone service with query operations and connection handling Updated index.ts with PineconeService to handle operations: getIndexStats, listVectorIds, fetchVectors, upsertVectors, updateVector, and deleteVectors. Added connection testing and API key handling functions. Error handling for invalid operations included. Only getIndexStats and listVectorIds are working correctly; other operations need debugging. * Fix operations and add query vector operation * Update placeholders for boolean fields Set placeholders for delete_all, include_values, and include_metadata to indicate "true (false by default)" for clearer defaults. --------- Co-authored-by: parthy007 <parthadhikari1812@gmail.com>
This commit is contained in:
namanmathur372
GitHub
parent
7d5000d498
commit
ad23daeca1
12 changed files with 859 additions and 80 deletions
5
marketplace/plugins/pinecone/.gitignore
vendored
Normal file
5
marketplace/plugins/pinecone/.gitignore
vendored
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
node_modules
|
||||
lib/*.d.*
|
||||
lib/*.js
|
||||
lib/*.js.map
|
||||
dist/*
|
||||
4
marketplace/plugins/pinecone/README.md
Normal file
4
marketplace/plugins/pinecone/README.md
Normal file
|
|
@ -0,0 +1,4 @@
|
|||
|
||||
# Pinecone
|
||||
|
||||
Documentation on: https://docs.tooljet.com/docs/data-sources/pinecone
|
||||
7
marketplace/plugins/pinecone/__tests__/index.js
Normal file
7
marketplace/plugins/pinecone/__tests__/index.js
Normal file
|
|
@ -0,0 +1,7 @@
|
|||
'use strict';
|
||||
|
||||
const pinecone = require('../lib');
|
||||
|
||||
describe('pinecone', () => {
|
||||
it.todo('needs tests');
|
||||
});
|
||||
72
marketplace/plugins/pinecone/lib/icon.svg
Normal file
72
marketplace/plugins/pinecone/lib/icon.svg
Normal file
|
|
@ -0,0 +1,72 @@
|
|||
<?xml version="1.0" encoding="iso-8859-1"?>
|
||||
<!-- Generator: Adobe Illustrator 19.0.0, SVG Export Plug-In . SVG Version: 6.00 Build 0) -->
|
||||
<svg version="1.1" id="Layer_1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px"
|
||||
viewBox="0 0 512 512" style="enable-background:new 0 0 512 512;" xml:space="preserve">
|
||||
<g>
|
||||
<g>
|
||||
<g>
|
||||
<rect x="445.176" y="222.066" width="15.208" height="32.606"/>
|
||||
<path d="M465.275,103.536V55.237c0-10.89-8.859-19.75-19.749-19.75H292.721c-10.89,0-19.75,8.86-19.75,19.75v48.298h-38.498
|
||||
V55.237c0-10.89-8.86-19.75-19.75-19.75H61.918c-10.891,0-19.75,8.86-19.75,19.75v48.298H0v372.977h512V103.536H465.275z
|
||||
M288.178,55.237c0-2.505,2.038-4.542,4.542-4.542h152.805c2.504,0,4.541,2.038,4.541,4.542v48.298H422.47V64.268h-15.208v39.268
|
||||
H288.178V55.237z M57.374,55.237c0-2.505,2.038-4.542,4.542-4.542h152.804c2.504,0,4.542,2.038,4.542,4.542v48.298h-29.126
|
||||
V64.268h-15.208v39.268H57.374V55.237z M496.792,461.305h-36.404V269.298H445.18v192.007H15.208V118.743h26.959h192.305h38.498
|
||||
h192.305h31.517V461.305z"/>
|
||||
<path d="M385.777,275.757c-1.028-0.078-2.067-0.118-3.092-0.118c-10.329,0-20.117,3.942-27.562,11.096
|
||||
c-0.054,0.051-0.105,0.101-0.159,0.15v-81.028h-81.027c0.05-0.054,0.099-0.105,0.149-0.158
|
||||
c7.838-8.156,11.84-19.33,10.978-30.656c-1.449-19.026-16.24-34.353-35.171-36.443c-1.494-0.164-3.008-0.248-4.5-0.248
|
||||
c-21.939,0-39.787,17.848-39.787,39.788c0,10.414,3.994,20.254,11.25,27.719h-81.031v99.334h7.604
|
||||
c6.248,0,12.237-2.492,16.431-6.835c5.122-5.303,12.213-8.006,19.621-7.435c11.548,0.878,21.213,10.196,22.481,21.673
|
||||
c0.787,7.121-1.381,13.948-6.105,19.223c-4.661,5.204-11.337,8.189-18.317,8.189c-6.631,0-12.846-2.602-17.502-7.328
|
||||
c-3.186-3.232-7.177-5.463-11.542-6.449c-0.757-0.172-1.534-0.259-2.307-0.259c-5.715,0-10.364,4.639-10.364,10.343v88.681
|
||||
h88.991c5.703,0,10.343-4.639,10.343-10.342c0-6.248-2.492-12.238-6.836-16.432c-5.285-5.103-7.994-12.256-7.435-19.622
|
||||
c0.879-11.548,10.197-21.212,21.675-22.48c0.943-0.105,1.895-0.157,2.831-0.157c13.553,0,24.58,11.027,24.58,24.58
|
||||
c0,6.606-2.585,12.805-7.281,17.456c-4.555,4.514-7.065,10.396-7.065,16.565c0,5.753,4.68,10.433,10.434,10.433h88.9v-81.031
|
||||
c7.465,7.255,17.304,11.25,27.719,11.25c11.299,0,22.105-4.83,29.646-13.251c7.537-8.415,11.142-19.728,9.893-31.037
|
||||
C420.13,291.995,404.803,277.205,385.777,275.757z M401.001,331.817c-4.661,5.205-11.338,8.19-18.317,8.19
|
||||
c-6.457,0-12.557-2.486-17.174-7c-4.843-4.737-11.287-7.344-18.148-7.344h-7.604v84.126h-67.227
|
||||
c0.262-0.342,0.551-0.671,0.868-0.987c7.6-7.528,11.785-17.564,11.785-28.259c0-21.94-17.848-39.788-39.788-39.788
|
||||
c-1.491,0-3.005,0.084-4.5,0.248c-18.931,2.093-33.722,17.419-35.17,36.442c-0.905,11.898,3.482,23.458,12.036,31.718
|
||||
c0.202,0.195,0.391,0.404,0.569,0.625h-67.295v-64.691c7.3,6.542,16.618,10.117,26.507,10.117
|
||||
c11.298,0,22.104-4.83,29.646-13.251c7.537-8.415,11.142-19.728,9.892-31.038c-2.092-18.93-17.419-33.72-36.444-35.168
|
||||
c-1.027-0.078-2.068-0.118-3.092-0.118c-9.906,0-19.212,3.571-26.508,10.116v-64.689l84.126-0.005v-7.604
|
||||
c0-6.856-2.608-13.299-7.344-18.144c-4.515-4.618-7.001-10.716-7.001-17.174c0-13.553,11.026-24.58,24.579-24.58
|
||||
c0.936,0,1.888,0.053,2.831,0.157c11.478,1.267,20.796,10.932,21.675,22.481c0.541,7.117-1.867,13.851-6.78,18.964
|
||||
c-4.831,5.026-7.491,11.525-7.491,18.3v7.604h84.126v84.126h7.604c6.775,0,13.273-2.66,18.301-7.491
|
||||
c5.061-4.863,11.901-7.314,18.963-6.78c11.548,0.878,21.213,10.196,22.481,21.675
|
||||
C407.893,319.717,405.725,326.542,401.001,331.817z"/>
|
||||
</g>
|
||||
</g>
|
||||
</g>
|
||||
<g>
|
||||
</g>
|
||||
<g>
|
||||
</g>
|
||||
<g>
|
||||
</g>
|
||||
<g>
|
||||
</g>
|
||||
<g>
|
||||
</g>
|
||||
<g>
|
||||
</g>
|
||||
<g>
|
||||
</g>
|
||||
<g>
|
||||
</g>
|
||||
<g>
|
||||
</g>
|
||||
<g>
|
||||
</g>
|
||||
<g>
|
||||
</g>
|
||||
<g>
|
||||
</g>
|
||||
<g>
|
||||
</g>
|
||||
<g>
|
||||
</g>
|
||||
<g>
|
||||
</g>
|
||||
</svg>
|
||||
|
||||
|
After Width: | Height: | Size: 3.6 KiB |
91
marketplace/plugins/pinecone/lib/index.ts
Normal file
91
marketplace/plugins/pinecone/lib/index.ts
Normal file
|
|
@ -0,0 +1,91 @@
|
|||
import { QueryError, QueryResult, QueryService, ConnectionTestResult } from '@tooljet-marketplace/common';
|
||||
import { SourceOptions, QueryOptions, Operation } from './types';
|
||||
import {
|
||||
getIndexStats,
|
||||
listVectorIds,
|
||||
fetchVectors,
|
||||
upsertVectors,
|
||||
updateVector,
|
||||
deleteVectors,
|
||||
quertVectors,
|
||||
} from './query_operations';
|
||||
import { Pinecone } from '@pinecone-database/pinecone';
|
||||
|
||||
export default class PineconeService implements QueryService {
|
||||
// Function to run the specified operation
|
||||
async run(sourceOptions: SourceOptions, queryOptions: QueryOptions, dataSourceId: string): Promise<QueryResult> {
|
||||
const operation = queryOptions.operation;
|
||||
const pinecone = await this.getConnection(sourceOptions);
|
||||
let result = {};
|
||||
|
||||
try {
|
||||
switch (operation) {
|
||||
case Operation.GetIndexStats:
|
||||
result = await getIndexStats(pinecone, queryOptions);
|
||||
break;
|
||||
case Operation.ListVectorIds:
|
||||
result = await listVectorIds(pinecone, queryOptions);
|
||||
break;
|
||||
case Operation.FetchVectors:
|
||||
result = await fetchVectors(pinecone, queryOptions);
|
||||
break;
|
||||
case Operation.UpsertVectors:
|
||||
result = await upsertVectors(pinecone, queryOptions);
|
||||
break;
|
||||
case Operation.UpdateVector:
|
||||
result = await updateVector(pinecone, queryOptions);
|
||||
break;
|
||||
case Operation.DeleteVectors:
|
||||
result = await deleteVectors(pinecone, queryOptions);
|
||||
break;
|
||||
case Operation.QueryVectors:
|
||||
result = await quertVectors(pinecone, queryOptions);
|
||||
break;
|
||||
default:
|
||||
throw new QueryError('Query could not be completed', 'Invalid operation', {});
|
||||
}
|
||||
} catch (error) {
|
||||
throw new QueryError('Query could not be completed', error?.message, {});
|
||||
}
|
||||
|
||||
return {
|
||||
status: 'ok',
|
||||
data: result,
|
||||
};
|
||||
}
|
||||
|
||||
// Function to test the Pinecone connection
|
||||
async testConnection(sourceOptions: SourceOptions): Promise<ConnectionTestResult> {
|
||||
const pinecone = await this.getConnection(sourceOptions);
|
||||
|
||||
try {
|
||||
const indexes = await pinecone.listIndexes();
|
||||
console.log('Indexes fetched:', indexes);
|
||||
|
||||
if (indexes.indexes.length > 0) {
|
||||
console.log('Connection successful, indexes found');
|
||||
return { status: 'ok' };
|
||||
} else {
|
||||
console.error('No indexes found');
|
||||
throw new QueryError('No indexes found', 'The index list is empty', {});
|
||||
}
|
||||
} catch (error) {
|
||||
console.error('Connection could not be established:', error.message);
|
||||
throw new QueryError('Connection could not be established', error?.message, {});
|
||||
}
|
||||
}
|
||||
|
||||
async getConnection(sourceOptions: SourceOptions): Promise<Pinecone> {
|
||||
const { apiKey } = sourceOptions;
|
||||
|
||||
if (!apiKey) {
|
||||
throw new QueryError('API key missing', 'No API key provided in source options', {});
|
||||
}
|
||||
|
||||
const pinecone = new Pinecone({
|
||||
apiKey: apiKey,
|
||||
});
|
||||
|
||||
return pinecone;
|
||||
}
|
||||
}
|
||||
34
marketplace/plugins/pinecone/lib/manifest.json
Normal file
34
marketplace/plugins/pinecone/lib/manifest.json
Normal file
|
|
@ -0,0 +1,34 @@
|
|||
{
|
||||
"$schema": "https://raw.githubusercontent.com/ToolJet/ToolJet/develop/plugins/schemas/manifest.schema.json",
|
||||
"title": "Pinecone Plugin",
|
||||
"description": "A schema defining Pinecone datasource",
|
||||
"type": "api",
|
||||
"source": {
|
||||
"name": "Pinecone",
|
||||
"kind": "pinecone",
|
||||
"exposedVariables": {
|
||||
"isLoading": false,
|
||||
"data": {},
|
||||
"rawData": {}
|
||||
},
|
||||
"options": {
|
||||
"apiKey": {
|
||||
"type": "string",
|
||||
"encrypted": true
|
||||
}
|
||||
}
|
||||
},
|
||||
"defaults": {},
|
||||
"properties": {
|
||||
"apiKey": {
|
||||
"label": "API Key",
|
||||
"key": "apiKey",
|
||||
"type": "password",
|
||||
"description": "Enter your Pinecone API Key",
|
||||
"helpText": "For generating API Key, visit: <a href='https://app.pinecone.io/organizations/-/projects/-/keys' target='_blank' rel='noreferrer'>Pinecone Console</a>"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"apiKey"
|
||||
]
|
||||
}
|
||||
294
marketplace/plugins/pinecone/lib/operations.json
Normal file
294
marketplace/plugins/pinecone/lib/operations.json
Normal file
|
|
@ -0,0 +1,294 @@
|
|||
{
|
||||
"$schema": "https://raw.githubusercontent.com/ToolJet/ToolJet/develop/plugins/schemas/operations.schema.json",
|
||||
"title": "Pinecone Datasource",
|
||||
"description": "A schema defining Pinecone datasource",
|
||||
"type": "database",
|
||||
"defaults": {},
|
||||
"properties": {
|
||||
"index": {
|
||||
"label": "Index",
|
||||
"key": "index",
|
||||
"type": "text",
|
||||
"description": "Enter the index name (e.g., example-index)",
|
||||
"placeholder": "example-index",
|
||||
"height": "36px"
|
||||
},
|
||||
"operation": {
|
||||
"label": "Operation",
|
||||
"key": "operation",
|
||||
"type": "dropdown-component-flip",
|
||||
"description": "Select an operation",
|
||||
"list": [
|
||||
{ "value": "get_index_stats", "name": "Get Index Stats" },
|
||||
{ "value": "list_vector_ids", "name": "List Vector IDs" },
|
||||
{ "value": "fetch_vectors", "name": "Fetch Vectors" },
|
||||
{ "value": "upsert_vectors", "name": "Upsert Vectors" },
|
||||
{ "value": "update_vector", "name": "Update a Vector" },
|
||||
{ "value": "delete_vectors", "name": "Delete Vectors" },
|
||||
{ "value": "query_vector", "name": "Query Vectors" }
|
||||
]
|
||||
},
|
||||
"get_index_stats": {
|
||||
"index": {
|
||||
"label": "Index",
|
||||
"key": "index",
|
||||
"type": "codehinter",
|
||||
"description": "Enter the index name (e.g., example-index)",
|
||||
"placeholder": "example-index",
|
||||
"height": "36px"
|
||||
}
|
||||
},
|
||||
"list_vector_ids": {
|
||||
"index": {
|
||||
"label": "Index",
|
||||
"key": "index",
|
||||
"type": "codehinter",
|
||||
"description": "Enter the index name (e.g., example-index)",
|
||||
"placeholder": "example-index",
|
||||
"height": "36px"
|
||||
},
|
||||
"prefix": {
|
||||
"label": "Prefix",
|
||||
"key": "prefix",
|
||||
"type": "codehinter",
|
||||
"description": "Enter a prefix to filter vector IDs",
|
||||
"placeholder": "document1#",
|
||||
"height": "36px"
|
||||
},
|
||||
"limit": {
|
||||
"label": "Limit",
|
||||
"key": "limit",
|
||||
"type": "codehinter",
|
||||
"description": "Enter the maximum number of vector IDs to return",
|
||||
"placeholder": "100",
|
||||
"height": "36px"
|
||||
},
|
||||
"paginationToken": {
|
||||
"label": "Pagination Token",
|
||||
"key": "paginationToken",
|
||||
"type": "codehinter",
|
||||
"description": "Enter next token for next set of vector IDs",
|
||||
"placeholder": "Tm90aGluzYB0byBZzWUGaGVyZQo=",
|
||||
"height": "36px"
|
||||
},
|
||||
"namespace": {
|
||||
"label": "Namespace",
|
||||
"key": "namespace",
|
||||
"type": "codehinter",
|
||||
"description": "Enter the namespace (optional)",
|
||||
"placeholder": "example-namespace",
|
||||
"height": "36px"
|
||||
}
|
||||
},
|
||||
"fetch_vectors": {
|
||||
"index": {
|
||||
"label": "Index",
|
||||
"key": "index",
|
||||
"type": "codehinter",
|
||||
"description": "Enter the index name (e.g., example-index)",
|
||||
"placeholder": "example-index",
|
||||
"height": "36px"
|
||||
},
|
||||
"ids": {
|
||||
"label": "IDs",
|
||||
"key": "ids",
|
||||
"type": "codehinter",
|
||||
"description": "Enter vector IDs as JSON array",
|
||||
"placeholder": "[\"id-1\", \"id-2\"]",
|
||||
"height": "36px"
|
||||
},
|
||||
"namespace": {
|
||||
"label": "Namespace",
|
||||
"key": "namespace",
|
||||
"type": "codehinter",
|
||||
"description": "Enter the namespace (optional)",
|
||||
"placeholder": "example-namespace",
|
||||
"height": "36px"
|
||||
}
|
||||
},
|
||||
"upsert_vectors": {
|
||||
"index": {
|
||||
"label": "Index",
|
||||
"key": "index",
|
||||
"type": "codehinter",
|
||||
"description": "Enter the index name (e.g., example-index)",
|
||||
"placeholder": "example-index",
|
||||
"height": "36px"
|
||||
},
|
||||
"vectors": {
|
||||
"label": "Vectors",
|
||||
"key": "vectors",
|
||||
"type": "codehinter",
|
||||
"description": "Enter vectors as JSON array",
|
||||
"placeholder": "[{\"id\": \"vec1\", \"values\": [0.1, 0.2, 0.3]}]",
|
||||
"height": "36px"
|
||||
},
|
||||
"namespace": {
|
||||
"label": "Namespace",
|
||||
"key": "namespace",
|
||||
"type": "codehinter",
|
||||
"description": "Enter the namespace (optional)",
|
||||
"placeholder": "example-namespace",
|
||||
"height": "36px"
|
||||
}
|
||||
},
|
||||
"update_vector": {
|
||||
"index": {
|
||||
"label": "Index",
|
||||
"key": "index",
|
||||
"type": "codehinter",
|
||||
"description": "Enter the index name (e.g., example-index)",
|
||||
"placeholder": "example-index",
|
||||
"height": "36px"
|
||||
},
|
||||
"id": {
|
||||
"label": "ID",
|
||||
"key": "id",
|
||||
"type": "codehinter",
|
||||
"description": "Enter vector ID to update",
|
||||
"placeholder": "id-3",
|
||||
"height": "36px"
|
||||
},
|
||||
"values": {
|
||||
"label": "Values",
|
||||
"key": "values",
|
||||
"type": "codehinter",
|
||||
"description": "Enter updated vector values as JSON array",
|
||||
"placeholder": "[4.0, 2.0]",
|
||||
"height": "36px"
|
||||
},
|
||||
"sparse_vector": {
|
||||
"label": "Sparse Vector",
|
||||
"key": "sparse_vector",
|
||||
"type": "codehinter",
|
||||
"description": "Enter sparse vector values",
|
||||
"placeholder": "{\"indices\": [1, 5], \"values\": [0.5, 0.5]}",
|
||||
"height": "36px"
|
||||
},
|
||||
"metadata": {
|
||||
"label": "Metadata",
|
||||
"key": "metadata",
|
||||
"type": "codehinter",
|
||||
"description": "Enter metadata",
|
||||
"placeholder": "{\"genre\": \"comedy\"}",
|
||||
"height": "36px"
|
||||
},
|
||||
"namespace": {
|
||||
"label": "Namespace",
|
||||
"key": "namespace",
|
||||
"type": "codehinter",
|
||||
"description": "Enter the namespace (optional)",
|
||||
"placeholder": "example-namespace",
|
||||
"height": "36px"
|
||||
}
|
||||
},
|
||||
"delete_vectors": {
|
||||
"index": {
|
||||
"label": "Index",
|
||||
"key": "index",
|
||||
"type": "codehinter",
|
||||
"description": "Enter the index name (e.g., example-index)",
|
||||
"placeholder": "example-index",
|
||||
"height": "36px"
|
||||
},
|
||||
"ids": {
|
||||
"label": "ID",
|
||||
"key": "id",
|
||||
"type": "codehinter",
|
||||
"description": "Enter vector IDs as JSON array",
|
||||
"placeholder": "[\"id-1\", \"id-2\"]",
|
||||
"height": "36px"
|
||||
},
|
||||
"delete_all": {
|
||||
"label": "Delete All",
|
||||
"key": "delete_all",
|
||||
"type": "codehinter",
|
||||
"description": "Set true to delete all vectors",
|
||||
"placeholder": "true (false by default)",
|
||||
"height":"36px"
|
||||
},
|
||||
"namespace": {
|
||||
"label": "Namespace",
|
||||
"key": "namespace",
|
||||
"type": "codehinter",
|
||||
"description": "Enter the namespace (optional)",
|
||||
"placeholder": "example-namespace",
|
||||
"height": "36px"
|
||||
},
|
||||
"filter": {
|
||||
"label": "Filter",
|
||||
"key": "filter",
|
||||
"type": "codehinter",
|
||||
"description": "Enter a filter query in JSON format",
|
||||
"placeholder": "{\"genre\": {\"$in\": [\"documentary\", \"action\"]}}",
|
||||
"height": "150px"
|
||||
}
|
||||
},
|
||||
"query_vector": {
|
||||
"index": {
|
||||
"label": "Index",
|
||||
"key": "index",
|
||||
"type": "codehinter",
|
||||
"description": "Enter the index name (e.g., example-index)",
|
||||
"placeholder": "example-index",
|
||||
"height": "36px"
|
||||
},
|
||||
"namespace": {
|
||||
"label": "Namespace",
|
||||
"key": "namespace",
|
||||
"type": "codehinter",
|
||||
"description": "Enter the namespace (optional)",
|
||||
"placeholder": "example-namespace",
|
||||
"height": "36px"
|
||||
},
|
||||
"top_k": {
|
||||
"label": "Top K",
|
||||
"key": "top_k",
|
||||
"type": "codehinter",
|
||||
"description": "Enter the number",
|
||||
"placeholder": "3",
|
||||
"height": "36px"
|
||||
},
|
||||
"filter": {
|
||||
"label": "Filter",
|
||||
"key": "filter",
|
||||
"type": "codehinter",
|
||||
"description": "Enter a filter query in JSON format",
|
||||
"placeholder": "{\"genre\": {\"$in\": [\"documentary\", \"action\"]}}",
|
||||
"height": "150px"
|
||||
},
|
||||
"include_values": {
|
||||
"label": "Include values",
|
||||
"key": "include_values",
|
||||
"type": "codehinter",
|
||||
"description": "Enter boolean values",
|
||||
"placeholder": "true (false by default)",
|
||||
"height": "36px"
|
||||
},
|
||||
"include_metadata": {
|
||||
"label": "Include metadata",
|
||||
"key": "include_metadata",
|
||||
"type": "codehinter",
|
||||
"description": "Enter boolean values",
|
||||
"placeholder": "true (false by default)",
|
||||
"height": "36px"
|
||||
},
|
||||
"vectors": {
|
||||
"label": "Vector",
|
||||
"key": "vectors",
|
||||
"type": "codehinter",
|
||||
"description": "Enter vector IDs as JSON array",
|
||||
"placeholder": "[\"0.3\", \"0.3\", \"0.3\", \"0.3\", \"0.3\"]",
|
||||
"height": "36px"
|
||||
},
|
||||
"sparse_vector": {
|
||||
"label": "Sparse Vector",
|
||||
"key": "sparse_vector",
|
||||
"type": "codehinter",
|
||||
"description": "Enter sparse vector values",
|
||||
"placeholder": "{\"indices\": [1, 5], \"values\": [0.5, 0.5]}",
|
||||
"height": "36px"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
181
marketplace/plugins/pinecone/lib/query_operations.ts
Normal file
181
marketplace/plugins/pinecone/lib/query_operations.ts
Normal file
|
|
@ -0,0 +1,181 @@
|
|||
import { QueryOptions } from './types';
|
||||
import { Pinecone } from '@pinecone-database/pinecone';
|
||||
|
||||
export async function getIndexStats(pinecone: Pinecone, options: QueryOptions): Promise<any> {
|
||||
const { index } = options;
|
||||
|
||||
if (!index) {
|
||||
throw new Error('Index name is required');
|
||||
}
|
||||
|
||||
try {
|
||||
const indexClient = pinecone.index(index);
|
||||
const stats = await indexClient.describeIndexStats();
|
||||
|
||||
return stats;
|
||||
} catch (error) {
|
||||
console.error('Error fetching index stats:', error);
|
||||
throw new Error(error?.message || 'An unexpected error occurred');
|
||||
}
|
||||
}
|
||||
|
||||
export async function listVectorIds(pinecone: Pinecone, options: QueryOptions): Promise<any> {
|
||||
const { index, prefix, limit, paginationToken, namespace } = options;
|
||||
|
||||
if (!index) {
|
||||
throw new Error('Index name is required');
|
||||
}
|
||||
|
||||
try {
|
||||
const indexClient = pinecone.index(index);
|
||||
|
||||
const listOptions = {
|
||||
prefix: prefix,
|
||||
limit: limit || 10,
|
||||
paginationToken: paginationToken,
|
||||
};
|
||||
|
||||
const client = namespace ? indexClient.namespace(namespace) : indexClient;
|
||||
|
||||
const vectors = await client.listPaginated(listOptions);
|
||||
|
||||
return vectors;
|
||||
} catch (error) {
|
||||
console.error('Error listing vector IDs:', error);
|
||||
throw new Error(error?.message || 'An unexpected error occurred');
|
||||
}
|
||||
}
|
||||
|
||||
export async function fetchVectors(pinecone: Pinecone, options: QueryOptions): Promise<any> {
|
||||
const { index, ids, namespace } = options;
|
||||
|
||||
if (!index || !ids) {
|
||||
throw new Error('Index name and vector IDs are required');
|
||||
}
|
||||
|
||||
const vectorIds = typeof ids === 'string' ? JSON.parse(ids) : ids;
|
||||
|
||||
try {
|
||||
const indexClient = pinecone.index(index);
|
||||
const client = namespace ? await indexClient.namespace(namespace) : indexClient;
|
||||
const vectors = await client.fetch(vectorIds);
|
||||
|
||||
return vectors;
|
||||
} catch (error) {
|
||||
throw new Error(error?.message || 'An unexpected error occurred');
|
||||
}
|
||||
}
|
||||
|
||||
export async function upsertVectors(pinecone: Pinecone, options: QueryOptions): Promise<any> {
|
||||
const { index, vectors, namespace } = options;
|
||||
const parsedVectors = typeof vectors === 'string' ? JSON.parse(vectors) : vectors;
|
||||
|
||||
if (!index || !vectors) {
|
||||
throw new Error('Index name and vectors are required');
|
||||
}
|
||||
|
||||
parsedVectors.forEach((vector) => {
|
||||
if (!vector.id || !Array.isArray(vector.values)) {
|
||||
throw new Error('Each vector must have an id and a values array');
|
||||
}
|
||||
});
|
||||
|
||||
try {
|
||||
const indexClient = pinecone.index(index);
|
||||
const client = namespace ? await indexClient.namespace(namespace) : indexClient;
|
||||
const upsertResponse = await client.upsert(parsedVectors);
|
||||
if (upsertResponse === undefined) {
|
||||
return 'Upsert Successful';
|
||||
} else {
|
||||
throw new Error('Upsert failed');
|
||||
}
|
||||
} catch (error) {
|
||||
throw new Error(error?.message || 'An unexpected error occurred');
|
||||
}
|
||||
}
|
||||
|
||||
export async function updateVector(pinecone: Pinecone, options: QueryOptions): Promise<any> {
|
||||
const { index, id, values, sparse_vector, metadata, namespace } = options;
|
||||
|
||||
if (!index || !id || (!values && !sparse_vector)) {
|
||||
throw new Error('Index name, vector ID, and either values or sparse vector are required');
|
||||
}
|
||||
|
||||
const valuesArray = typeof values === 'string' ? JSON.parse(values) : values;
|
||||
|
||||
try {
|
||||
const indexClient = pinecone.index(index);
|
||||
const client = namespace ? await indexClient.namespace(namespace) : indexClient;
|
||||
const updateResponse = await client.update({
|
||||
id,
|
||||
...(valuesArray && { values: valuesArray }),
|
||||
...(metadata && { metadata: JSON.parse(metadata) }),
|
||||
...(sparse_vector && { sparseValues: JSON.parse(sparse_vector) }),
|
||||
});
|
||||
|
||||
if (updateResponse === undefined) {
|
||||
return 'Update Successful';
|
||||
} else {
|
||||
throw new Error('Update failed');
|
||||
}
|
||||
} catch (error) {
|
||||
throw new Error(error?.message || 'An unexpected error occurred');
|
||||
}
|
||||
}
|
||||
|
||||
export async function deleteVectors(pinecone: Pinecone, options: QueryOptions): Promise<any> {
|
||||
const { index, id, delete_all, namespace, filter } = options;
|
||||
|
||||
if (!index) {
|
||||
throw new Error('Index name is required');
|
||||
}
|
||||
|
||||
try {
|
||||
const indexClient = pinecone.index(index);
|
||||
const client = namespace ? await indexClient.namespace(namespace) : indexClient;
|
||||
let deleteResponse;
|
||||
if (delete_all && delete_all.toLowerCase() === 'true') {
|
||||
deleteResponse = await client.deleteAll();
|
||||
} else if (filter) {
|
||||
deleteResponse = await client.deleteMany({
|
||||
filter: JSON.parse(filter),
|
||||
});
|
||||
} else {
|
||||
deleteResponse = await client.deleteMany(JSON.parse(id));
|
||||
}
|
||||
|
||||
if (deleteResponse === undefined) {
|
||||
return 'Delete Successful';
|
||||
} else {
|
||||
throw new Error('Delete failed');
|
||||
}
|
||||
} catch (error) {
|
||||
throw new Error(error?.message || 'An unexpected error occurred');
|
||||
}
|
||||
}
|
||||
|
||||
export async function quertVectors(pinecone: Pinecone, options: QueryOptions): Promise<any> {
|
||||
const { index, namespace, top_k, filter, include_values, include_metadata, vectors, sparse_vector } = options;
|
||||
|
||||
if (!index) {
|
||||
throw new Error('Index is required');
|
||||
}
|
||||
|
||||
const pineconeQueryOptions = {
|
||||
topK: Number(top_k),
|
||||
vector: JSON.parse(vectors),
|
||||
...(filter && { filter: JSON.parse(filter) }),
|
||||
...(include_values && { includeValues: include_values.toLowerCase() === 'true' }),
|
||||
...(include_metadata && { includeMetadata: include_metadata.toLowerCase() === 'true' }),
|
||||
...(sparse_vector && { sparseVector: JSON.parse(sparse_vector) }),
|
||||
};
|
||||
|
||||
try {
|
||||
const indexClient = pinecone.index(index);
|
||||
const client = namespace ? await indexClient.namespace(namespace) : indexClient;
|
||||
const queryResponse = await client.query(pineconeQueryOptions);
|
||||
return queryResponse;
|
||||
} catch (error) {
|
||||
throw new Error(error.message);
|
||||
}
|
||||
}
|
||||
44
marketplace/plugins/pinecone/lib/types.ts
Normal file
44
marketplace/plugins/pinecone/lib/types.ts
Normal file
|
|
@ -0,0 +1,44 @@
|
|||
export type SourceOptions = {
|
||||
apiKey: string;
|
||||
};
|
||||
|
||||
// Define the query options based on the available operations in the operations.json file.
|
||||
export type QueryOptions = {
|
||||
operation: Operation;
|
||||
index: string;
|
||||
ids?: string[];
|
||||
vectors?: string;
|
||||
string?: string[];
|
||||
id?: string;
|
||||
values?: number[];
|
||||
sparseValues?: SparseValues;
|
||||
setmetadata?: object;
|
||||
filter?: string;
|
||||
prefix?: string;
|
||||
limit?: number;
|
||||
paginationToken?: string;
|
||||
namespace?: string;
|
||||
delete_all?: string;
|
||||
metadata?: string;
|
||||
sparse_vector?: string;
|
||||
top_k?: string;
|
||||
include_metadata?: string;
|
||||
include_values?: string;
|
||||
};
|
||||
|
||||
// Define a type for sparse vectors used in the "update_vector" operation.
|
||||
export type SparseValues = {
|
||||
indices: number[];
|
||||
values: number[];
|
||||
};
|
||||
|
||||
// Enum for different operations.
|
||||
export enum Operation {
|
||||
GetIndexStats = 'get_index_stats',
|
||||
ListVectorIds = 'list_vector_ids',
|
||||
FetchVectors = 'fetch_vectors',
|
||||
UpsertVectors = 'upsert_vectors',
|
||||
UpdateVector = 'update_vector',
|
||||
DeleteVectors = 'delete_vectors',
|
||||
QueryVectors = 'query_vector',
|
||||
}
|
||||
28
marketplace/plugins/pinecone/package.json
Normal file
28
marketplace/plugins/pinecone/package.json
Normal file
|
|
@ -0,0 +1,28 @@
|
|||
{
|
||||
"name": "@tooljet-marketplace/pinecone",
|
||||
"version": "1.0.0",
|
||||
"main": "dist/index.js",
|
||||
"types": "dist/index.d.ts",
|
||||
"directories": {
|
||||
"lib": "lib",
|
||||
"test": "__tests__"
|
||||
},
|
||||
"files": [
|
||||
"lib"
|
||||
],
|
||||
"scripts": {
|
||||
"test": "echo \"Error: run tests from root\" && exit 1",
|
||||
"build": "ncc build lib/index.ts -o dist",
|
||||
"watch": "ncc build lib/index.ts -o dist --watch"
|
||||
},
|
||||
"homepage": "https://github.com/tooljet/tooljet#readme",
|
||||
"dependencies": {
|
||||
"@grpc/grpc-js": "^1.12.2",
|
||||
"@pinecone-database/pinecone": "^3.0.3",
|
||||
"@tooljet-marketplace/common": "^1.0.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@vercel/ncc": "^0.34.0",
|
||||
"typescript": "^4.7.4"
|
||||
}
|
||||
}
|
||||
11
marketplace/plugins/pinecone/tsconfig.json
Normal file
11
marketplace/plugins/pinecone/tsconfig.json
Normal file
|
|
@ -0,0 +1,11 @@
|
|||
{
|
||||
"extends": "../../tsconfig.json",
|
||||
"compilerOptions": {
|
||||
"outDir": "dist",
|
||||
"rootDir": "lib"
|
||||
},
|
||||
"exclude": [
|
||||
"node_modules",
|
||||
"dist"
|
||||
]
|
||||
}
|
||||
|
|
@ -1,84 +1,84 @@
|
|||
[
|
||||
{
|
||||
"name": "plivo",
|
||||
"description": "Plugin for Plivo APIs",
|
||||
"version": "1.0.0",
|
||||
"id": "plivo",
|
||||
"author": "Tooljet",
|
||||
"timestamp": "Thu, 02 Mar 2023 10:40:06 GMT"
|
||||
},
|
||||
{
|
||||
"name": "GitHub",
|
||||
"description": "Plugin for GitHub APIs",
|
||||
"version": "1.0.0",
|
||||
"id": "github",
|
||||
"author": "Tooljet",
|
||||
"timestamp": "Thu, 02 Mar 2023 11:52:32 GMT"
|
||||
},
|
||||
{
|
||||
"name": "OpenAI",
|
||||
"description": "Plugin for OpenAI APIs",
|
||||
"version": "1.0.0",
|
||||
"id": "openai",
|
||||
"author": "Tooljet",
|
||||
"timestamp": "Mon, 10 Apr 2023 06:33:21 GMT"
|
||||
},
|
||||
{
|
||||
"name": "AWS Textract",
|
||||
"description": "Plugin for AWS Textract machine-learning service",
|
||||
"version": "1.0.0",
|
||||
"id": "textract",
|
||||
"author": "Tooljet",
|
||||
"timestamp": "Wed, 19 Apr 2023 11:36:56 GMT"
|
||||
},
|
||||
{
|
||||
"name": "HarperDB",
|
||||
"repo": "",
|
||||
"description": "Plugin for HarperDB data source",
|
||||
"version": "1.0.0",
|
||||
"id": "harperdb",
|
||||
"author": "Tooljet",
|
||||
"timestamp": "Thu, 08 Jun 2023 09:50:05 GMT"
|
||||
},
|
||||
{
|
||||
"name": "AWS Redshift",
|
||||
"description": "Plugin for Amazon Redshift data warehouse",
|
||||
"version": "1.0.0",
|
||||
"id": "awsredshift",
|
||||
"author": "Tooljet",
|
||||
"timestamp": "Wed, 17 Jan 2024 20:05:16 GMT"
|
||||
},
|
||||
{
|
||||
"name": "aws-lambda",
|
||||
"description": "Plugin for aws-lambda",
|
||||
"version": "1.0.0",
|
||||
"id": "aws-lambda",
|
||||
"author": "Tooljet",
|
||||
"timestamp": "Tue, 05 Dec 2023 00:27:16 GMT"
|
||||
},
|
||||
{
|
||||
"name": "PocketBase",
|
||||
"description": "API plugin from pocketbase",
|
||||
"version": "1.0.0",
|
||||
"id": "pocketbase",
|
||||
"author": "Tooljet",
|
||||
"timestamp": "Mon, 18 Mar 2024 14:39:34 GMT"
|
||||
},
|
||||
{
|
||||
"name": "Supabase",
|
||||
"description": "Plugin for Supabase",
|
||||
"version": "1.0.0",
|
||||
"id": "supabase",
|
||||
"author": "Tooljet",
|
||||
"timestamp": "Mon, 25 Mar 2024 20:05:16 GMT"
|
||||
},
|
||||
{
|
||||
"name": "Engagespot",
|
||||
"description": "Plugin for engagespot APIs",
|
||||
"version": "1.0.0",
|
||||
"id": "engagespot",
|
||||
"author": "Tooljet",
|
||||
"timestamp": "Thu, 29 Feb 2024 09:46:21 GMT"
|
||||
{
|
||||
"name": "plivo",
|
||||
"description": "Plugin for Plivo APIs",
|
||||
"version": "1.0.0",
|
||||
"id": "plivo",
|
||||
"author": "Tooljet",
|
||||
"timestamp": "Thu, 02 Mar 2023 10:40:06 GMT"
|
||||
},
|
||||
{
|
||||
"name": "GitHub",
|
||||
"description": "Plugin for GitHub APIs",
|
||||
"version": "1.0.0",
|
||||
"id": "github",
|
||||
"author": "Tooljet",
|
||||
"timestamp": "Thu, 02 Mar 2023 11:52:32 GMT"
|
||||
},
|
||||
{
|
||||
"name": "OpenAI",
|
||||
"description": "Plugin for OpenAI APIs",
|
||||
"version": "1.0.0",
|
||||
"id": "openai",
|
||||
"author": "Tooljet",
|
||||
"timestamp": "Mon, 10 Apr 2023 06:33:21 GMT"
|
||||
},
|
||||
{
|
||||
"name": "AWS Textract",
|
||||
"description": "Plugin for AWS Textract machine-learning service",
|
||||
"version": "1.0.0",
|
||||
"id": "textract",
|
||||
"author": "Tooljet",
|
||||
"timestamp": "Wed, 19 Apr 2023 11:36:56 GMT"
|
||||
},
|
||||
{
|
||||
"name": "HarperDB",
|
||||
"repo": "",
|
||||
"description": "Plugin for HarperDB data source",
|
||||
"version": "1.0.0",
|
||||
"id": "harperdb",
|
||||
"author": "Tooljet",
|
||||
"timestamp": "Thu, 08 Jun 2023 09:50:05 GMT"
|
||||
},
|
||||
{
|
||||
"name": "AWS Redshift",
|
||||
"description": "Plugin for Amazon Redshift data warehouse",
|
||||
"version": "1.0.0",
|
||||
"id": "awsredshift",
|
||||
"author": "Tooljet",
|
||||
"timestamp": "Wed, 17 Jan 2024 20:05:16 GMT"
|
||||
},
|
||||
{
|
||||
"name": "aws-lambda",
|
||||
"description": "Plugin for aws-lambda",
|
||||
"version": "1.0.0",
|
||||
"id": "aws-lambda",
|
||||
"author": "Tooljet",
|
||||
"timestamp": "Tue, 05 Dec 2023 00:27:16 GMT"
|
||||
},
|
||||
{
|
||||
"name": "PocketBase",
|
||||
"description": "API plugin from pocketbase",
|
||||
"version": "1.0.0",
|
||||
"id": "pocketbase",
|
||||
"author": "Tooljet",
|
||||
"timestamp": "Mon, 18 Mar 2024 14:39:34 GMT"
|
||||
},
|
||||
{
|
||||
"name": "Supabase",
|
||||
"description": "Plugin for Supabase",
|
||||
"version": "1.0.0",
|
||||
"id": "supabase",
|
||||
"author": "Tooljet",
|
||||
"timestamp": "Mon, 25 Mar 2024 20:05:16 GMT"
|
||||
},
|
||||
{
|
||||
"name": "Engagespot",
|
||||
"description": "Plugin for engagespot APIs",
|
||||
"version": "1.0.0",
|
||||
"id": "engagespot",
|
||||
"author": "Tooljet",
|
||||
"timestamp": "Thu, 29 Feb 2024 09:46:21 GMT"
|
||||
},
|
||||
{
|
||||
"name": "salesforce",
|
||||
|
|
@ -95,5 +95,13 @@
|
|||
"id": "portkey",
|
||||
"author": "Portkey",
|
||||
"timestamp": "Sat, 29 Jun 2024 09:40:13 GMT"
|
||||
},
|
||||
{
|
||||
"name": "pinecone",
|
||||
"description": "api plugin from pinecone",
|
||||
"version": "1.0.0",
|
||||
"id": "pinecone",
|
||||
"author": "Tooljet",
|
||||
"timestamp": "Mon, 28 Oct 2024 08:08:28 GMT"
|
||||
}
|
||||
]
|
||||
Loading…
Reference in a new issue