mirror of
https://github.com/ToolJet/ToolJet
synced 2026-04-24 15:07:23 +00:00
5 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Akshay
|
8c9d79c012
|
Fix: Use loadSync for gRPC filesystem proto loading and add lightweight service discovery (#15225)
* fix(grpcv2): use loadSync for filesystem proto loading to prevent server crash protobufjs has an unfixed bug (protobufjs/protobuf.js#1098) where async Root.load() calls resolveAll() outside its try-catch in the finish() callback. When resolveAll() throws (e.g. unresolvable types), the error becomes an uncaught exception that crashes the Node.js process — the Promise never resolves/rejects. Switch from protoLoader.load() to protoLoader.loadSync() for all filesystem-based proto loading. With loadSync, resolveAll() errors propagate as normal synchronous throws caught by existing try-catch blocks. This is consistent with loadProtoFromRemoteUrl() which already uses loadSync. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(grpcv2): add filesystem proto discovery with lightweight scanning Add discoverServiceNames and discoverMethodsForServices to support two-phase service discovery from filesystem proto files. Uses protobufjs.parse() for lightweight name scanning (~30KB/file) and only loads full gRPC definitions for selected services, preventing OOM on large proto directories. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(DynamicSelector): add multi-select and autoFetch for grpcv2 filesystem services Add isMulti and autoFetch props to DynamicSelector. autoFetch triggers service discovery on mount without requiring a manual button click, and skips cache persistence to avoid false "Unsaved Changes" prompts. Multi-select renders services as chips with custom styles. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(data-sources): handle non-array elements in resolveKeyValuePair resolveKeyValuePair assumed all array option elements are sub-arrays (like metadata key-value pairs). Options like selected_services contain plain strings, causing arr.map crash during test connection. Guard with Array.isArray check and fall back to resolveValue for scalar elements. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(grpcv2): require service selection for filesystem mode in query manager Filesystem mode without selected services would fall back to full proto discovery (loading every file), defeating the purpose of the two-phase discovery. Now shows an error asking the user to select services in the datasource config instead. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(grpcv2): require selected services for filesystem test connection Test connection in filesystem mode now validates that at least one service is selected and uses a selected service for the connectivity check instead of picking an arbitrary one from the proto directory. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(grpcv2): simplify filesystem test connection to validate proto parsing Test connection for filesystem mode now just validates that proto files can be parsed and services discovered — no service selection required. Removes the meaningless waitForReady check which only tested TCP connectivity without validating anything about the proto definitions. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor(grpcv2): remove filesystem branch from discoverServices Filesystem mode never flows through discoverServices — it uses the two-phase discoverServiceNames + discoverMethodsForServices path. Remove the dead branch and add a comment documenting the filesystem flow for clarity. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(grpcv2): add comment documenting test connection behavior per mode Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(grpcv2): add TCP connectivity check for filesystem mode test connection Filesystem mode now falls through to checkFirstServiceConnection (waitForReady) like reflection and URL modes, instead of returning early after proto parsing. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * cleanup(DynamicForm): remove unused snake_case prop aliases for isMulti and autoFetch No plugin manifest uses is_multi or auto_fetch — the gRPC v2 manifest (the only consumer of these props) uses camelCase exclusively, and there is no transformation layer in the pipeline. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(DynamicSelector): suppress noAccessError flash during loading The no-access warning and red border briefly flashed on page reload because validateSelectedValue ran with an empty array before the fetch completed. Gate both on !isLoading so they only appear after data is actually loaded. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(DynamicSelector): skip cache validation for autoFetch on unrelated prop changes When autoFetch is enabled, cache is never persisted to options (to avoid "Unsaved Changes"). So every time selectedDataSource changes for unrelated reasons (e.g. title edit), the cache-checking useEffect finds no cache and calls validateSelectedValue([]), falsely triggering the no-access warning. Skip this effect for autoFetch since it has its own dedicated fetch/validation lifecycle. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(grpcv2): remove QueryResult wrapping from plugin invokeMethod returns grpcv2 plugin methods (discoverServiceNames, discoverMethodsForServices) were returning QueryResult-wrapped responses which got double-wrapped by DataSourcesService.invokeMethod, causing GRPCv2Component to crash with "servicesData.services.map is not a function" when opening filesystem proto queries. Plugin invokeMethod now returns raw data (arrays) instead of QueryResult objects. The server's invokeMethod always wraps with { status: 'ok', data } consistently. DynamicSelector adds an Array.isArray guard for plugins that return raw arrays vs { data: [...] }. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(DynamicSelector): skip access validation for autoFetch fields autoFetch fields (e.g. gRPC services) are discovered from proto files, not OAuth-scoped resources — "no access" warnings don't apply. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor(grpcv2): unify service discovery into single getServiceDefinitions entry point Consolidate discoverServices, discoverServiceNames, and discoverMethodsForServices into two clear methods: listServices (lightweight name scan for DS config) and getServiceDefinitions (full method discovery for query editor, all modes). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(grpcv2): prevent OOM on filesystem test connection and query fallback Filesystem test connection no longer parses proto files — just counts them with fast-glob and checks TCP connectivity via a raw gRPC client. Query execution fallback after server restart now uses the lightweight protobufjs.parse() scanner instead of the heavy protoLoader.loadSync() path. Removes two dead functions (discoverServicesIndividually, discoverServicesUsingFilesystem). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: update version to 3.20.95-lts across all components --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: gsmithun4 <gsmithun4@gmail.com> |
||
|
Akshay
|
ca71355d8e
|
Feature[LTS]: GRPC Plugin revamp (#13894)
* feat: Add new gRPC v2 plugin - Enhanced DataSourcesController to support invoking methods on data sources. - Introduced InvokeDataSourceMethodDto for method invocation requests. - Added WhitelistPluginGuard to restrict method invocation based on data source kind. - Updated IDataSourcesController and IDataSourcesService interfaces to include invoke method functionality. - Implemented invokeMethod in DataSourcesService to handle method calls for gRPC v2. - Added method whitelisting for gRPC v2 in the service layer. * refactor: invokeMethod to use method dispatch pattern * fix: improve error handling in testConnection method * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * refactor: update types and interface * fix: remove unnecessary border styles for improved UI consistency * refactor: simplify error handling - Introduced a new SCSS file for styling the gRPC v2 query editor components. - Styled request sections, server URL input, dropdown menus, and method buttons for better user experience. refactor(grpcv2): enhance error handling and type safety - Updated error handling in gRPC query service to use a new `toError` utility function for consistent error conversion. - Improved type definitions for gRPC client and service methods to ensure better type safety. - Refactored method existence validation and gRPC call execution to handle errors more gracefully. fix(grpcv2): improve service discovery and proto file loading - Enhanced service discovery logic to handle reflection and proto file clients more robustly. - Added checks for valid service constructors and improved error messages for better debugging. - Updated the `loadProtoFromRemoteUrl` function to handle HTTP errors more effectively. chore(grpcv2): add utility functions for type checks - Introduced utility functions `isRecord`, `hasProperty`, and `toError` for better type checking and error handling throughout the codebase. * feat: add LegacyBanner component for gRPC legacy tag display * fix: styling changes * fix: correct text in LegacyBanner component * feat: enhance GRPCv2 component with request handling and state management improvements * refactor: update metadata handling in GRPCv2 component and service * refactor: update GRPCv2 component to use raw_message instead of requestData * fix: Styling issues and labels * refactor: gRPC test connection checks for proto file and url * fix: improve error handling in service discovery --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Muhsin Shah C P
|
638fcf7223
|
[Feature] Now multiple users can login through RestAPI-Oauth 🚀 (#4052)
* Implemented multiple access token feature - working on refresh token case * worked on refresh token flow * added multiple token ability to openapi plugin too - fixed some bugs * changed tokenData to token_data - fixed some issues * added user type with user id * changed user type * Rewrote some function - added switch for enabling and disable multi auth - fixed some bugs - refactored the code * fixed view app issue * Fixed public app issue * cleaning some code * (public_app) add a check to avoid oauth login redirect when there is no access token * reverted all changes of openapi (temporary) - will add multi token functionality once done with restapi * fixed a bug * fixed a bug * refactored some code * changed the switch text * pr changes * changed token_data back to tokenData * cleaning code * removed token data from datasources query * removed some lines * added a comment |
||
|
Maurits Lourens
|
60f515d19c
|
Feature/2395 - add eslint to plugins (#2402)
* merge develop * Add eslint dependencies, configs and scripts to plugins project * run lint with Github action * ignore tests and dist folders * fun eslint with --fix and manual fixes, renamed __tests_ to __tests__ * add plugins packages folder to lint-staged config * fix lint issue |
||
|
Gandharv
|
2f3b441c0a
|
Move plugins to root (#1728)
* feat: move plugins to root * modify tsconfig * add .gitignore * delete old plugins file * add parcel * docker compose volume mount * add gcs * add typescript to plugins folder * gcs to ts * add dynamodb * add elastic search * add firestore * add gsheets * add graphql * add mongodb * mssql * add mysql * add postgresql * add redis * add s3 * add slack * add stripe * remove plugin related packages from pkgjson * add lib folder * add gitignore * remove typescript generated files * remove generated file * remove generated files * add twilio * add dist to docker compose cache binding * add dist prefix * cleanup - 1 * delete dist * rename to index.ts + add jest config * add it.todo in tests * test fixes * test file changes * fix type checks * add @tooljet/plugins to server package json * esm vs commonjs bug, reduce got to 11.8.2 from 12.0.0 * docker file npm package version fix * add typesense * cleaup - 2 * add sendgrid * add lerna build and clean script for all packages + tsconfig * cleanup -3 * add plugins build step * add missing plugins build step in npm run build * add mssql, mysql & postgres as singleton classes * add db connection to cache only if datasourceId is available * client: add data source schema/manifest files * add query operations files * logic for wrapping form with schema * add script to create index file * add @tooljet/plugins to frontend folder * cleanup 1 -frontend * cleanup - 2 // frontend // data queries * add client and index to gitignore * update gitignore * fix lint & test * update ci * fix unit, e2e * cleanup -3 * fix test * fix tests * fix indent * try npm ci * fix tests * fix typo * fix * rename file for server entry * heroku fix * add main and types entry points in pkg json * move common to root * cleanup - 4: remove redundant $ sign prefix * cleanup - 4: remove redundant $ sign prefix * update options in-sync before DOM is painted * change type cloud to cloud storage * update readme * update ci.yml * update ci yml * add pkg-lock.json * rename index.ts to server.ts * update lock files * add server package.lock * remove unused import * revert commit: add minio * add root dep * import server.ts * remove plugins build step * add npm shrinkwrap * update version - plugins * add new version - 0.0.8 * upgrade version * move to symlinked package * add lock file * feat: add icon inside package * add plugin creation docs * Remove seed * move icons to plugins folder * install pg dep * add react to packages * add seed cmd * revert change * add plugins build in lint, e2e, unit * e2e, lint use npm ci * update dockerfile for plugins * try combining release with web * limit memory on release * try executing seed script post transpile * try executing seed from server directory * update seed execution * add minio * add correct type * add minio to pkg json * remove old file * fix provider key * add python installable + npm ^7.2.0 (#1752) * add python installable + npm ^7.2.0 * add py to prod file * pin npm version to 7.20.0 * pin npm version to 7.20.0 * split into multi stage build and remove python for buildx * copy plugins from buider stage * update dependencies * add freetds dependency * update server dockerfile * update client dockerfile * update dev dockerfile and compose file * fix entrypoint * fix server dev dockerfile * update docker-compose * remove npm install on root dir on docker build * fix heroku script * make lerna prod dependency to enable prod builds * remove redundant env setup Co-authored-by: Akshay Sasidharan <akshaysasidharan93@gmail.com> Co-authored-by: navaneeth <navaneethpk@outlook.com> |