OpenMetadata/bootstrap/sql/migrations/native/1.12.4/postgres/schemaChanges.sql

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

119 lines
5.5 KiB
MySQL
Raw Normal View History

Mcp oauth (#25391) * Add OAuth MCP * Implement internal OAuth flow for MCP with database persistence This commit implements a redirect-free OAuth flow for the OpenMetadata MCP server that uses stored connector OAuth credentials internally, eliminating the need for external browser redirects. Key Features: - Internal OAuth authorization using stored connector credentials - Database persistence of OAuth tokens (survives container restarts) - Automatic token refresh when expired - PKCE support for authorization code flow - OAuth discovery metadata endpoint (RFC 8414) How It Works: 1. Admin performs one-time OAuth setup via /api/v1/mcp/oauth/setup 2. OAuth credentials (access token, refresh token) stored encrypted in database 3. MCP clients connect without browser - server uses stored credentials internally 4. Expired tokens automatically refreshed and re-persisted to database Tested With: - Snowflake OAuth (session:role:PUBLIC scope) - Container restart verification (credentials persist) - Automatic token refresh verification * feat: Add MCP OAuth database persistence with repositories and DAOs - Implement OAuthClientRepository, OAuthTokenRepository, OAuthAuthorizationCodeRepository - Add DAO methods in CollectionDAO for OAuth entities - Create database migration scripts for OAuth tables (oauth_client, oauth_access_token, oauth_refresh_token, oauth_authorization_code) - Add Fernet encryption for tokens and client secrets - Implement SHA-256 hashing for token lookups - Add OAuth connector plugin system (Snowflake, Databricks) - Add scope authorization and validation - Update ConnectorOAuthProvider to use database persistence - Add comprehensive tests for OAuth provider * Add MySQL migration for MCP OAuth tables (v1.12.1) - Create oauth_client, oauth_authorization_code, oauth_access_token, oauth_refresh_token tables - Convert Postgres schema to MySQL syntax - Add indexes for performance optimization - Tables manually applied in this session, migration framework integration needed * feat: Complete MCP OAuth implementation with critical fixes and MCP Inspector support 1. **Scope Validation Fix** - Set validScopes to null in McpServer to skip validation for connector-based OAuth - Modified RegistrationHandler to skip validation if validScopes is empty - Fixes: Client registration error "Invalid scope: api://apiId/.default" 2. **Metadata Endpoint URLs** - Fixed all OAuth discovery endpoints to include /mcp prefix - Updated OAuthHttpStatelessServerTransportProvider endpoint construction - Ensures proper OAuth metadata discovery 3. **Token Exchange Security** - Added client_id validation during token exchange - Added redirect_uri validation to prevent security vulnerabilities - Load authorization code from database for validation - Prevents authorization code interception attacks 4. **Time Unit Consistency** - Fixed deleteExpired methods to use seconds instead of milliseconds - Updated OAuthTokenRepository and OAuthAuthorizationCodeRepository - Enables proper cleanup of expired tokens and codes 5. **Authorization Code Loading** - Fixed loadAuthorizationCode to load all fields from database - Populates AuthorizationCode object with clientId, redirectUri, codeChallenge - Resolves: NullPointerException during token validation 6. **Connector Name Parameter Support** - Added connectorName field to AuthorizationParams - Extract connector_name from HTTP request in AuthorizationHandler - Priority: connector_name parameter > state (if not random hash) > default 7. **Default Connector Fallback** - Detect random hash in state parameter (64 hex chars for CSRF) - Default to test-snowflake-mcp connector for MCP Inspector testing - Enables MCP Inspector to work without manual URL modification 8. **MySQL Migration** - Added MySQL schema changes for OAuth tables - Matches PostgreSQL schema structure - Tables: oauth_clients, oauth_authorization_codes, oauth_access_tokens, oauth_refresh_tokens 9. **Documentation Cleanup** - Removed 12+ redundant and outdated documentation files - Created single comprehensive MCP_OAUTH_IMPLEMENTATION.md - Added .shell-fix-note for shell script compatibility guidance 10. **Test Script Organization** - Organized test scripts into scripts/mcp-oauth-tests/ - Added test-default-connector.sh for testing with MCP Inspector - Preserved all OAuth flow testing scripts - McpServer.java - Disabled scope validation for connector OAuth - RegistrationHandler.java - Skip empty validScopes - AuthorizationHandler.java - Extract connector_name parameter - AuthorizationParams.java - Added connectorName field - ConnectorOAuthProvider.java - Default connector logic, loadAuthorizationCode fix - OAuthHttpStatelessServerTransportProvider.java - Fixed endpoints, added validations - OAuthTokenRepository.java - Fixed time unit to seconds - OAuthAuthorizationCodeRepository.java - Fixed time unit to seconds - CollectionDAO.java - OAuth DAO registration - DatabaseServiceRepository.java - Database service queries - OAuthRecords.java - Database record types - Deleted: 15+ outdated documentation files - Deleted: Unused auth provider (OpenMetadataAuthProvider.java) - Deleted: Unused OAuth callback servlet - Added: Single comprehensive documentation file ✅ OAuth flow working end-to-end ✅ Client registration, authorization, token exchange successful ✅ Database persistence for all OAuth entities ✅ MCP Inspector compatibility with default connector ✅ Snowflake OAuth credentials configured for testing ⚠️ MCP Inspector SSE connection error (under investigation) - OAuth authentication completes successfully - Issue is with MCP protocol SSE connection, not OAuth Run MCP Inspector: ```bash npx @modelcontextprotocol/inspector http://localhost:8585/mcp ``` Test with default connector: ```bash ./test-default-connector.sh ``` 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: Add CORS preflight support and security fixes for MCP OAuth ## CORS Fix Allow OPTIONS requests without authentication in McpAuthFilter to support CORS preflight checks from web-based MCP clients. This enables proper CORS flow: 1. Browser sends OPTIONS preflight 2. Server responds with CORS headers (200 OK) 3. Browser sends actual POST request with Authorization header 4. Server authenticates and processes request Without this fix, OPTIONS requests were blocked with 401, preventing web clients from connecting to MCP endpoints. ## Security Fixes ### Critical Security Issues Fixed: 1. **Sensitive Token Logging** (95% severity) - Sanitize OAuth request parameters before logging - Remove client_secret, code, code_verifier, refresh_token, access_token from logs - Prevents credential leakage in log files 2. **Token Expiry Integer Overflow** (100% severity) - Changed all expiry timestamps from int/Integer to long/Long - Fixes 2038 problem (32-bit timestamp overflow) - Updated: AccessToken, RefreshToken, AuthorizationCode, ConnectorOAuthProvider, OAuthTokenRepository 3. **Hardcoded Default Connector** (80% severity) - Made default connector configurable via MCP_DEFAULT_CONNECTOR env var - Defaults to null in production (requires explicit connector_name) - Prevents unauthorized access to test credentials in production 4. **Missing Null Checks** (85% severity) - Added validation for token refresh response fields - Validates access_token and expires_in exist before use - Added bounds checking for expires_in (max 1 year) 5. **Missing Input Validation** (75% severity) - Added connector name format validation - Only allows: a-z, A-Z, 0-9, _, - characters - Prevents path traversal and injection attacks ## Documentation - Moved MCP docs to organized structure: openmetadata-mcp/docs/ - Created openmetadata-mcp/README.md with foundation documentation - Moved implementation guide and testing guide to docs/ directory ## Cleanup - Removed development test scripts (scripts/mcp-oauth-tests/) - Removed .shell-fix-note and test-default-connector.sh - Kept only clean final test script: test-mcp-with-token.sh Changes: - openmetadata-mcp/src/main/java/org/openmetadata/mcp/McpAuthFilter.java: OPTIONS CORS support - openmetadata-mcp/src/main/java/org/openmetadata/mcp/server/transport/OAuthHttpStatelessServerTransportProvider.java: Sanitized logging - openmetadata-mcp/src/main/java/org/openmetadata/mcp/server/auth/provider/ConnectorOAuthProvider.java: Multiple security fixes - openmetadata-mcp/src/main/java/org/openmetadata/mcp/McpServer.java: Configurable default connector - openmetadata-mcp/src/main/java/org/openmetadata/mcp/auth/*.java: Long timestamps - openmetadata-mcp/src/main/java/org/openmetadata/mcp/server/auth/repository/OAuthTokenRepository.java: Long timestamps Testing: - OAuth flow: ✅ Working with any OAuth-enabled connector - MCP protocol: ✅ Working via HTTP POST with JWT - Default connector: Configurable via MCP_DEFAULT_CONNECTOR env var - General solution: Works with ANY connector with OAuth credentials Test command: export MCP_DEFAULT_CONNECTOR=test-snowflake-mcp # For testing only ./test-mcp-with-token.sh * feat: MCP OAuth security hardening and production readiness Implemented security improvements and production configuration for MCP OAuth: - Added constant-time secret comparison to prevent timing attacks - Implemented token logging sanitization to protect sensitive credentials - Fixed timestamp overflow (Integer → Long) to prevent 2038 issues - Added input validation for connector names - Implemented HttpClient resource cleanup (AutoCloseable) - Added token refresh response validation with null checks - Replaced hardcoded base URL with dynamic SystemRepository configuration - Fixed MCP Inspector compatibility (removed unimplemented logging capability) - Added example credential files and test setup documentation - Removed commented code and unused files for cleaner codebase Security TODOs documented for future work: - Race condition in authorization code exchange (requires DB schema changes) - Rate limiting for OAuth endpoints (requires new infrastructure) Testing: - All changes tested with Snowflake OAuth connector - MCP Inspector connection verified working - Code formatted with spotless Breaking Changes: None * fix: Address security vulnerabilities from code review bots Implemented fixes based on automated code review bot findings: **Critical:** - SSRF prevention: Added URL validation in OAuthSetupHandler to block private IPs and validate schemes - ThreadLocal leak: Added try-finally cleanup in doGet() to prevent auth context leakage **High:** - Removed hardcoded JWT tokens and client secrets (replaced with dynamic UUIDs) - Added warning logs for missing connector names to improve auditability Security impact: Prevents internal network access, credential exposure, and auth state leakage. Testing: All changes formatted with spotless and validated. * fix: Optimize SSRF prevention per code review bot recommendations Improved SSRF mitigation based on detailed bot feedback: **Optimization:** - Refactored validateTokenEndpoint() → validateAndResolveTokenEndpoint() - Returns validated URI object to avoid double parsing - Integrates endpoint resolution and validation in single method - Reuses URI throughout method to prevent inconsistencies **Implementation Details:** - Validates URL scheme, host, and IP ranges - Blocks private IPs (10.x, 192.168.x, 172.16-31.x) - Blocks link-local addresses (169.254.x) - Validates before HTTP request and credential storage **Benefits:** - More efficient (single URI parse instead of two) - Safer (validated URI reused consistently) - Cleaner code (DRY principle) Based on GitHub Copilot autofix suggestion for SSRF vulnerability. * fix(mcp-oauth): Critical security fixes per code review bots - SSRF: Add DNS resolution and validate all resolved IPs for token endpoints - Race condition: Atomic authorization code exchange prevents replay attacks - Refresh token: Fix expiry check using ofEpochSecond instead of ofEpochMilli - Remove unrelated ingestion yaml files from PR Addresses: CodeQL, Copilot Autofix, Gitar bot feedback * fix(mcp-oauth): Address bot feedback - security and code quality - Remove shell scripts with hardcoded JWT tokens from PR (added to .gitignore) - Fix admin fallback: Use ingestion-bot instead of admin for security - Fix connector name validation: Fail refresh if connector name missing - Add TODO comments for hardcoded localhost URIs (requires MCPConfiguration wiring) Addresses bot feedback on security concerns and configuration flexibility * fix: SSRF - reconstruct URI from validated components * fix: CodeQL suppression, Y2038 bug, test provider safeguards * MCP OAuth: implement CORS development mode detection and token cleanup scheduler - Add development mode detection for CORS origins based on baseUrl - Development: allow localhost origins with warning - Production: empty allowedOrigins (same-origin only) with warning - Implement OAuth token cleanup scheduler with Quartz - OAuthTokenCleanupJob: deletes expired tokens and auth codes - OAuthTokenCleanupScheduler: runs cleanup hourly - Prevents unbounded token table growth * fix: SSRF with allowlist and rate limiting Use allowlist for OAuth endpoints, add rate limiting (10/5 req/min) * fix: SSRF, OAuth security, and MySQL schema bugs - SSRF: Remove user-provided tokenEndpoint, always infer from connector config using allowlist - Schema: Fix MySQL table names (plural), authorization codes schema, add missing tables - OAuth: Restore session redirect URI and re-enable nonce validation * fix: Duplicate clientId variable and missing user_name column in Postgres migration * security: Remove sensitive OAuth tokens and authorization codes from log statements * security: Remove sensitive client metadata from registration logs * chore: Remove connector OAuth infrastructure for user SSO implementation * feat: Add MCP user SSO OAuth MVP implementation - Updated database schema (MySQL + PostgreSQL) to use user_name instead of connector_name - Removed connector OAuth infrastructure (plugins, ConnectorOAuthProvider) - Created UserSSOOAuthProvider MVP skeleton with TODO markers - Added comprehensive IMPLEMENTATION_TODO.md tracking all remaining work - Added QUICK_START.md guide for setup instructions - Added Claude Desktop configuration example - Maintained backward compatibility with PAT authentication See openmetadata-mcp/docs/IMPLEMENTATION_TODO.md for complete implementation checklist * feat: Complete MCP OAuth SSO flow with database-backed state persistence This commit implements a robust OAuth SSO flow for MCP server integration that survives cross-domain redirects during SSO authentication (Google, etc). Key changes: - Add mcp_pending_auth_requests table for database-backed state storage - Add McpPendingAuthRequestRepository for managing pending auth requests - Add SSOCallbackServlet to handle SSO provider callbacks - Add handleDirectIdTokenFlow for already-authenticated users (pac4j token flow) - Add HtmlTemplates for secure error pages with XSS protection - Add Claude Desktop OAuth bridge script for stdio transport integration - Fix OIDC_CREDENTIAL_PROFILE constant shadowing issue - Fix Postgres schema references to non-existent connector_name column - Restore pac4j session attributes (State, Nonce, CodeVerifier) correctly The solution stores OAuth state in the database instead of HTTP sessions, which fail across cross-domain redirects due to SameSite cookie policy. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: Critical OAuth security fixes - thread safety, URL encoding, JWT validation, PKCE validation * fix: Complete ThreadLocal migration for currentRequest.getSession() * feat: Add development bypass for PKCE validation to enable local testing * feat: Add OAuth support with ID token validation, refresh tokens, and security fixes - Add JWKS-based ID token signature validation - Implement refresh token generation and exchange with rotation - Add redirect URI validation to prevent open redirect attacks - Fix clock skew logic and time unit consistency - Add comprehensive test coverage (15 tests) * fix: Critical OAuth security fixes - client validation, redirect URI validation, error handling, Fernet decryption - Add client ID validation in token exchange (prevents authorization code theft) - Add redirect URI validation in token exchange (RFC 6749 Section 4.1.3) - Fix time unit inconsistency in OAuthAuthorizationCodeRepository - Improve error handling to distinguish replay attacks from expired codes - Add user status validation in refresh token exchange - Fix session regeneration to prevent session fixation attacks - Add username/email validation in SSO callback handlers - Improve Fernet decryption error handling for key rotation scenarios All tests passing (15/15) * fix: Clean up pom.xml - fix malformed dependency and remove duplicate dropwizard-jersey * javacheck style fix * fix: Addressing issues raised by Gitar code review * fix: Merge McpAuthFilter changes - add impersonation support while preserving OAuth endpoints * docs: Add comprehensive README for MCP OAuth implementation * feat: Add MCP OAuth dynamic client registration * feat: Add OAuth token revocation endpoint (RFC 7009) * fix: OAuth basic auth flow - auto-redirect with code and optional scope enforcement * feat: Match MCP auth page design to OpenMetadata signin UI * fix: Support separate callback URLs for MCP OAuth and web login flows * feat: Add OAuth scope enforcement, domain validation and session handling for MCP Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * feat: Improve MCP OAuth login UI and add TODO for success page Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: MCP OAuth cleanup - security fixes, remove redundant scope system, improve error handling - Fix timing attacks in CSRF and PKCE validation using MessageDigest.isEqual() - Remove redundant @RequireScope system (OpenMetadata Authorizer handles permissions) - Make OAuth scopes provider-aware (Google/Okta/Azure) - Add baseUrl config to MCPConfiguration for cluster deployments - Delete duplicate RootOAuthEndpointsResource (handled by OAuthWellKnownFilter) - Fix silent failures: propagate errors instead of returning null/200 - Downgrade excessive logging to DEBUG level Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Update generated TypeScript types * fix: Move OAuth migrations from 1.12.1 to 1.12.0 - Consolidate OAuth schema tables into 1.12.0 migration - Add Snowflake backward compatibility migration to 1.12.0 - Remove empty 1.12.1 migration folder - Update README with security enhancements and permission model Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: critical OAuth security and reliability issues Fix ThreadLocal leak, atomic token rotation, PKCE validation, fail-closed error handling, and password sanitization * fix: URL encode authorization code * fix: MCP OAuth stateless transport compatibility and SSO initialization reliability * feat: Add MCP configuration to database settings system - Create mcpConfiguration.json schema for MCP-specific settings - Add MCP_CONFIGURATION to SettingsType enum - Add MCP configuration bootstrap logic to SettingsCache - Extend SecurityConfigurationManager with MCP config support - Add mcpConfiguration field to OpenMetadataApplicationConfig - Update MCPConfiguration.java with timeout settings and comments * feat: Complete McpServer dynamic configuration resolution - Add getBaseUrlFromConfig() to read from SecurityConfigurationManager with fallback - Add getAllowedOriginsFromConfig() for database-backed CORS configuration - Remove hardcoded baseUrl and CORS origins initialization - Remove System.setProperty for HTTP timeouts (will be handled per-request) - Fix SSO handler to use dynamic resolution via getInstance() - Fix NoSuchAlgorithmException import in UserSSOOAuthProvider - All configuration now comes from database via SecurityConfigurationManager * Update generated TypeScript types * feat: Add database-backed MCP configuration with dynamic reload - Add GET/PUT /api/v1/system/mcp/config API endpoints for MCP configuration management - Refactor SSOCallbackServlet to read claims/domains/validators dynamically from SecurityConfigurationManager - Add configuration reload support to OAuthHttpStatelessServerTransportProvider (volatile allowedOrigins, updateAllowedOrigins method) - Implement ConfigurationChangeListener pattern in SecurityConfigurationManager for component notification - Add HTTP timeout configuration (connectTimeout/readTimeout) to AuthenticationCodeFlowHandler from MCP config - All configuration stored in open_metadata_settings table with SecurityConfigurationManager as single source of truth * fix: Add volatile config fields, CopyOnWriteArrayList, null checks, and correct HTTP timeout properties * Remove hardcoded OAuth credentials and unrelated Snowflake migration * Fix HTTP timeout system properties and session regeneration null check * Implement cluster polling, DB-first loading, listener pattern, and fix race conditions * added unit tests * removed connector OAuth code * updated readme * fix: MCP OAuth cleanup — security fixes, migration move, and code quality - Move OAuth SQL migrations from 1.12.0 to 1.12.1 (release target) - Fix XSS in auth error page (no longer reflects exception messages into HTML) - Fix CSRF bypass in state validation (throw instead of return-after-write) - Fix token expiration check in BearerAuthenticator (millis vs seconds mismatch) - Require S256 code_challenge_method explicitly (reject null/plain) - Fix GetLineageTool: use VIEW_BASIC auth, add input validation, use singleton LineageRepository - Rename SESSION_GOOGLE_CALLBACK_URL to SESSION_SSO_CALLBACK_URL (provider-agnostic) - Remove 10-second config polling from SecurityConfigurationManager (use SettingsCache TTL) - Remove unnecessary synchronized on volatile field getters - Downgrade verbose LOG.info calls to LOG.debug (session state, admin principals, tokens) - Fix FQN imports in AuthenticationCodeFlowHandler (MCPConfiguration, Role) - URL-encode redirect parameters (id_token, email, name) - Remove invalid "default": null from defaultOAuthRole JSON schema - Add error logging in AuthorizationHandler.exceptionally() block Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * add TODOs for unfixed security review findings * fixed critical review issues: added client_secret validation, registration rate limiting, session regeneration bug, exact path matching, dead code removal * fixed auth filter 500→401 for invalid tokens, exact path matching in transport provider * added revocation client auth, redirect URI scheme validation, ID token validation in SSO flow, rate limiter race fix, downgraded PII logging to DEBUG * fix MCP config loading to use getSettingOrDefault, cache IdTokenValidator * google sso login working here * add basic auth login flow for MCP OAuth, fix web UI redirect_uri_mismatch * revert cosmetic UI formatting changes accidentally introduced in merge * fix CodeQL info exposure and GitarBot security findings: redirect_uri validation, pac4j race condition * harden MCP OAuth: fix error handling, remove dead code, prevent info leaks * remove dead code and harden MCP OAuth: delete 5 unused files, inline metadata handlers, add PKCE validation, fix error handling * fix GitarBot findings: restrict HTTP redirects to loopback, add token rate limiting, restore GET 405, deny-all CORS fallback, reduce JWK cache TTL * fix Azure SSO: always register callback servlet, use baseUrl for token exchange, show success page * security hardening: early user check, ID token audience validation, token rotation, shorter JWT TTL * LDAP support, allow native app redirect schemes, tolerate unknown registration fields * fix open redirect in MCP callback detection, check auth code expiry before consumption, warn on fallback baseUrl * null safety for PKCE, grant_type, and refresh_token params in token endpoint * fix RevocationHandler test exception type mismatch * add registration metadata length validation, fix loopback host check * fix MCP OAuth SSO callback for Okta: use registered redirect_uri, fix pac4j session attribute names, forward /callback to /mcp/callback * fix missing return in MCP callback error path, skip SSO registration for basic/ldap, improve comment * MCP OAuth security hardening: bcrypt secrets, atomic CAS rotation, XFF rate limiting, review fixes * fix XFF rate-limit bypass: validate IP format, cap map size to prevent heap exhaustion * move MCP OAuth migrations from 1.12.2 to 1.12.3, remove unused oauth_audit_log table, simplify * fix client_secret_basic removal, MySQL index idempotency, token auto-delete on decrypt failure * Update generated TypeScript types * Update generated TypeScript types * fix impersonation compatibility after McpAuthFilter deletion * hash authorization codes with SHA-256 before storing in DB --------- Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com> Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2026-03-19 03:03:25 +00:00
-- MCP OAuth 2.0 tables
CREATE TABLE IF NOT EXISTS oauth_clients (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
client_id VARCHAR(255) UNIQUE NOT NULL,
client_secret_encrypted TEXT,
client_name VARCHAR(255),
redirect_uris JSONB NOT NULL DEFAULT '[]'::jsonb,
grant_types JSONB NOT NULL DEFAULT '["authorization_code", "refresh_token"]'::jsonb,
token_endpoint_auth_method VARCHAR(50) NOT NULL DEFAULT 'client_secret_post',
scopes JSONB NOT NULL DEFAULT '["read", "write"]'::jsonb,
created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
CONSTRAINT oauth_clients_client_id_check CHECK (char_length(client_id) > 0)
);
CREATE INDEX IF NOT EXISTS idx_oauth_clients_client_id ON oauth_clients(client_id);
CREATE TABLE IF NOT EXISTS oauth_authorization_codes (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
code VARCHAR(512) UNIQUE NOT NULL,
client_id VARCHAR(255) NOT NULL,
user_name VARCHAR(255) NOT NULL,
code_challenge VARCHAR(255),
code_challenge_method VARCHAR(10),
redirect_uri TEXT NOT NULL,
scopes JSONB NOT NULL DEFAULT '[]'::jsonb,
expires_at BIGINT NOT NULL,
used BOOLEAN NOT NULL DEFAULT FALSE,
created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
CONSTRAINT oauth_authorization_codes_fk_client
FOREIGN KEY (client_id) REFERENCES oauth_clients(client_id) ON DELETE CASCADE,
CONSTRAINT oauth_authorization_codes_expires_check CHECK (expires_at > 0),
CONSTRAINT oauth_authorization_codes_challenge_method_check CHECK (
code_challenge_method IS NULL OR code_challenge_method IN ('S256')
)
);
CREATE INDEX IF NOT EXISTS idx_oauth_authz_codes_code ON oauth_authorization_codes(code);
CREATE INDEX IF NOT EXISTS idx_oauth_authz_codes_client_id ON oauth_authorization_codes(client_id);
CREATE INDEX IF NOT EXISTS idx_oauth_authz_codes_expires_at ON oauth_authorization_codes(expires_at);
CREATE TABLE IF NOT EXISTS oauth_access_tokens (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
token_hash VARCHAR(255) UNIQUE NOT NULL,
access_token_encrypted TEXT NOT NULL,
client_id VARCHAR(255) NOT NULL,
user_name VARCHAR(255) NOT NULL,
scopes JSONB NOT NULL DEFAULT '[]'::jsonb,
expires_at BIGINT NOT NULL,
created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
CONSTRAINT oauth_access_tokens_fk_client
FOREIGN KEY (client_id) REFERENCES oauth_clients(client_id) ON DELETE CASCADE,
CONSTRAINT oauth_access_tokens_expires_check CHECK (expires_at > 0)
);
CREATE INDEX IF NOT EXISTS idx_oauth_access_tokens_hash ON oauth_access_tokens(token_hash);
CREATE INDEX IF NOT EXISTS idx_oauth_access_tokens_client_id ON oauth_access_tokens(client_id);
CREATE INDEX IF NOT EXISTS idx_oauth_access_tokens_expires_at ON oauth_access_tokens(expires_at);
CREATE TABLE IF NOT EXISTS oauth_refresh_tokens (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
token_hash VARCHAR(255) UNIQUE NOT NULL,
refresh_token_encrypted TEXT NOT NULL,
client_id VARCHAR(255) NOT NULL,
user_name VARCHAR(255) NOT NULL,
scopes JSONB NOT NULL DEFAULT '[]'::jsonb,
expires_at BIGINT NOT NULL,
revoked BOOLEAN NOT NULL DEFAULT FALSE,
created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
CONSTRAINT oauth_refresh_tokens_fk_client
FOREIGN KEY (client_id) REFERENCES oauth_clients(client_id) ON DELETE CASCADE,
CONSTRAINT oauth_refresh_tokens_expires_check CHECK (expires_at > 0)
);
CREATE INDEX IF NOT EXISTS idx_oauth_refresh_tokens_hash ON oauth_refresh_tokens(token_hash);
CREATE INDEX IF NOT EXISTS idx_oauth_refresh_tokens_client_id ON oauth_refresh_tokens(client_id);
CREATE INDEX IF NOT EXISTS idx_oauth_refresh_tokens_revoked ON oauth_refresh_tokens(revoked);
CREATE INDEX IF NOT EXISTS idx_oauth_refresh_tokens_expires_at ON oauth_refresh_tokens(expires_at);
CREATE TABLE IF NOT EXISTS mcp_pending_auth_requests (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
auth_request_id VARCHAR(64) UNIQUE NOT NULL,
client_id VARCHAR(255) NOT NULL,
code_challenge VARCHAR(255) NOT NULL,
code_challenge_method VARCHAR(10) NOT NULL DEFAULT 'S256',
redirect_uri TEXT NOT NULL,
mcp_state VARCHAR(255),
scopes JSONB,
pac4j_state VARCHAR(64),
pac4j_nonce VARCHAR(255),
pac4j_code_verifier VARCHAR(255),
expires_at BIGINT NOT NULL,
created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX IF NOT EXISTS idx_mcp_pending_auth_request_id ON mcp_pending_auth_requests(auth_request_id);
CREATE INDEX IF NOT EXISTS idx_mcp_pending_auth_expires ON mcp_pending_auth_requests(expires_at);
CREATE INDEX IF NOT EXISTS idx_mcp_pending_auth_pac4j_state ON mcp_pending_auth_requests(pac4j_state);
Continuous indexing to handle failures (#26111) * Add Continuous Indexing * Add continuous Search indexing * Update to 1.12.3 * Make search index retry queue reliable with stale recovery, health checks, and silent failure coverage - Add entityType, retryCount, claimedAt columns to search_index_retry_queue table - Implement stale IN_PROGRESS recovery (10min threshold, 60s sweep interval) - Replace static isClientAvailable flag with cached ping health check (5s TTL) - Narrow catch blocks in resolveById/resolveByFqn to EntityNotFoundException - Use entityType hint for O(1) entity resolution instead of scanning all types - Switch from status-string-based retry to retryCount-based (< 3 retries → PENDING, ≥ 3 → FAILED) - Batch cascade reindex at 200 entities instead of accumulating up to 5000 - Add retry queue enqueue in catch blocks of createTimeSeriesEntity, updateTimeSeriesEntity, deleteTimeSeriesEntityById, bulkIndexPipelineExecutions, reindexAcrossIndices, and TestSuiteRepository.postCreate - Re-throw exceptions from indexTableColumns/deleteTableColumns to parent catch blocks - Add Micrometer counters for enqueued, processed (success/failure), and stale recovered * Add missing lineage call site and Add test * Review comments * Add resilience to search index retry worker: client availability checks, backoff, and error classification - Add exponential backoff when search client is unreachable so the worker does not burn retries during cluster outages (5s → 10s → … → 60s cap) - Classify errors using HTTP status codes from ES/OS exceptions: 4xx (except 429) are non-retryable and skip straight to FAILED; 429, 5xx, and IOException are retryable - Preserve first bulk failure detail in RuntimeException so error classification works for the bulk indexing path - Reorganize SearchIndexRetryWorker into clearly separated sections (lifecycle, main loop, record processing, entity resolution, reindexing, resilience, suspension, utilities) - Add isRetryableStatusCode utility to SearchIndexRetryQueue - Add integration tests: status code classification, retry exhaustion to FAILED, recovery from PENDING_RETRY_1, error detail preservation * Address review comments * Revert fqn size * Spotless * Address volatile review comments * Fix Failing Test * update review comments --------- Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com> Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
2026-03-18 10:53:04 +00:00
-- Incremental Search Retry Queue
-- Stores failed live-indexing operations for async background catch-up.
-- Keep this table intentionally minimal: entityId, entityFqn, failureReason, status.
CREATE TABLE IF NOT EXISTS search_index_retry_queue (
entityId VARCHAR(36) NOT NULL DEFAULT '',
entityFqn VARCHAR(1024) NOT NULL DEFAULT '',
failureReason TEXT,
status VARCHAR(32) NOT NULL DEFAULT 'PENDING',
entityType VARCHAR(256) NOT NULL DEFAULT '',
retryCount INT NOT NULL DEFAULT 0,
claimedAt TIMESTAMP NULL DEFAULT NULL,
PRIMARY KEY (entityId, entityFqn)
);
CREATE INDEX IF NOT EXISTS idx_search_index_retry_queue_status
ON search_index_retry_queue(status);
CREATE INDEX IF NOT EXISTS idx_search_index_retry_queue_claimed
ON search_index_retry_queue(claimedAt);