OpenMetadata/docker/development/docker-compose.yml

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

608 lines
31 KiB
YAML
Raw Normal View History

# Copyright 2021 Collate
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
version: "3.9"
volumes:
ingestion-volume-dag-airflow:
ingestion-volume-dags:
ingestion-volume-tmp:
es-data:
services:
mysql:
build:
context: ../../.
dockerfile: docker/mysql/Dockerfile_mysql
command: "--sort_buffer_size=10M"
container_name: openmetadata_mysql
restart: always
depends_on:
- elasticsearch
environment:
MYSQL_ROOT_PASSWORD: password
expose:
- 3306
ports:
- "3306:3306"
networks:
- local_app_net
healthcheck:
test: mysql --user=root --password=$$MYSQL_ROOT_PASSWORD --silent --execute "use openmetadata_db"
interval: 15s
timeout: 10s
retries: 10
volumes:
- ./docker-volume/db-data:/var/lib/mysql
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:9.3.0
container_name: openmetadata_elasticsearch
environment:
- discovery.type=single-node
Fix column filtering on Lineage (#25353) * Fix Column Filtering and add path preserve * Preserve only column with matching filter * Add Test * update param * Add UI work * Lanaguage * Add proper translations for column-filter locale keys (#25360) * Initial plan * Add proper translations for column-filter locale keys across all 18 languages Co-authored-by: karanh37 <33024356+karanh37@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: karanh37 <33024356+karanh37@users.noreply.github.com> * fix filtering * Fix ui : Dropdown filters (Domains, Owners, Tag, Tier, Service, etc.) were not showing in the Impact Analysis view and normal lineage view. * put back searchbox for column level * Fix query_filter not working for tag/domain/tier in lineage APIs -> table level filtering * fix: hasNodeLevelFilters bypassing ES filters causing empty results * fix: tag filter incorrectly sent to column_filter on table-level page * Fix Impact Analysis search and filtering with path preservation Summary of changes: - Backend: Path preservation for search, accurate pagination counts, wildcard query parsing, OR logic for name/displayName - Frontend: Column-level search now matches both table names and column names * Table level: Search → query_filter (matches table names) Column level: Search → column_filter only (matches column names) * Fix column impact analysis: depth-aware filtering, tag aggregation, and nested column support * address gitar bot feedback : lineage filter — add service to path preservation, fix OR semantics, rename preserve_paths, guard NPE on fromEntity * fix: use unfiltered depth counts in lineage pagination info, remove 10k doc fetch * fix: Impact Analysis — fix upstream BFS, always run BFS unfiltered and apply query filter as in-memory post-filter to support multi-depth traversal, fix column filter OR-within-type semantics, rename preserve_paths param, and add integration tests instead of passing queryFilter into the BFS (which blocked traversal through non-matching intermediate nodes), we now run BFS with no filter to discover the full graph topology, then apply the filter after all nodes and edges are collected using the existing applyInMemoryFiltersWithPathPreservationForEntityCount. * fix: lineage Impact Analysis — unfiltered BFS with post-filter for multi-depth traversal, upstream BFS direction fix, remove dead ES query column filter code, fix stale useCallback deps, add SDK methods and integration tests * fix: remove column_filter from UI calls where backend doesn't support it (exportAsync, platformLineage, dataQualityLineage, paginationInfo), fix stale useCallback deps in LineageProvider * fix: Impact Analysis — unfiltered BFS for multi-depth filter traversal, upstream direction fix, table/column tag separation, dead code cleanup, stale UI deps, node depth dropdown fix * fix: remove dead columnFilter plumbing from CustomControls, clear column filters on Table mode switch, fix QueryFilterParser search+filter OR logic, add search combo integration tests, log warn on tag fetch failure * fix: depth-based pagination sort * ui: performance optimization — avoids redundant lookups * handle matchesMultipleFiltersWithMetadata * fix: upstream/downstream count not updating in table view * fix UI changes * fix api issues * fix: Impact Analysis — move to ES-native filtering with unfiltered BFS, filtered pagination counts, tag name enrichment * address comments * fix: Impact Analysis — ES-native filtered traversal, batch tag enrichment, depth filtering with filters, SDK entityType support * fix tests * fix failing tests * fix backend test * add tests for code coverage * add tests for code coverage * fix: add id.keyword sub-field to ES index mappings to fix lineage filter dropdowns for topics, dashboards, and other non-table entities * address comments * fix service type filter case * address gitar bot feedback * fix tests * fix build * Fix the bugs * Fix the bugs * Fix all things related to Lineag, Impact Analysis * Update generated TypeScript types * Fix all things related to Lineag, Impact Analysis * Fix Mapping for ids for container and test suite * test: enhance lineage spec to cover all the missing cases (#26796) * test: enhance lineage spec to cover all the missing cases * fix searchIndex mapping * fix tests * added filter spec * fix filter issues * fix lineageSearchSelect * update database service filter tests * iterate over all the entity for service filter * update impact analysis fixes * update tests management * add missing test case * fix tests * fix column level lineage tests * fix apiEndpoint issue * improved lineage connection assertion * fix tests * fix column level linage issues * fix missing import * update test import from pages * fix mlModel spell issue * fix node pagination and right panel spec * refactor lineage tests to improve entity creation and visibility checks * fix license header * fix build * fix tests * fix tests * UI linter fixes * address comments * fix unit tests * remove redundant method * improve tests * fix impact analysis tests * fix impact analysis * Fix Export via Async and add tests * update tests * fix issues * Spotless fix * fix impact analysis * Fix issue with lineage export * Fix serviceType filtering * fix multiple calls issue * fix lint issues * fix uni tests * fix test issues * fix lineage settings spec * fix all the tests * Remove fix me * fix lint issue * fix failing specs --------- Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: karanh37 <33024356+karanh37@users.noreply.github.com> Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com> Co-authored-by: sonika-shah <58761340+sonika-shah@users.noreply.github.com> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> Co-authored-by: Sriharsha Chintalapani <harsha@getcollate.io> Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2026-04-06 16:01:15 +00:00
- ES_JAVA_OPTS=${ES_JAVA_OPTS:--Xms1024m -Xmx1024m}
- xpack.security.enabled=false
networks:
- local_app_net
expose:
- 9200
- 9300
ports:
- "9200:9200"
- "9300:9300"
healthcheck:
test: "curl -s http://localhost:9200/_cluster/health?pretty | grep status | grep -qE 'green|yellow' || exit 1"
interval: 15s
timeout: 10s
retries: 10
volumes:
- es-data:/usr/share/elasticsearch/data
execute-migrate-all:
build:
context: ../../.
dockerfile: docker/development/Dockerfile
container_name: execute_migrate_all
command: "./bootstrap/openmetadata-ops.sh -d migrate --force"
environment:
OPENMETADATA_CLUSTER_NAME: ${OPENMETADATA_CLUSTER_NAME:-openmetadata}
SERVER_PORT: ${SERVER_PORT:-8585}
SERVER_ADMIN_PORT: ${SERVER_ADMIN_PORT:-8586}
LOG_LEVEL: ${LOG_LEVEL:-INFO}
Upgrade to Dropwizard 5x and Jetty 12.1 (#24776) * Add support for Dropwizard 5.0 and Jetty 12.1.x * Dropwizard 5x and Jetty 12.1 upgrade * Fix test behavior * Fix rdf tests * revert enableVirtualThreads * fix tests * Fix Tests * Fix tests * Switch to jersey-jetty-connector for Jetty 12 compatibility - Replace jersey-apache-connector with jersey-jetty-connector - Jersey 3.1.4+ jersey-jetty-connector supports Jetty 12.0.x+ - Use JettyConnectorProvider and JettyHttpClientSupplier for HTTP client - Keep reasonable timeouts (30s connect, 2min read) to prevent CI hangs - Set SYNC_LISTENER_RESPONSE_MAX_SIZE for large responses This fixes the 1,093 InterruptedException test failures caused by using the default Jersey client (HttpURLConnection-based) which doesn't handle concurrent test execution properly. * Fix: Start Jetty HttpClient before use Jetty 12 HttpClient implements LifeCycle and must be explicitly started with httpClient.start() before use. This fixes the 163 InterruptedException test failures. * Fix: Force jetty-client to 12.1.1 for jersey-jetty-connector jersey-jetty-connector brings transitive jetty-client:12.0.22 but Dropwizard 5.0 uses Jetty 12.1.1. The ClientConnector.newTransport() API changed between 12.0.x and 12.1.x, causing NoSuchMethodError. Fix: Exclude transitive jetty-client and add explicit 12.1.x dependency. * Use Java 11+ HttpClient connector for tests (jersey-jnh-connector) Switch from the broken jersey-jetty-connector (incompatible with Jetty 12.1.x) to jersey-jnh-connector which uses Java's built-in java.net.http.HttpClient. This connector: - Natively supports all HTTP methods including PATCH - Works with Java 21 - No external dependencies required - Avoids compatibility issues with Jetty versions * Use Apache HttpClient 5.x connector for tests (jersey-apache5-connector) Switch from jersey-jetty-connector (incompatible with Jetty 12.1.x) to jersey-apache5-connector which uses Apache HttpClient 5.x. This connector: - Supports all HTTP methods including PATCH - Lenient with empty PUT request bodies - Has proper timeout support to prevent indefinite hangs - Works with Jetty 12.1.x * Fix tests * Fix docker compose * Fix tests * Fix tests - make url compatible * Add URL parsing * Fix URL decode * fix tests * fix test * fix tests * Fix integration with new dropwizard-5x changes --------- Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com> Co-authored-by: karanh37 <karanh37@gmail.com> Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
2026-01-12 20:18:29 +00:00
# Migration
MIGRATION_LIMIT_PARAM: ${MIGRATION_LIMIT_PARAM:-1200}
# OpenMetadata Server Authentication Configuration
AUTHORIZER_CLASS_NAME: ${AUTHORIZER_CLASS_NAME:-org.openmetadata.service.security.DefaultAuthorizer}
AUTHORIZER_REQUEST_FILTER: ${AUTHORIZER_REQUEST_FILTER:-org.openmetadata.service.security.JwtFilter}
AUTHORIZER_ADMIN_PRINCIPALS: ${AUTHORIZER_ADMIN_PRINCIPALS:-[admin]}
AUTHORIZER_ALLOWED_REGISTRATION_DOMAIN: ${AUTHORIZER_ALLOWED_REGISTRATION_DOMAIN:-["all"]}
AUTHORIZER_INGESTION_PRINCIPALS: ${AUTHORIZER_INGESTION_PRINCIPALS:-[ingestion-bot]}
AUTHORIZER_PRINCIPAL_DOMAIN: ${AUTHORIZER_PRINCIPAL_DOMAIN:-"open-metadata.org"}
AUTHORIZER_ALLOWED_DOMAINS: ${AUTHORIZER_ALLOWED_DOMAINS:-[]}
AUTHORIZER_ENFORCE_PRINCIPAL_DOMAIN: ${AUTHORIZER_ENFORCE_PRINCIPAL_DOMAIN:-false}
AUTHORIZER_ENABLE_SECURE_SOCKET: ${AUTHORIZER_ENABLE_SECURE_SOCKET:-false}
AUTHENTICATION_PROVIDER: ${AUTHENTICATION_PROVIDER:-basic}
2023-11-20 03:14:13 +00:00
AUTHENTICATION_RESPONSE_TYPE: ${AUTHENTICATION_RESPONSE_TYPE:-id_token}
CUSTOM_OIDC_AUTHENTICATION_PROVIDER_NAME: ${CUSTOM_OIDC_AUTHENTICATION_PROVIDER_NAME:-""}
AUTHENTICATION_PUBLIC_KEYS: ${AUTHENTICATION_PUBLIC_KEYS:-[http://localhost:8585/api/v1/system/config/jwks]}
AUTHENTICATION_AUTHORITY: ${AUTHENTICATION_AUTHORITY:-https://accounts.google.com}
AUTHENTICATION_CLIENT_ID: ${AUTHENTICATION_CLIENT_ID:-""}
AUTHENTICATION_CALLBACK_URL: ${AUTHENTICATION_CALLBACK_URL:-""}
AUTHENTICATION_JWT_PRINCIPAL_CLAIMS: ${AUTHENTICATION_JWT_PRINCIPAL_CLAIMS:-[email,preferred_username,sub]}
2024-06-24 17:45:19 +00:00
AUTHENTICATION_JWT_PRINCIPAL_CLAIMS_MAPPING: ${AUTHENTICATION_JWT_PRINCIPAL_CLAIMS_MAPPING:-[]}
AUTHENTICATION_ENABLE_SELF_SIGNUP: ${AUTHENTICATION_ENABLE_SELF_SIGNUP:-true}
AUTHENTICATION_CLIENT_TYPE: ${AUTHENTICATION_CLIENT_TYPE:-public}
#For OIDC Authentication, when client is confidential
OIDC_CLIENT_ID: ${OIDC_CLIENT_ID:-""}
OIDC_TYPE: ${OIDC_TYPE:-""} # google, azure etc.
OIDC_CLIENT_SECRET: ${OIDC_CLIENT_SECRET:-""}
OIDC_SCOPE: ${OIDC_SCOPE:-"openid email profile"}
OIDC_DISCOVERY_URI: ${OIDC_DISCOVERY_URI:-""}
OIDC_USE_NONCE: ${OIDC_USE_NONCE:-true}
OIDC_PREFERRED_JWS: ${OIDC_PREFERRED_JWS:-"RS256"}
OIDC_RESPONSE_TYPE: ${OIDC_RESPONSE_TYPE:-"code"}
OIDC_DISABLE_PKCE: ${OIDC_DISABLE_PKCE:-true}
OIDC_CALLBACK: ${OIDC_CALLBACK:-"http://localhost:8585/callback"}
OIDC_SERVER_URL: ${OIDC_SERVER_URL:-"http://localhost:8585"}
OIDC_CLIENT_AUTH_METHOD: ${OIDC_CLIENT_AUTH_METHOD:-"client_secret_post"}
OIDC_TENANT: ${OIDC_TENANT:-""}
OIDC_MAX_CLOCK_SKEW: ${OIDC_MAX_CLOCK_SKEW:-""}
OIDC_CUSTOM_PARAMS: ${OIDC_CUSTOM_PARAMS:-{}}
OIDC_MAX_AGE: ${OIDC_MAX_AGE:-"0"}
OIDC_PROMPT_TYPE: ${OIDC_PROMPT_TYPE:-"consent"}
OIDC_SESSION_EXPIRY: ${OIDC_SESSION_EXPIRY:-"604800"}
# For SAML Authentication
# SAML_DEBUG_MODE: ${SAML_DEBUG_MODE:-false}
# SAML_IDP_ENTITY_ID: ${SAML_IDP_ENTITY_ID:-""}
# SAML_IDP_SSO_LOGIN_URL: ${SAML_IDP_SSO_LOGIN_URL:-""}
# SAML_IDP_CERTIFICATE: ${SAML_IDP_CERTIFICATE:-""}
# SAML_AUTHORITY_URL: ${SAML_AUTHORITY_URL:-"http://localhost:8585/api/v1/saml/login"}
# SAML_IDP_NAME_ID: ${SAML_IDP_NAME_ID:-"urn:oasis:names:tc:SAML:2.0:nameid-format:emailAddress"}
# SAML_SP_ENTITY_ID: ${SAML_SP_ENTITY_ID:-"http://localhost:8585/api/v1/saml/metadata"}
# SAML_SP_ACS: ${SAML_SP_ACS:-"http://localhost:8585/api/v1/saml/acs"}
# SAML_SP_CERTIFICATE: ${SAML_SP_CERTIFICATE:-""}
# SAML_SP_CALLBACK: ${SAML_SP_CALLBACK:-"http://localhost:8585/saml/callback"}
# SAML_STRICT_MODE: ${SAML_STRICT_MODE:-false}
# SAML_SP_TOKEN_VALIDITY: ${SAML_SP_TOKEN_VALIDITY:-"3600"}
# SAML_SEND_ENCRYPTED_NAME_ID: ${SAML_SEND_ENCRYPTED_NAME_ID:-false}
# SAML_SEND_SIGNED_AUTH_REQUEST: ${SAML_SEND_SIGNED_AUTH_REQUEST:-false}
# SAML_SIGNED_SP_METADATA: ${SAML_SIGNED_SP_METADATA:-false}
# SAML_WANT_MESSAGE_SIGNED: ${SAML_WANT_MESSAGE_SIGNED:-false}
# SAML_WANT_ASSERTION_SIGNED: ${SAML_WANT_ASSERTION_SIGNED:-false}
# SAML_WANT_ASSERTION_ENCRYPTED: ${SAML_WANT_ASSERTION_ENCRYPTED:-false}
# SAML_WANT_NAME_ID_ENCRYPTED: ${SAML_WANT_NAME_ID_ENCRYPTED:-false}
# SAML_KEYSTORE_FILE_PATH: ${SAML_KEYSTORE_FILE_PATH:-""}
# SAML_KEYSTORE_ALIAS: ${SAML_KEYSTORE_ALIAS:-""}
# SAML_KEYSTORE_PASSWORD: ${SAML_KEYSTORE_PASSWORD:-""}
# For LDAP Authentication
# AUTHENTICATION_LDAP_HOST: ${AUTHENTICATION_LDAP_HOST:-}
# AUTHENTICATION_LDAP_PORT: ${AUTHENTICATION_LDAP_PORT:-}
# AUTHENTICATION_LOOKUP_ADMIN_DN: ${AUTHENTICATION_LOOKUP_ADMIN_DN:-""}
# AUTHENTICATION_LOOKUP_ADMIN_PWD: ${AUTHENTICATION_LOOKUP_ADMIN_PWD:-""}
# AUTHENTICATION_USER_LOOKUP_BASEDN: ${AUTHENTICATION_USER_LOOKUP_BASEDN:-""}
# AUTHENTICATION_USER_MAIL_ATTR: ${AUTHENTICATION_USER_MAIL_ATTR:-}
# AUTHENTICATION_LDAP_POOL_SIZE: ${AUTHENTICATION_LDAP_POOL_SIZE:-3}
# AUTHENTICATION_LDAP_SSL_ENABLED: ${AUTHENTICATION_LDAP_SSL_ENABLED:-}
# AUTHENTICATION_LDAP_TRUSTSTORE_TYPE: ${AUTHENTICATION_LDAP_TRUSTSTORE_TYPE:-TrustAll}
# AUTHENTICATION_LDAP_TRUSTSTORE_PATH: ${AUTHENTICATION_LDAP_TRUSTSTORE_PATH:-}
# AUTHENTICATION_LDAP_KEYSTORE_PASSWORD: ${AUTHENTICATION_LDAP_KEYSTORE_PASSWORD:-}
# AUTHENTICATION_LDAP_SSL_KEY_FORMAT: ${AUTHENTICATION_LDAP_SSL_KEY_FORMAT:-}
# AUTHENTICATION_LDAP_ALLOW_WILDCARDS: ${AUTHENTICATION_LDAP_ALLOW_WILDCARDS:-}
# AUTHENTICATION_LDAP_ALLOWED_HOSTNAMES: ${AUTHENTICATION_LDAP_ALLOWED_HOSTNAMES:-[]}
# AUTHENTICATION_LDAP_SSL_VERIFY_CERT_HOST: ${AUTHENTICATION_LDAP_SSL_VERIFY_CERT_HOST:-}
# AUTHENTICATION_LDAP_EXAMINE_VALIDITY_DATES: ${AUTHENTICATION_LDAP_EXAMINE_VALIDITY_DATES:-true}
# JWT Configuration
RSA_PUBLIC_KEY_FILE_PATH: ${RSA_PUBLIC_KEY_FILE_PATH:-"./conf/public_key.der"}
RSA_PRIVATE_KEY_FILE_PATH: ${RSA_PRIVATE_KEY_FILE_PATH:-"./conf/private_key.der"}
JWT_ISSUER: ${JWT_ISSUER:-"open-metadata.org"}
JWT_KEY_ID: ${JWT_KEY_ID:-"Gb389a-9f76-gdjs-a92j-0242bk94356"}
# OpenMetadata Server Pipeline Service Client Configuration
PIPELINE_SERVICE_CLIENT_ENDPOINT: ${PIPELINE_SERVICE_CLIENT_ENDPOINT:-http://ingestion:8080}
PIPELINE_SERVICE_CLIENT_HEALTH_CHECK_INTERVAL: ${PIPELINE_SERVICE_CLIENT_HEALTH_CHECK_INTERVAL:-300}
SERVER_HOST_API_URL: ${SERVER_HOST_API_URL:-http://openmetadata-server:8585/api}
PIPELINE_SERVICE_CLIENT_VERIFY_SSL: ${PIPELINE_SERVICE_CLIENT_VERIFY_SSL:-"no-ssl"}
PIPELINE_SERVICE_CLIENT_SSL_CERT_PATH: ${PIPELINE_SERVICE_CLIENT_SSL_CERT_PATH:-""}
# Database configuration for MySQL
DB_DRIVER_CLASS: ${DB_DRIVER_CLASS:-com.mysql.cj.jdbc.Driver}
DB_SCHEME: ${DB_SCHEME:-mysql}
DB_PARAMS: ${DB_PARAMS:-allowPublicKeyRetrieval=true&useSSL=false&serverTimezone=UTC}
DB_USE_SSL: ${DB_USE_SSL:-false}
DB_USER: ${DB_USER:-openmetadata_user}
DB_USER_PASSWORD: ${DB_USER_PASSWORD:-openmetadata_password}
DB_HOST: ${DB_HOST:-mysql}
DB_PORT: ${DB_PORT:-3306}
OM_DATABASE: ${OM_DATABASE:-openmetadata_db}
# ElasticSearch Configurations
ELASTICSEARCH_HOST: ${ELASTICSEARCH_HOST:- elasticsearch}
ELASTICSEARCH_PORT: ${ELASTICSEARCH_PORT:-9200}
ELASTICSEARCH_SCHEME: ${ELASTICSEARCH_SCHEME:-http}
ELASTICSEARCH_USER: ${ELASTICSEARCH_USER:-""}
ELASTICSEARCH_PASSWORD: ${ELASTICSEARCH_PASSWORD:-""}
SEARCH_TYPE: ${SEARCH_TYPE:- "elasticsearch"}
ELASTICSEARCH_CLUSTER_ALIAS: ${ELASTICSEARCH_CLUSTER_ALIAS:- "openmetadata"}
ELASTICSEARCH_TRUST_STORE_PATH: ${ELASTICSEARCH_TRUST_STORE_PATH:-""}
ELASTICSEARCH_TRUST_STORE_PASSWORD: ${ELASTICSEARCH_TRUST_STORE_PASSWORD:-""}
ELASTICSEARCH_CONNECTION_TIMEOUT_SECS: ${ELASTICSEARCH_CONNECTION_TIMEOUT_SECS:-5}
ELASTICSEARCH_SOCKET_TIMEOUT_SECS: ${ELASTICSEARCH_SOCKET_TIMEOUT_SECS:-60}
ELASTICSEARCH_KEEP_ALIVE_TIMEOUT_SECS: ${ELASTICSEARCH_KEEP_ALIVE_TIMEOUT_SECS:-600}
ELASTICSEARCH_BATCH_SIZE: ${ELASTICSEARCH_BATCH_SIZE:-100}
ELASTICSEARCH_PAYLOAD_BYTES_SIZE: ${ELASTICSEARCH_PAYLOAD_BYTES_SIZE:-10485760} #max payLoadSize in Bytes
ELASTICSEARCH_INDEX_MAPPING_LANG: ${ELASTICSEARCH_INDEX_MAPPING_LANG:-EN}
# AWS IAM Authentication for OpenSearch (auto-enabled when AWS_DEFAULT_REGION is set)
# Uses standard AWS env vars: https://docs.aws.amazon.com/cli/v1/userguide/cli-configure-envvars.html
AWS_DEFAULT_REGION: ${AWS_DEFAULT_REGION:-""}
AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID:-""}
AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY:-""}
AWS_SESSION_TOKEN: ${AWS_SESSION_TOKEN:-""}
SEARCH_AWS_SERVICE_NAME: ${SEARCH_AWS_SERVICE_NAME:-"es"}
#eventMonitoringConfiguration
EVENT_MONITOR: ${EVENT_MONITOR:-prometheus}
EVENT_MONITOR_BATCH_SIZE: ${EVENT_MONITOR_BATCH_SIZE:-10}
EVENT_MONITOR_PATH_PATTERN: ${EVENT_MONITOR_PATH_PATTERN:-["/api/v1/tables/*", "/api/v1/health-check"]}
EVENT_MONITOR_LATENCY: ${EVENT_MONITOR_LATENCY:-[]}
#pipelineServiceClientConfiguration
PIPELINE_SERVICE_CLIENT_CLASS_NAME: ${PIPELINE_SERVICE_CLIENT_CLASS_NAME:-"org.openmetadata.service.clients.pipeline.airflow.AirflowRESTClient"}
PIPELINE_SERVICE_IP_INFO_ENABLED: ${PIPELINE_SERVICE_IP_INFO_ENABLED:-false}
PIPELINE_SERVICE_CLIENT_HOST_IP: ${PIPELINE_SERVICE_CLIENT_HOST_IP:-""}
PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER: ${PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER:-"noop"}
#airflow parameters
AIRFLOW_USERNAME: ${AIRFLOW_USERNAME:-admin}
AIRFLOW_PASSWORD: ${AIRFLOW_PASSWORD:-admin}
AIRFLOW_TIMEOUT: ${AIRFLOW_TIMEOUT:-10}
AIRFLOW_TRUST_STORE_PATH: ${AIRFLOW_TRUST_STORE_PATH:-""}
AIRFLOW_TRUST_STORE_PASSWORD: ${AIRFLOW_TRUST_STORE_PASSWORD:-""}
FERNET_KEY: ${FERNET_KEY:-jJ/9sz0g0OHxsfxOoSfdFdmk3ysNmPRnH3TUAbz3IHA=}
#secretsManagerConfiguration
SECRET_MANAGER: ${SECRET_MANAGER:-db}
#parameters:
OM_SM_REGION: ${OM_SM_REGION:-""}
OM_SM_ACCESS_KEY_ID: ${OM_SM_ACCESS_KEY_ID:-""}
OM_SM_ACCESS_KEY: ${OM_SM_ACCESS_KEY:-""}
#email configuration:
OM_EMAIL_ENTITY: ${OM_EMAIL_ENTITY:-"OpenMetadata"}
OM_SUPPORT_URL: ${OM_SUPPORT_URL:-"https://slack.open-metadata.org"}
AUTHORIZER_ENABLE_SMTP : ${AUTHORIZER_ENABLE_SMTP:-false}
OPENMETADATA_SERVER_URL: ${OPENMETADATA_SERVER_URL:-""}
OPENMETADATA_SMTP_SENDER_MAIL: ${OPENMETADATA_SMTP_SENDER_MAIL:-""}
SMTP_SERVER_ENDPOINT: ${SMTP_SERVER_ENDPOINT:-""}
SMTP_SERVER_PORT: ${SMTP_SERVER_PORT:-""}
SMTP_SERVER_USERNAME: ${SMTP_SERVER_USERNAME:-""}
SMTP_SERVER_PWD: ${SMTP_SERVER_PWD:-""}
SMTP_SERVER_STRATEGY: ${SMTP_SERVER_STRATEGY:-"SMTP_TLS"}
#extensionConfiguration
OM_RESOURCE_PACKAGES: ${OM_RESOURCE_PACKAGES:-[]}
OM_EXTENSIONS: ${OM_EXTENSIONS:-[]}
# Heap OPTS Configurations
OPENMETADATA_HEAP_OPTS: ${OPENMETADATA_HEAP_OPTS:--Xmx1G -Xms1G}
# Mask passwords values in UI
MASK_PASSWORDS_API: ${MASK_PASSWORDS_API:-false}
#OpenMetadata Web Configuration
WEB_CONF_URI_PATH: ${WEB_CONF_URI_PATH:-"/api"}
#HSTS
WEB_CONF_HSTS_ENABLED: ${WEB_CONF_HSTS_ENABLED:-false}
WEB_CONF_HSTS_MAX_AGE: ${WEB_CONF_HSTS_MAX_AGE:-"365 days"}
WEB_CONF_HSTS_INCLUDE_SUBDOMAINS: ${WEB_CONF_HSTS_INCLUDE_SUBDOMAINS:-"true"}
WEB_CONF_HSTS_PRELOAD: ${WEB_CONF_HSTS_PRELOAD:-"true"}
#Frame Options
WEB_CONF_FRAME_OPTION_ENABLED: ${WEB_CONF_FRAME_OPTION_ENABLED:-false}
WEB_CONF_FRAME_OPTION: ${WEB_CONF_FRAME_OPTION:-"SAMEORIGIN"}
WEB_CONF_FRAME_ORIGIN: ${WEB_CONF_FRAME_ORIGIN:-""}
#Content Type
WEB_CONF_CONTENT_TYPE_OPTIONS_ENABLED: ${WEB_CONF_CONTENT_TYPE_OPTIONS_ENABLED:-false}
#XSS-Protection
WEB_CONF_XSS_PROTECTION_ENABLED: ${WEB_CONF_XSS_PROTECTION_ENABLED:-false}
WEB_CONF_XSS_PROTECTION_ON: ${WEB_CONF_XSS_PROTECTION_ON:-true}
WEB_CONF_XSS_PROTECTION_BLOCK: ${WEB_CONF_XSS_PROTECTION_BLOCK:-true}
#CSP
WEB_CONF_XSS_CSP_ENABLED: ${WEB_CONF_XSS_CSP_ENABLED:-false}
WEB_CONF_XSS_CSP_POLICY: ${WEB_CONF_XSS_CSP_POLICY:-"default-src 'self'"}
WEB_CONF_XSS_CSP_REPORT_ONLY_POLICY: ${WEB_CONF_XSS_CSP_REPORT_ONLY_POLICY:-""}
#Referrer-Policy
WEB_CONF_REFERRER_POLICY_ENABLED: ${WEB_CONF_REFERRER_POLICY_ENABLED:-false}
WEB_CONF_REFERRER_POLICY_OPTION: ${WEB_CONF_REFERRER_POLICY_OPTION:-"SAME_ORIGIN"}
#Permission-Policy
WEB_CONF_PERMISSION_POLICY_ENABLED: ${WEB_CONF_PERMISSION_POLICY_ENABLED:-false}
WEB_CONF_PERMISSION_POLICY_OPTION: ${WEB_CONF_PERMISSION_POLICY_OPTION:-""}
#Cache
WEB_CONF_CACHE_CONTROL: ${WEB_CONF_CACHE_CONTROL:-""}
WEB_CONF_PRAGMA: ${WEB_CONF_PRAGMA:-""}
depends_on:
elasticsearch:
condition: service_healthy
mysql:
condition: service_healthy
networks:
- local_app_net
openmetadata-server:
build:
context: ../../.
dockerfile: docker/development/Dockerfile
container_name: openmetadata_server
environment:
OPENMETADATA_CLUSTER_NAME: ${OPENMETADATA_CLUSTER_NAME:-openmetadata}
SERVER_PORT: ${SERVER_PORT:-8585}
SERVER_ADMIN_PORT: ${SERVER_ADMIN_PORT:-8586}
LOG_LEVEL: ${LOG_LEVEL:-INFO}
# OpenMetadata Server Authentication Configuration
AUTHORIZER_CLASS_NAME: ${AUTHORIZER_CLASS_NAME:-org.openmetadata.service.security.DefaultAuthorizer}
AUTHORIZER_REQUEST_FILTER: ${AUTHORIZER_REQUEST_FILTER:-org.openmetadata.service.security.JwtFilter}
AUTHORIZER_ADMIN_PRINCIPALS: ${AUTHORIZER_ADMIN_PRINCIPALS:-[admin]}
AUTHORIZER_ALLOWED_REGISTRATION_DOMAIN: ${AUTHORIZER_ALLOWED_REGISTRATION_DOMAIN:-["all"]}
AUTHORIZER_INGESTION_PRINCIPALS: ${AUTHORIZER_INGESTION_PRINCIPALS:-[ingestion-bot]}
AUTHORIZER_PRINCIPAL_DOMAIN: ${AUTHORIZER_PRINCIPAL_DOMAIN:-"open-metadata.org"}
AUTHORIZER_ALLOWED_DOMAINS: ${AUTHORIZER_ALLOWED_DOMAINS:-[]}
AUTHORIZER_ENFORCE_PRINCIPAL_DOMAIN: ${AUTHORIZER_ENFORCE_PRINCIPAL_DOMAIN:-false}
AUTHORIZER_ENABLE_SECURE_SOCKET: ${AUTHORIZER_ENABLE_SECURE_SOCKET:-false}
AUTHENTICATION_PROVIDER: ${AUTHENTICATION_PROVIDER:-basic}
AUTHENTICATION_RESPONSE_TYPE: ${AUTHENTICATION_RESPONSE_TYPE:-id_token}
CUSTOM_OIDC_AUTHENTICATION_PROVIDER_NAME: ${CUSTOM_OIDC_AUTHENTICATION_PROVIDER_NAME:-""}
AUTHENTICATION_PUBLIC_KEYS: ${AUTHENTICATION_PUBLIC_KEYS:-[http://localhost:8585/api/v1/system/config/jwks]}
AUTHENTICATION_AUTHORITY: ${AUTHENTICATION_AUTHORITY:-https://accounts.google.com}
AUTHENTICATION_CLIENT_ID: ${AUTHENTICATION_CLIENT_ID:-""}
AUTHENTICATION_CALLBACK_URL: ${AUTHENTICATION_CALLBACK_URL:-""}
AUTHENTICATION_JWT_PRINCIPAL_CLAIMS: ${AUTHENTICATION_JWT_PRINCIPAL_CLAIMS:-[email,preferred_username,sub]}
AUTHENTICATION_JWT_PRINCIPAL_CLAIMS_MAPPING: ${AUTHENTICATION_JWT_PRINCIPAL_CLAIMS_MAPPING:-[]}
AUTHENTICATION_ENABLE_SELF_SIGNUP : ${AUTHENTICATION_ENABLE_SELF_SIGNUP:-true}
AUTHENTICATION_CLIENT_TYPE: ${AUTHENTICATION_CLIENT_TYPE:-public}
#For OIDC Authentication, when client is confidential
OIDC_CLIENT_ID: ${OIDC_CLIENT_ID:-""}
OIDC_TYPE: ${OIDC_TYPE:-""} # google, azure etc.
OIDC_CLIENT_SECRET: ${OIDC_CLIENT_SECRET:-""}
OIDC_SCOPE: ${OIDC_SCOPE:-"openid email profile"}
OIDC_DISCOVERY_URI: ${OIDC_DISCOVERY_URI:-""}
OIDC_USE_NONCE: ${OIDC_USE_NONCE:-true}
OIDC_PREFERRED_JWS: ${OIDC_PREFERRED_JWS:-"RS256"}
OIDC_RESPONSE_TYPE: ${OIDC_RESPONSE_TYPE:-"code"}
OIDC_DISABLE_PKCE: ${OIDC_DISABLE_PKCE:-true}
OIDC_CALLBACK: ${OIDC_CALLBACK:-"http://localhost:8585/callback"}
OIDC_SERVER_URL: ${OIDC_SERVER_URL:-"http://localhost:8585"}
OIDC_CLIENT_AUTH_METHOD: ${OIDC_CLIENT_AUTH_METHOD:-"client_secret_post"}
OIDC_TENANT: ${OIDC_TENANT:-""}
OIDC_MAX_CLOCK_SKEW: ${OIDC_MAX_CLOCK_SKEW:-""}
OIDC_CUSTOM_PARAMS: ${OIDC_CUSTOM_PARAMS:-{}}
OIDC_MAX_AGE: ${OIDC_MAX_AGE:-"0"}
OIDC_PROMPT_TYPE: ${OIDC_PROMPT_TYPE:-"consent"}
OIDC_SESSION_EXPIRY: ${OIDC_SESSION_EXPIRY:-"604800"}
# For SAML Authentication
# SAML_DEBUG_MODE: ${SAML_DEBUG_MODE:-false}
# SAML_IDP_ENTITY_ID: ${SAML_IDP_ENTITY_ID:-""}
# SAML_IDP_SSO_LOGIN_URL: ${SAML_IDP_SSO_LOGIN_URL:-""}
# SAML_IDP_CERTIFICATE: ${SAML_IDP_CERTIFICATE:-""}
# SAML_AUTHORITY_URL: ${SAML_AUTHORITY_URL:-"http://localhost:8585/api/v1/saml/login"}
# SAML_IDP_NAME_ID: ${SAML_IDP_NAME_ID:-"urn:oasis:names:tc:SAML:2.0:nameid-format:emailAddress"}
# SAML_SP_ENTITY_ID: ${SAML_SP_ENTITY_ID:-"http://localhost:8585/api/v1/saml/metadata"}
# SAML_SP_ACS: ${SAML_SP_ACS:-"http://localhost:8585/api/v1/saml/acs"}
# SAML_SP_CERTIFICATE: ${SAML_SP_CERTIFICATE:-""}
# SAML_SP_CALLBACK: ${SAML_SP_CALLBACK:-"http://localhost:8585/saml/callback"}
# SAML_STRICT_MODE: ${SAML_STRICT_MODE:-false}
# SAML_SP_TOKEN_VALIDITY: ${SAML_SP_TOKEN_VALIDITY:-"3600"}
# SAML_SEND_ENCRYPTED_NAME_ID: ${SAML_SEND_ENCRYPTED_NAME_ID:-false}
# SAML_SEND_SIGNED_AUTH_REQUEST: ${SAML_SEND_SIGNED_AUTH_REQUEST:-false}
# SAML_SIGNED_SP_METADATA: ${SAML_SIGNED_SP_METADATA:-false}
# SAML_WANT_MESSAGE_SIGNED: ${SAML_WANT_MESSAGE_SIGNED:-false}
# SAML_WANT_ASSERTION_SIGNED: ${SAML_WANT_ASSERTION_SIGNED:-false}
# SAML_WANT_ASSERTION_ENCRYPTED: ${SAML_WANT_ASSERTION_ENCRYPTED:-false}
# SAML_WANT_NAME_ID_ENCRYPTED: ${SAML_WANT_NAME_ID_ENCRYPTED:-false}
# SAML_KEYSTORE_FILE_PATH: ${SAML_KEYSTORE_FILE_PATH:-""}
# SAML_KEYSTORE_ALIAS: ${SAML_KEYSTORE_ALIAS:-""}
# SAML_KEYSTORE_PASSWORD: ${SAML_KEYSTORE_PASSWORD:-""}
# For LDAP Authentication
# AUTHENTICATION_LDAP_HOST: ${AUTHENTICATION_LDAP_HOST:-}
# AUTHENTICATION_LDAP_PORT: ${AUTHENTICATION_LDAP_PORT:-}
# AUTHENTICATION_LOOKUP_ADMIN_DN: ${AUTHENTICATION_LOOKUP_ADMIN_DN:-""}
# AUTHENTICATION_LOOKUP_ADMIN_PWD: ${AUTHENTICATION_LOOKUP_ADMIN_PWD:-""}
# AUTHENTICATION_USER_LOOKUP_BASEDN: ${AUTHENTICATION_USER_LOOKUP_BASEDN:-""}
# AUTHENTICATION_USER_MAIL_ATTR: ${AUTHENTICATION_USER_MAIL_ATTR:-}
# AUTHENTICATION_LDAP_POOL_SIZE: ${AUTHENTICATION_LDAP_POOL_SIZE:-3}
# AUTHENTICATION_LDAP_SSL_ENABLED: ${AUTHENTICATION_LDAP_SSL_ENABLED:-}
# AUTHENTICATION_LDAP_TRUSTSTORE_TYPE: ${AUTHENTICATION_LDAP_TRUSTSTORE_TYPE:-TrustAll}
# AUTHENTICATION_LDAP_TRUSTSTORE_PATH: ${AUTHENTICATION_LDAP_TRUSTSTORE_PATH:-}
# AUTHENTICATION_LDAP_KEYSTORE_PASSWORD: ${AUTHENTICATION_LDAP_KEYSTORE_PASSWORD:-}
# AUTHENTICATION_LDAP_SSL_KEY_FORMAT: ${AUTHENTICATION_LDAP_SSL_KEY_FORMAT:-}
# AUTHENTICATION_LDAP_ALLOW_WILDCARDS: ${AUTHENTICATION_LDAP_ALLOW_WILDCARDS:-}
# AUTHENTICATION_LDAP_ALLOWED_HOSTNAMES: ${AUTHENTICATION_LDAP_ALLOWED_HOSTNAMES:-[]}
# AUTHENTICATION_LDAP_SSL_VERIFY_CERT_HOST: ${AUTHENTICATION_LDAP_SSL_VERIFY_CERT_HOST:-}
# AUTHENTICATION_LDAP_EXAMINE_VALIDITY_DATES: ${AUTHENTICATION_LDAP_EXAMINE_VALIDITY_DATES:-true}
# JWT Configuration
RSA_PUBLIC_KEY_FILE_PATH: ${RSA_PUBLIC_KEY_FILE_PATH:-"./conf/public_key.der"}
RSA_PRIVATE_KEY_FILE_PATH: ${RSA_PRIVATE_KEY_FILE_PATH:-"./conf/private_key.der"}
JWT_ISSUER: ${JWT_ISSUER:-"open-metadata.org"}
JWT_KEY_ID: ${JWT_KEY_ID:-"Gb389a-9f76-gdjs-a92j-0242bk94356"}
# OpenMetadata Server Pipeline Service Client Configuration
PIPELINE_SERVICE_CLIENT_ENDPOINT: ${PIPELINE_SERVICE_CLIENT_ENDPOINT:-http://ingestion:8080}
PIPELINE_SERVICE_CLIENT_HEALTH_CHECK_INTERVAL: ${PIPELINE_SERVICE_CLIENT_HEALTH_CHECK_INTERVAL:-300}
SERVER_HOST_API_URL: ${SERVER_HOST_API_URL:-http://openmetadata-server:8585/api}
PIPELINE_SERVICE_CLIENT_VERIFY_SSL: ${PIPELINE_SERVICE_CLIENT_VERIFY_SSL:-"no-ssl"}
PIPELINE_SERVICE_CLIENT_SSL_CERT_PATH: ${PIPELINE_SERVICE_CLIENT_SSL_CERT_PATH:-""}
# Database configuration for MySQL
DB_DRIVER_CLASS: ${DB_DRIVER_CLASS:-com.mysql.cj.jdbc.Driver}
DB_SCHEME: ${DB_SCHEME:-mysql}
DB_PARAMS: ${DB_PARAMS:-allowPublicKeyRetrieval=true&useSSL=false&serverTimezone=UTC}
DB_USE_SSL: ${DB_USE_SSL:-false}
DB_USER: ${DB_USER:-openmetadata_user}
DB_USER_PASSWORD: ${DB_USER_PASSWORD:-openmetadata_password}
DB_HOST: ${DB_HOST:-mysql}
DB_PORT: ${DB_PORT:-3306}
OM_DATABASE: ${OM_DATABASE:-openmetadata_db}
# ElasticSearch Configurations
ELASTICSEARCH_HOST: ${ELASTICSEARCH_HOST:- elasticsearch}
ELASTICSEARCH_PORT: ${ELASTICSEARCH_PORT:-9200}
ELASTICSEARCH_SCHEME: ${ELASTICSEARCH_SCHEME:-http}
ELASTICSEARCH_USER: ${ELASTICSEARCH_USER:-""}
ELASTICSEARCH_PASSWORD: ${ELASTICSEARCH_PASSWORD:-""}
SEARCH_TYPE: ${SEARCH_TYPE:- "elasticsearch"}
ELASTICSEARCH_CLUSTER_ALIAS: ${ELASTICSEARCH_CLUSTER_ALIAS:- "openmetadata"}
ELASTICSEARCH_TRUST_STORE_PATH: ${ELASTICSEARCH_TRUST_STORE_PATH:-""}
ELASTICSEARCH_TRUST_STORE_PASSWORD: ${ELASTICSEARCH_TRUST_STORE_PASSWORD:-""}
ELASTICSEARCH_CONNECTION_TIMEOUT_SECS: ${ELASTICSEARCH_CONNECTION_TIMEOUT_SECS:-5}
ELASTICSEARCH_SOCKET_TIMEOUT_SECS: ${ELASTICSEARCH_SOCKET_TIMEOUT_SECS:-60}
ELASTICSEARCH_KEEP_ALIVE_TIMEOUT_SECS: ${ELASTICSEARCH_KEEP_ALIVE_TIMEOUT_SECS:-600}
ELASTICSEARCH_BATCH_SIZE: ${ELASTICSEARCH_BATCH_SIZE:-100}
ELASTICSEARCH_PAYLOAD_BYTES_SIZE: ${ELASTICSEARCH_PAYLOAD_BYTES_SIZE:-10485760} #max payLoadSize in Bytes
ELASTICSEARCH_INDEX_MAPPING_LANG: ${ELASTICSEARCH_INDEX_MAPPING_LANG:-EN}
# AWS IAM Authentication for OpenSearch (auto-enabled when AWS_DEFAULT_REGION is set)
# Uses standard AWS env vars: https://docs.aws.amazon.com/cli/v1/userguide/cli-configure-envvars.html
AWS_DEFAULT_REGION: ${AWS_DEFAULT_REGION:-""}
AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID:-""}
AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY:-""}
AWS_SESSION_TOKEN: ${AWS_SESSION_TOKEN:-""}
SEARCH_AWS_SERVICE_NAME: ${SEARCH_AWS_SERVICE_NAME:-"es"}
#eventMonitoringConfiguration
EVENT_MONITOR: ${EVENT_MONITOR:-prometheus}
EVENT_MONITOR_BATCH_SIZE: ${EVENT_MONITOR_BATCH_SIZE:-10}
EVENT_MONITOR_PATH_PATTERN: ${EVENT_MONITOR_PATH_PATTERN:-["/api/v1/tables/*", "/api/v1/health-check"]}
EVENT_MONITOR_LATENCY: ${EVENT_MONITOR_LATENCY:-[]}
#pipelineServiceClientConfiguration
PIPELINE_SERVICE_CLIENT_CLASS_NAME: ${PIPELINE_SERVICE_CLIENT_CLASS_NAME:-"org.openmetadata.service.clients.pipeline.airflow.AirflowRESTClient"}
PIPELINE_SERVICE_IP_INFO_ENABLED: ${PIPELINE_SERVICE_IP_INFO_ENABLED:-false}
PIPELINE_SERVICE_CLIENT_HOST_IP: ${PIPELINE_SERVICE_CLIENT_HOST_IP:-""}
PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER: ${PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER:-"noop"}
#airflow parameters
AIRFLOW_USERNAME: ${AIRFLOW_USERNAME:-admin}
AIRFLOW_PASSWORD: ${AIRFLOW_PASSWORD:-admin}
AIRFLOW_TIMEOUT: ${AIRFLOW_TIMEOUT:-10}
AIRFLOW_TRUST_STORE_PATH: ${AIRFLOW_TRUST_STORE_PATH:-""}
AIRFLOW_TRUST_STORE_PASSWORD: ${AIRFLOW_TRUST_STORE_PASSWORD:-""}
FERNET_KEY: ${FERNET_KEY:-jJ/9sz0g0OHxsfxOoSfdFdmk3ysNmPRnH3TUAbz3IHA=}
#secretsManagerConfiguration
SECRET_MANAGER: ${SECRET_MANAGER:-db}
# AWS:
OM_SM_REGION: ${OM_SM_REGION:-""}
OM_SM_ACCESS_KEY_ID: ${OM_SM_ACCESS_KEY_ID:-""}
OM_SM_ACCESS_KEY: ${OM_SM_ACCESS_KEY:-""}
# Azure:
OM_SM_VAULT_NAME: ${OM_SM_VAULT_NAME:-""}
OM_SM_CLIENT_ID: ${OM_SM_CLIENT_ID:-""}
OM_SM_CLIENT_SECRET: ${OM_SM_CLIENT_SECRET:-""}
OM_SM_TENANT_ID: ${OM_SM_TENANT_ID:-""}
# GCP:
OM_SM_PROJECT_ID: ${OM_SM_PROJECT_ID:-""}
#email configuration:
OM_EMAIL_ENTITY: ${OM_EMAIL_ENTITY:-"OpenMetadata"}
OM_SUPPORT_URL: ${OM_SUPPORT_URL:-"https://slack.open-metadata.org"}
AUTHORIZER_ENABLE_SMTP : ${AUTHORIZER_ENABLE_SMTP:-false}
OPENMETADATA_SERVER_URL: ${OPENMETADATA_SERVER_URL:-""}
OPENMETADATA_SMTP_SENDER_MAIL: ${OPENMETADATA_SMTP_SENDER_MAIL:-""}
SMTP_SERVER_ENDPOINT: ${SMTP_SERVER_ENDPOINT:-""}
SMTP_SERVER_PORT: ${SMTP_SERVER_PORT:-""}
SMTP_SERVER_USERNAME: ${SMTP_SERVER_USERNAME:-""}
SMTP_SERVER_PWD: ${SMTP_SERVER_PWD:-""}
SMTP_SERVER_STRATEGY: ${SMTP_SERVER_STRATEGY:-"SMTP_TLS"}
#extensionConfiguration
OM_RESOURCE_PACKAGES: ${OM_RESOURCE_PACKAGES:-[]}
OM_EXTENSIONS: ${OM_EXTENSIONS:-[]}
# Heap OPTS Configurations
OPENMETADATA_HEAP_OPTS: ${OPENMETADATA_HEAP_OPTS:--Xmx1G -Xms1G}
MASK_PASSWORDS_API: ${MASK_PASSWORDS_API:-true}
expose:
- 8585
- 8586
- 5005
ports:
- "8585:8585"
- "8586:8586"
- "5005:5005"
depends_on:
elasticsearch:
condition: service_healthy
mysql:
condition: service_healthy
execute-migrate-all:
condition: service_completed_successfully
networks:
- local_app_net
healthcheck:
test: [ "CMD", "wget", "-q", "--spider", "http://localhost:8586/healthcheck" ]
ingestion:
build:
context: ../../.
dockerfile: ingestion/Dockerfile.ci
args:
INGESTION_DEPENDENCY: ${INGESTION_DEPENDENCY:-all}
container_name: openmetadata_ingestion
environment:
AIRFLOW__API__AUTH_BACKENDS: "airflow.api.auth.backend.basic_auth,airflow.api.auth.backend.session"
AIRFLOW__CORE__EXECUTOR: LocalExecutor
AIRFLOW__LOGGING__LOGGING_LEVEL: ${AIRFLOW_LOGGING_LEVEL:-DEBUG}
AIRFLOW__OPENMETADATA_AIRFLOW_APIS__DAG_GENERATED_CONFIGS: "/opt/airflow/dag_generated_configs"
# OpenLineage transport config (optional - enable for lineage via OL)
# AIRFLOW__OPENLINEAGE__TRANSPORT: '{"type": "http", "url": "http://openmetadata-server:8585/api/v1/openlineage/", "endpoint": "lineage", "auth": {"type": "api_key", "api_key": "<OM_JWT_TOKEN>"}}'
# AIRFLOW__OPENLINEAGE__NAMESPACE: local_airflow
DB_HOST: ${AIRFLOW_DB_HOST:-mysql}
DB_PORT: ${AIRFLOW_DB_PORT:-3306}
AIRFLOW_DB: ${AIRFLOW_DB:-airflow_db}
DB_SCHEME: ${AIRFLOW_DB_SCHEME:-mysql+mysqldb}
DB_USER: ${AIRFLOW_DB_USER:-airflow_user}
DB_PASSWORD: ${AIRFLOW_DB_PASSWORD:-airflow_pass}
# extra connection-string properties for the database
# EXAMPLE
# require SSL (only for Postgres)
# properties: "?sslmode=require"
DB_PROPERTIES: ${AIRFLOW_DB_PROPERTIES:-}
# To test the lineage backend
# AIRFLOW__LINEAGE__BACKEND: airflow_provider_openmetadata.lineage.backend.OpenMetadataLineageBackend
# AIRFLOW__LINEAGE__AIRFLOW_SERVICE_NAME: local_airflow
# AIRFLOW__LINEAGE__OPENMETADATA_API_ENDPOINT: http://openmetadata-server:8585/api
# AIRFLOW__LINEAGE__JWT_TOKEN: ...
# AIRFLOW__CORE__AUTH_MANAGER: airflow.providers.fab.auth_manager.fab_auth_manager.FabAuthManager
# AIRFLOW__CORE__SIMPLE_AUTH_MANAGER_ALL_ADMINS: true
## Secrets Manager
# To integrate Azure Key Vault
AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AZURE_KEY_VAULT_NAME: ${OM_SM_VAULT_NAME:-""}
AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AZURE_TENANT_ID: ${OM_SM_TENANT_ID:-""}
AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AZURE_CLIENT_ID: ${OM_SM_CLIENT_ID:-""}
AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AZURE_CLIENT_SECRET: ${OM_SM_CLIENT_SECRET:-""}
# To Integrate with AWS SSM
AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AWS_REGION: ${OM_SM_AWS_REGION:-""}
AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AWS_ACCESS_KEY_ID: ${OM_SM_AWS_ACCESS_KEY_ID:-""}
AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AWS_SECRET_ACCESS_KEY: ${OM_SM_AWS_SECRET_ACCESS_KEY:-""}
# To integrate GCP
AIRFLOW__OPENMETADATA_SECRETS_MANAGER__GCP_PROJECT_ID: ${OM_SM_PROJECT_ID:-""}
# Apps
ENABLE_APP_HelloPipelines: "true"
entrypoint: /bin/bash
command:
- "/opt/airflow/ingestion_dependency.sh"
depends_on:
elasticsearch:
condition: service_started
mysql:
condition: service_healthy
openmetadata-server:
condition: service_started
expose:
- 8080
ports:
- "8080:8080"
networks:
- local_app_net
volumes:
2022-09-30 12:50:38 +00:00
- ingestion-volume-dag-airflow:/opt/airflow/dag_generated_configs
- ingestion-volume-dags:/opt/airflow/dags
- ingestion-volume-tmp:/tmp
- /var/run/docker.sock:/var/run/docker.sock:z # Need 600 permissions to run DockerOperator
mock-oidc-provider:
build:
context: ./mock-oidc-provider
dockerfile: Dockerfile
container_name: mock_oidc_provider
environment:
PORT: "9090"
ISSUER: ${MOCK_OIDC_ISSUER:-http://localhost:9090}
INTERNAL_BASE_URL: ${MOCK_OIDC_INTERNAL_BASE_URL:-http://mock-oidc-provider:9090}
expose:
- 9090
ports:
- "9090:9090"
networks:
- local_app_net
healthcheck:
test: ["CMD", "wget", "-q", "--spider", "http://localhost:9090/health"]
interval: 10s
timeout: 5s
retries: 5
profiles:
- sso-test
networks:
local_app_net:
name: ometa_network
ipam:
driver: default
config:
- subnet: "172.16.239.0/24"