mirror of
https://github.com/open-metadata/OpenMetadata
synced 2026-05-24 09:39:11 +00:00
* chore(ingestion): drop pylint, expand ruff to Stage 2c
Replace pylint with a coherent ruff-only stack (Stage 2c of the modernize
roadmap). Pylint is dropped from dev deps and CI workflows; ruff selected
ruleset expanded to ~22 families covering style, bug catchers, hygiene,
and the pylint port (PLE/PLC/PLW/PLR with the noisy "too-many-X"
complexity caps + magic-value disabled).
What's selected (with rationale in pyproject.toml):
E, W, F, I, N — style + correctness baseline + naming
UP — pyupgrade (py>=3.10 modernizations)
B, C4, C90, RET, SIM, TRY — bug catchers
PIE, ICN, T20, TC, TID, PTH, PERF — hygiene
PLE, PLC, PLW, PLR — pylint port (PLR complexity caps ignored)
RUF — ruff-native (incl. RUF100 unused-noqa)
What's removed:
- .pylintrc (root) — duplicate of the ingestion pylint config
- [tool.pylint.*] block in ingestion/pyproject.toml (~140 lines)
- ingestion/plugins/{print_checker,import_checker}.py + tests + README
(replaced by built-in T20 + TID251 banned-api respectively)
- pylint dep from ingestion/setup.py and openmetadata-airflow-apis/pyproject.toml
- `make lint` Makefile target + the pylint invocation in py_format_check
- dead pylint TODO comment + ignored test entry in noxfile.py
Cwd-stable config: ruff is invoked both from the repo root (pre-commit,
CI) and from ingestion/ (`make py_format_check`). The `src`,
`extend-exclude`, and per-file-ignores entries are listed twice — once
relative to ingestion/ and once with the `ingestion/` prefix — so
first-party isort detection and exclusions match in both invocations.
Grandfathering: ran `ruff check --add-noqa` once + format-stable
iteration. ~12,130 noqa directives across ~1,400 files. Cleanup is
deferred to follow-up PRs that drop noqas one rule at a time.
Documentation sweep: replaced `make lint` references in CLAUDE.md,
AGENTS.md, DEVELOPER.md, copilot-instructions, and 6 SKILL files with
the apply+verify shape `make py_format && make py_format_check`.
`make py_format` is NOT a strict superset of pylint — it only applies
auto-fixable violations; `make py_format_check` catches the rest.
Basedpyright baseline regenerated: ruff format reflowed multi-line
signatures in ~70 files, shifting type-error column positions. The
basedpyright baseline matches by (file path, error code, range), so
column shifts caused 19 entries to mis-align. Net diff is small
(154 lines in/out of the 13MB baseline.json) — purely positional.
Verified locally:
- make py_format_check → All checks passed
- nox --no-venv -s static-checks → 0 errors, 0 warnings, 0 notes
* chore(ingestion): finish ruff swap — nox lint session + skill docs
Three remaining stale-tooling references after Stage 2c:
- `ingestion/noxfile.py` `lint` session was still calling `black --check`,
`isort --check-only`, `pycln --diff`. Those tools aren't installed
anywhere (we dropped them from dev deps). Replace with the ruff
equivalents that mirror `make py_format_check`.
- `skills/standards/code_style.md`: stack listed as `black + isort +
pycln`; line length claimed 88 (black default). Both wrong: stack is
ruff, line length is 120.
- `skills/connector-building/SKILL.md`: `make py_format` comment said
`# black + isort + pycln`. Same swap.
* chore(ingestion): keep main's baseline + globally ignore TRY400
Per gitar-bot's review on PR #27774:
1. Main's PR #27728 promoted ~60 `logger.warning()` → `logger.error()`
inside `except` blocks. Those changes landed on main with their own
baseline updates. Our PR doesn't promote anything — the merge from
origin/main brought those `error` calls along with their baseline
entries.
The bot interpreted the `# noqa: TRY400` we added next to those lines
as us silencing the rule case-by-case. Cleaner: globally ignore
TRY400 in pyproject.toml, with a comment explaining why the codebase's
`logger.error(...)` + separate `logger.debug(traceback.format_exc())`
pattern is intentional. Strip ~430 per-line `# noqa: TRY400` markers
from source.
2. Document that `S101` in `per-file-ignores` is a forward-looking
entry — flake8-bandit (`S`) is not yet selected, so the rule is
no-op today; the entry stays so when `S` lands later, tests don't
immediately error.
Reverts the platform pin and Linux Docker–generated baseline. Keep
main's baseline intact and let CI surface the exact column-shifted
entries; the team will decide whether to fix in-place (revert format
on affected files) or add per-line `# pyright: ignore` markers.
* chore(ingestion): regen baseline for new connector type debt
Main's baseline was stale relative to recently-added connectors
(McpConnection, CustomDriveConnection) that lack common attributes
like `hostPort`, `database`, `catalog` etc. — all sites that access
those attributes via the union-typed `serviceConnection.root.config`
fire `reportAttributeAccessIssue` errors that aren't baselined.
71 errors + 58 warnings absorbed. Local macOS regen; pushing to see
CI's drift count. Per the basedpyright-baseline-and-ci PR experience,
macOS↔Linux column drift on this size of regen has historically been
1-7 residuals.
5 KiB
5 KiB
Code Style Standards
Python
Imports
Order: stdlib → third-party → OpenMetadata generated → OpenMetadata internal
import json
import traceback
from functools import partial
from typing import Iterable, Optional
import requests
from sqlalchemy.engine import Engine
from metadata.generated.schema.entity.services.connections.database.myDbConnection import (
MyDbConnection,
)
from metadata.ingestion.api.models import Either
from metadata.ingestion.connections.connection import BaseConnection
from metadata.utils.logger import ingestion_logger
Naming
- Connector directory:
snake_case(e.g.,my_database) - Python classes:
PascalCase(e.g.,MyDatabaseSource) - JSON Schema file:
lowerCamelCase+Connection.json(e.g.,myDatabaseConnection.json) - Type enum:
PascalCase(e.g.,MyDatabase)
Type Annotations
- All function signatures must have type annotations
- Use
Optional[T]for nullable fields - Use
Iterable[Either[...]]for yield methods - Import types from
typingorcollections.abc
No Unnecessary Comments
- Do NOT add comments that describe what code obviously does
- Only comment complex business logic, non-obvious algorithms, or workarounds
- No Google-style docstrings with
Args:/Returns:on simple methods - If code needs a comment to be understood, refactor the code instead
Pydantic Models
When defining Pydantic models for API responses with aliased fields:
- Always set
model_config = ConfigDict(populate_by_name=True)when usingField(alias=...)— without it, constructing instances with Python attribute names raisesValidationError - Use
Optional[T]withField(None, alias=...)for nullable fields - Create list response wrapper models inheriting from a base OData/pagination response
from pydantic import BaseModel, ConfigDict, Field
class MyApiReport(BaseModel):
model_config = ConfigDict(populate_by_name=True)
id: str = Field(alias="Id")
name: str = Field(alias="Name")
description: Optional[str] = Field(None, alias="Description")
class MyApiListResponse(BaseModel):
value: List[MyApiReport] = Field(default_factory=list)
Error Messages
Include context in error messages:
# Good
raise ValueError(f"Cannot connect to {config.hostPort}: {exc}")
# Bad
raise ValueError("Connection failed")
JSON Schema
File Naming
Schema file names use lowerCamelCase:
myDatabaseConnection.json(notmy_database_connection.json)bigQueryConnection.json(notbig_query_connection.json)
Required Fields
Every connection schema must have:
$idwith full URI path$schema:http://json-schema.org/draft-07/schema#title: PascalCase connection namejavaType: Full Java class pathtype:"object"definitionsblock with type enumadditionalProperties: false
Property Conventions
- Use
titlefor UI labels - Use
descriptionfor help text - Use
format: "password"for secrets - Use
format: "uri"for URLs - Use
defaultvalues where sensible - Use
$refto compose from shared schemas
$ref Paths
Paths are relative from the schema file location:
- Auth:
./common/basicAuth.json - SSL:
../../../../security/ssl/verifySSLConfig.json#/definitions/sslConfig - Filters:
../../../../type/filterPattern.json#/definitions/filterPattern - Connection extras:
../connectionBasicType.json#/definitions/connectionOptions - Capability flags:
../connectionBasicType.json#/definitions/supportsMetadataExtraction
Copyright Header
All Python files must start with:
# Copyright 2025 OpenMetadata
# Licensed under the Collate Community License, Version 1.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/LICENSE
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
Formatting
- Python:
ruff(lint + format, runmake py_formatto apply,make py_format_checkto verify) - Java:
spotless(runmvn spotless:apply) - Line length: 120
Always run formatting before committing:
source env/bin/activate
make py_format
mvn spotless:apply
If make py_format fails because tools are not installed:
source env/bin/activate
make install_dev generate
make py_format
Pre-Merge Cleanup
CONNECTOR_CONTEXT.md is gitignored and stays local — no action needed for that file.
Before submitting a PR, verify:
- No scaffolding artifacts are staged (check
git status— CONNECTOR_CONTEXT.md should not appear) - Remove any
TODOcomments that are implementation instructions (keep only genuine future-work TODOs with ticket references) - Remove any temporary test fixtures or mock data files that aren't part of the test suite