mirror of
https://github.com/open-metadata/OpenMetadata
synced 2026-05-24 09:39:11 +00:00
84ed278720
* chore(ingestion): enable basedpyright across the codebase via baseline
Removes the ~25 paths from `[tool.basedpyright] ignore` (which excluded
roughly 90% of the codebase from type checking) and grandfathers the
existing violations into a baseline file. New violations in any
previously-ignored file now fail CI.
Changes:
- ingestion/pyproject.toml: drop the entire `ignore = [...]` block
- ingestion/setup.py: bump `basedpyright~=1.14` to `~=1.39.0`
- ingestion/.basedpyright/baseline.json (new, ~13MB): captures the
starting violation set (~18.8K errors + ~37.4K warnings) so the
migration is behavior-preserving. Regenerate with
`cd ingestion && basedpyright -p pyproject.toml --baselinefile
.basedpyright/baseline.json --writebaseline`. basedpyright analysis
has minor non-determinism (similar to ruff's), so re-running
--writebaseline a few times converges the baseline.
- ingestion/noxfile.py: pass `--baselinefile .basedpyright/baseline.json`
to the basedpyright invocation in the `static-checks` session so CI
honors the grandfathering. CI already runs the session via
`cd ingestion && nox --no-venv -s static-checks` (py-tests.yml).
- ingestion/Makefile: `make static-checks` now delegates to
`nox -s static-checks` so local invocations match CI exactly. Also
drops the dead Python 3.9 / OM_SKIP_SDK_PY39 branch (we require
Python >=3.10 since the previous modernization PR).
- .gitignore: add `.serena/` (local language-server cache)
* chore(ingestion): add nox to the dev dependency set
The static-checks Makefile target and the py-tests CI job both delegate
to `nox -s static-checks`, but nox was being installed as a separate
side step (`pip install nox` in `install_dev_env`, `uv pip install nox`
in the test-environment composite action). Listing it in dev extras
means a plain `pip install ingestion[dev]` brings it in.
* chore(ingestion): pin basedpyright analysis to py3.10; CI runs once
Following the basedpyright + multi-Python-version research:
- ingestion/pyproject.toml: add `pythonVersion = "3.10"` to
[tool.basedpyright] so type-checking always analyzes for the lowest
supported Python version. Forward-incompatible code (tomllib usage,
PEP 695 generics, etc.) is caught at type-check time regardless of
which Python interpreter runs the checker.
- .github/workflows/py-tests.yml: gate the "Run Static Checks" step on
`matrix.py-version == '3.10'`. With pythonVersion pinned, results are
identical across the matrix; running once avoids redundant work and
keeps the baseline file deterministic. Unit tests still run on the
full 3.10/3.11/3.12 matrix to verify runtime compatibility.
- ingestion/.basedpyright/baseline.json: regenerated cleanly with the
new pythonVersion config (~18.8K errors / ~37.3K warnings, similar
scale to the previous baseline). Aligns with the canonical
type-check-on-floor / test-on-matrix pattern used by Pydantic, CPython,
and other major Python projects.
* chore(ingestion): pin basedpyright pythonPlatform to Linux + regen baseline
CI's previous run still surfaced ~9 issues (2 errors + 7 warnings) that
weren't in the baseline. Root cause: my local environment differs from
CI's in three ways that affect type inference — Python interpreter
(3.11 vs 3.10), platform (Darwin vs Linux), and pip-resolved package
versions (couchbase, avro, trino, sqlalchemy stubs all differ slightly).
This commit closes the platform gap and regenerates the baseline from a
fresh CI-equivalent environment:
- ingestion/pyproject.toml: add `pythonPlatform = "Linux"` to
[tool.basedpyright] so type-checking uses the Linux subset of stdlib /
third-party stubs regardless of where the analyzer runs.
- ingestion/.basedpyright/baseline.json: regenerated against a fresh
Python 3.10 venv installed via `uv pip install ingestion[test]` (the
same install path CI's setup-openmetadata-test-environment composite
action uses). New scale: ~18.7K errors / ~37.5K warnings — same
ballpark as the previous baseline, with column positions now matching
CI's environment.
Local-developer note: when running `make static-checks` from a venv
that doesn't mirror CI exactly (e.g. macOS, Python 3.11, different
package versions), you may see drift errors. The supported workflow for
regenerating the baseline is to mirror CI:
python3.10 -m venv /tmp/ci-mirror
source /tmp/ci-mirror/bin/activate
uv pip install --upgrade pip "setuptools<81"
uv pip install --no-build-isolation "cx_Oracle>=8.3.0,<9"
uv pip install -e "ingestion[test]"
uv pip install "basedpyright~=1.39.0" nox
cd ingestion && basedpyright -p pyproject.toml \
--baselinefile .basedpyright/baseline.json --writebaseline
* chore(ingestion): drop pythonPlatform pin and regen baseline from CI-mirror
The previous attempt added `pythonPlatform = "Linux"` thinking it would
make the local-generated baseline match CI. It did the opposite — Linux
platform stubs activate additional conditional code paths that weren't
analyzed before, so CI saw 101 errors instead of the prior 2 errors.
Reverting:
- Drop `pythonPlatform = "Linux"` from [tool.basedpyright]. Without it,
basedpyright analyzes for the host platform; on CI's ubuntu-latest
runner that's Linux automatically, but type-stub coverage stays the
same as before (matching the
|
||
|---|---|---|
| .. | ||
| airflow-apis-tests.yml | ||
| auto-cherry-pick-labeled-prs.yaml | ||
| claude.yml | ||
| codeql.yml | ||
| docker-k8s-operator.yml | ||
| docker-openmetadata-db.yml | ||
| docker-openmetadata-ingestion-base-slim.yml | ||
| docker-openmetadata-ingestion-base.yml | ||
| docker-openmetadata-ingestion.yml | ||
| docker-openmetadata-postgres.yml | ||
| docker-openmetadata-server.yml | ||
| git-create-release-branch.yml | ||
| integration-tests-mysql-elasticsearch.yml | ||
| integration-tests-postgres-opensearch.yml | ||
| java-checkstyle.yml | ||
| label-connector.yml | ||
| maven-build-collate.yml | ||
| maven-sonar-build.yml | ||
| monitor-slack-link.yml | ||
| mysql-nightly-e2e.yml | ||
| openmetadata-service-unit-tests.yml | ||
| playwright-docs-check.yml | ||
| playwright-integration-tests-mysql.yml | ||
| playwright-integration-tests-postgres.yml | ||
| playwright-knowledge-graph-postgresql-e2e.yml | ||
| playwright-mysql-e2e-skip.yml | ||
| playwright-mysql-e2e.yml | ||
| playwright-postgresql-e2e-skip.yml | ||
| playwright-postgresql-e2e.yml | ||
| playwright-sso-login-nightly.yml | ||
| playwright-sso-tests.yml | ||
| postgresql-nightly-e2e.yml | ||
| publish-maven-package.yml | ||
| py-checkstyle.yml | ||
| py-cli-e2e-tests.yml | ||
| py-operator-build-test.yml | ||
| py-sonarcloud-nightly.yml | ||
| py-tests-postgres.yml | ||
| py-tests.yml | ||
| python-packages-publish.yml | ||
| security-scan.yml | ||
| stale.yml | ||
| team-labeler.yml | ||
| trivy-scan-ingestion-base-slim-image.yml | ||
| trivy-scan-ingestion-image.yml | ||
| trivy-scan-openmetadata-server.yml | ||
| typescript-type-generation.yml | ||
| ui-checkstyle.yml | ||
| update-playwright-e2e-docs.yml | ||
| validate-docker-compose-quickstart.yml | ||
| validate-jsons-yamls.yml | ||
| yarn-coverage.yml | ||