* fix(data-quality): table diff
- added handling for case-insensitive columns
- added handling for different numeric types (int/float/Decimal)
- added handling of boolean test case parameters
* add migrations for table diff
* add migrations for table diff
* removed cross type diff for now. it appears to be flaky
* fixed migrations
* use casefold() instead of lower()
* - implemented utils.get_test_case_param_value
- fixed params for case sensitive column
* handle bool test case parameters
* format
* testing
* format
* list -> List
* list -> List
* - change caseSensitiveColumns default to fase
- added migration to stay backward compatible
* - removed migration files
- updated logging message for table diff migration
* changed bool test case parameters default to always be false
* format
* docs: data diff
- added the caseSensitiveColumns parameter
requires: https://github.com/open-metadata/OpenMetadata/pull/18115
* fixed test_get_bool_test_case_param
* enable logging for debugging
* remove node-gyp
* add node-gyp globally before installing deps
* reduce quick type to 10
* revert quicktype and node-gyp changes for CI
* fix unit tests
* GEN-1493 - Fix opensearch pagination
* GEN-1494 - Add CI for py-tests with Postgres and Opensearch
* GEN-1494 - Add CI for py-tests with Postgres and Opensearch
* Fix backend tests and have index alias with clusterAlias appended
* Fix backend tests and have index alias with clusterAlias appended
* Fix failing tests
* alias setup
* fix suggestion not working due to alias
* fix getIndexOrAliasName method for multiple indexes
* update openmetadata.yaml
* update childAliases with clusterAlias
---------
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
* Limits
* Limits
* - Mismatched Types
* Update Limits config response
* Update Limits feature response
* Limits
* Limits
* - Mismatched Types
* Update Limits config response
* Update Limits feature response
* Limits: add entity resource enforcer
* Limits: fix rebase
* update limits enforcement
* Add OperationContext to limits
* chore: Bump versions to `1.4.0`
* chore: Bump Ingestion Versions to `1.4.0.1` for Release
* chore: Bump Ingestion Versions to `1.4.0.1` in Dockerfiles for Release
* Remove Retry From Abstract Event Consumer (#16405)
(cherry picked from commit f8ed079731)
* Fix Migrations: Add postgres migrations (#16403)
(cherry picked from commit 9416a7ac5f)
* Add Null Check for isAdmin (#16407)
* Remove Retry From Abstract Event Consumer
* - Add Check for null Or Empty in isAdmin
* - Fix Test
(cherry picked from commit fe2db2d63c)
* Fix OpenLineage ingestor (#16416)
* Fix OpenLineage ingestor
* py format
---------
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
* Minor: added whats new for 1.4.1 (#16420)
* Minor: added whats new for 1.4.1
* added note in to whats new
* Fix SSL issue (#16412)
* chore: Bump Versions for `1.4.1` Release
* chore(release): Prepare Branch for `1.4.2`
* [MINOR] partition migration issue with redshift servics (#16452)
* fix: partition migration issue with redshift servics
* chore: typo in sql comment
(cherry picked from commit 451d73593e)
* minor(ui): update what's new for 1.4.2 (#16457)
(cherry picked from commit d55981adfd)
* fix: ingestion for dbt > 1.8.0 resource_type is not an enum (#16415)
* fix: resource_type is not an enum
* feat: add log to display finis
* improve readability
* use getattr to be compatible
* format
* Add Cache Query Param for Limits
* Only Parse view query (#16470)
* add limit check during user creation via PUT
* add limit check during user creation via PUT
* MINOR: Kafka Setup SSL Arg Fix (#16469)
* Fix#16404 - Show Node level lineage by default (#16445)
* default to node layer
* update cypress
* code cleanup
* fix cypress
(cherry picked from commit f0cda8464f)
* Invalidate count of data asset after hard delete. add limit exception to ingestion client
* - Remove Change Description from Lineage (#16488)
(cherry picked from commit 9e5c5529a8)
* - Non Indexable fields should be remvoed at the end (#16499)
(cherry picked from commit f0b0f7a942)
* fix announcement not redirect from landing page (#16506)
* fix announcement not redirect from landing page
* minor changes
* change in cypress test
(cherry picked from commit ee7cddd169)
* Fix Schema Field Null Issue (#16510)
(cherry picked from commit 022772943f)
* feat(ui): limits integration with application (#16206)
* feat(ui): limits integration with application
* support pipelineSchedules via limit api
* enforce limit to all the modules
* update banner styling
* update
* support disable option for ManageButton
* limit version
* fix spotlight
* update tests
* Add name and version history to resource limits
Refactor the getEntityIcon function and add new icon mappings
* limit version
* hide access token tab
* fix version for all the entity
* fix tests
* fix DQ tests
* Add fallback for the icon
* Revert the fallback icon changes
* Apply the limit to the add ingestion button in the service details page
* Fix the data quality tab add test button not working
* fix banner styling
* minor fix
* Fix ingestion component unit test
* Add InlineAlert component
* update entityNameLabels mapping object
* Fix the incorrect link in LimitBanner
* update pricing page url
* Create the GlobalSettingsClassBase
* Update URLs for pricing page and upgrade options
* fix global settings uncaught error
* add parameters to the resource limit API
* implement inline alerts for service and alert creation form
* update PRIVILEGES for docker
* fix layout issues
* fix tests
---------
Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
* Add token limitations
* Add token limitations
* Add appType as part of schema in ingestion pipeline (#16519)
* #16489: fix the redirect issue to new tab for tags and glossary (#16512)
* fix the redirect issue to new tab for tags and glossary
* fix the redirect on cancel icon and unit test issue
* changes as per comments
(cherry picked from commit 8d312f0853)
* Fix #16229 - Tag and Service filters for test cases (#16484)
* fix: added test case support for tags (inherit from table/column)]"
* feat: add tag and service filter for test cases
* feat: add tier query param
* fix: tests
(cherry picked from commit 6b00dde902)
* fix: None type is not iterable (#16496)
(cherry picked from commit 656da03b14)
* minor(ui): refresh token for OIDC SSO (#16483)
* minor(ui): refresh token for OIDC SSO
* remove frame window timeout issue
* increase iFrame timeout for oidc
(cherry picked from commit 1a6c4c9720)
* feat(ui): support tag & tier filter for test case (#16502)
* feat(ui): support tag & tier filter for test case
* fix tag filter
* allow single select for tier
* added service name filter
* update cypress for tags, tier & service
* add specific add for filters
* fix tier api call
(cherry picked from commit 5b71d79e8a)
* minor: sanitize activity feed editor content (#16533)
* Add appType as part of schema in ingestion pipeline (#16519)
* Fixed quicksight conn (#16537)
* fix: saml auth for new user not created (#16543)
* fix: saml auth for new user not created
* doc: add comment
* Fix#16491 - fix lineage edge description update (#16538)
* fix lineage edge description update
* fix tests
(cherry picked from commit dff0aa8dbe)
* CYPRESS: fix announcement cypress (#16536)
* fix announcement cypress
* changes as per comments
* fix the cypress failure
(cherry picked from commit fcb87b5866)
* [MINOR] Fix Test Failure for EventRegistration
* [MINOR] Fix Test Failure for EventRegistration
* [MINOR] Fix Test Failure for EventRegistration
[MINOR] Fix Test Failure for EventRegistration
* Fix Event Handlers registration Issue (#16544)
* Fix Event Handlers Issue
* Review Comments
(cherry picked from commit d374e48b79)
* [MINOR] Fix Test Failure for EventRegistration
(cherry picked from commit 4563ad4fd1)
* Fix Topic Schema missing messageSchema (#16545)
(cherry picked from commit b612dd90c0)
* Add limits exception cache in rest client
* MINOR: Ignore Cluster Information from columns (#16495)
* minor: improve the block editor initial content history (#16540)
* Minor: fixed data quality page type issue (#16556)
* #16521: fix issue in userProfilePage for roles. teams and displayName (#16527)
* fix update on roles and backlink them in user profile page
* fix teams, displayName and profile pic issue
* sonar fix
* fix cypress issue
* minor changes
(cherry picked from commit 98945cb2db)
* Empty quick filters (#16402)
* initial commit for empty quick filters
* update progress
* fix field title
* cleanup
* add tests
* unit tests
* fix encoding of search query
* add cypress tests
* add cypress
* fix flaky cypress
* fix review comments
* revert tooltip changes
* fix tests
* fix tests
(cherry picked from commit 5930cd7a7a)
* Fix#16278 : Search to display Draft glossaryTerms on Explore page (#16462)
* Fix#16278 : Search to display Draft glossaryTerms as well on Explore page
* add term status quick filter
* change aggregation key for status field
* change aggregation key for status field
* add lowercase_normalizer in status filed for aggregate api
* add cypress tests
* fix cypress
---------
Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
(cherry picked from commit ae5e9d61cc)
* [FIX] GlossaryTerm reviewers should be user or team only (#16372)
* add teams as reviewer
* Check Users to be reviewers
* Reviewers can be a team or user
* Fix check by id or name
* Review can be team or user both
* Validate Reviewers
* add multi select control
* - Fix Reviewers
* - Centralize Reviewer Relationship to EntityRepository
* - Sort
* add team as reviewer for glossary terms
* locales
* cleanup
* - Update Reviewer should remove existing reviewers
* fix selectable owner control
* fix code smells
* fix reviewer issue
* add glossary cypress
* fix patch issue on reviewers set to null
* update cypress tests
* fix cypress
* fix cypress
* fix reviewers in glossary task and supported cypress
* fix pytest
* Fix
* fix cypress
* fix code smells
* Inherited Reviewers need to be present always
* filter out inherited users
* fix cypress
* fix backend tests failure
* fix backend tests failure -checkstyle
* restrict owner to accept task in case of reviewer present
* fix pytest
---------
Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
Co-authored-by: sonikashah <sonikashah94@gmail.com>
(cherry picked from commit 9ec3d94e3b)
* Add testSuite tags, domain field and check for TestCase limits
* fix owner not showing after refersh in teams page (#16567)
(cherry picked from commit 119fcf8959)
* [ISSUE-16503] Fix createUser to use EntityResource (#16549)
* Fix createUser to use EntityResource
* fix broken tests
* Fix Tests - 3
(cherry picked from commit aeb020ae3b)
* what's new for 1.4.2 (#16568)
(cherry picked from commit c86468d992)
* address feedbacks
* fix error for bots page
* update banner text
* allow force fetch limit
* fix ingestion schedule
* Revert "Merge branch '1.4.2' into limits"
This reverts commit 8e965207a2, reversing
changes made to 4d16531965.
* Merge 1.4.2 (#16578)
* fix explore page conflicts
* fix tests
---------
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: Chira Madlani <chirag@getcollate.io>
* fix subheader
* Updating glossary reviewers should propagate reviewers in glossary term (#16580)
* highlight inherited reviewer in glossary
* locales
* use glossary name for search query
* fix glossary version cypress
* add union datatype for subfields
* Adding reviewer to glossary also adds them as an assignee to the task
* add glossary approval cypress
---------
Co-authored-by: sonikashah <sonikashah94@gmail.com>
(cherry picked from commit 4c8bf1cac1)
* Update documentation for Search Index apis (#16539)
(cherry picked from commit d3123c4914)
* cypress: fixed flakiness and announcment cypress (#16579)
* fetch latest limit for create / delete operations
* guard datAsset limit got topic, dashboard, mlmodel etc
* Fix: Ensure correct index mapping in Elasticsearch for clusterAlias (#16589)
* Fix: Ensure correct index mapping in Elasticsearch for clusterAlias
* Fix: Ensure correct index mapping in Elasticsearch for clusterAlias
(cherry picked from commit 8723b8c36a)
* cypress: fixed cypress AUT for mysql (#16446)
* cypress: fixed cypress AUT for mysql
* minor fix
* skip announcment redirection cypress
* Minor: Ensure correct index mapping in Elasticsearch for clusterAlias (#16598)
(cherry picked from commit 04543722a6)
* Fix Postgres Application listing (#16600)
* Fix Postgres Application listing
* Fix Listing
(cherry picked from commit 77dfe1f6af)
* fix limit related issue
* Fix Automations limits invalidation during the uninstall
* cypress: fixed 1.4.2 AUT cypress (#16602)
* cypress: fixed 1.4.2 AUT cypress
* fix cypress around announcement,user,glossary, lineage and mydata
* searchIndexApplication fix and minor changes
---------
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
* test: add updateJWTTokenExpiryTime util (#16606)
(cherry picked from commit 8c173bed6a)
* OSS changes for adding automator cypress tests (#16611)
* Fix Test Suite Filter (#16615)
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
(cherry picked from commit 3db41f08e2)
* MINOR: Fix Profiler for SSL Enabled Source (#16613)
* Add Test Suite SSL (#16619)
* MINOR: Fix ssl connection in usage & lineage (#16625)
* Fix owner notification (#16629)
* - Fix Task notification not getting sent to owners
* - Fix Task notification not getting sent to owners
(cherry picked from commit cc2d581eb0)
* chore(release): Prepare Branch for `1.4.3`
* - Fix User Signup (#16667)
(cherry picked from commit b4cba8a850)
* - Fix User Signup - p2
(cherry picked from commit d9ae6f6db9)
* - Update What's new (#16669)
- fix vulnerability
(cherry picked from commit 1dcb1bd46f)
* Minor: Fix incorrect alert on signup page (#16666)
* Fix Application enforceLimits during install
* Wrap the add test button with limits wrapper for column profile tab
* fix errors
* fix tests
* fix pylint
* fix tests
* fix limits
* pylint
* fix schedule options
* fix glossary spec failure
* Add domain & tags to testSuite
* Update airflow-apis-tests-3_9.yml
---------
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
Co-authored-by: Chira Madlani <chirag@getcollate.io>
Co-authored-by: Pablo Takara <pjt1991@gmail.com>
Co-authored-by: Akash-Jain <15995028+akash-jain-10@users.noreply.github.com>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
Co-authored-by: Maxim Martynov <martinov_m_s_@mail.ru>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
Co-authored-by: Teddy <teddy.crepineau@gmail.com>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: Antoine Balliet <antoine.balliet@gorgias.com>
Co-authored-by: Suman Maharana <sumanmaharana786@gmail.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
Co-authored-by: sonika-shah <58761340+sonika-shah@users.noreply.github.com>
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
* Split ExternalSecretsManagerTest to new ExternalSecretsManagerTest and AWSBasedSecretsManagerTest
* implement SecretsManagerFactory to create GCPSecretsManager
* implement GCPSecretsManager
* implements gcp secret manager.
* Fix it for the GCP's rule.
* create a template of GCP
* fix compile error
* implements to use project_id
* add library for the google cloud secret manager
* add test code for using google credential in the docker container
* modify docker-compose.yml for GCP
* add google_crc32c module
* modify ways to get project id
* create a new docker-compose.yml for Google Cloud
* create a new document
* create compose file for gcp secret manager
* fix invalid styles and formats
* downgrade google library to avoid conflicting protoc versions
* Revert "fix(CI): update `run_local_docker.sh` script with cluster alias ES endpoint (#16604)"
This reverts commit baab52076f.
* Revert "chore(CI): Enable Elasticsearch cluster alias by default (#16599)"
This reverts commit a3524e14d3.
* WIP - MINOR - Clean ingestion dependencies
* test
* test
* Clean imports
* add pyiceberg for test
* Revert "add pyiceberg for test"
This reverts commit ab26942736.
* add pyiceberg for test
* Remove docker dep
* clean local docker sh
* MINOR - AKS Airflow troubleshooting docs
* Fix action
* clean local docker sh
* feat: use native backup tools
1. added mysqldump 8.3 to the ingestion container.
2. documented how to use native tools to back up and restore.
3. added deprecated message on the cli backup and restore.
* added deprecation notice for 1.3 backup
* removed 1.3.x deprecation notice
* added another backup page in 1.3 introducing SQL dump tools
* added --set-gtid-purged=OFF to the mysql dump process
* feat(nosql-profiler): row count
1. Implemented the NoSQLProfilerInterface as an entrypoint for the nosql profiler.
2. Added the NoSQLMetric as an abstract class.
3. Implemented the interface for the MongoDB database source.
4. Implemented an e2e test using testcontainers.
* added profiler support for mongodb connection
* doc
* use int_admin_ometa in test setup
* - fixed linting issue in gx
- removed unused inheritance
* moved the nosql function into the metric class
* formatting
* validate_compose: raise exception for bad status code.
* fixed import
* format
* feat: SAS Viya connector
* refactor SASCatalog to SAS
* add SAS logo to UI and connection documentation
* doc changes
* modify ingestion logic
* revert original changes
* added support for dataflow & perfect logic for reports/datatables
* add filter doc
* more updates to perfect ingestion for each asset type
* fix a bug with table lineage not created properly
* Delete ingestion/pipelines/sasCatalog.yaml
* precomit fix
* Conversion to database connector
* minor fixes
* make custom properties type generic
* Add SAS javaEnum
* add dummy variable for sas.yaml
---------
Co-authored-by: lizmc <liz.mcintosh@sas.com>
Co-authored-by: Shiyang Xiao <Shiyang.Xiao@sas.com>
* Fix supported characters in SM
* Update SM
* Fixes
* Fixes
* Improve class conversion exceptions
* Comments
* Rename noop to db secrets manager providee
* Update sm
* Fix
* db SM
* db SM
* Fix test
* UI
* Update openmetadata-ui/src/main/resources/ui/src/mocks/IngestionListTable.mock.ts
* update default
