* Fix#3061: Add validation to restrict region values to valid AWS regions
* address comments
* pyformat
* make aws fetch regions dynamically
* make fetching partitons dynamic
* Add sample data product ingestion and E2E test for domain data products
This adds TestDataProduct to the sample data ingestion and includes E2E
tests to verify domains and data products from sample data are properly
ingested. Also fixes TypeScript type errors in the test file by using
proper Domain['data'] type.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix sample data
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Suman Maharana <sumanmaharana786@gmail.com>
Co-authored-by: Anujkumar Yadav <anujf0510@gmail.com>
* test: add comprehensive debug logging to sample usage test
* Increase default lineage query parsing timeout from 10 to 30 sec
* test: add LineageParser 300s timeout test for CI performance measurement
* test: add lineage parser perf test to main test
* Correct query_log query with correct values and revert all debug code of test
* Revert parsing timeout to 10sec
* Update parsing timeout to 30sec and add debug tests to check time took on query
* Intentional failure to log query parsing time in CI
* Remove debug tests
* Initial Implementation of Adding Status and Reviewers to assets for workflows
* Update generated TypeScript types
* Copilot Review Comments Addressed
* Removed DataProduct Reviewer Inheritance as it is irrelevant
* Commit: Classification has status and reviewers, DataContract uses the same status enums, changed the logic to be APPROVED instead of Active, DataContract can have null status as seen in tests, Changed Workflow to use workflowStatus instead of status as it is contradicting with the approval status, Fixed Tests
* Default for reviewers is null
* Default for reviewers is createSchema
* Addressed CoPilots comments
* Update generated TypeScript types
* Workflow status to workflowStatus in db and migrations
* Revert "Workflow status to workflowStatus in db and migrations"
This reverts commit 676e878935.
* Changed status to entityStatus in the schema files
* Java Implementation of Default Status, Search Client improvements and Test fixes and new tests
* Adding entityStatus and reviewers in the searchIndex mappings and common attributes
* Data Migration scripts to change the glossaryTerm and dataContract structure
* Update generated TypeScript types
* Fixed zh/spreadsheet index json error
* Fix Postgre migration script
* Changed the entityStatus.json to status.json
Removed the duplicates of entityStatus in the indexMapping
Modified the sample data to take in EntityStatus.Approved instead of ContractStatus.Active
* Update generated TypeScript types
* dummy commit
* Fix UI Build Issues with the New EntityStatus
Fix py tests
* Migrations for all the entities that need entityStatus
* Update generated TypeScript types
* Removed Post Migration scripts
* Fix UI and py for entityStatus
* Update generated TypeScript types
* Fix: DataContractResourceTest
* Fix UI and py for importing entityStatus
* UI to show and fetch Reviewers
* cleanup
* Removed Overridden SetDefaultStatus in GlossaryTermRepository
* Removed unnecessary validation
* Added entityStatus in search_entity_index_mapping.json
* Fixed DataContractResourceTest
* mvn spotless apply and fix migration scripts
* fix tests
* fix type error
* fix advanced search tests
* Status comparison using enums and supportsStatus to supportsEntityStatus
* mvn spotless apply
* fix merge conflict
* update entity status
* fix tests
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: karanh37 <karanh37@gmail.com>
* Fix : query_cost_record_search_index Search exception for elasticsearch instance
* add sample query to cover test scenarios
* update mapping and fix test
* Fix explain turned by default, use dfs_query_then_fetch in cases of sharding of search cluster
* Add exact match configs
* Add exact match configs
* Update Logic to build search source builder with exact match priority
* Revert "Update Logic to build search source builder with exact match priority"
This reverts commit 175a2e9c6b.
* Revert "Add exact match configs"
This reverts commit 3fd5260661.
* revert display name change
* make boost mode as sum by defaul
* add more fqnparts for schema and database
* revert DFS_QUERY_THEN_FETCH since sharding wasn the issue
* use fqn split
* refactor fqn parsing
---------
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
* Fix explain turned by default, use dfs_query_then_fetch in cases of sharding of search cluster
* Add exact match configs
* Add exact match configs
* Update Logic to build search source builder with exact match priority
* Revert "Update Logic to build search source builder with exact match priority"
This reverts commit 175a2e9c6b.
* Revert "Add exact match configs"
This reverts commit 3fd5260661.
* revert display name change
* make boost mode as sum by defaul
* add more fqnparts for schema and database
* revert DFS_QUERY_THEN_FETCH since sharding wasn the issue
* use fqn split
* Refactor FQN Parts
---------
Co-authored-by: Sriharsha Chintalapani <harsha@getcollate.io>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
* feat: add query logger as an event listent in debug mode
* fix: added ingestion.src plugin to pylint
* minor: add partition sampled table
* test: added test for partitioned BQ table
* Remove log_query function from logger.py
* style: ran python linting
* Search: improve relevancy for plural/singular words, partial matches, exact matches
* apply to all indexes
* Fix other query patterns
* Revert changes of database and databaseSchema fields in TableIndex.getFields() and table index mapping
* add missing boost query builder in es
* fix ci
* add max_ngram_diff setting in di-assets index
* fix TestCaseResourceTest mvn test failure
---------
Co-authored-by: sonikashah <sonikashah94@gmail.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
* Add ER diag APIs and sample data
* fix pylint
* formatting fixes2
* fixed es client return
* fixed os client return
* supported TableDetailPage tabs as classBase for supporting collate only tabs
* Added schema Apis
* change the base class to .ts and move the component in the util files
* beautify function arguments
* Added optimizations
* Ingestion changes
* svg dimension change
* supported class base tab in databaseSchema
* supported classBase action button in schema table name column
* added further keys data for constraint modal
* fix sonar issue
* remove old method to override edit action on column and shifted to DisplayNameModal for fields
* supported table right panel component to further extends on collate side
* minor fix around duplicate constraint
* added support to update table constraints and column constraints in the UI
* code optimization and minor fixes
* review comments and multi col fix
* added queryFilter option in NodeSuggestion and tableConstrainst to fetch and use only in service tables
---------
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
* GEN-1322: API Entity - Remove Beta
* minor: add doc for the metadata pipeline
* api service refactor
* api service refactor backend changes
* add apiconnection in test service connection
* pytest fix
* fix java file formatting
* Fix casing of REST in ApiServiceRest.spec.ts
* Refactor REST to Rest in API classes
* minor change
* minor change
* minor change
* fix cashing for API to Api
* add playwright test for api service ingestion
* fix: playwright test
---------
Co-authored-by: harshsoni2024 <harshsoni2024@gmail.com>
* add sample data for api service
* pylint fix
* correct service connection json
* fix sample data
---------
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
* feat: added test defintion dimension + sample data for bounds
* chore: added migration for definition dimension
* style: ran python linting
* fix: rename dimension to dataQualityDimension
* fix: test definition dimension key
* add failed sample data
* format
* fixed masking pii data in test failed rows sample
* format
* failedRowsSamples -> failedRowsSample
* failedRowsSamples -> failedRowsSample
* fixed tests
* format
* wip
* added computePassedFailedRowCount to python client
* comment for loggerLevel
* format
* fixed tests
* tests for putting / deleting failed samples
* format
* format
* added test case for pii test
* changed method name to deleteTestCaseFailedRowsSample
* added getComputePassedFailedRowCount
