fix/escaped strings search (#1064)

I have problem with dynamic field in clickhouse json datatype.  My database looks for now like this
```
 │ CREATE TABLE click.log_json_better                                                ↴│
   │↳(                                                                                 ↴│
   │↳    `log` JSON,                                                                   ↴│
   │↳    `ingest_time` DateTime64(9) DEFAULT now64(9)                                  ↴│
   │↳) 
```
If some field contains string like bellow  
`SELECT log.message,  FROM click.log_json_better WHERE ...`
with result
```
log.message──┐
1. │ {"took":7,"errors":false,"items":[{"create":{...trunc # sometimes valid json, sometimes not 
```
but this is representation, select for whole log 
```
SELECT log
FROM click.log_json_better
WHERE (ingest_time = parseDateTime64BestEffort('2025-08-11T10:27:01.901588629Z', 9))...
```
truncated result with message part
```
trunc...4-536e18d9ad92"},"message":"{\"took\":7,\"errors\":false,\"items\...truc
```
<img width="1222" height="116" alt="image" src="https://github.com/user-attachments/assets/bee99ec4-708f-4a32-a7a0-4082bdc73a6c" />

so it's stored as escaped string and API is returning it as escaped string as well 
`"{\"took\":7,\"errors\":false...trunc`
<img width="1731" height="553" alt="image" src="https://github.com/user-attachments/assets/97a2b608-490f-47cb-909e-3d7f9e411cc0" />

Then when I  clicked on such row, it is escaped again and created query has no match in DB.
```
AND toString(`log`.message) = '{\\\"took\\\":4,...
```
<img width="1538" height="81" alt="image" src="https://github.com/user-attachments/assets/92f85860-2a1f-499b-b52a-f076e3fe0ecb" />

so I've made a simple fix with SqlString.raw. I've tried to match record  with md5, to not use whole field content for search query, but no luck with playing around with escaping and stripping escaping.

At first, I've removed `Search value/object key too large.` error, because this was problem that I've encountered as first, trowing error crashed whole UI. I've had no problem with string 10x larger than 1000 chars.
This commit is contained in:
Tomas Hulata 2025-08-14 05:58:48 +02:00 committed by GitHub
parent 823566f3ce
commit 4c459dc03e
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
3 changed files with 36 additions and 27 deletions

View file

@ -0,0 +1,5 @@
---
"@hyperdx/app": patch
---
handle escaped string search correctly

View file

@ -176,6 +176,26 @@ describe('processRowToWhereClause', () => {
expect(result).toBe("toString(dynamic_field)='quoted_value'");
});
it('should handle Dynamic columns with escaped values', () => {
const columnMap = new Map([
[
'dynamic_field',
{
name: 'dynamic_field',
type: 'Dynamic',
valueExpr: 'dynamic_field',
jsType: JSDataType.Dynamic,
},
],
]);
const row = { dynamic_field: '{\\"took\\":7, not a valid json' };
const result = processRowToWhereClause(row, columnMap);
expect(result).toBe(
'toString(dynamic_field)=\'{\\"took\\":7, not a valid json\'',
);
});
it('should handle long strings with MD5', () => {
const columnMap = new Map([
[
@ -334,26 +354,6 @@ describe('processRowToWhereClause', () => {
'valueExpr not found for test',
);
});
it('should throw error for large Dynamic values', () => {
const columnMap = new Map([
[
'dynamic_field',
{
name: 'dynamic_field',
type: 'Dynamic',
valueExpr: 'dynamic_field',
jsType: JSDataType.Dynamic,
},
],
]);
const row = { dynamic_field: 'a'.repeat(1001) };
expect(() => processRowToWhereClause(row, columnMap)).toThrow(
'Search value/object key too large.',
);
});
});
describe('useRowWhere', () => {

View file

@ -69,17 +69,21 @@ export function processRowToWhereClause(
return SqlString.format(`isNull(??)`, [column]);
}
if (value.length > 1000 || column.length > 1000) {
throw new Error('Search value/object key too large.');
console.warn('Search value/object key too large.');
}
// TODO: update when JSON type have new version
// will not work for array/object dyanmic data
return SqlString.format(`toString(?)=?`, [
// escaped strings needs raw, becuase sqlString will add another layer of escaping
// data other than array/object will alwayas return with dobule quote(because of CH)
// remove dobule qoute to search correctly
return SqlString.format(`toString(?)='?'`, [
SqlString.raw(valueExpr),
// data other than array/object will alwayas return with dobule quote(because of CH)
// remove dobule qoute to search correctly
value[0] === '"' && value[value.length - 1] === '"'
? value.slice(1, -1)
: value,
SqlString.raw(
value[0] === '"' && value[value.length - 1] === '"'
? value.slice(1, -1)
: value,
),
]);
default:
// Handle nullish values