mirror of
https://github.com/hyperdxio/hyperdx
synced 2026-04-21 13:37:15 +00:00
fix/escaped strings search (#1064)
I have problem with dynamic field in clickhouse json datatype. My database looks for now like this
```
│ CREATE TABLE click.log_json_better ↴│
│↳( ↴│
│↳ `log` JSON, ↴│
│↳ `ingest_time` DateTime64(9) DEFAULT now64(9) ↴│
│↳)
```
If some field contains string like bellow
`SELECT log.message, FROM click.log_json_better WHERE ...`
with result
```
log.message──┐
1. │ {"took":7,"errors":false,"items":[{"create":{...trunc # sometimes valid json, sometimes not
```
but this is representation, select for whole log
```
SELECT log
FROM click.log_json_better
WHERE (ingest_time = parseDateTime64BestEffort('2025-08-11T10:27:01.901588629Z', 9))...
```
truncated result with message part
```
trunc...4-536e18d9ad92"},"message":"{\"took\":7,\"errors\":false,\"items\...truc
```
<img width="1222" height="116" alt="image" src="https://github.com/user-attachments/assets/bee99ec4-708f-4a32-a7a0-4082bdc73a6c" />
so it's stored as escaped string and API is returning it as escaped string as well
`"{\"took\":7,\"errors\":false...trunc`
<img width="1731" height="553" alt="image" src="https://github.com/user-attachments/assets/97a2b608-490f-47cb-909e-3d7f9e411cc0" />
Then when I clicked on such row, it is escaped again and created query has no match in DB.
```
AND toString(`log`.message) = '{\\\"took\\\":4,...
```
<img width="1538" height="81" alt="image" src="https://github.com/user-attachments/assets/92f85860-2a1f-499b-b52a-f076e3fe0ecb" />
so I've made a simple fix with SqlString.raw. I've tried to match record with md5, to not use whole field content for search query, but no luck with playing around with escaping and stripping escaping.
At first, I've removed `Search value/object key too large.` error, because this was problem that I've encountered as first, trowing error crashed whole UI. I've had no problem with string 10x larger than 1000 chars.
This commit is contained in:
parent
823566f3ce
commit
4c459dc03e
3 changed files with 36 additions and 27 deletions
5
.changeset/fifty-phones-pretend.md
Normal file
5
.changeset/fifty-phones-pretend.md
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
---
|
||||
"@hyperdx/app": patch
|
||||
---
|
||||
|
||||
handle escaped string search correctly
|
||||
|
|
@ -176,6 +176,26 @@ describe('processRowToWhereClause', () => {
|
|||
expect(result).toBe("toString(dynamic_field)='quoted_value'");
|
||||
});
|
||||
|
||||
it('should handle Dynamic columns with escaped values', () => {
|
||||
const columnMap = new Map([
|
||||
[
|
||||
'dynamic_field',
|
||||
{
|
||||
name: 'dynamic_field',
|
||||
type: 'Dynamic',
|
||||
valueExpr: 'dynamic_field',
|
||||
jsType: JSDataType.Dynamic,
|
||||
},
|
||||
],
|
||||
]);
|
||||
|
||||
const row = { dynamic_field: '{\\"took\\":7, not a valid json' };
|
||||
const result = processRowToWhereClause(row, columnMap);
|
||||
expect(result).toBe(
|
||||
'toString(dynamic_field)=\'{\\"took\\":7, not a valid json\'',
|
||||
);
|
||||
});
|
||||
|
||||
it('should handle long strings with MD5', () => {
|
||||
const columnMap = new Map([
|
||||
[
|
||||
|
|
@ -334,26 +354,6 @@ describe('processRowToWhereClause', () => {
|
|||
'valueExpr not found for test',
|
||||
);
|
||||
});
|
||||
|
||||
it('should throw error for large Dynamic values', () => {
|
||||
const columnMap = new Map([
|
||||
[
|
||||
'dynamic_field',
|
||||
{
|
||||
name: 'dynamic_field',
|
||||
type: 'Dynamic',
|
||||
valueExpr: 'dynamic_field',
|
||||
jsType: JSDataType.Dynamic,
|
||||
},
|
||||
],
|
||||
]);
|
||||
|
||||
const row = { dynamic_field: 'a'.repeat(1001) };
|
||||
|
||||
expect(() => processRowToWhereClause(row, columnMap)).toThrow(
|
||||
'Search value/object key too large.',
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('useRowWhere', () => {
|
||||
|
|
|
|||
|
|
@ -69,17 +69,21 @@ export function processRowToWhereClause(
|
|||
return SqlString.format(`isNull(??)`, [column]);
|
||||
}
|
||||
if (value.length > 1000 || column.length > 1000) {
|
||||
throw new Error('Search value/object key too large.');
|
||||
console.warn('Search value/object key too large.');
|
||||
}
|
||||
// TODO: update when JSON type have new version
|
||||
// will not work for array/object dyanmic data
|
||||
return SqlString.format(`toString(?)=?`, [
|
||||
|
||||
// escaped strings needs raw, becuase sqlString will add another layer of escaping
|
||||
// data other than array/object will alwayas return with dobule quote(because of CH)
|
||||
// remove dobule qoute to search correctly
|
||||
return SqlString.format(`toString(?)='?'`, [
|
||||
SqlString.raw(valueExpr),
|
||||
// data other than array/object will alwayas return with dobule quote(because of CH)
|
||||
// remove dobule qoute to search correctly
|
||||
value[0] === '"' && value[value.length - 1] === '"'
|
||||
? value.slice(1, -1)
|
||||
: value,
|
||||
SqlString.raw(
|
||||
value[0] === '"' && value[value.length - 1] === '"'
|
||||
? value.slice(1, -1)
|
||||
: value,
|
||||
),
|
||||
]);
|
||||
default:
|
||||
// Handle nullish values
|
||||
|
|
|
|||
Loading…
Reference in a new issue