perf: optimize memory usage in denoised-rows query by conditionally including processedRows (#1177)

When the searching row limits is set very high (ex the max of 100k) the app quickly consumes all available memory and crashes.

This adds some improvements to help mitigate the problem: 

1. **QueryKey Issues** - The `queryKey` is generating a ton of extra entries every time the `processedRows` changes (which is every 5s when in live mode). The queryKey and result is cached regardless of if enabled is true or false. The base hashFn strategy is to [stringify the objects](2a00fb6504/packages/query-core/src/utils.ts (L216-L217)) which creates a very large string to be stored in memory. I tried to fix this by providing a custom `queryKeyHashFn` to `useQuery` but it was too slow, and the faster browser based hashing fns return a promise which isn't supported by `useQuery` at this time. The easiest solution I found was to short circuit the hash generation if we are not denoising.
2. **Sync `gcTime`** - We already set `gcTime` in `useOffsetPaginatedQuery` so I added that field here too, this helps keep the memory usage lower while denoising rows (but the memory still is much higher).

**The app still uses very high memory usage, just from the sheer number of rows being captured and processed**, but it doesn't crash anymore. There is definitely further optimizations we could make to reduce this. One solution that comes to mind is storing a hash/unique id of each row server side before sending to the client, then our app can leverage this key instead of a stringified object.

Before (after 1 min):
<img width="645" height="220" alt="Screenshot 2025-09-17 at 4 05 59 PM" src="https://github.com/user-attachments/assets/dab0ba34-4e92-42ce-90a0-fefadd9f0556" />

After (after 5 mins):
<img width="1887" height="940" alt="Screenshot 2025-09-17 at 3 52 23 PM" src="https://github.com/user-attachments/assets/bd969d2a-f0ec-4a5a-9858-409ff4a1eaa1" />

Fixes: HDX-2409
This commit is contained in:
Brandon Pereira 2025-09-23 08:08:41 -06:00 committed by GitHub
parent 8a1762e03f
commit ea5d2921ec
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
2 changed files with 15 additions and 1 deletions

View file

@ -0,0 +1,5 @@
---
"@hyperdx/app": patch
---
Improve memory efficiency in high row cound envs

View file

@ -10,6 +10,7 @@ import cx from 'classnames';
import { format, formatDistance } from 'date-fns';
import { isString } from 'lodash';
import curry from 'lodash/curry';
import ms from 'ms';
import { useHotkeys } from 'react-hotkeys-hook';
import {
Bar,
@ -1268,11 +1269,18 @@ function DBSqlRowTableComponent({
queryKey: [
'denoised-rows',
config,
processedRows,
denoiseResults,
// Only include processed rows if denoising is enabled
// This helps prevent the queryKey from getting extremely large
// and causing memory issues, when it's not used.
...(denoiseResults ? [processedRows] : []),
noisyPatternIds,
patternColumn,
],
queryFn: async () => {
if (!denoiseResults) {
return [];
}
// No noisy patterns, so no need to denoise
if (noisyPatternIds.length === 0) {
return processedRows;
@ -1296,6 +1304,7 @@ function DBSqlRowTableComponent({
}
return undefined;
},
gcTime: isLive ? ms('30s') : ms('5m'), // more aggressive gc for live data, since it can end up holding lots of data
enabled:
denoiseResults &&
noisyPatterns.isSuccess &&