Commit graph

62 commits

Author SHA1 Message Date
Mike Shi
d72d1d2d26
Add ingestion key check to otel collector via OpAMP (#825)
HDX-1698
2025-05-23 01:41:35 +00:00
Warren
c60b975e29
chore: bump node to v22.16.0 (#839)
Ref: HDX-1757
2025-05-22 17:05:30 +00:00
Warren
86465a2027
fix: map CLICKHOUSE_SERVER_ENDPOINT to exporter 'endpoint' (#838)
For users connecting to ClickHouse Cloud or a TLS endpoint, add the `secure=true` query parameter or use the HTTPS protocol. Providing the full URL via the `CLICKHOUSE_SERVER_ENDPOINT` in the exporter's endpoint field should resolve this issue

Ref: HDX-1743
2025-05-22 05:56:18 +00:00
Dan Hable
d9ab75710e
feat: normalize the text case of severity text (#834) 2025-05-22 04:42:00 +00:00
Warren
e7262d1288
feat: introduce all-one-one (auth vs noauth) multi-stage build (#821)
1. Merge 'fullstack' and 'local' (auth + noauth) builds into a single Dockerfile
2. Introduce 'all-in-one-auth' and 'all-in-one-noauth' build stages
3. Lock `IS_LOCAL_APP_MODE` env var
4. Fix bug in ctrl-c exit with docker run
5. Enable alerts in local mode (no-auth)
6. Build `common-utils` on the fly (no longer needing pulling pkg from npm)

Ref: HDX-1709
Ref: HDX-1713
Ref: HDX-1254
Ref: HDX-1729

To match v2 product definition, we are going to release three images:
- hyperdx/hyperdx (--target=prod): app only without any other deps (clickhouse, otelcol, mongodb), used in default compose + helm deployment
- hyperdx/hyperdx-all-in-one (--target=all-in-one-auth): all-in-one build + auth
- hyperdx/hyperdx-local (--target=all-in-one-noauth): all-in-one build + no-auth

Production impacts:
- hyperdx/hyperdx: none
- hyperdx/hyperdx-all-in-one: new
- hyperdx/hyperdx-local: add server components (alerts, saved searches, dashboards)
2025-05-19 22:35:48 +00:00
Mike Shi
15bb9e6c7e
chore: remove unused dependencies (#791)
- removes redis from stack (using in memory rate limiter)
- deletes unused miner code (moved to browser-based pyodide module)
- fixes rate limit bug

fixes HDX-1248
2025-05-06 03:13:20 +00:00
Dan Hable
1c54b59939
feat: Handle JSON-like structured logs better (#763)
Updates the OTEL pipeline to handle structured logs better. If the body content is an OTEL map, it will merge the body map into the log attributes map. If the body is a JSON object, it will parse the JSON string into an OTEL map, then merge the fields into the log attributes map.

Replacing the Body field doesn't work since the Clickhouse exporter schema defines Body as string, so any parsed out object ends up turning back into a string. At least as log resources, it's a lighter weight means of grouping and filtering in the UI.

Ref: HDX-1453
2025-04-21 18:11:00 +00:00
Dan Hable
19431d3fe9
Moving HDX-1514 to open source repo (#761)
1. Moves the HDX-1514 log inference change into the OSS repo.
2. Bumps the OTEL collector version to match
2025-04-18 19:09:14 +00:00
Warren
f5e9a07ef1
chore: bump node version to v22 (#744) 2025-04-15 16:56:56 +00:00
Warren
d32661032b
feat: introduce RUN_SCHEDULED_TASKS_EXTERNALLY (#733)
1. new env var RUN_SCHEDULED_TASKS_EXTERNALLY to opt out in-app task process
2. introduce new `prod-extended` build that includes mongodb process
3. GA k8s dashboard (only picking the connection)
4. bake check-alert task into fullstack app build
2025-04-02 02:21:49 +00:00
Warren
2e3eee4c9c
migration: rename default rrweb table to hyperdx_sessions (#677) 2025-03-13 05:28:23 +00:00
Warren
5b2cba019e
feat: scrape local otelcol + clickhouse metrics (#633)
<img width="1329" alt="Screenshot 2025-02-25 at 5 26 06 PM" src="https://github.com/user-attachments/assets/ae54c3de-3e4c-4452-84ef-dda05d23c39e" />


<img width="1321" alt="Screenshot 2025-02-25 at 5 28 06 PM" src="https://github.com/user-attachments/assets/b3eab865-d6da-44da-a2fe-79a3797790f9" />
2025-02-26 03:13:05 +00:00
Warren
9767658214
style: cleanup otel-collector config (#615)
1. add `CLICKHOUSE_USER` and `CLICKHOUSE_PASSWORD` env vars (clickhouse-exporter) - not used yet
2. disable debug exporter by default
2025-02-17 19:31:10 +00:00
Warren
7c3ac9b213
feat: introduce rrweb table to store session replay data (#586) 2025-01-31 20:22:58 +00:00
Warren
3f0e564b0d
feat: add example nginx ssl reverse proxy service (#535) 2024-12-17 18:29:54 +00:00
Warren
9993fb2097
DX: standalone app docker image (#532)
For better self-hosting experience, users should be able to run
```
docker run -e MONGO_URI=xxx -p 8080:8080 hyperdx/hyperdx:2-beta
```
to spin up the project that includes the server components
2024-12-16 23:13:16 +00:00
Warren
7a766f7977
style: remove aggregator related codes (#521) 2024-12-09 09:59:36 -08:00
Warren
6a24dcf405
feat: single app image (#519) 2024-12-06 16:27:23 -08:00
Warren
aa165fcc46 feat: move more codes 2024-11-21 21:44:33 -08:00
Warren
b16456fc39 feat: move v2 codes 2024-11-12 05:53:15 -07:00
Warren
a148c06fd4
chore: bump vector to v0.41.1 (#503) 2024-10-02 05:19:09 +00:00
Warren
f7ae1a44c8
perf: use route transform to reroute go spans (#489) 2024-08-28 00:08:01 -07:00
Warren
f1da5b6d0b
chore: bump vector to v0.40.1 (#490)
https://vector.dev/releases/0.40.0/

https://vector.dev/releases/0.40.1/
2024-08-27 04:18:54 +00:00
Warren
3b5ec64404
feat: support environment filtering (#482)
<img width="221" alt="Screenshot 2024-07-24 at 2 34 07 PM" src="https://github.com/user-attachments/assets/2a4f9113-c567-4ce8-9992-6c11645c2126">
2024-07-25 05:59:17 +00:00
Warren
4d6b362f9c
feat: support sentry sdk 'X-Sentry-Auth' header (#473) 2024-07-15 04:22:27 +00:00
Warren
ac88c52173
style: introduce AGGREGATOR_PAYLOAD_SIZE_LIMIT env var (#450) 2024-07-04 00:41:49 +00:00
Warren
a5e6f0195f
chore: bump vector to v0.39.0 (#449) 2024-07-01 23:52:52 +00:00
Warren
b192366f55
chore: bump node to v18.20.3 (#411) 2024-05-26 19:42:03 +00:00
Warren
148c92b504
perf: remove redundant otel-logs fields (timestamp + spanID + traceID) (#386)
Before:
<img width="500" alt="Screenshot 2024-04-26 at 4 42 29 PM" src="https://github.com/hyperdxio/hyperdx/assets/5959690/74654fcd-faac-4837-8866-25f6e3e3e69e">

After:
<img width="498" alt="Screenshot 2024-04-26 at 4 41 37 PM" src="https://github.com/hyperdxio/hyperdx/assets/5959690/9c34893f-d9f6-442d-a877-e04ea319471c">
2024-04-29 23:42:35 +00:00
Warren
f90a6873cd
DX: attach fake hdx_token on local mode (#387)
For local mode, it shouldn't require users to specify OTEL_EXPORTER_OTLP_HEADERS or HYPERDX_API_KEY
2024-04-29 22:10:39 +00:00
Mike Shi
a5a3d34b1d
Fix multi-arch local mode build (#384) 2024-04-23 23:14:12 +00:00
Warren
d1264d521c
fix: dns issue in local mode (#373) 2024-04-17 00:46:53 +00:00
Mike Shi
24d58d214f
Add HyperDX Local Mode docs (#363) 2024-04-02 02:15:00 -07:00
Warren
582c915ff8
feat: push hyperdx-local to ghcr + dockerhub (#362) 2024-04-02 07:03:49 +00:00
Mike Shi
c0ee0e19fb
Improve local mode (#361) 2024-04-01 22:42:24 -07:00
Warren
d3e270ad15
chore: bump vector to v0.37.0 (#357) 2024-03-29 17:30:01 +00:00
Mike Shi
9e4b6b4b11
Minimal local mode (#148)
Not ready for prime time yet, but does accomplish the basics of #7

A few things to iron out:
1. Clean up UI (remove log out, log patterns, etc.)
2. Tag and push image to ghcr
3. Document how to use this image
4. Allow user customization of ports
5. See if we can trim image size down a bit (it's ~2GB right now)
6. See if we can trim down memory usage (it's a bit under 1GB right now), I suspect ~half is from CH.
2024-03-29 00:46:30 +00:00
Warren
9615f54d28
fix: handle non-string type severity field (#354) 2024-03-29 00:18:58 +00:00
Warren
41d80de1f8
feat: parse legacy k8s v1 cluster events (#300) 2024-02-05 23:46:57 +00:00
Warren
a29e9794fa
Revert "feat: extract 'message' field in json body as _hdx_body" (#296)
Reverts hyperdxio/hyperdx#285

We need to handle the case when message body is json
2024-02-02 18:46:32 +00:00
Warren
e74e9af979
feat: extract 'message' field in json body as _hdx_body (#285) 2024-01-31 22:43:46 +00:00
Warren
8de2c5cf0d
feat: add new k8s.pod.status_phase metrics + handle py span ids (#264) 2024-01-22 02:08:05 +00:00
Warren
8e536e10e3
chore: bump vector to v0.35.0 (#254) 2024-01-19 23:43:03 +00:00
Warren
499c537ea5
style: inject ingestor url (otel config file) + aggregator/go-parser url (ingestor config file) through env vars (#241) 2024-01-17 01:54:25 +00:00
Warren
3e885bfa2c
fix: move span k8s tag to root (#226) 2024-01-11 07:05:03 +00:00
Warren
d5fcb570c6
feat: introduce go-parser service (#212)
Introduce a new separated go-parser service that obfuscates `db.statement` with sql lexer
2024-01-10 18:22:39 +00:00
Warren
b8133eb46f
feat: allow users to specify 'service.name' attr (flyio) (#169) 2023-12-29 18:11:13 +00:00
Warren
58d928cd7b
feat: transform k8s event semantic conventions (#168) 2023-12-29 18:08:11 +00:00
Warren
29d1e03071
fix: infer log level by the order of severity + metrics additional filtering (#155) 2023-12-15 14:17:30 -08:00
Warren
9dc7750080
fix: extend level inference scanning range (#149) 2023-12-11 06:48:20 +00:00