Elgato_dark/twenty
1
0
Fork 0
mirror of https://github.com/twentyhq/twenty synced 2026-04-21 13:37:22 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 17
twenty/package.json
Paul Rastoin 4ea2e32366
Refactor twenty client sdk provisioning for logic function and front-component (#18544) 
## 1. The `twenty-client-sdk` Package (Source of Truth)

The monorepo package at `packages/twenty-client-sdk` ships with:
- A **pre-built metadata client** (static, generated from a fixed
schema)
- A **stub core client** that throws at runtime (`CoreApiClient was not
generated...`)
- Both ESM (`.mjs`) and CJS (`.cjs`) bundles in `dist/`
- A `package.json` with proper `exports` map for
`twenty-client-sdk/core`, `twenty-client-sdk/metadata`, and
`twenty-client-sdk/generate`

## 2. Generation & Upload (Server-Side, at Migration Time)

**When**: `WorkspaceMigrationRunnerService.run()` executes after a
metadata schema change.

**What happens in `SdkClientGenerationService.generateAndStore()`**:
1. Copies the stub `twenty-client-sdk` package from the server's assets
(resolved via `SDK_CLIENT_PACKAGE_DIRNAME` — from
`dist/assets/twenty-client-sdk/` in production, or from `node_modules`
in dev)
2. Filters out `node_modules/` and `src/` during copy — only
`package.json` + `dist/` are kept (like an npm publish)
3. Calls `replaceCoreClient()` which uses `@genql/cli` to introspect the
**application-scoped** GraphQL schema and generates a real
`CoreApiClient`, then compiles it to ESM+CJS and overwrites
`dist/core.mjs` and `dist/core.cjs`
4. Archives the **entire package** (with `package.json` + `dist/`) into
`twenty-client-sdk.zip`
5. Uploads the single archive to S3 under
`FileFolder.GeneratedSdkClient`
6. Sets `isSdkLayerStale = true` on the `ApplicationEntity` in the
database

## 3. Invalidation Signal

The `isSdkLayerStale` boolean column on `ApplicationEntity` is the
invalidation mechanism:
- **Set to `true`** by `generateAndStore()` after uploading a new client
archive
- **Checked** by both logic function drivers before execution — if
`true`, they rebuild their local layer
- **Set back to `false`** by `markSdkLayerFresh()` after the driver has
successfully consumed the new archive

Default is `false` so existing applications without a generated client
aren't affected.

## 4a. Logic Functions — Local Driver

**`ensureSdkLayer()`** is called before every execution:
1. Checks if the local SDK layer directory exists AND `isSdkLayerStale`
is `false` → early return
2. Otherwise, cleans the local layer directory
3. Calls `downloadAndExtractToPackage()` which streams the zip from S3
directly to disk and extracts the full package into
`<tmpdir>/sdk/<workspaceId>-<appId>/node_modules/twenty-client-sdk/`
4. Calls `markSdkLayerFresh()` to set `isSdkLayerStale = false`

**At execution time**, `assembleNodeModules()` symlinks everything from
the deps layer's `node_modules/` **except** `twenty-client-sdk`, which
is symlinked from the SDK layer instead. This ensures the logic
function's `import ... from 'twenty-client-sdk/core'` resolves to the
generated client.

## 4b. Logic Functions — Lambda Driver

**`ensureSdkLayer()`** is called during `build()`:
1. Checks if `isSdkLayerStale` is `false` and an existing Lambda layer
ARN exists → early return
2. Otherwise, deletes all existing layer versions for this SDK layer
name
3. Calls `downloadArchiveBuffer()` to get the raw zip from S3 (no disk
extraction)
4. Calls `reprefixZipEntries()` which streams the zip entries into a
**new zip** with the path prefix
`nodejs/node_modules/twenty-client-sdk/` — this is the Lambda layer
convention path. All done in memory, no disk round-trip
5. Publishes the re-prefixed zip as a new Lambda layer via
`publishLayer()`
6. Calls `markSdkLayerFresh()`

**At function creation**, the Lambda is created with **two layers**:
`[depsLayerArn, sdkLayerArn]`. The SDK layer is listed last so it
overwrites the stub `twenty-client-sdk` from the deps layer (later
layers take precedence in Lambda's `/opt` merge).

## 5. Front Components

Front components are built by `app:build` with `twenty-client-sdk/core`
and `twenty-client-sdk/metadata` as **esbuild externals**. The stored
`.mjs` in S3 has unresolved bare import specifiers like `import {
CoreApiClient } from 'twenty-client-sdk/core'`.

SDK import resolution is split between the **frontend host** (fetching &
caching SDK modules) and the **Web Worker** (rewriting imports):

**Server endpoints**:
- `GET /rest/front-components/:id` —
`FrontComponentService.getBuiltComponentStream()` returns the **raw
`.mjs`** directly from file storage. No bundling, no SDK injection.
- `GET /rest/sdk-client/:applicationId/:moduleName` —
`SdkClientController` reads a single file (e.g. `dist/core.mjs`) from
the generated SDK archive via
`SdkClientGenerationService.readFileFromArchive()` and serves it as
JavaScript.

**Frontend host** (`FrontComponentRenderer` in `twenty-front`):
1. Queries `FindOneFrontComponent` which returns `applicationId`,
`builtComponentChecksum`, `usesSdkClient`, and `applicationTokenPair`
2. If `usesSdkClient` is `true`, renders
`FrontComponentRendererWithSdkClient` which calls the
`useApplicationSdkClient` hook
3. `useApplicationSdkClient({ applicationId, accessToken })` checks the
Jotai atom family cache for existing blob URLs. On cache miss, fetches
both SDK modules from `GET /rest/sdk-client/:applicationId/core` and
`/metadata`, creates **blob URLs** for each, and stores them in the atom
family
4. Once the blob URLs are cached, passes them as `sdkClientUrls`
(already blob URLs, not server URLs) to `SharedFrontComponentRenderer` →
`FrontComponentWorkerEffect` → worker's `render()` call via
`HostToWorkerRenderContext`

**Worker** (`remote-worker.ts` in `twenty-sdk`):
1. Fetches the raw component `.mjs` source as text
2. If `sdkClientUrls` are provided and the source contains SDK import
specifiers (`twenty-client-sdk/core`, `twenty-client-sdk/metadata`),
**rewrites** the bare specifiers to the blob URLs received from the host
(e.g. `'twenty-client-sdk/core'` → `'blob:...'`)
3. Creates a blob URL for the rewritten source and `import()`s it
4. Revokes only the component blob URL after the module is loaded — the
SDK blob URLs are owned and managed by the host's Jotai cache

This approach eliminates server-side esbuild bundling on every request,
caches SDK modules per application in the frontend, and keeps the
worker's job to a simple string rewrite.

## Summary Diagram

```
app:build (SDK)
  └─ twenty-client-sdk stub (metadata=real, core=stub)
       │
       ▼
WorkspaceMigrationRunnerService.run()
  └─ SdkClientGenerationService.generateAndStore()
       ├─ Copy stub package (package.json + dist/)
       ├─ replaceCoreClient() → regenerate core.mjs/core.cjs
       ├─ Zip entire package → upload to S3
       └─ Set isSdkLayerStale = true
              │
     ┌────────┴────────────────────┐
     ▼                             ▼
Logic Functions               Front Components
     │                             │
     ├─ Local Driver               ├─ GET /rest/sdk-client/:appId/core
     │   └─ downloadAndExtract     │    → core.mjs from archive
     │      → symlink into         │
     │        node_modules         ├─ Host (useApplicationSdkClient)
     │                             │    ├─ Fetch SDK modules
     └─ Lambda Driver              │    ├─ Create blob URLs
         └─ downloadArchiveBuffer  │    └─ Cache in Jotai atom family
            → reprefixZipEntries   │
            → publish as Lambda    ├─ GET /rest/front-components/:id
              layer                │    → raw .mjs (no bundling)
                                   │
                                   └─ Worker (browser)
                                        ├─ Fetch component .mjs
                                        ├─ Rewrite imports → blob URLs
                                        └─ import() rewritten source
```

## Next PR
- Estimate perf improvement by implementing a redis caching for front
component client storage ( we don't even cache front comp initially )
- Implem frontent blob invalidation sse event from server

---------

Co-authored-by: Charles Bochet <charlesBochet@users.noreply.github.com>
2026-03-24 18:10:25 +00:00

222 lines
7.2 KiB
JSON
Raw Blame History

{
"private": true,
"dependencies": {
"@apollo/client": "^4.0.0",
"@floating-ui/react": "^0.24.3",
"@linaria/core": "^6.2.0",
"@linaria/react": "^6.2.1",
"@radix-ui/colors": "^3.0.0",
"@sniptt/guards": "^0.2.0",
"@tabler/icons-react": "^3.31.0",
"@wyw-in-js/babel-preset": "^1.0.6",
"@wyw-in-js/vite": "^0.7.0",
"archiver": "^7.0.1",
"danger-plugin-todos": "^1.3.1",
"date-fns": "^2.30.0",
"date-fns-tz": "^2.0.0",
"deep-equal": "^2.2.2",
"file-type": "16.5.4",
"framer-motion": "^11.18.0",
"fuse.js": "^7.1.0",
"googleapis": "105",
"hex-rgb": "^5.0.0",
"immer": "^10.1.1",
"jotai": "^2.17.1",
"libphonenumber-js": "^1.10.26",
"lodash.camelcase": "^4.3.0",
"lodash.chunk": "^4.2.0",
"lodash.compact": "^3.0.1",
"lodash.escaperegexp": "^4.1.2",
"lodash.groupby": "^4.6.0",
"lodash.identity": "^3.0.0",
"lodash.isempty": "^4.4.0",
"lodash.isequal": "^4.5.0",
"lodash.isobject": "^3.0.2",
"lodash.kebabcase": "^4.1.1",
"lodash.mapvalues": "^4.6.0",
"lodash.merge": "^4.6.2",
"lodash.omit": "^4.5.0",
"lodash.pickby": "^4.6.0",
"lodash.snakecase": "^4.1.1",
"lodash.upperfirst": "^4.3.1",
"microdiff": "^1.3.2",
"next-with-linaria": "^1.3.0",
"planer": "^1.2.0",
"pluralize": "^8.0.0",
"react": "^18.2.0",
"react-dom": "^18.2.0",
"react-responsive": "^9.0.2",
"react-router-dom": "^6.30.3",
"react-tooltip": "^5.13.1",
"remark-gfm": "^4.0.1",
"rxjs": "^7.2.0",
"semver": "^7.5.4",
"slash": "^5.1.0",
"temporal-polyfill": "^0.3.0",
"ts-key-enum": "^2.0.12",
"tslib": "^2.8.1",
"type-fest": "4.10.1",
"typescript": "5.9.2",
"uuid": "^9.0.0",
"vite-tsconfig-paths": "^4.2.1",
"xlsx-ugnis": "^0.19.3",
"zod": "^4.1.11"
},
"devDependencies": {
"@babel/core": "^7.14.5",
"@babel/preset-react": "^7.14.5",
"@babel/preset-typescript": "^7.24.6",
"@chromatic-com/storybook": "^4.1.3",
"@graphql-codegen/cli": "^3.3.1",
"@graphql-codegen/client-preset": "^4.1.0",
"@graphql-codegen/typescript": "^3.0.4",
"@graphql-codegen/typescript-operations": "^3.0.4",
"@graphql-codegen/typescript-react-apollo": "^3.3.7",
"@nx/jest": "22.5.4",
"@nx/js": "22.5.4",
"@nx/react": "22.5.4",
"@nx/storybook": "22.5.4",
"@nx/vite": "22.5.4",
"@nx/web": "22.5.4",
"@oxlint/plugins": "^1.51.0",
"@sentry/types": "^8",
"@storybook-community/storybook-addon-cookie": "^5.0.0",
"@storybook/addon-coverage": "^3.0.0",
"@storybook/addon-docs": "^10.2.13",
"@storybook/addon-links": "^10.2.13",
"@storybook/addon-vitest": "^10.2.13",
"@storybook/icons": "^2.0.1",
"@storybook/react-vite": "^10.2.13",
"@storybook/test-runner": "^0.24.2",
"@swc-node/register": "^1.11.1",
"@swc/cli": "^0.7.10",
"@swc/core": "^1.15.11",
"@swc/helpers": "~0.5.19",
"@swc/jest": "^0.2.39",
"@testing-library/dom": "^10.4.0",
"@testing-library/jest-dom": "^6.6.3",
"@testing-library/react": "^16.3.0",
"@types/addressparser": "^1.0.3",
"@types/bcrypt": "^5.0.0",
"@types/bytes": "^3.1.1",
"@types/chrome": "^0.0.267",
"@types/deep-equal": "^1.0.1",
"@types/fs-extra": "^11.0.4",
"@types/graphql-fields": "^1.3.6",
"@types/inquirer": "^9.0.9",
"@types/jest": "^30.0.0",
"@types/lodash.camelcase": "^4.3.7",
"@types/lodash.compact": "^3.0.9",
"@types/lodash.escaperegexp": "^4.1.9",
"@types/lodash.groupby": "^4.6.9",
"@types/lodash.identity": "^3.0.9",
"@types/lodash.isempty": "^4.4.7",
"@types/lodash.isequal": "^4.5.7",
"@types/lodash.isobject": "^3.0.7",
"@types/lodash.kebabcase": "^4.1.7",
"@types/lodash.mapvalues": "^4.6.9",
"@types/lodash.omit": "^4.5.9",
"@types/lodash.pickby": "^4.6.9",
"@types/lodash.snakecase": "^4.1.7",
"@types/lodash.upperfirst": "^4.3.7",
"@types/ms": "^0.7.31",
"@types/node": "^24.0.0",
"@types/passport-google-oauth20": "^2.0.11",
"@types/passport-jwt": "^3.0.8",
"@types/passport-microsoft": "^2.1.0",
"@types/pluralize": "^0.0.33",
"@types/react": "^18.2.39",
"@types/react-datepicker": "^6.2.0",
"@types/react-dom": "^18.2.15",
"@types/supertest": "^2.0.11",
"@types/uuid": "^9.0.2",
"@typescript/native-preview": "^7.0.0-dev.20260116.1",
"@vitejs/plugin-react-swc": "4.2.3",
"@vitest/browser-playwright": "^4.0.18",
"@vitest/coverage-istanbul": "^4.0.18",
"@vitest/coverage-v8": "^4.0.18",
"@yarnpkg/types": "^4.0.0",
"chromatic": "^6.18.0",
"concurrently": "^8.2.2",
"danger": "^13.0.4",
"dotenv-cli": "^7.4.4",
"esbuild": "^0.25.10",
"http-server": "^14.1.1",
"jest": "29.7.0",
"jest-environment-jsdom": "30.0.0-beta.3",
"jest-environment-node": "^29.4.1",
"jest-fetch-mock": "^3.0.3",
"jsdom": "~22.1.0",
"msw": "^2.12.7",
"msw-storybook-addon": "^2.0.6",
"nx": "22.5.4",
"prettier": "^3.1.1",
"raw-loader": "^4.0.2",
"rimraf": "^5.0.5",
"source-map-support": "^0.5.20",
"storybook": "^10.2.13",
"storybook-addon-mock-date": "2.0.0",
"storybook-addon-pseudo-states": "^10.2.13",
"supertest": "^6.1.3",
"ts-jest": "^29.1.1",
"ts-loader": "^9.2.3",
"ts-node": "10.9.1",
"tsc-alias": "^1.8.16",
"tsconfig-paths": "^4.2.0",
"tsx": "^4.17.0",
"verdaccio": "^6.3.1",
"vite": "^7.0.0",
"vitest": "^4.0.18"
},
"engines": {
"node": "^24.5.0",
"npm": "please-use-yarn",
"yarn": ">=4.0.2"
},
"license": "AGPL-3.0",
"name": "twenty",
"packageManager": "yarn@4.9.2",
"resolutions": {
"graphql": "16.8.1",
"type-fest": "4.10.1",
"typescript": "5.9.2",
"graphql-redis-subscriptions/ioredis": "^5.6.0",
"@lingui/core": "5.1.2",
"@types/qs": "6.9.16",
"@wyw-in-js/transform@npm:0.6.0": "patch:@wyw-in-js/transform@npm%3A0.7.0#~/.yarn/patches/@wyw-in-js-transform-npm-0.7.0-ba641dc99f.patch",
"@wyw-in-js/transform@npm:0.7.0": "patch:@wyw-in-js/transform@npm%3A0.7.0#~/.yarn/patches/@wyw-in-js-transform-npm-0.7.0-ba641dc99f.patch"
},
"version": "0.2.1",
"nx": {},
"scripts": {
"docs:generate": "tsx packages/twenty-docs/scripts/generate-docs-json.ts",
"docs:generate-navigation-template": "tsx packages/twenty-docs/scripts/generate-navigation-template.ts",
"docs:generate-paths": "tsx packages/twenty-docs/scripts/generate-documentation-paths.ts",
"start": "npx concurrently --kill-others 'npx nx run-many -t start -p twenty-server twenty-front' 'npx wait-on tcp:3000 && npx nx run twenty-server:worker'"
},
"workspaces": {
"packages": [
"packages/twenty-front",
"packages/twenty-server",
"packages/twenty-emails",
"packages/twenty-ui",
"packages/twenty-utils",
"packages/twenty-zapier",
"packages/twenty-website",
"packages/twenty-docs",
"packages/twenty-e2e-testing",
"packages/twenty-shared",
"packages/twenty-sdk",
"packages/twenty-client-sdk",
"packages/twenty-apps",
"packages/twenty-cli",
"packages/create-twenty-app",
"packages/twenty-oxlint-rules"
]
},
"prettier": {
"singleQuote": true,
"trailingComma": "all",
"endOfLine": "lf"
}
}
Reference in a new issue View git blame Copy permalink