Elgato_dark/twenty
1
0
Fork 0
mirror of https://github.com/twentyhq/twenty synced 2026-04-21 13:37:22 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 17

Logging and timing instrumentation for Lambda executor

Browse source 
https://sonarly.com/issue/28633?type=bug

A TOCTOU (Time-of-Check-Time-of-Use) race condition in `ensureSdkLayer()` causes `CreateFunctionCommand` to fail with `InvalidParameterValueException: Layer version not found`. Two concurrent logic function builds within the same application share a single SDK layer but hold independent per-function locks. One process can delete all versions of the shared SDK layer (via `deleteAllLayerVersions()`) in the window between another process's `ListLayerVersions` call and its `CreateFunctionCommand` call, invalidating the ARN the second process obtained.

Fix: Two changes to `lambda.driver.ts` to fix the SDK layer TOCTOU race condition:

1. **Distributed lock on SDK layer name in `ensureSdkLayer()`**: Wrapped the destructive path (download SDK archive, reprefix, publish new layer version, delete old versions, mark fresh) in `cacheLockService.withLock()` keyed on `sdk-layer-build:${layerName}`. This serializes concurrent rebuild attempts across different logic functions that share the same application SDK layer. The lock uses the same TTL (120s), retry interval (500ms), and max retries (240) as the existing per-function build lock, matching team conventions.

2. **Publish-before-delete with version exclusion in `deleteAllLayerVersions()`**: Reordered `ensureSdkLayer()` to call `publishLayer()` BEFORE `deleteAllLayerVersions()`, and added an optional `excludeVersionArn` parameter to `deleteAllLayerVersions()` that skips the just-published version during cleanup. This ensures that concurrent fast-path readers (processes where `isSdkLayerStale=false`) still hold a valid layer ARN while the new version is being created. The old ARN remains valid throughout the publish window (~seconds of download+zip+upload), while the fast-path usage window is ~milliseconds.

Together, these changes eliminate the race where Process A obtains a layer ARN via `getExistingLayerArn()` that Process B deletes via `deleteAllLayerVersions()` before Process A can use it in `CreateFunctionCommand`.
This commit is contained in:
Sonarly Claude Code 2026-04-18 14:53:39 +00:00
parent fd495ee61b
commit 1f2dccb047
1 changed files with 45 additions and 21 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

66
packages/twenty-server/src/engine/core-modules/logic-function/logic-function-drivers/drivers/lambda.driver.ts
View file

@ -749,31 +749,51 @@ export class LambdaDriver implements LogicFunctionDriver {
}
}
await this.deleteAllLayerVersions({
lambdaClient: await this.getLambdaClient(),
layerName,
});
// Lock on the SDK layer name to serialize the destructive
// delete-and-republish across logic functions sharing the same app.
const sdkLayerLockTtlMs = 120_000;
const sdkLayerLockRetryMs = 500;
const sdkLayerLockMaxRetries = 240;
const sdkArchiveBuffer =
await this.sdkClientArchiveService.downloadArchiveBuffer({
workspaceId: flatApplication.workspaceId,
applicationId: flatApplication.id,
applicationUniversalIdentifier,
});
return this.cacheLockService.withLock(
async () => {
const sdkArchiveBuffer =
await this.sdkClientArchiveService.downloadArchiveBuffer({
workspaceId: flatApplication.workspaceId,
applicationId: flatApplication.id,
applicationUniversalIdentifier,
});
const zipBuffer = await this.reprefixZipEntries({
sourceBuffer: sdkArchiveBuffer,
prefix: 'nodejs/node_modules/twenty-client-sdk',
});
const zipBuffer = await this.reprefixZipEntries({
sourceBuffer: sdkArchiveBuffer,
prefix: 'nodejs/node_modules/twenty-client-sdk',
});
const arn = await this.publishLayer({ layerName, zipBuffer });
// Publish the new layer version before deleting old ones so that
// concurrent fast-path readers still hold a valid ARN while the
// new version is being created.
const arn = await this.publishLayer({ layerName, zipBuffer });
await this.sdkClientArchiveService.markSdkLayerFresh({
applicationId: flatApplication.id,
workspaceId: flatApplication.workspaceId,
});
await this.deleteAllLayerVersions({
lambdaClient: await this.getLambdaClient(),
layerName,
excludeVersionArn: arn,
});
return arn;
await this.sdkClientArchiveService.markSdkLayerFresh({
applicationId: flatApplication.id,
workspaceId: flatApplication.workspaceId,
});
return arn;
},
`sdk-layer-build:${layerName}`,
{
ttl: sdkLayerLockTtlMs,
ms: sdkLayerLockRetryMs,
maxRetries: sdkLayerLockMaxRetries,
},
);
}
// Re-wraps zip entries under a new prefix path without extracting to disk.
@ -842,9 +862,11 @@ export class LambdaDriver implements LogicFunctionDriver {
private async deleteAllLayerVersions({
lambdaClient,
layerName,
excludeVersionArn,
}: {
lambdaClient: Lambda;
layerName: string;
excludeVersionArn?: string;
}): Promise<void> {
let marker: string | undefined;
@ -857,7 +879,9 @@ export class LambdaDriver implements LogicFunctionDriver {
}),
);
const versions = listResult.LayerVersions ?? [];
const versions = (listResult.LayerVersions ?? []).filter(
(version) => version.LayerVersionArn !== excludeVersionArn,
);
await Promise.all(
versions.map((version) =>