From ad80a503543f3faf7b427f5dddb81ab8bd23a646 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Yannik=20S=C3=BC=C3=9F?= Date: Tue, 4 Nov 2025 12:10:11 +0100 Subject: [PATCH] feat: add Webmetic Visitor Intelligence (#15551) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit # Webmetic Visitor Intelligence Automatically sync B2B website visitor data into Twenty CRM. Identify anonymous companies visiting your website and track their engagement without forms or manual entry. Every hour, Webmetic enriches your CRM with actionable sales intelligence about who's researching your product before they ever fill out a contact form. ## Features - šŸ”„ **Hourly Automatic Sync**: Fetches visitor data every hour via cron trigger - šŸ¢ **Company Enrichment**: Creates and updates company records with visitor intelligence - šŸ“Š **Website Lead Tracking**: Records individual visits with detailed engagement metrics - šŸ“ˆ **Engagement Scoring**: Webmetic's proprietary scoring algorithm (0-100) identifies warm leads - šŸŽÆ **Sales Intelligence**: See which companies are actively researching your product - šŸŒ **Geographic Data**: Captures visitor city and country information - šŸ”— **UTM Parameter Tracking**: Full campaign attribution with utm_source, utm_medium, utm_campaign, utm_term, and utm_content - šŸ“„ **Page Journey Mapping**: Records complete navigation paths and scroll depth - āš” **Smart Deduplication**: Prevents duplicate records using session-based identification - šŸ” **Production-Ready**: Built with rate limiting, error handling, and idempotent operations ## Requirements - `twenty-cli` ā€” `npm install -g twenty-cli` - A Twenty API key (create one at `https://twenty.com/settings/api-webhooks` and name it **"Webmetic"**) - A Webmetic account with API access. Sign up at [webmetic.de](https://webmetic.de) - Node 18+ (for local development) ## Metadata prerequisites The app automatically creates the `websiteLead` custom object with all required fields on first run. No manual field provisioning is needed. **Created automatically:** - `websiteLead` object with 14 custom fields (TEXT, NUMBER, DATE_TIME types) - Company relation field (Many-to-One from websiteLead to Company) - All fields are idempotent ā€” safe to re-run without errors ## Quick start ### 1. Deploy the app ```bash twenty auth login cd packages/twenty-apps/webmetic twenty app sync ``` ### 2. Configure environment variables - **First, create a Twenty API key**: - Go to **Settings ā†’ API & Webhooks ā†’ API Keys** - Click **+ Create API Key** - Name it **"Webmetic"** - Copy the generated key - **Then configure the app**: - Open **Settings ā†’ Apps ā†’ Webmetic Visitor Intelligence ā†’ Configuration** - Provide values for the required keys: - `TWENTY_API_KEY` (required secret; paste the API key you just created) - `TWENTY_API_URL` (optional; defaults to `http://localhost:3000` for local dev, set to your production URL) - `WEBMETIC_API_KEY` (required secret; get from [app.webmetic.de/?menu=api_details](https://app.webmetic.de/?menu=api_details)) - `WEBMETIC_DOMAIN` (required; your website domain to track, e.g., `example.com`) - Save the configuration ### 3. Test the function - On the app page, select **Test your function** - Click **Run** - You should see a success summary showing companies and leads created - Check **Settings ā†’ Data Model ā†’ Website Leads** to verify the custom object was created - Navigate to **Website Leads** from the sidebar to view synced visitor data ### 4. Automatic hourly sync begins The cron trigger (`0 * * * *`) runs every hour on the hour, continuously syncing new visitor data. ## How it works ### Data flow ``` Webmetic API ā”€[hourly]ā†’ sync-visitor-data ā”€[create/update]ā†’ Twenty CRM ā”‚ ā”œā”€ā†’ Company records (with enrichment data) ā””ā”€ā†’ Website Lead records (linked to companies) ``` ### Sync process 1. **Cron Trigger**: Every hour at :00 (e.g., 1:00, 2:00, 3:00) 2. **Schema Validation**: Ensures `websiteLead` object and all fields exist (creates if missing) 3. **Fetch Visitors**: Queries Webmetic API `/company-sessions` endpoint for last hour 4. **Process Companies**: For each visitor's company: - Searches for existing company by domain - Creates new company or updates existing with latest data from Webmetic - Extracts employee count from ranges (e.g., "11-50" ā†’ 50) 5. **Create Website Leads**: For each session: - Checks for duplicate (by name: "Company - Date") - Creates lead record with engagement metrics - Links to company via relation field 6. **Rate Limiting**: 800ms delay between API calls (75 requests/minute max) ### Data captured **Company enrichment (from Webmetic):** - Name, domain, address (street, city, postcode, country) - Employee count (parsed from ranges) - LinkedIn URL (if available) - Tagline/short description **Website Lead tracking:** - Visit date and session duration - Page views count and pages visited (full navigation path) - Traffic source (Direct, or utm_source/utm_medium combination) - UTM campaign parameters (campaign, term, content) - Visitor location (city, country) - Engagement score (Webmetic's 0-100 scoring) - Average scroll depth percentage - Total user interaction events (clicks, etc.) ## Configuration reference | Variable | Required | Description | |----------|----------|-------------| | `TWENTY_API_KEY` | āœ… Yes | Your Twenty API key for authentication | | `TWENTY_API_URL` | āŒ No | Base URL of your Twenty instance (defaults to `http://localhost:3000`) | | `WEBMETIC_API_KEY` | āœ… Yes | Your Webmetic API key from [app.webmetic.de/?menu=api_details](https://app.webmetic.de/?menu=api_details) | | `WEBMETIC_DOMAIN` | āœ… Yes | Website domain to track (e.g., `example.com` without protocol) | ## API integration This app uses multiple Twenty CRM APIs: **REST API** (data operations): - `GET /rest/metadata/objects` ā€” Fetch object metadata with fields - `GET /rest/companies` ā€” Find existing companies by domain - `POST /rest/companies` ā€” Create new companies - `PATCH /rest/companies/:id` ā€” Update company data - `POST /rest/websiteLeads` ā€” Create website lead records **GraphQL Metadata API** (schema management): - `createOneObject` mutation ā€” Creates custom objects (if needed) - `createOneField` mutation ā€” Creates custom fields and relations ## Website Lead object structure The app creates a custom `websiteLead` object with the following fields: | Field | Type | Description | |-------|------|-------------| | `name` | TEXT | Lead identifier (Company Name - Date) | | `company` | RELATION | Many-to-One relation to Company object | | `visitDate` | DATE_TIME | When the visit occurred | | `pageViews` | NUMBER | Number of pages viewed during session | | `sessionDuration` | NUMBER | Visit length in seconds | | `trafficSource` | TEXT | Where visitor came from (utm_source/utm_medium or Direct) | | `pagesVisited` | TEXT | List of page URLs visited (ā†’ separated, max 1000 chars) | | `utmCampaign` | TEXT | UTM campaign parameter | | `utmTerm` | TEXT | UTM term parameter (keywords for paid search) | | `utmContent` | TEXT | UTM content parameter (for A/B testing) | | `visitorCity` | TEXT | Geographic city of visitor | | `visitorCountry` | TEXT | Geographic country of visitor | | `visitCount` | NUMBER | Total number of visits from this company | | `engagementScore` | NUMBER | Webmetic engagement score (0-100) | | `averageScrollDepth` | NUMBER | Average scroll percentage (0-100) | | `totalUserEvents` | NUMBER | Total count of user interactions (clicks, etc.) | ## Troubleshooting **Issue**: No data syncing after setup - **Solution**: Run "Test your function" to manually trigger a sync and check logs. Verify your `WEBMETIC_API_KEY` and `WEBMETIC_DOMAIN` are correct. **Issue**: "Duplicate Domain Name" error - **Solution**: This occurs if you previously deleted a company. Twenty maintains unique constraints on soft-deleted records. Either restore the company from trash or contact support. **Issue**: Missing fields on websiteLead object - **Solution**: The sync function recreates missing fields automatically. Run "Test your function" once to repair the schema. **Issue**: Empty linkedinLink on companies - **Solution**: Webmetic doesn't have LinkedIn data for that specific company. The mapping is working correctly; data availability depends on Webmetic's enrichment coverage. **Issue**: Employee count not matching Webmetic - **Solution**: Webmetic returns ranges (e.g., "11-50"). The app uses the maximum value (50) to better represent company size. **Issue**: Test shows "No new visitors in the last hour" - **Solution**: Normal if you have no traffic in the last 60 minutes. Wait for actual traffic or manually adjust the time range in code for testing. ## Rate limiting and performance - **Webmetic API**: No pagination used; fetches all visitors from last hour - **Twenty API**: 800ms delay between requests (75 requests/minute) - **Processing**: Handles 14+ companies with full enrichment in under 30 seconds - **Cron schedule**: `0 * * * *` (every hour on the hour) - **Duplicate prevention**: Checks existing leads by name before creating ## Development ### Local testing ```bash cd packages/twenty-apps/webmetic yarn install # Set up .env file cp .env.example .env # Edit .env with your credentials # Sync to local Twenty instance npx twenty-cli app sync # Watch for changes npx twenty-cli app dev ``` ### Manual trigger Use the Twenty UI test panel or trigger via API: ```bash curl -X POST http://localhost:3000/functions/sync-visitor-data \ -H "Authorization: Bearer YOUR_API_KEY" ``` ## Architecture notes - **100% programmatic schema**: Fields created via GraphQL Metadata API, not manifests - **Idempotent operations**: Safe to re-run without duplicates or errors - **Smart domain matching**: Normalizes domains (strips www, protocols) for matching - **Error resilience**: Individual company failures don't stop the entire sync - **Detailed logging**: Returns full execution log in response for debugging ## Contributing Built with šŸŗ and ā¤ļø in Munich by [Team Webmetic](https://webmetic.de) for Twenty CRM Hacktoberfest 2025. For issues or questions: - Webmetic API: [webmetic.de](https://webmetic.de) - Twenty CRM: [twenty.com/developers](https://twenty.com/developers) ## License MIT --- packages/twenty-apps/webmetic/.env.example | 4 + packages/twenty-apps/webmetic/.yarnrc.yml | 3 + packages/twenty-apps/webmetic/README.md | 237 +++++++++ packages/twenty-apps/webmetic/package.json | 40 ++ .../serverlessFunction.manifest.jsonc | 12 + .../sync-visitor-data/src/index.ts | 451 ++++++++++++++++++ .../sync-visitor-data/src/schemaSetup.ts | 325 +++++++++++++ .../sync-visitor-data/src/types.ts | 51 ++ packages/twenty-apps/webmetic/tsconfig.json | 26 + packages/twenty-apps/webmetic/yarn.lock | 273 +++++++++++ 10 files changed, 1422 insertions(+) create mode 100644 packages/twenty-apps/webmetic/.env.example create mode 100644 packages/twenty-apps/webmetic/.yarnrc.yml create mode 100644 packages/twenty-apps/webmetic/README.md create mode 100644 packages/twenty-apps/webmetic/package.json create mode 100644 packages/twenty-apps/webmetic/serverlessFunctions/sync-visitor-data/serverlessFunction.manifest.jsonc create mode 100644 packages/twenty-apps/webmetic/serverlessFunctions/sync-visitor-data/src/index.ts create mode 100644 packages/twenty-apps/webmetic/serverlessFunctions/sync-visitor-data/src/schemaSetup.ts create mode 100644 packages/twenty-apps/webmetic/serverlessFunctions/sync-visitor-data/src/types.ts create mode 100644 packages/twenty-apps/webmetic/tsconfig.json create mode 100644 packages/twenty-apps/webmetic/yarn.lock diff --git a/packages/twenty-apps/webmetic/.env.example b/packages/twenty-apps/webmetic/.env.example new file mode 100644 index 00000000000..4130599e6df --- /dev/null +++ b/packages/twenty-apps/webmetic/.env.example @@ -0,0 +1,4 @@ +TWENTY_API_KEY= +TWENTY_API_URL=http://localhost:3000 +WEBMETIC_API_KEY= +WEBMETIC_DOMAIN= diff --git a/packages/twenty-apps/webmetic/.yarnrc.yml b/packages/twenty-apps/webmetic/.yarnrc.yml new file mode 100644 index 00000000000..fda2aea431d --- /dev/null +++ b/packages/twenty-apps/webmetic/.yarnrc.yml @@ -0,0 +1,3 @@ +yarnPath: .yarn/releases/yarn-4.9.2.cjs + +nodeLinker: node-modules diff --git a/packages/twenty-apps/webmetic/README.md b/packages/twenty-apps/webmetic/README.md new file mode 100644 index 00000000000..fe9a7165578 --- /dev/null +++ b/packages/twenty-apps/webmetic/README.md @@ -0,0 +1,237 @@ +# Webmetic Visitor Intelligence + +Automatically sync B2B website visitor data into Twenty CRM. Identify anonymous companies visiting your website and track their engagement without forms or manual entry. Every hour, Webmetic enriches your CRM with actionable sales intelligence about who's researching your product before they ever fill out a contact form. + +## Features + +- šŸ”„ **Hourly Automatic Sync**: Fetches visitor data every hour via cron trigger +- šŸ¢ **Company Enrichment**: Creates and updates company records with visitor intelligence +- šŸ“Š **Website Lead Tracking**: Records individual visits with detailed engagement metrics +- šŸ“ˆ **Engagement Scoring**: Webmetic's proprietary scoring algorithm (0-100) identifies warm leads +- šŸŽÆ **Sales Intelligence**: See which companies are actively researching your product +- šŸŒ **Geographic Data**: Captures visitor city and country information +- šŸ”— **UTM Parameter Tracking**: Full campaign attribution with utm_source, utm_medium, utm_campaign, utm_term, and utm_content +- šŸ“„ **Page Journey Mapping**: Records complete navigation paths and scroll depth +- āš” **Smart Deduplication**: Prevents duplicate records using session-based identification +- šŸ” **Production-Ready**: Built with rate limiting, error handling, and idempotent operations + +## Requirements + +- `twenty-cli` ā€” `npm install -g twenty-cli` +- A Twenty API key (create one at `https://twenty.com/settings/api-webhooks` and name it **"Webmetic"**) +- A Webmetic account with API access. Sign up at [webmetic.de](https://webmetic.de) +- Node 18+ (for local development) + +## Metadata prerequisites + +The app automatically creates the `websiteLead` custom object with all required fields on first run. No manual field provisioning is needed. + +**Created automatically:** +- `websiteLead` object with 14 custom fields (TEXT, NUMBER, DATE_TIME types) +- Company relation field (Many-to-One from websiteLead to Company) +- All fields are idempotent ā€” safe to re-run without errors + +## Quick start + +### 1. Deploy the app + +```bash +twenty auth login +cd packages/twenty-apps/webmetic +twenty app sync +``` + +### 2. Configure environment variables + +- **First, create a Twenty API key**: + - Go to **Settings ā†’ API & Webhooks ā†’ API Keys** + - Click **+ Create API Key** + - Name it **"Webmetic"** + - Copy the generated key +- **Then configure the app**: + - Open **Settings ā†’ Apps ā†’ Webmetic Visitor Intelligence ā†’ Configuration** + - Provide values for the required keys: + - `TWENTY_API_KEY` (required secret; paste the API key you just created) + - `TWENTY_API_URL` (optional; defaults to `http://localhost:3000` for local dev, set to your production URL) + - `WEBMETIC_API_KEY` (required secret; get from [app.webmetic.de/?menu=api_details](https://app.webmetic.de/?menu=api_details)) + - `WEBMETIC_DOMAIN` (required; your website domain to track, e.g., `example.com`) + - Save the configuration + +### 3. Test the function + +- On the app page, select **Test your function** +- Click **Run** +- You should see a success summary showing companies and leads created +- Check **Settings ā†’ Data Model ā†’ Website Leads** to verify the custom object was created +- Navigate to **Website Leads** from the sidebar to view synced visitor data + +### 4. Automatic hourly sync begins + +The cron trigger (`0 * * * *`) runs every hour on the hour, continuously syncing new visitor data. + +## How it works + +### Data flow + +``` +Webmetic API ā”€[hourly]ā†’ sync-visitor-data ā”€[create/update]ā†’ Twenty CRM + ā”‚ + ā”œā”€ā†’ Company records (with enrichment data) + ā””ā”€ā†’ Website Lead records (linked to companies) +``` + +### Sync process + +1. **Cron Trigger**: Every hour at :00 (e.g., 1:00, 2:00, 3:00) +2. **Schema Validation**: Ensures `websiteLead` object and all fields exist (creates if missing) +3. **Fetch Visitors**: Queries Webmetic API `/company-sessions` endpoint for last hour +4. **Process Companies**: For each visitor's company: + - Searches for existing company by domain + - Creates new company or updates existing with latest data from Webmetic + - Extracts employee count from ranges (e.g., "11-50" ā†’ 50) +5. **Create Website Leads**: For each session: + - Checks for duplicate (by name: "Company - Date") + - Creates lead record with engagement metrics + - Links to company via relation field +6. **Rate Limiting**: 800ms delay between API calls (75 requests/minute max) + +### Data captured + +**Company enrichment (from Webmetic):** +- Name, domain, address (street, city, postcode, country) +- Employee count (parsed from ranges) +- LinkedIn URL (if available) +- Tagline/short description + +**Website Lead tracking:** +- Visit date and session duration +- Page views count and pages visited (full navigation path) +- Traffic source (Direct, or utm_source/utm_medium combination) +- UTM campaign parameters (campaign, term, content) +- Visitor location (city, country) +- Engagement score (Webmetic's 0-100 scoring) +- Average scroll depth percentage +- Total user interaction events (clicks, etc.) + +## Configuration reference + +| Variable | Required | Description | +|----------|----------|-------------| +| `TWENTY_API_KEY` | āœ… Yes | Your Twenty API key for authentication | +| `TWENTY_API_URL` | āŒ No | Base URL of your Twenty instance (defaults to `http://localhost:3000`) | +| `WEBMETIC_API_KEY` | āœ… Yes | Your Webmetic API key from [app.webmetic.de/?menu=api_details](https://app.webmetic.de/?menu=api_details) | +| `WEBMETIC_DOMAIN` | āœ… Yes | Website domain to track (e.g., `example.com` without protocol) | + +## API integration + +This app uses multiple Twenty CRM APIs: + +**REST API** (data operations): +- `GET /rest/metadata/objects` ā€” Fetch object metadata with fields +- `GET /rest/companies` ā€” Find existing companies by domain +- `POST /rest/companies` ā€” Create new companies +- `PATCH /rest/companies/:id` ā€” Update company data +- `POST /rest/websiteLeads` ā€” Create website lead records + +**GraphQL Metadata API** (schema management): +- `createOneObject` mutation ā€” Creates custom objects (if needed) +- `createOneField` mutation ā€” Creates custom fields and relations + +## Website Lead object structure + +The app creates a custom `websiteLead` object with the following fields: + +| Field | Type | Description | +|-------|------|-------------| +| `name` | TEXT | Lead identifier (Company Name - Date) | +| `company` | RELATION | Many-to-One relation to Company object | +| `visitDate` | DATE_TIME | When the visit occurred | +| `pageViews` | NUMBER | Number of pages viewed during session | +| `sessionDuration` | NUMBER | Visit length in seconds | +| `trafficSource` | TEXT | Where visitor came from (utm_source/utm_medium or Direct) | +| `pagesVisited` | TEXT | List of page URLs visited (ā†’ separated, max 1000 chars) | +| `utmCampaign` | TEXT | UTM campaign parameter | +| `utmTerm` | TEXT | UTM term parameter (keywords for paid search) | +| `utmContent` | TEXT | UTM content parameter (for A/B testing) | +| `visitorCity` | TEXT | Geographic city of visitor | +| `visitorCountry` | TEXT | Geographic country of visitor | +| `visitCount` | NUMBER | Total number of visits from this company | +| `engagementScore` | NUMBER | Webmetic engagement score (0-100) | +| `averageScrollDepth` | NUMBER | Average scroll percentage (0-100) | +| `totalUserEvents` | NUMBER | Total count of user interactions (clicks, etc.) | + +## Troubleshooting + +**Issue**: No data syncing after setup +- **Solution**: Run "Test your function" to manually trigger a sync and check logs. Verify your `WEBMETIC_API_KEY` and `WEBMETIC_DOMAIN` are correct. + +**Issue**: "Duplicate Domain Name" error +- **Solution**: This occurs if you previously deleted a company. Twenty maintains unique constraints on soft-deleted records. Either restore the company from trash or contact support. + +**Issue**: Missing fields on websiteLead object +- **Solution**: The sync function recreates missing fields automatically. Run "Test your function" once to repair the schema. + +**Issue**: Empty linkedinLink on companies +- **Solution**: Webmetic doesn't have LinkedIn data for that specific company. The mapping is working correctly; data availability depends on Webmetic's enrichment coverage. + +**Issue**: Employee count not matching Webmetic +- **Solution**: Webmetic returns ranges (e.g., "11-50"). The app uses the maximum value (50) to better represent company size. + +**Issue**: Test shows "No new visitors in the last hour" +- **Solution**: Normal if you have no traffic in the last 60 minutes. Wait for actual traffic or manually adjust the time range in code for testing. + +## Rate limiting and performance + +- **Webmetic API**: No pagination used; fetches all visitors from last hour +- **Twenty API**: 800ms delay between requests (75 requests/minute) +- **Processing**: Handles 14+ companies with full enrichment in under 30 seconds +- **Cron schedule**: `0 * * * *` (every hour on the hour) +- **Duplicate prevention**: Checks existing leads by name before creating + +## Development + +### Local testing + +```bash +cd packages/twenty-apps/webmetic +yarn install + +# Set up .env file +cp .env.example .env +# Edit .env with your credentials + +# Sync to local Twenty instance +npx twenty-cli app sync + +# Watch for changes +npx twenty-cli app dev +``` + +### Manual trigger + +Use the Twenty UI test panel or trigger via API: + +```bash +curl -X POST http://localhost:3000/functions/sync-visitor-data \ + -H "Authorization: Bearer YOUR_API_KEY" +``` + +## Architecture notes + +- **100% programmatic schema**: Fields created via GraphQL Metadata API, not manifests +- **Idempotent operations**: Safe to re-run without duplicates or errors +- **Smart domain matching**: Normalizes domains (strips www, protocols) for matching +- **Error resilience**: Individual company failures don't stop the entire sync +- **Detailed logging**: Returns full execution log in response for debugging + +## Contributing + +Built with šŸŗ and ā¤ļø in Munich by [Team Webmetic](https://webmetic.de) for Twenty CRM Hacktoberfest 2025. + +For issues or questions: +- Webmetic API: [webmetic.de](https://webmetic.de) +- Twenty CRM: [twenty.com/developers](https://twenty.com/developers) + +## License + +MIT diff --git a/packages/twenty-apps/webmetic/package.json b/packages/twenty-apps/webmetic/package.json new file mode 100644 index 00000000000..c98f03a6aa6 --- /dev/null +++ b/packages/twenty-apps/webmetic/package.json @@ -0,0 +1,40 @@ +{ + "$schema": "https://raw.githubusercontent.com/twentyhq/twenty/main/packages/twenty-cli/schemas/appManifest.schema.json", + "version": "0.0.1", + "license": "MIT", + "engines": { + "node": "^24.5.0", + "npm": "please-use-yarn", + "yarn": ">=4.0.2" + }, + "packageManager": "yarn@4.9.2", + "universalIdentifier": "8f3c5a1e-9b2d-4c7e-a5f3-1d8e9c2b4a6f", + "name": "Webmetic Visitor Intelligence", + "description": "Automatically sync B2B website visitor data into Twenty CRM. Identify companies visiting your website and track engagement without forms or manual entry.", + "env": { + "TWENTY_API_KEY": { + "description": "Twenty API key for authentication", + "isSecret": true + }, + "TWENTY_API_URL": { + "description": "Twenty API URL (base URL of your Twenty instance)", + "isSecret": false, + "value": "" + }, + "WEBMETIC_API_KEY": { + "description": "Webmetic API key (get from hub.webmetic.de)", + "isSecret": true + }, + "WEBMETIC_DOMAIN": { + "description": "Your website domain to track (e.g., example.com)", + "isSecret": false + } + }, + "dependencies": { + "axios": "^1.12.2", + "twenty-sdk": "0.0.3" + }, + "devDependencies": { + "@types/node": "^24.7.2" + } +} diff --git a/packages/twenty-apps/webmetic/serverlessFunctions/sync-visitor-data/serverlessFunction.manifest.jsonc b/packages/twenty-apps/webmetic/serverlessFunctions/sync-visitor-data/serverlessFunction.manifest.jsonc new file mode 100644 index 00000000000..ff66efe4c2e --- /dev/null +++ b/packages/twenty-apps/webmetic/serverlessFunctions/sync-visitor-data/serverlessFunction.manifest.jsonc @@ -0,0 +1,12 @@ +{ + "$schema": "https://raw.githubusercontent.com/twentyhq/twenty/main/packages/twenty-cli/schemas/serverlessFunction.schema.json", + "universalIdentifier": "f2d4b8e6-3a1c-4f7e-9b5d-8c3e2a1f6d4b", + "name": "sync-visitor-data", + "triggers": [ + { + "universalIdentifier": "c5e9a3b7-2d8f-4c1e-a6b3-9f2e5d8c1a7b", + "type": "cron", + "schedule": "0 * * * *" + } + ] +} diff --git a/packages/twenty-apps/webmetic/serverlessFunctions/sync-visitor-data/src/index.ts b/packages/twenty-apps/webmetic/serverlessFunctions/sync-visitor-data/src/index.ts new file mode 100644 index 00000000000..fbc6f5649f6 --- /dev/null +++ b/packages/twenty-apps/webmetic/serverlessFunctions/sync-visitor-data/src/index.ts @@ -0,0 +1,451 @@ +import axios from 'axios'; +import { WebmeticResponse, WebmeticCompany } from './types'; +import { ensureWebsiteLeadObjectExists, ensureWebsiteLeadFieldsExist } from './schemaSetup'; + +// Rate limiting helper: delay between API calls to avoid hitting limits +const sleep = (ms: number) => new Promise(resolve => setTimeout(resolve, ms)); +const RATE_LIMIT_DELAY = 800; + +export const main = async (): Promise => { + // Capture logs to return in response (create early so we can log everything) + const logs: string[] = []; + const log = (message: string) => { + console.log(message); + logs.push(message); + }; + + // Debug: Log all available environment variables + log('Available environment variables: ' + Object.keys(process.env).filter(k => k.includes('TWENTY') || k.includes('WEBMETIC')).join(', ')); + + const WEBMETIC_API_KEY = process.env.WEBMETIC_API_KEY; + const WEBMETIC_DOMAIN = process.env.WEBMETIC_DOMAIN; + const TWENTY_API_KEY = process.env.TWENTY_API_KEY; + + // Base URL without /rest for metadata operations + const TWENTY_API_BASE_URL = process.env.TWENTY_API_URL !== "" && process.env.TWENTY_API_URL !== undefined + ? process.env.TWENTY_API_URL + : "http://localhost:3000"; + + // REST URL for data operations + const TWENTY_API_URL = `${TWENTY_API_BASE_URL}/rest`; + + log('WEBMETIC_API_KEY: ' + (WEBMETIC_API_KEY ? '***SET***' : 'MISSING')); + log('WEBMETIC_DOMAIN: ' + (WEBMETIC_DOMAIN || 'MISSING')); + log('TWENTY_API_KEY: ' + (TWENTY_API_KEY ? '***SET***' : 'MISSING')); + log('TWENTY_API_BASE_URL: ' + TWENTY_API_BASE_URL); + log('TWENTY_API_URL (REST): ' + TWENTY_API_URL); + + if (!WEBMETIC_API_KEY || !WEBMETIC_DOMAIN || !TWENTY_API_KEY) { + return { + success: false, + error: 'Missing required environment variables', + WEBMETIC_API_KEY: !!WEBMETIC_API_KEY, + WEBMETIC_DOMAIN: !!WEBMETIC_DOMAIN, + TWENTY_API_KEY: !!TWENTY_API_KEY, + detailedLog: logs.join('\n') + }; + } + + // IMPORTANT: Ensure websiteLead object and all required fields exist before syncing data + // Everything is created programmatically via GraphQL (object + fields + relations) + log('\n=== Phase 1: Schema Setup ==='); + try { + // First, ensure the websiteLead object exists (create via GraphQL if missing) + await ensureWebsiteLeadObjectExists(TWENTY_API_KEY, TWENTY_API_BASE_URL); + + // Then, ensure all fields exist (create via GraphQL if missing) + await ensureWebsiteLeadFieldsExist(TWENTY_API_KEY, TWENTY_API_BASE_URL); + } catch (error: any) { + log('āŒ Failed to ensure schema exists: ' + error.message); + return { + success: false, + error: `Field setup failed: ${error.message}`, + detailedLog: logs.join('\n') + }; + } + + try { + // 1. Fetch visitor data from Webmetic API using company-sessions endpoint + log('\n=== Phase 2: Data Sync ==='); + log('=== Starting Webmetic Sync ==='); + log(`Fetching visitor data from Webmetic for domain: ${WEBMETIC_DOMAIN}`); + log('Time range: -1 hour to now'); + + // Build API params + const webmeticParams: any = { + domain: WEBMETIC_DOMAIN, + from_date: '-1 hour', + to_date: 'now', + }; + log(`šŸ“” Webmetic API request params: ${JSON.stringify(webmeticParams)}`); + + const webmeticResponse = await axios.get( + 'https://hub.webmetic.de/company-sessions', + { + headers: { + Authorization: WEBMETIC_API_KEY, + }, + params: webmeticParams, + } + ); + + log(`āœ… Webmetic API responded successfully`); + log(`šŸ“¦ Webmetic returned ${webmeticResponse.data.result?.length || 0} companies`); + + const companies = webmeticResponse.data.result; + + if (!Array.isArray(companies) || companies.length === 0) { + log('ā„¹ļø No new visitors in the last hour'); + return { + success: true, + companiesProcessed: 0, + sessionsProcessed: 0, + logCount: logs.length, + detailedLog: logs.join('\n') + }; + } + + const totalSessions = companies.reduce((sum, c) => sum + c.sessions.length, 0); + log(`\nšŸ“Š Found ${companies.length} companies with ${totalSessions} total sessions`); + log('='.repeat(50)); + + // 2. Process each company + const results = { + success: true, + companiesProcessed: 0, + companiesCreated: 0, + companiesUpdated: 0, + sessionsProcessed: 0, + websiteLeadsCreated: 0, + errors: [] as string[], + }; + + for (const company of companies) { + try { + log(`\nšŸ¢ Processing: ${company.company_name} (${company.sessions.length} sessions)`); + + // Extract domain from company_url + const domainMatch = company.company_url?.match(/^https?:\/\/(?:www\.)?([^\/]+)/); + const domain = domainMatch ? domainMatch[1] : company.company_url; + log(` Domain: ${domain}`); + + // 2a. Find or create Company in Twenty + let companyId: string; + + // Try to find existing company by domain + const existingCompaniesOptions = { + method: 'GET', + headers: { + Authorization: `Bearer ${TWENTY_API_KEY}`, + }, + url: `${TWENTY_API_URL}/companies?filter=domainName.primaryLinkUrl%5Beq%5D%3A%22${encodeURIComponent(domain)}%22`, + }; + + // Enhanced logging for debugging + log(` šŸ” Searching for existing company...`); + log(` šŸ“” API Call: GET ${existingCompaniesOptions.url}`); + log(` šŸ”‘ Auth header: Bearer ***SET***`); + + const existingCompaniesResponse = await axios(existingCompaniesOptions); + log(` āœ… Response status: ${existingCompaniesResponse.status}`); + log(` šŸ“¦ Response data structure: ${JSON.stringify(Object.keys(existingCompaniesResponse.data || {}))}`); + + await sleep(RATE_LIMIT_DELAY); // Rate limiting delay + + const existingCompanies = existingCompaniesResponse.data?.data?.companies || []; + log(` Found ${existingCompanies.length} existing companies with this domain`); + + // Build company data with proper validation + const companyData: any = { + name: company.company_name, + domainName: { + primaryLinkUrl: domain, + primaryLinkLabel: domain, + }, + }; + + // Only add address if at least one field has a value + if (company.address || company.city || company.postal_code || company.country) { + companyData.address = { + addressStreet1: company.address || '', + addressCity: company.city || '', + addressPostcode: company.postal_code || '', + addressCountry: company.country || '', + }; + } + + // Parse employee count (Webmetic returns ranges like "11-50" or "501-1000") + // Take the maximum value from the range for better company size representation + if (company.employee_count) { + const employeeStr = company.employee_count.trim(); + let employeeCount: number | null = null; + + if (employeeStr.includes('-')) { + // Parse range like "11-50" - take the maximum value (50) + const parts = employeeStr.split('-'); + const max = parseInt(parts[1]); + if (!isNaN(max) && max > 0) { + employeeCount = max; + } + } else { + // Single number + employeeCount = parseInt(employeeStr); + } + + if (employeeCount && !isNaN(employeeCount) && employeeCount > 0) { + companyData.employees = employeeCount; + } + } + + // Add LinkedIn (linkedinLink exists in schema) + if (company.linkedin) { + companyData.linkedinLink = { + primaryLinkUrl: company.linkedin, + primaryLinkLabel: 'LinkedIn', + }; + } + + // Note: tagline, companyEmail, companyPhone, and industryClassification fields don't exist in the standard schema + // If you need these fields, you'll need to create custom fields in Twenty first + + if (existingCompanies && existingCompanies.length > 0) { + // Company exists - update it + companyId = existingCompanies[0].id; + log(` āœ“ Found existing company (ID: ${companyId})`); + + // Don't include domainName in update (it's unique and already set) + const updateData = { ...companyData }; + delete updateData.domainName; + + const updateOptions = { + method: 'PATCH', + headers: { + Authorization: `Bearer ${TWENTY_API_KEY}`, + 'Content-Type': 'application/json', + }, + url: `${TWENTY_API_URL}/companies/${companyId}`, + data: updateData, + }; + await axios(updateOptions); + await sleep(RATE_LIMIT_DELAY); // Rate limiting delay + + results.companiesUpdated++; + log(` āœ“ Updated company data`); + } else { + // Create new company + log(` āž• Creating new company...`); + const newCompanyOptions = { + method: 'POST', + headers: { + Authorization: `Bearer ${TWENTY_API_KEY}`, + 'Content-Type': 'application/json', + }, + url: `${TWENTY_API_URL}/companies`, + data: companyData, + }; + const newCompanyResponse = await axios(newCompanyOptions); + await sleep(RATE_LIMIT_DELAY); // Rate limiting delay + + // Extract company ID from REST API response: { data: { createCompany: { id: "..." } } } + companyId = newCompanyResponse.data?.data?.createCompany?.id; + + if (!companyId) { + log(` āš ļø Warning: Could not extract company ID from response. Full response: ${JSON.stringify(newCompanyResponse.data)}`); + } + + results.companiesCreated++; + log(` āœ“ Created company (ID: ${companyId})`); + } + + // 2b. Create WebsiteLead records for each session + log(` šŸ“ Processing ${company.sessions.length} session(s)...`); + + if (!company.sessions || company.sessions.length === 0) { + log(` āš ļø No sessions found for this company`); + } + + for (const session of company.sessions) { + try { + log(` Processing session ${session.session_id}...`); + + // Build traffic source string + const trafficSource = session.utm_source + ? `${session.utm_source}${session.utm_medium ? '/' + session.utm_medium : ''}` + : 'Direct'; + + // Extract pages visited (limit length to prevent DB errors) + const pagesArray = session.user_data.map((ud) => ud.document_location); + let pagesVisited = pagesArray.join(' ā†’ '); + + // Truncate if too long (max ~1000 chars to be safe) + if (pagesVisited.length > 1000) { + pagesVisited = pagesVisited.substring(0, 997) + '...'; + } + + // Use Webmetic's lead score + const engagementScore = company.lead_score || 0; + + // Calculate average scroll depth + const scrollDepths = session.user_data + .map((ud) => ud.scroll_depth) + .filter((sd) => typeof sd === 'number' && !isNaN(sd)); + const averageScrollDepth = scrollDepths.length > 0 + ? Math.round(scrollDepths.reduce((sum, sd) => sum + sd, 0) / scrollDepths.length) + : 0; + + // Calculate total user events + const totalUserEvents = session.user_data + .reduce((sum, ud) => sum + (ud.user_events_count || 0), 0); + + // Use session_id as unique identifier (company name available via relation) + const leadName = session.session_id; + + log(` Lead name: ${leadName}`); + + // Check if WebsiteLead already exists for this session + log(` Checking if lead already exists...`); + const existingLeadsOptions = { + method: 'GET', + headers: { + Authorization: `Bearer ${TWENTY_API_KEY}`, + }, + url: `${TWENTY_API_URL}/websiteLeads?filter=name%5Beq%5D%3A%22${encodeURIComponent(leadName)}%22`, + }; + const existingLeadsResponse = await axios(existingLeadsOptions); + await sleep(RATE_LIMIT_DELAY); // Rate limiting delay + + const existingLeads = existingLeadsResponse.data?.data?.websiteLeads || existingLeadsResponse.data; + log(` Found ${existingLeads?.length || 0} existing leads`); + + if (!existingLeads || existingLeads.length === 0) { + // Create WebsiteLead record only if it doesn't exist + const leadData: any = { + name: leadName, + visitDate: session.timestamp, + pageViews: session.user_data.length, + sessionDuration: session.session_duration, + trafficSource: trafficSource, + pagesVisited: pagesVisited, + visitCount: company.sessions.length, + engagementScore: engagementScore, + }; + + // Add custom fields + if (averageScrollDepth > 0) { + leadData.averageScrollDepth = averageScrollDepth; + } + + if (totalUserEvents > 0) { + leadData.totalUserEvents = totalUserEvents; + } + + if (session.utm_campaign) { + leadData.utmCampaign = session.utm_campaign; + } + + if (session.utm_term) { + leadData.utmTerm = session.utm_term; + } + + if (session.utm_content) { + leadData.utmContent = session.utm_content; + } + + if (session.visitor_city) { + leadData.visitorCity = session.visitor_city; + } + + if (session.visitor_country) { + leadData.visitorCountry = session.visitor_country; + } + + // Link to company if we have the ID + if (companyId) { + leadData.companyId = companyId; + } + + log(` Creating WebsiteLead with data: ${JSON.stringify(leadData)}`); + + const createOptions = { + method: 'POST', + headers: { + Authorization: `Bearer ${TWENTY_API_KEY}`, + 'Content-Type': 'application/json', + }, + url: `${TWENTY_API_URL}/websiteLeads`, + data: leadData, + }; + const createResponse = await axios(createOptions); + await sleep(RATE_LIMIT_DELAY); // Rate limiting delay + + log(` WebsiteLead creation response: ${createResponse.status}`); + + results.websiteLeadsCreated++; + log(` āœ“ Created WebsiteLead: ${leadName}`); + } else { + log(` - WebsiteLead already exists: ${leadName}`); + } + } catch (sessionError: any) { + const errorDetails = sessionError?.response?.data || sessionError?.message || 'Unknown error'; + const errorMsg = `Error processing WebsiteLead for session ${session.session_id}: ${JSON.stringify(errorDetails, null, 2)}`; + console.error(errorMsg); + log(` āŒ ${errorMsg}`); + } + } + + results.companiesProcessed++; + results.sessionsProcessed += company.sessions.length; + + } catch (error: any) { + const errorDetails = error?.response?.data || error?.message || 'Unknown error'; + const errorMsg = `Error processing company ${company.company_name}: ${ + error instanceof Error ? error.message : 'Unknown error' + }`; + const detailedError = `${errorMsg} - Details: ${JSON.stringify(errorDetails, null, 2)}`; + + // Enhanced error logging + console.error(`\nāŒ ${detailedError}`); + log(`\nāŒ ${detailedError}`); + + // Log additional diagnostic info + if (error?.response) { + log(` šŸ” HTTP Status: ${error.response.status} ${error.response.statusText}`); + log(` šŸ” Request URL: ${error.config?.url || 'Unknown'}`); + log(` šŸ” Request Method: ${error.config?.method?.toUpperCase() || 'Unknown'}`); + log(` šŸ” Auth Header: ${error.config?.headers?.Authorization ? 'Present' : 'MISSING!'}`); + } + + results.errors.push(`${errorMsg} - Details: ${JSON.stringify(errorDetails)}`); + } + } + + log('\n' + '='.repeat(50)); + log('āœ… Sync Completed Successfully!'); + log('='.repeat(50)); + log(`šŸ“Š Summary:`); + log(` Companies Processed: ${results.companiesProcessed}`); + log(` Companies Created: ${results.companiesCreated}`); + log(` Companies Updated: ${results.companiesUpdated}`); + log(` Sessions Processed: ${results.sessionsProcessed}`); + log(` Website Leads Created: ${results.websiteLeadsCreated}`); + log(` Errors: ${results.errors.length}`); + + if (results.errors.length > 0) { + log('\nāš ļø Errors encountered:'); + results.errors.forEach((err, i) => log(` ${i + 1}. ${err}`)); + } + + log('='.repeat(50) + '\n'); + + // Return results with formatted log string for easier viewing + return { + ...results, + logCount: logs.length, + detailedLog: logs.join('\n') + }; + + } catch (error) { + console.error('Fatal error during sync:', error); + throw error; + } +} diff --git a/packages/twenty-apps/webmetic/serverlessFunctions/sync-visitor-data/src/schemaSetup.ts b/packages/twenty-apps/webmetic/serverlessFunctions/sync-visitor-data/src/schemaSetup.ts new file mode 100644 index 00000000000..83c521e190b --- /dev/null +++ b/packages/twenty-apps/webmetic/serverlessFunctions/sync-visitor-data/src/schemaSetup.ts @@ -0,0 +1,325 @@ +import axios from 'axios'; + +// Rate limiting helper: delay between API calls to avoid hitting limits +const sleep = (ms: number) => new Promise(resolve => setTimeout(resolve, ms)); + +/** + * Ensures the websiteLead object exists. Creates it if missing. + * Uses GraphQL metadata API. + */ +export async function ensureWebsiteLeadObjectExists(apiKey: string, apiBaseUrl: string): Promise { + const log = (msg: string) => console.log(`[Object Setup] ${msg}`); + + try { + // Check if websiteLead object already exists + const objectsResponse = await axios.get(`${apiBaseUrl}/rest/metadata/objects`, { + headers: { Authorization: `Bearer ${apiKey}` } + }); + await sleep(800); + + const objects = objectsResponse.data?.data?.objects || []; + const existingObject = objects.find((obj: any) => obj.nameSingular === 'websiteLead'); + + if (existingObject) { + log(`āœ“ websiteLead object already exists (ID: ${existingObject.id})`); + return existingObject; + } + + // Object doesn't exist, create it via GraphQL + log('websiteLead object not found, creating it via GraphQL...'); + + const mutation = ` + mutation CreateOneObject($input: CreateOneObjectInput!) { + createOneObject(input: $input) { + id + nameSingular + namePlural + labelSingular + labelPlural + } + } + `; + + const variables = { + input: { + object: { + nameSingular: 'websiteLead', + namePlural: 'websiteLeads', + labelSingular: 'Website Lead', + labelPlural: 'Website Leads', + description: 'B2B website visitor data from Webmetic', + icon: 'IconEye' + } + } + }; + + const metadataUrl = `${apiBaseUrl}/metadata`; + const response = await axios.post(metadataUrl, { + query: mutation, + variables: variables + }, { + headers: { + Authorization: `Bearer ${apiKey}`, + 'Content-Type': 'application/json' + } + }); + + const createdObject = response.data?.data?.createOneObject; + if (!createdObject) { + throw new Error('Failed to create websiteLead object - no data returned'); + } + + log(`āœ“ Created websiteLead object (ID: ${createdObject.id})`); + await sleep(1000); + + return createdObject; + + } catch (error: any) { + const errorMsg = error?.response?.data?.errors?.[0]?.message || error.message || ''; + + // If object already exists (race condition), fetch and return it + if (errorMsg.includes('already exists') || errorMsg.includes('duplicate')) { + log('Object already exists, fetching it...'); + + const objectsResponse = await axios.get(`${apiBaseUrl}/rest/metadata/objects`, { + headers: { Authorization: `Bearer ${apiKey}` } + }); + + const existingObject = objectsResponse.data?.data?.objects?.find( + (obj: any) => obj.nameSingular === 'websiteLead' + ); + + if (existingObject) { + log(`āœ“ Found existing websiteLead object (ID: ${existingObject.id})`); + return existingObject; + } + } + + console.error('[Object Setup] Error:', error?.response?.data || error.message); + throw new Error(`Failed to ensure websiteLead object exists: ${errorMsg}`); + } +} + +/** + * Ensures the websiteLead object exists with all required fields. + * Uses GraphQL metadata API for all field creation. + */ +export async function ensureWebsiteLeadFieldsExist(apiKey: string, apiBaseUrl: string): Promise { + const log = (msg: string) => console.log(`[Field Setup] ${msg}`); + + log('Checking if websiteLead fields exist...'); + + try { + // 1. Fetch all objects using REST API + const objectsResponse = await axios.get(`${apiBaseUrl}/rest/metadata/objects`, { + headers: { Authorization: `Bearer ${apiKey}` } + }); + await sleep(800); // Rate limiting + + const objects = objectsResponse.data?.data?.objects || []; + const websiteLeadObject = objects.find((obj: any) => obj.nameSingular === 'websiteLead'); + + if (!websiteLeadObject) { + throw new Error( + 'websiteLead object not found after creation. This should not happen.' + ); + } + + log(`āœ“ Found websiteLead object (ID: ${websiteLeadObject.id})`); + + // 2. Define all required fields + const requiredFields = [ + // TEXT fields + { name: 'trafficSource', type: 'TEXT', label: 'Traffic Source', description: 'Where visitor came from (utm_source/utm_medium or Direct)', icon: 'IconWorld' }, + { name: 'pagesVisited', type: 'TEXT', label: 'Pages Visited', description: 'List of page URLs visited (max 1000 chars)', icon: 'IconFileText' }, + { name: 'utmCampaign', type: 'TEXT', label: 'UTM Campaign', description: 'UTM campaign parameter', icon: 'IconTag' }, + { name: 'utmTerm', type: 'TEXT', label: 'UTM Term', description: 'UTM term parameter (keywords for paid search)', icon: 'IconSearch' }, + { name: 'utmContent', type: 'TEXT', label: 'UTM Content', description: 'UTM content parameter (for A/B testing)', icon: 'IconFileText' }, + { name: 'visitorCity', type: 'TEXT', label: 'Visitor City', description: 'Geographic city of visitor', icon: 'IconMapPin' }, + { name: 'visitorCountry', type: 'TEXT', label: 'Visitor Country', description: 'Geographic country of visitor', icon: 'IconWorld' }, + + // NUMBER fields + { name: 'pageViews', type: 'NUMBER', label: 'Page Views', description: 'Number of pages viewed during session', icon: 'IconEye' }, + { name: 'sessionDuration', type: 'NUMBER', label: 'Session Duration', description: 'Visit length in seconds', icon: 'IconClock' }, + { name: 'visitCount', type: 'NUMBER', label: 'Visit Count', description: 'Total number of visits from this company', icon: 'IconRepeat' }, + { name: 'engagementScore', type: 'NUMBER', label: 'Engagement Score', description: 'Webmetic engagement score (0-100)', icon: 'IconChartBar' }, + { name: 'averageScrollDepth', type: 'NUMBER', label: 'Avg Scroll Depth', description: 'Average scroll percentage (0-100)', icon: 'IconArrowDown' }, + { name: 'totalUserEvents', type: 'NUMBER', label: 'Total User Events', description: 'Total count of user interactions (clicks, etc.)', icon: 'IconClick' }, + + // DATE_TIME field + { name: 'visitDate', type: 'DATE_TIME', label: 'Visit Date', description: 'When the visit occurred', icon: 'IconCalendar' }, + ]; + + // 3. Check which fields exist + const existingFields = websiteLeadObject.fields || []; + const existingFieldNames = new Set(existingFields.map((f: any) => f.name)); + const missingFields = requiredFields.filter(f => !existingFieldNames.has(f.name)); + + if (missingFields.length === 0) { + log(`āœ“ All ${requiredFields.length} required fields already exist`); + + // Check if company relation exists + const companyRelation = existingFields.find((f: any) => f.name === 'company' && f.type === 'RELATION'); + if (!companyRelation) { + log(`āš ļø Company relation field is missing - will create it`); + await createCompanyRelation(apiKey, apiBaseUrl, websiteLeadObject.id); + } else { + log(`āœ“ Company relation field exists`); + } + + return; + } + + log(`Found ${missingFields.length} missing fields, creating them now...`); + + // 4. Create missing fields using GraphQL API + const metadataUrl = `${apiBaseUrl}/metadata`; + const mutation = ` + mutation CreateOneFieldMetadataItem($input: CreateOneFieldMetadataInput!) { + createOneField(input: $input) { + id + name + label + type + } + } + `; + + for (const field of missingFields) { + try { + log(` Creating field: ${field.name} (${field.type})...`); + + const variables = { + input: { + field: { + type: field.type, + objectMetadataId: websiteLeadObject.id, + name: field.name, + label: field.label, + description: field.description, + icon: field.icon, + isNullable: true + } + } + }; + + await axios.post(metadataUrl, { + query: mutation, + variables: variables + }, { + headers: { + Authorization: `Bearer ${apiKey}`, + 'Content-Type': 'application/json' + } + }); + + log(` āœ“ Created: ${field.name}`); + await sleep(500); // 500ms delay between field creations + + } catch (error: any) { + // If field already exists (race condition), that's OK + const errorMsg = error?.response?.data?.errors?.[0]?.message || error.message || ''; + if (errorMsg.includes('already exists') || errorMsg.includes('duplicate')) { + log(` - Field ${field.name} already exists (skipping)`); + } else { + throw new Error(`Failed to create field ${field.name}: ${errorMsg}`); + } + } + } + + log(`āœ“ Created ${missingFields.length} missing fields`); + + // 5. Create company relation if it doesn't exist + const companyRelation = existingFields.find((f: any) => f.name === 'company' && f.type === 'RELATION'); + if (!companyRelation) { + await createCompanyRelation(apiKey, apiBaseUrl, websiteLeadObject.id); + } + + log('āœ“ Field setup complete - all required fields exist'); + + } catch (error: any) { + console.error('[Field Setup] Error:', error?.response?.data || error.message); + throw error; + } +} + +/** + * Creates the company relation field (Many-to-One from WebsiteLead to Company). + * This creates both the forward relation (websiteLead.company) and the inverse (company.websiteLeads). + */ +async function createCompanyRelation(apiKey: string, apiBaseUrl: string, websiteLeadObjectId: string): Promise { + const log = (msg: string) => console.log(`[Field Setup] ${msg}`); + + try { + log(' Creating company relation field (Many-to-One)...'); + + // First, fetch the company object ID + const objectsResponse = await axios.get(`${apiBaseUrl}/rest/metadata/objects`, { + headers: { Authorization: `Bearer ${apiKey}` } + }); + + const companyObject = objectsResponse.data?.data?.objects?.find((obj: any) => obj.nameSingular === 'company'); + + if (!companyObject) { + throw new Error('Company object not found'); + } + + log(` Found company object (ID: ${companyObject.id})`); + + // Use GraphQL for relation creation (REST API doesn't support relations) + const mutation = ` + mutation CreateOneFieldMetadataItem($input: CreateOneFieldMetadataInput!) { + createOneField(input: $input) { + id + name + label + type + } + } + `; + + const variables = { + input: { + field: { + name: 'company', + label: 'Company', + type: 'RELATION', + objectMetadataId: websiteLeadObjectId, + isNullable: true, + relationCreationPayload: { + type: 'MANY_TO_ONE', + targetObjectMetadataId: companyObject.id, + targetFieldLabel: 'Website Leads', + targetFieldIcon: 'IconChartBar', + } + } + } + }; + + // GraphQL endpoint is at /metadata (not /rest/metadata) + const metadataUrl = `${apiBaseUrl}/metadata`; + + await axios.post(metadataUrl, { + query: mutation, + variables: variables + }, { + headers: { + Authorization: `Bearer ${apiKey}`, + 'Content-Type': 'application/json' + } + }); + + log(` āœ“ Created company relation (websiteLead.company ā†’ Company)`); + log(` āœ“ Also created inverse relation (Company.websiteLeads ā†’ websiteLead[])`); + await sleep(500); + + } catch (error: any) { + // If relation already exists, that's OK + const errorMsg = error?.response?.data?.errors?.[0]?.message || error.message || ''; + if (errorMsg.includes('already exists') || errorMsg.includes('duplicate')) { + log(` - Company relation already exists (skipping)`); + } else { + throw new Error(`Failed to create company relation: ${errorMsg}`); + } + } +} diff --git a/packages/twenty-apps/webmetic/serverlessFunctions/sync-visitor-data/src/types.ts b/packages/twenty-apps/webmetic/serverlessFunctions/sync-visitor-data/src/types.ts new file mode 100644 index 00000000000..5af7f79f5e2 --- /dev/null +++ b/packages/twenty-apps/webmetic/serverlessFunctions/sync-visitor-data/src/types.ts @@ -0,0 +1,51 @@ +export interface UserData { + timestamp: string; + document_location: string; + document_title: string; + scroll_depth: number; + referrer: string; + time_spent: number; + user_events_count?: number; +} + +export interface Session { + session_id: string; + timestamp: string; + utm_source?: string; + utm_medium?: string; + utm_campaign?: string; + utm_term?: string; + utm_content?: string; + visitor_city?: string; + visitor_country?: string; + session_duration: number; + session_time: number; + user_data: UserData[]; +} + +export interface WebmeticCompany { + company_id: string; + company_name: string; + company_url: string; + address?: string; + city?: string; + postal_code?: string; + country?: string; + phone_number?: string; + email_address?: string; + lead_score?: number; + employee_count?: string; + nace_code?: string; + linkedin?: string; + short_description_en?: string; + sessions: Session[]; +} + +export interface WebmeticResponse { + pagination: Array<{ + total: number; + page_total: number; + current: number; + }>; + result: WebmeticCompany[]; +} diff --git a/packages/twenty-apps/webmetic/tsconfig.json b/packages/twenty-apps/webmetic/tsconfig.json new file mode 100644 index 00000000000..17d258af134 --- /dev/null +++ b/packages/twenty-apps/webmetic/tsconfig.json @@ -0,0 +1,26 @@ +{ + "compileOnSave": false, + "compilerOptions": { + "sourceMap": true, + "declaration": true, + "outDir": "./dist", + "rootDir": ".", + "moduleResolution": "node", + "emitDecoratorMetadata": true, + "experimentalDecorators": true, + "importHelpers": true, + "strict": true, + "target": "es2018", + "module": "esnext", + "lib": ["es2020", "dom"], + "skipLibCheck": true, + "skipDefaultLibCheck": true, + "resolveJsonModule": true + }, + "exclude": [ + "node_modules", + "dist", + "**/*.test.ts", + "**/*.spec.ts" + ] +} diff --git a/packages/twenty-apps/webmetic/yarn.lock b/packages/twenty-apps/webmetic/yarn.lock new file mode 100644 index 00000000000..eb14171b7d4 --- /dev/null +++ b/packages/twenty-apps/webmetic/yarn.lock @@ -0,0 +1,273 @@ +# This file is generated by running "yarn install" inside your project. +# Manual changes might be lost - proceed with caution! + +__metadata: + version: 8 + cacheKey: 10c0 + +"@types/node@npm:^24.7.2": + version: 24.9.1 + resolution: "@types/node@npm:24.9.1" + dependencies: + undici-types: "npm:~7.16.0" + checksum: 10c0/c52f8168080ef9a7c3dc23d8ac6061fab5371aad89231a0f6f4c075869bc3de7e89b075b1f3e3171d9e5143d0dda1807c3dab8e32eac6d68f02e7480e7e78576 + languageName: node + linkType: hard + +"async-function@npm:^1.0.0": + version: 1.0.0 + resolution: "async-function@npm:1.0.0" + checksum: 10c0/669a32c2cb7e45091330c680e92eaeb791bc1d4132d827591e499cd1f776ff5a873e77e5f92d0ce795a8d60f10761dec9ddfe7225a5de680f5d357f67b1aac73 + languageName: node + linkType: hard + +"async-generator-function@npm:^1.0.0": + version: 1.0.0 + resolution: "async-generator-function@npm:1.0.0" + checksum: 10c0/2c50ef856c543ad500d8d8777d347e3c1ba623b93e99c9263ecc5f965c1b12d2a140e2ab6e43c3d0b85366110696f28114649411cbcd10b452a92a2318394186 + languageName: node + linkType: hard + +"asynckit@npm:^0.4.0": + version: 0.4.0 + resolution: "asynckit@npm:0.4.0" + checksum: 10c0/d73e2ddf20c4eb9337e1b3df1a0f6159481050a5de457c55b14ea2e5cb6d90bb69e004c9af54737a5ee0917fcf2c9e25de67777bbe58261847846066ba75bc9d + languageName: node + linkType: hard + +"axios@npm:^1.12.2": + version: 1.12.2 + resolution: "axios@npm:1.12.2" + dependencies: + follow-redirects: "npm:^1.15.6" + form-data: "npm:^4.0.4" + proxy-from-env: "npm:^1.1.0" + checksum: 10c0/80b063e318cf05cd33a4d991cea0162f3573481946f9129efb7766f38fde4c061c34f41a93a9f9521f02b7c9565ccbc197c099b0186543ac84a24580017adfed + languageName: node + linkType: hard + +"call-bind-apply-helpers@npm:^1.0.1, call-bind-apply-helpers@npm:^1.0.2": + version: 1.0.2 + resolution: "call-bind-apply-helpers@npm:1.0.2" + dependencies: + es-errors: "npm:^1.3.0" + function-bind: "npm:^1.1.2" + checksum: 10c0/47bd9901d57b857590431243fea704ff18078b16890a6b3e021e12d279bbf211d039155e27d7566b374d49ee1f8189344bac9833dec7a20cdec370506361c938 + languageName: node + linkType: hard + +"combined-stream@npm:^1.0.8": + version: 1.0.8 + resolution: "combined-stream@npm:1.0.8" + dependencies: + delayed-stream: "npm:~1.0.0" + checksum: 10c0/0dbb829577e1b1e839fa82b40c07ffaf7de8a09b935cadd355a73652ae70a88b4320db322f6634a4ad93424292fa80973ac6480986247f1734a1137debf271d5 + languageName: node + linkType: hard + +"delayed-stream@npm:~1.0.0": + version: 1.0.0 + resolution: "delayed-stream@npm:1.0.0" + checksum: 10c0/d758899da03392e6712f042bec80aa293bbe9e9ff1b2634baae6a360113e708b91326594c8a486d475c69d6259afb7efacdc3537bfcda1c6c648e390ce601b19 + languageName: node + linkType: hard + +"dunder-proto@npm:^1.0.1": + version: 1.0.1 + resolution: "dunder-proto@npm:1.0.1" + dependencies: + call-bind-apply-helpers: "npm:^1.0.1" + es-errors: "npm:^1.3.0" + gopd: "npm:^1.2.0" + checksum: 10c0/199f2a0c1c16593ca0a145dbf76a962f8033ce3129f01284d48c45ed4e14fea9bbacd7b3610b6cdc33486cef20385ac054948fefc6272fcce645c09468f93031 + languageName: node + linkType: hard + +"es-define-property@npm:^1.0.1": + version: 1.0.1 + resolution: "es-define-property@npm:1.0.1" + checksum: 10c0/3f54eb49c16c18707949ff25a1456728c883e81259f045003499efba399c08bad00deebf65cccde8c0e07908c1a225c9d472b7107e558f2a48e28d530e34527c + languageName: node + linkType: hard + +"es-errors@npm:^1.3.0": + version: 1.3.0 + resolution: "es-errors@npm:1.3.0" + checksum: 10c0/0a61325670072f98d8ae3b914edab3559b6caa980f08054a3b872052640d91da01d38df55df797fcc916389d77fc92b8d5906cf028f4db46d7e3003abecbca85 + languageName: node + linkType: hard + +"es-object-atoms@npm:^1.0.0, es-object-atoms@npm:^1.1.1": + version: 1.1.1 + resolution: "es-object-atoms@npm:1.1.1" + dependencies: + es-errors: "npm:^1.3.0" + checksum: 10c0/65364812ca4daf48eb76e2a3b7a89b3f6a2e62a1c420766ce9f692665a29d94fe41fe88b65f24106f449859549711e4b40d9fb8002d862dfd7eb1c512d10be0c + languageName: node + linkType: hard + +"es-set-tostringtag@npm:^2.1.0": + version: 2.1.0 + resolution: "es-set-tostringtag@npm:2.1.0" + dependencies: + es-errors: "npm:^1.3.0" + get-intrinsic: "npm:^1.2.6" + has-tostringtag: "npm:^1.0.2" + hasown: "npm:^2.0.2" + checksum: 10c0/ef2ca9ce49afe3931cb32e35da4dcb6d86ab02592cfc2ce3e49ced199d9d0bb5085fc7e73e06312213765f5efa47cc1df553a6a5154584b21448e9fb8355b1af + languageName: node + linkType: hard + +"follow-redirects@npm:^1.15.6": + version: 1.15.11 + resolution: "follow-redirects@npm:1.15.11" + peerDependenciesMeta: + debug: + optional: true + checksum: 10c0/d301f430542520a54058d4aeeb453233c564aaccac835d29d15e050beb33f339ad67d9bddbce01739c5dc46a6716dbe3d9d0d5134b1ca203effa11a7ef092343 + languageName: node + linkType: hard + +"form-data@npm:^4.0.4": + version: 4.0.4 + resolution: "form-data@npm:4.0.4" + dependencies: + asynckit: "npm:^0.4.0" + combined-stream: "npm:^1.0.8" + es-set-tostringtag: "npm:^2.1.0" + hasown: "npm:^2.0.2" + mime-types: "npm:^2.1.12" + checksum: 10c0/373525a9a034b9d57073e55eab79e501a714ffac02e7a9b01be1c820780652b16e4101819785e1e18f8d98f0aee866cc654d660a435c378e16a72f2e7cac9695 + languageName: node + linkType: hard + +"function-bind@npm:^1.1.2": + version: 1.1.2 + resolution: "function-bind@npm:1.1.2" + checksum: 10c0/d8680ee1e5fcd4c197e4ac33b2b4dce03c71f4d91717292785703db200f5c21f977c568d28061226f9b5900cbcd2c84463646134fd5337e7925e0942bc3f46d5 + languageName: node + linkType: hard + +"generator-function@npm:^2.0.0": + version: 2.0.1 + resolution: "generator-function@npm:2.0.1" + checksum: 10c0/8a9f59df0f01cfefafdb3b451b80555e5cf6d76487095db91ac461a0e682e4ff7a9dbce15f4ecec191e53586d59eece01949e05a4b4492879600bbbe8e28d6b8 + languageName: node + linkType: hard + +"get-intrinsic@npm:^1.2.6": + version: 1.3.1 + resolution: "get-intrinsic@npm:1.3.1" + dependencies: + async-function: "npm:^1.0.0" + async-generator-function: "npm:^1.0.0" + call-bind-apply-helpers: "npm:^1.0.2" + es-define-property: "npm:^1.0.1" + es-errors: "npm:^1.3.0" + es-object-atoms: "npm:^1.1.1" + function-bind: "npm:^1.1.2" + generator-function: "npm:^2.0.0" + get-proto: "npm:^1.0.1" + gopd: "npm:^1.2.0" + has-symbols: "npm:^1.1.0" + hasown: "npm:^2.0.2" + math-intrinsics: "npm:^1.1.0" + checksum: 10c0/9f4ab0cf7efe0fd2c8185f52e6f637e708f3a112610c88869f8f041bb9ecc2ce44bf285dfdbdc6f4f7c277a5b88d8e94a432374d97cca22f3de7fc63795deb5d + languageName: node + linkType: hard + +"get-proto@npm:^1.0.1": + version: 1.0.1 + resolution: "get-proto@npm:1.0.1" + dependencies: + dunder-proto: "npm:^1.0.1" + es-object-atoms: "npm:^1.0.0" + checksum: 10c0/9224acb44603c5526955e83510b9da41baf6ae73f7398875fba50edc5e944223a89c4a72b070fcd78beb5f7bdda58ecb6294adc28f7acfc0da05f76a2399643c + languageName: node + linkType: hard + +"gopd@npm:^1.2.0": + version: 1.2.0 + resolution: "gopd@npm:1.2.0" + checksum: 10c0/50fff1e04ba2b7737c097358534eacadad1e68d24cccee3272e04e007bed008e68d2614f3987788428fd192a5ae3889d08fb2331417e4fc4a9ab366b2043cead + languageName: node + linkType: hard + +"has-symbols@npm:^1.0.3, has-symbols@npm:^1.1.0": + version: 1.1.0 + resolution: "has-symbols@npm:1.1.0" + checksum: 10c0/dde0a734b17ae51e84b10986e651c664379018d10b91b6b0e9b293eddb32f0f069688c841fb40f19e9611546130153e0a2a48fd7f512891fb000ddfa36f5a20e + languageName: node + linkType: hard + +"has-tostringtag@npm:^1.0.2": + version: 1.0.2 + resolution: "has-tostringtag@npm:1.0.2" + dependencies: + has-symbols: "npm:^1.0.3" + checksum: 10c0/a8b166462192bafe3d9b6e420a1d581d93dd867adb61be223a17a8d6dad147aa77a8be32c961bb2f27b3ef893cae8d36f564ab651f5e9b7938ae86f74027c48c + languageName: node + linkType: hard + +"hasown@npm:^2.0.2": + version: 2.0.2 + resolution: "hasown@npm:2.0.2" + dependencies: + function-bind: "npm:^1.1.2" + checksum: 10c0/3769d434703b8ac66b209a4cca0737519925bbdb61dd887f93a16372b14694c63ff4e797686d87c90f08168e81082248b9b028bad60d4da9e0d1148766f56eb9 + languageName: node + linkType: hard + +"math-intrinsics@npm:^1.1.0": + version: 1.1.0 + resolution: "math-intrinsics@npm:1.1.0" + checksum: 10c0/7579ff94e899e2f76ab64491d76cf606274c874d8f2af4a442c016bd85688927fcfca157ba6bf74b08e9439dc010b248ce05b96cc7c126a354c3bae7fcb48b7f + languageName: node + linkType: hard + +"mime-db@npm:1.52.0": + version: 1.52.0 + resolution: "mime-db@npm:1.52.0" + checksum: 10c0/0557a01deebf45ac5f5777fe7740b2a5c309c6d62d40ceab4e23da9f821899ce7a900b7ac8157d4548ddbb7beffe9abc621250e6d182b0397ec7f10c7b91a5aa + languageName: node + linkType: hard + +"mime-types@npm:^2.1.12": + version: 2.1.35 + resolution: "mime-types@npm:2.1.35" + dependencies: + mime-db: "npm:1.52.0" + checksum: 10c0/82fb07ec56d8ff1fc999a84f2f217aa46cb6ed1033fefaabd5785b9a974ed225c90dc72fff460259e66b95b73648596dbcc50d51ed69cdf464af2d237d3149b2 + languageName: node + linkType: hard + +"proxy-from-env@npm:^1.1.0": + version: 1.1.0 + resolution: "proxy-from-env@npm:1.1.0" + checksum: 10c0/fe7dd8b1bdbbbea18d1459107729c3e4a2243ca870d26d34c2c1bcd3e4425b7bcc5112362df2d93cc7fb9746f6142b5e272fd1cc5c86ddf8580175186f6ad42b + languageName: node + linkType: hard + +"twenty-sdk@npm:0.0.3": + version: 0.0.3 + resolution: "twenty-sdk@npm:0.0.3" + checksum: 10c0/0a3c85c27edb22fb50f7eb0da4f9770e85729fce05e9e0118ad0cdfc36e42425c93340a6cd1c276daf30aeeaa612db0cd905831c0a8287a31bff3da5be9b0562 + languageName: node + linkType: hard + +"undici-types@npm:~7.16.0": + version: 7.16.0 + resolution: "undici-types@npm:7.16.0" + checksum: 10c0/3033e2f2b5c9f1504bdc5934646cb54e37ecaca0f9249c983f7b1fc2e87c6d18399ebb05dc7fd5419e02b2e915f734d872a65da2e3eeed1813951c427d33cc9a + languageName: node + linkType: hard + +"webmetic@workspace:.": + version: 0.0.0-use.local + resolution: "webmetic@workspace:." + dependencies: + "@types/node": "npm:^24.7.2" + axios: "npm:^1.12.2" + twenty-sdk: "npm:0.0.3" + languageName: unknown + linkType: soft