ToolJet/docs/versioned_docs/version-3.16.0-LTS/setup/workflow-temporal-to-bullmq-migration.md

---
id: workflow-temporal-to-bullmq-migration
title: Workflow Migration - Temporal to BullMQ
---

# Migrating Workflows from Temporal to BullMQ

This guide helps you migrate your ToolJet workflow scheduling system from the legacy Temporal-based architecture to the new BullMQ-based system.

:::note
This migration guide is for ToolJet version **v3.20.37-LTS and later**. Versions prior to v3.20.37-LTS used a Temporal-based workflow system.
:::

## Overview

ToolJet has replaced Temporal with BullMQ for workflow scheduling, significantly simplifying deployment while maintaining all existing functionality. This change eliminates the need for separate Temporal server infrastructure and provides built-in monitoring capabilities.

## Why Migrate?

### Benefits of BullMQ-based Workflows

- **Simplified Architecture**: No need for separate Temporal server deployment
- **Existing Infrastructure**: Leverages your existing Redis instance
- **Better Resource Management**: Flexible worker modes for optimized scaling
- **Improved Visibility**: Real-time job status tracking and retry capabilities

### Architecture Comparison

| Feature | Temporal (Old) | BullMQ (New) |
|---------|---------------|--------------|
| External Services | Temporal Server + Redis | Redis only |
| Deployment Complexity | High (multi-service) | Low (single-service) |
| Infrastructure Cost | Higher | Lower |

## How It Works

The new BullMQ-based workflow system operates as follows:

1. **Workflow Scheduling**: When you schedule a workflow, it's stored in PostgreSQL and a corresponding job is created in Redis using BullMQ
2. **Job Queues**: Two BullMQ queues manage workflows:
   - `workflow-schedule-queue`: Handles scheduled workflow triggers
   - `workflow-execution-queue`: Manages workflow execution
3. **Worker Processing**: ToolJet instances with `WORKER=true` pick up jobs from these queues and execute them
4. **Schedule Recovery**: On startup, the Schedule Bootstrap Service automatically loads all active schedules from PostgreSQL and recreates them in Redis, ensuring no workflows are lost during deployments
5. **State Updates**: The frontend polls workflow execution states every 3 seconds via batch API calls

## Migration Steps

### 1. Review Current Setup

Check your current deployment for Temporal-related configurations:

**Environment Variables to Remove:**
```bash
# Old Temporal variables - REMOVE THESE
ENABLE_WORKFLOW_SCHEDULING=true
WORKFLOW_WORKER=true
TOOLJET_WORKFLOWS_TEMPORAL_NAMESPACE=default
TEMPORAL_SERVER_ADDRESS=temporal:7233
```

**Services to Remove:**
- Temporal server containers/pods
- Temporal worker containers/pods

### 2. Set Up Redis

:::warning
**External Redis Requirement**: When running separate worker containers or multiple instances, an external stateful Redis instance is **required** for job queue coordination. The built-in Redis only works when the server and worker are in the same container instance (single instance deployment).
:::

**Redis Requirements:**
- **Persistence**: AOF (Append Only File) must be enabled
- **Memory Policy**: `maxmemory-policy` must be set to `noeviction` (required by BullMQ)
- **Version**: Redis 6.x or higher (Redis 7.x recommended)

**Example Redis Configuration:**
```conf
# Persistence
appendonly yes
appendfilename "appendonly.aof"
appendfsync everysec

# Memory Management
maxmemory-policy noeviction

# RDB Snapshots
save 900 1
save 300 10
save 60 10000
```

**Redis Setup by Platform:**

<details id="tj-dropdown">

<summary>Kubernetes (EKS, AKS, GKE, OpenShift)</summary>

Deploy stateful Redis:

```bash
kubectl apply -f https://tooljet-deployments.s3.us-west-1.amazonaws.com/kubernetes/redis-stateful.yaml
```

This creates:
- StatefulSet with persistent storage
- Headless Service
- ConfigMap with production-ready configuration
- Secret for password authentication

Configure environment variables:

```bash
REDIS_HOST=redis-service.default.svc.cluster.local
REDIS_PORT=6379
REDIS_PASSWORD=your-secure-password
```

</details>

<details id="tj-dropdown">

<summary>AWS ECS</summary>

Use Amazon ElastiCache for Redis:
1. Create Redis cluster with:
   - Engine version: Redis 7.x
   - Node type: cache.t3.medium or higher
   - Automatic failover enabled

2. Configure parameter group:
   - Set **maxmemory-policy** to **noeviction**
   - Set **appendonly** to **yes**

3. Add environment variables:

```bash
REDIS_HOST=your-elasticache-endpoint.cache.amazonaws.com
REDIS_PORT=6379
REDIS_PASSWORD=your-redis-password
```

</details>

<details id="tj-dropdown">

<summary>Azure Container Apps</summary>

Use Azure Cache for Redis:
1. Create Redis instance (Standard or Premium tier)
2. Configure Redis settings for AOF and noeviction policy
3. Add environment variables:

```bash
REDIS_HOST=your-redis.redis.cache.windows.net
REDIS_PORT=6379
REDIS_PASSWORD=your-redis-password
REDIS_TLS=true
```

</details>

<details id="tj-dropdown">

<summary>Google Cloud Run</summary>

Use Google Cloud Memorystore for Redis:
1. Create Redis instance with:
   - Redis version 7.x
   - High availability enabled
2. Configure Redis settings via **gcloud** CLI
3. Add environment variables:

```bash
REDIS_HOST=your-memorystore-ip
REDIS_PORT=6379
REDIS_PASSWORD=your-redis-password
```

</details>

### 3. Update Environment Variables

Add the new BullMQ workflow environment variables:

**Required Variables:**
```bash
# Worker Mode (required)
# Set to 'true' to enable job processing
WORKER=true

# Redis connection settings
REDIS_HOST=localhost         # Default: localhost
REDIS_PORT=6379              # Default: 6379
REDIS_USERNAME=              # Optional: Redis username (ACL)
REDIS_PASSWORD=              # Optional: Redis password
REDIS_DB=0                   # Optional: Redis database number (default: 0)
REDIS_TLS=false              # Optional: Enable TLS/SSL (set to 'true')
```

**Note:** Only `REDIS_HOST` and `REDIS_PORT` are required. Authentication and TLS are optional.

**Optional Variables:**
```bash
# Workflow Processor Concurrency (optional)
# Number of workflow jobs processed concurrently per worker
# Default: 5
TOOLJET_WORKFLOW_CONCURRENCY=5

# Workflow Timeout (optional)
# Maximum execution time for a workflow in seconds
# Default: 60
WORKFLOW_TIMEOUT_SECONDS=60

# Redis Configuration (optional)
REDIS_USERNAME=          # Redis username (ACL)
REDIS_DB=0              # Redis database number (default: 0)
REDIS_TLS=false         # Enable TLS/SSL (set to 'true')
```

### 4. Deploy Updated Configuration

Update your ToolJet deployment with the new configuration:

**For Kubernetes:**
```bash
# Update your deployment.yaml with new environment variables
kubectl apply -f deployment.yaml

# Restart pods to apply changes
kubectl rollout restart deployment/tooljet
```

**For Docker/Docker Compose:**
```bash
# Update your .env file or docker-compose.yml
docker-compose down
docker-compose up -d
```

**For AWS ECS:**
- Update task definition with new environment variables
- Create new revision and update service

**For Azure Container Apps:**
- Update environment variables in container app settings
- Save and restart

### 5. Remove Temporal Infrastructure

After confirming the new setup works:

**For Kubernetes:**
```bash
# Remove Temporal deployments
kubectl delete deployment temporal-server
kubectl delete deployment temporal-worker
kubectl delete service temporal-service
```

**For Docker Compose:**
```yaml
# Remove Temporal services from docker-compose.yml
# - temporal-server
# - temporal-worker
```

**For ECS:**
- Stop and delete Temporal task definitions
- Remove Temporal services

### 6. Verify Migration

1. **Check Workflow Scheduling**: Create a new scheduled workflow in ToolJet
2. **Verify Execution**: Trigger a workflow and confirm it executes successfully
3. **Check Logs**: Review application logs for any errors

## Scaling Workflows with Dedicated Workers

For production deployments with extensive workflow usage, it's recommended to deploy dedicated worker instances that only process jobs without serving HTTP traffic.

### Why Dedicated Workers?

- **Better Resource Allocation**: Separate compute resources for API and job processing
- **Independent Scaling**: Scale workers based on job queue depth
- **Improved Reliability**: HTTP server issues don't affect job processing
- **Cost Optimization**: Use different instance sizes for API vs workers

### Architecture

```
            ┌─────────────────────────────────────────────────────────────┐
            │                    ToolJet Deployment                       │
            ├─────────────────────────────────────────────────────────────┤
            │                                                             │
            │  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐       │
            │  │ Web Server   │  │  Worker 1    │  │  Worker 2    │       │
            │  │              │  │              │  │              │       │
            │  │ WORKER=true  │  │ WORKER=true  │  │ WORKER=true  │       │
            │  │              │  │              │  │              │       │
            │  │ HTTP Requests│  │ Process Jobs │  │ Process Jobs │       │
            │  └──────┬───────┘  └──────┬───────┘  └──────┬───────┘       │
            │         │                 │                 │               │
            │         └─────────────────┼─────────────────┘               │
            │                           │                                 │
            └───────────────────────────┼─────────────────────────────────┘
                                        │
                                        ▼
                               ┌─────────────────┐
                               │  External Redis │
                               │   (Stateful)    │
                               │                 │
                               │  - Job Queue    │
                               │  - Persistence  │
                               └─────────────────┘
```

### Deployment Configuration

#### Kubernetes Example

<details id="tj-dropdown">

<summary>ToolJet App Deployment (tooljet-deployment.yaml)</summary>
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tooljet-deployment
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  selector:
    matchLabels:
      component: tooljet
  template:
    metadata:
      labels:
        component: tooljet
    spec:
      imagePullSecrets:
        - name: docker-secret
      containers:
        - name: tooljet
          image: tooljet/tooljet:ee-lts-latest
          imagePullPolicy: Always
          args: ["npm", "run", "start:prod"]
          resources:
            limits:
              memory: "2000Mi"
              cpu: "2000m"
            requests:
              memory: "1000Mi"
              cpu: "1000m"
          ports:
            - containerPort: 3000
          readinessProbe:
            httpGet:
              port: 3000
              path: /api/health
            successThreshold: 1
            initialDelaySeconds: 10
            periodSeconds: 5
            failureThreshold: 6
          env:
            - name: PG_HOST
              valueFrom:
                secretKeyRef:
                  name: server
                  key: pg_host
            - name: PG_USER
              valueFrom:
                secretKeyRef:
                  name: server
                  key: pg_user
            - name: PG_PASS
              valueFrom:
                secretKeyRef:
                  name: server
                  key: pg_password
            - name: PG_DB
              valueFrom:
                secretKeyRef:
                  name: server
                  key: pg_db
            - name: LOCKBOX_MASTER_KEY
              valueFrom:
                secretKeyRef:
                  name: server
                  key: lockbox_key
            - name: SECRET_KEY_BASE
              valueFrom:
                secretKeyRef:
                  name: server
                  key: secret_key_base
            - name: TOOLJET_HOST
              valueFrom:
                secretKeyRef:
                  name: server
                  key: tj_host
            - name: REDIS_HOST
              value: redis-service.default.svc.cluster.local
            - name: REDIS_PORT
              value: "6379"
            - name: TOOLJET_DB
              value: "tooljet_db"
            - name: TOOLJET_DB_USER
              value: "replace_with_postgres_database_user"
            - name: TOOLJET_DB_HOST
              value: "replace_with_postgres_database_host"
            - name: TOOLJET_DB_PASS
              value: "replace_with_postgres_database_password"
            - name: PGRST_HOST
              value: localhost:3002
            - name: PGRST_SERVER_PORT
              value: "3002"
            - name: PGRST_JWT_SECRET
              value: "replace_jwt_secret_here"
            - name: PGRST_DB_PRE_CONFIG
              value: postgrest.pre_config
            - name: PGRST_DB_URI
              value: postgres://TOOLJET_DB_USER:TOOLJET_DB_PASS@TOOLJET_DB_HOST:port/tooljet_db
            - name: PGRST_LOG_LEVEL
              value: "info"
            - name: DEPLOYMENT_PLATFORM
              value: "k8s"
---
apiVersion: v1
kind: Service
metadata:
  name: tooljet-service
spec:
  type: LoadBalancer
  ports:
  - port: 80
    targetPort: 3000
  selector:
    component: tooljet
```

</details>

<details id="tj-dropdown">

<summary>Worker Deployment (tooljet-worker.yaml)</summary>
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tooljet-worker
spec:
  replicas: 2  # Scale based on job queue depth
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  selector:
    matchLabels:
      component: tooljet-worker
  template:
    metadata:
      labels:
        component: tooljet-worker
    spec:
      imagePullSecrets:
        - name: docker-secret
      containers:
        - name: tooljet-worker
          image: tooljet/tooljet:ee-lts-latest
          imagePullPolicy: Always
          args: ["npm", "run", "start:prod"]
          resources:
            limits:
              memory: "2000Mi"
              cpu: "2000m"
            requests:
              memory: "1000Mi"
              cpu: "1000m"
          # No ports - workers don't serve HTTP
          env:
            # Worker-specific environment variables
            - name: WORKER
              value: "true"
            - name: TOOLJET_WORKFLOW_CONCURRENCY
              value: "10"
            - name: REDIS_HOST
              value: redis-service.default.svc.cluster.local
            - name: REDIS_PORT
              value: "6379"
            # All other environment variables same as ToolJet app
            - name: PG_HOST
              valueFrom:
                secretKeyRef:
                  name: server
                  key: pg_host
            - name: PG_USER
              valueFrom:
                secretKeyRef:
                  name: server
                  key: pg_user
            - name: PG_PASS
              valueFrom:
                secretKeyRef:
                  name: server
                  key: pg_password
            - name: PG_DB
              valueFrom:
                secretKeyRef:
                  name: server
                  key: pg_db
            - name: LOCKBOX_MASTER_KEY
              valueFrom:
                secretKeyRef:
                  name: server
                  key: lockbox_key
            - name: SECRET_KEY_BASE
              valueFrom:
                secretKeyRef:
                  name: server
                  key: secret_key_base
            - name: TOOLJET_HOST
              valueFrom:
                secretKeyRef:
                  name: server
                  key: tj_host
            - name: TOOLJET_DB
              value: "tooljet_db"
            - name: TOOLJET_DB_USER
              value: "replace_with_postgres_database_user"
            - name: TOOLJET_DB_HOST
              value: "replace_with_postgres_database_host"
            - name: TOOLJET_DB_PASS
              value: "replace_with_postgres_database_password"
            - name: PGRST_HOST
              value: localhost:3002
            - name: PGRST_SERVER_PORT
              value: "3002"
            - name: PGRST_JWT_SECRET
              value: "replace_jwt_secret_here"
            - name: PGRST_DB_PRE_CONFIG
              value: postgrest.pre_config
            - name: PGRST_DB_URI
              value: postgres://TOOLJET_DB_USER:TOOLJET_DB_PASS@TOOLJET_DB_HOST:port/tooljet_db
            - name: PGRST_LOG_LEVEL
              value: "info"
            - name: DEPLOYMENT_PLATFORM
              value: "k8s"
```

</details>

**Key Points:**
- **ToolJet App**: Serves HTTP traffic on port 3000, `WORKER` is unset (defaults to false)
- **Worker**: Only processes jobs with `WORKER=true`, no ports exposed
- Both deployments use the same secrets and database configuration
- Worker has additional workflow-specific env var: `TOOLJET_WORKFLOW_CONCURRENCY`
- Update `REDIS_HOST` to point to your deployed Redis service

#### Docker Compose Example

<details id="tj-dropdown">

<summary>Docker Compose Configuration</summary>

```yaml
version: '3.8'

services:
  tooljet:
    tty: true
    stdin_open: true
    container_name: Tooljet-app
    image: tooljet/tooljet:ee-lts-latest
    platform: linux/amd64
    restart: always
    env_file: .env
    ports:
      - 80:80
    depends_on:
      - postgres
      - redis
    environment:
      SERVE_CLIENT: "true"
      PORT: "80"
    command: npm run start:prod

  tooljet-worker-1:
    tty: true
    stdin_open: true
    platform: linux/amd64
    container_name: tooljet-worker-1
    image: tooljet/tooljet:ee-lts-latest
    restart: always
    env_file: .env
    depends_on:
      - postgres
      - redis
    environment:
      WORKER: "true"
      TOOLJET_WORKFLOW_CONCURRENCY: 10
      REDIS_HOST: redis
      REDIS_PORT: 6379
    command: npm run start:prod

  redis:
    image: redis:7
    volumes:
      - redis-data:/data
    command: redis-server --appendonly yes --maxmemory-policy noeviction

volumes:
  redis-data:
```

</details>

**Key Points:**
- `tooljet` service: Web server with `WORKER` unset (defaults to false), serves HTTP on port 80
- `tooljet-worker-1` service: Dedicated worker with `WORKER=true`, no ports exposed
- Both services use the same `.env` file for shared configuration
- `env_file: .env` loads common environment variables (database credentials, secrets, etc.)
- Environment-specific variables are set directly in the `environment` section
- Redis configured with AOF persistence and `noeviction` policy

#### AWS ECS Example

<details id="tj-dropdown">

<summary>ECS Task Definitions</summary>

**Web Service Task Definition:**
```json
{
  "family": "tooljet-web",
  "containerDefinitions": [
    {
      "name": "tooljet",
      "image": "tooljet/tooljet:ee-lts-latest",
      "portMappings": [
        {
          "containerPort": 3000,
          "protocol": "tcp"
        }
      ],
      "environment": [
        {"name": "WORKER", "value": "false"},
        {"name": "REDIS_HOST", "value": "your-elasticache-endpoint"}
      ]
    }
  ]
}
```

**Worker Service Task Definition:**
```json
{
  "family": "tooljet-worker",
  "containerDefinitions": [
    {
      "name": "tooljet-worker",
      "image": "tooljet/tooljet:ee-lts-latest",
      "portMappings": [],  // No ports needed
      "environment": [
        {"name": "WORKER", "value": "true"},
        {"name": "TOOLJET_WORKFLOW_CONCURRENCY", "value": "10"},
        {"name": "REDIS_HOST", "value": "your-elasticache-endpoint"}
      ]
    }
  ]
}
```

</details>

### Worker Scaling Considerations

**When to scale workers:**
- Queue depth consistently > 100 jobs
- Job processing latency increases
- Workflows timing out

**Scaling strategies:**
- **Horizontal**: Add more worker replicas
- **Vertical**: Increase `TOOLJET_WORKFLOW_CONCURRENCY`
- **Hybrid**: Combine both approaches

**Monitoring metrics:**
- Job completion time
- Failed job count
- Redis memory usage
- Application logs

## Monitoring and Troubleshooting

### Common Issues

#### Workflows Not Executing

**Symptoms:** Workflows scheduled but not running

**Solutions:**
1. Check `WORKER=true` is set in at least one instance
2. Verify Redis connection:
   ```bash
   # From ToolJet container
   redis-cli -h $REDIS_HOST -p $REDIS_PORT ping
   ```
3. Check worker logs for errors
4. Verify `maxmemory-policy noeviction` in Redis

#### Jobs Failing Repeatedly

**Symptoms:** Workflow jobs fail repeatedly

**Solutions:**
1. Check application logs for error messages
2. Verify workflow node configurations
3. Check Redis memory usage (may be full)
4. Review `WORKFLOW_TIMEOUT_SECONDS` setting

#### Schedules Lost After Restart

**Symptoms:** Scheduled workflows don't trigger after restart

**Solutions:**
1. Check Schedule Bootstrap Service logs
2. Verify Redis persistence (AOF) is working
3. Confirm PostgreSQL connection is stable
4. Check Redis has sufficient memory

## FAQ

<details id="tj-dropdown">

<summary>Do I need to recreate my workflows?</summary>

No. All existing workflow definitions and schedules are stored in PostgreSQL and will continue to work with the new BullMQ system. The Schedule Bootstrap Service automatically loads them on startup.

</details>

<details id="tj-dropdown">

<summary>Can I use the built-in Redis for workflows?</summary>

Yes, but only for **single instance deployments** where the server and worker are in the same container. When running separate worker containers or multiple instances, an external Redis instance with proper persistence (AOF) and **maxmemory-policy noeviction** is **required** for job queue coordination.

</details>

<details id="tj-dropdown">

<summary>What happens to in-flight workflows during migration?</summary>

In-flight workflows in the old Temporal system will not be migrated. Complete or cancel them before migration. New schedules will trigger normally in the BullMQ system.

</details>

<details id="tj-dropdown">

<summary>Can I run both Temporal and BullMQ simultaneously?</summary>

No. ToolJet only supports one workflow engine at a time. Choose either Temporal (legacy) or BullMQ (recommended).

</details>

<details id="tj-dropdown">

<summary>How do I monitor workflow performance?</summary>

Monitor workflow performance using:
- Application logs for job execution details
- Redis metrics for queue depth and processing rate
- Workflow execution history in the ToolJet UI
- Database queries for job success/failure rates

</details>

<details id="tj-dropdown">

<summary>What Redis version is required?</summary>

Redis 6.x or higher is required. Redis 7.x is recommended for best performance and features.

</details>

## Support

If you encounter issues during migration:

- **Community**: Join our [Slack Community](https://join.slack.com/t/tooljet/shared_invite/zt-2rk4w42t0-ZV_KJcWU9VL1BBEjnSHLCA)
- **Email**: hello@tooljet.com
- **GitHub Issues**: [Report bugs](https://github.com/ToolJet/ToolJet/issues)