ToolJet/deploy/ec2/ee/setup_app
Akshay 18b831f783
Refactor: Workflows with BullMQ (#14275)
* refactor(workflows): migrate core modules from Temporal to BullMQ

Update main application modules to support BullMQ-based workflow scheduling:

- Remove Temporal worker bootstrap code from main.ts
- Migrate from @nestjs/bull to @nestjs/bullmq
- Add Bull Board dashboard at /jobs with basic auth
- Register BullMQ queues in WorkflowsModule
- Add IWorkflowScheduler interface for scheduler abstraction
- Create CE stubs for WorkflowSchedulerService and ScheduleBootstrapService
- Remove workflow dependencies from AppsModule (moved to WorkflowsModule)
- Add proper route exclusion for /jobs dashboard
- Support WORKER env var for conditional processor registration

This commit completes the migration from Temporal to BullMQ for workflow
scheduling, enabling simpler deployment and better horizontal scaling.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* chore: remove unused Temporal imports and commented code

Clean up migration artifacts:
- Remove unused imports from main.ts (TOOLJET_EDITIONS, getImportPath, ITemporalService, getTooljetEdition)
- Remove commented TemporalService references from WorkflowsModule
- Remove temporal.service from getProviders path array
- Add missing newlines at EOF for IWorkflowScheduler.ts and schedule-bootstrap.service.ts

This cleanup prepares the codebase for complete Temporal code removal in a future commit.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* build: add BullMQ and Bull Board dependencies

Add required packages for BullMQ-based workflow scheduling:

- @nestjs/bullmq: NestJS integration for BullMQ
- @bull-board/api, @bull-board/express, @bull-board/nestjs: Queue dashboard
- bullmq: Core BullMQ library
- express-basic-auth: Authentication for Bull Board dashboard

Note: @nestjs/bull is kept for backward compatibility during migration.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* chore: remove Temporal service files from CE

Remove deprecated Temporal-based workflow implementation files:
- server/src/modules/workflows/interfaces/ITemporalService.ts
- server/src/modules/workflows/services/temporal.service.ts

These files are replaced by IWorkflowScheduler interface and BullMQ-based
WorkflowSchedulerService implementation.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* feat: add comprehensive Redis configuration support for BullMQ

* refactor: remove Temporal setup and configuration from entrypoint scripts and Dockerfiles (#14294)

* refactor: remove Temporal setup and configuration from entrypoint scripts and Dockerfiles

* feat: integrate Redis support for BullMQ in preview environment

* remove worker execution logic from setup script

* Refactor: Centralise workflow execution through BullMQ (#14321)

* refactor: run all workflows through bullmq

* refactor: update imports

* chore: update subproject commit reference in server/ee

* feat: ablity to cancel workflow

* feat: implement workflow cancellation functionality with Redis support

* feat: add optional timeout parameter to requestCancellation method

* refactor: clean up formatting and add maintenance toggle event emission in AppsService

* feat: ability to cancel multiple inprogress executions

* feat: implement execution state management and display logic

* chore: update submodule commit reference

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Adish M <44204658+adishM98@users.noreply.github.com>
2025-11-05 17:54:38 +05:30

175 lines
4.8 KiB
Bash
Executable file

#!/bin/bash
# Load the .env file
source .env
# Check if LOCKBOX_MASTER_KEY is present or empty
if [[ -z "$LOCKBOX_MASTER_KEY" ]]; then
# Generate LOCKBOX_MASTER_KEY
LOCKBOX_MASTER_KEY=$(openssl rand -hex 32)
# Update .env file
awk -v key="$LOCKBOX_MASTER_KEY" '
BEGIN { FS=OFS="=" }
/^LOCKBOX_MASTER_KEY=/ { $2=key; found=1 }
1
END { if (!found) print "LOCKBOX_MASTER_KEY="key }
' .env > temp.env && mv temp.env .env
echo "Generated a secure master key for the lockbox"
else
echo "The lockbox master key already exists."
fi
# Check if SECRET_KEY_BASE is present or empty
if [[ -z "$SECRET_KEY_BASE" ]]; then
# Generate SECRET_KEY_BASE
SECRET_KEY_BASE=$(openssl rand -hex 64)
# Update .env file
awk -v key="$SECRET_KEY_BASE" '
BEGIN { FS=OFS="=" }
/^SECRET_KEY_BASE=/ { $2=key; found=1 }
1
END { if (!found) print "SECRET_KEY_BASE="key }
' .env > temp.env && mv temp.env .env
echo "Created a secret key for secure operations."
else
echo "The secret key base is already in place."
fi
# Check if PGRST_JWT_SECRET is present or empty
if [[ -z "$PGRST_JWT_SECRET" ]]; then
# Generate PGRST_JWT_SECRET
PGRST_JWT_SECRET=$(openssl rand -hex 32)
# Update .env file
awk -v key="$PGRST_JWT_SECRET" '
BEGIN { FS=OFS="=" }
/^PGRST_JWT_SECRET=/ { $2=key; found=1 }
1
END { if (!found) print "PGRST_JWT_SECRET="key }
' .env > temp.env && mv temp.env .env
echo "Generated a unique secret for PGRST authentication."
else
echo "The PGRST JWT secret is already generated and in place."
fi
# Function to generate a random password
generate_password() {
openssl rand -base64 12 | tr -d '/+' | cut -c1-16
}
# Check if PG_USER, PG_HOST, PG_PASS, PG_DB are present or empty
if [[ -z "$PG_USER" ]] || [[ -z "$PG_HOST" ]] || [[ -z "$PG_PASS" ]] || [[ -z "$PG_DB" ]]; then
# Prompt user for values
read -p "Enter PostgreSQL database username: " PG_USER
read -p "Enter PostgreSQL database hostname: " PG_HOST
read -p "Enter PostgreSQL database password: " PG_PASS
read -p "Enter PostgreSQL database name: " PG_DB
# Update .env file
awk -v pg_user="$PG_USER" -v pg_host="$PG_HOST" -v pg_pass="$PG_PASS" -v pg_db="$PG_DB" '
BEGIN { FS=OFS="=" }
/^PG_USER=/ { $2=pg_user; found=1 }
/^PG_HOST=/ { $2=pg_host; found=1 }
/^PG_PASS=/ { $2=pg_pass; found=1 }
/^PG_DB=/ { $2=pg_db; found=1 }
1
END {
if (!found) {
print "PG_USER="pg_user
print "PG_HOST="pg_host
print "PG_PASS="pg_pass
print "PG_DB="pg_db
}
}
' .env > temp.env && mv temp.env .env
echo "Successfully updated postgresql database values .env file"
fi
# Copy values from PG to TOOLJET_DB
TOOLJET_DB_USER=$PG_USER
TOOLJET_DB_HOST=$PG_HOST
TOOLJET_DB_PASS=$PG_PASS
# Update .env file for TOOLJET_DB
awk -v tj_user="$TOOLJET_DB_USER" -v tj_host="$TOOLJET_DB_HOST" -v tj_pass="$TOOLJET_DB_PASS" '
BEGIN { FS=OFS="=" }
/^TOOLJET_DB_USER=/ { $2=tj_user; found=1 }
/^TOOLJET_DB_HOST=/ { $2=tj_host; found=1 }
/^TOOLJET_DB_PASS=/ { $2=tj_pass; found=1 }
1
END { if (!found) print "TOOLJET_DB_USER="tj_user ORS "TOOLJET_DB_HOST="tj_host ORS "TOOLJET_DB_PASS="tj_pass }
' .env > temp.env && mv temp.env .env
echo "Successfully updated tooljet database values in the .env file"
# Construct PGRST_DB_URI with user-provided values
PGRST_DB_URI="postgres://$PG_USER:$PG_PASS@$PG_HOST/tooljet_db"
# Update .env file for PGRST_DB_URI
awk -v uri="$PGRST_DB_URI" '
BEGIN { FS=OFS="=" }
/^PGRST_DB_URI=/ { $2=uri; found=1 }
1
END { if (!found) print "PGRST_DB_URI="uri }
' .env > temp.env && mv temp.env .env
echo "Successfully updated PGRST database URI"
if [[ -z $PG_USER || -z $PG_PASS || -z $PG_HOST ]]
then
echo "Please set the required PG_USER, PG_PASS, and PG_HOST values within the .env file"
exit 1
fi
export $(grep -v '^#' .env | xargs)
if psql -d postgresql://$PG_USER:$PG_PASS@$PG_HOST/postgres -c 'select now()' > /dev/null 2>&1
then
echo "Successfully pinged the database!";
else
echo "Can't connect to the database. Kindly check the credenials provided in the .env file!"
exit 1
fi
if sudo systemctl start redis-server && sudo systemctl enable redis-server
then
echo "Successfully started Redis!"
else
echo "Failed to start and enable Redis"
fi
if sudo -E systemctl start openresty
then
echo "Successfully started reverse proxy!"
else
echo "Failed to start reverse proxy"
exit 1
fi
if sudo -E systemctl start postgrest
then
echo "Successfully started PostgREST server!"
else
echo "Failed to start PostgREST server"
exit 1
fi
TOOLJET_EDTION=ee npm --prefix server run db:setup
if sudo -E systemctl start nest && sudo journalctl -u nest -f
then
echo "The app will be served at ${TOOLJET_HOST}"
else
echo "Failed to start the server!"
exit 1
fi
sudo systemctl restart nest
sudo -E systemctl restart postgrest