import * as k8s from '@pulumi/kubernetes'; import * as kx from '@pulumi/kubernetesx'; import * as pulumi from '@pulumi/pulumi'; import { isDefined } from './helpers'; import { normalizeEnv, PodBuilder } from './pod-builder'; import { ServiceSecret } from './secrets'; type ProbeConfig = Omit< k8s.types.input.core.v1.Probe, 'httpGet' | 'exec' | 'grpc' | 'tcpSocket' > & { endpoint: string }; function normalizeEnvSecrets(envSecrets?: Record>) { return envSecrets ? Object.keys(envSecrets).map(name => ({ name, valueFrom: { secretKeyRef: { name: envSecrets[name].secret.record.metadata.name, key: envSecrets[name].key, }, }, })) : []; } export type ServiceSecretBinding> = { secret: ServiceSecret; key: keyof T | pulumi.Output; }; export class ServiceDeployment { private envSecrets: Record> = {}; constructor( protected name: string, protected options: { imagePullSecret?: k8s.core.v1.Secret; env?: kx.types.Container['env']; args?: kx.types.Container['args']; image: string; port?: number; /** Port to use for liveness, startup and readiness probes. */ probePort?: number; serviceAccountName?: pulumi.Output; livenessProbe?: string | ProbeConfig; readinessProbe?: string | ProbeConfig; startupProbe?: string | ProbeConfig; memory?: { limit?: string; requests?: string; }; cpu?: { limit?: string; requests?: string; }; volumes?: k8s.types.input.core.v1.Volume[]; volumeMounts?: k8s.types.input.core.v1.VolumeMount[]; /** * Enables /metrics endpoint on port 10254 */ exposesMetrics?: boolean; replicas?: number; pdb?: boolean; autoScaling?: { minReplicas?: number; maxReplicas: number; cpuAverageToScale: number; }; availabilityOnEveryNode?: boolean; command?: pulumi.Input[]>; }, protected dependencies?: Array, protected parent?: pulumi.Resource | null, ) {} withSecret>>( envVar: string, secret: ServiceSecret, key: keyof T, ) { this.envSecrets[envVar] = { secret, key }; return this; } withConditionalSecret>>( enabled: boolean, envVar: string, secret: ServiceSecret | null, key: keyof T, ) { if (enabled && secret) { this.envSecrets[envVar] = { secret, key }; } return this; } deployAsJob() { const { pb } = this.createPod(true); const job = new kx.Job( this.name, { spec: pb.asJobSpec(), }, { dependsOn: this.dependencies?.filter(isDefined) }, ); return { job }; } createPod(asJob: boolean) { const port = this.options.port || 3000; const probePort = this.options.probePort ?? port; const additionalEnv: any[] = normalizeEnv(this.options.env); const secretsEnv: any[] = normalizeEnvSecrets(this.envSecrets); let startupProbe: k8s.types.input.core.v1.Probe | undefined = undefined; let livenessProbe: k8s.types.input.core.v1.Probe | undefined = undefined; let readinessProbe: k8s.types.input.core.v1.Probe | undefined = undefined; if (this.options.livenessProbe) { livenessProbe = typeof this.options.livenessProbe === 'string' ? { initialDelaySeconds: 10, terminationGracePeriodSeconds: 60, periodSeconds: 10, failureThreshold: 5, timeoutSeconds: 5, httpGet: { path: this.options.livenessProbe, port: probePort, }, } : { ...this.options.livenessProbe, httpGet: { path: this.options.livenessProbe.endpoint, port: probePort, }, }; } if (this.options.readinessProbe) { readinessProbe = typeof this.options.readinessProbe === 'string' ? { initialDelaySeconds: 10, periodSeconds: 15, failureThreshold: 5, timeoutSeconds: 5, httpGet: { path: this.options.readinessProbe, port: probePort, }, } : { ...this.options.readinessProbe, httpGet: { path: this.options.readinessProbe.endpoint, port: probePort, }, }; } if (this.options.startupProbe) { startupProbe = typeof this.options.startupProbe === 'string' ? { initialDelaySeconds: 20, periodSeconds: 30, failureThreshold: 10, timeoutSeconds: 10, httpGet: { path: this.options.startupProbe, port: probePort, }, } : { ...this.options.startupProbe, httpGet: { path: this.options.startupProbe.endpoint, port: probePort, }, }; } if (this.options.exposesMetrics) { additionalEnv.push({ name: 'PROMETHEUS_METRICS', value: '1' }); } const topologySpreadConstraints: k8s.types.input.core.v1.TopologySpreadConstraint[] = []; if (this.options.availabilityOnEveryNode) { // This will ensure that services that has >1 replicas will be scheduled on every available node // and ensure that we are not exposed to downtime issues caused by node failures/restarts: topologySpreadConstraints.push({ maxSkew: 1, topologyKey: 'kubernetes.io/hostname', whenUnsatisfiable: 'ScheduleAnyway', labelSelector: { matchLabels: { app: this.name, }, }, }); } const resourcesLimits: { cpu?: string; memory?: string } = {}; const resourcesRequests: { cpu?: string; memory?: string } = {}; if (this.options?.cpu?.limit) { resourcesLimits.cpu = this.options?.cpu?.limit; } if (this.options?.cpu?.requests) { resourcesRequests.cpu = this.options?.cpu?.requests; } if (this.options?.memory?.limit) { resourcesLimits.memory = this.options?.memory?.limit; } if (this.options?.memory?.requests) { resourcesRequests.memory = this.options?.memory?.requests; } const pb = new PodBuilder({ restartPolicy: asJob ? 'Never' : 'Always', imagePullSecrets: this.options.imagePullSecret ? [{ name: this.options.imagePullSecret.metadata.name }] : undefined, terminationGracePeriodSeconds: 60, volumes: this.options.volumes, topologySpreadConstraints, serviceAccountName: this.options.serviceAccountName, containers: [ { livenessProbe, readinessProbe, startupProbe, volumeMounts: this.options.volumeMounts, imagePullPolicy: 'Always', env: [ { name: 'PORT', value: String(port) }, { name: 'POD_NAME', valueFrom: { fieldRef: { fieldPath: 'metadata.name', }, }, }, ] .concat(additionalEnv) .concat(secretsEnv), name: this.name, image: this.options.image, resources: { limits: resourcesLimits, requests: resourcesRequests, }, args: this.options.args, ports: { http: port, ...(this.options.exposesMetrics ? { metrics: 10_254, } : {}), }, command: this.options.command, }, ], }); return { pb }; } deploy() { const { pb } = this.createPod(false); const metadata: k8s.types.input.meta.v1.ObjectMeta = { annotations: {}, }; metadata.labels = { app: this.name, }; if (this.options.exposesMetrics) { metadata.annotations = { 'prometheus.io/port': '10254', 'prometheus.io/path': '/metrics', 'prometheus.io/scrape': 'true', }; } const deployment = new kx.Deployment( this.name, { spec: pb.asExtendedDeploymentSpec( { replicas: this.options.replicas ?? 1, strategy: { type: 'RollingUpdate', rollingUpdate: { maxSurge: this.options.replicas ?? 1, maxUnavailable: 0, }, }, }, { annotations: metadata.annotations, labels: metadata.labels, }, ), }, { dependsOn: this.dependencies?.filter(isDefined), parent: this.parent ?? undefined, }, ); if (this.options.pdb) { new k8s.policy.v1.PodDisruptionBudget(`${this.name}-pdb`, { spec: { minAvailable: 1, selector: deployment.spec.selector, }, }); } const service = createService(this.name, deployment); if (this.options.autoScaling) { new k8s.autoscaling.v2.HorizontalPodAutoscaler( `${this.name}-autoscaler`, { apiVersion: 'autoscaling/v2', kind: 'HorizontalPodAutoscaler', metadata: {}, spec: { scaleTargetRef: { name: deployment.metadata.name, kind: deployment.kind, apiVersion: deployment.apiVersion, }, metrics: [ { type: 'Resource', resource: { name: 'cpu', target: { type: 'Utilization', averageUtilization: this.options.autoScaling.cpuAverageToScale, }, }, }, ], minReplicas: this.options.autoScaling.minReplicas || this.options.replicas || 1, maxReplicas: this.options.autoScaling.maxReplicas, }, }, { dependsOn: [deployment, service], }, ); } return { deployment, service }; } } export function createService(name: string, deployment: kx.Deployment) { const labels = deployment.spec.selector.matchLabels; const ports = deployment.spec.template.spec.containers.apply(containers => { const ports: { name: string; port: number }[] = []; containers.forEach(container => { if (container.ports) { container.ports.forEach(port => { ports.push({ name: port.name || name, port: port.containerPort }); }); } }); return ports; }); return new k8s.core.v1.Service( name, { spec: { ports: ports, selector: labels, type: 'ClusterIP', }, }, {}, ); }