import NextImage from 'next/image'
import { Callout } from '@theguild/components'
import diagram from '../../../../public/docs/pages/self-hosting/diagram.png'

# Self-Hosting Hive

If you are not able to use Hive Cloud service, you can self-host it. The self-hosted version is free
and open-source. You can find the full
[source code on GitHub](https://github.com/graphql-hive/platform).

<Callout>
  **Self-hosting Hive** does currently not support all of the features as the Cloud version.

The following features are currently unavailable when self-hosting:

- High-Availability CDN (a replacement is available)
- Billing (payment integration)
- Usage rate-limiting and usage-estimation

</Callout>

## Pre-Requisites

The easiest way of running GraphQL Hive is using [Docker](https://www.docker.com/) and
[docker-compose](https://docs.docker.com/compose/).

All the services required for running GraphQL Hive are published to the GitHub container registry
([GraphQL Hive Docker Images](https://github.com/graphql-hive?tab=packages)).

Please make sure to install the Docker daemon as well as `docker-compose` on your machine before
proceeding.

## Self-Hosting Architecture

When self-hosting GraphQL Hive, you are responsible for running and maintaining the following
components:

- [PostgreSQL database](https://www.postgresql.org/)
- [Kafka](https://kafka.apache.org/) (or any other Kafka-compatible solution, like
  [Azure Event Hubs](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-about))
- [ClickHouse database](https://clickhouse.com/)
- [Redis](https://redis.io/)
- [SuperTokens instance](https://supertokens.com/)
- [S3 storage](https://aws.amazon.com/s3/) (or any other S3-compatible solution, like
  [minio](https://min.io/) or [CloudFlare R2](https://www.cloudflare.com/developer-platform/r2/))
- Hive microservices

<Callout type="info">
The defualt configuration of the self-hosted environment comes with a single instance of each database.

For production environments, we recommend replacing the in-cluster databases (PostgreSQL, Kafka and
ClickHouse) and storage (S3) with managed database/storage (Cloud-based or in-house).

[Avoid running databases in a containerized environment](https://patrobinson.github.io/2016/11/07/thou-shalt-not-run-a-database-inside-a-container/).

</Callout>

### Architecture Diagram

The following diagram shows the architecture of a self-hosted GraphQL Hive instance:

<NextImage
  alt="Self-hosting diagram"
  src={diagram}
  className="mt-10 w-full rounded-lg drop-shadow-md"
/>

### Services Overview

> The dependencies between the different services are described in the
> [`docker-compose.community.yml`](https://github.com/graphql-hive/platform/blob/main/docker/docker-compose.community.yml)
> file.

#### Databases / Storage

- **PostgreSQL database**: stores all core data, including: users, GraphQL schemas, schema checks,
  projects, targets, organizations and configurations.
- **Redis**: deployed as in-cluster, used for mainly for short-term caching.
- **ClickHouse database**: stores all GraphQL operations usage data.
- **Kafka**: high-throughput message queue, used for sending usage data to the usage-ingestor
  service.
- **S3 storage**: used for storing GraphQL schema artifacts (e.g. GraphQL SDL / Supergraph SDL and
  CDN access tokens).

#### Microservices

- **SuperTokens**: an open-source project used for authentication and authorization.
- `webapp`: the main GraphQL Hive web application.
- `server`: the main GraphQL Hive server, responsible for serving the GraphQL API and orchestrating
  calls to other services.
- **CDN**: a custom CDN service, responsible for serving the GraphQL schema artifacts. In
  self-hosted architecture, the CDN is provided as part of the `server`.
- `schema`: the GraphQL Hive schema service, responsible for validating, composing and building
  GraphQL schemas.
- `tokens`: the GraphQL Hive tokens service, responsible for generating and validating tokens.
- `usage`: the GraphQL Hive usage service, responsible for receiving usage data from a running
  GraphQL server, and prepare them for processing.
- `usage-ingestor`: the GraphQL Hive usage-ingestor service, responsible for processing usage data
  and storing them in ClickHouse.
- `emails`: the GraphQL Hive emails service, responsible for sending emails.
- `webhooks`: the GraphQL Hive webhooks service, responsible for sending webhooks to external
  services.
- `policy`: the GraphQL Hive policy service, responsible for validating and enforcing schema
  policies.

#### Utility services

In addition to the core services and databases, the default self-hosted architecture comes with the
following utility services:

- `zookeeper`: a centralized service and is used to maintain naming and configuration data for
  Kafka.
- `s3_provision_buckets`: a utility container that is used to provision the required S3 buckets.
- `s3_reverse_proxy`: a utility container that is used to proxy requests to the S3 storage.
- `migrations`: a utility container that is used to run database migrations for Hive's application
  (both PostgreSQL and Clickhouse).

## Quick Start Video

In this video you will learn how to run GraphQL Hive and publish your first schema on your machine
in less than 15 minutes.

<iframe
  src="https://www.youtube.com/embed/H57aej4cW2I"
  title="GraphQL Hive Self-Hosted Quick Start"
/>

## Running GraphQL Hive

First download the `docker-compose.community.yml` file from the
[GitHub repository](https://github.com/graphql-hive/platform/blob/main/docker/docker-compose.community.yml)
using `wget`. This's going to download the latest version available.

<Callout type="info">
  Note: make sure you've Docker installed to be able to continue with the setup process.
</Callout>

> You can also download the file directly from GitHub, if you don't have wget installed using the
> link below
>
> ```sh
> bash wget https://raw.githubusercontent.com/graphql-hive/platform/main/docker/docker-compose.community.yml
> ```

After downloading the `docker-compose.community.yml` file, this includes all the services required
for running Hive locally in a self hosted environment. But before we can spin it up, we first need
to set some environment variables.

<Callout>
  **Docker Tag.** Docker images are built and published for each single commit within the GraphQL
  Hive repository and tagged with the corresponding commit sha. You can also find all available
  images on the [Hive Docker Registry
  Overview](https://github.com/graphql-hive?tab=packages&repo_name=platform). We recommend using one
  of the latest versions. Right now we do not have a stable release strategy yet. However, every
  commit on the `main` branch can be considered as stable as it is the version we are running in
  production. After picking a version set the `DOCKER_TAG` environment variable to that value (e.g.
  `export DOCKER_TAG=":3b635063294a4636ee89579f57e531de04ef087f"`). We do not recommend using the
  tag `latest`. Instead use an exact commit from the `main` branch.
</Callout>

```bash
export DOCKER_REGISTRY="ghcr.io/graphql-hive/"
export DOCKER_TAG=":latest" # Pin this to an exact commit hash from our `main` branch, we publish a Docker image per each commit on main, and for PRs.
export HIVE_ENCRYPTION_SECRET=$(openssl rand -hex 16)
export HIVE_APP_BASE_URL="http://localhost:8080"
export HIVE_EMAIL_FROM="no-reply@graphql-hive.com"
export SUPERTOKENS_API_KEY=$(openssl rand -hex 16)
export CLICKHOUSE_USER="clickhouse"
export CLICKHOUSE_PASSWORD=$(openssl rand -hex 16)
export REDIS_PASSWORD=$(openssl rand -hex 16)
export POSTGRES_USER="postgres"
export POSTGRES_PASSWORD=$(openssl rand -hex 16)
export POSTGRES_DB="registry"
export MINIO_ROOT_USER="minioadmin"
export MINIO_ROOT_PASSWORD="minioadmin"
export CDN_AUTH_PRIVATE_KEY=$(openssl rand -hex 16)
```

<Callout type="info">
  We recommend saving the environment variables in a file that you can reuse later. E.g. a `.env` file that you
  can later on expand using `source .env`. We are using `openssl` here for generating passwords on the fly.

You can also embed the values within a new docker-compose file by running
`docker-compose -f docker-compose.community.yml config > docker-compose.with-env.yml` after setting
the environment variables.

</Callout>

After setting the environment variables, pull the required images for GraphQL Hive services. This's
going to take some time if you're doing it for the first time.

```bash
docker-compose -f docker-compose.community.yml pull
```

After it's done, you can start the GraphQL Hive services using `docker-compose`.

```bash
docker-compose -f docker-compose.community.yml up
# or of you embedded the environment variables into your docker-compose file and called it `docker-compose.with-env.yml`
docker-compose -f docker-compose.with-env.yml up
```

Wait until all the services are up and running.

Congratulations 🥳, you just started your own GraphQL Hive instance.

You'll notice the folder named `.hive` at your root directory that has been mounted from docker
representing the volumes for the different storages used by Hive services like `postgres` and
`minio`, etc... So yes data are currently local but we'll discuss that further more in the
`Next Steps` section.

> Note: deleting this directory will end up with a loss of data, as this directory is acting as a
> volume for storing the data from Hive dependencies: database and storage

## Testing GraphQL Hive

Visit `http://localhost:8080` in your browser and start using GraphQL Hive! The usage reporting
endpoint is bound to `http://localhost:8081`. The GraphQL API is bound to `http://localhost:8082`.
The artifacts bucket is bound to `http://localhost:8083`.

Firstly, you can head to `http://localhost:8080` in your browser and start using GraphQL Hive!
You'll need to Sign Up for an account by inputting your email and password. And once you do you'll
see your personal organization that has been automatically created for you by default.

### Creating first project and target

Now that we have GraphQL Hive setup locally, signed up and logged into our dashboard. We can start
creating our first project which we're going to push graphql schemas to.

So you're going to click on the `Create a Project` button on the top right, give it a name and
choose your project type. You're going to see all sorts of types `Single` which is the most common
and there're also `Federation` and `Stitching` but what's also nice is that you can define your own
custom endpoints for schema validation and schema building. I'm going to go with `Single` but you
can choose what best suits your needs. Then click `Create Project`, you're going to be redirected to
your project page where you're going to see 3 listed targets created for you by default:

- `development`
- `production`
- `staging`

We're going to use the `development` target in this guide and as soon as you click it you'll see
this message:

> Hive is waiting for your first schema.
>
> You can publish a schema with Hive CLI and Hive Client

If you already have a Nodejs project with a graphql schema that's great you can skip over the next
step, but if not so we're going to initialize a new package in our directory, you can use your
preferred package manager to do so:

```bash
# you can do the same with npm or yarn
pnpm init
```

Then create a `schema.graphql` file wherever you want, you can put it at the root for example with
any simple graphql schema like the following:

```graphql
type Query {
  hello: String!
}
```

You also need to have a git repository initilaized for your project, because Hive automatically uses
the last commit hash to determine who the author is and ties the schema with the commit hash. So
we're going to do that.

```bash
git init
```

Make sure you exclude the following in your `.gitignore` file, because you don't wanna push those in
your commits:

```
node_modules
.hive
hive.json
.env
```

Make sure to at least have a single commit so that Hive can associate the schema with it.

```bash
git add .
git commit -m"init"
```

Now that you've your Nodejs project setup, start by installing the GraphQL Hive CLI which we're
going to use to interact with Hive's server.

```bash
# you can do the same with npm or yarn
pnpm i -D @graphql-hive/cli
```

<Callout type="info">
  We recommend that you install the GraphQL Hive CLI as a dev dependency
</Callout>

After installing it, the CLI's bin is set to `hive` which means you can check if it was installed by
using this command:

```bash
# you can do the same with npm or yarn
pnpm hive --version
```

Make sure you get an output specifiying the installed version for your CLI.

And since we're using the Self Hosted version, we need to configure the Hive CLI to use our local
endpoint, by creating a `hive.json` file at the root of your directory. Then specify the `registry`
key to point it our local registry instead of the cloud version one and the value is going to be
`http://localhost:8082/graphql` where its running.

Additionally the schemas registry only allows trusted sources to be able to push to them, so we have
to create an access token by heading back to our project page, specifically on the target(in this
guide it's `development`) then go to the settings tab, if you scroll donw a bit you're going to see
the `Tokens` section, click the `Generate New Token` button, give it a name then we'll have to set
some permissions for that token, so expand the `All targets` section then give the `Registry` a
`Read & Write` access. Click the `Generate Token` button, then copy the token value. Head back to
the `hive.json` config file then add the following contents, while replacing the
`<your-access-token>` with the token value you just copied.

```json filename="hive.json"
{
  "registry": {
    "endpoint": "http://localhost:8082/graphql",
    "accessToken": "<your-access-token>"
  }
}
```

Now we're all setup to use the GraphQL Hive CLI against our Setup!

We can publish our first schema by using:

```bash
pnpm hive schema:publish ./schema.graphql
```

Go to the `Schema` tab in your project target and you're going to see your latest published schema.

## Bounded Ports Reference

| Port | Purpose                      |
| ---- | ---------------------------- |
| 8080 | Hive's Main App UI           |
| 8081 | The Usage Reporting Endpoint |
| 8082 | The GraphQL API              |
| 8083 | The Artifacts Bucket         |

## Next Steps

After doing your first testing with GraphQL Hive you should consider the following steps:

- Evaluate whether you want to run Databases yourself within Docker or instead use a managed
  database service.
- Set up backups for your data.
- Set up a CD pipeline for deploying GraphQL Hive to your Cloud Provider or bare-metal server of
  choice (e.g. a Kubernetes cluster or Docker Swarm)
- Set up a monitoring solution for your GraphQL Hive instance (leverage healthchecks, sentry error
  reporting, prometheus metrics, etc.).
- Configure the `emails` service to use a real email provider (by default it uses `sendmail`).
- Watch and follow the [GraphQL Hive GitHub repository](https://github.com/graphql-hive/platform)
  for new releases and updates.
- Set up a weekly reminder for updating your GraphQL Hive instance to the latest version and
  applying maintenance.
- Get yourself familiar with SuperTokens and follow their changelogs in order to keep your
  SuperTokens instance up-to-date.
- In order to use longer retention periods e.g. for conditional breaking changes or schema explorer
  overviews do the following: Open the postgres database, go to the table `organization`, find your
  organization and change `plan_name` to ENTERPRISE, `limit_operations_monthly` to 0 and
  `limit_retention_days` to a value between 0 and 360 in order to unlock full functionality.

You can review the configuration options of all the services in the corresponding Readme files.

- [`@hive/app`](https://github.com/graphql-hive/platform/tree/main/packages/web/app/README.md)
- [`@hive/server`](https://github.com/graphql-hive/platform/tree/main/packages/services/server/README.md)
- [`@hive/schema`](https://github.com/graphql-hive/platform/tree/main/packages/services/schema/README.md)
- [`@hive/webhooks`](https://github.com/graphql-hive/platform/tree/main/packages/services/webhooks/README.md)
- [`@hive/usage`](https://github.com/graphql-hive/platform/tree/main/packages/services/usage/README.md)
- [`@hive/usage-ingestor`](https://github.com/graphql-hive/platform/tree/main/packages/services/usage-ingestor/README.md)
- [`@hive/tokens`](https://github.com/graphql-hive/platform/tree/main/packages/services/tokens/README.md)
- [`@hive/storage`](https://github.com/graphql-hive/platform/tree/main/packages/services/storage/README.md)
- [`@hive/emails`](https://github.com/graphql-hive/platform/tree/main/packages/services/emails/README.md)
