not.bot Verify: Deployment Checklist

not.bot™ Verify: Deployment Checklist

Complete the Pre-flight Checklist before you start. Each phase below assumes the previous phase is done. Open the Operations & Reference Guide alongside this document if you need context for any step.

Pre-flight decision labels (A through K) appear throughout. Fill in the value you wrote down in the Pre-flight Checklist wherever you see one.

Phase 1: OpenBao

1.1 Install the OpenBao chart

The OpenBao image shipped in the chart (harbor.juliasocial-dev.com/library/openbao-bls, pinned by digest) already has the Chiakeys plugin baked in. No build step is required for the mainline deployment. If you need to rebuild the image from source for supply-chain validation, see Appendix A.

unzip not.bot_verify_deployment.zip
cd helm/openBao
helm dependency update
kubectl create namespace <A>
cd ../..
helm install openbao ./helm/openBao -n <A>

Verify: kubectl get pods -n <A> shows the openbao pod Running.

1.2 Initialize and unseal

kubectl port-forward svc/openbao 8200:8200 -n <A>

In a second terminal:

bao operator init -key-shares=1 -key-threshold=1

Save the Unseal Key and Root Token in your break-glass secret store. Losing the unseal key destroys the data in OpenBao.

bao operator unseal <UNSEAL_KEY>
read -rs -p "Enter Token: " && echo "$REPLY" | bao login -

OpenBao reseals on every pod restart. Keep the unseal key accessible to whoever is on call. Consider OpenBao auto-unseal if your environment supports it.

1.3 Create the secrets engine, namespace, and scoped token

bao namespace create <D>
export VAULT_NAMESPACE=<D>
bao secrets enable -path=chiakeys bls
bao secrets enable -path=secret -version=2 kv

The secret KV-v2 engine holds non-DID secrets that the admin service writes during normal operation — for example, the CA cert and key pair you upload when registering a Chia node in §6.2 are stored at secret/data/blockchain-nodes/<UUID>/certificate. The path name secret is fixed; the admin service is not configurable here.

Create policy.hcl:

path "chiakeys/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}
path "secret/data/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}
path "secret/metadata/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

Apply and create the scoped token:

bao policy write <D>-policy policy.hcl
bao token create -policy=<D>-policy -orphan -ttl=720h

Save the token. You will use it in Phase 5 (admin service) and never the root token.

-orphan is required. Without it, the new token is a child of the root token, and §1.4's bao token revoke -self cascades the revocation to every child — invalidating this scoped token before the admin service ever uses it. The failure surfaces later, when the admin service first writes to OpenBao, as a misleading permission denied on the chiakeys path. -orphan detaches the token so it survives the root revocation.

-ttl=720h (30 days) gives you a renewal window long enough to cover the deployment timeline. The default TTL in some OpenBao configurations is short enough to expire before you finish.

1.4 Revoke the root token

bao token revoke -self
rm -f ~/.vault-token

Phase 2: PostgreSQL

Connect to your PostgreSQL instance as a superuser and run:

Below, <F_USER>/<F> refer to the admin user name and password from Decision F, and <G_USER>/<G> refer to the signer user name and password from Decision G. The default names are notbot_admin and notbot_signer; you may keep them or substitute your own.

CREATE USER <F_USER> WITH PASSWORD '<F>';
CREATE USER <G_USER> WITH PASSWORD '<G>';
CREATE DATABASE <E> OWNER <F_USER>;

Grant the signer user read/write on tables created by the admin user (the admin service creates the schema on first startup):

\c <E>
ALTER DEFAULT PRIVILEGES FOR ROLE <F_USER>
  GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO <G_USER>;
ALTER DEFAULT PRIVILEGES FOR ROLE <F_USER>
  GRANT USAGE, SELECT ON SEQUENCES TO <G_USER>;
GRANT USAGE, CREATE ON SCHEMA public TO <G_USER>;

The current signature server runs its own database migrations at startup. That means the signer user needs CREATE on the target schema until the migration ownership model is changed. Without this grant, Phase 8 fails on PostgreSQL 15+ with permission denied for schema public while the signer runs migrations.

Verify: psql -h <HOST> -U <F_USER> -d <E> -c "SELECT 1;" succeeds with password F, and the same with <G_USER> and password G.

Phase 3: Keycloak

In your Keycloak admin console:

Create realm admin-service.
In that realm, create client admin-service with:
- Client authentication: ON
- Valid redirect URIs: https://<H>/*
- Web origins: https://<H>
Open the client's Credentials tab and copy the client secret.
Create at least one user in the realm. Set all of: username, email, first name, last name, password. Mark Email verified: ON and Temporary password: OFF. Modern Keycloak realms require email, firstName, and lastName by default for any user with the user role; missing any of these causes the eventual login (Phase 6.1) to fail with a misleading invalid_grant / Account is not fully set up error.

Decision H is the admin UI hostname. The redirect URI must use the same hostname you will set in app.baseUrl and the admin-service ingress later.

Phase 4: Chia node

The Chia node charts live in a public repo. Read readme.md in that repo for chart-level detail.

4.1 Generate the private CA

openssl genrsa -out private_ca.key 4096
openssl req -x509 -new -nodes -key private_ca.key -sha256 -days 3650 \
  -out private_ca.crt \
  -subj "/C=US/ST=Private/L=Private/O=Chia/OU=RPC/CN=Chia Private CA"
chmod 600 private_ca.key
chmod 644 private_ca.crt

Use the same CA and key on every Chia node. You will upload them again in Phase 6 when you register each node.

4.2 Build and push the node image

git clone https://github.com/GalactechsLLC/helm-chia-nodes.git
cd helm-chia-nodes
DOCKER_REPO="<YOUR_REGISTRY>/path" ./docker_build.sh

4.3 Create the environment values file

cd helm/chia-node

Create a new file named values-<your-suffix>.yaml. The suffix is yours to pick (commonly my-mainnet); the file does not exist in the repo — you are creating it. The README in the helm-chia-nodes repo documents the full values structure; the minimum overrides for not.bot Verify are:

chia-blockchain:
  nameOverride: "chia-node"
  fullnameOverride: "chia-node"
  image:
    repository: "<YOUR_REGISTRY>/path"
    tag: "<YOUR_TAG>"
  chia:
    ca:
      private_ca_crt: |
        -----BEGIN CERTIFICATE-----
        ...contents of private_ca.crt...
        -----END CERTIFICATE-----
      private_ca_key: |
        -----BEGIN PRIVATE KEY-----
        ...contents of private_ca.key...
        -----END PRIVATE KEY-----

Block scalar indentation must be consistent. Every line of the cert and key sits at the same indent under the field name. Wrong indentation produces a pod restart loop.

Volume size. The chart's default volume.size is 500Gi. See Operations & Reference Guide §5.6 for sizing context before changing this. Mainnet first-sync briefly co-locates a 121 GiB compressed checkpoint and the SQLite it extracts to — a smaller PVC will run out of space during extraction.

4.4 Create the namespace, update dependencies, diff, and deploy

The deploy scripts read CI_NAMESPACE from helm/env, which ships with CI_NAMESPACE="infrastructure". Create that namespace before running the scripts. If you want a different namespace, edit the env file first and create whatever you chose instead.

not.bot Verify operates on Chia mainnet only. The upstream helm-chia-nodes repo ships a separate helm/env_testnet file for testnet deploys — do not use it. Stick with ENV_FILE=./env.

kubectl create namespace infrastructure
cd ..   # back to helm/
CI_ENVIRONMENT=<your-suffix> ENV_FILE=./env ./update-chart.sh
CI_ENVIRONMENT=<your-suffix> ENV_FILE=./env ./diff.sh
CI_ENVIRONMENT=<your-suffix> ENV_FILE=./env ./upgrade.sh

The upstream scripts include a Is the Above Correct? [y/N] confirmation prompt. If you run them over a non-interactive shell (SSH without -t, CI pipeline, etc.) they will hang waiting for input. Either run from an interactive terminal, or pipe yes y to each:
yes y | CI_ENVIRONMENT=<your-suffix> ENV_FILE=./env ./upgrade.sh

Verify:

kubectl get pods -n infrastructure
kubectl logs -f <chia-node-pod> -n infrastructure

The pod reaches Running within a minute or two. A restart loop usually points at malformed CA YAML or a missing storage volume.

4.5 Wait for chain sync

The first deploy downloads a checkpoint from torrents.chia.net and walks forward to chain head. You cannot create the business DID or mint signature DIDs against an out-of-sync node. Continue deploying the admin service while this runs; sync only blocks the on-chain steps in Phase 6.

For redundancy, repeat steps 4.3 and 4.4 with a separate values file and release name to bring up additional nodes. Use the same private CA on each.

Phase 5: Admin service

5.1 Extract and configure

cd helm/admin-service

Open values.yaml. Fill in these blocks. Sensitive values are supplied through a Kubernetes Secret you create in §5.2; the chart references that Secret with *SecretKeyRef fields and does not render plaintext production credentials into its own Secret.

database:
  url: "jdbc:postgresql://<POSTGRES_HOST>:5432/<E>"
  username: "<F_USER>"
  adminPasswordSecretKeyRef:
    name: "admin-service-secrets"
    key: "POSTGRES_PASSWORD"

signerDatabase:
  host: "<POSTGRES_HOST>"
  port: 5432
  database: "<E>"
  username: "<G_USER>"
  passwordSecretKeyRef:
    name: "admin-service-secrets"
    key: "SIGNER_DB_PASSWORD"

keycloak:
  issuerUri: "http://keycloak.<KEYCLOAK_NAMESPACE>.svc.cluster.local:8080/realms/admin-service"
  externalUrl: "https://<KEYCLOAK_PUBLIC_HOSTNAME>"
  clientId: "admin-service"
  clientSecretSecretKeyRef:
    name: "admin-service-secrets"
    key: "KEYCLOAK_CLIENT_SECRET"

openbao:
  enabled: true
  addr: "http://openbao.<A>.svc.cluster.local:8200"
  namespace: "<D>"
  tokenSecretKeyRef:
    name: "admin-service-secrets"
    key: "OPENBAO_TOKEN"

app:
  baseUrl: "https://<H>"
  environment: "production"
  springProfile: "prod"

juliaServer:
  host: "identity.julia.social"
  port: 443
  secure: true
  baseUrl: ""   # empty derives from host/port/secure

If you use a different Kubernetes Secret name or key names in §5.2, update the four *SecretKeyRef blocks above to match.

TLS trust: the admin service fetches Keycloak's JWKS over HTTPS, so the JVM must trust whatever CA signed your Keycloak's cert. Public CAs (Let's Encrypt, DigiCert, etc.) are already trusted — no action needed. Internal CAs (corporate PKI, self-signed for evaluation) require adding the CA's public cert to the chart's extraCaCerts map:

extraCaCerts:
  my-internal-ca: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----

The chart's init container builds a custom JVM trust store containing both the image's default cacerts and any entries you list here. Leave extraCaCerts empty if Keycloak's cert is from a public CA. See Appendix B for the single-VM evaluation flow.

issuerUri is what the admin service backend uses to fetch Keycloak's public keys. It must point at your Keycloak — Keycloak is a prerequisite you provide, not part of not.bot Verify, so substitute <KEYCLOAK_NAMESPACE> and the service name with wherever your Keycloak actually lives. Examples:

In-cluster Keycloak in namespace identity: http://keycloak.identity.svc.cluster.local:8080/realms/admin-service

External Keycloak (managed, on another cluster, etc.): https://keycloak.example.com/realms/admin-service — drop the cluster.local suffix and use HTTPS if exposed publicly

Single-VM Appendix B deployment: https://keycloak.your-domain.example/realms/admin-service — must match the KC_HOSTNAME Keycloak is started with in §B.10 (Keycloak stamps that into every token's iss claim). The admin pod resolves this hostname to the in-cluster Keycloak via the CoreDNS hosts override (§B.12) and trusts the local CA via extraCaCerts (§B.13 §5.1). Do not point issuerUri at the cluster.local ExternalName here — the issuer claim would not match and login would fail.

externalUrl is what the operator's browser hits during login (must be HTTPS). A mismatch between the issuer claim in tokens Keycloak issues and the issuerUri here is the most common login failure. If logins fail with an issuer error, decode the JWT (e.g. jwt.io offline mode) and read the iss claim — issuerUri must equal whatever you find there.

5.2 Expose on the internal network

Operators reach the admin service through a LoadBalancer Service annotated for internal-only placement. The cloud provider provisions a private endpoint with a stable hostname; you point your internal DNS at it.

Single-VM / non-cloud deployments: the cloud-LB annotations below don't apply. See Appendix B for the ingress-nginx + local CA path (set service.type: ClusterIP and define an Ingress).

Create the namespace and the Secret referenced by §5.1:

kubectl create namespace <B>
kubectl create secret generic admin-service-secrets -n <B> \
  --from-literal=POSTGRES_PASSWORD='<F>' \
  --from-literal=SIGNER_DB_PASSWORD='<G>' \
  --from-literal=KEYCLOAK_CLIENT_SECRET='<CLIENT_SECRET_FROM_PHASE_3>' \
  --from-literal=OPENBAO_TOKEN='<SCOPED_TOKEN_FROM_STEP_1.3>'

The shipped admin-service image (harbor.juliasocial-dev.com/library/admin-service) is in Harbor's public library namespace and pulls anonymously. No imagePullSecret is needed for the mainline deployment. If you mirror the image into a private registry, create a docker-registry Secret in namespace B and set imagePullSecret in values.yaml to its name.

In values.yaml, set the service to LoadBalancer and add the internal-LB annotation for your cloud provider. Uncomment one:

service:
  type: LoadBalancer
  annotations:
    # AWS:   service.beta.kubernetes.io/aws-load-balancer-scheme: "internal"
    # GKE:   networking.gke.io/load-balancer-type: "Internal"
    # Azure: service.beta.kubernetes.io/azure-load-balancer-internal: "true"

After install (step 5.3), get the LB's private hostname:

kubectl get svc admin-service -n <B>

Create a CNAME in your internal DNS pointing H at that hostname. The hostname must match app.baseUrl from step 5.1 and the Keycloak redirect URIs from Phase 3.

TLS terminates at the LB. Add the certificate from Decision J to the service.annotations block above using your provider's annotation (aws-load-balancer-ssl-cert on AWS, managed cert resource on GCP, listener configuration on Azure). The admin service inside the pod serves plain HTTP on 8080.

5.3 Install and verify

cd ../..
helm install admin-service ./helm/admin-service -n <B>

Verify:

kubectl get pods -n <B> -l app.kubernetes.io/name=admin-service
kubectl logs -n <B> -l app.kubernetes.io/name=admin-service

Initial startup takes about 40 seconds. A pod stuck in CrashLoopBackOff traces back to one of the upstream dependencies; the log names which.

Browse to https://<H>. The Keycloak login page appears. Do not log in yet. Phase 6 covers first login.

Phase 6: Business identity setup

Confirm before starting:

At least one Chia node from Phase 4 reports synced.
You have DNS control for the domain value in your deployment-config.json.
You have an operator account in the Keycloak realm from Phase 3.

Browse to https://<H>. Click Login. Authenticate through Keycloak with the operator account. The dashboard shows the deployment as unconfigured.

6.2 Register Chia nodes

Register your Chia node(s) before uploading deployment-config.json. The admin UI rejects the upload until at least one blockchain node is configured, because the upload triggers an on-chain transaction (business DID creation) that needs a node to broadcast through.

For each node from Phase 4, open Blockchain → Add Node and fill in:

Host or IP Address: the in-cluster DNS name where the node's full_node RPC is reachable (e.g. chia-node.infrastructure.svc.cluster.local).
Port: 8555 (Chia mainnet full_node RPC).
CA Certificate (PEM Format): paste the contents of private_ca.crt from §4.1. Either paste the PEM block directly or use the "Upload .crt file" button.
Private Key (PEM Format): paste the contents of private_ca.key from §4.1. This is the CA's private key, not a client key. The admin service uses the CA cert+key as material to mint short-lived client certificates when it calls the chia node's RPC, and to delegate the same capability to your signature servers later. Treat this file with the same care as any private CA key.

The form also shows a Network row (read-only display, always "Mainnet" — not.bot Verify is mainnet-only) and a Use Secure Connection (SSL/TLS) checkbox that is pre-checked and disabled (the chia full_node RPC is mTLS-only, so SSL is mandatory). Neither requires operator action; the CA Certificate and Private Key fields below are what you actually fill in.

The chia full_node TLS certificate's only Subject Alternative Name is DNS:chia.net — a Chia design convention. The admin service handles this internally (overriding SNI to chia.net regardless of the Host you provide above). Generic HTTP clients connecting to chia full_node RPC would need to do the same; you do not need to configure anything for the admin service itself.

Wait until at least one node shows "synced" in the admin UI before continuing. The on-chain transaction in §6.3 requires a synced node. See Operations & Reference Guide §5.2 for what "synced" means and how long the first sync takes.

6.3 Upload deployment-config.json

In the admin interface, upload your deployment-config.json. The admin service contacts the billing server to register the deployment, which consumes the one-time API key from the file.

The API key is one-time-use, and ANY upload error consumes it. If the upload returns an error — whether the message says "Billing service is currently experiencing issues" (transient-looking), customerId or organizationName not recognized, or anything else — the key is consumed upstream regardless of what the error suggests. Retrying with the same deployment-config.json will always fail. Email support@julia.social for a re-issued welcome email containing a fresh key, then retry with the new file.

Common upload failures and what they actually mean:

The cluster cannot reach billingServerUrl over HTTPS. Fix egress, then request a re-issue.
"Billing service is currently experiencing issues" (or similar 5xx-ish wording). The upstream registration failed after the key was consumed. The error sounds transient but is permanent on your side — request a re-issue.
customerId or organizationName does not match Julia Social's records. Stop and contact support before retrying — your welcome email may have been generated incorrectly.
"You must register at least one blockchain node before uploading deployment-config.json" (or similar). You skipped §6.2 — go back and register a Chia node first, then re-upload.

6.4 Verify the business DID appeared

The §6.3 upload triggers business DID creation automatically — the admin service generates a BLS12-381 keypair in OpenBao and submits the DID transaction through a registered node. You don't click anything; the work happens in the background.

Open Business Identity in the admin UI and confirm the business DID is listed. It typically appears within a couple of minutes of the upload completing. If it's not visible after five minutes, check the admin-service logs for the on-chain transaction submission:

kubectl logs -n <B> -l app.kubernetes.io/name=admin-service | grep -i "business.*did"

Common reasons the DID does not appear: the registered Chia node fell out of sync between §6.2 and §6.3 (check Blockchain → node status), OpenBao sealed or unreachable (check kubectl get pods -n <A> and the chiakeys path with bao secrets list), or the admin service failed to write through the Vault path (look for permission denied in the logs — likely a policy or KV-v2 mount issue from §1.3).

6.5 Verify the domain

The admin interface displays a "Verify Domain" button. Clicking it opens a modal that walks through the DNS step. Under "Step 1: Add DNS Record" the modal shows the complete TXT record value (the julia-did= prefix plus the full DID) with a copy button — paste that value into a TXT record at the apex of your domain. Wait for propagation. Confirm with:

dig +short TXT <your-domain>

The output starts with julia-did=. Click Verify Domain in the admin interface.

If verification fails: check the value is character-for-character identical (some DNS UIs add quotes or split long values), wait longer for propagation, and retry.

Leave the TXT record in place.

Phase 7: Signature DID pool

The admin service maintains a pool of unallocated signature DIDs (target of at least two) in the background. DIDs are minted via Julia's identity service and recorded on chain. The pool refills automatically as signature servers consume DIDs at startup; you do not provision the pool manually.

Verify: open Signature DIDs in the admin UI and confirm at least one DID is listed with state "available." If you see zero DIDs more than five minutes after the §6.3 upload completes, check the admin-service logs for Julia identity service errors (look for POST request to Julia service endpoint: /new_signature_did or similar). If your admin UI build still exposes an Add Signature DID button and the pool has not populated automatically, click it once as an interim fallback, then confirm the new DID appears as available.

Phase 8: Signature servers

8.1 Configure

cd helm/signer-service

Edit values.yaml:

The signer needs a small bootstrap config — enough to contact the admin service at startup, after which it receives the rest of its config (DID assignment, OpenBao token, database credentials, Chia node info with TLS material) from the admin's /admin/signatureServerSetup response. The chart's values:

deployment:
  name: julia-signer
  replicaCount: 2
  image:
    repository: "harbor.juliasocial-dev.com/library/julia_signer"
    tag: "build_1776796141"
    pullPolicy: Always
  service:
    targetPort: 8080
  adminService:
    hostname: "admin-service.<B>.svc.cluster.local"
    port: 8080
    secure: false
  chiaNetwork: "Mainnet"
  rustLog: "info"
  apiKeySecret:
    name: "julia-signer-secrets"
    key: "API_KEY"
  ingress:
    enabled: true
    hosts:
      - host: <I>
        paths:
          - /(/+)?(.*)
    tls:
      - hosts:
          - <I>
        secretName: signer-tls

Retrieve the API key value from the admin database. The admin service generates a businesses.api_key row when you complete §6.3. There is no admin-UI surface for this value today, so you must query Postgres directly:

psql -h <POSTGRES_HOST> -U <F_USER> -d <E> -c "SELECT api_key FROM businesses LIMIT 1;"

The value begins with ltk_. Create the signer Secret in namespace C with that value:

kubectl create secret generic julia-signer-secrets \
  --from-literal=API_KEY='<businesses.api_key value, begins with ltk_>' \
  -n <C>

The chart reads that same Secret key into both env vars the signer requires: ADMIN_API_KEY (signer → admin authentication) and API_KEY (SDK adapter / not.bot app → signer authentication). This direct database query is an interim bootstrap path; a future release will move this credential to OpenBao-mediated setup so operators do not handle it.

The signer Ingress terminates TLS at the cluster edge using a pre-created Kubernetes TLS Secret. Create the Secret in the signer namespace (C) from Decision K:

kubectl create secret tls signer-tls \
  --cert=<path-to-K-cert> \
  --key=<path-to-K-key> \
  -n <C>

For cloud-managed certs (ACM, GCP managed cert) you have an alternative path: terminate TLS at a cloud LB above the Ingress and leave tls: [] empty here. Decision K's answer dictates which path to use.

Do NOT set CHIA_FULL_NODE, CHIA_FULL_NODE_PORT, CHIA_FULL_NODE_SSL, PRIVATE_CA_CRT, or PRIVATE_CA_KEY as overrides. The binary populates these itself from the admin response after startup. Setting CHIA_FULL_NODE_SSL in particular switches the TLS path from inline-PEM-from-admin (correct) to filesystem-paths (broken — the binary fails with No such file or directory looking for cert files that aren't there).

8.2 Install

kubectl create namespace <C>
cd ../..
helm install julia-signer ./helm/signer-service -n <C>

The shipped signer image (harbor.juliasocial-dev.com/library/julia_signer) is in Harbor's public library namespace and pulls anonymously. No imagePullSecret is needed.

Point internal DNS for I at your nginx ingress controller's internal address. This ingress is internal-only, reached by your SDK adapter, the same posture as the admin LB in Phase 5. It is not exposed to the public internet.

Single-VM / non-cloud deployments: use the local CA-signed cert from Appendix B §B.8 to create the signer-tls Secret in namespace C (the cert from §B.8 already covers Decision I's hostname). Run the kubectl create secret tls command from §8.1 above, pointing --cert and --key at /etc/verify-test/certs/server.crt and server.key respectively.

8.3 Verify

kubectl get pods -n <C> -l app.kubernetes.io/name=julia-signer
kubectl logs -n <C> -l app.kubernetes.io/name=julia-signer

The startup log shows DID assignment, OpenBao connection, database connection, domain credential delegation, and honest.bot™ credential acquisition. A failed step names itself. Common causes:

No available DIDs in the pool. The admin service mints to maintain the pool; if minting is failing, check that at least one Chia node is synced and OpenBao is unsealed.
OpenBao sealed or unreachable. Unseal, check NetworkPolicies.
Admin service unreachable from the signer namespace. Check service DNS and policies.
Cannot reach Julia Social. Check egress.

In the admin interface, Signature DIDs → the DID this server picked up shows "Assigned" with a recent heartbeat timestamp.

Phase 9: SDK integration

The SDK runs in your application backend, not in the cluster. Languages: Rust, JavaScript (Express), Python (FastAPI), Java (Spring MVC), Dart (shelf). Repo: https://github.com/julia-social/julia_web_sdk.

Set three environment variables on the host running your backend:

SIGNATURE_HOSTNAME=<I>
SIGNATURE_PORT=443
SIGNATURE_API_KEY=<api-key-from-admin-interface>

SIGNATURE_HOSTNAME must be the signature server load balancer (Decision I), not the host running the SDK adapter. Pointing both at the same host loops the WebSocket proxy back on itself.

Build the adapter for your framework following the SDK readme. Configure request_claims, require_site_pass, message_generator, on_success, on_failure, and expire_time. Mount the adapter's router into your web application.

Verify: call GET /signature/notbot from your frontend. The adapter returns a request ID. Build a universal link from it, scan with the not.bot app, complete a verification. Your on_success callback fires with the verified response.

Done

Your deployment is complete. The Operations & Reference Guide covers monitoring, alerting, scaling, failure modes, and troubleshooting for each component.

Appendix A: Rebuilding the OpenBao image from source

The chart's default image (harbor.juliasocial-dev.com/library/openbao-bls, pinned by digest) ships with the Chiakeys plugin (vault-plugin-secrets-bls) baked in. The mainline deployment uses it directly.

This appendix is for customers who need to rebuild the image from source — for example, to audit the plugin's provenance against a supply-chain review, or to mirror the image into an internal registry.

A.1 Build environment

You need:

Docker with buildx (docker buildx version)
Outbound access to github.com (to clone the plugin) and docker.io (to pull openbao/openbao:2.5.2 and golang:1.22-alpine)

You do not need Go or a C toolchain on your host. The Dockerfile below compiles the plugin inside an Alpine container so the resulting binary links against musl libc (matching the OpenBao runtime image) and uses a pinned Go version. Building on a glibc host (Ubuntu, Debian, RHEL) and copying the binary into the Alpine runtime image produces a binary that fails to load.

A.2 Multi-stage Dockerfile

Create a file named Dockerfile:

# Build stage — Alpine + pinned Go + C toolchain (cgo for blst)
FROM golang:1.22-alpine AS build
RUN apk add --no-cache git build-base
WORKDIR /src
RUN git clone https://github.com/mintgarden-io/vault-plugin-secrets-bls.git .
WORKDIR /src/cmd/vault-plugin-secrets-bls
ENV CGO_ENABLED=1
RUN go build -o /out/vault-plugin-secrets-bls

# Runtime stage — OpenBao with the plugin baked in
FROM openbao/openbao:2.5.2
COPY --from=build /out/vault-plugin-secrets-bls /openbao/plugins/

Build and push to your registry:

docker buildx build --platform linux/amd64 -t <YOUR_REGISTRY>/openbao-bls:<YOUR_TAG> --push .

Compute the plugin SHA-256 from inside the image (the chart's standalone config validates against this value):

docker run --rm --entrypoint sha256sum <YOUR_REGISTRY>/openbao-bls:<YOUR_TAG> /openbao/plugins/vault-plugin-secrets-bls

A.3 Use your image in the chart

Edit helm/openBao/values.yaml to point at your image and update the plugin SHA in the standalone config:

openbao:
  server:
    image:
      registry: <YOUR_REGISTRY>
      repository: openbao-bls
      tag: "<YOUR_TAG>"
    standalone:
      enabled: true
      config: |
        ...
        plugin "secret" "bls" {
          command     = "vault-plugin-secrets-bls"
          binary_name = "vault-plugin-secrets-bls"
          sha256sum   = "<SHA_FROM_A.2>"
        }

Verify with helm template helm/openBao before installing, then continue from §1.1.

Appendix B: Single-VM evaluation deployment

The mainline Preflight and Deployment Checklist assume you already operate a Kubernetes cluster with capacity, a managed PostgreSQL, a Keycloak you log into, a private container registry, real DNS, and a working TLS story. Most evaluators do not have that on day one — they want to spin up the whole stack on a single machine to see what they're buying. This appendix is the recipe for that scenario.

This is not a production deployment. The whole point of running everything on one host is that it's cheap to throw away and rebuild. You'll lose the deployment if the host dies, you have no HA story, OpenBao reseals on every host reboot and nobody else can unseal it, and you're trusting a self-signed CA you generated yourself. None of those are appropriate for production, but all of them are fine for "does this thing work for my use case?".

When you decide to go further, swap components one at a time — PostgreSQL is the easiest first move (point an ExternalName Service at managed Postgres instead of the local container), then Keycloak (your enterprise IAM), then DNS and TLS (real records and a real CA), then the cluster itself.

B.1 Host sizing

A single VM running every component plus its prereqs needs:

8 vCPU. Chia node wants 4 vCPU with headroom (it has to keep up with chain head; a node that falls behind takes the verification flow offline until it resyncs). The admin service is light (~250m). The signature server uses 500m–1 vCPU under MPC load. OpenBao, ingress-nginx, MetalLB, kubelet, containerd, Postgres, Keycloak fill the rest. 8 vCPU leaves headroom; 4 vCPU will run but get tight under load.
32 GiB RAM. Chia is the largest consumer and commonly needs several GiB during mainnet operation; leave headroom for sync, cache, and restarts. Keycloak likes 2 GiB. Postgres at this scale is 1 GiB. Admin, signer, OpenBao together are another 2–3 GiB. KIND control plane plus system overhead consumes the rest. A 16 GiB host can run a test rig, but 32 GiB leaves enough margin that one component's transient memory use does not destabilize the node.
500 GiB SSD. Dominated by the Chia chain database (~215 GiB after first sync, growing 5–10 GiB/month). Allow at least 500 GiB so peak disk pressure during checkpoint extraction (~310 GiB transient) fits and you have months of growth runway.

AWS reference: t3.2xlarge (8 vCPU / 32 GiB) with a 500 GiB gp3 EBS volume. For better Chia walk-forward throughput, bump gp3 IOPS to 6000 and throughput to 500 MiB/s during the first sync, then drop back to defaults afterward.

If you want to skip the Chia node entirely (and ship the deployment without on-chain operations — see Architecture & Privacy Guide for what that costs you), you can run on 4 vCPU / 16 GiB / 100 GiB. The other components are not the heavyweight ones.

B.2 Operating system and base packages

Ubuntu 22.04 LTS is the reference. Other Debian-derived distros work the same way; RHEL-family distros need the obvious package-manager substitutions.

sudo apt-get update
sudo apt-get install -y ca-certificates curl gnupg jq unzip openssl postgresql-client

postgresql-client is for your convenience — psql from the host to talk to the Postgres container in §B.9.

B.3 Docker

Install Docker CE with buildx and the compose plugin:

sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg \
  | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" \
  | sudo tee /etc/apt/sources.list.d/docker.list >/dev/null
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io \
                        docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker "$USER"   # log out + back in to pick this up

B.4 Kubernetes tooling

Install kubectl, helm, and the helm-diff plugin per Preflight items #1, #2, #3:

# kubectl (latest stable in the v1.32 channel)
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.32/deb/Release.key \
  | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.32/deb/ /" \
  | sudo tee /etc/apt/sources.list.d/kubernetes.list >/dev/null
sudo apt-get update
sudo apt-get install -y kubectl

# Helm 3
curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

# helm-diff plugin (chia-node deploy scripts need it)
helm plugin install https://github.com/databus23/helm-diff

Install the OpenBao CLI (Preflight #9) by downloading the matching linux-amd64 binary from the latest openbao/openbao GitHub release and putting it on PATH as /usr/local/bin/bao.

B.5 KIND cluster

KIND runs a Kubernetes cluster as a set of Docker containers on the host. For a single-VM deployment you want a single control-plane node with the host's port 80 and 443 mapped into the node — that's how ingress traffic from outside the VM reaches the cluster.

# Install KIND
curl -fsSL https://kind.sigs.k8s.io/dl/v0.30.0/kind-linux-amd64 -o /tmp/kind
sudo install -m 0755 /tmp/kind /usr/local/bin/kind
rm /tmp/kind

# Cluster config with port mappings for ingress
cat > /tmp/kind-config.yaml <<'EOF'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 80
    hostPort: 80
    protocol: TCP
  - containerPort: 443
    hostPort: 443
    protocol: TCP
EOF

kind create cluster --name notbot-test --config /tmp/kind-config.yaml --wait 120s
kubectl config use-context kind-notbot-test
kubectl cluster-info

extraPortMappings is what makes the cluster reachable from outside the VM. The ingress controller you install in §B.7 will bind to ports 80/443 inside the control-plane node, and KIND forwards from the host's 80/443 to those.

B.6 MetalLB for LoadBalancer Services

KIND has no cloud LB integration. Services of type LoadBalancer stay in Pending forever unless you provide an LB controller. MetalLB is the standard in-cluster answer; it picks an IP from a pool you configure and announces it via L2 (ARP) to the cluster's network.

For this deployment, MetalLB's role is narrower than you might expect — most external traffic enters through ingress-nginx (§B.7) on the host's port 80/443 mappings, not through a LoadBalancer IP. MetalLB is here so any chart that defaults to LoadBalancer (the admin service does, for one) gets a working IP instead of hanging in Pending.

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.14.9/config/manifests/metallb-native.yaml
kubectl -n metallb-system wait --for=condition=Available deployment/controller --timeout=300s
kubectl -n metallb-system rollout status daemonset/speaker --timeout=300s

# Derive an IP pool from KIND's docker network so MetalLB IPs are routable inside the host
DOCKER_SUBNET=$(docker network inspect kind \
  | python3 -c 'import sys,json; cfg=json.load(sys.stdin)[0]["IPAM"]["Config"]; print(next(c["Subnet"] for c in cfg if "." in c["Subnet"]))')
PREFIX=$(echo "$DOCKER_SUBNET" | cut -d. -f1-2)
cat <<EOF | kubectl apply -f -
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata: { name: notbot-pool, namespace: metallb-system }
spec:
  addresses: [ "${PREFIX}.255.200-${PREFIX}.255.250" ]
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata: { name: notbot-l2adv, namespace: metallb-system }
spec:
  ipAddressPools: [ notbot-pool ]
EOF

B.7 ingress-nginx for HTTPS termination

In a cloud deployment, your cloud's LB terminates TLS for you. KIND has no LB, so you terminate TLS yourself with an in-cluster ingress controller. ingress-nginx is the most widely-used; any controller with comparable feature support works.

Use the KIND-recommended manifests, which configure ingress-nginx to bind directly to the control-plane node's ports 80 and 443 via hostPort. Combined with the KIND extraPortMappings from §B.5, that means traffic to http(s)://<your-host-public-IP>/... lands on the ingress controller.

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.11.3/deploy/static/provider/kind/deploy.yaml
kubectl wait --namespace ingress-nginx \
  --for=condition=ready pod \
  --selector=app.kubernetes.io/component=controller \
  --timeout=300s

Once ingress-nginx is up, any Service you expose via an Ingress resource with a tls: block gets HTTPS termination using the TLS Secret you reference in that Ingress. You'll create the TLS Secret in §B.8.

For production: in a real cluster with a cloud LB, you don't need ingress-nginx unless your application requires its specific features. The admin-service and signer-service charts both work with a cloud LB doing TLS at L7 (or L4 with the pod doing TLS). Appendix B is the only place that talks about ingress-nginx.

B.8 Local CA and TLS certificates

For evaluation you generate a CA and use it to issue server certs for every hostname you want to reach over HTTPS. The CA's only job is to sign one cert (or a few). You install the CA's public cert in your OS trust store and in your browser's trust store so the cert chain validates.

sudo mkdir -p /etc/verify-test/{ca,certs}
sudo chmod 700 /etc/verify-test/ca

# CA key + cert
sudo openssl genrsa -out /etc/verify-test/ca/ca.key 4096
sudo openssl req -x509 -new -nodes -key /etc/verify-test/ca/ca.key -sha256 -days 3650 \
  -out /etc/verify-test/ca/ca.crt \
  -subj "/CN=Verify Eval CA"

# Server key + multi-SAN cert
SERVER_NAMES="DNS:admin.example.com,DNS:signer.example.com,DNS:keycloak.example.com,DNS:example.com"
sudo openssl genrsa -out /etc/verify-test/certs/server.key 4096
sudo openssl req -new -key /etc/verify-test/certs/server.key \
  -out /etc/verify-test/certs/server.csr \
  -subj "/CN=example.com" \
  -addext "subjectAltName=${SERVER_NAMES}"
sudo openssl x509 -req -in /etc/verify-test/certs/server.csr \
  -CA /etc/verify-test/ca/ca.crt -CAkey /etc/verify-test/ca/ca.key -CAcreateserial \
  -out /etc/verify-test/certs/server.crt -days 365 -sha256 \
  -extfile <(printf "subjectAltName=%s" "$SERVER_NAMES")

Replace example.com with whatever domain you control. The cert covers the apex and three subdomains — admin (Decision H), signer (Decision I), and Keycloak. All four DNS names resolve to the host's public IP.

Install the CA in the host's trust store so curl works without -k:

sudo cp /etc/verify-test/ca/ca.crt /usr/local/share/ca-certificates/verify-eval-ca.crt
sudo update-ca-certificates

You also need to install the CA in your laptop's browser trust store. Each browser has its own trust store; consult your browser's documentation. Firefox and Chrome both let you import a CA cert as "trusted to identify websites."

.app and other HSTS preload TLDs. Domains under .app, .dev, .bank, and a handful of others are in the Chromium HSTS preload list, which means the browser refuses to bypass an invalid cert warning at all — even with the CA installed, you must present a cert that validates cleanly. For evaluation, prefer a domain on a non-preloaded TLD (.com, .io, .test) or accept that you must get the local-CA install right before the browser will load anything.

B.9 PostgreSQL

Run Postgres as a Docker container on the KIND network. This puts it in DNS reach of the cluster pods at the container's name, and exposes it on the host's 127.0.0.1:5432 for your own psql use.

POSTGRES_PASSWORD=$(openssl rand -base64 24 | tr -d '/+=' | head -c 24)
echo "POSTGRES_PASSWORD=$POSTGRES_PASSWORD" >> ~/.verify-test-secrets

sudo mkdir -p /var/lib/postgresql-data
sudo chown 999:999 /var/lib/postgresql-data   # Postgres container's UID
sudo chmod 700 /var/lib/postgresql-data

docker run -d \
  --name postgres-verify \
  --network kind \
  --restart unless-stopped \
  -p 127.0.0.1:5432:5432 \
  -v /var/lib/postgresql-data:/var/lib/postgresql/data \
  -e POSTGRES_PASSWORD="$POSTGRES_PASSWORD" \
  postgres:16

Wait for it to be ready, then create the keycloak database:

KEYCLOAK_DB_PASSWORD=$(openssl rand -base64 24 | tr -d '/+=' | head -c 24)
echo "KEYCLOAK_DB_PASSWORD=$KEYCLOAK_DB_PASSWORD" >> ~/.verify-test-secrets

docker exec -i -e PGPASSWORD="$POSTGRES_PASSWORD" postgres-verify \
  psql -U postgres -v ON_ERROR_STOP=1 -v kc="$KEYCLOAK_DB_PASSWORD" <<'SQL'
CREATE DATABASE keycloak;
CREATE ROLE keycloak LOGIN;
SELECT format('ALTER ROLE keycloak WITH LOGIN PASSWORD %L', :'kc')\gexec
GRANT ALL PRIVILEGES ON DATABASE keycloak TO keycloak;
\c keycloak
GRANT ALL ON SCHEMA public TO keycloak;
ALTER SCHEMA public OWNER TO keycloak;
SQL

Expose Postgres into the cluster by DNS name with an ExternalName Service. Pods will resolve postgres.default.svc.cluster.local to the container:

kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata: { name: postgres, namespace: default }
spec:
  type: ExternalName
  externalName: postgres-verify
  ports: [ { port: 5432, targetPort: 5432 } ]
EOF

The verify database and its users (Decisions F and G) are created later by the mainline Deployment Checklist §2.

For production: swap to managed Postgres (RDS, Aurora, CloudSQL, Azure Database). Update the ExternalName Service to point at its endpoint, or have your admin-service values.yaml reference the managed hostname directly. The schema and users from DC §2 work identically.

B.10 Keycloak

Run Keycloak as a Docker container on the KIND network, backed by the Postgres container from §B.9.

KC_HOSTNAME="keycloak.example.com"   # match your DNS / TLS cert
KC_ADMIN_PASSWORD=$(openssl rand -base64 24 | tr -d '/+=' | head -c 24)
echo "KC_ADMIN_PASSWORD=$KC_ADMIN_PASSWORD" >> ~/.verify-test-secrets

docker run -d \
  --name keycloak \
  --network kind \
  --restart unless-stopped \
  -p 8080:8080 \
  -e KC_DB=postgres \
  -e KC_DB_URL="jdbc:postgresql://postgres-verify:5432/keycloak" \
  -e KC_DB_USERNAME=keycloak \
  -e KC_DB_PASSWORD="$KEYCLOAK_DB_PASSWORD" \
  -e KC_HOSTNAME="https://${KC_HOSTNAME}" \
  -e KC_HOSTNAME_STRICT=false \
  -e KC_HTTP_ENABLED=true \
  -e KC_PROXY_HEADERS=xforwarded \
  -e KEYCLOAK_ADMIN=admin \
  -e KEYCLOAK_ADMIN_PASSWORD="$KC_ADMIN_PASSWORD" \
  quay.io/keycloak/keycloak:25.0.6 \
  start-dev

Expose Keycloak into the cluster with another ExternalName Service:

kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata: { name: keycloak, namespace: default }
spec:
  type: ExternalName
  externalName: keycloak
  ports: [ { port: 8080, targetPort: 8080 } ]
EOF

Expose Keycloak to the browser through ingress-nginx with TLS termination from your local CA. First load the cert into a Secret in the namespace where the Ingress lives:

kubectl create secret tls verify-eval-tls \
  --cert=/etc/verify-test/certs/server.crt \
  --key=/etc/verify-test/certs/server.key \
  --namespace default

Then point an Ingress at Keycloak:

kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: keycloak
  namespace: default
spec:
  ingressClassName: nginx
  tls:
    - hosts: [ ${KC_HOSTNAME} ]
      secretName: verify-eval-tls
  rules:
    - host: ${KC_HOSTNAME}
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: keycloak
                port: { number: 8080 }
EOF

Browse to https://${KC_HOSTNAME}/admin and log in with admin / $KC_ADMIN_PASSWORD. This is the Keycloak admin console you'll use during Deployment Checklist §3 to create the admin-service realm.

For production: swap Keycloak for whatever IAM speaks OIDC. The admin service only needs an issuer URL, a client with a known secret, and a redirect URI. Any OIDC-compliant IdP (Okta, Auth0, Azure AD, Google Workspace) works.

B.11 Local container registry (optional)

You can skip this section. The shipped charts pull from harbor.juliasocial-dev.com/library/*, which is anonymously readable; the only reason to run a local registry is to push your own images (Appendix A rebuilds) and pull them without round-tripping to the internet, or to mirror Harbor images for an air-gapped test.

docker run -d --name registry-verify --network kind --restart unless-stopped \
  -p 127.0.0.1:5000:5000 registry:2

Tell KIND's containerd to trust this insecure registry (HTTP, no TLS):

docker exec notbot-test-control-plane bash -c '
cat >> /etc/containerd/config.toml <<EOF

[plugins."io.containerd.grpc.v1.cri".registry]
  config_path = "/etc/containerd/certs.d"
EOF
mkdir -p /etc/containerd/certs.d/registry-verify:5000
cat > /etc/containerd/certs.d/registry-verify:5000/hosts.toml <<EOF
[host."http://registry-verify:5000"]
EOF
systemctl restart containerd
'

Pull, retag, and push from the host as 127.0.0.1:5000/...; pull from inside the cluster as registry-verify:5000/....

B.12 DNS and routing

Point real DNS records at the VM's public IP for every hostname your local cert covers (the apex plus admin / signer / keycloak subdomains in the §B.8 example). Provider doesn't matter — anything that lets you set A records works.

In-cluster traffic uses the ExternalName Services from §B.9 and §B.10 (postgres.default.svc.cluster.local, keycloak.default.svc.cluster.local), so the cluster does not need to resolve your public hostnames internally.

If you want to evaluate without touching DNS, the alternative is to add entries to /etc/hosts on the operator's laptop pointing each hostname at the VM's public IP. That bypasses DNS but every laptop you test from needs the same hosts entries.

AWS EC2 evaluators: AWS doesn't NAT-loopback an instance's own Elastic IP by default. Commands from the EC2 host targeting hostnames that resolve to the EC2's own EIP (e.g. curl https://keycloak.your-domain.example:8443/... from the EC2 shell) hang and time out. Two fixes you'll need, one for the host and one for the cluster:

Host-side: add /etc/hosts entries mapping each hostname to 127.0.0.1:

sudo tee -a /etc/hosts <<EOF
127.0.0.1 keycloak.your-domain.example
127.0.0.1 admin.your-domain.example
127.0.0.1 signer.your-domain.example
EOF

Cluster-side: add a CoreDNS hosts entry for any hostname a pod needs to reach. The admin-service pod fetches Keycloak's JWKS over HTTPS at the public hostname — that lookup also resolves to the EC2 EIP and hits the same loopback wall, so the admin pod fails to log operators in.

Get the Keycloak container's IP on the kind docker network:
```
KC_IP=$(docker inspect keycloak --format '{{.NetworkSettings.Networks.kind.IPAddress}}')
echo "$KC_IP"
```
Patch the cluster's CoreDNS ConfigMap to add a hosts plugin entry, then restart the deployment. The full Corefile (replace the existing data.Corefile with this) is:
```
.:53 {
    errors
    health { lameduck 5s }
    ready
    hosts {
       <KC_IP> keycloak.your-domain.example
       fallthrough
    }
    kubernetes cluster.local in-addr.arpa ip6.arpa {
       pods insecure
       fallthrough in-addr.arpa ip6.arpa
       ttl 30
    }
    prometheus :9153
    forward . /etc/resolv.conf { max_concurrent 1000 }
    cache 30 {
       disable success cluster.local
       disable denial cluster.local
    }
    loop
    reload
    loadbalance
}
```
Then kubectl -n kube-system rollout restart deployment/coredns. Verify with kubectl run dns-test --rm -it --image=alpine:3.19 -- sh -c 'apk add bind-tools >/dev/null && nslookup keycloak.your-domain.example' — the answer should be the kind-network IP, not the EC2's public IP.

If you skip the cluster-side fix, you'll get all the way through admin-service install and only discover the problem when an operator tries to log in (the admin pod hangs on JWKS fetch and the browser hangs on the OAuth redirect-back). Add both entries before you deploy the admin service.

B.13 Continuing with the Deployment Checklist

At this point your single-VM rig satisfies the entire Preflight Checklist:

Kubernetes cluster + kubectl context: KIND (§B.5)
Helm 3, helm-diff plugin: §B.4
PostgreSQL 14+: §B.9
Keycloak 22+: §B.10
Container registry: skipped (using Harbor library) or §B.11
LoadBalancer support: MetalLB (§B.6)
Workstation tools (openssl, bao CLI): §B.2 / §B.4
DNS control: §B.12
TLS certificate (Decision J): §B.8

Continue from the Deployment Checklist's Phase 1. Three phase-specific notes for single-VM deployments:

§5.1 (admin-service trust store for Keycloak's local-CA cert). Your Keycloak is served from a cert signed by the local CA you created in §B.8. The JVM inside the admin-service pod does not trust that CA by default, so JWKS validation will fail unless you tell the chart to add the CA to the pod's trust store. In your admin-service values.yaml (or values overlay), set:
```
extraCaCerts:
  verify-eval-ca: |
    <paste the contents of /etc/verify-test/ca/ca.crt here>
```
Run cat /etc/verify-test/ca/ca.crt on the EC2 host to retrieve the PEM block. The chart's init container will import it into the admin pod's JVM trust store at startup.
§5.2 (Admin service LB). The mainline checklist annotates the admin service Service for an internal cloud LB. On KIND, set service.type: ClusterIP and reach the admin service through ingress-nginx — define an Ingress similar to the Keycloak one in §B.10, pointing at the admin service in your chosen namespace. TLS Secrets are namespace-scoped, so the verify-eval-tls Secret created in default (§B.10) is not visible to an Ingress in namespace B; recreate it there from the same local CA-signed cert before defining the Ingress:
```
kubectl create secret tls verify-eval-tls \
  --cert=/etc/verify-test/certs/server.crt \
  --key=/etc/verify-test/certs/server.key \
  --namespace <B>
```
then reference verify-eval-tls as the Ingress secretName.
§8 (Signature server). Same pattern — use an Ingress for Decision I, not a cloud LB. Back it with the signer-tls Secret you already created in namespace C in §8.1 (not verify-eval-tls, which exists only in default).

View this page as Markdown

not.bot Verify: Deployment Checklist