Skip to content
Docs
Development
Development with Kind

Development with a real Kubernetes cluster (kind)

Here we’ll cover how to run the Nstance operator locally and have it connecting to a kind (Kubernetes in Docker) cluster, and a local Nstance server using the dev-tmux provider, for fast development iteration and debugging.

If you are unfamiliar with the Nstance development environment in general, please read local setup first.

Terminal Tab/Window 1

Step 1.) Start an Nstance Server

The first thing you’ll want to do is ensure you have your nstance-server running with a dev-s3, as this will generate a fresh CA certificate, which the operator will require a copy of.

make clean-dev && make dev-tmux

This starts s3 + server only (no dev-k8s or operator), which is what you want since the operator will run separately against kind.

Terminal Tab/Window 2

Open a new/separate terminal window/tab and keep the nstance-server from step 1 running.

Step 2.a) Create a Kubernetes (kind) Cluster

Note that you must have docker/podman installed for this to work:

make kind-create

This creates a kind cluster named nstance-dev (with a default config generated at temp/kind-config.yaml if one doesn’t exist) and sets your kubectl context to kind-nstance-dev.

Step 2.b) Provision the Kubernetes (kind) Cluster

Provision the kind cluster with all required dependencies:

make kind-provision

Note that this may take a minute or so and requires an internet connection to download kubernetes manifests for cert-manager and CAPI.

This single command handles:

  • Installing cert-manager (required by CAPI for webhook TLS certificates)
  • Deploying Cluster API (CAPI) core components (CRDs, controllers, validation webhooks)
  • Deploying Nstance CRDs and CAPI RBAC
  • Creating the cluster CA ConfigMap (nstance-cluster-ca) from the CA cert generated by the server
  • Exporting the Kind kubeconfig to temp/operator/kubeconfig
  • Creating the operator config at temp/operator/config.yaml with shard addresses matching dev-tmux
  • Generating a registration nonce and storing it in a nstance-operator-nonce Secret

The nonce is valid for 3 hours by default. If it expires, re-run make kind-provision to generate a new one.

4. Run the Operator locally against the Kubernetes (kind) Cluster 

make dev-operator

This runs the operator with Air for hot-reload. Note the differences in runtime dependencies when run locally vs in-cluster:

namespace: (for Secrets & ConfigMaps)

  • in-cluster: /var/run/secrets/kubernetes.io/serviceaccount/namespace
  • locally: NSTANCE_NAMESPACE=default

kubeconfig:

  • in-cluster: /var/run/secrets/kubernetes.io/serviceaccount/.
  • locally: temp/operator/kubeconfig

nstance-operator config:

  • in-cluster: /etc/nstance/operator/config.yaml (Helm chart mounts from ConfigMap).
  • locally: temp/operator/config.yaml (specified via --config argument).

What Happens

The operator will:

  • Connect to the Kind cluster via the kubeconfig file
  • Load the cluster CA from the nstance-cluster-ca ConfigMap
  • Load or generate an Ed25519 keypair (stored in nstance-operator-key Secret)
  • Register with nstance-server using the nonce from nstance-operator-nonce Secret
  • Receive and store a client certificate in nstance-operator-cert Secret
  • Connect to the operator gRPC port (8993) with mTLS for sync/drain operations
  • Reconcile NstanceMachinePool, NstanceMachine, and NstanceShardGroup CRDs

This setup allows for rapid development cycles without needing to rebuild container images for each change.

Cleaning Up

Terminal Window/Tab 2

Ctrl-C to stop the nstance-operator then run delete the kind cluster:

make kind-delete

Terminal Window/Tab 1

Ctrl-C to stop the nstance-server then ensure the temp dir is cleaned up and processes are stopped:

make clean-dev
Local Development Setup