# Development with Kind
# Development with a real Kubernetes cluster (kind)

Here we'll cover how to run the Nstance operator locally and have it connecting to a [kind](https://kind.sigs.k8s.io/) (Kubernetes in Docker) cluster, and a local Nstance server using the `dev-tmux` provider, for fast development iteration and debugging.

If you are unfamiliar with the Nstance development environment in general, please read [local setup](./local-setup.md) first.

### Terminal Tab/Window 1

#### Step 1.) Start an Nstance Server

The first thing you'll want to do is ensure you have your `nstance-server` running with a `dev-s3`, as this will generate a fresh CA certificate, which the operator will require a copy of.

```bash
make clean-dev && make dev-tmux
```

This starts s3 + server only (no dev-k8s or operator), which is what you want since the operator will run separately against kind.

### Terminal Tab/Window 2

Open a new/separate terminal window/tab and keep the nstance-server from step 1 running.

#### Step 2.a) Create a Kubernetes (kind) Cluster

Note that you must have docker/podman installed for this to work:

```bash
make kind-create
```

This creates a kind cluster named `nstance-dev` (with a default config generated at `temp/kind-config.yaml` if one doesn't exist) and sets your kubectl context to `kind-nstance-dev`.

#### Step 2.b) Provision the Kubernetes (kind) Cluster

Provision the kind cluster with all required dependencies:

```bash
make kind-provision
```

Note that this may take a minute or so and requires an internet connection to download kubernetes manifests for cert-manager and CAPI.

This single command handles:
- Installing **cert-manager** (required by CAPI for webhook TLS certificates)
- Deploying **Cluster API (CAPI)** core components (CRDs, controllers, validation webhooks)
- Deploying **Nstance CRDs** and CAPI RBAC
- Creating the **cluster CA ConfigMap** (`nstance-cluster-ca`) from the CA cert generated by the server
- Exporting the **Kind kubeconfig** to `temp/operator/kubeconfig`
- Creating the **operator config** at `temp/operator/config.yaml` with shard addresses matching `dev-tmux`
- Generating a **registration nonce** and storing it in a `nstance-operator-nonce` Secret

The nonce is valid for 3 hours by default. If it expires, re-run `make kind-provision` to generate a new one.

### 4. Run the Operator locally against the Kubernetes (kind) Cluster

```bash
make dev-operator
```

This runs the operator with Air for hot-reload. Note the differences in runtime dependencies when run locally vs in-cluster:

**namespace**: (for Secrets & ConfigMaps)
  
- in-cluster: `/var/run/secrets/kubernetes.io/serviceaccount/namespace`
- locally: `NSTANCE_NAMESPACE=default`

**kubeconfig**:
  
- in-cluster: `/var/run/secrets/kubernetes.io/serviceaccount/`.
- locally: `temp/operator/kubeconfig`

**nstance-operator config**:
  
- in-cluster: `/etc/nstance/operator/config.yaml` (Helm chart mounts from ConfigMap).
- locally: `temp/operator/config.yaml` (specified via `--config` argument).

### What Happens

The operator will:
- Connect to the Kind cluster via the kubeconfig file
- Load the cluster CA from the `nstance-cluster-ca` ConfigMap
- Load or generate an Ed25519 keypair (stored in `nstance-operator-key` Secret)
- Register with nstance-server using the nonce from `nstance-operator-nonce` Secret
- Receive and store a client certificate in `nstance-operator-cert` Secret
- Connect to the operator gRPC port (8993) with mTLS for sync/drain operations
- Reconcile `NstanceMachinePool`, `NstanceMachine`, and `NstanceShardGroup` CRDs

This setup allows for rapid development cycles without needing to rebuild container images for each change.

### Cleaning Up

#### Terminal Window/Tab 2

Ctrl-C to stop the nstance-operator then run delete the kind cluster:

```bash
make kind-delete
```


#### Terminal Window/Tab 1

Ctrl-C to stop the nstance-server then ensure the temp dir is cleaned up and processes are stopped:

```bash
make clean-dev
```