Kubernetes - Custom Resource Definitions (CRDs)

Previous
Quiz

Kubernetes is a robust platform, but sometimes, its built-in resources—such as Pods, Deployments, and Services—aren't enough to meet the unique needs of an application or infrastructure. What if we need to define and manage resources that Kubernetes doesn't support out of the box? That's where Custom Resource Definitions (CRDs) come in.

CRDs allow us to extend Kubernetes by defining our own resource types, enabling us to treat custom objects just like native Kubernetes resources. Whether we're building a custom controller, managing application configurations, or integrating with external systems, CRDs give us the flexibility to shape Kubernetes to fit our needs.

In this chapter, we'll take a hands-on approach to understanding CRDs. We'll cover:

  • What CRDs are and why they matter
  • How to create and manage CRDs
  • Writing a simple custom controller to manage custom resources

What Are Custom Resource Definitions (CRDs)?

A Custom Resource Definition (CRD) extends Kubernetes by adding new resource types. Once a CRD is registered, we can create and manage instances of this custom resource just like we do with built-in resources.

For example, if we need to manage database configurations in Kubernetes, we can define a Database CRD and then create custom resources like my-database that Kubernetes understands.

When Should We Use CRDs?

  • When we need to store and manage application-specific configurations.
  • When Kubernetes' built-in resources are insufficient for our use case.
  • When we are building a Kubernetes-native application that requires custom logic.

Creating a Custom Resource Definition (CRD)

To define a custom resource in Kubernetes, we need to create a CustomResourceDefinition YAML file and apply it.

Define the CRD

Create a file called database-crd.yaml with the following content:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: databases.example.com
spec:
group: example.com
names:
kind: Database
plural: databases
singular: database
scope: Namespaced
versions:
- name: v1
served: true
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
engine:
type: string
version:
type: string

Apply the CRD

Run the following command to create the CRD in Kubernetes:

$ kubectl apply -f database-crd.yaml

Output

customresourcedefinition.apiextensions.k8s.io/databases.example.com created

To verify the CRD was created, run:

$ kubectl get crds

Output

NAME                    CREATED AT
databases.example.com   2025-03-26T10:37:42Z

Creating a Custom Resource Instance

Once the CRD is created, we can create a custom resource (CR) that follows its definition.

Define the Custom Resource

Create a file called my-database.yaml:

apiVersion: example.com/v1
kind: Database
metadata:
name: my-database
spec:
engine: postgres
version: "14"

Apply the Custom Resource:

$ kubectl apply -f my-database.yaml

Output

database.example.com/my-database created

To check if the resource was created, run:

$ kubectl get databases

Output

NAME          AGE
my-database   26s

To get details of our custom resource:

$ kubectl get database my-database -o yaml

Output

apiVersion: example.com/v1
kind: Database
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"example.com/v1","kind":"Database","metadata":{"annotations":{},"name":"my-database","namespace":"default"},"spec":{"engine":"postgres","version":"14"}}
creationTimestamp: "2025-03-26T10:44:30Z"
generation: 1
name: my-database
namespace: default
resourceVersion: "5168"
uid: aa98751b-0335-47be-af18-e9090bf2a90b
spec:
engine: postgres
version: "14"

Managing CRDs with a Custom Controller

Defining a Custom Resource Definition (CRD) allows us to create custom Kubernetes resources, but it doesn't provide any behavior on its own. To add logic and automate tasks related to our custom resources, we need a Custom Controller.

A controller watches for changes in CRDs and responds accordingly—just like Kubernetes' built-in controllers manage resources like Deployments and Services.

Choosing a Framework for Custom Controllers

There are multiple ways to build a Kubernetes controller:

  • Kubebuilder − A popular tool for scaffolding controllers in Go, commonly used for production-grade operators.
  • Operator SDK − A higher-level framework that builds on Kubebuilder and simplifies the process of writing Kubernetes Operators.
  • Client Libraries (controller-runtime) − A lower-level approach using controller-runtime directly.
  • Custom Scripts (Python, Bash, etc.) − Simple controllers can be built using kubectl and watch loops, but this approach lacks scalability.

For a proper implementation, we'll use Go with controller-runtime, as it is widely adopted in Kubernetes ecosystems.

Implement a Basic Controller

A simple Go-based controller listens for changes to a Database Custom Resource and logs them. Using an editor, create file database-controller.go and add the following content:

package main

import (
   "context"
   "fmt"
   "log"
   "os"

   "k8s.io/apimachinery/pkg/runtime"
   "k8s.io/apimachinery/pkg/runtime/schema"
   "sigs.k8s.io/controller-runtime/pkg/client"
   "sigs.k8s.io/controller-runtime/pkg/reconcile"
   ctrl "sigs.k8s.io/controller-runtime"
)

// Define GroupVersion for the CRD
var GroupVersion = schema.GroupVersion{Group: "example.com", Version: "v1"}

// Define the Database Custom Resource struct
type Database struct {
   client.Object
   Spec DatabaseSpec `json:"spec"`
}

type DatabaseSpec struct {
   Engine  string `json:"engine"`
   Version string `json:"version"`
}

// Implement DeepCopyObject to satisfy the client.Object interface
func (d *Database) DeepCopyObject() runtime.Object {
   copy := *d
   return &copy
}

// Define the Reconciler for the CRD
type DatabaseReconciler struct {
   client.Client
   Scheme *runtime.Scheme
}

func (r *DatabaseReconciler) Reconcile(ctx context.Context, req reconcile.Request) (reconcile.Result, error) {
   fmt.Println("Reconciling Database resource:", req.NamespacedName)
   return reconcile.Result{}, nil
}

func main() {
   // Create a controller manager
   mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
      Scheme: runtime.NewScheme(),
   })
   if err != nil {
      log.Fatal(err)
   }

   // Register the CRD in the scheme
   scheme := mgr.GetScheme()
   scheme.AddKnownTypes(GroupVersion, &Database{})

   // Add the custom reconciler to the manager
   err = ctrl.NewControllerManagedBy(mgr).
   For(&Database{}).
   Complete(&DatabaseReconciler{Client: mgr.GetClient(), Scheme: scheme})

   if err != nil {
      log.Fatal(err)
   }

   log.Println("Starting Database controller")
   if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil {
      os.Exit(1)
   }
}

Explanation

This Go program is a Kubernetes Custom Controller that watches a Custom Resource Definition (CRD) named Database. It listens for changes and logs when a Database resource is created, updated, or deleted.

Build and Run the Controller

To compile and run the controller, follow these steps:

Initialize a Go module (if not already done):

$ go mod init example.com/database-controller

Output

go: creating new go.mod: module example.com/database-controller

Install the necessary dependencies:

$ go get sigs.k8s.io/controller-runtime

Output

go: added k8s.io/utils v0.0.0-20241104100929-3ea5e8cea738
go: added sigs.k8s.io/controller-runtime v0.20.4
go: added sigs.k8s.io/structured-merge-diff/v4 v4.4.2
go: added sigs.k8s.io/yaml v1.4.0

$ go mod tidy

Output

go: finding module for package sigs.k8s.io/controller-runtime
go: finding module for package k8s.io/apimachinery/pkg/runtime
go: added sigs.k8s.io/controller-runtime v0.20.4
go: added sigs.k8s.io/structured-merge-diff/v4 v4.4.2
go: added sigs.k8s.io/yaml v1.4.0
go: downloading dependencies...

This output confirms that Go successfully initialized the module and fetched the required dependencies.

Build the controller:

$ go build -o database-controller database-controller.go

Run the controller (local development mode):

./database-controller

Output

2025/03/26 13:54:17 Starting Database controller
2025-03-26T13:54:17Z    INFO    controller-runtime.metrics      Starting metrics server
2025-03-26T13:54:17Z    INFO    controller-runtime.metrics      Serving metrics server  {"bindAddress": ":8080", "secure": false}
2025-03-26T13:54:17Z    INFO    Starting EventSource    {"controller": "database", "controllerGroup": "example.com", "controllerKind": "Database", "source": "kind source: *main.Database"}

This controller will now listen for changes to Database resources and log events accordingly. You can extend it by adding additional reconciliation logic.

Conclusion

In this chapter, we explored the fundamentals of CRDs, learned how to create and manage custom resources, and built a simple controller using Go and the controller-runtime library. The controller we implemented listens for changes to our custom Database resource and provides a foundation for more advanced automation.

Using CRDs and custom controllers, we can build Kubernetes-native applications, streamline complex operations, and enhance the overall management of cloud-native workloads. As you advance, consider integrating features like webhooks, status updates, and operator patterns to create more robust and intelligent controllers for your Kubernetes environment.

Print Page
Previous
Next