Patterns and Best Practices

This document describes patterns and best practices, to build and run operators, and how to implement them in terms of the Java Operator SDK (JOSDK).

See also best practices in Operator SDK.

Implementing a Reconciler

Reconcile All The Resources All the Time

The reconciliation can be triggered by events from multiple sources. It could be tempting to check the events and reconcile just the related resource or subset of resources that the controller manages. However, this is considered an anti-pattern for operators because the distributed nature of Kubernetes makes it difficult to ensure that all events are always received. If, for some reason, your operator doesn’t receive some events, if you do not reconcile the whole state, you might be operating with improper assumptions about the state of the cluster. This is why it is important to always reconcile all the resources, no matter how tempting it might be to only consider a subset. Luckily, JOSDK tries to make it as easy and efficient as possible by providing smart caches to avoid unduly accessing the Kubernetes API server and by making sure your reconciler is only triggered when needed.

Since there is a consensus regarding this topic in the industry, JOSDK does not provide event access from Reconciler implementations anymore starting with version 2 of the framework.

EventSources and Caching

As mentioned above during a reconciliation best practice is to reconcile all the dependent resources managed by the controller. This means that we want to compare a desired state with the actual state of the cluster. Reading the actual state of a resource from the Kubernetes API Server directly all the time would mean a significant load. Therefore, it’s a common practice to instead create a watch for the dependent resources and cache their latest state. This is done following the Informer pattern. In Java Operator SDK, informers are wrapped into an EventSource, to integrate it with the eventing system of the framework. This is implemented by the InformerEventSource class.

A new event that triggers the reconciliation is only propagated to the Reconciler when the actual resource is already in cache. Reconciler implementations therefore only need to compare the desired state with the observed one provided by the cached resource. If the resource cannot be found in the cache, it therefore needs to be created. If the actual state doesn’t match the desired state, the resource needs to be updated.

Idempotency

Since all resources should be reconciled when your Reconciler is triggered and reconciliations can be triggered multiple times for any given resource, especially when retry policies are in place, it is especially important that Reconciler implementations be idempotent, meaning that the same observed state should result in exactly the same outcome. This also means that operators should generally operate in stateless fashion. Luckily, since operators are usually managing declarative resources, ensuring idempotency is usually not difficult.

Sync or Async Way of Resource Handling

Depending on your use case, it’s possible that your reconciliation logic needs to wait a non-insignificant amount of time while the operator waits for resources to reach their desired state. For example, you Reconciler might need to wait for a Pod to get ready before performing additional actions. This problem can be approached either synchronously or asynchronously.

The asynchronous way is to just exit the reconciliation logic as soon as the Reconciler determines that it cannot complete its full logic at this point in time. This frees resources to process other primary resource events. However, this requires that adequate event sources are put in place to monitor state changes of all the resources the operator waits for. When this is done properly, any state change will trigger the Reconciler again and it will get the opportunity to finish its processing

The synchronous way would be to periodically poll the resources’ state until they reach their desired state. If this is done in the context of the reconcile method of your Reconciler implementation, this would block the current thread for possibly a long time. It’s therefore usually recommended to use the asynchronous processing fashion.

Why have Automatic Retries?

Automatic retries are in place by default and can be configured to your needs. It is also possible to completely deactivate the feature, though we advise against it. The main reason configure automatic retries for your Reconciler is due to the fact that errors occur quite often due to the distributed nature of Kubernetes: transient network errors can be easily dealt with by automatic retries. Similarly, resources can be modified by different actors at the same time, so it’s not unheard of to get conflicts when working with Kubernetes resources. Such conflicts can usually be quite naturally resolved by reconciling the resource again. If it’s done automatically, the whole process can be completely transparent.

Managing State

Thanks to the declarative nature of Kubernetes resources, operators that deal only with Kubernetes resources can operate in a stateless fashion, i.e. they do not need to maintain information about the state of these resources, as it should be possible to completely rebuild the resource state from its representation (that’s what declarative means, after all). However, this usually doesn’t hold true anymore when dealing with external resources, and it might be necessary for the operator to keep track of this external state so that it is available when another reconciliation occurs. While such state could be put in the primary resource’s status sub-resource, this could become quickly difficult to manage if a lot of state needs to be tracked. It also goes against the best practice that a resource’s status should represent the actual resource state, when its spec represents the desired state. Putting state that doesn’t strictly represent the resource’s actual state is therefore discouraged. Instead, it’s advised to put such state into a separate resource meant for this purpose such as a Kubernetes Secret or ConfigMap or even a dedicated Custom Resource, which structure can be more easily validated.

Stopping (or not) Operator in case of Informer Errors and Cache Sync Timeouts

It can be configured if the operator should stop in case of any informer error happens on startup. By default, if there ia an error on startup and the informer for example has no permissions list the target resources (both the primary resource or secondary resources) the operator will stop instantly. This behavior can be altered by setting the mentioned flag to false, so operator will start even some informers are not started. In this case - same as in case when an informer is started at first but experienced problems later - will continuously retry the connection indefinitely with an exponential backoff. The operator will just stop if there is a fatal error, currently that is when a resource cannot be deserialized. The typical use case for changing this flag is when a list of namespaces is watched by a controller. In is better to start up the operator, so it can handle other namespaces while there might be a permission issue for some resources in another namespace.

The stopOnInformerErrorDuringStartup has implication on cache sync timeout behavior. If true operator will stop on cache sync timeout. if false, after the timeout the controller will start reconcile resources even if one or more event source caches did not sync yet.