This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Operations

This section covers operations-related features for running and managing operators in production.

See the operations sample operator for a complete working example that demonstrates health probes, metrics, and Helm-based deployment.

1 - Configurations

The Java Operator SDK (JOSDK) provides abstractions that work great out of the box. However, we recognize that default behavior isn’t always suitable for every use case. Numerous configuration options help you tailor the framework to your specific needs.

Configuration options operate at several levels:

  • Operator-level using ConfigurationService
  • Reconciler-level using ControllerConfiguration
  • DependentResource-level using the DependentResourceConfigurator interface
  • EventSource-level where some event sources (like InformerEventSource) need fine-tuning to identify which events trigger the associated reconciler

Operator-Level Configuration

Configuration that impacts the entire operator is performed via the ConfigurationService class. ConfigurationService is an abstract class with different implementations based on which framework flavor you use (e.g., Quarkus Operator SDK replaces the default implementation). Configurations initialize with sensible defaults but can be changed during initialization.

For example, to disable CRD validation on startup and configure leader election:

Operator operator = new Operator( override -> override
        .checkingCRDAndValidateLocalModel(false)
        .withLeaderElectionConfiguration(new LeaderElectionConfiguration("bar", "barNS")));

Virtual Threads

Reconciliation is mostly about blocking: talking to the Kubernetes API server or to external systems. Virtual threads make such blocking calls much cheaper than platform threads, and the framework can be switched over to them with a single flag:

Operator operator = new Operator(override -> override.withUseVirtualThreads(true));

When enabled, reconciliations, dependent resource workflows and the framework’s internal housekeeping (starting the informers, for example) all run on virtual threads.

Enabling virtual threads does not remove the concurrency limits, parallelism is configured exactly as before: withConcurrentReconciliationThreads(int) still caps how many reconciliations run at the same time and withConcurrentWorkflowExecutorThreads(int) how many dependent resources of a workflow are processed concurrently. Only the threads backing those limits change. Since virtual threads are cheap, these limits can usually be raised significantly compared to what is reasonable with platform threads.

Things to keep in mind:

  • Virtual threads are only supported on Java 25 or later. Before Java 25 a virtual thread pins its carrier thread while it is inside a synchronized block, which can starve the carrier pool and even deadlock the operator on startup. JEP 491, delivered in Java 25, removed that pinning. This is checked at runtime: on an older JVM, including the ones where virtual threads exist (Java 21 to 24), a warning is logged and platform threads are used instead, so the same configuration works on any Java version the operator runs on.
  • The scheduled tasks (the polls of the polling event sources, which each create an executor of their own unless one is provided, and the retried and rescheduled reconciliations) run on virtual threads too. The JDK has no scheduled executor backed by virtual threads, so the default ones keep a single platform thread for the timing only and run each task they fire on a virtual thread. The number of those tasks running at the same time is not limited, so withConcurrentRetryAndRescheduleThreads(int) only applies to platform threads.
  • Virtual threads are always daemon threads, so unlike the threads of the platform thread pools they don’t keep the JVM running: if the main method of your application returns after starting the operator and nothing else keeps the JVM alive, call Operator.keepAlive(), see Keeping the JVM Alive.
  • A custom ExecutorService provided through withExecutorService(...) or withWorkflowExecutorService(...) is always used as is, the flag has no effect on it. The same goes for withRetryAndRescheduleExecutorService(...). Note however that overriding the flag, or withConcurrentRetryAndRescheduleThreads(int), of a configuration that provides its own retry and reschedule executor replaces that executor with the default one, so provide it again with withRetryAndRescheduleExecutorService(...) in that case.
  • The Kubernetes client the framework creates when none is provided also switches its internal task executor (used to dispatch informer events to their handlers and to deliver watch events) to virtual threads. A client you provide through withKubernetesClient(...) is used as is: to get the same behavior, configure it yourself, e.g. with new KubernetesClientBuilder().withTaskExecutor(Executors.newVirtualThreadPerTaskExecutor()) (the client doesn’t shut down an executor passed that way, use withTaskExecutorSupplier(...) if it should be shut down when the client is closed). Either way, the blocking calls your reconciler makes through the client work well on virtual threads without any client-side change.

Reconciler-Level Configuration

While reconcilers are typically configured using the @ControllerConfiguration annotation, you can also override configuration at runtime when registering the reconciler with the operator. You can either:

  • Pass a completely new ControllerConfiguration instance
  • Override specific aspects using a ControllerConfigurationOverrider Consumer (preferred)
Operator operator;
Reconciler reconciler;
...
operator.register(reconciler, configOverrider ->
        configOverrider.withFinalizer("my-nifty-operator/finalizer").withLabelSelector("foo=bar"));

Dynamically Changing Target Namespaces

A controller can be configured to watch a specific set of namespaces in addition of the namespace in which it is currently deployed or the whole cluster. The initial set can be provided programmatically, via the @Informer annotation, or read from an external configuration source (see the namespaces property). The framework supports dynamically changing the list of these namespaces while the operator is running. When a reconciler is registered, an instance of RegisteredController is returned, providing access to the methods allowing users to change watched namespaces as the operator is running.

A typical scenario would probably involve extracting the list of target namespaces from a ConfigMap or some other input but this part is out of the scope of the framework since this is use-case specific. For example, reacting to changes to a ConfigMap would probably involve registering an associated Informer and then calling the changeNamespaces method on RegisteredController.


public static void main(String[] args) {
    KubernetesClient client = new DefaultKubernetesClient();
    Operator operator = new Operator(client);
    RegisteredController registeredController = operator.register(new WebPageReconciler(client));
    operator.installShutdownHook();
    operator.start();

    // call registeredController further while operator is running
}

If watched namespaces change for a controller, it might be desirable to propagate these changes to InformerEventSources associated with the controller. In order to express this, InformerEventSource implementations interested in following such changes need to be configured appropriately so that the followControllerNamespaceChanges method returns true:


@ControllerConfiguration
public class MyReconciler implements Reconciler<TestCustomResource> {

   @Override
   public Map<String, EventSource> prepareEventSources(
      EventSourceContext<ChangeNamespaceTestCustomResource> context) {

    InformerEventSource<ConfigMap, TestCustomResource> configMapES =
        new InformerEventSource<>(InformerEventSourceConfiguration.from(ConfigMap.class, TestCustomResource.class)
            .withNamespacesInheritedFromController(context)
            .build());

    return EventSourceUtils.nameEventSources(configMapES);
  }

}

As seen in the above code snippet, the informer will have the initial namespaces inherited from controller, but also will adjust the target namespaces if it changes for the controller.

See also the integration test for this feature.

Keeping the JVM Alive

The operator normally keeps the JVM running through the non-daemon threads of its thread pools. If nothing else does, for example because all the threads of the operator are daemon threads as it is the case with virtual threads, the JVM exits as soon as the main method returns after starting the operator. Call Operator.keepAlive() to have a non-daemon thread keep the JVM running until the operator is stopped:

public static void main(String[] args) {
    Operator operator = new Operator();
    operator.register(new MyReconciler());
    operator.installShutdownHook();
    operator.start();
    operator.keepAlive();
}

DependentResource-level configuration

It is possible to define custom annotations to configure custom DependentResource implementations. In order to provide such a configuration mechanism for your own DependentResource implementations, they must be annotated with the @Configured annotation. This annotation defines 3 fields that tie everything together:

  • by, which specifies which annotation class will be used to configure your dependents,
  • with, which specifies the class holding the configuration object for your dependents and
  • converter, which specifies the ConfigurationConverter implementation in charge of converting the annotation specified by the by field into objects of the class specified by the with field.

ConfigurationConverter instances implement a single configFrom method, which will receive, as expected, the annotation instance annotating the dependent resource instance to be configured, but it can also extract information from the DependentResourceSpec instance associated with the DependentResource class so that metadata from it can be used in the configuration, as well as the parent ControllerConfiguration, if needed. The role of ConfigurationConverter implementations is to extract the annotation information, augment it with metadata from the DependentResourceSpec and the configuration from the parent controller on which the dependent is defined, to finally create the configuration object that the DependentResource instances will use.

However, one last element is required to finish the configuration process: the target DependentResource class must implement the ConfiguredDependentResource interface, parameterized with the annotation class defined by the @Configured annotation by field. This interface is called by the framework to inject the configuration at the appropriate time and retrieve the configuration, if it’s available.

For example, KubernetesDependentResource, a core implementation that the framework provides, can be configured via the @KubernetesDependent annotation. This set up is configured as follows:


@Configured(
        by = KubernetesDependent.class,
        with = KubernetesDependentResourceConfig.class,
        converter = KubernetesDependentConverter.class)
public abstract class KubernetesDependentResource<R extends HasMetadata, P extends HasMetadata>
        extends AbstractEventSourceHolderDependentResource<R, P, InformerEventSource<R, P>>
        implements ConfiguredDependentResource<KubernetesDependentResourceConfig<R>> {
  // code omitted
}

The @Configured annotation specifies that KubernetesDependentResource instances can be configured by using the @KubernetesDependent annotation, which gets converted into a KubernetesDependentResourceConfig object by a KubernetesDependentConverter. That configuration object is then injected by the framework in the KubernetesDependentResource instance, after it’s been created, because the class implements the ConfiguredDependentResource interface, properly parameterized.

For more information on how to use this feature, we recommend looking at how this mechanism is implemented for KubernetesDependentResource in the core framework, SchemaDependentResource in the samples or CustomAnnotationDep in the BaseConfigurationServiceTest test class.

Loading Configuration from External Sources

JOSDK ships a ConfigLoader that bridges any key-value configuration source to the operator and controller configuration APIs. This lets you drive operator behaviour from environment variables, system properties, YAML files, or any config library (MicroProfile Config, SmallRye Config, Spring Environment, etc.) without writing glue code by hand.

Architecture

The system is built around two thin abstractions:

  • ConfigProvider — a single-method interface that resolves a typed value for a dot-separated key:

    public interface ConfigProvider {
        <T> Optional<T> getValue(String key, Class<T> type);
    }
    
  • ConfigLoader — reads all known JOSDK keys from a ConfigProvider and returns Consumer<ConfigurationServiceOverrider> / Consumer<ControllerConfigurationOverrider<R>> values that you pass directly to the Operator constructor or operator.register().

The default ConfigLoader (no-arg constructor) stacks environment variables over system properties: environment variables win, system properties are the fallback.

// uses env vars + system properties out of the box
Operator operator = new Operator(ConfigLoader.getDefault().applyConfigs());

Built-in Providers

ProviderSourceKey mapping
EnvVarConfigProviderSystem.getenv()dots and hyphens → underscores, upper-cased (josdk.check-crd → JOSDK_CHECK_CRD)
PropertiesConfigProviderjava.util.Properties or .properties filekey used as-is; use PropertiesConfigProvider.systemProperties() to read Java system properties
YamlConfigProviderYAML filedot-separated key traverses nested mappings
AggregatePriorityListConfigProviderordered list of providersfirst non-empty result wins

All string-based providers convert values to the target type automatically. Supported types: String, Boolean, Integer, Long, Double, Duration (ISO-8601, e.g. PT30S).

Plugging in Any Config Library

ConfigProvider is a single-method interface, so adapting any config library takes only a few lines. As an example, here is an adapter for SmallRye Config:

public class SmallRyeConfigProvider implements ConfigProvider {

    private final SmallRyeConfig config;

    public SmallRyeConfigProvider(SmallRyeConfig config) {
        this.config = config;
    }

    @Override
    public <T> Optional<T> getValue(String key, Class<T> type) {
        return config.getOptionalValue(key, type);
    }
}

The same pattern applies to MicroProfile Config, Spring Environment, Apache Commons Configuration, or any other library that can look up typed values by string key.

Wiring Everything Together

Pass the ConfigLoader results when constructing the operator and registering reconcilers:

// Load operator-wide config from a YAML file via SmallRye Config
URL configUrl = MyOperator.class.getResource("/application.yaml");
var configLoader = new ConfigLoader(
    new SmallRyeConfigProvider(
        new SmallRyeConfigBuilder()
            .withSources(new YamlConfigSource(configUrl))
            .build()));

// applyConfigs() → Consumer<ConfigurationServiceOverrider>
Operator operator = new Operator(configLoader.applyConfigs());

// applyControllerConfigs(name) → Consumer<ControllerConfigurationOverrider<R>>
operator.register(new MyReconciler(),
    configLoader.applyControllerConfigs(MyReconciler.NAME));

Only keys that are actually present in the source are applied; everything else retains its programmatic or annotation-based default.

You can also compose multiple sources with explicit priority using AggregatePriorityListConfigProvider:

var configLoader = new ConfigLoader(
    new AggregatePriorityListConfigProvider(List.of(
        new EnvVarConfigProvider(),          // highest priority
        PropertiesConfigProvider.systemProperties(),
        new YamlConfigProvider(Path.of("config/operator.yaml"))  // lowest priority
    )));

Operator-Level Configuration Keys

All operator-level keys are prefixed with josdk..

General

KeyTypeDescription
josdk.check-crdBooleanValidate CRDs against local model on startup
josdk.close-client-on-stopBooleanClose the Kubernetes client when the operator stops
josdk.use-virtual-threadsBooleanRun the framework’s concurrent work on virtual threads (only supported on Java 25+, platform threads are used on older versions)
josdk.use-ssa-to-patch-primary-resourceBooleanUse Server-Side Apply to patch the primary resource
josdk.clone-secondary-resources-when-getting-from-cacheBooleanClone secondary resources on cache reads

Reconciliation

KeyTypeDescription
josdk.reconciliation.concurrent-threadsIntegerThread pool size for reconciliation
josdk.reconciliation.termination-timeoutDurationHow long to wait for in-flight reconciliations to finish on shutdown

Workflow

KeyTypeDescription
josdk.workflow.executor-threadsIntegerThread pool size for workflow execution

Retry and Reschedule

KeyTypeDescription
josdk.retry-and-reschedule.concurrent-threadsIntegerThread pool size for triggering retried and rescheduled reconciliations

Informer

KeyTypeDescription
josdk.informer.cache-sync-timeoutDurationTimeout for the initial informer cache sync
josdk.informer.stop-on-error-during-startupBooleanStop the operator if an informer fails to start

Dependent Resources

KeyTypeDescription
josdk.dependent-resources.ssa-based-create-update-matchBooleanUse SSA-based matching for dependent resource create/update

Events

See Recording Kubernetes Events for how events are recorded. The key below configures where events about cluster scoped resources go.

KeyTypeDescription
josdk.events.cluster-scoped-namespaceStringNamespace to record events about cluster scoped resources in (defaults to default); the API server only accepts default and kube-system

Recording events requires the operator’s service account to be allowed get, create and patch on events in the core ("") API group. All three are used: an event that does not exist yet is created, and an event that does is a repeat of one already recorded, which is patched to count the new occurrence instead of being recorded a second time. The generic Helm chart grants all three. If you write your own RBAC and a permission is missing, recording fails silently as far as reconciliation is concerned by default: the failure is swallowed so that it cannot break a reconciliation, and only shows up as a warning in the operator log. Kubernetes treats events as best effort, supplemental data, so this is usually what you want. Where an event that is not recorded must not go unnoticed, configure a recorder that propagates the failure to the reconciler instead, so that the reconciliation fails and is retried like any other failed one:

KubernetesClient client = new KubernetesClientBuilder().build();
Operator operator = new Operator(override -> override
        .withKubernetesClient(client)
        .withEventRecorder(DefaultEventRecorder.builder(new DefaultEventSink(client))
            .failOnError(true)
            .build()));

Every failure propagates then, transient ones included, such as the conflict two replicas run into when they count the same occurrence of an event at the same time.

Note that events live in a namespace of their own choosing, which is the namespace of the object they are about, and for cluster scoped objects the namespace configured above. A namespaced Role therefore has to grant the permission in every namespace events are recorded in — including default, or whatever josdk.events.cluster-scoped-namespace is set to, if the operator reconciles cluster scoped resources. A ClusterRole covers all of them at once.

The API server only accepts events about cluster scoped objects in the default and kube-system namespaces, so those are the only values josdk.events.cluster-scoped-namespace can usefully be set to. In particular, the namespace the operator itself runs in is rejected, and the event dropped with a warning in the operator log (or, with failOnError(true), the reconciliation failed).

Leader Election

Leader election is activated when at least one josdk.leader-election.* key is present. josdk.leader-election.lease-name is required when any other leader-election key is set. Setting josdk.leader-election.enabled=false suppresses leader election even if other keys are present.

KeyTypeDescription
josdk.leader-election.enabledBooleanExplicitly enable (true) or disable (false) leader election
josdk.leader-election.lease-nameStringRequired. Name of the Kubernetes Lease object used for leader election
josdk.leader-election.lease-namespaceStringNamespace for the Lease object (defaults to the operator’s namespace)
josdk.leader-election.identityStringUnique identity for this instance; defaults to the pod name
josdk.leader-election.lease-durationDurationHow long a lease is valid (default PT15S)
josdk.leader-election.renew-deadlineDurationHow long the leader tries to renew before giving up (default PT10S)
josdk.leader-election.retry-periodDurationHow often a candidate polls while waiting to become leader (default PT2S)

Controller-Level Configuration Keys

All controller-level keys are prefixed with josdk.controller.<controller-name>., where <controller-name> is the value returned by the reconciler’s name (typically set via @ControllerConfiguration(name = "...")).

General

KeyTypeDescription
josdk.controller.<name>.finalizerStringFinalizer string added to managed resources
josdk.controller.<name>.generation-awareBooleanSkip reconciliation when the resource generation has not changed
josdk.controller.<name>.label-selectorStringLabel selector to filter watched resources
josdk.controller.<name>.shard-selectorStringShard selector to filter watched resources for sharding across operator replicas
josdk.controller.<name>.max-reconciliation-intervalDurationMaximum interval between reconciliations even without events
josdk.controller.<name>.field-managerStringField manager name used for SSA operations
josdk.controller.<name>.trigger-reconciler-on-all-eventsBooleanTrigger reconciliation on every event, not only meaningful changes
josdk.controller.<name>.default-filtersBooleanWhen false, JOSDK’s internal update filters (generation-aware, finalizer-needed, marked-for-deletion) are not applied and the user’s onUpdateFilter becomes the sole filter

Watched Namespaces

KeyTypeDescription
josdk.controller.<name>.namespacesStringComma-separated list of namespaces the controller watches

Entries are trimmed and blank ones are ignored, so ns1, ns2 and ns1,ns2 are equivalent. Instead of a list of namespaces, the value can also be one of the two special values below, which have to be used on their own — combining them with a namespace name is an error:

ValueMeaning
JOSDK_ALL_NAMESPACESWatch the whole cluster (the default)
JOSDK_WATCH_CURRENTWatch only the namespace the operator is deployed in

Setting this property is equivalent to calling settingNamespaces on ControllerConfigurationOverrider and therefore replaces, rather than extends, the namespaces configured via the @Informer annotation. The set of watched namespaces can still be changed while the operator is running, see Dynamically Changing Target Namespaces.

josdk.controller.mycontroller.namespaces=team-a,team-b

Informer

KeyTypeDescription
josdk.controller.<name>.informer.label-selectorStringLabel selector for the primary resource informer (alias for label-selector)
josdk.controller.<name>.informer.shard-selectorStringShard selector for the primary resource informer (alias for shard-selector)
josdk.controller.<name>.informer.list-limitLongPage size for paginated informer list requests; omit for no pagination
josdk.controller.<name>.informer.comparable-resource-versionsBooleanWhether the resource versions of the primary resource can be treated as integers and thus compared
josdk.controller.<name>.informer.without-namespace-indexBooleanRemoves the namespace index the informer maintains; defaults to false. See Removing the Namespace Index

Retry

If any retry.* key is present, a GenericRetry is configured starting from the default limited exponential retry. Only explicitly set keys override the defaults.

KeyTypeDescription
josdk.controller.<name>.retry.max-attemptsIntegerMaximum number of retry attempts
josdk.controller.<name>.retry.initial-intervalLong (ms)Initial backoff interval in milliseconds
josdk.controller.<name>.retry.interval-multiplierDoubleExponential backoff multiplier
josdk.controller.<name>.retry.max-intervalLong (ms)Maximum backoff interval in milliseconds

Rate Limiter

The rate limiter is only activated when rate-limiter.limit-for-period is present and has a positive value. rate-limiter.refresh-period is optional and falls back to the default of 10 s.

KeyTypeDescription
josdk.controller.<name>.rate-limiter.limit-for-periodIntegerMaximum number of reconciliations allowed per refresh period. Must be positive to activate the limiter
josdk.controller.<name>.rate-limiter.refresh-periodDurationWindow over which the limit is counted (default PT10S)

2 - Logging

Contextual Info for Logging with MDC

Logging is enhanced with additional contextual information using MDC. The following attributes are available in most parts of reconciliation logic and during the execution of the controller:

MDC KeyValue added from primary resource
resource.apiVersion.apiVersion
resource.kind.kind
resource.name.metadata.name
resource.namespace.metadata.namespace
resource.resourceVersion.metadata.resourceVersion
resource.generation.metadata.generation
resource.uid.metadata.uid

For more information about MDC see this link.

MDC entries during event handling

Although, usually users might not require it in their day-to-day workflow, it is worth mentioning that there are additional MDC entries managed for event handling. Typically, you might be interested in it in your SecondaryToPrimaryMapper related logs. For InformerEventSource and ControllerEventSource the following information is present:

MDC KeyValue from Resource from the Event
eventsource.event.resource.name.metadata.name
eventsource.event.resource.uid.metadata.uid
eventsource.event.resource.namespace.metadata.namespace
eventsource.event.resource.kindresource kind
eventsource.event.resource.resourceVersion.metadata.resourceVersion
eventsource.event.actionaction name (e.g. ADDED, UPDATED, DELETED)
eventsource.namename of the event source

Note on null values

If a resource doesn’t provide values for one of the specified keys, the key will be omitted and not added to the MDC context. There is, however, one notable exception: the resource’s namespace, where, instead of omitting the key, we emit the MDCUtils.NO_NAMESPACE value instead. This allows searching for resources without namespace (notably, clustered resources) in the logs more easily.

Disabling MDC support

MDC support is enabled by default. If you want to disable it, you can set the JAVA_OPERATOR_SDK_USE_MDC environment variable to false when you start your operator.

3 - Metrics

Metrics

JOSDK provides built-in support for metrics reporting on what is happening with your reconcilers in the form of the Metrics interface which can be implemented to connect to your metrics provider of choice, JOSDK calling the methods as it goes about reconciling resources. By default, a no-operation implementation is provided thus providing a no-cost sane default. A micrometer-based implementation is also provided.

You can use a different implementation by overriding the default one provided by the default ConfigurationService, as follows:

Metrics metrics; // initialize your metrics implementation
Operator operator = new Operator(client, o -> o.withMetrics(metrics));

MicrometerMetricsV2

MicrometerMetricsV2 is the recommended micrometer-based implementation. It is designed with low cardinality in mind: all meters are scoped to the controller, not to individual resources. This avoids unbounded cardinality growth as resources come and go.

The simplest way to create an instance:

MeterRegistry registry; // initialize your registry implementation
Metrics metrics = MicrometerMetricsV2.newBuilder(registry).build();

Optionally, include a namespace tag on per-reconciliation counters (disabled by default to avoid unexpected cardinality increases in existing deployments):

Metrics metrics = MicrometerMetricsV2.newBuilder(registry)
        .withNamespaceAsTag()
        .build();

You can also supply a custom timer configuration for reconciliations.execution.duration:

Metrics metrics = MicrometerMetricsV2.newBuilder(registry)
        .withExecutionTimerConfig(builder -> builder.publishPercentiles(0.5, 0.95, 0.99))
        .build();

MicrometerMetricsV2 metrics

All meters use controller.name as their primary tag. Counters optionally carry a namespace tag when withNamespaceAsTag() is enabled.

Meter name (Micrometer)TypeTagsDescription
reconciliations.activegaugecontroller.nameNumber of reconciler executions currently executing
reconciliations.queuegaugecontroller.nameNumber of resources currently queued for reconciliation
custom_resourcesgaugecontroller.nameNumber of custom resources tracked by the controller
reconciliations.execution.durationtimercontroller.nameReconciliation execution duration with explicit bucket histogram
reconciliations.started.totalcountercontroller.name, namespace*Number of reconciliations started (including retries)
reconciliations.success.totalcountercontroller.name, namespace*Number of successfully finished reconciliations
reconciliations.failure.totalcountercontroller.name, namespace*Number of failed reconciliations
reconciliations.retries.totalcountercontroller.name, namespace*Number of reconciliation retries
events.receivedcountercontroller.name, event, action, namespace*Number of events received by the controller

* namespace tag is only included when withNamespaceAsTag() is enabled.

The execution timer uses explicit boundaries (10ms, 50ms, 100ms, 250ms, 500ms, 1s, 2s, 5s, 10s, 30s) to ensure compatibility with histogram_quantile() queries in Prometheus. This is important when using the OpenTelemetry Protocol (OTLP) registry, where publishPercentileHistogram() would otherwise produce Base2 Exponential Histograms that are incompatible with classic _bucket queries.

Note on Prometheus metric names: The exact Prometheus metric name suffix depends on the MeterRegistry in use. For PrometheusMeterRegistry the timer is exposed as reconciliations_execution_duration_seconds_*. For OtlpMeterRegistry (metrics exported via OpenTelemetry Collector), it is exposed as reconciliations_execution_duration_milliseconds_*.

Grafana Dashboard

A ready-to-use Grafana dashboard is available at observability/josdk-operator-metrics-dashboard.json. It visualizes all of the metrics listed above, including reconciliation throughput, error rates, queue depth, active executions, resource counts, and execution duration histograms and heatmaps.

The dashboard is designed to work with metrics exported via OpenTelemetry Collector to Prometheus, as set up by the observability sample (see below).

Exploring metrics end-to-end

The operations sample operator includes a full end-to-end test, OperationsE2E, that:

  1. Installs a local observability stack (Prometheus, Grafana, OpenTelemetry Collector) via observability/install-observability.sh. That imports also the Grafana dashboards.
  2. Runs two reconcilers that produce both successful and failing reconciliations over a sustained period
  3. Verifies that the expected metrics appear in Prometheus

This is a good starting point for experimenting with the metrics and the Grafana dashboard in a real cluster without having to deploy your own operator.

Aggregated Metrics

The AggregatedMetrics class provides a way to combine multiple metrics providers into a single metrics instance using the composite pattern. This is particularly useful when you want to simultaneously collect metrics data from different monitoring systems or providers.

You can create an AggregatedMetrics instance by providing a list of existing metrics implementations:

// create individual metrics instances
Metrics micrometerMetrics = MicrometerMetrics.withoutPerResourceMetrics(registry);
Metrics customMetrics = new MyCustomMetrics();
Metrics loggingMetrics = new LoggingMetrics();

// combine them into a single aggregated instance
Metrics aggregatedMetrics = new AggregatedMetrics(List.of(
    micrometerMetrics,
    customMetrics,
    loggingMetrics
));

// use the aggregated metrics with your operator
Operator operator = new Operator(client, o -> o.withMetrics(aggregatedMetrics));

This approach allows you to easily combine different metrics collection strategies, such as sending metrics to both Prometheus (via Micrometer) and a custom logging system simultaneously.

MicrometerMetrics (Deprecated)

Deprecated: MicrometerMetrics (V1) is deprecated as of JOSDK 5.3.0. Use MicrometerMetricsV2 instead. V1 attaches resource-specific metadata (name, namespace, etc.) as tags to every meter, which causes unbounded cardinality growth and can lead to performance issues in your metrics backend.

The legacy MicrometerMetrics implementation is still available. To create an instance that behaves as it historically has:

MeterRegistry registry; // initialize your registry implementation
Metrics metrics = MicrometerMetrics.newMicrometerMetricsBuilder(registry).build();

To collect metrics on a per-resource basis, deleting the associated meters after 5 seconds when a resource is deleted, using up to 2 threads:

MicrometerMetrics.newPerResourceCollectingMicrometerMetricsBuilder(registry)
        .withCleanUpDelayInSeconds(5)
        .withCleaningThreadNumber(2)
        .build();

Operator SDK metrics (V1)

The V1 micrometer implementation records the following metrics:

Meter nameTypeTag namesDescription
operator.sdk.reconciliations.executions.<reconciler name>gaugegroup, version, kindNumber of executions of the named reconciler
operator.sdk.reconciliations.queue.size.<reconciler name>gaugegroup, version, kindHow many resources are queued to get reconciled by named reconciler
operator.sdk.<map name>.sizegauge map sizeGauge tracking the size of a specified map (currently unused but could be used to monitor caches size)
operator.sdk.events.receivedcounter<resource metadata>, event, actionNumber of received Kubernetes events
operator.sdk.events.deletecounter<resource metadata>Number of received Kubernetes delete events
operator.sdk.reconciliations.startedcounter<resource metadata>, reconciliations.retries.last, reconciliations.retries.numberNumber of started reconciliations per resource type
operator.sdk.reconciliations.failedcounter<resource metadata>, exceptionNumber of failed reconciliations per resource type
operator.sdk.reconciliations.successcounter<resource metadata>Number of successful reconciliations per resource type
operator.sdk.controllers.execution.reconciletimer<resource metadata>, controllerTime taken for reconciliations per controller
operator.sdk.controllers.execution.cleanuptimer<resource metadata>, controllerTime taken for cleanups per controller
operator.sdk.controllers.execution.reconcile.successcountercontroller, typeNumber of successful reconciliations per controller
operator.sdk.controllers.execution.reconcile.failurecountercontroller, exceptionNumber of failed reconciliations per controller
operator.sdk.controllers.execution.cleanup.successcountercontroller, typeNumber of successful cleanups per controller
operator.sdk.controllers.execution.cleanup.failurecountercontroller, exceptionNumber of failed cleanups per controller

All V1 metrics start with the operator.sdk prefix. <resource metadata> refers to resource-specific metadata and depends on the considered metric and how the implementation is configured: group?, version, kind, [name, namespace?], scope where tags in square brackets ([]) won’t be present when per-resource collection is disabled and tags followed by a question mark are omitted if the value is empty. In the context of controllers’ execution metrics, these tag names are prefixed with resource..

4 - Leader Election

When running multiple replicas of an operator for high availability, leader election ensures that only one instance actively reconciles resources at a time. JOSDK uses Kubernetes Lease objects for leader election.

Enabling Leader Election

Programmatic Configuration

var operator = new Operator(o -> o.withLeaderElectionConfiguration(
    new LeaderElectionConfiguration("my-operator-lease", "operator-namespace")));

Or using the builder for full control:

import static io.javaoperatorsdk.operator.api.config.LeaderElectionConfigurationBuilder.aLeaderElectionConfiguration;

var config = aLeaderElectionConfiguration("my-operator-lease")
    .withLeaseNamespace("operator-namespace")
    .withIdentity(System.getenv("POD_NAME"))
    .withLeaseDuration(Duration.ofSeconds(15))
    .withRenewDeadline(Duration.ofSeconds(10))
    .withRetryPeriod(Duration.ofSeconds(2))
    .build();

var operator = new Operator(o -> o.withLeaderElectionConfiguration(config));

External Configuration

Leader election can also be configured via properties (e.g. environment variables or a config file).

See details under configurations page.

How It Works

  1. When leader election is enabled, the operator starts but does not process events until it acquires the lease.
  2. Once leadership is acquired, event processing begins normally.
  3. If leadership is lost (e.g. the leader pod becomes unresponsive), another instance acquires the lease and takes over reconciliation. The instance that lost the lead is terminated (System.exit(1)), so that it is restarted by Kubernetes and no two instances reconcile the same resources in parallel. This does not happen on a graceful shutdown (Operator.stop()), only when the lead is lost while the operator is running.

Identity and Namespace Inference

If not explicitly set:

  • Identity is resolved from the HOSTNAME environment variable, then the pod name, falling back to a random UUID.
  • Lease namespace defaults to the namespace the operator pod is running in.

RBAC Requirements

The operator’s service account needs permissions to manage Lease objects:

- apiGroups: ["coordination.k8s.io"]
  resources: ["leases"]
  verbs: ["create", "update", "get"]

JOSDK checks for these permissions at startup and throws a clear error if they are missing.

Sample E2E Test

A complete working example is available in the leader-election sample operator, including multi-replica deployment manifests and an E2E test that verifies failover behavior.

5 - Health Probes

Operators running in Kubernetes should expose health probe endpoints so that the kubelet can detect startup failures and runtime degradation. JOSDK provides the building blocks through its RuntimeInfo API.

RuntimeInfo

RuntimeInfo is available via operator.getRuntimeInfo() and exposes:

MethodPurpose
isStarted()true once the operator and all its controllers have fully started
allEventSourcesAreHealthy()true when every registered event source (informers, polling sources, etc.) reports a healthy status
unhealthyEventSources()returns a map of controller name → unhealthy event sources, useful for diagnostics
unhealthyInformerWrappingEventSourceHealthIndicator()returns a map of controller name → unhealthy informer-wrapping event sources, each exposing per-informer details via InformerHealthIndicator (hasSynced(), isRunning(), getTargetNamespace())

In most cases a single readiness probe backed by allEventSourcesAreHealthy() is sufficient: before the operator has fully started the informers will not have synced yet, so the check naturally covers the startup case as well. Once running, it detects runtime degradation such as a lost watch connection.

See also: ConfigurationService.stopOnInformerErrorDuringStartup

Fine-Grained Informer Diagnostics

For advanced use cases — such as exposing per-informer health in a diagnostic endpoint or logging which specific namespace lost its watch — unhealthyInformerWrappingEventSourceHealthIndicator() gives access to individual InformerHealthIndicator instances. Each indicator exposes hasSynced(), isRunning(), and getTargetNamespace(). This is typically not needed for a standard health probe but can be valuable for operational dashboards or troubleshooting.

Detecting Stale Informers

An informer whose watch is open but delivers nothing (for example, when the API server replica it’s connected to serves reads from a stale watch cache) still reports hasSynced() and isRunning(), so its status stays healthy. InformerHealthIndicator.getLastSyncResourceVersion() exposes the resource version of the last event, bookmark or list the informer processed, and returns null before the first list. Since events and bookmarks keep moving this value, a probe can remember it per informer and report the informer unhealthy once it has not changed for longer than a timeout of your choosing:

operator.getRuntimeInfo().getRegisteredControllers().forEach(controller ->
    controller.getControllerHealthInfo().informerEventSourceHealthIndicators()
        .forEach((eventSourceName, eventSource) ->
            eventSource.informerHealthIndicators().forEach((namespace, informer) -> {
              String resourceVersion = informer.getLastSyncResourceVersion();
              // compare with the value seen by the previous check
            })));

The API server makes no guarantees about how often it sends bookmarks, so set a conservative timeout (several minutes) to avoid restarting an operator that is simply watching a quiet cluster.

Setting Up a Probe Endpoint

The example below uses Jetty to expose a /healthz endpoint. Any HTTP server library works — the key is calling the RuntimeInfo methods to determine the response code.

import org.eclipse.jetty.server.Server;
import org.eclipse.jetty.server.handler.ContextHandler;

Operator operator = new Operator();
operator.register(new MyReconciler());

// start the health server before the operator so probes can be queried during startup
var health = new ContextHandler(new HealthHandler(operator), "/healthz");
Server server = new Server(8080);
server.setHandler(health);
server.start();

operator.start();

Where HealthHandler extends org.eclipse.jetty.server.Handler.Abstract and checks operator.getRuntimeInfo().allEventSourcesAreHealthy().

See the operations sample operator for a complete working example.

Kubernetes Deployment Configuration

Once your operator exposes the probe endpoint, configure probes in your Deployment manifest. Both the startup and readiness probes can point to the same /healthz endpoint — the startup probe simply uses a higher failureThreshold to give the operator time to initialize:

containers:
- name: operator
  ports:
  - name: probes
    containerPort: 8080
  startupProbe:
    httpGet:
      path: /healthz
      port: probes
    initialDelaySeconds: 1
    periodSeconds: 3
    failureThreshold: 20
  readinessProbe:
    httpGet:
      path: /healthz
      port: probes
    initialDelaySeconds: 5
    periodSeconds: 5
    failureThreshold: 3

The startup probe gives the operator time to start (up to ~60 s with the settings above). Once the startup probe succeeds, the readiness probe takes over and will mark the pod as not-ready if any event source becomes unhealthy.

Helm Chart Support

The generic Helm chart supports health probes out of the box. Enable them in your values.yaml:

probes:
  port: 8080
  startup:
    enabled: true
    path: /healthz
  readiness:
    enabled: true
    path: /healthz

All probe timing parameters (initialDelaySeconds, periodSeconds, failureThreshold) have sensible defaults and can be overridden.

6 - Generic Helm Chart

A generic, reusable Helm chart for deploying Java operators built with JOSDK is available at helm/generic-helm-chart.

It is intended as a template for operator developers — a starting point that covers common deployment patterns so you don’t have to write a chart from scratch. The chart is maintained on a best-effort basis. Contributions are more than welcome.

The chart is used in the operations sample operator E2E test to deploy the operator to a cluster via Helm.

What the Chart Provides

  • Deployment with security defaults (non-root user, read-only filesystem, no privilege escalation)
  • Dynamic RBAC (ClusterRole, ClusterRoleBinding, ServiceAccount) — permissions are generated automatically from the primary and secondary resources you declare in values.yaml
  • ConfigMap for operator configuration (config.yaml) and logging (log4j2.xml), mounted at /config
  • Leader election support (opt-in)
  • Extensibility via extra containers, init containers, volumes, and environment variables

Key Configuration

The most important values to set when adapting the chart for your operator:

image:
  repository: my-operator-image   # required
  tag: "latest"

# Custom resources your operator reconciles
primaryResources:
  - apiGroup: "sample.javaoperatorsdk"
    resources:
      - myresources

# Kubernetes resources your operator manages
secondaryResources:
  - apiGroup: ""
    resources:
      - configmaps
      - services

Primary resources get read/watch/patch permissions and status sub-resource access. Secondary resources get full CRUD permissions. Default verbs can be overridden per resource entry.

The ClusterRole always grants get, create and patch on events in the core API group, so recording events works without any extra configuration. All three are needed: an event is looked up by name before it is created, so that recording the same event again resolves to the event already recorded, and that event is then patched to count the new occurrence rather than recorded again. See Events for what this means if you write your own RBAC.

Operator Environment

The chart injects OPERATOR_NAMESPACE automatically. You can optionally set WATCH_NAMESPACE to restrict the operator to a single namespace, and add arbitrary environment variables:

operator:
  watchNamespace: ""       # empty = all namespaces
  env:
    - name: MY_CUSTOM_VAR
      value: "some-value"

Resource Defaults

resources:
  limits:
    cpu: 500m
    memory: 512Mi
  requests:
    cpu: 100m
    memory: 128Mi

See the full values.yaml for all available options.

Usage Example

A working example of how to use the chart can be found in the operations sample operator’s helm-values.yaml:

image:
  repository: operations-operator
  pullPolicy: Never
  tag: "latest"

nameOverride: "operations-operator"

resources: {}

primaryResources:
  - apiGroup: "sample.javaoperatorsdk"
    resources:
      - metricshandlingcustomresource1s
      - metricshandlingcustomresource2s

Install with:

helm install my-operator ./helm/generic-helm-chart -f my-values.yaml --namespace my-ns

Testing the Chart

The chart includes unit tests using the helm-unittest plugin. Run them with:

./helm/run-tests.sh