Dependent resources and workflows

• 1: Dependent resources
• 2: Workflows

1 - Dependent resources

Motivations and Goals

Most operators need to deal with secondary resources when trying to realize the desired state described by the primary resource they are in charge of. For example, the Kubernetes-native Deployment controller needs to manage ReplicaSet instances as part of a Deployment’s reconciliation process. In this instance, ReplicatSet is considered a secondary resource for the Deployment controller.

Controllers that deal with secondary resources typically need to perform the following steps, for each secondary resource:

flowchart TD

compute[Compute desired secondary resource based on primary state] --> A
A{Secondary resource exists?}
A -- Yes --> match
A -- No --> Create --> Done

match{Matches desired state?}
match -- Yes --> Done
match -- No --> Update --> Done

While these steps are not difficult in and of themselves, there are some subtleties that can lead to bugs or sub-optimal code if not done right. As this process is pretty much similar for each dependent resource, it makes sense for the SDK to offer some level of support to remove the boilerplate code associated with encoding these repetitive actions. It should be possible to handle common cases (such as dealing with Kubernetes-native secondary resources) in a semi-declarative way with only a minimal amount of code, JOSDK taking care of wiring everything accordingly.

Moreover, in order for your reconciler to get informed of events on these secondary resources, you need to configure and create event sources and maintain them. JOSDK already makes it rather easy to deal with these, but dependent resources makes it even simpler.

Finally, there are also opportunities for the SDK to transparently add features that are even trickier to get right, such as immediate caching of updated or created resources (so that your reconciler doesn’t need to wait for a cluster roundtrip to continue its work) and associated event filtering (so that something your reconciler just changed doesn’t re-trigger a reconciliation, for example).

Design

DependentResource vs. AbstractDependentResource

The new DependentResource interface lies at the core of the design and strives to encapsulate the logic that is required to reconcile the state of the associated secondary resource based on the state of the primary one. For most cases, this logic will follow the flow expressed above and JOSDK provides a very convenient implementation of this logic in the form of the AbstractDependentResource class. If your logic doesn’t fit this pattern, though, you can still provide your own reconcile method implementation. While the benefits of using dependent resources are less obvious in that case, this allows you to separate the logic necessary to deal with each secondary resource in its own class that can then be tested in isolation via unit tests. You can also use the declarative support with your own implementations as we shall see later on.

AbstractDependentResource is designed so that classes extending it specify which functionality they support by implementing trait interfaces. This design has been selected to express the fact that not all secondary resources are completely under the control of the primary reconciler: some dependent resources are only ever created or updated for example and we needed a way to let JOSDK know when that is the case. We therefore provide trait interfaces: Creator, Updater and Deleter to express that the DependentResource implementation will provide custom functionality to create, update and delete its associated secondary resources, respectively. If these traits are not implemented then parts of the logic described above is never triggered: if your implementation doesn’t implement Creator, for example, AbstractDependentResource will never try to create the associated secondary resource, even if it doesn’t exist. It is even possible to not implement any of these traits and therefore create read-only dependent resources that will trigger your reconciler whenever a user interacts with them but that are never modified by your reconciler itself - however note that read-only dependent resources rarely make sense, as it is usually simpler to register an event source for the target resource.

All subclasses of AbstractDependentResource can also implement the Matcher interface to customize how the SDK decides whether or not the actual state of the dependent matches the desired state. This makes it convenient to use these abstract base classes for your implementation, only customizing the matching logic. Note that in many cases, there is no need to customize that logic as the SDK already provides convenient default implementations in the form of DesiredEqualsMatcher and GenericKubernetesResourceMatcher implementations, respectively. If you want to provide custom logic, you only need your DependentResource implementation to implement the Matcher interface as below, which shows how to customize the default matching logic for Kubernetes resources to also consider annotations and labels, which are ignored by default:

public class MyDependentResource extends KubernetesDependentResource<MyDependent, MyPrimary>
        implements Matcher<MyDependent, MyPrimary> {
    // your implementation

    public Result<MyDependent> match(MyDependent actualResource, MyPrimary primary,
                                     Context<MyPrimary> context) {
        return GenericKubernetesResourceMatcher.match(this, actualResource, primary, context, true);
    }
}

Batteries included: convenient DependentResource implementations!

JOSDK also offers several other convenient implementations building on top of AbstractDependentResource that you can use as starting points for your own implementations.

One such implementation is the KubernetesDependentResource class that makes it really easy to work with Kubernetes-native resources. In this case, you usually only need to provide an implementation for the desired method to tell JOSDK what the desired state of your secondary resource should be based on the specified primary resource state.

JOSDK takes care of everything else using default implementations that you can override in case you need more precise control of what’s going on.

We also provide implementations that make it easy to cache (AbstractExternalDependentResource) or poll for changes in external resources (PollingDependentResource, PerResourcePollingDependentResource). All the provided implementations can be found in the io/javaoperatorsdk/operator/processing/dependent package of the operator-framework-core module.

Sample Kubernetes Dependent Resource

A typical use case, when a Kubernetes resource is fully managed - Created, Read, Updated and Deleted (or set to be garbage collected). The following example shows how to create a Deployment dependent resource:

@KubernetesDependent(labelSelector = WebPageManagedDependentsReconciler.SELECTOR)
class DeploymentDependentResource extends CRUDKubernetesDependentResource<Deployment, WebPage> {

    public DeploymentDependentResource() {
        super(Deployment.class);
    }

    @Override
    protected Deployment desired(WebPage webPage, Context<WebPage> context) {
        var deploymentName = deploymentName(webPage);
        Deployment deployment = loadYaml(Deployment.class, getClass(), "deployment.yaml");
        deployment.getMetadata().setName(deploymentName);
        deployment.getMetadata().setNamespace(webPage.getMetadata().getNamespace());
        deployment.getSpec().getSelector().getMatchLabels().put("app", deploymentName);

        deployment.getSpec().getTemplate().getMetadata().getLabels()
                .put("app", deploymentName);
        deployment.getSpec().getTemplate().getSpec().getVolumes().get(0)
                .setConfigMap(new ConfigMapVolumeSourceBuilder().withName(configMapName(webPage)).build());
        return deployment;
    }
}

The only thing that you need to do is to extend the CRUDKubernetesDependentResource and specify the desired state for your secondary resources based on the state of the primary one. In the example above, we’re handling the state of a Deployment secondary resource associated with a WebPage custom (primary) resource.

The @KubernetesDependent annotation can be used to further configure managed dependent resource that are extending KubernetesDependentResource.

See the full source code here .

Managed Dependent Resources

As mentioned previously, one goal of this implementation is to make it possible to declaratively create and wire dependent resources. You can annotate your reconciler with @Dependent annotations that specify which DependentResource implementation it depends upon. JOSDK will take the appropriate steps to wire everything together and call your DependentResource implementations reconcile method before your primary resource is reconciled. This makes sense in most use cases where the logic associated with the primary resource is usually limited to status handling based on the state of the secondary resources and the resources are not dependent on each other.

See Workflows for more details on how the dependent resources are reconciled.

This behavior and automated handling is referred to as “managed” because the DependentResource instances are managed by JOSDK, an example of which can be seen below:

@ControllerConfiguration(
        labelSelector = SELECTOR,
        dependents = {
                @Dependent(type = ConfigMapDependentResource.class),
                @Dependent(type = DeploymentDependentResource.class),
                @Dependent(type = ServiceDependentResource.class)
        })
public class WebPageManagedDependentsReconciler
        implements Reconciler<WebPage>, ErrorStatusHandler<WebPage> {

    // omitted code

    @Override
    public UpdateControl<WebPage> reconcile(WebPage webPage, Context<WebPage> context) {

        final var name = context.getSecondaryResource(ConfigMap.class).orElseThrow()
                .getMetadata().getName();
        webPage.setStatus(createStatus(name));
        return UpdateControl.patchStatus(webPage);
    }

}

See the full source code of sample here .

Standalone Dependent Resources

It is also possible to wire dependent resources programmatically. In practice this means that the developer is responsible for initializing and managing the dependent resources as well as calling their reconcile method. However, this makes it possible for developers to fully customize the reconciliation process. Standalone dependent resources should be used in cases when the managed use case does not fit. You can, of course, also use Workflows when managing resources programmatically.

You can see a commented example of how to do so here.

Creating/Updating Kubernetes Resources

From version 4.4 of the framework the resources are created and updated using Server Side Apply , thus the desired state is simply sent using this approach to update the actual resource.

Comparing desired and actual state (matching)

During the reconciliation of a dependent resource, the desired state is matched with the actual state from the caches. The dependent resource only gets updated on the server if the actual, observed state differs from the desired one. Comparing these two states is a complex problem when dealing with Kubernetes resources because a strict equality check is usually not what is wanted due to the fact that multiple fields might be automatically updated or added by the platform ( by dynamic admission controllers or validation webhooks, for example). Solving this problem in a generic way is therefore a tricky proposition.

JOSDK provides such a generic matching implementation which is used by default: SSABasedGenericKubernetesResourceMatcher This implementation relies on the managed fields used by the Server Side Apply feature to compare only the values of the fields that the controller manages. This ensures that only semantically relevant fields are compared. See javadoc for further details.

JOSDK versions prior to 4.4 were using a different matching algorithm as implemented in GenericKubernetesResourceMatcher.

Since SSA is a complex feature, JOSDK implements a feature flag allowing users to switch between these implementations. See in ConfigurationService.

It is, however, important to note that these implementations are default, generic implementations that the framework can provide expected behavior out of the box. In many situations, these will work just fine but it is also possible to provide matching algorithms optimized for specific use cases. This is easily done by simply overriding the match(...) method.

It is also possible to bypass the matching logic altogether to simply rely on the server-side apply mechanism if always sending potentially unchanged resources to the cluster is not an issue. JOSDK’s matching mechanism allows to spare some potentially useless calls to the Kubernetes API server. To bypass the matching feature completely, simply override the match method to always return false, thus telling JOSDK that the actual state never matches the desired one, making it always update the resources using SSA.

WARNING: Older versions of Kubernetes before 1.25 would create an additional resource version for every SSA update performed with certain resources - even though there were no actual changes in the stored resource - leading to infinite reconciliations. This behavior was seen with Secrets using stringData, Ingresses using empty string fields, and StatefulSets using volume claim templates. The operator framework has added built-in handling for the StatefulSet issue. If you encounter this issue on an older Kubernetes version, consider changing your desired state, turning off SSA for that resource, or even upgrading your Kubernetes version. If you encounter it on a newer Kubernetes version, please log an issue with the JOSDK and with upstream Kubernetes.

Telling JOSDK how to find which secondary resources are associated with a given primary resource

KubernetesDependentResource automatically maps secondary resource to a primary by owner reference. This behavior can be customized by implementing SecondaryToPrimaryMapper by the dependent resource.

See sample in one of the integration tests here .

Multiple Dependent Resources of Same Type

When dealing with multiple dependent resources of same type, the dependent resource implementation needs to know which specific resource should be targeted when reconciling a given dependent resource, since there could be multiple instances of that type which could possibly be used, each associated with the same primary resource. In this situation, JOSDK automatically selects the appropriate secondary resource matching the desired state associated with the primary resource. This makes sense because the desired state computation already needs to be able to discriminate among multiple related secondary resources to tell JOSDK how they should be reconciled.

There might be cases, though, where it might be problematic to call the desired method several times (for example, because it is costly to do so), it is always possible to override this automated discrimination using several means (consider in this priority order):

• Override the targetSecondaryResourceID method, if your DependentResource extends KubernetesDependentResource, where it’s very often possible to easily determine the ResourceID of the secondary resource. This would probably be the easiest solution if you’re working with Kubernetes resources.
• Override the selectTargetSecondaryResource method, if your DependentResource extends AbstractDependentResource. This should be relatively simple to override this method to optimize the matching to your needs. You can see an example of such an implementation in the ExternalWithStateDependentResource class.
• As last resort, you can implement your own getSecondaryResource method on your DependentResource implementation from scratch.

Sharing an Event Source Between Dependent Resources

Dependent resources usually also provide event sources. When dealing with multiple dependents of the same type, one needs to decide whether these dependent resources should track the same resources and therefore share a common event source, or, to the contrary, track completely separate resources, in which case using separate event sources is advised.

Dependents can therefore reuse existing, named event sources by referring to their name. In the declarative case, assuming a configMapSource EventSource has already been declared, this would look as follows:

 @Dependent(type = MultipleManagedDependentResourceConfigMap1.class, 
   useEventSourceWithName = "configMapSource")

A sample is provided as an integration test both: for managed

For standalone cases.

Bulk Dependent Resources

So far, all the cases we’ve considered were dealing with situations where the number of dependent resources needed to reconcile the state expressed by the primary resource is known when writing the code for the operator. There are, however, cases where the number of dependent resources to be created depends on information found in the primary resource.

These cases are covered by the “bulk” dependent resources feature. To create such dependent resources, your implementation should extend AbstractDependentResource (at least indirectly) and implement the BulkDependentResource interface.

Various examples are provided as integration tests .

To see how bulk dependent resources interact with workflow conditions, please refer to this integration test.

External State Tracking Dependent Resources

It is sometimes necessary for a controller to track external (i.e. non-Kubernetes) state to properly manage some dependent resources. For example, your controller might need to track the state of a REST API resource, which, after being created, would be refer to by its identifier. Such identifier would need to be tracked by your controller to properly retrieve the state of the associated resource and/or assess if such a resource exists. While there are several ways to support this use case, we recommend storing such information in a dedicated Kubernetes resources (usually a ConfigMap or a Secret), so that it can be manipulated with common Kubernetes mechanisms.

This particular use case is supported by the AbstractExternalDependentResource class that you can extend to suit your needs, as well as implement the DependentResourceWithExplicitState interface. Note that most of the JOSDK-provided dependent resource implementations such as PollingDependentResource or PerResourcePollingDependentResource already extends AbstractExternalDependentResource, thus supporting external state tracking out of the box.

See integration test as a sample.

For a better understanding it might be worth to study a sample implementation without dependent resources.

Please also refer to the docs for managing state in general.

Combining Bulk and External State Tracking Dependent Resources

Both bulk and external state tracking features can be combined. In that case, a separate, state-tracking resource will be created for each bulk dependent resource created. For example, if three bulk dependent resources associated with external state are created, three associated ConfigMaps (assuming ConfigMaps are used as a state-tracking resource) will also be created, one per dependent resource.

See integration test as a sample.

GenericKubernetesResource based Dependent Resources

In rare circumstances resource handling where there is no class representation or just typeless handling might be needed. Fabric8 Client provides GenericKubernetesResource to support that.

For dependent resource this is supported by GenericKubernetesDependentResource . See samples here.

Other Dependent Resource Features

Caching and Event Handling in KubernetesDependentResource

1. When a Kubernetes resource is created or updated the related informer (more precisely the InformerEventSource), eventually will receive an event and will cache the up-to-date resource. Typically, though, there might be a small time window when calling the getResource() of the dependent resource or getting the resource from the EventSource itself won’t return the just updated resource, in the case where the associated event hasn’t been received from the Kubernetes API. The KubernetesDependentResource implementation, however, addresses this issue, so you don’t have to worry about it by making sure that it or the related InformerEventSource always return the up-to-date resource.

2. Another feature of KubernetesDependentResource is to make sure that if a resource is created or updated during the reconciliation, this particular change, which normally would trigger the reconciliation again (since the resource has changed on the server), will, in fact, not trigger the reconciliation again since we already know the state is as expected. This is a small optimization. For example if during a reconciliation a ConfigMap is updated using dependent resources, this won’t trigger a new reconciliation. Such a reconciliation is indeed not needed since the change originated from our reconciler. For this system to work properly, though, it is required that changes are received only by one event source (this is a best practice in general) - so for example if there are two config map dependents, either there should be a shared event source between them, or a label selector on the event sources to select only the relevant events, see in related integration test .

“Read-only” Dependent Resources vs. Event Source

See Integration test for a read-only dependent here.

Some secondary resources only exist as input for the reconciliation process and are never updated by a controller (they might, and actually usually do, get updated by users interacting with the resources directly, however). This might be the case, for example, of a ConfigMapthat is used to configure common characteristics of multiple resources in one convenient place.

In such situations, one might wonder whether it makes sense to create a dependent resource in this case or simply use an EventSource so that the primary resource gets reconciled whenever a user changes the resource. Typical dependent resources provide a desired state that the reconciliation process attempts to match. In the case of so-called read-only dependents, though, there is no such desired state because the operator / controller will never update the resource itself, just react to external changes to it. An EventSource would achieve the same result.

Using a dependent resource for that purpose instead of a simple EventSource, however, provides several benefits:

• dependents can be created declaratively, while an event source would need to be manually created
• if dependents are already used in a controller, it makes sense to unify the handling of all secondary resources as dependents from a code organization perspective
• dependent resources can also interact with the workflow feature, thus allowing the read-only resource to participate in conditions, in particular to decide whether the primary resource needs/can be reconciled using reconcile pre-conditions, block the progression of the workflow altogether with ready post-conditions or have other dependents depend on them, in essence, read-only dependents can participate in workflows just as any other dependents.

2 - Workflows

Overview

Kubernetes (k8s) does not have the notion of a resource “depending on” on another k8s resource, at least not in terms of the order in which these resources should be reconciled. Kubernetes operators typically need to reconcile resources in order because these resources’ state often depends on the state of other resources or cannot be processed until these other resources reach a given state or some condition holds true for them. Dealing with such scenarios are therefore rather common for operators and the purpose of the workflow feature of the Java Operator SDK (JOSDK) is to simplify supporting such cases in a declarative way. Workflows build on top of the dependent resources feature. While dependent resources focus on how a given secondary resource should be reconciled, workflows focus on orchestrating how these dependent resources should be reconciled.

Workflows describe how as a set of dependent resources (DR) depend on one another, along with the conditions that need to hold true at certain stages of the reconciliation process.

Elements of Workflow

• Dependent resource (DR) - are the resources being managed in a given reconciliation logic.

• Depends-on relation - a B DR depends on another A DR if B needs to be reconciled after A.

• Reconcile precondition - is a condition on a given DR that needs to be become true before the DR is reconciled. This also allows to define optional resources that would, for example, only be created if a flag in a custom resource .spec has some specific value.

• Ready postcondition - is a condition on a given DR to prevent the workflow from proceeding until the condition checking whether the DR is ready holds true

• Delete postcondition - is a condition on a given DR to check if the reconciliation of dependents can proceed after the DR is supposed to have been deleted

• Activation condition - is a special condition meant to specify under which condition the DR is used in the workflow. A typical use-case for this feature is to only activate some dependents depending on the presence of optional resources / features on the target cluster. Without this activation condition, JOSDK would attempt to register an informer for these optional resources, which would cause an error in the case where the resource is missing. With this activation condition, you can now conditionally register informers depending on whether the condition holds or not. This is a very useful feature when your operator needs to handle different flavors of the platform (e.g. OpenShift vs plain Kubernetes) and/or change its behavior based on the availability of optional resources / features (e.g. CertManager, a specific Ingress controller, etc.).

  A generic activation condition is provided out of the box, called CRDPresentActivationCondition
  that will prevent the associated dependent resource from being activated if the Custom Resource Definition associated with the dependent’s resource type is not present on the cluster. See related integration test.

  To have multiple resources of same type with an activation condition is a bit tricky, since you don’t want to have multiple InformerEventSource for the same type, you have to explicitly name the informer for the Dependent Resource (@KubernetesDependent(informerConfig = @InformerConfig(name = "configMapInformer"))) for all resource of same type with activation condition. This will make sure that only one is registered. See details at low level api.

Result conditions

While simple conditions are usually enough, it might happen you want to convey extra information as a result of the evaluation of the conditions (e.g., to report error messages or because the result of the condition evaluation might be interesting for other purposes). In this situation, you should implement DetailedCondition instead of Condition and provide an implementation of the detailedIsMet method, which allows you to return a more detailed Result object via which you can provide extra information. The DetailedCondition.Result interface provides factory method for your convenience but you can also provide your own implementation if required.

You can access the results for conditions from the WorkflowResult instance that is returned whenever a workflow is evaluated. You can access that result from the ManagedWorkflowAndDependentResourceContext accessible from the reconciliation Context. You can then access individual condition results using the getDependentConditionResult methods. You can see an example of this in this integration test.

Defining Workflows

Similarly to dependent resources, there are two ways to define workflows, in managed and standalone manner.

Managed

Annotations can be used to declaratively define a workflow for a Reconciler. Similarly to how things are done for dependent resources, managed workflows execute before the reconcile method is called. The result of the reconciliation can be accessed via the Context object that is passed to the reconcile method.

The following sample shows a hypothetical use case to showcase all the elements: the primary TestCustomResource resource handled by our Reconciler defines two dependent resources, a Deployment and a ConfigMap. The ConfigMap depends on the Deployment so will be reconciled after it. Moreover, the Deployment dependent resource defines a ready post-condition, meaning that the ConfigMap will not be reconciled until the condition defined by the Deployment becomes true. Additionally, the ConfigMap dependent also defines a reconcile pre-condition, so it also won’t be reconciled until that condition becomes true. The ConfigMap also defines a delete post-condition, which means that the workflow implementation will only consider the ConfigMap deleted until that post-condition becomes true.

@Workflow(dependents = {
        @Dependent(name = DEPLOYMENT_NAME, type = DeploymentDependentResource.class,
                readyPostcondition = DeploymentReadyCondition.class),
        @Dependent(type = ConfigMapDependentResource.class,
                reconcilePrecondition = ConfigMapReconcileCondition.class,
                deletePostcondition = ConfigMapDeletePostCondition.class,
                activationCondition = ConfigMapActivationCondition.class,
                dependsOn = DEPLOYMENT_NAME)
})
@ControllerConfiguration
public class SampleWorkflowReconciler implements Reconciler<WorkflowAllFeatureCustomResource>,
    Cleaner<WorkflowAllFeatureCustomResource> {

  public static final String DEPLOYMENT_NAME = "deployment";

  @Override
  public UpdateControl<WorkflowAllFeatureCustomResource> reconcile(
      WorkflowAllFeatureCustomResource resource,
      Context<WorkflowAllFeatureCustomResource> context) {

    resource.getStatus()
        .setReady(
            context.managedWorkflowAndDependentResourceContext()  // accessing workflow reconciliation results
                .getWorkflowReconcileResult()
                .allDependentResourcesReady());
    return UpdateControl.patchStatus(resource);
  }

  @Override
  public DeleteControl cleanup(WorkflowAllFeatureCustomResource resource,
      Context<WorkflowAllFeatureCustomResource> context) {
    // emitted code

    return DeleteControl.defaultDelete();
  }
}

Standalone

In this mode workflow is built manually using standalone dependent resources . The workflow is created using a builder, that is explicitly called in the reconciler (from web page sample):

@ControllerConfiguration(
    labelSelector = WebPageDependentsWorkflowReconciler.DEPENDENT_RESOURCE_LABEL_SELECTOR)
public class WebPageDependentsWorkflowReconciler
    implements Reconciler<WebPage>, ErrorStatusHandler<WebPage> {

  public static final String DEPENDENT_RESOURCE_LABEL_SELECTOR = "!low-level";
  private static final Logger log =
      LoggerFactory.getLogger(WebPageDependentsWorkflowReconciler.class);

  private KubernetesDependentResource<ConfigMap, WebPage> configMapDR;
  private KubernetesDependentResource<Deployment, WebPage> deploymentDR;
  private KubernetesDependentResource<Service, WebPage> serviceDR;
  private KubernetesDependentResource<Ingress, WebPage> ingressDR;

  private final Workflow<WebPage> workflow;

  public WebPageDependentsWorkflowReconciler(KubernetesClient kubernetesClient) {
    initDependentResources(kubernetesClient);
    workflow = new WorkflowBuilder<WebPage>()
        .addDependentResource(configMapDR)
        .addDependentResource(deploymentDR)
        .addDependentResource(serviceDR)
        .addDependentResource(ingressDR).withReconcilePrecondition(new ExposedIngressCondition())
        .build();
  }

  @Override
  public Map<String, EventSource> prepareEventSources(EventSourceContext<WebPage> context) {
    return EventSourceUtils.nameEventSources(
        configMapDR.initEventSource(context),
        deploymentDR.initEventSource(context),
        serviceDR.initEventSource(context),
        ingressDR.initEventSource(context));
  }

  @Override
  public UpdateControl<WebPage> reconcile(WebPage webPage, Context<WebPage> context) {

    var result = workflow.reconcile(webPage, context);

    webPage.setStatus(createStatus(result));
    return UpdateControl.patchStatus(webPage);
  }
  // omitted code
}

Workflow Execution

This section describes how a workflow is executed in details, how the ordering is determined and how conditions and errors affect the behavior. The workflow execution is divided in two parts similarly to how Reconciler and Cleaner behavior are separated. Cleanup is executed if a resource is marked for deletion.

Common Principles

• As complete as possible execution - when a workflow is reconciled, it tries to reconcile as many resources as possible. Thus, if an error happens or a ready condition is not met for a resources, all the other independent resources will be still reconciled. This is the opposite to a fail-fast approach. The assumption is that eventually in this way the overall state will converge faster towards the desired state than would be the case if the reconciliation was aborted as soon as an error occurred.
• Concurrent reconciliation of independent resources - the resources which doesn’t depend on others are processed concurrently. The level of concurrency is customizable, could be set to one if required. By default, workflows use the executor service from ConfigurationService

Reconciliation

This section describes how a workflow is executed, considering first which rules apply, then demonstrated using examples:

Rules

1. A workflow is a Directed Acyclic Graph (DAG) build from the DRs and their associated depends-on relations.
2. Root nodes, i.e. nodes in the graph that do not depend on other nodes are reconciled first, in a parallel manner.
3. A DR is reconciled if it does not depend on any other DRs, or ALL the DRs it depends on are reconciled and ready. If a DR defines a reconcile pre-condition and/or an activation condition, then these condition must become true before the DR is reconciled.
4. A DR is considered ready if it got successfully reconciled and any ready post-condition it might define is true.
5. If a DR’s reconcile pre-condition is not met, this DR is deleted. All the DRs that depend on the dependent resource are also recursively deleted. This implies that DRs are deleted in reverse order compared the one in which they are reconciled. The reason for this behavior is (Will make a more detailed blog post about the design decision, much deeper than the reference documentation) The reasoning behind this behavior is as follows: a DR with a reconcile pre-condition is only reconciled if the condition holds true. This means that if the condition is false and the resource didn’t exist already, then the associated resource would not be created. To ensure idempotency (i.e. with the same input state, we should have the same output state), from this follows that if the condition doesn’t hold true anymore, the associated resource needs to be deleted because the resource shouldn’t exist/have been created.
6. If a DR’s activation condition is not met, it won’t be reconciled or deleted. If other DR’s depend on it, those will be recursively deleted in a way similar to reconcile pre-conditions. Event sources for a dependent resource with activation condition are registered/de-registered dynamically, thus during the reconciliation.
7. For a DR to be deleted by a workflow, it needs to implement the Deleter interface, in which case its delete method will be called, unless it also implements the GarbageCollected interface. If a DR doesn’t implement Deleter it is considered as automatically deleted. If a delete post-condition exists for this DR, it needs to become true for the workflow to consider the DR as successfully deleted.

Samples

Notation: The arrows depicts reconciliation ordering, thus following the reverse direction of the
depends-on relation: 1 --> 2 mean DR 2 depends-on DR 1.

Reconcile Sample

stateDiagram-v2
1 --> 2
1 --> 3
2 --> 4
3 --> 4

• Root nodes (i.e. nodes that don’t depend on any others) are reconciled first. In this example, DR 1 is reconciled first since it doesn’t depend on others. After that both DR 2 and 3 are reconciled concurrently, then DR 4 once both are reconciled successfully.
• If DR 2 had a ready condition and if it evaluated to as false, DR 4 would not be reconciled. However 1,2 and 3 would be.
• If 1 had a false ready condition, neither 2,3 or 4 would be reconciled.
• If 2’s reconciliation resulted in an error, 4 would not be reconciled, but 3 would be (and 1 as well, of course).

Sample with Reconcile Precondition

stateDiagram-v2
1 --> 2
1 --> 3
3 --> 4
3 --> 5

• If 3 has a reconcile pre-condition that is not met, 1 and 2 would be reconciled. However, DR 3,4,5 would be deleted: 4 and 5 would be deleted concurrently but 3 would only be deleted if 4 and 5 were deleted successfully (i.e. without error) and all existing delete post-conditions were met.
• If 5 had a delete post-condition that was false, 3 would not be deleted but 4 would still be because they don’t depend on one another.
• Similarly, if 5’s deletion resulted in an error, 3 would not be deleted but 4 would be.

Cleanup

Cleanup works identically as delete for resources in reconciliation in case reconcile pre-condition is not met, just for the whole workflow.

Rules

1. Delete is called on a DR if there is no DR that depends on it
2. If a DR has DRs that depend on it, it will only be deleted if all these DRs are successfully deleted without error and any delete post-condition is true.
3. A DR is “manually” deleted (i.e. it’s Deleter.delete method is called) if it implements the Deleter interface but does not implement GarbageCollected. If a DR does not implement Deleter interface, it is considered as deleted automatically.

Sample

stateDiagram-v2
1 --> 2
1 --> 3
2 --> 4
3 --> 4

• The DRs are deleted in the following order: 4 is deleted first, then 2 and 3 are deleted concurrently, and, only after both are successfully deleted, 1 is deleted.
• If 2 had a delete post-condition that was false, 1 would not be deleted. 4 and 3 would be deleted.
• If 2 was in error, DR 1 would not be deleted. DR 4 and 3 would be deleted.
• if 4 was in error, no other DR would be deleted.

Error Handling

As mentioned before if an error happens during a reconciliation, the reconciliation of other dependent resources will still happen, assuming they don’t depend on the one that failed. If case multiple DRs fail, the workflow would throw an ‘AggregatedOperatorException’ containing all the related exceptions.

The exceptions can be handled by ErrorStatusHandler

Waiting for the actual deletion of Kubernetes Dependent Resources

Let’s consider a case when a Kubernetes Dependent Resources (KDR) depends on another resource, on cleanup the resources will be deleted in reverse order, thus the KDR will be deleted first. However, the workflow implementation currently simply asks the Kubernetes API server to delete the resource. This is, however, an asynchronous process, meaning that the deletion might not occur immediately, in particular if the resource uses finalizers that block the deletion or if the deletion itself takes some time. From the SDK’s perspective, though, the deletion has been requested and it moves on to other tasks without waiting for the resource to be actually deleted from the server (which might never occur if it uses finalizers which are not removed). In situations like these, if your logic depends on resources being actually removed from the cluster before a cleanup workflow can proceed correctly, you need to block the workflow progression using a delete post-condition that checks that the resource is actually removed or that it, at least, doesn’t have any finalizers any longer. JOSDK provides such a delete post-condition implementation in the form of KubernetesResourceDeletedCondition

Also, check usage in an integration test.

In such cases the Kubernetes Dependent Resource should extend CRUDNoGCKubernetesDependentResource and NOT CRUDKubernetesDependentResource since otherwise the Kubernetes Garbage Collector would delete the resources. In other words if a Kubernetes Dependent Resource depends on another dependent resource, it should not implement GargageCollected interface, otherwise the deletion order won’t be guaranteed.

Explicit Managed Workflow Invocation

Managed workflows, i.e. ones that are declared via annotations and therefore completely managed by JOSDK, are reconciled before the primary resource. Each dependent resource that can be reconciled (according to the workflow configuration) will therefore be reconciled before the primary reconciler is called to reconcile the primary resource. There are, however, situations where it would be be useful to perform additional steps before the workflow is reconciled, for example to validate the current state, execute arbitrary logic or even skip reconciliation altogether. Explicit invocation of managed workflow was therefore introduced to solve these issues.

To use this feature, you need to set the explicitInvocation field to true on the @Workflow annotation and then call the reconcileManagedWorkflow method from the ManagedWorkflowAndDependentResourceContext retrieved from the reconciliation Context provided as part of your primary resource reconciler reconcile method arguments.

See related integration test for more details.

For cleanup, if the Cleaner interface is implemented, the cleanupManageWorkflow() needs to be called explicitly. However, if Cleaner interface is not implemented, it will be called implicitly. See related integration test.

While nothing prevents calling the workflow multiple times in a reconciler, it isn’t typical or even recommended to do so. Conversely, if explicit invocation is requested but reconcileManagedWorkflow is not called in the primary resource reconciler, the workflow won’t be reconciled at all.

Notes and Caveats

• Delete is almost always called on every resource during the cleanup. However, it might be the case that the resources were already deleted in a previous run, or not even created. This should not be a problem, since dependent resources usually cache the state of the resource, so are already aware that the resource does not exist and that nothing needs to be done if delete is called.
• If a resource has owner references, it will be automatically deleted by the Kubernetes garbage collector if the owner resource is marked for deletion. This might not be desirable, to make sure that delete is handled by the workflow don’t use garbage collected kubernetes dependent resource, use for example CRUDNoGCKubernetesDependentResource .
• No state is persisted regarding the workflow execution. Every reconciliation causes all the resources to be reconciled again, in other words the whole workflow is again evaluated.