CDAP Dataproc Operation Metrics Get More Specific

CDAP is an open source data integration platform, and this activity window is small but useful for operators who run pipelines on Google Cloud Dataproc. The main change is better response code accounting for Dataproc long running operations, not a new connector or a broad runtime rewrite. That matters because failed cluster create and delete paths often look similar until metrics carry enough context.

The important backend commit is the Dataproc response code change. It changes the Dataproc provisioner so operation outcomes can be counted with both a method and a status code. That is a small surface area change, but it hits the part of CDAP that operators watch when Spark or Hadoop clusters fail to appear.

The metric object in DataprocMetric.java now has optional method and statusCode fields. The builder also gained setters for both. This keeps the old metric API usable while allowing the provisioner to pass a response code that did not come from a thrown exception.

That distinction is practical. A Dataproc long running operation can finish with an operation error even when the status polling call itself succeeds. Without a way to pass that operation status into the metric layer, dashboards can count the poll as OK while the create operation failed.

The change in DataprocUtils.java moves status code selection into a helper. The order is clear: use an explicit status code from DataprocMetric, otherwise inspect an ApiException, otherwise use INTERNAL for generic exceptions, and use OK when there is no error.

That makes emitMetric more honest about operation results. It still preserves the existing tags such as region, status code, launch mode, and image version. It adds method only when present, which avoids changing every existing metric emission point.

For a pipeline platform, this is the right shape. Create, delete, status, and detail calls are not the same operational event. If they all collapse into the same status count, alerting ends up noisy. If CREATE errors and DELETE errors are tagged separately, the failure class is easier to route.

The provisioner logic in DataprocProvisioner.java now compares the previous cluster status with the current one. It emits the existing provisioner.clusterStatus.response.count metric after a status call. Then, only when the cluster reaches a terminal state, it tries to emit provisioner.operation.response.count.

The new terminal set is RUNNING, FAILED, and NOT_EXISTS. A previous status of CREATING maps to CREATE. A previous status of DELETING maps to DELETE. Other previous states do not produce an operation method, so the code does not invent a metric for every status transition.

This keeps polling from spamming operation metrics after a cluster has already settled. It also keeps operation metric failure from becoming a pipeline failure. If the lookup for operation metadata throws a retryable or runtime exception, the provisioner logs a warning and returns the cluster status it already found.

The client side addition in DataprocClient.java adds getLatestOperation. It builds a Dataproc operations filter using the cluster name and operation type, then returns the first operation in the page.

That is enough for the metric path, but it is also the weakest part of the patch. The method name says latest, while the implementation trusts the first value returned by the Dataproc operations page. If the API order is not stable, or if an old operation shares the same cluster name and type, the metric could reflect stale operation metadata.

The tests show the intended contract. DataprocClientTest.java covers a present operation, no operation, multiple operations, and an API exception. DataprocProvisionerTest.java covers create, delete, and error outcomes. One cleanup worth making later: two tests that expect no operation metric check a different metric name, so the assertion is weaker than it looks.

The July 13 submodule update changes only the cdap-ui pointer. That can still matter to console users, but the staged diff does not show REST API, schema, plugin, or runtime behavior in the main repo.

For data engineering readers, the Dataproc commit is the one to read. It changes observability around cluster lifecycle work, which sits directly on the pipeline execution path.

• Check whether dashboards already filter provisioner.clusterStatus.response.count. The new operation metric is provisioner.operation.response.count, so alert rules need a separate query.
• Watch cardinality on the method tag. Today it is only CREATE and DELETE, which is restrained.
• Validate Dataproc operation ordering before using latest semantics in incident reports. The code takes the first value returned by the operations page.