High CPU usage for NiFi Cluster due to operator controllers calls

[ggerla]

What steps will reproduce the bug?

I'm using nifikop to deploy a nifi cluster with 4 nifi nodes and 3 zookeper nodes.
I discovered that the nifi cluster is consuming between 7x to 10x times the cpu.
Looking to logs and statistics I found that the operator is doing a lot of calls to NiFi APIs and this generates the CPU consumption.
The main issue is that all these API calls are done during the reconcile loop with a frequency that is quite aggressive.

Killing the operator the CPU usage is collapsing

What is the expected behavior?

A Nifi Cluster with 4 nodes in idle usually consumes around 0,5 cpus

What do you see instead?

The same cluster with the operator consumes between 3 CPU and 5 CPU

Possible solution

Try to distinguish between a reconcile rerun during a the setup phase and the reconceile run interval during "idle" state, so that you can relax the frequancy of API calls during in idle state

NiFiKop version

v1.10.0

Golang version

1.19

Kubernetes version

v1.26.0

NiFi version

1.27.0

Additional context

No response

[juldrixx]

The frequencies of the reconcile loops are configurable through environment variables (https://github.com/konpyutaika/nifikop/blob/master/pkg/common/common.go#L75-L92) so you can increase them if you need.
But sure, the operator is making multiple calls for every resources to check that its definition match the one in NiFi, so logically the number of call is proportional to your number of resources.
And it can also be caused by a call that is lopping because it's unsuccessful. For example, a parameter is defined 2 times in the same NifiParameterContext, the operator will keep updating NiFi and generate a lot of call.
And it could be interesting to check which calls are made to see if it's "normal" call.

[ggerla]

thanks for the answer @juldrixx
I discovered that env vars looking into the code, so today I did some experiments.
In one cluster I tryed this configuration (trying to manage resources that I think I need)

• name: CLUSTER_TASK_RUNNING_REQUEUE_INTERVAL
  value: "60"
  • name: CLUSTER_TASK_TIMEOUT_REQUEUE_INTERVAL
    value: "120"
  • name: CLUSTER_TASK_NOT_READY_REQUEUE_INTERVAL
    value: "20"
  • name: CLUSTER_TASK_NO_NODE_INTERVAL
    value: "120"
  • name: USERS_REQUEUE_INTERVAL
    value: "60"
  • name: NODE_GROUP_AUTOSCALER_REQUEUE_INTERVAL
    value: "60"
  • name: REGISTRY_CLIENT_REQUEUE_INTERVAL
    value: "120"
  • name: PARAMETER_CONTEXT_REQUEUE_INTERVAL
    value: "120"
  • name: USER_GROUP_REQUEUE_INTERVAL
    value: "60"
  • name: DATAFLOW_REQUEUE_INTERVAL
    value: "120"
  • name: CONNECTION_REQUEUE_INTERVAL
    value: "120"

and this is the result

on another cluster I tryed this configuration (trying to relax everything)

• name: CLUSTER_TASK_RUNNING_REQUEUE_INTERVAL
  value: "120"
  • name: CLUSTER_TASK_TIMEOUT_REQUEUE_INTERVAL
    value: "120"
  • name: CLUSTER_TASK_NOT_READY_REQUEUE_INTERVAL
    value: "120"
  • name: CLUSTER_TASK_NO_NODE_INTERVAL
    value: "120"
  • name: USERS_REQUEUE_INTERVAL
    value: "120"
  • name: NODE_GROUP_AUTOSCALER_REQUEUE_INTERVAL
    value: "120"
  • name: REGISTRY_CLIENT_REQUEUE_INTERVAL
    value: "120"
  • name: PARAMETER_CONTEXT_REQUEUE_INTERVAL
    value: "120"
  • name: USER_GROUP_REQUEUE_INTERVAL
    value: "120"
  • name: DATAFLOW_REQUEUE_INTERVAL
    value: "120"
  • name: CONNECTION_REQUEUE_INTERVAL
    value: "120"

and this is the result

In both cases the cpu usage is higher than the one without the operator. In the second case I think it is acceptable (not in the first one). But what I'm "loosing" relaxing so much the reconcile period?

I'm using the operator to deploy the cluster, no other k8s resources are managed by the operator.

[wielkiborsuk]

I think it's impossible to set these variables via the nifikop helm chart - it would be nice to be able to set them in helm chart values.yml.

For the most part, simply having these config options should be enough. If it's only the cluster being controlled, having a reconciliation delay of 5min or even 15min should be acceptable.

[ggerla]

I think it's impossible to set these variables via the nifikop helm chart - it would be nice to be able to set them in helm chart values.yml.

yes you are right.

For the most part, simply having these config options should be enough. If it's only the cluster being controlled, having a reconciliation delay of 5min or even 15min should be acceptable.

I'm not sure about this. If you deploy a workflow resource into k8s, do you want to see it in NiFi soon or after 5 minutes? Same for the user.
I think the best solution can be to wait a notification from k8s for resource change and reconcile when notified.

[juldrixx]

This PR #543 was done to make it so the variables are settable

[wielkiborsuk]

That solves it for me :) Thank you
You might want to consider simply allowing additional env variables though. Instead of handing every single value separately in the template.

[juldrixx]

Made this also possible with #544 😄 and remove the other modifications with #545 (forgot to do it with the previous one)