SharedIndexInformer endlessly attempts to reconnect to the down Kube API server

[19Serhii99]

Describe the bug

I am testing behavior of the Fabric8 SharedIndexInformer during having network instability. I am trying to determine whether it is going to fail or recover when a connection is lost so that I could handle this scenario as neeeded. I ran kubectl proxy to control connectivity, so my master URL is http://127.0.0.1:8001.

try (SharedIndexInformer informer = client.pods()
  .inNamespace("$NAMESPACE")
  .withName("$POD_NAME")
  .runnableInformer(0L)) {
    informer.addEventHandler(...);
    informer.exceptionHandler(...);
}

When running the program, I terminate the proxy to break the connection, and then run kubectl proxy command again. Instead of invoking retryOnException we passed to exceptionHandler(....), it's endlessly attempting to reconnect, throwing lots of red logs (see below).

Of course, if you kubectl poxy again, it will reconnect and continue monitoring the pod state but if the server is down for long, then I would like it to fail sooner or later (retryAfterException would control it)

Fabric8 Kubernetes Client version

7.3.1

Steps to reproduce

1. Run kubectl proxy
2. Initialize and start a new SharedIndexInformer
3. Wait until it connects and receives first pod events
4. Terminate the proxy (Ctrl + C in the terminal)
5. Observe the logs - it's endlessly attempting to reconnect

Expected behavior

ExceptionHandler.retryAfterException is invoked

Runtime

Kubernetes (vanilla)

Kubernetes API Server version

other (please specify in additional context)

Environment

Windows, Azure

Fabric8 Kubernetes Client Logs

[vert.x-eventloop-thread-1] ERROR io.vertx.core.net.impl.ConnectionBase - Connection reset
[vert.x-eventloop-thread-3] INFO io.fabric8.kubernetes.client.dsl.internal.AbstractWatchManager - Unknown Watch error received 152 times without progress, will reconnect if possible
java.util.concurrent.CompletionException:
io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: no further information: /127.0.0.1:8001 at java.base/java.util.concurrent.CompletableFuture.encode Throwable (CompletableFuture.java:332)
at java.base/java.util.concurrent.CompletableFuture.complete Throwable (CompletableFuture.java:347)
at java.base/java.util.concurrent.Completable Future$UniCompose.tryFire (CompletableFuture.java:1141)
at java.base/java.util.concurrent.CompletableFuture.postComplete (CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally (CompletableFuture.java:2162)
at io.fabric8.kubernetes.client.vertx. VertxHttpClient.lambda$buildWebSocketDirect$3 (VertxHttpClient.java:103)
at io.vertx.core.impl.future. Future Impl$2.onFailure (FutureImpl.java:117)
at io.vertx.core.impl.future. Future Impl$ListenerArray.onFailure (FutureImpl.java:316)

Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: no further information: /127.0.0.1:8001 Caused by: java.net.ConnectException: Connection refused: no further information
at java.base/sun.nio.ch.Net.pollConnect (Native Method)
at java.base/sun.nio.ch.Net.pollConnectNow (Net.java:672)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect (SocketChannelImpl.java:946)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect (NioSocketChannel.java:336)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect (AbstractNioChannel.java:339)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:784)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:732)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:658)

Additional context

SharedIndexInformer is tested on Windows 11. K8S cluster is on Azure Kubernetes Service. Azure Kubernetes Service is on version 1.31.8

[shawkins]

This is currently the expected behavior - network level errors are not handled by the exceptionHandler. That is only for unexpected exceptions with running the Reflector. This would be an enhancement, and possibly require some other api changes, such as adding ExceptionHandler.isNetworkException.

[19Serhii99]

@shawkins If I skip creating a proxy and start an informer, it will fail to connect to the server as well but it will do invoke retryAfterExcetion which is not done for interruptions in the middle of the watching process...

[19Serhii99]

If this is expected behavior, are there any suggestions how to handle it on our side? We want to terminate the watching process if can't connect in X seconds. Some automation process is required

[shawkins]

If I skip creating a proxy and start an informer, it will fail to connect to the server as well but it will do invoke retryAfterExcetion which is not done for interruptions in the middle of the watching process...

That is correct. This is due to the special case of informer start-up - we don't consider the informer established until a watch attempt succeeds, so all exceptions prior to this point are sent to the ExceptionHandler to allow for the default handling

kubernetes-client/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/informers/impl/cache/Reflector.java

Line 60 in ed36981

private volatile ExceptionHandler handler = (b, t) -> b && !(t instanceof WatcherException);

- which is needed the assumed behavior of the blocking api. That is any exception, including a network problem, will cause the Informable.inform call to fail fast.

I can understand why you would want all exceptions to be handled by ExceptionHandler, even after it's successfully started, but generally once an informers up - it can just be left running.

If this is expected behavior, are there any suggestions how to handle it on our side? We want to terminate the watching process if can't connect in X seconds. Some automation process is required

There are status flags on the informer. In this case isWatching will return false. isWatching=false may happen ephemerally, so you'll want to see several false observations in a row before terminating the process based upon that.

[19Serhii99]

I terminated the proxy, and while it was endlessly printing stack traces in the terminal, I quickly checked the informer using the debugger and confirmed that both isRunning and isWatching are returning true. I do not see any other flags that might help identify connection loss.

[shawkins]

I do not see any other flags that might help identify connection loss.

Is running is expected to be true until the informer is stopped.

isWatching is not quite behaving as I remember it should. The watch reconnect is being handled by the AbstractWatchManager, not the Reflector, so from the Reflector's perspective the watch is still active. Only when there is a disruption over the watch protocol - http gone, unexpected message, etc. does the Reflector handling kick in. So unfortunately this is basically the same situation as you logged initially.

Let me check to see if we can relay the watch outages to the Reflector.