Error on syncing BlobStore: Could not download the blob with key

[wilke-christopher]

I have a Storm (2.8.0) cluster with 3 Nimbus and 3 Supervisor servers and replication count 3.

When submitting a new topolgy I sometimes get the ERROR Could not download the blob with key: ....

The error occurs when the LocalFsBlobStoreSynchronizer determines the missing blob keys during synchronization. But the function BlobStoreUtils.downloadMissingBlob throws a KeyAlreadyExistsException, which is logged with INFO. In my opinion, isSucess should be set to true at this point and the loop exited. The actual error is logged after all other Nimbus servers have been iterated over and each iteration has ended unsuccessfully with a KeyAlreadyExistsException.

The KeyAlreadyExistsException is thrown because the corresponding blob file already exists in the file system.

I am not sure if either LocalFsBlobStoreSynchronizer.getBlobStoreKeySet() returns an inconsistent key set or the blob file has already been created in parallel.

I have attached the relevant Nimbus log.

I am grateful for any help.

blobstore_error.log

[reiabreu]

Hi @wilke-christopher !
Is the topology upload successful despite the error?

[wilke-christopher]

Hi @reiabreu,
yes, it is. The topology gets started.

[reiabreu]

Had a look and I see the same errors on the uploads on the system I use. And the uploads are still successful, as you pointed out.
We'll have a deeper look at this and update this ticket when we have more information.

[wilke-christopher]

Great, thanks 👍