Integrate CRT with Iceberg S3 client

[guizmaii]

Feature Request / Improvement

Re-opening the following issue: #6739

Query engine

Other

Willingness to contribute

• I can contribute this improvement/feature independently
• I would be willing to contribute this improvement/feature with guidance from the Iceberg community
• I cannot contribute this improvement/feature at this time

[JeonDaehong]

Hello !

Thanks for the suggestion regarding the performance improvements — it sounds like a very interesting enhancement!

If possible, I’d be happy to take a look and try implementing this.

Before getting started, could you please clarify what specific changes would be needed in Iceberg to support the CRT-based S3 client?

For example:

• Are you suggesting changes to how FileIO or S3FileIO is configured?
• Are there any dependency or compatibility concerns we should be aware of?

It would be great to understand the scope of changes more clearly before moving forward.

[guizmaii]

Hi @JeonDaehong,

Thanks for showing interest in this improvement

I'm not familiar with the specifics of what needs to change. I don't know the internals of Iceberg, nor I'm an expert of Iceberg

What I've seen is that if I globally exclude the following dependency from my project: software.amazon.awssdk::apache-client, then the following code fails to execute:

iceberg/aws/src/main/java/org/apache/iceberg/aws/HttpClientProperties.java

Lines 235 to 241 in 2cdff36

	/**
	* Dynamically load the http client builder to avoid runtime deps requirements of both {@link
	* software.amazon.awssdk.http.urlconnection.UrlConnectionHttpClient} and {@link
	* software.amazon.awssdk.http.apache.ApacheHttpClient}, since including both will cause error
	* described in <a href="https://github.com/apache/iceberg/issues/6715">issue#6715</a>
	*/
	private <T> T loadHttpClientConfigurations(String impl) {

AFAICT, it seems like Iceberg only supports the UrlConnectionHttpClient and ApacheHttpClient implementations of the AWS SDK HTTP client. It'd probably be interesting in supporting Netty and AWS CRT

Hope it helps 🙂

[JeonDaehong]

@guizmaii

Thank you for the valuable insight!

Based on my brief investigation so far, it appears that, as you mentioned, Iceberg currently supports only ApacheHttpClient and UrlConnectionHttpClient through the HttpClientProperties class. Other AWS SDK HTTP client implementations like Netty or CrtHttpClient do not seem to be supported at this time.

I'll look into whether there's potential for Iceberg to be enhanced to allow more flexible configuration of SdkHttpClient.

[JeonDaehong]

If there's any contributor who is more familiar with this issue, I’d really appreciate any guidance or insights you could share.

I'll continue to look into it as time permits.:D

[SanjayMarreddi]

Hi @guizmaii @JeonDaehong
Just saw this issue and wanted to let you know that we are adding support for the Netty and Crt Async clients as part of this PR!