Question about your architecture

[cesaraugusto98]

Hi @zsvoboda Why do you use DBT + Spark between bronze layer to silver layer; and only uses dbt+trino moving from Silver to Gold? Is there any specific reason?

[zsvoboda]

Hi @cesaraugusto98, Yes, both bronze and silver schemas are managed by Spark (one logical db). The gold schema is managed by Postgres. You need to run CTAS queries that read data from Iceberg and store data in Postgres when moving data from silver to gold. Trino is much better for such federated queries.

[mtthsbrr]

Hi @zsvoboda. How about using only Trino and a data lake framework (Delta, Iceberg) for everything (bronze, silver, gold). Or am I missing something that only Spark would be capable of?

[zsvoboda]

This is certainly an option. However, Spark is more resilient for larger datasets (10s of billions of rows). I was running into memory issues with Trino myself.

[claytonjr]

This has been my issue as well. For trino/presto - it was a memory hog compared to spark. Also, remember that spark is based on mappers/reducers - larger data sets are basically the reason for it's existence. Kind regards, James Clayton

…

On 2/20/2023 3:40 AM, Zdenek Svoboda wrote: This is certainly an option. However, Spark is more resilient for larger datasets (10s of billions of rows). I was running into memory issues with Trino myself. — Reply to this email directly, view it on GitHub <#17 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABEEW5OY63W33FJI4QNICKDWYMUWVANCNFSM6AAAAAAU77FVXY>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[tnhgiang]

Hi @zsvoboda. I also have the same question as @mtthsbrr. Why we need postgres to manage the gold layer? Why don't we use Iceberg for all layers (bronze, silver and gold)?