Why not use DataFusion as the backend engine (rather than rewriting it all from scratch)?

[alamb]

Is your feature request related to a problem?

I read this blog. Very cool (and thanks @MrPowers) for pointing it out

https://dataengineeringcentral.substack.com/p/introduction-to-daft-vs-polars

The experience of reading from S3 and handling globs, etc looks pretty amazing with Daft 💯 nice work

I have a question: If your focus is on an amazing experience working with objects on remote object store, why build your own Rust / Arrow based vectorized query engine (yet again) as well when there are already exiting engines?

I can't help but notice there is a substantial amount of similarity between the current structure of your code and DataFusion (Arrow, LogicalPlan, PhysicalPlan, etc) -- likely as a consequence of it being fairly well understood how to build columnar query engines. It is

I also saw several issues that have started enumerating features that are missing in Daft that are already in DataFusion (along with tests, etc)

• Optimizer: simplify expression optimizer rule #2422
• Optimizer: eliminate nested union/concat #2425

Describe the solution you'd like

I propose (very selfishly, of course, as a PMC member) that you build on Apache DataFusion and better yet help us extend it to make it even better

You would likely get a full featured, fast engine, and instead of reinventing all the standard, low level operation for processing you could instead focus on making the DataFrame experience processing files remotely on object store amazing.

I can tell you from experience building and maturing a execution engine takes a lot of effort and it is great to do it with a big and established community.

Describe alternatives you've considered

I would also love to know if you have thought about DataFusion and chose a different approach anyways and any thoughts you were willing to share about why you made the choice (so we can see if we can make it easier for future startups to choose to build with DataFusion)

Additional Context

Happy hacking!

Would you like to implement a fix?

No

[maruschin]

Yeah, looks like a reinvention of datafusion

[MrPowers]

Speaking from experience, the DataFusion and Daft communities are both awesome to work with!

We have a weekly DataFusion call. Perhaps the Daft devs could join one week to brainstorm!

[samster25]

Hey @alamb! Really exciting to hear from you :) thanks for making this thread. Your kind words mean a lot for us and the team.

We’re huge fans of the Datafusion project and all the extremely exciting and inspiring work that’s been going on there. A lot of our work has been very obviously inspired by work in Datafusion.

 any thoughts you were willing to share about why you made the choice

When we first started transitioning Daft to Rust about 2 years ago the ecosystem was a lot more fragmented and it wasn’t yet clear what we should adopt At the time, Datafusion and arrow-rs were very young projects and didn’t support some of the requirements that we had.

The key things here were:

• Support for our own type system for our own logical types like images and tensors as well as out of band types like Python objects.
• We originally were executing kernels in a non-async environment
• Strict requirements on being able to serialize physical plans for a distributed environment.
• We wanted a transmute free library for our in-memory representation. (I was burned by arrow-cpp lol)

We ultimately decided on using arrow2 for the in-memory representation and kernels. But all the key query engine components were still in Python. As time went on, we moved more and more from Python land into Rust. Such as our DSL, Logical Plan and Physical Plan.

 (so we can see if we can make it easier for future startups to choose to build with DataFusion)

Today that is less clear. Datafusion has an amazing community and has made big leaps within the last 2 years. However we’ve also invested heavily in a lot of our own stack which makes a migration both costly and difficult to justify. If we were to start a new project today, I think our vote would have been unequivocally to start off with Datafusion. Datafusion makes it incredibly easy to get a project off the ground.

In reality however, we’re a small team with big distributed ambitions and unfortunately Datafusion integration is more of a nice-to-have right now rather than a core accelerant.

That being said, there are a few items on our roadmap which we are actively exploring, which should bring us much closer to the rest of the community.

1. Migration from arrow2 to arrow-rs arrow2 has been pretty much abandoned by the community. We would love to move Daft to arrow-rs and contribute back to it.
2. Leverage Datafusion's Logical Plan and Optimizer as you mentioned Datafusion already has a ton of optimizer rules and is quite robust. This not really an area we feel the need to innovate on. The key requirement there for us is being able to plug in our own type system, logical types and expressions.
3. Data Catalog and Table Format integrations - we’re very excited about this and have been on the forefront of integrating with a lot of these technologies (outside of the JVM). Our current integrations are in Python, but we do want to transition to more native Rust integrations and are in talks with the relevant Rust projects here.
4. Pluggable engines: we are just about to finish up with our new local execution engine, and at some point will be adding other engines such as a newly designed distributed engine, or even a proprietary one. Plugging in Datafusion here as a POC could be interesting, and we could even try that on a distributed runner.

I think (4) is potentially the most interesting story, but will obviously require a lot of work and investment from both Daft and Datafusion to make happen. We’ve spent a lot of time, effort, money, blood, sweat, tears, etc... into thinking about distributed and cloud computing and making Daft work at Terabyte -> Petabyte scale. I know there has been interest within the Datafusion community to extend into that space (e.g. datafusion-ray, ballista etc).

Happy to explore what a collaboration there would look like and maybe lay out what the benefits of adopting Datafusion would be for the Daft project.

Would love to attend the DataFusion call btw :)

[alamb]

Thanks for the response @samster25 🤗

When we first started transitioning Daft to Rust about 2 years ago the ecosystem was a lot more fragmented and it wasn’t yet clear what we should adopt At the time, Datafusion and arrow-rs were very young projects and didn’t support some of the requirements that we had.

This makes total sense 👍

Also makes sense. For anyone following along, here are some feature updates from the DataFusion side (I am absolutely trying to convince anyone reading this to build with DataFusion :) )

• Support for our own type system for our own logical types like images and tensors as well as out of band types like Python objects.

This is still an area that DataFuson is not as flexible as we might like. Thankfully @notfilippo, @jayzhan211, @findepi and others are working on this (see apache/datafusion#12853 and related)

• Strict requirements on being able to serialize physical plans for a distributed environment.

This is now fully supported via datafusion-ptoto

• We wanted a transmute free library for our in-memory representation. (I was burned by arrow-cpp lol)

I think strictly speaking arrow-rs has one use of transmute left (but it is used to create a floating point constant, so likely not the horror 😱 show it might sound like)

Over time we (well, really largely @tustvold) worked hard to bring the best of the design patterns from arrow2 into arrow-rs, and for what it is worth we haven't had any safety issues / crashes reported in over 2 years to my knowledge. There is still unsafe code (as there is in most performance critical code, including arrow2) but it is used carefully

(so we can see if we can make it easier for future startups to choose to build with DataFusion)

Today that is less clear. Datafusion has an amazing community and has made big leaps within the last 2 years. However we’ve also invested heavily in a lot of our own stack which makes a migration both costly and difficult to justify. If we were to start a new project today, I think our vote would have been unequivocally to start off with Datafusion. Datafusion makes it incredibly easy to get a project off the ground.

❤️

In reality however, we’re a small team with big distributed ambitions and unfortunately Datafusion integration is more of a nice-to-have right now rather than a core accelerant.

Indeed -- rewriting / replacing an already working system is not something to be done lightly. As you say, the core story would have to be that it accelerates your new features somehow.

That being said, there are a few items on our roadmap which we are actively exploring, which should bring us much closer to the rest of the community.

1. Migration from arrow2 to arrow-rs arrow2 has been pretty much abandoned by the community. We would love to move Daft to arrow-rs and contribute back to it.
2. Leverage Datafusion's Logical Plan and Optimizer as you mentioned Datafusion already has a ton of optimizer rules and is quite robust. This not really an area we feel the need to innovate on. The key requirement there for us is being able to plug in our own type system, logical types and expressions.

Maybe https://substrait.io/ or something similar could help here 🤔 (DataFusion supports substrait)

3. Data Catalog and Table Format integrations - we’re very excited about this and have been on the forefront of integrating with a lot of these technologies (outside of the JVM). Our current integrations are in Python, but we do want to transition to more native Rust integrations and are in talks with the relevant Rust projects here.

This is also an area I want to make easier for DataFusion too (iceberg and deltalake specifically). I think it is still too hard to integrate these pieces into a system that "just works" and reads iceberg tables, etc

We have some cool stuff coming up with the new FFI bindings from @timsaucer (see apache/datafusion#13175 / https://crates.io/crates/datafusion-ffi). It is still early

4. Pluggable engines: we are just about to finish up with our new local execution engine, and at some point will be adding other engines such as a newly designed distributed engine, or even a proprietary one. Plugging in Datafusion here as a POC could be interesting, and we could even try that on a distributed runner.

I think (4) is potentially the most interesting story, but will obviously require a lot of work and investment from both Daft and Datafusion to make happen. We’ve spent a lot of time, effort, money, blood, sweat, tears, etc... into thinking about distributed and cloud computing and making Daft work at Terabyte -> Petabyte scale. I know there has been interest within the Datafusion community to extend into that space (e.g. datafusion-ray, ballista etc).

Happy to explore what a collaboration there would look like and maybe lay out what the benefits of adopting Datafusion would be for the Daft project.

I think the basic story would be "you don't have to go reimplement the guts of the query engine" -- there are also several other people working on DataFusion based distributed engines, so I expect that supporting infrastructure for that will improve over time.

Would love to attend the DataFusion call btw :)

We would love to have you

[alamb]

I am sure you can do some pretty sweet things with $30M https://www.daft.ai/daft/blog/eventual-raises-30m-to-build-the-future-of-data

If you ever want to work alongside is in DataFusion we'll be happy to welcome you too

[rchowell]

Thank you @alamb!

Some interesting developments and follow-ups from when this thread was last active.

• Decoupling of our logical types from arrow (specifically arrow2) is still an ongoing challenge. It is extremely relevant now with the new geospatial and variant types. We understand a ton of effort has been put into arrow-rs variant, and we would have to duplicate this in arrow2 (AFAIK?) which could be the motivator we need to get onto arrow-rs.
• Levering datafusions logical planner would be excellent. Coming from the JVM, writing a SQL optimizer seemed silly when there is Calcite. Now, we do use sqlparser-rs, but have discussed our own parser. Something about the engine having more control over its syntax and AST is enticing, but for logical plans, I don't think there's a similar need. Then of course we keep our backend.
• For plan serialization, we recently embarked on our own proto journey. Our logical plans currently implement serde, but we wish to remove this. We chose our own protos over substrait for the sake of velocity rather than compatibility. The goal was to decouple our rust types from a specific serialization (aka remove serde), rather than the goal being substrait compatibility. I believe substrait would be very beneficial for daft compatibility, like our spark-connect project, but I believe we have much more work to do on our logical plan before this is feasible.
• For our SQL implementation, we added catalog and table interfaces just like the table provider work from spice.ai (@phillipleblanc). These interfaces make it possible for us to work with table formats like iceberg and delta purely from rust, but more work still needs to be done there to migrate each of these integrations.

Ideally we don't have to re-implement the guts of the compiler's logical layer, as well as its mechanisms for querying catalog metadata AND loading data from table sources via arrow. There are some other interesting problems too such as common expression representation for use in pushdowns (arrow compute expressions perhaps?).

In the long term, I still see us having our own,

• SQL grammar + AST
• Types and cast/coercion behavior.
• Serde (not 100% sure here.)
• Compiler backend (physical planner) and executor.

[alamb]

• Decoupling of our logical types from arrow (specifically arrow2) is still an ongoing challenge.

Yes -- this is also something DataFusion struggles with (using the Arrow type system directly) and to be honest I think we will be using the Arrow type system for quite a while

• Levering datafusions logical planner would be excellent. Coming from the JVM, writing a SQL optimizer seemed silly when there is Calcite. Now, we do use sqlparser-rs, but have discussed our own parser. Something about the engine having more control over its syntax and AST is enticing, but for logical plans, I don't think there's a similar need. Then of course we keep our backend.

I think people like SDF (now DBT) basically have their own frontend sql parsers and create DataFusion LogicalPlans directly so there is precedent to doing this

• For plan serialization, we recently embarked on our own proto journey. Our logical plans currently implement serde, but we wish to remove this. We chose our own protos over substrait for the sake of velocity rather than compatibility.

This is pretty much the same conclusion we came to in https://docs.rs/datafusion-proto/latest/datafusion_proto/

. I believe substrait would be very beneficial for daft compatibility, like our spark-connect project, but I believe we have much more work to do on our logical plan before this is feasible.

We have https://docs.rs/datafusion-substrait/latest/datafusion_substrait/ which is becoming quite feature rich thanks to the work of @gabotechs and others from DataDog and other companies

• For our SQL implementation, we added catalog and table interfaces just like the table provider work from spice.ai (@phillipleblanc). These interfaces make it possible for us to work with table formats like iceberg and delta purely from rust, but more work still needs to be done there to migrate each of these integrations.

I think the equivalent version of those interfaces is here in DataFusion: https://docs.rs/datafusion-catalog/48.0.0/datafusion_catalog/trait.CatalogProvider.html

It has taken some time but I think they are pretty reasonable now, and there are pre-created icerberg and delta-rs providers for Datafusion (they still need a bunch of work to help mature, but they do exist)

[rchowell]

This is super helpful information, and I really appreciate it!

It makes me want to re-visit an old discussion when daft-sql was first started. #2468 (comment). I do feel the custom frontend + datafusion logical plans would enable us to focus on our specific areas.

It has taken some time but I think they are pretty reasonable now, and there are pre-created icerberg and delta-rs providers for Datafusion (they still need a bunch of work to help mature, but they do exist)

The two challenges I see with those interfaces (and I might not be understanding entirely, so apologies if this is off the mark) is (1) requiring identifier normalization and (2) not supporting nested namespaces/schemas. I do like how you have the SchemaProvider (same style as calcite), whereas we flattened (catalog->table) more like the iceberg java interface. The flattening helps with nested namespaces, so there's no "schema" object.

For identifiers, there needs to be some means of capturing case-sensitive vs. case-insensitive as this semantic varies across catalogs/systems.

[alamb]

The two challenges I see with those interfaces (and I might not be understanding entirely, so apologies if this is off the mark) is (1) requiring identifier normalization and (2) not supporting nested namespaces/schemas.

I do think the Datafusion API supports nested identifiers -- multi-part identifiers are resolved to https://docs.rs/datafusion/latest/datafusion/common/enum.TableReference.html

For identifiers, there needs to be some means of capturing case-sensitive vs. case-insensitive as this semantic varies across catalogs/systems.

DataFusion has a setting that controls if the identifiers are normalized

From https://datafusion.apache.org/user-guide/configs.html

datafusion.sql_parser.enable_ident_normalization	true	When set to true, SQL parser will normalize ident (convert ident to lowercase when not quoted)

I am not quite sure if that is what you mean

[rchowell]

DataFusion is more accurate to SQL than most 😁 with <catalog>.<schema>.<table> whereas Iceberg et al are starting to use <catalog>.<namespace>+.<table> meaning DataFusion may need to update its TableReference to support thing like cat.ns1.ns2.tbl. These 4 part (or more) identifiers are not standard, but have been growing in popularity and can be quite handy.

The interesting bit on normalization is the ability to tell a catalog "this identifier was quoted" or "this identifier was not quoted" and pushing down the normalization (or lack thereof) to the catalog rather than always normalizing before talking to the catalog. This is a hard problem though, but there's something enticing about catalogs being responsible for their own name resolution given the user's original intention, rather than the engine interpreting first and normalizing before sending to the catalog for lookup.

[phillipleblanc]

I brought up the issue of having DataFusion support 4+ part identifiers (as we needed to support it in our project) here: apache/datafusion#13822

The main issue is that it creates ambiguity in parsing column references, and as you mentioned the identifier quoting issue.

For our SQL implementation, we added catalog and table interfaces just like the table provider work from spice.ai (@phillipleblanc). These interfaces make it possible for us to work with table formats like iceberg and delta purely from rust, but more work still needs to be done there to migrate each of these integrations.

I can't take credit for the interfaces :) But glad that our work was helpful for you to reference.