Skip to content

Move manifest list reader from spec crate #48

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Nov 9, 2024

Conversation

rdettai
Copy link
Contributor

@rdettai rdettai commented Nov 4, 2024

The manifest list reader should be in the same crate as the manifest file reader (main iceberg-rust).

@rdettai rdettai force-pushed the refacto-manifest-list-reader branch from 222049e to 53d78f9 Compare November 4, 2024 15:34
@JanKaul
Copy link
Owner

JanKaul commented Nov 6, 2024

I wonder if moving the manifest file reader out of the iceberg-rust-spec crate was premature, as the readers and writers are somewhat linked to the spec. I like the manifests method of the snapshot as it makes it obvious where to get the list of manifests from. Maybe we can move the manifest file reader back to the iceberg-rust-spec crate but create a separate file for it, so that it's not convoluted with the manifest_entry. And similarly with the ManifestListReader.

Let me know what you think.

@rdettai
Copy link
Contributor Author

rdettai commented Nov 6, 2024

It's nice to have the iceberg-rust-spec really focus on the spec: define how the files should look like and help you build valid Iceberg entities. I see a lot of benefit from decoupling it from object object_store: re-usability, readability, testability... But on the other hand I also think that having a higher level reader/writer that is object_store specific is very useful. Maybe some of the reading/writing logic can maybe be brought back, to have a more powerful spec API, but I would also have an object store specific reader/writer somewhere else.

@JanKaul
Copy link
Owner

JanKaul commented Nov 6, 2024

And there is the question about the naming scheme. So far I've called the reader of the manifest list "ManifestListReader" and the reader of the manifest "ManifestReader". If you would like to rename "ManifestReader" to "ManifestFileReader", it kind of breaks the naming scheme.
I'm not particularly opposed to it, I just think that there should be a universal naming scheme.

@rdettai
Copy link
Contributor Author

rdettai commented Nov 7, 2024

So far I've called the reader of the manifest list "ManifestListReader" and the reader of the manifest "ManifestReader". If you would like to rename "ManifestReader" to "ManifestFileReader", it kind of breaks the naming scheme.

Right, I forgot to rename some variables here and there (manifest_reader -> manifest_file_reder). Are you referring to this or some other place(s) where the naming scheme is broken?

@rdettai
Copy link
Contributor Author

rdettai commented Nov 9, 2024

After a discussion on the naming convention, we agreed that it's better to stick with Manifest for manifest files and ManifestList for manifest list files.

Copy link
Owner

@JanKaul JanKaul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. Thanks for taking this on

@JanKaul JanKaul merged commit ee4761b into JanKaul:main Nov 9, 2024
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants