Create mini-dataset to compare against private VLM scores

[bw4sz]

given the massive investment in AI models, there is some question on how an open source model competes against things like agentic object detector workflow from LandingAI, gemini and all the others. I think we could have a tiny cherrypicked (difficult) test set to compare against, both from a zero-shot and a with data perspective. That way it wouldn't be too expensive to run and review.

[Abhishek-Dimri]

Hey @bw4sz, just wanted to enquire about certain thing regarding this issue :

• Size & Scope: Should it be tiny (20-50), small (100+), and focus on zero-shot or fine-tuning too?
• Integration: Standalone benchmark or part of MillionTrees?
• Selection: Should we handpick data or randomly select through an algorithm?

[Abhishek-Dimri]

Would like you to specify the preferred methodology for this.

[bw4sz]

Thanks @Abhishek-Dimri , but this issue isn't really ready for outside contribution, still too early to really engage with it. If you are interested in the MillionTrees project, there are other pieces that are simpler and try to engage with documentation and the beginning user base here.

[Abhishek-Dimri]

Thank you for the clarification and guidance! I'm working on understanding the flow of the MillionTrees project better, and I’ll review the documentation to get a clearer picture.
I appreciate your support!