Skip to content

.Net: Bug: Qdrant DateTime Range filter set incorrectly #12934

@TsengSR

Description

@TsengSR

Describe the bug
When translating thje new DateTimeOffset range filter, all filters are set (Gt, Gte, Lt, Lte) always resulting in 0 results found when using Qdrant connector

To Reproduce

        IAsyncEnumerable<VectorSearchResult<Project>> results = collection.SearchAsync(embeddings, 10, new VectorSearchOptions<Project>()
        {
            Filter = project => project.PublishedOn > newerThan
        });

Expected behavior
Only the affected comparisions to be set

Platform

  • Language: C#
  • Source: Microsoft.SemanticKernel.Connectors.Qdrant, 1.62.0-preview
  • AI model: -
  • IDE: Visual Studio
  • OS: Windows

Additional context

The error lies in here

DateTimeOffset v => new FieldCondition
{
Key = property.StorageName,
DatetimeRange = new DatetimeRange
{
Gt = Timestamp.FromDateTimeOffset(v),
Gte = Timestamp.FromDateTimeOffset(v),
Lt = Timestamp.FromDateTimeOffset(v),
Lte = Timestamp.FromDateTimeOffset(v)
}
},

As per Qdrant DateTime range filtering documentation, the non-matching comparisions shall be null and only the ones be filled, which are part of the filter expression.

I did add some simple unit tests which where failing at first (expected and actual result count mismatch), then applied below fix to pass them.

    DateTimeOffset v => new FieldCondition
    {
        Key = property.StorageName,
        DatetimeRange = new DatetimeRange
        {
            Gt = comparison.NodeType == ExpressionType.GreaterThan ? Timestamp.FromDateTimeOffset(v) : null,
            Gte = comparison.NodeType == ExpressionType.GreaterThanOrEqual ? Timestamp.FromDateTimeOffset(v) : null,
            Lt = comparison.NodeType == ExpressionType.LessThan ? Timestamp.FromDateTimeOffset(v) : null,
            Lte = comparison.NodeType == ExpressionType.LessThanOrEqual ? Timestamp.FromDateTimeOffset(v) : null
        }
    },

I'd toss a PR towards this, but locally I can't test the other connectors as I only have Qdrant running locally and I'm not 100% certain how to set more than 1 operation at the same time or if its really required.

According to Qdrant DateTime range filtering documentation, 1 or more operatiors are allowed per DateTime range object, i.e.

{
  "key": "date",
  "range": {
    "gt": "2023-02-08T10:49:00Z",
    "gte": null,
    "lt": null,
    "lte": "2024-01-31 10:14:31Z"
  }
}

but currently one Condition is added per comparision expression.

Maybe I'll see if I can add more tests and in case of DateTimeOffset, lookup if there is already an condition existing and add it there?

Or in first revsion just leave it as is with multiple datetime conditions and hope there is no performance implicates having it in two separate conditions instead of one?

@roji
Any pointer how to proceed there? Just the simple fix first and see later if one can do it with a single condition?

Metadata

Metadata

Assignees

No one assigned

    Labels

    .NETIssue or Pull requests regarding .NET codebugSomething isn't workingtriage

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions