-
Notifications
You must be signed in to change notification settings - Fork 4.1k
Description
Describe the bug
When translating thje new DateTimeOffset range filter, all filters are set (Gt, Gte, Lt, Lte) always resulting in 0 results found when using Qdrant connector
To Reproduce
IAsyncEnumerable<VectorSearchResult<Project>> results = collection.SearchAsync(embeddings, 10, new VectorSearchOptions<Project>()
{
Filter = project => project.PublishedOn > newerThan
});
Expected behavior
Only the affected comparisions to be set
Platform
- Language: C#
- Source: Microsoft.SemanticKernel.Connectors.Qdrant, 1.62.0-preview
- AI model: -
- IDE: Visual Studio
- OS: Windows
Additional context
The error lies in here
semantic-kernel/dotnet/src/VectorData/Qdrant/QdrantFilterTranslator.cs
Lines 131 to 141 in 2d64013
DateTimeOffset v => new FieldCondition | |
{ | |
Key = property.StorageName, | |
DatetimeRange = new DatetimeRange | |
{ | |
Gt = Timestamp.FromDateTimeOffset(v), | |
Gte = Timestamp.FromDateTimeOffset(v), | |
Lt = Timestamp.FromDateTimeOffset(v), | |
Lte = Timestamp.FromDateTimeOffset(v) | |
} | |
}, |
As per Qdrant DateTime range filtering documentation, the non-matching comparisions shall be null
and only the ones be filled, which are part of the filter expression.
I did add some simple unit tests which where failing at first (expected and actual result count mismatch), then applied below fix to pass them.
DateTimeOffset v => new FieldCondition
{
Key = property.StorageName,
DatetimeRange = new DatetimeRange
{
Gt = comparison.NodeType == ExpressionType.GreaterThan ? Timestamp.FromDateTimeOffset(v) : null,
Gte = comparison.NodeType == ExpressionType.GreaterThanOrEqual ? Timestamp.FromDateTimeOffset(v) : null,
Lt = comparison.NodeType == ExpressionType.LessThan ? Timestamp.FromDateTimeOffset(v) : null,
Lte = comparison.NodeType == ExpressionType.LessThanOrEqual ? Timestamp.FromDateTimeOffset(v) : null
}
},
I'd toss a PR towards this, but locally I can't test the other connectors as I only have Qdrant running locally and I'm not 100% certain how to set more than 1 operation at the same time or if its really required.
According to Qdrant DateTime range filtering documentation, 1 or more operatiors are allowed per DateTime range object, i.e.
{
"key": "date",
"range": {
"gt": "2023-02-08T10:49:00Z",
"gte": null,
"lt": null,
"lte": "2024-01-31 10:14:31Z"
}
}
but currently one Condition
is added per comparision expression.
Maybe I'll see if I can add more tests and in case of DateTimeOffset, lookup if there is already an condition existing and add it there?
Or in first revsion just leave it as is with multiple datetime conditions and hope there is no performance implicates having it in two separate conditions instead of one?
@roji
Any pointer how to proceed there? Just the simple fix first and see later if one can do it with a single condition?