Skip to content

Improve the handling of inputs for TransformersMultiModal #1688

@RobinPicard

Description

@RobinPicard

The current approach in which we request users to include the asset tags in their text prompts causes problems for models for which the call to format_chat_template is expected to add them (for instance Qwen2.5VL). In general, we should review the way we turn user inputs into model inputs for this model to make sure it works with different types of models (so we need look at the format_input method of the type adapter).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions