Detection Image Function Not Working When Migrating MCP Server to semantic-kernel

[nblog]

I am attempting to migrate my existing MCP server implementation to use semantic-kernel as the backend. My goal is to replicate the image handling functionality that works with my previous mcp implementation.

Previous FastMCP Implementation (Working)

"""Example showing image handling with FastMCP."""

from mcp.server.fastmcp import FastMCP, Image

mcp = FastMCP("Image Example")

@mcp.tool()
def explain_image(image_path: str):
    return Image(data=open(image_path, mode="rb").read(), format="png")

@mcp.tool()
def detection_image(image_path: str):
    return [
        "How many cats are in the picture?",
        Image(data=open(image_path, mode="rb").read(), format="png"),
    ]

if __name__ == "__main__":
    mcp.run()

semantic-kernel Implementation (Issue Present)

import argparse
import logging
from typing import Any, Literal

from semantic_kernel import Kernel
from semantic_kernel.connectors.ai.open_ai import OpenAIChatCompletion
from semantic_kernel.functions import kernel_function
from semantic_kernel.prompt_template.input_variable import InputVariable
from semantic_kernel.prompt_template import InputVariable, KernelPromptTemplate, PromptTemplateConfig
from semantic_kernel.contents import FileReferenceContent, ImageContent, TextContent

logger = logging.getLogger(__name__)


def parse_arguments():
    parser = argparse.ArgumentParser(description="Run the Semantic Kernel MCP server.")
    parser.add_argument(
        "--transport",
        type=str,
        choices=["sse", "stdio"],
        default="stdio",
        help="Transport method to use (default: stdio).",
    )
    parser.add_argument(
        "--port",
        type=int,
        default=None,
        help="Port to use for SSE transport (required if transport is 'sse').",
    )
    return parser.parse_args()


def run(transport: Literal["sse", "stdio"] = "stdio", port: int | None = None) -> None:
    kernel = Kernel()

    @kernel_function()
    def explain_image(image_path: str):
        """Explain the image at the given path."""
        return ImageContent.from_image_file(image_path)

    @kernel_function()
    def detection_image(image_path: str):
        """Detect objects in the image at the given path."""
        return [
            TextContent(text="How many cats are in the picture?"),
            ImageContent.from_image_file(image_path),
        ]

    kernel.add_function("__builtins__", explain_image)
    kernel.add_function("__builtins__", detection_image)
    server = kernel.as_mcp_server(server_name="sk-test")

    if transport == "sse" and port is not None:
        import uvicorn
        from mcp.server.sse import SseServerTransport
        from starlette.applications import Starlette
        from starlette.routing import Mount, Route

        sse = SseServerTransport("/messages/")

        async def handle_sse(request):
            async with sse.connect_sse(request.scope, request.receive, request._send) as (read_stream, write_stream):
                await server.run(read_stream, write_stream, server.create_initialization_options())

        starlette_app = Starlette(
            debug=True,
            routes=[
                Route("/sse", endpoint=handle_sse),
                Mount("/messages/", app=sse.handle_post_message),
            ],
        )

        uvicorn.run(starlette_app, host="0.0.0.0", port=port)  # nosec
    elif transport == "stdio":
        import anyio
        from mcp.server.stdio import stdio_server

        async def handle_stdin(stdin: Any | None = None, stdout: Any | None = None) -> None:
            async with stdio_server() as (read_stream, write_stream):
                await server.run(read_stream, write_stream, server.create_initialization_options())

        anyio.run(handle_stdin)


if __name__ == "__main__":
    args = parse_arguments()
    run(transport=args.transport, port=args.port)

Result

• Calling explain_image: Works as expected
• Calling detection_image: Does not work

Request

Could you please help me identify why the detection_image function is not working as intended in the semantic-kernel implementation, even though it works in the original FastMCP-based implementation?
Any guidance or suggestions on how to properly return both a text and image content using semantic-kernel's MCP server would be greatly appreciated.

Thank you!

[moonbox3]

@eavanvalkenburg could you take a look?

[eavanvalkenburg]

So, looking through this, @nblog, I'm not seeing any obvious things wrong in your code, and since the first one works it 's mostly working as expected, can you share some logs of calling both functions to see if we can detect the difference, I was thinking it might be the image being handled differently by FastMCP, but since image works in the first one, thats not it, so I'll need some more info to further troubleshoot!

[nblog]

@eavanvalkenburg

there is nothing special in the vscode log.

2025-08-06 15:58:31.403 [info] 正在停止服务器 sk-test
2025-08-06 15:58:31.633 [info] 连接状态: 已停止
2025-08-06 15:58:31.635 [info] 正在启动服务器 sk-test
2025-08-06 15:58:31.635 [info] 连接状态: 正在启动
2025-08-06 15:58:31.638 [info] Starting server from LocalProcess extension host
2025-08-06 15:58:31.646 [info] 连接状态: 正在启动
2025-08-06 15:58:31.649 [info] 连接状态: 正在运行
2025-08-06 15:58:33.931 [info] Discovered 2 tools

I tried testing npx @modelcontextprotocol/inspector and found that semantic-kernel output the image as base64 data, while mcp was a image.

Semantic-Kernel

MCP

Do you need any further information from me?

[nblog]

@eavanvalkenburg

I think that's the problem.

{
  "content": [
    {
      "type": "text",
      "text": "图片中有几只猫？"
    },
    {
      "type": "image",
      "data": "iVBORw0KGgoA...

{
  "content": [
    {
      "type": "text",
      "text": "图片中有几只猫？"
    },
    {
      "type": "text",
      "text": "data:image/png;base64,iVBORw0K...

semantic-kernel seems to be returning text.

[eavanvalkenburg]

Thanks for that, that gave me the right spot to check and it was indeed not accurately parsing the returned types, I made a fix.

[eavanvalkenburg]

you could apply the same change to your local install and verify @nblog #12871

[eavanvalkenburg]

Fixed in release 1.35.2