Proposal of testing a difypkg

[Yeuoly]

Self Checks

• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:)
• Please do not modify this template :) and fill in all the required fields.

Content

As we all know, the plugin system has become one of the most important components of Dify. However, we’ve already encountered several challenges, such as security, integrity, dependency management, scalability, and availability.

At this stage, availability is our top concern. We’ve seen many issues caused by individual plugins rather than Dify itself. Most of these problems could have been caught earlier through CI integration testing before merging. This kind of testing not only prevents regressions but also provides plugin developers and AI agents (like Vibe Coding) a reliable way to verify that a plugin behaves as expected—rather than relying on screenshots or manual testing, which is often unreliable.

To achieve this, we need a way to test .difypkg files. I’ve considered using pytest or unittest, but since these are Python-specific, they only work for repos like dify-official-plugins (which is open-sourced and managed as a monorepo). For repositories like dify-plugins, we plan to deprecate them once direct uploads to the Marketplace are supported. That means traditional testing methods won’t apply there anymore. Also, since unit tests can’t validate full plugin behavior, we need integration testing instead.

Proposed Approach

I propose using pytest outside the plugin package to perform integration tests. Here’s a basic example:

import os
from dify_plugin.ci.integration_test import execute_tool

def test_google_search():
    with open("google.difypkg", "rb") as pkg:
        for message in execute_tool(pkg, "google_search", credentials={
            "api_key": os.getenv("GOOGLE_API_KEY")
        }, parameters={}):
            assert message

You can use a mock server if needed.

The execute_tool function will run the plugin in a real environment. As for internal implementation, we can add a CLI command in dify-plugin-daemon, such as:

dify plugin run xxx.difypkg

And package dify_plugin.ci.integration_test can use it to run plugin.

Of course, this is just a minimal draft. There are still many important details that need further discussion and exploration.

Let me know your thoughts. I’m open to suggestions or alternative approaches if you have a better idea. : )

[Yeuoly]

I've implemented a solution here langgenius/dify-plugin-sdks#144, tested already

[QuantumGhost]

To ensure we can experiment with this feature while minimizing concerns about stability guarantees, I suggest marking both the testing feature and related packages in dify_plugin as unstable. Specifically, my proposals are:

1. Move dify_plugin.integration.run into a clearly defined namespace, such as dify_plugin.unstable.integration.run.
2. Introduce a boolean flag --unstable for the dify-plugin command-line tool, making it mandatory when executing the dify-plugin plugin run command.

This approach gives us the flexibility to explore and iterate on integration testing mechanisms without committing to long-term stability right away.