Skip to content

[Feature][Dashboard] Add Transform Debug Interface in InLong Dashboard for Real-Time Validation #11797

Open
@Jzow

Description

@Jzow

Description

Currently, InLong lacks a user-friendly way to debug and validate Transform configurations in real-time. Users cannot verify whether the output of their Transform SQL aligns with expectations after configuring input/output types and transformation logic. This gap leads to potential misconfigurations and inefficiencies during data pipeline setup.

To address this, we propose adding a ​Transform Debug Interface within the InLong Dashboard. This interface will allow users to:

  1. Select input/output data types (JSON, CSV, Other).
  2. Write or upload Transform SQL.
  3. Provide test input data (e.g., sample records).
  4. Execute the transformation and view the processed output in real-time.
  5. Validate if the output matches the expected schema and content

Use case

​Scenario: A user configures a Transform SQL to convert a CSV log into a JSON format for downstream analytics.

Steps:

  • In the Dashboard, navigate to the Transform Debug tab.
  • Select CSV as input type and JSON as output type.
  • Input sample CSV data: timestamp,user_id,action\n2025-03-10,123,login.
  • Write Transform SQL: SELECT timestamp, user_id AS userId, action FROM SOURCE`.
  • Click "Run Debug" to see the transformed output: {"timestamp":"2023-01-01","userId":123,"action":"login"}

​Outcome: The user confirms the output schema matches their requirements before deploying the pipeline.

Are you willing to submit PR?

  • Yes, I am willing to submit a PR!

Code of Conduct

Metadata

Metadata

Assignees

Labels

stage/staleIssues or PRs that had no activity for a long timetype/feature

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions