Open
Description
Description
Currently, InLong
lacks a user-friendly way to debug and validate Transform
configurations in real-time. Users cannot verify whether the output of their Transform SQL
aligns with expectations after configuring input/output types and transformation logic. This gap leads to potential misconfigurations and inefficiencies during data pipeline setup.
To address this, we propose adding a Transform
Debug Interface within the InLong
Dashboard. This interface will allow users to:
- Select input/output data types (JSON, CSV, Other).
- Write or upload Transform SQL.
- Provide test input data (e.g., sample records).
- Execute the transformation and view the processed output in real-time.
- Validate if the output matches the expected schema and content
Use case
Scenario: A user configures a Transform SQL to convert a CSV
log into a JSON
format for downstream analytics.
Steps:
- In the Dashboard, navigate to the Transform Debug tab.
- Select
CSV
as input type andJSON
as output type. - Input sample
CSV
data:timestamp,user_id,action\n2025-03-10,123,login
. - Write Transform
SQL
:SELECT timestamp, user_id AS userId, action
FROM SOURCE`. - Click "Run Debug" to see the transformed output:
{"timestamp":"2023-01-01","userId":123,"action":"login"}
Outcome: The user confirms the output schema matches their requirements before deploying the pipeline.
Are you willing to submit PR?
- Yes, I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct