Skip to content

Commit 8e04720

Browse files
featzhangfeatzhang
authored
[INLONG-7030][Manager][Sort] Build tool for local debugging environment (#7031)
1 parent 3b46460 commit 8e04720

File tree

3 files changed

+291
-0
lines changed

3 files changed

+291
-0
lines changed

‎.gitignore

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -71,3 +71,13 @@ inlong-sdk/dataproxy-sdk-twins/dataproxy-sdk-cpp/release/bin/
7171

7272
# Docker mysql storage
7373
docker/docker-compose/mysql
74+
75+
# Directories for local debugging
76+
## The connectors for loading to Flink job
77+
/inlong-sort/connectors
78+
## The plugins for submitting job to Flink cluster
79+
/plugins
80+
## The jar for parsing job info
81+
/inlong-sort/sort-dist.jar
82+
## The directory for load plugins by manager
83+
/inlong-manager/plugins

‎inlong-tools/dev/README.md

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
<!--
2+
3+
Licensed to the Apache Software Foundation (ASF) under one
4+
or more contributor license agreements. See the NOTICE file
5+
distributed with this work for additional information
6+
regarding copyright ownership. The ASF licenses this file
7+
to you under the Apache License, Version 2.0 (the
8+
"License"); you may not use this file except in compliance
9+
with the License. You may obtain a copy of the License at
10+
11+
http://www.apache.org/licenses/LICENSE-2.0
12+
13+
Unless required by applicable law or agreed to in writing,
14+
software distributed under the License is distributed on an
15+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
16+
KIND, either express or implied. See the License for the
17+
specific language governing permissions and limitations
18+
under the License.
19+
20+
-->
21+
22+
## Apache InLong dev toolkit
23+
24+
### Overview
25+
26+
The Apache InLong project includes modules such as "Dashboard", "DataProxy", "Manager" and "TubeMQ".
27+
When debugging in a development environment such as `Intellij IDEA`,
28+
The environment is difficult to build.
29+
30+
Take the `inlong-manager` as an example, it depends on too many configurations and packages.
31+
When debugging locally in the IDE, multiple default directories need to be created, which is more complicated.
32+
So a script is urgently needed to quickly support local debugging.
33+
34+
Also, similar tools are required for local debugging of other modules.
35+
Temporarily named `InLong dev toolkit`, looking forward to adding more features.
36+
37+
### Use case
38+
39+
#### Help
40+
41+
```shell
42+
❯ inlong-tools/dev/inlong-dev-toolkit.sh
43+
####################################################################################
44+
# Welcome to use the Apache InLong dev toolkit! #
45+
# @2022-12-23 20:38:34 #
46+
####################################################################################
47+
48+
49+
inlong-tools/dev/inlong-dev-toolkit.sh help | h
50+
:help
51+
52+
inlong-tools/dev/inlong-dev-toolkit.sh manager | m
53+
:build manager local debugging environment
54+
55+
Have a nice day, bye!
56+
57+
```
58+
59+
#### Manager
60+
61+
Rely of Manager:
62+
63+
| Directory/File | link target | Rely modules |
64+
|--------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------|---------------------------------|
65+
| `./plugins` | `inlong-manager/manger-plugins/target/plugins` | `inlong-sort/sort-plugin` |
66+
| `./inlong-sort/connectors` | - | `inlong-sort/sort-connectors/*` | |
67+
| `./inlong-sort/connectors/sort-connector-${name}-${project.version}.jar` | `inlong-sort/sort-connectors/${connector.name}/target/sort-connector-${connector.name}-${project.version}.jar` | `inlong-sort/sort-connectors/*` |
68+
| `./sort-dist.jar` | `sort-dist/target/sort-dist-${project.version}.jar` | `inlong-sort/sort-dist` |
69+
70+
```shell
71+
❯ inlong-tools/dev/inlong-dev-toolkit.sh m
72+
Execute action: manager
73+
# start build manager local debugging environment ...
74+
current_version: 1.5.0-SNAPSHOT
75+
associate plugins directory: inlong-manager/manager-plugins/target/plugins
76+
associate sort dist: inlong-sort/sort-dist/target/sort-dist
77+
recreate connector dir: inlong-sort/connectors
78+
All connector names:
79+
hive mysql-cdc kafka jdbc pulsar iceberg postgres-cdc mongodb-cdc sqlserver-cdc oracle-cdc elasticsearch-6 elasticsearch-7 redis tubemq filesystem doris starrocks hudi
80+
associate connector: hive
81+
associate connector: mysql-cdc
82+
associate connector: kafka
83+
associate connector: jdbc
84+
associate connector: pulsar
85+
associate connector: iceberg
86+
associate connector: postgres-cdc
87+
associate connector: mongodb-cdc
88+
associate connector: sqlserver-cdc
89+
associate connector: oracle-cdc
90+
associate connector: elasticsearch-6
91+
associate connector: elasticsearch-7
92+
associate connector: redis
93+
associate connector: tubemq
94+
associate connector: filesystem
95+
associate connector: doris
96+
associate connector: starrocks
97+
associate connector: hudi
98+
build dev env of manager finished.
99+
Have a nice day, bye!
100+
```
101+
102+
### Development: add more function
103+
104+
#### Public variables
105+
106+
| inner variable | implication |
107+
|--------------------|-----------------------------------|
108+
| `script_dir` | the directory of this script file |
109+
| `script_file_name` | the script file name |
110+
| `base_dir` | the root directory of project |
111+
112+
#### Step 1. implement function
113+
114+
#### Step 2. register function
115+
116+
Create a global array variable:
117+
118+
- It is recommended to end with `_action`, such as `manager_action`.
119+
- It must have 4 elements.
120+
- The 1st element is the long opt command.
121+
- The 2nd element is the sort opt command.
122+
- The 3rd element is the details of the function.
123+
- The 4th element is the function name, which implements in Step 1.
124+
125+
Add the variable to the `actions` array, like:
126+
127+
```shell
128+
actions=(
129+
help_action
130+
manger_action
131+
)
132+
```
133+
134+
#### Step 3. ignore the temporary file
135+
136+
Add temporary files to `.gitignore`
137+
138+
> Notice: must base on project base directory

‎inlong-tools/dev/inlong-dev-toolkit.sh

Lines changed: 143 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,143 @@
1+
#!/bin/bash
2+
#
3+
# Licensed to the Apache Software Foundation (ASF) under one or more
4+
# contributor license agreements. See the NOTICE file distributed with
5+
# this work for additional information regarding copyright ownership.
6+
# The ASF licenses this file to You under the Apache License, Version 2.0
7+
# (the "License"); you may not use this file except in compliance with
8+
# the License. You may obtain a copy of the License at
9+
#
10+
# http://www.apache.org/licenses/LICENSE-2.0
11+
#
12+
# Unless required by applicable law or agreed to in writing, software
13+
# distributed under the License is distributed on an "AS IS" BASIS,
14+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15+
# See the License for the specific language governing permissions and
16+
# limitations under the License.
17+
# Initialize the configuration files of inlong components
18+
19+
script_dir=$(dirname "$0")
20+
script_file_name=$(basename "$0")
21+
22+
# the absolute dir of project
23+
base_dir=$(
24+
cd $script_dir
25+
cd ../..
26+
pwd
27+
)
28+
29+
help_action=(
30+
'help'
31+
'h'
32+
'help: show all actions'
33+
'welcome'
34+
)
35+
36+
manager_action=(
37+
'manager'
38+
'm'
39+
'build manager local debugging environment'
40+
'manager'
41+
)
42+
43+
actions=(
44+
help_action
45+
manager_action
46+
)
47+
48+
function welcome() {
49+
local_date_time=$(date +"%Y-%m-%d %H:%M:%S")
50+
echo '####################################################################################'
51+
echo '# Welcome to use Apache InLong dev toolkit ! #'
52+
echo '# @'$local_date_time' #'
53+
echo '####################################################################################'
54+
echo ''
55+
56+
# shellcheck disable=SC2068
57+
for action in ${actions[@]}; do
58+
# shellcheck disable=SC1087
59+
TMP=$action[@]
60+
TempB=("${!TMP}")
61+
name=${TempB[0]}
62+
simple_name=${TempB[1]}
63+
desc=${TempB[2]}
64+
65+
echo $script_dir'/'$script_file_name' '$name' | '$simple_name
66+
echo ' :'$desc
67+
done
68+
echo ''
69+
}
70+
71+
function manager() {
72+
echo '# start build manager local debugging environment ...'
73+
74+
project_version=$(mvn -q \
75+
-Dexec.executable=echo \
76+
-Dexec.args='${project.version}' \
77+
--non-recursive \
78+
exec:exec)
79+
80+
echo 'current_version: '"$project_version"
81+
#
82+
echo 'associate plugins directory: inlong-manager/manager-plugins/target/plugins'
83+
# plugins -> manager-plugins/target/plugins
84+
cd "$base_dir"
85+
rm -rf plugins
86+
ln -s inlong-manager/manager-plugins/target/plugins plugins
87+
#
88+
echo 'associate sort dist: inlong-sort/sort-dist/target/sort-dist'
89+
cd "$base_dir"/inlong-sort
90+
rm -rf sort-dist.jar
91+
ln -s sort-dist/target/sort-dist-"$project_version".jar sort-dist.jar
92+
# inlong-manager: plugins -> manager-plugins/target/plugins
93+
cd "$base_dir"/inlong-manager
94+
rm -rf plugins
95+
ln -s manager-plugins/target/plugins plugins
96+
# mkdir inlong-sort/connectors
97+
sort_connector_dir=$base_dir/inlong-sort/connectors
98+
echo 'recreate connector dir: '"$sort_connector_dir"
99+
# shellcheck disable=SC2086
100+
rm -rf $sort_connector_dir
101+
mkdir "$sort_connector_dir"
102+
cd "$sort_connector_dir"
103+
connector_names=$(grep '<module>' "$base_dir"/inlong-sort/sort-connectors/pom.xml | sed 's/<module>//g' | sed 's/<\/module>//g' | grep -v base)
104+
105+
echo 'All connector names: '
106+
echo $connector_names | tr -d '\n'
107+
108+
for connector_name in $connector_names; do
109+
echo 'associate connector: '"$connector_name"
110+
connector_suffix_name=$(echo "$connector_name" | sed 's/elasticsearch-6/elasticsearch6/g' | sed 's/elasticsearch-7/elasticsearch7/g')
111+
ln -s ../sort-connectors/"${connector_name}"/target/sort-connector-"${connector_suffix_name}"-"$project_version".jar sort-connector-"${connector_name}".jar
112+
done
113+
114+
echo 'build dev env of manager finished .'
115+
}
116+
117+
function main() {
118+
action=$1
119+
120+
if [ ! -n "$action" ]; then
121+
welcome
122+
fi
123+
124+
# shellcheck disable=SC2068
125+
for one_action in ${actions[@]}; do
126+
# shellcheck disable=SC1087
127+
TMP=$one_action[@]
128+
TempB=("${!TMP}")
129+
name=${TempB[0]}
130+
simple_name=${TempB[1]}
131+
function_name=${TempB[3]}
132+
desc=${TempB[4]}
133+
134+
if [[ X"$name" == X"$action" ]] || [[ X"$simple_name" == X"$action" ]]; then
135+
echo 'Execute action: '"$function_name"
136+
$function_name
137+
fi
138+
done
139+
140+
echo 'Have a nice day, bye!'
141+
}
142+
143+
main $1

0 commit comments

Comments
 (0)