Skip to content

Commit a7d90ee

Browse files
livi12138peacewong
livi12138
and
peacewong
authored
Create 2024-01-26-linkis1.3.0-adaptation-Huawei MRS-share .md (#786)
* Create 2024-01-26-linkis1.3.0-adaptation-Huawei MRS-share .md linkis分享 * Linkis 1.3.0 英文提交 * 修改authors * 增加作者 * Update 2024-01-26-linkis1.3.0-adaptation-Huawei MRS-share .md * 作者修改 * 回退作者修改 * Update authors.yml * Update authors.yml * Update authors.yml * Update authors.yml * Update authors.yml * 修改文件名 --------- Co-authored-by: peacewong <[email protected]>
1 parent f3195ea commit a7d90ee

File tree

4 files changed

+472
-5
lines changed

4 files changed

+472
-5
lines changed

‎blog/2024-01-26-linkis130-adaptation-Huawei-MRS-share.md

Lines changed: 228 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,228 @@
1+
---
2+
title: Linkis 1.3.0 adapts to Huawei MRS+SCRIPTIS actual combat sharing
3+
authors: [livi12138]
4+
tags: [blog,linki1.3.0,hadoop3.1.1,spark3.0.1,hive3.1.0]
5+
---
6+
## Overview
7+
The team needs to use SQL and Python syntax to analyze the data at the same time on the page. During the investigation, I found that Linkis can meet the needs. As a Huawei MRS is used, it is different from the open source software.
8+
It also carried out secondary development and adaptation. This article will share the experience, hoping to help students in need.
9+
10+
11+
## environment and version
12+
- JDK-1.8.0_112, Maven-3.5.2
13+
- Hadoop-3.1.1, spark-3.1.1, hive-3.1.0, zookerper-3.5.9 (Huawei MRS version)
14+
- Linkis-1.3.0
15+
- Scriptis-Web 1.1.0
16+
17+
## dependence adjustment and packaging
18+
First download the source code of 1.3.0 from the Linkis official website, and then adjust the dependent version
19+
#### Linkis outermost adjustment pom file
20+
21+
```xml
22+
<hadoop.version>3.1.1</hadoop.version>
23+
<zookerper.version>3.5.9</zookerper.version>
24+
<curaor.version>4.2.0</curaor.version>
25+
<guava.version>30.0-jre</guava.version>
26+
<json4s.version>3.7.0-M5</json4s.version>
27+
<scala.version>2.12.15</scala.version>
28+
<scala.binary.version>2.12</scala.binary.version>
29+
```
30+
#### linkis-engineplugin-hive的pom文件
31+
32+
```xml
33+
<hive.version>3.1.2</hive.version>
34+
```
35+
36+
#### linkis-engineplugin-spark的pom文件
37+
38+
```xml
39+
<spark.version>3.1.1</spark.version>
40+
```
41+
#### linkis-hadoop-common的pom文件
42+
```xml
43+
<dependency>
44+
<groupId>org.apache.hadoop</groupId>
45+
<artifactId>hadoop-hdfs</artifactId> <!-Just replace the line and replace it with <Arttifactid> Hadoop-HDFS-Client </Artifactid>->->
46+
<version>${hadoop.version}</version>
47+
</dependency>
48+
Modify the Hadoop-HDFS to:
49+
<dependency>
50+
<cepid> org.apache.hadoop </groupid>
51+
<Artifactid> Hadoop-HDFS-Client </Artifactid>
52+
<Version> $ {Hadoop.Version} </version>
53+
</dependency>
54+
```
55+
#### Linkis-Label-Common
56+
org.apache.linkis.manager.label.conf.labelcommonconfig
57+
Modify the default version, which is convenient for subsequent self -compiling scheduling components
58+
```
59+
Public Static Final Commonvars <string> Spark_engine_Version =
60+
Commonvars.apply ("wds.linkis.spark.engine.version", "3.1.1");
61+
62+
Public Static Final Commonvars <string> Hive_engine_Version =
63+
Commonvars.apply ("wds.linkis.hive.engine.version", "3.1.2");
64+
```
65+
66+
#### Linkis-computation-Governance-Common
67+
org.apache.linkis.governance.Common.conf.governanceCommonConf
68+
Modify the default version, which is convenient for subsequent self -compiling scheduling components
69+
70+
```
71+
Val spark_engine_version = Commonvars ("wds.linkis.spark.engine.version", "3.1.1")
72+
73+
VAL HIVE_ENGINE_VERSION = Commonvars ("wds.linkis.hive.engine.version", "3.1.2")
74+
```
75+
76+
#### Compilation
77+
78+
After the above configuration is adjusted, you can start compiling full amount, and execute the following commands in turn
79+
80+
```shell
81+
cd linkis-x.x.x
82+
MVN -N Install
83+
MVN CLEAN Install -DSKIPTESTS
84+
```
85+
86+
#### Compile Error
87+
88+
-If when you compile it, there is an error, try to enter a module alone to compile, see if there are errors, and adjust according to specific errors
89+
-Since the SCALA language is used in Linkis for code writing, it is recommended that you can configure the SCALA environment first to facilitate reading the source code
90+
-Aar package conflict is the most common problem, especially after upgrading Hadoop, please adjust the dependent version patiently
91+
92+
#### DatasphereStudio pom file
93+
94+
As we upgrade the version of Scala, the error will be reported when deploying.
95+
Conn to BML Now Exit Java.net.socketexception: Connection Reset. Here you need to modify the SCALA version and re -compile.
96+
1. Delete the low version of the DSS-Gateway-SUPPPPORT JAR package,
97+
2. Modify the scala version in DSS 1.1.0 to 2.12, compile it, get the new DSS-Gateway-SUPPPORT -.1.0.JAR, replace the linkis_installhome/lib/linkis-spaint-service/linkis-mg-gateway The original jar package of the Central Plains
98+
```xml
99+
<!-The SCALA environment is consistent->
100+
<scala.version> 2.12.15 </scala.version>
101+
```
102+
According to the adjustment of the dependent version above, you can solve most of the problems. If you have any problems, you need to carefully adjust the corresponding log.
103+
If a complete package can be compiled, it represents the full compilation of Linkis and can be deployed.
104+
105+
## deployment
106+
107+
- In in order to allow the engine node to have sufficient resource execution script, we have adopted multiple server deployments, and the general deployment structure is as follows.
108+
-SLB 1 load balancing is rotary inquiry
109+
-E ECS-WEB 2 Nginx, static resource deployment, background agent forwarding
110+
-ECS-APP 2 micro-service governance, computing governance, public enhancement and other node deployment
111+
-ECS-APP 4 Engineconnmanager node deployment
112+
113+
### linkis deployment
114+
115+
- At the use of multiple node deployments, we did not peel the code, or put the full amount on the server, but just modified the startup script to make it only start the service required
116+
117+
Refer to the official website single machine deployment example: https: //linkis.apache.org/zh-docs/1.3.0/dePlayment/dePlay-qick
118+
119+
#### Linkis Deployment Points Attention Point
120+
- 1. Deployment user: The startup user of the core process of Linkis. At the same time, this user will default as an administrator permissions. During the deployment process, the corresponding administrator login password, located in the linkis support specified in CONF/LINKIS-MG-Gateway.properties file file Submitted and executed users. The main process service of Linkis will be switched to the corresponding user through the SUDO -U $ {linkis-user}, and then executes the corresponding engine startup command, so the user of the engine linkis -ngine processes is the executor of the task.
121+
-The user defaults to the submission and executor of the task, if you want to change to the login user, you need to modify
122+
org.apache.linkis.entRance.Restful.entRANCERESTFAPI class
123+
json.put (taskConstant.execute_user, moduleuseuserutills.GetOperationUser (REQ));
124+
json.put (taskConstant.submit_user, SecurityFilter.getLoginusername (REQ));
125+
Change the above settings to the user and execute user to the Scriptis page to log in to the user
126+
- 2. Sudo -U $ {linkis -user} Switch to the corresponding user. If you use the login user, this command may fail, and you need to modify the command here.
127+
- org.apache.linkis.ecm.server.operator.EngineConnYarnLogOperator.sudoCommands
128+
```scala
129+
private def sudoCommands(creator: String, command: String): Array[String] = {
130+
Array(
131+
"/bin/bash",
132+
"-c",
133+
"sudo su " + creator + " -c \"source ~/.bashrc 2>/dev/null; " + command + "\""
134+
)
135+
} change into
136+
private def sudoCommands(creator: String, command: String): Array[String] = {
137+
Array(
138+
"/bin/bash",
139+
"-c",
140+
"\"source ~/.bashrc 2>/dev/null; " + command + "\""
141+
)
142+
}
143+
```
144+
- 3. Mysql's driver package must be Copy to/lib/linkis-commons/public-module/and/lib/linkis-spring-cloud-services/linkis-mg-gateway/
145+
146+
- 4. The default is to use static users and passwords. Static users are deploying users. Static passwords will generate a password string in execution deployment, stored at $ {linkis_home} /conf/linkis-mg-gateway.properties
147+
148+
- 5. database script execution, linkis itself needs to use the database, but when we execute the script of the inserted data of Linkis 1.3.0, we found an error. We directly deleted the data of the error part at that time.
149+
150+
- 6. Yarn's certification. When performing the spark task, the task will be submitted to the queue. The resource information of the queue will be obtained first to determine whether there is a resource to submit.
151+
For certification, if the file authentication is enabled, the file needs to be placed in the corresponding directory of the server, and the information is updated in the linkis_cg_rm_extRNAL_Resource_Provider library table.
152+
153+
### Install web front end
154+
- WEB side uses nginx as a static resource server, download the front -end installation package and decompress it, and place it on the directory corresponding to the Nginx server
155+
156+
### scriptis tool installation
157+
- Scriptis is a pure front -end project. As a component integrates in the web code component of DSS, we only need to compile the DSSWEB project for separate Scriptis modules, upload the compiled static resources to Visit, note: Linkis stand -by -machine deployment defaults to use session for verification. You need to log in to the Linkis management desk first, and then log in to Scriptis to use.
158+
159+
## Nginx deployment for example
160+
#### nginx.conf
161+
```
162+
upstream linkisServer{
163+
server ip:port;
164+
server ip:port;
165+
}
166+
Server {
167+
Listen 8088;# Access port
168+
Server_name localhost;
169+
#Charset Koi8-R;
170+
#access_log /var/log/nginx/host.access.log main;
171+
#Scriptis static resources
172+
local /scriptis {
173+
# Modify to your own front path
174+
alias/home/nginx/scriptis-weight; # static file directory
175+
#Root/Home/Hadoop/DSS/Web/DSS/Linkis;
176+
index index.html index.html;
177+
}
178+
#The default resource path points to the static resource of the front end of the management platform
179+
location / {
180+
# Modify to your own front path
181+
root/Home/Nginx/Linkis-Web/DIST; # r r r r
182+
#Root/Home/Hadoop/DSS/Web/DSS/Linkis;
183+
index index.html index.html;
184+
}
185+
186+
local /ws {
187+
Proxy_pass http:// linkisserver/api #back -end linkis address
188+
proxy_http_version 1.1;
189+
proxy_set_header upgrade $ http_upgrade;
190+
proxy_set_header connection upgrade;
191+
}
192+
193+
location /api {
194+
proxy_pass http:// linkisserver/api; #The address of the back end linkis
195+
proxy_set_header Host $host;
196+
proxy_set_header X-Real-IP $remote_addr;
197+
proxy_set_header x_real_ipP $remote_addr;
198+
proxy_set_header remote_addr $remote_addr;
199+
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
200+
proxy_http_version 1.1;
201+
proxy_connect_timeout 4s;
202+
proxy_read_timeout 600s;
203+
proxy_send_timeout 12s;
204+
proxy_set_header Upgrade $http_upgrade;
205+
proxy_set_header Connection upgrade;
206+
}
207+
208+
#error_page 404 /404.html;
209+
# redirect server error pages to the static page /50x.html
210+
#
211+
error_page 500 502 503 504 /50x.html;
212+
location = /50x.html {
213+
root /usr/share/nginx/html;
214+
}
215+
}
216+
217+
```
218+
## How to check the question
219+
- 1. There are more than 100 modules in Linkis, and the final service has 7 services, which are linkis-cg -ngineconnmanager, linkis-cg -ngineplugin, linkis-cg-entrance, linkis-cg-linkisManager,
220+
Linkis-Mg-Gateway, Linkis-Mg-Eureka, Linkis-PS-PublicService, each module has this different features. Among them, Linkis-CG-ENGINECONNMANAGER is responsible for managing the start-engine service, which will generate the corresponding engine script to pull up the engine. Services, so our team launched the Linkis-CG-ENGINECONNMANAGER alone on the server for sufficient resources to execute on the server.
221+
- 2. The execution of engines like JDBC, Spark.hedu and other engines require some JAR package support. When the linkis species is called material, these jar packs will be hit in the linkis-cg-oblmphin engine when packaging , Conf and LIB directory will appear. When starting this service, two packages will be uploaded to the configuration directory, which will generate two ZIP files. We use OSS to store these material information. Download it to the Linkis-CG-ENGINECONNMANAGER service, and then configure the following two configurations of wds.linkis.enginecoon.public.dir and wds.linkis.enginecoon.root.dir, then the bag will be pulled to WDS. Linkis.engineCoon.public.dir is the directory of wds.linkis.enginecoon.root.dir. .dir.
222+
- 3. If you want to check the engine log, you can see the directory under wds.linkis.enginecoon.root.dir configuration. Of course, the log information will be displayed on the log of the scriptis page. Just paste it to find it.
223+
224+
225+
226+
227+
228+

‎blog/authors.yml

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,8 +34,14 @@ ruY9527:
3434
url: https://github.com/ruY9527
3535
image_url: https://avatars.githubusercontent.com/u/43773582?v=4
3636

37+
livi12138:
38+
name: livi12138
39+
title: contributors
40+
url: https://github.com/livi12138
41+
image_url: https://avatars.githubusercontent.com/u/156271765?v=4
42+
3743
kevinWdong:
3844
name: kevinWdong
3945
title: contributors
4046
url: https://github.com/kongslove
41-
image_url: https://avatars.githubusercontent.com/u/42604208?v=4
47+
image_url: https://avatars.githubusercontent.com/u/42604208?v=4

0 commit comments

Comments
 (0)