HIVE-28983: Log HS2 and HMS PID and update hive-env.sh template #5884

Aggarwal-Raghav · 2025-06-21T08:38:30Z

What changes were proposed in this pull request?

Log the HS2 and HMS PID in their respective logs and give sample hms and hs2 HADOOP_OPTS in hive-env.sh.template.

Why are the changes needed?

During debugging prod issue, I saw the need for this.

If HMS/HS2 is crashing periodically then the heapdump will have the name in default format i.e. java_pid14784.hprof, it becomes difficult to identify whether it belongs to HMS or HS2 without loading it in any analyzer tool.
To get the gc logs for a particular HS2 or HMS crash requires extraction of timestamp from logs using text manipulation commands etc.

Does this PR introduce any user-facing change?

Yes, in HS2 and HMS logs PID will be logged

How was this patch tested?

On local setup

Aggarwal-Raghav · 2025-06-21T08:39:30Z

For example:

I can easily tell that PID 14784 belongs to HS2 heap dump and can easily fetch gc logs for that PID easily.

Aggarwal-Raghav · 2025-06-21T08:41:20Z

@ayushtkn @deniskuzZ , I would like to get you opinion on this small improvement patch. I have done this in our internal hive distribution and has helped a lot.

Aggarwal-Raghav · 2025-06-21T08:44:45Z

HMS log will look like

2025-06-21 13:00:24,689 INFO metastore.HiveMetaStore: Starting hive metastore on port 9083. PID is 7951

HS2 log will look like

2025-06-21 13:01:04,646 INFO server.HiveServer2: Starting HiveServer2. PID is: 8083

deniskuzZ · 2025-06-21T09:20:05Z

service/src/java/org/apache/hive/service/server/HiveServer2.java

@@ -1206,7 +1206,8 @@ public void startPrivilegeSynchronizer(HiveConf hiveConf) throws Exception {
  private static void startHiveServer2() throws Throwable {
    long attempts = 0, maxAttempts = 1;
    while (true) {
-      LOG.info("Starting HiveServer2");
+      long pid = ProcessHandle.current().pid();


Do we need a separate var for pid extraction?

We can inline it in LOG message in both HS2 and HMS, if you think that's best.

not sure, I am just used to inline when the var is not used in multiple places

okumin · 2025-06-22T05:52:39Z

conf/hive-env.sh.template

 #   if [ -z "$DEBUG" ]; then
-#     export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -Xms10m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:+UseParNewGC -XX:-UseGCOverheadLimit"
+#     export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -Xms10m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:+G1GC -XX:-UseGCOverheadLimit"


Can we remove -XX:+G1GC since it's the default with JDK 17+? I also see -XX:NewRatio is not recommended. However, as I don't know the intention of the DEBUG option, we may keep it.

-XX:+UseG1GC is the correct one?

Thanks for the review @okumin. Few points:

Yes -XX:+UseG1GC is the correct one. I doubled confirmed using:
java -XX:+UnlockExperimentalVMOptions -XX:+PrintFlagsFinal -version | grep -i "G1GC"

Regarding removing the GC flags and NewRatio, I am inclined towards removing every flag other than Xmx and Xms. Reason being, G1GC and ZGC doesn't require much tuning and tuning should be system specific + workload specifc which varies cluster to cluster.

I "want to believe" that the DEBUG option is there to showcase user the possibilities / kind of tuning users can do.

Let me know your thoughts based on that I will update the PR.

sorry, just to confirm, -XX:+G1GC was replaced with XX:+UseG1GC ? if yes, we still have -XX:+G1GC in beeline HADOOP_OPTS and need to replace

Addressed in ba2c9ca

...metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java

deniskuzZ

+1, pending tests

zhangbutao · 2025-06-27T14:28:17Z

conf/hive-env.sh.template

-# export HIVE_AUX_JARS_PATH=
+# if [ "$SERVICE" = "metastore" ]; then
+#   export HADOOP_HEAPSIZE=1024
+#   export HADOOP_OPTS="$HADOOP_OPTS -Dhive.log.dir=$HIVE_LOG_DIR -Dhive.log.file=hive$SERVICE.log \


@Aggarwal-Raghav Have you did the test using the latest master code?
I found that HADOOP_OPTS cannot make the Hive log directory take effect. It was only after I changed to HADOOP_CLIENT_OPTS that it worked. However, it was indeed possible to take effect by using HADOOP_OPTS in the past.

The screenshot shows that I deployed Hive 4.1.0 locally. I had to modify HADOOP_CLIENT_OPTS to make the log directory effective.

@zhangbutao , thanks for looking into this and yes, you are correct. Even in my setup I have HADOOP_CLIENT_OPTS for quite some time:
https://github.com/Aggarwal-Raghav/hive-mac-setup/blob/main/hive/hive-env.sh

But reason why I went ahead with HADOOP_OPTS:

In my prod clusters, we have HADOOP_OPTS set in hive-env.sh in Amabri UI and it's working as expected.

Even in Ambari opensource, HADOOP_OPTS is used:
https://github.com/apache/ambari/blob/trunk/ambari-server/src/main/resources/stacks/BIGTOP/3.2.0/services/HIVE/configuration/hive-env.xml#L376

Because of point [1] and [2], I stick to HADOOP_OPTS which are also present in older hive-env.sh.template in hive,

Let me know what are your thoughts on this? Should I change to HADOOP_CLIENT_OPTS in hive-env.sh.template?

I just want to confirm whether HADOOP_OPTS will make the Hive directory configuration take effect. Because it didn't seem to work in my test environment using latest branch, then I switched to using HADOOP_CLIENT_OPTS.

No, I also can't make HADOOP_OPTS work.

Sorry, I'd like to confirm again to avoid any misunderstanding due to my unclear description.

When I was testing the latest&branch-4.1 code, I found that the hive log parameters configured with HADOOP_OPTS (Dhive.log.dir=$HIVE_LOG_DIR -Dhive.log.file=hive$SERVICE.log) were not taking effect, and I didn't see any log file in the specified hive.log.dir. Only when I replaced HADOOP_OPTS with HADOOP_CLIENT_OPTS did this log directory take effect.

But I'm not sure where the problem lies that causes this situation, because before this, I was able to make the hive.log.dir parameter take effect by using HADOOP_OPTS.

So, if your current test results are the same as mine, it might be a problem caused by changes in the Hive code. If this issue is real, it would be best to fix it on the Hive side. Otherwise, similar downstream components like Ambari might have to modify their code to adapt to the new HADOOP_CLIENT_OPTS parameter, which could be quite troublesome.

Aggarwal-Raghav · 2025-06-27T19:20:59Z

Moving from HADOOP_OPTS => HADOOP_CLIENT_OPTS in hive-env.sh.template.
Reason:

JDK17 --add-opens changes were made in HADOOP_CLIENT_OPTS and same is used in Jenkinsfile as well.

@deniskuzZ @zhangbutao , can you please review once again?

sonarqubecloud · 2025-06-28T00:25:20Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

zhangbutao · 2025-06-28T01:32:41Z

Moving from HADOOP_OPTS => HADOOP_CLIENT_OPTS in hive-env.sh.template. Reason:

JDK17 --add-opens changes were made in HADOOP_CLIENT_OPTS and same is used in Jenkinsfile as well.

@deniskuzZ @zhangbutao , can you please review once again?

If this is the issue, could we add --add-opens to the HADOOP_OPTS setting to fix this problem?

hive/bin/ext/hiveserver2.sh

Lines 53 to 54 in 925df92

    
           export HADOOP_CLIENT_OPTS=" -Dproc_hiveserver2 --add-opens java.base/java.nio=ALL-UNNAMED --add-opens java.base/java.net=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED  --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.util.concurrent=ALL-UNNAMED --add-opens java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens java.base/java.util.regex=ALL-UNNAMED --add-opens java.base/java.lang.reflect=ALL-UNNAMED --add-opens java.base/java.io=ALL-UNNAMED $HADOOP_CLIENT_OPTS " 
        
           export HADOOP_OPTS="$HIVESERVER2_HADOOP_OPTS $HADOOP_OPTS"

cc @tanishq-chugh @ayushtkn

zhangbutao · 2025-06-28T01:57:17Z

Moving from HADOOP_OPTS => HADOOP_CLIENT_OPTS in hive-env.sh.template. Reason:

JDK17 --add-opens changes were made in HADOOP_CLIENT_OPTS and same is used in Jenkinsfile as well.

@deniskuzZ @zhangbutao , can you please review once again?

diff --git a/bin/ext/hiveserver2.sh b/bin/ext/hiveserver2.sh
index 3431f7de17..c45031b982 100644
--- a/bin/ext/hiveserver2.sh
+++ b/bin/ext/hiveserver2.sh
@@ -51,7 +51,7 @@ hiveserver2() {
     killAndWait $pid $timeout
   else
     export HADOOP_CLIENT_OPTS=" -Dproc_hiveserver2 --add-opens java.base/java.nio=ALL-UNNAMED --add-opens java.base/java.net=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED  --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.util.concurrent=ALL-UNNAMED --add-opens java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens java.base/java.util.regex=ALL-UNNAMED --add-opens java.base/java.lang.reflect=ALL-UNNAMED --add-opens java.base/java.io=ALL-UNNAMED $HADOOP_CLIENT_OPTS "
-    export HADOOP_OPTS="$HIVESERVER2_HADOOP_OPTS $HADOOP_OPTS"
+    export HADOOP_OPTS="$HIVESERVER2_HADOOP_OPTS --add-opens java.base/java.nio=ALL-UNNAMED --add-opens java.base/java.net=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED  --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.util.concurrent=ALL-UNNAMED --add-opens java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens java.base/java.util.regex=ALL-UNNAMED --add-opens java.base/java.lang.reflect=ALL-UNNAMED --add-opens java.base/java.io=ALL-UNNAMED $HADOOP_OPTS"
     commands=$(exec $HADOOP jar $JAR $CLASS -H | grep -v '-hiveconf' | awk '{print $1}')
     start_hiveserver2='Y'
     for i in "$@"; do
diff --git a/bin/ext/metastore.sh b/bin/ext/metastore.sh
index 251013b2bb..8b646d29e8 100644
--- a/bin/ext/metastore.sh
+++ b/bin/ext/metastore.sh
@@ -28,7 +28,8 @@ metastore() {
   # Append --add-opens args that is required for JDK-17
   export HADOOP_CLIENT_OPTS=" -Dproc_metastore --add-opens java.base/java.nio=ALL-UNNAMED --add-opens java.base/java.net=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED  --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.util.concurrent=ALL-UNNAMED --add-opens java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/java.util.regex=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED $HADOOP_CLIENT_OPTS "

-  export HADOOP_OPTS="$HIVE_METASTORE_HADOOP_OPTS $HADOOP_OPTS"
+  export HADOOP_OPTS="$HIVE_METASTORE_HADOOP_OPTS --add-opens java.base/java.nio=ALL-UNNAMED --add-opens java.base/java.net=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED  --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.util.concurrent=ALL-UNNAMED --add-opens java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/java.util.regex=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED $HADOOP_OPTS"
   exec $HADOOP jar $JAR $CLASS "$@"
 }

I did this change, and using HADOOP_OPTS in hive-env.sh to specify hive.log.dir, but it still didn't work.

Aggarwal-Raghav · 2025-06-28T05:04:27Z

@zhangbutao , the patch that you have sent above is not what I meant regarding add-opens. Let me rephrase, It seems HADOOP_OPTS are not picked up for starting hive services no matter where you set them i.e. bin/hive.sh, bin/hive-config.sh,bin/ext/hiveserver2.sh, etc or atleast I couldn't make them work. Confirmed from ps -ef | grep -i hiveserver2 output.

In HIVE-26473, all the add-opens were appended in HADOOP_CLIENT_OPTS, that's why I changed from HADOOP_OPTS => HADOOP_CLIENT_OPTS in hive-env.sh.template file.

zhangbutao · 2025-06-28T06:53:28Z

@zhangbutao , the patch that you have sent above is not what I meant regarding add-opens. Let me rephrase, It seems HADOOP_OPTS are not picked up for starting hive services no matter where you set them i.e. bin/hive.sh, bin/hive-config.sh,bin/ext/hiveserver2.sh, etc or atleast I couldn't make them work. Confirmed from ps -ef | grep -i hiveserver2 output.

In HIVE-26473, all the add-opens were appended in HADOOP_CLIENT_OPTS, that's why I changed from HADOOP_OPTS => HADOOP_CLIENT_OPTS in hive-env.sh.template file.

Yes, I understand what you mean.
I just don't understand why HADOOP_OPTS no longer takes effect after HIVE-26473 was merged.
For me, using HADOOP_CLIENT_OPTS at present is acceptable, but I think, in the future, there might be users who are confused about the issue of HADOOP_OPTS being ineffective.

Maybe we can address this issue later.

Aggarwal-Raghav · 2025-06-28T06:56:07Z

I just don't understand why HADOOP_OPTS no longer takes effect after HIVE-26473 was merged.

Just to clarify, i think it was not working way before HIVE-26473.

zhangbutao · 2025-06-28T07:00:53Z

I just don't understand why HADOOP_OPTS no longer takes effect after HIVE-26473 was merged.

Just to clarify, i think it was not working way before HIVE-26473.

OK. It might also be a issue caused by other PR. Because previously, when I conducted local tests, using HAODOOP_OPTS was always effective.

zhangbutao · 2025-06-28T08:16:39Z

@Aggarwal-Raghav I have just figured out the reason why the HADOOP_OPTS in my hive-env.sh file is ineffective.

Because in order for Hadoop 3.4.1 to run on JDK 17, I configured the HADOOP_OPTS parameter in the hadoop-env.sh file:
export HADOOP_OPTS="--add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/sun.net.util=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.lang.reflect=ALL-UNNAMED"

So when Hive starts, the value of the HADOOP_OPTS parameter in hive-env.sh is always overwritten by the parameter value in hadoop-env.sh.

Therefore, in order to prevent the parameters in hive-env.sh from being overridden by those in hadoop-env.sh, we can specify a clean hadoop-env.sh file for hive (a separate hadoop client for hive)

Back to this current PR, regarding whether we should change the HADOOP_OPTS in the hive-env.sh template to HADOOP_CLIENT_OPTS. My answer is yes.
See HIVE-19886

hive/service/src/java/org/apache/hive/service/server/HiveServer2.java

Lines 1469 to 1480 in 925df92

    
           for (String propKey : confProps.stringPropertyNames()) { 
        
             // save logging message for log4j output latter after log4j initialize properly 
        
             debugMessage.append("Setting " + propKey + "=" + confProps.getProperty(propKey) + ";\n"); 
        
             if ("hive.log.file".equals(propKey) || 
        
                 "hive.log.dir".equals(propKey) || 
        
                 "hive.root.logger".equals(propKey)) { 
        
               throw new IllegalArgumentException("Logs will be split in two " 
        
                   + "files if the commandline argument " + propKey + " is " 
        
                   + "used. To prevent this use to HADOOP_CLIENT_OPTS -D" 
        
                   + propKey + "=" + confProps.getProperty(propKey) 
        
                   + " or use the set the value in the configuration file" 
        
                   + " (see HIVE-19886)");

But I also want to hear the opinions of the other folks. :)

Thanks.

zhangbutao

+1 LGTM

zhangbutao · 2025-07-07T14:44:52Z

@okumin @deniskuzZ Thanks for the reivew!

HIVE-28983: Log HS2 and HMS PID and update hive-env.sh template

f7e2bc3

asf-ci-hive added the tests pending label Jun 21, 2025

deniskuzZ reviewed Jun 21, 2025

View reviewed changes

asf-ci-hive added tests passed and removed tests pending labels Jun 21, 2025

okumin reviewed Jun 22, 2025

View reviewed changes

Address review comment

ba2c9ca

asf-ci-hive added tests pending tests passed and removed tests passed tests pending labels Jun 23, 2025

deniskuzZ reviewed Jun 24, 2025

View reviewed changes

...metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java Show resolved Hide resolved

Review comment 2

311ad48

asf-ci-hive added tests pending tests unstable and removed tests passed tests pending labels Jun 26, 2025

deniskuzZ approved these changes Jun 27, 2025

View reviewed changes

asf-ci-hive added tests pending and removed tests unstable labels Jun 27, 2025

zhangbutao reviewed Jun 27, 2025

View reviewed changes

Move to HADOOP_CLIENT_OPTS

785603d

asf-ci-hive added tests unstable tests pending and removed tests pending tests unstable labels Jun 27, 2025

asf-ci-hive added tests passed and removed tests pending labels Jun 28, 2025

zhangbutao approved these changes Jul 3, 2025

View reviewed changes

zhangbutao merged commit 4ada928 into apache:master Jul 7, 2025
4 checks passed

HIVE-28983: Log HS2 and HMS PID and update hive-env.sh template #5884

HIVE-28983: Log HS2 and HMS PID and update hive-env.sh template #5884

Uh oh!

Conversation

Aggarwal-Raghav commented Jun 21, 2025

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

Aggarwal-Raghav commented Jun 21, 2025

Uh oh!

Aggarwal-Raghav commented Jun 21, 2025

Uh oh!

Aggarwal-Raghav commented Jun 21, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deniskuzZ Jun 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

deniskuzZ left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Aggarwal-Raghav commented Jun 27, 2025

Uh oh!

sonarqubecloud bot commented Jun 28, 2025

Quality Gate passed

Uh oh!

zhangbutao commented Jun 28, 2025

Uh oh!

zhangbutao commented Jun 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Aggarwal-Raghav commented Jun 28, 2025

Uh oh!

zhangbutao commented Jun 28, 2025

Uh oh!

Aggarwal-Raghav commented Jun 28, 2025

Uh oh!

zhangbutao commented Jun 28, 2025

Uh oh!

zhangbutao commented Jun 28, 2025

Uh oh!

zhangbutao left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

zhangbutao commented Jul 7, 2025

Uh oh!

Uh oh!

deniskuzZ Jun 23, 2025 •

edited

Loading

deniskuzZ left a comment •

edited

Loading

zhangbutao commented Jun 28, 2025 •

edited

Loading