HIVE-28655: Implement HMS Related Drop Stats Changes Part2 (param COLUMN_STAT_ACCURATE related changes) #5790

DanielZhu58 · 2025-04-28T23:21:21Z

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

dengzhhu653 · 2025-05-06T07:50:53Z

...ne-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java

@@ -7431,6 +7431,13 @@ public boolean delete_column_statistics_req(DeleteColumnStatisticsRequest req) t
      if (!isPartitioned || req.isTableLevel()) {
        ret = rawStore.deleteTableColumnStatistics(parsedDbName[CAT_NAME], parsedDbName[DB_NAME], tableName, colNames, engine);
        if (ret) {
+          Map<String, String> parameters = table.getParameters();


can we just remove the deleted columns from the COLUMN_STATS_ACCURATE? using the StatsSetupConst.removeColumnStatsState

dengzhhu653 · 2025-05-06T07:51:37Z

...ne-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java

+              }
+              partParams.put("COLUMN_STATS_ACCURATE", "false");
+              partition.setParameters(partParams);
+              rawStore.alterPartition(parsedDbName[CAT_NAME], parsedDbName[DB_NAME], tableName, partVals, partition, null);


alter the partitions in bulk?

...e-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java

...ne-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java

dengzhhu653 · 2025-05-21T05:00:28Z

...ne-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java

+          } else {
+            // remove the deleted column names in parameter COLUMN_STATS_ACCURATE
+            StatsSetupConst.removeColumnStatsState(table.getParameters(), colNames);
+          }


Need to persist the changed parameters by rawStore.alterTable()

dengzhhu653 · 2025-05-21T05:01:41Z

...ne-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java

+              } else {
+                // remove the deleted column names in parameter COLUMN_STATS_ACCURATE
+                StatsSetupConst.removeColumnStatsState(partParams, colNames);
+              }


need to persist the partition parameters after resetting the column stats

dengzhhu653 · 2025-05-22T08:14:12Z

...e-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java

@@ -10363,6 +10365,8 @@ private boolean deleteTableColumnStatisticsViaJdo(String catName, String dbName,
      }
      mStatsObjColl = (List<MTableColumnStatistics>) query.executeWithArray(params.toArray());
      pm.retrieveAll(mStatsObjColl);
+      Long tableID = directSql.getTableId(dbName, tableName);


is there any reason to mix the direct sql with the jdo together? this deleteTableColumnStatisticsViaJdo is for jdo query I guess. And we need to take care of the same in getSqlResult path.

...tore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java

…AT_ACCURATE After Dropping

sonarqubecloud · 2025-06-17T08:33:10Z

Quality Gate passed

Issues
5 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

dengzhhu653 · 2025-06-18T10:16:57Z

any ideas? cc @saihemanth-cloudera @soumyakanti3578

soumyakanti3578 · 2025-06-18T18:52:41Z

ql/src/test/results/clientpositive/llap/alter_table_column_stats.q.out

@@ -1105,7 +1105,6 @@ Retention:          	0
 #### A masked pattern was here ####
 Table Type:         	MANAGED_TABLE       	 
 Table Parameters:	 	 
-	COLUMN_STATS_ACCURATE	{\"BASIC_STATS\":\"true\"}


I see that we are removing COLUMN_STATS_ACCURATE here, however, in the corresponding q file, I see this comment:

-- rename a partition should not change its table, partition, and column stats alter table statsdb1.testpart1 partition (part = 'part1') rename to partition (part = 'part11'); describe formatted statsdb1.testpart1;

I just want us to be sure that this is what we intend to do as I see several other similar changes just below which were due to renaming/replacing columns. And if the tests don't make sense any more maybe we should consider updating the tests.

soumyakanti3578 · 2025-06-18T18:54:05Z

ql/src/test/results/clientpositive/llap/alter_table_column_stats.q.out

@@ -1476,7 +1471,7 @@ Database:           	statsdb1
 Table:              	testpart1           	 
 #### A masked pattern was here ####
 Partition Parameters:	 	 
-	COLUMN_STATS_ACCURATE	{\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"col1\":\"true\",\"col2\":\"true\"}}
+	COLUMN_STATS_ACCURATE	{}                  


Here we are printing empty COLUMN_STATS_ACCURATE. If it is easy to remove them, maybe we should do that?

asf-ci-hive added tests pending tests unstable and removed tests pending labels Apr 28, 2025

dengzhhu653 reviewed May 6, 2025

View reviewed changes

DanielZhu58 force-pushed the HIVE-28655-part2 branch from 0c0499d to 2318bb5 Compare May 7, 2025 08:59

asf-ci-hive added tests pending tests failed and removed tests unstable tests pending labels May 7, 2025

wecharyu reviewed May 12, 2025

View reviewed changes

...e-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java Outdated Show resolved Hide resolved

...ne-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java Outdated Show resolved Hide resolved

DanielZhu58 force-pushed the HIVE-28655-part2 branch from 2318bb5 to 60d8cf3 Compare May 20, 2025 08:55

asf-ci-hive added tests pending tests failed and removed tests failed tests pending labels May 20, 2025

DanielZhu58 force-pushed the HIVE-28655-part2 branch from 60d8cf3 to 882007d Compare May 21, 2025 00:45

asf-ci-hive added tests pending tests unstable and removed tests failed tests pending labels May 21, 2025

dengzhhu653 reviewed May 21, 2025

View reviewed changes

DanielZhu58 force-pushed the HIVE-28655-part2 branch from 882007d to 4b54b61 Compare May 22, 2025 07:26

asf-ci-hive added tests pending tests failed and removed tests unstable tests pending labels May 22, 2025

dengzhhu653 reviewed May 22, 2025

View reviewed changes

...tore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java Outdated Show resolved Hide resolved

asf-ci-hive added tests pending and removed tests unstable labels Jun 4, 2025

DanielZhu58 force-pushed the HIVE-28655-part2 branch from d0afa49 to 0e31251 Compare June 4, 2025 09:51

asf-ci-hive added tests failed tests pending and removed tests pending tests failed labels Jun 4, 2025

DanielZhu58 force-pushed the HIVE-28655-part2 branch from 0e31251 to fe37625 Compare June 11, 2025 10:14

asf-ci-hive added tests pending tests failed and removed tests failed tests pending labels Jun 11, 2025

DanielZhu58 force-pushed the HIVE-28655-part2 branch from fe37625 to 76ac5c5 Compare June 15, 2025 10:51

asf-ci-hive added tests pending tests unstable and removed tests failed tests pending labels Jun 15, 2025

HIVE-28655: Implement HMS Related Drop Stats Changes, Reset COLUMN_ST…

66640df

…AT_ACCURATE After Dropping

DanielZhu58 force-pushed the HIVE-28655-part2 branch from 76ac5c5 to 66640df Compare June 17, 2025 06:30

asf-ci-hive added tests pending and removed tests unstable labels Jun 17, 2025

asf-ci-hive added tests passed and removed tests pending labels Jun 17, 2025

DanielZhu58 requested review from dengzhhu653 and wecharyu June 17, 2025 18:49

soumyakanti3578 reviewed Jun 18, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HIVE-28655: Implement HMS Related Drop Stats Changes Part2 (param COLUMN_STAT_ACCURATE related changes) #5790

HIVE-28655: Implement HMS Related Drop Stats Changes Part2 (param COLUMN_STAT_ACCURATE related changes) #5790

DanielZhu58 commented Apr 28, 2025

Uh oh!

dengzhhu653 May 6, 2025

Uh oh!

dengzhhu653 May 6, 2025

Uh oh!

Uh oh!

Uh oh!

dengzhhu653 May 21, 2025

Uh oh!

dengzhhu653 May 21, 2025

Uh oh!

dengzhhu653 May 22, 2025 •

edited

Loading

Uh oh!

Uh oh!

sonarqubecloud bot commented Jun 17, 2025

Uh oh!

dengzhhu653 commented Jun 18, 2025

Uh oh!

soumyakanti3578 Jun 18, 2025 •

edited

Loading

Uh oh!

soumyakanti3578 Jun 18, 2025

Uh oh!

Uh oh!

HIVE-28655: Implement HMS Related Drop Stats Changes Part2 (param COLUMN_STAT_ACCURATE related changes) #5790

Are you sure you want to change the base?

HIVE-28655: Implement HMS Related Drop Stats Changes Part2 (param COLUMN_STAT_ACCURATE related changes) #5790

Conversation

DanielZhu58 commented Apr 28, 2025

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

dengzhhu653 May 6, 2025

Choose a reason for hiding this comment

Uh oh!

dengzhhu653 May 6, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

dengzhhu653 May 21, 2025

Choose a reason for hiding this comment

Uh oh!

dengzhhu653 May 21, 2025

Choose a reason for hiding this comment

Uh oh!

dengzhhu653 May 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sonarqubecloud bot commented Jun 17, 2025

Quality Gate passed

Uh oh!

dengzhhu653 commented Jun 18, 2025

Uh oh!

soumyakanti3578 Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

soumyakanti3578 Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dengzhhu653 May 22, 2025 •

edited

Loading

soumyakanti3578 Jun 18, 2025 •

edited

Loading