Genie OSS Spring Boot 3 Upgrade (#1229)

* Upgrade genie-common-internal to io.awspring.cloud:spring-cloud-aws-dependencies: 3.2.1

* upgrade genie-agent and genie-web to spring boot 3

* change the org.springframework.boot to 3.4.2

* move the subprojects block to the configure block

* upgrade genie-common-internal to JDK 17

* Revert "upgrade genie-common-internal to JDK 17"

This reverts commit 3227057.

* replace javax with jakarta

* replace jakarta with javax

* Fix status code in TitusAgentLauncherImpl

* javadoc and other minor fixes

* Remove TaskExecutorCustomizer&TaskSchedulerCustomizer in favor of ThreadLocalTaskExecutorCustomizer&ThreadLocalTaskSchedulerCustomizer

* move to new autoconfiguration registration

* enable debug mode to see autoconfiguration conditions report

* Fix genie-agent-app smoke tests

* Fix genie-app smoke test unsupported classes and methods

Fix genie-agent smoke test unsupported classes and methods

* Fix hibernate issues

* Move to new autoconfiguration registration for genie-app smoke test

* Upgrade to AWS SDK v2

* Fix threadSafe dependency

* Fix DigestUtils dependency

* Fix EC2MetadataUtils dependency

* getHostname by checking AWS environment

* Upgrade AwsAutoConfiguration

* Upgrade S3ProtocolResolverRegistrar

* Upgrade S3JobArchiverImpl

* S3TransferManagerFactory for creating and managing S3TransferManager instances with S3AsyncClient

* use S3TransferManagerFactory in S3JobArchiverImpl for sync uploading

* Fix ArgumentValidators

* Upgrade httpclient and commons-codec

* Upgrade AWSAutoConfiguration and RetryProperties for AWS SNS

* Amend ArgumentValidators

* remove unused imports

* Amend AwsAutoConfiguration

* Remove used imports

* Amend AwsAutoConfiguration

* Fix region not found error for GenieAgentApplicationSmokeTest

* dependecy updates with spring-boot-starter-web

* fix region not found error in GenieAgentApplicationSmokeTest

* fix sharedAgentTaskExecutor in AgentAutoConfiguration

* fix failures in GenieAppSmokeTest

* cleanup / refactor unused code

* fix setRegion in BucketProperties

* update BucketPropertiesSpec

* update S3ProtocolResolverSpec

* create S3TransferManagerFactorySpec

* update S3ClientFactory and S3ClientFactorySpec

* update SimpleStorageRangeResourceSpec

* remove S3ResourceLoaderProperties as it is not needed

* use different Spock BOM version for Spring Boot 2 and 3 modules to avoid compatibility issues with JUnit4

* Fix S3ProtocolResolverRegistrar and S3ProtocolResolverRegistrarSpec.groovy

* Fix S3JobArchiverImplSpec.groovy

* Fix AwsAutoConfigurationSpec.groovy

* Fix AwsAutoConfigurationTest

* Fix CommonServicesAutoConfigurationTest

* Fix AwsAutoConfigurationTest

* Fix :genie-common-internal:checkstyleMain

* Fix UserRestControllerSpec.groovy

* Fix AWSAutoConfigurationTest

* Fix GenieExceptionMapperTest

* Fix genie-web groovy test

* Fix Snippets

* Fix JobStateChangeSNSPublisherSpec.groovy

* Fix GenieCpuHealthIndicatorSpec.groovy

* Fix S3AttachmentServiceImplSpec.groovy

* Fix ClusterLeaderServiceCuratorImplSpec

* Fix ZookeeperAutoConfiguration

* Fix genie-web tests

* Fix JpaPersistenceServiceIntegrationTestBase

* Update rest-assured dependencies

* Fix application, cluster, command, job, criterion entities

* Fix jpa job repository

* Fix JpaPersistenceServiceImpl

* Fix JpaPersistenceServiceIntegrationTestBase

* Fix RestControllerIntegrationTestBase

* migrate off deprecated @MockBean annotation

* remove explicit aws.region property

* exclude S3TransferManagerAutoConfiguration from integration tests

* exclude S3TransferManagerAutoConfiguration from the genie-agent app

* exclude S3TransferManagerAutoConfiguration from the genie app

* checkstyle

* update io.zipkin.brave:brave to v6 that's aligned with Spring Boot 3

* update to latest Spring Boot 3

* Add threshold to avoid infinite loop waiting for job running and job done

* use deep stubs for mocks and disable bean overriding

* remove AutoConfiguration.imports from licencing

* use MySQLDialect for hibernate as MySQL8Dialect is deprecated

* Fix testForTooManyCommandArgs

* Fix canHandleBadInputToCreateCommand

* update grpc proto and other oss dependency versions

* add sleep to validate an assumption that requests are made too early

* exclude jetty from genie-web integration test

* reduce the max iterations from 100 to 60

* exclude CommonTracingAutoConfiguration.class from import to avoid bean conflicts with the custom mock

* use Spring Boot BOM recommendation and update relocated libraries

* remove redundant exclude

* Fix canCreateClusterWithoutId in ClusterRestControllerIntegrationTest

* Amend JobRestControllerIntegrationTest

* Amend ClusterRestControllerIntegrationTest

* Revert "add sleep to validate an assumption that requests are made too early"

This reverts commit 0434935.

* Amend ClusterRestControllerIntegrationTest

* upgrade grpc_version=1.50.2 and netty_version=4.1.86.Final

* cleanup

* Fix GRpcAgentJobKillServiceImpl and GRpcAgentJobKillServiceImplSpec

* Fix S3JobArchiverImpl and S3JobArchiverImplSpec

* Fix JobRestControllerIntegrationTest

* update to OSS Spring Boot 3.4.5 and align Netty version with OSS

* set netty_version=4.1.100.Final

* upgrade grpc_version to 1.63.1

* upgrade protobuf_version to 3.25.5

* patch update of protobuf plugin version

* fix some deprecation warnings

* migrate off deprecated Gradle Enterprise plugin

* update rest assured deps

* fix warn message placeholder

* fix some Gradle deprecation warnings

* turn off html.required for test reports

* Fix testSubmitJobMethodSuccess in JobRestControllerIntegrationTest

* clean up comments

* Turn off spring.jpa.hibernate.ddl-auto for now

* Add org.flywaydb:flyway-database-postgresql

* Revert "turn off html.required for test reports"

This reverts commit 9343acd.

* Update GRADLE_OPTS in genie-build.yml

* Fix to the Snippets issue

* Revert "update rest assured deps"

This reverts commit 02d62f3.

* Fix to the Snippets issue

* Fix to the Snippets issue

* Turn off html.enabled for test reports

* update org.hibernate.validator:hibernate-validator

* update rest assured deps

* fix AWSCredentialsProvider bean creation

* Amend LeaderElectionActuator

* Fix spring actuators package-info with jakarta

* Rebuild LeaderElectionActuator

* Amend LeaderElectionActuator to throw exceptions

* Amend GRpcAgentJobKillServiceImpl

* update SpringDoc as version 1 is incompatible with Spring Boot 3 (#1232)

* pick up OSS Spring Boot 3.4.6 (#1233)

---------

Co-authored-by: Binbing Hou <[email protected]>
Co-authored-by: Iuliia Sobolevska <[email protected]>
Co-authored-by: Zhuoran Dong <[email protected]>

Author: 4 people

.github/workflows/genie-build.yml

@@ -41,7 +41,7 @@ jobs:
       DOCKER_PASSWORD: ${{ secrets.ORG_NETFLIXOSS_DOCKERHUB_PASSWORD }}
       GRGIT_USER: ${{ secrets.GRGIT_USER }}
       GRGIT_PASS: ${{ secrets.GITHUB_TOKEN }}
-      GRADLE_OPTS: -Djava.io.tmpdir=/tmp/
+      GRADLE_OPTS: -Djava.io.tmpdir=/tmp/ --add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.base/java.io=ALL-UNNAMED
     steps:
       - uses: actions/checkout@v1
       - name: Configure MySQL database
@@ -95,7 +95,7 @@ jobs:
           ((!startsWith(github.ref, 'refs/tags/v')) && github.ref != 'refs/heads/master' && github.ref != 'refs/heads/dev-snapshot')
         run: |
           ./gradlew --stacktrace build codeCoverageReport coveralls
-          INTEGRATION_TEST_DB=mysql ./gradlew --stacktrace genie-web:integrationTest
+          INTEGRATION_TEST_DB=mysql ./gradlew --stacktrace genie-web:integrationTest genie-web:createAllMissingSnippets genie-web:asciidoctor
           INTEGRATION_TEST_DB=postgresql ./gradlew --stacktrace genie-web:integrationTest
           ./gradlew --stacktrace javadoc asciidoc dockerBuildAllImages
       - name: Publish snapshot

build.gradle

@@ -1,6 +1,5 @@
 import com.github.spotbugs.snom.SpotBugsTask
 import org.gradle.plugins.ide.idea.model.IdeaLanguageLevel
-import org.springframework.boot.gradle.plugin.SpringBootPlugin
 
 buildscript {
     repositories {
@@ -19,10 +18,10 @@ plugins {
     id "com.netflix.nebula.facet" version "11.0.0" apply false
     id "org.ajoberstar.grgit" version "4.1.1"
     id "org.ajoberstar.git-publish" version "3.0.1"
-    id "org.springframework.boot" version "${spring_boot_version}" apply false
+    id "org.springframework.boot" version "${spring_boot_3_version}" apply false
     id "org.asciidoctor.jvm.convert" version "3.3.2" apply false
     id "com.gorylenko.gradle-git-properties" version "2.3.2" apply false
-    id "com.google.protobuf" version "0.8.18" apply false
+    id "com.google.protobuf" version "0.8.19" apply false
     id "io.franzbecker.gradle-lombok" version "5.0.0" apply false
     id "nebula.node" version "2.0.1" apply false
     id "com.github.spotbugs" version "5.2.5" apply false
@@ -36,6 +35,8 @@ ext.javaProjects = subprojects.findAll {
     it.name != "genie-demo" && it.name != "genie-ddl" && it.name != "genie-docs"
 }
 
+def springBoot3ProjectNames = ['genie-agent', 'genie-agent-app', 'genie-app', 'genie-common-internal', 'genie-swagger', 'genie-ui', 'genie-web']
+
 allprojects {
     apply plugin: "jacoco"
     apply plugin: "idea"
@@ -44,6 +45,10 @@ allprojects {
     repositories {
         mavenCentral()
     }
+
+    tasks.withType(JavaExec) {
+        jvmArgs += ["--add-opens", "java.base/sun.nio.ch=ALL-UNNAMED", "--add-opens", "java.base/java.io=ALL-UNNAMED"]
+    }
 }
 
 idea {
@@ -54,8 +59,8 @@ idea {
     }
     module {
         languageLevel = new IdeaLanguageLevel(JavaVersion.VERSION_1_8)
-        downloadJavadoc true
-        downloadSources true
+        downloadJavadoc = true
+        downloadSources = true
     }
 }
 
@@ -93,14 +98,31 @@ configure((Set<Project>) ext.javaProjects) {
     }
 
     dependencyManagement {
+        if (springBoot3ProjectNames.contains(project.name)) {
+            imports {
+                mavenBom "org.springframework.boot:spring-boot-dependencies:${spring_boot_3_version}"
+                mavenBom "org.springframework.cloud:spring-cloud-dependencies:${spring_cloud_3_version}"
+                mavenBom "io.awspring.cloud:spring-cloud-aws-dependencies:${spring_cloud_aws_3_version}"
+                mavenBom "org.spockframework:spock-bom:2.4-M5-groovy-3.0"
+            }
+            dependencies {
+                dependency("io.awspring.cloud:spring-cloud-aws-starter:${spring_cloud_aws_3_version}")
+            }
+        } else {
+            imports {
+                mavenBom "org.springframework.boot:spring-boot-dependencies:${spring_boot_2_version}"
+                mavenBom "org.springframework.cloud:spring-cloud-dependencies:${spring_cloud_2_version}"
+                mavenBom "io.awspring.cloud:spring-cloud-aws-dependencies:${spring_cloud_aws_2_version}"
+                mavenBom "org.spockframework:spock-bom:2.0-groovy-3.0"
+            }
+        }
+
         imports {
-            mavenBom SpringBootPlugin.BOM_COORDINATES
-            mavenBom "org.springframework.cloud:spring-cloud-dependencies:${spring_cloud_version}"
-            mavenBom "io.awspring.cloud:spring-cloud-aws-dependencies:2.4.1"
+            mavenBom "software.amazon.awssdk:bom:2.30.17"
             mavenBom "com.google.protobuf:protobuf-bom:${protobuf_version}"
             mavenBom "com.squareup.okhttp3:okhttp-bom:4.9.2"
             mavenBom "io.grpc:grpc-bom:${grpc_version}"
-            mavenBom "org.spockframework:spock-bom:2.0-groovy-3.0"
+            mavenBom "io.netty:netty-bom:${netty_version}"
             mavenBom "org.testcontainers:testcontainers-bom:1.16.3"
         }
         dependencies {
@@ -128,10 +150,21 @@ configure((Set<Project>) ext.javaProjects) {
             dependency("org.codehaus.groovy:groovy-all:3.0.9")
             dependency("org.codehaus.janino:janino:3.1.6")
             dependency("org.dbunit:dbunit:2.7.3")
-            dependencySet(group: "org.springdoc", version: "1.6.6") {
-                entry "springdoc-openapi-hateoas"
-                entry "springdoc-openapi-ui"
-                entry "springdoc-openapi-webmvc-core"
+            dependencySet(group: "org.springdoc", version: "2.8.8") {
+                entry "springdoc-openapi-starter-webmvc-api"
+                entry "springdoc-openapi-starter-webmvc-ui"
+            }
+            dependencySet(group: "io.netty", version: "${netty_version}") {
+                entry "netty-buffer"
+                entry "netty-codec"
+                entry "netty-codec-http"
+                entry "netty-codec-http2"
+                entry "netty-common"
+                entry "netty-handler"
+                entry "netty-resolver"
+                entry "netty-transport"
+                entry "netty-transport-native-epoll"
+                entry "netty-transport-native-unix-common"
             }
         }
     }
@@ -156,7 +189,7 @@ configure((Set<Project>) ext.javaProjects) {
     }
 
     configurations {
-        all*.exclude group: "javax.servlet", module: "servlet-api"
+        all*.exclude group: "org.apache.groovy" // exclude groovy@4 dependencies
         all*.exclude group: "org.springframework.boot", module: "spring-boot-starter-logging"
     }
 
@@ -168,7 +201,18 @@ configure((Set<Project>) ext.javaProjects) {
         annotationProcessor("net.jcip:jcip-annotations")
         annotationProcessor("com.github.spotbugs:spotbugs-annotations")
 
-        implementation("javax.annotation:javax.annotation-api:1.3.2")
+        if (springBoot3ProjectNames.contains(project.name)) {
+            implementation("org.springframework.boot:spring-boot-actuator-autoconfigure")
+            implementation("io.awspring.cloud:spring-cloud-aws-starter-s3:${spring_cloud_aws_3_version}")
+            implementation("io.micrometer:micrometer-tracing-bridge-brave")
+            implementation("jakarta.servlet:jakarta.servlet-api")
+            implementation("jakarta.validation:jakarta.validation-api")
+            implementation("org.hibernate.validator:hibernate-validator")
+            implementation("jakarta.el:jakarta.el-api")
+            implementation("io.zipkin.reporter2:zipkin-reporter")
+        } else {
+            implementation("javax.annotation:javax.annotation-api:1.3.2")
+        }
 
         /*******************************
          * Compile Dependencies
@@ -205,15 +249,14 @@ configure((Set<Project>) ext.javaProjects) {
         testCompileOnly("com.github.spotbugs:spotbugs-annotations")
 
         testRuntimeOnly("jakarta.el:jakarta.el-api")
-        testRuntimeOnly("org.glassfish:jakarta.el")
         testRuntimeOnly("org.junit.jupiter:junit-jupiter-engine")
     }
 
-    task removeGeneratedSources(type: Delete) {
+    tasks.register('removeGeneratedSources', Delete) {
         project.generatedSourcesDir.deleteDir()
     }
 
-    task removeIdeaOutDirectory(type: Delete) {
+    tasks.register('removeIdeaOutDirectory', Delete) {
         file("${projectDir}/out").deleteDir()
     }
 
@@ -260,7 +303,12 @@ configure((Set<Project>) ext.javaProjects) {
                 systemProperty "spring.profiles.include", "githubactions"
             }
 
-            reports.html.destination(new File("${reporting.baseDir}/test/${task.name}"))
+            reports {
+                html.required = false
+                junitXml.required = true
+            }
+
+            // reports.html.outputLocation = new File("${reporting.baseDirectory}/test/${task.name}")
             (jacocoTestReport.getExecutionData() + files("$buildDir/jacoco/${task.name}.exec"))
 
             // Customize test logging on CI and everywhere else
@@ -276,7 +324,7 @@ configure((Set<Project>) ext.javaProjects) {
 
     licenseMain.dependsOn compileJava
     license {
-        ignoreFailures false
+        ignoreFailures = false
     }
 
     checkstyle {
@@ -326,9 +374,9 @@ task codeCoverageReport(type: JacocoReport, group: "Coverage reports") {
 
     reports {
         xml.getRequired().set(true)
-        xml.destination new File("${buildDir}/reports/jacoco/report.xml")
+        xml.outputLocation = new File("${buildDir}/reports/jacoco/report.xml")
         html.getRequired().set(true)
-        html.destination new File("${buildDir}/reports/jacoco/html")
+        html.outputLocation = new File("${buildDir}/reports/jacoco/html")
         csv.getRequired().set(false)
     }
     afterEvaluate {
@@ -371,7 +419,9 @@ tasks.coveralls {
  * Github Pages Tasks
  **********************************/
 
-task collectDocumentation(group: "documentation", description: "Copy the documentation from sub-projects") {
+tasks.register('collectDocumentation') {
+    group = "documentation"
+    description = "Copy the documentation from sub-projects"
     dependsOn "aggregateJavadocs"
 
     def restDocs = new File(project(":genie-web").buildDir, "docs/asciidoc")
@@ -484,22 +534,28 @@ ext.getDockerTags = { String appName, String projectVersion ->
     return tags
 }
 
-task dockerLogout(type: Exec, group: "Docker", description: "Logout of docker hub") {
+tasks.register('dockerLogout', Exec) {
+    group = "Docker"
+    description = "Logout of docker hub"
     onlyIf {
         rootProject.ext.isDockerRunning()
     }
     commandLine "docker", "logout"
 }
 
-task dockerLogin(type: Exec, group: "Docker", description: "Login to docker hub using DOCKER_USER and DOCKER_PASSWORD environment variables") {
+tasks.register('dockerLogin', Exec) {
+    group = "Docker"
+    description = "Login to docker hub using DOCKER_USER and DOCKER_PASSWORD environment variables"
     onlyIf {
         rootProject.ext.isDockerRunning()
     }
     dependsOn tasks.dockerLogout
     commandLine "docker", "login", "-u", System.getenv("DOCKER_USER") ?: "", "-p", System.getenv("DOCKER_PASSWORD") ?: ""
 }
 
-task dockerBuildAllImages(group: "Docker", description: "Container task for all docker image builds") {
+tasks.register('dockerBuildAllImages') {
+    group = "Docker"
+    description = "Container task for all docker image builds"
     onlyIf {
         rootProject.ext.isDockerRunning()
     }
@@ -508,10 +564,10 @@ task dockerBuildAllImages(group: "Docker", description: "Container task for all
     dependsOn ":genie-demo:dockerBuildClientImage"
 }
 
-gradleEnterprise {
+develocity {
     buildScan {
-        publishAlways()
-        termsOfServiceUrl = "https://gradle.com/terms-of-service"
-        termsOfServiceAgree = "yes"
+        publishing.onlyIf { true }
+        termsOfUseUrl = "https://gradle.com/terms-of-service"
+        termsOfUseAgree = "yes"
     }
 }

config/checkstyle/checkstyle.xml

@@ -304,7 +304,7 @@
 
     <!-- Checks for long lines. -->
     <module name="LineLength">
-        <property name="max" value="120"/>
+        <property name="max" value="150"/>
         <property name="ignorePattern" value="\$File.*\$"/>
     </module>
 

dependencies.lock

@@ -0,0 +1,12 @@
+{
+    "jacocoAgent": {
+        "org.jacoco:org.jacoco.agent": {
+            "locked": "0.8.12"
+        }
+    },
+    "jacocoAnt": {
+        "org.jacoco:org.jacoco.ant": {
+            "locked": "0.8.12"
+        }
+    }
+}

genie-agent-app/build.gradle

@@ -32,6 +32,7 @@ dependencies {
 
     implementation(project(":genie-agent"))
     implementation("org.springframework.boot:spring-boot-starter")
+    implementation("org.springframework.boot:spring-boot-starter-web")
 
     /*******************************
      * Compile Only Dependencies
@@ -46,6 +47,7 @@ dependencies {
      *******************************/
 
     testImplementation(project(":genie-test"))
+    testImplementation("org.springframework.boot:spring-boot-starter-web")
 }
 
 springBoot {