Merge pull request #4984 from cfpb/zencircle-docs-keyspaces

create platform-msk-keyspaces.md

Author: lchen-2101

README.md

@@ -171,7 +171,15 @@ The HMDA Platform can run locally using [`sbt`](https://www.scala-sbt.org/) with
     sbt:hmda-root> project hmda-platform
     sbt:hmda-platform> reStart
     ```
-
+### Building and running the .jar       
+To build JVM artifacts, from the sbt prompt first choose the project you want to build and use the assembly command:   
+   ```bash
+    sbt
+    sbt:root> project check-digit
+    sbt:check-digit>assembly
+   ```
+This task will create a fat jar, which can be executed on any JDK9 compliant JVM:   
+`java -jar target/scala-2.12/check-digit.jar`
 ### Running with docker compose
 
 The platform and it's dependency services, Kafka, Cassandra and PostgreSQL, can run locally using [Docker Compose](https://docs.docker.com/compose/).

docs/JavaInstall.md

@@ -1,154 +1,53 @@
-# HMDA Platform
-
-## Introduction
-
-For more information on HMDA, checkout the [About HMDA page](http://www.consumerfinance.gov/data-research/hmda/learn-more) on the CFPB website.
-
-## The HMDA Platform
-
-The new HMDA Platform (“V2”) was totally updated over the past year to utilize a more flexible and resilient approach (Kubernetes Microservices) in order to better serve our customers. This repository contains the code for the entirety of the HMDA platform backend. This platform has been designed to accommodate the needs of the HMDA filing process by financial institutions, as well as the data management and publication needs of the HMDA data asset.
-
-The HMDA Platform uses sbt's multi-project builds, each project representing a specific task. The platform is an Akka Cluster application that can be deployed on a single node or as a distributed application. For more information on how Akka Cluster is used, see the documentation [here](Documents/cluster.md)
+# [HMDA Platform](../README.md)
 
 The HMDA Platform is composed of the following modules:
-
 ### Parser (JS/JVM)
 
 Module responsible for reading incoming data and making sure that it conforms to the HMDA File Specification
 
 ### Data Validation
-
 Module responsible for validating incoming data by executing validation rules as per the Edit Checks documentation
 
 ### Persistence
-
 Module responsible for persisting information into the system. It becomes the system of record for HMDA data
 
 ### Cluster
-
 Module responsible for managing the various cluster roles, as well as starting the Hmda Platform
 
 ### API
-
 This module contains both public APIs for HMDA data for general use by third party clients and web applications, as well as endpoints for receiving data and providing information about the filing process for Financial Institutions
 
 ### API Model
-
 This module contains objects and JSON protocols for use by the API project
 
 ### Query
-
 This module is responsible for interacting with the back-end database, as well as conversion between model objects and database objects.
 
 ### Panel
-
 This module is responsible for parsing and persisting a CSV-format panel file
 
 ### Model (JS/JVM)
-
 This module is responsible for maintaining the objects used in our platform
 
 ### Census
-
 This module is responsible for geographic translation (e.g. state number -> state code)
 
 ### Publication
-
 This module generates Aggregate and Disclosure reports, as required by HMDA statute.
 
-## Dependencies
-
-
-## Dependencies
-
-### Java 19 SDK
-
-The HMDA Platform runs on the Java Virtual Machine (JVM), and requires the Java 19 JDK to build and run the project. This project is currently being built and tested on [Oracle JDK 19](https://www.oracle.com/java/technologies/javase/jdk19-archive-downloads.html). See [Oracle's JDK Install Overview](https://docs.oracle.com/en/java/javase/19/install/index.html) for install instructions.
-
-The HMDA Platform should also run on JDK 8.
-
-### Scala
-
-The HMDA Platform is written in [Scala](http://www.scala-lang.org/). To build it, you will need to [download](http://www.scala-lang.org/download/) and [install](http://www.scala-lang.org/download/install.html) Scala 2.12.x
-
-In addition, you'll need Scala's interactive build tool [sbt](https://www.scala-sbt.org/). Please refer to sbt's [installation instructions](https://www.scala-sbt.org/1.x/docs/Setup.html) to get started.
-
-## Project structure
-
-The HMDA Platform is divided into individual projects, each responsible for a subset of the functionality, as follows:
-
-### hmda-platform
-
-This is the main filing application, exposing the APIs necessary to upload, validate and store HMDA files. 
-
-### check-digit 
-
-Microservice that exposes functionality to create a check digit from a loan id, and to validate `Univeral Loan Identifiers` 
-
-
-## Building and Running
-
-### Running from the SBT prompt
-
-* To run the project from the `SBT` prompt for development purposes, issue the following commands on a terminal:
-
-```shell
-$ sbt
-sbt:root> project hmda-platform
-sbt:hmda-platform> reStart
-```
-
-
-### Building and running the .jar
-
-* To build JVM artifacts, from the sbt prompt first choose the project you want to build and use the assembly command:
-
-```shell
-$ sbt
-sbt:root> project check-digit
-sbt:check-digit>assembly
-```
-This task will create a `fat jar`, which can be executed on any `JDK9` compliant `JVM`:
-
-`java -jar target/scala-2.12/check-digit.jar`
-
-### Building and running the Docker image
-
-* To build a `Docker` image that runs the `hmda-platform` as a single node cluster, from the sbt prompt:
-
-```shell
-$sbt
-sbt:root> project hmda-platform
-sbt:hmda-platform> docker:publishLocal
-```
-This task will create a `Docker` image. To run a container with the `HMDA Platform` filing application as a single node cluster:
-
-`docker run -e CASSANDRA_CLUSTER_HOSTS=localhost --rm -ti -p 8080:8080 -p 8081:8081 -p 8082:8082 -p 19999:19999 hmda/hmda-platform` 
-
-The same approach can be followed to build and run Docker containers for the other microservices that form the HMDA Platform. 
-
-Certain environment variables can be passed in to set the log level of the micro service
-
-```
-ZOOKEEPER_LOG_LEVEL (Defaulted to WARN)
-KAFKA_LOG_LEVEL (Defaulted to INFO)
-CASSANDRA_LOG_LEVEL (Defaulted to INFO)
-PLATFORM_LOG_LEVEL (Defaulted to WARN)
-INSTITUTION_LOG_LEVEL (Defaulted to INFO)
-CHECKDIGIT_LOG_LEVEL (Defaulted to DEBUG)
-```
+### Docker run  
+To run a container with the `HMDA Platform` filing application as a single node cluster:  
+`docker run -e CASSANDRA_CLUSTER_HOSTS=localhost --rm -ti -p 8080:8080 -p 8081:8081 -p 8082:8082 hmda/hmda-platform` 
 
 ## Resources
 
 ### API Documentation
 
-* [HMDA Platform Public API Documentation](api/public-api.md)
-* [HMDA Platform ULI API Documentation](api/uli.md)
-* [HMDA Platform Filers API Documentation](api/filers-api.md)
-
 ### Development
-
-* [Local Kubernetes CI/CD](development/kubernetes.md)
+* [Local development](development/local-platform.md)
+* [Kubernetes](development/kubernetes.md)
+* [Specific MSK and Keyspaces configuration](development/platform-msk-keyspaces.md)
+* [Postman configuration](development/postman.md)
 
 ### Data Specifications
 

docs/README.md

@@ -0,0 +1,87 @@
+## Specific documentation related to AWS Keyspace and AWS MSK
+### MSK
+This includes the MSK configuration for hmda-platform, modified-lar, irs-publisher and email-service 
+
+- Required env variables for MSK 
+```
+export KAFKA_SECURITY="SASL_SSL" 
+export AWS_ACCESS_KEY_ID="acesskeyvalue1" 
+export AWS_SECRET_ACCESS_KEY="secretkeyvalue1" 
+```
+- To use plaintext KAFKA
+`unset KAFKA_SECURITY`
+
+### Keyspaces
+This includes the Keyspaces configuration for hmda-platform, modified-lar, irs-publisher and hmda-analytics
+- For Apache Cassandra(k8ssandra) with PlainTextAuthProvider use [persistence.conf](common/src/main/resources/persistence.conf)
+- For AWS Keyspace with PlainTextAuthProvider use [persistence-keyspace.conf](common/src/main/resources/persistence-keyspace.conf) (do not use for cluster deployment)
+- For AWS Keyspace with SigV4AuthProvider [persistence-keyspace-sigv4.conf](common/src/main/resources/persistence-keyspace-sigv4.conf)
+- List of platform related files to updated
+```
+hmda/src/main/resources/application.conf (local deployment)
+hmda/src/main/resources/application-kubernetes.conf (cluster deployment)
+hmda-analytics/src/main/resources/application.conf
+modified-lar/src/main/resources/application.conf
+irs-publisher/src/main/resources/application.conf
+```
+
+- Environment variables for Apache Cassandra(k8ssandra) with PlainTextAuthProvider use [persistence.conf](common/src/main/resources/persistence.conf)
+```
+export CASSANDRA_CLUSTER_HOSTS="10.x.x.x:9042"
+export CASSANDRA_JOURNAL_KEYSPACE=hmda2_journal
+export CASSANDRA_SNAPSHOT_KEYSPACE=hmda2_snapshot
+export CASSANDRA_CLUSTER_USERNAME=xxx
+export CASSANDRA_CLUSTER_DC=xxx
+export CASSANDRA_CLUSTER_PASSWORD=xxx
+```
+- Setup java truststore
+```
+curl https://certs.secureserver.net/repository/sf-class2-root.crt -O
+openssl x509 -outform der -in sf-class2-root.crt -out temp_file.der
+keytool -import -alias cassandra -keystore cassandra_truststore.jks -file temp_file.der
+...
+Trust this certificate? [no]:  yes
+Certificate was added to keystore
+```
+- Environment variables for AWS Keyspace with PlainTextAuthProvider use [persistence-keyspace.conf](common/src/main/resources/persistence-keyspace.conf)
+```
+export CASSANDRA_TRUSTSTORE_FILE="/tmp/cassandra_truststore.jks"
+export CASSANDRA_TRUSTSTORE_PASSWORD="xxx"
+export CASSANDRA_JOURNAL_KEYSPACE=hmda2_journal
+export CASSANDRA_SNAPSHOT_KEYSPACE=hmda2_snapshot
+export CASSANDRA_CLUSTER_DC="us-east-1"
+export CASSANDRA_CLUSTER_USERNAME="xxx"
+export CASSANDRA_CLUSTER_PASSWORD="xxx"
+```
+
+- Environment variables for AWS Keyspace with SigV4AuthProvider [persistence-keyspace-sigv4.conf](common/src/main/resources/persistence-keyspace-sigv4.conf)
+```
+export CASSANDRA_TRUSTSTORE_FILE="/tmp/cassandra_truststore.jks"
+export CASSANDRA_TRUSTSTORE_PASSWORD="xxx"
+export CASSANDRA_JOURNAL_KEYSPACE=hmda2_journal
+export CASSANDRA_SNAPSHOT_KEYSPACE=hmda2_snapshot
+export CASSANDRA_CLUSTER_DC="us-east-1"
+export CASSANDRA_CLUSTER_USERNAME="cassandrausernamevalue1"
+export CASSANDRA_CLUSTER_PASSWORD="xxx"
+export AWS_ACCESS_KEY_ID="acesskeyvalue" 
+export AWS_SECRET_ACCESS_KEY="secretkeyvalue" 
+```
+- Setup java truststore
+```
+curl https://certs.secureserver.net/repository/sf-class2-root.crt -O
+openssl x509 -outform der -in sf-class2-root.crt -out temp_file.der
+keytool -import -alias cassandra -keystore cassandra_truststore.jks -file temp_file.der
+...
+Trust this certificate? [no]:  yes
+Certificate was added to keystore
+```
+- Backward compatibility update configmap to accommodate for k8ssandra
+```
+kubectl patch configmap/cassandra-configmap \
+  --type merge \
+  -p '{ "data": { "cassandra-hosts": "cluster1-main-service.k8ssandra-operator:9042" }}'
+
+kubectl -n beta patch configmap/cassandra-configmap \
+  --type merge \
+  -p '{ "data": { "cassandra-hosts": "cluster1-main-service.k8ssandra-operator:9042" }}'
+```