Skip to content

Commit 0d87278

Browse files
chuckwondosharkinsspatial
chuckwondo
and
sharkinsspatial
authored
Prepare for release of MCP production deployment (#269)
* Fix id update typing for sentinel_ac_logger. * Use github vars for mcp-dev environment * Fix flake8 errors * Bump actions versions to remove node12 deprecation warnings * Fix deprecation warning for applyAspect The API @aws-cdk/core.ConstructNode.applyAspect is deprecated: Use "Aspects.of(construct).add(aspect)" instead. This API will be removed in the next major release * Use OIDC role for mcp-dev deployment * Upgrade Node and CDK - upgrade node to 16.x - upgrade to latest version of CDK v1 - fix dependency list error (appending should be extending) * Remove buckets on destroy * Bump Aurora Serverless to v11.18 (10.7 obsolete) * Restore subscription to Landsat SNS * Add mcp-production-deployment (#267) * Update Landsat SNS topic * Add mcp-production-deployment * Add HLS_LANDSAT_SNS_TOPIC_ENABLED env var (default: "true") to allow disabling of trigger --------- Co-authored-by: Sean Harkins <[email protected]>
1 parent c189ac7 commit 0d87278

16 files changed

+310
-123
lines changed

‎.github/workflows/mcp_dev_deployment.yml

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
name: mcp dev deployment
2+
3+
on:
4+
push:
5+
branches: [dev]
6+
workflow_dispatch:
7+
8+
env:
9+
AWS_DEFAULT_REGION: us-west-2
10+
11+
# See https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services#updating-your-github-actions-workflow
12+
permissions:
13+
id-token: write # required for requesting the JWT
14+
contents: read # required for actions/checkout
15+
16+
jobs:
17+
unit-tests:
18+
runs-on: ubuntu-20.04
19+
strategy:
20+
matrix:
21+
python: [3.8]
22+
steps:
23+
- uses: actions/checkout@v4
24+
- name: Setup Python
25+
uses: actions/setup-python@v4
26+
with:
27+
python-version: ${{ matrix.python }}
28+
- name: Install Tox
29+
run: pip install tox
30+
- name: Run Tox test environment
31+
# Run tox using the version of Python in `PATH`
32+
run: tox -e py
33+
mcp-dev-deployment:
34+
strategy:
35+
matrix:
36+
python: [3.8]
37+
runs-on: ubuntu-20.04
38+
needs: [unit-tests]
39+
environment:
40+
name: mcp-dev
41+
steps:
42+
- uses: actions/checkout@v4
43+
- name: Setup Python
44+
uses: actions/setup-python@v4
45+
with:
46+
python-version: ${{ matrix.python }}
47+
- name: Install tox
48+
run: pip install tox
49+
- name: Configure AWS credentials
50+
uses: aws-actions/configure-aws-credentials@v3
51+
with:
52+
role-to-assume: ${{ vars.AWS_ROLE_TO_ASSUME_ARN }}
53+
role-session-name: ${{ github.actor }}
54+
aws-region: us-west-2
55+
- name: Convert secrets to environment variables
56+
env:
57+
SECRETS_JSON: ${{ toJson(secrets) }}
58+
run: |
59+
while read -rd $'' line; do
60+
echo "$line" >> $GITHUB_ENV
61+
done < <(
62+
jq -r <<<"$SECRETS_JSON" 'to_entries|map("\(.key)=\(.value)\u0000")[]'
63+
)
64+
- name: Convert vars to environment variables
65+
env:
66+
VARS_JSON: ${{ toJson(vars) }}
67+
run: |
68+
while read -rd $'' line; do
69+
echo "$line" >> $GITHUB_ENV
70+
done < <(
71+
jq -r <<<"$VARS_JSON" 'to_entries|map("\(.key)=\(.value)\u0000")[]'
72+
)
73+
- name: Deploy with tox
74+
run: tox -v -e dev -r -- deploy --require-approval never
75+
- name: Run DB setup
76+
run: |
77+
setupdb=$(aws cloudformation describe-stacks \
78+
--stack-name $HLS_STACKNAME \
79+
--query "Stacks[0].Outputs[?OutputKey=='setupdbexport'].OutputValue" \
80+
--output=text
81+
)
82+
aws lambda invoke --function-name=$setupdb response.json

‎.github/workflows/mcp_production_deployment.yml

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
name: MCP production deployment
2+
3+
on:
4+
release:
5+
types: [ published ]
6+
7+
env:
8+
AWS_DEFAULT_REGION: us-west-2
9+
10+
# See https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services#updating-your-github-actions-workflow
11+
permissions:
12+
id-token: write # required for requesting the JWT
13+
contents: read # required for actions/checkout
14+
15+
jobs:
16+
unit-tests:
17+
runs-on: ubuntu-20.04
18+
strategy:
19+
matrix:
20+
python: [3.8]
21+
steps:
22+
- uses: actions/checkout@v4
23+
- name: Setup Python
24+
uses: actions/setup-python@v4
25+
with:
26+
python-version: ${{ matrix.python }}
27+
- name: Install Tox
28+
run: pip install tox
29+
- name: Run Tox test environment
30+
# Run tox using the version of Python in `PATH`
31+
run: tox -e py
32+
mcp-production-deployment:
33+
strategy:
34+
matrix:
35+
python: [3.8]
36+
runs-on: ubuntu-20.04
37+
needs: [unit-tests]
38+
environment:
39+
name: mcp-production
40+
steps:
41+
- uses: actions/checkout@v4
42+
- name: Setup Python
43+
uses: actions/setup-python@v4
44+
with:
45+
python-version: ${{ matrix.python }}
46+
- name: Install tox
47+
run: pip install tox
48+
- name: Configure AWS credentials
49+
uses: aws-actions/configure-aws-credentials@v3
50+
with:
51+
role-to-assume: ${{ vars.AWS_ROLE_TO_ASSUME_ARN }}
52+
role-session-name: ${{ github.actor }}
53+
aws-region: us-west-2
54+
- name: Convert secrets to environment variables
55+
env:
56+
SECRETS_JSON: ${{ toJson(secrets) }}
57+
run: |
58+
while read -rd $'' line; do
59+
echo "$line" >> $GITHUB_ENV
60+
done < <(
61+
jq -r <<<"$SECRETS_JSON" 'to_entries|map("\(.key)=\(.value)\u0000")[]'
62+
)
63+
- name: Convert vars to environment variables
64+
env:
65+
VARS_JSON: ${{ toJson(vars) }}
66+
run: |
67+
while read -rd $'' line; do
68+
echo "$line" >> $GITHUB_ENV
69+
done < <(
70+
jq -r <<<"$VARS_JSON" 'to_entries|map("\(.key)=\(.value)\u0000")[]'
71+
)
72+
- name: Deploy with tox
73+
run: tox -v -e dev -r -- deploy --require-approval never
74+
- name: Run DB setup
75+
run: |
76+
setupdb=$(aws cloudformation describe-stacks \
77+
--stack-name $HLS_STACKNAME \
78+
--query "Stacks[0].Outputs[?OutputKey=='setupdbexport'].OutputValue" \
79+
--output=text
80+
)
81+
aws lambda invoke --function-name=$setupdb response.json

‎.github/workflows/tox_tests.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,9 +12,9 @@ jobs:
1212
python: [3.8]
1313

1414
steps:
15-
- uses: actions/checkout@v2
15+
- uses: actions/checkout@v4
1616
- name: Setup Python
17-
uses: actions/setup-python@v2
17+
uses: actions/setup-python@v4
1818
with:
1919
python-version: ${{ matrix.python }}
2020
- name: Install Tox and any other packages

‎README.md

Lines changed: 69 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -1,78 +1,106 @@
1-
# hls-orchestration
1+
# HLS Orchestration
22

33
AWS Stack for processing HLS data.
44

55
![Alt text](/docs/HLS_architecture.png)
66

7-
For more detailed data flow diagrams and architecture see [architecture](/docs/architecture.md).
7+
For more detailed data flow diagrams and architecture see
8+
[architecture](/docs/architecture.md).
89

9-
For more details about all of the HLS project's components see [hls-project](https://github.com/nasa-impact/hls-project).
10+
For more details about all of the HLS project's components see
11+
[hls-project](https://github.com/nasa-impact/hls-project).
1012

13+
## Requirements
1114

12-
### Requirements
13-
Python>=3.7 \
14-
tox \
15-
aws-cli \
16-
jq \
17-
An IAM role with sufficient permissions for creating, destroying and modifying the relevant stack resources.
15+
- Python>=3.7
16+
- tox
17+
- aws-cli
18+
- jq
19+
- An IAM role with sufficient permissions for creating, destroying, and
20+
modifying the relevant stack resources.
1821

19-
### Environment Settings
20-
Environment variables are set in `environment.sh`. Copy `environment.sh.sample` to `environment.sh` and update the settings prior to running any commands. The following variables can be overridden from the calling shell's environment
21-
```
22-
$ export HLS_STACKNAME=<Name of your stack>
23-
$ export HLS_LAADS_TOKEN=<Token used for accessing the Laads Data>
24-
$ export HLS_SENTINEL_OUTPUT_BUCKET_ROLE_ARN=<GCC Role for accessing output bucket>
22+
## Environment Settings
23+
24+
Environment variables are set in `environment.sh`. Copy `environment.sh.sample`
25+
to `environment.sh` and update the settings prior to running any commands. The
26+
following variables can be overridden from the calling shell's environment:
27+
28+
```plain
29+
export HLS_STACKNAME=<Name of your stack>
30+
export HLS_LAADS_TOKEN=<Token used for accessing the Laads Data>
31+
export HLS_SENTINEL_OUTPUT_BUCKET_ROLE_ARN=<GCC Role for accessing output bucket>
2532
```
2633

27-
### Synth
34+
## Synth
35+
2836
Display generated cloud formation template that will be used to deploy.
37+
38+
```plain
39+
source environment.sh && tox -e dev -r -- synth
2940
```
30-
$ source ./environment.sh && tox -e dev -r -- synth
31-
```
3241

33-
### Diff
42+
## Diff
43+
3444
Display a diff of the current deployment and any changes created.
35-
```
36-
$ source ./environment.sh && tox -e dev -r -- diff
37-
```
3845

39-
### Deploy
40-
Deploy current version of stack.
41-
```
42-
$ source ./environment.sh && tox -e dev -r -- deploy
46+
```plain
47+
source environment.sh && tox -e dev -r -- diff
4348
```
4449

45-
The repository is configured to create automatic deployments to the `hls-development` stack when PRs are merged into the `dev` branch. This deployment uses [Github Actions Environments](https://docs.github.com/en/actions/reference/environments) to manage the environment configuration rather than the `environment.sh`.
50+
## Deploy
4651

47-
Deployments to GCC have restrictions over creating VPCs and the types of AMIs which can be utilized. To deploy to GCC your shell will require the following environment settings.
52+
Deploy current version of stack:
53+
54+
```plain
55+
source environment.sh && tox -e dev -r -- deploy
4856
```
57+
58+
The repository is configured to create automatic deployments to the
59+
`hls-development` stack when PRs are merged into the `dev` branch. This
60+
deployment uses
61+
[Github Actions Environments](https://docs.github.com/en/actions/reference/environments)
62+
to manage the environment configuration rather than the `environment.sh`.
63+
64+
Deployments to GCC have restrictions over creating VPCs and the types of AMIs
65+
which can be utilized. To deploy to GCC your shell will require the following
66+
environment settings:
67+
68+
```plain
4969
export GCC=true
5070
export AWS_DEFAULT_REGION=us-west-2
5171
export HLS_GCC_ACCOUNT=<The GCC account id>
5272
export HLS_GCC_VPCID=<The vpc id provided by GCC administrators>
5373
export HLS_GCC_BOUNDARY_ARN=<The boundary policy arn>
5474
```
5575

76+
## Setup Logging Database
5677

57-
### Setup Logging Database
58-
After `deploy` is run and the stack is created run
59-
```
60-
$ source ./environment.sh && ./scripts/setupdb.sh
78+
After `deploy` is run and the stack is created run:
79+
80+
```plain
81+
source environment.sh && scripts/setupdb.sh
6182
```
83+
6284
To bootstrap the logging database.
6385

64-
### Development
65-
For active stack development run
66-
```
67-
$ source ./environment.sh && tox -e dev -r -- version
68-
```
69-
This creates a local virtualenv in the directory `devenv`. To use it for development
86+
## Development
87+
88+
For active stack development run:
89+
90+
```plain
91+
source environment.sh && tox -e dev -r -- version
7092
```
71-
$ source devenv/bin/activate
93+
94+
This creates a local virtualenv in the directory `devenv`. To use it for development:
95+
96+
```plain
97+
source devenv/bin/activate
7298
```
7399

74-
### Tests
100+
## Tests
101+
75102
To run unit test for all included Lambda functions
76-
```
103+
104+
```plain
77105
tox -r
78106
```

‎cdk.context.json

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
{
2+
"acknowledged-issue-numbers": [
3+
19836
4+
]
5+
}

‎setup.py

Lines changed: 4 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,10 @@
11
"""Setup for hls-orchestration"""
2-
import os
3-
import subprocess
4-
import sys
2+
53

64
from setuptools import find_packages, setup
75

86
# Runtime requirements.
9-
aws_cdk_version = "1.65.0"
7+
aws_cdk_version = "1.203.0"
108
aws_cdk_reqs = [
119
"core",
1210
"aws-s3",
@@ -31,10 +29,9 @@
3129

3230
inst_reqs = [
3331
"boto3",
32+
*[f"aws_cdk.{x}=={aws_cdk_version}" for x in aws_cdk_reqs],
3433
]
3534

36-
inst_reqs.append([f"aws_cdk.{x}=={aws_cdk_version}" for x in aws_cdk_reqs])
37-
3835
extra_reqs = {
3936
"test": [
4037
"pytest",
@@ -49,6 +46,7 @@
4946
"flake8",
5047
"nodeenv",
5148
"isort",
49+
"mypy",
5250
"pre-commit",
5351
"pre-commit-hooks",
5452
],

‎stack/hlsconstructs/landsat_incomplete_step_function.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -83,4 +83,4 @@ def __init__(
8383
)
8484
)
8585

86-
self.addLambdasToRole(locals())
86+
self.add_lambdas_to_role(locals())

‎stack/hlsconstructs/landsat_mgrs_partials_step_function.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -136,4 +136,4 @@ def __init__(
136136
role_arn=self.steps_role.role_arn,
137137
)
138138

139-
self.addLambdasToRole(locals())
139+
self.add_lambdas_to_role(locals())

‎stack/hlsconstructs/landsat_mgrs_step_function.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -134,4 +134,4 @@ def __init__(
134134
role_arn=self.steps_role.role_arn,
135135
)
136136

137-
self.addLambdasToRole(locals())
137+
self.add_lambdas_to_role(locals())

‎stack/hlsconstructs/landsat_step_function.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -254,4 +254,4 @@ def __init__(
254254
)
255255
)
256256

257-
self.addLambdasToRole(locals())
257+
self.add_lambdas_to_role(locals())

0 commit comments

Comments
 (0)