Skip to content

[Feat] Add DynamoDB Reader/Writer #497

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Conversation

npow
Copy link

@npow npow commented Dec 28, 2022

@yalimpop yalimpop requested a review from Copilot May 13, 2025 15:23
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR introduces support for DynamoDB as an input and output for Metorikku, enabling both reading from and writing to DynamoDB.

  • New writer and reader implementations for DynamoDB
  • Updates to configuration files and factory methods to recognize DynamoDB
  • Addition of examples, documentation updates, and e2e test scripts for DynamoDB

Reviewed Changes

Copilot reviewed 21 out of 21 changed files in this pull request and generated no comments.

Show a summary per file
File Description
src/main/scala/com/yotpo/metorikku/output/writers/dynamodb/DynamodbOutputWriter.scala Adds the DynamoDB writer leveraging Spark’s DataFrameWriter with DynamoDB options
src/main/scala/com/yotpo/metorikku/input/readers/dynamodb/DynamodbInput.scala Implements the DynamoDB reader based on configurable options
src/main/scala/com/yotpo/metorikku/configuration/job/output/Dynamodb.scala Defines the output configuration for DynamoDB
src/main/scala/com/yotpo/metorikku/configuration/job/input/Dynamodb.scala Introduces the input configuration for DynamoDB
src/main/scala/com/yotpo/metorikku/configuration/job/Output.scala
src/main/scala/com/yotpo/metorikku/configuration/job/Input.scala
src/main/scala/com/yotpo/metorikku/configuration/metric/Output.scala
Integrates DynamoDB as a supported output and input type
examples/dynamodb/*.yaml Provides example configurations for DynamoDB input/output usage
e2e/* Adds new e2e test scripts and docker-compose files for DynamoDB, with one instance in the legacy elasticsearch directory
Comments suppressed due to low confidence (1)

e2e/elasticsearch/scripts/elasticsearch_test.sh:1

  • This script now queries DynamoDB (e.g. via curl to 'http://${DYNAMODB}/movies') rather than Elasticsearch; consider renaming the file to 'dynamodb_test.sh' to better reflect its purpose.
#!/bin/bash

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant