Skip to content

Feature/grid file etl #38

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
nick-j-roberts wants to merge 19 commits into
base: main
Choose a base branch
from
Open

Feature/grid file etl #38

nick-j-roberts wants to merge 19 commits into from

Conversation

@nick-j-roberts
Copy link

@nick-j-roberts nick-j-roberts commented Feb 15, 2024 •
edited
Loading

  • Gets rid of outdated ETL scripts which contain functionality now superseded by plugins
  • Adds more readmes to repo
  • Creates new ETL which:
    • queries meilisearch for top n storms per year
    • transforms data fetched from meilisearch into format expected by .grid file writer
    • creates a zipped package including a .grid file and a subdirectory of dss files pulled from s3
    • uploads zipped package to s3

Nick Roberts added 19 commits February 15, 2024 14:10
Add basic readmes for all plugins
91201d2
Get rid of ETLs which have functionality superceded by plugins
030e380
Delete docker files associated with deleted etls
72c58fe
Add ETL for creating ranked GRID package
f7799d1
Add basic readmes for ETLs
2fb7218
Update general repo readme
db4ede8
correct copy error in docker file
6aaeeb4
add more cli params for greater customization
0deaa4e
update docker image addresses
b76eb4d
correct typos or clarify information
187856b
specify DSS grids generated as a part of SST are precipitation values
e81b822
add reqs for s3 interactions and zarr to DSS conversion used in tempe…
e92be26 
…rature addition
add option to write out new DSS files with temperature for top storms…
7013936 
… and package in GRID
copy over new dependency in docker container
b5fa67f
add comment explaining why an hour is added
b2ff87a
create a default save location for DSS files generated with temperatu…
89eb762 
…re added
use existing temperature and precip DSS files if they exist in expect…
7dde4bd 
…ed location
Add more logging to see when zarr data is missing
2ca7e8d
Allow missing data during DSS generation stage; error when writing to…
46dc555 
… GRID
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@nick-j-roberts