Save ICON-EU GRIB files from DWD's FTP server to cloud object storage

[JackKelly]

Motivation

It seems that people are generally in favour of the idea of us archiving ICON-EU GRIB files on cloud object storage. For example, in this comment, @aldenks says:

I like the idea of archiving the complete gribs (e.g. on Source Coop) and then starting with a more limited surface variable/level set in the Zarr and seeing what people ask to add in. For AI WP model training I'd imagine almost everyone wants the whole domain at each timestep, meaning the native "chunking" of the grib files is actually optimal and we could expose it as a virtualzarr/virtual icechunk dataset and get equivalent read performance.

And, TBH, it's turning out to be quite hard to find a high-quality archive of ICON-EU data. (DWD delete GRIBs from their HTTP server after 24 hours). So archiving the GRIBs would be a useful contribution in its own right. And, before we know it, a year will have elapsed and we'll have a reasonable-sized archive.

And, if I've understood correctly, RegionJob.operational_update_jobs kind of expects to have access to an archive of GRIB files that spans from whenever operational_update_jobs was last run until now. Which could be more than 24 hours if, for example, the ICON-EU reformatters code fails on a Friday night, and I don't fix it until Monday. So I'm pausing work on PR #226, until we have an operational service that copies GRIBs and stores them on object storage.

Implementation

How would you like me to implement the script that grabs GRIBs from DWD's HTTP server and copies them to Source Coop?

I'm imagining a fairly simple script Python that you could run using cron (or similar). The script would:

• Take command line arguments for the source and destination. (e.g. source = https://opendata.dwd.de/weather/nwp/icon-eu/grib/, destination = s3://sourcecoop/dynamical/dwd/icon-eu)
• Check what's already in the destination
• Copy anything that's in the source that's not already in the destination
• Check the filesizes of the files in the source, and compare with the filesizes in the destination, and re-copy if any filesizes are too small (which indicates a partial transfer).
• Log using Python's standard logging mechanism

I'd probably use obstore instead of fsspec.

Does that sound like something I should build? Or would you like it implemented differently? (No worries either way!)

[aldenks]

@JackKelly +1 on the motivation and this sounds like a great thing to get to. Could we get your #226 even in very rough form on main before you start on this? just want to not have it get too out of step with main and give you a big merge headache, totally fine if its not complete as long as our repo checks are passing.

In terms of specific implementation, you can get a pretty easy deployment by, in your DynamicalDataset subclass:

• implement a new method, ill call it archive_grib_files
• overriding get_cli and calling super, then adding your own app.command that runs the copying process you described
• Return an addition CronJob from your dynamical dataset operational_kubernetes_resources function, calling the new command you created in get_cli

[JackKelly]

Hey @aldenks, this all sounds good, thanks for the reply... hopefully I'll have time to work on this again next week...

[JackKelly]

How much raw DWD data should we archive as GRIBs on Source CoOp? I'd be tempted to suggest we archive "everything", and we can delete stuff later if we run out of space?! The following NWPs appear to be freely available on DWD's HTTP server (with a 24-hour rolling archive):

• ICON-EU
  • ICON-EU-EPS (ensemble)
• ICON (global)
  • ICON-EPS (global ensemble)
• ICON-D2 (high res over Germany & neighbours)
  • ICON-D2-EPS
• ICON-ART (global NWP. "ART" stands for "Aerosols & Reactive Trace" gases. More details about ICON-ART.)
  • ICON-ART-EPS
  • ICON-ART-EU
  • ICON-ART-EU-EPS

Maybe ICON-ART is the lowest priority, because Schroedter-Homscheidt et al. showed at EMS2025 that higher resolution NWPs (without explicit modelling of aerosols) out-performed a lower-resolution aerosol-aware NWP.

And we maybe want at most 4 model runs per NWP per day?

[aldenks]

I like starting with everything besides ART, although I'd really just get ICON-EU going first and fast follow with the rest. Do you have an approx idea of the storage usage per day/year?

[JackKelly]

I've asked my friend who's been archiving ICON-EU about the storage space required... I'll get back to you...

To further refine the question of what to grab for each DWD NWP (except ICON-ART): Maybe we'd grab all variables for all model levels & surface variables. But we'd not grab the pressure level files?

[JackKelly]

In terms of specific implementation, you can get a pretty easy deployment by, in your DynamicalDataset subclass...

Given that the plan is to archive multiple DWD NWPs, should I put the "download & archive" code in a new DynamicalDataset subclass, called something like DwdArchiveDynamicalDataset? This would download gribs for ICON, ICON-EU, D2, and the ensemble versions. If I go this route, can I just delete the DwdArchiveTemplateConfig and DwdArchiveRegionJob files?

And can the operational system be configured to run DwdArchiveDynamicalDataset before DwdIconEuForecastDynamicalDataset (so DwdIconEuForecastDynamicalDataset can pull data from our archive)? This adds a little latency to the creation of the ICON-EU Zarr. But has the advantage that the code will do the right thing if DwdIconEuForecastDynamicalDataset fails and isn't fixed for a few days, and so has to back-fill from our archive. And reduces load on DWD's server!

[JackKelly]

Do you have an approx idea of the storage usage per day/year?

My friend's archive of ICON-EU is about 5 GiB per model run (they only store one model run per day, and aren't storing all model levels)

[JackKelly]

My friend recommends that we compress the DWD grib files using zstd, and concatenate zstd files.

Otherwise, if we store individual GRIBs then we'll end up with millions of objects, which slows down pipelines and makes it hard to rename, and increases the cost of object storage operations.

The main disadvantage of concatenation is that folks won't have such fine-grained control over how much data they download.

My friend stores entire ICON-EU model runs in a zstd file (one zstd file per model run). Which is perhaps a bit too coarse for our use-case? Alternatively, we could have smaller concatenations, e.g. 1 file per variable per model run? (i.e. one file would contain every timestep and every vertical level of a variable)?

Any thoughts?

[JackKelly]

On second thoughts: DWD already publishes GRIB files compressed using bzip2. So maybe we shouldn't decompress the grib2.bz2 files, and then compress using zstd, and then concat. Instead, it'd be simpler to just leave the GRIB files as .bz2, and combine multiple .bz2 files into a tar.

So, unless anyone objects, I'll concat all the files for each NWP variable. So, for each ICON-EU model run, you'd have one .tar file per variable. Each tar file would contain all the model levels and forecast steps for that variable, stored as .grib2.bz2 files within the tar. How does that sound?

[aldenks]

@JackKelly one downside to this is if we're making an "analysis" dataset from the archives then we can't download just the first few, shortest lead time, hours. Have you back of the enveloped the operations costs of not combining them?

[JackKelly]

Ah, yeah, good point! I haven't done any calcs yet (I must admit I'm not that experienced at doing those calcs! I still find cloud costs pretty confusing!). But maybe could just go ahead and store the individual files (without concat) and see how we go?

[JackKelly]

To clarify, here's my current plan. Please shout if this isn't the correct path to follow:

• Implement a new DwdArchiveDynamicalDataset that copies *.grib2.bz2 files from DWD's FTP server to Source Co-Op (on S3)
• The *.grib2.bz2 files will not be concatenated. The code will simply copy the files from DWD's FTP server to Source Co-Op. (With lots of logic to detect failed transfers & to retry).
• The *.grib2.bz2 files on Source Co-Op will be put into "directories". The directory structure will look like this:
  • s3://dynamical-source-co-op/nwp
  • /<provider>
  • /<model>
  • /<spatial grid> (because we know that DWD will soon switch from a regular lat lon grid to the native triangular grid)
  • /<init datetime in shortened ISO 8601 format: YYYY-MM-DDTHHZ>
  • /<NWP variable name (using the same name as the NWP provider)>
  • /<unmodified filename and unmodified file contents from the NWP provider>
• UPDATE: We'll only transfer the "model-level" and "single-level" files. We WON'T archive the pressure-level files.

For example:

s3://dynamical-source-co-op/nwp/dwd/icon-eu/regular-lat-lon/2025-10-08T00Z/t_2m/icon-eu_europe_regular-lat-lon_single-level_2025100800_000_T_2M.grib2.bz2

• My code will saturate the network bandwidth. To do so, it'll implement a multi-threaded FTP client in Python. As soon as a grib2.bz2 file finishes downloading from DWD's FTP server and is stored in RAM, that file will be sent to S3.

[aldenks]

that all sounds great with the exception that i the bucket/start of the path will probably be a little different, because we'll make a new repository on Source Coop (here it is https://source.coop/dynamical/dwd-icon-grib). I'll message you about how to write to it (see our shared slack)