Download the CORNEOUS 404 dataset issue

[Sherlyunl]

We are encountering difficulties when attempting to download the CORNEOUS 404 dataset (https://app.globus.org/file-manager?destination_id=0a0d322b-1adc-4c93-bfdb-18a6a848479e&destination_path=%2F&two_pane=true) to HPC. Currently, we can only download two files at a time, which is quite time-consuming given that we need to download a complete set for a 30-year period.

Please guide on how we might download the dataset more efficiently.

[amsnyder]

Hey @Sherlyunl - thanks for the question. I'm not sure why you are being limited to two files at a time - I do not experience the same unfortunately. A few options you could try to get around the issue are (in order of most to least recommended):

1. If you only need precipitation and temperature data (and not the other downscaled variables), you can read the data directly into your workflow (without doing any data transfers) from our public zarr store. This dataset is labeled conus404-hourly-ba-osn in our conus404 subcatalog of the HyTEST intake catalog.
2. If you have access to the USGS Hovenweep supercomputer, we have a copy of the data exactly as it appears in the data release stored in the following location: /caldera/hovenweep/projects/usgs/water/impd/hytest/conus404-tpBiasCorr
3. you could use a globus batch transfer to see if this would allow you to transfer more than one file (but the reads will still be slow because you are reading off tape drives)
4. If none of these work for you, I can try to put a copy of the files (matching what you see in the data release) onto the same location we have the zarrs stored and you could read the files directly using any tool that can interface with the AWS S3 API. I can give more info on this if we have to go this route.