Data Misalignment Issue when Importing Multiple Elements from CSV File

[jkmusyoka]

We recently encountered a serious issue while importing daily precipitation, maximum temperature (Tmax), and minimum temperature (Tmin) data from csv file into Climsoft during a project in Zambia.

The problem occurs when one or more of the first two elements in the dataset are missing. In these cases, values from subsequent columns are shifted into preceding columns, resulting in misaligned data. For example:

• If the Tmax value is missing and is the first element in the dataframe, the imported data becomes distorted: Tmin values are placed in the Tmax column, and rainfall values appear in the Tmin column.
• If both Tmax and Tmin values are missing, rainfall values are incorrectly imported into the Tmax column.

This occurs even when the column headers are correctly loaded in Climsoft, leading to incorrect data in the initial table.

I hope this is clear. Otherwise let me know so I can provide more context.

[smachua]

@jkmusyoka, Climsoft import functionality handles text delimited files well unless the wrong menu was chosen according to the data structure in that file. We would like to hear first from ZMD Climsoft managers (especially Consilia) about the problem encountered. Climsoft team aims to build capacity at the NMHSs for the sustainability of Climsoft operations and data processing in general.

[jkmusyoka]

@smachua I have alerted ZMD about this post and asked them to confirm this experience. In the meantime, @Patowhiz has reproduced the issue using some sample data I shared for investigation. I hope we proceed with resolving the issue.

[Patowhiz]

@jkmusyoka Sorry, I realised the branch I initially used to reproduce the bug was about two months old. After switching to a branch with the latest changes, we were unable to reproduce the issue.

Could you let us know which version you were using so we can retrace the bug? Also, could you test with the latest Climsoft release (v4.2.5, published on 8th August) and confirm whether the bug still occurs? We tested with that version and could not reproduce it.