Skip to content

Conversation

@rcjackson
Copy link
Collaborator

It was noted in #952 that the SMPS datastream has multiple ancillary QC variabiles that are related to QC checks from various machine learning algorithms. Therefore, I have added a function in the qcfilter accessor qcfilter.create_dummy_qc_variable to support merging together these QC flags into a dummy QC variable by performing the logical OR on the QC masks. qcfilter.create_dummy_qc_variable function supports specifying which ancillary QC flags to use and which tests to include in the mask. In addition, once the dummy variable is created, the various QC display functions and masking function will automatically detect the presence of this dummy variable when it is created.

Here is an example of its use:

`ds = act.io.arm.read_arm_netcdf(act.testssample_files.EXAMPLE_SMPS,
cleanup_qc=True)
var_name = 'merged_dN_dlogDp'
title = 'Merged Number Size Distribution'
cbar_title = 'dN/dlogD$_p$ (1/cm$^{3}$)'

# Create the dummy QC variable
ds.qcfilter.create_dummy_qc_variable(var_name, 
                                     rm_assessments=['Bad', 'Incorrect', 'Indeterminate', 'Suspect'])
display = act.plotting.TimeSeriesDisplay(ds, subplot_shape=(2,))

display.plot('merged_dN_dlogDp', cvd_friendly=True,
            norm=colors.LogNorm(vmin=100., vmax=10000.), set_title=title, cbar_label=cbar_title,
            ylabel='Pariticle Diameter (nm)', subplot_index=(0,))
display.axes[0].set_yscale('log')
# Plot the QC variable. Note that this function will automatically detect the dummy variable
display.qc_flag_block_plot('merged_dN_dlogDp', subplot_index=(1,))
display.axes[1].set_ylim([0, 21000])

`

@zssherman
Copy link
Collaborator

Left some comments, but also some PEP8 issues as well.

@zssherman
Copy link
Collaborator

@AdamTheisen Whenever you have time, this PR and the other should be good for review.

@AdamTheisen
Copy link
Collaborator

@zssherman I took a look at this one and am wondering if we need to revamp a little based on my comments above. I'll try and find some time to dig in deeper but I want to make sure that it's not going to conflict with any standards when we go to write out files.

@AdamTheisen
Copy link
Collaborator

@rcjackson just as a heads up, I was playing around with this and I added in some modifications to rely more on the already built functions and I think it's working pretty good. One of the big changes is that I am assuming the first variable in the list of ancillary QC variables is the main one to merge the other QC variables into, so we don't have to deal with dummy variable. It also brings over the test meanings and assessments resulting in slightly different results.

If you're okay with it, I'll update things accordingly and commit them to your PR/branch.

image

@rcjackson
Copy link
Collaborator Author

@rcjackson just as a heads up, I was playing around with this and I added in some modifications to rely more on the already built functions and I think it's working pretty good. One of the big changes is that I am assuming the first variable in the list of ancillary QC variables is the main one to merge the other QC variables into, so we don't have to deal with dummy variable. It also brings over the test meanings and assessments resulting in slightly different results.

If you're okay with it, I'll update things accordingly and commit them to your PR/branch.

image

Fine with me @AdamTheisen

to ensure the dataset was updated from ARM QC to the
correct standardized QC.
flag_type : boolean
flag_type : booleany
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added y after boolean that might be on accident

flag_type : booleany
Indicating the QC variable uses flag_values instead of
flag_masks.
ignore_dims : booleany
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here

@zssherman
Copy link
Collaborator

@rcjackson @AdamTheisen Two possible spelling mistakes and some linting errors.

@zssherman
Copy link
Collaborator

Looks good merging!

@zssherman zssherman merged commit 0d6c3c1 into ARM-DOE:main Dec 23, 2025
22 of 36 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Variables with multiple QC variables of varying dimensions

3 participants