autostart of gbs_prism has an error when launched prior to the run directory being synced off the staging server

[BenjaminJPerry]

Lab staff start the sequencer (usually EoD) and then want to setup the gbs_prism autostart. However, the "robocopy" sync of the run directory (created by the sequencer) only runs on the hour, and thus a situation arises where they have to wait for the run directory to be robocopied prior to being able to setup the gbs_prism autostart.

login-1.eri.agresearch.co.nz-20251113-16:44:41 | ~ ~ $ kinit

Password for [bairdh@agresearch.co.nz](mailto:bairdh@agresearch.co.nz):

login-1.eri.agresearch.co.nz-20251113-16:45:02 | ~ ~ $ module load gbs_prism

login-1.eri.agresearch.co.nz-20251113-16:45:33 | ~ ~ $ redun run $GBS_PRISM/pipeline.py main --run 251113_A01439_0424_AHGNGJDRX7

[redun] redun :: version 0.32.0

[redun] config dir: /nix/store/253gzpm26mqcqbj393i9df7isid57aw0-gbs-prism-bundle-2.4.0/config/redun.prod

[redun] Start Execution 37a56982-db23-4e48-8871-a27ee7737f06:  redun run /nix/store/253gzpm26mqcqbj393i9df7isid57aw0-gbs-prism-bundle-2.4.0/pipeline.py main --run 251113_A01439_0424_AHGNGJDRX7

[redun] Tasks will require namespace soon. Either set namespace in the `@task` decorator or with the module-level variable `redun_namespace`.

tasks needing namespace: barcode_yields.py:collate_barcode_yields, barcode_yields.py:plot_barcode_yields, bcl_convert.py:bcl_convert, bwa.py:bwa_aln_one, bwa.py:bwa_aln_all, bwa.py:bwa_samse_one, bwa.py:bwa_samse_all, collate_tags_reads.py:collate_tags_reads, collate_tags_reads.py:collate_tags_reads_kgdstats, cutadapt.py:cutadapt_one, cutadapt.py:cutadapt_all, dedupe.py:dedupe_one, dedupe.py:dedupe_all, tassel3.py:get_fastq_to_tag_count, tassel3.py:get_tag_count, tassel3.py:merge_taxa_tag_count, tassel3.py:tag_count_to_tag_pair, tassel3.py:tag_pair_to_tbt, tassel3.py:tbt_to_map_info, tassel3.py:map_info_to_hap_map, tag_count.py:_merge_results_and_counts, tag_count.py:create_consolidated_tag_count, demultiplex.py:demultiplex, fake_bcl_convert.py:fake_bcl_convert, fake_bcl_convert.py:real_or_fake_bcl_convert, fastq_sample.py:_sample_minsize_if_required, fastq_sample.py:fastq_sample_one, fastq_sample.py:fastq_sample_all, fastqc.py:fastqc_one, fastqc.py:fastqc_all, keyfiles.py:get_gbs_keyfiles, keyfiles.py:get_keyfile_for_tassel, keyfiles.py:get_keyfile_for_gbsx, kmer_analysis.py:kmer_analysis_one, kmer_analysis.py:kmer_analysis_all, multiqc.py:multiqc, sample_sheet.py:cook_sample_sheet, samtools.py:bam_stats_one, samtools.py:bam_stats_all, samtools.py:collate_mapping_stats, kgd.py:kgd, gupdate.py:import_gbs_read_tag_counts, gupdate.py:create_cohort_gbs_kgd_stats_import, gupdate.py:import_gbs_kgd_stats, gupdate.py:import_gbs_kgd_cohort_stats, gusbase.py:gusbase, tags.py:get_tags_reads_summary, tags.py:get_tags_reads_list, tags.py:get_tags_reads_cv, tags.py:get_tags_reads_plots, unblind.py:get_unblind_script, unblind.py:unblind_one, unblind.py:unblind_optional, unblind.py:unblind_all, unblind.py:unblind_each, warehouse.py:get_genophyle_export, warehouse.py:import_genophyle_gbs_import_file, warehouse.py:warehouse

[redun] Run    Job f257bf6e:  redun.root_task(expr=QuotedExpression(agr.gbs_prism.main(run='251113_A01439_0424_AHGNGJDRX7'))) on default

[redun] Run    Job 156c548d:  agr.gbs_prism.main(run='251113_A01439_0424_AHGNGJDRX7', path_context={'seq_root': '/agr/persist/projects/2024_illumina_sequencing_g/run_data', 'postprocessing_root': '/agr/persist/projects/2024_illumina_sequencing_g/postprocessing', 'keyfiles_dir': '/agr/persist/pro...) on default

[redun] Run    Job 43469787:  agr.gbs_prism.run_stage1(seq_root='/agr/persist/projects/2024_illumina_sequencing_g/run_data', postprocessing_root='/agr/persist/projects/2024_illumina_sequencing_g/postprocessing', gbs_backup_dir='/agr/scratch/projects/2024_illumina_sequencing_g/backups', keyfiles_dir='/agr/persist/projects/2024_illumina_sequencing_g/key-files', fastq_link_farm='/agr/persist/projects/2024_illumina_sequencing_g/fastq-link-farm', run='251113_A01439_0424_AHGNGJDRX7', job_context=<agr.redun.job_attributes.JobContext object at 0x7fba5c9aa4b0>) on default

[redun] *** Reject Job 43469787:  agr.gbs_prism.run_stage1(seq_root='/agr/persist/projects/2024_illumina_sequencing_g/run_data', postprocessing_root='/agr/persist/projects/2024_illumina_sequencing_g/postprocessing', gbs_backup_dir='/agr/scratch/projects/2024_illumina_sequencing_g/backups', keyfiles_dir='/agr/persist/projects/2024_illumina_sequencing_g/key-files', fastq_link_farm='/agr/persist/projects/2024_illumina_sequencing_g/fastq-link-farm', run='251113_A01439_0424_AHGNGJDRX7', job_context=<agr.redun.job_attributes.JobContext object at 0x7fba5c9aa4b0>):

[redun] Run    Job 7bef40b3:  agr.gbs_prism.recover(results=(SequencerRunError('no such directory /agr/persist/projects/2024_illumina_sequencing_g/run_data/251113_A01439_0424_AHGNGJDRX7'), SequencerRunError('no such directory /agr/persist/projects/2024_illu...) on default

[redun]

[redun] | JOB STATUS 2025/11/13 16:46:54

[redun] | TASK                                 PENDING RUNNING  FAILED  CACHED    DONE   TOTAL

[redun] |

[redun] | ALL                                        5       0       1       0       3       9

[redun] | agr.gbs_prism.main                         0       0       0       0       1       1

[redun] | agr.gbs_prism.recover                      0       0       0       0       1       1

[redun] | agr.gbs_prism.reports.create_reports       1       0       0       0       0       1

[redun] | agr.gbs_prism.run_stage1                   0       0       1       0       0       1

[redun] | agr.gbs_prism.run_stage2                   1       0       0       0       0       1

[redun] | agr.gbs_prism.run_stage3                   1       0       0       0       0       1

[redun] | agr.util.await_results                     1       0       0       0       0       1

[redun] | redun.root_task                            0       0       0       0       1       1

[redun] | warehouse                                  1       0       0       0       0       1

[redun]

[redun]

[redun] Execution duration: 0.66 seconds

(SequencerRunError('no such directory /agr/persist/projects/2024_illumina_sequencing_g/run_data/251113_A01439_0424_AHGNGJDRX7'),

SequencerRunError('no such directory /agr/persist/projects/2024_illumina_sequencing_g/run_data/251113_A01439_0424_AHGNGJDRX7'),

SequencerRunError('no such directory /agr/persist/projects/2024_illumina_sequencing_g/run_data/251113_A01439_0424_AHGNGJDRX7'),

SequencerRunError('no such directory /agr/persist/projects/2024_illumina_sequencing_g/run_data/251113_A01439_0424_AHGNGJDRX7'),

SequencerRunError('no such directory /agr/persist/projects/2024_illumina_sequencing_g/run_data/251113_A01439_0424_AHGNGJDRX7'))
"""

[tesujimath]

Well the existing behaviour is by design, and you should review carefully whether you really want to change it.

Current behaviour is:

1. check immediately to see if the top-level run directory exists, and if not, fail fast
2. otherwise wait for a run-in-progress to complete

This could be changed to:

1. wait for a long time for the top level directory to be created
2. then wait for the run to be completed

I would argue that this could be problematic. Any configuration error where the run directory is incorrect would not be discovered for hours, whereas the existing behaviour allows for a quick correction because of the fast fail.

There is surely a way for the staging sync to create top-level directories very quickly (every few minutes), and continue to run the full robocopy every hour or so. IMO that would be a much better solution, without this drawback.

[tesujimath]

I'm removing the bug label for now, as given my comment above that label seems incorrect. 😁

[oldrooster]

Agreed, this should be a straight forward script to create any new directories. what is the maximum tolerable interval that would work @BenjaminJPerry? e.g. 10 minutes?