Generate synthetic ADT messages that correlate with synthetic waveform data

[jeremyestein]

To allow for testing outside of the GAE.

Implements #136

[github-actions]

PR checklist

Default guide for a PR (if multiple PRs for the work, only keep one version of it and link to it on the other PRs)

• From the UCLH data science desktop, a validation run has been set off
• Check that content that reveals Epic IP (eg. Clarity/Caboodle queries) has not been checked into a public repo.
• load times
  in UCL teams has been populated with the run information
• During the run, glowroot has been checked for any queries which are taking a substantial proportion of the
  total processing time. This can be useful to identify indexes that are required.
• After the run, look for any unexpected errors in the etl_per_message_logging table, the error_search.sql file
  on the shared drive can be used for this \\sharefs6\UCLH6\EMAP\Shared\EmapSqlScripts\devops\error_search.sql.
  Create an issue if you find an unexpected exception and is not related to the changes you've made, otherwise
  fix them!
• After the run, populate the end time in
  load times
• Let Aasiyah know about the completed validation and give her information on the changes and where to start
  with the validation
• Check validation report and give any feedback to Aasiyah if there are any changes needed on her side,
  iterate on getting the validation to match at least 99% (validation and emap code).

[thompson318]

So far I'm having trouble running this. Comparing logs between develop this branch it seems I'm getting a new error with the waveform-generator.

waveform-generator-1 | Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
waveform-generator-1 | 2025-12-18 08:43:38.980 ERROR 1 --- [ main] o.s.boot.SpringApplication : Application run failed
waveform-generator-1 |
waveform-generator-1 | org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'hl7Generator' defined in URL [jar:file:/app/waveform-generator/target/waveform-generator.jar!/BOOT-INF/classes!/uk/ac/ucl/rits/inform/datasources/waveform_generator/Hl7Generator.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'publisher' defined in URL [jar:file:/app/waveform-generator/target/waveform-generator.jar!/BOOT-INF/lib/emap-interchange-2.7.jar!/uk/ac/ucl/rits/inform/interchange/messaging/Publisher.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'rabbitTemp' defined in class path resource [uk/ac/ucl/rits/inform/interchange/springconfig/DataSourceConfiguration.class]: Unsatisfied dependency expressed through method 'rabbitTemp' parameter 1; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'connectionFactory' defined in class path resource [uk/ac/ucl/rits/inform/interchange/springconfig/DataSourceConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.amqp.rabbit.connection.ConnectionFactory]: Factory method 'connectionFactory' threw exception; nested exception is java.lang.NullPointerException: Cannot invoke "java.lang.Integer.intValue()" because the return value of "org.springframework.boot.autoconfigure.amqp.RabbitProperties.getPort()" is null

I think this is because my hl7-reader fails with java.net.UnknownHostException: host.docker.internal. This doesn't matter on develop as the waveform reader doesn't need it, but it does on this branch. Presumably I have to configure my hl7-reader differently to run locally.

[jeremyestein]

So far I'm having trouble running this. Comparing logs between develop this branch it seems I'm getting a new error with the waveform-generator.

waveform-generator-1 | Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
waveform-generator-1 | 2025-12-18 08:43:38.980 ERROR 1 --- [ main] o.s.boot.SpringApplication : Application run failed
waveform-generator-1 |
waveform-generator-1 | org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'hl7Generator' defined in URL [jar:file:/app/waveform-generator/target/waveform-generator.jar!/BOOT-INF/classes!/uk/ac/ucl/rits/inform/datasources/waveform_generator/Hl7Generator.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'publisher' defined in URL [jar:file:/app/waveform-generator/target/waveform-generator.jar!/BOOT-INF/lib/emap-interchange-2.7.jar!/uk/ac/ucl/rits/inform/interchange/messaging/Publisher.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'rabbitTemp' defined in class path resource [uk/ac/ucl/rits/inform/interchange/springconfig/DataSourceConfiguration.class]: Unsatisfied dependency expressed through method 'rabbitTemp' parameter 1; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'connectionFactory' defined in class path resource [uk/ac/ucl/rits/inform/interchange/springconfig/DataSourceConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.amqp.rabbit.connection.ConnectionFactory]: Factory method 'connectionFactory' threw exception; nested exception is java.lang.NullPointerException: Cannot invoke "java.lang.Integer.intValue()" because the return value of "org.springframework.boot.autoconfigure.amqp.RabbitProperties.getPort()" is null

I think this is because my hl7-reader fails with java.net.UnknownHostException: host.docker.internal. This doesn't matter on develop as the waveform reader doesn't need it, but it does on this branch. Presumably I have to configure my hl7-reader differently to run locally.

The port seems to be coming through as null. Check that your config/waveform-generator-config-envs has a value for SPRING_RABBITMQ_PORT. If not, you might need to regenerate it with emap setup -g. If still not, check your global-configuration.yaml.

Unrelated to this, I'm about to push a bug fix for null locations into this branch.

[thompson318]

The port seems to be coming through as null. Check that your config/waveform-generator-config-envs has a value for SPRING_RABBITMQ_PORT. If not, you might need to regenerate it with emap setup -g. If still not, check your global-configuration.yaml.

Seem to have got this working, after adding the rabbitmq port. I also had to add values to the config/waveform-reader-config-envs. WAVEFORM_HL7_SAVE_ENABLED, WAVEFORM_HL7_TEST_DUMP_FILE, and WAVEFORM_HL7_SAVE_ARCHIVE_TIME_SLOT_MINUTES were all blank after running emap setup -g and caused the waveform-reader to break.

[thompson318]

When I say working, I mean running, but it is throwing lots of errors and doesn't seem to put anything in the queue. Errors:

waveform-generator-1 | 2025-12-18 14:12:41.203 ERROR 1 --- [ scheduling-1] u.a.u.r.i.d.waveform.LocationMapping : Could not map capsule location string SOMEWHERE_NOT_ICU
waveform-generator-1 | 2025-12-18 14:12:41.209 ERROR 1 --- [ scheduling-1] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task

[jeremyestein]

When I say working, I mean running, but it is throwing lots of errors and doesn't seem to put anything in the queue. Errors:

waveform-generator-1 | 2025-12-18 14:12:41.203 ERROR 1 --- [ scheduling-1] u.a.u.r.i.d.waveform.LocationMapping : Could not map capsule location string SOMEWHERE_NOT_ICU
waveform-generator-1 | 2025-12-18 14:12:41.209 ERROR 1 --- [ scheduling-1] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task

This was fixed very recently. Are you definitely running the latest version?

[jeremyestein]

The port seems to be coming through as null. Check that your config/waveform-generator-config-envs has a value for SPRING_RABBITMQ_PORT. If not, you might need to regenerate it with emap setup -g. If still not, check your global-configuration.yaml.

Seem to have got this working, after adding the rabbitmq port. I also had to add values to the config/waveform-reader-config-envs. WAVEFORM_HL7_SAVE_ENABLED, WAVEFORM_HL7_TEST_DUMP_FILE, and WAVEFORM_HL7_SAVE_ARCHIVE_TIME_SLOT_MINUTES were all blank after running emap setup -g and caused the waveform-reader to break.

Anything you add manually to config/waveform-reader-config-envs will get overwritten next time you run emap setup -g, IIRC. Values are supposed to go in your global-configuration.yaml and should get written into the former because of their presence in waveform-reader-config-envs.EXAMPLE.
Were they blank or absent in your global-configuration.yaml?

[thompson318]

This was fixed very recently. Are you definitely running the latest version?

I don't think I was. Now I get;

waveform-generator-1 | 2025-12-18 15:20:33.282 INFO 1 --- [ scheduling-1] u.a.u.r.i.d.w.Hl7Generator : Starting scheduled message dump (from 2024-01-02T12:00:15Z for 5000 milliseconds)
waveform-generator-1 | 2025-12-18 15:20:33.282 INFO 1 --- [ scheduling-1] u.a.u.r.i.d.w.Hl7Generator : Making HL7 messages starting at time 2024-01-02T12:00:15Z for 5000 milliseconds
waveform-generator-1 | 2025-12-18 15:20:33.285 INFO 1 --- [ scheduling-1] u.a.u.r.i.d.w.Hl7Generator : Not generating data for empty locations: [UCHT03ICURM02]
waveform-generator-1 | 2025-12-18 15:20:33.285 INFO 1 --- [ scheduling-1] u.a.u.r.i.d.w.Hl7Generator : Sending 200 HL7 messages
waveform-generator-1 | 2025-12-18 15:20:33.287 ERROR 1 --- [ scheduling-1] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task

Still doesn't seem to be anything reaching the rabbitmq queue.

[thompson318]

Were they blank or absent in your global-configuration.yaml?

Absent.

[jeremyestein]

Were they blank or absent in your global-configuration.yaml?

Absent.

This is a known usability deficit. You have to manually update your global-configuration.yaml when config variables have been added/removed in emap-setup/global-configuration-EXAMPLE.yaml . There's nothing to warn you if you haven't done this.
WAVEFORM_HL7_SAVE_ENABLED and WAVEFORM_HL7_SAVE_ARCHIVE_TIME_SLOT_MINUTES were added quite recently, but WAVEFORM_HL7_TEST_DUMP_FILE has been there over a year so I'm surprised that was absent.

[thompson318]

This "works" for me now, it was just a question of getting my configuration right. However, I now get multiple matches when querying the fakeuds created, this is a log message from the waveform-controller.

Ambiguous or non existent match: Wrong number of rows returned from database. 29 != 1, for T03^T03 SR01^SR01-01:2024-01-02 12:03:25+00:00

In summary the query is returning 29 possible csn matches for the patient in T03^T03 SR01^SR01-01 at 2024-01-02 12:03:25. I'll need to investigate.

EDIT: After digging around in my fakeuds, I think this may be due to too many entries in the hospital visit table with NULL discharge_datetimes. For example:

select * from fakeuds.schemaname.location_visit where location_id = 1369 and admission_datetime < '2024-01-02:14:18:55' and ( discharge_datetime > '2024-01-02:14:18:55' or discharge_datetime is null)

returned 20 rows. If I remove the or discharge_datetime is null clause I get 1 row. I also tested this with a modified waveform-controller and got the same result (the multiple matches were no longer seen).

EDIT: I think this happened because my fakeuds was not emptied from previous runs. Clearing the schemas in the fakeuds has solved the problem. (although I couldn't find an easy way to clear the schemas)

[thompson318]

I'm now able to run this and it works, enabling better development of the waveform controller. I had problems, firstly with getting the global configuration right, but that is mainly lack of familiarity with how emap setup works.
Second problem was caused by there being old data in the fakeuds schema. I was able to clear this out using an SQL script, but it would be good to document this somewhere, and I'm not sure what the best way to do it is. Alternatively, could we pass an option to clear the fakeuds schema on startup?