Ci test handler testing2 #2433

vaclav-ssvlabs · 2025-08-09T10:47:44Z

No description provided.

vaclav-ssvlabs · 2025-09-18T11:39:48Z

/test

ssv-test-bot · 2025-09-18T11:39:51Z

🟢 Test run for PR #2433

Branch: ci-test-handler-testing2
Commit: 2a71a70

Status

2025-09-18T11:39:51Z Starting test run
2025-09-18T11:39:52Z Test run submitted successfully for commit 2a71a70
2025-09-18T11:39:53Z container image for commit 2a71a7068f767db857088ace945a72882eb5871f is build and present in container registry
2025-09-18T11:39:54Z automatically allocated node range cluster=300,301,302,303 for test run on network=hoodi_stage
2025-09-18T11:39:56Z deploying commit 2a71a7068f767db857088ace945a72882eb5871f to ssv nodes 300,301,302,303
2025-09-18T11:40:42Z started test executor with tests=(sanity,e2m) and network=hoodi_stage

Click to expand executor logs

═══════════════════════════════════════════════════════════════════════════════════════════════════════
                🧪  WELCOME TO Executor - SSV's Distributed Test Automation Execution Layer  🧪            

                                  ___ __  __ ___  ___  _   _  _____  ___   ___ 
                                 | __|\\ \\/ /| __|/ __|| | | ||_   _|/ _ \\ | _ \\
                                 | _|  >  < | _|| (__ | |_| |  | | | (_) ||   /
                                 |___|/_/\\_\\|___|\\___| \\___/   |_|  \\___/ |_|_\\
                                        

═══════════════════════════════════════════════════════════════════════════════════════════════════════
{"time":"2025-09-18T11:40:44.229480614Z","level":"INFO","msg":"🐳 Docker mode: Using environment variables for config generation"}
{"time":"2025-09-18T11:40:44.238250845Z","level":"INFO","msg":"🌐 Orchestrator API Client initialized","endpoint":"http://aetheria-orchestrator.aetheria.svc:8080"}
{"time":"2025-09-18T11:40:44.238327363Z","level":"INFO","msg":"📝 Requesting configuration generation from orchestrator API","network":"hoodi_stage","clusters":[300,301,302,303],"tests":"(sanity,e2m)"}
{"time":"2025-09-18T11:40:44.333439575Z","level":"INFO","msg":"✅ Configuration generated successfully via API","resource_usage_id":"9d1f0717736506137b674da48e386807"}
{"time":"2025-09-18T11:40:44.334547306Z","level":"INFO","msg":"🚀 EXECUTOR Initialized Successfully"}
{"time":"2025-09-18T11:40:44.334626443Z","level":"INFO","msg":"⚙️ Test configuration loaded","network":"hoodi_stage","nodes":[300,301,302,303],"test_type":{"Sanity":{"ShouldRun":true,"E2M":true,"BulkAmount":0},"Bulk":{"ShouldRun":false,"E2M":false,"BulkAmount":0},"Isolated":{"Tests":null}}}
{"time":"2025-09-18T11:40:48.7415283Z","level":"INFO","msg":"Generated operator pubkeys file","module":"scanner","path":"/app/shared/data/operators/operator-pubkeys-hoodi_stage.json"}
{"time":"2025-09-18T11:40:48.74174152Z","level":"INFO","msg":"Loki Started...","module":"Loki"}
{"time":"2025-09-18T11:40:48.741798405Z","level":"INFO","msg":"HTTPRequester Started...","module":"HTTPRequester"}
{"time":"2025-09-18T11:40:48.741822929Z","level":"INFO","msg":"Validation client initialized","module":"main","loki_enabled":true,"e2m_enabled":true}
{"time":"2025-09-18T11:40:48.741895078Z","level":"INFO","msg":"🔑 KEY STEP: Starting Isolated Tests Concurrently if exists...","module":"main"}
{"time":"2025-09-18T11:40:48.741994304Z","level":"INFO","msg":"🔑 KEY STEP: Starting Sanity Test Suite","module":"main"}
{"time":"2025-09-18T11:40:48.742214959Z","level":"INFO","msg":"No isolated tests configured, skipping","module":"main"}
{"time":"2025-09-18T11:40:48.803723537Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":300}
{"time":"2025-09-18T11:40:48.820570948Z","level":"INFO","msg":"Available slots","test":"sanity","node":300,"slots":3000}
{"time":"2025-09-18T11:40:48.820660287Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":301}
{"time":"2025-09-18T11:40:48.828554967Z","level":"INFO","msg":"Available slots","test":"sanity","node":301,"slots":399}
{"time":"2025-09-18T11:40:48.828638036Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":302}
{"time":"2025-09-18T11:40:48.835598862Z","level":"INFO","msg":"Available slots","test":"sanity","node":302,"slots":299}
{"time":"2025-09-18T11:40:48.835706161Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":303}
{"time":"2025-09-18T11:40:48.843646954Z","level":"INFO","msg":"Available slots","test":"sanity","node":303,"slots":717}
{"time":"2025-09-18T11:41:23.970201868Z","level":"INFO","msg":"Extracted keyshares path","test":"sanity","module":"keys","path":"/app/shared/data/keyshares/0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754.json"}
{"time":"2025-09-18T11:41:23.970293149Z","level":"INFO","msg":"Key shares file generated successfully","test":"sanity","keyshare_path":"/app/shared/data/keyshares/0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754.json"}
{"time":"2025-09-18T11:41:25.574141038Z","level":"INFO","msg":"Allowance fetched successfully","test":"sanity","module":"contract","account":"0x91e32efb8139cd88cae0df30d2bf471294c6ed27","allowance":"99456893999200282400000000"}
{"time":"2025-09-18T11:41:25.57423214Z","level":"INFO","msg":"Threshold value","test":"sanity","module":"contract","threshold":"9000000000000000000000"}
{"time":"2025-09-18T11:41:25.574268906Z","level":"INFO","msg":"Sufficient allowance","test":"sanity","module":"contract","allowance margin":"6561000000000000000000000000000000000000000000000000000000000000000000000000000000000000"}
{"time":"2025-09-18T11:41:25.578631663Z","level":"INFO","msg":"Allowance fetched successfully","test":"sanity","module":"contract","account":"0x91e32efb8139cd88cae0df30d2bf471294c6ed27","allowance":"99456893999200282400000000"}
{"time":"2025-09-18T11:41:25.578791496Z","level":"INFO","msg":"Allowance","test":"sanity","module":"contract","allowance":"99456893999200282400000000"}
{"time":"2025-09-18T11:41:25.707418929Z","level":"INFO","msg":"Registering validator","test":"sanity","module":"contract","tx_hash":"0x8e7784029da94324ee9860c2e051ed4b20824773ba5f068cec6c6d8254a2ae08"}
{"time":"2025-09-18T11:41:25.711824625Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0x8e7784029da94324ee9860c2e051ed4b20824773ba5f068cec6c6d8254a2ae08"}
{"time":"2025-09-18T11:41:35.723820896Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":2,"sleep":20,"tx_hash":"0x8e7784029da94324ee9860c2e051ed4b20824773ba5f068cec6c6d8254a2ae08"}
{"time":"2025-09-18T11:41:55.731761148Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0x8e7784029da94324ee9860c2e051ed4b20824773ba5f068cec6c6d8254a2ae08"}
{"time":"2025-09-18T11:41:55.731900967Z","level":"INFO","msg":"✅ SUCCESS: Contract Register Successful","test":"sanity","txHash":"0x8e7784029da94324ee9860c2e051ed4b20824773ba5f068cec6c6d8254a2ae08","block_hash":"0x18a61ddc5c4c7951a16a1d1630bf346b28730ac6c7d2d28897933258c3117a80"}
{"time":"2025-09-18T11:41:55.750206919Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"sanity","module":"helpers","pubkey":"0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754","timeToSleep":10,"attempt":1}
{"time":"2025-09-18T11:42:05.759869618Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"sanity","module":"helpers","pubkey":"0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754","timeToSleep":20,"attempt":2}
{"time":"2025-09-18T11:42:25.782060417Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"sanity","module":"helpers","pubkey":"0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754","timeToSleep":40,"attempt":3}
{"time":"2025-09-18T11:43:05.827628827Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"sanity","module":"helpers","pubkey":"0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754","timeToSleep":80,"attempt":4}
{"time":"2025-09-18T11:44:25.918564632Z","level":"INFO","msg":"Registration epoch","module":"main","module":"e2m-validator","epoch":41620,"indices":"1102420"}
{"time":"2025-09-18T11:44:40.931329315Z","level":"ERROR","msg":"Validator did not attest 4 epochs after registration","test":"sanity","index":"1102420"}
{"time":"2025-09-18T11:44:42.492085048Z","level":"INFO","msg":"Liquidating cluster","test":"sanity","module":"contract","tx_hash":"0xc98aa37683cadf3b00fb30b247e607ffad5fadcb2d3884fa507bae8bfdeb8785"}
{"time":"2025-09-18T11:44:42.496911397Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0xc98aa37683cadf3b00fb30b247e607ffad5fadcb2d3884fa507bae8bfdeb8785"}
{"time":"2025-09-18T11:44:52.511972666Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0xc98aa37683cadf3b00fb30b247e607ffad5fadcb2d3884fa507bae8bfdeb8785"}
{"time":"2025-09-18T11:44:52.51211765Z","level":"INFO","msg":"✅ SUCCESS: Cluster Liquidation Successful","test":"sanity","txHash":"0xc98aa37683cadf3b00fb30b247e607ffad5fadcb2d3884fa507bae8bfdeb8785","block_hash":"0x732554d55a0bfe607d10961f9a76b3d6494bb510846963a8e2a9089b03e80872"}
{"time":"2025-09-18T11:44:52.531698992Z","level":"INFO","msg":"waiting for block to be found","module":"main","module":"e2m-validator","module":"e2m"}
{"time":"2025-09-18T11:45:12.541285204Z","level":"INFO","msg":"waiting for block to be found","module":"main","module":"e2m-validator","module":"e2m"}
{"time":"2025-09-18T11:45:32.549648044Z","level":"INFO","msg":"Liquidation epoch","module":"main","module":"e2m-validator","epoch":41621,"indices":["1102420"]}
{"time":"2025-09-18T11:45:47.569620048Z","level":"ERROR","msg":"Validator was not removed 2 epochs after liquidation","test":"sanity","index":"1102420"}
{"time":"2025-09-18T11:45:49.066455245Z","level":"INFO","msg":"Allowance fetched successfully","test":"sanity","module":"contract","account":"0x91e32efb8139cd88cae0df30d2bf471294c6ed27","allowance":"99456893999200282400000000"}
{"time":"2025-09-18T11:45:49.066599767Z","level":"INFO","msg":"Threshold value","test":"sanity","module":"contract","threshold":"9000000000000000000000"}
{"time":"2025-09-18T11:45:49.066708101Z","level":"INFO","msg":"Sufficient allowance","test":"sanity","module":"contract","allowance margin":"6561000000000000000000000000000000000000000000000000000000000000000000000000000000000000"}
{"time":"2025-09-18T11:45:49.11284951Z","level":"INFO","msg":"Reactivating cluster","test":"sanity","module":"contract","tx_hash":"0x85ff82dc75440b4de1f0a8becfcb96d59a3968e970e48b35c5a231ca81096739"}
{"time":"2025-09-18T11:45:49.116015484Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0x85ff82dc75440b4de1f0a8becfcb96d59a3968e970e48b35c5a231ca81096739"}
{"time":"2025-09-18T11:45:59.129411071Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":2,"sleep":20,"tx_hash":"0x85ff82dc75440b4de1f0a8becfcb96d59a3968e970e48b35c5a231ca81096739"}
{"time":"2025-09-18T11:46:19.141495295Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0x85ff82dc75440b4de1f0a8becfcb96d59a3968e970e48b35c5a231ca81096739"}
{"time":"2025-09-18T11:46:19.141605677Z","level":"INFO","msg":"✅ SUCCESS: Cluster Reactivation Successful","test":"sanity","txHash":"0x85ff82dc75440b4de1f0a8becfcb96d59a3968e970e48b35c5a231ca81096739","block_hash":"0x56656c7423aac655792e5d4639a302f529084e61fbee27b08fa0bcbeb23a8cd4"}
{"time":"2025-09-18T11:46:19.157406722Z","level":"INFO","msg":"waiting for block to be found","module":"main","module":"e2m-validator","module":"e2m"}
{"time":"2025-09-18T11:46:39.185574202Z","level":"INFO","msg":"Registration epoch","module":"main","module":"e2m-validator","epoch":41621,"indices":"1102420"}
{"time":"2025-09-18T11:46:54.197734436Z","level":"ERROR","msg":"Validator did not attest 4 epochs after reactivation","test":"sanity","index":"1102420"}
{"time":"2025-09-18T11:46:55.731573833Z","level":"INFO","msg":"Allowance fetched successfully","test":"sanity","module":"contract","account":"0x91e32efb8139cd88cae0df30d2bf471294c6ed27","allowance":"99447893999200282400000000"}
{"time":"2025-09-18T11:46:55.731707905Z","level":"INFO","msg":"Threshold value","test":"sanity","module":"contract","threshold":"9000000000000000000000"}
{"time":"2025-09-18T11:46:55.731756086Z","level":"INFO","msg":"Sufficient allowance","test":"sanity","module":"contract","allowance margin":"6561000000000000000000000000000000000000000000000000000000000000000000000000000000000000"}
{"time":"2025-09-18T11:46:55.767133754Z","level":"INFO","msg":"Removing validator","test":"sanity","module":"contract","tx_hash":"0x77670c942aebb85179d658ad0b9c11a4ff4a63744f22cc4d9b2de3b57485572b"}
{"time":"2025-09-18T11:46:55.770152628Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0x77670c942aebb85179d658ad0b9c11a4ff4a63744f22cc4d9b2de3b57485572b"}
{"time":"2025-09-18T11:47:05.778945538Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0x77670c942aebb85179d658ad0b9c11a4ff4a63744f22cc4d9b2de3b57485572b"}
{"time":"2025-09-18T11:47:05.779080535Z","level":"INFO","msg":"✅ SUCCESS: Contract Removal Successful","test":"sanity","blockHash":"0x038c838af13d8a337792b3e3ffd01cf9f676fa4b2fc460f1a6496015e10e45c9"}
{"time":"2025-09-18T11:47:05.795382041Z","level":"INFO","msg":"waiting for block to be found","module":"main","module":"e2m-validator","module":"e2m"}
{"time":"2025-09-18T11:47:25.820619177Z","level":"INFO","msg":"Removal epoch","module":"main","module":"e2m-validator","epoch":41621,"indices":"1102420"}
{"time":"2025-09-18T11:47:40.836360578Z","level":"ERROR","msg":"Validator was not removed 2 epochs after removal","test":"sanity","index":"1102420"}
{"time":"2025-09-18T11:47:40.836575141Z","level":"INFO","msg":"✅ SUCCESS: Sanity Test Suite Completed Successfully","module":"main","register":{"Contract":true,"E2M":false},"liquidate":{"Contract":true,"E2M":false},"reactivate":{"Contract":true,"E2M":false},"remove":{"Contract":true,"E2M":false}}
{"time":"2025-09-18T11:47:40.836626656Z","level":"INFO","msg":"⏳ PROGRESS: Waiting for all tests to finish","module":"main"}
{"time":"2025-09-18T11:47:40.836691448Z","level":"INFO","msg":"✅ SUCCESS: All Test Suites Completed Successfully","module":"main","passed_suites":["sanity"],"skipped_suites":["isolated"]}
{"time":"2025-09-18T11:47:40.836728765Z","level":"INFO","msg":"🔑 KEY STEP: Starting resource teardown...","module":"main","usage_id":"9d1f0717736506137b674da48e386807"}
{"time":"2025-09-18T11:47:40.845967549Z","level":"INFO","msg":"🌐 Orchestrator API Client initialized","module":"main","endpoint":"http://aetheria-orchestrator.aetheria.svc:8080"}
{"time":"2025-09-18T11:47:40.846070653Z","level":"INFO","msg":"🧹 Requesting resource teardown","module":"main","usage_id":"9d1f0717736506137b674da48e386807"}
{"time":"2025-09-18T11:47:40.884671669Z","level":"INFO","msg":"✅ SUCCESS: Completed resource teardown successfully","module":"main","usage_id":"9d1f0717736506137b674da48e386807"}

2025-09-18T11:47:44Z The test run finished successfully

Result - 🟢 Success

vaclav-ssvlabs · 2025-09-18T11:58:53Z

/test skip-ci

ssv-test-bot · 2025-09-18T15:03:02Z

🔴 Test run for PR #2433

Branch: ci-test-handler-testing2
Commit: 74b7038

Status

2025-09-18T15:03:02Z Starting test run
2025-09-18T15:03:03Z Test run submitted successfully for commit 74b7038
2025-09-18T15:03:04Z container image for commit 74b7038f0265e7f8da4425e9315fe4199c751b80 is build and present in container registry
2025-09-18T15:03:05Z automatically allocated node range cluster=312,313,314,315,316,317,318 for test run on network=hoodi_stage
2025-09-18T15:03:07Z deploying commit 74b7038f0265e7f8da4425e9315fe4199c751b80 to ssv nodes 312,313,314,315,316,317,318
2025-09-18T15:03:53Z started test executor with tests=(sanity,e2m) and network=hoodi_stage

Click to expand executor logs

═══════════════════════════════════════════════════════════════════════════════════════════════════════
                🧪  WELCOME TO Executor - SSV's Distributed Test Automation Execution Layer  🧪            

                                  ___ __  __ ___  ___  _   _  _____  ___   ___ 
                                 | __|\\ \\/ /| __|/ __|| | | ||_   _|/ _ \\ | _ \\
                                 | _|  >  < | _|| (__ | |_| |  | | | (_) ||   /
                                 |___|/_/\\_\\|___|\\___| \\___/   |_|  \\___/ |_|_\\
                                        

═══════════════════════════════════════════════════════════════════════════════════════════════════════
{"time":"2025-09-18T15:03:54.797790402Z","level":"INFO","msg":"🐳 Docker mode: Using environment variables for config generation"}
{"time":"2025-09-18T15:03:54.80792621Z","level":"INFO","msg":"🌐 Orchestrator API Client initialized","endpoint":"http://aetheria-orchestrator.aetheria.svc:8080"}
{"time":"2025-09-18T15:03:54.807963556Z","level":"INFO","msg":"📝 Requesting configuration generation from orchestrator API","network":"hoodi_stage","clusters":[312,313,314,315,316,317,318],"tests":"(sanity,e2m)"}
{"time":"2025-09-18T15:03:54.849953337Z","level":"INFO","msg":"✅ Configuration generated successfully via API","resource_usage_id":"589ec112e876c02612f127145ff8eb6b"}
{"time":"2025-09-18T15:03:54.850317463Z","level":"INFO","msg":"🚀 EXECUTOR Initialized Successfully"}
{"time":"2025-09-18T15:03:54.850345404Z","level":"INFO","msg":"⚙️ Test configuration loaded","network":"hoodi_stage","nodes":[312,313,314,315,316,317,318],"test_type":{"Sanity":{"ShouldRun":true,"E2M":true,"BulkAmount":0},"Bulk":{"ShouldRun":false,"E2M":false,"BulkAmount":0},"Isolated":{"Tests":null}}}
{"time":"2025-09-18T15:03:59.709841239Z","level":"INFO","msg":"Generated operator pubkeys file","module":"scanner","path":"/app/shared/data/operators/operator-pubkeys-hoodi_stage.json"}
{"time":"2025-09-18T15:03:59.709977045Z","level":"INFO","msg":"Loki Started...","module":"Loki"}
{"time":"2025-09-18T15:03:59.710030131Z","level":"INFO","msg":"HTTPRequester Started...","module":"HTTPRequester"}
{"time":"2025-09-18T15:03:59.710081676Z","level":"INFO","msg":"Validation client initialized","module":"main","loki_enabled":true,"e2m_enabled":true}
{"time":"2025-09-18T15:03:59.710143356Z","level":"INFO","msg":"🔑 KEY STEP: Starting Isolated Tests Concurrently if exists...","module":"main"}
{"time":"2025-09-18T15:03:59.710249245Z","level":"INFO","msg":"🔑 KEY STEP: Starting Sanity Test Suite","module":"main"}
{"time":"2025-09-18T15:03:59.71051788Z","level":"INFO","msg":"No isolated tests configured, skipping","module":"main"}
{"time":"2025-09-18T15:03:59.763627939Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":312}
{"time":"2025-09-18T15:03:59.776217362Z","level":"INFO","msg":"Available slots","test":"sanity","node":312,"slots":3000}
{"time":"2025-09-18T15:03:59.776310035Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":313}
{"time":"2025-09-18T15:03:59.783634945Z","level":"INFO","msg":"Available slots","test":"sanity","node":313,"slots":399}
{"time":"2025-09-18T15:03:59.783672869Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":314}
{"time":"2025-09-18T15:03:59.790962785Z","level":"INFO","msg":"Available slots","test":"sanity","node":314,"slots":299}
{"time":"2025-09-18T15:03:59.791063253Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":315}
{"time":"2025-09-18T15:03:59.798405753Z","level":"INFO","msg":"Available slots","test":"sanity","node":315,"slots":717}
{"time":"2025-09-18T15:04:38.419210249Z","level":"INFO","msg":"Extracted keyshares path","test":"sanity","module":"keys","path":"/app/shared/data/keyshares/0xa88938ed513411df0322826f6b017a2d1fb610c633e33ccad9dc1e46b274ad32a489f89e7b1b772ab518ba0cb089cee2.json"}
{"time":"2025-09-18T15:04:38.419321559Z","level":"INFO","msg":"Key shares file generated successfully","test":"sanity","keyshare_path":"/app/shared/data/keyshares/0xa88938ed513411df0322826f6b017a2d1fb610c633e33ccad9dc1e46b274ad32a489f89e7b1b772ab518ba0cb089cee2.json"}
{"time":"2025-09-18T15:05:50.438681196Z","level":"INFO","msg":"Allowance fetched successfully","test":"sanity","module":"contract","account":"0xd2dbd02e4efe087d7d195de828b9dd25f19a89c9","allowance":"999999999999999999999999999504928000000000000000000"}
{"time":"2025-09-18T15:05:50.438802648Z","level":"INFO","msg":"Threshold value","test":"sanity","module":"contract","threshold":"9000000000000000000000"}
{"time":"2025-09-18T15:05:50.43884282Z","level":"INFO","msg":"Sufficient allowance","test":"sanity","module":"contract","allowance margin":"4782969000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"}
{"time":"2025-09-18T15:05:50.442758246Z","level":"INFO","msg":"Allowance fetched successfully","test":"sanity","module":"contract","account":"0xd2dbd02e4efe087d7d195de828b9dd25f19a89c9","allowance":"999999999999999999999999999504928000000000000000000"}
{"time":"2025-09-18T15:05:50.442849023Z","level":"INFO","msg":"Allowance","test":"sanity","module":"contract","allowance":"999999999999999999999999999504928000000000000000000"}
{"time":"2025-09-18T15:05:50.53482658Z","level":"INFO","msg":"Registering validator","test":"sanity","module":"contract","tx_hash":"0xaddbdf4f399e59c0b26c2942d5d2fbaba1eeca9814613063eb0fc73ee46036c2"}
{"time":"2025-09-18T15:05:50.542191167Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0xaddbdf4f399e59c0b26c2942d5d2fbaba1eeca9814613063eb0fc73ee46036c2"}
{"time":"2025-09-18T15:06:00.546836625Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":2,"sleep":20,"tx_hash":"0xaddbdf4f399e59c0b26c2942d5d2fbaba1eeca9814613063eb0fc73ee46036c2"}
{"time":"2025-09-18T15:06:20.564003784Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0xaddbdf4f399e59c0b26c2942d5d2fbaba1eeca9814613063eb0fc73ee46036c2"}
{"time":"2025-09-18T15:06:20.564132321Z","level":"INFO","msg":"✅ SUCCESS: Contract Register Successful","test":"sanity","txHash":"0xaddbdf4f399e59c0b26c2942d5d2fbaba1eeca9814613063eb0fc73ee46036c2","block_hash":"0x946d3312c8b7f560e112e90ebcdbe5fdc71057172491e1e0966322ceaf8c7bea"}
{"time":"2025-09-18T15:06:20.582916289Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"sanity","module":"helpers","pubkey":"0xa88938ed513411df0322826f6b017a2d1fb610c633e33ccad9dc1e46b274ad32a489f89e7b1b772ab518ba0cb089cee2","timeToSleep":10,"attempt":1}
{"time":"2025-09-18T15:06:30.589446966Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"sanity","module":"helpers","pubkey":"0xa88938ed513411df0322826f6b017a2d1fb610c633e33ccad9dc1e46b274ad32a489f89e7b1b772ab518ba0cb089cee2","timeToSleep":20,"attempt":2}
{"time":"2025-09-18T15:06:50.611496696Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"sanity","module":"helpers","pubkey":"0xa88938ed513411df0322826f6b017a2d1fb610c633e33ccad9dc1e46b274ad32a489f89e7b1b772ab518ba0cb089cee2","timeToSleep":40,"attempt":3}
{"time":"2025-09-18T15:07:30.627074341Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"sanity","module":"helpers","pubkey":"0xa88938ed513411df0322826f6b017a2d1fb610c633e33ccad9dc1e46b274ad32a489f89e7b1b772ab518ba0cb089cee2","timeToSleep":80,"attempt":4}
{"time":"2025-09-18T15:08:50.640825109Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"sanity","module":"helpers","pubkey":"0xa88938ed513411df0322826f6b017a2d1fb610c633e33ccad9dc1e46b274ad32a489f89e7b1b772ab518ba0cb089cee2","timeToSleep":160,"attempt":5}
{"time":"2025-09-18T15:11:30.661945135Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"sanity","module":"helpers","pubkey":"0xa88938ed513411df0322826f6b017a2d1fb610c633e33ccad9dc1e46b274ad32a489f89e7b1b772ab518ba0cb089cee2","timeToSleep":320,"attempt":6}
{"time":"2025-09-18T15:16:50.67916876Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"sanity","module":"helpers","pubkey":"0xa88938ed513411df0322826f6b017a2d1fb610c633e33ccad9dc1e46b274ad32a489f89e7b1b772ab518ba0cb089cee2","timeToSleep":640,"attempt":7}
{"time":"2025-09-18T15:27:30.698569619Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"sanity","module":"helpers","pubkey":"0xa88938ed513411df0322826f6b017a2d1fb610c633e33ccad9dc1e46b274ad32a489f89e7b1b772ab518ba0cb089cee2","timeToSleep":1280,"attempt":8}
{"time":"2025-09-18T15:48:50.718827631Z","level":"ERROR","msg":"❌ FAILURE: Sanity Test Suite Failed","module":"main","error":"validator not found in exporter after 8 attempts"}
{"time":"2025-09-18T15:48:50.718951909Z","level":"INFO","msg":"⏳ PROGRESS: Waiting for all tests to finish","module":"main"}
{"time":"2025-09-18T15:48:50.71910686Z","level":"ERROR","msg":"❌ FAILURE: Test Execution Completed with Failures","module":"main","passed_suites":null,"failed_suites":["sanity"],"skipped_suites":["isolated"]}
{"time":"2025-09-18T15:48:50.719196571Z","level":"ERROR","msg":"❌ FAILURE: Suite Failure Details","module":"main","suite":"sanity","error":"validator not found in exporter after 8 attempts"}
{"time":"2025-09-18T15:48:50.719331304Z","level":"INFO","msg":"🔑 KEY STEP: Starting resource teardown...","module":"main","usage_id":"589ec112e876c02612f127145ff8eb6b"}
{"time":"2025-09-18T15:48:50.728117828Z","level":"INFO","msg":"🌐 Orchestrator API Client initialized","module":"main","endpoint":"http://aetheria-orchestrator.aetheria.svc:8080"}
{"time":"2025-09-18T15:48:50.728229595Z","level":"INFO","msg":"🧹 Requesting resource teardown","module":"main","usage_id":"589ec112e876c02612f127145ff8eb6b"}
{"time":"2025-09-18T15:48:50.759685227Z","level":"INFO","msg":"✅ SUCCESS: Completed resource teardown successfully","module":"main","usage_id":"589ec112e876c02612f127145ff8eb6b"}

2025-09-18T15:48:54Z The test run failed with error

Result - 🔴 Failure

yoaz-ssvlabs · 2025-09-19T07:29:48Z

/test tests=(sanity)

ssv-test-bot · 2025-09-19T07:29:52Z

🔴 Test run for PR #2433

Branch: ci-test-handler-testing2
Commit: 74b7038

Status

2025-09-19T07:29:52Z Starting test run
2025-09-19T07:29:53Z Test run submitted successfully for commit 74b7038
2025-09-19T07:29:54Z container image for commit 74b7038f0265e7f8da4425e9315fe4199c751b80 is build and present in container registry
2025-09-19T07:29:55Z automatically allocated node range cluster=304,305,306,307 for test run on network=hoodi_stage
2025-09-19T07:29:57Z deploying commit 74b7038f0265e7f8da4425e9315fe4199c751b80 to ssv nodes 304,305,306,307
2025-09-19T07:30:57Z started test executor with tests=(sanity) and network=hoodi_stage

Click to expand executor logs

═══════════════════════════════════════════════════════════════════════════════════════════════════════
                🧪  WELCOME TO Executor - SSV's Distributed Test Automation Execution Layer  🧪            

                                  ___ __  __ ___  ___  _   _  _____  ___   ___ 
                                 | __|\\ \\/ /| __|/ __|| | | ||_   _|/ _ \\ | _ \\
                                 | _|  >  < | _|| (__ | |_| |  | | | (_) ||   /
                                 |___|/_/\\_\\|___|\\___| \\___/   |_|  \\___/ |_|_\\
                                        

═══════════════════════════════════════════════════════════════════════════════════════════════════════
{"time":"2025-09-19T07:30:58.796843603Z","level":"INFO","msg":"🐳 Docker mode: Using environment variables for config generation"}
{"time":"2025-09-19T07:30:58.805040764Z","level":"INFO","msg":"🌐 Orchestrator API Client initialized","endpoint":"http://aetheria-orchestrator.aetheria.svc:8080"}
{"time":"2025-09-19T07:30:58.805115909Z","level":"INFO","msg":"📝 Requesting configuration generation from orchestrator API","network":"hoodi_stage","clusters":[304,305,306,307],"tests":"(sanity)"}
{"time":"2025-09-19T07:30:58.836777279Z","level":"INFO","msg":"✅ Configuration generated successfully via API","resource_usage_id":"35bd384c0b786d7ce9297f20eeef5338"}
{"time":"2025-09-19T07:30:58.838133366Z","level":"INFO","msg":"🚀 EXECUTOR Initialized Successfully"}
{"time":"2025-09-19T07:30:58.838215331Z","level":"INFO","msg":"⚙️ Test configuration loaded","network":"hoodi_stage","nodes":[304,305,306,307],"test_type":{"Sanity":{"ShouldRun":true,"E2M":false,"BulkAmount":0},"Bulk":{"ShouldRun":false,"E2M":false,"BulkAmount":0},"Isolated":{"Tests":null}}}
{"time":"2025-09-19T07:31:03.425790545Z","level":"INFO","msg":"Generated operator pubkeys file","module":"scanner","path":"/app/shared/data/operators/operator-pubkeys-hoodi_stage.json"}
{"time":"2025-09-19T07:31:03.425921779Z","level":"INFO","msg":"Loki Started...","module":"Loki"}
{"time":"2025-09-19T07:31:03.425950031Z","level":"INFO","msg":"HTTPRequester Started...","module":"HTTPRequester"}
{"time":"2025-09-19T07:31:03.425980096Z","level":"INFO","msg":"Validation client initialized","module":"main","loki_enabled":true,"e2m_enabled":false}
{"time":"2025-09-19T07:31:03.426046852Z","level":"INFO","msg":"🔑 KEY STEP: Starting Isolated Tests Concurrently if exists...","module":"main"}
{"time":"2025-09-19T07:31:03.426157863Z","level":"INFO","msg":"🔑 KEY STEP: Starting Sanity Test Suite","module":"main"}
{"time":"2025-09-19T07:31:03.426237081Z","level":"INFO","msg":"No isolated tests configured, skipping","module":"main"}
{"time":"2025-09-19T07:31:03.485874769Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":304}
{"time":"2025-09-19T07:31:03.501399931Z","level":"INFO","msg":"Available slots","test":"sanity","node":304,"slots":3000}
{"time":"2025-09-19T07:31:03.501485903Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":305}
{"time":"2025-09-19T07:31:03.511754792Z","level":"INFO","msg":"Available slots","test":"sanity","node":305,"slots":399}
{"time":"2025-09-19T07:31:03.511846112Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":306}
{"time":"2025-09-19T07:31:03.522257474Z","level":"INFO","msg":"Available slots","test":"sanity","node":306,"slots":299}
{"time":"2025-09-19T07:31:03.522342205Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":307}
{"time":"2025-09-19T07:31:03.531794315Z","level":"INFO","msg":"Available slots","test":"sanity","node":307,"slots":717}
{"time":"2025-09-19T07:32:20.468748304Z","level":"INFO","msg":"Extracted keyshares path","test":"sanity","module":"keys","path":"/app/shared/data/keyshares/0x84d7c46eb947f61c65a1c32971bbad4a1e832b1998675374fdebab49e6ab6d2cc0a7208acb3d176ac8b5df0cd3587f23.json"}
{"time":"2025-09-19T07:32:20.469700095Z","level":"INFO","msg":"Key shares file generated successfully","test":"sanity","keyshare_path":"/app/shared/data/keyshares/0x84d7c46eb947f61c65a1c32971bbad4a1e832b1998675374fdebab49e6ab6d2cc0a7208acb3d176ac8b5df0cd3587f23.json"}
{"time":"2025-09-19T07:32:22.120016308Z","level":"INFO","msg":"Allowance fetched successfully","test":"sanity","module":"contract","account":"0x5cc0dde14e7256340cc820415a6022a7d1c93a35","allowance":"9999999999999999999999643360199873254400000000"}
{"time":"2025-09-19T07:32:22.120101283Z","level":"INFO","msg":"Threshold value","test":"sanity","module":"contract","threshold":"9000000000000000000000"}
{"time":"2025-09-19T07:32:22.120130721Z","level":"INFO","msg":"Sufficient allowance","test":"sanity","module":"contract","allowance margin":"6561000000000000000000000000000000000000000000000000000000000000000000000000000000000000"}
{"time":"2025-09-19T07:32:22.125680229Z","level":"INFO","msg":"Allowance fetched successfully","test":"sanity","module":"contract","account":"0x5cc0dde14e7256340cc820415a6022a7d1c93a35","allowance":"9999999999999999999999643360199873254400000000"}
{"time":"2025-09-19T07:32:22.125799556Z","level":"INFO","msg":"Allowance","test":"sanity","module":"contract","allowance":"9999999999999999999999643360199873254400000000"}
{"time":"2025-09-19T07:32:22.158523204Z","level":"INFO","msg":"Registering validator","test":"sanity","module":"contract","tx_hash":"0xc63289dc481f71199a9489f6eb83f929eb3365a69517f672350f53962d8b690a"}
{"time":"2025-09-19T07:32:22.162111929Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0xc63289dc481f71199a9489f6eb83f929eb3365a69517f672350f53962d8b690a"}
{"time":"2025-09-19T07:32:32.168097284Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":2,"sleep":20,"tx_hash":"0xc63289dc481f71199a9489f6eb83f929eb3365a69517f672350f53962d8b690a"}
{"time":"2025-09-19T07:32:52.184376013Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0xc63289dc481f71199a9489f6eb83f929eb3365a69517f672350f53962d8b690a"}
{"time":"2025-09-19T07:32:52.184640015Z","level":"INFO","msg":"✅ SUCCESS: Contract Register Successful","test":"sanity","txHash":"0xc63289dc481f71199a9489f6eb83f929eb3365a69517f672350f53962d8b690a","block_hash":"0x001c645111a343fd72cbe3ca95d961374e1804d39abccaca60487cec6c9c2595"}
{"time":"2025-09-19T07:32:52.197248874Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"sanity","module":"helpers","pubkey":"0x84d7c46eb947f61c65a1c32971bbad4a1e832b1998675374fdebab49e6ab6d2cc0a7208acb3d176ac8b5df0cd3587f23","timeToSleep":10,"attempt":1}
{"time":"2025-09-19T07:33:02.205651609Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"sanity","module":"helpers","pubkey":"0x84d7c46eb947f61c65a1c32971bbad4a1e832b1998675374fdebab49e6ab6d2cc0a7208acb3d176ac8b5df0cd3587f23","timeToSleep":20,"attempt":2}
{"time":"2025-09-19T07:33:22.22148712Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"sanity","module":"helpers","pubkey":"0x84d7c46eb947f61c65a1c32971bbad4a1e832b1998675374fdebab49e6ab6d2cc0a7208acb3d176ac8b5df0cd3587f23","timeToSleep":40,"attempt":3}
{"time":"2025-09-19T07:34:02.253360799Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"sanity","module":"helpers","pubkey":"0x84d7c46eb947f61c65a1c32971bbad4a1e832b1998675374fdebab49e6ab6d2cc0a7208acb3d176ac8b5df0cd3587f23","timeToSleep":80,"attempt":4}
{"time":"2025-09-19T07:35:24.347361766Z","level":"INFO","msg":"Liquidating cluster","test":"sanity","module":"contract","tx_hash":"0x4bf5ee20b956f9c2ac663a2d510b458acb8ffbff962d5001c3400e84ffd355da"}
{"time":"2025-09-19T07:35:24.351339762Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0x4bf5ee20b956f9c2ac663a2d510b458acb8ffbff962d5001c3400e84ffd355da"}
{"time":"2025-09-19T07:35:34.35737033Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0x4bf5ee20b956f9c2ac663a2d510b458acb8ffbff962d5001c3400e84ffd355da"}
{"time":"2025-09-19T07:35:34.357523487Z","level":"INFO","msg":"✅ SUCCESS: Cluster Liquidation Successful","test":"sanity","txHash":"0x4bf5ee20b956f9c2ac663a2d510b458acb8ffbff962d5001c3400e84ffd355da","block_hash":"0xcc68ec61df496fe61c3d62df7f7abb60f2eddeb6beb7b5a0246b8fbecfa858b0"}
{"time":"2025-09-19T07:35:35.907495609Z","level":"INFO","msg":"Allowance fetched successfully","test":"sanity","module":"contract","account":"0x5cc0dde14e7256340cc820415a6022a7d1c93a35","allowance":"9999999999999999999999643360194910796200000000"}
{"time":"2025-09-19T07:35:35.90756911Z","level":"INFO","msg":"Threshold value","test":"sanity","module":"contract","threshold":"9000000000000000000000"}
{"time":"2025-09-19T07:35:35.90758841Z","level":"INFO","msg":"Sufficient allowance","test":"sanity","module":"contract","allowance margin":"6561000000000000000000000000000000000000000000000000000000000000000000000000000000000000"}
{"time":"2025-09-19T07:35:36.029736463Z","level":"INFO","msg":"Reactivating cluster","test":"sanity","module":"contract","tx_hash":"0xf1f38105c1ea8fd864fcf3489dddfcfabb32928ed59ce8361d4746602e3129b5"}
{"time":"2025-09-19T07:35:36.040116574Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0xf1f38105c1ea8fd864fcf3489dddfcfabb32928ed59ce8361d4746602e3129b5"}
{"time":"2025-09-19T07:35:46.044429668Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":2,"sleep":20,"tx_hash":"0xf1f38105c1ea8fd864fcf3489dddfcfabb32928ed59ce8361d4746602e3129b5"}
{"time":"2025-09-19T07:36:06.063193822Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0xf1f38105c1ea8fd864fcf3489dddfcfabb32928ed59ce8361d4746602e3129b5"}
{"time":"2025-09-19T07:36:06.063322679Z","level":"ERROR","msg":"Transaction verification failed","test":"sanity","blockHash":"0x02f1031a6d32e8b0dc25ca3ea6fc257051276b06601d0376416fd907181e2dc3"}
{"time":"2025-09-19T07:36:06.063373373Z","level":"ERROR","msg":"❌ FAILURE: Sanity Test Suite Failed","module":"main","error":"transaction verification failed"}
{"time":"2025-09-19T07:36:06.063388678Z","level":"INFO","msg":"⏳ PROGRESS: Waiting for all tests to finish","module":"main"}
{"time":"2025-09-19T07:36:06.063435556Z","level":"ERROR","msg":"❌ FAILURE: Test Execution Completed with Failures","module":"main","passed_suites":null,"failed_suites":["sanity"],"skipped_suites":["isolated"]}
{"time":"2025-09-19T07:36:06.063469678Z","level":"ERROR","msg":"❌ FAILURE: Suite Failure Details","module":"main","suite":"sanity","error":"transaction verification failed"}
{"time":"2025-09-19T07:36:06.063486057Z","level":"INFO","msg":"🔑 KEY STEP: Starting resource teardown...","module":"main","usage_id":"35bd384c0b786d7ce9297f20eeef5338"}
{"time":"2025-09-19T07:36:06.07098178Z","level":"INFO","msg":"🌐 Orchestrator API Client initialized","module":"main","endpoint":"http://aetheria-orchestrator.aetheria.svc:8080"}
{"time":"2025-09-19T07:36:06.07109988Z","level":"INFO","msg":"🧹 Requesting resource teardown","module":"main","usage_id":"35bd384c0b786d7ce9297f20eeef5338"}
{"time":"2025-09-19T07:36:36.104996571Z","level":"ERROR","msg":"❌ FAILURE: Failed to complete teardown","module":"main","error":"failed to make teardown request: Post \\"http://aetheria-orchestrator.aetheria.svc:8080/api/v1/teardown/35bd384c0b786d7ce9297f20eeef5338\\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)","usage_id":"35bd384c0b786d7ce9297f20eeef5338"}

2025-09-19T07:36:40Z The test run failed with error

Result - 🔴 Failure

vaclav-ssvlabs · 2025-09-19T13:30:20Z

/test

ssv-test-bot · 2025-09-19T13:30:24Z

🔴 Test run for PR #2433

Branch: ci-test-handler-testing2
Commit: 74b7038

Status

2025-09-19T13:30:23Z Starting test run
2025-09-19T13:30:25Z Test run submitted successfully for commit 74b7038
2025-09-19T13:30:26Z container image for commit 74b7038f0265e7f8da4425e9315fe4199c751b80 is build and present in container registry
2025-09-19T13:30:27Z automatically allocated node range cluster=312,313,314,315,316,317,318 for test run on network=hoodi_stage
2025-09-19T13:30:29Z deploying commit 74b7038f0265e7f8da4425e9315fe4199c751b80 to ssv nodes 312,313,314,315,316,317,318
2025-09-19T13:31:33Z commit 74b7038f0265e7f8da4425e9315fe4199c751b80 is deployed to ssv nodes 312,313,314,315,316,317,318

Click to expand executor logs

═══════════════════════════════════════════════════════════════════════════════════════════════════════
                🧪  WELCOME TO Executor - SSV's Distributed Test Automation Execution Layer  🧪            

{"time":"2025-09-19T13:31:35.8180388Z","level":"INFO","msg":"🐳 Docker mode: Using environment variables for config generation"}
                                  ___ __  __ ___  ___  _   _  _____  ___   ___ 
                                 | __|\\ \\/ /| __|/ __|| | | ||_   _|/ _ \\ | _ \\
                                 | _|  >  < | _|| (__ | |_| |  | | | (_) ||   /
                                 |___|/_/\\_\\|___|\\___| \\___/   |_|  \\___/ |_|_\\
                                        

═══════════════════════════════════════════════════════════════════════════════════════════════════════
{"time":"2025-09-19T13:31:35.828858091Z","level":"INFO","msg":"🌐 Orchestrator API Client initialized","endpoint":"http://aetheria-orchestrator.aetheria.svc:8080"}
{"time":"2025-09-19T13:31:35.828938032Z","level":"INFO","msg":"📝 Requesting configuration generation from orchestrator API","network":"hoodi_stage","clusters":[312,313,314,315,316,317,318],"tests":"(sanity)"}
{"time":"2025-09-19T13:31:35.895812969Z","level":"INFO","msg":"✅ Configuration generated successfully via API","resource_usage_id":"254ba142954382e3c12abb014240dbb6"}
{"time":"2025-09-19T13:31:35.897134918Z","level":"INFO","msg":"🚀 EXECUTOR Initialized Successfully"}
{"time":"2025-09-19T13:31:35.897231002Z","level":"INFO","msg":"⚙️ Test configuration loaded","network":"hoodi_stage","nodes":[312,313,314,315,316,317,318],"test_type":{"Sanity":{"ShouldRun":true,"E2M":false,"BulkAmount":0},"Bulk":{"ShouldRun":false,"E2M":false,"BulkAmount":0},"Isolated":{"Tests":null}}}
{"time":"2025-09-19T13:31:40.517967698Z","level":"INFO","msg":"Generated operator pubkeys file","module":"scanner","path":"/app/shared/data/operators/operator-pubkeys-hoodi_stage.json"}
{"time":"2025-09-19T13:31:40.518076973Z","level":"INFO","msg":"Loki Started...","module":"Loki"}
{"time":"2025-09-19T13:31:40.518097095Z","level":"INFO","msg":"HTTPRequester Started...","module":"HTTPRequester"}
{"time":"2025-09-19T13:31:40.518120775Z","level":"INFO","msg":"Validation client initialized","module":"main","loki_enabled":true,"e2m_enabled":false}
{"time":"2025-09-19T13:31:40.518154187Z","level":"INFO","msg":"🔑 KEY STEP: Starting Isolated Tests Concurrently if exists...","module":"main"}
{"time":"2025-09-19T13:31:40.518207393Z","level":"INFO","msg":"🔑 KEY STEP: Starting Sanity Test Suite","module":"main"}
{"time":"2025-09-19T13:31:40.518512092Z","level":"INFO","msg":"No isolated tests configured, skipping","module":"main"}
{"time":"2025-09-19T13:31:40.562128896Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":312}
{"time":"2025-09-19T13:31:40.588534513Z","level":"INFO","msg":"Available slots","test":"sanity","node":312,"slots":3000}
{"time":"2025-09-19T13:31:40.590633596Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":313}
{"time":"2025-09-19T13:31:40.599901577Z","level":"INFO","msg":"Available slots","test":"sanity","node":313,"slots":399}
{"time":"2025-09-19T13:31:40.599995551Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":314}
{"time":"2025-09-19T13:31:40.612097417Z","level":"INFO","msg":"Available slots","test":"sanity","node":314,"slots":299}
{"time":"2025-09-19T13:31:40.612217291Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":315}
{"time":"2025-09-19T13:31:40.624170706Z","level":"INFO","msg":"Available slots","test":"sanity","node":315,"slots":717}
{"time":"2025-09-19T13:32:58.087190913Z","level":"INFO","msg":"Extracted keyshares path","test":"sanity","module":"keys","path":"/app/shared/data/keyshares/0xa88938ed513411df0322826f6b017a2d1fb610c633e33ccad9dc1e46b274ad32a489f89e7b1b772ab518ba0cb089cee2.json"}
{"time":"2025-09-19T13:32:58.08729573Z","level":"INFO","msg":"Key shares file generated successfully","test":"sanity","keyshare_path":"/app/shared/data/keyshares/0xa88938ed513411df0322826f6b017a2d1fb610c633e33ccad9dc1e46b274ad32a489f89e7b1b772ab518ba0cb089cee2.json"}
{"time":"2025-09-19T13:34:14.584387412Z","level":"INFO","msg":"Allowance fetched successfully","test":"sanity","module":"contract","account":"0xd2dbd02e4efe087d7d195de828b9dd25f19a89c9","allowance":"999999999999999999999999999504925500000000000000000"}
{"time":"2025-09-19T13:34:14.584464906Z","level":"INFO","msg":"Threshold value","test":"sanity","module":"contract","threshold":"9000000000000000000000"}
{"time":"2025-09-19T13:34:14.584496879Z","level":"INFO","msg":"Sufficient allowance","test":"sanity","module":"contract","allowance margin":"4782969000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"}
{"time":"2025-09-19T13:34:14.588530312Z","level":"INFO","msg":"Allowance fetched successfully","test":"sanity","module":"contract","account":"0xd2dbd02e4efe087d7d195de828b9dd25f19a89c9","allowance":"999999999999999999999999999504925500000000000000000"}
{"time":"2025-09-19T13:34:14.588640188Z","level":"INFO","msg":"Allowance","test":"sanity","module":"contract","allowance":"999999999999999999999999999504925500000000000000000"}
{"time":"2025-09-19T13:34:14.74331125Z","level":"ERROR","msg":"Failed to register validator","test":"sanity","module":"contract","error":"execution reverted"}
{"time":"2025-09-19T13:34:14.74343471Z","level":"ERROR","msg":"❌ FAILURE: Sanity Test Suite Failed","module":"main","error":"execution reverted"}
{"time":"2025-09-19T13:34:14.743452166Z","level":"INFO","msg":"⏳ PROGRESS: Waiting for all tests to finish","module":"main"}
{"time":"2025-09-19T13:34:14.743499833Z","level":"ERROR","msg":"❌ FAILURE: Test Execution Completed with Failures","module":"main","passed_suites":null,"failed_suites":["sanity"],"skipped_suites":["isolated"]}
{"time":"2025-09-19T13:34:14.74352559Z","level":"ERROR","msg":"❌ FAILURE: Suite Failure Details","module":"main","suite":"sanity","error":"execution reverted"}
{"time":"2025-09-19T13:34:14.743541306Z","level":"INFO","msg":"🔑 KEY STEP: Starting resource teardown...","module":"main","usage_id":"254ba142954382e3c12abb014240dbb6"}
{"time":"2025-09-19T13:34:14.75181818Z","level":"INFO","msg":"🌐 Orchestrator API Client initialized","module":"main","endpoint":"http://aetheria-orchestrator.aetheria.svc:8080"}
{"time":"2025-09-19T13:34:14.751948017Z","level":"INFO","msg":"🧹 Requesting resource teardown","module":"main","usage_id":"254ba142954382e3c12abb014240dbb6"}
{"time":"2025-09-19T13:34:14.779920974Z","level":"INFO","msg":"✅ SUCCESS: Completed resource teardown successfully","module":"main","usage_id":"254ba142954382e3c12abb014240dbb6"}

2025-09-19T13:34:18Z The test run failed with error

Result - 🔴 Failure

2025-09-19T13:35:19Z Finished cleaning reserved resources

yoaz-ssvlabs · 2025-09-22T08:06:10Z

/test tests=(sanity)

ssv-test-bot · 2025-09-22T08:06:14Z

🔴 Test run for PR #2433

Branch: ci-test-handler-testing2
Commit: 74b7038

Status

2025-09-22T08:06:14Z Starting test run
2025-09-22T08:06:15Z Test run submitted successfully for commit 74b7038
2025-09-22T08:06:16Z container image for commit 74b7038f0265e7f8da4425e9315fe4199c751b80 is build and present in container registry
2025-09-22T08:06:17Z automatically allocated node range cluster=304,305,306,307 for test run on network=hoodi_stage
2025-09-22T08:06:18Z deploying commit 74b7038f0265e7f8da4425e9315fe4199c751b80 to ssv nodes 304,305,306,307
2025-09-22T08:07:18Z commit 74b7038f0265e7f8da4425e9315fe4199c751b80 is deployed to ssv nodes 304,305,306,307

Click to expand executor logs

═══════════════════════════════════════════════════════════════════════════════════════════════════════
                🧪  WELCOME TO Executor - SSV's Distributed Test Automation Execution Layer  🧪            

                                  ___ __  __ ___  ___  _   _  _____  ___   ___ 
                                 | __|\\ \\/ /| __|/ __|| | | ||_   _|/ _ \\ | _ \\
                                 | _|  >  < | _|| (__ | |_| |  | | | (_) ||   /
                                 |___|/_/\\_\\|___|\\___| \\___/   |_|  \\___/ |_|_\\
                                        

═══════════════════════════════════════════════════════════════════════════════════════════════════════
{"time":"2025-09-22T08:07:40.495065787Z","level":"INFO","msg":"🐳 Docker mode: Using environment variables for config generation"}
{"time":"2025-09-22T08:07:40.504025976Z","level":"INFO","msg":"🌐 Orchestrator API Client initialized","endpoint":"http://aetheria-orchestrator.aetheria.svc:8080"}
{"time":"2025-09-22T08:07:40.50413059Z","level":"INFO","msg":"📝 Requesting configuration generation from orchestrator API","network":"hoodi_stage","clusters":[304,305,306,307],"tests":"(sanity)"}
{"time":"2025-09-22T08:07:40.543305738Z","level":"INFO","msg":"✅ Configuration generated successfully via API","resource_usage_id":"9343e9dc9be5a53056ea82c116ac9e3e"}
{"time":"2025-09-22T08:07:40.544666591Z","level":"INFO","msg":"🚀 EXECUTOR Initialized Successfully"}
{"time":"2025-09-22T08:07:40.544761941Z","level":"INFO","msg":"⚙️ Test configuration loaded","network":"hoodi_stage","nodes":[304,305,306,307],"test_type":{"Sanity":{"ShouldRun":true,"E2M":false,"BulkAmount":0},"Bulk":{"ShouldRun":false,"E2M":false,"BulkAmount":0},"Isolated":{"Tests":null}}}
{"time":"2025-09-22T08:07:45.128794943Z","level":"INFO","msg":"Generated operator pubkeys file","module":"scanner","path":"/app/shared/data/operators/operator-pubkeys-hoodi_stage.json"}
{"time":"2025-09-22T08:07:45.129066724Z","level":"INFO","msg":"Registered parser","module":"main","component":"parser-client","parser_type":"consensus"}
{"time":"2025-09-22T08:07:45.129139141Z","level":"INFO","msg":"Registered parser","module":"main","component":"parser-client","parser_type":"flow-emits"}
{"time":"2025-09-22T08:07:45.1291877Z","level":"INFO","msg":"Validation client initialized","module":"main","loki_enabled":true,"e2m_enabled":false}
{"time":"2025-09-22T08:07:45.129251489Z","level":"INFO","msg":"🔑 KEY STEP: Starting Isolated Tests Concurrently if exists...","module":"main"}
{"time":"2025-09-22T08:07:45.12941099Z","level":"INFO","msg":"🔑 KEY STEP: Starting Sanity Test Suite","module":"main"}
{"time":"2025-09-22T08:07:45.129521456Z","level":"INFO","msg":"No isolated tests configured, skipping","module":"main"}
{"time":"2025-09-22T08:07:45.183388585Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":304}
{"time":"2025-09-22T08:07:45.194686526Z","level":"INFO","msg":"Available slots","test":"sanity","node":304,"slots":3000}
{"time":"2025-09-22T08:07:45.194785292Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":305}
{"time":"2025-09-22T08:07:45.205166139Z","level":"INFO","msg":"Available slots","test":"sanity","node":305,"slots":399}
{"time":"2025-09-22T08:07:45.205247591Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":306}
{"time":"2025-09-22T08:07:45.212004472Z","level":"INFO","msg":"Available slots","test":"sanity","node":306,"slots":299}
{"time":"2025-09-22T08:07:45.212094644Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":307}
{"time":"2025-09-22T08:07:45.221957293Z","level":"INFO","msg":"Available slots","test":"sanity","node":307,"slots":717}
{"time":"2025-09-22T08:08:59.713898771Z","level":"INFO","msg":"Extracted keyshares path","test":"sanity","module":"keys","path":"/app/shared/data/keyshares/0x88e97e72ea3a862b12b44028e65f2af490b8cc19d4b6f1915f1f80325cff9f02d8f5c6fd79f127c5fde7a90fcaa43b8d.json"}
{"time":"2025-09-22T08:08:59.713968415Z","level":"INFO","msg":"Key shares file generated successfully","test":"sanity","keyshare_path":"/app/shared/data/keyshares/0x88e97e72ea3a862b12b44028e65f2af490b8cc19d4b6f1915f1f80325cff9f02d8f5c6fd79f127c5fde7a90fcaa43b8d.json"}
{"time":"2025-09-22T08:09:03.055254142Z","level":"INFO","msg":"Allowance fetched successfully","test":"sanity","module":"contract","account":"0x5cc0dde14e7256340cc820415a6022a7d1c93a35","allowance":"9999999999999999999999598343194910796200000000"}
{"time":"2025-09-22T08:09:03.055365419Z","level":"INFO","msg":"Threshold value","test":"sanity","module":"contract","threshold":"9000000000000000000000"}
{"time":"2025-09-22T08:09:03.055398852Z","level":"INFO","msg":"Sufficient allowance","test":"sanity","module":"contract","allowance margin":"6561000000000000000000000000000000000000000000000000000000000000000000000000000000000000"}
{"time":"2025-09-22T08:09:03.062287711Z","level":"INFO","msg":"Allowance fetched successfully","test":"sanity","module":"contract","account":"0x5cc0dde14e7256340cc820415a6022a7d1c93a35","allowance":"9999999999999999999999598343194910796200000000"}
{"time":"2025-09-22T08:09:03.062382185Z","level":"INFO","msg":"Allowance","test":"sanity","module":"contract","allowance":"9999999999999999999999598343194910796200000000"}
{"time":"2025-09-22T08:09:03.252910108Z","level":"INFO","msg":"Registering validator","test":"sanity","module":"contract","tx_hash":"0x4a7329e1453a90c47b21f57e91dea5b0a6ffa7f0ed67e4ce23e0a13bba691da3"}
{"time":"2025-09-22T08:09:03.259064446Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0x4a7329e1453a90c47b21f57e91dea5b0a6ffa7f0ed67e4ce23e0a13bba691da3"}
{"time":"2025-09-22T08:09:13.275706118Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0x4a7329e1453a90c47b21f57e91dea5b0a6ffa7f0ed67e4ce23e0a13bba691da3"}
{"time":"2025-09-22T08:09:13.275778148Z","level":"ERROR","msg":"❌ FAILURE: Contract Registration Failed - Transaction verification failed","test":"sanity","txHash":"0x4a7329e1453a90c47b21f57e91dea5b0a6ffa7f0ed67e4ce23e0a13bba691da3"}
{"time":"2025-09-22T08:09:13.275796504Z","level":"ERROR","msg":"❌ FAILURE: Sanity Test Suite Failed","module":"main","error":"transaction verification failed"}
{"time":"2025-09-22T08:09:13.275814255Z","level":"INFO","msg":"⏳ PROGRESS: Waiting for all tests to finish","module":"main"}
{"time":"2025-09-22T08:09:13.275847909Z","level":"ERROR","msg":"❌ FAILURE: Test Execution Completed with Failures","module":"main","passed_suites":null,"failed_suites":["sanity"],"skipped_suites":["isolated"]}
{"time":"2025-09-22T08:09:13.275871894Z","level":"ERROR","msg":"❌ FAILURE: Suite Failure Details","module":"main","suite":"sanity","error":"transaction verification failed"}
{"time":"2025-09-22T08:09:13.275888496Z","level":"INFO","msg":"🔑 KEY STEP: Starting resource teardown...","module":"main","usage_id":"9343e9dc9be5a53056ea82c116ac9e3e"}
{"time":"2025-09-22T08:09:13.283745291Z","level":"INFO","msg":"🌐 Orchestrator API Client initialized","module":"main","endpoint":"http://aetheria-orchestrator.aetheria.svc:8080"}
{"time":"2025-09-22T08:09:13.283804292Z","level":"INFO","msg":"🧹 Requesting resource teardown","module":"main","usage_id":"9343e9dc9be5a53056ea82c116ac9e3e"}
{"time":"2025-09-22T08:09:13.336242567Z","level":"INFO","msg":"✅ SUCCESS: Completed resource teardown successfully","module":"main","usage_id":"9343e9dc9be5a53056ea82c116ac9e3e"}

2025-09-22T08:09:16Z The test run failed with error

Result - 🔴 Failure

yoaz-ssvlabs · 2025-09-22T08:21:43Z

/test help

ssv-test-bot · 2025-09-22T08:21:45Z

❓ Help

Available Commands

`/test` - Run Tests

To run tests, use the command /test followed by optional parameters.
Example:
/test cluster=300,301,302,303 tests=(sanity,e2m) text network=hoodi_stage

Available parameters (all optional):

cluster — Comma-separated operators IDs. If omitted (recommended), the workflow auto-allocates nodes. Current avaiable range 300-316.
tests — A list describing which tests to run. Default: (sanity,e2m)
network — Network configuration to use. Default: hoodi_stage

🧪 TESTS parameter

Format:

(type[,options]),(type[,options]),...

Supported test types:

sanity[,e2m] — Sanity tests (optional e2m validation)
bulk[,e2m][,amount] — Bulk tests (optional e2m validation; optional validator count as amount - default 3, max 10)
vouch,key:value — Vouch tests (requires all keys)

Isolated parameters for vouch (all required):

epochs:N — Number of epochs to examine
amountOfETHNodes:N — Number of ETH1 nodes vouch deployed with

💡 Usage examples

Default (no cluster, no network, no tests; defaults kick in):

/test

Simple bulk run with amount only, default network (hoodi_stage):

/test cluster=1,2,3,4 tests=(bulk,5)

With specific cluster, e2m on both sanity and bulk, fixed amount:

/test cluster=300,301,302,303 tests=(sanity,e2m),(bulk,e2m,10) network=hoodi_stage

Sanity plus a vouch run:

/test cluster=300,301,302,303 tests=(sanity),(vouch,epochs:5,amountOfETHNodes:4) network=hoodi_stage

Simple bulk run with default validators amount (3):

/test cluster=1,2,3,4 tests=(bulk) network=hoodi_stage

`/test skip-ci` - Skip CI Pipeline

⚠️ Restricted Command: Only members of authorized teams can use this command.

Bypasses the CI pipeline by setting the configured status check to success.
Example:

/test skip-ci

Important Notes:

Only members of authorized teams (configurable) can execute this command
Use with caution as it bypasses all automated checks

yoaz-ssvlabs · 2025-09-22T08:23:10Z

/test cluster=308,309,310,311 tests=(sanity)

ssv-test-bot · 2025-09-22T08:23:14Z

🟢 Test run for PR #2433

Branch: ci-test-handler-testing2
Commit: 74b7038

Status

2025-09-22T08:23:14Z Starting test run
2025-09-22T08:23:15Z Test run submitted successfully for commit 74b7038
2025-09-22T08:23:16Z container image for commit 74b7038f0265e7f8da4425e9315fe4199c751b80 is build and present in container registry
2025-09-22T08:23:17Z using user defined node range cluster=308,309,310,311 on network=hoodi_stage
2025-09-22T08:23:18Z deploying commit 74b7038f0265e7f8da4425e9315fe4199c751b80 to ssv nodes 308,309,310,311
2025-09-22T08:24:03Z commit 74b7038f0265e7f8da4425e9315fe4199c751b80 is deployed to ssv nodes 308,309,310,311

Click to expand executor logs

═══════════════════════════════════════════════════════════════════════════════════════════════════════
                🧪  WELCOME TO Executor - SSV's Distributed Test Automation Execution Layer  🧪            

                                  ___ __  __ ___  ___  _   _  _____  ___   ___ 
                                 | __|\\ \\/ /| __|/ __|| | | ||_   _|/ _ \\ | _ \\
                                 | _|  >  < | _|| (__ | |_| |  | | | (_) ||   /
                                 |___|/_/\\_\\|___|\\___| \\___/   |_|  \\___/ |_|_\\
                                        

═══════════════════════════════════════════════════════════════════════════════════════════════════════
{"time":"2025-09-22T08:24:06.880613404Z","level":"INFO","msg":"🐳 Docker mode: Using environment variables for config generation"}
{"time":"2025-09-22T08:24:06.886792158Z","level":"INFO","msg":"🌐 Orchestrator API Client initialized","endpoint":"http://aetheria-orchestrator.aetheria.svc:8080"}
{"time":"2025-09-22T08:24:06.886865802Z","level":"INFO","msg":"📝 Requesting configuration generation from orchestrator API","network":"hoodi_stage","clusters":[308,309,310,311],"tests":"(sanity)"}
{"time":"2025-09-22T08:24:06.923323477Z","level":"INFO","msg":"✅ Configuration generated successfully via API","resource_usage_id":"820e342a6d461fbcfc427372e9e73b84"}
{"time":"2025-09-22T08:24:06.924703012Z","level":"INFO","msg":"🚀 EXECUTOR Initialized Successfully"}
{"time":"2025-09-22T08:24:06.924850403Z","level":"INFO","msg":"⚙️ Test configuration loaded","network":"hoodi_stage","nodes":[308,309,310,311],"test_type":{"Sanity":{"ShouldRun":true,"E2M":false,"BulkAmount":0},"Bulk":{"ShouldRun":false,"E2M":false,"BulkAmount":0},"Isolated":{"Tests":null}}}
{"time":"2025-09-22T08:24:11.62627654Z","level":"INFO","msg":"Generated operator pubkeys file","module":"scanner","path":"/app/shared/data/operators/operator-pubkeys-hoodi_stage.json"}
{"time":"2025-09-22T08:24:11.626432427Z","level":"INFO","msg":"Registered parser","module":"main","component":"parser-client","parser_type":"consensus"}
{"time":"2025-09-22T08:24:11.626475946Z","level":"INFO","msg":"Registered parser","module":"main","component":"parser-client","parser_type":"flow-emits"}
{"time":"2025-09-22T08:24:11.626496127Z","level":"INFO","msg":"Validation client initialized","module":"main","loki_enabled":true,"e2m_enabled":false}
{"time":"2025-09-22T08:24:11.626548216Z","level":"INFO","msg":"🔑 KEY STEP: Starting Isolated Tests Concurrently if exists...","module":"main"}
{"time":"2025-09-22T08:24:11.626630724Z","level":"INFO","msg":"🔑 KEY STEP: Starting Sanity Test Suite","module":"main"}
{"time":"2025-09-22T08:24:11.626815572Z","level":"INFO","msg":"No isolated tests configured, skipping","module":"main"}
{"time":"2025-09-22T08:24:11.690457216Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":308}
{"time":"2025-09-22T08:24:11.703829736Z","level":"INFO","msg":"Available slots","test":"sanity","node":308,"slots":3000}
{"time":"2025-09-22T08:24:11.703990783Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":309}
{"time":"2025-09-22T08:24:11.712937129Z","level":"INFO","msg":"Available slots","test":"sanity","node":309,"slots":399}
{"time":"2025-09-22T08:24:11.713022446Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":310}
{"time":"2025-09-22T08:24:11.722016392Z","level":"INFO","msg":"Available slots","test":"sanity","node":310,"slots":299}
{"time":"2025-09-22T08:24:11.722094361Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":311}
{"time":"2025-09-22T08:24:11.730871464Z","level":"INFO","msg":"Available slots","test":"sanity","node":311,"slots":717}
{"time":"2025-09-22T08:24:49.553004792Z","level":"INFO","msg":"Extracted keyshares path","test":"sanity","module":"keys","path":"/app/shared/data/keyshares/0x88e97e72ea3a862b12b44028e65f2af490b8cc19d4b6f1915f1f80325cff9f02d8f5c6fd79f127c5fde7a90fcaa43b8d.json"}
{"time":"2025-09-22T08:24:49.553049219Z","level":"INFO","msg":"Key shares file generated successfully","test":"sanity","keyshare_path":"/app/shared/data/keyshares/0x88e97e72ea3a862b12b44028e65f2af490b8cc19d4b6f1915f1f80325cff9f02d8f5c6fd79f127c5fde7a90fcaa43b8d.json"}
{"time":"2025-09-22T08:24:52.904422548Z","level":"INFO","msg":"Allowance fetched successfully","test":"sanity","module":"contract","account":"0xceefd323dd28a8d9514eddfec45a6c81800a7d49","allowance":"9999999999999999999999831869028857427200000000"}
{"time":"2025-09-22T08:24:52.904539224Z","level":"INFO","msg":"Threshold value","test":"sanity","module":"contract","threshold":"9000000000000000000000"}
{"time":"2025-09-22T08:24:52.904571713Z","level":"INFO","msg":"Sufficient allowance","test":"sanity","module":"contract","allowance margin":"6561000000000000000000000000000000000000000000000000000000000000000000000000000000000000"}
{"time":"2025-09-22T08:24:52.908871322Z","level":"INFO","msg":"Allowance fetched successfully","test":"sanity","module":"contract","account":"0xceefd323dd28a8d9514eddfec45a6c81800a7d49","allowance":"9999999999999999999999831869028857427200000000"}
{"time":"2025-09-22T08:24:52.908971734Z","level":"INFO","msg":"Allowance","test":"sanity","module":"contract","allowance":"9999999999999999999999831869028857427200000000"}
{"time":"2025-09-22T08:24:52.980885693Z","level":"INFO","msg":"Registering validator","test":"sanity","module":"contract","tx_hash":"0x7293f05bccac7a27344f72d1e126db4533da4a4ada7184212b39e99ed96fa934"}
{"time":"2025-09-22T08:24:52.997928837Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0x7293f05bccac7a27344f72d1e126db4533da4a4ada7184212b39e99ed96fa934"}
{"time":"2025-09-22T08:25:03.005638217Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0x7293f05bccac7a27344f72d1e126db4533da4a4ada7184212b39e99ed96fa934"}
{"time":"2025-09-22T08:25:03.00580911Z","level":"INFO","msg":"✅ SUCCESS: Contract Register Successful","test":"sanity","txHash":"0x7293f05bccac7a27344f72d1e126db4533da4a4ada7184212b39e99ed96fa934","block_hash":"0xf66d68236d42a3c34eefb4617137395f30387dbfff36b766eb56b01f35d503a7"}
{"time":"2025-09-22T08:25:03.018519464Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"sanity","module":"helpers","pubkey":"0x88e97e72ea3a862b12b44028e65f2af490b8cc19d4b6f1915f1f80325cff9f02d8f5c6fd79f127c5fde7a90fcaa43b8d","timeToSleep":10,"attempt":1}
{"time":"2025-09-22T08:25:13.029968553Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"sanity","module":"helpers","pubkey":"0x88e97e72ea3a862b12b44028e65f2af490b8cc19d4b6f1915f1f80325cff9f02d8f5c6fd79f127c5fde7a90fcaa43b8d","timeToSleep":20,"attempt":2}
{"time":"2025-09-22T08:25:33.047417603Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"sanity","module":"helpers","pubkey":"0x88e97e72ea3a862b12b44028e65f2af490b8cc19d4b6f1915f1f80325cff9f02d8f5c6fd79f127c5fde7a90fcaa43b8d","timeToSleep":40,"attempt":3}
{"time":"2025-09-22T08:26:13.085536882Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"sanity","module":"helpers","pubkey":"0x88e97e72ea3a862b12b44028e65f2af490b8cc19d4b6f1915f1f80325cff9f02d8f5c6fd79f127c5fde7a90fcaa43b8d","timeToSleep":80,"attempt":4}
{"time":"2025-09-22T08:27:33.102499167Z","level":"INFO","msg":"Successfully retrieved validator index from exporter","test":"sanity","module":"helpers","pubkey":"0x88e97e72ea3a862b12b44028e65f2af490b8cc19d4b6f1915f1f80325cff9f02d8f5c6fd79f127c5fde7a90fcaa43b8d"}
{"time":"2025-09-22T08:27:33.37781925Z","level":"INFO","msg":"Validator event validation successful","module":"main","module":"loki-validator","validated_validators":1,"event":"ValidatorAdded"}
{"time":"2025-09-22T08:27:33.377901479Z","level":"INFO","msg":"SSV node log validation successful","test":"sanity","indices":["1092460"],"validators":1,"validate-type":"registration"}
{"time":"2025-09-22T08:27:36.960213133Z","level":"INFO","msg":"Liquidating cluster","test":"sanity","module":"contract","tx_hash":"0x5a1f11efccf1ef412783ffb3cce98b8dcd43dfae1d883d0078ff05b235fafc10"}
{"time":"2025-09-22T08:27:36.965664408Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0x5a1f11efccf1ef412783ffb3cce98b8dcd43dfae1d883d0078ff05b235fafc10"}
{"time":"2025-09-22T08:27:46.969766725Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":2,"sleep":20,"tx_hash":"0x5a1f11efccf1ef412783ffb3cce98b8dcd43dfae1d883d0078ff05b235fafc10"}
{"time":"2025-09-22T08:28:06.990295354Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0x5a1f11efccf1ef412783ffb3cce98b8dcd43dfae1d883d0078ff05b235fafc10"}
{"time":"2025-09-22T08:28:06.990427617Z","level":"INFO","msg":"✅ SUCCESS: Cluster Liquidation Successful","test":"sanity","txHash":"0x5a1f11efccf1ef412783ffb3cce98b8dcd43dfae1d883d0078ff05b235fafc10","block_hash":"0xb05cf0e3e30f9e877611420a2e100747dc24fad8f0d5b7c0133bf8461bef23c9"}
{"time":"2025-09-22T08:28:06.990678059Z","level":"INFO","msg":"Querying Loki for transaction hash validation","module":"main","module":"loki-validator","query":"{container=~\\"ssv-node-308|ssv-node-309|ssv-node-310|ssv-node-311\\"} |= \\"0x5a1f11efccf1ef412783ffb3cce98b8dcd43dfae1d883d0078ff05b235fafc10\\"","tx_hash":"0x5a1f11efccf1ef412783ffb3cce98b8dcd43dfae1d883d0078ff05b235fafc10"}
{"time":"2025-09-22T08:28:07.121247114Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":1,"max_retries":5,"delay_seconds":30,"start_unix":1758518886,"end_unix":1758529686}
{"time":"2025-09-22T08:28:37.174919358Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":2,"max_retries":5,"delay_seconds":60,"start_unix":1758518886,"end_unix":1758529717}
{"time":"2025-09-22T08:29:37.217101759Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":3,"max_retries":5,"delay_seconds":90,"start_unix":1758518886,"end_unix":1758529777}
{"time":"2025-09-22T08:31:07.332990033Z","level":"INFO","msg":"Event validation successful","module":"main","module":"loki-validator","valid_entries":4,"event":"ClusterLiquidated"}
{"time":"2025-09-22T08:31:07.333163569Z","level":"INFO","msg":"SSV node log validation successful","test":"sanity","indices":["1092460"],"validators":1,"validate-type":"liquidation"}
{"time":"2025-09-22T08:31:10.672499251Z","level":"INFO","msg":"Allowance fetched successfully","test":"sanity","module":"contract","account":"0xceefd323dd28a8d9514eddfec45a6c81800a7d49","allowance":"9999999999999999999999831869028857427200000000"}
{"time":"2025-09-22T08:31:10.672602721Z","level":"INFO","msg":"Threshold value","test":"sanity","module":"contract","threshold":"9000000000000000000000"}
{"time":"2025-09-22T08:31:10.672635338Z","level":"INFO","msg":"Sufficient allowance","test":"sanity","module":"contract","allowance margin":"6561000000000000000000000000000000000000000000000000000000000000000000000000000000000000"}
{"time":"2025-09-22T08:31:10.853789154Z","level":"INFO","msg":"Reactivating cluster","test":"sanity","module":"contract","tx_hash":"0x94f2500ebce941fa537abd51c0e98300b875e5ee772c272140d16142bd6656c1"}
{"time":"2025-09-22T08:31:10.857413446Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0x94f2500ebce941fa537abd51c0e98300b875e5ee772c272140d16142bd6656c1"}
{"time":"2025-09-22T08:31:20.875594482Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":2,"sleep":20,"tx_hash":"0x94f2500ebce941fa537abd51c0e98300b875e5ee772c272140d16142bd6656c1"}
{"time":"2025-09-22T08:31:40.897597574Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0x94f2500ebce941fa537abd51c0e98300b875e5ee772c272140d16142bd6656c1"}
{"time":"2025-09-22T08:31:40.897736589Z","level":"INFO","msg":"✅ SUCCESS: Cluster Reactivation Successful","test":"sanity","txHash":"0x94f2500ebce941fa537abd51c0e98300b875e5ee772c272140d16142bd6656c1","block_hash":"0xee6eb4efc4a4eb989459ee056a0b4601644bf30194bef87e798ced24ee682143"}
{"time":"2025-09-22T08:31:40.897904803Z","level":"INFO","msg":"Querying Loki for transaction hash validation","module":"main","module":"loki-validator","query":"{container=~\\"ssv-node-308|ssv-node-309|ssv-node-310|ssv-node-311\\"} |= \\"0x94f2500ebce941fa537abd51c0e98300b875e5ee772c272140d16142bd6656c1\\"","tx_hash":"0x94f2500ebce941fa537abd51c0e98300b875e5ee772c272140d16142bd6656c1"}
{"time":"2025-09-22T08:31:41.031065356Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":1,"max_retries":5,"delay_seconds":30,"start_unix":1758519100,"end_unix":1758529900}
{"time":"2025-09-22T08:32:11.085923045Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":2,"max_retries":5,"delay_seconds":60,"start_unix":1758519100,"end_unix":1758529931}
{"time":"2025-09-22T08:33:11.220856439Z","level":"INFO","msg":"Event validation successful","module":"main","module":"loki-validator","valid_entries":4,"event":"ClusterReactivated"}
{"time":"2025-09-22T08:33:11.221045918Z","level":"INFO","msg":"SSV node log validation successful","test":"sanity","indices":["1092460"],"validators":1,"validate-type":"reactivation"}
{"time":"2025-09-22T08:33:14.259182332Z","level":"INFO","msg":"Allowance fetched successfully","test":"sanity","module":"contract","account":"0xceefd323dd28a8d9514eddfec45a6c81800a7d49","allowance":"9999999999999999999999822869028857427200000000"}
{"time":"2025-09-22T08:33:14.25935485Z","level":"INFO","msg":"Threshold value","test":"sanity","module":"contract","threshold":"9000000000000000000000"}
{"time":"2025-09-22T08:33:14.259395862Z","level":"INFO","msg":"Sufficient allowance","test":"sanity","module":"contract","allowance margin":"6561000000000000000000000000000000000000000000000000000000000000000000000000000000000000"}
{"time":"2025-09-22T08:33:14.555173371Z","level":"INFO","msg":"Removing validator","test":"sanity","module":"contract","tx_hash":"0x911c698f21e2a198e9d1b88b377c6867931361eabe02cdcec7ccd74bab4e541c"}
{"time":"2025-09-22T08:33:14.558809084Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0x911c698f21e2a198e9d1b88b377c6867931361eabe02cdcec7ccd74bab4e541c"}
{"time":"2025-09-22T08:33:24.563382912Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":2,"sleep":20,"tx_hash":"0x911c698f21e2a198e9d1b88b377c6867931361eabe02cdcec7ccd74bab4e541c"}
{"time":"2025-09-22T08:33:44.571499Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0x911c698f21e2a198e9d1b88b377c6867931361eabe02cdcec7ccd74bab4e541c"}
{"time":"2025-09-22T08:33:44.571619228Z","level":"INFO","msg":"✅ SUCCESS: Contract Removal Successful","test":"sanity","blockHash":"0xa9ca2e96c409bfbf00e377331714f1bf63f1fb7d0dc6af1e85f5ff59f5f1c7e6"}
{"time":"2025-09-22T08:33:44.696705318Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":1,"max_retries":5,"delay_seconds":30,"start_unix":1758519224,"end_unix":1758530024}
{"time":"2025-09-22T08:34:14.787161962Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":2,"max_retries":5,"delay_seconds":60,"start_unix":1758519224,"end_unix":1758530054}
{"time":"2025-09-22T08:35:14.903181527Z","level":"INFO","msg":"Validator event validation successful","module":"main","module":"loki-validator","validated_validators":1,"event":"ValidatorRemoved"}
{"time":"2025-09-22T08:35:14.903244375Z","level":"INFO","msg":"SSV node log validation successful","test":"sanity","indices":["1092460"],"validators":1,"validate-type":"removal"}
{"time":"2025-09-22T08:35:14.903305769Z","level":"INFO","msg":"✅ SUCCESS: Sanity Test Suite Completed Successfully","module":"main","register":{"Contract":true,"E2M":false,"Loki":true},"liquidate":{"Contract":true,"E2M":false,"Loki":true},"reactivate":{"Contract":true,"E2M":false,"Loki":true},"remove":{"Contract":true,"E2M":false,"Loki":true}}
{"time":"2025-09-22T08:35:14.903315174Z","level":"INFO","msg":"⏳ PROGRESS: Waiting for all tests to finish","module":"main"}
{"time":"2025-09-22T08:35:14.903334559Z","level":"INFO","msg":"✅ SUCCESS: All Test Suites Completed Successfully","module":"main","passed_suites":["sanity"],"skipped_suites":["isolated"]}
{"time":"2025-09-22T08:35:14.903344336Z","level":"INFO","msg":"🔑 KEY STEP: Starting resource teardown...","module":"main","usage_id":"820e342a6d461fbcfc427372e9e73b84"}
{"time":"2025-09-22T08:35:14.910341588Z","level":"INFO","msg":"🌐 Orchestrator API Client initialized","module":"main","endpoint":"http://aetheria-orchestrator.aetheria.svc:8080"}
{"time":"2025-09-22T08:35:14.910378212Z","level":"INFO","msg":"🧹 Requesting resource teardown","module":"main","usage_id":"820e342a6d461fbcfc427372e9e73b84"}
{"time":"2025-09-22T08:35:14.94913904Z","level":"INFO","msg":"✅ SUCCESS: Completed resource teardown successfully","module":"main","usage_id":"820e342a6d461fbcfc427372e9e73b84"}

2025-09-22T08:35:18Z The test run finished successfully

Result - 🟢 Success

yoaz-ssvlabs · 2025-09-22T08:51:27Z

/test

ssv-test-bot · 2025-09-22T08:51:31Z

🟢 Test run for PR #2433

Branch: ci-test-handler-testing2
Commit: 74b7038

Status

2025-09-22T08:51:30Z Starting test run
2025-09-22T08:51:32Z Test run submitted successfully for commit 74b7038
2025-09-22T08:51:33Z container image for commit 74b7038f0265e7f8da4425e9315fe4199c751b80 is build and present in container registry
2025-09-22T08:51:34Z automatically allocated node range cluster=300,301,302,303 for test run on network=hoodi_stage
2025-09-22T08:51:35Z deploying commit 74b7038f0265e7f8da4425e9315fe4199c751b80 to ssv nodes 300,301,302,303
2025-09-22T08:52:20Z started test executor with tests=(sanity) and network=hoodi_stage

Click to expand executor logs

═══════════════════════════════════════════════════════════════════════════════════════════════════════
                🧪  WELCOME TO Executor - SSV's Distributed Test Automation Execution Layer  🧪            

                                  ___ __  __ ___  ___  _   _  _____  ___   ___ 
                                 | __|\\ \\/ /| __|/ __|| | | ||_   _|/ _ \\ | _ \\
                                 | _|  >  < | _|| (__ | |_| |  | | | (_) ||   /
{"time":"2025-09-22T08:52:22.426843656Z","level":"INFO","msg":"🐳 Docker mode: Using environment variables for config generation"}
                                 |___|/_/\\_\\|___|\\___| \\___/   |_|  \\___/ |_|_\\
                                        

═══════════════════════════════════════════════════════════════════════════════════════════════════════
{"time":"2025-09-22T08:52:22.434357771Z","level":"INFO","msg":"🌐 Orchestrator API Client initialized","endpoint":"http://aetheria-orchestrator.aetheria.svc:8080"}
{"time":"2025-09-22T08:52:22.434445618Z","level":"INFO","msg":"📝 Requesting configuration generation from orchestrator API","network":"hoodi_stage","clusters":[300,301,302,303],"tests":"(sanity)"}
{"time":"2025-09-22T08:52:22.47598644Z","level":"INFO","msg":"✅ Configuration generated successfully via API","resource_usage_id":"e479c3c4a314fbead381b0434fcb1d91"}
{"time":"2025-09-22T08:52:22.477056413Z","level":"INFO","msg":"🚀 EXECUTOR Initialized Successfully"}
{"time":"2025-09-22T08:52:22.477147327Z","level":"INFO","msg":"⚙️ Test configuration loaded","network":"hoodi_stage","nodes":[300,301,302,303],"test_type":{"Sanity":{"ShouldRun":true,"E2M":false,"BulkAmount":0},"Bulk":{"ShouldRun":false,"E2M":false,"BulkAmount":0},"Isolated":{"Tests":null}}}
{"time":"2025-09-22T08:52:26.681524104Z","level":"INFO","msg":"Generated operator pubkeys file","module":"scanner","path":"/app/shared/data/operators/operator-pubkeys-hoodi_stage.json"}
{"time":"2025-09-22T08:52:26.681815796Z","level":"INFO","msg":"Registered parser","module":"main","component":"parser-client","parser_type":"consensus"}
{"time":"2025-09-22T08:52:26.681874529Z","level":"INFO","msg":"Registered parser","module":"main","component":"parser-client","parser_type":"flow-emits"}
{"time":"2025-09-22T08:52:26.681920457Z","level":"INFO","msg":"Validation client initialized","module":"main","loki_enabled":true,"e2m_enabled":false}
{"time":"2025-09-22T08:52:26.681962683Z","level":"INFO","msg":"🔑 KEY STEP: Starting Isolated Tests Concurrently if exists...","module":"main"}
{"time":"2025-09-22T08:52:26.682018191Z","level":"INFO","msg":"🔑 KEY STEP: Starting Sanity Test Suite","module":"main"}
{"time":"2025-09-22T08:52:26.682066293Z","level":"INFO","msg":"No isolated tests configured, skipping","module":"main"}
{"time":"2025-09-22T08:52:26.731516643Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":300}
{"time":"2025-09-22T08:52:26.74319338Z","level":"INFO","msg":"Available slots","test":"sanity","node":300,"slots":3000}
{"time":"2025-09-22T08:52:26.7432892Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":301}
{"time":"2025-09-22T08:52:26.750202094Z","level":"INFO","msg":"Available slots","test":"sanity","node":301,"slots":399}
{"time":"2025-09-22T08:52:26.751218757Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":302}
{"time":"2025-09-22T08:52:26.757911733Z","level":"INFO","msg":"Available slots","test":"sanity","node":302,"slots":299}
{"time":"2025-09-22T08:52:26.758006086Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":303}
{"time":"2025-09-22T08:52:26.766672229Z","level":"INFO","msg":"Available slots","test":"sanity","node":303,"slots":717}
{"time":"2025-09-22T08:53:02.244147761Z","level":"INFO","msg":"Extracted keyshares path","test":"sanity","module":"keys","path":"/app/shared/data/keyshares/0x88e97e72ea3a862b12b44028e65f2af490b8cc19d4b6f1915f1f80325cff9f02d8f5c6fd79f127c5fde7a90fcaa43b8d.json"}
{"time":"2025-09-22T08:53:02.245212595Z","level":"INFO","msg":"Key shares file generated successfully","test":"sanity","keyshare_path":"/app/shared/data/keyshares/0x88e97e72ea3a862b12b44028e65f2af490b8cc19d4b6f1915f1f80325cff9f02d8f5c6fd79f127c5fde7a90fcaa43b8d.json"}
{"time":"2025-09-22T08:53:03.900144669Z","level":"INFO","msg":"Allowance fetched successfully","test":"sanity","module":"contract","account":"0x91e32efb8139cd88cae0df30d2bf471294c6ed27","allowance":"99375893999200282400000000"}
{"time":"2025-09-22T08:53:03.900251944Z","level":"INFO","msg":"Threshold value","test":"sanity","module":"contract","threshold":"9000000000000000000000"}
{"time":"2025-09-22T08:53:03.90027468Z","level":"INFO","msg":"Sufficient allowance","test":"sanity","module":"contract","allowance margin":"6561000000000000000000000000000000000000000000000000000000000000000000000000000000000000"}
{"time":"2025-09-22T08:53:03.90794545Z","level":"INFO","msg":"Allowance fetched successfully","test":"sanity","module":"contract","account":"0x91e32efb8139cd88cae0df30d2bf471294c6ed27","allowance":"99375893999200282400000000"}
{"time":"2025-09-22T08:53:03.908006147Z","level":"INFO","msg":"Allowance","test":"sanity","module":"contract","allowance":"99375893999200282400000000"}
{"time":"2025-09-22T08:53:04.000303739Z","level":"INFO","msg":"Registering validator","test":"sanity","module":"contract","tx_hash":"0xde1c217ee2e7ca4d406892a8a16c7b083a83835a167b10f08198ef1d29e284f6"}
{"time":"2025-09-22T08:53:04.005812126Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0xde1c217ee2e7ca4d406892a8a16c7b083a83835a167b10f08198ef1d29e284f6"}
{"time":"2025-09-22T08:53:14.016487288Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":2,"sleep":20,"tx_hash":"0xde1c217ee2e7ca4d406892a8a16c7b083a83835a167b10f08198ef1d29e284f6"}
{"time":"2025-09-22T08:53:34.023517024Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0xde1c217ee2e7ca4d406892a8a16c7b083a83835a167b10f08198ef1d29e284f6"}
{"time":"2025-09-22T08:53:34.023611492Z","level":"INFO","msg":"✅ SUCCESS: Contract Register Successful","test":"sanity","txHash":"0xde1c217ee2e7ca4d406892a8a16c7b083a83835a167b10f08198ef1d29e284f6","block_hash":"0x9e9fa9ac09165db2aaa256de02a176e5c04ffb637ae9c993e1a95bb297520d01"}
{"time":"2025-09-22T08:53:34.0439205Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"sanity","module":"helpers","pubkey":"0x88e97e72ea3a862b12b44028e65f2af490b8cc19d4b6f1915f1f80325cff9f02d8f5c6fd79f127c5fde7a90fcaa43b8d","timeToSleep":10,"attempt":1}
{"time":"2025-09-22T08:53:44.058132194Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"sanity","module":"helpers","pubkey":"0x88e97e72ea3a862b12b44028e65f2af490b8cc19d4b6f1915f1f80325cff9f02d8f5c6fd79f127c5fde7a90fcaa43b8d","timeToSleep":20,"attempt":2}
{"time":"2025-09-22T08:54:04.074746818Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"sanity","module":"helpers","pubkey":"0x88e97e72ea3a862b12b44028e65f2af490b8cc19d4b6f1915f1f80325cff9f02d8f5c6fd79f127c5fde7a90fcaa43b8d","timeToSleep":40,"attempt":3}
{"time":"2025-09-22T08:54:44.111258952Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"sanity","module":"helpers","pubkey":"0x88e97e72ea3a862b12b44028e65f2af490b8cc19d4b6f1915f1f80325cff9f02d8f5c6fd79f127c5fde7a90fcaa43b8d","timeToSleep":80,"attempt":4}
{"time":"2025-09-22T08:56:04.127272353Z","level":"INFO","msg":"Successfully retrieved validator index from exporter","test":"sanity","module":"helpers","pubkey":"0x88e97e72ea3a862b12b44028e65f2af490b8cc19d4b6f1915f1f80325cff9f02d8f5c6fd79f127c5fde7a90fcaa43b8d"}
{"time":"2025-09-22T08:56:04.537249373Z","level":"INFO","msg":"Validator event validation successful","module":"main","module":"loki-validator","validated_validators":1,"event":"ValidatorAdded"}
{"time":"2025-09-22T08:56:04.537324472Z","level":"INFO","msg":"SSV node log validation successful","test":"sanity","indices":["1092460"],"validators":1,"validate-type":"registration"}
{"time":"2025-09-22T08:56:06.360186408Z","level":"INFO","msg":"Liquidating cluster","test":"sanity","module":"contract","tx_hash":"0x9a64ceebd16a2886f9cb639945028871f67e7f28d935b52c6ccc79cf065c53f5"}
{"time":"2025-09-22T08:56:06.373271646Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0x9a64ceebd16a2886f9cb639945028871f67e7f28d935b52c6ccc79cf065c53f5"}
{"time":"2025-09-22T08:56:16.388493452Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0x9a64ceebd16a2886f9cb639945028871f67e7f28d935b52c6ccc79cf065c53f5"}
{"time":"2025-09-22T08:56:16.388639625Z","level":"INFO","msg":"✅ SUCCESS: Cluster Liquidation Successful","test":"sanity","txHash":"0x9a64ceebd16a2886f9cb639945028871f67e7f28d935b52c6ccc79cf065c53f5","block_hash":"0x548156e2f24c7a21618bd8fe984c3dbedab48e6339ea91e79db83f5c1cac3ba8"}
{"time":"2025-09-22T08:56:16.388794719Z","level":"INFO","msg":"Querying Loki for transaction hash validation","module":"main","module":"loki-validator","query":"{container=~\\"ssv-node-300|ssv-node-301|ssv-node-302|ssv-node-303\\"} |= \\"0x9a64ceebd16a2886f9cb639945028871f67e7f28d935b52c6ccc79cf065c53f5\\"","tx_hash":"0x9a64ceebd16a2886f9cb639945028871f67e7f28d935b52c6ccc79cf065c53f5"}
{"time":"2025-09-22T08:56:16.657755175Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":1,"max_retries":5,"delay_seconds":30,"start_unix":1758520576,"end_unix":1758531376}
{"time":"2025-09-22T08:56:46.831439176Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":2,"max_retries":5,"delay_seconds":60,"start_unix":1758520576,"end_unix":1758531406}
{"time":"2025-09-22T08:57:47.020210994Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":3,"max_retries":5,"delay_seconds":90,"start_unix":1758520576,"end_unix":1758531466}
{"time":"2025-09-22T08:59:17.244036829Z","level":"INFO","msg":"Event validation successful","module":"main","module":"loki-validator","valid_entries":4,"event":"ClusterLiquidated"}
{"time":"2025-09-22T08:59:17.244185127Z","level":"INFO","msg":"SSV node log validation successful","test":"sanity","indices":["1092460"],"validators":1,"validate-type":"liquidation"}
{"time":"2025-09-22T08:59:18.978853067Z","level":"INFO","msg":"Allowance fetched successfully","test":"sanity","module":"contract","account":"0x91e32efb8139cd88cae0df30d2bf471294c6ed27","allowance":"99375893999200282400000000"}
{"time":"2025-09-22T08:59:18.979010983Z","level":"INFO","msg":"Threshold value","test":"sanity","module":"contract","threshold":"9000000000000000000000"}
{"time":"2025-09-22T08:59:18.979050609Z","level":"INFO","msg":"Sufficient allowance","test":"sanity","module":"contract","allowance margin":"6561000000000000000000000000000000000000000000000000000000000000000000000000000000000000"}
{"time":"2025-09-22T08:59:19.13450286Z","level":"INFO","msg":"Reactivating cluster","test":"sanity","module":"contract","tx_hash":"0xba368cc0fb695132ab2d96873c09a8db7730ee8bc8755268e2e729903c41b605"}
{"time":"2025-09-22T08:59:19.138783491Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0xba368cc0fb695132ab2d96873c09a8db7730ee8bc8755268e2e729903c41b605"}
{"time":"2025-09-22T08:59:29.144424032Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0xba368cc0fb695132ab2d96873c09a8db7730ee8bc8755268e2e729903c41b605"}
{"time":"2025-09-22T08:59:29.144503865Z","level":"INFO","msg":"✅ SUCCESS: Cluster Reactivation Successful","test":"sanity","txHash":"0xba368cc0fb695132ab2d96873c09a8db7730ee8bc8755268e2e729903c41b605","block_hash":"0xd8e7ac043c3f1631072e825380186568b7172d8c1f64b374bdbf298b6664d986"}
{"time":"2025-09-22T08:59:29.144588313Z","level":"INFO","msg":"Querying Loki for transaction hash validation","module":"main","module":"loki-validator","query":"{container=~\\"ssv-node-300|ssv-node-301|ssv-node-302|ssv-node-303\\"} |= \\"0xba368cc0fb695132ab2d96873c09a8db7730ee8bc8755268e2e729903c41b605\\"","tx_hash":"0xba368cc0fb695132ab2d96873c09a8db7730ee8bc8755268e2e729903c41b605"}
{"time":"2025-09-22T08:59:29.519669909Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":1,"max_retries":5,"delay_seconds":30,"start_unix":1758520769,"end_unix":1758531569}
{"time":"2025-09-22T08:59:59.704225385Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":2,"max_retries":5,"delay_seconds":60,"start_unix":1758520769,"end_unix":1758531599}
{"time":"2025-09-22T09:00:59.919178538Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":3,"max_retries":5,"delay_seconds":90,"start_unix":1758520769,"end_unix":1758531659}
{"time":"2025-09-22T09:02:30.171300089Z","level":"INFO","msg":"Event validation successful","module":"main","module":"loki-validator","valid_entries":4,"event":"ClusterReactivated"}
{"time":"2025-09-22T09:02:30.171478627Z","level":"INFO","msg":"SSV node log validation successful","test":"sanity","indices":["1092460"],"validators":1,"validate-type":"reactivation"}
{"time":"2025-09-22T09:02:31.89986937Z","level":"INFO","msg":"Allowance fetched successfully","test":"sanity","module":"contract","account":"0x91e32efb8139cd88cae0df30d2bf471294c6ed27","allowance":"99366893999200282400000000"}
{"time":"2025-09-22T09:02:31.899978382Z","level":"INFO","msg":"Threshold value","test":"sanity","module":"contract","threshold":"9000000000000000000000"}
{"time":"2025-09-22T09:02:31.900078313Z","level":"INFO","msg":"Sufficient allowance","test":"sanity","module":"contract","allowance margin":"6561000000000000000000000000000000000000000000000000000000000000000000000000000000000000"}
{"time":"2025-09-22T09:02:32.194264965Z","level":"INFO","msg":"Removing validator","test":"sanity","module":"contract","tx_hash":"0x8e03b4aad5e4aab2c744fd7295cead931332291caf42ae16127e90207fb38277"}
{"time":"2025-09-22T09:02:32.202747698Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0x8e03b4aad5e4aab2c744fd7295cead931332291caf42ae16127e90207fb38277"}
{"time":"2025-09-22T09:02:42.213947734Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0x8e03b4aad5e4aab2c744fd7295cead931332291caf42ae16127e90207fb38277"}
{"time":"2025-09-22T09:02:42.214113509Z","level":"INFO","msg":"✅ SUCCESS: Contract Removal Successful","test":"sanity","blockHash":"0xc4702471d722c1319d0c470333fb895220488f81d6e0c092fbe166500ac2d57e"}
{"time":"2025-09-22T09:02:42.629615333Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":1,"max_retries":5,"delay_seconds":30,"start_unix":1758520962,"end_unix":1758531762}
{"time":"2025-09-22T09:03:12.832989266Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":2,"max_retries":5,"delay_seconds":60,"start_unix":1758520962,"end_unix":1758531792}
{"time":"2025-09-22T09:04:13.005524702Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":3,"max_retries":5,"delay_seconds":90,"start_unix":1758520962,"end_unix":1758531852}
{"time":"2025-09-22T09:05:43.227855083Z","level":"INFO","msg":"Validator event validation successful","module":"main","module":"loki-validator","validated_validators":1,"event":"ValidatorRemoved"}
{"time":"2025-09-22T09:05:43.227967213Z","level":"INFO","msg":"SSV node log validation successful","test":"sanity","indices":["1092460"],"validators":1,"validate-type":"removal"}
{"time":"2025-09-22T09:05:43.228063737Z","level":"INFO","msg":"✅ SUCCESS: Sanity Test Suite Completed Successfully","module":"main","register":{"Contract":true,"E2M":false,"Loki":true},"liquidate":{"Contract":true,"E2M":false,"Loki":true},"reactivate":{"Contract":true,"E2M":false,"Loki":true},"remove":{"Contract":true,"E2M":false,"Loki":true}}
{"time":"2025-09-22T09:05:43.228080602Z","level":"INFO","msg":"⏳ PROGRESS: Waiting for all tests to finish","module":"main"}
{"time":"2025-09-22T09:05:43.228117058Z","level":"INFO","msg":"✅ SUCCESS: All Test Suites Completed Successfully","module":"main","passed_suites":["sanity"],"skipped_suites":["isolated"]}
{"time":"2025-09-22T09:05:43.228131842Z","level":"INFO","msg":"🔑 KEY STEP: Starting resource teardown...","module":"main","usage_id":"e479c3c4a314fbead381b0434fcb1d91"}
{"time":"2025-09-22T09:05:43.233694972Z","level":"INFO","msg":"🌐 Orchestrator API Client initialized","module":"main","endpoint":"http://aetheria-orchestrator.aetheria.svc:8080"}
{"time":"2025-09-22T09:05:43.234054091Z","level":"INFO","msg":"🧹 Requesting resource teardown","module":"main","usage_id":"e479c3c4a314fbead381b0434fcb1d91"}
{"time":"2025-09-22T09:05:43.289622166Z","level":"INFO","msg":"✅ SUCCESS: Completed resource teardown successfully","module":"main","usage_id":"e479c3c4a314fbead381b0434fcb1d91"}

2025-09-22T09:05:46Z The test run finished successfully

Result - 🟢 Success

yoaz-ssvlabs · 2025-09-30T14:09:52Z

/test

ssv-test-bot · 2025-09-30T14:09:56Z

🟢 Test run for PR #2433

Branch: ci-test-handler-testing2
Commit: 74b7038

Status

2025-09-30T14:09:55Z Starting test run
2025-09-30T14:09:57Z Test run submitted successfully for commit 74b7038
2025-09-30T14:09:58Z container image for commit 74b7038f0265e7f8da4425e9315fe4199c751b80 is build and present in container registry
2025-09-30T14:09:59Z automatically allocated node range cluster=304,305,306,307 for test run on network=hoodi_stage
2025-09-30T14:10:01Z deploying commit 74b7038f0265e7f8da4425e9315fe4199c751b80 to ssv nodes 304,305,306,307
2025-09-30T14:11:01Z started test executor with tests=(sanity) and network=hoodi_stage

Click to expand executor logs

═══════════════════════════════════════════════════════════════════════════════════════════════════════
                🧪  WELCOME TO Executor - SSV's Distributed Test Automation Execution Layer  🧪            

                                  ___ __  __ ___  ___  _   _  _____  ___   ___ 
                                 | __|\\ \\/ /| __|/ __|| | | ||_   _|/ _ \\ | _ \\
                                 | _|  >  < | _|| (__ | |_| |  | | | (_) ||   /
                                 |___|/_/\\_\\|___|\\___| \\___/   |_|  \\___/ |_|_\\
                                        

═══════════════════════════════════════════════════════════════════════════════════════════════════════
{"time":"2025-09-30T14:11:02.865052847Z","level":"INFO","msg":"🐳 Docker mode: Using environment variables for config generation"}
{"time":"2025-09-30T14:11:02.908270394Z","level":"INFO","msg":"🌐 Orchestrator API Client initialized","endpoint":"http://aetheria-orchestrator.aetheria.svc:8080"}
{"time":"2025-09-30T14:11:02.908323288Z","level":"INFO","msg":"📝 Requesting configuration generation from orchestrator API","network":"hoodi_stage","clusters":[304,305,306,307],"tests":"(sanity)"}
{"time":"2025-09-30T14:11:02.939220734Z","level":"INFO","msg":"✅ Configuration generated successfully via API","resource_usage_id":"593bde78f79ebdca3aa737c49c12e086"}
{"time":"2025-09-30T14:11:02.940148249Z","level":"INFO","msg":"🚀 EXECUTOR Initialized Successfully"}
{"time":"2025-09-30T14:11:02.940200195Z","level":"INFO","msg":"⚙️ Test configuration loaded","network":"hoodi_stage","nodes":[304,305,306,307],"test_type":{"Sanity":{"ShouldRun":true,"E2M":false,"BulkAmount":0},"Bulk":{"ShouldRun":false,"E2M":false,"BulkAmount":0},"Isolated":{"Tests":null}}}
{"time":"2025-09-30T14:11:07.543031656Z","level":"INFO","msg":"Generated operator pubkeys file","module":"scanner","path":"/app/shared/data/operators/operator-pubkeys-hoodi_stage.json"}
{"time":"2025-09-30T14:11:07.543214617Z","level":"INFO","msg":"Registered parser","module":"main","component":"parser-client","parser_type":"consensus"}
{"time":"2025-09-30T14:11:07.543259238Z","level":"INFO","msg":"Registered parser","module":"main","component":"parser-client","parser_type":"flow-emits"}
{"time":"2025-09-30T14:11:07.543422979Z","level":"INFO","msg":"Validation client initialized","module":"main","loki_enabled":true,"e2m_enabled":false}
{"time":"2025-09-30T14:11:07.543495521Z","level":"INFO","msg":"🔑 KEY STEP: Starting Isolated Tests Concurrently if exists...","module":"main"}
{"time":"2025-09-30T14:11:07.543575461Z","level":"INFO","msg":"🔑 KEY STEP: Starting Sanity Test Suite","module":"main"}
{"time":"2025-09-30T14:11:07.54496242Z","level":"INFO","msg":"No isolated tests configured, skipping","module":"main"}
{"time":"2025-09-30T14:11:07.602178882Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":304}
{"time":"2025-09-30T14:11:07.61782081Z","level":"INFO","msg":"Available slots","test":"sanity","node":304,"slots":3000}
{"time":"2025-09-30T14:11:07.617921981Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":305}
{"time":"2025-09-30T14:11:07.628897381Z","level":"INFO","msg":"Available slots","test":"sanity","node":305,"slots":398}
{"time":"2025-09-30T14:11:07.62903112Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":306}
{"time":"2025-09-30T14:11:07.637262756Z","level":"INFO","msg":"Available slots","test":"sanity","node":306,"slots":297}
{"time":"2025-09-30T14:11:07.637332592Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":307}
{"time":"2025-09-30T14:11:07.643972505Z","level":"INFO","msg":"Available slots","test":"sanity","node":307,"slots":716}
{"time":"2025-09-30T14:12:20.429919121Z","level":"INFO","msg":"Extracted keyshares path","test":"sanity","module":"keys","path":"/app/shared/data/keyshares/0x84d7c46eb947f61c65a1c32971bbad4a1e832b1998675374fdebab49e6ab6d2cc0a7208acb3d176ac8b5df0cd3587f23.json"}
{"time":"2025-09-30T14:12:20.430003886Z","level":"INFO","msg":"Key shares file generated successfully","test":"sanity","keyshare_path":"/app/shared/data/keyshares/0x84d7c46eb947f61c65a1c32971bbad4a1e832b1998675374fdebab49e6ab6d2cc0a7208acb3d176ac8b5df0cd3587f23.json"}
{"time":"2025-09-30T14:12:23.163078368Z","level":"INFO","msg":"Allowance fetched successfully","test":"sanity","module":"contract","account":"0x5cc0dde14e7256340cc820415a6022a7d1c93a35","allowance":"9999999999999999999999580326194910796200000000"}
{"time":"2025-09-30T14:12:23.163188192Z","level":"INFO","msg":"Threshold value","test":"sanity","module":"contract","threshold":"9000000000000000000000"}
{"time":"2025-09-30T14:12:23.163222668Z","level":"INFO","msg":"Sufficient allowance","test":"sanity","module":"contract","allowance margin":"6561000000000000000000000000000000000000000000000000000000000000000000000000000000000000"}
{"time":"2025-09-30T14:12:28.193725994Z","level":"INFO","msg":"Allowance fetched successfully","test":"sanity","module":"contract","account":"0x5cc0dde14e7256340cc820415a6022a7d1c93a35","allowance":"9999999999999999999999580326194910796200000000"}
{"time":"2025-09-30T14:12:28.193862543Z","level":"INFO","msg":"Allowance","test":"sanity","module":"contract","allowance":"9999999999999999999999580326194910796200000000"}
{"time":"2025-09-30T14:12:28.408575513Z","level":"INFO","msg":"Registering validator","test":"sanity","module":"contract","tx_hash":"0xeb37139e89fb18b017a9eff8fa452fd834109d59bbbcbde7601b8d636ef20f61"}
{"time":"2025-09-30T14:12:28.413190413Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0xeb37139e89fb18b017a9eff8fa452fd834109d59bbbcbde7601b8d636ef20f61"}
{"time":"2025-09-30T14:12:38.427687985Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0xeb37139e89fb18b017a9eff8fa452fd834109d59bbbcbde7601b8d636ef20f61"}
{"time":"2025-09-30T14:12:38.427822053Z","level":"INFO","msg":"✅ SUCCESS: Contract Register Successful","test":"sanity","txHash":"0xeb37139e89fb18b017a9eff8fa452fd834109d59bbbcbde7601b8d636ef20f61","block_hash":"0x628922d4709e77721548ea8bb07cc6f121ceee976cce3760e71a719be7f6f2dd"}
{"time":"2025-09-30T14:12:38.44172546Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"sanity","module":"helpers","pubkey":"0x84d7c46eb947f61c65a1c32971bbad4a1e832b1998675374fdebab49e6ab6d2cc0a7208acb3d176ac8b5df0cd3587f23","timeToSleep":10,"attempt":1}
{"time":"2025-09-30T14:12:48.46225179Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"sanity","module":"helpers","pubkey":"0x84d7c46eb947f61c65a1c32971bbad4a1e832b1998675374fdebab49e6ab6d2cc0a7208acb3d176ac8b5df0cd3587f23","timeToSleep":20,"attempt":2}
{"time":"2025-09-30T14:13:08.485502016Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"sanity","module":"helpers","pubkey":"0x84d7c46eb947f61c65a1c32971bbad4a1e832b1998675374fdebab49e6ab6d2cc0a7208acb3d176ac8b5df0cd3587f23","timeToSleep":40,"attempt":3}
{"time":"2025-09-30T14:13:48.530448723Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"sanity","module":"helpers","pubkey":"0x84d7c46eb947f61c65a1c32971bbad4a1e832b1998675374fdebab49e6ab6d2cc0a7208acb3d176ac8b5df0cd3587f23","timeToSleep":80,"attempt":4}
{"time":"2025-09-30T14:15:08.54752791Z","level":"INFO","msg":"Successfully retrieved validator index from exporter","test":"sanity","module":"helpers","pubkey":"0x84d7c46eb947f61c65a1c32971bbad4a1e832b1998675374fdebab49e6ab6d2cc0a7208acb3d176ac8b5df0cd3587f23"}
{"time":"2025-09-30T14:15:08.931097402Z","level":"INFO","msg":"Validator event validation successful","module":"main","module":"loki-validator","validated_validators":1,"event":"ValidatorAdded"}
{"time":"2025-09-30T14:15:08.931194256Z","level":"INFO","msg":"SSV node log validation successful","test":"sanity","indices":["1104199"],"validators":1,"validate-type":"registration"}
{"time":"2025-09-30T14:15:11.785201713Z","level":"INFO","msg":"Liquidating cluster","test":"sanity","module":"contract","tx_hash":"0xce59aae3e8172f29b520f863863f6244d1f79efa4f0294eb46a37ce66d2aa248"}
{"time":"2025-09-30T14:15:11.789144556Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0xce59aae3e8172f29b520f863863f6244d1f79efa4f0294eb46a37ce66d2aa248"}
{"time":"2025-09-30T14:15:21.794738046Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":2,"sleep":20,"tx_hash":"0xce59aae3e8172f29b520f863863f6244d1f79efa4f0294eb46a37ce66d2aa248"}
{"time":"2025-09-30T14:15:41.820118128Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0xce59aae3e8172f29b520f863863f6244d1f79efa4f0294eb46a37ce66d2aa248"}
{"time":"2025-09-30T14:15:41.820268112Z","level":"INFO","msg":"✅ SUCCESS: Cluster Liquidation Successful","test":"sanity","txHash":"0xce59aae3e8172f29b520f863863f6244d1f79efa4f0294eb46a37ce66d2aa248","block_hash":"0x31b7eee6ddae3233ff469cf13c2a3339a836f82e7cbd48c3eefd9f320dee4f21"}
{"time":"2025-09-30T14:15:41.820468628Z","level":"INFO","msg":"Querying Loki for transaction hash validation","module":"main","module":"loki-validator","query":"{container=~\\"ssv-node-304|ssv-node-305|ssv-node-306|ssv-node-307\\"} |= \\"0xce59aae3e8172f29b520f863863f6244d1f79efa4f0294eb46a37ce66d2aa248\\"","tx_hash":"0xce59aae3e8172f29b520f863863f6244d1f79efa4f0294eb46a37ce66d2aa248"}
{"time":"2025-09-30T14:15:41.933514997Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":1,"max_retries":5,"delay_seconds":30,"start_unix":1759230941,"end_unix":1759241741}
{"time":"2025-09-30T14:16:12.019909109Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":2,"max_retries":5,"delay_seconds":60,"start_unix":1759230941,"end_unix":1759241771}
{"time":"2025-09-30T14:17:12.026563214Z","level":"ERROR","msg":"request failed","module":"HTTPRequester","error":"Get \\"https://loki.ops.ssvlabsinternal.com/loki/api/v1/query_range?query=%7Bcontainer%3D~%22ssv-node-304%7Cssv-node-305%7Cssv-node-306%7Cssv-node-307%22%7D+%7C%3D+%220xce59aae3e8172f29b520f863863f6244d1f79efa4f0294eb46a37ce66d2aa248%22+%7C+json+%7C+line_format+%22%7B%7B.container%7D%7D%3A%7B%7B.msg%7D%7D%22&start=1759230941&end=1759241832&limit=1000\\": unexpected EOF","request attempt #":0}
{"time":"2025-09-30T14:17:12.20391006Z","level":"INFO","msg":"Event validation successful","module":"main","module":"loki-validator","valid_entries":4,"event":"ClusterLiquidated"}
{"time":"2025-09-30T14:17:12.20398641Z","level":"INFO","msg":"SSV node log validation successful","test":"sanity","indices":["1104199"],"validators":1,"validate-type":"liquidation"}
{"time":"2025-09-30T14:17:14.768170948Z","level":"INFO","msg":"Allowance fetched successfully","test":"sanity","module":"contract","account":"0x5cc0dde14e7256340cc820415a6022a7d1c93a35","allowance":"9999999999999999999999580326194910796200000000"}
{"time":"2025-09-30T14:17:14.768250776Z","level":"INFO","msg":"Threshold value","test":"sanity","module":"contract","threshold":"9000000000000000000000"}
{"time":"2025-09-30T14:17:14.768287987Z","level":"INFO","msg":"Sufficient allowance","test":"sanity","module":"contract","allowance margin":"6561000000000000000000000000000000000000000000000000000000000000000000000000000000000000"}
{"time":"2025-09-30T14:17:14.802173538Z","level":"INFO","msg":"Reactivating cluster","test":"sanity","module":"contract","tx_hash":"0x7b65789c46a01a1a38418971af2e4165fc7979fd3afffe3360b68187b701492e"}
{"time":"2025-09-30T14:17:14.805629353Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0x7b65789c46a01a1a38418971af2e4165fc7979fd3afffe3360b68187b701492e"}
{"time":"2025-09-30T14:17:24.815325452Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":2,"sleep":20,"tx_hash":"0x7b65789c46a01a1a38418971af2e4165fc7979fd3afffe3360b68187b701492e"}
{"time":"2025-09-30T14:17:44.822190136Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0x7b65789c46a01a1a38418971af2e4165fc7979fd3afffe3360b68187b701492e"}
{"time":"2025-09-30T14:17:44.822365998Z","level":"INFO","msg":"✅ SUCCESS: Cluster Reactivation Successful","test":"sanity","txHash":"0x7b65789c46a01a1a38418971af2e4165fc7979fd3afffe3360b68187b701492e","block_hash":"0xb0d99a3b5b0e7d4e5b8f4df391c756b33e0841e314ae3faaf0d9acc60c5a46b9"}
{"time":"2025-09-30T14:17:44.822536826Z","level":"INFO","msg":"Querying Loki for transaction hash validation","module":"main","module":"loki-validator","query":"{container=~\\"ssv-node-304|ssv-node-305|ssv-node-306|ssv-node-307\\"} |= \\"0x7b65789c46a01a1a38418971af2e4165fc7979fd3afffe3360b68187b701492e\\"","tx_hash":"0x7b65789c46a01a1a38418971af2e4165fc7979fd3afffe3360b68187b701492e"}
{"time":"2025-09-30T14:17:44.927395706Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":1,"max_retries":5,"delay_seconds":30,"start_unix":1759231064,"end_unix":1759241864}
{"time":"2025-09-30T14:18:15.011447856Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":2,"max_retries":5,"delay_seconds":60,"start_unix":1759231064,"end_unix":1759241894}
{"time":"2025-09-30T14:19:15.166461418Z","level":"INFO","msg":"Event validation successful","module":"main","module":"loki-validator","valid_entries":4,"event":"ClusterReactivated"}
{"time":"2025-09-30T14:19:15.166589853Z","level":"INFO","msg":"SSV node log validation successful","test":"sanity","indices":["1104199"],"validators":1,"validate-type":"reactivation"}
{"time":"2025-09-30T14:19:17.592701719Z","level":"INFO","msg":"Allowance fetched successfully","test":"sanity","module":"contract","account":"0x5cc0dde14e7256340cc820415a6022a7d1c93a35","allowance":"9999999999999999999999571326194910796200000000"}
{"time":"2025-09-30T14:19:17.59280183Z","level":"INFO","msg":"Threshold value","test":"sanity","module":"contract","threshold":"9000000000000000000000"}
{"time":"2025-09-30T14:19:17.592836536Z","level":"INFO","msg":"Sufficient allowance","test":"sanity","module":"contract","allowance margin":"6561000000000000000000000000000000000000000000000000000000000000000000000000000000000000"}
{"time":"2025-09-30T14:19:17.805289515Z","level":"INFO","msg":"Removing validator","test":"sanity","module":"contract","tx_hash":"0x5e2e75084f11808a61053fd34f4db32aa85d043e1d1dc25d4cedd73b31a60780"}
{"time":"2025-09-30T14:19:17.809786811Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0x5e2e75084f11808a61053fd34f4db32aa85d043e1d1dc25d4cedd73b31a60780"}
{"time":"2025-09-30T14:19:27.815832807Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0x5e2e75084f11808a61053fd34f4db32aa85d043e1d1dc25d4cedd73b31a60780"}
{"time":"2025-09-30T14:19:27.815994579Z","level":"INFO","msg":"✅ SUCCESS: Contract Removal Successful","test":"sanity","blockHash":"0xdca22f9b5503ff946677df362e5db918cb5c768d4f2a399dfd531dbd622a1f6c"}
{"time":"2025-09-30T14:19:27.91463331Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":1,"max_retries":5,"delay_seconds":30,"start_unix":1759231167,"end_unix":1759241967}
{"time":"2025-09-30T14:19:58.025654991Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":2,"max_retries":5,"delay_seconds":60,"start_unix":1759231167,"end_unix":1759241997}
{"time":"2025-09-30T14:20:58.124490085Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":3,"max_retries":5,"delay_seconds":90,"start_unix":1759231167,"end_unix":1759242058}
{"time":"2025-09-30T14:22:28.554814438Z","level":"INFO","msg":"Validator event validation successful","module":"main","module":"loki-validator","validated_validators":1,"event":"ValidatorRemoved"}
{"time":"2025-09-30T14:22:28.554964843Z","level":"INFO","msg":"SSV node log validation successful","test":"sanity","indices":["1104199"],"validators":1,"validate-type":"removal"}
{"time":"2025-09-30T14:22:28.555162784Z","level":"INFO","msg":"✅ SUCCESS: Sanity Test Suite Completed Successfully","module":"main","register":{"Contract":true,"E2M":false,"Loki":true},"liquidate":{"Contract":true,"E2M":false,"Loki":true},"reactivate":{"Contract":true,"E2M":false,"Loki":true},"remove":{"Contract":true,"E2M":false,"Loki":true}}
{"time":"2025-09-30T14:22:28.555252831Z","level":"INFO","msg":"⏳ PROGRESS: Waiting for all tests to finish","module":"main"}
{"time":"2025-09-30T14:22:28.555410926Z","level":"INFO","msg":"✅ SUCCESS: All Test Suites Completed Successfully","module":"main","passed_suites":["sanity"],"skipped_suites":["isolated"]}
{"time":"2025-09-30T14:22:28.555462819Z","level":"INFO","msg":"🔑 KEY STEP: Starting resource teardown...","module":"main","usage_id":"593bde78f79ebdca3aa737c49c12e086"}
{"time":"2025-09-30T14:22:28.563697251Z","level":"INFO","msg":"🌐 Orchestrator API Client initialized","module":"main","endpoint":"http://aetheria-orchestrator.aetheria.svc:8080"}
{"time":"2025-09-30T14:22:28.563811473Z","level":"INFO","msg":"🧹 Requesting resource teardown","module":"main","usage_id":"593bde78f79ebdca3aa737c49c12e086"}
{"time":"2025-09-30T14:22:28.589533599Z","level":"INFO","msg":"✅ SUCCESS: Completed resource teardown successfully","module":"main","usage_id":"593bde78f79ebdca3aa737c49c12e086"}

2025-09-30T14:22:31Z The test run finished successfully

Result - 🟢 Success

yoaz-ssvlabs · 2025-10-19T12:09:27Z

/test tests=(sanity,e2m),(bulk,4)

ssv-test-bot · 2025-10-19T12:09:32Z

🟢 Test run for PR #2433

Branch: ci-test-handler-testing2
Commit: 74b7038

Status

2025-10-19T12:09:32Z Starting test run
2025-10-19T12:09:33Z Test run submitted successfully for commit 74b7038
2025-10-19T12:09:34Z container image for commit 74b7038f0265e7f8da4425e9315fe4199c751b80 is build and present in container registry
2025-10-19T12:09:35Z automatically allocated node range cluster=300,301,302,303 for test run on network=hoodi_stage
2025-10-19T12:09:36Z deploying commit 74b7038f0265e7f8da4425e9315fe4199c751b80 to ssv nodes 300,301,302,303
2025-10-19T12:10:37Z started test executor with tests=(sanity,e2m),(bulk,4) and network=hoodi_stage

Click to expand executor logs

═══════════════════════════════════════════════════════════════════════════════════════════════════════
                🧪  WELCOME TO Executor - SSV's Distributed Test Automation Execution Layer  🧪            

                                  ___ __  __ ___  ___  _   _  _____  ___   ___ 
                                 | __|\\ \\/ /| __|/ __|| | | ||_   _|/ _ \\ | _ \\
                                 | _|  >  < | _|| (__ | |_| |  | | | (_) ||   /
                                 |___|/_/\\_\\|___|\\___| \\___/   |_|  \\___/ |_|_\\
                                        

═══════════════════════════════════════════════════════════════════════════════════════════════════════
{"time":"2025-10-19T12:11:16.618632686Z","level":"INFO","msg":"🐳 Docker mode: Using environment variables for config generation"}
{"time":"2025-10-19T12:11:16.632266533Z","level":"INFO","msg":"🌐 Orchestrator API Client initialized","endpoint":"http://aetheria-orchestrator.aetheria.svc:8080"}
{"time":"2025-10-19T12:11:16.632298453Z","level":"INFO","msg":"📝 Requesting configuration generation from orchestrator API","network":"hoodi_stage","clusters":[300,301,302,303],"tests":"(sanity,e2m),(bulk,4)"}
{"time":"2025-10-19T12:11:16.723310361Z","level":"INFO","msg":"✅ Configuration generated successfully via API","resource_usage_id":"d7a46a7b7d9c00fb3e7d469742f1a0ae"}
{"time":"2025-10-19T12:11:16.723829938Z","level":"INFO","msg":"🚀 EXECUTOR Initialized Successfully"}
{"time":"2025-10-19T12:11:16.723860041Z","level":"INFO","msg":"⚙️ Test configuration loaded","network":"hoodi_stage","nodes":[300,301,302,303],"test_type":{"Sanity":{"ShouldRun":true,"E2M":true,"BulkAmount":0},"Bulk":{"ShouldRun":true,"E2M":false,"BulkAmount":4},"Isolated":{"Tests":null}}}
{"time":"2025-10-19T12:11:21.207321126Z","level":"INFO","msg":"Generated operator pubkeys file","module":"scanner","path":"/app/shared/data/operators/operator-pubkeys-hoodi_stage.json"}
{"time":"2025-10-19T12:11:21.207579091Z","level":"INFO","msg":"Registered parser","module":"main","component":"parser-client","parser_type":"consensus"}
{"time":"2025-10-19T12:11:21.207796308Z","level":"INFO","msg":"Registered parser","module":"main","component":"parser-client","parser_type":"flow-emits"}
{"time":"2025-10-19T12:11:21.207935836Z","level":"INFO","msg":"Validation client initialized","module":"main","loki_enabled":true,"e2m_enabled":true}
{"time":"2025-10-19T12:11:21.208003753Z","level":"INFO","msg":"🔑 KEY STEP: Starting Isolated Tests Concurrently if exists...","module":"main"}
{"time":"2025-10-19T12:11:21.208093644Z","level":"INFO","msg":"🔑 KEY STEP: Starting Sanity Test Suite","module":"main"}
{"time":"2025-10-19T12:11:21.208409809Z","level":"INFO","msg":"No isolated tests configured, skipping","module":"main"}
{"time":"2025-10-19T12:11:21.248241525Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":300}
{"time":"2025-10-19T12:11:21.258943046Z","level":"INFO","msg":"Available slots","test":"sanity","node":300,"slots":3000}
{"time":"2025-10-19T12:11:21.259038971Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":301}
{"time":"2025-10-19T12:11:21.265169385Z","level":"INFO","msg":"Available slots","test":"sanity","node":301,"slots":756}
{"time":"2025-10-19T12:11:21.265245438Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":302}
{"time":"2025-10-19T12:11:21.272234902Z","level":"INFO","msg":"Available slots","test":"sanity","node":302,"slots":452}
{"time":"2025-10-19T12:11:21.272311569Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":303}
{"time":"2025-10-19T12:11:21.278624788Z","level":"INFO","msg":"Available slots","test":"sanity","node":303,"slots":1073}
{"time":"2025-10-19T12:12:33.861524307Z","level":"INFO","msg":"Extracted keyshares path","test":"sanity","module":"keys","path":"/app/shared/data/keyshares/0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754.json"}
{"time":"2025-10-19T12:12:33.861679488Z","level":"INFO","msg":"Key shares file generated successfully","test":"sanity","keyshare_path":"/app/shared/data/keyshares/0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754.json"}
{"time":"2025-10-19T12:12:35.760861555Z","level":"INFO","msg":"Allowance fetched successfully","test":"sanity","module":"contract","account":"0x91e32efb8139cd88cae0df30d2bf471294c6ed27","allowance":"99303309999200282400000000"}
{"time":"2025-10-19T12:12:35.760951481Z","level":"INFO","msg":"Threshold value","test":"sanity","module":"contract","threshold":"9000000000000000000000"}
{"time":"2025-10-19T12:12:35.76097048Z","level":"INFO","msg":"Sufficient allowance","test":"sanity","module":"contract","allowance margin":"6561000000000000000000000000000000000000000000000000000000000000000000000000000000000000"}
{"time":"2025-10-19T12:12:35.766580753Z","level":"INFO","msg":"Allowance fetched successfully","test":"sanity","module":"contract","account":"0x91e32efb8139cd88cae0df30d2bf471294c6ed27","allowance":"99303309999200282400000000"}
{"time":"2025-10-19T12:12:35.76665957Z","level":"INFO","msg":"Allowance","test":"sanity","module":"contract","allowance":"99303309999200282400000000"}
{"time":"2025-10-19T12:12:35.887281726Z","level":"INFO","msg":"Registering validator","test":"sanity","module":"contract","tx_hash":"0xa0608db97ebe10d0891afdc2a036121cd9ca206a83e7487517ce7309fe43a9d6"}
{"time":"2025-10-19T12:12:35.892483143Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0xa0608db97ebe10d0891afdc2a036121cd9ca206a83e7487517ce7309fe43a9d6"}
{"time":"2025-10-19T12:12:45.900791427Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":2,"sleep":20,"tx_hash":"0xa0608db97ebe10d0891afdc2a036121cd9ca206a83e7487517ce7309fe43a9d6"}
{"time":"2025-10-19T12:13:05.907192016Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0xa0608db97ebe10d0891afdc2a036121cd9ca206a83e7487517ce7309fe43a9d6"}
{"time":"2025-10-19T12:13:05.907287837Z","level":"INFO","msg":"✅ SUCCESS: Contract Register Successful","test":"sanity","txHash":"0xa0608db97ebe10d0891afdc2a036121cd9ca206a83e7487517ce7309fe43a9d6","block_hash":"0xd734b24ba78d88afacc5cda240b52250939180343ee9281c9fe6785ef218973f"}
{"time":"2025-10-19T12:13:05.925519838Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"sanity","module":"helpers","pubkey":"0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754","timeToSleep":10,"attempt":1}
{"time":"2025-10-19T12:13:15.937936885Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"sanity","module":"helpers","pubkey":"0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754","timeToSleep":20,"attempt":2}
{"time":"2025-10-19T12:13:35.977965265Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"sanity","module":"helpers","pubkey":"0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754","timeToSleep":40,"attempt":3}
{"time":"2025-10-19T12:14:15.994962589Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"sanity","module":"helpers","pubkey":"0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754","timeToSleep":80,"attempt":4}
{"time":"2025-10-19T12:15:36.116789899Z","level":"INFO","msg":"Successfully retrieved validator index from exporter","test":"sanity","module":"helpers","pubkey":"0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754"}
{"time":"2025-10-19T12:15:36.205216167Z","level":"INFO","msg":"Registration epoch","module":"main","module":"e2m-validator","epoch":48600,"indices":"1102420"}
{"time":"2025-10-19T12:15:36.366798453Z","level":"INFO","msg":"Validator event validation successful","module":"main","module":"loki-validator","validated_validators":1,"event":"ValidatorAdded"}
{"time":"2025-10-19T12:15:36.366885282Z","level":"INFO","msg":"SSV node log validation successful","test":"sanity","indices":["1102420"],"validators":1,"validate-type":"registration"}
{"time":"2025-10-19T12:15:51.248083847Z","level":"INFO","msg":"Validator did not attest","module":"main","module":"e2m-validator","epoch":48599,"indices":["1102420"]}
{"time":"2025-10-19T12:16:36.408056094Z","level":"INFO","msg":"Validator did not attest","module":"main","module":"e2m-validator","epoch":48600,"indices":["1102420"]}
{"time":"2025-10-19T12:22:52.460890417Z","level":"INFO","msg":"Validator did not attest","module":"main","module":"e2m-validator","epoch":48601,"indices":["1102420"]}
{"time":"2025-10-19T12:23:07.49523867Z","level":"INFO","msg":"E2M registration validation successful - validators are attesting","module":"main","module":"e2m-validator","validation_type":"registration","indices":["1102420"],"registered_epoch":48600,"validated_epoch":48601,"epochs_waited":1}
{"time":"2025-10-19T12:23:07.49532776Z","level":"INFO","msg":"E2M validation successful","test":"sanity","indices":["1102420"],"validate-type":"registration"}
{"time":"2025-10-19T12:23:07.49553422Z","level":"INFO","msg":"Starting consensus validation with epoch iteration logic","module":"main","module":"loki-validator","validator_index":"1102420","registered_epoch":48600,"max_epoch_offset":4}
{"time":"2025-10-19T12:23:07.495644797Z","level":"INFO","msg":"Checking epoch for duties","module":"main","module":"loki-validator","epoch_offset":0,"current_epoch":"48600","validator_index":"1102420"}
{"time":"2025-10-19T12:23:07.505965307Z","level":"WARN","msg":"No attesters found in duties response","module":"main","module":"loki-validator","epoch_offset":0,"epoch":"48600"}
{"time":"2025-10-19T12:23:07.506022879Z","level":"INFO","msg":"Checking epoch for duties","module":"main","module":"loki-validator","epoch_offset":1,"current_epoch":"48601","validator_index":"1102420"}
{"time":"2025-10-19T12:23:07.513698076Z","level":"INFO","msg":"Successfully found duties, building duty ID for consensus validation","module":"main","module":"loki-validator","duty_id":"COMMITTEE-300_301_302_303-e48601-s1555262","final_epoch":"48601","slot":"1555262"}
{"time":"2025-10-19T12:23:07.646235047Z","level":"INFO","msg":"Consensus leader validation passed","module":"main","module":"loki-validator","slot":1555262,"leader":302}
{"time":"2025-10-19T12:23:07.646431197Z","level":"INFO","msg":"Consensus timing validation comparison","module":"main","module":"loki-validator","actual_time_diff":46914000,"expected_consensus_time":41000000,"lower_bound":28700000,"upper_bound":53300000,"margin_percent":30}
{"time":"2025-10-19T12:23:07.646454989Z","level":"INFO","msg":"Consensus timing validation passed - within acceptable range","module":"main","module":"loki-validator"}
{"time":"2025-10-19T12:23:07.646482995Z","level":"INFO","msg":"Consensus time validation completed successfully","module":"main","module":"loki-validator","final_epoch":"48601"}
{"time":"2025-10-19T12:23:07.646536974Z","level":"INFO","msg":"Consensus timing validation successful","test":"sanity","indices":["1102420"],"epoch":48600}
{"time":"2025-10-19T12:23:09.786969165Z","level":"INFO","msg":"Liquidating cluster","test":"sanity","module":"contract","tx_hash":"0xc11b5f74f9a52363a68c55b8c555d0722c34d426e2e52f9457e2a4f561f394ef"}
{"time":"2025-10-19T12:23:09.791516372Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0xc11b5f74f9a52363a68c55b8c555d0722c34d426e2e52f9457e2a4f561f394ef"}
{"time":"2025-10-19T12:23:19.798467958Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0xc11b5f74f9a52363a68c55b8c555d0722c34d426e2e52f9457e2a4f561f394ef"}
{"time":"2025-10-19T12:23:19.798594449Z","level":"INFO","msg":"✅ SUCCESS: Cluster Liquidation Successful","test":"sanity","txHash":"0xc11b5f74f9a52363a68c55b8c555d0722c34d426e2e52f9457e2a4f561f394ef","block_hash":"0x6e781f4c50f09514ea645cb91f0c759b821601d161b74439e86285d4982d6ec9"}
{"time":"2025-10-19T12:23:19.798759376Z","level":"INFO","msg":"Querying Loki for transaction hash validation","module":"main","module":"loki-validator","query":"{container=~\\"ssv-node-300|ssv-node-301|ssv-node-302|ssv-node-303\\"} |= \\"0xc11b5f74f9a52363a68c55b8c555d0722c34d426e2e52f9457e2a4f561f394ef\\"","tx_hash":"0xc11b5f74f9a52363a68c55b8c555d0722c34d426e2e52f9457e2a4f561f394ef"}
{"time":"2025-10-19T12:23:19.807696714Z","level":"INFO","msg":"waiting for block to be found","module":"main","module":"e2m-validator","module":"e2m"}
{"time":"2025-10-19T12:23:19.935181367Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":1,"max_retries":5,"delay_seconds":30,"start_unix":1760865799,"end_unix":1760876599}
{"time":"2025-10-19T12:23:39.828158887Z","level":"INFO","msg":"Liquidation epoch","module":"main","module":"e2m-validator","epoch":48602,"indices":["1102420"]}
{"time":"2025-10-19T12:23:50.017915644Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":2,"max_retries":5,"delay_seconds":60,"start_unix":1760865799,"end_unix":1760876629}
{"time":"2025-10-19T12:24:50.019368213Z","level":"ERROR","msg":"request failed","module":"HTTPRequester","error":"Get \\"https://loki.ops.ssvlabsinternal.com/loki/api/v1/query_range?query=%7Bcontainer%3D~%22ssv-node-300%7Cssv-node-301%7Cssv-node-302%7Cssv-node-303%22%7D+%7C%3D+%220xc11b5f74f9a52363a68c55b8c555d0722c34d426e2e52f9457e2a4f561f394ef%22+%7C+json+%7C+line_format+%22%7B%7B.container%7D%7D%3A%7B%7B.msg%7D%7D%22&start=1760865799&end=1760876690&limit=1000\\": unexpected EOF","request attempt #":0}
{"time":"2025-10-19T12:24:50.151589357Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":3,"max_retries":5,"delay_seconds":90,"start_unix":1760865799,"end_unix":1760876690}
{"time":"2025-10-19T12:26:20.288949532Z","level":"INFO","msg":"Event validation successful","module":"main","module":"loki-validator","valid_entries":4,"event":"ClusterLiquidated"}
{"time":"2025-10-19T12:26:20.289088682Z","level":"INFO","msg":"SSV node log validation successful","test":"sanity","indices":["1102420"],"validators":1,"validate-type":"liquidation"}
{"time":"2025-10-19T12:29:25.85987369Z","level":"INFO","msg":"E2M liquidation validation successful - validators are liquidated","module":"main","module":"e2m-validator","validation_type":"liquidation","indices":["1102420"],"liquidated_epoch":48602,"validated_epoch":48602,"epochs_waited":0,"assigned":1,"missed":1}
{"time":"2025-10-19T12:29:25.859999567Z","level":"INFO","msg":"E2M validation successful","test":"sanity","indices":["1102420"],"validate-type":"liquidation"}
{"time":"2025-10-19T12:29:27.911997373Z","level":"INFO","msg":"Allowance fetched successfully","test":"sanity","module":"contract","account":"0x91e32efb8139cd88cae0df30d2bf471294c6ed27","allowance":"99303309999200282400000000"}
{"time":"2025-10-19T12:29:27.912127786Z","level":"INFO","msg":"Threshold value","test":"sanity","module":"contract","threshold":"9000000000000000000000"}
{"time":"2025-10-19T12:29:27.912185794Z","level":"INFO","msg":"Sufficient allowance","test":"sanity","module":"contract","allowance margin":"6561000000000000000000000000000000000000000000000000000000000000000000000000000000000000"}
{"time":"2025-10-19T12:29:27.989240673Z","level":"INFO","msg":"Reactivating cluster","test":"sanity","module":"contract","tx_hash":"0x6ad72f0c9354ee44c55c975d6a6185324d36738ddc0bc962f9ed51bad5c2c5d2"}
{"time":"2025-10-19T12:29:27.994702334Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0x6ad72f0c9354ee44c55c975d6a6185324d36738ddc0bc962f9ed51bad5c2c5d2"}
{"time":"2025-10-19T12:29:38.007033749Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":2,"sleep":20,"tx_hash":"0x6ad72f0c9354ee44c55c975d6a6185324d36738ddc0bc962f9ed51bad5c2c5d2"}
{"time":"2025-10-19T12:29:58.034002522Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0x6ad72f0c9354ee44c55c975d6a6185324d36738ddc0bc962f9ed51bad5c2c5d2"}
{"time":"2025-10-19T12:29:58.034106638Z","level":"INFO","msg":"✅ SUCCESS: Cluster Reactivation Successful","test":"sanity","txHash":"0x6ad72f0c9354ee44c55c975d6a6185324d36738ddc0bc962f9ed51bad5c2c5d2","block_hash":"0x0266958e7bd3185b51d2d37e2823a6cb81d79d7d818badc4b3e875fb33e8d7d4"}
{"time":"2025-10-19T12:29:58.034192873Z","level":"INFO","msg":"Validating reactivation - checking that validators resume attesting","module":"main","module":"e2m-validator","indices":["1102420"],"block_hash":"0x0266958e7bd3185b51d2d37e2823a6cb81d79d7d818badc4b3e875fb33e8d7d4"}
{"time":"2025-10-19T12:29:58.034301074Z","level":"INFO","msg":"Querying Loki for transaction hash validation","module":"main","module":"loki-validator","query":"{container=~\\"ssv-node-300|ssv-node-301|ssv-node-302|ssv-node-303\\"} |= \\"0x6ad72f0c9354ee44c55c975d6a6185324d36738ddc0bc962f9ed51bad5c2c5d2\\"","tx_hash":"0x6ad72f0c9354ee44c55c975d6a6185324d36738ddc0bc962f9ed51bad5c2c5d2"}
{"time":"2025-10-19T12:29:58.04302665Z","level":"INFO","msg":"Registration epoch","module":"main","module":"e2m-validator","epoch":48603,"indices":"1102420"}
{"time":"2025-10-19T12:29:58.257116644Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":1,"max_retries":5,"delay_seconds":30,"start_unix":1760866198,"end_unix":1760876998}
{"time":"2025-10-19T12:30:13.085498021Z","level":"INFO","msg":"Validator did not attest","module":"main","module":"e2m-validator","epoch":48602,"indices":["1102420"]}
{"time":"2025-10-19T12:30:28.361703607Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":2,"max_retries":5,"delay_seconds":60,"start_unix":1760866198,"end_unix":1760877028}
{"time":"2025-10-19T12:31:28.492271693Z","level":"INFO","msg":"Event validation successful","module":"main","module":"loki-validator","valid_entries":4,"event":"ClusterReactivated"}
{"time":"2025-10-19T12:31:28.492365737Z","level":"INFO","msg":"SSV node log validation successful","test":"sanity","indices":["1102420"],"validators":1,"validate-type":"reactivation"}
{"time":"2025-10-19T12:35:44.088931568Z","level":"INFO","msg":"Validator did not attest","module":"main","module":"e2m-validator","epoch":48603,"indices":["1102420"]}
{"time":"2025-10-19T12:42:15.475949272Z","level":"INFO","msg":"E2M registration validation successful - validators are attesting","module":"main","module":"e2m-validator","validation_type":"registration","indices":["1102420"],"registered_epoch":48603,"validated_epoch":48604,"epochs_waited":1}
{"time":"2025-10-19T12:42:15.476151995Z","level":"INFO","msg":"E2M validation successful","test":"sanity","indices":["1102420"],"validate-type":"reactivation"}
{"time":"2025-10-19T12:42:15.476228014Z","level":"INFO","msg":"Starting consensus validation with epoch iteration logic","module":"main","module":"loki-validator","validator_index":"1102420","registered_epoch":48603,"max_epoch_offset":4}
{"time":"2025-10-19T12:42:15.476344228Z","level":"INFO","msg":"Checking epoch for duties","module":"main","module":"loki-validator","epoch_offset":0,"current_epoch":"48603","validator_index":"1102420"}
{"time":"2025-10-19T12:42:15.486540609Z","level":"WARN","msg":"No attesters found in duties response","module":"main","module":"loki-validator","epoch_offset":0,"epoch":"48603"}
{"time":"2025-10-19T12:42:15.486604995Z","level":"INFO","msg":"Checking epoch for duties","module":"main","module":"loki-validator","epoch_offset":1,"current_epoch":"48604","validator_index":"1102420"}
{"time":"2025-10-19T12:42:15.497836494Z","level":"INFO","msg":"Successfully found duties, building duty ID for consensus validation","module":"main","module":"loki-validator","duty_id":"COMMITTEE-300_301_302_303-e48604-s1555347","final_epoch":"48604","slot":"1555347"}
{"time":"2025-10-19T12:42:15.598851798Z","level":"INFO","msg":"Consensus leader validation passed","module":"main","module":"loki-validator","slot":1555347,"leader":303}
{"time":"2025-10-19T12:42:15.598994693Z","level":"INFO","msg":"Consensus timing validation comparison","module":"main","module":"loki-validator","actual_time_diff":32305000,"expected_consensus_time":9000000,"lower_bound":6300000,"upper_bound":11700000,"margin_percent":30}
{"time":"2025-10-19T12:42:15.599104997Z","level":"ERROR","msg":"Consensus timing validation failed","test":"sanity","indices":["1102420"],"epoch":48603,"error":"consensus timing outside acceptable range: actual=32.305ms, expected=9ms (±30%), bounds=[6.3ms, 11.7ms]"}
{"time":"2025-10-19T12:42:17.600048045Z","level":"INFO","msg":"Allowance fetched successfully","test":"sanity","module":"contract","account":"0x91e32efb8139cd88cae0df30d2bf471294c6ed27","allowance":"99294309999200282400000000"}
{"time":"2025-10-19T12:42:17.600206528Z","level":"INFO","msg":"Threshold value","test":"sanity","module":"contract","threshold":"9000000000000000000000"}
{"time":"2025-10-19T12:42:17.600261574Z","level":"INFO","msg":"Sufficient allowance","test":"sanity","module":"contract","allowance margin":"6561000000000000000000000000000000000000000000000000000000000000000000000000000000000000"}
{"time":"2025-10-19T12:42:17.669258786Z","level":"INFO","msg":"Removing validator","test":"sanity","module":"contract","tx_hash":"0xd202ed1c607d0cbb115173f2ef8456f658ee76387463baf7a2d0006fe1734e96"}
{"time":"2025-10-19T12:42:17.673504025Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0xd202ed1c607d0cbb115173f2ef8456f658ee76387463baf7a2d0006fe1734e96"}
{"time":"2025-10-19T12:42:27.688098892Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0xd202ed1c607d0cbb115173f2ef8456f658ee76387463baf7a2d0006fe1734e96"}
{"time":"2025-10-19T12:42:27.688173587Z","level":"INFO","msg":"✅ SUCCESS: Contract Removal Successful","test":"sanity","blockHash":"0xc1e59bb6f6226eb830f4f1655aa4b941d4cbf2fdacfa3c15906ed1f8ce7ede3b"}
{"time":"2025-10-19T12:42:27.697843688Z","level":"INFO","msg":"waiting for block to be found","module":"main","module":"e2m-validator","module":"e2m"}
{"time":"2025-10-19T12:42:27.837941701Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":1,"max_retries":5,"delay_seconds":30,"start_unix":1760866947,"end_unix":1760877747}
{"time":"2025-10-19T12:42:47.707535084Z","level":"INFO","msg":"Removal epoch","module":"main","module":"e2m-validator","epoch":48605,"indices":"1102420"}
{"time":"2025-10-19T12:42:57.899542098Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":2,"max_retries":5,"delay_seconds":60,"start_unix":1760866947,"end_unix":1760877777}
{"time":"2025-10-19T12:43:57.90059713Z","level":"ERROR","msg":"request failed","module":"HTTPRequester","error":"Get \\"https://loki.ops.ssvlabsinternal.com/loki/api/v1/query_range?query=%7Bcontainer%3D~%22ssv-node-300%7Cssv-node-301%7Cssv-node-302%7Cssv-node-303%22%7D+%7C%3D+%220xd202ed1c607d0cbb115173f2ef8456f658ee76387463baf7a2d0006fe1734e96%22&start=1760866947&end=1760877837&limit=1000\\": unexpected EOF","request attempt #":0}
{"time":"2025-10-19T12:43:57.996726251Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":3,"max_retries":5,"delay_seconds":90,"start_unix":1760866947,"end_unix":1760877837}
{"time":"2025-10-19T12:45:28.096665526Z","level":"INFO","msg":"Validator event validation successful","module":"main","module":"loki-validator","validated_validators":1,"event":"ValidatorRemoved"}
{"time":"2025-10-19T12:45:28.09678406Z","level":"INFO","msg":"SSV node log validation successful","test":"sanity","indices":["1102420"],"validators":1,"validate-type":"removal"}
{"time":"2025-10-19T12:48:33.93008148Z","level":"INFO","msg":"Validator was not removed","module":"main","module":"e2m-validator","epoch":48605,"indices":["1102420"]}
{"time":"2025-10-19T12:55:05.237259624Z","level":"INFO","msg":"E2M removal validation successful - validators stopped attesting","module":"main","module":"e2m-validator","validation_type":"removal","indices":["1102420"],"removed_epoch":48605,"validated_epoch":48606,"epochs_waited":1}
{"time":"2025-10-19T12:55:05.237348193Z","level":"INFO","msg":"E2M validation successful","test":"sanity","indices":["1102420"],"validate-type":"removal"}
{"time":"2025-10-19T12:55:05.237485816Z","level":"INFO","msg":"✅ SUCCESS: Sanity Test Suite Completed Successfully","module":"main","register":{"Contract":true,"E2M":true,"Loki":true},"liquidate":{"Contract":true,"E2M":true,"Loki":true},"reactivate":{"Contract":true,"E2M":true,"Loki":false},"remove":{"Contract":true,"E2M":true,"Loki":true}}
{"time":"2025-10-19T12:55:05.23750047Z","level":"INFO","msg":"🔑 KEY STEP: Starting Bulk Test Suite","module":"main"}
{"time":"2025-10-19T12:55:05.254283636Z","level":"INFO","msg":"Getting available slots","test":"bulk","node":300}
{"time":"2025-10-19T12:55:05.262664931Z","level":"INFO","msg":"Available slots","test":"bulk","node":300,"slots":3000}
{"time":"2025-10-19T12:55:05.262768401Z","level":"INFO","msg":"Getting available slots","test":"bulk","node":301}
{"time":"2025-10-19T12:55:05.26957096Z","level":"INFO","msg":"Available slots","test":"bulk","node":301,"slots":756}
{"time":"2025-10-19T12:55:05.269612305Z","level":"INFO","msg":"Getting available slots","test":"bulk","node":302}
{"time":"2025-10-19T12:55:05.276490652Z","level":"INFO","msg":"Available slots","test":"bulk","node":302,"slots":452}
{"time":"2025-10-19T12:55:05.276576193Z","level":"INFO","msg":"Getting available slots","test":"bulk","node":303}
{"time":"2025-10-19T12:55:05.283291579Z","level":"INFO","msg":"Available slots","test":"bulk","node":303,"slots":1073}
{"time":"2025-10-19T12:56:00.056161551Z","level":"INFO","msg":"Extracted keyshares path for bulk keyshare","test":"bulk","module":"keys","path":"/app/shared/data/keyshares/keyshares-1760878559.json"}
{"time":"2025-10-19T12:56:03.866637235Z","level":"INFO","msg":"Allowance fetched successfully","test":"bulk","module":"contract","account":"0x91e32efb8139cd88cae0df30d2bf471294c6ed27","allowance":"99294309999200282400000000"}
{"time":"2025-10-19T12:56:03.867716266Z","level":"INFO","msg":"Threshold value","test":"bulk","module":"contract","threshold":"9000000000000000000000"}
{"time":"2025-10-19T12:56:03.867786618Z","level":"INFO","msg":"Sufficient allowance","test":"bulk","module":"contract","allowance margin":"6561000000000000000000000000000000000000000000000000000000000000000000000000000000000000"}
{"time":"2025-10-19T12:56:04.046709156Z","level":"INFO","msg":"Bulk Registering validators","test":"bulk","module":"contract","tx_hash":"0x3eee1b0cfd91bfbe3b4a9a759bdb577d333778626dabb287175709424c66c737"}
{"time":"2025-10-19T12:56:04.046764049Z","level":"INFO","msg":"Bulk registration transaction sent","test":"bulk","txHash":"0x3eee1b0cfd91bfbe3b4a9a759bdb577d333778626dabb287175709424c66c737"}
{"time":"2025-10-19T12:56:04.055575268Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"bulk","module":"contract","attempt":1,"sleep":10,"tx_hash":"0x3eee1b0cfd91bfbe3b4a9a759bdb577d333778626dabb287175709424c66c737"}
{"time":"2025-10-19T12:56:14.062308037Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"bulk","module":"contract","attempt":2,"sleep":20,"tx_hash":"0x3eee1b0cfd91bfbe3b4a9a759bdb577d333778626dabb287175709424c66c737"}
{"time":"2025-10-19T12:56:34.083985632Z","level":"INFO","msg":"Transaction receipt received","test":"bulk","module":"contract","tx_hash":"0x3eee1b0cfd91bfbe3b4a9a759bdb577d333778626dabb287175709424c66c737"}
{"time":"2025-10-19T12:56:34.084134954Z","level":"INFO","msg":"✅ SUCCESS: Bulk Registration Successful","test":"bulk","blockHash":"0xc32835aa658725deddca5bd956cfc2cf2d94a473cedc62050923c44182cfb006","Validators Amount":4}
{"time":"2025-10-19T12:56:34.099403837Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"bulk","module":"helpers","pubkey":"0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754","timeToSleep":10,"attempt":1}
{"time":"2025-10-19T12:56:44.115229078Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"bulk","module":"helpers","pubkey":"0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754","timeToSleep":20,"attempt":2}
{"time":"2025-10-19T12:57:04.139555512Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"bulk","module":"helpers","pubkey":"0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754","timeToSleep":40,"attempt":3}
{"time":"2025-10-19T12:57:44.164930209Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"bulk","module":"helpers","pubkey":"0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754","timeToSleep":80,"attempt":4}
{"time":"2025-10-19T12:59:04.187992783Z","level":"INFO","msg":"Successfully retrieved validator index from exporter","test":"bulk","module":"helpers","pubkey":"0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754"}
{"time":"2025-10-19T12:59:04.197943876Z","level":"INFO","msg":"Successfully retrieved validator index from exporter","test":"bulk","module":"helpers","pubkey":"0x86d6aa42d120173270727af286965c371f919f423558997f914b30b20bba45b041561650b531d386ed85f65828e0bde1"}
{"time":"2025-10-19T12:59:04.206624084Z","level":"INFO","msg":"Successfully retrieved validator index from exporter","test":"bulk","module":"helpers","pubkey":"0x88e97e72ea3a862b12b44028e65f2af490b8cc19d4b6f1915f1f80325cff9f02d8f5c6fd79f127c5fde7a90fcaa43b8d"}
{"time":"2025-10-19T12:59:04.214926616Z","level":"INFO","msg":"Successfully retrieved validator index from exporter","test":"bulk","module":"helpers","pubkey":"0xa88938ed513411df0322826f6b017a2d1fb610c633e33ccad9dc1e46b274ad32a489f89e7b1b772ab518ba0cb089cee2"}
{"time":"2025-10-19T12:59:04.231893782Z","level":"INFO","msg":"Registration epoch","module":"main","module":"e2m-validator","epoch":48607,"indices":"1102420,1092007,1092460,1092461"}
{"time":"2025-10-19T12:59:04.415752742Z","level":"INFO","msg":"Validator event validation successful","module":"main","module":"loki-validator","validated_validators":4,"event":"ValidatorAdded"}
{"time":"2025-10-19T12:59:04.415880684Z","level":"INFO","msg":"SSV node log validation successful","test":"bulk","indices":["1102420","1092007","1092460","1092461"],"validators":4,"validate-type":"registration"}
{"time":"2025-10-19T12:59:19.278859683Z","level":"INFO","msg":"Validator did not attest","module":"main","module":"e2m-validator","epoch":48606,"indices":["1102420","1092007","1092460","1092461"]}
{"time":"2025-10-19T13:01:34.7959622Z","level":"INFO","msg":"Validator did not attest","module":"main","module":"e2m-validator","epoch":48607,"indices":["1102420","1092007","1092460","1092461"]}
{"time":"2025-10-19T13:07:51.286190606Z","level":"INFO","msg":"E2M registration validation successful - validators are attesting","module":"main","module":"e2m-validator","validation_type":"registration","indices":["1102420","1092007","1092460","1092461"],"registered_epoch":48607,"validated_epoch":48608,"epochs_waited":1}
{"time":"2025-10-19T13:07:51.286380467Z","level":"INFO","msg":"E2M validation successful","test":"bulk","indices":["1102420","1092007","1092460","1092461"],"validate-type":"registration"}
{"time":"2025-10-19T13:07:51.286409085Z","level":"INFO","msg":"Starting consensus validation with epoch iteration logic","module":"main","module":"loki-validator","validator_index":"1102420","registered_epoch":48607,"max_epoch_offset":4}
{"time":"2025-10-19T13:07:51.286496274Z","level":"INFO","msg":"Checking epoch for duties","module":"main","module":"loki-validator","epoch_offset":0,"current_epoch":"48607","validator_index":"1102420"}
{"time":"2025-10-19T13:07:51.29678308Z","level":"WARN","msg":"No attesters found in duties response","module":"main","module":"loki-validator","epoch_offset":0,"epoch":"48607"}
{"time":"2025-10-19T13:07:51.296922249Z","level":"INFO","msg":"Checking epoch for duties","module":"main","module":"loki-validator","epoch_offset":1,"current_epoch":"48608","validator_index":"1102420"}
{"time":"2025-10-19T13:07:51.320837403Z","level":"INFO","msg":"Successfully found duties, building duty ID for consensus validation","module":"main","module":"loki-validator","duty_id":"COMMITTEE-300_301_302_303-e48608-s1555485","final_epoch":"48608","slot":"1555485"}
{"time":"2025-10-19T13:07:51.479034461Z","level":"INFO","msg":"Consensus leader validation passed","module":"main","module":"loki-validator","slot":1555485,"leader":301}
{"time":"2025-10-19T13:07:51.479128755Z","level":"INFO","msg":"Consensus timing validation comparison","module":"main","module":"loki-validator","actual_time_diff":36747000,"expected_consensus_time":32000000,"lower_bound":22400000,"upper_bound":41600000,"margin_percent":30}
{"time":"2025-10-19T13:07:51.47917056Z","level":"INFO","msg":"Consensus timing validation passed - within acceptable range","module":"main","module":"loki-validator"}
{"time":"2025-10-19T13:07:51.479189945Z","level":"INFO","msg":"Consensus time validation completed successfully","module":"main","module":"loki-validator","final_epoch":"48608"}
{"time":"2025-10-19T13:07:51.479225922Z","level":"INFO","msg":"Consensus timing validation successful","test":"bulk","indices":["1102420","1092007","1092460","1092461"],"epoch":48607}
{"time":"2025-10-19T13:07:53.977029979Z","level":"INFO","msg":"Bulk Removing validators","test":"bulk","module":"contract","tx_hash":"0x73af6aa8dfbd68a17d840f39735e14de04efc5271b1b2932a6de12ce3d32a7b1"}
{"time":"2025-10-19T13:07:53.977160394Z","level":"INFO","msg":"Bulk removal transaction sent","test":"bulk","txHash":"0x73af6aa8dfbd68a17d840f39735e14de04efc5271b1b2932a6de12ce3d32a7b1"}
{"time":"2025-10-19T13:07:53.981621559Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"bulk","module":"contract","attempt":1,"sleep":10,"tx_hash":"0x73af6aa8dfbd68a17d840f39735e14de04efc5271b1b2932a6de12ce3d32a7b1"}
{"time":"2025-10-19T13:08:03.993828324Z","level":"INFO","msg":"Transaction receipt received","test":"bulk","module":"contract","tx_hash":"0x73af6aa8dfbd68a17d840f39735e14de04efc5271b1b2932a6de12ce3d32a7b1"}
{"time":"2025-10-19T13:08:03.994014219Z","level":"INFO","msg":"✅ SUCCESS: Bulk Removal Successful","test":"bulk","blockHash":"0xecb51e1161e858a307f96590bf310e5711febb1ab5bad564fde412c95d0eb439","Validators Amount":4}
{"time":"2025-10-19T13:08:04.002591132Z","level":"INFO","msg":"waiting for block to be found","module":"main","module":"e2m-validator","module":"e2m"}
{"time":"2025-10-19T13:08:04.14288224Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":1,"max_retries":5,"delay_seconds":30,"start_unix":1760868483,"end_unix":1760879283}
{"time":"2025-10-19T13:08:24.118372848Z","level":"INFO","msg":"Removal epoch","module":"main","module":"e2m-validator","epoch":48609,"indices":"1102420,1092007,1092460,1092461"}
{"time":"2025-10-19T13:08:34.21470909Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":2,"max_retries":5,"delay_seconds":60,"start_unix":1760868483,"end_unix":1760879314}
{"time":"2025-10-19T13:09:34.217053565Z","level":"ERROR","msg":"request failed","module":"HTTPRequester","error":"Get \\"https://loki.ops.ssvlabsinternal.com/loki/api/v1/query_range?query=%7Bcontainer%3D~%22ssv-node-300%7Cssv-node-301%7Cssv-node-302%7Cssv-node-303%22%7D+%7C%3D+%220x73af6aa8dfbd68a17d840f39735e14de04efc5271b1b2932a6de12ce3d32a7b1%22&start=1760868483&end=1760879374&limit=1000\\": unexpected EOF","request attempt #":0}
{"time":"2025-10-19T13:09:34.337350223Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":3,"max_retries":5,"delay_seconds":90,"start_unix":1760868483,"end_unix":1760879374}
{"time":"2025-10-19T13:11:04.447288988Z","level":"INFO","msg":"Validator event validation successful","module":"main","module":"loki-validator","validated_validators":4,"event":"ValidatorRemoved"}
{"time":"2025-10-19T13:11:04.447373883Z","level":"INFO","msg":"SSV node log validation successful","test":"bulk","indices":["1102420","1092007","1092460","1092461"],"validators":4,"validate-type":"removal"}
{"time":"2025-10-19T13:14:10.442667303Z","level":"INFO","msg":"Validator was not removed","module":"main","module":"e2m-validator","epoch":48609,"indices":["1102420","1092007","1092460","1092461"]}
{"time":"2025-10-19T13:20:41.949895858Z","level":"INFO","msg":"E2M removal validation successful - validators stopped attesting","module":"main","module":"e2m-validator","validation_type":"removal","indices":["1102420","1092007","1092460","1092461"],"removed_epoch":48609,"validated_epoch":48610,"epochs_waited":1}
{"time":"2025-10-19T13:20:41.950029527Z","level":"INFO","msg":"E2M validation successful","test":"bulk","indices":["1102420","1092007","1092460","1092461"],"validate-type":"removal"}
{"time":"2025-10-19T13:20:41.950208267Z","level":"INFO","msg":"✅ SUCCESS: Bulk Test Suite Completed Successfully","module":"main","register":{"Contract":true,"E2M":true,"Loki":true},"remove":{"Contract":true,"E2M":true,"Loki":true}}
{"time":"2025-10-19T13:20:41.95026666Z","level":"INFO","msg":"⏳ PROGRESS: Waiting for all tests to finish","module":"main"}
{"time":"2025-10-19T13:20:41.950332335Z","level":"INFO","msg":"✅ SUCCESS: All Test Suites Completed Successfully","module":"main","passed_suites":["sanity","bulk"],"skipped_suites":["isolated"]}
{"time":"2025-10-19T13:20:41.950444992Z","level":"INFO","msg":"🔑 KEY STEP: Starting resource teardown...","module":"main","usage_id":"d7a46a7b7d9c00fb3e7d469742f1a0ae"}
{"time":"2025-10-19T13:20:41.96018522Z","level":"INFO","msg":"🌐 Orchestrator API Client initialized","module":"main","endpoint":"http://aetheria-orchestrator.aetheria.svc:8080"}
{"time":"2025-10-19T13:20:41.960353828Z","level":"INFO","msg":"🧹 Requesting resource teardown","module":"main","usage_id":"d7a46a7b7d9c00fb3e7d469742f1a0ae"}
{"time":"2025-10-19T13:20:42.01364482Z","level":"INFO","msg":"✅ SUCCESS: Completed resource teardown successfully","module":"main","usage_id":"d7a46a7b7d9c00fb3e7d469742f1a0ae"}

2025-10-19T13:20:45Z The test run finished successfully

Result - 🟢 Success

yoaz-ssvlabs · 2025-10-20T07:46:23Z

/test tests=(sanity,e2m),(bulk,7)

ssv-test-bot · 2025-10-20T07:46:26Z

🟢 Test run for PR #2433

Branch: ci-test-handler-testing2
Commit: 74b7038

Status

2025-10-20T07:46:26Z Starting test run
2025-10-20T07:46:27Z Test run submitted successfully for commit 74b7038
2025-10-20T07:46:28Z container image for commit 74b7038f0265e7f8da4425e9315fe4199c751b80 is build and present in container registry
2025-10-20T07:46:29Z automatically allocated node range cluster=300,301,302,303 for test run on network=hoodi_stage
2025-10-20T07:46:31Z deploying commit 74b7038f0265e7f8da4425e9315fe4199c751b80 to ssv nodes 300,301,302,303
2025-10-20T07:47:45Z started test executor with tests=(sanity,e2m),(bulk,7) and network=hoodi_stage

Click to expand executor logs

═══════════════════════════════════════════════════════════════════════════════════════════════════════
                🧪  WELCOME TO Executor - SSV's Distributed Test Automation Execution Layer  🧪            

                                  ___ __  __ ___  ___  _   _  _____  ___   ___ 
                                 | __|\\ \\/ /| __|/ __|| | | ||_   _|/ _ \\ | _ \\
                                 | _|  >  < | _|| (__ | |_| |  | | | (_) ||   /
                                 |___|/_/\\_\\|___|\\___| \\___/   |_|  \\___/ |_|_\\
                                        

═══════════════════════════════════════════════════════════════════════════════════════════════════════
{"time":"2025-10-20T07:47:47.462942876Z","level":"INFO","msg":"🐳 Docker mode: Using environment variables for config generation"}
{"time":"2025-10-20T07:47:47.480123332Z","level":"INFO","msg":"🌐 Orchestrator API Client initialized","endpoint":"http://aetheria-orchestrator.aetheria.svc:8080"}
{"time":"2025-10-20T07:47:47.480174031Z","level":"INFO","msg":"📝 Requesting configuration generation from orchestrator API","network":"hoodi_stage","clusters":[300,301,302,303],"tests":"(sanity,e2m),(bulk,7)"}
{"time":"2025-10-20T07:47:47.596457065Z","level":"INFO","msg":"✅ Configuration generated successfully via API","resource_usage_id":"d36304548488d99aa00efea3ab2556e1"}
{"time":"2025-10-20T07:47:47.596987474Z","level":"INFO","msg":"🚀 EXECUTOR Initialized Successfully"}
{"time":"2025-10-20T07:47:47.597032261Z","level":"INFO","msg":"⚙️ Test configuration loaded","network":"hoodi_stage","nodes":[300,301,302,303],"test_type":{"Sanity":{"ShouldRun":true,"E2M":true,"BulkAmount":0},"Bulk":{"ShouldRun":true,"E2M":false,"BulkAmount":7},"Isolated":{"Tests":null}}}
{"time":"2025-10-20T07:47:52.248048115Z","level":"INFO","msg":"Generated operator pubkeys file","module":"scanner","path":"/app/shared/data/operators/operator-pubkeys-hoodi_stage.json"}
{"time":"2025-10-20T07:47:52.248232107Z","level":"INFO","msg":"Registered parser","module":"main","component":"parser-client","parser_type":"consensus"}
{"time":"2025-10-20T07:47:52.248300822Z","level":"INFO","msg":"Registered parser","module":"main","component":"parser-client","parser_type":"flow-emits"}
{"time":"2025-10-20T07:47:52.248368593Z","level":"INFO","msg":"Validation client initialized","module":"main","loki_enabled":true,"e2m_enabled":true}
{"time":"2025-10-20T07:47:52.248412301Z","level":"INFO","msg":"🔑 KEY STEP: Starting Isolated Tests Concurrently if exists...","module":"main"}
{"time":"2025-10-20T07:47:52.248455482Z","level":"INFO","msg":"🔑 KEY STEP: Starting Sanity Test Suite","module":"main"}
{"time":"2025-10-20T07:47:52.248577431Z","level":"INFO","msg":"No isolated tests configured, skipping","module":"main"}
{"time":"2025-10-20T07:47:52.306532375Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":300}
{"time":"2025-10-20T07:47:52.31841493Z","level":"INFO","msg":"Available slots","test":"sanity","node":300,"slots":3000}
{"time":"2025-10-20T07:47:52.318473316Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":301}
{"time":"2025-10-20T07:47:52.333320682Z","level":"INFO","msg":"Available slots","test":"sanity","node":301,"slots":756}
{"time":"2025-10-20T07:47:52.334179258Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":302}
{"time":"2025-10-20T07:47:52.342243489Z","level":"INFO","msg":"Available slots","test":"sanity","node":302,"slots":452}
{"time":"2025-10-20T07:47:52.342327085Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":303}
{"time":"2025-10-20T07:47:52.349163252Z","level":"INFO","msg":"Available slots","test":"sanity","node":303,"slots":1073}
{"time":"2025-10-20T07:48:29.604965652Z","level":"INFO","msg":"Extracted keyshares path","test":"sanity","module":"keys","path":"/app/shared/data/keyshares/0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754.json"}
{"time":"2025-10-20T07:48:29.605058304Z","level":"INFO","msg":"Key shares file generated successfully","test":"sanity","keyshare_path":"/app/shared/data/keyshares/0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754.json"}
{"time":"2025-10-20T07:48:31.302300543Z","level":"INFO","msg":"Allowance fetched successfully","test":"sanity","module":"contract","account":"0x91e32efb8139cd88cae0df30d2bf471294c6ed27","allowance":"99294309999200282400000000"}
{"time":"2025-10-20T07:48:31.302388709Z","level":"INFO","msg":"Threshold value","test":"sanity","module":"contract","threshold":"9000000000000000000000"}
{"time":"2025-10-20T07:48:31.302410547Z","level":"INFO","msg":"Sufficient allowance","test":"sanity","module":"contract","allowance margin":"6561000000000000000000000000000000000000000000000000000000000000000000000000000000000000"}
{"time":"2025-10-20T07:48:31.310601701Z","level":"INFO","msg":"Allowance fetched successfully","test":"sanity","module":"contract","account":"0x91e32efb8139cd88cae0df30d2bf471294c6ed27","allowance":"99294309999200282400000000"}
{"time":"2025-10-20T07:48:31.310681786Z","level":"INFO","msg":"Allowance","test":"sanity","module":"contract","allowance":"99294309999200282400000000"}
{"time":"2025-10-20T07:48:31.472945455Z","level":"INFO","msg":"Registering validator","test":"sanity","module":"contract","tx_hash":"0xc13751d258aceb31fa73bcfd83e22d88aad2edbe4938872acb100cde26ecbc28"}
{"time":"2025-10-20T07:48:31.477372963Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0xc13751d258aceb31fa73bcfd83e22d88aad2edbe4938872acb100cde26ecbc28"}
{"time":"2025-10-20T07:48:41.483230958Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0xc13751d258aceb31fa73bcfd83e22d88aad2edbe4938872acb100cde26ecbc28"}
{"time":"2025-10-20T07:48:41.483357916Z","level":"INFO","msg":"✅ SUCCESS: Contract Register Successful","test":"sanity","txHash":"0xc13751d258aceb31fa73bcfd83e22d88aad2edbe4938872acb100cde26ecbc28","block_hash":"0x69922ec97d678115f7253746204a672ae0525fee68969cc69331b496d2a73ca4"}
{"time":"2025-10-20T07:48:41.49947313Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"sanity","module":"helpers","pubkey":"0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754","timeToSleep":10,"attempt":1}
{"time":"2025-10-20T07:48:51.515751916Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"sanity","module":"helpers","pubkey":"0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754","timeToSleep":20,"attempt":2}
{"time":"2025-10-20T07:49:11.541800536Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"sanity","module":"helpers","pubkey":"0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754","timeToSleep":40,"attempt":3}
{"time":"2025-10-20T07:49:51.584323876Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"sanity","module":"helpers","pubkey":"0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754","timeToSleep":80,"attempt":4}
{"time":"2025-10-20T07:51:11.613982019Z","level":"INFO","msg":"Successfully retrieved validator index from exporter","test":"sanity","module":"helpers","pubkey":"0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754"}
{"time":"2025-10-20T07:51:11.631731064Z","level":"INFO","msg":"Registration epoch","module":"main","module":"e2m-validator","epoch":48784,"indices":"1102420"}
{"time":"2025-10-20T07:51:11.915655566Z","level":"INFO","msg":"Validator event validation successful","module":"main","module":"loki-validator","validated_validators":1,"event":"ValidatorAdded"}
{"time":"2025-10-20T07:51:11.915718388Z","level":"INFO","msg":"SSV node log validation successful","test":"sanity","indices":["1102420"],"validators":1,"validate-type":"registration"}
{"time":"2025-10-20T07:51:26.674763942Z","level":"INFO","msg":"Validator did not attest","module":"main","module":"e2m-validator","epoch":48783,"indices":["1102420"]}
{"time":"2025-10-20T07:54:12.286911275Z","level":"INFO","msg":"Validator did not attest","module":"main","module":"e2m-validator","epoch":48784,"indices":["1102420"]}
{"time":"2025-10-20T08:00:28.439002282Z","level":"INFO","msg":"E2M registration validation successful - validators are attesting","module":"main","module":"e2m-validator","validation_type":"registration","indices":["1102420"],"registered_epoch":48784,"validated_epoch":48785,"epochs_waited":1}
{"time":"2025-10-20T08:00:28.43915639Z","level":"INFO","msg":"Starting consensus validation with epoch iteration logic","module":"main","module":"loki-validator","validator_index":"1102420","registered_epoch":48784,"max_epoch_offset":4}
{"time":"2025-10-20T08:00:28.439248839Z","level":"INFO","msg":"Checking epoch for duties","module":"main","module":"loki-validator","epoch_offset":0,"current_epoch":"48784","validator_index":"1102420"}
{"time":"2025-10-20T08:00:28.439210176Z","level":"INFO","msg":"E2M validation successful","test":"sanity","indices":["1102420"],"validate-type":"registration"}
{"time":"2025-10-20T08:00:28.449257449Z","level":"WARN","msg":"No attesters found in duties response","module":"main","module":"loki-validator","epoch_offset":0,"epoch":"48784"}
{"time":"2025-10-20T08:00:28.449365051Z","level":"INFO","msg":"Checking epoch for duties","module":"main","module":"loki-validator","epoch_offset":1,"current_epoch":"48785","validator_index":"1102420"}
{"time":"2025-10-20T08:00:28.459952106Z","level":"INFO","msg":"Successfully found duties, building duty ID for consensus validation","module":"main","module":"loki-validator","duty_id":"COMMITTEE-300_301_302_303-e48785-s1561129","final_epoch":"48785","slot":"1561129"}
{"time":"2025-10-20T08:00:28.635944669Z","level":"INFO","msg":"Consensus leader validation passed","module":"main","module":"loki-validator","slot":1561129,"leader":301}
{"time":"2025-10-20T08:00:28.636071571Z","level":"INFO","msg":"Consensus timing validation comparison","module":"main","module":"loki-validator","actual_time_diff":38566000,"expected_consensus_time":33000000,"lower_bound":23100000,"upper_bound":42900000,"margin_percent":30}
{"time":"2025-10-20T08:00:28.63609176Z","level":"INFO","msg":"Consensus timing validation passed - within acceptable range","module":"main","module":"loki-validator"}
{"time":"2025-10-20T08:00:28.636115281Z","level":"INFO","msg":"Consensus time validation completed successfully","module":"main","module":"loki-validator","final_epoch":"48785"}
{"time":"2025-10-20T08:00:28.636145392Z","level":"INFO","msg":"Consensus timing validation successful","test":"sanity","indices":["1102420"],"epoch":48784}
{"time":"2025-10-20T08:00:30.881168369Z","level":"INFO","msg":"Liquidating cluster","test":"sanity","module":"contract","tx_hash":"0xdbe0ffe71d95bb5fc50c44bd83c9b25c41125c0c65845655f5839a81c0808b8f"}
{"time":"2025-10-20T08:00:30.886510261Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0xdbe0ffe71d95bb5fc50c44bd83c9b25c41125c0c65845655f5839a81c0808b8f"}
{"time":"2025-10-20T08:00:40.893780749Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0xdbe0ffe71d95bb5fc50c44bd83c9b25c41125c0c65845655f5839a81c0808b8f"}
{"time":"2025-10-20T08:00:40.893924699Z","level":"INFO","msg":"✅ SUCCESS: Cluster Liquidation Successful","test":"sanity","txHash":"0xdbe0ffe71d95bb5fc50c44bd83c9b25c41125c0c65845655f5839a81c0808b8f","block_hash":"0xeaf15314e54cdbcb3814569794ba555f372c6dab0b91f0af456564964eeec3f7"}
{"time":"2025-10-20T08:00:40.894157365Z","level":"INFO","msg":"Querying Loki for transaction hash validation","module":"main","module":"loki-validator","query":"{container=~\\"ssv-node-300|ssv-node-301|ssv-node-302|ssv-node-303\\"} |= \\"0xdbe0ffe71d95bb5fc50c44bd83c9b25c41125c0c65845655f5839a81c0808b8f\\"","tx_hash":"0xdbe0ffe71d95bb5fc50c44bd83c9b25c41125c0c65845655f5839a81c0808b8f"}
{"time":"2025-10-20T08:00:40.900855922Z","level":"INFO","msg":"waiting for block to be found","module":"main","module":"e2m-validator","module":"e2m"}
{"time":"2025-10-20T08:00:41.038133854Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":1,"max_retries":5,"delay_seconds":30,"start_unix":1760936440,"end_unix":1760947240}
{"time":"2025-10-20T08:01:00.92582018Z","level":"INFO","msg":"Liquidation epoch","module":"main","module":"e2m-validator","epoch":48786,"indices":["1102420"]}
{"time":"2025-10-20T08:01:11.124492368Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":2,"max_retries":5,"delay_seconds":60,"start_unix":1760936440,"end_unix":1760947271}
{"time":"2025-10-20T08:02:11.1595482Z","level":"ERROR","msg":"request failed","module":"HTTPRequester","error":"Get \\"https://loki.ops.ssvlabsinternal.com/loki/api/v1/query_range?query=%7Bcontainer%3D~%22ssv-node-300%7Cssv-node-301%7Cssv-node-302%7Cssv-node-303%22%7D+%7C%3D+%220xdbe0ffe71d95bb5fc50c44bd83c9b25c41125c0c65845655f5839a81c0808b8f%22+%7C+json+%7C+line_format+%22%7B%7B.container%7D%7D%3A%7B%7B.msg%7D%7D%22&start=1760936440&end=1760947331&limit=1000\\": unexpected EOF","request attempt #":0}
{"time":"2025-10-20T08:02:11.448754681Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":3,"max_retries":5,"delay_seconds":90,"start_unix":1760936440,"end_unix":1760947331}
{"time":"2025-10-20T08:03:41.561963828Z","level":"INFO","msg":"Event validation successful","module":"main","module":"loki-validator","valid_entries":4,"event":"ClusterLiquidated"}
{"time":"2025-10-20T08:03:41.562159387Z","level":"INFO","msg":"SSV node log validation successful","test":"sanity","indices":["1102420"],"validators":1,"validate-type":"liquidation"}
{"time":"2025-10-20T08:07:17.10619848Z","level":"INFO","msg":"Validator was not liquidated","module":"main","module":"e2m-validator","epoch":48786,"indices":["1102420"]}
{"time":"2025-10-20T08:13:18.421221738Z","level":"INFO","msg":"E2M liquidation validation successful - validators are liquidated","module":"main","module":"e2m-validator","validation_type":"liquidation","indices":["1102420"],"liquidated_epoch":48786,"validated_epoch":48787,"epochs_waited":1,"assigned":0,"missed":0}
{"time":"2025-10-20T08:13:18.421399853Z","level":"INFO","msg":"E2M validation successful","test":"sanity","indices":["1102420"],"validate-type":"liquidation"}
{"time":"2025-10-20T08:13:20.293076778Z","level":"INFO","msg":"Allowance fetched successfully","test":"sanity","module":"contract","account":"0x91e32efb8139cd88cae0df30d2bf471294c6ed27","allowance":"99294309999200282400000000"}
{"time":"2025-10-20T08:13:20.293212802Z","level":"INFO","msg":"Threshold value","test":"sanity","module":"contract","threshold":"9000000000000000000000"}
{"time":"2025-10-20T08:13:20.293275608Z","level":"INFO","msg":"Sufficient allowance","test":"sanity","module":"contract","allowance margin":"6561000000000000000000000000000000000000000000000000000000000000000000000000000000000000"}
{"time":"2025-10-20T08:13:20.440505802Z","level":"INFO","msg":"Reactivating cluster","test":"sanity","module":"contract","tx_hash":"0x292acc344b1a93ab9e5192ae2c96ab4d83dd187ac6a9d9ce75e06b0631801121"}
{"time":"2025-10-20T08:13:20.44441461Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0x292acc344b1a93ab9e5192ae2c96ab4d83dd187ac6a9d9ce75e06b0631801121"}
{"time":"2025-10-20T08:13:30.452545194Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0x292acc344b1a93ab9e5192ae2c96ab4d83dd187ac6a9d9ce75e06b0631801121"}
{"time":"2025-10-20T08:13:30.452926593Z","level":"INFO","msg":"✅ SUCCESS: Cluster Reactivation Successful","test":"sanity","txHash":"0x292acc344b1a93ab9e5192ae2c96ab4d83dd187ac6a9d9ce75e06b0631801121","block_hash":"0xf60b69e662c202466a12a39f371d8b787393c06096da34538c4b2a1a265e4d9b"}
{"time":"2025-10-20T08:13:30.453208025Z","level":"INFO","msg":"Validating reactivation - checking that validators resume attesting","module":"main","module":"e2m-validator","indices":["1102420"],"block_hash":"0xf60b69e662c202466a12a39f371d8b787393c06096da34538c4b2a1a265e4d9b"}
{"time":"2025-10-20T08:13:30.453285775Z","level":"INFO","msg":"Querying Loki for transaction hash validation","module":"main","module":"loki-validator","query":"{container=~\\"ssv-node-300|ssv-node-301|ssv-node-302|ssv-node-303\\"} |= \\"0x292acc344b1a93ab9e5192ae2c96ab4d83dd187ac6a9d9ce75e06b0631801121\\"","tx_hash":"0x292acc344b1a93ab9e5192ae2c96ab4d83dd187ac6a9d9ce75e06b0631801121"}
{"time":"2025-10-20T08:13:30.461022063Z","level":"INFO","msg":"waiting for block to be found","module":"main","module":"e2m-validator","module":"e2m"}
{"time":"2025-10-20T08:13:30.760809252Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":1,"max_retries":5,"delay_seconds":30,"start_unix":1760937210,"end_unix":1760948010}
{"time":"2025-10-20T08:13:50.476550837Z","level":"INFO","msg":"Registration epoch","module":"main","module":"e2m-validator","epoch":48788,"indices":"1102420"}
{"time":"2025-10-20T08:14:00.81312749Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":2,"max_retries":5,"delay_seconds":60,"start_unix":1760937210,"end_unix":1760948040}
{"time":"2025-10-20T08:14:05.504797484Z","level":"INFO","msg":"Validator did not attest","module":"main","module":"e2m-validator","epoch":48787,"indices":["1102420"]}
{"time":"2025-10-20T08:15:00.865060498Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":3,"max_retries":5,"delay_seconds":90,"start_unix":1760937210,"end_unix":1760948100}
{"time":"2025-10-20T08:16:30.978065967Z","level":"INFO","msg":"Event validation successful","module":"main","module":"loki-validator","valid_entries":4,"event":"ClusterReactivated"}
{"time":"2025-10-20T08:16:30.978179619Z","level":"INFO","msg":"SSV node log validation successful","test":"sanity","indices":["1102420"],"validators":1,"validate-type":"reactivation"}
{"time":"2025-10-20T08:19:51.714081079Z","level":"INFO","msg":"Validator did not attest","module":"main","module":"e2m-validator","epoch":48788,"indices":["1102420"]}
{"time":"2025-10-20T08:26:07.977027311Z","level":"INFO","msg":"E2M registration validation successful - validators are attesting","module":"main","module":"e2m-validator","validation_type":"registration","indices":["1102420"],"registered_epoch":48788,"validated_epoch":48789,"epochs_waited":1}
{"time":"2025-10-20T08:26:07.97723511Z","level":"INFO","msg":"E2M validation successful","test":"sanity","indices":["1102420"],"validate-type":"reactivation"}
{"time":"2025-10-20T08:26:07.97711931Z","level":"INFO","msg":"Starting consensus validation with epoch iteration logic","module":"main","module":"loki-validator","validator_index":"1102420","registered_epoch":48788,"max_epoch_offset":4}
{"time":"2025-10-20T08:26:07.977324157Z","level":"INFO","msg":"Checking epoch for duties","module":"main","module":"loki-validator","epoch_offset":0,"current_epoch":"48788","validator_index":"1102420"}
{"time":"2025-10-20T08:26:07.994305779Z","level":"WARN","msg":"No attesters found in duties response","module":"main","module":"loki-validator","epoch_offset":0,"epoch":"48788"}
{"time":"2025-10-20T08:26:07.994456825Z","level":"INFO","msg":"Checking epoch for duties","module":"main","module":"loki-validator","epoch_offset":1,"current_epoch":"48789","validator_index":"1102420"}
{"time":"2025-10-20T08:26:08.014805718Z","level":"INFO","msg":"Successfully found duties, building duty ID for consensus validation","module":"main","module":"loki-validator","duty_id":"COMMITTEE-300_301_302_303-e48789-s1561268","final_epoch":"48789","slot":"1561268"}
{"time":"2025-10-20T08:26:08.226629765Z","level":"INFO","msg":"Consensus leader validation passed","module":"main","module":"loki-validator","slot":1561268,"leader":300}
{"time":"2025-10-20T08:26:08.226766526Z","level":"INFO","msg":"Consensus timing validation comparison","module":"main","module":"loki-validator","actual_time_diff":56774000,"expected_consensus_time":48000000,"lower_bound":33600000,"upper_bound":62400000,"margin_percent":30}
{"time":"2025-10-20T08:26:08.226812046Z","level":"INFO","msg":"Consensus timing validation passed - within acceptable range","module":"main","module":"loki-validator"}
{"time":"2025-10-20T08:26:08.226858755Z","level":"INFO","msg":"Consensus time validation completed successfully","module":"main","module":"loki-validator","final_epoch":"48789"}
{"time":"2025-10-20T08:26:08.226918656Z","level":"INFO","msg":"Consensus timing validation successful","test":"sanity","indices":["1102420"],"epoch":48788}
{"time":"2025-10-20T08:26:10.00142581Z","level":"INFO","msg":"Allowance fetched successfully","test":"sanity","module":"contract","account":"0x91e32efb8139cd88cae0df30d2bf471294c6ed27","allowance":"99285309999200282400000000"}
{"time":"2025-10-20T08:26:10.001505118Z","level":"INFO","msg":"Threshold value","test":"sanity","module":"contract","threshold":"9000000000000000000000"}
{"time":"2025-10-20T08:26:10.001539261Z","level":"INFO","msg":"Sufficient allowance","test":"sanity","module":"contract","allowance margin":"6561000000000000000000000000000000000000000000000000000000000000000000000000000000000000"}
{"time":"2025-10-20T08:26:10.300395796Z","level":"INFO","msg":"Removing validator","test":"sanity","module":"contract","tx_hash":"0x1173dd923c887e345789f335a299e71d1b47a3008a09db8bb786c830c4df061e"}
{"time":"2025-10-20T08:26:10.303778687Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0x1173dd923c887e345789f335a299e71d1b47a3008a09db8bb786c830c4df061e"}
{"time":"2025-10-20T08:26:20.311176875Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0x1173dd923c887e345789f335a299e71d1b47a3008a09db8bb786c830c4df061e"}
{"time":"2025-10-20T08:26:20.311253253Z","level":"INFO","msg":"✅ SUCCESS: Contract Removal Successful","test":"sanity","blockHash":"0xbbf1763726953376986002c408fea4c9fe49979998f73395e04456bf8391f9f2"}
{"time":"2025-10-20T08:26:20.319429059Z","level":"INFO","msg":"waiting for block to be found","module":"main","module":"e2m-validator","module":"e2m"}
{"time":"2025-10-20T08:26:20.475085745Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":1,"max_retries":5,"delay_seconds":30,"start_unix":1760937980,"end_unix":1760948780}
{"time":"2025-10-20T08:26:40.328339519Z","level":"INFO","msg":"Removal epoch","module":"main","module":"e2m-validator","epoch":48790,"indices":"1102420"}
{"time":"2025-10-20T08:26:50.538768766Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":2,"max_retries":5,"delay_seconds":60,"start_unix":1760937980,"end_unix":1760948810}
{"time":"2025-10-20T08:27:50.539844474Z","level":"ERROR","msg":"request failed","module":"HTTPRequester","error":"Get \\"https://loki.ops.ssvlabsinternal.com/loki/api/v1/query_range?query=%7Bcontainer%3D~%22ssv-node-300%7Cssv-node-301%7Cssv-node-302%7Cssv-node-303%22%7D+%7C%3D+%220x1173dd923c887e345789f335a299e71d1b47a3008a09db8bb786c830c4df061e%22&start=1760937980&end=1760948870&limit=1000\\": unexpected EOF","request attempt #":0}
{"time":"2025-10-20T08:27:50.674152535Z","level":"INFO","msg":"Validator event validation successful","module":"main","module":"loki-validator","validated_validators":1,"event":"ValidatorRemoved"}
{"time":"2025-10-20T08:27:50.674272195Z","level":"INFO","msg":"SSV node log validation successful","test":"sanity","indices":["1102420"],"validators":1,"validate-type":"removal"}
{"time":"2025-10-20T08:32:41.934415675Z","level":"INFO","msg":"Validator was not removed","module":"main","module":"e2m-validator","epoch":48790,"indices":["1102420"]}
{"time":"2025-10-20T08:38:58.363553348Z","level":"INFO","msg":"E2M removal validation successful - validators stopped attesting","module":"main","module":"e2m-validator","validation_type":"removal","indices":["1102420"],"removed_epoch":48790,"validated_epoch":48791,"epochs_waited":1}
{"time":"2025-10-20T08:38:58.363785517Z","level":"INFO","msg":"E2M validation successful","test":"sanity","indices":["1102420"],"validate-type":"removal"}
{"time":"2025-10-20T08:38:58.36408407Z","level":"INFO","msg":"✅ SUCCESS: Sanity Test Suite Completed Successfully","module":"main","register":{"Contract":true,"E2M":true,"Loki":true},"liquidate":{"Contract":true,"E2M":true,"Loki":true},"reactivate":{"Contract":true,"E2M":true,"Loki":true},"remove":{"Contract":true,"E2M":true,"Loki":true}}
{"time":"2025-10-20T08:38:58.36415183Z","level":"INFO","msg":"🔑 KEY STEP: Starting Bulk Test Suite","module":"main"}
{"time":"2025-10-20T08:38:58.387087394Z","level":"INFO","msg":"Getting available slots","test":"bulk","node":300}
{"time":"2025-10-20T08:38:58.394769433Z","level":"INFO","msg":"Available slots","test":"bulk","node":300,"slots":3000}
{"time":"2025-10-20T08:38:58.394854942Z","level":"INFO","msg":"Getting available slots","test":"bulk","node":301}
{"time":"2025-10-20T08:38:58.401806061Z","level":"INFO","msg":"Available slots","test":"bulk","node":301,"slots":756}
{"time":"2025-10-20T08:38:58.401885651Z","level":"INFO","msg":"Getting available slots","test":"bulk","node":302}
{"time":"2025-10-20T08:38:58.408673251Z","level":"INFO","msg":"Available slots","test":"bulk","node":302,"slots":452}
{"time":"2025-10-20T08:38:58.408753982Z","level":"INFO","msg":"Getting available slots","test":"bulk","node":303}
{"time":"2025-10-20T08:38:58.415827774Z","level":"INFO","msg":"Available slots","test":"bulk","node":303,"slots":1073}
{"time":"2025-10-20T08:40:09.297525744Z","level":"INFO","msg":"Extracted keyshares path for bulk keyshare","test":"bulk","module":"keys","path":"/app/shared/data/keyshares/keyshares-1760949609.json"}
{"time":"2025-10-20T08:40:12.998888746Z","level":"INFO","msg":"Allowance fetched successfully","test":"bulk","module":"contract","account":"0x91e32efb8139cd88cae0df30d2bf471294c6ed27","allowance":"99285309999200282400000000"}
{"time":"2025-10-20T08:40:12.999041505Z","level":"INFO","msg":"Threshold value","test":"bulk","module":"contract","threshold":"9000000000000000000000"}
{"time":"2025-10-20T08:40:12.999109774Z","level":"INFO","msg":"Sufficient allowance","test":"bulk","module":"contract","allowance margin":"6561000000000000000000000000000000000000000000000000000000000000000000000000000000000000"}
{"time":"2025-10-20T08:40:13.088611007Z","level":"INFO","msg":"Bulk Registering validators","test":"bulk","module":"contract","tx_hash":"0x5b40433973ccb056951501a8f6c3b561729295b0549109e07c0acce6ed73bec3"}
{"time":"2025-10-20T08:40:13.088677356Z","level":"INFO","msg":"Bulk registration transaction sent","test":"bulk","txHash":"0x5b40433973ccb056951501a8f6c3b561729295b0549109e07c0acce6ed73bec3"}
{"time":"2025-10-20T08:40:13.093840522Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"bulk","module":"contract","attempt":1,"sleep":10,"tx_hash":"0x5b40433973ccb056951501a8f6c3b561729295b0549109e07c0acce6ed73bec3"}
{"time":"2025-10-20T08:40:23.135238732Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"bulk","module":"contract","attempt":2,"sleep":20,"tx_hash":"0x5b40433973ccb056951501a8f6c3b561729295b0549109e07c0acce6ed73bec3"}
{"time":"2025-10-20T08:40:43.160137018Z","level":"INFO","msg":"Transaction receipt received","test":"bulk","module":"contract","tx_hash":"0x5b40433973ccb056951501a8f6c3b561729295b0549109e07c0acce6ed73bec3"}
{"time":"2025-10-20T08:40:43.160243854Z","level":"INFO","msg":"✅ SUCCESS: Bulk Registration Successful","test":"bulk","blockHash":"0x3fe67a79bd564605886137b85520ec8d40a30378eb560f3efc3474a2d178f01d","Validators Amount":7}
{"time":"2025-10-20T08:40:43.186803397Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"bulk","module":"helpers","pubkey":"0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754","timeToSleep":10,"attempt":1}
{"time":"2025-10-20T08:40:53.195265368Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"bulk","module":"helpers","pubkey":"0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754","timeToSleep":20,"attempt":2}
{"time":"2025-10-20T08:41:13.216188617Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"bulk","module":"helpers","pubkey":"0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754","timeToSleep":40,"attempt":3}
{"time":"2025-10-20T08:41:53.235599029Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"bulk","module":"helpers","pubkey":"0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754","timeToSleep":80,"attempt":4}
{"time":"2025-10-20T08:43:13.309046552Z","level":"INFO","msg":"Successfully retrieved validator index from exporter","test":"bulk","module":"helpers","pubkey":"0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754"}
{"time":"2025-10-20T08:43:13.315673429Z","level":"INFO","msg":"Successfully retrieved validator index from exporter","test":"bulk","module":"helpers","pubkey":"0x84d7c46eb947f61c65a1c32971bbad4a1e832b1998675374fdebab49e6ab6d2cc0a7208acb3d176ac8b5df0cd3587f23"}
{"time":"2025-10-20T08:43:13.324694397Z","level":"INFO","msg":"Successfully retrieved validator index from exporter","test":"bulk","module":"helpers","pubkey":"0x86d6aa42d120173270727af286965c371f919f423558997f914b30b20bba45b041561650b531d386ed85f65828e0bde1"}
{"time":"2025-10-20T08:43:13.334119387Z","level":"INFO","msg":"Successfully retrieved validator index from exporter","test":"bulk","module":"helpers","pubkey":"0x87aa97b4388e95df2a4a06bf7e638fe3a86c097159bfe9e8f4483fc8028025fc9a69b0121edd7da55bbed0615610ed3c"}
{"time":"2025-10-20T08:43:13.340723666Z","level":"INFO","msg":"Successfully retrieved validator index from exporter","test":"bulk","module":"helpers","pubkey":"0x88e97e72ea3a862b12b44028e65f2af490b8cc19d4b6f1915f1f80325cff9f02d8f5c6fd79f127c5fde7a90fcaa43b8d"}
{"time":"2025-10-20T08:43:13.350470974Z","level":"INFO","msg":"Successfully retrieved validator index from exporter","test":"bulk","module":"helpers","pubkey":"0x89dadaea25c2fa19d4c45e1ba993c85445be5f19c32b5d50af109802ef6cacd492075907be0d585e7e707a138d3e7c20"}
{"time":"2025-10-20T08:43:13.359964686Z","level":"INFO","msg":"Successfully retrieved validator index from exporter","test":"bulk","module":"helpers","pubkey":"0xa88938ed513411df0322826f6b017a2d1fb610c633e33ccad9dc1e46b274ad32a489f89e7b1b772ab518ba0cb089cee2"}
{"time":"2025-10-20T08:43:13.377315642Z","level":"INFO","msg":"Registration epoch","module":"main","module":"e2m-validator","epoch":48792,"indices":"1102420,1104199,1092007,1104251,1092460,1102518,1092461"}
{"time":"2025-10-20T08:43:13.620686428Z","level":"INFO","msg":"Validator event validation successful","module":"main","module":"loki-validator","validated_validators":7,"event":"ValidatorAdded"}
{"time":"2025-10-20T08:43:13.620824175Z","level":"INFO","msg":"SSV node log validation successful","test":"bulk","indices":["1102420","1104199","1092007","1104251","1092460","1102518","1092461"],"validators":7,"validate-type":"registration"}
{"time":"2025-10-20T08:43:28.425021103Z","level":"INFO","msg":"Validator did not attest","module":"main","module":"e2m-validator","epoch":48791,"indices":["1102420","1104199","1092007","1104251","1092460","1102518","1092461"]}
{"time":"2025-10-20T08:45:28.825845957Z","level":"INFO","msg":"Validator did not attest","module":"main","module":"e2m-validator","epoch":48792,"indices":["1102420","1104199","1092007","1104251","1092460","1102518","1092461"]}
{"time":"2025-10-20T08:51:45.208933443Z","level":"INFO","msg":"Validator did not attest","module":"main","module":"e2m-validator","epoch":48793,"indices":["1102420","1104199","1092007","1104251","1092460","1102518","1092461"]}
{"time":"2025-10-20T08:52:15.326610535Z","level":"INFO","msg":"E2M registration validation successful - validators are attesting","module":"main","module":"e2m-validator","validation_type":"registration","indices":["1102420","1104199","1092007","1104251","1092460","1102518","1092461"],"registered_epoch":48792,"validated_epoch":48793,"epochs_waited":1}
{"time":"2025-10-20T08:52:15.326682148Z","level":"INFO","msg":"E2M validation successful","test":"bulk","indices":["1102420","1104199","1092007","1104251","1092460","1102518","1092461"],"validate-type":"registration"}
{"time":"2025-10-20T08:52:15.326769164Z","level":"INFO","msg":"Starting consensus validation with epoch iteration logic","module":"main","module":"loki-validator","validator_index":"1102420","registered_epoch":48792,"max_epoch_offset":4}
{"time":"2025-10-20T08:52:15.326806346Z","level":"INFO","msg":"Checking epoch for duties","module":"main","module":"loki-validator","epoch_offset":0,"current_epoch":"48792","validator_index":"1102420"}
{"time":"2025-10-20T08:52:15.337042485Z","level":"WARN","msg":"No attesters found in duties response","module":"main","module":"loki-validator","epoch_offset":0,"epoch":"48792"}
{"time":"2025-10-20T08:52:15.337123736Z","level":"INFO","msg":"Checking epoch for duties","module":"main","module":"loki-validator","epoch_offset":1,"current_epoch":"48793","validator_index":"1102420"}
{"time":"2025-10-20T08:52:15.349664045Z","level":"INFO","msg":"Successfully found duties, building duty ID for consensus validation","module":"main","module":"loki-validator","duty_id":"COMMITTEE-300_301_302_303-e48793-s1561382","final_epoch":"48793","slot":"1561382"}
{"time":"2025-10-20T08:52:15.496238055Z","level":"INFO","msg":"Consensus leader validation passed","module":"main","module":"loki-validator","slot":1561382,"leader":302}
{"time":"2025-10-20T08:52:15.496286381Z","level":"INFO","msg":"Consensus timing validation comparison","module":"main","module":"loki-validator","actual_time_diff":37800000,"expected_consensus_time":33000000,"lower_bound":23100000,"upper_bound":42900000,"margin_percent":30}
{"time":"2025-10-20T08:52:15.49629247Z","level":"INFO","msg":"Consensus timing validation passed - within acceptable range","module":"main","module":"loki-validator"}
{"time":"2025-10-20T08:52:15.496301397Z","level":"INFO","msg":"Consensus time validation completed successfully","module":"main","module":"loki-validator","final_epoch":"48793"}
{"time":"2025-10-20T08:52:15.496312662Z","level":"INFO","msg":"Consensus timing validation successful","test":"bulk","indices":["1102420","1104199","1092007","1104251","1092460","1102518","1092461"],"epoch":48792}
{"time":"2025-10-20T08:52:17.35557327Z","level":"INFO","msg":"Bulk Removing validators","test":"bulk","module":"contract","tx_hash":"0x2023e2c5a7c26c05158e101af6822727250eafacf4c5f7b2fd05bcac43972497"}
{"time":"2025-10-20T08:52:17.355679184Z","level":"INFO","msg":"Bulk removal transaction sent","test":"bulk","txHash":"0x2023e2c5a7c26c05158e101af6822727250eafacf4c5f7b2fd05bcac43972497"}
{"time":"2025-10-20T08:52:17.361411402Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"bulk","module":"contract","attempt":1,"sleep":10,"tx_hash":"0x2023e2c5a7c26c05158e101af6822727250eafacf4c5f7b2fd05bcac43972497"}
{"time":"2025-10-20T08:52:27.376228166Z","level":"INFO","msg":"Transaction receipt received","test":"bulk","module":"contract","tx_hash":"0x2023e2c5a7c26c05158e101af6822727250eafacf4c5f7b2fd05bcac43972497"}
{"time":"2025-10-20T08:52:27.376349438Z","level":"INFO","msg":"✅ SUCCESS: Bulk Removal Successful","test":"bulk","blockHash":"0xc10ef8ae8ecd753e490728cd6e77b280efc2fee16c5d2e99825f6bfe3c94691f","Validators Amount":7}
{"time":"2025-10-20T08:52:27.385071265Z","level":"INFO","msg":"waiting for block to be found","module":"main","module":"e2m-validator","module":"e2m"}
{"time":"2025-10-20T08:52:27.523261994Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":1,"max_retries":5,"delay_seconds":30,"start_unix":1760939547,"end_unix":1760950347}
{"time":"2025-10-20T08:52:47.420575369Z","level":"INFO","msg":"Removal epoch","module":"main","module":"e2m-validator","epoch":48794,"indices":"1102420,1104199,1092007,1104251,1092460,1102518,1092461"}
{"time":"2025-10-20T08:52:57.60659248Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":2,"max_retries":5,"delay_seconds":60,"start_unix":1760939547,"end_unix":1760950377}
{"time":"2025-10-20T08:53:57.607461761Z","level":"ERROR","msg":"request failed","module":"HTTPRequester","error":"Get \\"https://loki.ops.ssvlabsinternal.com/loki/api/v1/query_range?query=%7Bcontainer%3D~%22ssv-node-300%7Cssv-node-301%7Cssv-node-302%7Cssv-node-303%22%7D+%7C%3D+%220x2023e2c5a7c26c05158e101af6822727250eafacf4c5f7b2fd05bcac43972497%22&start=1760939547&end=1760950437&limit=1000\\": unexpected EOF","request attempt #":0}
{"time":"2025-10-20T08:53:57.721286996Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":3,"max_retries":5,"delay_seconds":90,"start_unix":1760939547,"end_unix":1760950437}
{"time":"2025-10-20T08:55:27.856141161Z","level":"INFO","msg":"Validator event validation successful","module":"main","module":"loki-validator","validated_validators":7,"event":"ValidatorRemoved"}
{"time":"2025-10-20T08:55:27.856328034Z","level":"INFO","msg":"SSV node log validation successful","test":"bulk","indices":["1102420","1104199","1092007","1104251","1092460","1102518","1092461"],"validators":7,"validate-type":"removal"}
{"time":"2025-10-20T08:58:03.530556528Z","level":"INFO","msg":"Validator was not removed","module":"main","module":"e2m-validator","epoch":48794,"indices":["1102420","1104199","1092007","1104251","1092460","1102518","1092461"]}
{"time":"2025-10-20T09:04:49.887906155Z","level":"INFO","msg":"E2M removal validation successful - validators stopped attesting","module":"main","module":"e2m-validator","validation_type":"removal","indices":["1102420","1104199","1092007","1104251","1092460","1102518","1092461"],"removed_epoch":48794,"validated_epoch":48795,"epochs_waited":1}
{"time":"2025-10-20T09:04:49.887992087Z","level":"INFO","msg":"E2M validation successful","test":"bulk","indices":["1102420","1104199","1092007","1104251","1092460","1102518","1092461"],"validate-type":"removal"}
{"time":"2025-10-20T09:04:49.888071165Z","level":"INFO","msg":"✅ SUCCESS: Bulk Test Suite Completed Successfully","module":"main","register":{"Contract":true,"E2M":true,"Loki":true},"remove":{"Contract":true,"E2M":true,"Loki":true}}
{"time":"2025-10-20T09:04:49.888094083Z","level":"INFO","msg":"⏳ PROGRESS: Waiting for all tests to finish","module":"main"}
{"time":"2025-10-20T09:04:49.888157834Z","level":"INFO","msg":"✅ SUCCESS: All Test Suites Completed Successfully","module":"main","passed_suites":["bulk","sanity"],"skipped_suites":["isolated"]}
{"time":"2025-10-20T09:04:49.888178169Z","level":"INFO","msg":"🔑 KEY STEP: Starting resource teardown...","module":"main","usage_id":"d36304548488d99aa00efea3ab2556e1"}
{"time":"2025-10-20T09:04:49.897840054Z","level":"INFO","msg":"🌐 Orchestrator API Client initialized","module":"main","endpoint":"http://aetheria-orchestrator.aetheria.svc:8080"}
{"time":"2025-10-20T09:04:49.897925622Z","level":"INFO","msg":"🧹 Requesting resource teardown","module":"main","usage_id":"d36304548488d99aa00efea3ab2556e1"}
{"time":"2025-10-20T09:04:49.957910192Z","level":"INFO","msg":"✅ SUCCESS: Completed resource teardown successfully","module":"main","usage_id":"d36304548488d99aa00efea3ab2556e1"}

2025-10-20T09:04:53Z The test run finished successfully

Result - 🟢 Success

yoaz-ssvlabs · 2025-10-20T10:06:21Z

/test tests=(bulk,6)

ssv-test-bot · 2025-10-20T10:06:25Z

🟢 Test run for PR #2433

Branch: ci-test-handler-testing2
Commit: 74b7038

Status

2025-10-20T10:06:25Z Starting test run
2025-10-20T10:06:26Z Test run submitted successfully for commit 74b7038
2025-10-20T10:06:27Z container image for commit 74b7038f0265e7f8da4425e9315fe4199c751b80 is build and present in container registry
2025-10-20T10:06:28Z automatically allocated node range cluster=304,305,306,307 for test run on network=hoodi_stage
2025-10-20T10:06:29Z deploying commit 74b7038f0265e7f8da4425e9315fe4199c751b80 to ssv nodes 304,305,306,307
2025-10-20T10:07:14Z started test executor with tests=(bulk,6) and network=hoodi_stage

Click to expand executor logs

═══════════════════════════════════════════════════════════════════════════════════════════════════════
                🧪  WELCOME TO Executor - SSV's Distributed Test Automation Execution Layer  🧪            

                                  ___ __  __ ___  ___  _   _  _____  ___   ___ 
                                 | __|\\ \\/ /| __|/ __|| | | ||_   _|/ _ \\ | _ \\
                                 | _|  >  < | _|| (__ | |_| |  | | | (_) ||   /
                                 |___|/_/\\_\\|___|\\___| \\___/   |_|  \\___/ |_|_\\
                                        

═══════════════════════════════════════════════════════════════════════════════════════════════════════
{"time":"2025-10-20T10:07:16.457827506Z","level":"INFO","msg":"🐳 Docker mode: Using environment variables for config generation"}
{"time":"2025-10-20T10:07:16.468259179Z","level":"INFO","msg":"🌐 Orchestrator API Client initialized","endpoint":"http://aetheria-orchestrator.aetheria.svc:8080"}
{"time":"2025-10-20T10:07:16.46830042Z","level":"INFO","msg":"📝 Requesting configuration generation from orchestrator API","network":"hoodi_stage","clusters":[304,305,306,307],"tests":"(bulk,6)"}
{"time":"2025-10-20T10:07:16.518508329Z","level":"INFO","msg":"✅ Configuration generated successfully via API","resource_usage_id":"70f185f53d419a8be2f77f9e8dbe339d"}
{"time":"2025-10-20T10:07:16.519510967Z","level":"INFO","msg":"🚀 EXECUTOR Initialized Successfully"}
{"time":"2025-10-20T10:07:16.519569886Z","level":"INFO","msg":"⚙️ Test configuration loaded","network":"hoodi_stage","nodes":[304,305,306,307],"test_type":{"Sanity":{"ShouldRun":false,"E2M":false,"BulkAmount":0},"Bulk":{"ShouldRun":true,"E2M":false,"BulkAmount":6},"Isolated":{"Tests":null}}}
{"time":"2025-10-20T10:07:20.209729723Z","level":"INFO","msg":"Generated operator pubkeys file","module":"scanner","path":"/app/shared/data/operators/operator-pubkeys-hoodi_stage.json"}
{"time":"2025-10-20T10:07:20.209857956Z","level":"INFO","msg":"Registered parser","module":"main","component":"parser-client","parser_type":"consensus"}
{"time":"2025-10-20T10:07:20.209888675Z","level":"INFO","msg":"Registered parser","module":"main","component":"parser-client","parser_type":"flow-emits"}
{"time":"2025-10-20T10:07:20.209950518Z","level":"INFO","msg":"Validation client initialized","module":"main","loki_enabled":true,"e2m_enabled":false}
{"time":"2025-10-20T10:07:20.209975275Z","level":"INFO","msg":"🔑 KEY STEP: Starting Isolated Tests Concurrently if exists...","module":"main"}
{"time":"2025-10-20T10:07:20.210004607Z","level":"INFO","msg":"🔑 KEY STEP: Starting Bulk Test Suite","module":"main"}
{"time":"2025-10-20T10:07:20.210127538Z","level":"INFO","msg":"No isolated tests configured, skipping","module":"main"}
{"time":"2025-10-20T10:07:20.259214758Z","level":"INFO","msg":"Getting available slots","test":"bulk","node":304}
{"time":"2025-10-20T10:07:20.268751193Z","level":"INFO","msg":"Available slots","test":"bulk","node":304,"slots":3000}
{"time":"2025-10-20T10:07:20.268812793Z","level":"INFO","msg":"Getting available slots","test":"bulk","node":305}
{"time":"2025-10-20T10:07:20.276019429Z","level":"INFO","msg":"Available slots","test":"bulk","node":305,"slots":756}
{"time":"2025-10-20T10:07:20.276082672Z","level":"INFO","msg":"Getting available slots","test":"bulk","node":306}
{"time":"2025-10-20T10:07:20.282992856Z","level":"INFO","msg":"Available slots","test":"bulk","node":306,"slots":452}
{"time":"2025-10-20T10:07:20.283060517Z","level":"INFO","msg":"Getting available slots","test":"bulk","node":307}
{"time":"2025-10-20T10:07:20.289880139Z","level":"INFO","msg":"Available slots","test":"bulk","node":307,"slots":1073}
{"time":"2025-10-20T10:08:52.728166327Z","level":"INFO","msg":"Extracted keyshares path for bulk keyshare","test":"bulk","module":"keys","path":"/app/shared/data/keyshares/keyshares-1760954932.json"}
{"time":"2025-10-20T10:08:55.708105589Z","level":"INFO","msg":"Allowance fetched successfully","test":"bulk","module":"contract","account":"0x5cc0dde14e7256340cc820415a6022a7d1c93a35","allowance":"9999999999999999999999562326194910796200000000"}
{"time":"2025-10-20T10:08:55.708157058Z","level":"INFO","msg":"Threshold value","test":"bulk","module":"contract","threshold":"9000000000000000000000"}
{"time":"2025-10-20T10:08:55.708169009Z","level":"INFO","msg":"Sufficient allowance","test":"bulk","module":"contract","allowance margin":"6561000000000000000000000000000000000000000000000000000000000000000000000000000000000000"}
{"time":"2025-10-20T10:08:55.816592588Z","level":"INFO","msg":"Bulk Registering validators","test":"bulk","module":"contract","tx_hash":"0x1dda1daa13f38bb9218739208ac31d4adb2f481da2bc84283f32fc7f4b249471"}
{"time":"2025-10-20T10:08:55.816640278Z","level":"INFO","msg":"Bulk registration transaction sent","test":"bulk","txHash":"0x1dda1daa13f38bb9218739208ac31d4adb2f481da2bc84283f32fc7f4b249471"}
{"time":"2025-10-20T10:08:55.820862416Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"bulk","module":"contract","attempt":1,"sleep":10,"tx_hash":"0x1dda1daa13f38bb9218739208ac31d4adb2f481da2bc84283f32fc7f4b249471"}
{"time":"2025-10-20T10:09:05.836064554Z","level":"INFO","msg":"Transaction receipt received","test":"bulk","module":"contract","tx_hash":"0x1dda1daa13f38bb9218739208ac31d4adb2f481da2bc84283f32fc7f4b249471"}
{"time":"2025-10-20T10:09:05.836147201Z","level":"INFO","msg":"✅ SUCCESS: Bulk Registration Successful","test":"bulk","blockHash":"0x5d895048fb90a6d8174ed2783b120deddd2cd603e0659aca37bd7f724094c3f1","Validators Amount":6}
{"time":"2025-10-20T10:09:05.851255301Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"bulk","module":"helpers","pubkey":"0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754","timeToSleep":10,"attempt":1}
{"time":"2025-10-20T10:09:15.866522783Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"bulk","module":"helpers","pubkey":"0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754","timeToSleep":20,"attempt":2}
{"time":"2025-10-20T10:09:35.907939016Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"bulk","module":"helpers","pubkey":"0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754","timeToSleep":40,"attempt":3}
{"time":"2025-10-20T10:10:15.942676171Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"bulk","module":"helpers","pubkey":"0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754","timeToSleep":80,"attempt":4}
{"time":"2025-10-20T10:11:35.962056881Z","level":"INFO","msg":"Successfully retrieved validator index from exporter","test":"bulk","module":"helpers","pubkey":"0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754"}
{"time":"2025-10-20T10:11:35.97456186Z","level":"INFO","msg":"Successfully retrieved validator index from exporter","test":"bulk","module":"helpers","pubkey":"0x84d7c46eb947f61c65a1c32971bbad4a1e832b1998675374fdebab49e6ab6d2cc0a7208acb3d176ac8b5df0cd3587f23"}
{"time":"2025-10-20T10:11:35.986530108Z","level":"INFO","msg":"Successfully retrieved validator index from exporter","test":"bulk","module":"helpers","pubkey":"0x86d6aa42d120173270727af286965c371f919f423558997f914b30b20bba45b041561650b531d386ed85f65828e0bde1"}
{"time":"2025-10-20T10:11:36.043371809Z","level":"INFO","msg":"Successfully retrieved validator index from exporter","test":"bulk","module":"helpers","pubkey":"0x87aa97b4388e95df2a4a06bf7e638fe3a86c097159bfe9e8f4483fc8028025fc9a69b0121edd7da55bbed0615610ed3c"}
{"time":"2025-10-20T10:11:36.094515096Z","level":"INFO","msg":"Successfully retrieved validator index from exporter","test":"bulk","module":"helpers","pubkey":"0x88e97e72ea3a862b12b44028e65f2af490b8cc19d4b6f1915f1f80325cff9f02d8f5c6fd79f127c5fde7a90fcaa43b8d"}
{"time":"2025-10-20T10:11:36.149888885Z","level":"INFO","msg":"Successfully retrieved validator index from exporter","test":"bulk","module":"helpers","pubkey":"0xa88938ed513411df0322826f6b017a2d1fb610c633e33ccad9dc1e46b274ad32a489f89e7b1b772ab518ba0cb089cee2"}
{"time":"2025-10-20T10:11:36.515704248Z","level":"INFO","msg":"Validator event validation successful","module":"main","module":"loki-validator","validated_validators":6,"event":"ValidatorAdded"}
{"time":"2025-10-20T10:11:36.515791192Z","level":"INFO","msg":"SSV node log validation successful","test":"bulk","indices":["1102420","1104199","1092007","1104251","1092460","1092461"],"validators":6,"validate-type":"registration"}
{"time":"2025-10-20T10:11:38.3673777Z","level":"INFO","msg":"Bulk Removing validators","test":"bulk","module":"contract","tx_hash":"0x96f89fd112371683822a45c6ae4257d8979618e6cf5d0c64d0c99479cc985e95"}
{"time":"2025-10-20T10:11:38.367462993Z","level":"INFO","msg":"Bulk removal transaction sent","test":"bulk","txHash":"0x96f89fd112371683822a45c6ae4257d8979618e6cf5d0c64d0c99479cc985e95"}
{"time":"2025-10-20T10:11:38.372477726Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"bulk","module":"contract","attempt":1,"sleep":10,"tx_hash":"0x96f89fd112371683822a45c6ae4257d8979618e6cf5d0c64d0c99479cc985e95"}
{"time":"2025-10-20T10:11:48.378038056Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"bulk","module":"contract","attempt":2,"sleep":20,"tx_hash":"0x96f89fd112371683822a45c6ae4257d8979618e6cf5d0c64d0c99479cc985e95"}
{"time":"2025-10-20T10:12:08.398704549Z","level":"INFO","msg":"Transaction receipt received","test":"bulk","module":"contract","tx_hash":"0x96f89fd112371683822a45c6ae4257d8979618e6cf5d0c64d0c99479cc985e95"}
{"time":"2025-10-20T10:12:08.398847874Z","level":"INFO","msg":"✅ SUCCESS: Bulk Removal Successful","test":"bulk","blockHash":"0x416673e6d508268b557b5ee9fded65b0472f458abf3ec166248b92942c81d3e1","Validators Amount":6}
{"time":"2025-10-20T10:12:08.548981753Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":1,"max_retries":5,"delay_seconds":30,"start_unix":1760944328,"end_unix":1760955128}
{"time":"2025-10-20T10:12:38.610266993Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":2,"max_retries":5,"delay_seconds":60,"start_unix":1760944328,"end_unix":1760955158}
{"time":"2025-10-20T10:13:38.612649306Z","level":"ERROR","msg":"request failed","module":"HTTPRequester","error":"Get \\"https://loki.ops.ssvlabsinternal.com/loki/api/v1/query_range?query=%7Bcontainer%3D~%22ssv-node-304%7Cssv-node-305%7Cssv-node-306%7Cssv-node-307%22%7D+%7C%3D+%220x96f89fd112371683822a45c6ae4257d8979618e6cf5d0c64d0c99479cc985e95%22&start=1760944328&end=1760955218&limit=1000\\": unexpected EOF","request attempt #":0}
{"time":"2025-10-20T10:13:38.744706698Z","level":"INFO","msg":"Validator event validation successful","module":"main","module":"loki-validator","validated_validators":6,"event":"ValidatorRemoved"}
{"time":"2025-10-20T10:13:38.744822323Z","level":"INFO","msg":"SSV node log validation successful","test":"bulk","indices":["1102420","1104199","1092007","1104251","1092460","1092461"],"validators":6,"validate-type":"removal"}
{"time":"2025-10-20T10:13:38.744903654Z","level":"INFO","msg":"✅ SUCCESS: Bulk Test Suite Completed Successfully","module":"main","register":{"Contract":true,"E2M":false,"Loki":true},"remove":{"Contract":true,"E2M":false,"Loki":true}}
{"time":"2025-10-20T10:13:38.74492429Z","level":"INFO","msg":"⏳ PROGRESS: Waiting for all tests to finish","module":"main"}
{"time":"2025-10-20T10:13:38.744969051Z","level":"INFO","msg":"✅ SUCCESS: All Test Suites Completed Successfully","module":"main","passed_suites":["bulk"],"skipped_suites":["isolated"]}
{"time":"2025-10-20T10:13:38.744991071Z","level":"INFO","msg":"🔑 KEY STEP: Starting resource teardown...","module":"main","usage_id":"70f185f53d419a8be2f77f9e8dbe339d"}
{"time":"2025-10-20T10:13:38.754467877Z","level":"INFO","msg":"🌐 Orchestrator API Client initialized","module":"main","endpoint":"http://aetheria-orchestrator.aetheria.svc:8080"}
{"time":"2025-10-20T10:13:38.7545638Z","level":"INFO","msg":"🧹 Requesting resource teardown","module":"main","usage_id":"70f185f53d419a8be2f77f9e8dbe339d"}
{"time":"2025-10-20T10:13:38.81199598Z","level":"INFO","msg":"✅ SUCCESS: Completed resource teardown successfully","module":"main","usage_id":"70f185f53d419a8be2f77f9e8dbe339d"}

2025-10-20T10:13:42Z The test run finished successfully

Result - 🟢 Success

yoaz-ssvlabs · 2025-10-20T10:50:00Z

/test tests=(sanity,e2m)

ssv-test-bot · 2025-10-20T10:50:04Z

🟢 Test run for PR #2433

Branch: ci-test-handler-testing2
Commit: 74b7038

Status

2025-10-20T10:50:04Z Starting test run
2025-10-20T10:50:05Z Test run submitted successfully for commit 74b7038
2025-10-20T10:50:06Z container image for commit 74b7038f0265e7f8da4425e9315fe4199c751b80 is build and present in container registry
2025-10-20T10:50:08Z automatically allocated node range cluster=308,309,310,311 for test run on network=hoodi_stage
2025-10-20T10:50:09Z deploying commit 74b7038f0265e7f8da4425e9315fe4199c751b80 to ssv nodes 308,309,310,311
2025-10-20T10:50:54Z started test executor with tests=(sanity,e2m) and network=hoodi_stage

Click to expand executor logs

═══════════════════════════════════════════════════════════════════════════════════════════════════════
                🧪  WELCOME TO Executor - SSV's Distributed Test Automation Execution Layer  🧪            

                                  ___ __  __ ___  ___  _   _  _____  ___   ___ 
                                 | __|\\ \\/ /| __|/ __|| | | ||_   _|/ _ \\ | _ \\
                                 | _|  >  < | _|| (__ | |_| |  | | | (_) ||   /
                                 |___|/_/\\_\\|___|\\___| \\___/   |_|  \\___/ |_|_\\
                                        

═══════════════════════════════════════════════════════════════════════════════════════════════════════
{"time":"2025-10-20T10:50:56.499214872Z","level":"INFO","msg":"🐳 Docker mode: Using environment variables for config generation"}
{"time":"2025-10-20T10:50:56.510552296Z","level":"INFO","msg":"🌐 Orchestrator API Client initialized","endpoint":"http://aetheria-orchestrator.aetheria.svc:8080"}
{"time":"2025-10-20T10:50:56.510655246Z","level":"INFO","msg":"📝 Requesting configuration generation from orchestrator API","network":"hoodi_stage","clusters":[308,309,310,311],"tests":"(sanity,e2m)"}
{"time":"2025-10-20T10:50:56.564918035Z","level":"INFO","msg":"✅ Configuration generated successfully via API","resource_usage_id":"501bdd2898fed713e7efd9ceb2e65a1f"}
{"time":"2025-10-20T10:50:56.566453527Z","level":"INFO","msg":"🚀 EXECUTOR Initialized Successfully"}
{"time":"2025-10-20T10:50:56.566536341Z","level":"INFO","msg":"⚙️ Test configuration loaded","network":"hoodi_stage","nodes":[308,309,310,311],"test_type":{"Sanity":{"ShouldRun":true,"E2M":true,"BulkAmount":0},"Bulk":{"ShouldRun":false,"E2M":false,"BulkAmount":0},"Isolated":{"Tests":null}}}
{"time":"2025-10-20T10:51:01.191598949Z","level":"INFO","msg":"Generated operator pubkeys file","module":"scanner","path":"/app/shared/data/operators/operator-pubkeys-hoodi_stage.json"}
{"time":"2025-10-20T10:51:01.191814268Z","level":"INFO","msg":"Registered parser","module":"main","component":"parser-client","parser_type":"consensus"}
{"time":"2025-10-20T10:51:01.193071673Z","level":"INFO","msg":"Registered parser","module":"main","component":"parser-client","parser_type":"flow-emits"}
{"time":"2025-10-20T10:51:01.193111836Z","level":"INFO","msg":"Validation client initialized","module":"main","loki_enabled":true,"e2m_enabled":true}
{"time":"2025-10-20T10:51:01.193159384Z","level":"INFO","msg":"🔑 KEY STEP: Starting Isolated Tests Concurrently if exists...","module":"main"}
{"time":"2025-10-20T10:51:01.193215649Z","level":"INFO","msg":"🔑 KEY STEP: Starting Sanity Test Suite","module":"main"}
{"time":"2025-10-20T10:51:01.193269155Z","level":"INFO","msg":"No isolated tests configured, skipping","module":"main"}
{"time":"2025-10-20T10:51:01.258495568Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":308}
{"time":"2025-10-20T10:51:01.269635121Z","level":"INFO","msg":"Available slots","test":"sanity","node":308,"slots":3000}
{"time":"2025-10-20T10:51:01.269688947Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":309}
{"time":"2025-10-20T10:51:01.277046037Z","level":"INFO","msg":"Available slots","test":"sanity","node":309,"slots":756}
{"time":"2025-10-20T10:51:01.277146155Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":310}
{"time":"2025-10-20T10:51:01.284957694Z","level":"INFO","msg":"Available slots","test":"sanity","node":310,"slots":452}
{"time":"2025-10-20T10:51:01.285105563Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":311}
{"time":"2025-10-20T10:51:01.304963395Z","level":"INFO","msg":"Available slots","test":"sanity","node":311,"slots":1073}
{"time":"2025-10-20T10:52:24.708802515Z","level":"INFO","msg":"Extracted keyshares path","test":"sanity","module":"keys","path":"/app/shared/data/keyshares/0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754.json"}
{"time":"2025-10-20T10:52:24.70889698Z","level":"INFO","msg":"Key shares file generated successfully","test":"sanity","keyshare_path":"/app/shared/data/keyshares/0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754.json"}
{"time":"2025-10-20T10:52:26.411240896Z","level":"INFO","msg":"Allowance fetched successfully","test":"sanity","module":"contract","account":"0xceefd323dd28a8d9514eddfec45a6c81800a7d49","allowance":"9999999999999999999999813819028857427200000000"}
{"time":"2025-10-20T10:52:26.411345868Z","level":"INFO","msg":"Threshold value","test":"sanity","module":"contract","threshold":"9000000000000000000000"}
{"time":"2025-10-20T10:52:26.411378515Z","level":"INFO","msg":"Sufficient allowance","test":"sanity","module":"contract","allowance margin":"6561000000000000000000000000000000000000000000000000000000000000000000000000000000000000"}
{"time":"2025-10-20T10:52:26.414972223Z","level":"INFO","msg":"Allowance fetched successfully","test":"sanity","module":"contract","account":"0xceefd323dd28a8d9514eddfec45a6c81800a7d49","allowance":"9999999999999999999999813819028857427200000000"}
{"time":"2025-10-20T10:52:26.415071275Z","level":"INFO","msg":"Allowance","test":"sanity","module":"contract","allowance":"9999999999999999999999813819028857427200000000"}
{"time":"2025-10-20T10:52:26.485836695Z","level":"INFO","msg":"Registering validator","test":"sanity","module":"contract","tx_hash":"0x0353b4c0807c32ec92053226c253b0106e089358db6621b4dbccd19ffb7764ae"}
{"time":"2025-10-20T10:52:26.489294736Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0x0353b4c0807c32ec92053226c253b0106e089358db6621b4dbccd19ffb7764ae"}
{"time":"2025-10-20T10:52:36.492664576Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":2,"sleep":20,"tx_hash":"0x0353b4c0807c32ec92053226c253b0106e089358db6621b4dbccd19ffb7764ae"}
{"time":"2025-10-20T10:52:56.511549798Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0x0353b4c0807c32ec92053226c253b0106e089358db6621b4dbccd19ffb7764ae"}
{"time":"2025-10-20T10:52:56.511632868Z","level":"INFO","msg":"✅ SUCCESS: Contract Register Successful","test":"sanity","txHash":"0x0353b4c0807c32ec92053226c253b0106e089358db6621b4dbccd19ffb7764ae","block_hash":"0x52ac1af3760a635767ef3b295148e7d9dfc06ca13f1ad81d154a8530192ed197"}
{"time":"2025-10-20T10:52:56.525906862Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"sanity","module":"helpers","pubkey":"0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754","timeToSleep":10,"attempt":1}
{"time":"2025-10-20T10:53:06.539029538Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"sanity","module":"helpers","pubkey":"0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754","timeToSleep":20,"attempt":2}
{"time":"2025-10-20T10:53:26.564095073Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"sanity","module":"helpers","pubkey":"0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754","timeToSleep":40,"attempt":3}
{"time":"2025-10-20T10:54:06.601055366Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"sanity","module":"helpers","pubkey":"0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754","timeToSleep":80,"attempt":4}
{"time":"2025-10-20T10:55:26.617356312Z","level":"INFO","msg":"Successfully retrieved validator index from exporter","test":"sanity","module":"helpers","pubkey":"0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754"}
{"time":"2025-10-20T10:55:26.637307071Z","level":"INFO","msg":"Registration epoch","module":"main","module":"e2m-validator","epoch":48812,"indices":"1102420"}
{"time":"2025-10-20T10:55:26.953363893Z","level":"INFO","msg":"Validator event validation successful","module":"main","module":"loki-validator","validated_validators":1,"event":"ValidatorAdded"}
{"time":"2025-10-20T10:55:26.953450149Z","level":"INFO","msg":"SSV node log validation successful","test":"sanity","indices":["1102420"],"validators":1,"validate-type":"registration"}
{"time":"2025-10-20T10:55:41.680717954Z","level":"INFO","msg":"Validator did not attest","module":"main","module":"e2m-validator","epoch":48812,"indices":["1102420"]}
{"time":"2025-10-20T10:59:42.512796184Z","level":"INFO","msg":"Validator did not attest","module":"main","module":"e2m-validator","epoch":48813,"indices":["1102420"]}
{"time":"2025-10-20T11:06:13.829788083Z","level":"INFO","msg":"E2M registration validation successful - validators are attesting","module":"main","module":"e2m-validator","validation_type":"registration","indices":["1102420"],"registered_epoch":48812,"validated_epoch":48814,"epochs_waited":2}
{"time":"2025-10-20T11:06:13.82985271Z","level":"INFO","msg":"E2M validation successful","test":"sanity","indices":["1102420"],"validate-type":"registration"}
{"time":"2025-10-20T11:06:13.830102581Z","level":"INFO","msg":"Starting consensus validation with epoch iteration logic","module":"main","module":"loki-validator","validator_index":"1102420","registered_epoch":48812,"max_epoch_offset":4}
{"time":"2025-10-20T11:06:13.830246661Z","level":"INFO","msg":"Checking epoch for duties","module":"main","module":"loki-validator","epoch_offset":0,"current_epoch":"48812","validator_index":"1102420"}
{"time":"2025-10-20T11:06:13.855424809Z","level":"WARN","msg":"No attesters found in duties response","module":"main","module":"loki-validator","epoch_offset":0,"epoch":"48812"}
{"time":"2025-10-20T11:06:13.855497797Z","level":"INFO","msg":"Checking epoch for duties","module":"main","module":"loki-validator","epoch_offset":1,"current_epoch":"48813","validator_index":"1102420"}
{"time":"2025-10-20T11:06:13.864452332Z","level":"WARN","msg":"No attesters found in duties response","module":"main","module":"loki-validator","epoch_offset":1,"epoch":"48813"}
{"time":"2025-10-20T11:06:13.864551246Z","level":"INFO","msg":"Checking epoch for duties","module":"main","module":"loki-validator","epoch_offset":2,"current_epoch":"48814","validator_index":"1102420"}
{"time":"2025-10-20T11:06:13.874158625Z","level":"INFO","msg":"Successfully found duties, building duty ID for consensus validation","module":"main","module":"loki-validator","duty_id":"COMMITTEE-308_309_310_311-e48814-s1562077","final_epoch":"48814","slot":"1562077"}
{"time":"2025-10-20T11:06:13.997274939Z","level":"INFO","msg":"Consensus leader validation passed","module":"main","module":"loki-validator","slot":1562077,"leader":309}
{"time":"2025-10-20T11:06:13.997389712Z","level":"INFO","msg":"Consensus timing validation comparison","module":"main","module":"loki-validator","actual_time_diff":45450000,"expected_consensus_time":38000000,"lower_bound":26600000,"upper_bound":49400000,"margin_percent":30}
{"time":"2025-10-20T11:06:13.997410418Z","level":"INFO","msg":"Consensus timing validation passed - within acceptable range","module":"main","module":"loki-validator"}
{"time":"2025-10-20T11:06:13.997430633Z","level":"INFO","msg":"Consensus time validation completed successfully","module":"main","module":"loki-validator","final_epoch":"48814"}
{"time":"2025-10-20T11:06:13.99745378Z","level":"INFO","msg":"Consensus timing validation successful","test":"sanity","indices":["1102420"],"epoch":48812}
{"time":"2025-10-20T11:06:15.815035541Z","level":"INFO","msg":"Liquidating cluster","test":"sanity","module":"contract","tx_hash":"0x06cc4ef58fb6c6cb7a23c4db52bf621577ab7f8eb25550cb9a9297c4d47997e1"}
{"time":"2025-10-20T11:06:15.818378513Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0x06cc4ef58fb6c6cb7a23c4db52bf621577ab7f8eb25550cb9a9297c4d47997e1"}
{"time":"2025-10-20T11:06:25.829131468Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":2,"sleep":20,"tx_hash":"0x06cc4ef58fb6c6cb7a23c4db52bf621577ab7f8eb25550cb9a9297c4d47997e1"}
{"time":"2025-10-20T11:06:45.853990808Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0x06cc4ef58fb6c6cb7a23c4db52bf621577ab7f8eb25550cb9a9297c4d47997e1"}
{"time":"2025-10-20T11:06:45.854132554Z","level":"INFO","msg":"✅ SUCCESS: Cluster Liquidation Successful","test":"sanity","txHash":"0x06cc4ef58fb6c6cb7a23c4db52bf621577ab7f8eb25550cb9a9297c4d47997e1","block_hash":"0x215cdacbaf6d36edbf5761775b77d5d14be7685024442e30a27b9c3d5332bbef"}
{"time":"2025-10-20T11:06:45.854340984Z","level":"INFO","msg":"Querying Loki for transaction hash validation","module":"main","module":"loki-validator","query":"{container=~\\"ssv-node-308|ssv-node-309|ssv-node-310|ssv-node-311\\"} |= \\"0x06cc4ef58fb6c6cb7a23c4db52bf621577ab7f8eb25550cb9a9297c4d47997e1\\"","tx_hash":"0x06cc4ef58fb6c6cb7a23c4db52bf621577ab7f8eb25550cb9a9297c4d47997e1"}
{"time":"2025-10-20T11:06:45.863424166Z","level":"INFO","msg":"Liquidation epoch","module":"main","module":"e2m-validator","epoch":48815,"indices":["1102420"]}
{"time":"2025-10-20T11:06:45.990737019Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":1,"max_retries":5,"delay_seconds":30,"start_unix":1760947605,"end_unix":1760958405}
{"time":"2025-10-20T11:07:16.086587032Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":2,"max_retries":5,"delay_seconds":60,"start_unix":1760947605,"end_unix":1760958435}
{"time":"2025-10-20T11:08:16.095724834Z","level":"ERROR","msg":"request failed","module":"HTTPRequester","error":"Get \\"https://loki.ops.ssvlabsinternal.com/loki/api/v1/query_range?query=%7Bcontainer%3D~%22ssv-node-308%7Cssv-node-309%7Cssv-node-310%7Cssv-node-311%22%7D+%7C%3D+%220x06cc4ef58fb6c6cb7a23c4db52bf621577ab7f8eb25550cb9a9297c4d47997e1%22+%7C+json+%7C+line_format+%22%7B%7B.container%7D%7D%3A%7B%7B.msg%7D%7D%22&start=1760947605&end=1760958496&limit=1000\\": unexpected EOF","request attempt #":0}
{"time":"2025-10-20T11:08:16.258507634Z","level":"INFO","msg":"Event validation successful","module":"main","module":"loki-validator","valid_entries":4,"event":"ClusterLiquidated"}
{"time":"2025-10-20T11:08:16.258613599Z","level":"INFO","msg":"SSV node log validation successful","test":"sanity","indices":["1102420"],"validators":1,"validate-type":"liquidation"}
{"time":"2025-10-20T11:12:32.124740574Z","level":"INFO","msg":"Validator was not liquidated","module":"main","module":"e2m-validator","epoch":48815,"indices":["1102420"]}
{"time":"2025-10-20T11:19:03.448686465Z","level":"INFO","msg":"E2M liquidation validation successful - validators are liquidated","module":"main","module":"e2m-validator","validation_type":"liquidation","indices":["1102420"],"liquidated_epoch":48815,"validated_epoch":48816,"epochs_waited":1,"assigned":0,"missed":0}
{"time":"2025-10-20T11:19:03.448791909Z","level":"INFO","msg":"E2M validation successful","test":"sanity","indices":["1102420"],"validate-type":"liquidation"}
{"time":"2025-10-20T11:19:05.356120723Z","level":"INFO","msg":"Allowance fetched successfully","test":"sanity","module":"contract","account":"0xceefd323dd28a8d9514eddfec45a6c81800a7d49","allowance":"9999999999999999999999813819028857427200000000"}
{"time":"2025-10-20T11:19:05.356255026Z","level":"INFO","msg":"Threshold value","test":"sanity","module":"contract","threshold":"9000000000000000000000"}
{"time":"2025-10-20T11:19:05.356301493Z","level":"INFO","msg":"Sufficient allowance","test":"sanity","module":"contract","allowance margin":"6561000000000000000000000000000000000000000000000000000000000000000000000000000000000000"}
{"time":"2025-10-20T11:19:05.486412443Z","level":"INFO","msg":"Reactivating cluster","test":"sanity","module":"contract","tx_hash":"0x79af240403d90c126d03e9a1398bfdc43fabf3881e6dca239d025594e7e1f194"}
{"time":"2025-10-20T11:19:05.49030936Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0x79af240403d90c126d03e9a1398bfdc43fabf3881e6dca239d025594e7e1f194"}
{"time":"2025-10-20T11:19:15.502704204Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0x79af240403d90c126d03e9a1398bfdc43fabf3881e6dca239d025594e7e1f194"}
{"time":"2025-10-20T11:19:15.502766016Z","level":"INFO","msg":"✅ SUCCESS: Cluster Reactivation Successful","test":"sanity","txHash":"0x79af240403d90c126d03e9a1398bfdc43fabf3881e6dca239d025594e7e1f194","block_hash":"0x1a3732cdcdc11af1163ca9f4fc5d74b0f82a7bbe9d55d8e2c4a0a09e1fe9561f"}
{"time":"2025-10-20T11:19:15.502889848Z","level":"INFO","msg":"Querying Loki for transaction hash validation","module":"main","module":"loki-validator","query":"{container=~\\"ssv-node-308|ssv-node-309|ssv-node-310|ssv-node-311\\"} |= \\"0x79af240403d90c126d03e9a1398bfdc43fabf3881e6dca239d025594e7e1f194\\"","tx_hash":"0x79af240403d90c126d03e9a1398bfdc43fabf3881e6dca239d025594e7e1f194"}
{"time":"2025-10-20T11:19:15.502918153Z","level":"INFO","msg":"Validating reactivation - checking that validators resume attesting","module":"main","module":"e2m-validator","indices":["1102420"],"block_hash":"0x1a3732cdcdc11af1163ca9f4fc5d74b0f82a7bbe9d55d8e2c4a0a09e1fe9561f"}
{"time":"2025-10-20T11:19:15.512623332Z","level":"INFO","msg":"waiting for block to be found","module":"main","module":"e2m-validator","module":"e2m"}
{"time":"2025-10-20T11:19:16.070947729Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":1,"max_retries":5,"delay_seconds":30,"start_unix":1760948355,"end_unix":1760959155}
{"time":"2025-10-20T11:19:35.534208139Z","level":"INFO","msg":"waiting for block to be found","module":"main","module":"e2m-validator","module":"e2m"}
{"time":"2025-10-20T11:19:55.548851654Z","level":"INFO","msg":"Registration epoch","module":"main","module":"e2m-validator","epoch":48817,"indices":"1102420"}
{"time":"2025-10-20T11:19:56.078563057Z","level":"ERROR","msg":"request failed","module":"HTTPRequester","error":"Get \\"https://loki.ops.ssvlabsinternal.com/loki/api/v1/query_range?query=%7Bcontainer%3D~%22ssv-node-308%7Cssv-node-309%7Cssv-node-310%7Cssv-node-311%22%7D+%7C%3D+%220x79af240403d90c126d03e9a1398bfdc43fabf3881e6dca239d025594e7e1f194%22+%7C+json+%7C+line_format+%22%7B%7B.container%7D%7D%3A%7B%7B.msg%7D%7D%22&start=1760948355&end=1760959186&limit=1000\\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)","request attempt #":0}
{"time":"2025-10-20T11:20:06.079459557Z","level":"ERROR","msg":"request failed","module":"HTTPRequester","error":"Get \\"https://loki.ops.ssvlabsinternal.com/loki/api/v1/query_range?query=%7Bcontainer%3D~%22ssv-node-308%7Cssv-node-309%7Cssv-node-310%7Cssv-node-311%22%7D+%7C%3D+%220x79af240403d90c126d03e9a1398bfdc43fabf3881e6dca239d025594e7e1f194%22+%7C+json+%7C+line_format+%22%7B%7B.container%7D%7D%3A%7B%7B.msg%7D%7D%22&start=1760948355&end=1760959186&limit=1000\\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)","request attempt #":1}
{"time":"2025-10-20T11:20:09.527250533Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":2,"max_retries":5,"delay_seconds":60,"start_unix":1760948355,"end_unix":1760959186}
{"time":"2025-10-20T11:20:10.589730667Z","level":"INFO","msg":"Validator did not attest","module":"main","module":"e2m-validator","epoch":48816,"indices":["1102420"]}
{"time":"2025-10-20T11:21:09.825797016Z","level":"INFO","msg":"Event validation successful","module":"main","module":"loki-validator","valid_entries":4,"event":"ClusterReactivated"}
{"time":"2025-10-20T11:21:09.825875768Z","level":"INFO","msg":"SSV node log validation successful","test":"sanity","indices":["1102420"],"validators":1,"validate-type":"reactivation"}
{"time":"2025-10-20T11:25:26.683450854Z","level":"INFO","msg":"Validator did not attest","module":"main","module":"e2m-validator","epoch":48817,"indices":["1102420"]}
{"time":"2025-10-20T11:31:57.944380007Z","level":"INFO","msg":"E2M registration validation successful - validators are attesting","module":"main","module":"e2m-validator","validation_type":"registration","indices":["1102420"],"registered_epoch":48817,"validated_epoch":48818,"epochs_waited":1}
{"time":"2025-10-20T11:31:57.944834995Z","level":"INFO","msg":"E2M validation successful","test":"sanity","indices":["1102420"],"validate-type":"reactivation"}
{"time":"2025-10-20T11:31:57.944503171Z","level":"INFO","msg":"Starting consensus validation with epoch iteration logic","module":"main","module":"loki-validator","validator_index":"1102420","registered_epoch":48817,"max_epoch_offset":4}
{"time":"2025-10-20T11:31:57.945008693Z","level":"INFO","msg":"Checking epoch for duties","module":"main","module":"loki-validator","epoch_offset":0,"current_epoch":"48817","validator_index":"1102420"}
{"time":"2025-10-20T11:31:57.957523477Z","level":"WARN","msg":"No attesters found in duties response","module":"main","module":"loki-validator","epoch_offset":0,"epoch":"48817"}
{"time":"2025-10-20T11:31:57.957698077Z","level":"INFO","msg":"Checking epoch for duties","module":"main","module":"loki-validator","epoch_offset":1,"current_epoch":"48818","validator_index":"1102420"}
{"time":"2025-10-20T11:31:57.969950439Z","level":"INFO","msg":"Successfully found duties, building duty ID for consensus validation","module":"main","module":"loki-validator","duty_id":"COMMITTEE-308_309_310_311-e48818-s1562188","final_epoch":"48818","slot":"1562188"}
{"time":"2025-10-20T11:31:58.077116483Z","level":"INFO","msg":"Consensus leader validation passed","module":"main","module":"loki-validator","slot":1562188,"leader":308}
{"time":"2025-10-20T11:31:58.077315354Z","level":"INFO","msg":"Consensus timing validation comparison","module":"main","module":"loki-validator","actual_time_diff":36213000,"expected_consensus_time":20000000,"lower_bound":14000000,"upper_bound":26000000,"margin_percent":30}
{"time":"2025-10-20T11:31:58.077417128Z","level":"ERROR","msg":"Consensus timing validation failed","test":"sanity","indices":["1102420"],"epoch":48817,"error":"consensus timing outside acceptable range: actual=36.213ms, expected=20ms (±30%), bounds=[14ms, 26ms]"}
{"time":"2025-10-20T11:31:59.786451422Z","level":"INFO","msg":"Allowance fetched successfully","test":"sanity","module":"contract","account":"0xceefd323dd28a8d9514eddfec45a6c81800a7d49","allowance":"9999999999999999999999804819028857427200000000"}
{"time":"2025-10-20T11:31:59.786495716Z","level":"INFO","msg":"Threshold value","test":"sanity","module":"contract","threshold":"9000000000000000000000"}
{"time":"2025-10-20T11:31:59.786517748Z","level":"INFO","msg":"Sufficient allowance","test":"sanity","module":"contract","allowance margin":"6561000000000000000000000000000000000000000000000000000000000000000000000000000000000000"}
{"time":"2025-10-20T11:31:59.875372506Z","level":"INFO","msg":"Removing validator","test":"sanity","module":"contract","tx_hash":"0xb481ffc7998405ddad302ffecbabd56898d443973f2c481625bbc706ed44d51f"}
{"time":"2025-10-20T11:31:59.88479143Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0xb481ffc7998405ddad302ffecbabd56898d443973f2c481625bbc706ed44d51f"}
{"time":"2025-10-20T11:32:09.894340779Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":2,"sleep":20,"tx_hash":"0xb481ffc7998405ddad302ffecbabd56898d443973f2c481625bbc706ed44d51f"}
{"time":"2025-10-20T11:32:29.903884475Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0xb481ffc7998405ddad302ffecbabd56898d443973f2c481625bbc706ed44d51f"}
{"time":"2025-10-20T11:32:29.904066782Z","level":"INFO","msg":"✅ SUCCESS: Contract Removal Successful","test":"sanity","blockHash":"0x0aabe57e178b2cdf05a90aed0bd65eb86ee59198b277da2483a80fdd05bbc3e8"}
{"time":"2025-10-20T11:32:29.911257186Z","level":"INFO","msg":"Removal epoch","module":"main","module":"e2m-validator","epoch":48819,"indices":"1102420"}
{"time":"2025-10-20T11:32:30.079012937Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":1,"max_retries":5,"delay_seconds":30,"start_unix":1760949149,"end_unix":1760959949}
{"time":"2025-10-20T11:33:00.155218987Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":2,"max_retries":5,"delay_seconds":60,"start_unix":1760949149,"end_unix":1760959980}
{"time":"2025-10-20T11:34:00.219715787Z","level":"INFO","msg":"Validator event validation successful","module":"main","module":"loki-validator","validated_validators":1,"event":"ValidatorRemoved"}
{"time":"2025-10-20T11:34:00.219790447Z","level":"INFO","msg":"SSV node log validation successful","test":"sanity","indices":["1102420"],"validators":1,"validate-type":"removal"}
{"time":"2025-10-20T11:38:16.047730804Z","level":"INFO","msg":"Validator was not removed","module":"main","module":"e2m-validator","epoch":48819,"indices":["1102420"]}
{"time":"2025-10-20T11:44:32.212068143Z","level":"INFO","msg":"E2M removal validation successful - validators stopped attesting","module":"main","module":"e2m-validator","validation_type":"removal","indices":["1102420"],"removed_epoch":48819,"validated_epoch":48820,"epochs_waited":1}
{"time":"2025-10-20T11:44:32.212183395Z","level":"INFO","msg":"E2M validation successful","test":"sanity","indices":["1102420"],"validate-type":"removal"}
{"time":"2025-10-20T11:44:32.212277115Z","level":"INFO","msg":"✅ SUCCESS: Sanity Test Suite Completed Successfully","module":"main","register":{"Contract":true,"E2M":true,"Loki":true},"liquidate":{"Contract":true,"E2M":true,"Loki":true},"reactivate":{"Contract":true,"E2M":true,"Loki":false},"remove":{"Contract":true,"E2M":true,"Loki":true}}
{"time":"2025-10-20T11:44:32.212294859Z","level":"INFO","msg":"⏳ PROGRESS: Waiting for all tests to finish","module":"main"}
{"time":"2025-10-20T11:44:32.212328271Z","level":"INFO","msg":"✅ SUCCESS: All Test Suites Completed Successfully","module":"main","passed_suites":["sanity"],"skipped_suites":["isolated"]}
{"time":"2025-10-20T11:44:32.21233801Z","level":"INFO","msg":"🔑 KEY STEP: Starting resource teardown...","module":"main","usage_id":"501bdd2898fed713e7efd9ceb2e65a1f"}
{"time":"2025-10-20T11:44:32.222674685Z","level":"INFO","msg":"🌐 Orchestrator API Client initialized","module":"main","endpoint":"http://aetheria-orchestrator.aetheria.svc:8080"}
{"time":"2025-10-20T11:44:32.22282076Z","level":"INFO","msg":"🧹 Requesting resource teardown","module":"main","usage_id":"501bdd2898fed713e7efd9ceb2e65a1f"}
{"time":"2025-10-20T11:44:32.260157693Z","level":"INFO","msg":"✅ SUCCESS: Completed resource teardown successfully","module":"main","usage_id":"501bdd2898fed713e7efd9ceb2e65a1f"}

2025-10-20T11:44:36Z The test run finished successfully

Result - 🟢 Success

yoaz-ssvlabs · 2025-10-22T07:33:06Z

/test tests=(sanity,e2m)

ssv-test-bot · 2025-10-22T07:33:10Z

🟢 Test run for PR #2433

Branch: ci-test-handler-testing2
Commit: 74b7038

Status

2025-10-22T07:33:10Z Starting test run
2025-10-22T07:33:11Z Test run submitted successfully for commit 74b7038
2025-10-22T07:33:12Z container image for commit 74b7038f0265e7f8da4425e9315fe4199c751b80 is build and present in container registry
2025-10-22T07:33:13Z automatically allocated node range cluster=304,305,306,307 for test run on network=hoodi_stage
2025-10-22T07:33:15Z deploying commit 74b7038f0265e7f8da4425e9315fe4199c751b80 to ssv nodes 304,305,306,307
2025-10-22T07:34:00Z commit 74b7038f0265e7f8da4425e9315fe4199c751b80 is deployed to ssv nodes 304,305,306,307

Click to expand executor logs

═══════════════════════════════════════════════════════════════════════════════════════════════════════
                🧪  WELCOME TO Executor - SSV's Distributed Test Automation Execution Layer  🧪            

                                  ___ __  __ ___  ___  _   _  _____  ___   ___ 
                                 | __|\\ \\/ /| __|/ __|| | | ||_   _|/ _ \\ | _ \\
                                 | _|  >  < | _|| (__ | |_| |  | | | (_) ||   /
                                 |___|/_/\\_\\|___|\\___| \\___/   |_|  \\___/ |_|_\\
                                        

═══════════════════════════════════════════════════════════════════════════════════════════════════════
{"time":"2025-10-22T07:34:48.484870608Z","level":"INFO","msg":"🐳 Docker mode: Using environment variables for config generation"}
{"time":"2025-10-22T07:34:48.502428993Z","level":"INFO","msg":"🌐 Orchestrator API Client initialized","endpoint":"http://aetheria-orchestrator.aetheria.svc:8080"}
{"time":"2025-10-22T07:34:48.502552625Z","level":"INFO","msg":"📝 Requesting configuration generation from orchestrator API","network":"hoodi_stage","clusters":[304,305,306,307],"tests":"(sanity,e2m)"}
{"time":"2025-10-22T07:34:48.626214989Z","level":"INFO","msg":"✅ Configuration generated successfully via API","resource_usage_id":"e3df1702f722737a30fb0b0bdc1f80eb"}
{"time":"2025-10-22T07:34:48.627090068Z","level":"INFO","msg":"🚀 EXECUTOR Initialized Successfully"}
{"time":"2025-10-22T07:34:48.627138659Z","level":"INFO","msg":"⚙️ Test configuration loaded","network":"hoodi_stage","nodes":[304,305,306,307],"test_type":{"Sanity":{"ShouldRun":true,"E2M":true,"BulkAmount":0},"Bulk":{"ShouldRun":false,"E2M":false,"BulkAmount":0},"Isolated":{"Tests":null}}}
{"time":"2025-10-22T07:34:53.6442727Z","level":"INFO","msg":"Generated operator pubkeys file","module":"scanner","path":"/app/shared/data/operators/operator-pubkeys-hoodi_stage.json"}
{"time":"2025-10-22T07:34:53.644497062Z","level":"INFO","msg":"Registered parser","module":"main","component":"parser-client","parser_type":"consensus"}
{"time":"2025-10-22T07:34:53.644557283Z","level":"INFO","msg":"Registered parser","module":"main","component":"parser-client","parser_type":"flow-emits"}
{"time":"2025-10-22T07:34:53.644698617Z","level":"INFO","msg":"Validation client initialized","module":"main","loki_enabled":true,"e2m_enabled":true}
{"time":"2025-10-22T07:34:53.644756933Z","level":"INFO","msg":"🔑 KEY STEP: Starting Isolated Tests Concurrently if exists...","module":"main"}
{"time":"2025-10-22T07:34:53.644845772Z","level":"INFO","msg":"🔑 KEY STEP: Starting Sanity Test Suite","module":"main"}
{"time":"2025-10-22T07:34:53.645094924Z","level":"INFO","msg":"No isolated tests configured, skipping","module":"main"}
{"time":"2025-10-22T07:34:53.698753992Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":304}
{"time":"2025-10-22T07:34:53.710646231Z","level":"INFO","msg":"Available slots","test":"sanity","node":304,"slots":3000}
{"time":"2025-10-22T07:34:53.71165214Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":305}
{"time":"2025-10-22T07:34:53.718019816Z","level":"INFO","msg":"Available slots","test":"sanity","node":305,"slots":755}
{"time":"2025-10-22T07:34:53.718107047Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":306}
{"time":"2025-10-22T07:34:53.724244384Z","level":"INFO","msg":"Available slots","test":"sanity","node":306,"slots":451}
{"time":"2025-10-22T07:34:53.724318441Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":307}
{"time":"2025-10-22T07:34:53.730801885Z","level":"INFO","msg":"Available slots","test":"sanity","node":307,"slots":1072}
{"time":"2025-10-22T07:37:02.025808753Z","level":"INFO","msg":"Extracted keyshares path","test":"sanity","module":"keys","path":"/app/shared/data/keyshares/0x87aa97b4388e95df2a4a06bf7e638fe3a86c097159bfe9e8f4483fc8028025fc9a69b0121edd7da55bbed0615610ed3c.json"}
{"time":"2025-10-22T07:37:02.026758924Z","level":"INFO","msg":"Key shares file generated successfully","test":"sanity","keyshare_path":"/app/shared/data/keyshares/0x87aa97b4388e95df2a4a06bf7e638fe3a86c097159bfe9e8f4483fc8028025fc9a69b0121edd7da55bbed0615610ed3c.json"}
{"time":"2025-10-22T07:37:03.878917652Z","level":"INFO","msg":"Allowance fetched successfully","test":"sanity","module":"contract","account":"0x5cc0dde14e7256340cc820415a6022a7d1c93a35","allowance":"9999999999999999999999562326194910796200000000"}
{"time":"2025-10-22T07:37:03.879024701Z","level":"INFO","msg":"Threshold value","test":"sanity","module":"contract","threshold":"9000000000000000000000"}
{"time":"2025-10-22T07:37:03.879051711Z","level":"INFO","msg":"Sufficient allowance","test":"sanity","module":"contract","allowance margin":"6561000000000000000000000000000000000000000000000000000000000000000000000000000000000000"}
{"time":"2025-10-22T07:37:03.885665094Z","level":"INFO","msg":"Allowance fetched successfully","test":"sanity","module":"contract","account":"0x5cc0dde14e7256340cc820415a6022a7d1c93a35","allowance":"9999999999999999999999562326194910796200000000"}
{"time":"2025-10-22T07:37:03.885770592Z","level":"INFO","msg":"Allowance","test":"sanity","module":"contract","allowance":"9999999999999999999999562326194910796200000000"}
{"time":"2025-10-22T07:37:03.985083072Z","level":"INFO","msg":"Registering validator","test":"sanity","module":"contract","tx_hash":"0x74f43e693039405bdb145368a58611034a8479e2ef29a647c6a709dc5ce6469f"}
{"time":"2025-10-22T07:37:03.990002029Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0x74f43e693039405bdb145368a58611034a8479e2ef29a647c6a709dc5ce6469f"}
{"time":"2025-10-22T07:37:14.00152915Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0x74f43e693039405bdb145368a58611034a8479e2ef29a647c6a709dc5ce6469f"}
{"time":"2025-10-22T07:37:14.001680826Z","level":"INFO","msg":"✅ SUCCESS: Contract Register Successful","test":"sanity","txHash":"0x74f43e693039405bdb145368a58611034a8479e2ef29a647c6a709dc5ce6469f","block_hash":"0x9fb2fc15f78c79e837fe743d1dc5aae73b56a725e8f7fb3ecaf93ff152c0d4e0"}
{"time":"2025-10-22T07:37:14.020615718Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"sanity","module":"helpers","pubkey":"0x87aa97b4388e95df2a4a06bf7e638fe3a86c097159bfe9e8f4483fc8028025fc9a69b0121edd7da55bbed0615610ed3c","timeToSleep":10,"attempt":1}
{"time":"2025-10-22T07:37:24.051251972Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"sanity","module":"helpers","pubkey":"0x87aa97b4388e95df2a4a06bf7e638fe3a86c097159bfe9e8f4483fc8028025fc9a69b0121edd7da55bbed0615610ed3c","timeToSleep":20,"attempt":2}
{"time":"2025-10-22T07:37:44.066194814Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"sanity","module":"helpers","pubkey":"0x87aa97b4388e95df2a4a06bf7e638fe3a86c097159bfe9e8f4483fc8028025fc9a69b0121edd7da55bbed0615610ed3c","timeToSleep":40,"attempt":3}
{"time":"2025-10-22T07:38:24.180045904Z","level":"INFO","msg":"Failed to get data for validator from exporter - retrying...","test":"sanity","module":"helpers","pubkey":"0x87aa97b4388e95df2a4a06bf7e638fe3a86c097159bfe9e8f4483fc8028025fc9a69b0121edd7da55bbed0615610ed3c","timeToSleep":80,"attempt":4}
{"time":"2025-10-22T07:39:44.19737674Z","level":"INFO","msg":"Successfully retrieved validator index from exporter","test":"sanity","module":"helpers","pubkey":"0x87aa97b4388e95df2a4a06bf7e638fe3a86c097159bfe9e8f4483fc8028025fc9a69b0121edd7da55bbed0615610ed3c"}
{"time":"2025-10-22T07:39:44.216787298Z","level":"INFO","msg":"Registration epoch","module":"main","module":"e2m-validator","epoch":49232,"indices":"1104251"}
{"time":"2025-10-22T07:39:44.462203677Z","level":"INFO","msg":"Validator event validation successful","module":"main","module":"loki-validator","validated_validators":1,"event":"ValidatorAdded"}
{"time":"2025-10-22T07:39:44.462323611Z","level":"INFO","msg":"SSV node log validation successful","test":"sanity","indices":["1104251"],"validators":1,"validate-type":"registration"}
{"time":"2025-10-22T07:39:59.271342766Z","level":"INFO","msg":"Validator did not attest","module":"main","module":"e2m-validator","epoch":49231,"indices":["1104251"]}
{"time":"2025-10-22T07:41:29.582217735Z","level":"INFO","msg":"Validator did not attest","module":"main","module":"e2m-validator","epoch":49232,"indices":["1104251"]}
{"time":"2025-10-22T07:47:45.894479332Z","level":"INFO","msg":"E2M registration validation successful - validators are attesting","module":"main","module":"e2m-validator","validation_type":"registration","indices":["1104251"],"registered_epoch":49232,"validated_epoch":49233,"epochs_waited":1}
{"time":"2025-10-22T07:47:45.894679095Z","level":"INFO","msg":"E2M validation successful","test":"sanity","indices":["1104251"],"validate-type":"registration"}
{"time":"2025-10-22T07:47:45.894737745Z","level":"INFO","msg":"Starting consensus validation with epoch iteration logic","module":"main","module":"loki-validator","validator_index":"1104251","registered_epoch":49232,"max_epoch_offset":4}
{"time":"2025-10-22T07:47:45.894854783Z","level":"INFO","msg":"Checking epoch for duties","module":"main","module":"loki-validator","epoch_offset":0,"current_epoch":"49232","validator_index":"1104251"}
{"time":"2025-10-22T07:47:45.915786033Z","level":"WARN","msg":"No attesters found in duties response","module":"main","module":"loki-validator","epoch_offset":0,"epoch":"49232"}
{"time":"2025-10-22T07:47:45.91596354Z","level":"INFO","msg":"Checking epoch for duties","module":"main","module":"loki-validator","epoch_offset":1,"current_epoch":"49233","validator_index":"1104251"}
{"time":"2025-10-22T07:47:45.930753074Z","level":"INFO","msg":"Successfully found duties, building duty ID for consensus validation","module":"main","module":"loki-validator","duty_id":"COMMITTEE-304_305_306_307-e49233-s1575458","final_epoch":"49233","slot":"1575458"}
{"time":"2025-10-22T07:47:46.060590729Z","level":"INFO","msg":"Consensus leader validation passed","module":"main","module":"loki-validator","slot":1575458,"leader":306}
{"time":"2025-10-22T07:47:46.060752864Z","level":"INFO","msg":"Consensus timing validation comparison","module":"main","module":"loki-validator","actual_time_diff":38870000,"expected_consensus_time":34000000,"lower_bound":23800000,"upper_bound":44200000,"margin_percent":30}
{"time":"2025-10-22T07:47:46.060777253Z","level":"INFO","msg":"Consensus timing validation passed - within acceptable range","module":"main","module":"loki-validator"}
{"time":"2025-10-22T07:47:46.060801421Z","level":"INFO","msg":"Consensus time validation completed successfully","module":"main","module":"loki-validator","final_epoch":"49233"}
{"time":"2025-10-22T07:47:46.060843112Z","level":"INFO","msg":"Consensus timing validation successful","test":"sanity","indices":["1104251"],"epoch":49232}
{"time":"2025-10-22T07:47:47.902796602Z","level":"INFO","msg":"Liquidating cluster","test":"sanity","module":"contract","tx_hash":"0xbc5374aa602e4cca8d39d675d08dc94d5cbf3aa69d59de3c3d2fab3ff052c46a"}
{"time":"2025-10-22T07:47:47.906244973Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0xbc5374aa602e4cca8d39d675d08dc94d5cbf3aa69d59de3c3d2fab3ff052c46a"}
{"time":"2025-10-22T07:47:57.918295297Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":2,"sleep":20,"tx_hash":"0xbc5374aa602e4cca8d39d675d08dc94d5cbf3aa69d59de3c3d2fab3ff052c46a"}
{"time":"2025-10-22T07:48:17.939025495Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0xbc5374aa602e4cca8d39d675d08dc94d5cbf3aa69d59de3c3d2fab3ff052c46a"}
{"time":"2025-10-22T07:48:17.939274564Z","level":"INFO","msg":"✅ SUCCESS: Cluster Liquidation Successful","test":"sanity","txHash":"0xbc5374aa602e4cca8d39d675d08dc94d5cbf3aa69d59de3c3d2fab3ff052c46a","block_hash":"0x2f87a8c0a2a920acd7daeaa929928435fedd7637a7d109f0fa9f450c6f595954"}
{"time":"2025-10-22T07:48:17.939503184Z","level":"INFO","msg":"Querying Loki for transaction hash validation","module":"main","module":"loki-validator","query":"{container=~\\"ssv-node-304|ssv-node-305|ssv-node-306|ssv-node-307\\"} |= \\"0xbc5374aa602e4cca8d39d675d08dc94d5cbf3aa69d59de3c3d2fab3ff052c46a\\"","tx_hash":"0xbc5374aa602e4cca8d39d675d08dc94d5cbf3aa69d59de3c3d2fab3ff052c46a"}
{"time":"2025-10-22T07:48:17.949281412Z","level":"INFO","msg":"Liquidation epoch","module":"main","module":"e2m-validator","epoch":49234,"indices":["1104251"]}
{"time":"2025-10-22T07:48:18.045805149Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":1,"max_retries":5,"delay_seconds":30,"start_unix":1761108497,"end_unix":1761119297}
{"time":"2025-10-22T07:48:48.105094786Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":2,"max_retries":5,"delay_seconds":60,"start_unix":1761108497,"end_unix":1761119328}
{"time":"2025-10-22T07:49:48.130163035Z","level":"ERROR","msg":"request failed","module":"HTTPRequester","error":"Get \\"https://loki.ops.ssvlabsinternal.com/loki/api/v1/query_range?query=%7Bcontainer%3D~%22ssv-node-304%7Cssv-node-305%7Cssv-node-306%7Cssv-node-307%22%7D+%7C%3D+%220xbc5374aa602e4cca8d39d675d08dc94d5cbf3aa69d59de3c3d2fab3ff052c46a%22+%7C+json+%7C+line_format+%22%7B%7B.container%7D%7D%3A%7B%7B.msg%7D%7D%22&start=1761108497&end=1761119388&limit=1000\\": unexpected EOF","request attempt #":0}
{"time":"2025-10-22T07:49:48.325860403Z","level":"INFO","msg":"Event validation successful","module":"main","module":"loki-validator","valid_entries":4,"event":"ClusterLiquidated"}
{"time":"2025-10-22T07:49:48.325922402Z","level":"INFO","msg":"SSV node log validation successful","test":"sanity","indices":["1104251"],"validators":1,"validate-type":"liquidation"}
{"time":"2025-10-22T07:54:19.268371759Z","level":"INFO","msg":"Validator was not liquidated","module":"main","module":"e2m-validator","epoch":49234,"indices":["1104251"]}
{"time":"2025-10-22T08:00:35.443780994Z","level":"INFO","msg":"E2M liquidation validation successful - validators are liquidated","module":"main","module":"e2m-validator","validation_type":"liquidation","indices":["1104251"],"liquidated_epoch":49234,"validated_epoch":49235,"epochs_waited":1,"assigned":0,"missed":0}
{"time":"2025-10-22T08:00:35.444002376Z","level":"INFO","msg":"E2M validation successful","test":"sanity","indices":["1104251"],"validate-type":"liquidation"}
{"time":"2025-10-22T08:00:37.443558622Z","level":"INFO","msg":"Allowance fetched successfully","test":"sanity","module":"contract","account":"0x5cc0dde14e7256340cc820415a6022a7d1c93a35","allowance":"9999999999999999999999562326194910796200000000"}
{"time":"2025-10-22T08:00:37.443647246Z","level":"INFO","msg":"Threshold value","test":"sanity","module":"contract","threshold":"9000000000000000000000"}
{"time":"2025-10-22T08:00:37.443672795Z","level":"INFO","msg":"Sufficient allowance","test":"sanity","module":"contract","allowance margin":"6561000000000000000000000000000000000000000000000000000000000000000000000000000000000000"}
{"time":"2025-10-22T08:00:37.510452954Z","level":"INFO","msg":"Reactivating cluster","test":"sanity","module":"contract","tx_hash":"0x3a3d93dd2310f20163ff0fde94331673a5903e4f1ed480a40b7c5b300778785c"}
{"time":"2025-10-22T08:00:37.514692426Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0x3a3d93dd2310f20163ff0fde94331673a5903e4f1ed480a40b7c5b300778785c"}
{"time":"2025-10-22T08:00:47.522526456Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":2,"sleep":20,"tx_hash":"0x3a3d93dd2310f20163ff0fde94331673a5903e4f1ed480a40b7c5b300778785c"}
{"time":"2025-10-22T08:01:07.530347031Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0x3a3d93dd2310f20163ff0fde94331673a5903e4f1ed480a40b7c5b300778785c"}
{"time":"2025-10-22T08:01:07.530426838Z","level":"INFO","msg":"✅ SUCCESS: Cluster Reactivation Successful","test":"sanity","txHash":"0x3a3d93dd2310f20163ff0fde94331673a5903e4f1ed480a40b7c5b300778785c","block_hash":"0x35d9328d20a970b97afafebb00bbe1f74f9adb808383d9cbb6258e2e9a57130b"}
{"time":"2025-10-22T08:01:07.530573274Z","level":"INFO","msg":"Validating reactivation - checking that validators resume attesting","module":"main","module":"e2m-validator","indices":["1104251"],"block_hash":"0x35d9328d20a970b97afafebb00bbe1f74f9adb808383d9cbb6258e2e9a57130b"}
{"time":"2025-10-22T08:01:07.53073841Z","level":"INFO","msg":"Querying Loki for transaction hash validation","module":"main","module":"loki-validator","query":"{container=~\\"ssv-node-304|ssv-node-305|ssv-node-306|ssv-node-307\\"} |= \\"0x3a3d93dd2310f20163ff0fde94331673a5903e4f1ed480a40b7c5b300778785c\\"","tx_hash":"0x3a3d93dd2310f20163ff0fde94331673a5903e4f1ed480a40b7c5b300778785c"}
{"time":"2025-10-22T08:01:07.541510243Z","level":"INFO","msg":"waiting for block to be found","module":"main","module":"e2m-validator","module":"e2m"}
{"time":"2025-10-22T08:01:07.698255848Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":1,"max_retries":5,"delay_seconds":30,"start_unix":1761109267,"end_unix":1761120067}
{"time":"2025-10-22T08:01:27.556248116Z","level":"INFO","msg":"Registration epoch","module":"main","module":"e2m-validator","epoch":49236,"indices":"1104251"}
{"time":"2025-10-22T08:01:37.850116059Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":2,"max_retries":5,"delay_seconds":60,"start_unix":1761109267,"end_unix":1761120097}
{"time":"2025-10-22T08:01:42.606283344Z","level":"INFO","msg":"Validator did not attest","module":"main","module":"e2m-validator","epoch":49235,"indices":["1104251"]}
{"time":"2025-10-22T08:02:37.852390036Z","level":"ERROR","msg":"request failed","module":"HTTPRequester","error":"Get \\"https://loki.ops.ssvlabsinternal.com/loki/api/v1/query_range?query=%7Bcontainer%3D~%22ssv-node-304%7Cssv-node-305%7Cssv-node-306%7Cssv-node-307%22%7D+%7C%3D+%220x3a3d93dd2310f20163ff0fde94331673a5903e4f1ed480a40b7c5b300778785c%22+%7C+json+%7C+line_format+%22%7B%7B.container%7D%7D%3A%7B%7B.msg%7D%7D%22&start=1761109267&end=1761120157&limit=1000\\": unexpected EOF","request attempt #":0}
{"time":"2025-10-22T08:02:37.955226724Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":3,"max_retries":5,"delay_seconds":90,"start_unix":1761109267,"end_unix":1761120157}
{"time":"2025-10-22T08:04:08.085818021Z","level":"INFO","msg":"Event validation successful","module":"main","module":"loki-validator","valid_entries":4,"event":"ClusterReactivated"}
{"time":"2025-10-22T08:04:08.085893628Z","level":"INFO","msg":"SSV node log validation successful","test":"sanity","indices":["1104251"],"validators":1,"validate-type":"reactivation"}
{"time":"2025-10-22T08:06:58.675261903Z","level":"INFO","msg":"Validator did not attest","module":"main","module":"e2m-validator","epoch":49236,"indices":["1104251"]}
{"time":"2025-10-22T08:13:29.997033371Z","level":"INFO","msg":"E2M registration validation successful - validators are attesting","module":"main","module":"e2m-validator","validation_type":"registration","indices":["1104251"],"registered_epoch":49236,"validated_epoch":49237,"epochs_waited":1}
{"time":"2025-10-22T08:13:29.997252157Z","level":"INFO","msg":"E2M validation successful","test":"sanity","indices":["1104251"],"validate-type":"reactivation"}
{"time":"2025-10-22T08:13:29.997291111Z","level":"INFO","msg":"Starting consensus validation with epoch iteration logic","module":"main","module":"loki-validator","validator_index":"1104251","registered_epoch":49236,"max_epoch_offset":4}
{"time":"2025-10-22T08:13:29.99741249Z","level":"INFO","msg":"Checking epoch for duties","module":"main","module":"loki-validator","epoch_offset":0,"current_epoch":"49236","validator_index":"1104251"}
{"time":"2025-10-22T08:13:30.010282636Z","level":"WARN","msg":"No attesters found in duties response","module":"main","module":"loki-validator","epoch_offset":0,"epoch":"49236"}
{"time":"2025-10-22T08:13:30.010413276Z","level":"INFO","msg":"Checking epoch for duties","module":"main","module":"loki-validator","epoch_offset":1,"current_epoch":"49237","validator_index":"1104251"}
{"time":"2025-10-22T08:13:30.02213876Z","level":"INFO","msg":"Successfully found duties, building duty ID for consensus validation","module":"main","module":"loki-validator","duty_id":"COMMITTEE-304_305_306_307-e49237-s1575585","final_epoch":"49237","slot":"1575585"}
{"time":"2025-10-22T08:13:30.182488457Z","level":"INFO","msg":"Consensus leader validation passed","module":"main","module":"loki-validator","slot":1575585,"leader":305}
{"time":"2025-10-22T08:13:30.1827839Z","level":"INFO","msg":"Consensus timing validation comparison","module":"main","module":"loki-validator","actual_time_diff":46956000,"expected_consensus_time":39000000,"lower_bound":27300000,"upper_bound":50700000,"margin_percent":30}
{"time":"2025-10-22T08:13:30.182873859Z","level":"INFO","msg":"Consensus timing validation passed - within acceptable range","module":"main","module":"loki-validator"}
{"time":"2025-10-22T08:13:30.1829309Z","level":"INFO","msg":"Consensus time validation completed successfully","module":"main","module":"loki-validator","final_epoch":"49237"}
{"time":"2025-10-22T08:13:30.183097069Z","level":"INFO","msg":"Consensus timing validation successful","test":"sanity","indices":["1104251"],"epoch":49236}
{"time":"2025-10-22T08:13:31.974761242Z","level":"INFO","msg":"Allowance fetched successfully","test":"sanity","module":"contract","account":"0x5cc0dde14e7256340cc820415a6022a7d1c93a35","allowance":"9999999999999999999999553326194910796200000000"}
{"time":"2025-10-22T08:13:31.974872495Z","level":"INFO","msg":"Threshold value","test":"sanity","module":"contract","threshold":"9000000000000000000000"}
{"time":"2025-10-22T08:13:31.974932209Z","level":"INFO","msg":"Sufficient allowance","test":"sanity","module":"contract","allowance margin":"6561000000000000000000000000000000000000000000000000000000000000000000000000000000000000"}
{"time":"2025-10-22T08:13:32.333204497Z","level":"INFO","msg":"Removing validator","test":"sanity","module":"contract","tx_hash":"0x2bea32fd1191ec7da5706391cd6de6dd70a213e2cfdee8c0aec342494798f33d"}
{"time":"2025-10-22T08:13:32.337087403Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0x2bea32fd1191ec7da5706391cd6de6dd70a213e2cfdee8c0aec342494798f33d"}
{"time":"2025-10-22T08:13:42.34978427Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0x2bea32fd1191ec7da5706391cd6de6dd70a213e2cfdee8c0aec342494798f33d"}
{"time":"2025-10-22T08:13:42.349959919Z","level":"INFO","msg":"✅ SUCCESS: Contract Removal Successful","test":"sanity","blockHash":"0x3501384b072bcdd8b6251eff0fbf48d71b4cf4f20ab00b0778df50179412737a"}
{"time":"2025-10-22T08:13:42.360049505Z","level":"INFO","msg":"waiting for block to be found","module":"main","module":"e2m-validator","module":"e2m"}
{"time":"2025-10-22T08:13:42.506790196Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":1,"max_retries":5,"delay_seconds":30,"start_unix":1761110022,"end_unix":1761120822}
{"time":"2025-10-22T08:14:02.369558216Z","level":"INFO","msg":"Removal epoch","module":"main","module":"e2m-validator","epoch":49238,"indices":"1104251"}
{"time":"2025-10-22T08:14:12.565495231Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":2,"max_retries":5,"delay_seconds":60,"start_unix":1761110022,"end_unix":1761120852}
{"time":"2025-10-22T08:15:12.567473385Z","level":"ERROR","msg":"request failed","module":"HTTPRequester","error":"Get \\"https://loki.ops.ssvlabsinternal.com/loki/api/v1/query_range?query=%7Bcontainer%3D~%22ssv-node-304%7Cssv-node-305%7Cssv-node-306%7Cssv-node-307%22%7D+%7C%3D+%220x2bea32fd1191ec7da5706391cd6de6dd70a213e2cfdee8c0aec342494798f33d%22&start=1761110022&end=1761120912&limit=1000\\": unexpected EOF","request attempt #":0}
{"time":"2025-10-22T08:15:12.67497156Z","level":"INFO","msg":"No logs found, retrying after delay (nodes may still be processing)","module":"main","module":"loki-validator","attempt":3,"max_retries":5,"delay_seconds":90,"start_unix":1761110022,"end_unix":1761120912}
{"time":"2025-10-22T08:16:42.795873092Z","level":"INFO","msg":"Validator event validation successful","module":"main","module":"loki-validator","validated_validators":1,"event":"ValidatorRemoved"}
{"time":"2025-10-22T08:16:42.796054217Z","level":"INFO","msg":"SSV node log validation successful","test":"sanity","indices":["1104251"],"validators":1,"validate-type":"removal"}
{"time":"2025-10-22T08:19:48.627966242Z","level":"INFO","msg":"Validator was not removed","module":"main","module":"e2m-validator","epoch":49238,"indices":["1104251"]}
{"time":"2025-10-22T08:26:04.952887485Z","level":"INFO","msg":"E2M removal validation successful - validators stopped attesting","module":"main","module":"e2m-validator","validation_type":"removal","indices":["1104251"],"removed_epoch":49238,"validated_epoch":49239,"epochs_waited":1}
{"time":"2025-10-22T08:26:04.953079954Z","level":"INFO","msg":"E2M validation successful","test":"sanity","indices":["1104251"],"validate-type":"removal"}
{"time":"2025-10-22T08:26:04.953370157Z","level":"INFO","msg":"✅ SUCCESS: Sanity Test Suite Completed Successfully","module":"main","register":{"Contract":true,"E2M":true,"Loki":true},"liquidate":{"Contract":true,"E2M":true,"Loki":true},"reactivate":{"Contract":true,"E2M":true,"Loki":true},"remove":{"Contract":true,"E2M":true,"Loki":true}}
{"time":"2025-10-22T08:26:04.953423479Z","level":"INFO","msg":"⏳ PROGRESS: Waiting for all tests to finish","module":"main"}
{"time":"2025-10-22T08:26:04.953580674Z","level":"INFO","msg":"✅ SUCCESS: All Test Suites Completed Successfully","module":"main","passed_suites":["sanity"],"skipped_suites":["isolated"]}
{"time":"2025-10-22T08:26:04.95362641Z","level":"INFO","msg":"🔑 KEY STEP: Starting resource teardown...","module":"main","usage_id":"e3df1702f722737a30fb0b0bdc1f80eb"}
{"time":"2025-10-22T08:26:04.965821757Z","level":"INFO","msg":"🌐 Orchestrator API Client initialized","module":"main","endpoint":"http://aetheria-orchestrator.aetheria.svc:8080"}
{"time":"2025-10-22T08:26:04.965972533Z","level":"INFO","msg":"🧹 Requesting resource teardown","module":"main","usage_id":"e3df1702f722737a30fb0b0bdc1f80eb"}
{"time":"2025-10-22T08:26:04.995152672Z","level":"INFO","msg":"✅ SUCCESS: Completed resource teardown successfully","module":"main","usage_id":"e3df1702f722737a30fb0b0bdc1f80eb"}

2025-10-22T08:26:08Z The test run finished successfully

Result - 🟢 Success

yoaz-ssvlabs · 2025-11-23T08:38:17Z

/test help

ssv-test-bot · 2025-11-23T08:38:20Z

❓ Help

Available Commands

`/test` - Run Tests

To run tests, use the command /test followed by optional parameters.
Example:
/test cluster=300,301,302,303 tests=(sanity,e2m) network=hoodi_stage

Available parameters (all optional):

cluster — Comma-separated operator IDs (4 or 7 numbers in ascending order). If omitted (recommended), the workflow auto-allocates nodes. Valid range: 300-333.
tests — A list describing which tests to run. Default: (sanity,e2m)
network — Network configuration to use. Default: hoodi_stage

🧪 TESTS parameter

Format:

(type[,options]),(type[,options]),...

Supported test types:

sanity[,e2m][,threshold] — Sanity tests (optional e2m validation, optional threshold testing)
bulk[,e2m][,threshold][,amount] — Bulk tests (optional e2m/threshold validation; optional validator count as amount - default 3, max 10)
vouch,key:value — Vouch tests (requires all keys)
fee-recipient,key:value — Fee recipient validation tests (requires newAddress, optional ownerAddress)

Isolated parameters for vouch (all required):

epochs:N — Number of epochs to examine
amountOfETHNodes:N — Number of ETH1 nodes vouch deployed with

Isolated parameters for fee-recipient:

newAddress:0x... — Required - New fee recipient address to set and validate
ownerAddress:0x... — Optional - Cluster owner address (defaults to cluster owner from config)

💡 Usage examples

Default (no cluster, no network, no tests; defaults kick in):

/test

Simple bulk run with amount only, default network (hoodi_stage):

/test cluster=300,301,302,303 tests=(bulk,5)

With specific cluster, e2m on both sanity and bulk, fixed amount:

/test cluster=300,301,302,303 tests=(sanity,e2m),(bulk,e2m,10) network=hoodi_stage

Sanity plus a vouch run:

/test cluster=300,301,302,303 tests=(sanity),(vouch,epochs:5,amountOfETHNodes:4) network=hoodi_stage

Simple bulk run with default validators amount (3):

/test cluster=300,301,302,303 tests=(bulk) network=hoodi_stage

Sanity with threshold testing:

/test cluster=300,301,302,303 tests=(sanity,threshold) network=hoodi_stage

Bulk with threshold and custom amount:

/test cluster=300,301,302,303 tests=(bulk,threshold,5) network=hoodi_stage

Combined e2m and threshold testing:

/test cluster=300,301,302,303 tests=(sanity,e2m,threshold),(bulk,e2m,threshold,10) network=hoodi_stage

Fee recipient validation test:

/test cluster=300,301,302,303 tests=(fee-recipient,newAddress:0xd2DBd02e4efe087d7d195de828b9Dd25f19A89C9) network=hoodi_stage

Fee recipient test with custom owner address:

/test cluster=300,301,302,303 tests=(fee-recipient,newAddress:0xd2DBd02e4efe087d7d195de828b9Dd25f19A89C9,ownerAddress:0x5cc0dde14e7256340cc820415a6022a7d1c93a35) network=hoodi_stage

Combined sanity and fee recipient tests:

/test cluster=300,301,302,303 tests=(sanity),(fee-recipient,newAddress:0xd2DBd02e4efe087d7d195de828b9Dd25f19A89C9) network=hoodi_stage

`/test skip-ci` - Skip CI Pipeline

⚠️ Restricted Command: Only members of authorized teams can use this command.

Bypasses the CI pipeline by setting the configured status check to success.
Example:

/test skip-ci

Important Notes:

Only members of authorized teams (configurable) can execute this command
Use with caution as it bypasses all automated checks

yoaz-ssvlabs · 2025-11-23T08:46:51Z

/test tests=(sanity,e2m),(bulk,15)

ssv-test-bot · 2025-11-23T08:46:55Z

🔴 Test run for PR #2433

Branch: ci-test-handler-testing2
Commit: 74b7038

Status

2025-11-23T08:46:55Z Starting test run
2025-11-23T08:46:56Z Test run submitted successfully for commit 74b7038
2025-11-23T08:46:56Z container image for commit 74b7038f0265e7f8da4425e9315fe4199c751b80 is build and present in container registry
2025-11-23T08:46:58Z automatically allocated node range cluster=300,301,302,303 for test run on network=hoodi_stage
2025-11-23T08:46:59Z deploying commit 74b7038f0265e7f8da4425e9315fe4199c751b80 to ssv nodes 300,301,302,303
2025-11-23T08:48:14Z commit 74b7038f0265e7f8da4425e9315fe4199c751b80 is deployed to ssv nodes 300,301,302,303

Click to expand executor logs

═══════════════════════════════════════════════════════════════════════════════════════════════════════
                🧪  WELCOME TO Executor - SSV's Distributed Test Automation Execution Layer  🧪            

                                  ___ __  __ ___  ___  _   _  _____  ___   ___ 
                                 | __|\\ \\/ /| __|/ __|| | | ||_   _|/ _ \\ | _ \\
                                 | _|  >  < | _|| (__ | |_| |  | | | (_) ||   /
                                 |___|/_/\\_\\|___|\\___| \\___/   |_|  \\___/ |_|_\\
                                        

═══════════════════════════════════════════════════════════════════════════════════════════════════════
{"time":"2025-11-23T08:48:16.166124603Z","level":"INFO","msg":"🐳 Docker mode: Using environment variables for config generation"}
{"time":"2025-11-23T08:48:16.186596073Z","level":"INFO","msg":"🌐 Orchestrator API Client initialized","endpoint":"http://aetheria-orchestrator.aetheria.svc:8080"}
{"time":"2025-11-23T08:48:16.186743249Z","level":"INFO","msg":"📝 Requesting configuration generation from orchestrator API","network":"hoodi_stage","clusters":[300,301,302,303],"tests":"(sanity,e2m),(bulk,15)"}
{"time":"2025-11-23T08:48:16.262645573Z","level":"INFO","msg":"✅ Configuration generated successfully via API","resource_usage_id":"def527d99f136e6bf34179aaa0f69a55"}
{"time":"2025-11-23T08:48:16.26398099Z","level":"INFO","msg":"🚀 EXECUTOR Initialized Successfully"}
{"time":"2025-11-23T08:48:16.264049265Z","level":"INFO","msg":"⚙️ Test configuration loaded","network":"hoodi_stage","nodes":[300,301,302,303],"test_type":{"Sanity":{"ShouldRun":true,"E2M":true,"Threshold":false,"BulkAmount":0},"Bulk":{"ShouldRun":true,"E2M":false,"Threshold":false,"BulkAmount":15},"Isolated":{"Tests":null}}}
{"time":"2025-11-23T08:48:20.724249851Z","level":"INFO","msg":"Generated operator pubkeys file","module":"scanner","path":"/app/shared/data/operators/operator-pubkeys-hoodi_stage.json"}
{"time":"2025-11-23T08:48:20.724549683Z","level":"INFO","msg":"Registered parser","module":"main","component":"parser-client","parser_type":"consensus"}
{"time":"2025-11-23T08:48:20.724606936Z","level":"INFO","msg":"Registered parser","module":"main","component":"parser-client","parser_type":"flow-emits"}
{"time":"2025-11-23T08:48:20.724642149Z","level":"INFO","msg":"Validation client initialized","module":"main","loki_enabled":true,"e2m_enabled":true}
{"time":"2025-11-23T08:48:20.72469456Z","level":"INFO","msg":"🔑 KEY STEP: Starting Isolated Tests Concurrently if exists...","module":"main"}
{"time":"2025-11-23T08:48:20.724786009Z","level":"INFO","msg":"🔑 KEY STEP: Starting Sanity Test Suite","module":"main"}
{"time":"2025-11-23T08:48:20.724877413Z","level":"INFO","msg":"No isolated tests configured, skipping","module":"main"}
{"time":"2025-11-23T08:48:20.775750312Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":300}
{"time":"2025-11-23T08:48:20.794456123Z","level":"INFO","msg":"Available slots","test":"sanity","node":300,"slots":3000}
{"time":"2025-11-23T08:48:20.795470337Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":301}
{"time":"2025-11-23T08:48:20.807938483Z","level":"INFO","msg":"Available slots","test":"sanity","node":301,"slots":755}
{"time":"2025-11-23T08:48:20.808023585Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":302}
{"time":"2025-11-23T08:48:20.816577446Z","level":"INFO","msg":"Available slots","test":"sanity","node":302,"slots":451}
{"time":"2025-11-23T08:48:20.816677288Z","level":"INFO","msg":"Getting available slots","test":"sanity","node":303}
{"time":"2025-11-23T08:48:20.823731191Z","level":"INFO","msg":"Available slots","test":"sanity","node":303,"slots":1072}
{"time":"2025-11-23T08:48:59.950673266Z","level":"INFO","msg":"Extracted keyshares path","test":"sanity","module":"keys","path":"/app/shared/data/keyshares/0x8038d417227d331ed9f539b22c36caeea8ee9b70df5b69b4c763b00ecd314ff76bd5880bd6629b66883b929e63474e67.json"}
{"time":"2025-11-23T08:48:59.951792564Z","level":"INFO","msg":"Key shares file generated successfully","test":"sanity","keyshare_path":"/app/shared/data/keyshares/0x8038d417227d331ed9f539b22c36caeea8ee9b70df5b69b4c763b00ecd314ff76bd5880bd6629b66883b929e63474e67.json"}
{"time":"2025-11-23T08:49:02.936472771Z","level":"INFO","msg":"Allowance fetched successfully","test":"sanity","module":"contract","account":"0x91e32efb8139cd88cae0df30d2bf471294c6ed27","allowance":"99123284439116557600000000"}
{"time":"2025-11-23T08:49:02.936704397Z","level":"INFO","msg":"Threshold value","test":"sanity","module":"contract","threshold":"9000000000000000000000"}
{"time":"2025-11-23T08:49:02.936770259Z","level":"INFO","msg":"Sufficient allowance","test":"sanity","module":"contract","allowance margin":"6561000000000000000000000000000000000000000000000000000000000000000000000000000000000000"}
{"time":"2025-11-23T08:49:02.941727288Z","level":"INFO","msg":"Allowance fetched successfully","test":"sanity","module":"contract","account":"0x91e32efb8139cd88cae0df30d2bf471294c6ed27","allowance":"99123284439116557600000000"}
{"time":"2025-11-23T08:49:02.941837006Z","level":"INFO","msg":"Allowance","test":"sanity","module":"contract","allowance":"99123284439116557600000000"}
{"time":"2025-11-23T08:49:02.983590655Z","level":"INFO","msg":"Registering validator","test":"sanity","module":"contract","tx_hash":"0x96805b1d041874aa921acfd66cac847bec9a5849884feb751c052957644a8824"}
{"time":"2025-11-23T08:49:02.98741905Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0x96805b1d041874aa921acfd66cac847bec9a5849884feb751c052957644a8824"}
{"time":"2025-11-23T08:49:13.000515295Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":2,"sleep":20,"tx_hash":"0x96805b1d041874aa921acfd66cac847bec9a5849884feb751c052957644a8824"}
{"time":"2025-11-23T08:49:33.013181686Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0x96805b1d041874aa921acfd66cac847bec9a5849884feb751c052957644a8824"}
{"time":"2025-11-23T08:49:33.013347993Z","level":"INFO","msg":"✅ SUCCESS: Contract Register Successful","test":"sanity","txHash":"0x96805b1d041874aa921acfd66cac847bec9a5849884feb751c052957644a8824","block_hash":"0x5dcda86442e3a64dd0cae3099e14ea96870357c4b6b327cfe5060a8a20553a69"}
{"time":"2025-11-23T08:49:33.019160824Z","level":"ERROR","msg":"❌ FAILURE: Sanity Test Suite Failed","module":"main","error":"ErrTransport: Get \\"http://ssv-node-exporter-stable.ssv:16000/v1/validators?pubkeys=0x8038d417227d331ed9f539b22c36caeea8ee9b70df5b69b4c763b00ecd314ff76bd5880bd6629b66883b929e63474e67\\": dial tcp 10.43.144.107:16000: connect: connection refused"}
{"time":"2025-11-23T08:49:33.019283289Z","level":"INFO","msg":"🔑 KEY STEP: Starting Bulk Test Suite","module":"main"}
{"time":"2025-11-23T08:49:33.030192816Z","level":"INFO","msg":"Getting available slots","test":"bulk","node":300}
{"time":"2025-11-23T08:49:33.04079647Z","level":"INFO","msg":"Available slots","test":"bulk","node":300,"slots":3000}
{"time":"2025-11-23T08:49:33.040879098Z","level":"INFO","msg":"Getting available slots","test":"bulk","node":301}
{"time":"2025-11-23T08:49:33.048533938Z","level":"INFO","msg":"Available slots","test":"bulk","node":301,"slots":755}
{"time":"2025-11-23T08:49:33.048664746Z","level":"INFO","msg":"Getting available slots","test":"bulk","node":302}
{"time":"2025-11-23T08:49:33.057265991Z","level":"INFO","msg":"Available slots","test":"bulk","node":302,"slots":451}
{"time":"2025-11-23T08:49:33.05735457Z","level":"INFO","msg":"Getting available slots","test":"bulk","node":303}
{"time":"2025-11-23T08:49:33.064760476Z","level":"INFO","msg":"Available slots","test":"bulk","node":303,"slots":1072}
{"time":"2025-11-23T08:51:16.86418282Z","level":"INFO","msg":"Extracted keyshares path for bulk keyshare","test":"bulk","module":"keys","path":"/app/shared/data/keyshares/keyshares-1763887876.json"}
{"time":"2025-11-23T08:51:22.130862429Z","level":"INFO","msg":"Allowance fetched successfully","test":"bulk","module":"contract","account":"0x91e32efb8139cd88cae0df30d2bf471294c6ed27","allowance":"99123284439116557600000000"}
{"time":"2025-11-23T08:51:22.130973294Z","level":"INFO","msg":"Threshold value","test":"bulk","module":"contract","threshold":"9000000000000000000000"}
{"time":"2025-11-23T08:51:22.131009398Z","level":"INFO","msg":"Sufficient allowance","test":"bulk","module":"contract","allowance margin":"6561000000000000000000000000000000000000000000000000000000000000000000000000000000000000"}
{"time":"2025-11-23T08:51:22.263802776Z","level":"ERROR","msg":"❌ FAILURE: Bulk Test Suite Failed","module":"main","error":"execution reverted"}
{"time":"2025-11-23T08:51:22.263909226Z","level":"INFO","msg":"⏳ PROGRESS: Waiting for all tests to finish","module":"main"}
{"time":"2025-11-23T08:51:22.263993155Z","level":"ERROR","msg":"❌ FAILURE: Test Execution Completed with Failures","module":"main","passed_suites":null,"failed_suites":["sanity","bulk"],"skipped_suites":["isolated"]}
{"time":"2025-11-23T08:51:22.264451792Z","level":"ERROR","msg":"❌ FAILURE: Suite Failure Details","module":"main","suite":"sanity","error":"ErrTransport: Get \\"http://ssv-node-exporter-stable.ssv:16000/v1/validators?pubkeys=0x8038d417227d331ed9f539b22c36caeea8ee9b70df5b69b4c763b00ecd314ff76bd5880bd6629b66883b929e63474e67\\": dial tcp 10.43.144.107:16000: connect: connection refused"}
{"time":"2025-11-23T08:51:22.265216554Z","level":"ERROR","msg":"❌ FAILURE: Suite Failure Details","module":"main","suite":"bulk","error":"execution reverted"}
{"time":"2025-11-23T08:51:22.265360828Z","level":"INFO","msg":"🔑 KEY STEP: Starting resource teardown...","module":"main","usage_id":"def527d99f136e6bf34179aaa0f69a55"}
{"time":"2025-11-23T08:51:22.275153252Z","level":"INFO","msg":"🌐 Orchestrator API Client initialized","module":"main","endpoint":"http://aetheria-orchestrator.aetheria.svc:8080"}
{"time":"2025-11-23T08:51:22.275258829Z","level":"INFO","msg":"🧹 Requesting resource teardown","module":"main","usage_id":"def527d99f136e6bf34179aaa0f69a55"}
{"time":"2025-11-23T08:51:22.374253689Z","level":"INFO","msg":"✅ SUCCESS: Completed resource teardown successfully","module":"main","usage_id":"def527d99f136e6bf34179aaa0f69a55"}

2025-11-23T08:51:25Z The test run failed with error

Result - 🔴 Failure

test 2

2a71a70

Copilot AI review requested due to automatic review settings August 9, 2025 10:47

vaclav-ssvlabs requested review from a team as code owners August 9, 2025 10:47

vaclav-ssvlabs changed the base branch from main to stage August 9, 2025 10:47

This comment was marked as outdated.

Sign in to view

ssvlabs deleted a comment from ssv-test-bot bot Aug 12, 2025

ssvlabs deleted a comment from ssv-test-bot bot Aug 14, 2025

ssvlabs deleted a comment from ssv-test-bot bot Sep 18, 2025

Update main.go

74b7038

ssvlabs deleted a comment from ssv-test-bot bot Nov 23, 2025

Ci test handler testing2 #2433

Are you sure you want to change the base?

Ci test handler testing2 #2433

Uh oh!

Conversation

vaclav-ssvlabs commented Aug 9, 2025

Uh oh!

This comment was marked as outdated.

Uh oh!

vaclav-ssvlabs commented Sep 18, 2025

Uh oh!

ssv-test-bot bot commented Sep 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟢 Test run for PR #2433

Status

Result - 🟢 Success

Uh oh!

vaclav-ssvlabs commented Sep 18, 2025

Uh oh!

ssv-test-bot bot commented Sep 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔴 Test run for PR #2433

Status

Result - 🔴 Failure

Uh oh!

yoaz-ssvlabs commented Sep 19, 2025

Uh oh!

ssv-test-bot bot commented Sep 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔴 Test run for PR #2433

Status

Result - 🔴 Failure

Uh oh!

vaclav-ssvlabs commented Sep 19, 2025

Uh oh!

ssv-test-bot bot commented Sep 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔴 Test run for PR #2433

Status

Result - 🔴 Failure

Uh oh!

yoaz-ssvlabs commented Sep 22, 2025

Uh oh!

ssv-test-bot bot commented Sep 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔴 Test run for PR #2433

Status

Result - 🔴 Failure

Uh oh!

yoaz-ssvlabs commented Sep 22, 2025

Uh oh!

ssv-test-bot bot commented Sep 22, 2025

Available Commands

/test - Run Tests

/test skip-ci - Skip CI Pipeline

Uh oh!

yoaz-ssvlabs commented Sep 22, 2025

Uh oh!

ssv-test-bot bot commented Sep 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟢 Test run for PR #2433

Status

Result - 🟢 Success

Uh oh!

yoaz-ssvlabs commented Sep 22, 2025

Uh oh!

ssv-test-bot bot commented Sep 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟢 Test run for PR #2433

Status

Result - 🟢 Success

Uh oh!

yoaz-ssvlabs commented Sep 30, 2025

Uh oh!

ssv-test-bot bot commented Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟢 Test run for PR #2433

Status

Result - 🟢 Success

Uh oh!

ssv-test-bot bot commented Sep 18, 2025 •

edited

Loading

ssv-test-bot bot commented Sep 18, 2025 •

edited

Loading

ssv-test-bot bot commented Sep 19, 2025 •

edited

Loading

ssv-test-bot bot commented Sep 19, 2025 •

edited

Loading

ssv-test-bot bot commented Sep 22, 2025 •

edited

Loading

`/test` - Run Tests

`/test skip-ci` - Skip CI Pipeline

ssv-test-bot bot commented Sep 22, 2025 •

edited

Loading

ssv-test-bot bot commented Sep 22, 2025 •

edited

Loading

ssv-test-bot bot commented Sep 30, 2025 •

edited

Loading

ssv-test-bot bot commented Oct 19, 2025 •

edited

Loading

ssv-test-bot bot commented Oct 20, 2025 •

edited

Loading

ssv-test-bot bot commented Oct 20, 2025 •

edited

Loading

ssv-test-bot bot commented Oct 20, 2025 •

edited

Loading

ssv-test-bot bot commented Oct 22, 2025 •

edited

Loading

`/test` - Run Tests

`/test skip-ci` - Skip CI Pipeline

ssv-test-bot bot commented Nov 23, 2025 •

edited

Loading