Test "test_schema_await_with_unreachable_node" is flaky

[Lorak-mmk]

It sometimes fails on dropping keyspace.
I did not investigate it a lot, but I see some possible issues with it:

• It assumes that dropping must work, but it is intended that even creating keyspace won't always work (see callsites of run_some_ddl_with_unreachable_node). Perhaps we should have a separate session for dropping?
• Test case 4 assumes that control connection is still opened to node 0. It could be broken by case 3.

[Lorak-mmk]

Happened again: https://github.com/scylladb/scylla-rust-driver/actions/runs/18713576618/job/53367519182?pr=1445

[wprzytula]

Failed again: https://github.com/scylladb/scylla-rust-driver/actions/runs/19110912929/job/54607451260?pr=1458

[Lorak-mmk]

After reading the test again I don't see how I came to the original conclusions.
Proxy rules exclude control connection, so it should never get broken.
Proxy rules are disabled before DROP KEYSPACE, so it should not get affected.
Proxy rules only match requests containing system.local, so they should not prevent CREATE KEYSPACE from running.

Both of the failing runs above are for Cassandra - maybe there is some Cassandra-specific issue?

[Lorak-mmk]

I had an idea that maybe a connection is dropped in one test case, and session is not refilling it before next test case.
However, run sent by @wprzytula is failing on a first test case, which is really surprising.
Posting the log archive of this run below so that it doesn't get lost.

logs_49215947399.zip

I looked at it, and afaict this is what is happening:

1. Session is correctly established, with connections to all 3 nodes through proxt (let's call them nodes 0, 1, 2).
2. We enter the first test case, and change proxy rules accordingly
3. CREATE KEYSPACE is sent to node 1 (as expected), and reaches the server
4. Non-error result is sent back from the server, and reaches the driver, so the keyspace was successfully created
5. Driver tries to await schema agreement, so it send a query to system.local to all 3 nodes
6. Query to node 1 is intercepted by the proxy, and connection dropped
7. Other queries complete successfully
8. Driver does not attempt another agreement check
9. Driver sends DROP KEYSPACE
10. Connection that was broken in step 6 is re-opened. I don't think it matters, but adding it for completeness
11. Driver receives error from DROP KEYSPACE
12. Driver receives two EVENTs related to schema changes in other tests
13. Driver receives EVENT about keyspace that we created in step 3 (!)
14. Driver reports error about keyspace we tried to drop not existing.

So, I see potentially two issues:

• Issue with a test: DROP KEYSPACE needs schema with keyspace to be propagated to the node we send it to. We can achieve this by either performing schema agreement after clearing proxy rules, or sending DROP KEYSPACE to the same node that we sent CREATE KEYSPACE to
• Issue in the driver, not connected to those failures. check_schema_agreement_with_required_node correctly returns errors. await_schema_agreement_indefinitely uses ? so it propagates the errors always. Maybe this is not a good approach? Errors like broken connections are potentially transient, so another check after some time could succeed. I'll open new issue about that.