fix: properly rewrite errors for watch api, part 2

[miparnisari]

This is a continuation of #2640

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.41%. Comparing base (45508c2) to head (8cb6c91).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2656      +/-   ##
==========================================
- Coverage   79.47%   79.41%   -0.06%     
==========================================
  Files         455      455              
  Lines       47156    47158       +2     
==========================================
- Hits        37471    37444      -27     
- Misses       6934     6954      +20     
- Partials     2751     2760       +9     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[vroldanbet]

It would seem like Unavailable is a better error condition

From https://grpc.io/docs/guides/status-codes/

The service is currently unavailable. This is most likely a transient condition, which can be corrected by retrying with a backoff. Note that it is not always safe to retry non-idempotent operations.

Whereas FailedPrecondition says this

The operation was rejected because the system is not in a state required for the operation’s execution. For example, the directory to be deleted is non-empty, an rmdir operation is applied to a non-directory, etc. Service implementors can use the following guidelines to decide between FAILED_PRECONDITION, ABORTED, and UNAVAILABLE: (a) Use UNAVAILABLE if the client can retry just the failing call. (b) Use ABORTED if the client should retry at a higher level (e.g., when a client-specified test-and-set fails, indicating the client should restart a read-modify-write sequence). (c) Use FAILED_PRECONDITION if the client should not retry until the system state has been explicitly fixed. E.g., if an “rmdir” fails because the directory is non-empty, FAILED_PRECONDITION should be returned since the client should not retry unless the files are deleted from the directory.

[miparnisari]

Yeah i only had one reason for using FailedPrecondition, which was that we didn't alarm on it 😆 i've updated to use Unavailable now.

[tstirrat15]

LGTM