Would we consider running benchmarks for common Tap/Target combos?

[kgpayne]

In an effort to i) better inform Tap/Target users on the Hub and ii) better track and celebrate improvements to Tap/Target performance, would it be feasible to run and publish benchmark figures for common Tap/Target pairs (e.g. Postgres to Snowflake)?

Some thoughts:

• Benchmarks likely only need to run monthly or at major versions to be useful.
• This might initially be primarily marketing-driven, in the same vein as porting priority Taps/Targets to the SDK.
• Down the line, we might fund it using a sponsorship model akin to Patreon or Git Sponsors.
• We may be able to source anonymous real-world performance metrics via our extract-load block construct in meltano run as an alternative to formal benchmarking. I believe @pandemicsyn has already thought about monitoring/telemetry of tap/target pairs (including RAM consumption etc.).

[tayloramurphy]

@kgpayne in general I like the idea! Anything we can do to show the quality of specific connectors I think would be good. We'd be somewhat limited by access to systems with representative data, but perhaps with our managed offering we could get some of that data.

[aaronsteers]

I too really like the idea. To ensure the data is meaningful, ideally we'd isolate taps and targets separately for benchmarking.

This can be done more easily after BATCH messages are delivered, since those would allow taps to spin down after they finish emitting data, rather than becoming blocked on the target backpressure.

[aaronsteers]

We don't currently parse info metrics messages. We would likely need to start doing so.

See related conversation in the SDK:

• Tap and Target performance benchmarks sdk#906