Reservoir Pooling Sampling

[dcramer]

I'm considering not building sentry-relay and just pushing this project more, but one of the primary reasons for wanting a relay for myself (and some orgs like Disqus) is to do sampling before it gets sent to the server.

What's your thoughts on adding something like reservoir sampling to the app?

[heynemann]

I read the wikipedia entry on reservoir sampling. We would still need to store all the errors in memory, right (in order to do the sampling)?

If you take a look at https://github.com/heynemann/cyclops/blob/master/cyclops/handlers/router.py#L40 you'll see that we already do a kind of sampling. If we are being bombarded by the same URL, during URL_CACHE_EXPIRATION seconds we'll allow at most MAX_CACHE_USES to be processed.

I know that our documentation is lacking, but in our load test with a HUGE number of messages (1.5 million in 5 minutes, I think), nearly 5% of those reached sentry. The ones that did, reached it in a controlled manner that allowed sentry to digest them.

I'm not saying I'm against adding reservoir sampling, but how would you do it without increasing the memory used (we have SEVERAL products that are going to be using the same farm)? Besides, do you think we could do it as a configuration? If I want sampling I can enable it.

Thanks a LOT for considering contributing. I also think that having one relay is less confusing for users than having multiple. I'd gladly give you commit status in cyclops, if you'd like it.

We are in very early stages, though. We wanted to prove first that it could work. Now that it does work, we need to:

• make sure that it's tested
• that all interfaces are covered
• that we store requests in memory, but in an extensible way, to allow people to implement other types of storage

If you are willing, welcome aboard!

[heynemann]

Hey David, we just updated Cyclops docs and added a comprehensive test suite to it.

In the light of that I said in the previous post, do you still think we should add reservoir sampling?

[heynemann]

Hey @dcramer,

I know this it's been a while, but have you considered adding cyclops to sentry's docs? I haven't checked it there, so might be asking for something already done.

We've been using cyclops for an year now and haven't had to restart either its servers or sentry's for a HUGE amount of messages.

At the time of this writing we have processed over 128 million messages in cyclops. I would say that's battle tested.

Thanks a lot for all your great work with Sentry.

[dcramer]

It hasn't been a huge deal since we're aggressively improving performance/rate limiting on the server and other than network latency (which isnt a probably in local network) it's extremely fast.

I think Cyclops is listed somewhere in the docs, but I forget where.

[heynemann]

Do you have any statistics on the amount of trafic you are getting (number of errors/sec)?

Because at our last tests even with a big server (24 cores and 64GB RAM) running sentry with celery we got very low error/sec processing.

I'm guessing we configured it wrong. That's why I'm curious to how hard you are hitting. Might be worth investigating what we are doing wrong.