error: Error -3 while decompressing data: incorrect header check

[gregwjacobs]

Found an issue occuring when testing use of cyclops with Raven client 5.20.0

I get the following error in -vvv debug logs of cyclops:

2016-06-09 11:25:57,150 - tornado.application - ERROR - Uncaught exception POST /api/22/store/ (10.26.10.38)
HTTPRequest(protocol='http', host='prod-vm-sentry-cyclops-01', method='POST', uri='/api/22/store/', version='HTTP/1.1', remote_ip='10.26.10.38', headers={'Content-Length': '261', 'Accept-Encoding': 'identity', 'Content-Encoding': 'deflate', 'Connection': 'close', 'X-Sentry-Auth': 'Sentry sentry_timestamp=1465485957.17, sentry_client=raven-python/5.8.1, sentry_version=6, sentry_key=xxxxxxxxxxxxxxxxx, sentry_secret=yyyyyyyyyyyyyyyyyy', 'User-Agent': 'raven-python/5.8.1', 'Host': 'prod-vm-sentry-cyclops-01', 'Content-Type': 'application/octet-stream'})
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/tornado/web.py", line 1320, in _stack_context_handle_exception
    raise_exc_info((type, value, traceback))
  File "/usr/local/lib/python2.7/dist-packages/tornado/web.py", line 1503, in wrapper
    result = method(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/cyclops/handlers/router.py", line 174, in post
    self.backend_request(project_id)
  File "/usr/local/lib/python2.7/dist-packages/cyclops/handlers/router.py", line 99, in backend_request
    payload = loads(decompress(b64decode(self.request.body)))
error: Error -3 while decompressing data: incorrect header check
2016-06-09 11:25:57,151 - tornado.access - ERROR - 500 POST /api/22/store/ (10.26.10.38) 1.38ms

I send the following test payload via raven to produce this error (which is sane, for sentry 8.x.x versions I use in labs prior.

>>> DSN = 'http://XXXXXXXX:YYYYYYY@prod-vm-sentry-cyclops-01/22'
>>> import raven
>>> client = raven.Client(dsn=DSN)
>>> client.captureMessage("This is a test from raven via cyclops")
'a5fcc36f71e04dff94de11f1ec034c30'

I get a error 500 returned by the cyclops server (as expected given the exception) to the test client.

I presently am using raven client 5.20.00 and Cyclops was installed from the pip 1.1.1 version

Name: raven
Version: 5.20.0
Summary: Raven is a client for Sentry (https://getsentry.com)
Home-page: https://github.com/getsentry/raven-python

How else can I help resolve this. Do changes in Raven payloads require changes in cyclops,or is this an issue with compression/decompression and/or base64 encode/code libs of the client or server?

[gregwjacobs]

I also am seeing:

2016-06-09 10:57:29,724 - root - INFO - -- Cyclops started listening in 0.0.0.0:80 --
2016-06-09 10:57:50,245 - tornado.application - ERROR - Uncaught exception POST /api/22/store/ (10.26.10.38)
HTTPRequest(protocol='http', host='prod-vm-sentry-cyclops-01', method='POST', uri='/api/22/store/', version='HTTP/1.1', remote_ip='10.26.10.38', headers={'Content-Length': '252', 'Accept-Encoding': 'identity', 'Content-Encoding': 'deflate', 'Connection': 'close', 'X-Sentry-Auth': 'Sentry sentry_timestamp=1465484270.22, sentry_client=raven-python/5.8.1, sentry_version=6, sentry_key=xxxxxxxxxxxxxxxxxxxxx, sentry_secret=yyyyyyyyyyyyyyyyyyyyyy', 'User-Agent': 'raven-python/5.8.1', 'Host': 'prod-vm-sentry-cyclops-01', 'Content-Type': 'application/octet-stream'})
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/tornado/web.py", line 1320, in _stack_context_handle_exception
    raise_exc_info((type, value, traceback))
  File "/usr/local/lib/python2.7/dist-packages/tornado/web.py", line 1503, in wrapper
    result = method(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/cyclops/handlers/router.py", line 174, in post
    self.backend_request(project_id)
  File "/usr/local/lib/python2.7/dist-packages/cyclops/handlers/router.py", line 99, in backend_request
    payload = loads(decompress(b64decode(self.request.body)))
  File "/usr/lib/python2.7/base64.py", line 76, in b64decode
    raise TypeError(msg)
TypeError: Incorrect padding

Still looking deeper. So far I can confirm wireshark can properly decompress and decode the base64 payload and the JSON data is formatted correctly.

[gregwjacobs]

Using:

> python --version
> Python 2.7.6

[gregwjacobs]

I found the likely issue. This patch gets system working.

/cyclops/handlers/router.py

5c5
< import zlib
---
> from zlib import decompress
99c99
<             payload = loads(zlib.decompress(self.request.body, zlib.MAX_WBITS|32))
---
>             payload = loads(decompress(b64decode(self.request.body)))

Looks like the logic ideally should do the following

• Handle direct payloads (does that already)
• Handle variant types of compressed payloads (use the zlib.MAX_WBITS|32 parameter?
• Auto-detect if base64 encoded and decode, else just pass it onward

I can work on a Pull Request with a clean fix if you'd like?

[gregwjacobs]

I should note, this was done directly, no use of the nginx instance in front of cyclops. Is it possible the data presented via nginx differs hitting cyclops directly? (will test on my end also)

[gregwjacobs]

I confirmed this still occurs when being used verbatim with cyclops setup example (behind nginx reverse proxy)

2016-06-14 10:31:03,247 - tornado.application - ERROR - Uncaught exception POST /api/22/store/ (127.0.0.1)
HTTPRequest(protocol='http', host='gateway', method='POST', uri='/api/22/store/', version='HTTP/1.0', remote_ip='127.0.0.1', headers={'Content-Length': '433', 'Accept-Encoding': 'identity', 'Content-Encoding': 'deflate', 'Host': 'gateway', 'X-Sentry-Auth': 'Sentry sentry_timestamp=1465914663.07, sentry_client=raven-python/5.20.0, sentry_version=6, sentry_key=xxxxxxxxxxxxxxxxx, sentry_secret=yyyyyyyyyyyyyy', 'User-Agent': 'raven-python/5.20.0', 'Connection': 'close', 'Content-Type': 'application/octet-stream'})
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/tornado/web.py", line 1320, in _stack_context_handle_exception
    raise_exc_info((type, value, traceback))
  File "/usr/local/lib/python2.7/dist-packages/tornado/web.py", line 1503, in wrapper
    result = method(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/cyclops/handlers/router.py", line 174, in post
    self.backend_request(project_id)
  File "/usr/local/lib/python2.7/dist-packages/cyclops/handlers/router.py", line 99, in backend_request
    payload = loads(decompress(b64decode(self.request.body)))
  File "/usr/lib/python2.7/base64.py", line 76, in b64decode
    raise TypeError(msg)
TypeError: Incorrect padding