Skip to content

bug: inputstreams not closed when inputstream not fully consumed #14

@rutchkiwi

Description

@rutchkiwi

Context

I'm got a project where I return s3 input streams as :body, as a way to send s3 data through my app without loading the entire file in memory all at once. This works fine most of the time. But when the http request is aborted early, .close is not being called on the inputStream. In the case of java s3, not closing the inputSteams is quite bad as in that case, the library stops working after 50 open requests.
Example:

.. 
(:require [amazonica.aws.s3 :as s3])
...
  (GET "/stream" _
    {:status  200
     :body    (:input-stream (s3/get-object {:endpoint region} bucket key))})
curl http://localhost:4000/stream --compressed | head -c 5 > /dev/null

The above will not correctly close the s3 stream - I've verified this using a debugger.

Suspicious code

https://github.com/clj-commons/ring-gzip-middleware/blob/4328168b640ce5006efcb11b371e5f1c9250204e/src/ring/middleware/gzip.clj#L37C1-L53C14

(defn piped-gzipped-input-stream [in]
  (let [pipe-in (piped-gzipped-input-stream*)
        pipe-out (PipedOutputStream. pipe-in)]
    ; separate thread to prevent blocking deadlock
    (future
      (with-open [out (if @flushable-gzip?
                        (GZIPOutputStream. pipe-out true)
                        (GZIPOutputStream. pipe-out))]
        (if (seq? in)
          (doseq [string in]
            (io/copy (str string) out) <-------- io/copy throws IOException when out is closed early
            (.flush out))
          (io/copy in out)))
      (when (instance? Closeable in)
        (.close ^Closeable in))) <--- in that case, .close isn't called! (should be in a finally)
    pipe-in))

Thanks for your attention!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions