-
Notifications
You must be signed in to change notification settings - Fork 18
Open
Description
Context
I'm got a project where I return s3 input streams as :body, as a way to send s3 data through my app without loading the entire file in memory all at once. This works fine most of the time. But when the http request is aborted early, .close is not being called on the inputStream. In the case of java s3, not closing the inputSteams is quite bad as in that case, the library stops working after 50 open requests.
Example:
..
(:require [amazonica.aws.s3 :as s3])
...
(GET "/stream" _
{:status 200
:body (:input-stream (s3/get-object {:endpoint region} bucket key))})
curl http://localhost:4000/stream --compressed | head -c 5 > /dev/null
The above will not correctly close the s3 stream - I've verified this using a debugger.
Suspicious code
(defn piped-gzipped-input-stream [in]
(let [pipe-in (piped-gzipped-input-stream*)
pipe-out (PipedOutputStream. pipe-in)]
; separate thread to prevent blocking deadlock
(future
(with-open [out (if @flushable-gzip?
(GZIPOutputStream. pipe-out true)
(GZIPOutputStream. pipe-out))]
(if (seq? in)
(doseq [string in]
(io/copy (str string) out) <-------- io/copy throws IOException when out is closed early
(.flush out))
(io/copy in out)))
(when (instance? Closeable in)
(.close ^Closeable in))) <--- in that case, .close isn't called! (should be in a finally)
pipe-in))
Thanks for your attention!
Metadata
Metadata
Assignees
Labels
No labels