java.lang.IllegalStateException - Channel is closed

[donbonifacio]

I can't replicate this and don't understand it yet. Sometimes I have the java.lang.IllegalStateException - Channel is closed exception when getting results. For example:

(let [sql (str "select model from " (:table config) " where id = $1")]
    (async-go config sql
      (let [db (config/get-connection config)
            response (async/<! (query! db [sql model-id]))]

(It's not only onquery!, also got this on update!.)

Any ideas?

[donbonifacio]

Here's the full stacktrace with your code:

at com.github.pgasync.impl.netty.NettyPgProtocolStream.send(NettyPgProtocolStream.java:83)
at com.github.pgasync.impl.PgConnection.query(PgConnection.java:91)
at com.github.pgasync.impl.PgConnectionPool.lambda$query$14(PgConnectionPool.java:77)
at com.github.pgasync.impl.PgConnectionPool$$Lambda$10/847936826.accept(:0)
at com.github.pgasync.impl.PgConnectionPool.getConnection(PgConnectionPool.java:143)
at com.github.pgasync.impl.PgConnectionPool.query(PgConnectionPool.java:76)
at postgres.async/execute!(async.clj:46)
at postgres.async/query!(async.clj:57)
at clojure.lang.AFn.applyToHelper(AFn.java:160)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.core/apply(core.clj:628)
at postgres.async.impl/result-chan(impl.clj:17)
at clojure.lang.RestFn.invoke(RestFn.java:439)
at postgres.async/query!(async.clj:55)

[alaisi]

Thanks, I'll try to reproduce the error. The connection pool needs connection validation support to catch cases like this.

[donbonifacio]

@alaisi here's my use case: when my (config/get-connection ...) is called for the first time, it calles open-db and puts the conn-pool on an atom. All next calls will get that pool. This is on a long running web server, and all requests go through that global pool.

[donbonifacio]

More info: we have 2 services running but we only have this problem on one of them. After checking out heroku I noticed that the postgres versions are different. On PostgreSQL 9.4.3 we never got that error, we only have it on PostgreSQL 9.4.1. That's the only significant difference between the two services.

[alaisi]

Tracked issue: alaisi/postgres-async-driver#7

[donbonifacio]

Hey @alaisi ,
Any news on this? Is there any workaround or something I can help with?

[alaisi]

Hi,

can you test with the the master branch version to see if it fixes the problems you've been having. Configure the connection pool with:

(open-db {:hostname ...
          :validation-query "select 1" })

[donbonifacio]

It's not easy to test this, we get this randomly on production and don't have a always fail test scenario.

I did run this on one of our services and got no errors but a test suite that usually takes 6s ended up taking 65s.

[alaisi]

Thanks for the info. I'm looking into the performance issue.

Is the test suite using a single db on an atom or is an instance opened/closed for each test or ns? close-db is now blocking until all sockets are closed and IO thread is stopped (usually several seconds) so that could be one possible cause.

[donbonifacio]

Hum.. on this particular service we do use component and start/stop the system on every test scenario that needs the DB. Any workaround for this?

[alaisi]

You could wrap close-db in (thread) or (thread-call) to get similiar semantics as before, eg. the component is considered closed when the db has started to shut down.

[donbonifacio]

@alaisi I tried close on another thread and the speed is back to normal. So we'll adjust our tests to a global conn.