It turned out our config was missing a few important parameters to start the Evictor thread in DBCP.
See
http://commons.apache.org/dbcp/configuration.htmltimeBetweenEvictionRunsMillis should be a positive value (-1 by default) to make the evictor thread run.
It's the evictor thread that will run the validation query and remove invalid connections from the pool.
K.