Unable to connect after a few hours


#1

I’m running the AWS marketplace instance on a p2.xlarge. I have a single table with only 5000 rows of data. I’m running 2 java apps which connect to mapd via JDBC (one app performs only inserts, the other only queries) and pool connections.

After some period of time (around 6 hours or so), I am no longer able to connect to the database via mapdql (the process just hangs indefinitely with no output), nor via a new JDBC connection (again it just hangs). The apps themselves will continue running unaffected (presumably because of the pooled connections). However, if I stop one and start it again, it is no longer able to connect. In order to allow additional connections, I need to restart the mapd server (I kill the process and a new one starts automatically).

There are no relevant messages in syslog, but is there somewhere else I need to look to try and solve this. I have been running another mapd instance with a much larger dataset handling queries and a set of bulk inserts but never saw any issues. Any ideas as to why I’m seeing this issue?

Thanks,

Julian


Use of the JDBC Driver in a connection pool
Unable to connect to Hostname or Port
#2

Hi,

Sorry to hear you are experiencing an issue.

MapD by default allows for only 8 concurrent thrift sessions (what you would be using using to connect via JDBC), this is controlled by the tthreadpool-size parameter on the server start. Based on the description of your issue it sounds like over time you are opening and not closing these sessions completely and eventually MapD is running out.

Short term workaround is to increase the threadpool-size parameter to allow more sessions (this is not an issue so long as you do not have backend rendering in use, which from your description you do not).

Long term fix is for you to review the log and see if you are seeing sessions being created that never get closed. Perhaps just seeing when new sessions are established will help you identify which client it may be that is causing the issue.

regards


StreamInsert benchmarking cannot have more than 8 producers (resolved in v3.3.0)
#3

Dwayne,

Thanks for this, it gives me a probable explanation for the behaviour:

The app connection pools are configured to with a minimum and maximum size. The minima in the 2 apps are 2 & 4 for the query and insert apps respectively (maxima are 8 for each) so I automatically start off with 6 sessions and at that point have capacity to connect from mapdql. However, as the query app is used it generally receives multiple simultaneous requests from the ui so those initial 2 sessions will likely grow - once it hits 4 no additional sessions will be available and I’ll see the hanging sessions.

Additional questions:

  • Which log were you talking about when you said to review sessions being created?
  • Where do I change the threadpool-size parameter?
  • Should I raise a github issue to request that when no further thrift sessions are available, an error detailing the reason is returned rather than just hanging?

Thanks again,

Julian


#4

Hi,

Yes from your description it is more than likely the connection pools that is holding all your connections.

Logs are in <MAPDHOME>/data/mapd_logs/, the log you are interested to look at connects and disconnects would be the INFO logs specifically.

Add the tthreadpool-size=nn to the main part of your mapd.conf assuming you are using systemd to run your system.

We are aware of the poor messaging situation when the thrift thread pool is exhausted, and are planning to remove the limitation in a more complete way in an up coming release. But feel free to raise an issue as a tracker, or even better if you think you can take the coding for the change on.

regards