StreamInsert benchmarking cannot have more than 8 producers (resolved in v3.3.0)


#1

I’m benchmarking ingestion throughput and am utilizing multiple simultaneous StreamInsert instances to insert data into a MapD database running on Amazon p2.8xlarge. I’m getting great benchmarks with up to 8 simultaneous producers but as soon as I add any additional producers the 9th/10th/etc… hang until one of the first 8 producer completes. I thought maybe this was due to having 8 gpus on a p2.8xlarge so I spun up a p2.16xlarge thinking 16 gpus might give me a different result but it had the same limitation. Given this, is there a way to tweak the configuration of MapD core in such a way that I could have more than 8 simultaneous StreamInsert producers or is there some other recommended approach? Thanks in advance for your assistance, Adam M.


#2

Hi,

What you are seeing is the limitation of number of concurrent connections.

Discussion here Unable to connect after a few hours

For older versions you can increase the limit, in new version (upcoming 3.3.0) the limitation has been removed.

regards


#3

Dwayne -
Thanks for the quick response.
I updated the mapd.conf.in file:

sudo vi /opt/mapd/systemd/mapd.conf.in
threadpool-size = 30

and added the threadpool-size entry into the file.

Then stopped and restarted mapd & mapd-web processes:

sudo systemctl stop mapd_server
sudo systemctl stop mapd_web_server

sudo systemctl start mapd_server
sudo systemctl start mapd_web_server

but it did not seem to have any change in behavior as my 9th producer still hangs until another one finishes. Do I have the property right, did I edit the wrong file or did I miss something else in the stop/start process?

Thanks again for the help on this,
Adam M.


#4

I also just tried:

tthreadpool-size = 30

with an extra ‘t’ in front since I thought that might not have been a typo in the other thread you referenced along with a stop/start but that did not resolve the issue either.


#5

I guess you have to disable backend rendering to have more than 8 concurrent sessions


#6

I think the issue here is that a different config file is being used. The mapd.conf.in file is the input template for the config file, which by default will get placed at /var/lib/mapd/mapd.conf. Since you’re using systemd, you might be able to determine the path to your config file by running:

systemctl status mapd_server | grep config

That being said, I am in the process of finishing the v3.3.0 builds, which remove this restriction. They should be up in the next hour or so.


#7

Thank you Andrew for getting the v3.3.0 builds up on the community AMI with this restriction removed. I spun up a new instance earlier this morning and verified that I no longer have this constraint with v3.3.0. Thanks again!