SQLImporter behavior: long wait before starting ingesting


#1

MapD-3.1.1

I cannot understand why the SQLImporter waits so long before starting importing data: between 10 and 40 minutes.

This is an example of data ingesting:

 java -cp /opt/mapd/mapd-ce-3.1.1/bin/mapd-1.0-SNAPSHOT-jar-with-dependencies.jar:/opt/jdbc_driver/MySql/mysql-connector-java-5.1.39/mysql-connector-java-5.1.39-bin.jar com.mapd.utility.SQLImporter -t <tablename> -sp ... -su ... -c  jdbc:mysql://servername:3306/databasename -ss "select * from <tablename>"

2017-07-06 15:36:09 INFO SQLImporter:executeQuery:188 - Connecting to database url :jdbc:mysql://servername:3306/databasename

Thu Jul 06 15:36:10 CEST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.

I have about 25 millions rows in the table to be imported: after a while, the import begins and it is fine.


#2

Hi,

This behavior is normally associated with the DB you are extracting the data from.

I suspect your mysql takes along time before it allows the resultset to be read on that select * you are running. We by default request setFetchSize to 10,000 if no parameter is set, but some DB’s do not start passing records until they are fully ready.

The more complex your select statement the slower mysql is going to probably be producing the result set. Are all your extracts just select *

regards


#3

I confirm this is a behavior unrelated to MapD: I am further investigating on MySQL side.


#4

yes you are right it’s the normal behaviour of mysql