Error while dumping data from Spark to MapD?


#1

I am using Spark cluster to read the data from Azure data lake store and using some sparkSQL do some query over the data and I want to dump the data in MapD for Visualization in MapD immerse . I am able to read data from Azure data lake store but while dumping it to MapD i am getting some error which i am not able to rectify is it because of spark or because of MapD. I have already created the table in the MapD to dump the data .

I am getting this error
Exception in thread “main” java.sql.SQLException: Query failed : Syntax error at: "
at com.mapd.jdbc.MapDStatement.executeUpdate(MapDStatement.java:87)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.createTable(JdbcUtils.scala:692)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:89)
at org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:518)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:215)
at com.biz.Read_from_ADL.App.main(App.java:25)

And my dataset Schema is
root
|-- CurrencyPair: string (nullable = true)
|-- ExchangeName: string (nullable = true)
|-- MachineTime: string (nullable = true)
|-- OrderId: string (nullable = true)
|-- OrderSide: string (nullable = true)
|-- Price: double (nullable = true)
|-- Quantity: double (nullable = true)

public class App {
public static void main(String[] args) throws Exception {
	SparkSession spark = SparkSession.builder().appName("Java Spark SQL basic example").master("local[2]")
			.getOrCreate();

	Dataset<Row> df = spark.read().json("C:\\Users\\test\\Downloads\\BITFINEX_DSHBTC_ORDER.json");

	// df.show();

	df.createOrReplaceTempView("trade");
	Dataset<Row> sqlDF = spark.sql("SELECT * FROM trade");
	sqlDF.show();
	sqlDF.printSchema();
	sqlDF.write().format("jdbc").option("url", "jdbc:mapd:**.**.**.***:9091:mapd")
			.option("driver", "com.mapd.jdbc.MapDDriver").option("dbtable", "expample").option("user", "mapd")
			.option("password", "HyperInteractive").save();

}

}


#2

Hi,

It is reporting that he command being submitted by spark has syntax errors.

The error info is not very helpful here so I have added some additional error reporting and uploaded a new jdbc driver here

https://s3-us-west-2.amazonaws.com/mapd-artifact/mapdjdbc-1.0-SNAPSHOT-jar-with-dependencies.jar

Which should give us some more info to go on.

Please rerun with this driver and lets see what it is reporting the sql command is.

regards