Errors after SELECT


#1

Hello,

I have read line by line the CentOS installation howto (http://docs.mapd.com/latest/getting-started/centos7recipe/) but I’m getting errors after executing the “SELECT” sentence. These errors are:

  1. in the “mapdql” console:
    [root@myserver mapd]# bin/mapdql
    Password:
    User mapd connected to database mapd
    mapdql> SELECT origin_city AS “Origin”, dest_city AS “Destination”, AVG(airtime) AS “Average Airtime” FROM flights_2008_10k WHERE distance < 175 GROUP BY origin_city, dest_city;
    Thrift: Wed Jun 28 08:50:21 2017 TSocket::write_partial() send() <Host: localhost Port: 9091>Broken pipe
    Thrift: Wed Jun 28 08:50:21 2017 TSocket::open() connect() <Host: localhost Port: 9091>Connection refused
    Thrift: Wed Jun 28 08:50:21 2017 TSocket::open() connect() <Host: localhost Port: 9091>Connection refused
    Thrift error: connect() failed: Connection refused
    mapdql>

  2. at /var/log/messages:
    Jun 28 08:50:21 myserver mapd_server: F0628 08:50:21.453549 2351 NvidiaKernel.cpp:93] Check failed: cuModuleLoadDataEx(&module_, image, num_options, options, option_vals) == CUDA_SUCCESS (300 vs. 0)
    Jun 28 08:50:21 myserver mapd_server: *** Check failure stack trace: ***
    Jun 28 08:50:21 myserver mapd_server: @ 0x19b155a google::LogMessage::Fail()
    Jun 28 08:50:21 myserver mapd_server: @ 0x19b14b1 google::LogMessage::SendToLog()
    Jun 28 08:50:21 myserver mapd_server: @ 0x19b0e7c google::LogMessage::Flush()
    Jun 28 08:50:21 myserver mapd_server: @ 0x19b3e15 google::LogMessageFatal::~LogMessageFatal()
    Jun 28 08:50:21 myserver mapd_server: @ 0xf9d0b8 GpuCompilationContext::GpuCompilationContext()
    Jun 28 08:50:21 myserver mapd_server: @ 0xf96d58 Executor::optimizeAndCodegenGPU()
    Jun 28 08:50:21 myserver mapd_server: @ 0xf992d5 Executor::compileWorkUnit()
    Jun 28 08:50:21 myserver mapd_server: @ 0xf4f49a Executor::ExecutionDispatch::compile()
    Jun 28 08:50:21 myserver mapd_server: @ 0xf401b8 Executor::executeWorkUnit()
    Jun 28 08:50:21 myserver mapd_server: @ 0xfdf6a8 RelAlgExecutor::executeWorkUnit()
    Jun 28 08:50:21 myserver mapd_server: @ 0xfe032d RelAlgExecutor::executeCompound()
    Jun 28 08:50:21 myserver mapd_server: @ 0xfe2c9d RelAlgExecutor::executeRelAlgStep()
    Jun 28 08:50:21 myserver mapd_server: @ 0xfe348f RelAlgExecutor::executeRelAlgSeq()
    Jun 28 08:50:21 myserver mapd_server: @ 0xfe4445 RelAlgExecutor::executeRelAlgQuery()
    Jun 28 08:50:21 myserver mapd_server: @ 0xe57e75 MapDHandler::execute_rel_alg()
    Jun 28 08:50:21 myserver mapd_server: @ 0xe5dcef (unknown)
    Jun 28 08:50:21 myserver mapd_server: @ 0xe5e68b MapDHandler::sql_execute_impl()
    Jun 28 08:50:21 myserver mapd_server: @ 0xe60d81 MapDHandler::sql_execute()
    Jun 28 08:50:21 myserver mapd_server: @ 0xdb462e MapDProcessor::process_sql_execute()
    Jun 28 08:50:21 myserver mapd_server: @ 0xda61b7 MapDProcessor::dispatchCall()
    Jun 28 08:50:21 myserver mapd_server: @ 0xda028c apache::thrift::TDispatchProcessor::process()
    Jun 28 08:50:21 myserver mapd_server: @ 0x2f5d45f apache::thrift::server::TConnectedClient::run()
    Jun 28 08:50:21 myserver mapd_server: @ 0x2f388a5 apache::thrift::concurrency::ThreadManager::Task::run()
    Jun 28 08:50:21 myserver mapd_server: @ 0x2f38de5 apache::thrift::concurrency::ThreadManager::Worker::run()
    Jun 28 08:50:21 myserver mapd_server: @ 0x2f5778b apache::thrift::concurrency::PthreadThread::threadMain()
    Jun 28 08:50:21 myserver mapd_server: @ 0x7efc3e664dc5 start_thread
    Jun 28 08:50:21 myserver mapd_server: @ 0x7efc3becf76d __clone
    Jun 28 08:50:21 myserver systemd: mapd_server.service: main process exited, code=killed, status=6/ABRT
    Jun 28 08:50:22 myserver systemd: Unit mapd_server.service entered failed state.
    Jun 28 08:50:22 myserver systemd: mapd_server.service failed.
    Jun 28 08:50:22 myserver systemd: mapd_server.service holdoff time over, scheduling restart.
    Jun 28 08:50:22 myserver systemd: Started MapD database server.
    Jun 28 08:50:22 myserver systemd: Starting MapD database server…

I accept all connections from “localhost”… and also I have run this example with iptables down and fails too…

I load CUDA/7.5 environment with “module load cuda/7.5” (I have also configured cuda/7.0 and cuda/8.0)

Inside “mapdql”, if I execute “\gpu” and run “SELECT”, I don’t receive nothing, but if I execute “\cpu” and run the same “SELECT”, I receive:
Origin|Destination|Average Airtime
West Palm Beach|Tampa|33.818182
Norfolk|Baltimore|36.071429
Ft. Myers|Orlando|28.666667
Indianapolis|Chicago|39.538462
Tampa|West Palm Beach|33.250000
Orlando|Ft. Myers|32.583333
Austin|Houston|33.055556
Chicago|Indianapolis|32.700000
Baltimore|Norfolk|31.714286
Houston|Austin|29.611111

so it seems problem is with nvidia and cuda. I suppose normal execution should be with “\gpu”…

Could anybody help me?

Thanks.


#2

Thanks for trying MapD.

Could you run nvidia-smi -q and post the output here?

I’m specifically interested in seeing what GPU model and what driver version you are using. We require Kepler or newer and do not run on older cards, such as the Fermi Tesla C2070.

We also require at least driver 375.51 in order to support all features (such as rendering).


#3

In my server there are two GPUs: one GeForce GTX 680 and one GeForce GTX 1080 Ti

This is the output of a “nvidia-smi -q” (with CUDA/7.5 environment loaded)

==============NVSMI LOG==============

Timestamp : Wed Jun 28 13:56:11 2017
Driver Version : 375.51

Attached GPUs : 2
GPU 0000:02:00.0
Product Name : GeForce GTX 680
Product Brand : GeForce
Display Mode : N/A
Display Active : N/A
Persistence Mode : Disabled
Accounting Mode : N/A
Accounting Mode Buffer Size : N/A
Driver Model
Current : N/A
Pending : N/A
Serial Number : N/A
GPU UUID : GPU-2750c43f-7842-59ae-8777-9bc794390c44
Minor Number : 0
VBIOS Version : 80.04.28.00.21
MultiGPU Board : N/A
Board ID : N/A
GPU Part Number : N/A
Inforom Version
Image Version : N/A
OEM Object : N/A
ECC Object : N/A
Power Management Object : N/A
GPU Operation Mode
Current : N/A
Pending : N/A
GPU Virtualization Mode
Virtualization mode : N/A
PCI
Bus : 0x02
Device : 0x00
Domain : 0x0000
Device Id : 0x118010DE
Bus Id : 0000:02:00.0
Sub System Id : 0x353C1458
GPU Link Info
PCIe Generation
Max : N/A
Current : N/A
Link Width
Max : N/A
Current : N/A
Bridge Chip
Type : N/A
Firmware : N/A
Replays since reset : 0
Tx Throughput : N/A
Rx Throughput : N/A
Fan Speed : 38 %
Performance State : P8
Clocks Throttle Reasons : N/A
FB Memory Usage
Total : 1996 MiB
Used : 567 MiB
Free : 1429 MiB
BAR1 Memory Usage
Total : N/A
Used : N/A
Free : N/A
Compute Mode : Default
Utilization
Gpu : N/A
Memory : N/A
Encoder : N/A
Decoder : N/A
Ecc Mode
Current : N/A
Pending : N/A
ECC Errors
Volatile
Single Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Texture Shared : N/A
Total : N/A
Double Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Texture Shared : N/A
Total : N/A
Aggregate
Single Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Texture Shared : N/A
Total : N/A
Double Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Texture Shared : N/A
Total : N/A
Retired Pages
Single Bit ECC : N/A
Double Bit ECC : N/A
Pending : N/A
Temperature
GPU Current Temp : 37 C
GPU Shutdown Temp : N/A
GPU Slowdown Temp : N/A
Power Readings
Power Management : N/A
Power Draw : N/A
Power Limit : N/A
Default Power Limit : N/A
Enforced Power Limit : N/A
Min Power Limit : N/A
Max Power Limit : N/A
Clocks
Graphics : N/A
SM : N/A
Memory : N/A
Video : N/A
Applications Clocks
Graphics : N/A
Memory : N/A
Default Applications Clocks
Graphics : N/A
Memory : N/A
Max Clocks
Graphics : N/A
SM : N/A
Memory : N/A
Video : N/A
Clock Policy
Auto Boost : N/A
Auto Boost Default : N/A
Processes : N/A

GPU 0000:03:00.0
Product Name : GeForce GTX 1080 Ti
Product Brand : GeForce
Display Mode : Disabled
Display Active : Disabled
Persistence Mode : Disabled
Accounting Mode : Disabled
Accounting Mode Buffer Size : 1920
Driver Model
Current : N/A
Pending : N/A
Serial Number : N/A
GPU UUID : GPU-9323406f-e04b-a1d9-e222-747d8f1e1468
Minor Number : 1
VBIOS Version : 86.02.39.00.2A
MultiGPU Board : No
Board ID : 0x300
GPU Part Number : N/A
Inforom Version
Image Version : G001.0000.01.04
OEM Object : 1.1
ECC Object : N/A
Power Management Object : N/A
GPU Operation Mode
Current : N/A
Pending : N/A
GPU Virtualization Mode
Virtualization mode : None
PCI
Bus : 0x03
Device : 0x00
Domain : 0x0000
Device Id : 0x1B0610DE
Bus Id : 0000:03:00.0
Sub System Id : 0x36091462
GPU Link Info
PCIe Generation
Max : 2
Current : 1
Link Width
Max : 16x
Current : 16x
Bridge Chip
Type : N/A
Firmware : N/A
Replays since reset : 0
Tx Throughput : 0 KB/s
Rx Throughput : 0 KB/s
Fan Speed : 29 %
Performance State : P8
Clocks Throttle Reasons
Idle : Active
Applications Clocks Setting : Not Active
SW Power Cap : Not Active
HW Slowdown : Not Active
Sync Boost : Not Active
Unknown : Not Active
FB Memory Usage
Total : 11172 MiB
Used : 728 MiB
Free : 10444 MiB
BAR1 Memory Usage
Total : 256 MiB
Used : 2 MiB
Free : 254 MiB
Compute Mode : Default
Utilization
Gpu : 0 %
Memory : 0 %
Encoder : 0 %
Decoder : 0 %
Ecc Mode
Current : N/A
Pending : N/A
ECC Errors
Volatile
Single Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Texture Shared : N/A
Total : N/A
Double Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Texture Shared : N/A
Total : N/A
Aggregate
Single Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Texture Shared : N/A
Total : N/A
Double Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Texture Shared : N/A
Total : N/A
Retired Pages
Single Bit ECC : N/A
Double Bit ECC : N/A
Pending : N/A
Temperature
GPU Current Temp : 34 C
GPU Shutdown Temp : 96 C
GPU Slowdown Temp : 93 C
Power Readings
Power Management : Supported
Power Draw : 8.97 W
Power Limit : 250.00 W
Default Power Limit : 250.00 W
Enforced Power Limit : 250.00 W
Min Power Limit : 125.00 W
Max Power Limit : 300.00 W
Clocks
Graphics : 164 MHz
SM : 164 MHz
Memory : 405 MHz
Video : 544 MHz
Applications Clocks
Graphics : N/A
Memory : N/A
Default Applications Clocks
Graphics : N/A
Memory : N/A
Max Clocks
Graphics : 1936 MHz
SM : 1936 MHz
Memory : 5505 MHz
Video : 1708 MHz
Clock Policy
Auto Boost : N/A
Auto Boost Default : N/A
Processes
Process ID : 3395
Type : C+G
Name : /soft/mapd/bin/mapd_server
Used GPU Memory : 726 MiB


#4

Ahh yes, that would also explain it: we don’t currently support mixing different models of GPUs, or at least different generations.

Try setting the num-gpus and start-gpu options in either the config file or at the command line, depending on how you’re running mapd_server. If you followed the install recipe exactly and placed the storage directory at /var/lib/mapd, then the config file should be at /var/lib/mapd/mapd.conf.

Add the following lines to select the second GPU (GPU 1 in nvidia-smi, should be the 1080Ti):

num-gpus = 1
start-gpu = 1

Place those in the main/unmarked section of the config file, above [web].


#5

Applying “num-gpus” and “start-gpu” parameters, MapD-Core works like a charm!!!

Thanks a lot!!!