Exception: Columnar conversion not supported for variable length types


#21

OK I’ve created the sharded tables and successfully loaded data into them via the COPY command.

mapdql> \t
stories
stories_shard_#1
stories_shard_#2
stories_shard_#3
stories_shard_#4
stories_shard_#5
stories_shard_#6
stories_shard_#7
stories_shard_#8
stories_shard_#9
stories_shard_#10
profiles
profiles_shard_#1
profiles_shard_#2
profiles_shard_#3
profiles_shard_#4
profiles_shard_#5
profiles_shard_#6
profiles_shard_#7
profiles_shard_#8
profiles_shard_#9
profiles_shard_#10

mapdql> select count(*) from profiles;
EXPR$0
1010930

mapdql> select count(*) from stories;
EXPR$0
1400966

But I’m still getting the same error as before, but now I’m getting it with only one day’s worth of data loaded as opposed to getting it when I had 6 days of data loaded. So it seems like the sharding had the opposite effect as intended?

mapdql> select count(p.id) from stories s, profiles p where s.dma = '501' and s.profile_id = p.id and 'interest:hispanic' = ANY p.audience_ids;
Exception: Columnar conversion not supported for variable length types

#22

Hi,

Sorry about that, as mentioned above this a new feature we are just rolling out and it is not finalized yet.

Will take a look at it and increase my sample array sizes to see if I can reproduce the issue and go from there.

thanks for your patience

regards


#23

Hi,

Could I get you to gather a little more info for me.

Please confirm you are loading exactly the same data in the same order as you were with the 6 day load.

Please run the query causing the issue.

Then in mapdql run the following

\clear_gpu

then re run only the problem causing query.

then run

\memory_gpu

And include the full output here

This will let us see what is going on fragment wise on the GPU’s

regards


#24

Hi,

How many GPUs do you have in this machine.

Please reset the shard count to match the number of GPUs and give the load and query a retry.

This is not going to solve the super high volume question (which should be addressed next week) but it should allow you to at least load num_of_GPUs x your current max load amount.

regards


#25

@dwayneberry yes, it’s the same exact data in the same exact order as the 6 day load (it’s scripted).

mapdql> select count(p.id) from stories s, profiles p where s.dma = '501' and s.profile_id = p.id and 'interest:hispanic' = ANY p.audience_ids;
Exception: Columnar conversion not supported for variable length types

mapdql> \clear_gpu
MapD Server GPU memory Cleared 

mapdql> select count(p.id) from stories s, profiles p where s.dma = '501' and s.profile_id = p.id and 'interest:hispanic' = ANY p.audience_ids;
Exception: Columnar conversion not supported for variable length types

mapdql> \memory_gpu
MapD Server GPU Detailed Memory Usage 
Slabs Contents:  GPU_MGR:0
Slab St.Page   Pages  Touch
   0       0   15797      7 FREE 
   0   15797    1093      8 PC:  0 USED - Chunk: 2,2,2,0,
   0   16890    1093      9 PC:  0 USED - Chunk: 2,2,14,0,
   0   17983 4176321      0 FREE 
--------------------

mapdql> \memory_summary
MapD Server Memory Usage 
CPU RAM IN BUFFER USE :      9.52 MB
GPU VRAM USAGE (in MB's)
GPU     MAX    ALLOC    IN-USE     FREE
 0 11311.88  2048.00      1.07 11310.81

This is on a p2.xlarge on AWS which has only one GPU.

$ nvidia-smi
Thu Jul 27 13:38:08 2017       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.66                 Driver Version: 375.66                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K80           Off  | 0000:00:1E.0     Off |                    0 |
| N/A   64C    P0    58W / 149W |   2112MiB / 11439MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0     17382    C   ./bin/mapd_server                             2110MiB |
+-----------------------------------------------------------------------------+

#26

I’m trying to do this, but when I drop the database and recreate it, the old tables are still there:

./mapdql mapd
Password: 
User mapd connected to database mapd
mapdql> drop database insights_profile;
mapdql> create database insights_profile;
mapdql> \q
User mapd disconnected from database mapd

./mapdql insights_profile
Password: 
User mapd connected to database insights_profile
mapdql> \t
stories
stories_shard_#1
stories_shard_#2
stories_shard_#3
stories_shard_#4
stories_shard_#5
stories_shard_#6
stories_shard_#7
stories_shard_#8
stories_shard_#9
stories_shard_#10
profiles
profiles_shard_#1
profiles_shard_#2
profiles_shard_#3
profiles_shard_#4
profiles_shard_#5
profiles_shard_#6
profiles_shard_#7
profiles_shard_#8
profiles_shard_#9
profiles_shard_#10

If I try to drop any of the tables I can’t.

mapdql> drop table stories;
Exception: Sqlite3 Error: attempt to write a readonly database
mapdql> drop table profiles;
Exception: Sqlite3 Error: attempt to write a readonly database

It seems stuck. I’ll trash this instance and spin up a new one in the mean time to test the different shard value. Stay tuned…


#27

@dwayneberry ok I spun up a new instance and set the shard count to match the number of GPUs (1). Now the query returns with 1 day’s worth of data.

mapdql> select count(p.id) from stories s, profiles p where s.dma = '501' and s.profile_id = p.id and 'interest:hispanic' = ANY p.audience_ids;
EXPR$0
28541
mapdql> \timing
mapdql> select count(p.id) from stories s, profiles p where s.dma = '501' and s.profile_id = p.id and 'interest:hispanic' = ANY p.audience_ids;
EXPR$0
28541
1 rows returned.
Execution time: 86 ms, Total time: 86 ms

#28

I loaded 6 days worth of data with the shard count set to 1, and it fails at 6 days worth of data as it did before.

mapdql> select count(p.id) from stories s, profiles p where s.dma = '501' and s.profile_id = p.id and 'interest:hispanic' = ANY p.audience_ids;
Exception: Columnar conversion not supported for variable length types

mapdql> \memory_summary
MapD Server Memory Usage 
CPU RAM IN BUFFER USE :    233.82 MB
GPU VRAM USAGE (in MB's)
GPU     MAX    ALLOC    IN-USE     FREE
 0 11311.88  2048.00    225.49 11086.39

mapdql> 
mapdql> \memory_gpu
MapD Server GPU Detailed Memory Usage 
Slabs Contents:  GPU_MGR:0
Slab St.Page   Pages  Touch
   0       0   29237    474 FREE 
   0   29237   29236    425 PC:  0 USED - Chunk: 2,4,6,0,2,
   0   58473   45429    472 FREE 
   0  103902   29236    419 PC:  0 USED - Chunk: 2,4,1,0,
   0  133138  156281    473 FREE 
   0  289419   71709    475 PC:  0 USED - Chunk: 2,2,2,0,
   0  361128   71709    476 PC:  0 USED - Chunk: 2,2,14,0,
   0  432837    1862      0 FREE 
   0  434699  259906    424 PC:  0 USED - Chunk: 2,4,6,0,1,
   0  694605 3499699      0 FREE 
--------------------

#29

Hi

This sharding work around is only going to help if you have more than one GPU.

You will have to wait for the full fix.

Thanks for the additional information, it does put a ? around some of the assumptions as it looks like you are still well within one fragment. We will be looking at that in parallel.

regards


#30

@dwayneberry Cool, thanks. Let me know if you need any additional info.


#31

@dwayneberry just checking in to see if you have any updates on this issue? Also let me know if it would be helpful to open an issue on the Github repo for this (don’t want it to get lost in the weeds of the forum).

Thanks!


#32

Hi

Work is underway, we will keep you up to date on progress.

regards


#33

Hi Dwayne,
I met the same error, but unlikely this is related to the volume of data.

I have 5 little tables from 600 to 6000 rows, and from 3 to 5 columns.

I can join all the tables, and add a last filter on a text column without problems.
But if I create a view with the same query, and then add the last filter to the view, I get the error:

Columnar conversion not supported for variable length types

I am keen to share other details in private mode