How does MapD handle large dataset with "enable_watchdog=false"?


#1

This topic Is there a memory replacement mechanism for mapd? said:

The first option is to turn the watch dog off which will allow the query to run in stages on the GPU. MapD will then orchestrate the transfer of data through the various layers of our data abstraction to move the data onto the GPU for execution.
The second option is to set option allow-cpu-retry to direct queries, that do not fit in the GPU memory available, to fall back and be executed on the CPU.

The second option is easy to understand and I really found the if-clause branch in MapD-Core code. But I’m confused by the first, does “in stage” mean, for each relational algebra operation, transfer all required data from cpu memory to gpu, do query and copy results back to cpu?

If it is, can MapD handle rel-alg operations with two large columns which exceed gpu memory, or will fallback to cpu memory? (For example, not actual situation, inner join with two 20GB column on 20GB gpu mem)