Determining available capacity


#1

Is there any method to determine how much of the GPU/CPU memory capacity is currently in use by a dataset? As an example, we currently have 2 years of data loaded into an AWS p2.8xlarge (approx 1.3B rows in a single table). All queries so far run extremely quickly, but we’d like to be able to see how much RAM are we currently using, so we can estimate how much data more we could hold, or whether we could use a smaller instance size.

I appreciate there may be subtleties around the queries that are run potentially needing additional memory depending on their complexity, but I’m happy to assume that our current queries are relatively static.

Thanks,

Julian


#2

Hi,

Please give the mapdql memory commands a trial \memory_summary (and \memory_gpu for more detail than you probably want).

documented here https://www.mapd.com/docs/latest/mapd-core-guide/mapdql/?highlight=mapdql

The log will also make noise once queries start requiring data to be evicted so it is good to review.

Regards


#3

It’d be good to get these things into a virtual table and displayable by charting and Immerse, ideally with some amount of history to line this up with how various queries and perf line up with data movement into and out of GPU memory.