Increase the size of raidStorage


#1

I’m using the community edition of MapD AMI on AWS, and I’ve filled up the raid storage and was wondering what steps I need to do increase the storage size, any help or direction would be appreciated, thanks


#2

Hi @gridcell, I’m not sure if this is exactly what you’re looking for, but you might find the AWS documentation on resizing EBS volumes to be helpful here: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-modify-volume.html


#3

The current raidStorage size was 100GB which I have now hit and am now getting the error when running and sql statements.

Exception: Exception occurred: java.lang.RuntimeException: org.apache.thrift.TApplicationException: Inconsistency in num tuples within fragment

I’ve increased the size of the volumes in AWS console but doesn’t seem to be working with raid0

df -h

/dev/xvda1 300G 97G 204G 33% /
devtmpfs 30G 0 30G 0% /dev
tmpfs 29G 0 29G 0% /dev/shm
tmpfs 29G 3.0G 26G 11% /run
tmpfs 29G 0 29G 0% /sys/fs/cgroup
/dev/md0 99G 94G 0 100% /raidStorage
tmpfs 5.8G 0 5.8G 0% /run/user/1000

lsblk

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 300G 0 disk
└─xvda1 202:1 0 300G 0 part /
xvdb 202:16 0 200G 0 disk
└─md0 9:0 0 100G 0 raid0 /raidStorage
xvdc 202:32 0 200G 0 disk
└─md0 9:0 0 100G 0 raid0 /raidStorage
xvdd 202:48 0 200G 0 disk
└─md0 9:0 0 100G 0 raid0 /raidStorage
xvde 202:64 0 200G 0 disk
└─md0 9:0 0 100G 0 raid0 /raidStorage


#4

Have you tried growing the array or assembling from scratch to match your new sizing?
Something like: mdadm --assemble --update=devicesize --scan ?


#5

Most importantly, please BACKUP your data at the filesystem level before trying any of these commands.
I also see the following suggestions from the web to grow the raid0 volume and then follow it up by resizing your filesystem to make use of the added capacity:

mdadm --grow /dev/md0 --size=max

resize2fs /dev/md0 (assuming ext3/4)


#6

Am I able to copy my data directory from the old server and move to a new MapD EC2 instance (with more storage)?


#7

Yes, copying the MapD data folder and conf file to the new server should work.


#8

How to increase available disk space after launch
One way to add more free space would be to replace the existing RAID0 array (composed of /dev/xvdb ~ /dev/xvde) with a new larger volume (composed of /dev/xvdf ~ /dev/xvdi).

  • Using AWS consoles or other means, attach four new volumes to the running instance. You can add the volumes while the instancec is running.

  • Create a new logical volume via steps detailed below using mdadm, mkfs, /etc/fstab, and mount.

  • Stop mapd

  • Copy /raidStorage to the new volume.

  • Unmount the older, smaller volume from /raidStorage

  • Remount the new, larger volume at /raidStorage.

  • Reboot (if needed)

  • If mapd isn’t running, start it. Verify it’s condition. It should be running on the larger volume, now.

  • Using AWS consoles or other means, detach the old volumes and delete them.

Run these steps as root (sudo bash)…

  • Assumes new volumes are xvdf,h,i & j
    mdadm --verbose --create /dev/md1 --level=0 --raid-devices=4 /dev/xvd[f-i]
    echo “$(mdadm --detail --scan)” | tee -a /etc/mdadm.conf
    NOTE go edit /etc/mdadm.conf to remove duplicate entries.

  • Create file systems and mount point info
    mke2fs -t ext4 -F /dev/md1
    mkdir /raidStorage_new
    echo ‘/dev/md1 /raidStorage_new ext4 defaults,auto,noatime 0 0’ | tee -a /etc/fstab
    mount /raidStorage_new

  • shutdown mapd
    systemctl stop mapd_server
    systemctl stop mapd_web_server
    systemctl stop mapd_xorg

  • cp the data from old to new volume
    cp -r /raidStorage /raidStorage_new

  • Fix ownership
    chown -R -v ec2-user:ec2-user /raidStorage_new

  • In /etc/fstab, swap the /dev/md0 & /dev/md1 pointers to /raidStorage and /raidStorage_new so that /dev/md1 mounts to /raidStorage

  • Reboot server
    /sbin/reboot

  • start mapd, if needed
    systemctl start mapd_xorg
    systemctl start mapd_server
    systemctl start mapd_web_server

  • wait for mapd_server to start
    while netstat -lnt | awk ‘4 ~ /:9091/ {exit 1}’; do sleep 10; done
    echo “done”


#9

So I ended up creating a new instance and restoring data from the original source, since the server reboot never recovered.

My issue now is I have the new system running (community version) and everything was fine able to query add records, etc, I had to restart MapD, now when I run say…

SELECT * FROM activities limit 1;

It just hangs (~70M records in the table), any ideas?

Commands I used to restart:

sudo systemctl stop mapd_web_server
sudo systemctl stop mapd_server

sudo systemctl start mapd_server
sudo systemctl start mapd_web_server


#10

Hil @gridcell,

That is the right sequence. Output from “ps” and “df” would be helpful.

Regards,
Veda


#11

Can we just resize the existing volumes ?


#12

Was this issue resolved?