How to run MapD as a Service after Github Installation


#1

Hi All,

I am installing the MapD in CentOS 7.

  1. Would like to know how we can run the MapD as a Service. so we can manage MapD server as service (systemctl)- Start, Stop,Status

  2. At present, we can start MapD Server with command “./bin/mapd_server”. However not sure how to stop the MapD Server gracefully.

  3. When changed the Data path using “./bin/initdb /opt/Data”. Not able to start the MapD server.

Kindly assist.

Regards,
Sumit


#2

do you mean OS version of mapd?

you should have a subdirectory under directory contanining various sources called systemd

just run the script install_mapd_systemd.sh inside of it with appropriate permissions


#3

Hi @aznable,

I am getting following Error when i am running the script

[sumit@home01 systemd]$ sudo ./install_mapd_systemd.sh
MAPD_PATH: MapD install directory
[/opt/mapd-core-master]: /opt/mapd-core-master/build

MAPD_STORAGE: MapD data and configuration storage directory
[/var/lib/mapd]: /opt/datafiles

MAPD_USER: user MapD will be run as
[root]:

MAPD_GROUP: group MapD will be run as
[root]:

MAPD_PATH: /opt/mapd-core-master/build
MAPD_STORAGE: /opt/datafiles
MAPD_USER: root
MAPD_GROUP: root
/opt/mapd-core-master/build/bin/initdb: error while loading shared libraries: libarrow.so.0: cannot open shared object file: No such file or directory

[sumit@home01 systemd] ll /opt/mapd-core-master/build/bin/ total 186448 -rw-r--r--. 1 root root 24428417 Jul 9 16:08 calcite-1.0-SNAPSHOT-jar-with-dependencies.jar -rwxr-xr-x. 1 root root 2037744 Jul 9 16:03 DataGen -rwxr-xr-x. 1 root root 2981279 Jul 9 16:03 generate_cert -rwxr-xr-x. 1 root root 49941912 Jul 9 16:07 initdb -rwxr-xr-x. 1 root root 3680136 Jul 9 16:04 KafkaImporter -rw-r--r--. 1 root root 30594538 Jul 9 16:08 mapd-1.0-SNAPSHOT-jar-with-dependencies.jar -rw-r--r--. 1 root root 3709729 Jul 9 16:08 mapdjdbc-1.0-SNAPSHOT-jar-with-dependencies.jar -rwxr-xr-x. 1 root root 2855680 Jul 9 16:03 mapdql -rwxr-xr-x. 1 root root 52043040 Jul 9 16:07 mapd_server -rwxr-xr-x. 1 root root 11571031 Jul 9 16:02 mapd_web_server -rwxr-xr-x. 1 root root 2731936 Jul 9 16:03 StreamImporter -rwxr-xr-x. 1 root root 2290080 Jul 9 16:03 StreamInsert -rwxr-xr-x. 1 root root 2038224 Jul 9 16:03 StreamInsertSimple [sumit@home01 systemd]

You mentioned about appropriate permission, please let us know if any other permission i need to give.

Please advise.


#4

it’s not able to find libarrow

try
ldd initdb | grep arrow

the output should somethign like this
libarrow.so.0 => /usr/local/mapd-deps/lib/libarrow.so.0 (0x00007fc95b530000)


#5

Hi Sumit,

I just pushed an update to the readme with instructions for how to generate packages. One thing it mentions is the option to prefer linking to static libraries instead of the shared ones, which will take care of issues like libarrow.so.0 not being found. Plus, since you’re building on CentOS, you should also be able to copy these generated packages to other machines with recent Linux distros (main thing is must have same or newer version of glibc).


#6

Hi @andrew

I am getting the Error during the building process. Please refer the detail trace for reference.

MapD_Issue.log.zip (23.1 KB)


#7

Could you do a git pull to grab recent commits from the repository? Looks to be the same issue as MapD Building Error : [ 35%] Built target mapd_java_components which was resolved with this commit: https://github.com/mapd/mapd-core/commit/dd901ba9134d1bf97075cc6178482781680d9233


#8

i downloaded latest code today and verified that this section is already available.

Tests/CMakeLists.txt

60 target_link_libraries(ExecuteTest {EXECUTE_TEST_LIBS}) 61 target_link_libraries(RunQueryLoop {EXECUTE_TEST_LIBS} bcrypt)
62 target_link_libraries(ImportTest gtest {EXECUTE_TEST_LIBS}) 63 target_link_libraries(AlterColumnTest gtest {EXECUTE_TEST_LIBS})
64 target_link_libraries(PlanTest gtest {EXECUTE_TEST_LIBS}) 65 target_link_libraries(UpdelStorageTest gtest {EXECUTE_TEST_LIBS})
66 target_link_libraries(StorageTest gtest {EXECUTE_TEST_LIBS}) 67 target_link_libraries(StoragePerfTest gtest {EXECUTE_TEST_LIBS})
68 target_link_libraries(TopKTest {EXECUTE_TEST_LIBS}) 69 target_link_libraries(MapDQLCommandTest gtest {EXECUTE_TEST_LIBS} {Boost_LIBRARIES}) 70 target_link_libraries(DBObjectPrivilegesTest gtest {EXECUTE_TEST_LIBS} {Boost_LIBRARIES}) **71 target_link_libraries(GeoTypesTest gtest {EXECUTE_TEST_LIBS})**’’

However still i am getting same error.


#9

Let me share with you one more observation and may be it will help to understand the issue better way.

Issue 1: Not able to run as a service when building code with “cmake -DPREFER_STATIC_LIBS=off”
Observation:
a) Building is successful
b) when running the script “./install_mapd_systemd.sh” as root user , failing with Error Code " /opt/mapd-core-master/build-package/bin/initdb: error while loading shared libraries: libarrow.so.0: cannot open shared object file: No such file or directory"

MAPD_STORAGE: MapD data and configuration storage directory
[/var/lib/mapd]:
MAPD_USER: user MapD will be run as
[root]:
MAPD_GROUP: group MapD will be run as
[root]:
MAPD_PATH: /opt/mapd-core-master/build-package
MAPD_STORAGE: /var/lib/mapd
MAPD_USER: root
MAPD_GROUP: root
/opt/mapd-core-master/build-package/bin/initdb: error while loading shared libraries: libarrow.so.0: cannot open shared object file: No such file or directory

C) Edit the script and deleted the sudo from Line number 40 ({vars["MAPD_PATH"]}/bin/initdb {vars[‘MAPD_DATA’]}) and script executed without any Error.

[root@sumithome01 systemd]# ./install_mapd_systemd.sh
MAPD_PATH: MapD install directory
/opt/mapd-core-master: /opt/mapd-core-master/build-package

MAPD_STORAGE: MapD data and configuration storage directory
[/var/lib/mapd]:
MAPD_USER: user MapD will be run as
[root]:
MAPD_GROUP: group MapD will be run as
[root]:
MAPD_PATH: /opt/mapd-core-master/build-package
MAPD_STORAGE: /var/lib/mapd
MAPD_USER: root
MAPD_GROUP: root
Thrift: Wed Jul 11 07:37:41 2018 TSocket::open() connect() <Host: localhost Port: 39093>Connection refused
Thrift: Wed Jul 11 07:37:41 2018 TSocket::open() connect() <Host: localhost Port: 39093>Connection refused
[root@sumithome01 systemd]#

d) However now when i am executing start, it is starting without any Error But the status command is showing the Error and MapD is not running in system.

[root@sumithome01 systemd]# systemctl start mapd_server
[root@sumithome01 systemd]#
[root@sumithome01 systemd]# systemctl status mapd_server
● mapd_server.service - MapD database server
Loaded: loaded (/usr/lib/systemd/system/mapd_server.service; disabled; vendor preset: disabled)
Active: failed (Result: start-limit) since Wed 2018-07-11 07:38:27 IST; 1min 12s ago
Process: 23361 ExecStart=/opt/mapd-core-master/build-package/bin/mapd_server --config /var/lib/mapd/mapd.conf (code=exited, status=127)
Main PID: 23361 (code=exited, status=127)

Jul 11 07:38:27 sumithome01 systemd[1]: Unit mapd_server.service entered failed state.
Jul 11 07:38:27 sumithome01 systemd[1]: mapd_server.service failed.
Jul 11 07:38:27 sumithome01 systemd[1]: mapd_server.service holdoff time over, scheduling restart.
Jul 11 07:38:27 sumithome01 systemd[1]: start request repeated too quickly for mapd_server.service
Jul 11 07:38:27 sumithome01 systemd[1]: Failed to start MapD database server.
Jul 11 07:38:27 sumithome01 systemd[1]: Unit mapd_server.service entered failed state.
Jul 11 07:38:27 sumithome01 systemd[1]: mapd_server.service failed.

e) If i am running only the command " /opt/mapd-core-master/build-package/bin/mapd_server --config /var/lib/mapd/mapd.conf" then the MapD Process is starting and able to access the MapD Database.

[root@sumithome01 systemd]# /opt/mapd-core-master/build-package/bin/mapd_server --config /var/lib/mapd/mapd.conf
E0711 07:40:21.055378 23477 MapDHandler.cpp:155] This build isn’t CUDA enabled, will run on CPU
Thrift: Wed Jul 11 07:40:21 2018 TSocket::open() connect() <Host: localhost Port: 9093>Connection refused
Thrift: Wed Jul 11 07:40:21 2018 TSocket::open() connect() <Host: localhost Port: 9093>Connection refused
E0711 07:40:21.276934 23477 MapDHandler.cpp:192] No GPUs detected, falling back to CPU mode

^C
[root@sumithome01 systemd]#

Issue 2. Not able to build the source code "[root@sumithome01 build-package]# cmake -DPREFER_STATIC_LIBS=on -DCMAKE_BUILD_TYPE=release -DENABLE_CUDA=off -DMAPD_IMMERSE_DOWNLOAD=off -DENABLE_AWS_S3=off "

Observation:
1. compilation is not successful and getting Error
[ 93%] Built target mapd_java_components
make: *** [all] Error 2
[root@sumithome01 build-package]#
2. Shared the complete trace in earlier thread and this issue continued with just downloaded the code from git hub.
3. Checked the “Tests/CMakeLists.txt” file also and code is as mentioned in above before reply thread.

Note
Additional Information
1. Using the Centos : CentOS-7-x86_64-Everything-1804
2. Iptables are OFF
3. SELINUX is in disabled State
4. All MapD installations process are executed as root user.
5. MapD Dependency are " /usr/local/mapd-deps/20180703/"

Regards,
Sumit


#10

In the build dir where you ran cmake with -DPREFER_STATIC_LIBS=on, could you run make VERBOSE=1 (no -j) and provide the output? That will tell us exactly which portion of the build failed and the linker command for it. I have not been able to reproduce the curl linking issue on master.

Also it seems like you’re compiling from the .zip downloaded from GitHub. Could you instead clone it with git clone https://github.com/mapd/mapd-core? It shouldn’t change anything, but just eliminates a variable.


If you’re still having problems building with static linking enabled, the missing shared libraries and status=127 issues can be resolved by setting the environment variable LD_LIBRARY_PATH. In your case you’ll want to set it to

LD_LIBRARY_PATH=/usr/local/mapd-deps/20180703/lib64:/usr/local/mapd-deps/20180703/lib

For the systemd service, you can do this by by adding the following to the [service] section of /usr/lib/systemd/system/mapd_server.service:

Environment="LD_LIBRARY_PATH=/usr/local/mapd-deps/20180703/lib64:/usr/local/mapd-deps/20180703/lib"

#11

@andrew

  1. Tried to compile with VERBOSE=1 and observed the issue. please find the enclosed error Log.

  2. systemd service will try today and update you on results.Building_MapD_with VERBOSE.txt (328.9 KB)

/sumit


#12

Ok, now I can reproduce: I missed that you were setting -DENABLE_AWS_S3=off. Try setting this to on and try rebuilding.

It looks like this is a bug in our build config, geo import depends on curl but isn’t explicitly linking to it, instead is using the version that gets linked in if you enable S3 support.

I’ve opened an issue for this, should be a quick fix. https://github.com/mapd/mapd-core/issues/228


#13

@andrew

it is working now and able to generate RPM file. Please find the enclosed file for reference. i will continue to do further checks.

MapD_RPM_Creation.txt (35.9 KB)