Building core - dependencies not installed CENTOS 7


#1

After running:

sudo yum groupinstall -y "Development Tools"
sudo yum install -y \
    zlib-devel \
    epel-release \
    libssh \
    openssl-devel \
    ncurses-devel \
    git \
    maven \
    java-1.8.0-openjdk-devel \
    java-1.8.0-openjdk-headless \
    gperftools \
    gperftools-devel \
    gperftools-libs \
    python-devel \
    wget \
    curl \
    environment-modules

and

curl -OJ https://internal-dependencies.mapd.com/mapd-deps/deploy.sh
sudo bash deploy.sh  

I only get an empty folder :

/usr/local/mapd-deps/20171002/

and themapd-deps-centos.sh script fails with :

/common-functions.sh: No such file or directory

Any help ?


#2

Hi,

Sorry to hear your issue. I just ran deploy methods on a clean centos 7.3 machine and it worked fine?

deploy.sh should populate a file called /usr/local/mapd-deps/20170926/

Would it be possible for you to capture the output from deploy.sh so we can try to identify your issue?

Did you run the mapd-deps-centos.sh from with in the scripts directory? You should be able to see the common-functions.sh file in that same directory.

Regards


#3

Hi dwayneberry,
thanks for your support.

Indeed, I didn’t were into the /script folder, so I run “scripts/mapd-deps-centos.sh” from within the script folder.

But after a long successful output, I had an error:

Creating glxew.html
--------------------------------------------------------------------
cp -f src/header.html ../doc/glxew.html
echo -e "<h2>Supported GLX Extensions</h2>\n" >> ../doc/glxew.html
bin/make_html.pl extensions/gl/GLX_* >> ../doc/glxew.html
cat src/footer.html >> ../doc/glxew.html
perl -i -pe 's#<a href="glxew.html">(.*)</a>#\1#' ../doc/glxew.html
perl -e "s/GLEW_MAJOR/1/g;s/GLEW_MINOR/13/g;s/GLEW_MICRO/0/g;" -p src/glew.rc > ../build/glew.rc
perl -e "s/GLEW_MAJOR/1/g;s/GLEW_MINOR/13/g;s/GLEW_MICRO/0/g;" -p src/glewinfo.rc > ../build/glewinfo.rc
perl -e "s/GLEW_MAJOR/1/g;s/GLEW_MINOR/13/g;s/GLEW_MICRO/0/g;" -p src/visualinfo.rc > ../build/visualinfo.rc
make[1]: Leaving directory `/opt/mapd/mapd-core-master/scripts/glew-1.13.0/auto'
+ make DESTDIR=/usr/local/mapd-deps/20171003 GLEW_DEST=
cc -DGLEW_NO_GLU -DGLEW_BUILD -O2 -Wall -W -Iinclude -fPIC -Wcast-qual -ansi -pedantic  -o tmp/linux/default/shared/glew.o -c src/glew.c
In file included from src/glew.c:38:0:
include/GL/glxew.h:98:22: fatal error: X11/Xlib.h: No such file or directory
 #include <X11/Xlib.h>
                      ^
compilation terminated.
make: *** [tmp/linux/default/shared/glew.o] Error 1
[root@gecc11 scripts]# 

How should i proceed?


#4

Hi Camgia,

Sorry for the confusion. If you use the deploy.sh script to download the prebuilt dependencies, then you do not need to run the mapd-deps-centos.sh script.

Could you run the deploy.sh script again and send us the last few lines of output?

curl -OJ https://internal-dependencies.mapd.com/mapd-deps/deploy.sh
sudo bash deploy.sh  

If it finishes successfully, be sure to log out and log back in so that the module command is available, and then run module load mapd-deps in order to set the environment variables required to use the dependencies.


#5

Thanks andrew,
for clarifying me that.

I got to run deploy.sh successfully, but now, - after a reboot - , running :

module load mapd-deps

I get:

ModuleCmd_Load.c(213):ERROR:105: Unable to locate a modulefile for 'mapd-deps'

the modules currently available are:

[root@gecc11 scripts]# module avail   

-------------------------------------------------------- /usr/share/Modules/modulefiles ---------------------------------------------------------
dot         module-git  module-info modules     null        use.own

--------------------------------------------------------------- /etc/modulefiles ----------------------------------------------------------------
mpi/openmpi-x86_64
[root@gecc11 scripts]#

In addition it seems that the mapd-deps folder is fully populated:

[root@gecc11 20171003]# ls -la /usr/local/mapd-deps/20171003/
total 36
drwxr-xr-x. 10 root root    98 Oct  3 10:13 .
drwxr-xr-x.  3 root root    21 Oct  3 09:42 ..
drwxr-xr-x.  2 root root  4096 Oct  3 10:29 bin
drwxr-xr-x.  3 root root    22 Oct  3 10:13 doc
drwxr-xr-x. 22 root root  4096 Oct  3 10:29 include
drwxr-xr-x.  7 root root 16384 Oct  3 10:29 lib
drwxr-xr-x.  3 root root  4096 Oct  3 10:13 lib64
drwxr-xr-x.  3 root root    54 Oct  3 10:24 libexec
drwxr-xr-x.  3 root root    17 Oct  3 09:54 man
drwxr-xr-x. 20 root root  4096 Oct  3 10:24 share

#6

What are the last few lines of output when running deploy.sh? After extracting the deps, the script is supposed to place a symlink in /etc/modulefiles/mapd-deps/, which will make it available via module. However, if any of the steps fail, the script will exit and should show the failed command.

In addition to the last few lines of deploy.sh's output, could you also run the following:

ls -l /usr/local/mapd-deps/
ls -l /usr/local/mapd-deps/20170926/
ls -l /etc/modulefiles/
ls -l /etc/modulefiles/mapd-deps/

#7

Last few lines of

sudo bash deploy.sh

++ for i in '/etc/profile.d/*.sh'
++ '[' -r /etc/profile.d/cuda_path.sh ']'
++ '[' ehxB '!=' ehxB ']'
++ . /etc/profile.d/cuda_path.sh
+++ export CUDA_HOME=/usr/local/cuda-8.0
+++ CUDA_HOME=/usr/local/cuda-8.0
+++ export PATH=/usr/local/cuda-8.0/bin:/usr/local/sbin:/sbin:/bin:/usr/sbin:/usr/bin
+++ PATH=/usr/local/cuda-8.0/bin:/usr/local/sbin:/sbin:/bin:/usr/sbin:/usr/bin
+++ export PATH=/usr/local/cuda-8.0/lib64:/usr/local/cuda-8.0/bin:/usr/local/sbin:/sbin:/bin:/usr/sbin:/usr/bin
+++ PATH=/usr/local/cuda-8.0/lib64:/usr/local/cuda-8.0/bin:/usr/local/sbin:/sbin:/bin:/usr/sbin:/usr/bin
+++ export LD_LIBRARY_PATH=/usr/local/cudnn/cuda/lib64:
+++ LD_LIBRARY_PATH=/usr/local/cudnn/cuda/lib64:
+++ export PYTHONPATH=/your/tensorflow/path:.
+++ PYTHONPATH=/your/tensorflow/path:.
++ for i in '/etc/profile.d/*.sh'
++ '[' -r /etc/profile.d/cuda.sh ']'
++ '[' ehxB '!=' ehxB ']'
++ . /etc/profile.d/cuda.sh
+++ '[' -x /usr/bin/nvcc ']'
+++ '[' -x /usr/libexec/cuda/open64/bin/nvopencc ']'
+++ '[' -d /usr/include/cuda ']'
[root@gecc11 scripts]# 

and

[root@gecc11 modulefiles]# ls -l /usr/local/mapd-deps/
total 0
drwxr-xr-x. 10 root root 98 Oct  3 10:13 20171003
[root@gecc11 modulefiles]# ls -l /usr/local/mapd-deps/20171003/
total 36
drwxr-xr-x.  2 root root  4096 Oct  3 10:29 bin
drwxr-xr-x.  3 root root    22 Oct  3 10:13 doc
drwxr-xr-x. 22 root root  4096 Oct  3 10:29 include
drwxr-xr-x.  7 root root 16384 Oct  3 10:29 lib
drwxr-xr-x.  3 root root  4096 Oct  3 10:13 lib64
drwxr-xr-x.  3 root root    54 Oct  3 10:24 libexec
drwxr-xr-x.  3 root root    17 Oct  3 09:54 man
drwxr-xr-x. 20 root root  4096 Oct  3 10:24 share
[root@gecc11 modulefiles]# ls -l /etc/modulefiles/
total 0
drwxr-xr-x. 2 root root 27 Mar 10  2017 mpi
[root@gecc11 modulefiles]# ls -l /etc/modulefiles/mapd-deps/
ls: cannot access /etc/modulefiles/mapd-deps/: No such file or directory
[root@gecc11 modulefiles]#

#8

Hi,

Please do a which on deploy.sh it looks like you are not running the deploy.sh that was curled.

Please go to a clean directory

curl -OJ https://internal-dependencies.mapd.com/mapd-deps/deploy.sh

then

sudo bash `pwd`/deploy.sh

regards


#9

Dwayneberry,

after recurling deploy.sh into a clean directory it worked and the dependency are fully working.

Great support, thank you!