Mapd core build faile on Centos 7


#1

encountered errors when building mapd-deps

1.newly installed centos 7
2.curl -OJ https://internal-dependencies.mapd.com/mapd-deps/deploy.sh
3.bash deploy.sh
got errors:
20171218/lib/liblldbCore.a
xz: (stdin): Compressed data is corrupt
tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now

thanks for any help


#2

It sounds like the dependencies didn’t download completely. Could you try removing /usr/local/mapd-deps and re-running deploy.sh?

sudo rm -rf /usr/local/mapd-deps
bash deploy.sh

#3

thank you, I will remove it and try again


#4

After clean and re-download deps package, mapd-deps module build success

When I run make following instructions, seems header file “completion_hints_types.h” missed or not properly generated

-- Build files have been written to: /home/develop/mapd/mapd-core/build
[  0%] Generating gen-cpp/MapD.cpp, gen-cpp/MapD.h, gen-cpp/mapd_constants.cpp, gen-cpp/mapd_types.cpp, gen-cpp/completion_hints_constants.cpp, gen-cpp/completion_hints_types.cpp
Scanning dependencies of target mapd_thrift
[  0%] Generating gen-cpp/MapD.cpp, gen-cpp/MapD.h, gen-cpp/mapd_constants.cpp, gen-cpp/mapd_types.cpp, gen-cpp/completion_hints_constants.cpp, gen-cpp/completion_hints_types.cpp
[  1%] Building CXX object CMakeFiles/mapd_thrift.dir/gen-cpp/MapD.cpp.o
In file included from /home/develop/mapd/mapd-core/build/gen-cpp/MapD.h:12:0,
                 from /home/develop/mapd/mapd-core/build/gen-cpp/MapD.cpp:7:
/home/develop/mapd/mapd-core/build/gen-cpp/mapd_types.h:19:36: fatal error: completion_hints_types.h: No such file or directory
 #include "completion_hints_types.h"
                                    ^
compilation terminated.
make[2]: *** [CMakeFiles/mapd_thrift.dir/gen-cpp/MapD.cpp.o] Error 1
make[1]: *** [CMakeFiles/mapd_thrift.dir/all] Error 2

#5

Sorry about that: one of the commands for generating the Thrift bindings wasn’t set to generate everything, though it only was an issue if compiling single threaded.

I’ve pushed a fix for it, which you should be able to pick up by running git pull.


#6

Hi andrew, thanks for highly response

After pull from latest git code, previous error fixed, but some errors in test code:

…/Utils/libUtils.a(Regexp.cpp.o): In function bool boost::regex_match<char const*, std::allocator<boost::sub_match<char const*> >, char, boost::regex_traits<char, boost::cpp_regex_traits<char> > >(char const*, char const*, boost::match_results<char const*, std::allocator<boost::sub_match<char const*> > >&, boost::basic_regex<char, boost::regex_traits<char, boost::cpp_regex_traits<char> > > const&, boost::regex_constants::_match_flags)': /usr/local/mapd-deps/20171218/include/boost/regex/v4/regex_match.hpp:50: undefined reference toboost::re_detail_106200::perl_matcher<char const*, std::allocator<boost::sub_match<char const*> >, boost::regex_traits<char, boost::cpp_regex_traits > >::match()’
…/Utils/libUtils.a(Regexp.cpp.o): In function boost::re_detail_106200::perl_matcher<char const*, std::allocator<boost::sub_match<char const*> >, boost::regex_traits<char, boost::cpp_regex_traits<char> > >::perl_matcher(char const*, char const*, boost::match_results<char const*, std::allocator<boost::sub_match<char const*> > >&, boost::basic_regex<char, boost::regex_traits<char, boost::cpp_regex_traits<char> > > const&, boost::regex_constants::_match_flags, char const*)': /usr/local/mapd-deps/20171218/include/boost/regex/v4/perl_matcher.hpp:382: undefined reference toboost::re_detail_106200::perl_matcher<char const*, std::allocator<boost::sub_match<char const*> >, boost::regex_traits<char, boost::cpp_regex_traits > >::construct_init(boost::basic_regex<char, boost::regex_traits<char, boost::cpp_regex_traits > > const&, boost::regex_constants::_match_flags)'
collect2: error: ld returned 1 exit status
make[2]: *** [Tests/UtilTest] Error 1
make[1]: *** [Tests/CMakeFiles/UtilTest.dir/all] Error 2

I disabled test to avoid error.

here’s new error when compiling with one thread:

[ 0%] Generating gen-cpp/MapD.cpp, gen-cpp/MapD.h, gen-cpp/mapd_constants.cpp, gen-cpp/mapd_types.cpp, gen-cpp/completion_hints_constants.cpp, gen-cpp/completion_hints_types.cpp
[ERROR:/home/develop/mapd/mapd-core/mapd.thrift:50] (last token was ‘TDatum’)
Type “TDatum” has not been defined.
make[2]: *** [gen-cpp/MapD.cpp] Error 1
make[1]: *** [CMakeFiles/mapd_thrift.dir/all] Error 2
make: *** [all] Error 2

When compiling with multi-threads, another error occurs around 91% progress

[ERROR] Failed to execute goal on project calcite: Could not resolve dependencies for project com.mapd:calcite:jar:1.0-SNAPSHOT: The following artifacts could not be resolved: org.apache.thrift:libthrift:jar:0.9.3, org.apache.calcite:calcite-core:jar:1.14.0, org.apache.calcite.avatica:avatica-core:jar:1.10.0, com.google.protobuf:protobuf-java:jar:3.1.0, org.apache.httpcomponents:httpclient:jar:4.5.2, commons-codec:commons-codec:jar:1.9, org.apache.calcite:calcite-linq4j:jar:1.14.0, commons-dbcp:commons-dbcp:jar:1.4, commons-pool:commons-pool:jar:1.5.4, org.apache.commons:commons-lang3:jar:3.2, com.esri.geometry:esri-geometry-api:jar:2.0.0, com.fasterxml.jackson.core:jackson-core:jar:2.6.3, com.fasterxml.jackson.core:jackson-annotations:jar:2.6.3, com.fasterxml.jackson.core:jackson-databind:jar:2.6.3, com.google.guava:guava:jar:19.0, net.hydromatic:aggdesigner-algorithm:jar:6.0, commons-lang:commons-lang:jar:2.4, commons-logging:commons-logging:jar:1.1.3, org.codehaus.janino:janino:jar:2.7.6, org.codehaus.janino:commons-compiler:jar:2.7.6: Could not transfer artifact org.apache.thrift:libthrift:jar:0.9.3 from/to central (https://repo.maven.apache.org/maven2): GET request of: org/apache/thrift/libthrift/0.9.3/libthrift-0.9.3.jar from central failed: Tag mismatch! -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :calcite
make[2]: *** [bin/mapd-1.0-SNAPSHOT-jar-with-dependencies.jar] Error 1
make[1]: *** [CMakeFiles/mapd_java_components.dir/all] Error 2


#7

Hi
still stuck here

in main branch, I got this:

Import/libCsvImport.a(Importer.cpp.o): In function Importer_NS::openGDALDataset(std::string const&)': Importer.cpp:(.text+0x2d36): undefined reference toOGRSFDriverRegistrar::Open(char const*, int, OGRSFDriver**)‘
Import/libCsvImport.a(Importer.cpp.o): In function Importer_NS::Importer::readVerticesFromGDALGeometryZ(std::string const&, OGRPolygon*, Importer_NS::PolyData2d&, bool, long, long)': Importer.cpp:(.text+0x14b3e): undefined reference toOGRLineString::getPoint(int, OGRPoint*) const’

with tag v3.4.0:

[ 0%] Generating gen-cpp/MapD.cpp, gen-cpp/MapD.h, gen-cpp/mapd_constants.cpp, gen-cpp/mapd_types.cpp, gen-cpp/completion_hints_constants.cpp, gen-cpp/completion_hints_types.cpp
[ERROR:/home/develop/mapd/mapd-core/mapd.thrift:50] (last token was ‘TDatum’)
Type “TDatum” has not been defined.


#8

Hi,

Something’s not adding up here. After running the deploy.sh script, did you logout, log back in, and then run module load mapd-deps in order to “load” the dependencies? It seems like some dependencies such as thrift and gdal are missing. Quick test would be to try running

thrift --version

If command thrift does not exist, then the dependencies haven’t been loaded.

On CentOS we use Environment Modules to manage the dependencies. The reason for this is that we require certain versions of packages (such as gcc and boost) which may interfere with other packages and projects on your system. Running module load mapd-deps modifies a few environment variables (such as PATH and LD_LIBRARY_PATH) to expose the dependencies to your environment.

After loading the dependencies, be sure to start with a completely clean build dir so that there are no artifacts of the failed builds. Assuming you have not modified any code, in the mapd-core repo run:

git clean -f -f -d -x ; git reset --hard origin/master

#9

Hi
I found some problem
when I log in, “thrift --version” produce

Thrift version 0.9.1

After “module load mapd-deps”, it becomes:

Thrift version 0.10.0

I DID log out after running deploy.sh due to SSH session timeout.
The thrift linking error is still here, what should I do, runing deploy or add module load to /etc/profile or “.bashrc”?


#10

Did you run the git commands above to completely clean your clone of the mapd-core repo? Even better would be to remove it and git clone https://github.com/mapd/mapd-core again to be completely sure that it’s clean.

The java components sometimes leave files outside CMake’s build dir, which can cause problems if the environment isn’t set up quite right. We do have a make target to clean these up, but in this case it’s easiest to start with a guaranteed fresh clone.


#11

yeah

I clear all CMAKE config and git repo, download deploy and pull repo again , but still has the same error

I will initial the instance, and try my luck from very beginning ~


#12

Hi

Seems something wrong with my previous YUM installed package, there are two thrift versions in my environment: 0.9.1/0.10.0.

I initialize the OS, rebuild following instructs, everything compiles fine
Thank for your help