Custom Extension Function - initdb error


#1

Hi,

I’m trying to create a custom extension function, by adding a new function into the ExtensionFunction.hpp file from the mapd-core source (master).

It compiles fine. However, when I try to do initdb it fails with the following error:

F0620 08:17:37.094236  6104 Calcite.cpp:75] Check failed: calciteDirectObject_ 
*** Check failure stack trace: ***
@     0x7f92ad4035cd  google::LogMessage::Fail()
@     0x7f92ad405433  google::LogMessage::SendToLog()
@     0x7f92ad40315b  google::LogMessage::Flush()
@     0x7f92ad405e1e  google::LogMessageFatal::~LogMessageFatal()
@           0x61642a  Calcite::runJNI()
@           0x61728a  Calcite::Calcite()
@           0x574216 
_ZN9__gnu_cxx13new_allocatorI7CalciteE9constructIS1_JiRNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEiEEEvPT_DpOT0_
@           0x573dfc  _ZNSt16allocator_traitsISaI7CalciteEE9constructIS0_JiRNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEiEEEvRS1_PT_DpOT0_
@           0x5738f3  _ZNSt23_Sp_counted_ptr_inplaceI7CalciteSaIS0_ELN9__gnu_cxx12_Lock_policyE2EEC1IJiRNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEiEEES1_DpOT_
@           0x572ecd  std::__shared_count<>::__shared_count<>()
@           0x5722e9  _ZNSt12__shared_ptrI7CalciteLN9__gnu_cxx12_Lock_policyE2EEC1ISaIS0_EJiRNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEiEEESt19_Sp_make_shared_tagRKT_DpOT0_
@           0x57194c  _ZNSt10shared_ptrI7CalciteEC1ISaIS0_EJiRNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEiEEESt19_Sp_make_shared_tagRKT_DpOT0_
@           0x570d82  _ZSt15allocate_sharedI7CalciteSaIS0_EJiRNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEiEESt10shared_ptrIT_ERKT0_DpOT1_
@           0x56fde4  _ZSt11make_sharedI7CalciteJiRNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEiEESt10shared_ptrIT_EDpOT0_
@           0x56db9f  main
@     0x7f92acccc830  __libc_start_main
@           0x56d149  _start
@              (nil)  (unknown)
Aborted (core dumped)

I’ve been unable to get any further with deciphering the error message. I’d appreciate any suggestions.

Thanks, Owen


SEGFAULT due to new glibc
#2

Hi,

This normally indicates you had an issue compiling the java portion of MapD.

Would it be possible for you to share the code changes you added to ExtensionFunction.hpp

We would recommend you go to the java directory and do a mvn clean and then a mvn install.

Please then share the output of these.

regards


#3

Thanks Dwayne, I’ll take a look and let you know how I get on.


#4

Hi Dwayne,

So I ran mvn clean and mvn install in the main java directory but this fails with or without my changes (although the main make-based build succeeds fine). The error message starts as follows:

[INFO] 96 errors 
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] MapD Java .......................................... SUCCESS [  0.453 s]
[INFO] thrift ............................................. FAILURE [  4.764 s]
[INFO] mapdjdbc ........................................... SKIPPED
[INFO] calcite ............................................ SKIPPED
[INFO] utility ............................................ SKIPPED
[INFO] mapd ............................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 5.386 s
[INFO] Finished at: 2017-06-21T07:01:18+00:00
[INFO] Final Memory: 16M/304M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.2:compile (default-compile) on project thrift: Compilation failure: Compilation failure:
[ERROR] /home/ubuntu/mapd-core/java/thrift/src/gen/com/mapd/thrift/server/TRenderResult.java:[467,63] method hashCode in class java.lang.Object cannot be applied to given types;
[ERROR] required: no arguments
[ERROR] found: long
[ERROR] reason: actual and formal argument lists differ in length
[ERROR] /home/ubuntu/mapd-core/java/thrift/src/gen/com/mapd/thrift/server/TRenderResult.java:[469,63] method hashCode in class java.lang.Object cannot be applied to given types;
[ERROR] required: no arguments
[ERROR] found: long
[ERROR] reason: actual and formal argument lists differ in length
[ERROR] /home/ubuntu/mapd-core/java/thrift/src/gen/com/mapd/thrift/server/TRenderResult.java:[471,63] method hashCode in class java.lang.Object cannot be applied to given types;
[ERROR] required: no arguments
[ERROR] found: long
[ERROR] reason: actual and formal argument lists differ in length
[ERROR] /home/ubuntu/mapd-core/java/thrift/src/gen/com/mapd/thrift/server/TDatumVal.java:[406,63] method hashCode in class java.lang.Object cannot be applied to given types;
[ERROR] required: no arguments
[ERROR] found: long
[ERROR] reason: actual and formal argument lists differ in length
[ERROR] /home/ubuntu/mapd-core/java/thrift/src/gen/com/mapd/thrift/server/TDatumVal.java:[408,63] method hashCode in class java.lang.Object cannot be applied to given types;
[ERROR] required: no arguments
[ERROR] found: double

Cheers, Owen


#5

Hmm, my node seems to have gotten in a funky state. I just get segfaults on initdb now. Even with a clean git clone and new data directory.

I’ll maybe rebuild the node, and try again later : /


#6

Hi Owen

Did you do an OS update on the ‘node’? We have noticed that today a push of glibc on many platforms is not playing nice with java and has broken JNI interfaces causing segfaults.

We are working on a solution at present and I would suggets you wait for a push to the repo before you try to proceed at present. Basically you will be running into segfaults all over the place.

regards


#7

Appreciate the heads-up Dwayne! I’m guessing that’s probably it.


#8

I just experienced similar symptoms after performing a yum update on the kernel. (Unstable behavior, initdb crash, and lots of java error logs). I downgraded to the previous kernel version (as a way to back out glibc) and the problems ceased. So try downgrading kernel versions and retest.


#9

Hi @owenrh were you able to give this a try again with the 3.1.1 version of MapD? That segfault issue should be fixed now.


#10

Great thanks. Yeah, I can confirm the segfault is now gone.

I’m just back to the original error now, which I hope to dive into later in the week.


#11

Is it possible for you to share the function you added to ExtensionFunctions.hpp?


#12

So I’m just coming back onto this work, having previously got dragged off onto another task.

I’m just trying to perform the mvn clean install on the unmodified codebase (v3.2.2).

I get the same error messages ‘actual and formal argument lists differ in length’, etc.

Is the mvn clean install still required when adding new Extension Functions?

Thanks, Owen


#13

Ideally not if I can avoid if for the time being : /


#14

So I cleaned the code, git cloned, made my modifications and compiled and have successfully run up the MapD server.

This is without going into the /java directory and performing the mvn clean install.

I think my issue was a Thrift related. My extension function had used a uint64_t argument, which it doesn’t look like Thrift supports. If I change this to a int64_t type it compiles and runs fine.