Documentation for MapD Core for Developers


#1

Hi,

I am a grad student working on improving MapD Core by adding some interesting spatial functionality. In order to do that, I am first understanding the existing code/code structure of MapD. I have following questions related to the code of MapD Core:

  1. Is this the right place for development related questions? Or there’s a mapd-core-dev forum?

  2. Are there any documents that detail the architecture of MapD Core or show the way it functions? I have already looked at the MapD Core Guide, but it is more of a user guide. Also, the docs link mentioned on github gives 404 error.

Any help related to “how to better understand MapD Core code?” will be greatly appreciated.

Thank you!
-Harshada


#2

Hi Harshada, and thanks for joining. I’m excited to learn more about your spatial functionality. Are you looking to add this functionality to the open source Core, or are you building these features separately? Just curious - if you’re looking to contribute to the open source project, let’s make sure we have a GitHub issue for your functionality.

  1. This is our one community forum, for now. We’re talking about adding a more traditional dev mailing list or forum, but this is where we’re starting.

  2. Thanks for the link 404, we’ll fix that. Let me ping some folks internally to make sure I get you the right URL.


#3

Hi Aaron,

Thanks for the reply.

I am planning to add some essential spatial functionality to MapD such as spatial kNN, spatial join, etc. I will be able to better plan these functionalities once I know the existing code. Also, I would be very happy to release my changes through the open source project :slight_smile: I will contact the team when I have something to release.

Looking forward to the correct docs url.

Thanks,
Harshada


#4

Cool. Even better though if you bring the team in as you’re developing it, through a GitHub issue, sharing your design and interim steps in your development. We do not want to get to the end of your work and realize there is some reason why we can’t accept your pull request. If you open the issue, I’ll make sure we get the right committer in the project to shepherd your work through.

We really appreciate your participation in the project! Thank you!!


#5

That would be awesome! I can’t wait to get started! :slight_smile:

Still waiting for the docs link.

Thanks,
Harshada


#6

Yep, still tracking down the doc link. Sorry for the delay!


#7

So, the repo that that link was supposed to point to was the documentation you’ve already seen (the Core Guide). The link was just supposed to show you where the docs come from, not provide more detail. We’re fixing the link (thanks again for bringing it to our attention), but I realize that doesn’t help you with what you’re looking for.

We are working on a deeper set of core docs, but I don’t have a firm date for when they will be ready. It is not imminent though.

In the meantime, we have published a couple blog posts that go into some more detail:

And we’re talking about writing more, including one that goes into detail on the executor. I realize you’re just getting started, but if you have questions or specific areas we could help further explain, I’d like to hear them. It will help us ensure the docs we do produce cover the areas you’re most interested in.

Thanks again @Harshada!


#8

Thank you for the links Aaron!

I have created a new issue for this feature here. Is there anything else I need to do to get started the collaboration with the mapd-core dev team?

Thanks,
Harshada


#9

Nope, I see Dimitri already responded to your issue, so you’re good. Thank you!


#10

Harshada,
You also should read through the “Contributing” section in https://github.com/mapd/mapd-core. You’re welcome of course to clone and do any sort of research you like, but if you wish to have your work merged into master than there is a contribution agreement to review and sign.
Initial code into master is not far off, could be 3 weeks or a bit more depending on the work goes. And this would be a basic start towards SFS, there also would be plenty of room for you to help. And it would be a bit easier after there are the basic geometry types and first few operators as a model to develop and experiment with others.

Bill


#11

Thanks for the clarification Bill. It seems the basic spatial geometry types and operators are not part of mapd-core, yet. They are likely to be there in about 3 weeks.

Thanks,
Harshada


#12

@Harshada, and interested geogeeks.

As you’ve hopefully seen already, the core Geo enhancements landed at 4.0 release. These include geo types with web mercator and geographic projections, and ST_DISTANCE and ST_CONTAINS support (see MapD Geo documentation

The geospatial world is a big one, so if you’re still up for it there’s plenty of room to contribute. ST_INTERSECTS (testing for intersections) could be an interesting place to start. Or ST_AREA or ST_INTERSECTION (generating new geometries) if you are more ambitious!

Also of note, Evan Miller has a nice project implementing more geo goodies on GPU, including a broader set of projection ST_TRANSFORMs: