Apache SDAP (Incubating) - Dataset Relevancy Mining & Discovery

Mining and Utilizing Dataset Relevancy from Oceanographic Datasets to Improve Data Discovery and Access

MUDROD is a semantic discovery and search project part of Apache SDAP.

Software requirements:

  • Java 8
  • Git
  • Apache Maven 3.X
  • Elasticsearch v5.X
  • Kibana v4
  • Apache Spark v2.0.0
  • Apache Tomcat 7.X


Docker Container

We strongly advise all users to save time and effort by consulting the Dockerfile documentation for guidance on how to quickly use Docker to deploy Mudrod.

From source

  1. Ensure you have Elasticsearch running locally and that the configuration in config.xml reflects your ES cluster.
  2. Update the svmSgdModel configuration option in config.xml. There is a line in config.xml that looks likeIt needs to be changed to an absolute filepath on your system. For example:
  3. (Optional) Depending on your computer's configuration you might run into an error when starting the application: “Service 'sparkDriver' could not bind on port 0”. The easiest fix is to export the environment variable SPARK_LOCAL_IP= and then start the service.
$ git clone https://github.com/mudrod/mudrod.git
$ cd mudrod
$ mvn clean install
$ cd service
$ mvn tomcat7:run

You will now be able to access the Mudrod Web Application at http://localhost:8080/mudrod-service. N.B. The service should not be run this way in production.


In another window...

$ cd mudrod
$ ./core/target/appassembler/bin/mudrod-engine -h
usage: MudrodEngine: 'logDir' argument is mandatory. User must also
       provide an ingest method. [-a] [-esHost <host_name>] [-esPort
       <port_num>] [-esTCPPort <port_num>] [-f] [-h] [-l] -logDir
       </path/to/log/directory> [-p] [-s] [-v]
 -a,--addSimFromMetadataAndOnto                          begin adding
                                                         metadata and
                                                         ontology results
 -esHost,--elasticSearchHost <host_name>                 elasticsearch
                                                         cluster unicast
 -esPort,--elasticSearchHTTPPort <port_num>              elasticsearch
                                                         HTTP/REST port
 -esTCPPort,--elasticSearchTransportTCPPort <port_num>   elasticsearch
                                                         transport TCP
 -f,--fullIngest                                         begin full ingest
                                                         Mudrod workflow
 -h,--help                                               show this help
 -l,--logIngest                                          begin log ingest
                                                         without any
                                                         processing only
 -logDir,--logDirectory </path/to/log/directory>         the log directory
                                                         to be processed
                                                         by Mudrod
 -p,--processingWithPreResults                           begin processing
 -s,--sessionReconstruction                              begin session
 -v,--vocabSimFromLog                                    begin similarity
                                                         calulation from
                                                         web log Mudrod

Deploying to Apache Tomcat (or any other Servlet container)

Once you have built the codebase as above, merely copy the genrated .war artifact to the servlet deployment directory. In Tomcat (for example), this would look as follows

$ cp mudrod/service/target/mudrod-service-${version}-SNAPSHOT.war $CATALINA_HOME/webapps/

Once Tomcat hot deploys the .war artifact, you will be able to browse to the running application similar to what is shown above http://localhost:8080/mudrod-service

Mudrod Wiki


Java API Documentation

$ mvn javadoc:aggregate
$ open target/site/apidocs/index.html

REST API Documentation

$ mvn clean install
$ open service/target/miredot/index.html

The REST API documentation can also be seen at https://sdap.apache.org/miredot.


This source code is licensed under the Apache License v2.0, a copy of which is shipped with this project.

<para name="svmSgdModel">file:///Users/user/githubprojects/mudrod/core/src/main/resources/javaSVMWithSGDModel</para>
<para name="svmSgdModel">file://YOUNEEDTOCHANGETHIS</para>

Download Details:

Author: apache

Official Github:  https://github.com/apache/incubator-sdap-mudrod 

License: Apache-2.0 license

#apache  #data  #data-analysis #data-science 

Apache SDAP (Incubating) - Dataset Relevancy Mining & Discovery
1.10 GEEK