Tutorials

Quick Start

This tutorial will help you deploy your first Concord computations in our simple virtualized development environment.

At the moment Concord does not support running in a local mode and depends on Apache Mesos in order to run. For ease of use we have created a virtual environment to try out Concord with. In order to complete this tutorial you must have Vagrant installed. Please follow the tutorials on their website

Setup Concord Virtualized Environment

Clone the getting started project and start the Vagrant virtual machine. If you’re prompted, the password for vagrant is vagrant.

$ git clone https://github.com/concord/getting-started.git concord-getting-started
$ cd concord-getting-started
$ ./bootstrap_vagrant.sh # only run this when you want to reconfigure
$ vagrant ssh

If you go to your http://localhost:5050 on your browser, you should be able to see Mesos running Concord (Concord may take a few moments to start up).

Deploying Computations

The vagrant box ships with a sample word example that can be located in the /vagrant folder. An example has been provided for every programming language that Concord currently supports.

Python / Ruby

To deploy scripting languages (such as python and ruby):

vagrant@vagrant-ubuntu-trusty-64:~$ cd /vagrant/py
vagrant@vagrant-ubuntu-trusty-64:/vagrant/py$ concord deploy word_counter.json
INFO:cmd.utils:Connecting to:localhost:2181
INFO:cmd.deploy:Sending computation to: 172.17.0.110:11211
INFO:cmd.deploy:Verify with the mesos host: 172.17.0.110 that the service is running
vagrant@vagrant-ubuntu-trusty-64:/vagrant/py$ concord deploy sentence_splitter.json
...
vagrant@vagrant-ubuntu-trusty-64:/vagrant/py$ concord deploy sentence_generator.json
...

C++

For C++ you will need to compile the project. This project uses CMake to manage its build system. For ease of use we have conviently created a script that will build and prepare this C++ example in one step:

vagrant@vagrant-ubuntu-trusty-64:~$ cd /vagrant/cpp
vagrant@vagrant-ubuntu-trusty-64:/vagrant/cpp$ ./compile.bash
[ 33%] Built target sentence_generator
[100%] [100%] Built target word_counter
Built target sentence_splitter
vagrant@vagrant-ubuntu-trusty-64:/vagrant/cpp$ concord deploy word_counter.json
INFO:cmd.utils:Connecting to:localhost:2181
	INFO:cmd.deploy:Sending computation to: 172.17.0.110:11211
INFO:cmd.deploy:Verify with the mesos host: 172.17.0.110 that the service is running
vagrant@vagrant-ubuntu-trusty-64:/vagrant/cpp$ concord deploy sentence_splitter.json
...
vagrant@vagrant-ubuntu-trusty-64:/vagrant/cpp$ concord deploy sentence_generator.json
...

JVM

If you are using one of the provided JVM languages such as Scala or Java you can run the pre_package script located in the projects respective folder. This script compiles the project (using sbt), creates a fat jar, then finally copies all build artifacts and manifest files into a target directory:

vagrant@vagrant-ubuntu-trusty-64:~$ cd /vagrant/java
vagrant@vagrant-ubuntu-trusty-64:/vagrant/java$ ./pre_package_concord.bash
...
[info] Done packaging.
[success] Total time: 4 s, completed Jan 13, 2016 11:10:42 AM
vagrant@vagrant-ubuntu-trusty-64:~$ cd target/concord
vagrant@vagrant-ubuntu-trusty-64:~$ concord deploy word_counter.json
INFO:cmd.utils:Connecting to:localhost:2181
INFO:cmd.deploy:Sending computation to: 172.17.0.110:11211
INFO:cmd.deploy:Verify with the mesos host: 172.17.0.110 that the service is running

vagrant@vagrant-ubuntu-trusty-64:/vagrant/java$ concord deploy sentence_splitter.json
...
vagrant@vagrant-ubuntu-trusty-64:/vagrant/java$ concord deploy sentence_generator.json
...

Check the Mesos Dashboard

Go to http://localhost:5050 in your browser and you will see the three computations running on Concord:

mesoscomps

You should be able to find the logs by clicking into the Sandbox for each task you’ve started, and navigating to the stderr link. Your word counter program is working correctly if you see output like this:

I1013 10:34:38.125651 5611 ComputationFacade.cc:28] Computation Facade init. I1013 10:34:38.125705 5611 word_counter.cc:15] Init word count sink I1013 10:34:40.099395 5611 word_counter.cc:25] Key values stored: (offer -> 12500)(trail -> 12500)(sorry -> 12500)(giant -> 12500)(hurry -> 12500)(saved -> 12500)(stock -> 12500)(beach -> 12500) I1013 10:34:41.475951 5611 word_counter.cc:25] Key values stored: (offer -> 25000)(trail -> 25000)(sorry -> 25000)(giant -> 25000)(hurry -> 25000)(saved -> 25000)(stock -> 25000)(beach -> 25000) I1013 10:34:42.988759 5611 word_counter.cc:25] Key values stored: (offer -> 37500)(trail -> 37500)(sorry -> 37500)(giant -> 37500)(hurry -> 37500)(saved -> 37500)(stock -> 37500)(beach -> 37500) I1013 10:34:44.361919 5611 word_counter.cc:25] Key values stored: (offer -> 50000)(trail -> 50000)(sorry -> 50000)(giant -> 50000)(hurry -> 50000)(saved -> 50000)(stock -> 50000)(beach -> 50000) I1013 10:34:45.807272 5611 word_counter.cc:25] Key values stored: (offer -> 62500)(trail -> 62500)(sorry -> 62500)(giant -> 62500)(hurry -> 62500)(saved -> 62500)(stock -> 62500)(beach -> 62500) I1013 10:34:46.977836 5611 word_counter.cc:25] Key values stored: (offer -> 75000)(trail -> 75000)(sorry -> 75000)(giant -> 75000)(hurry -> 75000)(saved -> 75000)(stock -> 75000)(beach -> 75000) I1013 10:34:48.291836 5611 word_counter.cc:25] Key values stored: (offer -> 87500)(trail -> 87500)(sorry -> 87500)(giant -> 87500)(hurry -> 87500)(saved -> 87500)(stock -> 87500)(beach -> 87500)


Missing Logs
If you do not see computation output in the stderr logs, navigate to the folder next to the stderr link. Any computation logs should be in a file ending in '.log'.