Tutorials

Cluster Install

This document contains detailed instructions on how to install Concord. For ease of use we’ve created our own installer that will simplify the cluster install process. The installer creates a single logical Apache Mesos cluster out of a given set of machines. Only the necessary dependencies for any single machine will be installed, along with Concord itself.

Preparation

Your going to need to install ansible and a couple of other dependencies in order to run the install scripts. We’ve created a simple script that you can source so you won’t pollute your local environment with any installer dependencies. Unpack the tarball and cd into the concord-install directory. Then source the bootstraping script like so:

$ source source_ansible_bash

Creating your hosts file

Ansible needs some information about what computers to run particular commands against. Machines can be designated as one of four categories: zookeepers, mesos_masters, concord_schedulers, and mesos_agents. Create a file called hosts which defines your cluster configuration. An example of a hosts file would look like this:

[zookeepers]
ccord-1.us-central1-b.your-domain.com             internal_ip=10.240.0.20  zk_id=1
ccord-2.us-central1-b.your-domain.com             internal_ip=10.240.0.24  zk_id=2
ccord-3.us-central1-b.your-domain.com             internal_ip=10.240.0.18  zk_id=3

[mesos_masters]
ccord-1.us-central1-b.your-domain.com             internal_ip=10.240.0.20 public_ip=x.x.x.x mesos_quorum=2
ccord-2.us-central1-b.your-domain.com             internal_ip=10.240.0.24 public_ip=x.x.x.x mesos_quorum=2
ccord-3.us-central1-b.your-domain.com             internal_ip=10.240.0.18 public_ip=x.x.x.x mesos_quorum=2

[concord_schedulers]
ccord-1.us-central1-b.your-domain.com             internal_ip=10.240.0.20 public_ip=x.x.x.x
ccord-2.us-central1-b.your-domain.com             internal_ip=10.240.0.24 public_ip=x.x.x.x
ccord-3.us-central1-b.your-domain.com             internal_ip=10.240.0.18 public_ip=x.x.x.x

[mesos_slaves]
ccord-4.us-central1-b.your-domain.com             internal_ip=10.240.0.2
ccord-5.us-central1-b.your-domain.com             internal_ip=10.240.0.15
ccord-6.us-central1-b.your-domain.com             internal_ip=10.240.0.9
ccord-7.us-central1-b.your-domain.com             internal_ip=10.240.0.3
ccord-8.us-central1-b.your-domain.com             internal_ip=10.240.0.21
ccord-9.us-central1-b.your-domain.com             internal_ip=10.240.0.16


Watch out!
If you don't name your ansible hosts file "hosts" , then you will have to pass the "-i" flag to the ansible-playbook command below, specifying the name of your hosts file. Also make sure this file exists at the root of the installer directory.

Run the installer

All thats left to do is to run the ansible installer. Each machine that you’ve defined in your hosts file will only have installed the sole dependencies it needs to perform its role.

$ ansible-playbook playbooks/provision_cluster.yml

To verify that Mesos and Concord have successfully installed and that all of the necessary services have started, navigate to the ip address of a mesos-master designated machine at port 5050 from a web browser. Once there you should see the Apache Mesos cluster homepage. Click on the frameworks tab and ensure that the Concord framework is registered. You should see something like this:

Frameworks tab

Finally, computations run in clusters provisioned this way must notify the framework of a runtime environment. You’ll need to add this line to your computation manifest file before deploying:

{
  ...
  "docker_container": "concord/runtime_executor:<current_version>"
  ...
}

A list of available runtimes can be found on our dockerhub. Use the runtime version that matches the version of the scheduler your using. To find the scheduler version grep for the string concord_scheduler_version in the install scripts.

To test running computations against the cluster, check out the walkthrough. For those that are looking for ready made examples, check out our getting-started repo on github.

Client Libraries
With the exception of C++ developers, you'll need to install the client libraries of your choice onto the machines designated as mesos agents. Without this your computations will crash when attempting to load the concord client library.