DIY DSP, Part 2

Last updated: March 3, 2017

Overview

After determining your basic requirements, you are now ready to build your DSP. We suggest a base system at first, with a single SSP, then after you get that operational you can then begin to scale.

Deploying Base System

We recommend building a base system first, using 4 different servers. These servers needed will be the following:

  1. 1 8-core, 16Gbyte RAM, 128Gbyte disk system for the bidder plus zerospike
  2. 1 8 core, 16Gbyte RAM, 128Gbyte disk system for Crosstalk+MySQL+Campaign Manager
  3. 1 8 core, 16Gbyte RAM, 128Gbyte system for Zookeeper+Kafka
  4. 1 12 core, 32Gbyte RAM, 2Tbyte system system for Elastic Search

When completed, you can horizontally scale more bidders, and you can also add more servers and provide fault tolerance.

Each of these servers must have Docker running.

Naming Conventions

It is important to create meaningful hostnames so that you don't get confused at what you are looking at later. It is not useful to name one a 'bidder' and another 'crosstalk' because it is possible to run multiple bidders on multiple servers.

In addition, when you begin to scale horizontally you will be replicating hardware

In general, name your system by its region + its size. For example region EU might have these servers:

  • EU-8c16g-1, EU-8c16g-2, EU-8c16g3 are the three EU 8-core systems
  • EU-12c32g-1 is the EU 12 core system

Configure Stack

There are five Docker stack files. They handle the following services:

  1. RTB - Bidder+Crosstalk+Zerospike
  2. PROXY - HaProxy
  3. KAFKA - Kafka+Zookeeper
  4. ELK - Logstash+ElasticSearch
  5. WEB - MySQL+Campaign Manager
In the sections above we will place RTB and PROXY on EU-8c16g-1; KAFKA will be placed on EU-8c16g-2; WEB will be placed on EU-8c16g3 and ELK will be placed on EU-12c32g-1.

Generate Stack Files

To make it easy, we have a Docker container that will generate the 5 stack files for you. Here's an example run:

	$docker run
	REGION-NAME: EU
	RTB: EU-8c16g-1
	PROXY: EU-8c16g-1 
	KAFKA: EU-8c16g-2
	WEB: EU-8c16g-3
	ELK: EU-12c32g-1
	.... generating ....
	creating: docker-compose-rtb.yml
	creating: docker-compose-haproxy.yml
	creating: docker-compose-kafka.yml
	creating: docker-compose-web.yml
	creating: docker-compose-elk.yml
	Done!

Stack Deploy

Now that the stack files have been generated for you, now it's time to deploy.

Create Tokens

Now we create the Master and worker docker nodes. Follow these steps:

  1. Make Master Node.

    Copy the 5 compose files to EU-8c16g-1. Then type this command:

    	$docker swarm init 
    	$docker network create --driver=overlay --attachable kafka_net
    	$docker network create --driver=overlay --subnet 10.10.9.0/24 --attachable frontend
    	
    

  2. On each of the other 3 nodes start Docker as a worker node

Webr

The first service to start is the Web. This will bring up MySQL, Ruby on Rails, and Crosstalk. At this stage the campaign Manager will be operational. On EU-8c16g-1 execute the following command:

	$docker stack deploy -c docker-compose-web.yml

Kafka

The next thing to bring up is Kafka. This will start Zookeeper addition Kafka. All the other services depend on these. On EU-8c16g-1 execute the following command:

	$docker stack deploy -c docker-compose-kafka.yml

ELK

The next thing to start is ELK. This will bring up Logstash, Kibana and ElasticSearch On EU-8c16g-1 execute the following command:

	$docker stack deploy -c docker-compose-elk.yml

RTB

The next set of services to bring up is the bidder(s). This starts the RTB4FREE service and when it comes up, it will be ready for business, once campaigns are loaded. On EU-8c16g-1 execute the following command:

	$docker stack deploy -c docker-compose-rtb.yml

HA Proxy

Next service to bring up is HA Proxy. Bringing up the Proxy will cause bid requests to be routed to your bidders. On EU-8c16g-1 execute the following command:

	$docker stack deploy -c docker-compose-haproxy.yml

Basic Tests

At this point the system is operational. However some basic tests should be done to ensure the system is functioning normally.

Bids

Test bids

Wins

Test wins

Pixels

Test Pixels

Events

Test Events

Logs

Check Logs