RTB4FREE Details - Layer 1 (Bidder)

Last updated: Feb 21, 2017

Overview

This project provides a Docker container of an RTB engine. Simply start the container and you will be operational with a stand alone bidder in 5 minutes.

A production bidding enterprise requires multiple containers running RTB4FREE, running behind a load balancer, using a campaign manager, and a data management platform. A complete DSP using RTB4FREE is done in "layers", and has a different Docker compose file to set up and maintain it. The three layers are:

  1. Bidder - the RTB part of the system, it connects to the exchanges and makes bids.
  2. Campaign Manager - set up and maintain campaigns.
  3. Data Management Platform - the big data analytics.

This page will show you how to operate the Bidder part of the DSP. By going through the exercises on this page you will get an idea of how a real DSP is set up.

Features at a glance:
  1. Supports the IAB recommendations for GDPR compliance
  2. openRTB 2.5 Compliant
  3. Supports many SSPs
  4. Supports SSL
  5. Supports GZIP Compressed Bids
  6. Simplified Bidder Management
  7. Open Source Campaign Manager
  8. Docker Microservices Architecture
  9. Commerical support and customization is available from this link

Because this is a Docker deployment you must have a working knowledge of Docker. You need Docker and Docker docker-compose installed. For information on Docker : look here.

It is a good idea to go through the steps on this page to edit campaigns to gain an understanding of the flow of control. However, after you have run the execises on this page, use the production campaign manager we provide if you plan to really use this bidder for production.

Here is an overview of the RTB4FREE Microservices Architecture

Quickstart

RTB4FREE is available as a set of Docker containers. You can get a test system up and running in no time. Out of the box, the campaigns are created on disk, but there is also a Docker container for a production campaign manager written with Ruby on Rails and Javascript.

First some prerequisites for running the RTB4FREE bidder as a demo:

  1. Demo mode will log all of its activitites to the directory ./logs. So make sure you create the directory ./logs before you start the container!
  2. The container will be using port 8080 internally and will map to your host's 8080 port.
  3. The RTB4FREE Docker container will use a file-based JSON file of campaigns that is located in the container at /database.json. Later, we will show you how to override it with your own campaigns.

Download docker-compose.yml from here. To view it look here.

Use this command to bring the system up in a console window:

$docker-compose up -d

Use docker ps to make sure all the services have been up for at least 30 seconds before proceeding to the next step:

$docker ps

Web Request

Now let's send a bid request to the system using the simulator built into the bidder. Simply point your web broswer from your local machine to http://localhost:8080/exchange.html. Select the send bid button and watch the bid request/response window.

Connect to the Bidder...

When you connect to the bidder you will get a login screen similar to this:

Select the Simulator...

Choose Send Sample Bids, a new window appears like this:

Send a Sample Bid...

The Display ad is already selected, simply depress orange Send Bid Request button and you should see the results of the transaction like this:

Send the Win Notification...

Notice the X-Time, this is the time it took to process the message. The Win URL is shown next to the Send Win button. Pressing that will show the ad displayed on the left, and the creative that was served:

Clicking the ad will cause the actual ad to appear.

Curl Request

Now let's send a bid request to the system using curl. We will login to the bidder's Docker container and send a curl command using the bash shell.

In order to execute commands within the docker container for the bidder you have to find its id first. In the examples we use the term BIDDER-CONTAINER-ID to denote the id of the bidder's container. To do that, use the docker ps command. Here is an example:

$docker ps
CONTAINER ID        IMAGE                   COMMAND                  CREATED             STATUS                    PORTS                                                                                                           NAMES
b01c78abf834        jacamars/rtb4free:v1    "bash -c './wait-for…"   3 minutes ago       Up 38 seconds             0.0.0.0:7000->7000/tcp, 0.0.0.0:7379->7379/tcp, 0.0.0.0:8080->8080/tcp, 6000-6002/tcp, 0.0.0.0:8155->8155/tcp   1_bidder_1
22a2f5df9708        jacamars/zerospike:v1   "bash -c './wait-for…"   3 hours ago         Up 44 seconds             0.0.0.0:6000-6002->6000-6002/tcp, 0.0.0.0:7001->7001/tcp                                                        1_zerospike_1
a411d20d4c61        ches/kafka              "/start.sh"              3 hours ago         Up 44 seconds             7203/tcp, 0.0.0.0:9092->9092/tcp                                                                                1_kafka_1
2e063997e8d7        jplock/zookeeper        "/opt/zookeeper/bin/…"   3 hours ago         Up 45 seconds (healthy)   2888/tcp, 0.0.0.0:2181->2181/tcp, 3888/tcp     

$docker exec -it b01c78abf834 ./sendbid

Note that we used CONTAINER ID: b01c78abf834. We will refer to this in the documentation as BIDDER-CONTAINER-ID. Make sure you make the correct substitution.

Sample result of the docker command is shown below:

$docker exec -it b01c78abf834 ./sendbid
{"seatbid":[{"seat":"99999999","bid":[{"impid":"1","id":"35c22289-06e2-48e9-a0cd-94aeb79fab43","price":0.01,"adid":"ben:payday","nurl":"http://localhost/rtb/win/junk1.com/SITE/nexage/${AUCTION_PRICE}/42.378/-71.227/ben:payday/stroer-test/35c22289-06e2-48e9-a0cd-94aeb79fab43","cid":"ben:payday","crid":"stroer-test","iurl":"http://localhost/images/320x50.jpg?adid=ben:payday&bidid=35c22289-06e2-48e9-a0cd-94aeb79fab43","adomain": ["originator.com"],"adm":"%3Ca%20href%3D'http%3A%2F%2Flocalhost%2Fcallback%3Ftype%3Dredirect%2Fexchange%3Dnexage%2Fad_id%3Dben%3Apayday%2Fcreative_id%3Dstroer-test%2Fprice%3D%24%7BAUCTION_PRICE%7D%2Flat%3D42.378%2Flon%3D-71.227%2Fbid_id%3D35c22289-06e2-48e9-a0cd-94aeb79fab43%3Furl%3Dhttp%3A%2F%2Flocalhost%2Fforward%3F99201'%3E%3Cimg%20src%3D'http%3A%2F%2Flocalhost%2Fimages%2F320x50.jpg%3Fadid%3Dben%3Apayday%26amp%3B%2338%3Bbidid%3D35c22289-06e2-48e9-a0cd-94aeb79fab43'%20height%3D'50'%20width%3D'%7Bcreative_ad_width%7D'%3E%3C%2Fa%3E%3Cimg%20src%3D'http%3A%2F%2Flocalhost%2Fcallback%3Ftype%3Dpixel%2Fexchange%3Dnexage%2Fad_id%3Dben%3Apayday%2Fcreative_id%3Dstroer-test%2F35c22289-06e2-48e9-a0cd-94aeb79fab43%2Fprice%3D%24%7BAUCTION_PRICE%7D%2Flat%3D42.378%2Flon%3D-71.227%2Fbid_id%3D35c22289-06e2-48e9-a0cd-94aeb79fab43'%20height%3D'1'%20width%3D'1'%3E"}]}],"id":"35c22289-06e2-48e9-a0cd-94aeb79fab43","bidid":"35c22289-06e2-48e9-a0cd-94aeb79fab43","cur":"USD"} 

System Console

There is a system console that will allow you to review the running status of your bidder. An example of what it looks like is here:

You can find the the RTB4FREE System Console documentation here.

Dynamic Campaigns

In the previous section we used campaigns loaded from the file /database.json in the container's file system. Now we will use your own custom database.json. There are four steps to using your own campaigns:

  1. Download a copy of database.json. We will edit it shortly.
  2. You need to modify docker-compose.yml and un-comment out the /database.json line:
    	volumes:
    	  - "./logs:/logs"
    	  # - "./database.json:/database.json"
    

    When you are done, it should look like this:

    	volumes:
    	  - "./logs:/logs"
    	  - "./database.json:/database.json"
    
  3. In your own directory we will configure the database.json we downloaded to add our own paramaters..
  4. Then we reload database.json into the container "zerospike".
  5. Then we will reload the affected campaign from Zerospike into the bidder.

Load DB

Run "docker ps" command to find the name of the bidder container. On our system it was called "ben_bidder_1". If you are logged in as ubuntu, it would likely be named "ubuntu_bidder_1". Now, that you know the container name, you can tell it to load the database:

$docker exec -it BIDDER-CONTAINER-ID ./rtb-commands -loaddb database.json

As the file /home/ubuntu/database.json is loaded into Zerospike it is printed on the console. The database is loaded into Zerospike, however, the bidder doesn't know about it yet. For this we will use another command. Before moving to the next step, you can list the campaigns available to load into the bidder by using the following command:

$docker exec -it  BIDDER=CONTAINER-ID ./rtb-commands -listcamps

Edit & Reload

Now that basic operations have been defined, you can now edit /home/ubuntu/database.json and load the changes into Zerospike, and then, into the bidder. In this example we will change the price of ben/ben:payday/smartypants from 1.0 to 2.0. Use your editor to modify /home/ubuntu/database.json to change the price field in the ben:payday/smartypants from 1.0 to 2.0. It looks like this:

	...
	"impid" : "smartypants",
	"w" : 300,
	"h" : 250,
	"subtemplate" : "iframe",
	"attributes" : [ ],
	"cur" : "USD",
	"price" : 1.0,            <- change this to "price" : 2.0,
	"adm_override" : false,
	"capFrequency" : 0,
	"capTimeout" : "0"
}
Now do the following:
$docker exec -it  BIDDER-CONTAINER-ID  /bin/sh
vi database.json      <- make sure the price is changed inside the container for ben/ben:payday/smarypants!
:q

#./rtb-commands -loaddb database.json
#./rtb-commands -loadcamps -ids ben:payday

Test

Now we will send a bid to the Bidder and check to see it is bidding at the new price. Execute this command:

	$docker exec -it b01c78abf834 ./sendbid

Look for this field: "price":2.0. The previous time you sent this same bid, its price was 1.0

This somewhat mechanical way of editing campaigns will work for testing, but in a production environment this is pretty clunky. Instead use the production campaign manager to coordinate the campaigns on your real DSP.

Advanced Logging

So far, you have been logging all of the bids, requests, wins, clicks, events, etc. to disk. This is convenient for testing, but in real DSP operations you will want a logging service. We provide logging with Kafka. Therefore, we need to override the bidder's Campaigns/payday.json file to point the logs to Kafka instead of to disk.

After the bidder comes up, we will then verify the Kafka logging is working.

Here's an overview of the Kafka-based logging done with RTB4FREE

Logging Overview

All of the logging done by the bidder is done in JSON format (except the application log). Each log element is one line long terminated by a carriage return. Click here for a sample bid request log:

Now we will switch the logging to use Kafka. More information about Kafka can be found here. But, in a nutshell you will be logging events to Kafka topics, and then you will use Kafka-based subscribers to read the data as it is published. Below is the section in Campaigns/payday.json in the bidder container where the logging is set to disk:

"zeromq" : {
	"bidchannel" : "file://logs/bids",
	"winchannel" : "file://logs/wins",
	"requests" : "file://logs/requests",
	"clicks" : "file://logs/clicks",
	"pixels" : "file://logs/pixels",
	"videoevents": "file://logs/videoevents",
	"postbackevents": "file://logs/postbackevents",
	"status" : "file://logs/status",
	"reasons" : "file://logs/reasons",
	  
	"commands": "tcp://$PUBSUB:6001&commands",
	"responses": "tcp://$PUBSUB:6000&responses",
	"xfrport": "6002",
	"requeststrategy" : "$REQUESTSTRATEGY"
},

Now we will setup our own payday.json that sets up logging to Kafka. Download the kafka-based file here: payday.json This payday.json will be used to override the /Campaigns/payday.json file in the container. The relevant portions that changed are shown below:

"zeromq" : {
	"bidchannel" : "kafka://[$BROKERLIST]&topic=bids",
	"winchannel" : "kafka://[$BROKERLIST]&topic=wins",
	"requests" : "kafka://[$BROKERLIST]&topic=requests",
	"clicks" : "kafka://[$BROKERLIST]&topic=clicks",
	"pixels" : "kafka://[$BROKERLIST]&topic=pixels",
	"videoevents": "kafka://[$BROKERLIST]&topic=videoevents",
	"postbackevents": "kafka://[$BROKERLIST]&topic=postbackevents",
	"status" : "kafka://[$BROKERLIST]&topic=status",
	"reasons" : "kafka://[$BROKERLIST]&topic=reasons",
	  
	"commands": "tcp://$PUBSUB:6001&commands",
	"responses": "tcp://$PUBSUB:6000&responses",
	"xfrport": "6002",  
	"requeststrategy" : "$REQUESTSTRATEGY"
 },

Note the topic names, if you want to tap into the channel, pay attention to the topic.

Also note $PUBSUB, $BROKERLIST, and $REQUESTSTRATEGY. These are environment variables used to override the standard configuration file entries. You can use your own set of environment variables alongside with a set of "built-in" enviornment variables we defined: like $PUBSUB, $BROKERLIST, and $REQUESTSTRATEGY.

The difference is, with our pre-defined variables, they have default valuse so if the variable not defined it has a default value (instead of ""). The list of predefined variables and their defaults can be found here

Logging Setup

The docker-compose.yml file needs to be modified so that the Campaigns/payday.json file is used. Here is what the bidder section looks like where you are using your own database.json, but using the file logging:

		bidder:
		image: "jacamars/rtb4free:v1"
		environment:
		  BROKERLIST: "kafka:9092"
		  PUBSUB: "zerospike"
		  EXTERNAL: "http://localhost:8080"
		  FREQGOV: "false"
		ports:
		  - "8080:8080"
		  - "8155:8155"
		  - "7379:7379"
		  - "7000:7000"
		volumes:
		  - "./logs:/logs"
		  - "./database.json:/database.json"
		#  - "./payday.json:/Campaigns/payday.json"

Now you need to un-comment out the payday.json part of the volume. Your bidder section should look like this:

		bidder:
		image: "jacamars/rtb4free:v1"
		environment:
		  BROKERLIST: "kafka:9092"
		  PUBSUB: "zerospike"
		  EXTERNAL: "http://localhost:8080"
		  FREQGOV: "false"
		ports:
		  - "8080:8080"
		  - "8155:8155"
		  - "7379:7379"
		  - "7000:7000"
		volumes:
		  - "./logs:/logs"
		  - "./database.json:/database.json"
		  - "./payday.json:/Campaigns/payday.json"	

As before, place payday.json in your local directory, alongside your database.json file.

Now we are ready to bring the services up with docker-compose:

$docker-compose up

Verification

Now that the system is up, lets make sure the system works and is logging correctly. We will send a bid and make sure the bids, requests, wins and clicks all are logging correctly.

First, use another console window and load the campaigns:

$docker exec -it BIDDER-CONTAINER-ID /bin/sh
#./load-database
#./rtb-commands -loadcamps -ids ben:payday

Now that the database is loaded, let's send a bid and make sure the bidder responds:

$docker exec -it  BIDDER-CONTAINER-ID   ./sendbid

You should have seen the bid come back. Now we are ready to watch a bid in the console. In yet another console window start the watch-kafka program:

$docker exec -it BIDDER-CONTAINER-ID /bin/sh
#./watch-kafka 'kafka://[kafka:9092]&topic=bids&reader=demo'

The program will now wait. In the other window where you sent the bid. Send another bid. In this window where watch-kafka is running, you should see the bid logged to the console.

To see other topics, just change the &topic=bids to the appropriate topic. For example, to see wind use &topic=wins. Use the browser based bid test program and send a bid, then a win and you should see the log message come out. Or use, the send-win script in the container like so:

$docker exec -it  BIDDER-CONTAINER-ID   ./sendwin

Source Code

The RTB4FREE source code for all the services is located here.