RTB4FREE Details

Last updated: Feb 3, 2017

Overview

This project is for those fairly familiar with RTB. With a basic understanding of RTB, JAVA, Maven and Docker. There is a blog post which details the development of a multi-bidder DSP here: "Building Your Own DSP Parts, One and Two "

Features at a glance:

  1. openRTB 2.5 Compliant
  2. Supports many SSPs
  3. Works with Doubleclick (Google Adx)
  4. Supports SSL
  5. Supports GZIP Compressed Bids
  6. Simplified Bidder Management
  7. Simple Built-in Campaign Manager, easily replaced with your own
  8. Simple logging formats of requests, bids, wins, pixel fires. Easily load into HDFS or Elastic Search
  9. Commerical support and customization is available from this link

Quickstart

The following brings the XRTB system up and running on your local machine, presuming you have Git, Maven, and Java 1.8 installed on your local machine and Aerospike is already running.

In one window:

$git clone https://github.com/benmfaul/XRTB.git
$cd XRTB
$mkdir logs
$mvn assembly:assembly -DdescriptorId=jar-with-dependencies  -Dmaven.test.skip=true

RTB4FREE has two configuration files. The sample database is called "database.json" which is used to initialize the Zerospike system's database The second configuration file is Campaigns/payday.json which sets up the operational parameters for your bidder. Neither of these files exist after you do the GIT clone (or subsequent GIT pull). You need to make these two files on your instanceby copying the samples:

$cd XRTB
$cp sampledb.json database.json
$cp Campaigns/samplecfg.json Campaigns/payday.json
									

If you forget this last step, RTB4FREE will not start. These files are kept local on your instance so that changes you make to Campaigns/payday.json and database.json don't block your ability to GIT pull to get updates for the bidder.

Now you have it all built you need to create the Docker images that contain the RTB4FREE and Zerospike services.

	$docker build zerospike -f Docker.zerospike .
	$docker build rtb4free  -f Docker.rtb4free .

Note, if you have your own repository (or use the Docker respository) you can tag them first. Let's say your repo is MyDSP, then you would do this:

		$docker build zerospike -f MyDSP/Docker.zerospike .
		$docker build rtb4free  -f MyDSP/Docker.rtb4free .

After building you should push them to the repo:

		$docker push MyDSP/rtb4free
		$docker push MyDSP/zerospike 

Now you are ready to start the bidder+zerospike. Use the following command:

	$docker-compose up.

You should see Zerospike, RTB4FREE bidde. Kafka and Zookeeper all come up.

You can send a command to bidder bu doing the following:
	$curl -X POST -d @../SampleBids/nexage.txt http://localhost:8080/rtb/bids/nexage --header "Content-Type:application/json"

You should see the JSON returned for the bid request which will look something like this: <!--#include virtual="COMPONENT/rawbid.ssi" -->

Configuring

database.json

Discuss config in general

Campaigns/payday.json

Discuss Payday.json

Environment Vars

The Campaings/payday.json and database.json are standardized config files. As a general rule you do not want to edit these. Instead, use substitution macros in database.json and environment variables in Campaigns/payday.json to override settings from the Docker compose file. This section details how to work with these 2 files.

Database.json

Database.json is a JSON formatted file defining the internal structure of campaigns within the RTB4FREE system. Basically it is an array of "campaigns". And each campaign is an array of "creatives". within both "campaigns" and "creatives" there is an array called "attributes" that are predefined selection operations you can apply against values in a bid request. These attributes define the rules for selecting whether a bid request is going to be bid on. Each of the types of creatives, video, banner, native, etc., has some predefined items like width or height as well.

The form of the database.json file for campaigns look like this:

[
   {campaign},
   {campaign},
   {campaign}...
]

The form of the campaign looks like this:

	{ 
		"adid": "the ad it",
		some more standard campaign variables...,
		"attribibutes":[rule,rule,rule,...],
		"creatives": [creative,creative,creative,...]	
	}

The form of the creative looks like this:

	{
		"impid": "the creative's impression id",
		some more standard creative attributes...,
		"attributes": [rule,rule,rule,...]
	}

The rules provide the selection criteria. You can examine them in detail here.

Below is a sample campaign with 1 banner creative, and a rule on the campaign

	[{
		"isAdx": false,
		"owner": "ben",
		"adId": "block-test",
		"adomain": "originator.com",
		"attributes": [{
		  "value": "index",
		  "op": "EQUALS",
		  "notPresentOk": false,
		  "bidRequestValues": ["exchange"]
		}],
		"creatives": [{
		  "forwardurl": "http://localhost:8080/contact.html?{site_id}&adid={ad_id}&crid={creative_id}",
		  "imageurl": "http://localhost:8080/images/320x50.jpg?adid={ad_id}&bidid={bid_id}",
		  "impid": "blocker",
		  "w": 50,
		  "h": 50,
		  "attributes": [],
		  "currency": "",
		  "cur": "USD",
		  "price": 1.0,
		  "adm_override": false,
		  "status": "active"
		}],

Macro Definitions

To eliminate the need to hard code things in the campaigns and creatives, and to allow you to pass information about the bid request, macro substitutions are provided for you. Here is a list of Macro definitions understood by the RTB4FREE bidder:

Macro Does

Note, the RTB exchange can make substitutions in the ADM field as it delivers it to the web site. The RTB exchange will reflect the ADM back on the win notification, and you can ask for the RTB exchange to also substitute fields as well. See the RTB 2.5 specification for supported macro names.

These are the macros substituted by the Exchange, not the bidder. so these will return via the RTB win notification and as the ad is delivered into the web page.

Macro Does

Environment Variables

The Docker compose files all support environment variables. There are 5 compose files used: docker-compose-rtb.yml, docker-compose-kafka.yml, docker-compose.web.yml, docker-compose-elk.yml, and docker-compose-haproxy.yml.

Each has its own set of environment parameters

docker-compose-rtb.yml

The following table are the environment variables available by default in the RTB yaml file.

Variable Default Description
ADMINPORT0A separate login port for admin use, separate from the bidder.
BROKERLIST[localhost:9092]List of Kafka cluster members.
CONCURRENCY3Number of threads used to resolve campaigns.
EXTERNALlocalhost:8080The Internet side of the load balancer. Your DSP address.
EVENTlocalhostWhere app install postback events are sent.
INITPORT6002Zerospike port for receiving initial data.
PIXELlocalhostThe pixel host. Where creatives call back
PUBPORT6000Where 0MQ publishers connect.
PUBSUBlocalhostWhere the Zerospike host is.
SUBPORT6001Where 0MQ subscribers connect.
VIDEOlocalhostWhere video events are sent.
WINlocalhostWhere win events are sent.

docker-compose-web.yml

The following describes those environment variables available in the services of the WEB yaml file.

Service Variable Default Description
CrosstalkCONTROL8100Crosstalk control/API port.
CrosstalkJDBC"jdbc:mysql://localhost/rtb4free?user=ben&password=password"Login string to MySQL

docker-compose-kafka.yml

docker-compose-elk.yml

docker-compose-haproxy.yml

Theory of Operation

Zerospike service is used as the shared context between all bidders and crosstalk. All shared data is kept in Zerospike, and all bidders connect to this service to share data. Specifically, the response to a bid request, a 'bid', is stored in Zerospike after it is made, because on the win notification, a completely separate bidder may process the win, and the original bid must be retrieved as quickly as possible to complete the transaction. A database query is far to slow fo this.

Use of ZeroMQ

ZeroMQ is the system used for publish/subscribe for RTB4FREE. Commands are sent to running bidders over a ZeroMQ publish connections. Likewise responses to commands are sent back on another ZeroMQ publish channel. Click-through, wins, and bid notification is sent on other ZeroMQ publish connections. Zerospike is the broker for all ZeroMQ connections

Zerospike Based Shared Database

A database of campaigns is kept in a ConcurrentHashMap and stored in memory cache on each bidder. This allows the bidders to maintain a shared database. The hashmap is stored on disk by Zerospike to a memory mapped database, for use when other bidders come on line.

Configuration

A configuration file is used to set up the basic operating parameters of the bidder (such as ZeroMQ channels) and to load any initial campaigns from the Database in Aerospike.

Upon loading the configuration file into the Configuration class, the campaigns are created, using a set of Node objects that describe the JSON name to look for in the RTB bid, and the acceptable values for that constraint.

Receive Bid

When the RTBBidder starts, it creates a an HTTP handler based on Jetty that handles all the HTTP requests coming into the bidder. The handler will process mundane gets/posts to retrieve resources like images and javascript files placed in the ./web directory. In addition, the bidder will produce a BidRequest object from the JSON payload of the HTTP post. The URI will determine the kind of exchange, e.g. Nexage.

Note, each bid request is on a thread started by JETTY, For each one of these threads, N number of threads will be created for N campaigns. The number of total threads is limited by a configuration parameter "maxConnections". When max connections is reached, the bid request will result in a no-bid.

Campaign Select

Once the Handler determines the bid request and instantiates the BidRequest object, the BidRequest object will then determine which, if any of the campaigns are to be selected. If no campaign was selected, the Handler will return an HTTP 204 code to indicate no reply. Each of the campaigns is loaded into a future task to hold it, and then the tasks are started. When the tasks join, 0 or more of the campaigns may match the bid request. In this case, the campaign is chosen among the set at random.

Note, the RTBServer will place an X-REASON header in the HTTP that explains why the bidder did not bid on the request.

Also note, the RTBServer always places an X-TIME header in the HTPP that describes the time the bidder spent processing a bid request (in milliseconds).

Create Bid Response

The BidRequest object then produces a BidResponse that is usable for this bid request. The bid is first recorded in Aerospike as a Map, then the JSON form is serialized and returned to the Handler. The bid response will then be written to the HTTP response. Note, it is possible to also record the bid requests and the bids in respective ZeroMQ publish channel. This way these messages can be analyzed for further review by other programs subscribing to the ZeroMQ channel.

Win the Auction

If the exchange accepts the bid and awards the win, a win notification is sent to the bidder. The handler will take that notification, which is an encoded URI of information such as auction price, lat, lon, campaign attributes etc. and writes this information to the ZeroMQ channel so that the win can be recorded by some downstream service. The ADM field of the original bid is pulled from Aerospike, using the object-id of the winning bid and returned to the exchange with the ad markup, the referer url and the pixel url.

Ad Served

When the user's screen receives the ad, the pixel URL is fired, and URI encoded GET is read by the Handler to associate the loading of the page in the web browser with the winning bid and this information is sent to a ZeroMQ 'clicks' channel, so that it can be reconciled by some downstream service with the originating bid.

User Clicks the Ad

When the user clicks on the ad, the referrer URL is fired and this is also handled by the handler. The handler then uses the URI encoding to transmit the information to a ZeroMQ channel, usually called 'clicks', for further processing and accounting downstream.

Zeromq Commands

The RTB Bidding engine uses ZeroMQ pub/sub for all communications, such as commands to load the bidder.

Simulator

The Bids Tester provides you with the ability to send test bids to your campaigns loaded in the bidder. You fill out an HTML page for what the bid should look like, press Test button and the bid request is sent. The bidders response JSON is returned plus a visual display of your tracking pixel.

The simulator reads a sample campaign construct in the ./web directory called "./web/config.json" file. This file sets up those parameters you can change through the web page. This file looks exactly like the ../Campaigns file. So look in the README.md in ./Campaigns directory for more information.


WARNING: THIS SIMULATOR IS DESIGNED TO WORK WITH THE database.json AND Campaigns/payday.json FILES ORIGINALLY PROVIDED.

Presuming you did not modify the configuration files and you are running RTB4FREE on your localhost.com you can type following: http://localhost.com:8080/xrtb/simulator/exchange.html

How Fast is the Bidder

After you get your site running, you probably want to know how fast your bidder is.

The actual number of bids per second is highly dependent on a number of factors. How much memory does your system have, how many cores, and what is your network performance?

There is a program in the tools directory to give you a raw number of queries per second. To use it start the RTB bidder in one window. Then in another window, run the following command:

$tools/maxload

This presumes your RTBbidder is running on localhost. If your bidder is on another host then do this:

$tools/maxload -host your-ip-address-or-hostname

By default 10 threads will be used to generate as much traffic as possible. If you have more cores, then up the threads. For example, to 16 threads to 192.168.1.100:

$tools/maxload -host 192.168.1.100 -threads 16

RTB4FREE is developed on an IBM Think Station (12 cores). On this system, also running X Window System and the Unity desktop, the bidder will sustain 6,500 bids per second.

Branding

After you get your site running, you probably want to change the name of the System Console, Campaign Administrator, and the index.html pages to your own branding.

All of these variables can be modified with "BRANDING" environment variables passed to the bidder in the docker-compose file.

SSL

If your SSP requires an SSL connection, you should load your SSL credentials into your load balancer.