Core Open Source Software Developed by RTB4FREE

Last updated: April 18, 2017

Bidder

The Bidder engine is the key component of the RTB4FREE platform. It interfaces directly with the SSP exchanges using OpenRTB protocols or proprietary protocols, if necessary, such as Google AdX or OpenX. The bidder is is a Java version 1.8, and supports OpenRTB version 2.5. The source code with program description can be found here: https://github.com/benmfaul/XRTB

The Bidder features include:

  • OpenRTB 2.5 Compliant.
  • Supports many SSPs.
  • Supports SSL.
  • Supports GZIP Compressed Bids.
  • Simplified standalone Bidder Management.

The Bidder can operate as a stand-alone system. Campaigns and Exchange interfaces can be defined in it's configuration files - see the github source or the demo instructions on this site for detail on how to set up these files. A sample configuration file, stored at Campaigns/payday.json.

Click here to view a sample configuration file:

Bidder Console

Each bidder instance has a console at URL http://bidder_ip_address:8080/admin login that will show operational status and metrics. Log in using root and default password "startrekisbetterthanstarwars", set in the bidder configuration file. (btw, it isn't - PL)

Production Configuration

To support higher transaction rates beyond the capacity of a single instance, bidders can be deployed as a farm of instances, typically front ended by a load balancer like HAProxy. The SSP traffic load is distributed among the farm. This configuration requires the RTB4FREE Crosstalk component which loads configruation information to each bidder instance, and coordinates their activity. It also requires the RTB Zerospike component which acts as a store for shared data among the bidders.

The RTB4FREE Campaign Manager component is usually deployed as a web user interface to allow multiple users to define campaigns, update bidder configurations and view campaign level metrics. Campaign configurations are stored in a MySQL database. The Campaign Manager communicates with theb Crosstalk API when a configuration update is required. Crosstalk will extract data from the MySQL database, then update each bidder in the farm.

Logging

For a base configuration, logging all of the bids, requests, wins, clicks, events, etc. are made to disk. This is convenient for testing, but in real DSP operations you will want a logging service. We provide logging to Kafka by overriding the bidder's Campaigns/payday.json file to point the logs to Kafka instead of to a file. Any micro-service that requires access to log data can connect to Kafka as a consumer, and subscribe to the required log topic.

Here's an overview of the Kafka-based logging done with RTB4FREE

All of the logging done by the bidder is done in JSON format (except the application log). Each log element is one line long terminated by a carriage return. Click here to view a sample bid request log:

Now we will switch the logging to use Kafka. More information about Kafka can be found here. But, in a nutshell you will be logging events to Kafka topics, and then you will use Kafka-based subscribers to read the data as it is published. Below is the section in Campaigns/payday.json in the bidder container where the logging is set to disk:

"zeromq" : {
	"bidchannel" : "file://logs/bids",
	"winchannel" : "file://logs/wins",
	"requests" : "file://logs/requests",
	"clicks" : "file://logs/clicks",
	"pixels" : "file://logs/pixels",
	"videoevents": "file://logs/videoevents",
	"postbackevents": "file://logs/postbackevents",
	"status" : "file://logs/status",
	"reasons" : "file://logs/reasons",
	  
	"commands": "tcp://$PUBSUB:6001&commands",
	"responses": "tcp://$PUBSUB:6000&responses",
	"xfrport": "6002",
	"requeststrategy" : "$REQUESTSTRATEGY"
},

Now we will setup our own payday.json that sets up logging to Kafka. Download the kafka-based file here: payday.json This payday.json will be used to override the /Campaigns/payday.json file in the container. The relevant portions that changed are shown below:

"zeromq" : {
	"bidchannel" : "kafka://[$BROKERLIST]&topic=bids",
	"winchannel" : "kafka://[$BROKERLIST]&topic=wins",
	"requests" : "kafka://[$BROKERLIST]&topic=requests",
	"clicks" : "kafka://[$BROKERLIST]&topic=clicks",
	"pixels" : "kafka://[$BROKERLIST]&topic=pixels",
	"videoevents": "kafka://[$BROKERLIST]&topic=videoevents",
	"postbackevents": "kafka://[$BROKERLIST]&topic=postbackevents",
	"status" : "kafka://[$BROKERLIST]&topic=status",
	"reasons" : "kafka://[$BROKERLIST]&topic=reasons",
	  
	"commands": "tcp://$PUBSUB:6001&commands",
	"responses": "tcp://$PUBSUB:6000&responses",
	"xfrport": "6002",  
	"requeststrategy" : "$REQUESTSTRATEGY"
 },

Note the topic names, if you want to tap into the channel, pay attention to the topic.

Also note $PUBSUB, $BROKERLIST, and $REQUESTSTRATEGY. These are environment variables used to override the standard configuration file entries. You can use your own set of environment variables alongside with a set of "built-in" enviornment variables we defined: like $PUBSUB, $BROKERLIST, and $REQUESTSTRATEGY.

The difference is, with our pre-defined variables, they have default valuse so if the variable not defined it has a default value (instead of ""). The list of predefined variables and their defaults can be found here

All the RTB4FREE configuration files are designed to work in concert with Docker compose files so that key parameters can be changed with environment variables. See the Deploy section for details.