RTB4FREE Bidder

(NOTE: THIS IS A WORK IN PROGRESS)

A Real Time Bidder (RTB 2.3) engine written in Java 1.8

The RTB4FREE project contains 3 major components:
  • Real Time Bidding engine
  • Simulator for sending test bids
  • Administration system for campaign definition

This project is for those fairly familiar with RTB. With a basic understanding of RTB, this project will get you up and running with a commercial grade bidder in a short period of time.

A production bidding enterprise would most likely require multiple bidding engines running behind a firewall. This project does not provide NGINX or AWS Load Balancer (or similar) infrastructure for this, you will need to tailor the administration of the XRTB to deal with these production issues. The XRTB comes out of the box ready to run in a multi-server bidding farm, you just need to lash it up and administer it. There is a blog post which details the development of a multi-bidder DSP here: "Building Your Own DSP Parts, One, Two and Three "


QUICK START - WITH NO AEROSPIKE

RTB4FREE comes in 2 versions. One is a standalone system, and is intended for running on a single instance. It requires no Aerospike support. Instead it uses an embedded Cache2k cache. Use this version if you just want to play around with the system. If you plan to build a production DSP, use the Aerospike Enabled version. Look here for the stand-alone RTB4FREE server with no Aerospike. The rest of this page is devoted to building and running the Aerospike version of RTB4FREE.

BUILDING THE SYSTEM

The following brings the XRTB system up and running on your local machine, presuming you have Git Aerospike, Maven, and Java 1.8 installed on your local machine and Aerospike is already running.

In one window:

$git clone https://github.com/benmfaul/XRTB.git
cd XRTB
$mvn assembly:assembly -DdescriptorId=jar-with-dependencies  -Dmaven.test.skip=true

Now you have it all built. But, the RTB4FREE system comes out of the box using rtb4free.com:8080 for all it's operations. Obviously, for your RTB system that's not going to work. Your RTB4FREE bidder has to be open on the Internet for the exchanges to contact. So, you setup RTB4FREE's web access to use your address using the tools/config-website program For example, let's say your host's IP address is 199,188.62.6 you need to modify all the web based files to use this. This is easy, edit build.xml and look for the "config-website" target. For example sake, let's say your RTB address 192.188.62.6 on port 8080, ()the default port). See below:

$tools/config-website -address 192.188.62.6
$tools/load-database
$tools/rtb4free
You should see the RTB4FREE bidder come up, and print some logging information. In another window execute this command:
$cd XRTB
$curl -X POST -d @../SampleBids/nexage.txt http://192,188.62.6:8080/rtb/bids/nexage --header "Content-Type:application/json"
You should see the JSON returned for the bid request which will look something like this: {"seatbid":[{"seat":"seat1","bid":[{"impid":"1","id":"35c22289-06e2-48e9-a0cd-94aeb79fab43","price":1.0,"adid":"ben:payday" ,"nurl":"http://rtb4free.com:8080/rtb/win/smaato/${AUCTION_PRICE}/42.378/-71.227/ben:payday/23-2-skiddoo/35c22289-06e2-48e9-a0 cd-94aeb79fab43","cid":"ben:payday","crid":"23-2-skiddoo","iurl":"http://rtb4free.com:8080/images/320x50.jpg?adid=ben:payday&b idid=35c22289-06e2-48e9-a0cd-94aeb79fab43","adomain": ["originator.com"],"adm":"<ad xmlns:xsi=\"http://www.w3.org/2001/XMLS chema-instance\" xsi:noNamespaceSchemaLocation=\"smaato_ad_v0.9.xsd\" modelVersion=\"0.9\"><imageAd><clickUrl>http://localho st:8080/redirect/exchange=smaato/ben:payday/creative_id=23-2-skiddoo/price=${AUCTION_PRICE}/lat=42.378/lon=-71.227/bid_id=35 c22289-06e2-48e9-a0cd-94aeb79fab43?url=http://rtb4free.com:8080/contact.html?99201&amp;adid=ben:payday&amp;crid=23-2-skiddoo</c lickUrl><imgUrl>http://rtb4free.com:8080/images/320x50.jpg?adid=ben:payday&amp;bidid=35c22289-06e2-48e9-a0cd-94aeb79fab43</imgU rl><width>320</width><height>50</height><toolTip></toolTip><additionalText></additionalText><beacons><beacon>http://localhos t:8080/pixel/exchange=smaato/ad_id=ben:payday/creative_id=23-2-skiddoo/35c22289-06e2-48e9-a0cd-94aeb79fab43/price=${AUCTION_ PRICE}/lat=42.378/lon=-71.227/bid_id=35c22289-06e2-48e9-a0cd-94aeb79fab43</beacon></beacons></imageAd></ad>"}]}],"id":"35c22 289-06e2-48e9-a0cd-94aeb79fab43","bidid":"35c22289-06e2-48e9-a0cd-94aeb79fab43"}

Modifying Your Bidder for Actual Operation

Startup Parameters

Now that your bidder is running in the initial configuration, you can modify the startup to run for real.

The startup parameters can be modified by hand by editing Campaigns/payday.json. However you can use the system Console for the bidder to make the necessary changes. Of critical importance is to change the Aerospike location (if necessary), and modify the win, click and pixel fire URLs to the actual IP address of the bidder instead of rtb4free.com!

  1. First, start the bidder using the default configuration.
  2. Login to the "root" login of the Console
  3. Follow the directions for modifying the configuration.
  4. Save the configuration (it will be called Campaigns/payday.json.new)
  5. Copy Campaigns/payday.json.new over Campaigns/payday.json
  6. Restart the bidder.

Campaigns

Remember, the campaigns are loaded into Aerospike. The initial campaigns can be loaded into Aerospike using the tools/load-databaseprogram, The sample database is stored as a JSON file called ./database.json. You can modify it by hand and reload using tools/database, but that is really tedious. Instead, use the Campaign Manager, which is described here.

Macro Definitions

Here is a list of Macro definitions understood by the RTB4FREE bidder.

        {ad_id}					Substituted from the campaign id.
        {app_id}				The id of the app in the bid request.
        {app_name}				The name of the app in the bid request.
        {app_bunde}				The app.bundle of the bid request.
        {adsize}				Size of the creative, hxw
        {bid_id}				Substituted from the bid's object id field.
        {bidder_ip}				Gives you the bidder ip address that handled the bid request.
        {brand}					The device brand.
        {cachebuster}				A random numnber.
        {creative_ad_height}			The height of the creative bid.
        {creative_ad_price}			The price that was bid.
        {creative_ad_width}			The creative width of the bid.
        {creative_forward_url}			The forward url (the actual markup sent on the bid.
        {creative_id}           		Substituted from campaign's creative's id.
        {dnt}					The value of device.dnt (do not track) in the bid request.
        {exchange}				The exchange where the bid came from
        {gps}					Device GPS coordinates lat,lon
        {ifa}					The value of device.ifa in the bid request.
        {isp}					The ISP name if the device identifies it.
        {lat}					Latitude of the device
        {lon}					Longitude of the device
        {make}					The device make
        {model}					The device model
        {os}					The device operating system
        {osv}					The device operating system version
        {page_url}				The page url, contents of site.content.url or if undefined site.pahe, or if app: app,content.url
        {pixel_url}				The pixel fire url of this bid request
        {publisher}				The publisher from the bid request
        {redirect_url}				The redirect url used on this bid
        {site_domain}				Substituted from the bid request's site domain
	{site_id}				Substituted from the bid request's site id.
        {site_name}				Substituted from the bid request's site name.
        {timestamp}				Timestamp of the bid
        {ua}					Device user agent
        {win_url}				The win fire url of the bid response.

Note, the RTB exchange can make substitutions in the ADM field as it delivers it to the web site. The RTB exchange will reflect the ADM back on the win notification, and you can ask for the RTB exchange to also substitute fields as well. See the RTB 2.3 specification for supported macro names.

These are the macros substituted by the Exchange, not the bidder. so these will return via the RTB win notification and as the ad is delivered into the web page.


        {AUCTION_ID}        ID of the bid request; from “id” attribute.
        {AUCTION_BID_ID}    ID of the bid; from “bidid” attribute.
        {AUCTION_IMP_ID}    ID of the impression just won; from “impid” attribute.
        {AUCTION_SEAT_ID}   ID of the bidder’s seat for whom the bid was made.
        {AUCTION_AD_ID}     ID of the ad markup the bidder wishes to serve; from “adid” attribute.
        {AUCTION_PRICE}     Settlement price using the same currency and units as the bid.
        {AUCTION_CURRENCY}  The currency used in the bid (explicit or implied); for confirmation only.

Running the Bidder

Now that the configuration file is setup, now you start the bidder. The tools/rtb4free script will start the bidder and load the sample Campaigns/payday.json as the configuration file:

$tools/rtb4free

However you can start RTB4FREE as a service by using systemctl. Copy the file rtb4free.service to /etc/systemd/system and then run sudo systemctl daemon-reload. Now you can start, stop, and restart RTB4FREE as a service:

$sudo systemctl start rtb4free

The log file is in /var/logs/rtb4free.log.

RUNNING THE SIMULATOR

The Bids Tester provides you with the ability to send test bids to your campaigns loaded in the bidder. You fill out an HTML page for what the bid should look like, press Test button and the bid request is sent. The bidders response JSON is returned plus a visual display of your tracking pixel.

The simulator reads a sample campaign construct in the ./web directory called "./web/config.json" file. This file sets up those parameters you can change through the web page. This file looks exactly like the ../Campaigns file. So look in the README.md in ./Campaigns directory for more information.


WARNING: THIS SIMULATOR IS DESIGNED TO WORK WITH THE database.json AND Campaigns/payday.json FILES ORIGINALLY PROVIDED.

Presuming you did not modify the configuration files and you are running RTB4FREE on your rtb4free.com.com you can type following: http://rtb4free.com.com:8080/xrtb/simulator/exchange.html


THEORY OF OPERATION

Use of Aerospike

Aerospike is used as the shared context between all bidders. All shared data is kept in Aerospike, and all bidders connect to this instance to share data. Specifically, the response to a bid request, a 'bid', is stored in Aerospike after it is made, because on the win notification, a completely separate bidder may process the win, and the original bid must be retrieved as quickly as possible to complete the transaction. A database query is far to slow fo this,

Use of ZeroMQ

ZeroMQ is the system used for publish/subscribe for RTB4FREE. Commands are sent to running bidders over a ZeroMQ publish connections. Likewise responses to commands are sent back on another ZeroMQ publish channel. Click-through, wins, and bid notification is sent on other ZeroMQ publish connections.

Aerospike Based Shared Database

A database of Users and their campaigns is kept in a ConcurrentHashMap and stored in Aerospike. This allows the bidders to maintain a shared database. The HashMap is actually backed in Aerospike using a Map bin.

Configuration

A configuration file is used to set up the basic operating parameters of the bidder (such as ZeroMQ channels) and to load any initial campaigns from the Database in Aerospike.

Upon loading the configuration file into the Configuration class, the campaigns are created, using a set of Node objects that describe the JSON name to look for in the RTB bid, and the acceptable values for that constraint.

Receive Bid

When the RTBBidder starts, it creates a an HTTP handler based on Jetty that handles all the HTTP requests coming into the bidder. The handler will process mundane gets/posts to retrieve resources like images and javascript files placed in the ./web directory. In addition, the bidder will produce a BidRequest object from the JSON payload of the HTTP post. The URI will determine the kind of exchange, e.g. Nexage.

Note, each bid request is on a thread started by JETTY, For each one of these threads, N number of threads will be created for N campaigns. The number of total threads is limited by a configuration parameter "maxConnections". When max connections is reached, the bid request will result in a no-bid.

Campaign Select

Once the Handler determines the bid request and instantiates the BidRequest object, the BidRequest object will then determine which, if any of the campaigns are to be selected. If no campaign was selected, the Handler will return an HTTP 204 code to indicate no reply. Each of the campaigns is loaded into a future task to hold it, and then the tasks are started. When the tasks join, 0 or more of the campaigns may match the bid request. In this case, the campaign is chosen among the set at random.

Note, the RTBServer will place an X-REASON header in the HTTP that explains why the bidder did not bid on the request.

Also note, the RTBServer always places an X-TIME header in the HTPP that describes the time the bidder spent processing a bid request (in milliseconds).

Create Bid Response

The BidRequest object then produces a BidResponse that is usable for this bid request. The bid is first recorded in Aerospike as a Map, then the JSON form is serialized and returned to the Handler. The bid response will then be written to the HTTP response. Note, it is possible to also record the bid requests and the bids in respective ZeroMQ publish channel. This way these messages can be analyzed for further review by other programs subscribing to the ZeroMQ channel.

Win the Auction

If the exchange accepts the bid and awards the win, a win notification is sent to the bidder. The handler will take that notification, which is an encoded URI of information such as auction price, lat, lon, campaign attributes etc. and writes this information to the ZeroMQ channel so that the win can be recorded by some downstream service. The ADM field of the original bid is pulled from Aerospike, using the object-id of the winning bid and returned to the exchange with the ad markup, the referer url and the pixel url.

Mobile Ad Served

When the mobile user's screen receives the ad, the pixel URL is fired, and URI encoded GET is read by the Handler to associate the loading of the page in the web browser with the winning bid and this information is sent to a ZeroMQ 'clicks' channel, so that it can be reconciled by some downstream service with the originating bid.

User Clicks the Ad

When the mobile user clicks on the ad, the referrer URL is fired and this is also handled by the handler. The handler then uses the URI encoding to transmit the information to a ZeroMQ channel, usually called 'clicks', for further processing and accounting downstream.

SENDING BIDS TO THE RTB4FREE SERVER

After starting the RTB server you can send it a test bid by pointing your browser to http://rtb4free.com.com:8080/xrtb/simulator/exchange

(The test page presumes you are using the Campaigns/payday.json campaign definition file)

The test page will let you change the bid request attributes domain and the geo.country field in a canned bid. You can also add other constraints to customize the bid request

The test page supports DISPLAY, VIDEO and NATIVE CONTENT ad bid requests. Choose the kind of bid you want first by pressing the appropriate Display or Native or Video button first

If you like, you can use the JavaScript window to set the values of the bid request. For example, you can override the id by using bid.id = '11111'; in the window. Push the Free Form "Show" button and the bid is shown as a JavaScript object. Make the changes you want in that object and then hit the 'Test' button

Press the Test button - The X-TIME of the bid/response will be shown. The NURL will be returned and the image of the advertisement is displayed. Below this you can see the contents of the bid/response. If the server bid, you can send a win notification by pressing the "Send Win" button.This will also cause the contents of the ADM to be displayed as it is seen in the user's web browser and what the HTML was that produced it.

For display ads, you can see the actual graphic and if you press the ad, the pixel and reder handlers will be called.

If the bid was a video ad, the content of the ad is shown on the left, and on the right, the video tag XML is shown as sent by the bidder

----> To see a no bid, select GER for geo.country. The X-REASON will then be displayed on the page.

if you want to run the simulator on the rtb4free web site, click here:http://rtb4free.com:8080/xrtb/simulator/exchange.html


ZeroMQ COMMANDS

The RTB Bidding engine uses ZeroMQ pub/sub for all communications. Specifically, logs are published, wins are published, the bid requests can be published and you can also publish bids.

In addition, the RTB4FREE servers subscribe to a commands channel and you can get status information, pause and restart servers, set throttles, create. delete and update campaigns in the servers in real time

In sending commands, you can send commands to ALL the servers, or a subset of the servers, or to an individual server. The concept of a shard key is used with the servers. Each server is addressed in the ZeroMQ pub/sub with its shardkey + the hostname + port number. The servers understand wild card naming, so if you have 5 servers you can organize them by types. For example, servers 1-3 are dedicated to BigCo campaigns and servers 4-5 are dedicated to everyone else. So using a shard key of "BIGCO" and "NOT-BIGCO", your 5 server names might look like: BIGCO:host1:8080, BIGCO:host2:8080, BIGCO:host3:8080, NOT-BIGCO:host4:8080, and NOT-BIGCO:host5:8080

You can send a command to all of the bidders in the commands by using the name field of "*". Just BIGCO by using "BIGCO*".

Note, the shard key for the RTB4FREE.com demonstration bidder is 'demo'. The entire instance-name is "demo:rtb4free.com:8080"

The following will show you an overview of the commands that you can issue from the command line tool. Looking in the package for the commands, you can see how to programmatically issue these commands.

The basic commands are:

The commands are Java objects, all derived from the com.xrtb.commands.BasicCommand class. (See here for the JAVADOC). The objects are serialized and then sent to the ZeroMQ commands topic. The responses to the commands are transmitted back via the responsestopic.

In order to make communications with the bidding system easier, a program called tools.Commander is provided so that you can easily send and receive commands with the bidding engines. In the following sub-sections each of these commands is discussed and the appropriate Commander use is shown. To start the Commander, use from the top directory:

$tools/rtb-commander
    
(Note, this presumes you are using Aerospike and RTB4FREE on rtb4free.com. If you are using a different configuration, see the JAVADOC for the Commander, ot use tools/rtb-commander -h) The Commander will reply with a prompt, then you specify the appropriate response on STDIN. Also remember, that when a command asks for which bidder to send to, if you use the carriage return the command is sent to ALL bidders. To send to a particular bidder, use it's instance name. Also, you can use JAVA regular expressions for the name. An example: 'demo(.*)' will match all bidders beginning with the shard name 'demo'.

1 - Echo

The echo command will prompt the bidders to reply with their status. The status will include a list of the campaigns it is running as well as summary statistics about the use of the system. Example use:
$tools/rtb-commander
RTB4FREE Commander
Command: (1=Echo, 2=Load, 3=Delete, 4=Stop Campaign, 5=Start Campaign, 6=Start Bidder, 7=Stop Bidder
?? 1
Echo to?

The command has 1 argument, 'to'. Put the instance name of which bidder you want to respond, otherwise, simply pressing cr will cause all bidders to respond. You can also use JAVA regular expressions. For example, specifying demo(.*) will match all bidders that have 'demo' in their shard key.

The best way to find out what systems are running is to issue the 1 (echo) and simply press CR. All of the bidders will answer. Then you can simply issue the ECHO command to whatever bidders you would like using the wild-card feature

You can also use ZeroMQ to send the bidder(s) commands to add campaigns, delete campaigns, start the bidder, stop the bidder and retrieve statistics. Each command is a JSON string. Look in the package com.xrtb.commands for examples on the actual command formats. However, here are the basics. The bidders listen on the 'commands' topic, and return values on 'responses' topic. Each of the commands and their responses may have additional fields. Here are examples

The echo command retrieves status information about a bidder's state. All of its campaigns are listed, as well as some summary statistics. The class implementing this is com.xrtb.commands.Echo.java. An example echo response from the Commander program can be seen below:

     
$tools/rtb-commander
RTB4FREE Commander
(1=Echo, 2=Load JSON Database into Aerosppike, 3=Delete Campaign from Aerospike,
4=Stop Campaign, 5=Start Campaign, 6=Start Bidder, 7=Stop Bidder,
8=Exit Commander)
??
??<<<<<{
  "instance";"bidder-name",
  "campaigns": [
    {
      "adId": "ben:payday",
      "price": 5.0,
      "adomain": "originator.com",
      "template": {
        "default": "999",
        "exchange": {
          "mopub": "\u003ca href\u003d\u0027mopub template here\u0027 \u003c/a\u003e",
          "mobclix": "\u003ca href\u003d\u0027mobclix template here\u0027 \u003c/a\u003e",
          "nexage": "\u003ca href\u003d\u0027{redirect_url}/{campaign_id}/{pub}/{bid_id}?url\u003d{campaign_forward_url}\u0027\u003e\u003cimg src\u003d\u0027{pixel_url}/{pub}/{ad_id}/{bid_id}/${AUCTION_PRICE}/{creative_id}\u0027 height\u003d\u00271\u0027 width\u003d\u00271\u0027\u003e\u003c/img\u003e\u003cimg src\u003d\u0027{campaign_image_url}\u0027 height\u003d\u0027{campaign_ad_height}\u0027 width\u003d\u0027{campaign_ad_width}\u0027\u003e\u003c/img\u003e\u003c/a\u003e"
        }
      },
      "attributes": [
        {
          "operator": 4,
          "value": [
            "chive.com",
            "junk.com"
          ],
          "lval": [
            "chive.com",
            "junk.com"
          ],
          "op": "NOT_MEMBER",
          "notPresentOk": true,
          "bidRequestValues": [
            "site",
            "domain"
          ]
        },
        {
          "operator": 3,
          "value": [
            "USA",
            "MEX"
          ],
          "lval": [
            "USA",
            "MEX"
          ],
          "op": "MEMBER",
          "notPresentOk": true,
          "bidRequestValues": [
            "user",
            "geo",
            "country"
          ]
        },
        {
          "operator": 3,
          "value": [
            "CA",
            "NY",
            "MA"
          ],
          "lval": [
            "CA",
            "NY",
            "MA"
          ],
          "op": "MEMBER",
          "notPresentOk": true,
          "bidRequestValues": [
            "rtb4free",
            "geocode",
            "state"
          ]
        }
      ],
      "creatives": [
        {
          "forwardurl": "http://rtb4free.com.com:8080/forward?{site_id}",
          "encodedFurl": "http%3A%2F%2Frtb4free.com.com%3A8080%2Fforward%3F%7Bsite_id%7D",
          "imageurl": "http://rtb4free.com.com:8080/images/320x50.jpg?adid\u003d{ad_id}\u0026#38;bidid\u003d{bid_id}",
          "encodedIurl": "http%3A%2F%2Frtb4free.com.com%3A8080%2Fimages%2F320x50.jpg%3Fadid%3D%7Bad_id%7D%26%2338%3Bbidid%3D%7Bbid_id%7D",
          "impid": "23skiddoo",
          "w": 320.0,
          "h": 50.0
        },
        {
          "forwardurl": "http://rtb4free.com.com:8080/forward?{site_id}",
          "encodedFurl": "http%3A%2F%2Frtb4free.com.com%3A8080%2Fforward%3F%7Bsite_id%7D",
          "imageurl": "http://rtb4free.com.com:8080/images/320x50.gif?adid\u003d{ad_id}\u0026#38;bidid\u003d{bid_id}",
          "encodedIurl": "http%3A%2F%2Frtb4free.com.com%3A8080%2Fimages%2F320x50.gif%3Fadid%3D%7Bad_id%7D%26%2338%3Bbidid%3D%7Bbid_id%7D",
          "impid": "66skiddoo",
          "w": 640.0,
          "h": 480.0
        }
      ],
      "date": [
        20130205,
        20200101
      ]
    }
  ],
  "percentage": 100,
  "stopped": false,
  "bid": 0,
  "nobid": 0,
  "error": 0,
  "handled": 1,
  "unknown": 0,
  "cmd": 5,
  "to": "*",
  "msg": "undefined",
  "status": "ok",
  "type": "status"
}
??

2 - Load Campaigns into Aerospike Database

This command will load a JSON formatted campaign from file into the Aerospike database. The file contains an array of User records (user with one or more campaigns).

                
$tools/rtb-commander
RTB4FREE Commander
(1=Echo, 2=Load JSON Database into Aerospike, 3=Delete Campaign from Aerospike,
4=Stop Campaign, 5=Start Campaign, 6=Start Bidder, 7=Stop Bidder,
8=Exit Commander)
??2
Filename to load:database.json
[ {
  "name" : "ben",
  "directory" : "web/users/ben",
  "origin" : 1431197706849,
  "lastAccess" : 1431197706849,
  "campaigns" : [ {
    "adId" : "ben:payday",
    "adomain" : "originator.com",
    "template" : {
      "default" : "999",
      "exchange" : {
        "mopub" : "",
        "mobclix" : "",
        "nexage" : "",
        "privatex" : ""
      }
    },
    "attributes" : [ {
      "value" : [ "chive.com", "junk.com" ],
      "op" : "NOT_MEMBER",
      "notPresentOk" : true,
      "bidRequestValues" : [ "site", "domain" ]
    }, {
      "value" : [ "USA", "MEX" ],
      "op" : "MEMBER",
      "notPresentOk" : true,
      "bidRequestValues" : [ "user", "geo", "country" ]
    }, {
      "value" : [ "CA", "NY", "MA" ],
      "op" : "MEMBER",
      "notPresentOk" : true,
      "bidRequestValues" : [ "rtb4free", "geocode", "state" ]
    } ],
    "creatives" : [ {
      "impid" : "iAmAnAppWall",
      "w" : 0.0,
      "h" : 0.0,
      "attributes" : [ ],
      "currency" : "US",
      "price" : 10.0,
      "nativead" : {
        "assets" : [ {
          "title" : {
            "text" : "Install BOA"
          },
          "link" : {
            "url" : "http://i.am.a/URL"
          }
        }, {
          "img" : {
            "url" : "http://cdn.mobad.com/ad.png",
            "w" : 64,
            "h" : 64
          }
        }, {
          "link" : {
            "url" : "http://i.am.a/URL"
          },
          "data" : {
            "value" : "Install",
            "type" : 2
          }
        }, {
          "data" : {
            "value" : "5",
            "type" : 3
          }
        } ],
        "link" : {
          "url" : "deeplink://deeplink/url/into/app",
          "fallback" : "http: //i.am.a/URL",
          "clicktrackers" : [ "http: //a.com/a", "http: //b.com/b" ]
        },
        "imptrackers" : [ "http: //a.com/a", "http: //b.com/b" ],
        "nativeAdType" : 2
      }
    }, {
      "impid" : "iAmStreamingContentVideo",
      "w" : 0.0,
      "h" : 0.0,
      "attributes" : [ ],
      "currency" : "US",
      "price" : 10.5,
      "nativead" : {
        "assets" : [ {
          "title" : {
            "text" : "Install BOA"
          }
        }, {
          "img" : {
            "url" : "http://cdn.mobad.com/ad.png",
            "w" : 64,
            "h" : 64
          }
        }, {
          "video" : {
            "value" : "",
            "duration" : 25,
            "linearity" : 1,
            "protocol" : "VAST 2.0"
          }
        }, {
          "data" : {
            "value" : "Click",
            "type" : 2
          }
        } ],
        "link" : {
          "url" : "deeplink://deeplink/url/into/app",
          "fallback" : "http: //i.am.a/URL",
          "clicktrackers" : [ "http: //a.com/a", "http: //b.com/b" ]
        },
        "imptrackers" : [ "http: //a.com/a", "http: //b.com/b" ],
        "nativeAdType" : 6
      }
    }, {
      "forwardurl" : "http://rtb4free.com.com:8080/forward?{site_id}",
      "imageurl" : "http://rtb4free.com.com:8080/images/320x50.jpg?adid={ad_id}&bidid={bid_id}",
      "impid" : "23skiddoo",
      "w" : 320.0,
      "h" : 50.0,
      "attributes" : [ ],
      "currency" : "US",
      "price" : 5.0
    }, {
      "forwardurl" : "http://rtb4free.com.com:8080/forward?{site_id}",
      "imageurl" : "http://rtb4free.com.com:8080/images/320x50.gif?adid={ad_id}&bidid={bid_id}",
      "impid" : "66skiddoo",
      "w" : 640.0,
      "h" : 480.0,
      "attributes" : [ ],
      "currency" : "US",
      "price" : 4.0
    }, {
      "impid" : "iAmVideo",
      "w" : 400.0,
      "h" : 200.0,
      "attributes" : [ ],
      "adm" : [ "http://rtb4free.com.com:8080/vast/onion270.xml" ],
      "currency" : "US",
      "videoProtocol" : 2,
      "videoDuration" : 30,
      "videoLinearity" : 1,
      "videoMimeType" : "application/x-shockwave-flash",
      "price" : 3.0
    } ],
    "date" : [ 20130205, 20200101 ]
  }, {
    "adId" : "ben:fyber",
    "adomain" : "originator.com",
    "template" : {
      "default" : "999",
      "exchange" : {
        "mopub" : "",
        "mobclix" : "",
        "nexage" : ""
      }
    },
    "attributes" : [ {
      "value" : "EUR",
      "op" : "MEMBER",
      "notPresentOk" : true,
      "bidRequestValues" : [ "cur" ]
    }, {
      "value" : "seat1",
      "op" : "MEMBER",
      "notPresentOk" : true,
      "bidRequestValues" : [ "wseat" ]
    } ],
    "creatives" : [ {
      "impid" : "1",
      "w" : 640.0,
      "h" : 480.0,
      "attributes" : [ ],
      "adm" : [ "http://rtb4free.com.com:8080/vast/onion270.xml" ],
      "currency" : "EUR",
      "videoProtocol" : 3,
      "videoDuration" : 10,
      "videoLinearity" : 1,
      "price" : 6.91156
    } ],
    "date" : [ 20130205, 20200101 ]
  }, {
    "adId" : "ben:extended-device",
    "adomain" : "originator.com",
    "template" : {
      "default" : "999",
      "exchange" : {
        "mopub" : "",
        "mobclix" : "",
        "nexage" : ""
      }
    },
    "attributes" : [ {
      "value" : [ "chive.com", "junk.com" ],
      "op" : "NOT_MEMBER",
      "notPresentOk" : true,
      "bidRequestValues" : [ "site", "domain" ]
    }, {
      "value" : [ "USA", "MEX" ],
      "op" : "MEMBER",
      "notPresentOk" : true,
      "bidRequestValues" : [ "user", "geo", "country" ]
    } ],
    "creatives" : [ {
      "forwardurl" : "http://rtb4free.com.com:8080/forward?{site_id}",
      "imageurl" : "http://rtb4free.com.com:8080/images/320x50.jpg?adid={ad_id}&bidid={bid_id}",
      "impid" : "23skiddoo",
      "w" : 320.0,
      "h" : 50.0,
      "attributes" : [ ],
      "currency" : "US",
      "price" : 2.0
    }, {
      "forwardurl" : "http://rtb4free.com.com:8080/forward?{site_id}",
      "imageurl" : "http://rtb4free.com.com:8080/images/320x50.gif?adid={ad_id}&bidid={bid_id}",
      "impid" : "66skiddoo",
      "w" : 640.0,
      "h" : 480.0,
      "attributes" : [ ],
      "currency" : "US",
      "price" : 1.0
    } ],
    "date" : [ 20130205, 20200101 ]
  } ]
} ]
Database init complete.
RTB4FREE Commander (1=Echo, 2=Load into DB, 3=Delete from DB, 4=Stop Campaign, 5=Start Campaign, 6=Start Bidder, 7=Stop Bidder, 8=Exit Commander)
??

3 - Delete Campaign from Aerospike Database.

This command deletes a campaign from the Database (not the bidder memory).

$tools/rtb-commander
RTB4FREE Commander
(1=Echo, 2=Load JSON Database into Aerospike, 3=Delete Campaign from Aerospike,
4=Stop Campaign, 5=Start Campaign, 6=Start Bidder, 7=Stop Bidder,
8=Exit Commander)
Campaign to Delete from Aerospike:ben:payday
??3
Ok, campaign deleted
RTB4FREE Commander
(1=Echo, 2=Load JSON Database into Aerospike, 3=Delete Campaign from Aerospike,
4=Stop Campaign, 5=Start Campaign, 6=Start Bidder, 7=Stop Bidder,
8=Exit Commander)
??


4 - Stop A Campaign (Unload Campaign from Bidder Memory) Memory).

This command will unload a campaign from the bidders memory, thus stopping the campaign

             
$java -cp "libs/*" tools.Commands       
RTB4FREE Commander        
(1=Echo, 2=Load JSON Database into Aerospike, 3=Delete Campaign from Aerospike,
4=Stop Campaign, 5=Start Campaign, 6=Start Bidder, 7=Stop Bidder,
8=Exit Commander)
??4
Which campaign to stop:ben:payday
Which bidder to notify: 
??<<<<<{
  "cmd" : -1,
  "from" : "demo:ben-OptiPlex-780:8080",
  "to" : "c9cf37f8-77b4-4a09-ab40-c4cae519ba57",
  "id" : "na",
  "msg" : "Campaign deleted",
  "status" : "ok",
  "type" : "status",
  "target" : "na"
}

5 - Start a Campaign in the Bidder. (Load campaign from Aerospike into Bidder)/h2>

This command loads a campaign from the Database into the bidder memory, thus starting it

                
$tools/rtb-commander             
RTB4FREE Commander
(1=Echo, 2=Load JSON Database into Aerospike, 3=Delete Campaign from Aerospike,
4=Stop Campaign, 5=Start Campaign, 6=Start Bidder, 7=Stop Bidder,
8=Exit Commander)
??<<<<<{
  "cmd" : -1,
  "from" : "demo:ben-OptiPlex-780:8080",
  "to" : "7d9ad779-16c1-433c-b670-4c4876e984b6",
  "id" : "na",
  "msg" : "undefined",
  "status" : "ok",
  "type" : "status",
  "target" : "na"
}
??

6 - Start Bidding (Resume the Bidders)

Issuing this command will cause the bidder to respond to bid requests by processing loaded campaigns

$tools/rtb-commander              
RTB4FREE Commander
(1=Echo, 2=Load JSON Database into Aerospike, 3=Delete Campaign from Aerospike,
4=Stop Campaign, 5=Start Campaign, 6=Start Bidder, 7=Stop Bidder,
8=Exit Commander)
??
Which bidder to stop:
RTB4FREE Commander
(1=Echo, 2=Load JSON Database into Aerospike, 3=Delete Campaign from Aerospike,
4=Stop Campaign, 5=Start Campaign, 6=Start Bidder, 7=Stop Bidder,
8=Exit Commander)
??<<<<<{
  "cmd" : -1,
  "from" : "demo:ben-OptiPlex-780:8080",
  "to" : "7d9ad779-16c1-433c-b670-4c4876e984b6",
  "id" : "na",
  "msg" : "stopped",
  "status" : "ok",
  "type" : "status",
  "target" : "na"
}
??

7 - Stop Bidding (Pause the Bidders)

Issuing this command will cause the bidder to stop responding to bid requests by sending NOBIDS

$tools/rtb-commander             
RTB4FREE Commander
(1=Echo, 2=Load JSON Database into Aerospike, 3=Delete Campaign from Aerospike,
4=Stop Campaign, 5=Start Campaign, 6=Start Bidder, 7=Stop Bidder,
8=Exit Commander)
??
Which bidder to stop:
RTB4FREE Commander
(1=Echo, 2=Load JSON Database into Aerospike, 3=Delete Campaign from Aerospike,
4=Stop Campaign, 5=Start Campaign, 6=Start Bidder, 7=Stop Bidder,
8=Exit Commander)
??<<<<<{
  "cmd" : -1,
  "from" : "demo:ben-OptiPlex-780:8080",
  "to" : "7d9ad779-16c1-433c-b670-4c4876e984b6",
  "id" : "na",
  "msg" : "stopped",
  "status" : "ok",
  "type" : "status",
  "target" : "na"
}
??

HOW FAST IS THE BIDDER?

After you get your site running, you probably want to know how fast your bidder.

The actual number of bids per second is highly dependent on a number of factors. How much memory does your system have, how many cores, and what is your network performance?

There is a program in the tools directory to give you a raw number of queries per second. To use it start the RTB bidder in one window. Then in another window, run the following command:

$tools/maxload

This presumes your RTBbidder is running on rtb4free.com. If your bidder is on another host then do this:

$tools/maxload -host your-ip-address-or-hostname

By default 10 threads will be used to generate as much traffic as possible. If you have more cores, then up the threads. For example, to 16 threads to 192.168.1.100:

$tools/maxload -host 192.168.1.100 -threads 16

RTB4FREE is developed on an IBM Think Station (12 cores). On this system, also running X Window System and the Unity desktop, the bidder will sustain 6,500 bids per second.


BRAND YOUR RTB4FREE SITE

After you get your site running, you probably want to change the name of the System Console, Campaign Administrator, and the index.html pages to your own branding. You probably also want to change the tab icon from RTB4FREE's little green alien.

The index.html is located in ./XRTB/www

The Campaign Administrator (login.html), System Console (admin.html) and the Bid Simulator are located in ./XRTB/web.

All of the references to the brand name and the host address of the RTB4FREE system are located in: ./XRTB/www/SSI. This directory contains Server Side Includes where all the dependent references are made. The file names and what they do are located here:

  • bidloader.ssi - Identifies the ZeroMQ for monitoring bids. Set to rtb4free.com:7379
  • brand_icon.ssi - Identifies the icon to use on the browser tab. Set to images/alien.png (located in www/images)
  • brand_name.ssi - Identifies the brand of this DSP, set to 'RTB4FREE - JAVA Based RTB Bidder'
  • clickloader.ssi - Identifies the ZeroMQ for monitoring clicks. Set to rtb4free.com:7379
  • logger.ssi - Identifies the ZeroMQ for the log functions Set to rtb4free.com:7379
  • winloader.ssi
Of course database.json and Campaigns.payday.json are also set up to use rtb4free.com too, and they must be changed.

You can modify all these files, by habd, or you can use tools/config-website to do it for you. You need to determine the following information ahead of time:

  • The actual IP address of your bidder instance. For example, we will presume 192.168.1.100.
  • Your brand name is My-DSP for example purposes
  • You placed your browser icon i ./XRTB/www/images and overwrote alien.png
  • Your Aerospike server is also running on 192.168.1.100
  • We will presume your win, pixel, and click endpoints will be on this same IP address. If not modify these as appropriate in Campaigns/payday.json

Now we will configure the system by running the web configuration tool:

$tools/config-website -address 192.168.1.100 -brand My-DSP -aerospike 192.168.1.100:3000

SETTING UP SSL

If your SSP requires an SSL connection, you can set that up very easily. To do this, you will need a keystore file, the pasword, and the manager password for the keystore and then determine what port you want SSL to be available on.

The next step is set up the ssl object in the configuration file. For example, here is the setup for Campaigns/payday.json

"ssl": {
	"setKeyStorePath":"data/keystore.jks",
	"setKeyStorePassword": "password",
	"setKeyManagerPassword": "password"
},
					

Note, the keystore file is in the ./data directory, named "keystore.jks".

The port for SSL is not defined in the ssl object. Instead it is defined with the -x argument to the bidder start. For example. to start the bidder you might use the following command with:

$tools/rtb4free -x 8081					
					

This would set the HTTPS port to be 8081 (also, the default value). Remember, RTB4FREE standard HTTP port is 8080.

If you do not want SSL support, simply delete the ssl object in the configuration JSON file.

To not enable HTTP (only use HTTPS) use the -p to set the HTTP port to 0 and set the -x port. Example:

$tools/rtb4free -p 0 -x 8081
					

Obtaining a Free SSL Certificate

If you want to obtain a free SSL certificate for use with your RTB4FREE, check out letsencrypt. Here is an example of how to obtain a free SSL certificate using letencrypt. Basically, we run Nginx, on our server first, obtain the certificates, and then convert them to a java key store.

First get Nginx running on your system and obtain the certificate using the directions here.

Now copy the files from /etc/letsenctypt/{live/yourdomain}/to a convenient directory. From there do the following:

openssl pkcs12 -export -out keystore.pkcs12 -in fullchain.pem -inkey privkey.pem
keytool -importkeystore -srckeystore keystore.pkcs12 -srcstoretype PKCS12 -destkeystore keystore.jks	
					

Now copy the keystore.jks file to wherever your ssl directive says it will live. Don't forget to change the passwords as appropriate.