Using RTB4FREE With Adx (DoubleClick)

This article describes how to use RTB4FREE with Google (tm) Ad Exchange. This will show you how you can set up RTB4FREE to work with Adx using the Adx protocol based on PROTOBUF.

Bear in mind, Adx is NOT openRTB. This is a mapping of *some* of the Adx bid request attributes to an openRTB specification which allows RTB4FREE to use it's RTB campaign software to participate in the Adx bidding process. A couple of things to be aware of:

  1. IP Addresses - Google masks off the last octet of the IP address to ensure privacy.
  2. Requires SSL - Google Adx only does SSL. And anything you serve up from a third party site needs to be SSL too. Don't use a self signed certificate, it won't work.
  3. NO WIN NOTIFICATION - Google does not provide a direct WIN notification. You read that right... You have to encode your own win notification directly into your VAST or banner ad. If you don't handle this yourself then, you get no notification. However, we will show you how to do this. For banner ads, you can use the impression tracking url.
  4. Do Not Track - Doesn't exist in Google land. There is no mapping to the dnt attribute
  5. Encrypted location - Location data is encrypted to start with (we will decrypt it). We also add in city, state, county, and country information ad well.
  6. It's Not JSON - Adx is PROTOBUF based. We convert the protobuf and use our campaign software with it, then we build an Adx bid resposne and send it back to Adx.
  7. IMPORTANT - Do NOT serve the same RTB4FREE creative definitions with any other exchange besides Adx. Make sure your Campaign attributes array has an Adx restriction in it:
     "attributes" : [ {
            "value" : "adx",
            "op" : "EQUALS",
            "notPresentOk" : false,
            "bidRequestValues" : [ "exchange" ]
          } ]
    			

    Adx Bid Request attributes and values are radically different. Creative categories, vendor types, carrier ids, etc. do not match with openRTB. Keep your creatives separate!


Setup

Obtain your Adx Account

First, of course, you need a Google Adx account. Here's some background on Adx

Get a Real SSL Certificate

Second, you are going to need a REAL SSL certificate for your bidder. Adx ONLY does SSL. You can get a free certificate using LETSENCRYPT. Don't even try a self signed certificate. It won't work. Look here for letsencrypt with RTB4FREE

There is a python script that some people use to test their Adx server. It's here. It's also out of date, and does not work with SSL.

When you are ready to test, then execute it against Google's traffic. Since your account is in test mode in the beginning anyway, it won't cost you.

Get the E_KEY and I_KEY for Decrypting Google Adx Prices and Hyperlocal Data

Third, you need an Adx e_key and i_key. Once you have that, then you need to configure the adx endpoint in the seats object the configuration file (e.g. Campaigns/payday.json):

"seats": [
	{
		"name": "adx",
		"id": "adx-seat-id",
		"bid": "/rtb/bids/adx=com.xrtb.exchanges.adx.DoubleClick",
		"extension": {
			"e_key": "e_key in base64 goes here",
			"i_key": "i_key_in_base64 goes here"
		}
	},
	...
	

Then add the template for Adx in the templates part of the configuration file (e.g. Campaigns/payday.json):<.p>

"template": {
		"default": "",
		"exchange": {
			"adx": "{creative_forward_url}",
			"smartyads": "{creative_forward_url}",
			...
		}
	},	
	

Dealing with Absence of Win Notifications

Fourth - Now you have to deal with the fact Adx does not have a win notification.

You can create a custom pixel fire and get RTB4FREE to handle the win notification by formatting the way RTB4FREE processes wins.

You have to add the following win url to your banner ad or native ad. You can use the macros if RTB4FREE is serving up the ad. Otherwise you have to substitute the values yourself. Here's a version where RTB4FREE serves it up:

<img src="{win_url}/{exchange}/{ad_id}/{creative_id}/%%WINNING_PRICE%%/{lat}/{lon}/{bid_id}">
	

You can also put this into the adxTrackingUrl part of the creative definition as shown below (adxTrackingUrl is discussed in the next section:

adxTrackingUrl="{win_url}/{exchange}/%%WINNING_PRICE%%/{lat}/{lon}/{ad_id}/{creative_id}/{bid_id}",
	

If RTB4FREE will not be able to substitute it (it's buried in a third party site, for example), you have to do the substitutions yourself:

<img src="https://myrtb.com:8081/adx/YOUR_AD_ID/crOUR_CRID/%%WINNING_PRICE%%/0/0/NA"/>
	

If you handle the win notification yourself, you will have to decrypt the winning price yourself.

If you are doing a video ad, and you want to know about the WIN then you need to add an extra Impression object to your VAST like so:

<Impression> <![CDATA[{win_url}/{exchange}/%%WINNING_PRICE%%/{lat}/{lon}/{ad_id}/{creative_id}/{bid_id}]]> </Impression>

If you are not using RTB4FREE do serve the VAST tag, then you will need to encode the Win impression yourself.

The value of the macro {win_url} is defined in the RTB4FREE startup file (Campaigns/payday.json by default) in the app object as win_url. See below:

"app": {
	"password": "startrekisbetterthanstarwars",
	"stopped": false,
	"ttl": 300,
	"pixel-tracking-url": "https://yoururl.com:8081/pixel",
	"winurl": "https://yoururlcom:8081/rtb/win",
	"redirect-url": "https://yoururlcom:8081/redirect",
	...
	
	

Important Note: Make sure you set the winurl to use https! Otherwise Google will not serve the ad. Everything must be SSL. And note, in this example we presume you are starting RTB4FREE on the default SSL port of 8081. If you start RTB4FREE on a different SSL port (discussed below with the -x option) you need to set to that port number


You Need Some Extra Stuff In Your Creative

You will need to declare the vendor type that is serving the ad, the category of your ad, and the attributes of the creative.

In addition, your banner ad will need to include the %%CLICK_URL_ESC%% or %%CLICK_URL_UNESC%% Google macro. In addition, you will also need to provide the click through url.

Below you can see the adxCreativeExtensions in an Adx creative

{ "forwardurl" : "<script src='https://yoururl.com/app2.js' google='%%CLICK_URL_UNESC%%https://yoururl.com' data-width='300' data-height='250' data-sid='51376' data-appname='{app_name}' data-aid='{ifa}' data-idfa='{ifa}' data-lat='{lat}' data-lon='{lon}'></script>", "imageurl" : "", "impid" : "is-adx-banner", "w" : 300.0, "h" : 250.0, "attributes" : [ { "value": "adx", "op" : "EQUALS", "notPresentOk" : false, "bidRequestValues": [ "exchange" ] } ], "price" : 150000.0, "adm_override" : false, "adxCreativeExtensions" : { "adxVendorType" : 42, "adxCategory" : 19, "adxClickThroughUrl" : "https://yourclickthroughurlhere.com", "adxTrackingUrl": "{win_url}/{exchange}/%%WINNING_PRICE%%/{lat}/{lon}/{ad_id}/{creative_id}/{bid_id}", "attributes" : [ 21 ] }, }

In the example above, here are the fields of adxCreativeExtensions explained:

The example below is a complete example of a Campaign, one creative video and the other creative Banner, which demonstrates the Adx campaign:

"campaigns" : [ { "owner" : "ben", "adId" : "ben:adxtest", "adomain" : "originator.com", "attributes" : [ ], "creatives" : [ { "forwardurl" : ""<script src='https://yoururl.com/app2.js' google='%%CLICK_URL_UNESC%%https://yoururl.com' data-width='300' data-height='250' data-sid='51376' data-appname='{app_name}' data-aid='{ifa}' data-idfa='{ifa}' data-lat='{lat}' data-lon='{lon}'></script>"", "imageurl" : "", "impid" : "is-adx-banner", "w" : 300.0, "h" : 250.0, "attributes" : [ { "value" : "adx", "op" : "EQUALS", "notPresentOk" : false, "bidRequestValues" : [ "exchange" ] } ], "price" : 10000000, "adm_override" : false, "adxCreativeExtensions" : { "adxCategory" : 19, "adxVendorType" : 42, "adxTrackingUrl": "{win_url}/{exchange}/%%WINNING_PRICE%%/{lat}/{lon}/{ad_id}/{creative_id}/{bid_id}", "attributes" : [ 21 ] }, "capFrequency" : 0, "capTimeout" : "0" }, { "impid" : "is-adx-video", "w" : 300, "h" : 250, "attributes" : [ ], "adm" : [ "https://tag.youturl.com/vast?env=app&sid=51376&width={creative_ad_width}&height={creative_ad_height}&dnt={dnt}&ua={ua}&ip={ip}&idfa={ifa}&aid={ifa}&appstoreurl=[APP_STOREURL]&appname={app_name}&appversion=[APP_VER]&bundleid={app_bundle}&loclat={lat}&loclong={lon}&rnd={cachebuster}" ], "videoProtocol" : 2, "videoDuration" : 30, "videoLinearity" : 1, "videoMimeType" : "video/mp4", "price" : 10000000, "adm_override" : false, "adxCreativeExtensions" : { "adxClickThroughUrl" : "https://yoururl.com", "adxCategory" : 19, "adxVendorType" : 42, "attributes" : [ 21 ] }, "adm_override" : true, "capFrequency" : 0 } ], "date" : [ 20130205, 20200101 ] }

As a side note, notice that Adx Banner is created with the forwardurl attribute in the creative; however, the Video creative is defined as the first element of the adm[] array attribute.


Start RTB4FREE Using SSL

Fifth - Time to start RTB4FREE with SSL. RTB4FREE will use port 8081 as the default SSL port. If you want to change the port, then start rtb4free with the -x portnum option. Example, start it on port 9000

$tools/rtb4free -x 9000
	

Geolocation

Google geolocation is not like openRTB. First, the geo information in the native Adx bid request is encrypted. So RTB4FREE decrypts that for you using the e_key and i_key. But this just gets you to the latitude and longitude.

A postal code may be provided by the Adx request, and it will be mapped to device.geo.zip object in openRTB. However, there is no direct information from the request to identify the country, or the city. Instead, Google provides a geolocation identifier, Which is mapped in a CSV file provided here. Unfortunately, this cannot be used directly because Google uses the ISO-2 city designation, not the ISO-3 country codes that RTB does. So we provide this for you in a combined file called 'data/adxgeo.csv'. This provides City, Country and Zipcode information mapped to the Google geo identifier. Note that many of the geo codes in the file map to a 'Postal Code' not cities, byt a postal region

However we have another mapping file that will map US zipcodes to City, states, and county. It is located in 'data/zip_code_states.csv'. You will need to provide these 2 files to RTB4FREE to use this enhanced Geo capability. This will go into the "lists" section of the configuration file (.e.g. Campaigns/payday.json'). Here's what it should look like:

	"lists": [
		{ "filename": "data/adxgeo.csv", "name": "@ADXGEO", "type": "com.xrtb.exchanges.adx.AdxGeoCodes" },
		{ "filename": "data/zip_codes_states.csv", "name": "@ZIPCODES", "type": "com.xrtb.tools.LookingGlass" }
	],
	"ssl": {
		"setKeyStorePath":"data/keystore.jks",
		"setKeyStorePassword": "password",
		"setKeyManagerPassword": "password"	
	},	
	

If you change the location of the files or the file names, make sure you retain the "name" keys exactly as shown above.


Filter Mappings

There is not a one-to-one mapping of Adx objects to RTB objects. The table below sets out what attributes RTB4FREE has mapped. You can use these openRTB analogs in creating your campaigns.

Notes:

The bcat (blocked categories) and battr (blocked attributes) and allowed vendor extensions (allowedvendors) will be automatically filtered for you, you do not need to create any filters for these:

  1. battr - The attributes for the creative will be matched against the battr array from the Bid Request. If there is an intersection, RTB4FREE will send a NO BID to Adx
  2. allowedvendortype - The allowed vendor types declared in the Bid Request will be checked against the adxVendorType, and will bid only if the vendor type is found.
  3. bcat - The excluded product categories are matched against the adxCategory, and if there are no matches, then RTB4FREE will send the bid.

It may not be what you would like, and there may be some attributes you think should be added. Feel free to add them... These are just our best estimate.

openRTB Adx
openrtb.id AdxBidRequest.id
openrtb.bcat BidRequest.AdSlot.excluded_sensitive_category + BidRequest.AdSlot.excluded_product_category (Refer to enum ContentCategory.)
openrtb.imp[0].id AdxBidRequest.AdSlot.id
openrtb.imp[0].banner.battr AdxBidRequest.AdSlot.excluded_attributes
openrtb.imp[0].banner.pos AdxBidRequest.AdSlot.SlotVisibility
openrtb.imp[0].banner.h AdxBidRequest.AdSlot.height
openrtb.imp[0].banner.w AdxBidRequest.AdSlot.width
openrtb.imp[0].video.battr AdxBidRequest.AdSlot.excluded_attributes
openrtb.imp[0].video.h AdxBidRequest.AdSlot.height
openrtb.imp[0].video.w AdxBidRequest.AdSlot.width
openrtb.imp[0].video.pos AdxBidRequest.AdSlot.SlotVisibility
openrtb.imp[0].video.minduration AdxBidRequest.AdSlot.Video.min_duration
openrtb.imp[0].video.maxduration AdxBidRequest.AdSlot.Video.max_duration
openrtb.device.carrier AdxBidRequest.Mobile.carrier
openrtb.device.devicetype AdxBidRequest.Mobile.device_type
openrtb.device.device.geo.lat AdxBidRequest.[encrypted_]hyperlocal_set.center_point.latitude]
openrtb.device.device.geo.lon AdxBidRequest.[encrypted_]hyperlocal_set.center_point.longitude]
openrtb.device.device.geo.city AdxBidRequest.geo_criteria_id via geo-table.csv], See Appendix A for a link to the codes. (http://www.unece.org/cefact/locode/service/location.htm).
openrtb.device.device.geo.country AdxBidRequest.geo_criteria_id via geo-table.csv]
openrtb.device.device.geo.zip AdxBidRequest.postal_code
openrtb.device.device.h AdxBidRequest.Mobile.height
openrtb.device.device.w AdxBidRequest.Mobile.width
openrtb.device.ifa AdxBidRequest.Mobile.encrypted_advertising_id (decrypted and converted to string)
openrtb.device.language AdxBidRequest.language
openrtb.device.make AdxBidRequest.Mobile.brand
openrtb.device.model AdxBidRequest.Mobile.model
openrtb.device.os AdxBidRequest.Device.platform
openrtb.device.osv catenate AdxBidRequest.Device.os.versionMajor + AdxBidRequest.Device.os.versionMinor + AdxBidRequest.Device.versionMicro
openrtb.device.os AdxBidRequest.Device.platform
openrtb.app.name AdxBidRequest.Mobile.app.name
openrtb.app.bundle AdxBidRequest.Mobile.app.name
openrtb.app.id AdxBidRequest.Mobile.app.id
openrtb.app.content.url AdxBidRequest.url
openrtb.site.id AdxBidRequest.seller_network_id
openrtb.site.page AdxBidRequest.url

Macro Substitutions

The RTB4FREE macros are available for use with Adx, except for {dnt}. Do not track flag is not supported in Ad Exchange.

Look here for the list of supported macro substitutions.


Result Codes

Unlike openRTB, Adx will tell you the result of your bid. In subsequent bid requests, it will send you an array of bid ids, with the reason code of why it did not bid.

This is a very helpful feature. The logs/request file contains not only the bid request log records, but also, the feedback message.

Here is an example feedback message in the log file:

{"feedback":"587537e70a824fa7e5855651563c","code":10}
	

To take advantage of this information, find the bid with the id 587537e70a824fa7e5855651563c, then you know the campaign id and creative id. Then from there, look up the code using this list:

Looking at the example feedback, the code is 10. Which means: "Creative was not approved"

Modifying the Code

All the code for the Adx is located in the package com.exchanges.adx. The main class is AdxBidRequest.java. This is where the mapping from Adx to RTB occurs. Basically this is how it works:

  1. The Bid Request Comes in and the constructor is called with the InputStream. It's actually an InputStream of Protobuf, so it is parsed into an Google BidRequest object, named 'internal'.
  2. A JSON object is constructed to hold the RTB equivalent - called root. We query internal and make RTB objects in root.
  3. A static Map of lambda functions are stored using major keys of the RTB specification (like site, app, etc). These actually do the mapping. Modify the appropriate lambda in the 'Command' interface at the top of the class as you see fit, or, add add lambdas as you see fit to handle other objects.
  4. Finally, at the end of the constructor, a few odds and ends are mapped. Also, a base64 version of the protobuf is stored at the root.protobuf key.
  5. After the constructor completes, an openRTB representation is stored in 'root'.
  6. Since AdxBidRequest extends BidRequest, and relies on the standard JACKSON object named 'root', the rest of RTB4FREE is none the wiser it is dealing with Adx.
  7. If RTB4FREE decides to bid, it will call the buildNewBidResponse() method, which AdxBidRequest overrides to return the appropriate Protobuf version of the Bid Response