Bidder Management

Last updated: Feb 3, 2017

Overview

The Bidder Management System Console is a web page that allows you to connect to a bidder, and from there, manage all the bidders connected to the same Aerospike instance.

Recall, that all bidders use an Aerospike database to share their context information

Operational control of the bidder is through the System Console,available at http://youripaddress:8080/admin.html. The login name is 'root' and the password is defined in the file Campaigns/payday.json with the attribute app.password

After you login to the Bidder Management system - as root, you can control all the bidders from that page. Once you login, you will be presented with a page that looks like this:

At the top of the screen you can see the green Save button, and the Red logout button.

You are presented with 6 tabs to choose from, for various operations. You start off in System Status tab. The tabs are:

  1. Users - Add and delete logins to the system
  2. Configuration - Change the startup file (for the bidder you are logged into.)
  3. System Status - The real-time status of the bidders
  4. Campaigns - Shows what campaigns are running on the bidder you are logged into
  5. Blacklist - Shows the current system wide blacklist, if any
  6. Exchanges - Shows the requests/bids/wins counts for all the exchanges connected to the bidders

System Status

The System Status tab is the most frequently used tab on the application. THere are four main sections: Network status, Operational status, Commands, and the console

Network Status

This status shows the bidders currently running and the current count of requests, bids, wins, etc. for that bidder From here you can get a feeling for what the current situation is on the bidder. Here is an example of the System status:

Each bidder that in the Aerospike cluster is shown as a row. The system you are logged into is at the top of the list. In this case 'ben-Thinkstation-S20:8080'. The other system in the cluster is 'piX-ben-pi3B:8080' (a Raspberry Pi). The columns have the individual performance items listed, and the last row is the total. Here is an explanation of the items:

  1. Total - The total number of HTTP requests sent.
  2. Request - The number of RTB requests received.
  3. Fraud - The number of RTB requests deemed fraudulent (requires Forensiq key)
  4. Bid - The number of bids made
  5. Nobid - THe number of no bids made.
  6. Win - The number of wins made
  7. Pixel - The number of pixel fires received.
  8. Click - The number of click through's received
  9. Errors - Number of openRTB parsing errors received
  10. %Bid - The percentage of bids (bid/requests)
  11. %Pixel - The percentage of pixels (pixel/win)
  12. %Click - The percentage of clicks received (click/win)
  13. %Fraud - The percentage of fraudulent requests (frau/requests)
  14. Ad Spend - The approximate ad spend, in dollars.
  15. QPS - The last measured queries per second (last minute)
  16. X-time(avg) - Last average X-time. X-time is the number of milliseconds to resolve a request to a campaign

Operational Status

This status shows the overview status of the bidders in the cluster as a row of systems like so:

the members of the cluster is shown as a row. You are logged into the top row. Here the two bidders are shown as to their running status, how many campaigns they have loaded, whether to show why they aren't bidding on a campaign, and a check box that is used for issuing commands. Here is an overview of the status information:

  1. Running - If the bidder is capable of serving bids, it is 'Running'. If this is false, it means the bidder is actually online, but will no-bid to all bid requests.
  2. Campaigns - This shows how many campaigns are loaded into the bidder
  3. Log Level - This shows the log level. 1-3 are operational levels. A negative number means log to stdout, normally it logs to 0MQ. Anything over level 3 degrades performance.
  4. Show No Bid Reason - This will emit the reason why a campaign does not bid on each request. Seriously degrades performance, used for debugging
  5. Select - Check this to issue a command to this particular bidder (next sub-section).

Commands

This is a row of buttons that allows you to send a command to the bidder (selected from above). Below is the list of buttons:

Here are the functions:

  1. Stop Bidder - Will place the bidder into 'no bid' mode for every request.
  2. Start Bidder - Will start a previously stopped bidder.
  3. Change Log Level - Will allow you to change the log to a different level.
  4. Change No Bid Reason - Turns on/off the Print No Bid Reason on the selected bidders.

Console

This section shows a running log of the messages for the bidder you are logged into. The most recent log messages are at the top:

The Heartbeat message is the most useful message and shows the various operational items about the running bidder. Here is an overview of the fields:

  1. openfiles - How many file descriptors the bidder has open.
  2. cpu - The relative load on the cpu.
  3. mem - How much virtual memory the bidder is using and the percentage of the total
  4. freedsk - How much relative free disk is on the system.
  5. low-on-threads - Is JETTY running low on network threads?
  6. qps - Last measured queries per second.
  7. avgBidTime - Measures number of ms to make a bid
  8. avgForensic - Measures vgg number of ms to make a call to Forensiq
  9. wins - Running count of wins
  10. pixels - Running count of pixels
  11. clicks - Running count of clicks
  12. exchanges - A map of wins. bids, requests by exchanges.
  13. stopped - Whether the bidder is in no-bid mode.
  14. campaigns - Current number of campaigns.

Users

The User's tab allows you to update the authorized users of the system (except for root, which is done by hand before loading Aerospike). Clicking the User tab will show a list of users like below:

To add a user, simply fill in the top line, and then press Add link to the far right.

Deleting a user is accomplished by pressing the Del link to the far right of the row.

To modify a user, simply change the value in the edit field for that user.

When done, press the green Save button in the title field.

Configuration

The Configuration tab allows you to set start parameters for the bidder you are currently logged into. Normally, this file is ./Campaigns/payday.json. When you save the new configuration, it will be called Campaigns/payday.json.new. In order for the changes to take effect you will have to restart the bidder. Below are the sections of the file you can edit:

  1. Seats - These are the entry points into the bidder where your SSP ('exchange') connects.
  2. Symbolic Links - These are lists, hashmaps, and CIDR lists stored in either the bidders' memory or in Aerospike and REDIS or REDIS compatible caches. These lists can be used symbolically in campaign rules to deal with large and unwieldy data sets. can use with your campaigns
  3. Forensiq - If you have a Forensiq account. here is where you set up your API key and your threshholds for testing fraud.
  4. SSL - If you are using SSL with RTB4FREE, here is where you tell the bidder the location of the keystore and passwords of your SSL certificate.
  5. Application - This section allows you to control the applications behavior, such as cache TTL values, pixel, win, and click locations.
  6. Logging - Here you can set the verbosity of the log, and whether or not to print why the bidder doesn't bid on requests.
  7. Geotags - Here you can set up the States and Zipcodes files for augmenting the RTB request.
  8. Aerospike - This section allows you to set up the Aerospike connection for this bidder.
  9. ZeroMQ - Here you can define the logging behavior of requests, bids, wins, clicks, pixels., and status data.
  10. Initial Campaigns - With this section you can define which campaigns are automatically started when the bidder begins execution.
  11. Templates - If your exchange requires non standard ADMs, you can define them here.

Seats

The Seats list defines seat-ids used for each of the exchanges you are bidding on. The seat-id is assigned by the exchange - it's how they know whom is bidding. The name attribute defines the name of the exchange, as it will appear in all the logs. The id is the actual id name the bidder sends to the exchange as the seat id - how the exchange knows who you are. The bid attribute tells the bidder where the JAVA class is for that exchange.

The E-Key and I-Key fields are crypto keys supplied by Google for use with the Adx Exchange.

If you want to connect to an SSP that RTB4FREE does not already have configured, follow these steps:

  • Is the SSP using vanilla openRTB? If so follow these directions:
    1. Determine the name you will use in logging requests, usually the exchange name. Consider we are defining a seat for Waardx. We would likely use waardx for the name.
    2. Determine the endpoint. For example consider the SSP Waardx, the endpoint would be /rtb/bids/waardx
    3. We will use the Generic driver so the Java Classname is com.xrtb.exchanges.Generic.
    4. Determine if the SSP is using an HTML encoded ADM string. If it is, append &usesEncodedAdm to the class name, otherwise don't append it.
    5. Add a seat-id, make it unique if the SSP doesn't provide you with one. Here, we come up with waardx-id<

    Here's an example definition for Waardx:

    Notice that all of the Bid Endpoints are using the same driver, but only Waardx is using the encoded ADM flag. The default is not to do encoded ADMs.

    Another useful option is &rlog. With this you can override the percentage logging defined in requestlogstrategy. If your percentage was set at 10%, but you wanted to keep 100% of the requests from this exhange use the form &rlog=100. For example, log 100% of waardx and set usesEncodedAdm the Bid Endpoint field would be /rtb/bids/waardx.Generic&rlog=100&usesEncodedAdm

  • Is the SSP using a non standard openRTB or extending openRTB, then you will need to write your own class and extend com.xrtb.pojo.BidRequest.java. You can use one of the other predefined classes like com.xrtb.exchanges.Atomx.java to implement your non-standard or extended SSP. Add special processing to the overidden method parseSpecial() in your class to add your secret sauce. Look at com.xrtb.exchanges.Stroeb for an example where an extension is added to the Bid Response.
  • If your SSP is a radical departure from openRTB, look at the package com.xrtb.exchanges.adx for an example of how Google (tm) Adx is implemented.
  • Don't forget to define a template for your new SSP endpoint, if the default does not suffice.

In the example below you can see the definition of all the exchanges that RTB4FREE supports by directly using a class that defines the SSP endpoint specifically..

To delete an exchange, depress the Del link to the right of the exchange's definition.

To add an exchange fill out the fields and press the Add link.

List Symbolics

This section defines the use of lists, sets and key-value pairs loaded in the bidder for use in Constraints defined in the Campaign Manager. In the constraint field you can use the value of the RTB bid request as the key for a lookup in the bidder's memory. Then the value returned is used in place of the value of the bid request.

There are three places that these symbols can be loaded: the bidder, Aerospike, and REDIS or REDIS-compatible key/value stores. They can call be used together. To reference the bidders memory use '@' as the first character of the symbolic name. To reference an Aerospike object use '#' as the first character of the symbolic name. For REDIS, use the '$' character as the beginning of the symbolic name.

The following describes the objects available using these three kinds of symbols:

  1. Bidder's Heapspace.

    You can load specialized objects into the bidder's memory in the List section. The following specialized objects are available:

    • NavMap - Defines a NavigableMap in JAVA. This is used specifically for CIDR lists of the form of the familiar CIDR IP address and mask: e.g. 34.34.0.0/10; Also you can specify a range 34.34.1.0-34.34.99.255. When querying the CIDR object with an IP address (in dotted form) it returns TRUE if in a range specified in the NavMap, or false if it is not within the range. Note, NavMaps are not supported in Aerospike.

      As an example, consider the example where a CIDR list, stored as a Navigable Map in the bidder's memory. To target a device whose address is in within the CIDR ranges in the map called BigMap you would use:

      request.device.ip MEMBER_OF @BigMap
    • Membership - Defines a Set in JAVA. This is simply a list. You can test membership in the bidders memory by the symbol name. For example, to test that a user.id is in the Set MyList use:
      request.user.id MEMBER_OF @MyList
    • AdxGeoCodes - This defines a CSV file that maps ISO region codes to a region, it's 2 char and 3 char ISO country codes. The CSV file contains the code, the name, country_and, name, isocode, iso2 code, type, mode, and iso3 name. It is used strictly for Adx exchanges. Querying this object will return an object of the form:
      1. object.code
      2. object.name
      3. object.country_and_name
      4. object.isocode
      5. object.iso2
      6. object.type
      7. object.mode
      8. object.iso3
    • LookingGlass - This is a straight JavaHashMap. It is a CSV file, the key will be the first entry, the object returned is a list of the other items, with the key being list[0]. In the example, you query are querying @ZIPCODES it by the first entry in the CSV list, which is the Zipcode and it returns an object that is an array with the lat, lon, city, state and county.
  2. REDIS and REDIS-Compatible Key/Value Stores

    You can reference objects in a REDIS cache if you have a REDIS connection defined in the REDIS sub-section of the Application. The following data stores are available:

    1. Set

      You can load a Set into REDIS, and then refer to it by the symbolic name such as $setname

      .

      The bidder considers the Set as addressable with a MEMBER or NOT_MEMBER operator. The values are all considered strings. For example consider a set called $Wallmart that contains all the zipcodes within 5 miles of a Wallmart store. To target a zipcode that is in this set use:

      request.device.ip MEMBER_OF $Wallmart
    2. Key/Value

      You can refer to a Key/Value pair using any of the non-set related commands, such as EQUALS, NOT_EQUALS, etc. You can also use the REDIS get, incr and decr operations. For example, if users are being counted using a key, based on the cookie id, you can use the form:

      $$incr:request.user.id LESS_THAN 3

      The first $ makes this a REDIS operation, the $incr is the operation. The key is the value in request.user.id.

    3. HSet

      HSets can be accessed with any of the non-set related commands.

      As an example consider a set called Zipcodes, that uses the zipcode as a key, and returns the current temperature. So to target zipcodes where the temperature is less than 70 degrees you could use the rule:

      $Zipcodes:request.device.zipcode LESS_THAN 70

      The $ sign indicates this is the REDIS command, the Zipcodes: indicates which set is to be used, and the key is the value of request.device.zipcode. Note, if the key does not exist, then the operation returns False.

    4. Bloom Filter

      The special bloom filter is addressable with the MEMBER or NOT_MEMBER operations, only.

      For example, say you want to target only those users with cookies in the cookie filter called MyCookies use:

      request.user.id MEMBER_OF $MyCookies

Forensiq

The Forensiq section is where you set up your BOT fraud detection information. You must have a Forensiq account and an API key to use this feature. Below is a sample definition:

The Connections item defines the maximum number of connections to use to connect to Forensiq. If you define too few connections, the campaign manager blocks for this request until a channel opens up. Keep an eye on the average forensiq access time in the console's Heartbeat message to keep this as low as possible.

The Threshhold is a percentage at which you decide to no-bid. Forensiq will return a value between 0=100 which is there confidence level that the bid request is from a BOT. If the value is 65, then forensiq is 65% sure it is a BOT. So you decide what the percentage level you are willing to bid at.

The CK field defins the API key provided by Forensiq.

The Endpoint field identifies the endpoint Forensiq giave you to query for BOTs.

The Bid On Error field defines that if the Forensiq Endpoint returns an error, or connection is refused, should you bid anyway. Set to true if you want to bid on the Forensiq error.:

SSL

If you are using SSL with RTB4FREE, use the SSL section to identify where the keystore file is, the key password, and the management password. Example:

Application

In this section the application parameters for the bidder are set up, such as the endpoints for clicks, wins, and pixels. the cache TTL, number of JETTY threads, whether or not multi-bids per request is supported, and whether or not a deadman swtich is to be used. Example

is an explanation of the fields:

  • Deadman Switch - If set, then this key must be available in the Aerospike cache or the bidder will remain in a no-bid state.
  • Multi Bids - The default is one bid per request. If multiple campaigns could bid, only 1 is chosen to bid, at random. If multi bids is set to true, then if multiple bids could be done, then multiple bids could respond to a single request.
  • Jetty Threads - This sets the number of web access threads in the web server.
  • Stop On Load -
  • TTL - This denotes in minutes the shelf life of a bid, before it is deleted from the win resolution cache
  • Pixel Track URL - Denotes where the pixel tracking endpoint is.
  • Win URL - Denotes where the win processing url is.
  • Redirect URL -Denotes where the click redirect endpoint is.
  • Admin Port - Set to 0 when admin functions share the same port as the RTB requests (usually 8080). Set to a different port if you want admin port to reside somewhere else.
  • Adm SSL - Set to True if SSL is configured AND admin port is not on the same port as the RTB requests.
  • Password - This is the root password used for logging into the Admin Console.

Logging

This section handles the logging of RTB4FREE. There are 2 settings. Verbosity denotes a number from -5 to 5. The higher the absolute value of the Verbosity, the chattier the logs are. 3 is operational. Levels 4 and 5 are very chatty and cause degraded performance. Logging is set to either a file, or a ZeroMQ channel. The log will also log to STDOUT if the log level is set to a negative number.

The No Bid Reason flag when set to True will log why a campaign does not bid on a request. Set to False for operational use, as logging this information causes performance degradation. Example:

Geotags

In this section you can set up additional Geotag information to be injected into the RTB bid request, based on the device.geo.lat and device.geo.lon entries found in the RTB bid request. There are two files already set up with the information you need.

If you use these 2 files, then depending on if the lat/lon is in the bid request, a new object is paced into the bid request called 'rtb4free.geocode'. The rtb4free.geocode attributes are city, state, county and zipcode. All are strings. You can use these attributes in creating additional constraints for your campaigns in the Campaign Manager.. Below is how to set up Gettagging:

Aerospike

Aerospike is used for all the shared contexts needed between multiple bidders, and is a place where large key/value items are kept used frequently by the bidder (like IP frequency capping).

Set the three parameters needed to connect to the Aerospike server with this section. The Hostname defines the host where Aerospike resides. The Max Connections field states how many connections the Aerospike client in the bidder will allow. The default is 300, and works well as long as QPS on an individual bidder is below 10,000 QPS. The Port field defines the port Aerospike server is listening on. The default port is 3000. Example:

Zeromq

The "zeromq" object defines the logging system used by RTB4FREE. The bidchannel, winchannel, clicks,logger,responses, pixels, and requests fields are the Zeromq channels (or files names) for the various messages. Example:

The Win Channel sets forth the file or ZeroMQ port/topic used to log win objects. It can be in the form of either:

  • file://filespecification

    If this doem is ued, then the log is placed in the file specified by 'filespecification'. The file grows without bounds.

  • file://filespecification&time=n

    In this form the file is defined, but, and the n defines the integer number of minutes to log before creating a new log file. The log files are in the form filespec-YYYY-MM-DD-HH:MM

  • tcp://*port-number&wins

    In this form the TCP port for Zeromq is defined, along with its topic name.

  • http://hostname:port/wins

    In this form the wins will be send to an HTTP web server, e.g. Logstash, as a POST. Optionally, you can add the option &time=n which will log the requests every n milliseconds. Example: http://localhost:9999/wins&time=100 means batch up wins and send them every 100 milliseconds.

The Bid Channel field defines either the file, HTTP endpoint or the ZeroMQ port/topic used to log bids made by the bidder. The same format as the Win Channel is used here too.

The Command Response field defines either the file, HTTP endpoint or the ZeroMQ port/topic used to send command responses of the RTB4FREE bidder. The same format as the Win Channel is used here too.

The No Bid Channel defines the file, HTTP endpoint or the ZeroMQ port/topic used to send requests that the bidder did not respond to. The same format as the Win Channel is used here too.

The Requests Channel defines the file, HTTP endpoint or the ZeroMQ port/topic used to send RTB bid requests processed by the bidder. The same format as the Win Channel is used here too.

The Req Log Strategy defines the strategy for logging requests. It can be all, bids, and wins. meaning log all requests in the requests log as received, or, just those requests that were bid on, or, those requests that were won.

The Clicks Channel defines the file, HTTP or the ZeroMQ port/topic used to the clicks that the bidder received. The same format as the Win Channel is used here too.

The Pixels Channel defines the file, HTTP endpoint or the ZeroMQ port/topic used to send pixel fires that the bidder received. The same format as the Win Channel is used here too.

The Status Channel defines the file, HTTP endpoint or the ZeroMQ port/topic used to send the 1 minute status message the bidder generates. The same format as the Win Channel is used here too.

The Subscriber Hosts defines a list of addresses where commands can come from.

The Command Port denotes the TCP connection where commands are received by the bidder

Initial Campaigns

The "campaigns" object is an array of campaign objects that specify what campaigns to preload.

The "Campaign Owner Name" is the Campaign owner - the user login name, the "Campaign Ad Id" is the adid of the campaign to preload. In the Campaigns/payday.json file, for demo purposes there is one campaign pre-loaded for you called "ben:payday", from the campaigns owner "ben". Note, the id field field accepts JAVA regular expressions. In the example the campaign that matches 'ben:payday' is loaded. To load all campaigns use '(.*). To load only campaigns prefixed with 'ben', then use 'ben(.*)'.

To add a new campaign to preload, simply fill in the 2 fields and press the Add link. To remove a campaign from the preload list, press the Del link by the campaign to remove.

Template

The Template section is where the ADM field of the RTB bid is constructed, based on the values in the bid request and the campaign/creative that is actually going to bid. It is a constructed using RTB4FREE macro definitions.

For those SSP's (Exchanges) that are listed below, you should not need to modify any of the parameters.

If you do modify the creatives, you are basically specifying the values in the ADM field through the RTB4FREE macros. You can find the definition of the macros here.

The SMAATO directive is different however. In the SMAATO template the components of the bid response is an XML string. The JAVA class that handles the SMAATO exchange will format the return values using the items specified as JavaScript. In the example below you can see the JavaScript as: richMediaBeacon='%%smaato_ct_url%%'; script='{creative_forward_url}'; clickurl='{redirect_url}/exchange={pub}/{ad_id}/creative_id={creative_id}/price=${AUCTION_PRICE}/lat={lat}/lon={lon}/bid_id={bid_id}?url={creative_forward_url}'; imageurl='{creative_image_url}'; pixelurl='{pixel_url}/exchange={pub}/ad_id={ad_id}/creative_id={creative_id}/{bid_id}/price=${AUCTION_PRICE}/lat={lat}/lon={lon}/bid_id={bid_id}';

The JavaScript objects needed by the SMAATO class is richMediaBeacon, script, imageUrl and clickUrl. It is not\ recommended you modify these otherwise your ad may not work properly.

For the most part, the exchanges are defined just using {creative_forward_url}. This macro points to the definition of your ad as specified in the Creatives section of the Campaign Manager. You can also use macros inside the Creative definition of your ad. Below is an example of the setup.

If you need to edit a template definition, just change the appropriate entry.

To delete a template definition, press the Del link to the right of the entry.

To add a template, simply in the top line, define both the Exchange and the Template field, and press the Add link to the right.

Note, if you define a Seat, but you don't define a template, Default will be be used.

Campaigns

This tab will show you all the campaigns and their status, on the bidder that you are currently logged into. You can also find out why your campaigns/creatives are not bidding. The page is broken into two tabs. The first tab shows the campaigns loaded into the Aerospike server, the campaigns loaded into the bidder (green) and those campaigns that could be started (grey). The second tab is a tree control that shows the breakdown of reasons why the exchange/campaign/creatives are not bidding on the bid requests coming in on that exchange. Below is an example:



Control

In the example above there are four campaign loaded into the bidder, for user 'ben'. If the buttons grayed out, then the campaigns available to run (Loaded into Aerospike), but are not loaded into the bidder. The green buttons show loaded/running campaigns.

To stop a campaign, press the green button. It will unload from the bidder and it's button will turn grey.

To start a campaign, press the grey button. It will load into the bidder and turn green.

No Bid Reasons

Frequently you will start a campaign, but for some reason you aren't bidding like you think you should be. The second tab "Campaign Performance by Exchange" provides a handy tree control that will help you determine why your campaigns aren't bidding. Consider the following example:

In the tab "Campaign Performance by Exchange" we see that the exchange republer has received 1,437 requests but has not placed any bids. Let's find out why the no-bids by clicking on the control republer. The tree control opens up to the next level showing 4 campaigns running, none of which are bidding on this exchange:

Now selecting and clicking on Campaign 46, we see there are 3 creatives in this campaign: 115, 116 and 117. Also, we see Global, which relates to global constraints of Campaign 46, and applied to all creatives:

We can see Creative 115 has been given 1,361 bids, but has not bid on anything. Selecting and clicking on Creative 115, we see the following reasons for not bidding:

Reasons for not bidding:

  • 79% of the requests considered, the bid price is below the bid floor.
  • 20% of the requests were of the wrong size.

In, general, the Global tree shows those constraints where the Campaign was not allowed to bid:

Further reasons for not bidding, applied to all campaigns is stored in Global. The total of 132 means that of the 3 creatives considered (115, 116, 117) 132 requests were satisfied, but the global constraints were not satisfied, resulting in no-nids.

  • 37% requests result in no-bid because the device.devicetype did not match the global constraint set for that Campaign.
  • 63% of the requests result in no-bid on because device.os did not match those specified in the campaign.

Blacklist

The blacklist is a system wide (stored in Aerospike) that is a list of domains for all the bidders to blacklist. The blacklist is loaded into Aerospike as set forth here.a This page shows a list of all the system-wide blacklist:

Blacklisted domains will not be visible to the campaigns, and as such, they do not need their own blacklist for these common entries.

Exchanges

This tab will show you the summary statistics for each exchange that each bidder in the cluster has processed RTB messages for. Each bidder in the cluster is shown, at the top of the row, followed by its \ exchanges. The row will skip a line and then the next bidder's information is shown. As seen below: