This is the Data Management Platform (DMP), the third layer of the DSP. Layer 1 is the Bidder, Layer 2 is the Campaign Manager and Layer 3 is the Data Management Platform. Features at a glance:
Because this is a Docker deployment you must have a working knowledge of Docker. You need Docker and Docker docker-compose installed. For information on Docker , look here.
All bidder logs are sent to Kafka, as shown in the following diagram.
The system configuration depends on many parameters, including
The RTB4FREE architucture can scale to support high data processing rates by horizontally scaling any of the following components.
The initial Kibana view is Discover.
The RTB log records can be searched by selecting the corresponding index from the dropdown list, then entering a Lucene query in the seach input box. The following record types can be selected.
Kibana contains a report builder that let's you create various display widgets, such as charts, graphs, maps, heat maps, tag clouds, etc. A sample time series graph showing request rate over time is shown below.
You can combine various visualizations into a dashboard. This dashboard shows bidder transaction activity as well as bidder performance metrics in a single view. Any data in the logs can be visualized with these tools.
Data logged into Elasticsearch can be made available to users of the Campaign Manger. The following Campaign Manager view shows each campaign's usage using data extracted from Elasticsearch.
If you have special processing needs, you can build you own app to consume and process the data. An easy method is to build your own Kafka consumer and subscribe to the data topics you wish to process.
The open source secor project let's you read any of the RTB logs and store them in cloud storage, such as Amazon S3 or Google Cloud Storage. This option offers unlimited storage without worrying about disk space. You can then download the data for offline data analysis using tools like Hadoop.
If you need to process analytics in real time, a custom Kafka consumer can read the log stream and perform real-time analysis and store in your own database.
The RTB4FREE source code for all the services is located here.