Performance of RTB4FREE

In this blog post we show you the performance characteristics of RTB4FREE running on various Ubuntu 16.04 systems. In each case, we are using Oracle Java 1.8 with 4G allocated to the bidder, except in the case of the Raspberry PI and the Scrap computer, which both ran 1G RAM allocated ot the bidder.

Note, your mileage may vary!

Results

Here's a chart of RTB4FREE performance on various computers.

In general, RTB4FREE, Aerospike and the test program all run on the same machine. Which is not the most optimal setup. But as you will see, the performance is quite acceptable nonetheless. The campaign database has 2 campaigns with approcimately 10 total creatives. A bid request is used which is guaranteed to always produce a bid - this is a real test, we aren't seeing how fast we can process no bids.

Computer Queries Per Second Cost
AWS Micro Instance 300 (note-1) $.013/hour
Raspberry Pi 3 650 (Note-2) $39
10 Year Old Scrap (Dual core, 1.9Ghz, 2Gb RAM) 950 $29
Asus K53e Notebook 2,800 $200 (used) on Amazon
Dell Optiplex 755 3,500 $129 (used) on PC Liquidators
AWS mx4xLarge 7,500 $.23/hour
m4.10xLarge 11,000 $2.30/hour
IBM Thinkstation S20 12,000 $339 (used) on EBay
c4.8xLarge 32,000 $1.65/hour
Intel Xeon E5 2650v2 50,000 $8,000

Note 1. The microinstance runs 300 QPS for about 10 seconds, then performance drops off suddenly to 5-10 QPS. In general, Amazon will throttle the CPU on any microinstance that shows high CPU usage after 10 seconds. But hey, what do you expect for the free tier, or at $9.50/month after a year of free use. Still. Throttling is quite aggressive.

Note 2. The Raspberry Pi-3 cannot run Aerospike server, as it is a 32 bit machine. RTB4FREE can run without Aerospike in a single stand-alone configuration, which uses Cache2K for the database and cache. Note this test with the Raspberry was done while still running the X desktop applications. The network was also a WiFi connection pretty close to the the end of the range. Still, 650 QPS per second - and beats the performance of the AWS microinstance hands down.

Run The Simulation Yourself

Ok, so how does your hardware stack up? Here's how to run the benchmark.

  1. Install Aerospike and Load

    Ok, we presume you are running Ubuntu 16. So execute the following commands:

    $wget -O aerospike.tgz 'http://aerospike.com/download/server/latest/artifact/ubuntu16'
    $tar -xvf aerospike.tgz
    $sudo cd aerospike-server-community-*-ubuntu16*
    $sudo ./asinstall
    $sudo service aerospike start
    
  2. Download RTB4FREE and Build
    $git clone https://github.com/benmfaul/XRTB.git
    cd XRTB
    $mvn assembly:assembly -DdescriptorId=jar-with-dependencies  -Dmaven.test.skip=true
    
    
    
  3. Load the Aerospike Database With the Campaigns
    cd XRTB
    $tools/load-database
    
  4. Make RTB4FREE a Service And Start
    $cd XRTB
    $cp rtb4free.service /etc/systemd/system
    $sudo systemctl daemon-reload 
    $sudo service rtb4free start
    
  5. 
    
  6. Run The Simulation
    $cd XRTB
    $tools/maxload -threads 16
    

    The program will open threads (in this case 16 total) one per second and transmit as many bid requests as it can to the RTB4FREE bidder. Increase the thread count to see how increasing the load works on your system.