Catchpoint - Benchmark Feature

Prev Next

What Happens When We Talk About Benchmark Feature


Website-Side Behaviour

image.png

  • Benchmark endpoints load within an iframe and initiate requests after the page’s onload event.
  • Requests are delayed by approximately 1 second post-onload to avoid impacting user experience.
  • Requests are made sequentially, not in parallel.
  • Currently, we test across 7 CDNs:
    • Amazon
    • Azure
    • CDN1 (Akamai)
    • Fastly
    • Cloudflare
    • Google
    • CDN Networks

image.png


🌐 CDN-Side Configuration

Benchmark tests use specific URLs managed by Catchpoint to evaluate CDN performance.

πŸ”Ή 1 Packet Object URLs

These are lightweight requests used to measure latency:

https://lpulse.perflib.com/ipulse/ad/cp.com/creative/1p.gif
https://kpulse.perflib.com/ipulse/ad/cp.com/creative/1p.gif
https://jpulse.perflib.com/ipulse/ad/cp.com/creative/1p.gif
https://apulse.perflib.com/ipulse/ad/cp.com/creative/1p.gif
https://xpulse.perflib.com/ipulse/ad/cp.com/creative/1p.gif
https://cpulse.perflib.com/ipulse/ad/cp.com/creative/1p.gif
https://ctchpnt.akamaized.net/ipulse/ad/cp.com/creative/1p.gif

πŸ”Ή 100k Packet Object URLs

Used to measure throughput and larger payload handling:

https://lpulse.perflib.com/ipulse/ad/cp.com/creative/100.jpg
https://kpulse.perflib.com/ipulse/ad/cp.com/creative/100.jpg
https://jpulse.perflib.com/ipulse/ad/cp.com/creative/100.jpg
https://apulse.perflib.com/ipulse/ad/cp.com/creative/100.jpg
https://xpulse.perflib.com/ipulse/ad/cp.com/creative/100.jpg
https://cpulse.perflib.com/ipulse/ad/cp.com/creative/100.jpg
https://ctchpnt.akamaized.net/ipulse/ad/cp.com/creative/100.jpg
  • They all are configured to go to the same origin server, and in all of them we try to do the following:

    • Cache the assets as long as possible through HTTP headers, and whatever cache settings each CDN has
    • Enable serve from cache, and do not validate with origin where possible
    • Enable specific HTTP headers to allow our JavaScript to gather data

βš™οΈ CDN Configuration Goals

  • Maximise caching via HTTP headers and CDN-specific settings.
  • Serve from cache without origin validation where possible.
  • Enable HTTP headers to support JavaScript-based data collection.

⚠️ Note: Despite these configurations, there's a ~0.2% to 0.4% chance that a CDN edge server may not have the asset cached and will call the origin. To maintain benchmark integrity, such cases are excluded from results.


πŸš€ Recent Benchmark Feature Updates

  • Option to deploy benchmarks without Catchpoint RUM tag.
  • Added Throughput as a new metric.
  • Added ability to report HTTP Version.
  • Improved test execution and reporting to minimise user impact.

πŸ”’ Private Benchmarks

image.png

Private benchmarks allow you to run custom tests from real user devices, comparing performance across CDNs and origin servers.

Benefits:

  • Evaluate delivery infrastructure from real-world user perspectives.
  • Identify regional performance differences.
  • Determine the best-performing CDN or origin strategy.

Private benchmark results are excluded from the public dataset.