Testing Overview

Using two WordPress images, we performed a series of five speed and stability tests. These being:

1. TTFB (Time to First Byte)
   1. Best of 3
   2. Average of 3
2. DOM Content Loaded
   1. Best of 5
   2. Average of 5
3. Max Parallel Clients
4. Load Test
5. Soak Test

The definition and results of each test are outlined below.

Scoring & Ranking

Our goal was to rank each host against one another. Within each test, hosts were ranked on their performance, or given a rank between 1 and 5. Using pairwise ranking, 1st place awarded 5 points, 2nd place awarded 4 points, etc.

Pairwise Scoring Outline

5 participants:

1. First Place = 5
2. Second Place = 4
3. Third Place = 3
4. Fourth Place = 2
5. Fifth Place = 1

WordPress Images

We tested two versions of a WordPress site. One, the current, standard install or image – basically what you get when you install WordPress for the first time. Although the site looks pretty empty, it’s still running the all important WordPress processes in the background. Our second WordPress image was a small, one-page site with a couple images and paragraphs of text, running the Salient Theme and it’s required plugins. See the WordPress images section for a brief, visual overview of each site.

Test A: Standard WordPress Image

April 20, 2021

• WordPress 5.7
• Theme Twenty Twenty-One version 1.2
• No Plugins

Test B: Custom WordPress Image

April 22, 2021

• WordPress 5.7.1
• Theme Salient 13.05
• Plugins
  • Defender Pro
  • Salient Core
  • Salient Demo Importer
  • Salient Home Slider
  • Salient Nectar Slider
  • Salient Portfolio
  • Salient Shortcodes
  • Salient Social
  • Salient Widgets
  • Salient WPBakery Page Builder
  • Smush Pro
  • WPMU DEV Dashboard

Test #1: TTFB (Time to First Byte)

Tests

• TTFB – Best of 3
• TTFB – Average of 3

Testing Site:

• https://tools.keycdn.com/performance

Definition:
TTFB measures the duration from the user or client making an HTTP request to the first byte of the page being received by the client’s browser. This time is made up of the socket connection time, the time taken to send the HTTP request, and the time taken to get the first byte of the page. Although sometimes misunderstood as a post-DNS calculation, the original calculation of TTFB in networking always includes network latency in measuring the time it takes for a resource to begin loading.Often, a smaller (faster) TTFB size is seen as a benchmark of a well-configured server application. For example, a lower time to first byte could point to fewer dynamic calculations being performed by the webserver, although this is often due to caching at either the DNS, server, or application level. More commonly, a very low TTFB is observed with statically served web pages, while larger TTFB is often seen with larger, dynamic data requests being pulled from a database. Source: https://en.wikipedia.org/wiki/Time_to_first_byte

TTFB Example

TTFB Best Chart

In this test, each site was measured using the TTFB three times. This result represents the single fastest time from all three tests.

TTFB Average Chart

In this test, each site was measured using the TTFB three times. This result represents the averaged total response times of all locations.

Test #2: DOM Content Loaded

Tests

• DOM Content Loaded – Best of 5
• DOM Content Loaded – Average of 5

Definition
The DOMContentLoaded event is fired when the initial HTML document has been completely loaded and parsed, without waiting for stylesheets, images, and subframes to finish loading. A very different event load should be used only to detect a fully-loaded page. It is an incredibly common mistake to use load where DOMContentLoaded would be much more appropriate, so be cautious. Source: https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event

DOM Content Loaded Example

DOM Content Loaded Best Chart

DOM Content Loaded Average Chart

Test #3: Max Parallel Clients (Until Failure, Maximum 250)

Tests

• 1 – 250 client load over 1 min period, or until 50% failure rate.

Testing Site

• https://loader.io/tests

Definition
Server performance over a 1 minute period, when there is a constant build of 1 to 250 clients.
In computer science, the number of concurrent users (sometimes abbreviated CCU) for a resource in a location, with the location being a computing network or a single computer, refers to the total number of people simultaneously accessing or using the resource. The resource can, for example, be a computer program, a file, or the computer as a whole. Source: https://en.wikipedia.org/wiki/Concurrent_user

From WPMUDEV: Here’s a good analogy we like to use: “If you prefer to have a bar serving beer to 10 clients and then closing it down because the 11th started a fire, fine by us.” “We’d prefer a bar that serves 140 people in a timely manner. Even if it is a tad slower.”

Max Parallel Clients Example

Max Parallel Clients Chart

Test #4: Load

Testing Site

• https://k6.io/

Definition
Load Testing is a type of Performance Testing used to determine a system’s behavior under both normal and peak conditions. Load Testing is used to ensure that the application performs satisfactorily when many users access it at the same time. Source: https://k6.io/docs/test-types/load-testing/

Load Chart

Test #5: Soak

Testing Site

• https://k6.io/

Definition
While load testing is primarily concerned with performance assessment, and stress testing is concerned with system stability under extreme conditions, soak testing is concerned with reliability over a long time. The soak test uncovers performance and reliability issues stemming from a system being under pressure for an extended period. Reliability issues typically relate to bugs, memory leaks, insufficient storage quotas, incorrect configuration or infrastructure failures. Performance issues typically relate to incorrect database tuning, memory leaks, resource leaks or a large amount of data. With soak test you can simulate days worth of traffic in only a few hours. Source: https://k6.io/docs/test-types/soak-testing/

Soak Chart

Overview

And the winner is…(bap-ba-da-da)… Frankie Dog Hosting. We did not plan this. 

Frankie Dog Hosting Places

1. TTFB Best, Second Place
2. TTFB Average, Winner
3.  DOM Content Loaded Best, Winner
4.  DOM Content Loaded Average, Winner
5. Max Parallel Clients, Third Place
6. Load, Third Place
7. Soak, Fourth Place

Scoring Spreadsheet