Research Case Study

During the early stages of The Daily Telegraph navigation redesign, a few concepts needed to be validated and compared against each other and the baseline (the existing nav).

To make sure we had a good representative sample that captured success rates as well as the time spent per task, I organised a quantitative, task-based remote test.

For this, I turned the designs for two new concepts into clickable prototypes, then set two separate task-based tests in Loop11 and divided the participant traffic equally between the tests using a redirect page.

All users were taken through the exact same tasks on the A, B and C (existing) designs. The C design test results became the baseline. The A and B concepts’ results were compared to each other, and also the baseline success rate and time spent per task.

The winning concept undertook an in-lab usability test with 5 desktop and 5 mobile users to catch any usability issues that could be fixed to further improve the solution.

As part of a redesign of all our Mastheads subscription page, we wanted to ensure it was easy for users to compare the value for money of our subscription offers. There was an assumption that the existing pricing was somehow confusing and that the benefits of each offer weren’t clear to the users.

We organised some qualitative remote usability testing that proved the assumption that the pricing was confusing. We were able to report the findings to the marketing team, supported by video snippets showing users struggling with the prices for about 10 minutes, with several even getting their calculators out to make sense of the pricing.

We also organised quantitative, task-based A/B testing to analyse the best possible layout for the offers. This involved us asking users to find a particular offer as we captured the fail vs success rates, as well as time spent per task on each design.

As a result of the tests we ran, we made recommendations and informed decisions on the “offer” layouts and "calls to action".

When The Daily Telegraph and The Australian first introduced a paywall we wanted to see what users’ initial reaction was about the breach message.

Knowing that users only spend about 5 seconds on a page before they make a decision to leave or stay on it, we created a 5 seconds test. We placed different breach messages in front of users for 5 seconds only and asked them to tell us what they remembered about the page, what options they had there and why they thought they might have been shown it.

The results we gathered helped us communicate to the marketing team the importance of the right messages, options and call to actions on the page.

How would scannability and findability be affected if we changed our headlines’ font from lowercase to uppercase on The Daily Telegraph and Herald Sun home pages?

To make sure we had a good representative sample that captured success rates as well as time spent per task, I organised for a quantitative, task-based remote A/B test.

The participants were divided between the two concepts and were given a set of tasks to check for scannability and readability of the two fonts.

We asked the users to find a story on a homepage that contained the word ‘strawberry’ and the word ‘watermelon’, capturing the fail rate and the time spent per task as users scanned for the words.

We then asked them to find a story about postnatal depression, capturing the fail rate and the time spent per task as users read through the story snippets. For this test the participants actually needed to read - not scan - the headlines, as the story snippet did not actually contain the words ‘postnatal depression’.

Testing with users recruited by an agency was proving a little inaccurate in previous testing sessions. To avoid a ‘pretend user’ and ‘professional testers’ scenario, I decided to build our very own research panels for The Australian, The Daily Telegraph and Herald Sun.

I created an invitation for our Daily Telegraph, The Australian and Herald Sun users to opt-in for research activities with us and placed it on our homepages.

Soon after we had accurate, pre-screened panels that we used for all sort of research activities. Having the panels ensured we captured accurate data from our real users and reduced the in-lab research cost by 50%

Building and using the panels involved working closely with the legal department to ensure we don’t breach privacy policy and comply with all the legal requirements while contacting our users, inviting them in and paying them for attendance. It also involved me in managing the panels, keeping them up to date and teaching the UX team members to wisely use the panels for their research activities.