Why may ‘control’ be outperforming the new version?
After reviewing the failed experiment, Product outlined several hypotheses for why the redesign may have underperformed. These assumptions helped shape where I focused my research and validation efforts:
Language Relevance: The terms “rent” and “rental” are underutilized in the treatment, potentially reducing alignment with user intent to lease.
Hero Imagery: Visuals may trigger less effective emotional responses, impacting engagement. Unconscious emotions trigger different behaviors.
Search Priority on Mobile: The control version prioritizes search functionality, resulting in higher user interaction. On mobile treatment, the search is not as prevalent.
Component Positioning: Layout may not guide users effectively through the “find a home” journey. Treatment emphasizes content heavily vs. functionality.
Mobile Optimization: Dense copy and oversized imagery reduce usability and visibility on smaller screens.
Accessibility: Coral text on white background fails AAA compliance, potentially impacting readability.
Page Performance
During this phase, engineering flagged a major performance issue tied to page load speed. The homepage hero image was significantly slowing load times because the asset alone was 12.6 MB.
This was especially concerning on mobile, where the majority of our traffic was coming from.
Engineering recommended converting the image from PNG to WebP format and reducing the file size to under 200 KB.
Scroll Depth
While reviewing FullStory heatmaps, I noticed mobile users (who represented the majority of our traffic) weren’t scrolling very far down the homepage. This immediately raised questions around content prioritization and screen real estate.
With limited scroll depth on mobile, shortening the affordable housing banner became an opportunity to help users reach key conversion CTAs sooner.
Product also noted that the treatment experience showed slightly lower average scroll depth than the control, potentially because users were engaging with top features more.
Prioritizing search visibility on mobile
I noticed in the original redesign that the search bar was pushed below the hero image which draws attention away from it — exactly what we don't want when over 80% of users are navigating the website on their mobile device. I planned to reposition the search higher within the hero experience to keep the primary action front and center.
Validating the homepage through Maze testing
Since search was the most important action on the homepage, and often a user’s first impression of the brand, I wanted additional validation before recommending another relaunch. I used Maze to quickly gather qualitative feedback on the redesigned mobile homepage and to better understand how users were interacting with the updated search experience.
The test included 54 participants and focused on three areas:
10-second impression test
Users were shown the homepage for 10 seconds and asked what they believed the website was used for.
58% correctly identified the website as a home rental platform, while 31% assumed it was for apartment rentals — revealing an opportunity to better clarify Progress Residential’s offering through messaging and imagery.
A/B preference test
Users compared the original homepage against the redesign and selected which search experience they preferred.
69% preferred the redesigned homepage, describing it as cleaner, easier to read, and more focused on search. Users who preferred the original experience appreciated its filtering options and more direct “Find My Home” CTA.
Prototype task
Users were asked to search for available homes in Dallas, TX.
96% successfully completed the task using the search bar, reinforcing that users naturally gravitated toward the primary action we wanted them to take.
Key Takeaway
While the original homepage performed better in the live experiment, Maze testing revealed that users overwhelmingly preferred the redesigned search experience.