UX Project: Win/Loss Report
role: ui/ux LEAD designer
TOOL: SKETCH, BALSAMIQ, PHOTOSHOP
Win/Loss report is a L size 4-6 week design project. This project introduces a new type of report in T1.
Win rate is an important metric for users who want to monitor their campaign bid opportunities. Win rate can inform users on whether they need to increase or decrease their bids across a campaign or strategy.
Design Process Summary
- Product Manager holds a kick off meeting with UX Designer to discuss the business requirements and project goals
- UX Designer and Product Manager conduct user research with users
- Product Manager and UX Designer brainstorm on different ideas for the reporting chart
- Product Manager presents their early design ideas during the UI planning meeting to get feedback on technical feasibility
- Product Manager regroups with designer to focus on the design problem and sketch out simple concept for UX Designer to prototype
- UX Designer works on very lo-fidelity mockups
- UX Designer writes up questions for validating designs with users
- Product Manager schedules users for user testing I
- Product Manager and UX Designer sync up on designs and questions to finalize the mockups for testing
- Product Manager and UX Designer conduct user testing and record audio from their session
- UX Designer transcribes audio to identify user insights and area which need further design thinking and discussion
- Product Manager and UX Designer run a design session to discuss user researching findings and unresolved design decisions
- UX Designer works on prototypes for user testing II and writes questions, schedules and recruits users for testing
- UX Designer and Product Manager discuss the questions and finalize the designs for testing
- UX Designer conducts the user testing session
- UX Designer to finalize the design
- UX Designer uploads design deliverables in the ticket
- Product Manager to finalize product requirements and attends poker planning to push the ticket into a sprint
Product Manager brief on UX Designer of the business goal and design requirements. Our goal for this project is easily delivering different data includes targeting match, creative match, bid rate and win rate to the users, and allowing them to toggle between strategies. We have 4 views need to design: Campaign Daily and Summary view, Strategy Daily and Summary view
Research on Win Rate Report
After the kick-off meeting, UX Designer did more research on the different chart: bar chart, scatter chart and funnel: Examples found on across Google Image search containing the keyword 'Win Loss Reporting'. Most common uses were in finance, sports and managing sales/client prospecting efforts. Charts varied from bar, spark, lines.
Brainstorm with Product Manager
Inspired by the last two charts, we came up with few ideas:
- Campaign Daily: Line chart
- Campaign Summary:
- View by Bar chart: could show bid rates, but no average bid amounts.
- Scatter + New Funnel A
- Scatter + New Funnel B
- Scatter + Old Funnel(Left aligned)
- Strategy Daily(up to 30 days): stacked bar by each day instead of line chartwith rate on Y (right side)/ Line with two metrics
We sketched out four different charts to display the data for Campaign Summary page:
- Vertical stacked bar chart: User can compare the win rate between different strategies, but there is a space limit.
- Scatter chart + horizontal stacked bar chart/funnel chart: See the waterfall comparison for each strategy, but can't compare between different strategies
Low-Fidelity Mockups with Balsamiq
Discuss with Product Manager about the user testing questions and Product Manager set up user tests
Ask user to introduce themselves, remind them we are testing our designs and not testing them and ask for permission to record the session.
Show user blank M&M Screen
Q1: Where do you expect to find the win/loss report?
Observe: What will user select? Campaign or Strategy or Chart icon on campaign
Show user either report with A-Bars or B - Scatter
Q2: How would you interpret this graph?
Q3: How would you use these data points?
Observe: Listen for cues that they understand how to read graph, interpret % and tell us how they would change their campaign settings based on this data
Q4: Compare to the first chart, which one is easier for you to interpret?
Show user all the funnel graph options
Q5: Which visual is easier for you to interpret the results
Show user any report they saw and ask the following
Q6. How would you switch from daily/summary
Q7: If you want to make a change your campaign based on this chart, where will you go to edit it?
Q8: How do you feel being able to edit here?
We Set up with 8 users in one day, showing bar chart and scatter chart to the users alternatively.
6 out of 8 users think funnel/scatter chart is easier to read.
3 out of 6 users think funnel/scatter chart is useful after they saw the funnel chart(not the scatter one).
4 out of 8 users think they would like to compare between strategies.
6 out of 8 users got confused of the stacked bar.
8 out of 8 users didn’t see the summary/daily toggle until it got pointed out.
For funnel/Chart, users have their favorite: 1, 5 (they want to see the numbers)
The users think inline edit is nice to have, but not necessary.
Stacked Bar chart: Didn’t represent the waterfall relationship between the data points
Someone brought up a table chart idea.
UX Designer brought up a few design questions after discussed with Product Manager
- Bar chart
- good for compare at strategy level
- How to make the bar more clear to read?
- Group bar(horizontal funnel)
- Color and interaction
- Add Table chart
- Scatter + Funnel chart
- What is the default Highest dropdown?
- User’s preference?
- How to make users understand the funnel/scatter chart when there is no strategy selected?
- Make the text clear: “ Click a dot/Strategy to view the opportunity funnel”
- Show the funnel right next to the scatter, make the funnel grey out.
- New idea: Add a table chart, have a scatter, funnel, table slideshow
- How to show the strategy name in a scatter view?/How to find a specific strategy in a scatter view?
- Hover to show the name
- Search box
- Can we still compare between strategies in this view?
- What is the default Highest dropdown?
Design requirements after the meeting:
- User needs the ability to compare the strategies under one campaign
- User needs the ability to click in each strategy to see the win rate
Hi-Fi mockups: (Spacing and color)
- One screen to show scatter and table(color block match to the dots, maybe an icon/something to trigger the funnel)
- One screen to show funnel(Change colors, and change the arrows) and table
- One screen to show scatter and funnel
- scatter and table:
- When hover on a dot, the strategy in the table will be highlighted; When hover on the strategy in the table, the dot will be highlighted.
- When click a dot, trigger the funnel
- when click a strategy in the table, dot disappear.(?)
- Funnel and scatter:
- Click two dots, show different funnels.
- Reset button go back to the initial stage.
Second round of the user test
- Interaction with the scatter?
- Interaction with the table? (sort/click on the chart? Click on the whole bar?)
- Interaction with Funnel?
- Looking at the scatter graph, how would you interact with it?
- (If no one says they will click on the dot) When you click on the dot, what do you expect to happen?
- For who saw this graph, please don’t answer it. For whom haven’t seen this graph, please tell us how would you interpret this funnel?
- How would you interact with the table?
- (Point on the icon) What do you think these icons do?
- (If no one says) When you click on the row, what do you expect to happen?
- (If no one says) When you click on the chart icon, what do you expect to happen?
We rounded up with a group of 8 users, 30 mins questions and answers while we went through the clickable mockups.
- What would you do when you first come to this screen?
- 6 users will click a strategy dot
- 1 user will scroll down strategies & win rates box
- 1 user will click view by to select strategy/exchange
- Scatter view: What do you expect to see when you click on the dot?
- 8 users expect to view funnel/ details
- Funnel view: What do you expect to happen when you click on the toggle? Table icon
- 3 users have no idea
- 1 user expects to see different visualization
- 1 user expects to collapse funnel
- 1 user expects to see daily view report
- 1 user expects to recalculate the win rate through the funnel
- 1 user expects to see strategy metrics
- Table view: What do you expect to happen when you click on the chart icon?
- 4 users expect to see the same interaction as click the dot
- 2 users expect that the strategy dot is highlighted on the chart
- 2 users expect that the strategy data would populate on the chart
- Scatter view: What do you expect to happen if you click on another grey dot?
- 6 users expect that the view will switch to that strategy funnel
- 1 user expects that strategy is highlighted in lower MM screen
- 1 user expects that the strategy is highlighted on the right side chart
- Table view: What do you expect to happen if you click on the strategy row?
- 6 users expect to see the same interaction as dot
- 1 user expect to see dot highlighted on chart
- 1 user expect to see Max bid: strategy type
Overall, for the Funnel View, 8 of 8 users didn't know what to expect when they click on the toggle(table icon). So how do we make the toggle information more clear? Adding a tooltip?
After this new feature releases, this has been the second popular report in T1. Users love the chart and the interaction.