Project Overview
SafeAds AI aims to empower Instagram users to report ads more easily and effectively by increasing awareness of the bias in Ins AI system and providing clear reporting channels. Through user research and design iterations, we seek to create a feature that promotes a safer and more user-centric advertising experience.
Team
2 Product Manager
2 UX Researcher
Timeline
Spring 2023
(3 months)
My role
UX Researcher
UX/UI Designer
Skills
Competitive Audits
Prototyping
User Interview
INITIAL PROJECT GOAL
How might we harness everyday users' power to detect harmful biases in algorithmic systems?
OVERVIEW
Process Introduction
Our project aims to explore the concept of 'everyday algorithm auditing' and its potential for empowering and incentivizing everyday users to detect harmful algorithmic biases in AI systems. We seek to evaluate whether and how users can assist AI/ML teams in addressing these issues. To accomplish this, we focus on the Instagram Ads algorithm system, as it is one of the most widely used social media platforms.
We introduced the ‘Uncertainty Model’ to frame the research and design process. The ‘Uncertainty Model’ is derived from Agile Methodologies - drives the project with the cycle of ‘research-design-evaluation’.
Uncertainty Model
My Contribution
I participated in the entire research process, from conducting user interviews and contextual inquiry to creating models, Crazy 8’s, storyboards, and speed dating. As the only UX designer in the team, I also design the solution from paper prototype to final design.
Final Solution
Empowering everyday users to report ads by increasing awareness about the bias in the Instagram algorithm system. Here is the final solution:
Pop-up ChatAI
Less options to choose
Instant feedback
RESEARCH
Background Research
We started our research process with a series of background research including experiential research, literature review, and academic informational research.
#1 Experiential Research & Informational Research
Some Screenshots of Experiential Research
Some Screenshots of Informational Research
#2 Data Analysis
Sources: UCRE-Twitter dataset, ImageRoulette, TwitterCropping and PortrialAI
Methods: Import the UCRE-Twitter dataset to Tableau, and then use the Tableau filter to delete bad data. After understanding the meaning of each column, we used Tableau to explore and analyze the data. Finally, we generated 5 data visualization and generated insights based on our discussion.
Data Analysis of Twitter
#3 Usability Testing
The purpose of this usability testing is to learn about how people react to algorithms in digital products. The digital product we used was Instagram, specifically ads. Users will be asked to complete three tasks related to algorithm auditing on Instagram Ads.
Here are three common themes across the observation & think-aloud:
Ignorance
Relevancy
Privacy
Participants appeared to be ignoring the Instagram ads that were displayed on their feeds.
Users who frequently engaged with content related to a specific topic were more likely to find the corresponding ads relevant.
Some participants expressed concerns about the use of their personal data for the targeting of Instagram ads.
NARROWING & REFRAMING RESEARCH GOALS
Activity #1 Walking the Wall
As a team, we thoroughly review all background research materials and engage in insightful discussion sessions to exchange and consolidate our collective insights.
Team Discussion Notes
Activity #2 Reverse Assumptions
Our team employed the reverse assumption method to challenge our preconceptions about the project's boundaries and uncover innovative ideas that we hadn't previously considered.
Reverse Assumptions
Activity #3 Define Our Project
We utilize affinity diagramming to analyze our ideas, identify areas of opportunity, and define the scope of our future research.
Define Our Project
RESEARCH
Contextual Research
We observe that most people ignore ads at the moment, so if we make ads more actionable, it may even attract more users to the platform. Our goal is to better understand:
1. What are the primary reasons why social media users ignore ads?
2. In what kind of situation people will love to interact with the ads?
Where Are We Now?
Directed Storytelling
We conducted 4 sessions of directed storytelling + semi-structured interviews with 8 frequent Instagram users. Having the participants lead the conversation and talk about their daily use of Instagram, we took recordings and organized interpretation notes.
Conducted Directed Storytelling
Interpretation Notes
We organized the interpretation notes (168 in total) into the affinity diagram and summarized key insights based on the diagram. The notes are categorized into 3 sessions: Ins functions, ads reporting, and user expectations.
Affinity Diagraming Interpretation Notes
SYNTHESIZE
Empathy Map & Journey Map
Based on the background research and affinity diagram, we decided to use the empathy map & journey map as a model to conclude our findings. We concluded 4 key insights from the analyzing and synthesizing steps.
Journey Map
Empathy Map
Key Insights
We generate three key insights after analyze
Insight #1
📝
Users believe that they must be told what exact action will be taken or procedures will be followed when they report an ad.
A professor who use Ins:
“There are too many steps and I need to look at every reason and choose one”
A graduate student:
“I just want to relax while I am using Instagram”
Insight #2
📝
Personalized ads can be effective but may lead to privacy concerns among users.
An undergraduate student:
“I feel they are tracking my viewing history in other websites”
Insight #3
📝
Users ignore ads because they use instagram for recreation or connecting others and viewing or reporting ads does not add to that.
A graduate student:
“I use Instagram for connection with my friend, not for reporting ads”
Instagram influencer:
“Skip ad is much easier than reporting the ad”
IDEATE & TEST
Walk the Wall & Crazy 8’s
Then each team member generated eight potential solution scenarios for various needs and shared them on our Miro board. We then labeled the needs we addressed to enable effective categorization and ranking.
Walk the Wall & My Crazy 8's Ideas
Storyboard & Speed Dating
We each used storyboards to ideate solutions based on specific user needs and conducted 8 speed-dating sessions for further investigation.
User Needs #3 Storyboards
Users need rewards
Users don’t want waste their time
Users don’t trust the reporting mechanic
Users tend to dislike features that do not provide any tangible rewards or benefits, as they are perceived as a waste of time.
Users generally do not appreciate features or activities that consume their time without providing any meaningful value or benefit.
The user shared their opinions that they don't believe the reported ads would be appropriately reviewed and action taken by Instagram based on the information provided in the storyboards.
SOLUTION
Paper Prototyping
Sketch & Paper Prototype
Lo-fi Digital Prototype
Initial Solution
Usability Testing
After conducting tests with 8 Instagram users, we identified two key problems. After iteration we successfully arrived at our final solution.
