Having their own software for augmented reality allows Facebook to avoid dependency on Google for Android software updates
What is research?
What Research is ?
Research is simply systematic inquiry. You want to know more about a particular topic, so you go through a process to increase your knowledge. The type of process depends on who you are and what you need to know.
A lot of personal research these days begins with a Google query (“Who is Mihaly Csikszentmihalyi?”) and ends on a Wikipedia page. Finding information is relatively easy. The knowledge already exists. You just have to find a trustworthy source for it. Assessing credibility is the hard part.
Pure research is carried out to create new human knowledge, whether to uncover new facts or fundamental principles. The researcher or investigator wants to advance a particular field, such as neuroscience, by answering a particular question, such as “Why do humans sleep?” Pure research is based on observation or experimentation. The results are published in peer-reviewed journals. This is science. Rigorous standards and methodologies exist to preserve objectivity and ensure the credibility of conclusions. (Things get squishy when corporations fund ostensibly pure research, as they frequently do.)
Applied research borrows ideas and techniques from pure research to serve a specific real-world goal, such as creating a supersoldier or improving the quality of hospital care or finding new ways to market pork-flavored soda. While ethics are just as important, methods can be more relaxed. Maybe this means changing up the questions you ask partway through a study, or making the most of an imperfect sample group because you’re tight on time. The research is successful to the extent that it contributes to the stated goal. As with pure research, sometimes you accidentally discover something valuable you weren’t even looking for, and that’s a fantastic bonus.
And then there is design research.
is a broad term with a long history. In the 1960s, design research referred to the study of design itself, its purpose and processes. This is still how the term is used in academia today. There are various institutes of design research around the world, mostly involved in large existential or small theoretical questions couched in highly specialized academic language. If you’re interested in transformative concepts of spatial intelligence or the poetics of the sustainable kitchen, this field is for you. Read More
However, when practicing industrial or interactive design-ers refer to design research, they typically mean research that is integral to the design work itself—inquiries that are part of designing, not about design. This research focuses largely on understanding the people for whom we’re designing, often re-ferred to by the dehumanizing but instrumental term end users. Research is a core part of user-centered design.
Design research both inspires imagination and informs intuition through a variety of methods with related intents: to expose patterns underlying the rich reality of people’s behaviors and experiences, to explore reactions to probes and prototypes, and to shed light on the unknown through iterative hypothesis and experiment.
For a design to be successful, it must serve the needs and desires of actual humans. Strangely, simply being human is insufficient for understanding most of our fellows. That’s why Suri’s description could apply equally to the research extrater-restrials conduct on unsuspecting abductees (including their reactions to probes!). Design research requires us to approach familiar people and things as though they are unknown to us to see them clearly. We need to peel away our assumptions like a gray alien shedding its encounter suit.
Imagine you’re working with a well-established science and technology museum. Let’s call it the Fantastic Science Center. And this museum just received a grant for the vague purpose of improving its use of the web, which could mean anything from designing a new brochure website to creating interactive science education activities for remote students to developing mobile apps that complement the physical exhibits for visitors with smartphones. How do you prioritize alternatives and ensure the project succeeds? Throughout this book we’ll look at ways you can employ research techniques to ensure the museum (or any organization) makes the best use of your time and its resources.
Asking your own questions and knowing how to find the answers is a critical part of being a designer. If you rely on other people to set the agenda for inquiry, you might end up caught between fuzzy focus groups and an algorithm that scientifically chooses a drop shadow from among forty-one shades of blue. Discovering how and why people behave as they do and what opportunities that presents for your business or organization will open the way to more innovative and appropriate design solutions than asking how they feel or merely tweaking your current design based on analytics.
You will find that when you ask the hard questions, your job gets much easier. You will have stronger arguments, clarity of purpose, and the freedom to innovate that only comes with truly knowing your constraints.
What research is not
Research is not asking people what they like
As you start interviewing people involved in business and design decisions, you might hear them refer to what they do or don’t like. “Like” is not a part of the critical thinker’s vocabulary. On some level, we all want the things we do to be liked (particularly on Facebook), so it’s easy to treat likability as a leading success indicator. But the concept of “liking” is as subjective as it is empty. It is a superficial and self-reported mental state unmoored from any particular behavior. This means you can’t get any useful insights from any given individual reporting that they like or hate a particular thing. I like horses, but I’m not going to buy any online. Read More at
Quash all liking, and hating too. Plenty of people habitually engage in activities they claim to hate.
Research is not a political tool
Don’t let your methods be guided by a desire to appear smart or conform to anyone else’s picture of research. Some clients will argue for doing interviews in a usability lab even when it isn’t appropriate, just because it feels research-y. You’ll need to explain to them why interviews with method and purpose are more valuable than having a social conversation with a random person—and why it really doesn’t matter where you do them, as long as they’re done right.
In the best case, you can use the real-world facts and insights you gather to bring an external perspective to internal debates and power struggles that threaten your ability to get good work done. At the very least, it’s up to everyone participating in the research to hold the line and not let interpersonal dynamics influence your findings. Watch out for those who would use information gathering for political purposes or as a popularity contest.
In addition to executives who prefer the authoritative appearance of experimentation, you may run into sample-size queens who dispute the validity or utility of applied qualitative research. These people are often pollsters and marketers who run a lot of surveys. Avoid arguments about statistical significance; you will not win. Instead, keep the focus on gathering useful insights. Read More at
People who make design decisions at any level benefit from asking more and better questions. Many of them also need a little guidance on what to do with the answers. Within are ideas and techniques for you to use in making your projects and design solutions better and more successful. It is a sampler rather than a survey, and a biased sampler in that I have included only the topics and approaches I personally have found most useful in my design career. Most of these are what we do at Mule in the beginning of a client project.
It is also a pointed book, and that point will help you cut through the laziness, arrogance, and politics that prevent more research.
Research is just another name for critical thinking. With a little encouragement, everyone on your team can open their minds and embrace it. And together, we can fix it so no one contemplating a web project ever mentions focus groups again.
Research is a discipline with many applications. This chapter introduces the core practices and fundamental ideas and techniques you will use repeatedly in many situations. We’ll cover who should do research, different types of research and when to use them, and roles within each set of research activities. To help counter any skepticism about the business value of research, we’ll also review some common objections and how to overcome them.
Who should do research? Everyone!
Ideally, everyone who is on the design team should also partici-pate in the research.
If you are a sole practitioner, well, that’s easy. You will have excellent direct experience and can tailor the process and documentation to suit your needs. (Just be particularly mindful of your personal biases.) If you work with other people, involve them from the start. Presenting them with the world’s most stunning report will give them a terrific reference document, but it’s far less likely to inspire them to approach their work differently. (Do you disagree? Perhaps you are an economist.)
When you find yourself making a case for a skeuomorphic, bronze astrolabe interface based on the research you’ve all done together, you’ll be able to spend less time explaining the rationale and more time focused on the merit of the conclusion. “As you saw in the interviews, we found that our target group of amateur astronomers exclusively uses nineteenth-century equipment for stargazing….”
People who have a hand in collecting the insights will look for opportunities to apply them. Being the smart person is more fun than obeying the smart person, which is how the researcher/ designer dynamic can feel if designers are merely the recipients of the analysis.
At my first design agency job, the research director was a charming PhD anthropologist with a penchant for vivid, striped shirts. Despite being fresh out of academia, he was much more of a scout troop leader than a fusty professor. Interviews and us-ability tests were scavenger hunts and mysteries with real-world implications. Unlike heinous, contrived team-building activi-ties—rope courses and trust falls—doing research together actually did make our team more collaborative. We were learning interesting, valuable new things, and everyone had something different to contribute. The content strategist would notice the vocabulary real people used and the developer had good questions about personal technology habits. The visual designer was just really into motorcycles, and that helped sometimes too.
Someone needs to be the research lead—the person who keeps everyone on track and on protocol and takes ultimate responsibility for the quality of the work. If you take this on it might mean that you’re the primary researcher, gathering the data for others to help you analyze, or you could have more of an ensemble approach. The most important thing is that everyone involved knows the purpose or goal of the research, their role, and the process.
Find your purpose
One of our maxims at Mule is that every design project ulti-mately amounts to a series of decisions. What are the most important features? What is the best navigation scheme? How big should the logo be?
For any given project, we include only the research activities that support the specific decisions we anticipate. If the client has only identified an audience and wants to explore ways to better serve them (“What can we offer of value to high school science teachers?”), our research will be more open-ended than if the design problem is already well defined (“How can we get high school science teachers to download and use our lesson plans?”). Read more at
This has been playing out on the fields of “mobile first.” Many organizations are seeing a significant increase in their mobile traffic. They know they have to do something different for users on mobile devices, but aren’t quite sure what. So, they’re looking for ideas, or should be. It’s too soon to jump to fine-tuning solutions. For example, should the Fantastic Science Center, our fictional museum client, rewrite all of the exhibit descriptions for a mobile audience, or build a native event reservation app, or encourage school group students to post exhibit photos to Facebook from their phones? Organizational research will tell you which interactions benefit the museum most, while user research will indicate which are most plausible and the circumstances under which they will take place. Maybe you will discover that school district policy prohibits students from using their phones on field trips, but parents are likely to take photos of family visits to share with their Facebook friends. In that case, parents are the ones to target with a social media marketing campaign.
There are many, many ways of classifying research, depend-ing on who is doing the classification. Researchers are always thinking up more classifications. Academic classifications may be interesting in the abstract, but we care about utility, what helps get the job done. Research is a set of tools. We want to make sure we can find the right one fast, but we aren’t too con-cerned with the philosophy of how the toolbox is organized.
To choose the best research tool for your project, you’ll need to know what decisions are in play (the purpose) and what you’re asking about (the topic). Then you can find the best ways to gather background information, determine the project’s goals and requirements, understand the project’s current context, and evaluate potential solutions.
Generative or exploratory research: “What’s up with…?”
This is the research you do before you even know what you’re doing. It leads to ideas and helps define the problem. Don’t think of this as just the earliest research. Even if you’re working on an existing product or service, you might be looking for ideas for additional features or other enhancements, or new products you could bring to an audience you’re already serving.
Generative research can include interviews, field observation, and reviewing existing literature—plus feeling fancy about say-ing “generative research.”
Maybe the museum is trying to decide how to allocate that grant money and has discovered that a lot of parents who re-cently had their first child are coming to the website and you want to figure out what else you can offer them. Your question might be, “What’s up with new parents anyway?” Your goal would be to see the new parent experience from their eyes, to understand what they do and what they need. Your generative research activities might include interviewing new parents on the phone, following new parents around on a typical day, or looking at the questions new parents ask on social websites. Read More at
Once you’ve gathered information, the next step is to comb through it and determine the most commonly voiced unmet needs. This sort of research and analysis helps point out useful problems to solve. Your thinking might lead to a hypothesis, such as “Local parents of young children would value an app that offers ideas for science events and activities based on their child’s developmental milestones.” Then you can do further (descriptive) research on how parents recognize and commemorate those milestones.
Descriptive and explanatory: “What and how?”
Descriptive research involves observing and describing the char-acteristics of what you’re studying. This is what you do when you already have a design problem and you need to do your homework to fully understand the context to ensure that you design for the audience instead of yourself. While the activities can be very similar to generative research, descriptive research differs in the high-level question you’re asking. You’ve moved past “What is a good problem to solve?” to “What is the best way to solve the problem I’ve identified?”
At Mule, we’ve done a lot of design work for eye health orga-nizations. Despite the fact that several of us have really terrible vision (and very stylish glasses), none of us had any expertise beyond whether the chart looks sharper through lens number one or lens number two. The Glaucoma Research Foundation offered a clear design problem to solve: how to create useful, accurate educational materials for people who had been newly diagnosed with an eye disease. So, a round of descriptive re-search was in order.
To inform our design recommendations, we interviewed ophthalmologists and patients, and reviewed a large quantity of frankly horrifying literature. (Please, have your eyes examined regularly.) By understanding both the doctor and patient priori-ties and experiences, we were able to create online resources full of clear information that passed clinical muster and didn’t provoke anxiety.
For the Fantastic Science Center, descriptive research comes into play once we’ve identified a design problem, such as providing an online robotics course for students around the world. Maybe this supports the organizational goal to create a global robot army. It would be important to understand how online learning would best fit into the lives of the target students. For example, do they have their own equipment or do they share? How do target users find out about new online activities? How do the needs of students who only have mobile devices com-pare to those who have access to a laptop or desktop? Which activities are they already engaged in that might compete with or complement such a course? Read More at
Evaluative research: “Are we getting close?”
Once you have a very clear idea of the problem you’re trying to solve, you can begin to define potential solutions. And once you have ideas for potential solutions, you can test them to make sure they work and meet the requirements you’ve identified. This is research you can, and should, do in an ongoing and iterative way as you move through design and development. The most common type of evaluative research is usability testing, but any time you put a proposed design solution in front of your client, you really are doing some evaluative research.
Causal research: “Why is this happening?”
Once you have implemented the solutions you proposed, and have a website or application up and running out in the world, you might start noticing that people are using it in a certain way, possibly a way that isn’t exactly what you’d hoped. Or perhaps, something really terrific is happening and you want to replicate the success in other parts of your operation. For example, you’ve noticed that ever since the Fantastic Science Center redesign launched, tickets for the Friday evening science-loving singles event are selling better, but ticket sales have completely dropped off for the Sunday afternoon film program. You need to do some causal research.
Establishing a cause-and-effect relationship can be tricky. Causal research often includes looking at analytics and conduct-ing multivariate testing (see Chapter 9). This means reviewing your site traffic to see how visitors are entering and moving around the site and what words they might be searching for, as well as trying design and language variations to see which ones are more effective. Causal research might show that your film program traffic all came from one referring website that no longer links to you. Or, you might have to expand beyond looking at site performance to see what else is going on. Maybe a competing organization started an event with a very similar name and confused your target audience.
As long as you’re clear about your questions and your expectations, don’t fret too much about the classification of the research you want to undertake. Remain open to learning at every stage of the process. And share this love of learning with your team. Your research will benefit from a collaborative approach that includes assigning specific responsibilities to different people. Must Read
Research roles represent clusters of tasks, not individual people. Often one person will take multiple roles on a study, or a single role can be shared.
The author plans and writes the study. This includes the prob-lem statement and questions, and the interview guide or test script.
The interviewer is the person who interacts directly with the test participants.
The coordinator plans how time will be used during the study and schedules sessions, including arranging times with the participants.
The notetaker is responsible for capturing the data during a test session. It is best that the interviewer and notetaker be two separate people so that the interviewer can devote full attention
to the participant. If this is impossible, the interviewer can ar-range to record the session as unobtrusively as possible. Having both written notes and a recording makes data analysis easier.
The recruiter screens potential participants and identifies the respondents who would be good subjects. The recruiter and scheduler can easily be the same person.
The analyst reviews the gathered data to look for patterns and insights. More than one person should have this role.
The documenter reports the findings once the research study is complete.
Often it’s useful for clients or other available team members to watch the research in progress. This is appropriate as long as the presence of the observers will not influence the research itself. As a substitute, you can make raw recordings available.
You can change roles with each set of activities if that works best, or develop a routine that allows you to focus on the infor-mation gathering. Just as with design and coding, every time you complete some research, you’ll have ideas for how to do it better next time and you’ll have found new ways to incorporate learning into your work.
Listen. Be interested. Ask questions. Write clearly. And prac-tice. Whatever your day job is, adding research skills will make you better at it.
The research process
For the purposes of this section, what matters is that everyone working together has a shared understanding of how the work will proceed. This can be as simple as a checklist.
In addition to organizing the efforts of your immediate team, you might need to get approval to do research at all, either from the client or from decision-makers in your organization. Handle this as early as possible so you can focus on the work rather than defending it.
Sometimes the people you’re working with or for will consider research somewhere between a threat and a nuisance.
You might have to get a certain amount of what they call organizational buy-in to proceed. This could start with agreement from your immediate team, but the whole point of doing research is to have a stronger basis for decision-making, so if another level of decision-making, such as executive fiat, trumps the research, you will have wasted your time. Get ready to advocate for your research project—before you start it.
The objections you will hear
Here is a handy list of objections and responses.
We don’t have time
You don’t have time to be wrong about your assumptions. What are your key assumptions? What if they’re all wrong? How much work would you have to redo? How long would that take?
We don’t have the expertise, or the budget
You have what it takes, thanks to this book! Even with little or no budget, you can usually locate some related research online, wrangle representative users to interview, and do a little usability testing. Applying some critical thinking to your assumptions costs nothing, but a change in your habits can offer tremendous returns. Must read
The CEO is going to dictate what we do anyway
You’re going to fight to change that dictatorial culture. Reasonably accurate and well-documented research has been known to sway even the most magnificent and well-fed egos. And if the leadership really does have a “damn the facts, full speed ahead” attitude, get a different job.
One research methodology is superior (qualitative vs.quantitative)
What you need to find out determines the type of research you need to conduct. It’s that simple.
You need to be a scientist
This isn’t pure science we’re talking about here. This is applied research. You just need to have (or develop) a few qualities in common with a good scientist:
• Your desire to find out needs to be stronger than your desire to predict. Otherwise you’ll be a mess of confirmation bias, looking for answers that confirm what you already assume.
• You need to be able to depersonalize the work. There are no hurt feelings or bruised toes in research, only findings.
• You need to be a good communicator and a good analytical thinker. Otherwise questions and reports get muddy, and results will be worse. This is just a set of skills that most people can develop if they have the right attitude.
You need infrastructure
I suspect you own or can borrow a laptop and have access to the internet. That is all you need.
It will take too long
Upfront research can provide a basis for decision-making that makes the rest of the work go much faster. Nothing slows down design and development projects as much as arguing over per-sonal opinions or wasting effort solving the wrong problem. And you can start small. A couple of weeks can mean very little to your overall schedule while adding significantly to your po-tential for success.
You can find out everything you need in beta
There are a lot of things you can find out in beta: what func-tionality is working, whether users have a hard time finding core features. But there are also a lot of things that are very helpful to know before you start designing or coding at all, and you can find those out pretty fast: what your target audience is doing right now to solve the problems your product or service purports to solve, whether people want this product at all, and what your organization has to do to support it. Must Read
Again, it’s a matter of where you want to invest and what you have to lose. Don’t waste anyone’s time or effort on untested assumptions if you don’t have to.
We know the issue/users/app/problem inside and out already
Unless this knowledge comes from recent inquiry specific to your current goals, a fresh look will be helpful. Familiarity breeds assumptions and blind spots. Plus, if you are very famil-iar with your users it will be very easy for you to find some to talk to.
And who is the “we” in this case? In the absence of a mind meld, the client’s experience with the users or the business problem doesn’t transfer to the designer. Shared understand-ing is key.
Research will change the scope of the project
It’s better to adjust the scope intentionally at the start than be surprised when new information pops up down the road to amend your plans. Research is an excellent prophylactic against unexpected complexity.
Research will get in the way of innovation
Relevance to the real world is what separates innovation from invention. Understanding why and how people do what they do today is essential to making new concepts fit into their lives tomorrow.
Actual reasons behind the objections
At the root of most of these objections is a special goo made up of laziness and fear.
I don’t want to be bothered
Unless you are naturally curious about people, research can seem like annoying homework at first, but once you get into it, you’ll find it totally fun and useful. A little knowledge opens up a whole world of new problems to solve and new ways to solve the problems at hand. That makes your work more rewarding. If research is one more thing tossed on your already overfull plate, then someone needs to ask the “Who should be doing this?” question again—but the problem is you being too busy, not research being unimportant. Research needs to be integrated into process and workflow or it will get shoved in a corner. If your project has a project manager, talk with them about finding ways to make it work.
I am afraid of being wrong
The cult of the individual genius designer/developer/entre-preneur is strong. In certain “rockstar knows best” cultures, wanting to do research can come across as a sign of weakness or lack of confidence. Fight this.
Information and iteration are the keys to a successful design.
Research is just one set of inputs.
I am very uncomfortable talking to people
You are creating a system or a service actual people are going to have to use. This system will be talking to people on your behalf, so it’s only fair that you talk to people on its behalf. That said, some people on your team will have more comfort and skills when it comes to interacting with your research subjects, so consider that when you’re deciding who does what.
Having to respond to challenges and objections before you can get to work might feel like a waste of time, but it can be very useful in its own right. Describing the goals and potential of your research to people who aren’t sold on the value will actually help you focus and better articulate what you hope to uncover.
These discussions will give you a better understanding of the environment you’re working in. Research is all about context. Must Read
Research in any situation
“Poor user experiences inevitably come from poorly informed design teams.”
—Jared M. Spool, founder of User Interface Engineering
Design happens in context. And research is simply understand-ing that context.
Research happens in a context as well. Your professional environment should inform the research activities you choose and how you go about them.
The following contexts and situations aren’t mutually ex-clusive. You might be in some that overlap. Just remember that no matter what situation you’re in, you can do or participate in some useful research.
On the one hand, as a freelancer, you can do whatever the hell you want as part of your design and development process as long as you deliver what the client expects when the client needs it. On the other hand, if someone is hiring you as a solo web designer, they may balk at paying for something that falls outside of their concept of that gig.
If you are doing work, you need to get paid for it. “Just tossing in the research” is a terrible mistake designers who want to do good work make all the time. Instead, you have to commit to research personally as part of how you work, make your case for it, and be sure to include it in your fee. Research will make your design stronger and enhance your ability to defend your decisions to the client.
If you’re being brought in as a contractor to work as part of an internal team, make sure you have access to all of the information and insight required to do your job. Contractors run the risk of being pigeonholed. You’re the designer, why do you need to bother with research? When information is distributed on a “need-to-know” basis, you’re the best judge of what you need to know to get the job done. Must read
At a client services agency
If you are at an agency, you have the opportunity to improve your process with each new project. A certain amount of re-search is built in simply because you have to scope the work to bid on the job and understand the client’s needs (or do an awe-some job of faking it) to land the gig. The better you do these things, the better time you’ll have doing the work.
At Mule, and at many other agencies, the first phase of work on any new project is an intensive period including all the research that’s useful and practical. We talk to stakeholders, interview users, review competitors, and sometimes conduct a round of benchmark usability testing. Sometimes we do this in a matter of a few weeks. We need to get up to speed quickly, so working collaboratively is essential.
As with freelancing, coming in from the outside puts us at a definite advantage because we are external to existing processes and political considerations. Some clients bring us in to ask the questions they know need to be asked, but are not in a position to ask themselves.
Falling back on ignorance can be a position of strength. Asking naive questions can cut right to the heart of assumptions and open people up to thinking about problems in a new way. “How does that benefit the business?” and “Why do you do it that way?” are a couple of terrific questions that can be very tricky for someone on the inside to get away with.
In-house at an established company
In an established organization of any size, politics are a huge consideration. Challenging the assumptions of those in power can be incredibly threatening to those people. It can also be the best thing you can do to ensure that your work succeeds—if you don’t get fired. (See Chapter 4 for more on introducing even the most stubborn organization to the joys of research.)
If you’re at an organization that genuinely embraces critical thinking and clear communication in the design process, that’s terrific. I hope you’re also taking very good care of your unicorn desk mate. Otherwise, proceed with open eyes and discretion.
An existing product means that a glorious data trove exists: customer service! Customer service is where actual, individual human needs and expectations crash headfirst into reality. If you have ready access to the customer service representatives, talk to them. You will make friends. Customer service staff have so much expertise and often get very little respect within an organization. And they have to communicate with unhappy people all day, every day. A conversation with someone who respects and values their contribution is likely to be a good time for all. Must Read
In addition to, or instead of, direct access to the customer ser-vice people, get hold of the inbound support requests. This will be a fantastic source of insights into the ways different types of customers think about their needs and the language they use to describe them. You can also practice seeing through what people say they want to what they actually need. Steve in Louisville may be asking for a more informative error message, but what he really wants is to be able to reorder his usual pizza and salad with a different credit card and no error message.
You don’t just want insights; you also want a way to put those insights back into the product.
It’s very helpful to have a clear idea of how product and mar-keting decisions are made in your company. Do you have a truly customer-centered culture? When leaders talk about research, are they talking about market research, or do they have a more holistic view? Is there a market research group? Is design or user experience research already part of your process?
In-house at a startup
When you have a small startup team, you don’t have to worry too much about understanding your own organization beyond knowing what you have to do to keep the lights on. In the early stages it should be easy to share information with the team, as long as you aren’t growing so fast that people and perspectives start getting left out.
You do need to have some clarity around your audience and the business context you’re operating in. You’re trying to intro-duce something new into the world. Who needs it and what is important to those people? When you’re discussing the initial design and development of your product, discuss the role of research with the team. Document and review assumptions to identify the areas in which doing some research might be the most beneficial. Get some early agreement on how research will be involved, keep track of your assumptions, and adopt a skeptical point of view.
The approach and biases of the founder and the investors might dominate, so if you aren’t one of those, you will have to be very clear about the value of research to your endeavor and savvy about how to make your case.
Working with an agile development team
Agile is a popular software development philosophy with the goal of building better software faster in a productive, collaborative working environment. Many short iterations of two or three weeks replace the traditional approach of multi-month or multi-year projects broken into distinct phases.
On the surface, agile seems antithetical to design. The agile manifesto explicitly values “responding to change over following a plan.” Design is planning. However, any work with complex ideas and dependencies requires holding some ideas outside the development process. You can’t cave in completely to the seduc-tive solipsism that agile offers, or you’ll be tunneling efficiently and collaboratively toward the center of the earth. While flex-ibility and responsiveness are certainly virtues that many project teams could use more of, let’s not discount the importance of having some sort of plan.
From a user experience perspective, the primary problem with Agile is that it’s focused on the process, not the outcomes. It doesn’t offer guidance on what to build, only how. Perhaps your team is more efficient and happier making a lot of stuff together, but how do you know that stuff is the best it could be, meeting real user needs and fit to compete in the marketplace?
If you’re always reacting without a framework, you need some guiding mandates. Which customers do you listen to and why? Which user stories do you prioritize? What are you ulti-mately building toward?
Research is not antithetical to moving fast and shipping con-stantly. You’ll need to do some upfront work for background and strategy and the overall framework. Then, as the work progresses, do continual research.
It might sound counterintuitive, but the most effective ap-proach may be to decouple the research planning from the development process—that is, don’t wait to start coding until you’ve answered all your research questions. Once you have some basic tools and processes in place, such as observation guides, interview guides, recording equipment, and questions for analysis, you can take a Mad Libs approach and fill in your actual questions and prototypes on the fly.
Jeff Patton describes this continuous user research process in his article “Twelve Emerging Best Practices for Adding UX Work to Agile Development”. He offers a tidy three-point summary:
• Aggressively prioritize the highest-value users.
• Analyze and model data quickly and collaboratively.
• Defer less urgent research and complete it while the software is being constructed.
In other words, focus only on the essential user types, deal with your data as soon as you get it, involve your team in the analysis, and do the less important stuff later.
This of course opens up the questions of who are the highest-value users and what are the more or less urgent research activi-ties. Prioritize those user types whose acceptance of the product is critical to success and those who least resemble the software developers on your team. Go learn about them.
Recruiting and scheduling participants is the most difficult part, so always be recruiting. Set up windows of time with different participants every three weeks. to understand their behavior before the next round of development or do some usability testing on the current state of the application.
Use what you learn from the initial user research and analysis to create personas that inform high-level sketches and user stories. Then, when the team is working on a feature that has a lot more engineering complexity than interaction design complexity, you can fit in additional evaluative research.
Throughout the development cycle, the designers can use research to function as a periscope, keeping an eye out for new insights about users and competitive opportunities while doing usability testing on whatever is ready.
Just enough rigor
Professional researchers are not unlike journalists. While many people have sufficient skills to observe, analyze, and write, it’s allegiance to a set of standards that sets the pros apart. In addition to being professional and respectful in your work, there are just a few responsibilities to keep in mind.
Cover your bias
Wherever there is research there is bias. Your perspective is colored by your habits, beliefs, and attitudes. Any study you design, run, or analyze will have at least a little bit of bias. Your group of participants will be imperfectly representative. Your data gathering will be skewed. Your analysis will be colored by selective interpretation.
Don’t give up!
You can’t eliminate it completely—but the simple act of not-ing potential or obvious bias in your research process or results will allow you to weigh the results more appropriately. In lieu of a trained eye, use the following bias checklist, or make your own. Grade hard.
Design in this case refers to the design of the studies themselves, how they are structured and conducted. This is the bias that creeps into studies when you don’t acknowledge bias, or if you include or leave out information based on personal goals or preferences.
Since we’re talking about quick and dirty qualitative research, sampling bias is almost unavoidable. Counter it by being mindful in the general conclusions you draw.
If your app for science-minded new parents is intended to serve men and women in equal numbers but all your subjects are women, that’s a biased sample.
Conducting unbiased interviews is difficult. Inserting one’s opinions is easy. Make sure that interviewers remain as neutral as possible. This is something to watch out for particularly at the beginning of interviews when you are trying to establish rapport. Maybe the interviewer is super enthusiastic about one aspect of the museum. Practice interviews and critiques with an internal team are the best way to develop a neutral interviewing style.
This is one of the biggest issues with onsite lab usability tests, because going onsite feels special and can be exciting or even daunting to a participant. If the Fantastic Science Center is invit-ing you in to their facility, offering you snacks, and writing you a check, it is very possible you will be gentler in your evalua-tions. To decrease sponsor bias without being deceptive, use a general description of the organization and goals of the study without naming the specific company until and unless it appears in materials you are evaluating. (Once you get to the point of showing a website design featuring the Fantastic Science Center logo, the secret will be out.)
For example, begin a phone interview with “We’re interested in how you select and plan activities for your family,” rather than “We want you to tell us what would entice you to visit the Fantastic Science Center.”
Social desirability bias
Everyone wants to look their best. People want to be liked. It can be hard to admit to an interviewer that you don’t floss or pay off your credit card bill every month, so participants will sometimes give the answers that put them in the best light. Emphasize the need for honesty and promise confidentiality.
The Hawthorne effect
The behavior of the people you are studying might change just because you are there. Staff who typically goof around and chat during the day might clam up and shuffle files if you’re hanging about to observe their workflow. Do your best to blend into the background and encourage research participants to go about their normal day.
The ethics of user research
What harm can come of asking people how they decide what to have for dinner or how they use their phones to find directions? We aren’t talking about clinical trials of dangerous, new cancer drugs, but all research that includes people and their personal information should be conducted ethically and conscientiously. It’s our responsibility as professionals to proceed without de-ceiving or injuring any of the participants.
Below is a starter set of ethical concerns you should keep in mind whenever you are doing research.
The project as a whole
Maybe this goes without saying, but it is worth saying neverthe-less. Is your overall goal, the project that the research supports, ethical? Will your success lead to harm for others? If it will, don’t participate in it. Designers have a role to play as gatekeepers. You should be intentional about your position. Conducting a completely above-the-board study on women to induce them to buy a diet aid with dangerous side effects doesn’t make it right.
The goals or methods of the research
A certain amount of user research and usability requires keep-ing certain facts from the participants. Usually this is benign, such as hiding the name and description of the product you’re designing, but sometimes it’s a problem. Will concealing these facts lead those users to participate in anything they might not otherwise agree to? Are you tricking them or setting some un-realistic expectation about the real world? Are you presenting false information as true?
Consent and transparency
Informed consent is the rule. This means that participants must understand and agree in advance to the overall goals of any study and how their information will be recorded, used, or shared. Let them know if they are being watched by unseen observers. Make sure that research participants are of sound mind and able to give consent to participate. This means that working with underage research participants is very tricky, and requires the parents’ consent.
Safety and privacy
Ensure that participants know what is required of them in ad-vance and will be comfortable and not fatigued. Verify that your presence in a home or workplace will not lead to any risks or danger. For example, if you’re observing someone taking care of small children, make sure that your actions don’t distract in any way that would interfere with proper care.
And for the love of all humanity, never, ever agree to do telephone interviews when anyone involved is driving. Not participants, not interviewers, not passive observers. No one. As soon as you learn that someone is on the phone while driv-ing, end the call, and follow up by email or another means to reschedule if necessary.
Be a skeptic
Get in the habit of asking a lot of questions. Question all your assumptions and determine whether you need to check your facts. If you’re constantly on the lookout for threats and poten-tial points of failure, you and your products will be stronger. This is a type of critical thinking that will serve you well at all times. You need to be aware of how much you don’t know and what that means.
Awareness of your own limits will allow you to be as effective as possible within them.
There are many good reasons why people get master’s degrees and PhDs and become professional analysts and researchers, and there are plenty of reasons why companies benefit from hiring those people. Specialized, educated, and trained researchers cultivate a deep curiosity, have a broad base of relevant knowl-edge, and gain academic and professional experience conduct-ing ethical and methodical studies. As a designer or developer, you might have good reasons to avoid DIY and hire a trained professional.
• A large, complex project.
• A large, complex organization.
• Complex or sensitive subject matter.
• A very specialized or challenging user base, such as children or neurosurgeons.
• Heinous organizational politics.
• Lack of team members with the time or inclination to acquire additional skills and duties.
Skilled, trained professional researchers have rigor. They can apply precise critical thinking in the face of common distractions and pressures, such as the enthusiasm of their team or their manager’s personal preferences. The best researchers are like Mr. Spock, with just enough humor and humanity to temper their logical thought processes and allow them to roll with im-perfect circumstances. You want rigorous, not rigid.
In the absence of a trained professional, how do you ensure you are being sufficiently rigorous? You’re an amateur attempt-ing these stunts on the open road instead of a closed course; how do you make sure you and your work don’t go up in flames?
You borrow the methods of America’s greatest amateur, Benjamin Franklin: discipline and checklists.
Discipline requires you to be ever-watchful for bad habits, shoddy thinking, and other human frailties that will undermine your efforts. Checklists substitute the experience of others for your own. Discipline also requires that you don’t deviate from the checklists until you have sufficient experience yourself.
Here is the first checklist, that of best practices. Go over these again and again until you know them by heart, and then post them visibly so you never have to rely on memory.
1. Phrase questions clearly
This refers not to the questions you’re asking, but the big ques-tion you’re trying to answer. Unless you know and can clearly state what you’re trying to find out and why, applied research is a pointless exercise.
2. Set realistic expectations
A successful study is preceded by expectation-setting for ev-eryone involved, including the questions to be answered, the methods to be used, and the decisions to be informed by the findings. This is particularly important if you need to request time or budget especially for the work. If your research work doesn’t meet the expectations of the stakeholders, they will treat you like you’ve wasted time and money. Ask team members and managers what they hope for. Tell them what to expect.
3. Be prepared
Research is like kitchen work: the better you prep, the faster and cleaner the work goes. If you don’t prepare, you end up with a huge mess and a kitchen on fire. Get your process and materials in order before you start. Set these up so they’re easy to reuse as needed.
4. Allow sufficient time for analysis
You need a little time for things to click into place. After doing the research, it’s tempting to just forge ahead to solutions with-out giving yourself enough time to digest. Again, a bit more time here can save lots later on.
5. Take dictation
Notes or it didn’t happen. Effective research requires effective reporting, and sharing your results and recommendations with others. A good report doesn’t have to be arduous to compile or read. It needs to be sufficiently informative and very clear to anyone who needs to make decisions based on the research.
You may be doing your own research to save time and money, but be honest with yourself and your team about your capacity for maintaining this level of rigor. Otherwise you risk wasting both time and money, as well as spreading misinformation and decreasing the overall reputation of research as a necessary input into the work.
Can you commit?
Good. Then onward.
How much research is enough?
“There are things we know that we know. There are known unknowns—that is to say, there are things that we now know we don’t know. But there are also unknown unknowns—there are things we do not know we don’t know.”
—Donald Rumsfeld, former US secretary of defense
In addition to offering the clarity and confidence necessary to design, research is essential to reducing your risk—the risk you incur by relying on assumptions that turn out to be wrong or by failing to focus on what’s most important to your business and your users. However, some assumptions are higher-risk than others.
To make the best use of your time and truly do just enough research, try to identify your highest-priority questions—your assumptions that carry the biggest risk.
Ask this question: given our stated business goals, what po-tential costs do we incur—what bad thing will happen—if, six months from now, we realize:
• We are solving the wrong problem.
• We were wrong about how much organizational support we have for this project.
• We don’t have a particular competitive advantage we thought we had, or we didn’t see a particular competitive advantage before our competitor copied us.
• We were working on features that excited us but don’t actu-ally matter that much to our most important customers.
• We failed to reflect what is actually most important to our users.
• Our users don’t really understand the labels we’re using.
• We missed a key aspect of our users’ environments.
• We were wrong about our prospective users’ habits and preferences.
If there is no risk associated with an assumption—for ex-ample, if you are working on a technical proof of concept that really, truly doesn’t have to satisfy any real-world users—then you don’t need to spend time investigating that assumption. Must Read
On the other hand, maybe the success of the new design for the Fantastic Science Center’s online store depends on the as-sumption that many people who shop online value the ability to publicly share their transactions. You could conduct research to understand the social sharing practices and motivations of people who shop online before diving into design and development. Or you could go ahead and design based on an optimistic assumption, then see what happens. At risk are the time and money to design and build the functionality, as well as the organization’s reputation. (“They just told everyone on the internet about the robot I bought my kid for her birthday. Not cool!”)
Better understanding of online shoppers mitigates the risk by validating the assumption and informing your design with real user priorities. In addition, you might uncover opportunities to provide something of even greater value to that same audience.
All it takes to turn potential hindsight into happy foresight is keeping your eyes open and asking the right questions. Failing isn’t the only way to learn.
That satisfying click
No matter how much research you do, there will still be things you wish you’d known, and there are some things you can only learn once your design is out there in the world. Design is an iterative process. Questions will continue to crop up. Some of them you can answer with research and some you can only answer with design. Even with research, you’ll need to create a few iterations of the wrong thing to get to the right thing. There is no answer to the question of enough, other than the point at which you feel sufficiently informed and inspired. The topics in this book can only offer a starter kit of known unknowns.
That said, one way to know you’ve done enough research is to listen for the satisfying click. That’s the sound of the pieces falling into place when you have a clear idea of the problem you need to solve and enough information to start working on the solution. The click will sound at different times depending on the problem at hand and the people working on it.
Patterns will begin to emerge from the data. Those patterns will become the answers you need to move forward. This will be very satisfying on a neurochemical level, especially when you start out with a lot of uncertainty. Since human brains are pattern recognition machines, you might start seeing the pat-terns you want to see that aren’t actually there. Collaborating with a team to interpret the data will reduce the risk of overly optimistic interpretation.
If you don’t have enough information, or what you’re finding doesn’t quite hold together, the pieces will rattle around in your head. Ask a few more questions or talk to a few more people. Talk through the results. The pieces will fall into place.