THE GREAT Topic Subtopic CourRSsES' Professional | Research & Analytics Effective Research Methods for Any Project Course Guidebook Professor Amanda M. Rosen Webster UniversityPublished by The Great Courses Corporate Headquarters 4840 Westfields Boulevard | Suite 500 | Chantilly, Virginia | 20151-2299 Phone 1.800.832.2412 | Fax 703.378.3819 | www.thegreatcourses.com Copyright © The Teaching Company, 2019 Printed in the United States of America This book is in copyright. All rights reserved. Without limiting the rights under copyright reserved above, no part of this publication may be reproduced, stored in or introduced into a retrieval system, or transmitted, in any form, or by any means (electronic, mechanical, photocopying, recording, or otherwise), without the prior written permission of The Teaching Company.Amanda M. Rosen, PhD Associate Professor of Politics and International Relations Webster University manda M. Rosen is an Associate Professor of Politics and International Relations and a fellow in the Institute for Human Rights and Humanitarian Studies at Webster University. She holds a BA in Political and Economic Studies of Europe from Duke University as well as an MA and a PhD in Political Science from The Ohio State University. Dr. Rosen’s dissertation was on the impact of international policy making on national and subnational action on climate change. She regularly teaches research methods at both undergraduate and graduate levels. Dr. Rosen’s research specialization is the scholarship of teaching and learning, with particular focus on games and simulations, experiential learning, human rights education, and transparent teaching. She also works on climate change policy making, issues of human security, and international human rights of marriage and the family.Effective Research Methods for Any Project Dr. Rosen’s work can be found in the Journal of Political Science Education; PS: Political Science & Politics; International Studies Perspectives; and Politics & Policy. She has authored chapters in Human Rights in Higher Education: Institutional, Classroom, and Community Approaches to Teaching Social Justice as well as in Learning from Each Other: Refining the Practice of Teaching in Higher Education. She is also a cofounder of the Active Learning in Political Science blog. Dr. Rosen’s work on teaching methods has been recognized with numerous teaching awards, including the International Studies Association's Deborah Gerner Innovative Teaching Award, the William T. Kemper Award for Teaching Excellence, and the CQ Press Award for Teaching Innovation. She has led numerous workshops on teaching and pedagogy at conferences and universities and serves as the vice president and program chair for the International Studies Association's International Education section.Table of Contents INTRODUCTION Professor Biography ....... See eee ee eed CourseScope. ee ed LECTURES 1 Why Research Methods Matter 2 Characteristics of Good Research. . . 3 Doing Research Ethically.. 2.2... .... 00000. 20 4 From Topic of Interest to Research Question... .. 2... 30 5S What’s Already Known? The LiteratureReview........ 38 6 Generating Hypotheses and Theories.......... . 48 7 Selecting aResearch Design... 2. 57 8 Measuring Concepts andPhenomena............. 67 9 Choosing Populations, Samples, and Cases... ... .. 75 10° TheClassicExperiment................., . 84 TI The Value of Quasi Experiments................ 94 12 Designing and ConductingaSurvey........... . 102 13 Understanding Election Polls... 2. 112 14 Research by Case Study... 121Effective Research Methods for Any Project 15 Interpretivism and Field Research... 2.20.0... . 130 16 Applied, Evaluative, and Action Research... . 2... 141 17 Gathering andPreparingData.. 22 151 18 Using Statistics to InterpretData.. 2 162 19 Statistical Inferences fromData.... 2.2... 2... 170 20 Assessing Correlation and Causation........... . 180 21 From Bivariate to Multivariate Analysis ........ . 188 22 Foundations of Qualitative Analysis... 2... 2... . 197 23 Qualitative Analysis Variations... 2.2... 205 24 The Art of Presenting YourFindings........... . 213 SUPPLEMENTARY MATERIAL QZ. ee ve. 223 Answers and Explanations... 2.0.2.0... 0.00000. 232 Bibliography 2 ee 244 Image Credits. eee . 255Effective Research Methods for Any Project his course will give you the tools you need to conduct research— whether you have to do a project for work or because there is a question you've always wanted to answer, a puzzle you want to solve, or a program you want to evaluate. Research isn’t just for academics and scientists; it’s for everyone. You will learn guidelines and systematic methods that will show you the value of knowing the ins and outs of research in your everyday life and give you the tools to answer questions both large and small. The first part of the course focuses on the fundamentals of research, taking the scientific method as the framework for everything that follows. You will learn what good research looks like so that you can evaluate your own work as well as the claims and findings of others. You will discover what sound practice in research looks like, and you will be able to see whether others follow it. You will follow the path of ethics in research, seeing what research looked like before the sound ethical principles of the Belmont Report existed to guide the study of human behavior. Topics will quickly become research questions, and thoroughly reviewing the literature will help crystalize the hypotheses, theories, and other ideas that will allow you to start designing your research project The second part of the course turns to the many design tools you will put in your toolbox so that you can bring them out as needed for any individual question or project. You will learn how to select the best research design for your project and how to measure your concepts and variables so that you can actually observe them. You will also give some thought as to who you are studying and learn why you don’t have to study every member of that group to be able to say something interesting about it. All the classic research designsEffective Research Methods for Any Project will compete for use in your project—from experiments to surveys and from case studies to fieldwork. After you are introduced to the interpretation of election polls, you will consider the interpretivist challenge to positivist research as well as applied, action, and evaluation research With your design chosen, it's time to move on to the third part of the course: data analysis and presentation. First, you will learn how to get your data in order and ready for analysis. Then, you will review a variety of quantitative and qualitative analysis techniques. Whatever kind of data you have, you will learn some tools for how to test your hypotheses and assess your claims. Along the way, you will learn why you cannot infer causation from correlation and why statistical significance is so important to interpreting your results. You will also learn multiple different ways to evaluate qualitative data. Finally, you will learn how to communicate all you have learned, as the best way to advance knowledge is to share it. And the best part is you don’t need to have strong math or science skills to do any of this. By the end of this course, you will have the tools you need to start your next stage in life as a full-fledged researcher.il Why Research ethods MatterEffective Research Methods for Any Project esearch methods include all the many ways researchers systematically study anything to better understand it, uncover new information about it, or explain why things happen as they do. While research is incremental and cumulative and requires continual testing, retesting, and debate, it is ultimately through the application of sound research methods and principles that we advance our understanding of the world. What makes the study of research methods so important is that it helps us see the inner workings of how knowledge is created. And it turns out that how we discover information about the world matters just as much as what we learn. The world of scientific history is full of cases where preexisting and incorrect beliefs are challenged by the application of sound research methods. The idea that the world was hollow and flat and full of sea monsters was once held by the experts of the day, but it was discredited thanks to rigorous research. Aristotle thought that the heart, not the brain, controlled sensation and cunt ona Cae cat meee eeletis Race eeonerd human organs.| Why Research Methods Matter TYPES OF RESEARCH Although there are many ways to classify research, it is generally divided into 2 broad categories: basic research and applied research. Basic research—also known as fundamental, or pure, research—aims at answering questions that lead to gains in knowledge, understanding, and prediction. It’s about advancing our general knowledge of how the world works. Applied research aims at answering questions that lead to direct and practical applications in the world. Applied research uses the findings of basic research to develop practical solutions aimed at changing the world, Applied research must be grounded in the findings of basic research; the 2 are therefore interrelated Another way of looking at research is focusing on the type of inquiry: exploratory, descriptive, or explanatory. Exploratory research focuses on initial exploration. This is typically done as a precursor to other types of research to learn more about a topic or problem or what others have said about it. Maybe you want to study something about a general topic but don’t know exactly what to focus on. Doing some exploratory research by reviewing the scholarly literature and seeing what kind of data and other studies exist would be a good start. Descriptive research is when you attempt to describe something in detail but are not necessarily aiming at explaining why something is the way it is. Descriptive research aims at answering questions of who, what, when, and where. Explanatory research can be a natural follow-up to descriptive research. It aims at questions of why and how; the goal is to evaluate cause and effect and explain why and how things work the way they do.Effective Research Methods for Any Project Still another distinction in types of research is between quantitative and qualitative research Quantitative research focuses on analyzing large amounts of data that can be readily expressed in numbers. Qualitative work can also have numbers but usually involves data that is not easily boiled down to numbers and therefore tends to focus on a smaller number of cases or subjects. Quantitative work generally allows for greater breadth in the number of cases and variables that you study, while qualitative work sacrifices some breadth for the sake of depth, really digging into each case and variable that you study. Scholars and practitioners can get into heated debates over which general approach is better. Different disciplines focus on different approaches to research, and the nature of your project will often dictate which kind of research you do. But all these approaches are simply different forms of inquiry, and that’s the key point: Regardless of the type of research you choose, ultimately you are trying to explore, describe, understand, explain, evaluate, or predict something about the world. Because research is about inquiry, it’s important to understand that you don’t always find what you expect. Take Alexander Fleming, who in 1928 discovered penicillin while investigating bacteria. In fact, many of the most important discoveries in the world were found largely by accident. Some discoverers actually end up regretting that they made their discoveries. Alfred Nobel—for whom many top prizes are named, including the Nobel Peace Prize—invented dynamite as a safer explosive for construction projects, never intending that it would be used as a weapon of war.| Why Research Methods Matter When you do research, you have to be open Albert Einstein's to both the unintended and the undesired. discoveries in You can’t go into a research project thinking + you already know the answer and only physics set tne looking for information that confirms your — foundation forthe preexisting ideas. You have to be open to _ invention ofthe what you find—to what the data tells you— atomic bomb, and even if it means your initial ideas are wrong he expressed deep This is a fundamental principle of research —_‘TeMorse over how and methods: You never set out to prove his work was used. yourself correct. If anything, your job is to try your hardest to disprove your ideas. Only if you subject your initial thoughts to the most rigorous of tests, and they pass, can you have any confidence that your ideas might be right. If instead you set out to only find the information that confirms what you already believe, then you aren't doing research; you are just pushing a point of view or trying to make yourself look good. It’s human nature to believe a study that confirms your existing beliefs, even if the study was poorly or unsystematically done. But the study that challenges your beliefs is the one that really deserves your attention. There are tons of research methods this course could cover, considering how many disciplines and areas of human endeavor depend on research. The course will focus, however, on empirical research that uses the scientific method as a way to find answers to questions. While not all research is grounded in the scientific method, many of the principles of empirical, scientific research apply to all kinds of questions and problems. Therefore, the concepts, principles, and techniques that are covered in this course will aid you in your research, no matter what area you want to explore.Effective Research Methods for Any Project THE SCIENTIFIC METHOD OF RESEARCH The scientific method is a systematic approach to research that relies on observation, hypothesis formation, prediction, and hypothesis testing. In its essence, it is simply a process to follow that will get you from whatever research question you might have to an answer, regardless of your discipline, profession, or topic of study. The scientific method can be described in the following 6 steps. However, these steps should not be seen as a pure, literal path to follow. In some cases, you may do step 2 first and then go back to step 1. In other cases, it’s a more cyclical process, where different steps feed into each other and you go back and forth between The origins of the scientific method can be traced back to Aristotle, butit has been refined over the centuries—most notably between the 16th and 18th centuries by luminaries such as Francis Bacon and Sir Isaac Newton, but in other important ways by scientists of the 20th and 21st centuries as our understanding of reality and our ability to observe it them, and one project leads have developed. directly into the next. 1. Ask a question. It’s not enough to just have a topic of interest; research relies on questions. Questions give you a focused reason for study, so at some point early in the process, you need to figure out what yours might be. 2. Observe. This means you have to start making observations about the world around you—both so that you understand the context for your question and so that you know what others have already discovered. It might be that your question has already been answered; it’s only through reading what is already known, called conducting a literature review, that you will find that out. There’s excitement in asking new questions that nobody has answered before, but keep in mind that how you observe—the order and method you follow—are of vital importance.| Why Research Methods Matter 3. Form a hypothesis. Not all research involves hypothesis- testing. Some research aims at exploration or description, in which case steps 1 and 2 might be sufficient. But if you are trying to explain cause and effect, then generating a hypothesis is key. For purposes of research, a hypothesis is defined as an educated guess of the answer to your question, grounded in the observations you have completed. It’s typically a testable statement of the relationship between 2 or more concepts, and it clearly expresses how you anticipate that they are related or impact each other. In the third step, depending on your study, you might be making predictions about behavior that you can then test or you might start developing a theory that something produces a certain outcome. Other times you might not want to start predicting or theorizing until after you've completed your test, or multiple tests, and analyzed the results. That’s the main difference between 2 other common classifications of research: deductive and inductive. © Deductive research starts with theories and tests them. © Inductive research starts with observation and then builds theories from the results, 4. Test your hypothesis. This is the meat of research methods. In this step, you first have to design a good study to see whether your hypothesis holds water and then carry it out. That means choosing your methodology, figuring out how to measure your concepts and variables, identifying a relevant population and sample, gathering your data, and perhaps applying for funding. You haveEffective Research Methods for Any Project to rigorously test your hypothesis, considering and evaluating any alternative hypotheses and explanations. Only if your hypothesis survives the testing process can you report with any confidence—although usually still with some level of uncertainty—that your initial ideas might have had some merit. Analyze your results. Now it’s time to analyze the data you gather and interpret it. Does it support or refute your hypothesis? Pay careful attention to data that contradicts your initial views; you might need to revise your hypothesis or theory or develop entirely new ideas. This is also where you want to consider the implications of your results, particularly for applied research purposes. It's at this stage that you figure out if you've actually found an answer to your question or if more work needs to be done. Typically, in research, itis this is also where you discover the next 3 or 5 or 8 questions you want always better to be to answer, now that you've embedded _clear about your yourself in this work. uncertainty and + to be honest and Communicate your results. Research is all about communication. Sharing careful about the your findings is an essential part of — extentofyourclaims. the research process, for 3 reasons: Ifyou overstate your It allows your work to undergo peer findings, your work review, it might have a positive impact won't be as valued on others, and it helps you get simple 1 . recognition for the hard work of doing as if you report it research. factually, and it may be disregarded altogether.LECTURE 1 | Why Research Methods Matter READINGS Adams, Khan, and Raeside, Research Methods for Business and Social Science Students, chs. 1 and 3. Edmonds and Kennedy, An Applied Guide to Research Designs: Quantitative, Qualitative, and Mixed Methods, ch. 1.erica cg ectecnela lS Good Research _| Characteristics of Good Research o evaluate research properly, you have to evaluate the research findings and the claims they make about how the world works. And to do that, you need to establish some criteria of good research so that you know what to look for when reading the work of others. Identifying those criteria will also help you know how to structure your research to ensure that it, too, is of the highest quality. KEY CHARACTERISTICS OF GOOD RESEARCH While there are other characteristics of good research, the following 5 key ones can be reasonably applied to most fields, projects, and methodologies and offer a good start to understanding the fundamentals of how to approach a research project. Research should be systematic. Research follows a set of sound procedures in the process of discovery and finding answers. Proper research has to have systematic procedures for a researcher to follow. Systematic research keeps you honest and ensures that you can stand behind the results you come up with. By using sound methods, you can demonstrate that your findings are not due to your personal preferences or laziness, but due to following a set, defendable process. To make your research systematic, start by following the scientific method and ensuring that you have a solid research design. That research design will lay out the specific procedures you will follow in your work, such as an experiment, a survey, a case study, or an analysis of an existing database. Correspondingly, when you evaluate the work of other researchers, you should look for a clearly outlined methodology section in their writing or presentation, in which they explain the process they followed to get their results. If there is no discussion of the methodology or if it seems very haphazard, thats a sign that the research was not conducted systemically and doesn’t deserve your trust.Effective Research Methods for Any Project Research should be objective. Researchers must be neutral toward their study and remain objective throughout the research process. If you start out wanting to prove yourself correct or certain that a particular point of view is better than another, then you are setting yourself up for biased results and, in turn, poorly conducted research. Everyone is entitled to their opinion, but research isn’t about opinions. Sometimes you get results you don't like, and you have to be open to that. The reason is that research can be seen as a quest for truth. To the extent that an objective truth exists—a highly debatable prospect itself—that is what you seek. You are looking to find out the answer to a research question by observing the world around you. It’s fine to start out with a guess as to what that answer is, but you must be open to the idea that your guess is wrong. Your quest is not to prove your ideas right, but to figure out answers to questions about how the world works. Only by aiming for objectivity can you achieve this. One kind of research, known as hypothesis testing, aims at vigorously testing hypotheses in an attempt to prove them wrong. If you challenge your guess at every turn and it still holds up, then you can have some level of confidence that it might be correct. But if you go in with an idea of an answer and then either subject it to weak tests or, worse, look for evidence proving it correct, then you are Not conducting research. You are cherry-picking data to support your biases or to make yourself look good. That's not good research. Research should be empirical. Evidence should be grounded in observations and data. Research, fundamentally, is based on what you can observe in the world, not what you hope or wish to be true about the world. Your findings and conclusions must be based on an analysis of data that to the greatest extent possible represents an objective observation of the| Characteristics of Good Research When evaluating the research of others, you should pay attention to any conflicts of interest that researchers might have. If they are affiliated with a think tank or company, consider what impact that might have on their work. Their jobs and income might depend on producing a certain set of results, and this should be suspect. They certainly can still produce objective work—but you need to know about their potential biases so you can judge for yourself. The researcher should disclose any potential conflicts of interest so you can evaluate their methods and results with this in mind. Another thing to look for is charged language. If the language in someone's work relies on nonneutral descriptive terms, that may be evidence of bias and lack of objectivity. For your own work, be honest with yourself. If you have a preference toward finding a particular answer or result, will you be willing to admit it if you turn out to be wrong? If the answer is no, then this might not be the best project for you. Starting out by trying to prove your ideas correct is going to lead to results that no one will trust, so it's best to focus on a question or project that is of great interest to you but whose eventual findings or answer you are neutral toward. world around you. This isn’t always easy. Some things you might want to observe you simply can’t; other times human or technological failings prevent you from getting completely accurate observations. Research requires acknowledging and wrestling these limitations to acquire accurate observations about the world you can then analyze. This doesn’t mean that your results will necessarily be consistent or that they will yield clear answers to your questions. There is often a high degree of uncertainty and ambiguity in what you do. Many of the things you study are going to turn out in a way you can’t predict with any precision, at least some percentage of the time. But just because the data can’t give you a definitive, foolproof answer doesn’t mean that the research isn’t accurate or useful.Effective Research Methods for Any Project Research should be cumulative. Research builds on what other researchers have already discovered and helps build a scientific consensus. Research is an incremental process. You tend to want to answer the big questions entirely by yourself, but to do research well, you usually need to start small and work collaboratively with other experts. Research requires you to assess, critique, and build on previous work. If you don’t acknowledge and build on earlier findings, your work will not be taken seriously. It’s fine to critique or reject the work that has come before, but you still need to know it so that your About 97% of climate scientists argue that climate change is a real, man-made problem. One of the grounds for this claim is a 2013 study of peer-reviewed articles that took a position on global warming. The study evaluated the abstracts of almost 12,000 articles written by 29,000 authors and published in 1,980 journals. Of those that took a position on anthropogenic global warming, 97.1% endorsed the scientific consensus, A single paper or study claiming that global warming or climate change is real and manmade would not necessarily convince anyone to take action on such a complex global issue. But knowing that 12,000 articles claimed this is a much more persuasive claim.| Characteristics of Good Research audience knows how to situate your work with the other research they have read. And it is only by replicating and confirming previously published work that you can really come to a consensus about how the world works; a single study is not enough. Research should be transparent. You need to be open and transparent about all aspects of your research: the systematic methods you follow, any conflicts of interest, the empirical nature of your data, and how the work builds on and relates to other findings. Typically, this entails writing up your specific methodology and making your data available to others. This lets other researchers replicate your results; they can take your data and follow your procedures and should be able to come up with the same findings as | you did. This keeps you honest as a Academics and researcher and lets others have more _ scientists generally confidence in your work. Transparency aren't trained to write Ensures that research can pe vetted for the public, yet true efore, during, and after publication. 5 The norm of transparency ensures that _ Wansparency oe research—which is often the basis of — thatthey communicate society's most important decisions— _ their findings not only is based on genuine, accurate, and to their peers but also ethical work, even if it sometimes to all interested and takes a while to root out work that « fails to live up to that standard. affected parties. DETERMINING WHETHER GOOD RESEARCH HAS BEEN DONE How can you determine whether research has been done well or not? Make sure that the author has consistently and accurately cited his or her sources of evidence. If the study attributes an asserted fact to what “some people say” but provides no quotations or citations, the author might be making an assumption rather than reportingEffective Research Methods for Any Project a genuine finding. If the author refers to a study or an expert's opinion, there should be some kind of citation— even just a website or link—that lets you read the original source. Good researchers check their sources and cite originals, so if the citation record is spotty, you should question the work. Consider the reliability of the author’s sources of evidence. Whoever wrote the study that the author is citing might have invented the information, creating what is now often called fake news. You may have to check the original source’s citations to make sure they themselves are accurate. Make sure those studies aren't being misrepresented. It’s easy to choose a chart, graph, or statistic from a larger study and manipulate it to tell the story you want to tell. If the original source is a personal blog, an opinion page in a newspaper, or some other form of commentary, then it may be unreliable: there may not be checks on that source to ensure its accuracy or objectivity. Be sure to watch out for any signs of ideological or other bias. And keep in mind that crowd-sourced cites like Wikipedia are also problemati they may attempt to achieve objectivity by making themselves open to public scrutiny, but the information they provide is easily manipulated and may or may not be policed vigilantly. Evaluate the nature of any experts that the author relies ‘on. Who are these experts? Do they have credentials in the specific topic at hand? Whom you count as an “expert” matters in terms of the results you get. When you are judging the merits of research, be especially wary of findings based on personal experience. We all tend to use our personal experience—or that of people around us—as evidence for some greater phenomenon. But that does not mean these experiences are representative on the whole of the outcomes experienced by the general population. It is very easy to fall into the trap of relying on what you have experienced and therefore think you know with certainty. Personal experience can illustrate a| Characteristics of Good Research phenomenon, but by itself it is not evidence to be used in research. If you see a study that relies on a few first- person experiences to justify a finding, watch out. Check the author’s research to see how fairly it treats alternative claims and findings. Does it conduct a rigorous test of those claims? Does the presentation leave room for the possibility that the author may be wrong? Or does the data seem cherry-picked to support a single conclusion? Strong research should give alternative arguments a fair shot and acknowledge the limitations and nuances of the author's findings and claims. People love for things to be binary—black and white, good and evil, good and bad research—but that isn’t how the world works. We live in a world of nuance and subtlety, where every observation has qualifications and exceptions. If you see words like “always” and “never,” or other language that consistently ignores nuances and presents findings in binary terms rather than as points along a possible continuum, you should be suspicious. READINGS Krathwohl, Methods of Educational and Social Science Research, ch. 3. Little, New Directions in the Philosophy of Social Science.WY 'y Doing Rese 3 Ethically Y Ct| Doing Research Ethically esearch ethics are the codes, norms, and principles of acceptable behavior for those who conduct research. Your task in research is to uncover new ideas and information, but at the same time, you are charged with making sure that the benefits of that new information outweigh the costs and that you do everything possible to ensure no one is harmed in the course of your research. It also means that you engage in honesty about your work. THE NUREMBERG CODE Prior to World War Il, there were very few federal regulations in the United States or widespread international principles regarding ethical research. The revelations of Nazi experimentation led to the first such set of regulations: the Nuremberg Code. These are 10 principles of human subjects research. During World War II, the Nazis engaged in extensive experimentation on prisoners in concentration camps. They exposed people to frigid temperatures to check the effects of hypothermia; used toxic gases and chemicals on prisoners to test different treatments for chemical burns; and forcibly sterilized people to identify the most effective techniques to control people's abilities to procreate—among Many other research wrongs. 1. Subjects must give their voluntary consent to participate in research. 2. Research should be for the benefit of society and not achievable through other means. 3. Experiments should be designed based on existing knowledge such that the anticipated results will justify the experiment. 4, Research should avoid unnecessary suffering and harmEffective Research Methods for Any Project 5. Do not do any research where it is known beforehand that death or disabling injury is likely, except perhaps if the experimenter is also the subject. 6. Risk should never exceed the “humanitarian importance of the problem” being investigated. 7. Preparation and proper facilities should aim at protecting subjects from even the remote chance of harm. 8. Only “scientifically qualified persons” should conduct experiments, and only with the highest degree of care. 9. Subjects should be able to end the experiment if they feel they cannot continue. 10. The researcher must be prepared to terminate the experiment if they have reason to believe continuation will cause harm to the subject. These 10 principles laid the groundwork for the protections to come. But ethical violations did not stop in 1947.| Doing Research Ethically In 1964, the World Medical Association adopted the Declaration of Helsinki to establish ethical principles of medical research. It states, “While the primary purpose of medical research is to generate new knowledge, this goal can never ‘take precedence over the rights and interests of individual research subjects.” THE BELMONT REPORT In the United States, the National Research Act of 1974 established the National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research, which commissioned The Belmont Report to “identify the basic ethical principles” for the conduct of human subjects research. The US department of Health and Human Services adopted the report’s ideas as the Common Rule, the ethical principles regulating US research on human subjects. Issued in 1979, The Belmont Report is named after the Smithsonian Institution’s Belmont Conference Center, where the initial discussions took place. It identifies 3 comprehensive principles of ethical research on human subjects. Respect for persons. This means recognizing individuals as “autonomous agents” that have the ability to decide for themselves whether to participate in a research project. It also means that those who might not be able to act completely autonomously—children or those with “diminished capacity”—require special protection. How much protection is required will depend on the person and the potential harm of the study but should be determined with regard to this principle of respect. Beneficence. Like the Hippocratic oath, this principle requires that researchers do no harm. Subjects must be protected from unnecessary risks of harm, including physical, mental, psychological, social, economic, and legal harms. Additionally, beneficence requires that research maximize the general benefits to society and minimize harms to individual subjects.Effective Research Methods for Any Project Justice. This final principle recognizes the long history of research that used the poor and disadvantaged for subjects, exposing them to harm while doing work that would ultimately benefit the wealthy or advantaged Justice, then, calls for fairness in terms of access to research and its benefits as well as an equitable distribution of the potential risks to individuals. Drawing on these 3 principles, The Be/mont Report generated a set of applications that would be required of researchers. Researchers must solicit informed consent of their potential subjects. Subjects must be given enough information about the project’s purposes, procedures, risks, and benefits to make an informed decision about whether they want to participate. This information must be distributed in such a way that the potential subjects can genuinely understand what their participation entails. Those unable to provide consent may be Essentially, informed able to do so if a third party is able consent is about making to act in that person's best interest sure that subject and potentially provide informed ja . consent on their behalf. And as _ Participation is genuinely always, special consideration must _ voluntary: Participants be given to vulnerable populations. must be able to choose to be involved, and they also The proposed ch stb a propo researc mu “ must be free to withdraw justifiable as maximizing societal ° naan benefits and minimizing risks. In _ their participation at other words, there must be value to —_ any time without facing society for the research: You can’t consequences for doing so. experiment on people purely to satisfy your own curiosity or to gain personal benefit. All kinds of harm need to be considered— not just physical harm, but also psychological, mental, legal, economic, social, and reputational harm. This entails protecting the identities of your subjects and even keeping the fact that they participated in your study confidential. You may need to have procedures for| Doing Research Ethically Sometimes you might not want your subjects to know everything about your study before you start, because this might influence how they respond in your experiment. Deception is sometimes necessary for research to work. The Belmont Report says that when deception is needed for the research to be valid, the subjects still need to know of any potential risks. If the deception poses additional risks, that can’t be withheld from them. You also must debrief the participants at some point and tell them about the deception. securing your data, particularly if it contains identifying information. If your research does pose a risk, or is on vulnerable populations, then you must be extra cautious. There must be fair procedures and equitable outcomes in how subjects Many US-based are selected for research. You can’t pharmaceutical offer the benefits of your research to companies have particular people or groups and deny ° them to others on the basis of personal moved their clinical preference. Justice requires that ifcertain Studies overseas populations are overly burdened and because itis cheaper the research you wish to do could pose and easier to find aoe non harm in those areas, then ey participants. And urdened populations should be calle . on first as subjects for the research, While they are still As always, vulnerable populations Supposed to comply deserve special attention and should — with US laws, that only be included in the research if their _doesn'talways participation is absolutely necessary for happen. It’s not h , - the study's success, always clear if every The Belmont Report is an important regulation applies ifit milestone in the establishment of ethical differs with the law in principles for research, but it Colona with another country. And the government’s Common Rule) applies only to the United States. Research [nsomecases, the outside the US may have different rules, _ Tules may representa so if your research is international, you culture clash.Effective Research Methods for Any Project need to consider the rules in your host country, as well as the potential benefits and harms that might be unique to that local population. Also, just because some practices are permitted by law, that doesn’t mean they are ethical. Follow the principles of respect for persons, beneficence, and justice regardless of where you conduct your research—but realize that those principles may entail additional protections depending on where you are. ETHICAL BEHAVIOR OF RESEARCHERS Regardless of the type of research you do, you must consider how to behave ethically as a researcher. Don’t make up data. Falsifying or distorting data to make your results look better or support a story you want to tell is an incredible ethical violation. Don’t put your personal gain above those you study. One way to avoid this is to craft your studies so that no matter what the data show, you have something interesting to say. This will limit the temptation to fabricate results. If you have any conflicts of interest or reasons for bias, disclose them. It is important to be honest about any commitments in your life that might prevent you from being fully objective. Always acknowledge any sources of funding for your work so that your audience can make their own determination about whether this might cause bias in your work. Be sure to properly attribute and protect the intellectual property of others. If you use any words—or, perhaps more importantly, ideas—of another researcher, you must give that person proper credit. Cite thoroughly and often; you never want your reader to have to guess which ideas are yours and which you borrowed. Be clear in your own head about your own contributions and make sure you properly give credit to others.| Doing Research Ethically INSTITUTIONAL REVIEW BOARDS Research in the United States on humans that is conducted by the government or any entity that receives government funding, directly or indirectly, must be approved by an independent ethical review board, usually called an Institutional Review Board (IRB), whose job is to make sure that the research you want to do follows the principles of The Belmont Report. In 1981, US federal agencies started requiring IRBs to engage in oversight of research. IRBs can be based in universities or other institutes of research, or they can be independent or focused on commercial research. All must be registered with the federal government. IRBs are not restricted to the US, however; they are found all over the world, although the specific regulations may differ by country. If you want to do Not every researcher necessarily needs IRB any beseech on approval. If you are doing a personal project human subjects for a local organization and want to conduct —_ and your workis in a survey of your membership, you probably any way funded by don’t have to get IRB approval first. But the US government, that doesn’t mean you are free to violate you must have IRB the principles of The Belmont Report. Always take care to protect the rights of approval before you your subjects, even if you don’t have to get _ Start yourresearch. permission first. Some categories of research are exempt from IRB approval. These include some educational research, research involving publicly or anonymously available records, and research evaluating public programs, as well as a few other areas that are deemed to be very low risk. However, you don’t get to decide that your research is exempt. Unless you're doing research that obviously needs no approval, such as a personal project for a local group, you still have to submit an application to the IRB. They are the ones that will determine that your research is exempt.Effective Research Methods for Any Project You aren't allowed to begin data collection until you have IRB approval. You aren't allowed to have any contact with your subjects until you’ve convinced the IRB that you are aware of the potential risks your subjects may face and that you are prepared and able to protect your subjects from harm, including psychological, emotional, and reputational harm Once you file your application, it will be reviewed by the IRB and designated to one of 3 categories: exempt, expedited review, or full board review. Exempt research is that which by its nature imposes no foreseeable risk to subjects. Expedited research poses minimal risk to subjects, meaning that the harm they face is no greater than that which they might experience in their normal, daily life. A full board review covers everything else, including anything that involves intentional deception, invasive procedures, sensitive or stressful topics, or potential for more than minimal harm to the subjects. Regardless of the level of the review, the IRB will evaluate the research proposal to make sure that risks to subjects have been minimized, that the researchers are ready and able to care for their subjects and their data to ensure minimized harm, and that any potential harm is outweighed by the potential benefits of the research. Ifyou violate the basic ethical principles outlined in this lecture, consequences can vary. If you've published your work, it may be retracted. You or your institution might lose federal funding. If you violate ethical principles after an IRB has approved your research, the IRB may lose its standing and ability to grant approval to other projects. You could be reprimanded, suspended, or fired. Certainly, your reputation will take a hit.LECTURE 3 | Doing Research Ethically READINGS Granberg and Galliher, A Most Human Enterprise. Krathwohl, Methods of Educational and Social Science Research, ch. 10. Nakray, Alston, and Whittenbury, eds., Socia/ Science Research Ethics for a Globalizing World. National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research, The Belmont Report. Zimmer and Kinder-Kurlanda, /nternet Research Ethics for the Social Age.From Topic-of Interest to Research Question| From Topic of Interest to Research Question ost research starts with a topic. But while starting with a topic is fine, you should not end with a topic. Ultimately, you need to come up with a question that is worth answering. Moving from a topic to a question is an important step in the research process. SELECTING A TOPIC How do you go about selecting a topic in the first place? Sometimes a topic is chosen for you. You might have been asked by a supervisor to evaluate a particular program. Colleagues may have brought you in on their project, or a client may ask for assistance in solving a problem. But what if you need to do research and produce a paper, presentation, talk, or poster; you have complete control over the topic; and you don’t know where to start? The best way to start is to think about something that interests you or your intended audience. Whatever topic you choose, all things being equal, you want it to be something that you genuinely want to explore in great depth. Otherwise, you might find yourself prone to procrastination and lackluster effort. If nothing comes to mind immediately, then try a quick exercise. Take out a piece of paper and make 2 columns, one of which you label “topics | know a lot about” and the other you label “topics I'd like to learn more about.” Write down as many as you can think of in both columns. Be as broad or as narrow as you like, and write down anything that comes to mind; this is a brainstorming session, so don’t censor yourself right now. Then, see if there are any natural connections between any 2 items. If you still are having trouble finding something, then look to recent research in your field. Skim through the table of contents in academic journals, a conference program, or the recent publication list of some publishers. See what kinds of topics are pursued by recent authors and note which ones sound the most interesting to you.Effective Research Methods for Any Project If you really find yourself struggling to find a general topic, you might want to jump straight to a research question, as you might find that easier. CHARACTERISTICS OF GOOD RESEARCH QUESTIONS While the specifics may vary from one field to another, good research questions tend to have 3 main characteristics: They are unanswered, appropriate in scope, and empirical. What constitutes a good research question can depend on your discipline and field. Many projects call for describing historical events in detail. In other projects, however, you might want to focus a description of events on causes and explanations. In such a case, your question should not ask what happened but why it happened. Good research questions are unanswered. Research— whether it is scientific, historical, journalistic, or something else—aims at producing new, original knowledge or answers. That new knowledge might be based on original data or new interpretations of existing data. As you explore the literature, you may discover that someone has already answered your question. That may mean you need to find another one, but just because someone else has written one answer to your question doesn’t mean your question has been definitively answered. There is lots of room for debate in research-based fields. Read the other researcher's work and evaluate it. You may find that you disagree with his or her assumptions or conclusions. Maybe his or her answer applies in some cases, but not yours. All this means is that your area of interest is further along in the research process than you thought—and your incremental contribution starts a bit further down the line. The idea that good research questions should be unanswered means that your question should be about something that doesn’t have an easy, factual answer. Also| From Topic of Interest to Research Question It’s normal to start with answered, make sure that your question allows you to enter a debate, solve a puzzle, fill a gap . in knowledge or understanding, solve a __ factual questions problem, or otherwise make a contribution so you can gather to your audience. The best way to do the information this. is by doing a literature review, which necessary to ask a basically means you need to read through what other scholars have already written good, unanswered ‘on this subject so that you can figure out question. where your ideas can fit in. Good research questions are appropriate in scope. Sometimes you need to adjust the anticipated product to meet the scope of your question. Sometimes one research paper becomes 2 or a program-wide evaluation spurs into individual evaluations that are narrower in scope. If this happens, make sure that changing theEffective Research Methods for Any Project ultimate product is fine with the editor, client, supervisor, or whoever else is expecting it. If it’s not, you might have to limit your approach to the expected product or adjust the size of your product so you have the space to consider all aspects of your question without sacrificing necessary detail. Another aspect of scope has to do with considering your own abilities and limitations. You need to think about the kinds of data that you would need to answer your question. Does it exist? Do you have the time and resources to acquire it? Do you have the skills to interpret it? You have to consider your training, skills, and resources. Time is always a factor: You may have a set deadline and want to make sure whatever question you choose is one that can be answered in the time frame you have. Other factors to consider include whether or not your project requires extensive work with numbers or people; you may have to adjust your research question if you dislike interviewing strangers or if working with numbers scares you. Finally, is your question going to require funding for you to be able to complete the project? Projects that are large in scale or require specialized equipment or personnel often need grants. Good research questions are empirical. Empirical research questions ask how the world works. They are based on events, phenomena, and actors that can be observed in the world. While empirical work is about objective processes that aim at observing how the world actually works, normative questions and claims are subjectively based judgments about how you might want the world to work. Normative questions are those that proscribe or judge behavior, By their nature, these questions are asked from a subjective standpoint and tend to produce subjective answers. This is fine in your everyday life. But when it comes to research, it can be a problem. If you base your findings on what you hope to be true or what you think might be good or bad, then you are likely going to cherry-pick information to support your own ideas and conclusions. It is also very difficult to provide evidence| From Topic of Interest to Research Question Anorm is a shared code of behavior. It’s a set of usually unwritten rules a group has for interacting. All members of the group know the rules and often don’t even think about them even though they regularly obey them. There are norms about everything—behavior in bathrooms, eating, standing in lines, greeting people, and much more. You can violate norms, but there may be social, if not legal, consequences for doing so. Norms change by location, culture, group, and time. fora Subjective statement. None of this Plenty of research means that normative questions are i i i bad. Some fields, such as ethics, focus pects aim Se primarily on normative questions. But at making predictions in terms of using the scientific method, about the future, but you cannot focus solely on normative _ to do that, you usually questions because you need to make need to understand observations about the world—in other the world of the past words, engage in empirical work—to . get data. Normative questions are often and present; in other essential to motivating your work, but —_ words, you need to do to answer these questions, you have to —_ empirical work that ask empirical questions, too. Only after allows you to make empirical research can you answer the those predictions. normative question. Ask normative questions, as they motivate your work and help you develop recommendations for action, but then move quickly to the empirical question that will allow you to effectively answer the normative question. That is essential to solid scientific research.Effective Research Methods for Any Project TIPS ON WRITING A GOOD RESEARCH QUESTION Start your question with the 5 Ws and H: who, what, where, when, why, and how. Who, where, and when tend to be more sparsely used, as these might lead you to ask previously answered, trivial questions. But for comparative, action, or evaluative research, these can work just fine. For example, ask, “Who are the people most harmed by this policy?” Why and how are the best for asking questions about causes and outcomes. What is a great starter for questions about policy, practice, and scope. For example, ask, “What kind of policy might solve X problem?” Try to avoid yes/no questions. The key to all of these question starters is that they are hard to answer with a simple yes or no. Yes/no questions are fine in some situations but can often lead you down the path of trivial answers. Typically, once you know that an answer is yes or no, you immediately want to know why. So, instead of asking, “Is a carbon tax the most effective policy for addressing climate change?” you could ask, “What is the most effective policy for addressing climate change?” Then, no matter what the answer is, you have something interesting to say. That also helps you avoid ethical issues about wanting to try to prove a particular answer correct. Don’t make assumptions about the answer. Avoid writing a question that presumes a specific answer. For example, you could ask, “Why are eggs a healthier breakfast option than cereal?” But that presumes that eggs are healthier. Depending on what you mean by “healthier,” that may or may not be true. Instead, ask, “What is the healthiest breakfast food option?” Eggs might be the answer—but you aren’t presuming that from the start.| From Topic of Interest to Research Question Don’t complicate things unnecessarily. Your question might be “Is my program meeting its outcomes?” or “What are the causes of this phenomenon?” You might revise your question later to be more specific, but you don’t have to write the perfect question from the outset. Do the best you can, but get something down on paper s0 you can start doing a literature review. You will have plenty of opportunities to revise your question as you learn more. Remember, it’s better to answer a narrow question well than answer a large question poorly. Use the literature. The scholarly literature on a topic is a great resource. A lot of scholarly works will end with the authors suggesting questions for future research. You can look to those for ideas on what your questions should be. The work of journalists, policy makers, and entrepreneurs may also inspire you. READINGS Henderson, “Norms.” Krathwohl, Methods of Educational and Social Science Research, ch. 5. Powner, Empirical Research and Writing, ch. 1.A What’s Already Known? The | Literature Review| What's Already Known? The Literature Review he literature review, often referred to as the lit review, is where you dig deep into what scholars and other experts have already learned on your topic. Then, you can build your work on top of it. This lets you add to the conversation with the assurance that you are making a genuine and valued contribution. WHAT TO LOOK FOR Research is conducted at least twice in any study. The second time is when you get your actual data—the statistics, surveys, experimental results, reports, or historical analysis that will let you assess your claims. It occurs much later in the process, after you've designed your entire study. The first time is when you review the scholarly literature on your question and find out what is already known on the subject. The body of scholarly work on a given question or topic is called the literature. It's the work that scholars and scientists have completed and published ina particular area. Aliterature review, then, refers to both the process of finding and evaluating this body of work for your chosen topic or question as well as the section of your report or presentation where you share your conclusions about the literature. First, you find the literature, read it, and then draw conclusions about what you do and do not know. Later, you will produce a section of your report—written or oral—that explains the key claims of the literature: the themes, debates, gaps, methods, and overall conclusions drawn from this review of scholarly work. Conducting a literature review—and demonstrating you have done so by including it when you share your work—is required to have your work taken seriously by the research community.Effective Research Methods for Any Project You need to understand what ideas are out there and their strengths and flaws before you are able to advance understanding of the world. If you don’t do the literature review, then you don’t even know whom you are challenging. There are a number of practical things you can learn from reading the work of other scholars that will help you build your own study. You can learn where mistakes were made and identify best practices that will help you build your research design. Specifically, you will find common and accepted definitions for the concepts and variables you plan to use. You will learn which concepts and variables are important in the first place. Many ofthe You will see which cases have been studied —_ questions you and which haven’t—perhaps providing a clue —_ might have for where you can make a contribution. You'll about designing learn what kinds of qualitative or quantitative your study can methods are generally employed to study this n question and what sources of data are out beansweredina there. You might even find an unanswered _ thorough review research question or untested hypothesis that of the literature. you can tackle. Reviewing the literature also helps you in bigger-picture tasks, too. Collectively, the literature tells you about the state of theory and practice in this particular area. It can help you identify the debates scholars have, the scope and limitations of existing theory, unexplored gaps of understanding, practical examples where policies and programs have or have not worked, and connections to other fields or areas of study. A thorough read of the literature is what starts many researchers on their projects; they see something that other scholars have missed or a way to resolve an existing debate, and that’s their point of entry into the discussion. 40| What's Already Known? The Literature Review HOW TO FIND THE SOURCES YOU NEED Scholarly sources are the key to any literature review. A scholarly source is a book or journal article that is written by an expert, is written for other scholars, and has undergone peer review. In other words, other experts have evaluated the work and deemed it worthy of publication, usually in a double- blind process, which means that neither the author nor the reviewers know the identities of those involved. The goal is to ensure that the work is evaluated based purely on merit. Typically, nonscholarly sources are avoided in the literature review. A nonscholarly source is a book, article, or other source that has not undergone scholarly peer review. Examples would be a newspaper article, magazine, blog, social media post, or self-published book. Even if the author is an expert, if the source is not peer reviewed, it’s not typically considered a scholarly source. Some of these sources can still be useful, but they are only rarely considered part of the literature.Effective Research Methods for Any Project These 3 criteria are used to identify a true scholarly source. Scholarly sources are written by experts. Look at the author’s credentials. The author will typically hold a terminal degree in the field—that is, the ultimate degree that one can obtain—or be in the process of acquiring one. The author also will usually have an affiliation with a university, think tank, or reputable research-based agency or organization. If you can’t find any information about the author's credentials, or they don’t meet that level, then it’s probably not a scholarly source. Scholars write scholarly sources for other scholars. If the intended audience is the public, students, government officials, company executives, or friends and family members, then it’s usually not considered a scholarly source. If there’s a lot of indecipherable jargon and charts; few pictures; no colorful, glossy paper; and citations and footnotes galore, then you've probably found a scholarly source. This will vary by discipline, but scholarly sources generally have a high bar of entry for the average reader. Scholarly sources have undergone peer review. A good way of determining whether this is the case is by examining the publisher. Is it a university press like Princeton, Oxford, or Stanford? That’s usually a good sign. Likewise, an academic journal such as Chemical Reviews will conduct peer review of all submissions. If you review the publisher information, it should have a clear section that discusses its review process. All 3 criteria must be met for a source to be appropriate for the literature review. WHERE TO LOOK FOR SOURCES One way to find the sources you need is to find just one or 2 highly relevant and recent journal articles on your topic. As previously mentioned, scholarly articles are usually full of 42| What's Already Known? The Literature Review citations and footnotes, so why do the work of tracking down a ton of citations when other scholars have already done that work for you? Find one or 2 really good, recent articles on your research question and you've got the key to the core of the literature. Go to the bibliography and track down the articles there. Then, do the same with those articles. Keep going, and note which articles keep popping up in the bibliography. The ones that keep showing up are likely the core of the literature ‘on the subject. Once you start doing a lot of research in a particular area, you'll get to know who the big names are and which articles or books are considered the most central to the current debate. But when you are just getting started in a new area of research, this snowball method of finding sources is really useful. Another thing you can do is check the number of citations for a given article. Google Scholar will report that information if you do a search using the article’s title. A high citation count means that the article has been consistently cited by other scholars in their work—a good sign that you've found a core part of the literature Another option is to ask a mentor. Your mentor may already know the big names in the field and can direct you to some of the core works you need to know. To find the one or 2 articles in the first place, you have a few options. You can head to your local university library, if there's ‘one near you, and browse the stacks of bound journals. If you go to the website for the professional association in your field, it should identify some of the top journals in your discipline. You could pull out a recent edition and start browsing until you find a relevant article. Searching databases on the internet might be quicker, though. Try a catchall database like EBSCO, Academic Search Premier, or JSTOR. Enter a few relevant search terms re)Effective Research Methods for Any Project and narrow the responses to scholarly sources. You can use the provided subject headings in an article’s description to further narrow your results. That's great for anyone with access to a university library, but many people don’t have that option. Still, you'll be able to find some sources available through free search databases, such as Google Scholar. There are also subscription services that let you rent journal access on a monthly basis, such as DeepDyve, or ones like Academia.edu that allow authors to upload their papers. Your public library may have some resources, and it’s possible that you may have alumni access privileges at an institution you previously attended. Some journals, particularly those in the natural sciences, have moved to open-access formats, allowing free public access to their publications. WHAT TO DO WITH YOUR SOURCES Journal articles can seem difficult to decipher at first, but they follow a fairly standard format once you know what to look for. In August 2017, the journal Science published an article called “Protecting Unauthorized Immigrant Mothers Improves Their Children’s Mental Health.” The authors ask their research question in the first paragraph: They want to know the impact of a parent’s unauthorized status on a child’s well-being. The last sentence of the first paragraph begins the literature review section of the article, as the authors note that their study is filling a gap in existing literature. The literature review continues for the next 3 paragraphs, as the authors note relevant studies that connect parental immigration status to child development and note the methodological issues that have prevented a full understanding of this question. They then outline the nature of their study—the subjects, sources of data, specific variables used, method of analysis—before turning to their results. They close with the implications of their results and possible steps for future research.| What's Already Known? The Literature Review It’s important to take notes during this process so you can find your way back later to any sources you found and any important ideas that occurred to you along the way. First, before you write down the source’s research findings, note its bibliographical information as well as the page or paragraph number of any material that interests you. If you want to save yourself time later, upload the information into a citation manager system—now included in many word processing programs. That will create a bibliography with a single click. But for now, just make sure you know exactly where each idea and finding came from so you don’t have to go back and Doing the literature look it up again later. review up front can Next, for each article or book, take save yougtonoLume notes on its claims, research design, 4nd frustration later. and implications. Note exactly what the authors claim to contribute as well as the details of their research methodology used to evaluate those claims. Try to paraphrase whenever possible. As you read, you may question the process or results, particularly as you comb through multiple articles on the subject. Note those issues, too. Also note any connections or contradictions to other articles you've read. At this stage, it is really important to clearly distinguish between what the authors are saying and what you think about what the authors are saying With the Science article, you would note the full citation and then mark the research question, the gap it claims to be filling, the methodology the authors used, and the key finding. If you have any questions, ideas, comments, or connections to note, put those in your notes as well, but use a different color, highlight them, or bold them to make it clear that it is your analysis and commentary and not what’s found in the article itself. Repeat this process with all of your sources. Depending on the size of your project, you may consult 15, 50, or 100 or more sources. You want to be comprehensive; use the sources you are reading as a guide. If they are citing an average of 25 sources, then that’s a good number to aim for. rtEffective Research Methods for Any Project HOW TO DRAW AND PRESENT YOUR CONCLUSIONS The final step is to draw and eventually present your conclusions about the literature. Remember that the literature tells you about the state of knowledge on your chosen topic. That means the point is not simply to report that scholar X said this and scholar Y said that. A list of sources and their contribution and findings—the notes you just took—is called an annotated bibliography. It is a source-by-source list of the important characteristics and findings of each book or article. It is an initial first step to writing a literature review, but the literature review itself requires you to go further. Instead, you need to identify the core findings in the literature. This means you have to explain what is known about the answers to your research question. This might be consensus on a set of definitions, or a methodology that scholars tend to use to study the question, or agreement on which range of theories are most germane to the subject. But you also have to explain what is unknown. If everything is already known, then there is no need for your project. There are a lot of comparisons to make as you synthesize your sources and write your literature review. You might look for common themes across the sources that keep popping up, or you might look for gaps—issues or ideas that the authors seem to miss but that you think are very relevant and perhaps worth exploring in your own project. Another way to organize your thoughts is to look for debates between authors. Where do they disagree, and is there room for you to enter the debate with your own contribution? Are the different theoretical perspectives you encounter in contradiction to each other? If so, that might be a way to organize your review of the literature.| What's Already Known? The Literature Review One last thing to look for are methodological choices and debates. You might note that the dominant methodology used to study similar questions to yours is quantitative— which means that a qualitative approach might provide new insights. This means that a literature review is not simply a paragraph about what each author has to say. Instead, you should focus your writing by one of those methods, using themes, gaps, debates, or methodology to organize your thoughts. You cite sources as a type of evidence to support your argument that this theme, gap, debate, or methodology is a useful way of understanding the current scholarly approach to your topic. READINGS Adams, Khan, and Raeside, Research Methods for Business and Social Science Students, ch. 4. Garrard, Health Sciences Literature Review Made Easy. Powner, Empirical Research and Writing, ch. 3 Walliman, Social Research Methods, ch. 5. 47Generating Hypotheses and Theories| Generating Hypotheses and Theories enerating hypotheses and theories is typically the third step in the process of doing research after identifying a problem or question and reviewing the literature. At this point, it’s time to start considering possible answers. THEORIES Theories are generalized explanations for what we observe in the world. They play an important role in driving basic research. Observation tells us what to expect. We all know that what goes up must come down. Theory helps us understand why that is the case. Research isn’t always about building or testing theory. Sometimes you just want to observe or describe, rather than explain or understand. Sometimes, as in action research, your goal isn’t to explain broader patterns but to find solutions for specific people, groups, or organizations. But a large swath of research is focused on explanation. And frequently, understanding why can help you apply ideas to new cases and events—and even let you make predictions. Any time you are interested in why or how things happen or want to fit an event into a larger pattern or trend, you are going to want theory. Theories have to be tested by data. While you can use inductive reasoning to form theories from observation and data, there are dangers if you stop there. It can lead to data fitting—basing your theories on the data you have, which may not be comprehensive. Instead, you observe, form theories based on those observations, and then test those theories against new data. Parsimony is valued in theories. This is a quality drawn from the Occam’s razor principle: that the simplest explanation is usually the best. Parsimony means that when everything else is equal, simpler explanations are preferred to complex ones. The idea is to maximize the story that a few variables can tell. If you can explain some outcome using just one or 49Effective Research Methods for Any Project 2 variables, that is considered a stronger theory than if you need 10 or 12 variables. But you don’t want to oversimplify and lose the causal power of your theory. If the explanation really does require those 10 or 12 variables, then it’s better to keep a complex-but-accurate theory than to go with a simpler-but-less-accurate one, To develop a theory, start by thinking systematically about how and why one variable of interest affects another variable of interest. Theory is often about unwrapping the causal mechanisms that get you from point A to point B. To develop your theory, first look to the literature. See what kinds of explanations are already offered and examine whether the evidence in the literature supports those explanations. Are there gaps that aren't explained? Cases that seem to contradict the general findings? Debates that are not yet resolved? All of these are fertile grounds to spark your own ideas. Another way is to engage in some observation. At an early stage of your research, examining data can be a useful way to generate theories as well as hypotheses. Eventually, you will want to subject those theories and hypotheses to proper tests, but initially you need some knowledge on which to base your theories. The key is to ensure that your theory helps answer your research question, aims at explaining general principles— not just specific cases—and advances understanding of the phenomenon of interest. HYPOTHESES Hypotheses are testable statements that provide the researcher's best guess at an answer to a research question. Hypotheses are typically derived from theories, although sometimes it works in reverse and testing hypotheses helps researchers build theories.| Generating Hypotheses and Theories In science, researchers set out to disprove, not prove. When you prove something to be true, that means it’s final and definitive. In many research areas, almost everything is uncertain and conditional. Researchers are rarely 100% positive about anything. You find evidence that supports a theory or hypothesis, or refutes it, but there is no single piece of evidence that can prove your ideas correct. Instead, you set out to disprove ideas. It’s what you are left withthe ideas that have been tested but not disproven—that form the foundation of scientific understanding. Most research is actually testing what is called the null hypothesis rather than a hypothesis itself. Every hypothesis has a null hypothesis. A null hypothesis simply states that no relationship exists between the variables. And that is what you are actually setting out to disprove. If you can disprove the null hypothesis, then that means that some kind of relationship does exist—perhaps the one you hypothesized or perhaps something else you weren't expecting. fin your study you find evidence that supports your hypothesis, note that this does not mean that you are automatically correct. You may be right, but you have not proven your case. Why is it that if you find evidence to support your hypothesis, you shouldn't assume your hypothesis is definitely correct? It’s possible that it’s only correct for whatever data you looked at. It might be that if you had more data—or data from different sources or data over a longer time frame—the results would be different. Your data might be incomplete, missing, or flawed. There might be alternative explanations that account for the relationship you suggested.Effective Research Methods for Any Project Theory and hypothesis intersect closely. Theory, by increasing your understanding of why and how things happen—and sometimes the causal mechanisms underlying the why and how—can also help you assess your hypotheses and findings. HOW TO WRITE A STRONG HYPOTHESIS Some research focuses just on developing theories—and that’s fine. But ultimately, if your goal is to advance knowledge or to be reasonably confident that your ideas are correct, you need to examine empirical evidence. That doesn't always require a hypothesis to test; sometimes you are looking for themes, or describing your findings, or engaging in critical analysis. Other times you are doing research to figure out the variables that might play a role in affecting your outcomes of interest. You can’t really write a hypothesis, because you don’t yet know what variables to use. This is completely reasonable; hypothesis testing might come later, or not at all. So, if your project doesn’t lend itself to hypothesis testing, that’s fine, but a lot of research does involve constructing tests to evaluate hypotheses. And having a hypothesis can be an advantage, because it provides guidance for your research. Just as a research question provides more guidance than a simple topic, a hypothesis can help guide you toward a specific research design and provide a framework for your analysis. If your research is focused on developing and testing hypotheses, consider these 4 rules for how to write a strong hypothesis. Hypotheses are typically probabilistic. If you state a hypothesis in an all-or-nothing sort of way, then there’s no room for nuance, random chance, or exceptions. To write hypotheses probabilistically, that usually means using words like “more likely” or “less likely” rather than “always” or “never.”| Generating Hypotheses and Theories Hypotheses are written in a generalizable way—that is, in such a way that they apply to broad groups or concepts, not specific ones. Whenever possible, hypotheses should state a specific proposed relationship between variables. This means that your hypothesis should specifically state something you expect to find when you examine the empirical evidence. There are generally 2 broad categories of relationships: correlation and causation. © Inahypothesis that focuses on correlation, you are stating that you expect to find a relationship of some kind between 2 variables. You might indicate what kind of relationship you expect to find. Do you expect that as one variable increases so does the other? Or do you expect the second variable to decrease? © Ina hypothesis that focuses on causation, you are investigating whether or not one variable causes change in the other variable. In other words, you aren’t just looking to see whether there are patterns in the values of 2 or more variables. You are looking to see if, specifically, change in one variable causes change in another. The variables that you think are affecting some outcome of interest to you are called independent variables, usually referred to as x. In an experiment, x is the variable you are manipulating. For example, it is the drug that you are giving to subjects in medical research The other type of variable—the outcome, or effect—is called the dependent variable, usually referred to as y. This is whatever phenomenon you are studying— the thing that is changing and you want to understand why itis changing. In a study where you administer drugs for medical research, y is whether or not the medical condition improves.Effective Research Methods for Any Project Austriap-born fans Perse Oyred if sifiability is Talore laren vat distinguishes CitecRicoaa Strong hypotheses should be falsifiable; that is, they are written in such a way that they can be disproven. There are a few templates you can use to construct your hypotheses that follow these 4 rules. They won't fit for everything you want to study, but they can be a good help to get you started. Value 1 of x is more likely to lead to value 1 of y than value 2 of x. This template says that one category—or value—of one variable, x, is creating a particular effect on a second variable, y, and some second category of x is not having that effect. You are stating a specific relationship between the variables: x, is having this effect, but x, is not. Both x, and x, are values on your independent variable. This clear| Generating Hypotheses and Theories relationship gives you something to test. Sometimes you have more than one x to consider; you can always add more x’s into your template. There is a [ ] relationship between x and y. This template is easier to write, but you are more restricted in when you can use it. Fill in the blank with “positive,” “negative,” “curvilinear,” or other descriptors of the relationship and you are done; there is no messing around with values. A positive relationship is one where the 2 variables vary in the same direction: As one goes up, the other goes up—and as one goes down, so does the other. Anegative relationship is one where the 2 variables vary in opposite directions: As one goes up, the other goes down, and vice versa. There are 2 restrictions to using this template. First, these kinds of hypotheses only establish correlation, not causation. If you just want to test whether 2 variables are related to each other, then this can work fine, but if your goal is to figure out if one causes the other, this hypothesis style doesn’t clarify that for you. Second, for this style to work, your variables need to be at a certain level of measurement; that is, it needs to make sense that a variable can go up or down—increase or decrease. Some variables you may want to study don’t make sense in that way. There are plenty of other ways to write hypotheses. Don’t feel bound to these templates. But when you are just getting started, it helps to have a format to follow, so use these as a starting point.Effective Research Methods for Any Project Science in theory and science in practice can differ widely. Researchers set out to disprove theories and hypotheses and design tests based on hypotheses. In the real world of research, though, this doesn’t always happen; there are a lot of incentives in place to not do this. For example, it’s much more difficult to publish negative results than positive ones. In other words, if your evidence confirms your null hypothesis and rejects your hypothesis, that is very important to know—and exactly the kind of falsifying that Karl Popper would applaud. But it’s not as interesting or appealing as results that support a hypothesis, and therefore it is much more difficult to acquire grant funding for such research or to get it published. It's also true that sometimes scientists will just run various models on their large data sets and then when they hit on a finding, pretend that this was their hypothesis all along. This can lead to data fitting, and it doesn’t necessarily advance the state of knowledge, But it can be an effective way to get results that can be published. And for many researchers—particularly academics at big research universities—getting published is one of their primary goals. Their careers depend on it. There are strong reasons to follow the rules of scientific inquiry when it comes to research. Ethics demand it, and for many of the projects you are doing, itis just as important to know what doesn’t work as what does. There are always going to be incentives to avoid or bend the rules. Resist those incentives. READINGS Powner, Empirical Research and Writing, ch. 2. Thornton, “Karl Popper.” Walliman, Social Research Methods, ch. 6.Selecting a Research DesignEffective Research Methods for Any Project he design of your research procedures encompasses your overall approach to research as well as the specific components of how it will be carried out. There is a wide range of approaches to research design, and there are some key factors to consider when choosing an overall approach for your project. There are many ways of categorizing different research methods, but a straightforward and common way of organizing these methods puts them into 2 categories: quantitative and qualitative. © Quantitative analysis generally involves using statistical tools to analyze large amounts of numerical data on a large number of cases. Quantitative designs tend to produce data useful for quantitative analysis. © Qualitative analysis typically looks at a smaller number of cases more in depth, using data that can’t always be reduced to numbers. Qualitative designs tend to produce data useful for qualitative analysis. QUANTITATIVE APPROACHES TO RESEARCH Quantitative approaches include experimental designs; observational designs, such as _ cross-sectional and longitudinal studies; and large-scale survey research. Experimental designs—proper, controlled . experiments, with random assignment, Experiments are controls, and treatments—allow you to Widely usedin the be very precise about the connections __ natural sciences between your variables. They let you and psychology, do all kinds of things that other designs and the principles of don’t; you can control how variables are . I desi manipulated for your subjects, letting you experimental esign really isolate the impact of variables that Can be applied to you think matter. The researcher has alot _ surveys and other of control over how the data is collected. _ kinds of studies.LECTURE 7 | Selecting a Research Design Because of this, if you can answer your research question using a proper experiment, you probably should. But plenty of questions and projects do not lend themselves to experimenting, so you might need to pick another method, such as cross-sectional or longitudinal studies. With these observational designs, you can’t control your subjects the same way you can with experiments. Instead of manipulating variables and seeing what happens, you have to simply observe the world as it is and figure out what you can about how it works. The typical approach is to take a large data set with information on a host of variables and conduct statistical analysis to look for correlation or causation. Cross-sectional studies examine a set of cases at a single point in time, as if you were making a cut across space to look at a wide cross section of a population.Effective Research Methods for Any Project Longitudinal studies look at a case over time. They let you study trends and changes over time by gathering multiple points of data about a case over a period of time. This can be particularly useful as variables change in value. You can combine both of these methods and do a design that is both cross sectional and longitudinal. In such a study, you would compare different units to each other over time. You can also do correlational studies that focus on looking for relationships between variables, often in existing data sets or ones that you build Large-scale survey research is actually a type of cross- sectional design, but it is important enough in its own right to be discussed separately, as surveys are a commonly used tool in research. In survey research, you distribute questionnaires to large numbers of people to determine their characteristics, behaviors, attitudes, values, or beliefs. Rather than experimenting on people and observing the results, you are letting people self- report and then examining what they say. You might also compare groups across demographic indicators, which is why many surveys ask about respondents’ age, gender, race, income, and education level QUALITATIVE APPROACHES TO RESEARCH Qualitative approaches generally focus on a smaller number of cases. Quantitative methods tend to sacrifice depth for breadth; you don’t generally know a lot about any single case or subject, but the high variation in subjects generally gives you more room to draw conclusions about a wider population. Qualitative methods, on the other hand, allow for richer description and can be better at identifying causal mechanisms that help you understand why one variable might create change in another.