explore the scientific problem solving process

How it works

For Business

Join Mind Tools

Article • 5 min read

Using the Scientific Method to Solve Problems

How the scientific method and reasoning can help simplify processes and solve problems.

By the Mind Tools Content Team

The processes of problem-solving and decision-making can be complicated and drawn out. In this article we look at how the scientific method, along with deductive and inductive reasoning can help simplify these processes.

‘It is a capital mistake to theorize before one has information. Insensibly one begins to twist facts to suit our theories, instead of theories to suit facts.’ Sherlock Holmes

The Scientific Method

The scientific method is a process used to explore observations and answer questions. Originally used by scientists looking to prove new theories, its use has spread into many other areas, including that of problem-solving and decision-making.

The scientific method is designed to eliminate the influences of bias, prejudice and personal beliefs when testing a hypothesis or theory. It has developed alongside science itself, with origins going back to the 13th century. The scientific method is generally described as a series of steps.

• observations/theory
• explanation/conclusion

The first step is to develop a theory about the particular area of interest. A theory, in the context of logic or problem-solving, is a conjecture or speculation about something that is not necessarily fact, often based on a series of observations.

Once a theory has been devised, it can be questioned and refined into more specific hypotheses that can be tested. The hypotheses are potential explanations for the theory.

The testing, and subsequent analysis, of these hypotheses will eventually lead to a conclus ion which can prove or disprove the original theory.

Applying the Scientific Method to Problem-Solving

How can the scientific method be used to solve a problem, such as the color printer is not working?

1. Use observations to develop a theory.

In order to solve the problem, it must first be clear what the problem is. Observations made about the problem should be used to develop a theory. In this particular problem the theory might be that the color printer has run out of ink. This theory is developed as the result of observing the increasingly faded output from the printer.

2. Form a hypothesis.

Note down all the possible reasons for the problem. In this situation they might include:

• The printer is set up as the default printer for all 40 people in the department and so is used more frequently than necessary.
• There has been increased usage of the printer due to non-work related printing.
• In an attempt to reduce costs, poor quality ink cartridges with limited amounts of ink in them have been purchased.
• The printer is faulty.

All these possible reasons are hypotheses.

3. Test the hypothesis.

Once as many hypotheses (or reasons) as possible have been thought of, then each one can be tested to discern if it is the cause of the problem. An appropriate test needs to be devised for each hypothesis. For example, it is fairly quick to ask everyone to check the default settings of the printer on each PC, or to check if the cartridge supplier has changed.

4. Analyze the test results.

Once all the hypotheses have been tested, the results can be analyzed. The type and depth of analysis will be dependant on each individual problem, and the tests appropriate to it. In many cases the analysis will be a very quick thought process. In others, where considerable information has been collated, a more structured approach, such as the use of graphs, tables or spreadsheets, may be required.

5. Draw a conclusion.

Based on the results of the tests, a conclusion can then be drawn about exactly what is causing the problem. The appropriate remedial action can then be taken, such as asking everyone to amend their default print settings, or changing the cartridge supplier.

Inductive and Deductive Reasoning

The scientific method involves the use of two basic types of reasoning, inductive and deductive.

Inductive reasoning makes a conclusion based on a set of empirical results. Empirical results are the product of the collection of evidence from observations. For example:

‘Every time it rains the pavement gets wet, therefore rain must be water’.

There has been no scientific determination in the hypothesis that rain is water, it is purely based on observation. The formation of a hypothesis in this manner is sometimes referred to as an educated guess. An educated guess, whilst not based on hard facts, must still be plausible, and consistent with what we already know, in order to present a reasonable argument.

Deductive reasoning can be thought of most simply in terms of ‘If A and B, then C’. For example:

• if the window is above the desk, and
• the desk is above the floor, then
• the window must be above the floor

It works by building on a series of conclusions, which results in one final answer.

Social Sciences and the Scientific Method

The scientific method can be used to address any situation or problem where a theory can be developed. Although more often associated with natural sciences, it can also be used to develop theories in social sciences (such as psychology, sociology and linguistics), using both quantitative and qualitative methods.

Quantitative information is information that can be measured, and tends to focus on numbers and frequencies. Typically quantitative information might be gathered by experiments, questionnaires or psychometric tests. Qualitative information, on the other hand, is based on information describing meaning, such as human behavior, and the reasons behind it. Qualitative information is gathered by way of interviews and case studies, which are possibly not as statistically accurate as quantitative methods, but provide a more in-depth and rich description.

The resultant information can then be used to prove, or disprove, a hypothesis. Using a mix of quantitative and qualitative information is more likely to produce a rounded result based on the factual, quantitative information enriched and backed up by actual experience and qualitative information.

In terms of problem-solving or decision-making, for example, the qualitative information is that gained by looking at the ‘how’ and ‘why’ , whereas quantitative information would come from the ‘where’, ‘what’ and ‘when’.

It may seem easy to come up with a brilliant idea, or to suspect what the cause of a problem may be. However things can get more complicated when the idea needs to be evaluated, or when there may be more than one potential cause of a problem. In these situations, the use of the scientific method, and its associated reasoning, can help the user come to a decision, or reach a solution, secure in the knowledge that all options have been considered.

Join Mind Tools and get access to exclusive content.

This resource is only available to Mind Tools members.

Already a member? Please Login here

Team Management

Learn the key aspects of managing a team, from building and developing your team, to working with different types of teams, and troubleshooting common problems.

Sign-up to our newsletter

Subscribing to the Mind Tools newsletter will keep you up-to-date with our latest updates and newest resources.

Subscribe now

Business Skills

Personal Development

Leadership and Management

Member Extras

Most Popular

Newest Releases

SWOT Analysis

SMART Goals

Mind Tools Store

About Mind Tools Content

Discover something new today

How to stop procrastinating.

Overcoming the Habit of Delaying Important Tasks

What Is Time Management?

Working Smarter to Enhance Productivity

How Emotionally Intelligent Are You?

Boosting Your People Skills

Self-Assessment

What's Your Leadership Style?

Learn About the Strengths and Weaknesses of the Way You Like to Lead

Recommended for you

The addie model.

Developing Learning Sessions from the Ground Up

Business Operations and Process Management

Strategy Tools

Customer Service

Business Ethics and Values

Handling Information and Data

Project Management

Knowledge Management

Self-Development and Goal Setting

Time Management

Presentation Skills

Learning Skills

Career Skills

Communication Skills

Negotiation, Persuasion and Influence

Working With Others

Difficult Conversations

Creativity Tools

Self-Management

Work-Life Balance

Stress Management and Wellbeing

Coaching and Mentoring

Change Management

Managing Conflict

Delegation and Empowerment

Performance Management

Leadership Skills

Developing Your Team

Talent Management

Problem Solving

Decision Making

Member Podcast

If you're seeing this message, it means we're having trouble loading external resources on our website.

If you're behind a web filter, please make sure that the domains *.kastatic.org and *.kasandbox.org are unblocked.

To log in and use all the features of Khan Academy, please enable JavaScript in your browser.

Biology library

Course: biology library   >   unit 1, the scientific method.

• Controlled experiments
• The scientific method and experimental design

Introduction

• Make an observation.
• Ask a question.
• Form a hypothesis , or testable explanation.
• Make a prediction based on the hypothesis.
• Test the prediction.
• Iterate: use the results to make new hypotheses or predictions.

Scientific method example: Failure to toast

1. make an observation..

• Observation: the toaster won't toast.

2. Ask a question.

• Question: Why won't my toaster toast?

3. Propose a hypothesis.

• Hypothesis: Maybe the outlet is broken.

4. Make predictions.

• Prediction: If I plug the toaster into a different outlet, then it will toast the bread.

5. Test the predictions.

• Test of prediction: Plug the toaster into a different outlet and try again.
• If the toaster does toast, then the hypothesis is supported—likely correct.
• If the toaster doesn't toast, then the hypothesis is not supported—likely wrong.

Logical possibility

Practical possibility, building a body of evidence, 6. iterate..

• Iteration time!
• If the hypothesis was supported, we might do additional tests to confirm it, or revise it to be more specific. For instance, we might investigate why the outlet is broken.
• If the hypothesis was not supported, we would come up with a new hypothesis. For instance, the next hypothesis might be that there's a broken wire in the toaster.

show/hide words to know

Biased: when someone presents only one viewpoint. Biased articles do not give all the facts and often mislead the reader.

Conclusion: what a person decides based on information they get through research including experiments.

Method: following a certain set of steps to make something, or find an answer to a question. Like baking a pie or fixing the tire on a bicycle.

Research: looking for answers to questions using tools like the scientific method.

What is the Scientific Method?

If you have ever seen something going on and wondered why or how it happened, you have started down the road to discovery. If you continue your journey, you are likely to guess at some of your own answers for your question. Even further along the road you might think of ways to find out if your answers are correct. At this point, whether you know it or not, you are following a path that scientists call the scientific method. If you do some experiments to see if your answer is correct and write down what you learn in a report, you have pretty much completed everything a scientist might do in a laboratory or out in the field when doing research. In fact, the scientific method works well for many things that don’t usually seem so scientific.

The Flashlight Mystery...

Like a crime detective, you can use the elements of the scientific method to find the answer to everyday problems. For example you pick up a flashlight and turn it on, but the light does not work. You have observed that the light does not work. You ask the question, Why doesn't it work? With what you already know about flashlights, you might guess (hypothesize) that the batteries are dead. You say to yourself, if I buy new batteries and replace the old ones in the flashlight, the light should work. To test this prediction you replace the old batteries with new ones from the store. You click the switch on. Does the flashlight work? No?

What else could be the answer? You go back and hypothesize that it might be a broken light bulb. Your new prediction is if you replace the broken light bulb the flashlight will work. It’s time to go back to the store and buy a new light bulb. Now you test this new hypothesis and prediction by replacing the bulb in the flashlight. You flip the switch again. The flashlight lights up. Success!

If this were a scientific project, you would also have written down the results of your tests and a conclusion of your experiments. The results of only the light bulb hypothesis stood up to the test, and we had to reject the battery hypothesis. You would also communicate what you learned to others with a published report, article, or scientific paper.

More to the Mystery...

Not all questions can be answered with only two experiments. It can often take a lot more work and tests to find an answer. Even when you find an answer it may not always be the only answer to the question. This is one reason that different scientists will work on the same question and do their own experiments.

In our flashlight example, you might never get the light to turn on. This probably means you haven’t made enough different guesses (hypotheses) to test the problem. Were the new batteries in the right way? Was the switch rusty, or maybe a wire is broken. Think of all the possible guesses you could test.

No matter what the question, you can use the scientific method to guide you towards an answer. Even those questions that do not seem to be scientific can be solved using this process. Like with the flashlight, you might need to repeat several of the elements of the scientific method to find an answer. No matter how complex the diagram, the scientific method will include the following pieces in order to be complete.

The elements of the scientific method can be used by anyone to help answer questions. Even though these elements can be used in an ordered manner, they do not have to follow the same order. It is better to think of the scientific method as fluid process that can take different paths depending on the situation. Just be sure to incorporate all of the elements when seeking unbiased answers. You may also need to go back a few steps (or a few times) to test several different hypotheses before you come to a conclusion. Click on the image to see other versions of the scientific method. 

• Observation – seeing, hearing, touching…
• Asking a question – why or how?
• Hypothesis – a fancy name for an educated guess about what causes something to happen.
• Prediction – what you think will happen if…
• Testing – this is where you get to experiment and be creative.
• Conclusion – decide how your test results relate to your predictions.
• Communicate – share your results so others can learn from your work.

Other Parts of the Scientific Method…

Now that you have an idea of how the scientific method works there are a few other things to learn so that you will be able test out your new skills and test your hypotheses.

• Control - A group that is similar to other groups but is left alone so that it can be compared to see what happened to the other groups that are tested.
• Data - the numbers and measurements you get from the test in a scientific experiment.
• Independent variable - a variable that you change as part of your experiment. It is important to only change one independent variable for each experiment. 
• Dependent variable - a variable that changes when the independent variable is changed.
• Controlled Variable - these are variables that you never change in your experiment.

Practicing Observations and Wondering How and Why...

It is really hard not to notice things around us and wonder about them. This is how the scientific method begins, by observing and wondering why and how. Why do leaves on trees in many parts of the world turn from green to red, orange, or yellow and fall to the ground when winter comes? How does a spider move around their web without getting stuck like its victims? Both of these questions start with observing something and asking questions. The next time you see something and ask yourself, “I wonder why that does that, or how can it do that?” try out your new detective skills, and see what answer you can find. 

Try Out Your Detective Skills

Now that you have the basics of the scientific method, why not test your skills? The Science Detectives Training Room will test your problem solving ability. Step inside and see if you can escape the room. While you are there, look around and see what other interesting things might be waiting. We think you find this game a great way to learn the scientific method. In fact, we bet you will discover that you already use the scientific method and didn't even know it.

After you've learned the basics of being a detective, practice those skills in The Case of the Mystery Images . While you are there, pay attention to what's around you as you figure out just what is happening in the mystery photos that surround you.

Ready for your next challenge? Try Science Detectives: Case of the Mystery Images for even more mysteries to solve. Take your scientific abilities one step further by making observations and formulating hypothesis about the mysterious images you find within.

Acknowledgements:  

We thank John Alcock for his feedback and suggestions on this article.

Science Detectives - Mystery Room Escape was produced in partnership with the Arizona Science Education Collaborative (ASEC) and funded by ASU Women & Philanthropy.

Flashlight image via Wikimedia Commons - The Oxygen Team

Read more about: Using the Scientific Method to Solve Mysteries

View citation, bibliographic details:.

• Article: Using the Scientific Method to Solve Mysteries
• Author(s): CJ Kazilek and David Pearson
• Publisher: Arizona State University School of Life Sciences Ask A Biologist
• Site name: ASU - Ask A Biologist
• Date published: October 8, 2009
• Date accessed: March 24, 2024
• Link: https://askabiologist.asu.edu/explore/scientific-method

CJ Kazilek and David Pearson. (2009, October 08). Using the Scientific Method to Solve Mysteries . ASU - Ask A Biologist. Retrieved March 24, 2024 from https://askabiologist.asu.edu/explore/scientific-method

Chicago Manual of Style

CJ Kazilek and David Pearson. "Using the Scientific Method to Solve Mysteries ". ASU - Ask A Biologist. 08 October, 2009. https://askabiologist.asu.edu/explore/scientific-method

MLA 2017 Style

CJ Kazilek and David Pearson. "Using the Scientific Method to Solve Mysteries ". ASU - Ask A Biologist. 08 Oct 2009. ASU - Ask A Biologist, Web. 24 Mar 2024. https://askabiologist.asu.edu/explore/scientific-method

Do you think you can escape our Science Detectives Training Room ?

Using the Scientific Method to Solve Mysteries

Be part of ask a biologist.

By volunteering, or simply sending us feedback on the site. Scientists, teachers, writers, illustrators, and translators are all important to the program. If you are interested in helping with the website we have a Volunteers page to get the process started.

Share to Google Classroom

• school Campus Bookshelves
• menu_book Bookshelves
• perm_media Learning Objects
• login Login
• how_to_reg Request Instructor Account
• hub Instructor Commons
• Download Page (PDF)
• Download Full Book (PDF)
• Periodic Table
• Physics Constants
• Scientific Calculator
• Reference & Cite
• Tools expand_more
• Readability

selected template will load here

This action is not available.

1.2: Scientific Approach for Solving Problems

• Last updated
• Save as PDF
• Page ID 358114

Learning Objectives

• To identify the components of the scientific method

Scientists search for answers to questions and solutions to problems by using a procedure called the scientific method . This procedure consists of making observations, formulating hypotheses, and designing experiments, which in turn lead to additional observations, hypotheses, and experiments in repeated cycles (Figure \(\PageIndex{1}\)).

Observations can be qualitative or quantitative. Qualitative observations describe properties or occurrences in ways that do not rely on numbers. Examples of qualitative observations include the following: the outside air temperature is cooler during the winter season, table salt is a crystalline solid, sulfur crystals are yellow, and dissolving a penny in dilute nitric acid forms a blue solution and a brown gas. Quantitative observations are measurements, which by definition consist of both a number and a unit. Examples of quantitative observations include the following: the melting point of crystalline sulfur is 115.21 °C, and 35.9 grams of table salt—whose chemical name is sodium chloride—dissolve in 100 grams of water at 20 °C. An example of a quantitative observation was the initial observation leading to the modern theory of the dinosaurs’ extinction: iridium concentrations in sediments dating to 66 million years ago were found to be 20–160 times higher than normal. The development of this theory is a good exemplar of the scientific method in action (see Figure \(\PageIndex{2}\) below).

After deciding to learn more about an observation or a set of observations, scientists generally begin an investigation by forming a hypothesis , a tentative explanation for the observation(s). The hypothesis may not be correct, but it puts the scientist’s understanding of the system being studied into a form that can be tested. For example, the observation that we experience alternating periods of light and darkness corresponding to observed movements of the sun, moon, clouds, and shadows is consistent with either of two hypotheses:

• Earth rotates on its axis every 24 hours, alternately exposing one side to the sun, or
• The sun revolves around Earth every 24 hours.

Suitable experiments can be designed to choose between these two alternatives. For the disappearance of the dinosaurs, the hypothesis was that the impact of a large extraterrestrial object caused their extinction. Unfortunately (or perhaps fortunately), this hypothesis does not lend itself to direct testing by any obvious experiment, but scientists collected additional data that either support or refute it.

After a hypothesis has been formed, scientists conduct experiments to test its validity. Experiments are systematic observations or measurements, preferably made under controlled conditions—that is, under conditions in which a single variable changes. For example, in the dinosaur extinction scenario, iridium concentrations were measured worldwide and compared. A properly designed and executed experiment enables a scientist to determine whether the original hypothesis is valid. Experiments often demonstrate that the hypothesis is incorrect or that it must be modified. More experimental data are then collected and analyzed, at which point a scientist may begin to think that the results are sufficiently reproducible (i.e., dependable) to merit being summarized in a law , a verbal or mathematical description of a phenomenon that allows for general predictions. A law simply says what happens; it does not address the question of why.

One example of a law, the Law of Definite Proportions , which was discovered by the French scientist Joseph Proust (1754–1826), states that a chemical substance always contains the same proportions of elements by mass. Thus sodium chloride (table salt) always contains the same proportion by mass of sodium to chlorine, in this case 39.34% sodium and 60.66% chlorine by mass, and sucrose (table sugar) is always 42.11% carbon, 6.48% hydrogen, and 51.41% oxygen by mass. Some solid compounds do not strictly obey the law of definite proportions. The law of definite proportions should seem obvious—we would expect the composition of sodium chloride to be consistent—but the head of the US Patent Office did not accept it as a fact until the early 20th century.

Whereas a law states only what happens, a theory attempts to explain why nature behaves as it does. Laws are unlikely to change greatly over time unless a major experimental error is discovered. In contrast, a theory, by definition, is incomplete and imperfect, evolving with time to explain new facts as they are discovered. The theory developed to explain the extinction of the dinosaurs, for example, is that Earth occasionally encounters small- to medium-sized asteroids, and these encounters may have unfortunate implications for the continued existence of most species. This theory is by no means proven, but it is consistent with the bulk of evidence amassed to date. Figure \(\PageIndex{2}\) summarizes the application of the scientific method in this case.

Example \(\PageIndex{1}\)

Classify each statement as a law, a theory, an experiment, a hypothesis, a qualitative observation, or a quantitative observation.

• Ice always floats on liquid water.
• Birds evolved from dinosaurs.
• Hot air is less dense than cold air, probably because the components of hot air are moving more rapidly.
• When 10 g of ice were added to 100 mL of water at 25 °C, the temperature of the water decreased to 15.5 °C after the ice melted.
• The ingredients of Ivory soap were analyzed to see whether it really is 99.44% pure, as advertised.

Given : components of the scientific method

Asked for : statement classification

Strategy: Refer to the definitions in this section to determine which category best describes each statement.

• This is a general statement of a relationship between the properties of liquid and solid water, so it is a law.
• This is a possible explanation for the origin of birds, so it is a hypothesis.
• This is a statement that tries to explain the relationship between the temperature and the density of air based on fundamental principles, so it is a theory.
• The temperature is measured before and after a change is made in a system, so these are quantitative observations.
• This is an analysis designed to test a hypothesis (in this case, the manufacturer’s claim of purity), so it is an experiment.

Exercise \(\PageIndex{1}\)

• Measured amounts of acid were added to a Rolaids tablet to see whether it really “consumes 47 times its weight in excess stomach acid.”
• Heat always flows from hot objects to cooler ones, not in the opposite direction.
• The universe was formed by a massive explosion that propelled matter into a vacuum.
• Michael Jordan is the greatest pure shooter ever to play professional basketball.
• Limestone is relatively insoluble in water but dissolves readily in dilute acid with the evolution of a gas.
• Gas mixtures that contain more than 4% hydrogen in air are potentially explosive.

qualitative observation

quantitative observation

Because scientists can enter the cycle shown in Figure \(\PageIndex{1}\) at any point, the actual application of the scientific method to different topics can take many different forms. For example, a scientist may start with a hypothesis formed by reading about work done by others in the field, rather than by making direct observations.

It is important to remember that scientists have a tendency to formulate hypotheses in familiar terms simply because it is difficult to propose something that has never been encountered or imagined before. As a result, scientists sometimes discount or overlook unexpected findings that disagree with the basic assumptions behind the hypothesis or theory being tested. Fortunately, truly important findings are immediately subject to independent verification by scientists in other laboratories, so science is a self-correcting discipline. When the Alvarezes originally suggested that an extraterrestrial impact caused the extinction of the dinosaurs, the response was almost universal skepticism and scorn. In only 20 years, however, the persuasive nature of the evidence overcame the skepticism of many scientists, and their initial hypothesis has now evolved into a theory that has revolutionized paleontology and geology.

Chemists expand their knowledge by making observations, carrying out experiments, and testing hypotheses to develop laws to summarize their results and theories to explain them. In doing so, they are using the scientific method.

1.2 The Process of Science

Learning objectives.

• Identify the shared characteristics of the natural sciences
• Understand the process of scientific inquiry
• Compare inductive reasoning with deductive reasoning
• Describe the goals of basic science and applied science

Like geology, physics, and chemistry, biology is a science that gathers knowledge about the natural world. Specifically, biology is the study of life. The discoveries of biology are made by a community of researchers who work individually and together using agreed-on methods. In this sense, biology, like all sciences is a social enterprise like politics or the arts. The methods of science include careful observation, record keeping, logical and mathematical reasoning, experimentation, and submitting conclusions to the scrutiny of others. Science also requires considerable imagination and creativity; a well-designed experiment is commonly described as elegant, or beautiful. Like politics, science has considerable practical implications and some science is dedicated to practical applications, such as the prevention of disease (see Figure 1.15 ). Other science proceeds largely motivated by curiosity. Whatever its goal, there is no doubt that science, including biology, has transformed human existence and will continue to do so.

The Nature of Science

Biology is a science, but what exactly is science? What does the study of biology share with other scientific disciplines? Science (from the Latin scientia, meaning "knowledge") can be defined as knowledge about the natural world.

Science is a very specific way of learning, or knowing, about the world. The history of the past 500 years demonstrates that science is a very powerful way of knowing about the world; it is largely responsible for the technological revolutions that have taken place during this time. There are however, areas of knowledge and human experience that the methods of science cannot be applied to. These include such things as answering purely moral questions, aesthetic questions, or what can be generally categorized as spiritual questions. Science cannot investigate these areas because they are outside the realm of material phenomena, the phenomena of matter and energy, and cannot be observed and measured.

The scientific method is a method of research with defined steps that include experiments and careful observation. The steps of the scientific method will be examined in detail later, but one of the most important aspects of this method is the testing of hypotheses. A hypothesis is a suggested explanation for an event, which can be tested. Hypotheses, or tentative explanations, are generally produced within the context of a scientific theory . A generally accepted scientific theory is thoroughly tested and confirmed explanation for a set of observations or phenomena. Scientific theory is the foundation of scientific knowledge. In addition, in many scientific disciplines (less so in biology) there are scientific laws , often expressed in mathematical formulas, which describe how elements of nature will behave under certain specific conditions. There is not an evolution of hypotheses through theories to laws as if they represented some increase in certainty about the world. Hypotheses are the day-to-day material that scientists work with and they are developed within the context of theories. Laws are concise descriptions of parts of the world that are amenable to formulaic or mathematical description.

Natural Sciences

What would you expect to see in a museum of natural sciences? Frogs? Plants? Dinosaur skeletons? Exhibits about how the brain functions? A planetarium? Gems and minerals? Or maybe all of the above? Science includes such diverse fields as astronomy, biology, computer sciences, geology, logic, physics, chemistry, and mathematics ( Figure 1.16 ). However, those fields of science related to the physical world and its phenomena and processes are considered natural sciences . Thus, a museum of natural sciences might contain any of the items listed above.

There is no complete agreement when it comes to defining what the natural sciences include. For some experts, the natural sciences are astronomy, biology, chemistry, earth science, and physics. Other scholars choose to divide natural sciences into life sciences , which study living things and include biology, and physical sciences , which study nonliving matter and include astronomy, physics, and chemistry. Some disciplines such as biophysics and biochemistry build on two sciences and are interdisciplinary.

Scientific Inquiry

One thing is common to all forms of science: an ultimate goal “to know.” Curiosity and inquiry are the driving forces for the development of science. Scientists seek to understand the world and the way it operates. Two methods of logical thinking are used: inductive reasoning and deductive reasoning.

Inductive reasoning is a form of logical thinking that uses related observations to arrive at a general conclusion. This type of reasoning is common in descriptive science. A life scientist such as a biologist makes observations and records them. These data can be qualitative (descriptive) or quantitative (consisting of numbers), and the raw data can be supplemented with drawings, pictures, photos, or videos. From many observations, the scientist can infer conclusions (inductions) based on evidence. Inductive reasoning involves formulating generalizations inferred from careful observation and the analysis of a large amount of data. Brain studies often work this way. Many brains are observed while people are doing a task. The part of the brain that lights up, indicating activity, is then demonstrated to be the part controlling the response to that task.

Deductive reasoning or deduction is the type of logic used in hypothesis-based science. In deductive reasoning, the pattern of thinking moves in the opposite direction as compared to inductive reasoning. Deductive reasoning is a form of logical thinking that uses a general principle or law to predict specific results. From those general principles, a scientist can deduce and predict the specific results that would be valid as long as the general principles are valid. For example, a prediction would be that if the climate is becoming warmer in a region, the distribution of plants and animals should change. Comparisons have been made between distributions in the past and the present, and the many changes that have been found are consistent with a warming climate. Finding the change in distribution is evidence that the climate change conclusion is a valid one.

Both types of logical thinking are related to the two main pathways of scientific study: descriptive science and hypothesis-based science. Descriptive (or discovery) science aims to observe, explore, and discover, while hypothesis-based science begins with a specific question or problem and a potential answer or solution that can be tested. The boundary between these two forms of study is often blurred, because most scientific endeavors combine both approaches. Observations lead to questions, questions lead to forming a hypothesis as a possible answer to those questions, and then the hypothesis is tested. Thus, descriptive science and hypothesis-based science are in continuous dialogue.

Hypothesis Testing

Biologists study the living world by posing questions about it and seeking science-based responses. This approach is common to other sciences as well and is often referred to as the scientific method. The scientific method was used even in ancient times, but it was first documented by England’s Sir Francis Bacon (1561–1626) ( Figure 1.17 ), who set up inductive methods for scientific inquiry. The scientific method is not exclusively used by biologists but can be applied to almost anything as a logical problem-solving method.

The scientific process typically starts with an observation (often a problem to be solved) that leads to a question. Let’s think about a simple problem that starts with an observation and apply the scientific method to solve the problem. One Monday morning, a student arrives at class and quickly discovers that the classroom is too warm. That is an observation that also describes a problem: the classroom is too warm. The student then asks a question: “Why is the classroom so warm?”

Recall that a hypothesis is a suggested explanation that can be tested. To solve a problem, several hypotheses may be proposed. For example, one hypothesis might be, “The classroom is warm because no one turned on the air conditioning.” But there could be other responses to the question, and therefore other hypotheses may be proposed. A second hypothesis might be, “The classroom is warm because there is a power failure, and so the air conditioning doesn’t work.”

Once a hypothesis has been selected, a prediction may be made. A prediction is similar to a hypothesis but it typically has the format “If . . . then . . . .” For example, the prediction for the first hypothesis might be, “ If the student turns on the air conditioning, then the classroom will no longer be too warm.”

A hypothesis must be testable to ensure that it is valid. For example, a hypothesis that depends on what a bear thinks is not testable, because it can never be known what a bear thinks. It should also be falsifiable , meaning that it can be disproven by experimental results. An example of an unfalsifiable hypothesis is “Botticelli’s Birth of Venus is beautiful.” There is no experiment that might show this statement to be false. To test a hypothesis, a researcher will conduct one or more experiments designed to eliminate one or more of the hypotheses. This is important. A hypothesis can be disproven, or eliminated, but it can never be proven. Science does not deal in proofs like mathematics. If an experiment fails to disprove a hypothesis, then we find support for that explanation, but this is not to say that down the road a better explanation will not be found, or a more carefully designed experiment will be found to falsify the hypothesis.

Each experiment will have one or more variables and one or more controls. A variable is any part of the experiment that can vary or change during the experiment. A control is a part of the experiment that does not change. Look for the variables and controls in the example that follows. As a simple example, an experiment might be conducted to test the hypothesis that phosphate limits the growth of algae in freshwater ponds. A series of artificial ponds are filled with water and half of them are treated by adding phosphate each week, while the other half are treated by adding a salt that is known not to be used by algae. The variable here is the phosphate (or lack of phosphate), the experimental or treatment cases are the ponds with added phosphate and the control ponds are those with something inert added, such as the salt. Just adding something is also a control against the possibility that adding extra matter to the pond has an effect. If the treated ponds show lesser growth of algae, then we have found support for our hypothesis. If they do not, then we reject our hypothesis. Be aware that rejecting one hypothesis does not determine whether or not the other hypotheses can be accepted; it simply eliminates one hypothesis that is not valid ( Figure 1.18 ). Using the scientific method, the hypotheses that are inconsistent with experimental data are rejected.

In recent years a new approach of testing hypotheses has developed as a result of an exponential growth of data deposited in various databases. Using computer algorithms and statistical analyses of data in databases, a new field of so-called "data research" (also referred to as "in silico" research) provides new methods of data analyses and their interpretation. This will increase the demand for specialists in both biology and computer science, a promising career opportunity.

Visual Connection

In the example below, the scientific method is used to solve an everyday problem. Which part in the example below is the hypothesis? Which is the prediction? Based on the results of the experiment, is the hypothesis supported? If it is not supported, propose some alternative hypotheses.

• My toaster doesn’t toast my bread.
• Why doesn’t my toaster work?
• There is something wrong with the electrical outlet.
• If something is wrong with the outlet, my coffeemaker also won’t work when plugged into it.
• I plug my coffeemaker into the outlet.
• My coffeemaker works.

In practice, the scientific method is not as rigid and structured as it might at first appear. Sometimes an experiment leads to conclusions that favor a change in approach; often, an experiment brings entirely new scientific questions to the puzzle. Many times, science does not operate in a linear fashion; instead, scientists continually draw inferences and make generalizations, finding patterns as their research proceeds. Scientific reasoning is more complex than the scientific method alone suggests.

Basic and Applied Science

The scientific community has been debating for the last few decades about the value of different types of science. Is it valuable to pursue science for the sake of simply gaining knowledge, or does scientific knowledge only have worth if we can apply it to solving a specific problem or bettering our lives? This question focuses on the differences between two types of science: basic science and applied science.

Basic science or “pure” science seeks to expand knowledge regardless of the short-term application of that knowledge. It is not focused on developing a product or a service of immediate public or commercial value. The immediate goal of basic science is knowledge for knowledge’s sake, though this does not mean that in the end it may not result in an application.

In contrast, applied science or “technology,” aims to use science to solve real-world problems, making it possible, for example, to improve a crop yield, find a cure for a particular disease, or save animals threatened by a natural disaster. In applied science, the problem is usually defined for the researcher.

Some individuals may perceive applied science as “useful” and basic science as “useless.” A question these people might pose to a scientist advocating knowledge acquisition would be, “What for?” A careful look at the history of science, however, reveals that basic knowledge has resulted in many remarkable applications of great value. Many scientists think that a basic understanding of science is necessary before an application is developed; therefore, applied science relies on the results generated through basic science. Other scientists think that it is time to move on from basic science and instead to find solutions to actual problems. Both approaches are valid. It is true that there are problems that demand immediate attention; however, few solutions would be found without the help of the knowledge generated through basic science.

One example of how basic and applied science can work together to solve practical problems occurred after the discovery of DNA structure led to an understanding of the molecular mechanisms governing DNA replication. Strands of DNA, unique in every human, are found in our cells, where they provide the instructions necessary for life. During DNA replication, new copies of DNA are made, shortly before a cell divides to form new cells. Understanding the mechanisms of DNA replication enabled scientists to develop laboratory techniques that are now used to identify genetic diseases, pinpoint individuals who were at a crime scene, and determine paternity. Without basic science, it is unlikely that applied science could exist.

Another example of the link between basic and applied research is the Human Genome Project, a study in which each human chromosome was analyzed and mapped to determine the precise sequence of DNA subunits and the exact location of each gene. (The gene is the basic unit of heredity represented by a specific DNA segment that codes for a functional molecule.) Other organisms have also been studied as part of this project to gain a better understanding of human chromosomes. The Human Genome Project ( Figure 1.19 ) relied on basic research carried out with non-human organisms and, later, with the human genome. An important end goal eventually became using the data for applied research seeking cures for genetically related diseases.

While research efforts in both basic science and applied science are usually carefully planned, it is important to note that some discoveries are made by serendipity, that is, by means of a fortunate accident or a lucky surprise. Penicillin was discovered when biologist Alexander Fleming accidentally left a petri dish of Staphylococcus bacteria open. An unwanted mold grew, killing the bacteria. The mold turned out to be Penicillium , and a new critically important antibiotic was discovered. In a similar manner, Percy Lavon Julian was an established medicinal chemist working on a way to mass produce compounds with which to manufacture important drugs. He was focused on using soybean oil in the production of progesterone (a hormone important in the menstrual cycle and pregnancy), but it wasn't until water accidentally leaked into a large soybean oil storage tank that he found his method. Immediately recognizing the resulting substance as stigmasterol, a primary ingredient in progesterone and similar drugs, he began the process of replicating and industrializing the process in a manner that has helped millions of people. Even in the highly organized world of science, luck—when combined with an observant, curious mind focused on the types of reasoning discussed above—can lead to unexpected breakthroughs.

Reporting Scientific Work

Whether scientific research is basic science or applied science, scientists must share their findings for other researchers to expand and build upon their discoveries. Communication and collaboration within and between sub disciplines of science are key to the advancement of knowledge in science. For this reason, an important aspect of a scientist’s work is disseminating results and communicating with peers. Scientists can share results by presenting them at a scientific meeting or conference, but this approach can reach only the limited few who are present. Instead, most scientists present their results in peer-reviewed articles that are published in scientific journals. Peer-reviewed articles are scientific papers that are reviewed, usually anonymously by a scientist’s colleagues, or peers. These colleagues are qualified individuals, often experts in the same research area, who judge whether or not the scientist’s work is suitable for publication. The process of peer review helps to ensure that the research described in a scientific paper or grant proposal is original, significant, logical, and thorough. Grant proposals, which are requests for research funding, are also subject to peer review. Scientists publish their work so other scientists can reproduce their experiments under similar or different conditions to expand on the findings.

There are many journals and the popular press that do not use a peer-review system. A large number of online open-access journals, journals with articles available without cost, are now available many of which use rigorous peer-review systems, but some of which do not. Results of any studies published in these forums without peer review are not reliable and should not form the basis for other scientific work. In one exception, journals may allow a researcher to cite a personal communication from another researcher about unpublished results with the cited author’s permission.

As an Amazon Associate we earn from qualifying purchases.

This book may not be used in the training of large language models or otherwise be ingested into large language models or generative AI offerings without OpenStax's permission.

Want to cite, share, or modify this book? This book uses the Creative Commons Attribution License and you must attribute OpenStax.

Access for free at https://openstax.org/books/concepts-biology/pages/1-introduction

• Authors: Samantha Fowler, Rebecca Roush, James Wise
• Publisher/website: OpenStax
• Book title: Concepts of Biology
• Publication date: Apr 25, 2013
• Location: Houston, Texas
• Book URL: https://openstax.org/books/concepts-biology/pages/1-introduction
• Section URL: https://openstax.org/books/concepts-biology/pages/1-2-the-process-of-science

© Jan 8, 2024 OpenStax. Textbook content produced by OpenStax is licensed under a Creative Commons Attribution License . The OpenStax name, OpenStax logo, OpenStax book covers, OpenStax CNX name, and OpenStax CNX logo are not subject to the Creative Commons license and may not be reproduced without the prior and express written consent of Rice University.

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

3 The Process of Science in Biology

By the end of this section, you will be able to do the following:

• Identify the shared characteristics of the natural sciences
• Summarize the steps of the scientific method
• Compare inductive reasoning with deductive reasoning
• Describe the goals of basic science and applied science

What is biology? In simple terms, biology is the study of living organisms and their interactions with one another and their environments. This is a very broad definition because the scope of biology is vast. Biologists may study anything from the microscopic or submicroscopic view of a cell to ecosystems and the whole living planet (Figure 1). Listening to the daily news, you will quickly realize how many aspects of biology we discuss every day. For example, recent news topics include Escherichia coli (Figure 2) outbreaks in spinach and Salmonella contamination in peanut butter. Other subjects include efforts toward finding a cure for AIDS, Alzheimer’s disease, and cancer. On a global scale, many researchers are committed to finding ways to protect the planet, solve environmental issues, and reduce the effects of climate change. All of these diverse endeavors are related to different facets of the discipline of biology.

The Process of Science

Biology is a science, but what exactly is science? What does the study of biology share with other scientific disciplines? We can define science (from the Latin scientia , meaning “knowledge”) as the knowledge that covers general truths or the operation of general laws, especially when acquired and tested by the scientific method. It becomes clear from this definition that applying the scientific method plays a major role in science. The scientific method is a method of research with defined steps that include experiments and careful observation.

We will examine the scientific method steps in detail later, but one of the most important aspects of this method is the testing of hypotheses by means of repeatable experiments. A hypothesis is a suggested explanation for an event, that is both testable and falsifiable. Although using the scientific method is inherent to science, it is inadequate in determining what science is. This is because it is relatively easy to apply the scientific method to disciplines such as physics and chemistry, but when it comes to disciplines like archaeology, psychology, and geology, the scientific method becomes less applicable as repeating experiments becomes more difficult.

These areas of study are still sciences, however. Consider archaeology—even though one cannot perform repeatable experiments, hypotheses may still be supported. For instance, an archaeologist can hypothesize that an ancient culture existed based on finding a piece of pottery. He or she could make further hypotheses about various characteristics of this culture, which could be correct or false through continued support or contradictions from other findings.

It can take a while before a hypothesis becomes proven and widely accepted by the scientific community. In general, scientists attribute different degrees of confidence in scientific evidence-based the quality or quantity of the research and data supporting a given conclusion. Some scientists, especially in medicine, have codified these different sources of information into a hierarchy of scientific evidence (Figure 3) [ 1 ].

With enough evidence, a concept or explanation can become the highest form of scientific understanding: a theory .

What Is a Scientific Theory?

A scientific theory [ 2 ] is a broad explanation of events that is widely accepted by the scientific community. To become a theory, an explanation must be strongly supported by a great deal of evidence.

People commonly use the word theory to describe a guess or hunch about how or why something happens. For example, you might say, “I think a woodchuck dug this hole in the ground, but it’s just a theory.” Using the word theory in this way is different from the way it is used in science. A scientific theory is not just a guess or hunch that may or may not be true. In science, a theory is an explanation that has a high likelihood of being correct because it is so well supported by evidence.

What is a scientific theory?

Natural Sciences

What would you expect to see in a museum of natural sciences? Frogs? Plants? Dinosaur skeletons? Exhibits about how the brain functions? A planetarium? Gems and minerals? Maybe all of the above? Science includes such diverse fields as astronomy, biology, computer sciences, geology, logic, physics, chemistry, and mathematics (Figure 4). However, scientists consider those fields of science related to the physical world and its phenomena and processes in natural sciences . Thus, a museum of natural sciences might contain any of the items listed above.

There is no complete agreement when it comes to defining what the natural sciences include, however. For some experts, the natural sciences are astronomy, biology, chemistry, earth science, and physics. Other scholars choose to divide natural sciences into life sciences , which study living things and include biology, and physical sciences , which study nonliving matter and include astronomy, geology, physics, and chemistry. Some disciplines such as biophysics and biochemistry build on life and physical sciences and are interdisciplinary. Some refer to natural sciences as “hard science” because they rely on the use of quantitative data. Social sciences that study society and human behavior are more likely to use qualitative assessments to drive investigations and findings.

Not surprisingly, the natural science of biology has many branches or subdisciplines. Cell biologists study cell structure and function, while biologists who study anatomy investigate the structure of an entire organism. Those biologists studying physiology, however, focus on the internal functioning of an organism. Some areas of biology focus on only particular types of living things. For example, botanists explore plants, while zoologists specialize in animals.

Scientific Reasoning

One thing is common to all forms of science: an ultimate goal is “to know.” Curiosity and inquiry are the driving forces for the development of science. Scientists seek to understand the world and the way it operates. To do this, they use two methods of logical thinking: inductive reasoning and deductive reasoning.

Inductive reasoning is a form of logical thinking that uses related observations to arrive at a general conclusion. This type of reasoning is common in descriptive science. A life scientist such as a biologist makes observations and records them. These data can be qualitative or quantitative, and one can supplement the raw data with drawings, pictures, photos, or videos. From many observations, the scientist can infer conclusions (inductions) based on evidence. Inductive reasoning involves formulating generalizations inferred from careful observation and analyzing a large amount of data. Brain studies provide an example. In this type of research, scientists observe many live brains while people are engaged in a specific activity, such as viewing images of food. The scientist then predicts the part of the brain that “lights up” during this activity to be the part controlling the response to the selected stimulus, in this case, images of food. Excess absorption of radioactive sugar derivatives by active areas of the brain causes the various areas to “light up”. Scientists use a scanner to observe the resultant increase in radioactivity. Then, researchers can stimulate that part of the brain to see if similar responses result.

Deductive reasoning or deduction is the type of logic used in hypothesis-based science . In deductive reasoning, the pattern of thinking moves in the opposite direction as compared to inductive reasoning. Deductive reasoning is a form of logical thinking that uses a general principle or law to forecast specific results. From those general principles, a scientist can extrapolate and predict the specific results that would be valid as long as the general principles are valid. Studies in climate change can illustrate this type of reasoning. For example, scientists may predict that if the climate becomes warmer in a particular region, then the distribution of plants and animals should change.

Both types of logical thinking are related to the two main pathways of scientific study: descriptive science and hypothesis-based science. Descriptive (or discovery) science , which is usually inductive, aims to observe, explore, and discover, while hypothesis-based science , which is usually deductive, begins with a specific question or problem and a potential answer or solution that one can test. The boundary between these two forms of study is often blurred, and most scientific endeavors combine both approaches. The fuzzy boundary becomes apparent when thinking about how easily observation can lead to specific questions. For example, a gentleman in the 1940s observed that the burr seeds that stuck to his clothes and his dog’s fur had a tiny hook structure. On closer inspection, he discovered that the burrs’ gripping device was more reliable than a zipper. He eventually experimented to find the best material that acted similarly and produced the hook-and-loop fastener popularly known today as Velcro. Descriptive science and hypothesis-based science are in continuous dialogue.

The Scientific Method

Biologists study the living world by posing questions about it and seeking science-based responses. Known as the scientific method, this approach is common to other sciences as well. The scientific method was used even in ancient times, but England’s Sir Francis Bacon (1561–1626) first documented it (Figure 5). He set up inductive methods for scientific inquiry. The scientific method is not used only by biologists; researchers from almost all fields of study can apply it as a logical, rational problem-solving method.

The scientific process typically starts with an observation (often a problem to solve) that leads to a question. Let’s think about a simple problem that starts with an observation and apply the scientific method to solve the problem. One Monday morning, a student arrives at class and quickly discovers that the classroom is too warm. That is an observation that also describes a problem: the classroom is too warm. The student then asks a question: “Why is the classroom so warm?”

Proposing a Hypothesis

Recall that a hypothesis is a suggested explanation that can be tested and falsified. A good hypothesis is specific and includes clear variables that can be measured. For a given question, one can propose several hypotheses.

Let’s consider an example. You notice the classroom you teach in is warmer than usual. One hypothesis might be, “The temperature of the classroom is warmer because no one turned on the air conditioning.” Alternatively, a second hypothesis might be, “The temperature in the classroom is warmer because there is a power failure, and so the air conditioning doesn’t work.” While the cause is the same–no air conditioning–the variables are different: the air conditioner status (on/off) versus the power supply (present/absent). To find a solution, you need to isolate the problem.

The example above might seem simplistic–trouble-shooting an HVAC is not science… right? In fact, it is science or at least one part of the scientific process. And it illustrates the generality of scientific thinking in humans. Science is simply a methodology for problem-solving and collecting knowledge. It’s not the only system of knowledge or even the best per se , but it is one we regularly employ in our daily lives without even realizing it.

Once you have selected a hypothesis, you can make a prediction. A prediction is similar to a hypothesis but it typically has the format “If . . . then . . . .” For example, the prediction for the first hypothesis might be, “ If you turn on the air conditioning, then the classroom will no longer be too warm.” Note how this relates to the testability and falsifiability of the original hypothesis. You are testing the hypothesis by flipping the air conditioner switch. If you switch it on, and nothing happens, then the hypothesis is falsified, and it’s time to call an electrician to test the second hypothesis.

Testing a Hypothesis

To test a hypothesis, researchers design an experiment or analysis designed to validate or reject the hypothesis. In addition to the original hypothesis, researchers typically identify a null hypothesis. A null hypothesis represents the expectation if the proposed explanation is wrong. In our example from above, the competing null hypothesis would be “the power failure and loss of air conditioning does not cause the room to be warm.”

There are many types of experiments and analyses researchers conduct to test hypotheses. The general structure of most of these experiments or analyses, however, involves examining the effect of one variable on another. A variable is any part of the experiment that can vary or change during the course of the experiment. The variable of interest is referred to as the dependent variable . In our example, the dependent variable would be the temperature of the classroom. The independent variable is the condition the researcher purposefully changes to see how it affected the dependent variable. In our example, the independent variable would be the status of the air conditioner. Variables other than the independent variable that might nonetheless affect the dependent variable are referred to as confounding factors . A well-designed experiment will attempt to minimize the effect of confounding factors so that the researcher can be confident that the independent variable is the one causing the change in the dependent variable. It is not always possible to eliminate every confounding factor in a single experiment, however, and researchers must often run multiple experiments to ensure that something other than what they think is going on is actually occurring.

The most basic experimental design involves two groups, a control group and an experimental group . The control group represents the unmanipulated study condition, while the experimental group is somehow manipulated to test the effect of the independent variable. Otherwise, differences between the groups are limited to reduce any potential confounding variables. If the experimental group’s results differ from the control group, the difference must be due to the hypothesized manipulation, rather than some outside factor.  If the groups do not differ, then the independent variable has no effect, and the null hypothesis would be supported.

Look for the variables and controls in the examples that follow. To test the first hypothesis, the student would find out if the air conditioning is on. If the air conditioning is turned on but does not work, there should be another reason, and the student should reject this hypothesis. To test the second hypothesis, the student could check if the lights in the classroom are functional. If so, there is no power failure and the student should reject this hypothesis. The students should test each hypothesis by carrying out appropriate experiments. Be aware that rejecting one hypothesis does not determine whether or not one can accept the other hypotheses. It simply eliminates one hypothesis that is not valid (Figure 5). Using the scientific method, the student rejects hypotheses that are inconsistent with experimental data.

While this “warm classroom” example is based on observational results, other hypotheses and experiments might have clearer controls. For instance, a student might attend class on Monday and realize she had difficulty concentrating on the lecture. One observation to explain this occurrence might be, “When I eat breakfast before class, I am better able to pay attention.” The student could then design an experiment with a control to test this hypothesis.

To determine if the results of their experiment are significant, researchers use a variety of statistical analyses. Statistical analyses help researchers determine whether the observations from their experiments are meaningful or due to random chance. For example, if a researcher observes a difference between the control group and experimental group, should they treat it as a real effect of the independent variable or simply random chance? A result is considered to have statistical significance when it is very unlikely to have occurred given the null hypothesis. Statistical results themselves are not entirely objective and can depend on many assumptions including the null hypothesis itself. A researcher must consider potential biases in their analyses, just as they do confounding variables in their experimental design. Two factors that play a major role in the power of an experiment to detect meaningful statistical differences are sample size and replication. Sample size refers to the number of observations within each treatment, while replication refers to the number of repeated times the same experiment treatment is tried. In general, the bigger the sample size and the more replication, the more confidence a researcher can have in the outcome of their study.

In hypothesis-based science, researchers predict specific results from a general premise. We call this type of reasoning deductive reasoning: deduction proceeds from the general to the particular. However, the reverse of the process is also possible: sometimes, scientists reach a general conclusion from a number of specific observations. We call this type of reasoning inductive reasoning, and it proceeds from the particular to the general. Researchers often use inductive and deductive reasoning in tandem to advance scientific knowledge.

VISUAL CONNECTION

In the example below, the scientific method is used to solve an everyday problem. Order the scientific method steps (numbered items) with the process of solving the everyday problem (lettered items). Based on the results of the experiment, is the hypothesis correct? If it is incorrect, propose some alternative hypotheses.

Answer: 1: C; 2: F; 3: A; 4: B; 5: D; 6: E. The original hypothesis is incorrect, as the coffee maker works when plugged into the outlet. Alternative hypotheses include that the toaster might be broken or that the toaster wasn’t turned on.

The scientific method may seem too rigid and structured. It is important to keep in mind that, although scientists often follow this sequence, there is flexibility. Sometimes an experiment leads to conclusions that favor a change in approach. Often, an experiment brings entirely new scientific questions to the puzzle. Many times, science does not operate in a linear fashion. Instead, scientists continually draw inferences and make generalizations, finding patterns as their research proceeds. Scientific reasoning is more complex than the scientific method alone suggests. Notice, too, that we can apply the scientific method to solving problems that aren’t necessarily scientific in nature.

Two Types of Science: Basic Science and Applied Science

The scientific community has been debating for the last few decades about the value of different types of science. Is it valuable to pursue science for the sake of simply gaining knowledge, or does scientific knowledge only have worth if we can apply it to solving a specific problem or to bettering our lives? This question focuses on the differences between two types of science: basic science and applied science.

Basic science or “pure” science seeks to expand knowledge regardless of the short-term application of that knowledge. It is not focused on developing a product or a service of immediate public or commercial value. The immediate goal of basic science is knowledge for knowledge’s sake, although this does not mean that, in the end, it may not result in a practical application.

In contrast, applied science or “technology,” aims to use science to solve real-world problems, making it possible, for example, to improve a crop yield, find a cure for a particular disease, or save animals threatened by a natural disaster (Figure 6). In applied science, the problem is usually defined for the researcher.

Some individuals may perceive applied science as “useful” and basic science as “useless.” A question these people might pose to a scientist advocating knowledge acquisition would be, “What for?” However, a careful look at the history of science reveals that basic knowledge has resulted in many remarkable applications of great value. Many scientists think that a basic understanding of science is necessary before researchers develop an application therefore, applied science relies on the results that researchers generate through basic science. Other scientists think that it is time to move on from basic science in order to find solutions to actual problems. Both approaches are valid. It is true that there are problems that demand immediate attention; however, scientists would find few solutions without the help of the wide knowledge foundation that basic science generates.

One example of how basic and applied science can work together to solve practical problems occurred after the discovery of DNA structure led to an understanding of the molecular mechanisms governing DNA replication. DNA strands, unique in every human, are in our cells, where they provide the instructions necessary for life. When DNA replicates, it produces new copies of itself, shortly before a cell divides. Understanding DNA replication mechanisms enabled scientists to develop laboratory techniques that researchers now use to identify genetic diseases, pinpoint individuals who were at a crime scene, and determine paternity. Without basic science, it is unlikely that applied science would exist.

Another example of the link between basic and applied research is the Human Genome Project, a study in which researchers analyzed and mapped each human chromosome to determine the precise sequence of DNA subunits and each gene’s exact location. (The gene is the basic unit of heredity. An individual’s complete collection of genes is his or her genome.) Researchers have studied other less complex organisms as part of this project in order to gain a better understanding of human chromosomes. The Human Genome Project (Figure 7) relied on basic research with simple organisms and, later, with the human genome. An important end goal eventually became using the data for applied research, seeking cures and early diagnoses for genetically related diseases.

While scientists usually carefully plan research efforts in both basic science and applied science, note that some discoveries are made by serendipity , that is, by means of a fortunate accident or a lucky surprise. Scottish biologist Alexander Fleming discovered penicillin when he accidentally left a petri dish of Staphylococcus bacteria open. An unwanted mold grew on the dish, killing the bacteria. Fleming’s curiosity to investigate the reason behind the bacterial death, followed by his experiments, led to the discovery of the antibiotic penicillin, which is produced by the fungus Penicillium . Even in the highly organized world of science, luck—when combined with an observant, curious mind—can lead to unexpected breakthroughs.

Reporting Scientific Work

Whether scientific research is basic science or applied science, scientists must share their findings in order for other researchers to expand and build upon their discoveries. Collaboration with other scientists—when planning, conducting, and analyzing results—are all important for scientific research. For this reason, important aspects of a scientist’s work are communicating with peers and disseminating results to peers. Scientists can share results by presenting them at a scientific meeting or conference, but this approach can reach only the select few who are present. Instead, most scientists present their results in peer-reviewed manuscripts that are published in scientific journals. Peer-reviewed manuscripts are scientific papers that a scientist’s colleagues or peers review. These colleagues are qualified individuals, often experts in the same research area, who judge whether or not the scientist’s work is suitable for publication. The process of peer review helps to ensure that the research in a scientific paper or grant proposal is original, significant, logical, and thorough. Grant proposals, which are requests for research funding, are also subject to peer review. Scientists publish their work so other scientists can reproduce their experiments under similar or different conditions to expand on the findings. The experimental results must be consistent with the findings of other scientists.

A scientific paper is very different from creative writing. Although creativity is required to design experiments, there are fixed guidelines when it comes to presenting scientific results. First, scientific writing must be brief, concise, and accurate. A scientific paper needs to be succinct but detailed enough to allow peers to reproduce the experiments.

The scientific paper consists of several specific sections—introduction, materials and methods, results, and discussion. This structure is sometimes called the “IMRaD” format. There are usually acknowledgment and reference sections as well as an abstract (a concise summary) at the beginning of the paper. There might be additional sections depending on the type of paper and the journal where it will be published. For example, some review papers require an outline.

The introduction starts with brief, but broad, background information about what is known in the field. A good introduction also gives the rationale of the work. It justifies the work carried out and also briefly mentions the end of the paper, where the researcher will present the hypothesis or research question driving the research. The introduction refers to the published scientific work of others and therefore requires citations following the style of the journal. Using the work or ideas of others without proper citation is plagiarism .

The materials and methods section includes a complete and accurate description of the substances the researchers use and the methods and techniques they use to gather data. The description should be thorough enough to allow another researcher to repeat the experiment and obtain similar results, but it does not have to be verbose. This section will also include information on how the researchers made measurements and the types of calculations and statistical analyses they used to examine raw data. Although the materials and methods section gives an accurate description of the experiments, it does not discuss them.

Some journals require a results section followed by a discussion section, but it is more common to combine both. If the journal does not allow combining both sections, the results section simply narrates the findings without any further interpretation. The researchers present results with tables or graphs, but they do not present duplicate information. In the discussion section, the researchers will interpret the results, describe how variables may be related, and attempt to explain the observations. It is indispensable to conduct an extensive literature search to put the results in the context of previously published scientific research. Therefore, researchers include proper citations in this section as well.

Finally, the conclusion section summarizes the importance of the experimental findings. While the scientific paper almost certainly answers one or more scientific questions that the researchers stated, any good research should lead to more questions. Therefore, a well-done scientific paper allows the researchers and others to continue and expand on the findings.

Review articles do not follow the IMRAD format because they do not present original scientific findings or primary literature. Instead, they summarize and comment on findings that were published as primary literature and typically include extensive reference sections.

Review of the scientific process

Introductory Biology: Evolutionary and Ecological Perspectives Copyright © by Various Authors - See Each Chapter Attribution is licensed under a Creative Commons Attribution 4.0 International License , except where otherwise noted.

Share This Book

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Front Psychol
• PMC10151488

Using process features to investigate scientific problem-solving in large-scale assessments

1 School of Foreign Languages, Zhejiang University of Finance and Economics, Hangzhou, Zhejiang, China

2 Educational Testing Service, Princeton, NJ, United States

3 Google, New York, NY, United States

Burcu Arslan

Associated data.

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Introduction

This study investigates the process data from scientific inquiry tasks of fair tests [requiring test-takers to manipulate a target variable while keeping other(s) constant] and exhaustive tests (requiring test-takers to construct all combinations of given variables) in the National Assessment of Educational Progress program.

We identify significant associations between item scores and temporal features of preparation time, execution time, and mean execution time.

Reflecting, respectively, durations of action planning and execution, and execution efficiency, these process features quantitatively differentiate the high- and low-performing students: in the fair tests, high-performing students tended to exhibit shorter execution time than low-performing ones, but in the exhaustive tests, they showed longer execution time; and in both types of tests, high-performing students had shorter mean execution time than low-performing ones.

This study enriches process features reflecting scientific problem-solving process and competence and sheds important light on how to improve performance in large-scale, online delivered scientific inquiry tasks.

1. Introduction

The past two decades have witnessed an increasing use of computers and relevant technologies in classroom teaching and learning ( Hoyles and Noss, 2003 ) and a swift transition from traditional paper-and-pencil tests to digitally-based assessments (DBAs) ( Zenisky and Sireci, 2002 ; Scalise and Gifford, 2006 ) that accommodate advancement of educational technologies. Along with these trends, the National Assessment of Educational Progress (NAEP) 1 began to use hand-held tablets to administer math assessments in the U.S. in 2017, so did other disciplines afterward. Capable of recording multi-dimensional data, DBAs offer ample opportunities to systematically investigate U.S. students’ problem-solving processes through well-designed technology-enhanced items (TEIs) ( National Assessment Governing Board, 2015 ). TEIs refer broadly to computer-aided items that incorporate technology beyond simple option selections as test-takers’ response method ( Koedinger and Corbett, 2006 ). In a TEI, test-takers are asked to interact with computers by performing a series of actions to solve one (or multiple) problem. For example, in scientific inquiry TEIs of fair tests ( Chen and Klahr, 1999 ), students are asked to adjust a target variable in an experimental setting or condition while keeping other(s) constant, to reveal effect or outcome of the target variable. In another type of scientific inquiry TEIs, exhaustive tests ( Montgomery, 2000 ; Black, 2007 ), students are required to construct all possible combinations of given variables to investigate what combination(s) leads to a specific outcome (see Section 2 for details). In both types of tests, students need to apply the control-of-variables strategy (CVS, see Section 2 for details), a domain-general processing skill to design controlled experiments in a multi-variable system ( Kuhn and Dean, 2005 ; Kuhn, 2007 ).

Beyond final responses, interactive actions of students are captured as process data . 2 Such data help (re)construct problem-solving processes, reflect durations (or frequencies) of major problem-solving stages, and infer how students deploy strategies they seem to know ( Pedaste et al., 2015 ; Provasnik, 2021 ), all of which provide additional clues of students’ problem-solving behaviors ( Kim et al., 2007 ; Ebenezer et al., 2011 ; Gobert et al., 2012 ). For example, in drag-and-drop (D&D) items, a popular type of TEIs, students drag some objects from source locations and drop them into target positions on screen. Compared to conventional multiple-choice items, such items can better represent construct-relevant skills, strengthen measurement, improve engagement/motivation of test-takers, and reduce interference of random guessing ( Bryant, 2017 ; Arslan et al., 2020 ).

Despite the advantages, process data have long been treated as by-products in educational assessments. Until recently, scholars have begun to investigate whether (and if so, how) process data inform (meta)cognitive processes and students’ strategies during problem solving ( Guo et al., 2019 ; Tang et al., 2019 ; Gong et al., 2021 , 2022 ). By reviewing pioneering studies on NAEP process data before its formal transition to DBA, Bergner and von Davier (2019) proposed a hierarchical framework that divides process data use into five levels based on its relative importance to outcome: Level 1 , process data are irrelevant/ignored and only response data are considered; Level 2 , process data are incorporated as auxiliary to understanding outcome; Level 3 , process data are incorporated as essential to understanding outcome; Level 4 , process data are outcome and incorporated into scoring rubrics; and Level 5 , process data are outcome and incorporated into measurement models.

Most published process data studies remain up to level 2 of this framework; they directly use students’ actions, action sequences, and (partial/rough) durations of answering processes to interpret item outcome (e.g., answer change behaviors, Liu et al., 2015 ; response time, Lee and Jia, 2014 ; or action sequences, Han et al., 2019 ; Ulitzsch et al., 2021 ). Before explicitly revealing correlations between process data and individual performance, inferences from these studies remain auxiliary rather than essential . In other words, discovering process features and their relatedness to test-takers’ performance is a prerequisite for using process features to understand or interpret individual performance, thus reaching higher levels of the framework.

This study aims to fulfill this prerequisite by investigating process data from scientific inquiry tasks (see Supplementary materials ) and related research questions therein in a three-step procedure:

Define time-related features to illustrate action planning and executing stages of scientific problem solving . Many early studies have examined action-related features that reflect conceptual formation ( Jonassen, 2000 ; Lesh and Harel, 2011 ), response strategies, and internal (individual dispositions) or external (testing circumstances) factors probably affecting students’ choices of strategies ( Griffiths et al., 2015 ; Lieder and Griffiths, 2017 ; Moon et al., 2018 ). However, the time needed for problem solving has been largely undervalued ( Dostál, 2015 ). As an informative indicator of problem solving stages, temporal information helps characterize patterns of students, and infer (meta)cognitive processes occurring at various stages of problem solving.

We propose three temporal features to reflect, respectively, the major stages of scientific problem solving (see Figure 1 ). In an assessment setting, preparation time ( PT ) is defined as the time difference (duration) between the moment students enter a test scene and when they make their first answer-related event. It denotes the duration while students understand instructions and conceptually plan their actions, before making any. Execution time ( ET ) is defined as the time difference between students’ first and last answer-related events. It measures the duration while students execute their planned actions. Mean execution time per ( MET ) is measured as ET divided by the number of answer related events. 3 ET reflects total efforts of students casted to construct their answers, including setting up answers and revising or (possibly) reviewing their choices, whereas MET reflects average effort over total events. Controlling for answer-related events, MET indicates the efficiency of action execution. Our study examines whether these temporal features significantly correlate item scores and characterize high/low-performing students in test scenes. 4

Proposed process features [preparation time (PT) and execution time (ET)] and corresponding major stages of scientific problem solving (understanding and planning, and executing planned actions, denoted by colored bars) in a scientific inquiry test item. Vertical lines denote the times when a test-taker enters and exits the task, vertical bars denote answer-related actions. See Supplementary materials for review of scientific problem solving processes.

Explore correlations between process features and item scores . This is the missing link in many existing studies of problem solving; some cannot verify such correlations, since the categorical features (e.g., action sequences) used cannot fit for correlation tests, whereas others, directly assuming such correlations, skip this step and use process features to inform/interpret performance. Neither approach is complete. Our study focuses on detecting correlations between continuous process features and item scores and explaining feature differences across score groups.

We apply two statistical tests to detect correlations. First, the Kruskal-Wallis test ( Kruskal and Wallis, 1952 ) compares process features across score groups, and reports whether (at least) one of the multiple samples is significantly distinct from others. As a non-parametric version of ANOVA, this test does not require a normal distribution of the residual values of features. Extended from the Mann–Whitney test, this test is an omnibus test applicable to small-scale, independent samples from multiple groups. Second, we conduct the omnibus ANOVA between score groups, and log-transformed (base e ) the process features to meet the normality assumption. This test is applicable to large-scale datasets. We use both methods to cross-validate obtained results by each method.

Use process features to characterize performance (or competence) differences between students and/or tasks . After verifying correlations between process features and item scores, we further investigate: (a) whether there exist differences (or similarities) in the process features across score groups and/or inquiry tasks ; and (b) whether the observed differences (or similarities) characterize problem solving performance (or competence) between high- and low-performing students and between inquiry tasks. Answers to these questions further foster these features as informative indicators of students’ performance and pave the way for incorporating them into scoring rubrics and measurement models aiming to classify and interpret students’ behaviors.

In the following sections, we first review the CVS strategies and scientific inquiry tasks, and then define the process metrics and analysis plans. After reporting the analysis results, we answer the abovementioned questions, summarize our contributions to scientific inquiry and problem solving research, and point out the general procedure of process data use in educational assessments.

2. Control-of-variables strategies and scientific inquiry tasks

Control-of-variables strategy (CVS) 5 has been widely studied in science assessments. CVS refers to the skill used to design controlled experiments in a multi-variable system. To avoid confounded experiments, all variables but those under investigation must be controlled in a way to meet task requirements. In the Next Generation Science Standards (NGSS), CVS and multivariate reasoning are viewed as two key scientific thinking skills. Central to early science instruction ( Klahr and Nigam, 2004 ) (around grades 4–8), CVS cannot develop routinely without practice or instruction ( Schwichow et al., 2016 ), making it a critical issue in development of scientific thinking ( Kuhn and Dean, 2005 ). Children, adolescents, and adults with low science inquiry skills show difficulty in applying CVS in scientific problem solving ( Chen and Klahr, 1999 ).

In large-scale assessments like NAEP, CVS is often assessed by two types of scientific inquiry tasks: fair tests and exhaustive tests. A fair test (see examples in Section 3.1) refers to a controlled investigation carried out to answer a scientific question about the effect of a target variable. To control for confounding factors and be scientifically sound, students are expected to apply the CVS to meet the fair test requirement that: (a) all other variable(s) are kept constant; and (b) only the target one(s) changes across conditional sets for comparison. In such a “fair” setting, the effect of the target variable(s) can be observed and less interfered by other variables. To properly complete the task, students need to choose, among possible combinations of different levels of the target and other variables, one (or a few) condition that meets the requirement. There are studies of CVS in scientific inquiry using small-scale participants and response/survey data ( Kuhn, 2007 ). A recent meta-analysis of intervention studies (partially) designed to enhance CVS skills revealed that instruction/intervention (e.g., cognitive conflict and demonstration) influences achievement in scientific inquiry tasks ( Schwichow et al., 2016 ).

An exhaustive test (a.k.a. all-pair or combinatorial test ) (see examples in Section 3.2) requires test-takers to construct, physically or mentally, (nearly) all possible combinations of given variables to address an inquiry of what condition(s) induces a specific outcome. Similar to fair tests, students in exhaustive tests need to control the given variables by setting up combinations exhaustively or nearly so (in an open-ended case). Though not explicitly mentioned in NGSS, exhaustive testing is essentially related to CVS or at least a case of multivariate reasoning. How to conduct exhaustive tests is usually taught and learned relatively late in science education (around grades 9–12). Such tests have also been adopted in other fields than educational assessments, e.g., software engineering and business ( Grindal et al., 2005 ).

3. Materials and methods

Our study makes use of the 2018 NAEP science pilot tasks (see Supplementary materials ). It adopted four tests, respectively, from four tasks in the repertoire: two fair tests administered on fourth- and eighth-graders, respectively (the primary and middle school bands, per NGSS), and two exhaustive tests on twelfth-graders (the high school band). Table 1 shows the samples of these tests.

Basic information of the testlets investigated in this paper.

Due to various reasons (e.g., early quit or data capture glitches), data of some students were missing. The rightmost column records the number of students whose process and response data were used for analyses.

Two criteria lead to the choice of these tasks. First, the sampled tests should cover most science subfields and grades in the NAEP sample. However, given that lower grade students have not been taught to solve exhaustive tasks, no such tests were administered on fourth-graders. Second, since fair tests were administered mostly on eighth-graders and exhaustive tests on twelfth-graders, it is impossible to select fair tests and exhaustive tests administered on students of the same grade. Nonetheless, since all the NAEP fair and exhaustive tests were designed by content experts following similar constructs and the only difference was that each task fell into one of the science disciplines (physical, life, earth/space sciences), the chosen tests in our study are representative.

3.1. The fair tests and scoring rubrics

The fair test 1 came from an earth/space science task. Its cover task 6 is as follows. A city near a mountain suffers from strong north wind each year. The government plans to test the wind-blocking effect of three types of trees. Each type can be planted at the foot (low), side (medium), or peak (high) of the northern ridge of the mountain to reduce wind speed, and there is no interaction between tree type and mountain position (e.g., there is no preference for one type of trees to be planted at a specific position).

The whole task is presented to students through multiple scenes, some involving items. The first few scenes help students understand, represent, and explore relevant issues. After them comes the fair test scene, in which students are asked to design a controlled experiment to investigate the wind-blocking effects of the three types of trees. The follow-up scenes ask them to interpret/revisit their answers and apply their knowledge in novel conditions. Students went through these scenes in the same order and could not freely jump around.

In the fair test scene (see Figure 2 ), students need to drag each type of the trees and drop it at one of the four virtual mountains resembling the real one near the city; students can drop the trees at the foot (low), side (middle), or peak (high) of the northern ridge of the mountain. Each mountain can hold one type of the trees, and each type can only be planted at one mountain. Students can move trees from one mountain to another, or from one position of a mountain to another position of the same or different mountain. After making final selections, students click an on-screen “Submit” button to initiate the experiment. Then, the wind speeds before and after passing over each of the mountains with/without trees are shown on screen as experimental results.

Example answers of the fair test 1. “Low,” “Medium,” “High” denote tree positions (foot, side, peak) in the northern ridge of a virtual mountain. “None” means no tree planted. In (A) , the first “Low” indicates that one type of trees is planted at the foot of the mountain, the second and third “Low” indicate that the other two types of trees are planted on the second and third mountains, and “None” means that the fourth mountain has no trees planted. Since the scoring rubric (see Table 2 ) does not specify tree type and ignores the mountain without trees, submitted answers can be simply denoted by the tree positions in the mountains with trees. In this way, answer (A) can be denoted as “Low; Low; Low”, answer (B) as “Medium; High; High”, and answer (C) as “Low, High, Medium”.

This fair test has two variables: tree type (with three levels, corresponding to the three tree types) and tree position (with three levels, low, middle, and high). To conduct a fair test showing the effect of tree type, students must keep the tree positions across mountains identical. Table 2 shows the scoring rubric of the test. Since students can never plant the same type of trees on two mountains or at two positions of one mountain, the rubric focuses mainly on the types of trees planted on mountains. In addition, no matter how students plant trees, one mountain is left with no trees. A complete comparison on the effect of tree type needs a baseline condition of no trees, but students are not required to explicitly set up this condition in this test. Therefore, although there are in principle 3 × 3 × 3 ×  P (4,3) = 648 ways of tree planting and 3 ×  P (4,3) = 72 in which match the fair test requirement ( P means permutation), the matching answers can be classified into three types: (a) those having the three types of trees all planted at the “Low” positions of any three out of the four mountains; (b) all at the “Middle” positions; and (c) all at the “High” positions. These answers receive a full score (3). Answers having trees planted at two distinct positions of any three mountains has a partial score (2), and those having trees planted at three distinct positions of any three mountains receives the lowest score (1).

Scoring rubrics of the fair tests 1 and 2.

The fair test 2 comes from a physical science task. Its cover task is as follows. A bakery shop is developing a new product. The bakers want to test which of the three ingredients (white candy, butter, and honey) has the most acceptable sweetness in the new product. Each ingredient has three amounts to choose: 50, 100, and 200 milligrams. After instruction scenes, in the fair test scene, nine piles of the three ingredients with the three amounts are shown on the left side of the screen (see Figure 3A ), and students can drag three of these piles into the three slots on the right side of the screen to show the effect of ingredients on the sweetness of the product. Students can move the piles from one slot to another. After making final choices, students click on an on-screen “Submit” button to initiate the experiment, and the sweetness of each choice is shown on the screen.

Example answers of the fair test 2. (A) Nine piles of ingredients for selection: 50, 100, and 200 are milligrams, and each pile is marked by an index of 1–9. (B) A choice of three piles, denoted by 2–5–8, matching the fair test requirement. (C) A choice of three piles, 1–5–9, partially matching the requirement. (D) A choice of three piles, 7–8–6, not matching the requirement.

This fair test has two variables: ingredient type (white candy, butter, and honey) and ingredient amount (50, 100, and 200 milligrams). To show the effect of ingredients, one needs to keep ingredient amount identical across conditions. Among a total of C (9,3) ×  P (3,3) = 504 choices of three piles of ingredients ( C means combination), 3 ×  P (3,3) = 18 match the fair test requirement. Table 2 shows the scoring rubric of the test. Answers matching the fair test requirement receive a full score (3), and others receive a partial (2) or the lowest score (1).

3.2. The exhaustive tests and scoring rubric

The exhaustive test 1 comes from a life science task. Its cover task is as follows. Farmers are trying to cultivate flowers with a special color. They do this in a natural way or using one or two types of fertilizers (A and B). After scenes for students to understand related issues, represent and explore different conditions, there comes the exhaustive test scene, in which students are asked to design an experiment to show which way has the highest chance to cultivate flowers with a target color. They can set up a condition by selecting (or not) any (or both) fertilizer, and save it by clicking an on-screen “Save” button. They can also remove a saved condition by clicking it and an on-screen “Delete” button. After saving some conditions, they can click on an on-screen “Submit” button to submit all the saved conditions at that moment as final answers. This test requires four variable combinations (see Figure 4 ). The follow-up scenes ask students to review their answers and apply their knowledge in similar domains. Students went through these scenes in the same order and could not freely jump around the scenes.

All the combinations in the exhaustive test 1: (A) None; (B) : A; (C) : B; (D) : A + B.

Table 3 shows the scoring rubric of this test. The four scales are based on the types of the saved answers, especially whether they include some hard-to-foresee ones (e.g., Figures 4A , ​ ,D). D ). An exhaustive answer covering all combinations in Figure 4 receives the full score (4), whereas answers lacking one, two, or three of the combinations receive lower scores 3, 2, and 1. The validity of the rubric (whether it can reasonably reflect students’ intuitive conceptions and clarify students with various levels of problem solving skills) is beyond the scope of this paper.

Scoring rubrics of the exhaustive tests 1 and 2.

For the exhaustive test 1, the original rubric also evaluates whether students give a proper interpretation of submitted answers. Here, students are rescored based only on their saved conditions. “Loc.” stands for location.

The exhaustive test 2 comes from an earth science task. Its cover task is as follows. Two cities (A and B) plan to build a transmission tower to broadcast television signals. To evaluate signal quality on the land between the cities, they segment the land into 14 regions, each having four locations for signal sampling (see Figure 5 ). After instructions, students are asked to select at most 15 locations (out of 42) in the 13 regions (one region with one location therein being chosen is used as a demo) to test the signal coverage. They can select a location by clicking on it and deselect it by clicking on it again. If 15 locations are already chosen, students must deselect some chosen locations before making new selection(s). After choosing some locations (not necessarily 15), students can click on an on-screen “Submit” button to submit the chosen locations as final answers.

Example answers in the exhaustive test 2. Squares marked as 1–14 are the regions between City A and City B. Round dots in a region are locations for sample taking. Region 6 is the demo region with a chosen location marked in red, and the others are marked in grey. Green dots are students’ chosen locations. In this answer, there is at least one chosen location in all 13 regions except region 6, and there is at least one additional chosen location in one of the three regions adjacent to City A (region 5) and City B (region 11). This answer has a score of 4 (see Table 3 ).

In this exhaustive test, students need to: (a) select at least one location in each of the 13 regions to test signal quality; and (b) choose two additional locations, respectively, in the three regions adjacent to each city to evaluate the signal sources in the two cities. It is challenging to foresee both aspects of requirements. Table 3 shows the scoring rubric of the test. The four scales are dependent on whether students fulfil both, either, or none of the two aspects of requirement. Whether this rubric is valid is not the focus of this paper.

3.3. Process-based measures

We define and measure three temporal features: preparation time ( PT ), execution time ( ET ), and mean execution time per answer-related event ( MET ). All of them are calculated based on time stamps of answer-related events. In the fair tests, answer-related events include: drag a type of trees (or a pile of ingredients) and drop it on a position of a virtual mountain (or a slot), or move a type of tree (or a pile of ingredients) from one mountain (position) (or one slot) to another; in the exhaustive tests, such events include: select one or two fertilizers (or a number of locations), and save or cancel a condition. The ending time point of ET is not the moment when students click on the “Submit” button, because after executing the last answer-related event, they can review their answers, thus moving into the next stage of problem solving. Also, executing actions may involve planning bounded to prior actions, which is different from the conceptual planning of related actions before making any. Therefore, we limited ET as the duration between the first and last answer related events. In addition to answer-related events, other factors (e.g., mouse or computer speed) might affect the efficiency of action execution. Since the tests were administered on site using the same model of tablets, the influence of these factors was minimal.

3.4. Preprocessing and analysis plan

Before analysis, we first remove missing values. Then, for each process feature in a data set, we adopt a 98% winsorization estimation ( Dixon, 1960 ) (set the values <1% of the whole values to the value at 1%, and those >99% to the value at 99%) to adjust spurious outliers. Winsorization is independent of data distribution and preserves the exact proportion of data points, thus being more flexible than other outlier removal methods that presume a normal distribution of data points.

For response data, we first show score distributions among students and summarize how many students appropriately applied the CVS in each test, and then show the most frequent (top 10) submitted answers.

For process data, we conduct the Kruskal–Wallis test to compare the duration features across score groups. If a significant value of p is reported by the test, we adopt another non-parametric test, the Wilcoxon signed-rank test, on pair-wised score groups to clarify which pair(s) of score groups have different means of the features. These two tests, implemented using kruskal.test and wilcox.test functions in the stats package in R 3.6.1 ( R Core Team, 2019 ), provide quantitative evidence on the relation between item scores and process features. Since there are three Kruskal–Wallis tests on the three measures, respectively, the critical p- value for identifying significance is set to 0.05/3 ≈ 0.0167.

To cross-validate the results of the Kurskal–Wallis and Wilcoxon signed-rank tests, we also conduct the omnibus ANOVA and pair-wised t tests (if the omnibus ANOVA test reports a significant value of p ) between score groups. The log-transformed (base e ) features pass the normality test (we use the Shapairo–Wilk’s method to test normality, and the p -values are all above 0.05, indicating that the distributions of the log-transformed data are not significantly distinct from a normal distribution). The ANOVA results are shown in the Supplementary materials .

4.1. The fair tests

The two fair tests show similar trends in score distribution and top 10 frequent submitted answers.

In the fair test 1, 41.4% of the students received the lowest score (1), 29.1% received a partial score (2), and only 29.5% properly applied the CVS and got a full score (3). In other words, the majority (over 70%) of the students failed to properly apply the CVS in this test. Figure 6A illustrates the top 10 frequently-submitted answers in this test. “Low; Low; Low” was the most frequent correct answer, and other correct ones (e.g., “Medium; Medium; Medium” and “High; High; High”) were less frequent; “Low; Medium; High,” an answer with totally-varied tree positions, was the most common incorrect answer, and its variants (e.g., “High; Medium; Low” or “Low; High; Medium”) were also common, all receiving the lowest score (1); and the answers having a partial score (2) (e.g., “Medium; Low; Medium”) were less frequent.

Top 10 frequent answers of the fair test 1 (A) and those of the fair test 2 (B) . Values on top of the bars are numbers of students, and those inside brackets are proportions.

In the fair test 2, only 28.8% of the students properly applied the CVS and got the full score (3), and most students had either the lowest score (1) (33.3%) or the partial score (2) (37.9%). Figure 6B shows that “1,4,7” was the most frequent correct answer, so was “3,6,9,” but other variants (e.g., “2,5,8” and “9,6,3”) were less frequent. “1,5,9,” an answer with totally varied ingredient amounts, was the most frequent incorrect answer. Others (e.g., “1,2,3,” “7,8,9” or “9,8,7”) that kept ingredient type consistent but varied ingredient amount were also frequent. Students who submitted these answers applied the CVS on a wrong variable. Other answers (e.g., “1,2,4” or “1,6,9”) that partially controlled the target variable of ingredient amount could not get the full score.

Table 4 shows the means and standard errors of the process features across score groups. As for the fair test 1, the Kruskal–Wallis tests report significant differences of these features across score groups ( PT , χ 2  = 12.2, df = 2, p  < 0.005; ET , χ 2  = 89.916, df = 2, p  < 0.001; MET , χ 2  = 64.776, df = 2, p  < 0.001). The omnibus ANOVA tests show similar results [ PT , F (2,1604) = 5.943, p  < 0.005; ET , F (2,1604) = 51.7, p  < 0.001; MET , F (2,1604) = 38.93, p  < 0.001].

Means and standard errors of PT , ET , and MET in each score group of the two fair tests.

Values (in seconds) outside brackets are means and those inside brackets are standard errors.

As for the fair test 2, the Kruskal–Wallis tests report marginally significant differences in PT ( χ 2  = 7.824, df = 2, p  = 0.02) and MET ( χ 2  = 6.600, df = 2, p  = 0.037) and significant differences in ET ( χ 2  = 78.111, df = 2, p  < 0.001) between score groups. The omnibus ANOVA tests show non-significant results for PT [ F (2,1987) = 2.744, p  = 0.065], but significant and marginally significant results for ET [ F (2,1987) = 37.53, p  < 0.001] and MET [ F (2,1987) = 3.451, p  = 0.032].

Table 5 shows the Wilcoxon signed-rank test results. In both fair tests, the students with higher scores had shorter ET than those with lower scores; and the full score students had shorter PT and MET than the lowest score students, but such differences were not statistically significant when the partial score group was involved.

Wilcoxon signed-rank test results between pair-wised score groups of the two fair tests.

“1” to “3” refer to score groups. Values outside brackets are test statistics, and those inside are p values. Significant (having p values <0.0167) results are marked in bold. Supplementary Table S1 shows the omnibus ANOVA and pair-wise t- tests results.

4.2. The exhaustive tests

The two exhaustive tests show similar results.

In the exhaustive test 1, 25.2% of the students received the lowest score (1), 33.9% properly applied the CVS and received the full score (4), and the rest got the partially high (3) (34.1%) or low (2) (6.8%) scores. In other words, the majority (over 65%) of the students failed to properly apply the CVS. Among the top 10 frequent answers (see Figure 7 ), “A; B; A + B; None” and its variants “A; A + B; B; None” and “A + B; A; B; None” received the full score, but they were less frequent than “A + B,” “B,” “A,” and “None,” which were the most frequent incorrect answers with the lowest score. Answers having partially high (e.g., “A; A + B; None”) or low (e.g., “A; A + B”) scores were less frequent.

Top 10 frequent answers of the exhaustive test 1. Values on top of bars are numbers of students, and those inside brackets are percentages of students.

In the exhaustive test 2, many students received the lowest score (1) (26.8%) and partially low score (2) (60.6%), and only 5.8% properly applied the CVS and got the full score (4), and 6% got the partially high score (3). Due to extremely numerous cases of submitted answers and equivalence of submitted answers, we discuss the frequent answers based on Figure 4 and the scoring rubric in Table 3 . Most students got the partially low score (1), their submitted answers did not ensure that at least one location in each of the 13 regions was chosen; instead, they chose over 2 locations in the three regions adjacent to City A/City B, indicating that they failed to figure out the two requirements (see Section 3.2) of this test.

Table 6 shows the means and standard errors of the process features across score groups. In the exhaustive test 1, the Kruskal–Wallis tests report significant feature differences across score groups ( PT , χ 2  = 133.57, df = 3, p  < 0.001; ET , χ 2  = 498.49, df = 3, p  < 0.001; MET , χ 2  = 258.97, df = 3, p  < 0.001). The omnibus ANOVA tests show similar results [ PT , F (3,2721) = 65.4, p  < 0.001; ET , F (3,2721) = 224.5, p  < 0.001; MET , F (3,2721) = 78.4, p  < 0.001].

Means and standard errors of PT , ET , and MET across the score groups of the exhaustive tests 1 and 2.

Values (in seconds) outside brackets are means and those inside are standard errors.

In the exhaustive test 2, the Kruskal–Wallis tests report marginally significant difference in PT ( χ 2  = 10.317, df = 3, p  < 0.017), and significantly differences in ET ( χ 2  = 440.33, df = 3, p  < 0.001) and MET ( χ 2  = 158.79, df = 3, p  < 0.001). The omnibus ANOVA tests also show (marginally) significant differences in PT [ F (3,2942) = 3.094, p  = 0.026], ET [ F (3,2942) = 185.7, p  < 0.001], and MET [ F (3,2942) = 45.58, p  < 0.001].

Table 7 shows the Wilcoxon signed-rank test results. Similar to the fair tests, the full score students had shorter PT and MET than other low score students; but unlike the fair tests, the full score students had longer ET than most of the other score students. The patterns might not be consistent when partial score groups were involved.

Wilcoxon signed-rank test results between pair-wised score groups of the two exhaustive tests.

“1” to “4” refer to score groups. Values outside brackets are test statistics, and those inside are p- values. Significant (having p- values <0.0167) results are marked in bold. Supplementary Table S2 shows the omnibus ANOVA and pair-wise t- tests results.

5. Discussions

5.1. problem solving processes of high- and low-performing students.

This study examined two fair tests and two exhaustive tests from the NAEP scientific inquiry tasks, which require students to apply the control-of-variables strategy to design controlled experiments. We propose three process features to reflect the major stages of problem solving and use them to investigate performances of students having various levels of problem solving competency. In both types of tests, high- and low-performing students exhibited distinct response and process patterns.

In terms of response, more than 70% of the fourth- and eighth-graders failed to properly apply the control-of-variables strategy in the fair tests, and over 80% of the twelfth-graders failed to do so in the exhaustive tests. These are consistent with previous literature ( Chen and Klahr, 1999 ).

In the fair test 1, the most common strategy was to vary tree position in mountain, e.g., “Low; Medium; High” (and its variations) (see Figure 6A ). In the fair test 2, the most common strategy was to vary ingredient amount, e.g., “1,5,9” (and its variations) (see Figure 6B ). These similar results are in line with early observations in response data (e.g., Shimoda et al., 2002 ): students adopting inappropriate strategies failed to recognize that variation in this extraneous variable actually interfered the effect of the target variable.

In the exhaustive test 1, the common wrong strategies were to save (and submit) only one of the four possible conditions. In the exhaustive test 2, the common wrong strategies were to select locations mainly in the regions adjacent to a city but ignore those in between. These inappropriate strategies reveal that: the low-performing students in these tests failed to conceive an exhaustive set of test data for the controlled experiments, probably due to lacking intention or required skills, and as a consequence, they simply submitted a subset of test data or some guessed answers. These results are in line with early studies ( Tschirgi, 1980 ).

In terms of process, consistent patterns are evident in the process features. As for preparation time, in the fair tests, compared to students with the lowest score, those with a full score tended to spend shorter preparation time before making their first answer-related action. Longer preparation in students with the lowest score indicates that they needed more time to understand the test and plan their activities, whereas high-performing students could efficiently do so. This difference at the planning stage reveals that whether a student can properly solve a problem depends on whether he/she efficiently grasps the instructions and plans the activities before any is made. Apparently contradictive to the intuition that longer planning leads to better outcome, our finding is supported by results from other time-constrained tasks, e.g., a shorter pre-writing pause (duration between the moment a student entered the item and when he/she made the first typing event) in high-performing students in a time-constrained writing test, indicating efficient task planning ( Zhang et al., 2017 ).

Patterns in preparation time between the high- and low-performing students were not consistent in the exhaustive tests. In the exhaustive test 1, students with the full score spent less preparation time than those with the lowest score, but in the exhaustive test 2, such pattern disappeared. The number of exhaustive combinations in the exhaustive test 1 (4) is much smaller than that in the exhaustive test 2 (15). Therefore, in the exhaustive test 2, both students with lower scores and those with the full score might not be able to foresee all required combinations at the planning stage, so they simply started right away to make selections and think along with the process of answer formation. This leads to non-significant difference in preparation time between the high- and low-performing students in this test.

As for execution time and mean execution time, in the two fair tests, most students with the lowest score spent longer execution time in conducting the drag-and-drop actions than those with higher scores (see Tables 4 , ​ ,5). 5 ). In these tests, the minimum number of actions required to construct an answer was just 3: drag and drop each type of trees (or three piles of different ingredients) respectively at the same (or different) positions of three mountains (or the three slots). There were two situations that caused longer execution time in students with the lowest scores: they either spent more time in executing individual actions or kept revising their choices, 7 both reflecting hesitation or uncertainty during the action execution stage of problem solving. The process feature of mean execution time (see Tables 4 , ​ ,5) 5 ) explicitly reveals that on average, students with the lowest score spent more time on conducting each of their answer-related actions, i.e., they were less efficient in action execution than those with the full score.

Unlike the fair tests, in the exhaustive tests, most students with the lowest score showed shorter execution time than those with higher scores in answer formulation (see Tables 6 , ​ ,7). 7 ). According to Table 3 , low scores in these tests correspond to incomplete submissions. The longest execution time of most full score students suggests that they were well motivated and had endeavored in constructing and saving all possible conditions, even at the cost of spending more time in total. By contrast, the shorter execution times of the lower score students were mostly caused by two cases: (1) they did not spend much time exploring the conditions and finished the test with lack-of-thinking results, which reflected low motivation/engagement or lack of reasonable understanding; (2) without realizing that they needed to submit all possible conditions, some students left the test after submitting just one condition (consistent with the frequent wrong answers).

As for mean execution time, in the exhaustive test 1, though spending more time in problem solving, most students with the full score showed shorter mean execution time than those with the lower scores (see Tables 5 , ​ ,6). 6 ). This indicates that most of high-performing students efficiently formulated their answers. In the exhaustive test 2, although spending longer time in selecting multiple locations for comparison, most high-performing students had smaller or comparable mean execution time to that of low-performing students, who submitted incomplete answers. To sum up, in both tests, high-performing students tended to be more efficient in executing multiple answer-related actions than low-performing ones.

5.2. Process features and problem-solving competency

In all four tests, most students who properly applied the control-of-variables strategy (thus having high problem-solving competency) enacted more goal-oriented behaviors ( Shimoda et al., 2002 ). In the fair tests, they quickly grasped the goal at the planning stage, and efficiently set up the conditions matching the fair test requirement; in the exhaustive tests, with a clear goal in mind, they persistently constructed all the conditions for comparison within a longer execution time. By contrast, students having low problem-solving competency were confused about the target variable while formulating answers in the fair tests; in the exhaustive tests, they either ignored or did not fully understand the goal, and tended to drop before submitting enough conditions.

The proposed process features of execution time and mean execution time reflect the level differences in goal-orientation and motivation between students, which are crucial to problem solving ( Gardner, 2006 ; Dörner and Güss, 2013 ; Güss et al., 2017 ). The contrasting patterns of execution time between the two types of tests reveal different characteristics of the solutions and execution stages therein; the fair tests need conditions matching the fair test requirement, yet the exhaustive tests request all possible conditions. They also reveal that task property could influence how students deploy strategies that they seem to know, which echoes the knowledge-practice integration in NGSS.

The consistent patterns of mean execution time in high-performing students across the two types of tests indicate that both types of tests require similar control-of-variables strategies and high-performing students can efficiently apply such strategies in solving apparently-distinct problems. This suggests that the capabilities of doing analogical reasoning and employing key skills and related abilities across tasks of various contents are critical in scientific problem solving.

Most of the above discussions concern the full and lowest score students, because the statistical tests report consistent results between these groups in each test. Inconsistent results exist between partial score groups or between a partial score group and the full (or the lowest) score group. This inconsistency is due to several reasons. First, some partial score groups contained fewer students than others. Second, as in the scoring rubrics, the response difference between the full (or the lowest) and a partial score is smaller than that between the full and the lowest scores. Both of these factors decimated the statistical power of the analyses. Third, due to lacking empirical bases ( Parshall and Brunner, 2017 ), the predefined rubrics might not be able to clearly differentiate students with different levels of problem solving competency. The reliability of scoring rubrics is worth further investigation, but it is beyond the scope of the current study.

The discussions on problem solving process and competency based on process features of high- and low-performing students in different tests provide useful insights on teaching and learning of the control-of-variables strategies and related skills as well as applying them in similar scientific inquiry tasks. For example, comparing a specific student’s performance with the typical patterns of high-performing students can reveal on which problem solving stage the student needs to improve efficiency; comparing high- and low-performing students’ process patterns can also reveal on which aspects the low-performing students need to polish, e.g., how to allocate time and effort in different problem-solving stages in order to improve overall performance in scientific inquiry tasks.

5.3. Precision of process features

The temporal features of preparation time and execution time roughly estimate the process of action planning and that of action execution, respectively. In addition to individual differences, other factors may “contaminate” these features, especially in complex tasks requiring careful thinking and multiple answer formulation stages; e.g., students may change part of their answers during the problem solving process, and execution time may cover the time of answer change.

Answer change is part of action execution. In all four tests, most students conducted answer change through drag-and-drop actions. For example, in the fair test 1, the minimum number of drag-and-drop actions for correctly answering the question is 3, but only 11% of the students conducted exactly 3 drag-and-drop actions, and more than 50% conducted 3 to 6 actions; in the exhaustive test 1, the minimum number of saved conditions for a correct answer is 4, but only 23% of the students saved exactly 4 cases, and more than 90% saved 4–6 cases. In addition, answer change actions are often intertwined with answer formulation actions, indicating that the purpose of such actions is to correct execution error and stick to planned actions. In this sense, answer change is part of action execution, and their durations should be included into execution time.

However, students might occasionally clear all the answers and re-answer the question from scratch. In this case, they could spend some time to re-plan their actions, but such time is embedded in the current definition of execution time. In the four tests, very few (<1%) students went through such re-planning and re-execution process, but in complex tasks, such cases may be ample. To better clarify such cases, we need to improve the precision of process features by examining drag-and-drop action sequences and their time stamps to clearly identify whether a student re-planned. We leave such modification to future work.

5.4. Procedure of process data use

In addition to the process features and insights on scientific problem solving, this study lays out a general procedure of using process data to study test-takers’ performance or competency:

Discover or define process features that could (potentially) inform test-takers’ performance or competency . This step is often based on prior hypotheses or existing studies;

Demonstrate correlation or relatedness between process features and test-takers’ performance . This step is critical in two aspects. First, it verifies whether the features are related to performance in the target dataset. Second, it bridges the first and third steps; only after relatedness or correlation between test-takers’ performance and the process features is validated would analyses on these features and derived understandings become meaningful.

Understand or characterize test-takers’ performance, or incorporate process features into scoring rubrics, cognitive or measurement models . Understanding test-takers’ performance is based on defined or discovered features in the first step. In our study, the proposed features characterize high- and low-performing (or common vs. abnormal) test-takers. The observed consistent patterns of process features also pave the way for incorporating those features into scoring rubrics, e.g., specific values or ranges of values of process features correspond to various scales of scores. Moreover, the quantitative process features as in our study could serve as important components in cognitive or measurement models to predict, classify, or interpret test-takers’ performance.

6. Conclusion

This study proposes three process features and an analytical procedure of process data use. Based on four scientific inquiry tasks, we investigate how students apply the control-of-variables strategy in typical fair and exhaustive tests and how the process features characterize high- and low-performing students in these tasks. Although (meta)cognitive processes cannot be observed directly from process data, the proposed features have proven values in elucidating the planning and executing stage of problem solving, characterizing students’ performance patterns, and revealing relatedness among capacities (the control-of-variables strategy), test properties (the fair and exhaustive tests), and performance (answers, scores, and answering process). Our study demonstrates that process data provide unique windows to interpret students’ performance beyond scores, and that a combination of analytical procedures and process data helps infer students’ problem-solving strategies, fill in the gap in early studies, and stimulate future work on process features reflecting problem-solving performance.

Data availability statement

Ethics statement.

The studies involving human participants were reviewed and approved by NCES. Written informed consent to participate in this study was provided by the participants’ legal guardian/next of kin.

Author contributions

TG and LS designed the study. TG collected the data and conducted the analysis and wrote the manuscript. TG, LS, YJ, and BA discussed the results. LS, YJ, and BA edited the manuscript. All authors contributed to the article and approved the submitted version.

Conflict of interest

TG was employed by the company Google.

The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Acknowledgments

We thank Madeleine Keehner, Kathleen Scalise, Christopher Agard, and Gary Feng from ETS for guidance on this work. Preliminary results of the paper were reported on the 13th International Conference on Educational Data Mining (EDM 2020).

1 https://nces.ed.gov/

2 Such data include, but are not limited to: student events and their time stamps , e.g., drag-and-drop, (de)selection, or tool-use actions during answer formulation, text (re)typing or editing behaviors during keyboard-based writing, or navigations across pages, scenes, or questions during on-screen reading; and system events and their time stamps , e.g., entering/leaving scenes, (de)activating on-scene tools or popping-up messages.

3 In addition to durations, one can measure numbers/sequence of answer-related events in the test scene. Such features have some uncertainties: greater/fewer events do not necessarily require more/less effort, more events sometimes indicate low competency, and it is non-trivial to align such features with performance (scores). The duration features proposed in our study can overcome these uncertainties by explicitly measuring students’ planning and executing stages and relating them to performance.

4 Some recent studies begun to touch upon duration features. For example, Arslan et al. (2020) reported no significant correlations between item scores and preparation (and execution) times in math items. Jiang et al. (2021) investigated action sequences and response strategies derived. One can also measure ratios between durations. If durations can reflect the planning and executing stages during problem solving and characterize performance patterns of students, ratios between durations can further reveal relative cognitive loads between the planning and executing processes. We leave such ratio-based features in future work.

5 a.k.a. "isolation of variables" ( Inhelder and Piaget, 1958 ), "vary one thing at a time" ( Tschirgi, 1980 ), or "control of variables strategy" ( Chen and Klahr, 1999 ).

6 Due to privacy and secure nature of NAEP data, we present conceptually equivalent cover tasks to maintain task security. The cover tasks have similar underlying structures and require similar cognitive processes to solve, but do not connect to specific science contents as the real tasks.

7 Both can be identified from event logs, and action frequency data can further clarify which situation is more popular.

Supplementary material

The Supplementary material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyg.2023.1131019/full#supplementary-material

• Arslan B., Jiang Y., Keehner M., Gong T., Katz I. R., Yan F. (2020). The effect of drag-and-drop item features on test-taker performance and response strategies . Educ. Meas. Issues Pract. 39 , 96–106. doi: 10.1111/emip.12326 [ CrossRef ] [ Google Scholar ]
• Bergner Y., von Davier A. A. (2019). Process data in NAEP: past, present, and future . J. Educ. Behav. Stat. 44 , 706–732. doi: 10.3102/1076998618784700 [ CrossRef ] [ Google Scholar ]
• Black R. (2007). Pragmatic software testing: Becoming an effective and efficient test professional . New York: Wiley. [ Google Scholar ]
• Bryant W. (2017). Developing a strategy for using technology-enhanced items in large-scale standardized tests . Pract. Assess. Res. Eval. 22 , 1–10. doi: 10.7275/70yb-dj34 [ CrossRef ] [ Google Scholar ]
• Chen Z., Klahr D. (1999). All other things being equal: acquisition and transfer of the control-of-variables strategy . Child Dev. 70 , 1098–1120. doi: 10.1111/1467-8624.00081, PMID: [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Dixon W. J. (1960). Simplified estimation from censored normal samples . Ann. Math. Stat. 31 , 385–391. doi: 10.1214/aoms/1177705900 [ CrossRef ] [ Google Scholar ]
• Dörner D., Güss C. D. (2013). PSI: a computational architecture of cognition, motivation, and emotion . Rev. Gen. Psychol. 17 , 297–317. doi: 10.1037/a0032947 [ CrossRef ] [ Google Scholar ]
• Dostál J. (2015). Theory of problem solving . Procedia Soc. Behav. Sci. 174 , 2798–2805. doi: 10.1016/j.sbspro.2015.01.970 [ CrossRef ] [ Google Scholar ]
• Ebenezer J., Kaya O. N., Ebenezer D. L. (2011). Engaging students in environmental research projects: perceptions of fluency with innovative technologies and levels of scientific inquiry abilities . J. Res. Sci. Teach. 48 , 94–116. doi: 10.1002/tea.20387 [ CrossRef ] [ Google Scholar ]
• Gardner E. A. (2006). Instruction in mastery goal orientation: developing problem solving and persistence for clinical settings . J. Nurs. Educ. 45 , 343–347. doi: 10.3928/01484834-20060901-03, PMID: [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Gobert J. D., Sao Pedro M. A., Baker R. S. J. D., Toto E., Montalvo O. (2012). Leveraging educational data mining for real-time performance assessment of scientific inquiry skills within microworlds . J. Educ. Data Mining 4 , 104–143. doi: 10.5281/zenodo.3554645 [ CrossRef ] [ Google Scholar ]
• Gong T., Jiang Y., Saldivia L. E., Agard C. (2021). Using Sankey diagrams to visualize drag and drop action sequences in technology-enhanced items . Behaiv. Res. Methods 54 , 117–132. doi: 10.3758/s13428-021-01615-4, PMID: [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Gong T., Zhang M., Li C. (2022). Association of keyboarding fluency and writing performance in online-delivered assessment . Assess. Writ. 51 :100575. doi: 10.1016/j.asw.2021.100575 [ CrossRef ] [ Google Scholar ]
• Griffiths T. L., Lieder F., Goodman N. D. (2015). Rational use of cognitive resources: levels of analysis between the computational and the algorithmic . Top. Cogn. Sci. 7 , 217–229. doi: 10.1111/tops.12142, PMID: [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Grindal M., Offutt J., Andler S. F. (2005). Combination testing strategies: a survey . Soft. Testing, Verif. Reliab. 15 , 167–199. doi: 10.1002/stvr.319 [ CrossRef ] [ Google Scholar ]
• Guo H., Zhang M., Deane P. D., Bennett R. E. (2019). Writing process differences in subgroups reflected in keystroke logs . J. Educ. Behav. Stat. 44 , 571–596. doi: 10.3102/1076998619856590 [ CrossRef ] [ Google Scholar ]
• Güss C. D., Burger M. L., Dörner D. (2017). The role of motivation in complex problem solving . Front. Psychol. 8 :851. doi: 10.3389/fpsyg.2017.00851, PMID: [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Han Z., He Q., von Davier M. (2019). Predictive feature generation and selection using process data from PISA interactive problem-solving items: an application of random forests . Front. Psychol. 10 :2461. doi: 10.3389/fpsyg.2019.02461, PMID: [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Hoyles C., Noss R. (2003). “ What can digital technologies take from and bring to research in mathematics education? ” in Second international handbook of mathematics education . eds. Bishop A., Clements M. A. K., Keitel-Kreidt C., Kilpatrick J., Leung F. K.-S. (Dordrecht: Springer; ), 323–349. [ Google Scholar ]
• Inhelder B., Piaget J. (1958). The growth of logical thinking from childhood to adolescence: An essay on the construction of formal operational structures . London: Routledge and Kegan Paul. [ Google Scholar ]
• Jiang Y., Gong T., Saldivia L. E., Cayton-Hodges G., Agard C. (2021). Using process data to understand problem-solving strategies and processes in large-scale mathematics assessments . Large-Scale Assess. Educ. 9 , 1–31. doi: 10.1186/s40536-021-00095-4 [ CrossRef ] [ Google Scholar ]
• Jonassen D. H. (2000). Toward a design theory of problem solving . Educ. Technol. Res. Dev. 48 , 63–85. doi: 10.1007/BF02300500 [ CrossRef ] [ Google Scholar ]
• Kim M. C., Hannafin M. J., Bryan L. A. (2007). Technology-enhanced inquiry tools in science education: an emerging pedagogical framework for classroom practice . Sci. Educ. 91 , 1010–1030. doi: 10.1002/sce.20219 [ CrossRef ] [ Google Scholar ]
• Klahr D., Nigam M. (2004). The equivalence of learning paths in early science instruction: effects of direct instruction and discovery learning . Psychol. Sci. 15 , 661–667. doi: 10.1111/j.0956-7976.2004.00737.x, PMID: [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Koedinger K. R., Corbett A. (2006). “ Cognitive tutors: technology bringing learning science to the classroom ” in The Cambridge handbook of the learning sciences . ed. Sawyer K. (Cambridge, MA: Cambridge University Press; ), 61–78. [ Google Scholar ]
• Kruskal W. H., Wallis W. A. (1952). Use of ranks in one-criterion variance analysis . J. Am. Stat. Assoc. 47 , 583–621. doi: 10.1080/01621459.1952.10483441 [ CrossRef ] [ Google Scholar ]
• Kuhn D. (2007). Reasoning about multiple variables: control of variables is not the only challenge . Sci. Educ. 91 , 710–726. doi: 10.1002/sce.20214 [ CrossRef ] [ Google Scholar ]
• Kuhn D., Dean D. (2005). Is developing scientific thinking all about learning to control variables? Psychol. Sci. 16 , 866–870. doi: 10.1111/j.1467-9280.2005.01628.x [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Lee Y., Jia Y. (2014). Using response time to investigate students’ test-taking behaviors in a NAEP computer-based study . Large-Scale Assess. Educ. 2 , 1–24. doi: 10.1186/s40536-014-0008-1 [ CrossRef ] [ Google Scholar ]
• Lesh R., Harel G. (2011). Problem solving, modeling, and local conceptual development . Math. Think. Learn. 5 , 157–189. doi: 10.1080/10986065.2003.9679998 [ CrossRef ] [ Google Scholar ]
• Lieder F., Griffiths T. L. (2017). Strategy selection as rational metareasoning . Psychol. Rev. 124 , 762–794. doi: 10.1037/rev0000075, PMID: [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Liu O. L., Bridgeman B., Gu L., Xu J., Kong N. (2015). Investigation of response changes in the GRE revised general test . Educ. Psychol. Meas. 75 , 1002–1020. doi: 10.1177/0013164415573988, PMID: [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Montgomery D. C. (2000). Design and analysis of experiments (5th). Indianapolis, IN: Wiley Text Books. [ Google Scholar ]
• Moon J. A., Keehner M., Katz I. R. (2018). Affordances of item formats and their effects on test-taker cognition under uncertainty . Educ. Meas. Issues Pract. 38 , 54–62. doi: 10.1111/emip.12229 [ CrossRef ] [ Google Scholar ]
• National Assessment Governing Board . (2015). Science framework for the 2015 national assessment of educational progress. Washington, DC. Available at: https://www.nagb.gov/naep-frameworks/science/2015-science-framework.html
• Parshall C. G., Brunner B. (2017). “ Content development and review ” in Testing in the professions: Credentialing policies and practice . eds. Davis-Becker S., Buckendahl C. W. (Abingdon: Routledge; ), 85–104. [ Google Scholar ]
• Pedaste M., Mäeots M., Siiman L. A., de Jong T., van Riesen S. A. N., Kamp E. T., et al.. (2015). Phases of inquiry-based learning: definitions and the inquiry cycle . Educ. Res. Rev. 14 , 47–61. doi: 10.1016/j.edurev.2015.02.003 [ CrossRef ] [ Google Scholar ]
• Provasnik S. (2021). Process data, the new frontier for assessment development: rich new soil or a quixotic quest? Large Scale Assess. Educ. 9 :1. doi: 10.1186/s40536-020-00092-z [ CrossRef ] [ Google Scholar ]
• R Core Team . (2019). R: A language and environment for statistical computing . R Foundation for Statistical Computing, Vienna, Austria. [ Google Scholar ]
• Scalise K., Gifford B. (2006). Computer-based assessment in E-learning: a framework for constructing “intermediate constraint” questions and tasks for technology platforms . J. Technol. Learn. Assess. 4 , 3–44. Available at: https://ejournals.bc.edu/index.php/jtla/article/view/1653 [ Google Scholar ]
• Schwichow M., Croker S., Zimmerman C., Höffler T. N., Haertig H. (2016). Teaching the control-of-variables strategy: a meta analysis . Dev. Rev. 39 , 37–63. doi: 10.1016/j.dr.2015.12.001 [ CrossRef ] [ Google Scholar ]
• Shimoda T. A., White B. Y., Frederiksen J. R. (2002). Student goal orientation in learning inquiry skills with modifiable software advisors . Sci. Educ. 86 , 244–263. doi: 10.1002/sce.10003 [ CrossRef ] [ Google Scholar ]
• Tang X., Wang Z., He Q., Liu J., Ying Z. (2019). Latent feature extraction for process data via multidimensional scaling . Psychometrika 85 , 378–397. doi: 10.1007/s11336-020-09708-3 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Tschirgi J. E. (1980). Sensible reasoning: a hypothesis about hypotheses . Child Dev. 51 , 1–10. doi: 10.2307/1129583 [ CrossRef ] [ Google Scholar ]
• Ulitzsch E., He Q., Pohl S. (2021). Using sequence mining techniques for understanding incorrect behavioral patterns on interactive tasks . J. Educ. Behav. Stat. 47 , 3–35. doi: 10.3102/10769986211010467 [ CrossRef ] [ Google Scholar ]
• Zenisky A. L., Sireci S. G. (2002). Technological innovations in large-scale assessment . Appl. Meas. Educ. 15 , 337–362. doi: 10.1207/S15324818AME1504_02 [ CrossRef ] [ Google Scholar ]
• Zhang M., Zou D., Wu A. D., Deane P., Li C. (2017). “ An investigation of the writing processes in timed task condition using keystrokes ” in Understanding and investigating writing processes in validation research . eds. Zumbo B. D., Hubley A. M. (New York, NY: Springer; ), 321–339. [ Google Scholar ]

• school Campus Bookshelves
• menu_book Bookshelves
• perm_media Learning Objects
• login Login
• how_to_reg Request Instructor Account
• hub Instructor Commons
• Download Page (PDF)
• Download Full Book (PDF)
• Periodic Table
• Physics Constants
• Scientific Calculator
• Reference & Cite
• Tools expand_more
• Readability

selected template will load here

This action is not available.

1.3: Scientific Inquiry

• Last updated
• Save as PDF
• Page ID 44278

Learning Objectives

• Compare inductive reasoning with deductive reasoning
• Describe the process of scientific inquiry

One thing is common to all forms of science: an ultimate goal “to know.” Curiosity and inquiry are the driving forces for the development of science. Scientists seek to understand the world and the way it operates. Two methods of logical thinking are used: inductive reasoning and deductive reasoning.

Inductive reasoning is a form of logical thinking that analyzes trends or relationships in data to arrive at a general conclusion. A scientist makes observations and records them. These data can be qualitative (descriptive) or quantitative (consisting of numbers), and the raw data can be supplemented with drawings, pictures, photos, or videos. From many observations, a scientist can draw conclusions based on evidence. In other words, inductive reasoning involves making generalizations from careful observation and the analysis of a large amount of individual data points. Generalizations arrived at through inductive reasoning are not always correct.

Deductive reasoning is another form of logical thinking that begins from a general principle or law and applies it to a specific circumstance to predict specific results. From a set of general principles, a scientist can extrapolate and predict specific results that will always be correct as long as the general principles they start from are correct.

Deductive reasoning and inductive reasoning move in opposite directions – inductive reasoning goes from individual observations to broad generalizations while deductive reasoning goes from general principles to specific decisions or predictions.

Both types of logical thinking are related to the two main pathways of scientific study: descriptive science and hypothesis-based science. Descriptive science (or discovery science) aims to observe, explore, and discover, while h ypothesis-based science begins with a specific question or problem and a potential answer or solution that can be tested. Inductive reasoning is used most often in descriptive science, while deductive reasoning is used most often in hypothesis-based science. The boundary between these two forms of study is often blurred, because most scientific endeavors combine both approaches. Observations lead to questions, questions lead to forming a hypothesis as a possible answer to those questions, and then the hypothesis is tested. Thus, descriptive science and hypothesis-based science are in continuous dialogue.

Hypothesis Testing

Biologists study the living world by posing questions about it and seeking science-based responses. This approach is common to other sciences as well and is often referred to as the scientific method. The scientific method was used even in ancient times, but it was first documented by England’s Sir Francis Bacon (1561–1626) (Figure 1), who set up inductive methods for scientific inquiry. The scientific method is not exclusively used by biologists but can be applied to almost anything as a logical problem-solving method.

The scientific process typically starts with an observation (often a problem to be solved) that leads to a question. Let’s think about a simple problem that starts with an observation and apply the scientific method to solve the problem. One Monday morning, a student arrives at class and quickly discovers that the classroom is too warm. That is an observation that also describes a problem: the classroom is too warm. The student then asks a question: “Why is the classroom so warm?”

Recall that a hypothesis is a suggested explanation that can be tested. To solve a problem, several hypotheses may be proposed. For example, one hypothesis might be, “The classroom is warm because no one turned on the air conditioning.” But there could be other responses to the question, and therefore other hypotheses may be proposed. A second hypothesis might be, “The classroom is warm because there is a power failure, and so the air conditioning doesn’t work.”

Once a hypothesis has been selected, a prediction may be made. A prediction is similar to a hypothesis but it typically has the format “If . . . then . . . .” For example, the prediction for the first hypothesis might be, “ If the student turns on the air conditioning, then the classroom will no longer be too warm.”

A hypothesis must be testable to ensure that it is valid. For example, a hypothesis that depends on what a bear thinks is not testable, because it can never be known what a bear thinks. It should also be falsifiable , meaning that it can be disproven by experimental results. An example of an unfalsifiable hypothesis is “Botticelli’s Birth of Venus is beautiful.” There is no experiment that might show this statement to be false. To test a hypothesis, a researcher will conduct one or more experiments designed to eliminate one or more of the hypotheses. This is important. A hypothesis can be disproven, or eliminated, but it can never be proven. Science does not deal in proofs like mathematics. If an experiment fails to disprove a hypothesis, then we find support for that explanation, but this is not to say that down the road a better explanation will not be found, or a more carefully designed experiment will be found to falsify the hypothesis.

Scientific inquiry has not displaced faith, intuition, and dreams. These traditions and ways of knowing have emotional value and provide moral guidance to many people. But hunches, feelings, deep convictions, old traditions, or dreams cannot be accepted directly as scientifically valid. Instead, science limits itself to ideas that can be tested through verifiable observations. Supernatural claims that events are caused by ghosts, devils, God, or other spiritual entities cannot be tested in this way.

Practice Question

Your friend sees this image of a circle of mushrooms and excitedly tells you it was caused by fairies dancing in a circle on the grass the night before. Can your friend’s explanation be studied using the process of science?

[reveal-answer q=”665464″] Show Answer [/reveal-answer] [hidden-answer a=”665464″]In theory, you might try to observe the fairies. But fairies are magical or supernatural beings. We have never observed them using any verifiable method, so scientists agree that they cannot be studied using scientific tools. Instead, science has an explanation supported by strong evidence: “fairy rings” result when a single colony of fungus spreads out into good habitat over a period of many years. The core area is clear of mushrooms because the soil nutrients have been partly depleted there. This idea can be evaluated with repeated observations over time using chemical soil tests and other verifiable measurements.[/hidden-answer]

Each experiment will have one or more variables and one or more controls. A variable is any part of the experiment that can vary or change during the experiment. A control is a part of the experiment that does not change. Look for the variables and controls in the example that follows. As a simple example, an experiment might be conducted to test the hypothesis that phosphate limits the growth of algae in freshwater ponds. A series of artificial ponds are filled with water and half of them are treated by adding phosphate each week, while the other half are treated by adding a salt that is known not to be used by algae. The variable here is the phosphate (or lack of phosphate), the experimental or treatment cases are the ponds with added phosphate and the control ponds are those with something inert added, such as the salt. Just adding something is also a control against the possibility that adding extra matter to the pond has an effect. If the treated ponds show lesser growth of algae, then we have found support for our hypothesis. If they do not, then we reject our hypothesis. Be aware that rejecting one hypothesis does not determine whether or not the other hypotheses can be accepted; it simply eliminates one hypothesis that is not valid (Figure 2). Using the scientific method, the hypotheses that are inconsistent with experimental data are rejected.

In the example below, the scientific method is used to solve an everyday problem. Which part in the example below is the hypothesis? Which is the prediction? Based on the results of the experiment, is the hypothesis supported? If it is not supported, propose some alternative hypotheses.

• My toaster doesn’t toast my bread.
• Why doesn’t my toaster work?
• There is something wrong with the electrical outlet.
• If something is wrong with the outlet, my coffeemaker also won’t work when plugged into it.
• I plug my coffeemaker into the outlet.
• My coffeemaker works.

[practice-area rows=”4″][/practice-area] [reveal-answer q=”41039″]Show Answer[/reveal-answer] [hidden-answer a=”41039″]The hypothesis is #3 (there is something wrong with the electrical outlet), and the prediction is #4 (if something is wrong with the outlet, then the coffeemaker also won’t work when plugged into the outlet). The original hypothesis is not supported, as the coffee maker works when plugged into the outlet. Alternative hypotheses may include (1) the toaster might be broken or (2) the toaster wasn’t turned on.[/hidden-answer]

In practice, the scientific method is not as rigid and structured as it might at first appear. Sometimes an experiment leads to conclusions that favor a change in approach; often, an experiment brings entirely new scientific questions to the puzzle. Many times, science does not operate in a linear fashion; instead, scientists continually draw inferences and make generalizations, finding patterns as their research proceeds. Scientific reasoning is more complex than the scientific method alone suggests.

Contributors and Attributions

• Concepts of Biology. Provided by : OpenStax CNX. Located at : http://cnx.org/contents/[email protected] . License : CC BY: Attribution . License Terms : http://cnx.org/contents/[email protected]
• Practice Question (Scientific Inquiry). Provided by : Open Learning Initiative. Located at : https://oli.cmu.edu/jcourse/workbook/activity/page?context=434a5c2680020ca6017c03488572e0f8 . Project : Introduction to Biology (Open + Free). License : CC BY-NC-SA: Attribution-NonCommercial-ShareAlike

• SUGGESTED TOPICS
• The Magazine
• Newsletters
• Managing Yourself
• Managing Teams
• Work-life Balance
• The Big Idea
• Data & Visuals
• Reading Lists
• Case Selections
• HBR Learning
• Topic Feeds
• Account Settings
• Email Preferences

Share Podcast

Do You Understand the Problem You’re Trying to Solve?

To solve tough problems at work, first ask these questions.

• Apple Podcasts
• Google Podcasts

Problem solving skills are invaluable in any job. But all too often, we jump to find solutions to a problem without taking time to really understand the dilemma we face, according to Thomas Wedell-Wedellsborg , an expert in innovation and the author of the book, What’s Your Problem?: To Solve Your Toughest Problems, Change the Problems You Solve .

In this episode, you’ll learn how to reframe tough problems by asking questions that reveal all the factors and assumptions that contribute to the situation. You’ll also learn why searching for just one root cause can be misleading.

Key episode topics include: leadership, decision making and problem solving, power and influence, business management.

HBR On Leadership curates the best case studies and conversations with the world’s top business and management experts, to help you unlock the best in those around you. New episodes every week.

• Listen to the original HBR IdeaCast episode: The Secret to Better Problem Solving (2016)
• Find more episodes of HBR IdeaCast
• Discover 100 years of Harvard Business Review articles, case studies, podcasts, and more at HBR.org .

HANNAH BATES: Welcome to HBR on Leadership , case studies and conversations with the world’s top business and management experts, hand-selected to help you unlock the best in those around you.

Problem solving skills are invaluable in any job. But even the most experienced among us can fall into the trap of solving the wrong problem.

Thomas Wedell-Wedellsborg says that all too often, we jump to find solutions to a problem – without taking time to really understand what we’re facing.

He’s an expert in innovation, and he’s the author of the book, What’s Your Problem?: To Solve Your Toughest Problems, Change the Problems You Solve .

  In this episode, you’ll learn how to reframe tough problems, by asking questions that reveal all the factors and assumptions that contribute to the situation. You’ll also learn why searching for one root cause can be misleading. And you’ll learn how to use experimentation and rapid prototyping as problem-solving tools.

This episode originally aired on HBR IdeaCast in December 2016. Here it is.

SARAH GREEN CARMICHAEL: Welcome to the HBR IdeaCast from Harvard Business Review. I’m Sarah Green Carmichael.

Problem solving is popular. People put it on their resumes. Managers believe they excel at it. Companies count it as a key proficiency. We solve customers’ problems.

The problem is we often solve the wrong problems. Albert Einstein and Peter Drucker alike have discussed the difficulty of effective diagnosis. There are great frameworks for getting teams to attack true problems, but they’re often hard to do daily and on the fly. That’s where our guest comes in.

Thomas Wedell-Wedellsborg is a consultant who helps companies and managers reframe their problems so they can come up with an effective solution faster. He asks the question “Are You Solving The Right Problems?” in the January-February 2017 issue of Harvard Business Review. Thomas, thank you so much for coming on the HBR IdeaCast .

THOMAS WEDELL-WEDELLSBORG: Thanks for inviting me.

SARAH GREEN CARMICHAEL: So, I thought maybe we could start by talking about the problem of talking about problem reframing. What is that exactly?

THOMAS WEDELL-WEDELLSBORG: Basically, when people face a problem, they tend to jump into solution mode to rapidly, and very often that means that they don’t really understand, necessarily, the problem they’re trying to solve. And so, reframing is really a– at heart, it’s a method that helps you avoid that by taking a second to go in and ask two questions, basically saying, first of all, wait. What is the problem we’re trying to solve? And then crucially asking, is there a different way to think about what the problem actually is?

SARAH GREEN CARMICHAEL: So, I feel like so often when this comes up in meetings, you know, someone says that, and maybe they throw out the Einstein quote about you spend an hour of problem solving, you spend 55 minutes to find the problem. And then everyone else in the room kind of gets irritated. So, maybe just give us an example of maybe how this would work in practice in a way that would not, sort of, set people’s teeth on edge, like oh, here Sarah goes again, reframing the whole problem instead of just solving it.

THOMAS WEDELL-WEDELLSBORG: I mean, you’re bringing up something that’s, I think is crucial, which is to create legitimacy for the method. So, one of the reasons why I put out the article is to give people a tool to say actually, this thing is still important, and we need to do it. But I think the really critical thing in order to make this work in a meeting is actually to learn how to do it fast, because if you have the idea that you need to spend 30 minutes in a meeting delving deeply into the problem, I mean, that’s going to be uphill for most problems. So, the critical thing here is really to try to make it a practice you can implement very, very rapidly.

There’s an example that I would suggest memorizing. This is the example that I use to explain very rapidly what it is. And it’s basically, I call it the slow elevator problem. You imagine that you are the owner of an office building, and that your tenants are complaining that the elevator’s slow.

Now, if you take that problem framing for granted, you’re going to start thinking creatively around how do we make the elevator faster. Do we install a new motor? Do we have to buy a new lift somewhere?

The thing is, though, if you ask people who actually work with facilities management, well, they’re going to have a different solution for you, which is put up a mirror next to the elevator. That’s what happens is, of course, that people go oh, I’m busy. I’m busy. I’m– oh, a mirror. Oh, that’s beautiful.

And then they forget time. What’s interesting about that example is that the idea with a mirror is actually a solution to a different problem than the one you first proposed. And so, the whole idea here is once you get good at using reframing, you can quickly identify other aspects of the problem that might be much better to try to solve than the original one you found. It’s not necessarily that the first one is wrong. It’s just that there might be better problems out there to attack that we can, means we can do things much faster, cheaper, or better.

SARAH GREEN CARMICHAEL: So, in that example, I can understand how A, it’s probably expensive to make the elevator faster, so it’s much cheaper just to put up a mirror. And B, maybe the real problem people are actually feeling, even though they’re not articulating it right, is like, I hate waiting for the elevator. But if you let them sort of fix their hair or check their teeth, they’re suddenly distracted and don’t notice.

But if you have, this is sort of a pedestrian example, but say you have a roommate or a spouse who doesn’t clean up the kitchen. Facing that problem and not having your elegant solution already there to highlight the contrast between the perceived problem and the real problem, how would you take a problem like that and attack it using this method so that you can see what some of the other options might be?

THOMAS WEDELL-WEDELLSBORG: Right. So, I mean, let’s say it’s you who have that problem. I would go in and say, first of all, what would you say the problem is? Like, if you were to describe your view of the problem, what would that be?

SARAH GREEN CARMICHAEL: I hate cleaning the kitchen, and I want someone else to clean it up.

THOMAS WEDELL-WEDELLSBORG: OK. So, my first observation, you know, that somebody else might not necessarily be your spouse. So, already there, there’s an inbuilt assumption in your question around oh, it has to be my husband who does the cleaning. So, it might actually be worth, already there to say, is that really the only problem you have? That you hate cleaning the kitchen, and you want to avoid it? Or might there be something around, as well, getting a better relationship in terms of how you solve problems in general or establishing a better way to handle small problems when dealing with your spouse?

SARAH GREEN CARMICHAEL: Or maybe, now that I’m thinking that, maybe the problem is that you just can’t find the stuff in the kitchen when you need to find it.

THOMAS WEDELL-WEDELLSBORG: Right, and so that’s an example of a reframing, that actually why is it a problem that the kitchen is not clean? Is it only because you hate the act of cleaning, or does it actually mean that it just takes you a lot longer and gets a lot messier to actually use the kitchen, which is a different problem. The way you describe this problem now, is there anything that’s missing from that description?

SARAH GREEN CARMICHAEL: That is a really good question.

THOMAS WEDELL-WEDELLSBORG: Other, basically asking other factors that we are not talking about right now, and I say those because people tend to, when given a problem, they tend to delve deeper into the detail. What often is missing is actually an element outside of the initial description of the problem that might be really relevant to what’s going on. Like, why does the kitchen get messy in the first place? Is it something about the way you use it or your cooking habits? Is it because the neighbor’s kids, kind of, use it all the time?

There might, very often, there might be issues that you’re not really thinking about when you first describe the problem that actually has a big effect on it.

SARAH GREEN CARMICHAEL: I think at this point it would be helpful to maybe get another business example, and I’m wondering if you could tell us the story of the dog adoption problem.

THOMAS WEDELL-WEDELLSBORG: Yeah. This is a big problem in the US. If you work in the shelter industry, basically because dogs are so popular, more than 3 million dogs every year enter a shelter, and currently only about half of those actually find a new home and get adopted. And so, this is a problem that has persisted. It’s been, like, a structural problem for decades in this space. In the last three years, where people found new ways to address it.

So a woman called Lori Weise who runs a rescue organization in South LA, and she actually went in and challenged the very idea of what we were trying to do. She said, no, no. The problem we’re trying to solve is not about how to get more people to adopt dogs. It is about keeping the dogs with their first family so they never enter the shelter system in the first place.

In 2013, she started what’s called a Shelter Intervention Program that basically works like this. If a family comes and wants to hand over their dog, these are called owner surrenders. It’s about 30% of all dogs that come into a shelter. All they would do is go up and ask, if you could, would you like to keep your animal? And if they said yes, they would try to fix whatever helped them fix the problem, but that made them turn over this.

And sometimes that might be that they moved into a new building. The landlord required a deposit, and they simply didn’t have the money to put down a deposit. Or the dog might need a $10 rabies shot, but they didn’t know how to get access to a vet.

And so, by instigating that program, just in the first year, she took her, basically the amount of dollars they spent per animal they helped went from something like $85 down to around $60. Just an immediate impact, and her program now is being rolled out, is being supported by the ASPCA, which is one of the big animal welfare stations, and it’s being rolled out to various other places.

And I think what really struck me with that example was this was not dependent on having the internet. This was not, oh, we needed to have everybody mobile before we could come up with this. This, conceivably, we could have done 20 years ago. Only, it only happened when somebody, like in this case Lori, went in and actually rethought what the problem they were trying to solve was in the first place.

SARAH GREEN CARMICHAEL: So, what I also think is so interesting about that example is that when you talk about it, it doesn’t sound like the kind of thing that would have been thought of through other kinds of problem solving methods. There wasn’t necessarily an After Action Review or a 5 Whys exercise or a Six Sigma type intervention. I don’t want to throw those other methods under the bus, but how can you get such powerful results with such a very simple way of thinking about something?

THOMAS WEDELL-WEDELLSBORG: That was something that struck me as well. This, in a way, reframing and the idea of the problem diagnosis is important is something we’ve known for a long, long time. And we’ve actually have built some tools to help out. If you worked with us professionally, you are familiar with, like, Six Sigma, TRIZ, and so on. You mentioned 5 Whys. A root cause analysis is another one that a lot of people are familiar with.

Those are our good tools, and they’re definitely better than nothing. But what I notice when I work with the companies applying those was those tools tend to make you dig deeper into the first understanding of the problem we have. If it’s the elevator example, people start asking, well, is that the cable strength, or is the capacity of the elevator? That they kind of get caught by the details.

That, in a way, is a bad way to work on problems because it really assumes that there’s like a, you can almost hear it, a root cause. That you have to dig down and find the one true problem, and everything else was just symptoms. That’s a bad way to think about problems because problems tend to be multicausal.

There tend to be lots of causes or levers you can potentially press to address a problem. And if you think there’s only one, if that’s the right problem, that’s actually a dangerous way. And so I think that’s why, that this is a method I’ve worked with over the last five years, trying to basically refine how to make people better at this, and the key tends to be this thing about shifting out and saying, is there a totally different way of thinking about the problem versus getting too caught up in the mechanistic details of what happens.

SARAH GREEN CARMICHAEL: What about experimentation? Because that’s another method that’s become really popular with the rise of Lean Startup and lots of other innovation methodologies. Why wouldn’t it have worked to, say, experiment with many different types of fixing the dog adoption problem, and then just pick the one that works the best?

THOMAS WEDELL-WEDELLSBORG: You could say in the dog space, that’s what’s been going on. I mean, there is, in this industry and a lot of, it’s largely volunteer driven. People have experimented, and they found different ways of trying to cope. And that has definitely made the problem better. So, I wouldn’t say that experimentation is bad, quite the contrary. Rapid prototyping, quickly putting something out into the world and learning from it, that’s a fantastic way to learn more and to move forward.

My point is, though, that I feel we’ve come to rely too much on that. There’s like, if you look at the start up space, the wisdom is now just to put something quickly into the market, and then if it doesn’t work, pivot and just do more stuff. What reframing really is, I think of it as the cognitive counterpoint to prototyping. So, this is really a way of seeing very quickly, like not just working on the solution, but also working on our understanding of the problem and trying to see is there a different way to think about that.

If you only stick with experimentation, again, you tend to sometimes stay too much in the same space trying minute variations of something instead of taking a step back and saying, wait a minute. What is this telling us about what the real issue is?

SARAH GREEN CARMICHAEL: So, to go back to something that we touched on earlier, when we were talking about the completely hypothetical example of a spouse who does not clean the kitchen–

THOMAS WEDELL-WEDELLSBORG: Completely, completely hypothetical.

SARAH GREEN CARMICHAEL: Yes. For the record, my husband is a great kitchen cleaner.

You started asking me some questions that I could see immediately were helping me rethink that problem. Is that kind of the key, just having a checklist of questions to ask yourself? How do you really start to put this into practice?

THOMAS WEDELL-WEDELLSBORG: I think there are two steps in that. The first one is just to make yourself better at the method. Yes, you should kind of work with a checklist. In the article, I kind of outlined seven practices that you can use to do this.

But importantly, I would say you have to consider that as, basically, a set of training wheels. I think there’s a big, big danger in getting caught in a checklist. This is something I work with.

My co-author Paddy Miller, it’s one of his insights. That if you start giving people a checklist for things like this, they start following it. And that’s actually a problem, because what you really want them to do is start challenging their thinking.

So the way to handle this is to get some practice using it. Do use the checklist initially, but then try to step away from it and try to see if you can organically make– it’s almost a habit of mind. When you run into a colleague in the hallway and she has a problem and you have five minutes, like, delving in and just starting asking some of those questions and using your intuition to say, wait, how is she talking about this problem? And is there a question or two I can ask her about the problem that can help her rethink it?

SARAH GREEN CARMICHAEL: Well, that is also just a very different approach, because I think in that situation, most of us can’t go 30 seconds without jumping in and offering solutions.

THOMAS WEDELL-WEDELLSBORG: Very true. The drive toward solutions is very strong. And to be clear, I mean, there’s nothing wrong with that if the solutions work. So, many problems are just solved by oh, you know, oh, here’s the way to do that. Great.

But this is really a powerful method for those problems where either it’s something we’ve been banging our heads against tons of times without making progress, or when you need to come up with a really creative solution. When you’re facing a competitor with a much bigger budget, and you know, if you solve the same problem later, you’re not going to win. So, that basic idea of taking that approach to problems can often help you move forward in a different way than just like, oh, I have a solution.

I would say there’s also, there’s some interesting psychological stuff going on, right? Where you may have tried this, but if somebody tries to serve up a solution to a problem I have, I’m often resistant towards them. Kind if like, no, no, no, no, no, no. That solution is not going to work in my world. Whereas if you get them to discuss and analyze what the problem really is, you might actually dig something up.

Let’s go back to the kitchen example. One powerful question is just to say, what’s your own part in creating this problem? It’s very often, like, people, they describe problems as if it’s something that’s inflicted upon them from the external world, and they are innocent bystanders in that.

SARAH GREEN CARMICHAEL: Right, or crazy customers with unreasonable demands.

THOMAS WEDELL-WEDELLSBORG: Exactly, right. I don’t think I’ve ever met an agency or consultancy that didn’t, like, gossip about their customers. Oh, my god, they’re horrible. That, you know, classic thing, why don’t they want to take more risk? Well, risk is bad.

It’s their business that’s on the line, not the consultancy’s, right? So, absolutely, that’s one of the things when you step into a different mindset and kind of, wait. Oh yeah, maybe I actually am part of creating this problem in a sense, as well. That tends to open some new doors for you to move forward, in a way, with stuff that you may have been struggling with for years.

SARAH GREEN CARMICHAEL: So, we’ve surfaced a couple of questions that are useful. I’m curious to know, what are some of the other questions that you find yourself asking in these situations, given that you have made this sort of mental habit that you do? What are the questions that people seem to find really useful?

THOMAS WEDELL-WEDELLSBORG: One easy one is just to ask if there are any positive exceptions to the problem. So, was there day where your kitchen was actually spotlessly clean? And then asking, what was different about that day? Like, what happened there that didn’t happen the other days? That can very often point people towards a factor that they hadn’t considered previously.

SARAH GREEN CARMICHAEL: We got take-out.

THOMAS WEDELL-WEDELLSBORG: S,o that is your solution. Take-out from [INAUDIBLE]. That might have other problems.

Another good question, and this is a little bit more high level. It’s actually more making an observation about labeling how that person thinks about the problem. And what I mean with that is, we have problem categories in our head. So, if I say, let’s say that you describe a problem to me and say, well, we have a really great product and are, it’s much better than our previous product, but people aren’t buying it. I think we need to put more marketing dollars into this.

Now you can go in and say, that’s interesting. This sounds like you’re thinking of this as a communications problem. Is there a different way of thinking about that? Because you can almost tell how, when the second you say communications, there are some ideas about how do you solve a communications problem. Typically with more communication.

And what you might do is go in and suggest, well, have you considered that it might be, say, an incentive problem? Are there incentives on behalf of the purchasing manager at your clients that are obstructing you? Might there be incentive issues with your own sales force that makes them want to sell the old product instead of the new one?

So literally, just identifying what type of problem does this person think about, and is there different potential way of thinking about it? Might it be an emotional problem, a timing problem, an expectations management problem? Thinking about what label of what type of problem that person is kind of thinking as it of.

SARAH GREEN CARMICHAEL: That’s really interesting, too, because I think so many of us get requests for advice that we’re really not qualified to give. So, maybe the next time that happens, instead of muddying my way through, I will just ask some of those questions that we talked about instead.

THOMAS WEDELL-WEDELLSBORG: That sounds like a good idea.

SARAH GREEN CARMICHAEL: So, Thomas, this has really helped me reframe the way I think about a couple of problems in my own life, and I’m just wondering. I know you do this professionally, but is there a problem in your life that thinking this way has helped you solve?

THOMAS WEDELL-WEDELLSBORG: I’ve, of course, I’ve been swallowing my own medicine on this, too, and I think I have, well, maybe two different examples, and in one case somebody else did the reframing for me. But in one case, when I was younger, I often kind of struggled a little bit. I mean, this is my teenage years, kind of hanging out with my parents. I thought they were pretty annoying people. That’s not really fair, because they’re quite wonderful, but that’s what life is when you’re a teenager.

And one of the things that struck me, suddenly, and this was kind of the positive exception was, there was actually an evening where we really had a good time, and there wasn’t a conflict. And the core thing was, I wasn’t just seeing them in their old house where I grew up. It was, actually, we were at a restaurant. And it suddenly struck me that so much of the sometimes, kind of, a little bit, you love them but they’re annoying kind of dynamic, is tied to the place, is tied to the setting you are in.

And of course, if– you know, I live abroad now, if I visit my parents and I stay in my old bedroom, you know, my mother comes in and wants to wake me up in the morning. Stuff like that, right? And it just struck me so, so clearly that it’s– when I change this setting, if I go out and have dinner with them at a different place, that the dynamic, just that dynamic disappears.

SARAH GREEN CARMICHAEL: Well, Thomas, this has been really, really helpful. Thank you for talking with me today.

THOMAS WEDELL-WEDELLSBORG: Thank you, Sarah.  

HANNAH BATES: That was Thomas Wedell-Wedellsborg in conversation with Sarah Green Carmichael on the HBR IdeaCast. He’s an expert in problem solving and innovation, and he’s the author of the book, What’s Your Problem?: To Solve Your Toughest Problems, Change the Problems You Solve .

We’ll be back next Wednesday with another hand-picked conversation about leadership from the Harvard Business Review. If you found this episode helpful, share it with your friends and colleagues, and follow our show on Apple Podcasts, Spotify, or wherever you get your podcasts. While you’re there, be sure to leave us a review.

We’re a production of Harvard Business Review. If you want more podcasts, articles, case studies, books, and videos like this, find it all at HBR dot org.

This episode was produced by Anne Saini, and me, Hannah Bates. Ian Fox is our editor. Music by Coma Media. Special thanks to Maureen Hoch, Adi Ignatius, Karen Player, Ramsey Khabbaz, Nicole Smith, Anne Bartholomew, and you – our listener.

See you next week.

• Subscribe On:

Latest in this series

This article is about leadership.

• Decision making and problem solving
• Power and influence
• Business management

Partner Center