From question to solution: Part 1. The scientific method August 2, 2021
By Stijn Huygh
What do you do when a client comes to you with a specific question? You try to come up with a solution and propose it to the client. This is of course the simple answer, which ignores the complete process of problem-solving. For simple problems, one can get away with a non-systematic approach and tap from the same barrel of known truths and assumed knowledge. This can happen because the path from question to the solution is one straight line without many hurdles to take. However, when the problem becomes highly complex, and there aren’t many known truths, the simplest way to a solution is always a systematic approach. This is where the scientific method of problem investigation comes into play.
‘Science is more than a body of knowledge. It’s a way of thinking. A way of sceptically interrogating the universe with a fine understanding of human fallibility
– Carl Sagan –
The scientific method is applicable in much more situations than just scientific research. It forms the basis of a systematic and logical approach to tackle problems, be it scientific or business problems, to interrogate your own solutions, and to pinpoint the fallacies in your assumed knowledge. The scientific method exists of a few simple steps:
- Identify and analyse the problem
You have an observation, a question from your client, or a problem you came across at any point. This often is not the true problem and needs to be translated to a kind of research question.
What do we need to solve? Within which framework needs this to be solved? What are the business and technological restrictions? What measurable or quantifiable properties do/already exist to identify if the problem is solved? What data do we require, and can we collect on our problem? Asking these questions about your problem, or even just what, who, where, when, and how, can help us to translate your problem into a researchable problem.
- Form a hypothesis
Now that a researchable problem has been identified and you posed the research question, the next step is to form a hypothesis. To be able to tackle your research question, a statement is made that forms an educated proposition or solution to your problem.
“When we do A, then B will happen.”, “If we obtain C, we can calculate D”, …
Just as before, this hypothesis should have a measurable or quantifiable outcome. This allows to prove or disprove the hypothesis and work towards a solution.
- Conduct experiments to test the hypothesis
The time has come to define an experiment that will be used to prove or disprove the hypothesis. One of the biggest risks in setting up an experiment, is known as confirmation bias. One is more likely to have a one-sided approach to testing the current hypothesis. The search for evidence is often directed to evidence consistent with the expected result, and not to find possible fallacies in your research.
Other risks that arise are sampling errors, such as survivorship and sampling bias. The selected subjects or properties are not representative of the true problem or population. It is easy to eliminate certain subjects as they did not pass a certain hurdle beforehand, or only sample properties that have a greater tendency to prove the hypothesis.
The experiment should be objective, and just as likely to prove as to disprove your statement to truly work towards a solution.
- Analyse Data
Upon completing the experiment, the results should be analysed thoroughly. Does the data confirm the hypothesis or is it rejected? Are there any remaining issues? Even when the proposed solution has been disproven by the experiment, it can result in additional insights to form a new hypothesis.
Try to detect what problems did get solved and if any new problems were identified?
- Conclude on the results
No matter the outcome of the experiment, the results should be well documented for later reference and should be communicated within your team to learn and grow. One should assess if further research is likely to deliver the expected solution and then one can propose a new hypothesis with the earlier conclusions. Never let a failed attempt deter you from continuing. The thing I have learned is that a challenge is not an obstacle, but rather the opportunity to prove your perseverance and determination.
Now the steps of the scientific method have been laid out it would be interesting to have a small example that applies this approach to a fictional problem.
Imagine a navigation application based on GPS. You get feedback from your users, that the arrow indicator on the map drifts and jumps away from the route. You identify this as an issue, as it significantly reduces the user experience and can lead to confusing and dangerous situations.
Step 1a: Identify and analyse the problem
There is an issue in the location detection, which causes fluctuations in the UI of the application.
“Is it possible to address the location detection issues to improve the user experience of our users?”
Step 2a: Form a hypothesis
You find that the GPS data received in the application often has bad accuracy in an urban environment. You pose that:
“If we filter out GPS data with bad accuracy, the fluctuations of the arrow indicator will no longer be there.”
Step 3a: Conduct experiments to test the hypothesis
As an experiment, you filter out the GPS data with bad accuracy and request the users that have reported the problem to join a beta group. This beta-group will evaluate if the problem has become better, or worse. You receive a lot of feedback from your beta testers.
Step 4a: Analyse Data
The result from your survey comes back as the following:
“The arrow indicator is more stable and no longer jumps over the screen, but now it often just stands still while I’m driving.”
Step 5a: Conclude the results
The results of the test are clear, the problem itself has been fixed. However, there is an unwanted side effect, the location becomes stale. The GPS locations with bad accuracy are indeed the problem but filtering them out is not the solution. It is time to go back to the drawing board and reiterate through the method.
Step 1b: Identify and analyse the problem
As the conclusions from the originally proposed solution identified a new problem, you will need to reiterate the steps to find a proper solution. The bad location updates were indeed the root cause of the problem, but filtering them out results in the risk of the location becoming stale.
The newly identified problem is:
“Can we improve the bad location updates, without filtering them out?”
Step 2b: Form a hypothesis
The application has a predicted route, of which there is a significant chance that the car of the user will be.
“If we project the location on the predicted route, the fluctuations of the arrow indicator will no longer be there, and the location will not become stale”
Step 3b: Conduct experiments to test the hypothesis
As before, you implement the solution and ask the beta group to give their feedback. You again get a lot of feedback which you can process in the next step.
Step 4b: Analyse Data
The result of the survey came back positive.
“The arrow indicator is slightly less stable than with the previous solution, but no longer stands still when I’m driving”
Step 5b: Conclude the results
The new results indicate that the proposed solution in step 2b is valid, and you can continue with that.
The scientific method helps form a framework in which you can work. The process itself has the inherent tendency to bring you closer to the solution. New information and ideas are gathered along the way, the proposed solutions are interrogated to pinpoint pain points and the true problems tend to surface during investigation.
In this first part of the blog post we discussed how the scientific method works, what the points are for which you need to pay attention, and we have applied this method to a fictional problem. In part 2 we will discuss the solution to a real-world problem and how we applied this method to come to a working solution. The question we solved was “Is it possible to detect arrival and departure of a train in a station with a time resolution of a couple of seconds?”. This does seem to be an easy one but looks are deceiving…