How do I stop over-analyzing and actually make a decision?
Ever had that feeling of being trapped in some kind of time loop when analyzing data? You start out with the best of intentions, everything is going fine, until you realize that you have been doing this analysis for about 3 days and you have around 50 slides in PowerPoint and 5 Excel files with 30 sheets.
It’s Data Analysis Paralysis: the inability to move past unnecessary information while performing a data analysis. It’s actually a very frequent issue, with most first-time managers dealing with this. While it might be frequent, it’s not really good: for the one making the analysis, for the teams that have to sit through 50 slides of data, and for the customers or stakeholders that requested it.
If you are also a victim of this bad habit, let us clear the air and help you understand what this is, why it is happening, and how to deal with it and avoid it. In this article we will talk about:
- Getting Stuck in Spreadsheets
- Before you start: EDA – Exploratory Data Analysis
- DIG: Description, Introspection and Goal Setting
- The Hard STOP
Getting Stuck in Spreadsheets
To better understand the concept of data paralysis, we need to look at the first thing people usually do when the “symptoms” start showing: getting stuck in multiple spreadsheets. Let’s assume you just got the job and you are a bit more comfortable with data analysis (and if you are not, you can check out these articles for a quick overview on data for the non-analytical person: I’m Not Good with Data: The Non-Analytical Lead’s Guide and Data Types: A Beginner’s Guide). You get your stuff in order, grab your coffee, and start digging into data. You feel empowered and strong, like nothing in the world can stop you now: you’re doing research, looking up terms, and checking formulas… everything you can think of to make this analysis the best one you’ve made. After some time, you realize: it’s been 20 hours, and you have 3 Excel files and between them maybe 50 worksheets.
This is not good!
For people just starting out with this kind of task, it’s very common to just get sucked into the frenzy and start doing a lot of things that don’t really make any sense at some point. In part, this is because you feel the need to dig deeper and deeper into insights, and maybe find something outstanding that would really make people go, “WOW, what a valuable point.” On the other side, it’s a behavior most often associated with people who don’t really know how to tackle the subject, and because of this, they just keep going.
Here’s your most valuable insight, from me to you: if you arrive at this point, just close all those Excel files and take at least 2 hours away. Nothing good is going to come out of it. It’s too much data, too many potentially unnecessary leads, and nothing of actual value (and even if there was something of value, it was so far back that you no longer remember what that was).
I work with data almost daily and I teach others how to do it as well. Trust me when I say I know exactly what is going through your mind right now. And it’s not about how good you are with data in general; it’s about not knowing what your direction is or what you want to achieve. To get you on the right track, let’s review my personal checklist for data analysis, something I use every time to avoid data paralysis:
- Step 0: What is the ask? This is something people, ironically, don’t pay enough attention to; they just start pounding on those numbers hoping to get some ideas. Before you even begin any type of analysis, one thing (and one thing ONLY) must be 100% clear: what is it that you need to check.
- Step 1: Understand your data. Once a goal is clear, people just start making pivots and tables, but very few take a step back and say, “Hey, wait a minute, do I know what this data is about?” The idea is to have a complete understanding of said data, meaning: what every column is about, what each number says, where they got the data, how it is connected to other things, etc.
- Step 2: Always start with the general and then move to the specific. I find this very helpful because it helps me avoid the mistake of going directly to very specific, very niche situations. You want to have a good overview first on what is going on, and then move to specifics.
- Step 3: Create your own framework. Every person is different, and the way we look at and understand data is, of course, different. Don’t assume every person in this world will analyze data in the exact same way. We have different styles because we relate to data differently: some will need a more personal approach to understanding it, some are more “cut and slice,” some prefer to go for very specific analytics, etc. This is why you need to develop your own style, something you are OK with and that works for you.
Before you start: EDA – Exploratory Data Analysis
Credits here to Mr. John Tukey for the outstanding work he did way back in 1962 when he first introduced the concept as a philosophy (The Future of Data Analysis), and then expanded it in his 1977 book Exploratory Data Analysis. The basic concept of this is simple: you “interview” the data to make sure you understand its shape, check for any flaws in it, uncover initial patterns, and test assumptions before performing any complex analytics on it. There are 2 very important reasons we do this:
- Garbage In – Garbage Out: Check for duplicate values, missing fields, and corrupted entries that will ultimately skew the data.
- Wasted Time: What’s even the point of spending 20 hours on a complex analysis if, by the end of it, you realize that you have a lot of missing values and because of this nothing makes any sense?
To achieve EDA, you will need to cover 4 stages:
- High-Level Orientation: Checking row counts, column names, data types, and missing values.
- Summary Statistics: Mean, median, max, and standard deviation to understand the central tendency and the spread of your numbers.
- Check for Anomalies and Outliers: Look for extreme values that will impact your averages.
- Pattern and Correlation: Using very simple visual tools (histograms, scatter plots, box plots), check how different variables interact with each other.
Remember this: EDA is a time-limited activity – you don’t need to spend too much time on it. Once you check the data and make sure it’s clean, you are good to go, so you need to STOP the EDA and move to the next step.
DIG: Description, Introspection and Goal Setting
With a clean dataset, you are now able to do more solid work, and a good way to go about it without getting sucked into paralysis is DIG. This is a structured data analysis model designed to bridge the gap between raw data and actual business decisions. This will help you avoid the “spreadsheet madness” and also give you a nice way of going about data analysis. Let’s look at each step and see what they are all about.
Description is all about the foundation. At this stage, you establish the facts without jumping to conclusions. The core of it is to summarize your raw numbers into clear and objective statements about past or current performance.
The core question to ask during description is, “What is the reality right now?” Your role here is to aggregate the metrics, check the trends, calculate the averages and percentages, and list all the key performance outputs.
Example: Average ticket resolution time increased from 24.56 minutes per ticket to 35.6 minutes per ticket in the last quarter.
DO NOT: Speculate or create a narrative here. Just stick to the facts and don’t add anything that is not based purely on objective evidence. When you start creating a narrative, you are corrupting the analysis and going back to that “spreadsheet madness.”
Introspection acts like your diagnostic stage, where you start digging into the operational mechanics behind the numbers. It’s similar to your basic Root Cause Analysis, in which you look at different factors instead of accepting the surface-level issue: processes, workload, handoffs, external factors, etc.
The core question we ask here is, “What factors or operational root causes drove these specific results?” Here, we apply different RCA tools to uncover what is behind the number (5 Whys, Fishbone, process mapping, etc.) and we examine the context, not just the isolated outputs. If you need a bit of help with process mapping, you can start out with the easiest of them, the SIPOC.
Example: Average ticket resolution time increased because a software update created 30% more complex escalation tickets, while onboarding 3 new team members slowed down the first-line triage.
DO NOT: Stop at superficial correlations or blame individual people for the root causes. You need to make sure that whatever issue you think is actually causing the problem is the correct and real one.
Goal Setting is all about what you need to achieve. It’s essential to have a clear goal; otherwise, you are just creating more spreadsheets that have no use. The task you have here is to make sure your analysis will have actual operational impact, clear recommendations, or help with specific decisions that must be made. During goal setting, you convert your findings into measurable targets and specific, targeted interventions. As a rule of thumb, don’t go crazy with this; people can’t handle too many struggles, so pick the ones that have the highest impact and present those.
The core question here would be, “What specific action, target, or decision will actually help solve what we just learned?” In a nutshell, this is where we establish clear timelines and targets, transform the analysis into proper KPIs that people can track and measure (your success rate, if you will), assign operational owners, etc.
Example: Update first-tier documentation by Friday (to be done by the process SME), perform a workshop re-check on the SOP with the Trainer/TL for the new people, and target reducing the handling time to below 25 minutes by the end of the next 30 days.
DO NOT: Create vague goals (e.g., “improve customer satisfaction”), have recommendations without timelines or clear owners, or have no plan whatsoever. If there’s anything stakeholders hate more than anything, it is an incomplete analysis, one that does not have any clear action points at the end.
The Hard STOP
And finally, we have the hard stop. This is all about you and what you personally must do each time you stray from the path: stop. If you feel things are not going in the right direction, stop, assess, and go back. It’s a bit harder to do than people think, but it’s a necessary step. Paralysis appears very often, and even for more experienced data analysts, it can be a challenge. However, the trick to it is to stick to your plan, follow the framework, and always go back to your scope.
Here are a few things I do to avoid paralysis and force myself to execute a hard stop:
- Always look at my goal: If I do this deep dive, will it get me closer to my goal? If I can’t reach it in 10-20 minutes, then I just stop. There’s no point in it; I’m just wasting time.
- Create a “plan” of what I am going to analyze: Before I begin, I keep a notepad or Excel table with all the checks I want to make. I do this to cross-check and always go back when I feel I’m no longer sticking to the plan.
- I use the 10-second rule: If people can’t understand in 10 seconds what this is about, there’s no point in doing it. With smaller attention spans and too much visual noise going on around us, it’s no surprise that people are less likely to understand what you are trying to say. If they catch the main idea in 10 seconds or less, you are good to go. If you need to explain for 4 minutes what the issue is… then it’s not worth it.
While data paralysis is not something you can just get rid of overnight, we have plenty of tools and tricks to help us mitigate its impact. With enough practice and patience, you will reach a point where your expertise will significantly increase and data will no longer be such a complicated topic.
Until the next one, stay healthy, happy, and safe!