No project has perfect data

For the jaded and and road-weary consultants, this will sound like an understatement. In fact, it’s usually like an Easter egg hunt where the team has a good idea where the data eggs might be, but can’t be 100% sure until they really start looking for it. Looking in multiple IT systems, hard-drives, filing cabinets, and dashboards. 

Most clients have trouble with their data

While there is a lot of talk of BIG DATA and the revolution it will have in predictive analytics, in reality, many organizations have trouble patching together their SMALL DATA. I would guess that only 1/3 of companies can can easily satisfy the consulting team’s data request.

Oddly, lack of data = consulting project

For a bit of circular logic. . . it is often because the data is difficult to find, that the client has not really solved the problem yet. As Seth Godin remarked in a blog post about “perfect problems”:

The only problems you have left are the perfect ones. The imperfect ones, the ones with a clearly evident solution, well, if they were important, you’ve solved them already

So, consultants should be weirdly thankful. Client’s data mess = work, projects, billing and money.

Collecting data can be painfully slow

Even when the location of the data is clear, it’s fairly common to spend several days hunting down the right people to get the data. Consultants often go to the client site just to request (read: pester, nag) the client to give them the data. Clients could save 5-10% of fees if they would just get the data to the consultants quicker.

Good data is hard to find

In my experience, the larger, the more geographically dispersed, and the older the company = the messier the data. Using the analogy of data flow like plumbing . . . the larger and older the house, the more it leaks.  For those new to consulting, get ready to start digging for the data. Just some of the crazy examples from my past:

• Several late nights alone typing shipping data from paper invoices into an excel spreadsheet
• Consolidated data from 60+ separate emails (daily reports) into one excel

Ask any analyst. They will have their own hazing story of collecting data in some crazy way. As long as companies don’t make the effort to do this, they will continue to pay $$$$ / hour for this mundane task to get done. 

There are so many reasons that clients might (oh trust me, they definitely do) have data problems:

#1-3. IT needs to update and standardize

Too often IT only makes tactical repairs and spends their energy and budget just playing catch up. Too often, clients customize their enterprise resource planning (e.g., SAP, Oracle) to match their process, instead of listening to the systems integrators and stick with best practices. “Oh, we like to do it our way” is usually code words for messy data down the road.

#4. Sometimes legacy = bad 

We are all creatures of habit, so are organizations. I cannot tell you how often I hear the phrase, “Oh that’s just the way we do it.” or it’s corollary, “we tried that before, but it failed.” Those are all signs of trouble and poor planning. There is always someone – let’s call them Timmy – who knows how things really work. It’s a kind of resigned organizational stasis, exhaustion, which frankly signals a failing bureaucracy. 

Talk about disaster planning: What happens if Timmy leaves?

#5-7. Time to clean the data

The customer master (where you list all the key information of your clients) needs to be clean because it is used for billing, accounting, and other customer-relationship activities (e.g., sales calls, marketing direct mail). Too often these are a bit of a mess. For example, there will be 4 ways to spell Wal-mart, Wallmart, Walmart, Wall Mart.  With junky data, it is hard to analyze anything.

From my experience:

• Collecting data = 1/3 of the time
• Cleaning/validating data = 1/3 of the time
• Doing the actual value-added analysis (the real work) = 1/3 of the time 

#8-9. Figure out the roles / responsibilities

Who’s job is it anyways?  If it is everyone’s job, then in effect, it is no one’s job.  It’s not a good sign when veteran office workers who are uncomfortable using basic excel commands like sort and pivot. While it seems basic, sometimes real-time analysis is just not valued enough to put into job descriptions and performance reviews.  Inevitably, it is the manager’s fault. 

Insights matter, not data 

“Data” is a bit of a misnomer because frankly data by itself is kinda useless. It’s like having a big bag of flour, water, yeast, sugar, salt, butter, and egg in random portions, in random containers. Good luck making bread.

All data is not the same, of course. In my mind, there is a progression / hierarchy of the value of information which looks something like this. It starts as noise, gets organized into data. That data turns into information as it is structured, cleaned, rearranged, and sorted so it makes some sense. Analysis takes shape as information is pivoted, correlated, appended, hypothesis-tested. Insights are the gems and diamonds. They are rare, valuable, and often very polished. Only analysis and insights should be presented to the client.

TIP: Know what you are looking for

Too often, management consultants (myself included) are unclear on what we are trying to prove, so we drop an large, brutish, and onerous data request that ineloquently says “we’ll take everything; give us whatever you got.” 

Instead, a professional has some hypotheses of what the answer could be. . . then working from right to left on the diagram above, reverse-engineers the analysis, information, and ultimately, the data she needs.

TIP: Be creative 

Sometimes, consultants have to uncover, create, cleanse, triangulate or even create data to answer key questions. Creativity is needed here. It is also a great way to “wow” the client. Some of consulting tools used to find new data include: surveys, interviews, focus groups, workshops, financial comparisons, observations, estimates, simulations, business models, benchmarks, maturity models, and others.

What data war stories do you have?  Please share. . . 

Related posts:

• Data analysis in 20 minutes
• 21 Data visualization tips
• Links