The patient is going to receive worse customer service and more importantly worse care if the hospital and clinic don’t share data-when, where, and why was he admitted, what issues did he present, what treatment did he receive, etc. Imagine a patient admitted to a hospital ER, receiving treatment and then being released with a requirement to attend an outpatient clinic for additional treatment and checkups. There must be a data-sharing culture within the organization so that data can be joined, such as combining customers’ clickstream with their transactional history. The mechanics of getting those numbers became a huge time-sink for the analysts and a source of stress as to whether they would get their numbers or have to wait another 10 hours to rerun those VLOOKUPS. It turned them from analysts into Microsoft data engineers. My team helped to bring that whole dataset into a MySQL relational database. We wrote queries to crunch the numbers for them, allowing them to focus on analysis, trends, and presentation of that data-a far better use of their time. Now that they have better tools and more time, they are producing deeper, richer analyses. Excel had been an appropriate tool, but company hypergrowth changed that. They had stretched the tool and approach as far as they could go. This worked well initially, but as the company’s sales and customer base were scaling rapidly, the data got larger and larger, the Excel file approached 300 MB, their computers maxed out their RAM, and the VLOOKUPS would take 10 hours or more, frequently crash, and had to be restarted.
They sucked down huge amounts of raw data from different sources and ran VLOOKUPS (an Excel function to find cross-references in the data) to join all that data to get a top-level look at the numbers. For instance, for a long while, the financial analysts at Warby Parker were using Excel to compute the key metrics reported to senior management. There are many options, such as relational databases, NoSQL stores, or Hadoop. The data must be in a form that can be joined to other enterprise data when necessary. Having accurate, timely, and relevant data, though, is not sufficient to count as data-driven. Prerequisite #2: Data must be accessible and queryable. A small amount of clean, trustworthy data can be far more valuable than petabytes of junk. The hard truth is that data alone is not enough.
Some people, especially certain big data vendors and service providers, pimp big data as a panacea: if you collect everything, somewhere in there are diamonds (or golden nuggets or needles or one of many other metaphors) that will make any company successful. In the next chapter, I’ll cover aspects of data quality in much more detail.Įven if you do have quality data, and even if you have a lot of quality data, you will only get so far and despite the hype that you might hear, it does not make you data-driven. In my experience, this is entirely plausible.
I often hear that data scientists spend 80% of their time obtaining, cleaning, and preparing data, and only 20% of their time building models, analyzing, visualizing, and drawing conclusions from that data (for example, and ). There can be subtle hidden biases that can sway your conclusions, and cleaning and massaging data can be a tough, time-consuming, and expensive operation. It also has to be timely, accurate, clean, unbiased and perhaps most importantly, it has to be trustworthy. The dataset has to be relevant to the question at hand. Of course, it can’t just be any data it has to be the right data. Prerequisite #1: An organization must be collecting data.ĭata undoubtedly is a key ingredient. Let’s get a couple of obvious prerequisites out of the way.
0 kommentar(er)
