When is a problem not a problem?

One of your colleagues comes to you with a problem. There is a field in one of the marts that is not being populated. The data is being collected in your production database, it is just not being passed into your mart.

An analysis of your mart produces interesting results. The field has never had the data added. This is where we get into the world of the defect vs feature.

I have heard the term 'feature' politely used for something that ought to work, but was either not tested, overlooked at testing or the problem was discovered so late that a decision was made to implement without it. There is significantly less desire within your IS department to correct a 'feature' than a true deviation from agreed process ('if-it-ain't-broke-don't-fix-it'). The longer the time span between implementation and discovery of the feature, the more reluctant your colleagues will become to fix it.

It is very easy to become cynical when faced by vigorous challenges from your technical areas around remediation of features. You may have tried to fix the problem before, but with the advent of new disciplines like data quality, there is much more support available for getting these problems remediated.

So the answer is, a problem is always a problem - no matter how orwellian the description is. Some may choose to reframe the issue, but the truth is that it never worked properly. The 'feature' is still a problem. Screw your courage to the sticking post. Assert yourself. Engage your data quality section. Get it fixed.

4 fundamental questions for business intelligence

When setting up a business intelligence project, most people tend to start with systems, people and processes. It is very common to consider the 'hows' before the 'whys'. Taking a step back and asking some more fundamental questions may be more beneficial.

Cost

Any piece of information can be captured. Is it financially feasible to do so? Compare the costs to the anticipated benefits of acquiring the information. What people generally miss from a good cost benefit analysis is the cost of acquiring the money. This will depend on how your organisation is funded. If you have shareholders, the cost of acquiring money (dividend payments) is probably far more expensive than loaning from a bank (interest rates). But when you factor in financial acquisition costs, your benefit analysis may be far less than you thought. As new technologies become established, their cost comes down. Could you wait until this happens or are you losing a potential opportunity to gain an advantage over competitors?

Ethics

Is it ethically appropriate to collect the information? Do your customers or colleagues know you are capturing this information? Do they have a right to know? If they found out, what would their reaction be? What are the rules about this information? What are the regulatory obligations and constraints? 

Ownership

Who will own this new data? Ownership and accountability should be ascertained right at project inception. I guarantee that as soon as something has been built without an accountable owner agreed beforehand, your colleagues will head for the hills. If they can get something built without being held accountable, then they will. Without an accountable business owner, it becomes so much harder to get cooperation from the rest of the organisation when the data requires remediation. 

Access

This is like accountability, only the other way round. A new report or data set becomes available and EVERYONE wants access. Do they really need it? Is the information politically sensitive? Could the performance of other colleagues be derived from this information? What could be the repercussions of general circulation? How valuable would the insight be to an external company? How vulnerable is this data to theft?

Consider carefully before you start. There are inherent risks - both financial and human - to collecting new data that need to be carefully considered.

Security considerations for data quality tooling

One of the more unexpected places you will experience resistance to implementing a data quality toolset could be from your own internal IT department. If they take security and system performance seriously, they will want to know all about your new software and how it interacts with all the databases and marts.

All data quality tooling uses ODBC or JDBC connections to access your enterprise data sources. Each different connection will require a userID and password. If you are planning to implement a desktop only solution, prepare for a long fight. Desktop versions of data quality systems rely upon your PC having these connections set up and the passwords embedded. This could cause problems if other people gain access to your desktop. Some will be able to use your connections with other programs - like MS access to query your data and save it anywhere. Even if the O/JDBC connections are not accessible outside the data quality suite of programs, the embedded ETL capabilities of the data quality software may pose a security risk that may prove a step too far for some system administrators. 

The solution is to implement a server version of your data quality software. This involves installing your software on a central server. The ODBC/JDBC connections are similarly centralised. Users then have a desktop program that interfaces with the server, and cannot access it until a password is keyed in. This is far more secure, but will effectively treble the set-up costs, especially if you build a failover solution. A failover option may be mandatory if you work in a highly governed business like pharma or banking.

In this information age, the value of data has never been higher. With all current crime trends pointing to internal colleagues stealing data and selling on to other companies as being the biggest data security threat, it is important that the capability of data quality tooling is not perceived as too great a risk. All of these risks can be successfully mitigated with the correct infrastructure implemented and governance controls around appropriate access to data. 

Trust, a bridge to better quality

Remember my post on organisational entropy? One of the key secrets to lowering the entropy with everyone in your organisation is the building of trust. You can tell when trust is lost with any individual, because every interaction becomes almost impossible. Working in the quality and governance space is a highly sensitive dynamic, for who will trust you with their business problems if they believe you are not trustworthy?

You can take the role of a critical friend; you can become a master of body language and unspoken communication; you can even learn NLP. But these are all superfluous without the core principles to building trust. Here is my take on building trust:

Deliver to your promise

Say what you are going to do, when you will do it, and deliver. When you can consistently do this, you are well on the way to building trust. Broken promises damage trust. If you find that you cannot deliver on something, go immediately to your customer, apologise and let them know what you can do. The earlier you can tell your customer about a potential problem, the better they manage expectations elsewhere.

Keep things private and personal, appropriately

Nothing destroys trust more than if you blurt out gossip that is told to you in confidence. This is particularly damaging if you talk about one department's troubles to another one. Word gets around that you can't be discreet, and that can cause problems. However, this is not the same as keeping secrets from colleagues who should be informed. If someone tells you of criminal activities, report them to the appropriate officials, and not to your friends on coffee break.

Delegate to educate

When you delegate something, you are also telling your colleague that you trust them. But don't just drop it and run. Use it as an opportunity to coach them in your area of expertise. Share the decision making that you would make if you were doing it yourself. Build that rapport.

Deliver together

If you have worked on something with someone, present the results together. Make sure they know they are being recognised as a key contributor. 

Take responsibility appropriately 

Don't blame others when things go wrong. At the same time, don't accept blame when it is not your fault. 

When dealing with data quality or governance issues, you are in a position of trust. Therefore, be worthy of that trust. It is one of the most important things you can build with your colleagues. Be consistent, be effective, be reliable and fair. The rewards are great, as Emerson said...... 

"Trust men and they will be true to you; treat them greatly and they will show themselves great."

Is your SCV really a SCV?

My business is full of jargon. You can't avoid it. And along with the jargon comes a whole rack of TLA's.  In this case, TLA stands for "Three Letter Acronym". Things like Treating Customers Fairly (TCF) and Extract Transformation and Load (ETL) to name a couple.

But the one TLA that causes more furrowed brows than anything I have seen is SCV.... the Single Customer View.

As businesses grow, they acquire other companies. They build separate databases to service different customers. As time goes by, you can end up with one customer who can have their personal details on a bewildering array of platforms and marts. There is significant risk of your marketing data and your customer insight being fragmented. 

The best solution is to build a separate data mart with all of your customer insight data merged. There are a number of master data management tools that can help you do this. They are purpose-built for the job, and can cut down a lot of the legwork. With the proliferation of these solutions being increasingly widespread, consider that you may not be the only one in your organisation who is building a single customer view.

This brings me to the crux of my point. A single customer view should be just that - single. Having 5 or 6 different single customer views throughout your organisation is bound to create reconciliation problems further downstream.

So when someone asks you to build a SCV, your first priority is to make sure that it hasn't been done somewhere else before you start. And the second priority is that it should be the last single customer view you create.

Privacy - sharing the responsibility

Business intelligence is right at the cutting edge of the all our concerns about privacy. Facebook and google are prime examples of organisations that collect an enormous amount of personal information. One of the main challenges to business intelligence is the ethics of collecting information, and where it crosses into intrusion and espionage.

We now know that not only did Google unethically collect this information, but they did not delete the information correctly. It makes you wonder what information is being collected that is not currently being used for anything. This goes even further when you look at the amount of personal information collected by mobile devices like smartphones. Steve Jobs had an interesting take on the problem.

I would take this a step further and say that sharing of information and total transparency is the key. There are significant advantages to be gained to giving customers control of the information that you hold about them. Giving them control of their information makes them accountable for the quality of it. Wouldn't it be great if every 6 months we sent our customers an email for them to check their contact data and revise their privacy agreement and preferred contact methods? Carrying out the process could entitle them to discounts or extra services.

More importantly, when everyone works together, trust is built. Customers get back the control and businesses get a feedback loop to keep details updated and quality high.

What gets fixed first?

Dogs played a special role in ancient Greece. They had guard dogs, hunting dogs and dogs for herding. They even put a mythical guard dog at the gates of the dead - cerberus. Choosing the right puppy from a litter was an important skill. Making sure you had the very best of the litter could mean the difference between life and death.

One hunting dog expert advocated a special test to find out which puppy was the best. He decided to let the mother choose by taking the pups away from their mother, then placing taking an oil-soaked string in a circle around them and setting it on fire. The mother would jump over the ring and rescue each pup... one by one... in order of their merit. The first pup saved by the mother was obviously the best one.

Many people build data quality reporting tools with some kind of severity measure to ascertain priority. However, you will always get colleagues who will inflate the severity just to get their issue fixed first. There are others who will understate the risk to avoid repercussions. Naturally, some errors just stand out as being the most important. But when you have a massive list of data quality errors, go back to your consumers and ask them.... Which puppy gets saved first?