Brave new world... are we there yet?

Following recent high-profile terrorist activities, one of my friends asked me whether it was possible to leverage the latest IT solutions to monitor every email, chat room and the internet message for terrorist-related activity. 

Setting the legal, privacy and personal freedom issues aside, let's take a pragmatic approach. It is easy to look at the new big data solutions and the amazing hardware that is available and say "in theory, it's possible to be done".  

In theory......

Firstly, have a good honest look at the data on your systems. How good is it? Perhaps 80-90% correct?

Let's assume that there are about 1,000 terrorists in the UK. We have approximately 64 million inhabitants. If we have access to every piece of information that is exchanged, even if the data is 99.9% accurate, we will probably need to arrest and question 64,000 people, just to capture 1,000 terrorists. 

That would annoy a lot of people!

Now given that your computer has perhaps 90% correct data and that others are far worse than you... How much data do you think is actually correct on the internet? How much information that people exchange through the internet is truth? It's a darn sight less than 90%.

Then you have people using code words, different languages and 16 bit encryption.

We can build the most powerful computers in the world, and the most sophisticated solutions, but unless the data is correct, we don't stand a cat in hell's chance of getting anything right. A system is only as good as the data that it holds.

So don't expect minority-report analysis of our data with predictions of where bad things will happen next any time soon. For if the truth is known, these amazing new systems will probably to fall apart when we put real-world, inconsistent data into them.