Medical Big Data – big problem or big opportunity?

Over the past few years, data has seen exponential growth. This growth has been so dramatic that petabytes are now normal, exabytes are on the horizon and we are now beginning to grasp what a zettabyte is. To put this data explosion into perspective, one needs to only look at the proliferation of the Internet, social media, and e-commerce in India. Social media is growing at 100 per cent and is likely to touch 45 million users by 2012; 30 million Indians who are online consumers are members of social networking sites, and 80 per cent of India’s 65 million Internet users are searching or transacting online. These numbers have undoubtedly fuelled the growth of unstructured data.

If one were to go through any of the numerous forecasts and IT predictions for 2012, there is one common issue that finds mention at the top of everyone’s list: ‘Big Data’. So, what exactly is ‘Big Data’, and why is everybody talking about it all of a sudden? According to Wikipedia, “Big Data are datasets that grow so large that they become awkward to work with using on-hand data management tools.

Many of the reports on Big Data would also refer to the healthcare industry as being one of the biggest contributors to this trend. This does not mean that patient numbers have dramatically increased overnight, or that healthcare organisations have suddenly doubled their business. The fact is that the Big Data phenomenon is not merely about the volume of data or the trajectory of its growth, but rather, about the untapped potential of already existing data sets which have so far not been stored, managed and analysed in an optimal fashion. Today, given the advancement of technology and the reduction in storage costs, we now have increased access to the information available in non-traditional forms. This, in turn, has given rise to the Big Data phenomenon.

The healthcare industry does in fact account for the proliferation of a large amount of unstructured data. Increasing interest by healthcare organisations in collecting, storing and processing patient data, medical records and information to determine trends and carry out research, has led to its own set of storage, security, and management challenges, with significant implications for the sector. This is because the proper management, storage, sharing and analysis of data is imperative for saving lives, improving the way healthcare is delivered, and making healthcare more reliable, affordable and accessible.

Yet the latest studies also show that Big Data can generate big value and create new business opportunities if mined wisely. In a report by McKinsey Global Institute (MGI) published in early 2011, MGI found that if healthcare providers were to use Big Data creatively and effectively to drive efficiency and quality, the sector could create more than $300 billion in new value every year. However, managing, preserving and manipulating Big Data is no easy task, especially with today’s rapid growth of medical technologies, mobile devices and network connectivity, all of which significantly accelerate the volume of medical Big Data and add to its complexity.

In fact, in Gartner’s ‘Top Predictions for IT Organisations and Users for 2012 and Beyond’, Gartner forecasts that more than 85 per cent of Fortune 500 organisations will fail to effectively exploit Big Data for competitive advantage between now and 2015. Fortunately, this prediction need not apply to everyone. That’s because the right storage infrastructure, namely an infrastructure that simplifies the management and retention of data can go a long way towards reducing the complexity of Big Data management.

Irrespective of whether the medical Big Data is structured or unstructured, investing in the right storage infrastructure that enables everything to be centrally managed and consolidated with high levels of availability, scalability and compliance is the need of the hour. With this level of storage, healthcare organisations can create big value from medical Big Data by collecting, accessing and protecting it in a timely manner that enhances efficiency, productivity and service quality. In other words, the answer to whether your medical Big Data will become a big problem or a big value is entirely dependent on the competence and intelligence of your storage infrastructure.

Another important aspect being compounded by the Big Data challenge is that of business and operational continuity, which is crucial to the healthcare sector. Protecting invaluable medical records and maintaining business continuity is the bottom line of all healthcare organisations. However, these goals have been complicated by today’s fast-evolving technology and the rapid proliferation of medical Big Data.

An indestructible DR solution – the need of the hour

As organisations begin working on an IT plan for 2012, an area of growing concern is that of operational continuity even under extreme circumstances. The spate of natural disasters that we have seen in the recent past such as the Japan catastrophe and the more recent floods in Thailand, have once again underlined the need for organisations to ensure that they have an effective disaster recovery plan in place. Closer to home, the recent fire at the Columbia Asia Hospital in Bangalore, and at Kolkata’s AMRI Hospital, have demonstrated that apart from the risks to patients and the tragic loss of lives, there is the added risk of losing valuable, and confidential medical data, patient records and information which could have far reaching implications for all those affected.

The first step for organisations looking to adapt to the Big Data challenge therefore, is to begin doing things a little differently. Organisations need to change their perspective on unstructured data from being unwanted to being an asset they can mine to create new value.

On a more practical level, organisations now need to move away from investing in backup as part of their strategic direction. Adding storage layers only increases complexity, and organisations need to adopt the latest technologies that will help them move to a backup-free approach to unstructured content or Big Data.

Today, given the possibilities that come from infrastructure compression (e.g. smaller cluster systems, storage, and analytics) that is less costly, more and more organisations can now take advantage of Big Data architecture and tools. Big Data is not just another passing trend that has technologists salivating at a new gold rush, but is about the promise of everyday people interacting with confidence in technologies to answer questions that may require analysing enormous quantities of data to make their work more efficient, secure, and of far greater value to customers, patients, and stakeholders.