The amount of data that businesses are having to store, manage, protect and backup is increasing at an exponential rate of between 40% and 60% per year on average. For unstructured data in the enterprise, the growth rate can be as high as 80%. This is causing huge issues for IT departments within every industry - IT budgets are staying relatively flat, but the amount of data that needs to be stored is increasing massively, which has pushed IT teams to store data on disparate systems to stay within budget.
Despite best efforts and significant investments, organisations lack visibility into ownership and usage of unstructured or 'dark' data. As a result, businesses are struggling to gain control of runaway costs, whilst security and compliance risks are increasing. Gartner estimates that by 2018, 25% of organisations will manage all their unstructured data using information governance and storage management policies - up from less than 1% today - to improve business value, reduce risk and lower management cost.
As the amount of data that we are storing grows, the risk of ransomware or viruses becomes increasingly important, and therefore businesses need to make sure they have a robust plan for backup and data recovery. Historically, many smaller and mid-sized businesses have used file shares from low-cost NAS (Network Attached Storage) devices as targets for backups; however these are at risk of being encrypted via a ransomware attack. In response to this, the platforms used are now utilising WORM (write once, read many) policies to protect the data - or API-based access to mitigate against the threats posed.
When you start storing vast amounts of data within a single system it is not only outside threats that have to be considered - the durability of the objects within the system becomes critical. There is no point having a system that can hold petabytes of data if the system then loses this data due to disk failure or file system corruption. The concept of RAID at this scale is no longer capable of supporting systems, especially those that are geographically diverse. As such, we've had to find a new system to protect data (at this scale most systems use Erasure Coding or a similar technology to provide protection across data centres and not just across disks).
Storing data in one place is a good start to a data strategy; however to ensure organisations can gain business insight from the data it needs to be available in different formats - this might be NFS (Network File System), CIFS (Common Internet File System) or HDFS (Hadoop Distributed File System), amongst others.
IT departments don’t want the strain of managing different data sets while trying to keep these consistent, therefore centralising the data into a single system that can provide multiple protocols for various business demands can provide simplification of management, and analysis based on live data.
Storing data that isn’t for compliance or business intelligence offers no value and is purely an expense - the associated costs reduce the capabilities of investing in new technologies and solutions that enable the business to reach its objectives. Analysing the unstructured data that organisations hold, and cleansing the data that doesn’t provide any benefit, is critical to ensure a successful data strategy.
Whilst it is great having a platform that addresses the need to have data presented in the correct format, it has limited use cases if the platform holding that data does not scale, or is complex to manage. Having a system that will scale linearly is critical to ensure a successful data lake. As such, modern data lake systems are built with a scale-out architecture that allows IT departments to take a building block approach to the solutions they need. This allows them to have a flexible solution that can scale in terms of performance, capacity - or both - depending on how they wish to deploy going forwards.
There are many platforms within the industry that can provide some of this functionality, however to provide the complete functionality to address all of the issues listed above - and achieve the goals laid out - Ultima believe that DellEMC’s Isilon platform is the most suitable for enterprises (*other solutions are available from Ultima!). Isilon is capable of scaling up to cope with petabytes of data, so can cope with the demands of the largest enterprise, as well as having a low starting point, making it accessible for businesses who don’t have the same magnitude of data to store.
Building these platforms and being able to realise the value of the data that resides within them can become a very complex process; however Ultima has the capabilities to help you interrogate, analyse and optimise the data you have as part of our Storage Advisory Service, or a wider Data Centre Optimisation engagement.
- By Matthew Beale (Storage Solutions Specialist)