Building your big data toolbox

When thinking about ‘big data’ for most people the first thing that comes to mind is technology. And they ask themselves what kind of tools and solutions they need to be able to generate the necessary advanced insights to move their business forward. Unfortunately answering this question is not so easy, since the amount of technologies and companies that claim to have the answer to your data challenge have exploded in the last three years. And without any background and experience in IT-technologies it will be very hard to know where to start.

To give an example, only two years ago the open-source ‘Hadoop’ was referred to as the solution you need to ‘do’ big data. However, this is not just one piece of software, since it is actually an ecosystem of multiple completely different modules and a range of additional software packages and languages that can be installed alongside it. As a result, it is often a bridge too far for most business managers to get their mind around what Hadoop actually does. On top of that, the big data landscape is changing rapidly, and you will probably always be one step behind reality. And to make matters worse, when looking into technical solutions it is essential to make sure that you have, or can get access to the right resources to ‘work’ with these technologies, which is often the hardest challenge. So what to do now?

Three important insights
Before answering this question we would like to share three important insights you need to take into account when considering the solutions and technologies you want to use for your company’s data challenge.

What is your organisations’ maturity level?
Firstly, have a critical look at the data analytics maturity level of your company at this moment. If this is still relatively low, there is no need to directly go for the most advanced, flexible and robust solutions. It is important to keep the balance between the added value of the data insights and the investments you need to make in technology and resources to be able to generate those insights. It would not be the first time that a company has invested large amounts of money to implement complex technology and get it working (sometimes failing hopelessly here), without having a clear strategy on how to use it and benefit from it. So a solid business case (or design) is essential, maybe critical for your digital development.

What are the ‘tools’ designed for? And what not!
Secondly, be aware that there is not one ‘tool’ or technology that can solve all of your problems. Every company has a different situation, with different needs and different people. There is not one tool that covers all the aspects of data process management that you need in your organisation. You need to work with multiple technologies and tools at the same time to develop an effective data-ecosystem for your organisation. The first trap that a number of companies fall into is that they start to use tools for purposes they were not designed for. With the support of smart technical employees or consultants, you can create a lot of work-arounds to do more with tools than the original intent, but you may find yourself in a situation where things get very complex, slow at one point and more expensive than you expected. By that time, it is rather difficult to get back easily, since you have started the change in your company and everybody is expecting results from these new efforts. When you bump into a limitation of one tool, simply look for another one that is designed especially for this purpose, although always avoid that your data ecosystem will start to look like a plate of spaghetti. The challenge in establishing a good data ecosystem creating a well-oiled data exchange between the different tools and data domains, which is something that Adversitement have specialised in over the last few years, since it is such an important aspect to develop a good data ecosystem.

What is our current data ecosystem like? And why?
Thirdly, do not start from scratch, but analyse your current data ecosystem first. Most companies already have a data ecosystem in place, but still some managers tend to ignore their current situation when designing the future. While attacking the problems and inefficiencies in their current ecosystem, they can generate most added value on a short-term notice. Besides that, setting up data streams between multiple tools or domains to create new combinations and insights within the current ecosystem can already be a large step upwards for your company’s data maturity level. As said, be aware for spaghetti, and know that free or cheap software licenses may get you started, but will not be ‘cheap’ in the long run, so be willing to invest in technologies that are able to support you in the things that are essential for your company’s success in data-driven business.

Get business users involved
So with these three points in mind, back to the question on what to do when considering the big data tooling you need. The first step is to analyse the needs for the business users within your department. What data is needed to gain insights or to automate processes, and why is this important? If your company’s data maturity level is very low, the business user might not be able to give answers to these questions. In that case do not hesitate to ask for expert support from external parties to help your organisation to find the right answers. Besides that, a fresh look from somebody outside your company gives you insights you may have overlooked.

Analyse your status
With this information you can analyse your current tooling and data domains. Is the needed data available? Can we create ways to obtain the needed data without extra tooling? If so, then you can move forward directly. If not, you need to see if there are tools and technologies on the market, what would be the costs for licenses, implementation and maintenance, and if this investment is worth it? Are the solutions and the way you use them compliant with the current data regulations? For all these steps you can let your IT department support you as an advisor, or hire an external specialist to guide you in the process, help you with the selection, implementation, governance and the data integration within the eco-system.

We hope that the insights outlined here help you when considering the tooling and technologies for your organisation’s data challenge. We invite you to do a free Digital Business Excellence Scan if you want to get some understanding what the data maturity level of your organisation is.