Issues, Challenges and Solutions of Big Data in Information Management: An Overview

Big data is too voluminous and requires the use of computer technology and the right data processing application to capture, store effectively, analyse, and present big data to enable the business to have clearer visibility of trends, make plans and decision for future direction. The data revolution is reshaping the way knowledge is produced, business is conducted, humanitarian assistance is handled, public officials are elected, and governance is enacted. Big data provides unprecedented insights and opportunities across all industries, and it raises concerns that must be addressed. This article discusses the challenges and solutions for big data as an important tool for the benefit of the public. It suggests that big data and data analytics if used properly, can provide real-time actionable information that can be used to identify problems and needs, offer services, and provide feedback on the effectiveness of policy action. Introduction The world is experiencing a huge data revolution. The explosion of data is directly connected to the arrival of the digital age. The term “Big Data” refers to the vast amounts of data in which traditional data processing procedures and tools would not be able to handle. It emerged in the 1990s and gained momentum in the early 2000s and has been variously defined and operationalized. Clearly, size often comes to mind when referring to big data. It is commonly defined as the astonishing amount of structured and unstructured data that are being generated, captured, and stored at an amazing speed. It is obvious that data comes from literally everywhere, every time and from all sorts of devices. Data is often produced and accessible in real time, and it arises from the merging of different sources. Organizations have been relying on these sources of data to describe, interpret, and forecast and provision economic and business activities and to decide for the next direction. Today, various efficient and intelligent techniques are available to help the organization in providing the best interpretation of this large volume of data from different types of heterogeneous sources, to be International Journal of Academic Research in Business and Social Sciences Vol. 8 , No. 12, Dec, 2018, E-ISSN: 2222-6990 © 2018 HRMARS 1384 processed and analysed and presented in an understandable, visual and decent manner to suit the business language and stakeholders’ objectives. Big data is categorized as having three features, the 3Vs; which are value, velocity and variety. Volume refers to the sheer amount of data, velocity is the speed with which it is being delivered in real-time, and variety is the number of different sources. With the combination of increased volume, velocity and variety, organizations need to reconsider, rethink, and repurpose their ways of working with innovation, the marketplace and communication. Their entire business processes may need to change as well. Characteristics of Big Data Management As one of the current trend terms in the world today, there is no exact way to define big data. The term is often used with related concepts such as Business Intelligence (BI) and data mining. These three terms are about analysing data, but big data concepts differed from the others two concept where the data volumes, number of transactions and the number of data sources is bigger and much complex and acquired special methods and technologies to analysed the data. The three major characteristics that define big data are volume, variety and velocity. The first characteristic is volume, and it became as the first and important characteristic because of in every organization, they have many large archived data along the way from the starts of their business but they can't process the data to help them make decisions and plan their future business. This has represented as the most immediate challenge to the conventional IT structures. The said situation faced by the organization what inspired the IT researchers to come out with big data. The ability of big data to process large amounts of information is the main attraction of big data analytics to create value from relevant data. The second major characteristic in big data is velocity and it refers to the pace of increasing speed of the new data is generated and the pace at which data been transfer around. In the current electronic and digital era like business processes, machines, networks and human interaction with things like social media sites, mobile devices, etc. Social media messages going to viral in seconds in 1999, WalMart’s data warehouse stored 1,000 terabytes (1,000,000 gigabytes) of data. In the year 2012, it had access to over 2.5 petabytes (2,500,000 gigabytes) of data. Every minute of every day, we upload hundreds of hours of videos on YouTube, we send over 200 million emails through Gmail. Reacting quickly enough to deal with data generate pace is a challenge for most organizations. Twitter is one of the most used online diary worldwide and it can be as the best example of the velocity of big data. It produces around 6,000 tweets every second that corresponds to over 350,000 tweets sent per minute, 500 million tweets per day and approximately 200 billion tweets per year. The third characteristic of big data is variety as the sources and types of data is different and big data is not only a structured data and for sure is not an easy task to put big data into a relational database. Today data comes in the form of emails, photos, videos, monitoring devices, PDFs, audio, etc. This variation of unstructured data creates problems for storage, mining and analysing data. Dealing with International Journal of Academic Research in Business and Social Sciences Vol. 8 , No. 12, Dec, 2018, E-ISSN: 2222-6990 © 2018 HRMARS 1385 a variation of structured and unstructured data significantly increases the difficulty of storing and analysing big data. Around 90% of data generated is data in unstructured form. This will conclude that the “big” in big data is not solely about volume because big data involves of a large amount of data. But, big data not only refer to data volume because it does not only mean you have a lot of data but the data also coming to you at a very fast pace and it comes with a variation of forms and variety of sources. It is important to say that today’s big data may not be tomorrow’s big data because the technologies always evolve. As a perspective, if the organization facing a significant challenge to manage with data’s volume, velocity and variety, the organization need to have data management with the assistance of technologies and techniques. Big Data Presents Challenges and Opportunities Big data is not just about volume and from various sources; it is about its other characteristics such as size, speed of data, structure and quality and new-generation analytic technologies that help organizations get more value from their information assets. It helps organizations to provide business insights on customer behaviour and patterns that can be used to improve operations, anticipate opportunities and business growth, and to detect for any possibility of issues or problems. Today, organizations are generating, receiving, processing, and storing remarkable amount of data to and from a wide range of resources such as databases and Internet. The process of managing, handling and storing huge amount of data is known as big data management. Big data has many features and varies in nature, they can be simple or complex, structured or unstructured, secure or with a very minimum security. This makes the management and storage of big data becomes more challenging and extremely important, and would definitely require the help of technology and techniques. No matter how challenging it can be, organizations must put in the necessary control measures to solve or lower the risk imposed so that the data stored are available, retrievable, and can be used to make ad-hock decisions and plans for the future. The main purposes of the management of big data are to store and handle available data in a simple and understandable manner with easier and flexible retrieval methods to make the right and strategic decisions for enhancing their businesses. Issues and Challenges of Big Data in Information Management New technology comes with uncertainty as it may not be understood well by all organizations especially during its early introduction, adoption, and implementation. Big data is not an exceptional. Organizations lack of information and understanding on the fundamental of big data such as what is actually is, the benefits, and the infrastructure requirements. Uncertainties are the outcome from unknown opportunities, challenges and emerging risks. While opportunities create new values, risks create threats to the organization. International Journal of Academic Research in Business and Social Sciences Vol. 8 , No. 12, Dec, 2018, E-ISSN: 2222-6990 © 2018 HRMARS 1386 Lack of Business Sponsorship and Management’s Support When introducing any new technology in the organization, it is important to obtain top management’s buy-in. This is very essential if it involves changes in the organizational culture and structure as well as amount of time and money to invest. Hence, big data as any new technology needs support from top management and stakeholders. Since investing on big data wisely would give organizations competitive advantage, top management should have the vision to see the impact of big data in the organization’s future. Big data is not only a technology, but it involves innovation, cultural change, analytical mindset and new skillset. It also requires more effort to educate people on how to treat the data. Data Privacy and Security Organizations benefit from many conveniences and breakthroughs due to Big Data-powered applications and services. However, one of the most sensitive issues that organizations feel reluctant to adopt big data