Developing big data analytics architecture for spatial data

Abstract

In the mobile and Internet era, massive scale data is generated from disparate sources with spatial components. Traditional methods often fail to handle high volume, high speed, and variety of complex data in this fast and growing modern view of data generation and consumption. The modern technologies like Big Data and Big Data Analytics (BDA) have a huge potential to handle massive scale data with high scalability and low latency. Though the modern big data management tools are highly efficient, they offer limited functions and methods for spatial data management. In addition, in modern application development, only one specific big data tool would not be able to manage big data efficiently and effectively. Hence, it is highly enviable to exploit the potential features of big data tools and technologies and propose integrated frameworks and architectures built on top of more than one technology to develop robust and powerful applications including geospatial data. The research presents an open source and scalable big data analytics architecture for spatial data management. The main goal of the research work is to solve a wide range of data problems by offering batch, iterative, and interactive computations in a unified architecture. The proposed architecture is built and implemented on top of open source big data frameworks. In comparison with the existing platforms and architectures, the proposed architecture is in-memory, cost-effective, and open source. The architecture is realized for the agriculture domain. As a proof of concept, the spatial analytics applications are developed using agricultural real-life datasets. The main focus of the research is to develop a big data analytics framework to load, store, process, and query spatial data at scale. The framework is developed to enable interactive query processing on spatial and non-spatial data via a web based REST interface and distributed APIs. In comparison with the existing frameworks, the proposed framework is implemented on an integrated big data infrastructure with a new input data source, i.e. NoSQL database. The framework provides efficient and scalable solutions for spatial data by developing distributed APIs for spatial operations such as location search, proximity search, and K Nearest Neighbor (KNN) search. The application layer is implemented to accelerate ad-hoc query processing via a common web based REST interface. The user interface diverts the user requests to the suitable framework either Cassandra or Spark, i.e. low latency queries are executed on Cassandra and aggregated and complex queries are performed on the Spark cluster. The framework is evaluated by analyzing the performance of the analytical operations in terms of latency against the variable size of data. The performance of the framework is compared with the baseline technology, i.e. Cassandra for low latency queries.