The traditional MapReduce frameworks were originally designed for processing data within a single cluster and are not suitable for handling geo-distributed data. Consequently, alternative approaches such as Hierarchical and Geo-Hadoop have been proposed to address this limitation. However, these approaches still face challenges in efficiently managing inter-cluster data transfer, particularly considering the heterogeneity of clusters and varying bandwidth among them. Moreover, the need to transmit results to a central global reducer for geo-distributed MapReduce operations adds unnecessary complexity. To tackle these issues, we introduce Extended Cross-MapReduce (ECMR), a framework that integrates resource heterogeneity and network links in geo-distributed MapReduce workflows. ECMR optimizes data management and determines the necessary data volume for generating final results. To enhance performance, ECMR leverages the overlap between data transfer and execution time by utilizing multiple global reducers and grouping temporary results that require data transfer over the Internet. In ECMR, we propose a bipartite graph and extend the Gale-Shapley algorithm to determine the optimal number of clusters and select the most suitable locations for global reducers. Through extensive experimental evaluations conducted on a real testbed, we demonstrate the effectiveness of our proposed ECMR method. The results exhibit significant improvements over traditional Hierarchical and Geo-Hadoop approaches, achieving reductions of up to 81% and 85% in overall makespan, respectively.

Saeed Mirpour Marzuni

Adel Toosi

Abdorreza Savadi

Mahmud Naghibzadeh

David Taniar

ACM Transactions on Internet Technology

ACM TRANSACTIONS ON INTERNET TECHNOLOGY (TOIT) 汇集了许多计算学科，包括计算机软件工程、计算机编程语言、中间件、数据库管理、安全性、知识发现和数据挖掘、网络和分布式系统、通信、性能和可扩展性等。TOIT 将涵盖各个学科的结果和作用以及它们之间的关系。

ACM Transactions on Internet Technology (TOIT) brings together research from various computer science disciplines, including software engineering, programming languages, middleware, database management, security, knowledge discovery and data mining, networking and distributed systems, communication, performance, and scalability. TOIT focuses on the results and interrelations of these disciplines.

《ACM Transactions on Internet Technology (TOIT)》汇集了多个计算机学科的研究成果，包括计算机软件工程、编程语言、中间件、数据库管理、安全性、知识发现与数据挖掘、网络与分布式系统、通信、性能与可扩展性等。TOIT 关注这些学科的研究成果及其相互关系。

Category	Quartile
计算机科学	3
计算机科学, 计算机信息系统	3
计算机科学, 计算机软件工程	3

Optimizing Geo-Distributed Data Processing with Resource Heterogeneity over the Internet

ivySCI AI Smartly Parses PDF, Answers Researchers' Questions, and Helps You Understand Papers in Seconds

Journal Info

Category	Quartile
COMPUTER SCIENCE, SOFTWARE ENGINEERING	1