Enhancing Task Management in Apache Spark Through Energy-Efficient Data Segregation and Time-Based Scheduling

The rise of smart cities as solutions to urban challenges has garnered significant attention in recent years. With technological advancements, particularly in wireless communication and artificial intelligence, smart cities aim to optimize decision-making processes and improve citizen services. This study explores the integration of extensive infrastructure and networked Internet of Things (IoT) devices to collect data and enhance city performance. With urban populations steadily increasing, the need for efficient resource management and sustainability practices becomes paramount. However, challenges such as energy trading, privacy concerns, and security issues persist. To address these challenges, big data analytics (BDA) systems are crucial, necessitating efficient task scheduling strategies. This study proposes a Dynamic Smart Flow Scheduler (DSFS) system for Apache Spark, showcasing significant improvements in resource efficiency and task optimization. By reducing resource consumption and task execution, the proposed approach enhances system performance, scalability, and sustainability.