Energy Measurement System for Data Lake - Université de Pau et des Pays de l'Adour Access content directly
Conference Papers Year : 2024

Energy Measurement System for Data Lake

Abstract

Data Lakes are increasingly deployed as a solution for Big Data analytics. Recent improvements in Data Lake technology have focused on improving data access, governance, and discoverability. However, the energy consumption of data operations, a non-trivial issue for eco-conscious organizations, is currently overlooked. Furthermore, existing monitoring tools do not adequately address the complexities of Data Lake architectures. This paper presents the initial phase of developing a system for measuring energy in Data Lake pipeline operations. The novelty of our solution lies in the fact that we define four measures to assess the power usage of crucial hardware components in a Data Lake context: CPU, RAM, NIC, and storage devices. To validate our approach, we developed a monitoring tool grounded in real-world datasets from a Data Lake benchmark.
Fichier principal
Vignette du fichier
ACIIDS_2024_paper_63-1.pdf (678.74 Ko) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-04549466 , version 1 (17-04-2024)

Identifiers

  • HAL Id : hal-04549466 , version 1

Cite

Philippe Roose, Hernán H Álvarez Valera, Alexandre Maurice, Franck Ravat, Jiefu Song, et al.. Energy Measurement System for Data Lake: An Initial Approach. ACIIDS 2024 - 16th Asian Conference on Intelligent Information and Database Systems, Apr 2024, Ras Al Khaimah, United Arab Emirates. à paraître. ⟨hal-04549466⟩
76 View
23 Download

Share

Gmail Facebook X LinkedIn More