Digital Library[ Search Result ]
Development of Big Data Platform Operation and Management System Considering HPC Environments
Jae-Hyuck Kwak, Jieun Choi, Eunkyu Byun, Sangwan Kim
http://doi.org/10.5626/JOK.2020.47.3.240
Software technologies in traditional computational science and big data fields have evolved into different forms, but the growth of big data technology and recent advances in artificial intelligence technology have broken down boundaries between the two fields and have lead to generalized, high-performance computing environments. However, as these two areas of software stack were built and developed independently, it is not easy to integrate and operate them seamlessly in high-performance computing environment. In this paper, we developed a big data platform operation and management system considering high-performance computing environment. The system is an extension of Ambari, an open-source Hadoop platform operations management system that also provides installation management for Lustre, configuration of the Hadoop-on-Lustre execution environment, YARN job monitoring with user-defined and dynamic monitoring metrics as well as a web-based interface for high-performance computing resource monitoring.
Dynamic Memory Allocation for Scientific Workflows in Containers
Theodora Adufu, Jieun Choi, Yoonhee Kim
The workloads of large high-performance computing (HPC) scientific applications are steadily becoming “bursty” due to variable resource demands throughout their execution life-cycles. However, the over-provisioning of virtual resources for optimal performance during execution remains a key challenge in the scheduling of scientific HPC applications. While over-provisioning of virtual resources guarantees peak performance of scientific application in virtualized environments, it results in increased amounts of idle resources that are unavailable for use by other applications. Herein, we proposed a memory resource reconfiguration approach that allows the quick release of idle memory resources for new applications in OS-level virtualized systems, based on the applications resource-usage pattern profile data. We deployed a scientific workflow application in Docker, a light-weight OS-level virtualized system. In the proposed approach, memory allocation is fine-tuned to containers at each stage of the workflows execution life-cycle. Thus, overall memory resource utilization is improved.
A Design of a Distributed Computing Problem Solving Environment for Dietary Data Analysis
Jieun Choi, Younsun Ahn, Yoonhee Kim
Recently, wellness has become an issue related to improvements in personal health and quality of life. Data that are accumulated daily, such as meals and momentum records, in addition to body measurement information such as body weight, BMI and blood pressure have been used to analyze the personal health data of an individual. Therefore, it has become possible to prevent potential disease and to analyze dietary or exercise patterns. In terms of food and nutrition, analyses are performed to evaluate the health status of an individual using dietary data. However, it is very difficult to process the large amount of dietary data. An analysis of dietary data includes four steps, and each step contains a series of iterative tasks that are executed over a long time. This paper proposes a problem solving environment that automates dietary data analysis, and the proposed framework increases the speed with which an experiment can be conducted.
Search
Journal of KIISE
- ISSN : 2383-630X(Print)
- ISSN : 2383-6296(Electronic)
- KCI Accredited Journal
Editorial Office
- Tel. +82-2-588-9240
- Fax. +82-2-521-1352
- E-mail. chwoo@kiise.or.kr