Scalable Parallel Algorithms for High-Performance Computing Systems
DOI:
https://doi.org/10.63282/3050-9262.IJAIDSML-V5I1P102Keywords:
HPC, Parallel Computing, Task Decomposition, Load Balancing, Communication Optimization, Fault Tolerance, Distributed Memory, Scalability, High-Speed Networks, Processor SynchronizationAbstract
High-Performance Computing (HPC) systems are essential for solving complex computational problems in various fields, including scientific research, engineering, and data analytics. The increasing demand for faster and more efficient computations has driven the development of scalable parallel algorithms. This paper provides a comprehensive overview of the current state of scalable parallel algorithms, focusing on their design, implementation, and performance optimization in HPC systems. We discuss key challenges, recent advancements, and future directions in the field. The paper also includes detailed algorithms, performance metrics, and case studies to illustrate the practical application and effectiveness of these algorithms
References
[1] Abdullah-Al-Mamun, A., Haider, C. M. R., Wang, J., & Aref, W. G. (2022). The "AI+R"-tree: An instance-optimized R-tree. arXiv preprint arXiv:2207.00550. https://arxiv.org/abs/2207.00550
[2] Álvarez Cid-Fuentes, J., Álvarez, P., Solà, S., Ishii, K., Morizawa, R. K., & Badia, R. M. (2021). ds-array: A distributed data structure for large scale machine learning. arXiv preprint arXiv:2104.10106. https://arxiv.org/abs/2104.10106
[3] Cantini, R., Marozzo, F., Orsino, A., Talia, D., Trunfio, P., Badia, R. M., Ejarque, J., & Vazquez, F. (2022). Block size estimation for data partitioning in HPC applications using machine learning techniques. arXiv preprint arXiv:2211.10819. https://arxiv.org/abs/2211.10819
[4] Peng, H., Ding, C., Geng, T., Choudhury, S., Barker, K., & Li, A. (2023). Evaluating emerging AI/ML accelerators: IPU, RDU, and NVIDIA/AMD GPUs. arXiv preprint arXiv:2311.04417. https://arxiv.org/abs/2311.04417
[5] Gu, A. (2024, September 11). AI will force a transformation of tech infrastructure. The Wall Street Journal. https://www.wsj.com/articles/ai-will-force-a-transformation-of-tech-infrastructure-c261f556
[6] NVIDIA. (n.d.). High performance computing (HPC) and AI. Retrieved from https://www.nvidia.com/en-us/high-performance-computing/hpc-and-ai/
[7] IBM. (n.d.). High performance computing (HPC) and AI. Retrieved from https://www.ibm.com/think/topics/hpc-ai
[8] Run:ai. (n.d.). HPC and AI: Better together. Retrieved from https://www.run.ai/guides/hpc-clusters/hpc-and-ai
[9] ZINFI Technologies. (2023, March 15). The role of artificial intelligence in high-performance computing. Retrieved from https://www.zinfi.com/blog/artificial-intelligence-in-high-performance-computing/
[10] DataBank. (2023, June 10). Why machine learning models demand high performance computing (HPC). Retrieved from https://www.databank.com/resources/blogs/why-machine-learning-models-demand-high-performance-computing-hpc/
[11] World Wide Technology. (2023, March 20). High performance computing (HPC) helped build AI capabilities of today. Retrieved from https://www.wwt.com/article/high-performance-computing-hpc-helped-build-ai-capabilities-of-today
[12] Sarbu, P. C. (2018). HPC-suitable data structures for machine learning and other applications. International HPC Summer School. Retrieved from https://www.hpc-training.org/moodle/pluginfile.php/1776/mod_data/content/486/C03_PaulCristian_Sarbu_ihpcss18.pdf
[13] Lim, S. (2019). Methods for accelerating machine learning in high performance computing systems. University of Oregon. Retrieved from https://www.cs.uoregon.edu/Reports/AREA-201901-Lim.pdf
[14] Penguin Computing. (n.d.). InsightHPC: HPC and AI solutions. Retrieved from https://www.penguinsolutions.com/solutions/hpc/insighthpc
[15] IDTechEx. (2025, January 5). CPUs, GPUs, and AI: Exploring high-performance computing hardware. Retrieved from https://www.edge-ai-vision.com/2025/01/cpus-gpus-and-ai-exploring-high-performance-computing-hardware/
[16] Deloitte. (2021). High performance computing in AI. Retrieved from https://www2.deloitte.com/us/en/pages/consulting/articles/nvidia-alliance-high-performance-computing-in-ai.html
[17] Configr. (2023, August 1). Mastering sparse data structures: Efficient strategies for handling zero-rich datasets at scale. Retrieved from https://configr.medium.com/mastering-sparse-data-structures-efficient-strategies-for-handling-zero-rich-datasets-at-scale-0333f2ce24e0
[18] Álvarez Cid-Fuentes, J., Álvarez, P., Solà, S., Ishii, K., Morizawa, R. K., & Badia, R. M. (2021). ds-array: A distributed data structure for large scale machine learning. arXiv preprint arXiv:2104.10106. https://arxiv.org/abs/2104.10106
[19] Cantini, R., Marozzo, F., Orsino, A., Talia, D., Trunfio, P., Badia, R. M., Ejarque, J., & Vazquez, F. (2022). Block size estimation for data partitioning in HPC applications using machine learning techniques. arXiv preprint arXiv:2211.10819. https://arxiv.org/abs/2211.10819
[20] Peng, H., Ding, C., Geng, T., Choudhury, S., Barker, K., & Li, A. (2023). Evaluating emerging AI/ML accelerators: IPU, RDU, and NVIDIA/AMD GPUs. arXiv preprint arXiv:2311.04417. https://arxiv.org/abs/2311.04417
