Hybrid Accelerator Selection for Generative AI Workloads: A Cost-Effective Approach Based on Model Type and Pipeline Stage

Abstract

With the increasing adoption of machine learning, AI, and large language models in production environments, the requirement for computational acceleration continues to intensify. However, defaulting to expensive off chip accelerators, such as GPUs and TPUs for all workloads leads to unnecessary cost inefficiencies. This paper presents a framework for hybrid accelerator selection that considers model characteristics, pipeline stages, and workload requirements. We demonstrate that strategic use of on-chip acceleration (CPU-based SIMD) for smaller models and latency-insensitive workloads, combined with selective deployment of off-chip accelerators for compute-intensive tasks, can significantly reduce infrastructure costs while maintaining performance requirements.