Colossal-AI Team Open-Sources SwiftInfer: A TensorRT-Based Implementation of the StreamingLLM Algorithm

Colossal-AI team has open-sourced SwiftInfer, a TensorRT-based implementation of the StreamingLLM algorithm. StreamingLLM stabilizes text generation quality during multi-round conversations by employing a sliding-window-based attention module. SwiftInfer combines the strengths of StreamingLLM with TensorRT inference optimization, resulting in a 46% improvement in inference performance for large language models.