High-Speed Large Language Model Serving on PCs with Consumer-Grade GPUs(github.com)
377 points by dataminer | 78 comments
GitHub repository for PowerInfer, an SJTU-IPADS engine for high-speed large language model inference on PCs with consumer-grade GPUs.