LLM擅长文本生成应用程序，如聊天和代码完成模型，能够高度理解和流畅。但是它们的大尺寸也给推理带来了挑战。有很多个框架和包可以优化LLM推理和服务，所以在本文中我将整理一些常用的推理引擎并进行比较。 TensorRT-LLM TensorRT-LLM是NV发布的一个推理引擎。

New tools for filtering malicious prompts, detecting ungrounded outputs, and evaluating the safety of models will make generative AI safer to use. Both extremely promising and extremely risky, ...

After adding the Mistral Large LLM as an option for its Azure AI services, Microsoft has included the Mistral Small LLM as well. It is supposed to be used for high-volume workloads with low latency.