Skip to main content

Chat models

Features (natively supported)

All ChatModels implement the Runnable interface, which comes with default implementations of all methods, ie. ainvoke, batch, abatch, stream, astream. This gives all ChatModels basic support for async, streaming and batch, which by default is implemented as below:

  • Async support defaults to calling the respective sync method in asyncio's default thread pool executor. This lets other async functions in your application make progress while the ChatModel is being executed, by moving this call to a background thread.
  • Streaming support defaults to returning an Iterator (or AsyncIterator in the case of async streaming) of a single value, the final result returned by the underlying ChatModel provider. This obviously doesn't give you token-by-token streaming, which requires native support from the ChatModel provider, but ensures your code that expects an iterator of tokens can work for any of our ChatModel integrations.
  • Batch support defaults to calling the underlying ChatModel in parallel for each input by making use of a thread pool executor (in the sync batch case) or asyncio.gather (in the async batch case). The concurrency can be controlled with the max_concurrency key in RunnableConfig.

Each ChatModel integration can optionally provide native implementations to truly enable async or streaming. The table shows, for each integration, which features have been implemented with native support.

ModelInvokeAsync invokeStreamAsync streamTool callingStructured outputPython Package
AzureChatOpenAIlangchain-openai
BedrockChatlangchain-community
ChatAnthropiclangchain-anthropic
ChatAnyscalelangchain-community
ChatBaichuanlangchain-community
ChatCoherelangchain-cohere
ChatCozelangchain-community
ChatDatabrickslangchain-community
ChatDeepInfralangchain-community
ChatEverlyAIlangchain-community
ChatFireworkslangchain-fireworks
ChatFriendlilangchain-community
ChatGooglePalmlangchain-community
ChatGroqlangchain-groq
ChatHuggingFacelangchain-community
ChatHunyuanlangchain-community
ChatJavelinAIGatewaylangchain-community
ChatKineticalangchain-community
ChatKonkolangchain-community
ChatLiteLLMlangchain-community
ChatLiteLLMRouterlangchain-community
ChatMLXlangchain-community
ChatMLflowAIGatewaylangchain-community
ChatMaritalklangchain-community
ChatMistralAIlangchain-mistralai
ChatMlflowlangchain-community
ChatOctoAIlangchain-community
ChatOllamalangchain-community
ChatOpenAIlangchain-openai
ChatPerplexitylangchain-community
ChatPremAIlangchain-community
ChatSparkLLMlangchain-community
ChatTongyilangchain-community
ChatVertexAIlangchain-google-vertexai
ChatYandexGPTlangchain-community
ChatYuan2langchain-community
ChatZhipuAIlangchain-community
ErnieBotChatlangchain-community
GPTRouterlangchain-community
GigaChatlangchain-community
JinaChatlangchain-community
LlamaEdgeChatServicelangchain-community
MiniMaxChatlangchain-community
PaiEasChatEndpointlangchain-community
PromptLayerChatOpenAIlangchain-community
QianfanChatEndpointlangchain-community
SolarChatlangchain-community
VolcEngineMaasChatlangchain-community

Was this page helpful?


You can leave detailed feedback on GitHub.