easydel.inference.tools.tool_calling_mixin

easydel.inference.tools.tool_calling_mixin#

Mixin class for tool calling functionality in inference servers.

class easydel.inference.tools.tool_calling_mixin.ToolCallingMixin[source]#

Bases: object

Mixin class providing tool calling functionality for inference API servers.

This mixin provides: - Tool parser initialization and management - Tool call extraction for batch responses - Tool call extraction for streaming responses - Tool listing and metadata endpoints

Classes using this mixin should have: - self.tool_parsers: dict[str, ToolParser] - self.tool_parser_name: str - self.enable_function_calling: bool

create_tool_execution_placeholder() → JSONResponse[source]#

Create a placeholder response for tool execution endpoints.

Returns

JSONResponse with NOT_IMPLEMENTED status

create_tools_response(model_names: list[str]) → dict[str, Any][source]#

Create a standardized tools response for listing endpoints.

Parameters

model_names – List of available model names

Returns

Dictionary with tools information for each model

extract_tool_calls_batch(response_text: str, request: ChatCompletionRequest, model_name: str) → tuple[easydel.inference.openai_api_modules.ChatMessage, str][source]#

Extract tool calls from a batch response.

Parameters

  • response_text – The generated text response

  • request – The original chat completion request

  • model_name – The model name to get the appropriate parser

Returns

Tuple of (ChatMessage with potential tool calls, finish_reason)

extract_tool_calls_streaming(model_name: str, previous_text: str, current_text: str, delta_text: str, previous_token_ids: list[int] | None = None, current_token_ids: list[int] | None = None, delta_token_ids: list[int] | None = None, request: easydel.inference.openai_api_modules.ChatCompletionRequest | None = None) → easydel.inference.openai_api_modules.DeltaMessage | None[source]#

Extract tool calls from streaming response.

Parameters

  • model_name – The model name to get the appropriate parser

  • previous_text – Previously accumulated text

  • current_text – Current accumulated text

  • delta_text – New text in this chunk

  • previous_token_ids – Previous token IDs (optional)

  • current_token_ids – Current token IDs (optional)

  • delta_token_ids – Delta token IDs (optional)

  • request – The original request (optional)

Returns

DeltaMessage with tool call information or None

get_tool_parser_for_model(model_name: str) → easydel.inference.tools.abstract_tool.ToolParser | None[source]#

Get the tool parser for a specific model.

Parameters

model_name – Name of the model

Returns

ToolParser instance or None if not available

initialize_tool_parsers(model_processors: dict[str, Any], tool_parser_name: str, enable_function_calling: bool) → dict[str, easydel.inference.tools.abstract_tool.ToolParser][source]#

Initialize tool parsers for models.

Parameters

  • model_processors – Dictionary mapping model names to their processors/tokenizers

  • tool_parser_name – Name of the tool parser to use (e.g., “hermes”, “qwen”)

  • enable_function_calling – Whether to enable function calling

Returns

Dictionary mapping model names to their tool parsers

tool_parsers: dict[str, easydel.inference.tools.abstract_tool.ToolParser]#
Contents