easydel.inference.vinference.api_models#
- class easydel.inference.vinference.api_models.ChatCompletionRequest(*, model: str, messages: List[ChatMessage], function_call: Optional[str] = 'none', temperature: Optional[float] = 1, top_p: Optional[float] = 1.0, n: Optional[int] = 1, stream: Optional[bool] = False, stop: Optional[Union[str, List[str]]] = None, max_tokens: Optional[int] = 16, presence_penalty: Optional[float] = 0.0, frequency_penalty: Optional[float] = 0.0, logit_bias: Optional[Dict[str, float]] = None, user: Optional[str] = None)[source]#
Bases:
BaseModel- frequency_penalty: Optional[float]#
- function_call: Optional[str]#
- logit_bias: Optional[Dict[str, float]]#
- max_tokens: Optional[int]#
- messages: List[ChatMessage]#
- model: str#
- model_config: ClassVar[ConfigDict] = {}#
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- n: Optional[int]#
- presence_penalty: Optional[float]#
- stop: Optional[Union[str, List[str]]]#
- stream: Optional[bool]#
- temperature: Optional[float]#
- top_p: Optional[float]#
- user: Optional[str]#
- class easydel.inference.vinference.api_models.ChatCompletionResponse(*, id: str = <factory>, object: str = 'chat.completion', created: int = <factory>, model: str, choices: ~typing.List[~easydel.inference.vinference.api_models.ChatCompletionResponseChoice], usage: ~easydel.inference.vinference.api_models.UsageInfo)[source]#
Bases:
BaseModel- choices: List[ChatCompletionResponseChoice]#
- created: int#
- id: str#
- model: str#
- model_config: ClassVar[ConfigDict] = {}#
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- object: str#
- class easydel.inference.vinference.api_models.ChatCompletionResponseChoice(*, index: int, message: ChatMessage, finish_reason: Optional[Literal['stop', 'length', 'function_call']] = None)[source]#
Bases:
BaseModel- finish_reason: Optional[Literal['stop', 'length', 'function_call']]#
- index: int#
- message: ChatMessage#
- model_config: ClassVar[ConfigDict] = {}#
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- class easydel.inference.vinference.api_models.ChatCompletionStreamResponse(*, id: str = <factory>, object: str = 'chat.completion.chunk', created: int = <factory>, model: str, choices: ~typing.List[~easydel.inference.vinference.api_models.ChatCompletionStreamResponseChoice], usage: ~easydel.inference.vinference.api_models.UsageInfo)[source]#
Bases:
BaseModel- choices: List[ChatCompletionStreamResponseChoice]#
- created: int#
- id: str#
- model: str#
- model_config: ClassVar[ConfigDict] = {}#
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- object: str#
- class easydel.inference.vinference.api_models.ChatCompletionStreamResponseChoice(*, index: int, delta: DeltaMessage, finish_reason: Optional[Literal['stop', 'length', 'function_call']] = None)[source]#
Bases:
BaseModel- delta: DeltaMessage#
- finish_reason: Optional[Literal['stop', 'length', 'function_call']]#
- index: int#
- model_config: ClassVar[ConfigDict] = {}#
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- class easydel.inference.vinference.api_models.ChatMessage(*, role: str, content: Union[str, List[Mapping[str, str]]])[source]#
Bases:
BaseModel- content: Union[str, List[Mapping[str, str]]]#
- model_config: ClassVar[ConfigDict] = {}#
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- role: str#
- class easydel.inference.vinference.api_models.CountTokenRequest(*, model: str, conversation: Union[str, ChatMessage])[source]#
Bases:
BaseModel- conversation: Union[str, ChatMessage]#
- model: str#
- model_config: ClassVar[ConfigDict] = {}#
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- class easydel.inference.vinference.api_models.DeltaMessage(*, role: Optional[str] = None, content: Optional[Union[str, List[Mapping[str, str]]]] = None)[source]#
Bases:
BaseModel- content: Optional[Union[str, List[Mapping[str, str]]]]#
- model_config: ClassVar[ConfigDict] = {}#
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- role: Optional[str]#
- class easydel.inference.vinference.api_models.UsageInfo(*, prompt_tokens: int = 0, completion_tokens: Optional[int] = 0, total_tokens: int = 0, tokens_pre_second: float = 0, processing_time: float = 0.0, first_iter_flops: float = 0.0, iter_flops: float = 0.0)[source]#
Bases:
BaseModel- completion_tokens: Optional[int]#
- first_iter_flops: float#
- iter_flops: float#
- model_config: ClassVar[ConfigDict] = {}#
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- processing_time: float#
- prompt_tokens: int#
- tokens_pre_second: float#
- total_tokens: int#