easydel.inference.esurge.runners.states

easydel.inference.esurge.runners.states#

class easydel.inference.esurge.runners.states.CachedRequestState(req_id: str, prompt_token_ids: list[int], sampling_params: SamplingParams, generator: PRNGKey, page_ids: tuple[list[int], ...], num_computed_tokens: int, output_token_ids: list[int], num_prompt_tokens: int = -1)[source]#

Bases: object

Represents the state of a single request, compatible with JAX.

classmethod from_dict(data: dict[str, Any]) → T#

Deserializes a dictionary into a PyTree object.

classmethod from_json(json_str: str) → T#

Deserializes a JSON string into a PyTree object.

generator: PRNGKey#
get_token_id(idx: int) → int[source]#
num_computed_tokens: int#
num_prompt_tokens: int = -1#
property num_tokens: int#
output_token_ids: list[int]#
page_ids: tuple[list[int], ...]#
prompt_token_ids: list[int]#
replace(**kwargs)#

Creates a new instance with specified fields replaced.

req_id: str#
sampling_params: SamplingParams#
to_dict() → dict[str, Any]#

Serializes the PyTree object to a dictionary.

to_json(**kwargs) → str#

Serializes the PyTree object to a JSON string.

Contents