easydel.modules.minimax_text_v1.__init__

class easydel.modules.minimax_text_v1.__init__.MiniMaxText01Config(vocab_size=32000, hidden_size=4096, intermediate_size=14336, num_hidden_layers=32, num_attention_heads=32, num_key_value_heads=8, hidden_act='silu', max_position_embeddings=131072, initializer_range=0.02, rms_norm_eps=1e-05, use_cache=True, pad_token_id=None, bos_token_id=None, eos_token_id=None, tie_word_embeddings=False, rope_theta=1000000.0, sliding_window=None, attention_dropout=0.0, num_experts_per_tok=2, num_local_experts=8, output_router_logits=False, router_aux_loss_coef=0.001, router_jitter_noise=0.0, **kwargs)

Bases: EasyDeLBaseConfig

This is the configuration class to store the configuration of a [MiniMaxText01Model]. It is used to instantiate an MiniMaxText01 model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the MiniMaxText01. Configuration objects inherit from [PretrainedConfig] and can be used to control the model outputs. Read the documentation from [PretrainedConfig] for more information. :param vocab_size: Vocabulary size of the MiniMaxText01 model. Defines the number of different tokens that can be represented by the

inputs_ids passed when calling [MiniMaxText01Model]

Parameters

• hidden_size (int, optional, defaults to 4096) – Dimension of the hidden representations.

• intermediate_size (int, optional, defaults to 14336) – Dimension of the MLP representations.

• num_hidden_layers (int, optional, defaults to 32) – Number of hidden layers in the Transformer encoder.

• num_attention_heads (int, optional, defaults to 32) – Number of attention heads for each attention layer in the Transformer encoder.

• num_key_value_heads (int, optional, defaults to 8) – This is the number of key_value heads that should be used to implement Grouped Query Attention. If num_key_value_heads=num_attention_heads, the model will use Multi Head Attention (MHA), if num_key_value_heads=1 the model will use Multi Query Attention (MQA) otherwise GQA is used. When converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed by meanpooling all the original heads within that group. For more details checkout [this paper](https://arxiv.org/pdf/2305.13245.pdf). If it is not specified, will default to `8.

• hidden_act (str or function, optional, defaults to “silu”) – The non-linear activation function (function or string) in the decoder.

• max_position_embeddings (int, optional, defaults to 4096*32) – The maximum sequence length that this model might ever be used with. MiniMaxText01’s sliding window attention allows sequence of up to 4096*32 tokens.

• initializer_range (float, optional, defaults to 0.02) – The standard deviation of the truncated_normal_initializer for initializing all weight matrices.

• rms_norm_eps (float, optional, defaults to 1e-05) – The epsilon used by the rms normalization layers.

• use_cache (bool, optional, defaults to True) – Whether or not the model should return the last key/values attentions (not used by all models). Only relevant if config.is_decoder=True.

• pad_token_id (int, optional) – The id of the padding token.

• bos_token_id (int, optional, defaults to 1) – The id of the “beginning-of-sequence” token.

• eos_token_id (int, optional, defaults to 2) – The id of the “end-of-sequence” token.

• tie_word_embeddings (bool, optional, defaults to False) – Whether the model’s input and output word embeddings should be tied.

• rope_theta (float, optional, defaults to 1000000.0) – The base period of the RoPE embeddings.

• sliding_window (int, optional) – Sliding window attention window size. If not specified, will default to 4096.

• attention_dropout (float, optional, defaults to 0.0) – The dropout ratio for the attention probabilities.

• num_experts_per_tok (int, optional, defaults to 2) – The number of experts to route per-token, can be also interpreted as the top-k routing parameter

• num_local_experts (int, optional, defaults to 8) – Number of experts per Sparse MLP layer.

• output_router_logits (bool, optional, defaults to False) – Whether or not the router logits should be returned by the model. Enabeling this will also allow the model to output the auxiliary loss. See [here]() for more details

• router_aux_loss_coef (float, optional, defaults to 0.001) – The aux loss factor for the total loss.

• router_jitter_noise (float, optional, defaults to 0.0) – Amount of noise to add to the router.

`python >>> from transformers import MiniMaxText01Model, MiniMaxText01Config >>> # Initializing a MiniMaxText01 style configuration >>> configuration = MiniMaxText01Config() >>> # Initializing a model from the MiniMaxText01 style configuration >>> model = MiniMaxText01Model(configuration) >>> # Accessing the model configuration >>> configuration = model.config `

get_partition_rules(*args, **kwargs)

Get the partition rules for distributing the MiniMaxText01 model parameters across multiple devices.

These rules define how parameters should be partitioned when using techniques like Fully Sharded Data Parallelism (FSDP), Sharded Parallelism (SP), and Tensor Parallelism (TP). Each rule consists of a regex pattern matching parameter names and a corresponding PartitionSpec.

Returns

A tuple of tuples where each inner tuple contains:

• A regex pattern matching parameter names

• A PartitionSpec object specifying how to partition matching parameters

Return type

tuple

keys_to_ignore_at_inference = ['past_key_values']

model_type: str = 'MiniMaxText01'

class easydel.modules.minimax_text_v1.__init__.MiniMaxText01ForCausalLM(*args: Any, **kwargs: Any)

Bases: EasyDeLBaseModule

class easydel.modules.minimax_text_v1.__init__.MiniMaxText01Model(*args: Any, **kwargs: Any)

Bases: EasyDeLBaseModule