easydel.modules.deepseek_v2.modeling_deepseek

class easydel.modules.deepseek_v2.modeling_deepseek.DeepseekV2Attention(*args: Any, **kwargs: Any)

Bases: UnifiedAttention

DeepSeek V2 Multi-head Latent Attention.

Inherits MLA implementation from UnifiedAttention base class.

define_network(config: DeepseekV2Config, dtype: dtype, param_dtype: dtype, precision: Precision, rngs: Rngs)

Define MLA-specific network structure.

projection_mapping: ClassVar[dict[str, str]] = {'mla_kv_a_layernorm': 'kv_a_layernorm', 'mla_kv_a_proj_with_mqa': 'kv_a_proj_with_mqa', 'mla_kv_b_proj': 'kv_b_proj', 'mla_q_a_layernorm': 'q_a_layernorm', 'mla_q_a_proj': 'q_a_proj', 'mla_q_b_proj': 'q_b_proj', 'mla_q_proj': 'q_proj', 'output_projection': 'o_proj'}

class easydel.modules.deepseek_v2.modeling_deepseek.DeepseekV2DecoderLayer(*args: Any, **kwargs: Any)

Bases: Module

Single DeepSeek V2 transformer block with MLA attention and optional MoE MLP.

class easydel.modules.deepseek_v2.modeling_deepseek.DeepseekV2ForCausalLM(*args: Any, **kwargs: Any)

Bases: BaseCausalLMModule[DeepseekV2Model, DeepseekV2Config]

DeepseekV2 model with a language modeling head for causal language modeling tasks.

This model extends the base DeepseekV2Model by adding a linear language modeling head on top of the transformer model. It’s designed for generative tasks and can be used for text generation.

class easydel.modules.deepseek_v2.modeling_deepseek.DeepseekV2MLP(*args: Any, **kwargs: Any)

Bases: Module

Standard DeepSeek V2 feed-forward block for dense layers.

class easydel.modules.deepseek_v2.modeling_deepseek.DeepseekV2MLPMoE(*args: Any, **kwargs: Any)

Bases: Module

Mixture-of-experts feed-forward used in DeepSeek V2 MoE layers.

reform_param: ClassVar = {'down_proj$': {'inverse_spliter': <function DeepseekV2MLPMoE.<lambda>>, 'splits': [{'name': 'down_proj.kernel', 'spliter': <function DeepseekV2MLPMoE.<lambda>>}]}, 'gate_up_proj$': {'inverse_spliter': <function DeepseekV2MLPMoE.<lambda>>, 'splits': [{'name': 'gate_proj.kernel', 'spliter': <function DeepseekV2MLPMoE.<lambda>>}, {'name': 'up_proj.kernel', 'spliter': <function DeepseekV2MLPMoE.<lambda>>}]}}

class easydel.modules.deepseek_v2.modeling_deepseek.DeepseekV2MoE(*args: Any, **kwargs: Any)

Bases: BaseMoeModule

Wraps gating and experts to apply DeepSeek V2 mixture-of-experts feed-forward.

class easydel.modules.deepseek_v2.modeling_deepseek.DeepseekV2Model(*args: Any, **kwargs: Any)

Bases: EasyDeLBaseModule

DeepSeek V2 decoder stack connecting embeddings, decoder layers, and final norm.

property frequencies

Compute RoPE frequencies using config’s get_basic_frequencies method.

get_decoder() → Module

Returns the decoder part of the model’s graph definition. For DeepseekV2Model, this is the model itself.

get_embedding() → Module

Returns the embedding layer of the module.

get_encoder() → Module

Returns the encoder part of the model’s graph definition. For DeepseekV2Model (decoder-only), this is not applicable.

get_lm_head() → Module

Returns the language model head of the module. DeepseekV2Model does not include the lm_head.

class easydel.modules.deepseek_v2.modeling_deepseek.MoEGate(*args: Any, **kwargs: Any)

Bases: Module

Router that scores tokens and selects experts for DeepSeek V2 MoE blocks.