合并lora权重到基础模型权重

合并lora权重到基础模型权重

无论是用NEMO_ENABLE_USER_MODULES=1 CUDA_VISIBLE_DEVICES=0,1 automodel --nproc-per-node=2 train-Qwen-Qwen-2.5-0.5B-Instruct.yaml执行
继续预训练，还是SFT微调，都支持lora微调方法，使用lora微调后会得到adapter权重文件，合并lora权重到基础模型权重可以使用AutoModel提供的merge_lora.py脚本，指令如下所示

python /opt/Automodel/tools/merge_lora.py\--base-model /hmlp/data/storage/Qwen2.5-0.5B-Instruct/V1/Qwen2.5-0.5B-Instruct\--adapter-path /hmlp/data/finetune/8e418d7f-b009-4f4f-a1a3-a19fc27b7f0c/f1a28b50-b8e5-4318-8545-5a8b57b67be8/output/epoch_0_step_229/model\--output-dir /hmlp/data/finetune/8e418d7f-b009-4f4f-a1a3-a19fc27b7f0c/f1a28b50-b8e5-4318-8545-5a8b57b67be8/merge/Qwen2.5-0.5B-Instruct-v4

• base-model：指定基础模型的 HuggingFace 模型名或本地路径

• adapter-path：指定 PEFT adapter 目录路径，目录下需要包含 adapter_config.json 和 adapter 权重文件

• output-dir：指定合并后模型的输出目录，合并后的模型权重和配置会保存到这里，默认也会把 tokenizer 一起保存

• qlora：告诉脚本这是 QLoRA 场景，类型: 布尔开关 默认值: False。根据训练时是否包含如下配置判断是否是QLoRA 场景

  quantization:load_in_4bit:Trueload_in_8bit:Falsebnb_4bit_compute_dtype:bfloat16bnb_4bit_use_double_quant:Truebnb_4bit_quant_type:nf4

  • 开启后，基础模型会以 4-bit 方式加载
  • 然后先反量化，再做 merge
  • 用于避免直接对量化权重做 naive merge 带来的效果下降

• dtype：指定合并后模型权重的数据类型，默认为 float16。可选值如下

  • float16
  • bfloat16
  • float32

• device： 指定模型加载时使用的设备映射，默认为 “auto”，表示自动检测和使用可用的 GPU 设备。也可以指定为 “cpu” 或 “cuda”。

  --device cpu--device auto

  • 在 --qlora 模式下，这个参数实际上没有生效，因为代码里固定用了 device_map={“”: 0}，见 tools/merge_lora.py:233-234

• no-save-tokenizer：不保存 tokenizer，类型: 布尔开关 默认值: False

• trust-remote-code：加载模型时允许执行 HuggingFace 仓库中的自定义代码。 必填: 否 类型: 布尔开关 默认值: False。对一些自定义模型结构是必须的

• model-class：显式指定使用哪个 transformers 的 AutoModel* 类来加载基础模型，必填: 否 默认值: None

  --model-class AutoModel--model-class AutoModelForCausalLM

  • 如果不传，脚本会尝试从 adapter_config.json 的 task_type 自动推断
  • 映射关系定义在 TASK_TYPE_TO_AUTO_CLASS，见 tools/merge_lora.py:76
  • 如果既没有显式传入，也无法识别 task_type，则回退到 AutoModelForCausalLM

model-class自动推断模型类规则

–model-class 不传时，脚本按下面顺序决定加载类，见 tools/merge_lora.py:86-121

• 优先使用你显式传入的 --model-class
• 否则读取 adapter_path/adapter_config.json 里的 task_type
• 再映射到对应的 transformers Auto 类
• 如果还是不行，默认使用 AutoModelForCausalLM

当前支持的 task_type 映射为

• CAUSAL_LM -> AutoModelForCausalLM
• SEQ_CLS -> AutoModelForSequenceClassification
• SEQ_2_SEQ_LM -> AutoModelForSeq2SeqLM
• TOKEN_CLS -> AutoModelForTokenClassification
• QUESTION_ANS -> AutoModelForQuestionAnswering
• FEATURE_EXTRACTION -> AutoModel

checkpoint.save_consolidated

checkpoint:save_consolidated:true

当为 true 时，将一个与 HuggingFace 兼容的单个检查点写入model/consolidated/，可直接由 Transformers、vLLM 等加载。需要 safetensors 格式。
此参数并不会自动合并lora权重生成完整模型

• save_consolidated: true 的本质 是“额外保存一份可直接 from_pretrained 加载的完整 HF 格式模型权重”
• 不会在 LoRA/PEFT 微调时自动合并 base model + adapter 生成最终完整模型。 原因是 PEFT 场景下 is_peft=True，而 consolidated 导出逻辑明确排除了 PEFT。
• 训练保存的仍然主要是 adapter 权重

合并后模型部署

合并后模型部署会报如下错误:

INFO 06-01 20:26:39 [__init__.py:239] Automatically detected platform cuda. INFO 06-01 20:26:44 [api_server.py:1043] vLLM API server version 0.8.5 INFO 06-01 20:26:44 [api_server.py:1044] args: Namespace(subparser='serve', model_tag='/hmlp/data/finetune/ead13c8f-2a8f-4952-b0ca-ce6bc70b5e1d/f1a28b50-b8e5-4318-8545-5a8b57b67be8/merge/Qwen-2.5-0.5B-Instruct-0.3-V6', config='', host='0.0.0.0', port=38109, uvicorn_log_level='info', disable_uvicorn_access_log=False, allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, enable_ssl_refresh=False, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='/hmlp/data/finetune/ead13c8f-2a8f-4952-b0ca-ce6bc70b5e1d/f1a28b50-b8e5-4318-8545-5a8b57b67be8/merge/Qwen-2.5-0.5B-Instruct-0.3-V6', task='auto', tokenizer=None, hf_config_path=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, load_format='auto', download_dir=None, model_loader_extra_config={}, use_tqdm_on_load=True, config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', max_model_len=None, guided_decoding_backend='auto', reasoning_parser=None, logits_processor_pattern=None, model_impl='auto', distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=2, data_parallel_size=1, enable_expert_parallel=False, max_parallel_loading_workers=None, ray_workers_use_nsight=False, disable_custom_all_reduce=False, block_size=None, gpu_memory_utilization=0.25, swap_space=4, kv_cache_dtype='auto', num_gpu_blocks_override=None, enable_prefix_caching=None, prefix_caching_hash_algo='builtin', cpu_offload_gb=0, calculate_kv_scales=False, disable_sliding_window=False, use_v2_block_manager=True, seed=None, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_token=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config={}, limit_mm_per_prompt={}, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=None, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=None, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', speculative_config=None, ignore_patterns=[], served_model_name=['Qwen-2.5-0.5B-Instruct-0.3 2070948db0fc473488dcc8c0f9250b8e'], qlora_adapter_name_or_path=None, show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, max_num_batched_tokens=None, max_num_seqs=None, max_num_partial_prefills=1, max_long_partial_prefills=1, long_prefill_token_threshold=0, num_lookahead_slots=0, scheduler_delay_factor=0.0, preemption_mode=None, num_scheduler_steps=1, multi_step_stream_outputs=True, scheduling_policy='fcfs', enable_chunked_prefill=None, disable_chunked_mm_input=False, scheduler_cls='vllm.core.scheduler.Scheduler', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', worker_extension_cls='', generation_config='auto', override_generation_config=None, enable_sleep_mode=False, additional_config=None, enable_reasoning=False, disable_cascade_attn=False, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False, enable_server_load_tracking=False, dispatch_function=<function ServeSubcommand.cmd at 0x7faf290a0540>) INFO 06-01 20:26:44 [config.py:2968] Downcasting torch.float32 to torch.float16. INFO 06-01 20:26:49 [config.py:717] This model supports multiple tasks: {'classify', 'generate', 'score', 'embed', 'reward'}. Defaulting to 'generate'. INFO 06-01 20:26:49 [config.py:1770] Defaulting to use mp for distributed inference INFO 06-01 20:26:49 [config.py:2003] Chunked prefill is enabled with max_num_batched_tokens=8192. Traceback (most recent call last): File "/usr/local/bin/vllm", line 10, in <module> sys.exit(main()) ^^^^^^ File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/main.py", line 53, in main args.dispatch_function(args) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/serve.py", line 27, in cmd uvloop.run(run_server(args)) File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 109, in run return __asyncio.run( ^^^^^^^^^^^^^^ File "/usr/lib/python3.12/asyncio/runners.py", line 195, in run return runner.run(main) ^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run return self._loop.run_until_complete(task) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 61, in wrapper return await main ^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1078, in run_server async with build_async_engine_client(args) as engine_client: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__ return await anext(self.gen) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 146, in build_async_engine_client async with build_async_engine_client_from_engine_args( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__ return await anext(self.gen) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 178, in build_async_engine_client_from_engine_args async_llm = AsyncLLM.from_vllm_config( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 150, in from_vllm_config return cls( ^^^^ File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 97, in __init__ self.tokenizer = init_tokenizer_from_configs( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/vllm/transformers_utils/tokenizer_group.py", line 101, in init_tokenizer_from_configs return TokenizerGroup( ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/vllm/transformers_utils/tokenizer_group.py", line 23, in __init__ self.tokenizer = get_tokenizer(self.tokenizer_id, **tokenizer_config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/vllm/transformers_utils/tokenizer.py", line 221, in get_tokenizer tokenizer = AutoTokenizer.from_pretrained( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/transformers/models/auto/tokenization_auto.py", line 1009, in from_pretrained return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/transformers/tokenization_utils_base.py", line 2062, in from_pretrained return cls._from_pretrained( ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/transformers/tokenization_utils_base.py", line 2302, in _from_pretrained tokenizer = cls(*init_inputs, **init_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/transformers/models/qwen2/tokenization_qwen2_fast.py", line 120, in __init__ super().__init__( File "/usr/local/lib/python3.12/dist-packages/transformers/tokenization_utils_fast.py", line 178, in __init__ super().__init__(**kwargs) File "/usr/local/lib/python3.12/dist-packages/transformers/tokenization_utils_base.py", line 1452, in __init__ self._set_model_specific_special_tokens(special_tokens=self.extra_special_tokens) File "/usr/local/lib/python3.12/dist-packages/transformers/tokenization_utils_base.py", line 1190, in _set_model_specific_special_tokens self.SPECIAL_TOKENS_ATTRIBUTES = self.SPECIAL_TOKENS_ATTRIBUTES + list(special_tokens.keys()) ^^^^^^^^^^^^^^^^^^^ AttributeError: 'list' object has no attribute 'keys'

• 上述报错原因是tokenizer_config.json文件中extra_special_tokens字段类型不被VLLM识别失败

• tokenizer_config.json文件在做微调时，输出的Lora文件中就已经出问题了，没有参考基础模型的tokenizer_config.json。但llama factory那边的checkpoint里面基本上与基础模型保持一致的

• automodel微调后产生的权重文件哪些没有问题，但tokenizer文件是有问题的，没有参考基础模型里面的tokenizer。问题其实不是合并脚本的问题，是微调后tokenizer就出问题

英伟达工程师给的临时解决方案

使用如下新的merge_lora.py脚本

# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Merge a LoRA/QLoRA adapter into a base HuggingFace model and save the result. Supports both dense and Mixture-of-Experts (MoE) models. For QLoRA adapters the base model is loaded in 4-bit, dequantized, and only then merged so that the adapter delta is applied to the correct weight representation (avoids the "naive merge" quality degradation described in https://kaitchup.substack.com/p/lora-adapters-when-a-naive-merge). Usage examples -------------- Standard LoRA merge (dense):: python tools/merge_lora.py \ --base-model meta-llama/Llama-3.2-1B \ --adapter-path checkpoints/adapter/ \ --output-dir merged_model/ QLoRA merge (dequantize then merge):: python tools/merge_lora.py \ --base-model meta-llama/Llama-3.2-1B \ --adapter-path checkpoints/adapter/ \ --output-dir merged_model/ \ --qlora MoE model merge:: python tools/merge_lora.py \ --base-model deepseek-ai/DeepSeek-V2-Lite \ --adapter-path checkpoints/adapter/ \ --output-dir merged_model/ Embedding / non-CausalLM model merge (auto-detected from adapter task_type):: python tools/merge_lora.py \ --base-model BAAI/bge-base-en-v1.5 \ --adapter-path checkpoints/adapter/ \ --output-dir merged_model/ Explicit model class override:: python tools/merge_lora.py \ --base-model BAAI/bge-base-en-v1.5 \ --adapter-path checkpoints/adapter/ \ --output-dir merged_model/ \ --model-class AutoModel """ import argparse import copy import gc import json import logging import os import torch logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s") logger = logging.getLogger(__name__) TASK_TYPE_TO_AUTO_CLASS = { "CAUSAL_LM": "AutoModelForCausalLM", "SEQ_CLS": "AutoModelForSequenceClassification", "SEQ_2_SEQ_LM": "AutoModelForSeq2SeqLM", "TOKEN_CLS": "AutoModelForTokenClassification", "QUESTION_ANS": "AutoModelForQuestionAnswering", "FEATURE_EXTRACTION": "AutoModel", } def _resolve_auto_cls(adapter_path: str, model_class: str | None = None): """Return the ``transformers.AutoModel*`` class to use for loading. Resolution order: 1. Explicit *model_class* string (e.g. ``"AutoModel"``). 2. ``task_type`` field in the adapter's ``adapter_config.json``. 3. Fall back to ``AutoModelForCausalLM``. """ import transformers if model_class is not None: cls = getattr(transformers, model_class, None) if cls is None: raise ValueError( f"Unknown model class '{model_class}'. " f"Must be an attribute of the `transformers` package " f"(e.g. AutoModelForCausalLM, AutoModel)." ) return cls config_path = os.path.join(adapter_path, "adapter_config.json") if os.path.isfile(config_path): with open(config_path, "r") as f: adapter_cfg = json.load(f) task_type = adapter_cfg.get("task_type") if task_type and task_type in TASK_TYPE_TO_AUTO_CLASS: cls_name = TASK_TYPE_TO_AUTO_CLASS[task_type] logger.info("Detected task_type=%s → using %s", task_type, cls_name) return getattr(transformers, cls_name) if task_type: logger.warning( "Unrecognised task_type '%s' in adapter config; falling back to AutoModelForCausalLM.", task_type, ) return transformers.AutoModelForCausalLM def dequantize_model(model, dtype=torch.float16, device="cpu"): """Replace every ``bitsandbytes.nn.Linear4bit`` with a plain ``nn.Linear``. The 4-bit weights are dequantized using the stored ``quant_state`` so that the full-precision delta from the LoRA adapter is applied correctly. Args: model: A HuggingFace model loaded with 4-bit quantization. dtype: Target dtype for the dequantized weights. device: Device to place the new linear layers on. Returns: The same model object, modified in-place. """ import bitsandbytes as bnb from bitsandbytes.functional import dequantize_4bit with torch.no_grad(): for name, module in model.named_modules(): if isinstance(module, bnb.nn.Linear4bit): logger.info("Dequantizing %s", name) quant_state = copy.deepcopy(module.weight.quant_state) quant_state[2] = dtype weights = dequantize_4bit(module.weight.data, quant_state=quant_state, quant_type="nf4").to(dtype) new_module = torch.nn.Linear( module.in_features, module.out_features, bias=module.bias is not None, dtype=dtype ) new_module.weight = torch.nn.Parameter(weights) if module.bias is not None: new_module.bias = torch.nn.Parameter(module.bias.data.to(dtype)) new_module.to(device=device, dtype=dtype) # Walk to the parent and swap the child module. parts = name.rsplit(".", 1) if len(parts) == 2: parent = model.get_submodule(parts[0]) setattr(parent, parts[1], new_module) else: setattr(model, name, new_module) model.is_loaded_in_4bit = False return model _V5_RUNTIME_TOKENIZER_KEYS = frozenset({"backend", "is_local"}) _SPECIAL_TOKENS_LIST_KEYS = ("additional_special_tokens", "extra_special_tokens") def _load_json_file(path: str) -> dict | None: """Load a JSON file, returning ``None`` when missing or unreadable.""" if not os.path.isfile(path): return None try: with open(path, "r") as f: return json.load(f) except (json.JSONDecodeError, OSError) as e: logger.warning("Could not read %s: %s", path, e) return None def _merge_special_token_lists(base_config: dict, adapter_config: dict) -> list | None: """Merge special-token lists from both configs, preserving order.""" merged: list = [] seen: set = set() for key in _SPECIAL_TOKENS_LIST_KEYS: for cfg in (base_config, adapter_config): tokens = cfg.get(key) if not tokens: continue for token in tokens: if token not in seen: seen.add(token) merged.append(token) return merged or None def _merge_tokenizer_configs(base_config: dict | None, adapter_config: dict | None) -> dict | None: """Merge base-model and adapter ``tokenizer_config.json`` contents. The base config is used as the foundation so fields such as ``added_tokens_decoder`` and ``chat_template`` are preserved. Adapter values override the base for keys explicitly present in the adapter config. Transformers v5 runtime-only keys (``backend``, ``is_local``) are dropped. """ if base_config is None and adapter_config is None: return None if base_config is None: merged = dict(adapter_config) elif adapter_config is None: merged = dict(base_config) else: merged = dict(base_config) for key, value in adapter_config.items(): if key in _V5_RUNTIME_TOKENIZER_KEYS: continue if key == "added_tokens_decoder": decoder = dict(base_config.get("added_tokens_decoder") or {}) decoder.update(value or {}) merged["added_tokens_decoder"] = decoder elif key in _SPECIAL_TOKENS_LIST_KEYS: continue else: merged[key] = value special_tokens = _merge_special_token_lists(base_config, adapter_config) if special_tokens is not None: merged["additional_special_tokens"] = special_tokens for key in _V5_RUNTIME_TOKENIZER_KEYS: merged.pop(key, None) extra_tokens = merged.pop("extra_special_tokens", None) if extra_tokens and "additional_special_tokens" not in merged: merged["additional_special_tokens"] = extra_tokens return merged def _apply_merged_tokenizer_config(base_model: str, adapter_path: str, output_dir: str) -> None: """Write a merged ``tokenizer_config.json`` into *output_dir*.""" base_config = _load_json_file(os.path.join(base_model, "tokenizer_config.json")) adapter_config = _load_json_file(os.path.join(adapter_path, "tokenizer_config.json")) saved_config = _load_json_file(os.path.join(output_dir, "tokenizer_config.json")) merged = _merge_tokenizer_configs(base_config, adapter_config) if merged is None: if saved_config is None: logger.warning("No tokenizer_config.json found in base model, adapter, or output directory") return if saved_config and "tokenizer_class" not in merged and "tokenizer_class" in saved_config: merged["tokenizer_class"] = saved_config["tokenizer_class"] config_path = os.path.join(output_dir, "tokenizer_config.json") with open(config_path, "w") as f: json.dump(merged, f, indent=2, ensure_ascii=False) f.write("\n") logger.info("Merged tokenizer_config.json saved to %s", config_path) def _clean_quantization_config(output_dir): """Remove ``quantization_config`` from the saved ``config.json``.""" config_path = os.path.join(output_dir, "config.json") if not os.path.exists(config_path): return with open(config_path, "r") as f: config_data = json.load(f) changed = False for key in ("quantization_config", "pretraining_tp"): if key in config_data: config_data.pop(key) changed = True if changed: with open(config_path, "w") as f: json.dump(config_data, f, indent=2) def merge_lora( base_model: str, adapter_path: str, output_dir: str, *, qlora: bool = False, dtype: str = "float16", device: str = "auto", save_tokenizer: bool = True, trust_remote_code: bool = False, model_class: str | None = None, ): """Load a base model, apply a LoRA adapter, merge, and save. Args: base_model: HuggingFace model name or path. adapter_path: Path to the PEFT adapter directory. output_dir: Where to write the merged model. qlora: If True, load the base model in 4-bit and dequantize before merging. dtype: Weight dtype for the merged model (``float16``, ``bfloat16``, ``float32``). device: ``auto``, ``cpu``, or ``cuda:N``. save_tokenizer: Whether to save the tokenizer alongside the model. trust_remote_code: Passed through to ``from_pretrained``. model_class: Explicit ``transformers`` Auto class name (e.g. ``"AutoModel"``, ``"AutoModelForCausalLM"``). When ``None`` the class is inferred from the adapter's ``task_type``. """ from peft import PeftModel from transformers import AutoTokenizer auto_cls = _resolve_auto_cls(adapter_path, model_class) torch_dtype = getattr(torch, dtype) # --- Load base model --- load_kwargs = dict( trust_remote_code=trust_remote_code, ) if qlora: from transformers import BitsAndBytesConfig bnb_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch_dtype, bnb_4bit_use_double_quant=True, ) load_kwargs["quantization_config"] = bnb_config load_kwargs["device_map"] = {"": 0} logger.info("Loading base model in 4-bit for QLoRA merge (%s): %s", auto_cls.__name__, base_model) else: load_kwargs["torch_dtype"] = torch_dtype load_kwargs["device_map"] = device logger.info("Loading base model (%s): %s", auto_cls.__name__, base_model) model = auto_cls.from_pretrained(base_model, **load_kwargs) # --- Dequantize if QLoRA --- if qlora: logger.info("Dequantizing model weights from 4-bit to %s", dtype) model = dequantize_model(model, dtype=torch_dtype, device="cuda") # --- Apply adapter and merge --- logger.info("Loading adapter from %s", adapter_path) model = PeftModel.from_pretrained(model, adapter_path) logger.info("Merging adapter into base model") model = model.merge_and_unload() # --- Save --- os.makedirs(output_dir, exist_ok=True) logger.info("Saving merged model to %s", output_dir) model.save_pretrained(output_dir, safe_serialization=True) if qlora: _clean_quantization_config(output_dir) if save_tokenizer: try: tokenizer_source = base_model if os.path.isfile(os.path.join(adapter_path, "tokenizer_config.json")): tokenizer_source = adapter_path tokenizer = AutoTokenizer.from_pretrained(tokenizer_source, trust_remote_code=trust_remote_code) tokenizer.save_pretrained(output_dir) _apply_merged_tokenizer_config(base_model, adapter_path, output_dir) logger.info("Tokenizer saved to %s", output_dir) except Exception as e: logger.warning("Could not save tokenizer: %s", e) logger.info("Merge complete.") # Free memory del model torch.cuda.empty_cache() gc.collect() def parse_args() -> argparse.Namespace: parser = argparse.ArgumentParser(description="Merge a LoRA / QLoRA adapter into a HuggingFace base model.") parser.add_argument( "--base-model", "-m", required=True, help="HuggingFace model name/path (e.g. meta-llama/Llama-3.2-1B or /path/to/model).", ) parser.add_argument( "--adapter-path", "-a", required=True, help="Path to the PEFT adapter directory (must contain adapter_config.json).", ) parser.add_argument( "--output-dir", "-o", required=True, help="Directory to save the merged model.", ) parser.add_argument( "--qlora", action="store_true", default=False, help="Load base model in 4-bit and dequantize before merging (for QLoRA adapters).", ) parser.add_argument( "--dtype", choices=["float16", "bfloat16", "float32"], default="float16", help="Weight dtype for the merged model (default: float16).", ) parser.add_argument( "--device", default="auto", help="Device map for model loading (default: auto). Use 'cpu' for CPU-only.", ) parser.add_argument( "--no-save-tokenizer", action="store_true", default=False, help="Skip saving the tokenizer.", ) parser.add_argument( "--trust-remote-code", action="store_true", default=False, help="Trust remote code when loading the model.", ) parser.add_argument( "--model-class", default=None, help=( "Explicit transformers Auto class name (e.g. AutoModel, AutoModelForCausalLM). " "When omitted, inferred from the adapter's task_type in adapter_config.json." ), ) return parser.parse_args() def main(): args = parse_args() merge_lora( base_model=args.base_model, adapter_path=args.adapter_path, output_dir=args.output_dir, qlora=args.qlora, dtype=args.dtype, device=args.device, save_tokenizer=not args.no_save_tokenizer, trust_remote_code=args.trust_remote_code, model_class=args.model_class, ) if __name__ == "__main__": main()