LLM SYSTEMS

DPO Preference Packing: Dense Masks and Sparse Execution

Ilho Ahn

Jun 18, 2026

A DPO systems note separating shared-prompt layout requirements from dense and sparse execution, with recorded Qwen3-8B training-cost comparisons.

Figure 1. Recorded sparse/dense packed cost ratios for the 200-step check and response-long repeats. The right panel shows aggregate held-out differences from the 200-step check; its separate metric scales and near-zero values do not establish log-probability or per-pair agreement. Open full-size figure.

Rafailov et al.’s Direct Preference Optimization (DPO) trains on a preferred and a rejected response to the same prompt [3]. A conventional pairwise layout processes that prompt twice. Sharing it removes duplicate tokens. Correctness requires keeping the responses independent; the resulting cost also depends on how the attention backend executes the mask.

This note examines response-context preservation and execution cost in a trainer built with the TRL library and LoRA adapters. The main recorded comparison used Qwen3-8B on two GB10 nodes. Its 200-step sparse run used less step time and allocated memory than the dense packed run, while the reported held-out metric differences were small. The numerical validation records needed to establish implementation equivalence are no longer available for this revision.

Summary

• Three paths separate the comparison: vanilla pairwise inputs, a shared prompt with dense masked attention, and the same packed layout with sparse block skipping.
• In the 200-step Qwen3-8B + LoRA comparison, sparse/dense ratios were 0.8066 for median step time and 0.6490 for rank-summed CUDA allocated memory: reductions of 19.34% and 35.10% in those recorded measures.
• Held-out sparse-minus-dense differences were -0.0000118 in DPO loss and +0.00195 in reward accuracy. Small aggregate differences do not establish token-level or per-pair agreement.
• Short checks and response-long repeats also recorded lower cost. Reward-accuracy differences had opposite signs across the two UltraFeedback variants, so the evidence does not support a general quality improvement.
• The scope is a finite-run implementation study. Original logs, complete configurations, and numerical parity-test outputs are unavailable; the retained summaries do not establish downstream quality or long-run convergence.

Main resources

Prefix Sharing for DPO arXiv:2410.20305 Preference Packing arXiv:2602.24082 Qwen3-8B Qwen/Qwen3-8B H4 UltraFeedback HuggingFaceH4/ultrafeedback_binarized

Experimental Setup

Prior work and implementation scope

Wang and Hegde’s prefix-sharing work already combined a shared prompt with branch-aware masking and FlexAttention block skipping. Cho’s Preference Packing likewise studied a shared-prompt layout for preference optimization [1] [2]. The implementation work recorded here applied these ideas to a TRL trainer and collator, then compared dense and sparse execution under LoRA and distributed training.

LoRA trains low-rank adapters while retaining the base weights; FSDP (Fully Sharded Data Parallel) distributes training state across devices. Table 1 lists the retained conditions for the main Qwen3-8B checks [4] [5].

Component	Recorded condition
Model and adaptation	Qwen/Qwen3-8B, LoRA, bf16
Distributed setting	Two GB10 nodes, FSDP, one GPU per node; 100GB CUDA memory cap per node
Reference policy	Base model with adapters temporarily disabled through model.disable_adapter()
200-step data	H4 UltraFeedback binarized, 512/512 train/eval slice
Measured outcomes	Step time, CUDA allocated memory, held-out DPO loss, reward margin, and reward accuracy

Table 1. Retained conditions for the main distributed comparisons. These are a partial experiment specification; missing settings are listed under Limitations.

Layout requirements

Figure 2 separates input layout from backend execution. In the packed sequence, both responses use the same prompt, and each response can attend only to that prompt and its own preceding tokens.

Figure 2. Layout-level comparison of the three paths. Vanilla pairwise duplicates prompt-side work, dense packed uses the packed layout with dense masked attention, and the sparse-backed path passes the branch structure to sparse block skipping.

prompt + chosen_branch + rejected_branch

The implementation account identifies three details needed to preserve the ordinary pairwise computation:

• Branch-aware masking: prevent either response from seeing the other response’s tokens.
• Aligned positions: give response tokens the same position_ids they would have in their separate prompt + response sequence.
• Explicit log-prob gathering: select the logits that predict each response token, including the branch start.

The last detail matters because the first token of either response is predicted from the final prompt logits. A simple one-position shift over the packed sequence would instead predict the rejected branch’s first token from the end of the chosen branch.

These are the layout requirements described by the implementation. The prior note reported successful log-prob checks, but retained neither their maximum error and tolerance nor the tested cases or gradient comparisons. The available record therefore supports the implementation rationale, without independently establishing numerical equivalence.

Dense and sparse execution

The dense masked path can compute scores for positions it later masks:

scores = q @ k.T
scores = scores.masked_fill(...)

A simple score-matrix size proxy makes the length trade-off visible. For a prompt of length $p$ and two responses of equal length $r$:

\[C_{\mathrm{pairwise}} = 2(p+r)^2\] \[C_{\mathrm{packed,dense}} = (p+2r)^2.\]

At $p = 1024$ and $r = 1024$, the packed/pairwise proxy ratio is 1.125. This is a dense attention-size illustration, not a total training-time prediction: it omits projections, feed-forward layers, communication, and backend-specific optimizations. Sharing the prompt can reduce those other token-dependent costs even when the dense attention proxy grows.

FlexAttention can use a block mask to skip fully masked blocks [6]. In the sparse packed path, that includes blocks crossing between response branches. This is the execution distinction examined in the recorded comparisons.

Reading the measurements

Time and memory ratios divide the named numerator by its baseline; lower values mean lower recorded cost. All held-out deltas below are sparse minus dense. Reward accuracy tracks the fraction of pairs for which the chosen response has the higher implicit DPO reward; reward margin tracks that reward difference. They are preference-training diagnostics, not an external judge’s assessment of generated answers. The exact tie rule and aggregation settings are not retained.

Rank-summed memory was defined as the sum of per-rank torch.cuda.max_memory_allocated() peaks within each optimizer step, followed by the maximum of those step sums. Peaks on different ranks need not occur at the same instant. This measure excludes allocator reserved memory, CPU/UMA host allocation, model-load peaks, and other system processes; it is not total machine memory usage. The underlying peak-reset instrumentation cannot now be checked.

Results

Short mechanism checks

Table 2 compares the three execution paths in 5-step checks labeled prompt-long, balanced, and response-long. Those labels describe the relative prompt/response lengths; their exact token-length settings are not retained. The medians indicate lower cost for sparse packing, but do not recover the individual regime results or timing variability.

Comparison	Median step-time ratio	Rank-summed memory ratio
sparse packed / dense packed	0.7528	0.6701
sparse packed / vanilla pairwise	0.6395	0.6046
dense packed / vanilla pairwise	0.8495	0.9022

Table 2. Recorded 5-step mechanism checks. Each ratio divides the first layout by the second; lower is better. Values are medians across the three length regimes, so they summarize those checks rather than establish an improvement in every regime.

The response-long repeats extended the sparse/dense comparison to 20 steps per run, using 64/192 train/eval slices and seeds 17 and 23. The recorded median step-time ratio was 0.7647, and the rank-summed memory ratio was 0.5968. Mean reward-accuracy delta was +0.00260, with range [0.0, +0.00521]. These two seeds support a repeat check in that setting; they are not repeats of the separate 200-step experiment.

UltraFeedback comparisons

Table 3 separates the two UltraFeedback variants rather than relying only on their aggregate [7] [8]. The cost ratios were below one for both, while reward accuracy moved in opposite directions. These are two variants from the same dataset family, not two independent task families.

Dataset	Step-time ratio	Rank-summed memory ratio	Reward accuracy delta
Argilla UltraFeedback	0.8097	0.6521	+0.054688
H4 UltraFeedback	0.8005	0.6719	-0.015625
Recorded aggregate median ratios; mean delta	0.8051	0.6620	+0.019531

Table 3. UltraFeedback comparisons labeled as 20-step runs in Figure 3. Cost ratios are sparse/dense; accuracy deltas are sparse minus dense. The positive mean accuracy delta does not describe both datasets.

Average response lengths were roughly 338-416 tokens for training and 364-405 for evaluation. Exact slice sizes for these transfer checks are not retained in the text.

Figure 3 collects the recorded distributed cost ratios. Its rows summarize different checks and should not be treated as a common set of repeated measurements. The additional larger-eval row has incomplete condition metadata, preserved separately in the Appendix.

Figure 3. Recorded Qwen3-8B + LoRA 2-node FSDP cost comparisons. Ratios are sparse packed / dense packed, so shorter bars are more efficient; blue bars show step time and gray bars show rank-summed allocated memory. Rows summarize different checks.

The 200-step comparison

The longest recorded comparison used the H4 UltraFeedback 512/512 train/eval slice from Table 1. Table 4 separates execution cost from the held-out metric differences summarized in Figure 1. Cost values are sparse/dense ratios; held-out values are sparse-minus-dense deltas.

Measure	Value
Median step time	0.8066
Rank-summed allocated memory	0.6490
Held-out DPO loss	-0.0000118
Held-out reward accuracy	+0.00195
Held-out reward margin	+0.0000236

Table 4. Recorded 200-step comparison. The first two rows are sparse/dense cost ratios; the remaining rows are sparse-minus-dense held-out deltas. No uncertainty interval or numerical equivalence threshold is available.

The time and rank-summed memory ratios correspond to 19.34% and 35.10% reductions relative to dense packed training. They describe the reported step and allocation measures; without startup/compile timings they do not establish the reduction in end-to-end job time.

The held-out deltas were small at this endpoint. Aggregate accuracy can remain close even if individual pair decisions change, and close loss values do not establish matching token log-probs or gradients. The result supports a recorded cost reduction with similar aggregate diagnostics in this finite run. The separately recorded, ambiguously labeled “final loss” values are retained in the Appendix and are not used to reconstruct held-out deltas.

Limitations

• Incomplete measurement records. Absolute seconds per step and peak GB, timing variability, compile/warmup treatment, CUDA synchronization, and memory-reset details cannot be recovered from the retained summaries. This particularly limits interpretation of the 5-step checks.
• Incomplete training specification. Effective pair batch size, accumulation, learning rate, DPO temperature, LoRA configuration, exact length/truncation settings, software/model/data revisions, and the seed/repeat count for the 200-step comparison are not retained. Only the response-long repeat explicitly records two seeds.
• Restricted baseline and quality claims. The reported gains are within this implementation’s comparisons. The actual vanilla/dense backend and matched optimization settings cannot be audited, so the results do not establish an advantage over an optimized SDPA or FlashAttention baseline. No human win-rate, generated-answer quality, or convergence beyond 200 steps was tested.
• Restricted implementation coverage. The recorded reference path disables LoRA adapters. Full fine-tuning with a separate frozen reference model is outside this study. Hardware and data coverage remain limited to the reported GB10 setting and Intel Orca-style / UltraFeedback-family data.

Appendix: Incompletely specified records

Final loss. The earlier note separately recorded dense 0.69298 and sparse 0.69336 as “final loss.” Their difference is +0.00038, whereas the explicitly labeled held-out DPO loss delta is -0.0000118. The final-loss evaluation split and aggregation are unknown. They may refer to a different statistic; the record does not justify relabeling them as training loss or treating them as the source of the held-out delta.

Max-rank memory. An additional sparse/dense memory ratio of 0.6046 was labeled “max-rank.” Its exact aggregation across ranks and optimizer steps was not retained. The main comparison uses the rank-summed measure defined above.

Larger-eval row. Figure 3 preserves an additional row labeled “Larger eval robustness,” with 512 / 256 / 256 eval, step-time ratio 0.820, and memory ratio 0.688. The dataset-to-count mapping and aggregation cannot be recovered. These displayed, rounded ratios remain in the figure as a retained record, without using that row to claim broader evaluation coverage.

Experiment Resources

• Available evidence: numerical summaries and figures retained in this note. Original logs, full configurations, and numerical parity-test outputs were unavailable for the September 6, 2026 revision; no new training or evaluation was run.
• Implementation availability: the previously linked repository returned HTTP 404 on unauthenticated access on September 6, 2026. It is not currently an accessible reproduction artifact.

References

1. Wang, Franklin and Hegde, Sumanth. Accelerating Direct Preference Optimization with Prefix Sharing. arXiv:2410.20305, 2024. arXiv
2. Cho, Jaekyung. Preference Packing: Efficient Preference Optimization for Large Language Models. arXiv:2602.24082, 2026. arXiv
3. Rafailov, Rafael et al. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arXiv:2305.18290, 2023. arXiv
4. Qwen Team. Qwen3-8B. Hugging Face model repository. Model card
5. Hu, Edward J. et al. LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685, 2021. arXiv
6. PyTorch. FlexAttention. PyTorch documentation. Docs
7. Argilla. UltraFeedback Binarized Preferences Cleaned. Hugging Face dataset repository. Dataset card
8. Hugging Face H4. UltraFeedback Binarized. Hugging Face dataset repository. Dataset card

Citation

Text citation:

Ilho Ahn, "DPO Preference Packing: Dense Masks and Sparse Execution", Mini Research, June 18, 2026.

BibTeX:

@article{ahn2026preferencepackingdpo,
  author = {Ilho Ahn},
  title = {DPO Preference Packing: Dense Masks and Sparse Execution},
  journal = {Mini Research},
  year = {2026},
  month = jun,
  url = {https://muted-color.github.io/research/2026/06/18/preference-packing-dpo-efficiency/}
}