LLM EVAL

Reporting Fixed-Budget Tool-Use DPO as Recipe-Checkpoint Profiles

Ilho Ahn

A fixed-budget tool-use DPO comparison is better reported as a recipe-checkpoint profile than as a single recipe winner.

Tool-use DPO results are often compressed into a single improvement over an SFT baseline. That summary was too thin in this experiment. Function-call structure, call-decision behavior, and instruction-following guardrails moved in different directions.

This note compares DPO negative recipes under the same pair budget and optimizer-step budget, using a Qwen3-8B tool-use SFT checkpoint as the shared reference [18]. The main result is not a recipe ranking. Tool-use DPO should be reported as a recipe-checkpoint profile. A profile records the negative recipe, fixed budget, reported checkpoint, intended metric, guardrail regression, seed scope, pair-sampling scope, benchmark coverage, data-quality gate, and overlap checks together.

Tool-use DPO is a preference-optimization setup where chosen/rejected pairs are built from tool-call outputs or tool-use decisions, and DPO is used to update the policy [13]. This note does not propose a new objective. It asks which evaluation axes move under fixed-budget negative recipes and checkpoints.

Key evaluation resources

Qwen3-8B Qwen/Qwen3-8B When2Call nvidia/When2Call

Summary

  • The main reporting unit is a recipe-checkpoint profile, not a single recipe winner.
  • The comparison used a fixed budget of 3000 pairs, 375 optimizer steps, DPO beta 0.1, LR 5e-6, LoRA rank 16, and effective batch size 8.
  • At the preferred step50 checkpoint, clean structural DPO was +3.33 points above clean behavior DPO on BFCL core, but -6.67 points lower on When2Call behavior accuracy and -5.31 points lower on When2Call macro F1.
  • Clean-vs-unfiltered effects within the same source family were small or uncertain. In this setting, semantic filtering behaved more like a pair-quality gate than a downstream performance axis.
  • The final checkpoint was not automatically the best reporting point: in the reported clean conditions, it worsened the IFEval guardrail trade-off relative to step50.
  • Additional seeds and an independent pool-B replicate preserved the source-axis sign pattern, but the robustness claim remains narrow.

Repository and Paper

GitHub repository

Artifact release muted-color/tool-use-dpo-fixed-budget-report

Paper

Paper PDF paper.pdf

This section is only for the GitHub repository and paper PDF. The result tables in this note and Figure 1 can be checked against the public artifact repository, using sanitized evaluation outputs and aggregate tables. The repository is an artifact-level verification release for the reported tables, confidence intervals, and Pareto figure. It is not an end-to-end retraining bundle.

For the full paper-style report and appendix, see the separate paper PDF. The result snapshot is pinned by the arxiv-v1 tag, which currently points to commit 3f4799e. For a quick check, start with reproduced/tables/ and reproduced/figures/ in the repository.

Evaluation Design

Table 1 fixes the unit of interpretation for this note. A fixed budget means the same number of preference pairs and optimizer steps. It does not mean identical source distributions or identical loss-bearing token counts.

Profile field Value in this note Interpretive role
Negative recipe clean structural, clean behavior,
same-budget unfiltered controls		 Separates source-axis movement instead of ranking one recipe as the winner.
Fixed budget 3000 pairs, 375 steps,
beta 0.1, LR 5e-6, LoRA rank 16		 Fixes pair, step, and DPO recipe settings without making an equal-token causal claim.
Reference Shared Qwen3-8B tool-use SFT checkpoint Removes variation from changing the DPO reference across recipes.
Intended metric BFCL core, When2Call behavior accuracy,
When2Call macro F1		 Separates function-call structure from call-decision behavior.
Guardrail metric IFEval prompt-level strict accuracy Reports intended gain together with instruction-following regression.
Table 1. Core fields of the reporting profile used in this note. The paper and repository manifest also include seed scope, pair-sampling scope, data-quality gates, benchmark coverage, and overlap checks in the same profile.

The evaluation is read across three axes. BFCL core is mainly a structural function-calling axis, focused on function selection and argument correctness [11]. When2Call evaluates call-decision behavior such as tool call, follow-up question, and unable-to-answer decisions [15]. IFEval prompt-strict is not the primary intended metric; it is a guardrail metric for measuring how much DPO perturbs instruction following [20].

Results

The results are easiest to read in four groups. First, the source-axis split between BFCL and When2Call. Second, whether clean filtering itself explains downstream performance. Third, how checkpoint selection changes the guardrail trade-off. Fourth, how far the seed and pair-pool robustness checks can support the claim.

Source-Axis Comparison

Table 2 compares the structural and behavior recipes at the preferred step50 checkpoint. Values are percentage-point deltas computed as structural - behavior. A positive BFCL row means the structural recipe scored higher; a negative When2Call row means the behavior recipe scored higher.

Comparison Metric Delta CI95 low CI95 high How to read it
clean structural
minus clean behavior		 BFCL core +3.33 +1.53 +5.67 Structural recipe moved BFCL more.
clean structural
minus clean behavior		 W2C behavior acc. -6.67 -10.75 -3.02 Behavior recipe moved call-decision accuracy more.
clean structural
minus clean behavior		 W2C macro F1 -5.31 -8.85 -2.34 Behavior recipe moved macro F1 more.
unfiltered structural
minus unfiltered behavior		 BFCL core +2.67 +1.02 +4.86 The same direction appears in the unfiltered control.
unfiltered structural
minus unfiltered behavior		 W2C macro F1 -4.21 -7.48 -1.23 The When2Call signal remains stronger for behavior negatives.
Table 2. Source-axis deltas at the preferred step50 checkpoint, computed with grouped bootstrap. This is a source-native fixed-budget recipe comparison, not a prompt-matched causal ranking of source quality.

The stable observation is simple. Structural negatives moved BFCL more, while behavior negatives moved When2Call more. This does not mean that the structural source is better, or that the behavior source is better. It means the two recipes fill different evaluation axes.

There is an important caveat. The behavior condition is tied to a When2Call-style data and evaluation family, and BFCL core also contains relevance/irrelevance-style buckets. Table 2 should therefore be read as a fixed-budget recipe profile under the represented decision regime, not as a source-superiority result.

Role of Clean Filtering

Semantic filtering was used as a pair-quality gate: whether the rejected output was actually worse than the chosen output, schema-valid, and not an equivalent alternative. But when clean and unfiltered variants are compared within the same source family, Table 3 shows small or uncertain downstream differences.

Comparison Metric Delta CI95 low CI95 high
clean structural
minus unfiltered structural		 BFCL core +0.33 0.00 +1.08
clean behavior
minus unfiltered behavior		 W2C macro F1 +0.55 -0.83 +1.81
Table 3. Same-budget clean-vs-unfiltered comparison. The clean gate reduced pair-level risk, but it did not appear to be a general downstream performance axis in this setting.

This is not a reason to drop filtering. Tool-use negatives easily mix optional/default/no-op differences, harmless normalization, acceptable alternate tools, and chosen/reference-suspect cases. The narrower point is that filtering should not be summarized as a performance improvement unless there is separate evidence for that claim. Here, quality control and downstream movement are reported separately.

Checkpoint and Guardrail Trade-Off

Figure 1 plots intended-axis gain against IFEval prompt-strict regression, while Table 4 gives the corresponding absolute scores. Even within the same recipe, moving from step50 to the final checkpoint changes both the intended metric and the guardrail metric.

Two-panel Pareto figure comparing BFCL core and When2Call macro F1 gains against IFEval prompt-strict regression
Figure 1. Pareto view of intended-axis gain and IFEval prompt-strict regression under a fixed-budget tool-use DPO setting. The left panel uses BFCL core as the intended metric; the right panel uses When2Call macro F1. Arrows show the movement from the step50 checkpoint to the final checkpoint within the same condition.

For the clean structural condition, step50 had BFCL core +4.67 points and IFEval prompt-strict -6.25 points. The final checkpoint changed to BFCL core +2.67 points and IFEval prompt-strict -11.46 points. In this condition, the final checkpoint reduced intended gain and increased guardrail regression.

For the clean behavior condition, step50 had When2Call macro F1 +4.86 points and IFEval prompt-strict -5.21 points. The final checkpoint had When2Call macro F1 +4.68 points and IFEval prompt-strict -7.29 points. The intended metric was nearly preserved, but the guardrail metric deteriorated.

Condition Checkpoint BFCL core W2C macro F1 IFEval prompt strict
SFT baseline SFT best 0.667 0.481 0.635
clean structural step50 0.713 0.477 0.573
clean structural final 0.693 0.479 0.521
clean behavior step50 0.680 0.530 0.583
clean behavior final 0.660 0.528 0.562
Table 4. Absolute scores for the baseline and clean conditions. Deltas alone can hide the starting point and the size of guardrail regression, so the reporting profile should include absolute scores as well.

This does not imply that early checkpoints are always better. It does show that automatically reporting the final checkpoint can hide the trade-off between intended-axis gain and guardrail regression. In this setting, checkpoint selection is part of the empirical claim.

Robustness and Coverage Scope

Additional checks were used to narrow the claim, not to turn the source-axis result into a general robustness claim. Table 5 separates what was checked from what it can support.

Check Evidence Boundary
Training-seed check With the same pair pool, seed2 and seed3 preserved the source-axis pattern. Including the original seed, 18/18 sign checks matched the expected direction. Supports fixed-pool training-seed stability, not broad data-distribution robustness.
Independent pool-B replicate Clean structural selected 3000 rows from 9988 candidates; clean behavior selected 3000 rows from 4294 candidates. Pair-id and content-hash overlap with pool-A were both 0. Across pool-A and pool-B step50/best checkpoints, 12/12 sign checks matched the expected direction. Supports sign stability under one independent pair-pool reconstruction, not a causal ranking of source quality.
When2Call coverage audit The audited public labeled configs contained 27,952 rows and 0 direct-answer gold rows. Limits the When2Call behavior claim to the represented tool-call, follow-up-question, and unable-to-answer slices.
Table 5. Robustness and coverage checks used to bound the claim. The checks support sign stability in the evaluated setup, but they do not establish broad distribution robustness or source-level superiority.

Interpretation

The shortest interpretation is that tool-use DPO was not a one-axis improvement problem. Structural negatives moved function-call correctness more. Behavior negatives moved the represented call-decision slice more. Clean filtering reduced pair-quality risk, but the evidence for a general same-budget downstream improvement was weak. Checkpoint selection changed both intended gain and guardrail regression.

The conclusion is therefore about reporting, not recipe selection. Public tool-use DPO results should state:

  • which negative recipe was used
  • the pair count, step count, DPO recipe, and reference checkpoint
  • why the reported checkpoint was selected
  • how the intended metric and guardrail metric moved
  • what seed and pair-sampling robustness was checked
  • what limitations remain from the clean gate and benchmark coverage
  • which exact overlap keys were checked, and what those checks do not guarantee

Without this profile, a +x points improvement is easy to over-interpret. In tool-use evaluation, formatting a tool call correctly and deciding whether a tool should be called are not the same capability.

Limitations

The DPO training pools are source-native rather than prompt-matched. The behavior condition is confounded with the When2Call-style data and evaluation family. The source-axis result is therefore a fixed-budget recipe comparison, not a causal ranking of source quality.

The fixed-budget condition is also limited. Pair count, optimizer steps, DPO recipe, and reference checkpoint were fixed, but token count and source distribution were not identical. All DPO conditions come from one Qwen3-8B SFT reference, one QLoRA DPO recipe, and one pair/step budget [3] [2].

The IFEval slice should be read as a guardrail diagnostic. Bootstrap intervals address evaluation-sample uncertainty, but they do not cover all uncertainty from training stochasticity, data sampling, or benchmark construction.

References

  1. Chowdhury, Sayak Ray, Kini, Anush, and Natarajan, Nagarajan. Provably Robust DPO: Aligning Language Models with Noisy Feedback. ICML, 2024. arXiv
  2. Dettmers, Tim et al. QLoRA: Efficient Finetuning of Quantized LLMs. NeurIPS, 2023. arXiv
  3. Hu, Edward J. et al. LoRA: Low-Rank Adaptation of Large Language Models. ICLR, 2022. arXiv
  4. Jung, Sunghee et al. DiaTool-DPO: Multi-Turn Direct Preference Optimization for Tool-Augmented Large Language Models. SIGDIAL, 2025. ACL Anthology
  5. Lee, Shinbok et al. FunctionChat-Bench: Comprehensive Evaluation of Language Models' Generative Capabilities in Korean Tool-use Dialogs. arXiv:2411.14054, 2024. arXiv
  6. Li, Minghao et al. API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs. EMNLP, 2023. DOI: 10.18653/v1/2023.emnlp-main.187
  7. Liang, Percy et al. Holistic Evaluation of Language Models. Transactions on Machine Learning Research, 2023.
  8. Liu, Weiwen et al. ToolACE: Winning the Points of LLM Function Calling. arXiv:2409.00920, 2024. arXiv
  9. Liu, Zuxin et al. APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets. NeurIPS, 2024.
  10. Pan, Yu et al. What Matters in Data for DPO? arXiv:2508.18312, 2025. arXiv
  11. Patil, Shishir G. et al. The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models. ICML, 2025. PMLR; Project page
  12. Qin, Yujia et al. ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs. ICLR, 2024. OpenReview
  13. Rafailov, Rafael et al. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS, 2023. arXiv
  14. Rafailov, Rafael et al. Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms. arXiv:2406.02900, 2024. arXiv
  15. Ross, Hayley, Mahabaleshwarkar, Ameya Sunil, and Suhara, Yoshi. When2Call: When (not) to Call Tools. NAACL, 2025. DOI: 10.18653/v1/2025.naacl-long.174; Dataset card
  16. Schick, Timo et al. Toolformer: Language Models Can Teach Themselves to Use Tools. NeurIPS, 2023.
  17. Yan, Fanjia. A Function Calling Perspective on Scalable Large Language Model Agent Evaluation. Master's thesis, UC Berkeley EECS, 2025. Technical report
  18. Yang, An et al. Qwen3 Technical Report. arXiv:2505.09388, 2025. arXiv; Qwen Team. Qwen3-8B. Hugging Face model repository. Model card
  19. Zhang, Jianguo et al. xLAM: A Family of Large Action Models to Empower AI Agent Systems. NAACL, 2025. arXiv
  20. Zhou, Jeffrey et al. Instruction-Following Evaluation for Large Language Models. arXiv:2311.07911, 2023. arXiv

Citation

Text citation:

Ilho Ahn, "Reporting Fixed-Budget Tool-Use DPO as Recipe-Checkpoint Profiles", Mini Research, June 27, 2026.

BibTeX:

@article{ahn2026toolusedporeportingprofile,
  author = {Ilho Ahn},
  title = {Reporting Fixed-Budget Tool-Use DPO as Recipe-Checkpoint Profiles},
  journal = {Mini Research},
  year = {2026},
  month = jun,
  url = {https://muted-color.github.io/research/2026/06/27/tool-use-dpo-fixed-budget-reporting-profile/}
}