LLM TOOL USE

DiaTool-DPO Reconstruction: Preference Accuracy and Tool-Use Behavior

Ilho Ahn

Jul 29, 2026

A DiaTool-DPO reconstruction connecting preference ranking, repeated training responses, and changes in missing-field handling and complete-call success.

Figure 1. Local sensitivity comparison: prefix-synchronized DPO minus primary DPO, in percentage points. Missing-field handling (Slot) improved, while Call and Completion declined and auxiliary rollout exact success stayed at 15/28. This single-run comparison changed prefixes, later content, and lengths together. Open full-size figure.

A tool-using assistant must decide when it has enough information to act. For example, a booking request without a date requires a follow-up question before a tool call. In FunctionChat-Bench, this missing-field behavior is evaluated on the Slot axis. Learning to rank a supplied dialogue correctly and generating that follow-up question are different capabilities.

This reconstruction of Jung et al.’s DiaTool-DPO examines how those capabilities align [1]. The primary model ranked held-out preference pairs with 97.86% accuracy and improved all four local benchmark axes over its SFT starting point. Under a later reconstruction, Slot scores were higher, while Call and Completion were lower and auxiliary complete-call success was unchanged. The useful observation is this uneven behavioral response across two constructions of the training data.

Summary

• Preference fit: chosen trajectories scored above rejected trajectories on 97.86% of 840 held-out pairs from 135 source groups, with no source overlap with training. This evaluates supplied dialogue pairs.
• Generated behavior: the primary DPO model improved local Slot performance from 54.07% to 61.09% (239→270/442), with 43 item-level improvements and 12 regressions. This is a separate evaluation of generated responses, not the same accuracy measure as preference ranking.
• Training-signal distribution: the most frequent rejected first response covered 49.13% of the inspected partial-call pairs, compared with 8.18% for the most frequent chosen response. The difference motivates inspecting what contrasts the data repeatedly presents; it does not prove shortcut learning.
• Local sensitivity: the condition with synchronized prefixes for 2,530 pairs scored 68.78% on Slot, versus 61.09% for primary DPO in one run. Call and Completion declined, and auxiliary rollout success remained 15/28 for both DPO variants (Figure 1).
• Scope: the alternative also changed later content, tokenization, and lengths. Each DPO condition was trained once, and original run artifacts are unavailable for this revision. The results support a local sensitivity finding, not an isolated prefix effect or a generally better reconstruction rule.

Experimental Setup

Dialogue paths and preference training

DiaTool-DPO trains on preferred and rejected trajectories: complete sequences of assistant, user, and tool turns. Jung et al. define three query types: call immediately when required arguments are known, ask for missing fields before calling, or respond without a tool call when the request is unsupported [1]. Figure 2 summarizes this structure. The three query types were originally described over five conceptual dialogue states; they are task definitions, not observed internal model states.

Figure 2. DiaTool-DPO pairs complete dialogue paths for immediate calls, questions about missing required fields, or responses without a tool call. The objective combines early-turn weighting, normalization by total turn weight, and a reward-gap margin. This is a conceptual method diagram; its geometry encodes no counts or effect sizes.

Each pair shares the initial request and tool context, but its later turns and lengths can differ. Rejected paths include premature calls, redundant questions, incorrect refusals, and unsupported calls. The objective scores assistant turns relative to the reference policy, gives earlier turns more weight, normalizes by total turn weight, and applies a chosen/rejected reward-gap margin. User and tool messages provide context.

The reconstruction followed the public method and reference implementation, retaining DPO’s policy-versus-reference comparison [2] [3]. It used separately reconstructed Korean preference data and a locally trained starting checkpoint. Appendix Table 1 records the reconstruction choices; the local comparisons below carry the main findings.

Local training and evaluation conditions

Supervised fine-tuning (SFT) provided the starting tool-use policy. Both DPO conditions continued the same local rank-8 LoRA adapter, which updates a small set of trainable parameters while retaining the base model. Table 1 summarizes the recorded settings. The public SFT artifact documents the starting model, not the later DPO outcomes.

Reconstructed SFT checkpoint

LLaMA 3 8B DiaTool Korean SFT LoRA soleaf/Llama-3-8B-DiaTool-Ko-SFT-LoRA

Field	Recorded condition	Role in this study
Starting model	Llama 3 8B Instruct [4] + reconstructed tool-use SFT LoRA	Shared local reference for SFT-versus-DPO comparison
Preference data	Easy 8,357 + Hard 8,437 = 16,794 pairs	Reported pair counts match the public specification
DPO training	1 epoch, total batch 8, LR 1e-7, bf16, max length 8,192	Settings recorded for the reconstruction
Parameter update	Continued rank-8 LoRA [5]	The paper does not separately specify whether PEFT was used
Evaluation set	FunctionChat-Bench, 1,306 items at the evaluated revision [6]	Call 670, Completion 71, Slot 442, Relevance 123

Table 1. Recorded local training and evaluation conditions. The two DPO conditions shared the reconstructed SFT starting point. The evaluated benchmark revision is specified; complete run and decoding configurations are not available for this revision.

FunctionChat-Bench separates four behaviors [6]:

• Call: produce the correct tool and arguments. The local implementation used normalized tool-call exact match.
• Completion: convey the tool’s returned result appropriately.
• Slot: request missing required information before a call.
• Relevance: respond appropriately when no tool supports the request or a tool call is unnecessary.

Completion, Slot, and Relevance used a rubric-based judge. These benchmark items evaluate an assistant response in a supplied context. The separate 28-case free-running rollout check lets earlier generated turns affect later context and evaluates complete-call success. Improvement on a fixed-context Slot item can therefore be meaningful without establishing improvement across an entire dialogue.

The evaluated repository revision contains 1,306 items across Singlecall, Dialog, and CallDecision. The original FunctionChat paper described 700 items [6], and DiaTool-DPO does not pin its evaluation commit or item count. This is another reason to treat the paper comparison in the Appendix as context rather than a matched reproduction.

Macro is the unweighted mean of the four axis percentages. Because their sample counts and scoring mechanisms differ, Macro summarizes the benchmark rather than a pooled pass rate. Main-result changes are reported in percentage points; the relative-gain calculation used for the paper comparison is given in the Appendix.

Identical-response judgment adjustments and the difference from the public SFT card’s score snapshot are documented in the Appendix.

Results

Preference fit and missing-field handling

The primary model’s held-out preference accuracy was 97.86% on a 5% source-group validation split: 840 pairs from 135 groups, with zero source overlap with training. This shows that the learned score ordering extended to held-out source groups under the reconstructed pair distribution. The responses in this check were already supplied to the model; it did not test whether the model would generate the desired dialogue path.

Table 2 shows the separate generation-based evaluation. All four local axes improved over SFT. On Slot, the net gain was 31 passing items: 43 previously failing items improved and 12 previously passing items regressed. The model therefore improved missing-field handling in this evaluation while retaining substantial room for improvement. The high preference-ranking accuracy and the Slot result answer different questions and should not be compared as a single accuracy gap.

Evaluation axis	SFT	Primary DPO	Change
Call n=670	63.88% (428)	68.81% (461)	+4.93 pp
Completion n=71	85.92% (61)	92.96% (66)	+7.04 pp
Slot n=442	54.07% (239)	61.09% (270)	+7.01 pp
Relevance n=123, locally adjusted	21.95% (27)	27.64% (34)	+5.69 pp
Macro	56.45%	62.62%	+6.17 pp

Table 2. Local SFT-to-DPO results. Parentheses give passing-item counts; changes are percentage points computed from counts before rounding. Relevance uses the identical-response judgment reconciliation described in the Appendix. Macro is an unweighted average across axes, not a pooled item accuracy.

Repeated contrasts in the primary preference data

The first data audit checked schema, state paths, loaders, duplicates, masks, and pair validity. The subsequent diagnostic examined whether the reconstructed data repeatedly contrasted a small set of response patterns.

The inspected subset contained 2,530 pairs for missing-field queries, labeled Hard Type2 partial-call in the original record. They were part of 7,832 Type2 pairs and derived from 562 source trajectories. A partial-call rejection calls a tool before all required fields have been gathered. An exact first-response pattern groups identical assistant content + tool_calls; it does not group paraphrases. Top-1 and top-5 shares are the fractions of inspected pairs covered by the most common one and five patterns. Source-equal weighting gives each source trajectory the same total weight, preventing sources with many derived pairs from dominating the count.

Table 3 separates two observations. First, the endpoint implicit rewards relative to the frozen SFT reference were -0.038 for chosen trajectories and -6.021 for rejected trajectories. The score contrast was dominated by lower rejected scores. Second, rejected first responses were more concentrated than chosen ones, and that difference remained under source-equal weighting.

Diagnostic and population	Observation	Interpretive boundary
Chosen implicit reward held-out preference pairs	-0.038	Near the SFT reference under the trajectory score
Rejected implicit reward held-out preference pairs	-6.021	Further below the SFT reference than the chosen score
Rejected share of absolute reward magnitude	99.38%	Recorded decomposition of reward magnitudes
Chosen first response, top-1 / top-5 exact text + tool calls	8.18% / 28.50%	Hard Type2 partial-call subset; source-equal values were 8.19% / 28.83%
Rejected first response, top-1 / top-5 exact text + tool calls	49.13% / 71.30%	Source-equal values were 49.47% / 71.71%

Table 3. Primary-reconstruction diagnostics. Reward rows use 840 held-out pairs from 135 source groups; response-concentration rows use the 2,530 Hard Type2 partial-call pairs. Source-equal weighting gives each source trajectory the same total weight. The two groups measure preference fit and data repetition, respectively.

Lower rejected reward is a valid DPO optimization path [3]. The concentration finding identifies repeated supervision that can be inspected; it does not show that the model relied on those patterns as a shortcut [7]. Together, these observations motivate testing the sensitivity of generated behavior to dialogue construction. They do not identify the cause of the primary model’s remaining Slot errors.

Changing where the paired dialogues diverge

The alternative reconstruction synchronized chosen and rejected prefixes through the turn immediately before their behaviors diverged. It changed the 2,530 Hard Type2 partial-call pairs; the other 14,264 pairs, starting SFT model, objective, and optimizer-update budget were held fixed in the recorded setup. Synchronizing the prefix also changed later trajectory content, tokenization, and branch lengths. The exact recorded length changes are retained in the Appendix.

A coverage check on this alternative dataset found that 2,521 / 2,530 pairs (99.64%) diverged with only one required field still missing. At source level, all inspected pairs from 559 / 562 sources (99.47%) had that property. This locates the contrastive decision mostly at the final missing field. It does not mean those source dialogues lacked earlier states with several missing fields, and it is separate from the primary-data concentration measurements in Table 3.

Behavioral changes in the sensitivity run

Figure 1 and Table 4 compare the primary and alternative conditions. Slot improved by +7.69 percentage points, from 270 to 304 passing items out of 442. At the same time, Call lost 16 passing items and Completion lost one. The higher Macro score therefore summarizes a trade-off across behaviors, not a uniform improvement.

Model	Call n=670	Completion n=71	Slot n=442	Relevance n=123, adjusted	Macro
Primary DPO	68.81% (461)	92.96% (66)	61.09% (270)	27.64% (34)	62.62%
Prefix-synchronized DPO	66.42% (445)	91.55% (65)	68.78% (304)	28.46% (35)	63.80%
Change	-2.39 pp	-1.41 pp	+7.69 pp	+0.81 pp	+1.18 pp

Table 4. Local data-construction sensitivity comparison. Each DPO condition was trained once. Changes are prefix-synchronized minus primary DPO, computed from counts before rounding; positive values favor the alternative on that axis. Slot scores retain the 442-item judgment snapshot, separate from the filtered transition audit below.

The paired transition audit supports a change in the generated Slot responses, beyond comparing two aggregate scores. The recorded judge-based improvement/regression counts were 0/1 for Completion, 45/11 for Slot, and 3/2 for Relevance. After excluding three Slot cases where identical outputs received inconsistent judgments, the filtered Slot audit contained 44 improvements and 9 regressions. That exclusion changes the audited population; those counts are not a recomputation of the 442-item score row in Table 4.

Both DPO variants completed 15/28 auxiliary rollouts successfully. This small check showed no increase in complete-call success, despite the positive Slot movement. It does not establish zero effect on dialogue success. With one training run per condition and simultaneous changes to prefixes, content, and lengths, the observed shift is associated with the combined reconstruction change; its separate causes and sensitivity to training randomness remain unresolved.

Interpretation

The contribution of this reconstruction is a connected assessment of preference fit, the contrasts presented by the data, and generated tool-use behavior.

• Preference fit: held-out ranking showed that the model learned the reconstructed preference distinction. The reward decomposition described how chosen and rejected scores differed relative to SFT.
• Contrast coverage: repeated rejected responses in the primary data and final-missing-field divergence in the alternative data exposed properties that pair counts and a valid dialogue schema do not describe. These are separate distribution diagnostics, not an established shortcut mechanism.
• Behavioral transfer: paired output changes supported a local Slot improvement, while Call, Completion, and complete-call success prevented that improvement from being read as a general gain in tool use.

The practical implication is to evaluate reconstructed preference data at all three levels. A schema and a high preference score describe what was supplied and learned; axis-level responses and free-running dialogues reveal which behaviors changed. This study supports that distinction through a specific reconstruction and one sensitivity comparison, without selecting a generally superior data-construction rule.

Limitations

• Single-run sensitivity: the primary and alternative DPO conditions were each trained once. Seed variation was not measured, and no random-rewrite or placebo control isolated prefix synchronization from the other changes.
• Judgment uncertainty: the recorded Slot agreement with an independent judge was 98.19% (Cohen’s kappa 0.957). This is evidence about that subset; it does not establish agreement with the paper’s judge or validate every local score. Identical-output inconsistencies were handled as described above and in the Appendix.
• Different evaluation roles: the 28 rollouts are a small auxiliary check, separate from the fixed-context, teacher-forced turn evaluation described by DiaTool-DPO. Equal success counts on this check do not establish equivalent dialogue policies.
• Unavailable run artifacts: this revision uses retained tables, figures, and the public SFT documentation. Original DPO logs, paired outputs, full generation/judge configurations, and the paper’s original preference data are unavailable for a fresh audit. The local concentration measurements cannot establish whether the paper’s data had the same patterns.

Appendix: Reconstruction and comparison context

Recorded reconstruction scope

Component	Public specification	Local reconstruction / boundary
Preference-pair counts and state-transition paths	Counts and state-transition paths reported by trajectory type	Rebuilt to match the reported table Boundary: Structurally aligned
Objective and DPO settings	Early-turn weight, turn-weight normalization, margin, and training settings	Applied from the reported specification Boundary: Training specification aligned
Continued pretraining / SFT starting point	Training procedure and source described	Separately trained from public data Boundary: Not the same checkpoint
Korean preference data	Generation procedure and total scale described	Public prompts with local generation models and rules Boundary: Reconstructed data
Source sampling and partial-reveal distribution	Not fully determined by the public description	Local sources and partial-reveal rules Boundary: Potential sensitivity axis
Branch-level surface concentration	Not a required measurement in the paper	Measured as a post-hoc diagnostic Boundary: Local evidence only

Appendix Table 1. Recorded reconstruction choices relative to DiaTool-DPO [1]. Matching the reported pair counts, state paths, and settings establishes the intended structural comparison; it does not establish identity of checkpoints, trajectories, or evaluation conditions.

Paper comparison

For Macro scores $M_{\mathrm{SFT}}$ and $M_{\mathrm{DPO}}$, the relative gain $g$ is

\[g = \frac{M_{\mathrm{DPO}}-M_{\mathrm{SFT}}}{M_{\mathrm{SFT}}}\times 100\%.\]

The recorded local Macro relative gain was +10.93%, numerically close to the paper’s +10.78%. Their behavioral composition differed: local Slot rose by +7.01 points and Completion by +7.04, while the paper reported +27.80 and -2.80, respectively. Appendix Figure 1 and Appendix Table 2 preserve that comparison.

This difference originally motivated the data diagnostics. It cannot quantify a reproduction shortfall: the checkpoints and preference trajectories differed, the paper’s evaluation revision and detailed scoring implementation were not pinned, and the local Call score used exact match whereas the original benchmark described judge-based rubrics. A similar Macro ratio cannot resolve those differences.

Appendix Figure 1. Contextual comparison of two reported profiles. The top strip shows relative Macro gains on a 0–12% scale; the lower plot shows axis changes in percentage points. Checkpoints, preference data, and scoring conditions differ, and the paper's evaluation revision is not pinned. These columns do not estimate a common effect.

Evaluation axis	Paper change
Call	+1.40 pp
Completion	-2.80 pp
Slot	+27.80 pp
Relevance	+8.70 pp
Macro	+8.80 pp

Appendix Table 2. SFT-to-DPO changes calculated from DiaTool-DPO's reported scores. Values are retained for context alongside the local results in Table 2, not as a matched benchmark comparison.

Judgment snapshots and data-length changes

SFT score snapshot. The public SFT card lists Relevance 21.14% and Macro 56.25%. This note retains the later locally adjusted values 21.95% and 56.45%. The prior audit recorded two inconsistent judgments among 42 byte-identical Relevance responses and assigned identical outputs the same local judgment. The card and adjusted table are separate score snapshots; the adjustment logs are no longer available to recheck.

Alternative-data lengths. Under the training tokenizer and chat-template rendering, full-dataset chosen tokens changed by -0.73% and rejected tokens by -0.09%. Within the changed pairs, combined tokens changed by -2.30%, and the mean chosen-minus-rejected length gap fell from 41.03 to 28.36 tokens. Neither dataset contained trajectories longer than 8,192 tokens. These changes are part of the sensitivity condition, so the comparison does not isolate prefix synchronization.

Experiment Resources

• Reconstructed SFT adapter and model card, with its training report. These document the Instruct-based continued-pretraining-to-SFT lineage and rank-8 LoRA. The card names gpt-oss-120b for its local judged axes; it does not supply the full DPO scoring configuration.
• Evaluated FunctionChat-Bench revision. The retained local axis counts are 670 Call, 71 Completion, 442 Slot, and 123 Relevance items.
• The September 6, 2026 revision reorganizes retained results and checks accessible primary sources. No new training, generation, or model judging was performed.

References

1. Jung, S. et al. DiaTool-DPO: Multi-Turn Direct Preference Optimization for Tool-Augmented Large Language Models. SIGDIAL 2025, 2025. ACL Anthology
2. Kakao. DiaTool-DPO Reference Implementation. GitHub repository
3. Rafailov, Rafael et al. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023, 2023. arXiv
4. Meta. Meta-Llama-3-8B-Instruct. Hugging Face model card
5. Hu, Edward J. et al. LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022, 2022. arXiv
6. Lee, Shinbok et al. FunctionChat-Bench: Comprehensive Evaluation of Language Models' Generative Capabilities in Korean Tool-use Dialogs. 2024. arXiv; evaluated repository revision
7. Geirhos, Robert et al. Shortcut Learning in Deep Neural Networks. Nature Machine Intelligence, 2, 665-673, 2020. DOI

Citation

Text citation:

Ilho Ahn, "DiaTool-DPO Reconstruction: Preference Accuracy and Tool-Use Behavior", Mini Research, July 29, 2026.

BibTeX:

@article{ahn2026diatooldpopublicreconstruction,
  author = {Ilho Ahn},
  title = {DiaTool-DPO Reconstruction: Preference Accuracy and Tool-Use Behavior},
  journal = {Mini Research},
  year = {2026},
  month = jul,
  url = {https://muted-color.github.io/research/2026/07/29/diatool-dpo-public-reconstruction-preference-trajectories/}
}