LLM TOOL USE

Preference Trajectories and Slot Behavior in a DiaTool-DPO Reconstruction

Ilho Ahn

Jul 29, 2026

An analysis of preference trajectories and Slot behavior under a public-information reconstruction of DiaTool-DPO.

Featured summary. Numerically similar Macro gains did not distinguish the different Completion and Slot changes. This axis-level difference motivates the preference-trajectory diagnostics below; the profiles use different checkpoints, data, and judge backends and are not one common effect estimate.

The local DiaTool-DPO model improved over its SFT baseline on all four FunctionChat-Bench axes, but its axis-level profile differed from the paper. This note examines that difference through the structure and distribution of tool-use preference trajectories, with Slot behavior—whether the assistant requests missing required fields before calling a tool—as the main diagnostic. The public-information reconstruction defines the experimental condition and its limits; the paper and local profiles use different checkpoints, preference data, and judge backends and are not one common effect estimate [1].

Summary

• Public pair counts, state paths, objective components, and DPO settings were matched; the unavailable checkpoint, preference data, and judge backend were not identical.
• Macro relative improvement was numerically similar (+10.93% locally; +10.78% in the paper), but the two conditions are not one comparable effect estimate.
• Slot improved by +7.01 points locally, substantially less than the paper’s +27.80-point change; this is the main boundary on effect reproduction.
• Local trajectory diagnostics and one post-hoc run indicate data-construction sensitivity, but do not establish shortcut learning or a generally superior reconstruction rule.

DiaTool-DPO Method

Jung et al.’s DiaTool-DPO treats tool use as a choice of dialogue flow rather than a preference between isolated responses [1]. The paper defines three query types over five conceptual states: Type1 calls immediately when all required arguments are available, Type2 gathers missing fields before calling, and Type3 refuses when no available tool supports the request. The states describe the task structure proposed by the paper, not model states observed directly.

Each training example pairs a chosen full trajectory with a rejected one. They share the initial request and tool context, but later turns and lengths may differ. Rejected paths include redundant questions, premature calls, incorrect refusals, and unsupported tool calls. User and tool messages provide context while assistant turns are scored. The Easy and Hard subsets contain 16,794 pairs in total [1]. Figure 1 summarizes the resulting training flow.

Figure 1. A request and tool set lead to an immediate call, filling N missing required fields before a call, or refusal. DiaTool-DPO pairs the intended path with a counter-trajectory and scores the pair at trajectory level. The geometry is conceptual and does not encode frequency or effect size.

The objective aggregates policy-versus-reference scores across assistant turns. It gives earlier actions more influence, normalizes each trajectory by its total turn weight, and requires a margin between chosen and rejected scores. These adjustments accommodate the structural length differences between Slot-filling and rejection paths [1].

Experimental Setup

Reconstruction Scope

The reconstruction implemented the published trajectory structure and objective components using the reference implementation as a public resource [2]. It retained ordinary DPO’s policy-versus-reference comparison [3], but several artifacts needed for an exact reproduction were not public.

Table 1 separates specifications taken from the paper from choices made during the reconstruction. This boundary is central to the result: matching a published trajectory schema does not imply matching the unpublished trajectory distribution.

Component	Public specification	Local reconstruction / boundary
Preference-pair counts and state-transition paths	Counts and state-transition paths reported by trajectory type	Rebuilt to match the reported table Boundary: Structurally aligned
Objective and DPO settings	Early-turn weight, turn-weight normalization, margin, and training settings	Applied from the reported specification Boundary: Training specification aligned
CPT/SFT starting point	Training procedure and source described	Separately trained from public data Boundary: Not the same checkpoint
Korean preference data	Generation procedure and total scale described	Public prompts with local generation models and rules Boundary: Reconstructed data
Source sampling and partial-reveal distribution	Not fully determined by the public description	Local sources and partial-reveal rules Boundary: Potential sensitivity axis
Branch-level surface concentration	Not a required measurement in the paper	Measured as a post-hoc diagnostic Boundary: Local evidence only

Table 1. Scope of structural alignment against the published specification [1]. “Aligned” applies to reported preference-pair counts, state-transition paths, and settings, not checkpoint or data identity.

Evaluation Design

The primary comparison used the same local SFT lineage for the SFT baseline and reconstructed DPO model. The SFT adapter reconstructed for this project is publicly available, which makes the local starting point inspectable even though it is not the paper’s checkpoint.

Reconstructed SFT checkpoint

LLaMA 3 8B DiaTool Korean SFT LoRA soleaf/Llama-3-8B-DiaTool-Ko-SFT-LoRA

Table 2 summarizes the shared local comparison contract and the boundary of each condition.

Field	Reconstruction condition	Interpretive role
Starting model	Llama 3 8B Instruct [4] + reconstructed tool-use SFT LoRA	Shared local reference for SFT-versus-DPO comparison
Preference data	Easy 8,357 + Hard 8,437 = 16,794 pairs	Matches the public pair-count specification
DPO training	1 epoch, total batch 8, LR 1e-7, bf16, max length 8,192	Uses the reported optimization settings
Parameter update	Continued rank-8 LoRA [5]	The paper does not separately specify whether PEFT was used
Evaluation set	FunctionChat-Bench, 1,306 items at the evaluated revision [6]	Call 670, Completion 71, Slot 442, Relevance 123

Table 2. Primary reconstruction and evaluation conditions. Reported pair counts and optimization settings follow DiaTool-DPO [1]. The comparison is internally controlled against the reconstructed SFT starting point, not against the paper's original checkpoint.

Call is normalized tool-call exact match. Completion, Slot, and Relevance use a fixed-rubric judge following the benchmark’s four-axis evaluation design [6]. Macro is the unweighted mean of these four axes despite their different sample counts and judgment mechanisms, so it is used only as a benchmark summary. Relative Macro gain is computed as

Relative Macro gain (Macro_DPO − Macro_SFT) / Macro_SFT × 100%

For Relevance, byte-identical responses were assigned the same local judgment. A manual audit found two inconsistent judgments among 42 identical-response cases. This adjustment did not change the main conclusion.

Results

Aggregate Gain and Behavioral Composition

The reconstructed DPO model scored above its SFT baseline on all four evaluation axes. Figure 2 shows why the Macro result is insufficient: the local and paper relative Macro gains were numerically close, but the behavioral components moved differently. The two profiles come from different checkpoints, preference data, and judge backends, so their values are not one common effect estimate. Table 3 retains the exact values.

Figure 2. The top strip compares relative Macro gain on a common 0–12% scale; the zero-centered plot below shows SFT-to-DPO changes for each evaluation axis in percentage points. Similar aggregate ratios were composed differently, including a larger reported Slot change and a reported Completion decrease. The local and paper profiles come from separate checkpoints, preference data, and judge backends and are not one common effect estimate.

Evaluation axis	SFT	Reconstructed DPO	Local change	Paper change
Call n=670	63.88% (428)	68.81% (461)	+4.93 pp	+1.40 pp
Completion n=71	85.92% (61)	92.96% (66)	+7.04 pp	-2.80 pp
Slot n=442	54.07% (239)	61.09% (270)	+7.01 pp	+27.80 pp
Relevance n=123, locally adjusted	21.95% (27)	27.64% (34)	+5.69 pp	+8.70 pp
Macro	56.45%	62.62%	+6.17 pp	+8.80 pp

Table 3. SFT-to-DPO changes in the local reconstruction and changes reported by DiaTool-DPO [1]. Higher is better on all four axes. The columns do not estimate one common effect because the checkpoints, preference data, and judge backends differ.

The local Macro relative gain was +10.93%; the paper’s was +10.78%. In the paper, Slot was the largest positive change while Completion decreased. Locally, Completion and Call contributed more and Slot increased less. Similar aggregate ratios therefore did not distinguish the different behavioral composition.

At the item level, the local Slot comparison contained 43 improvements and 12 regressions. This confirms a positive aggregate Slot movement, but not the magnitude or behavioral composition reported in the paper. The difference between the local profile and the paper’s Slot-dominant profile motivates the trajectory-level diagnostics below.

Preference-Trajectory Diagnostics

The initial data audit covered schema, state paths, loaders, duplicates, masks, and pair-level semantic validity. Two additional measurements were added after the behavioral difference appeared: branch-level surface concentration and divergence-state coverage.

The surface diagnostics use the 2,530 Hard Type2 partial-call target pairs within 7,832 Type2 pairs, derived from 562 source trajectories. An exact surface groups only identical first-assistant content + tool_calls; source-equal weighting assigns the same total weight to each source. The remaining-one row is separate: it measures divergence coverage in the prefix-synchronized post-hoc reconstruction.

On a 5% source-group validation split, chosen reward exceeded rejected reward for 97.86% of pairs. The split contained 840 pairs derived from 135 source groups, with zero source overlap with training. The trajectory-level implicit rewards were recorded relative to the frozen SFT reference. They diagnose where the learned margin formed; they do not directly measure generated behavior. Figure 3 summarizes the Hard Type2 surface-concentration and divergence-state diagnostics; Table 4 reports them alongside the reward diagnostics and interpretation boundaries.

Figure 3. Chosen and rejected trajectories share the user query but diverge into different behaviors. Exact-surface concentrations come from the primary reconstruction; remaining-one coverage comes from the prefix-synchronized post-hoc reconstruction. The box geometry is conceptual and does not establish shortcut learning.

Diagnostic	Observation	Interpretive boundary
Chosen-arm implicit reward	-0.038	Small recorded trajectory-reward change
Rejected-arm implicit reward	-6.021	Most of the recorded margin formed through the rejected arm
Rejected share of absolute reward magnitude	99.38%	A reward decomposition, not behavioral evidence
Chosen first-assistant exact surface, top-1 / top-5	8.18% / 28.50%	Hard Type2 partial-call subset; source-equal values were 8.19% / 28.83%
Rejected first-assistant exact surface, top-1 / top-5	49.13% / 71.30%	Source-equal values were 49.47% / 71.71%
One Slot remaining at partial-call divergence	2,521 / 2,530 (99.64%)	Prefix-synchronized post-hoc reconstruction; source-level all-remaining-one coverage was 559 / 562 (99.47%)

Table 4. Diagnostics from three populations: reward rows use the 5% source-group validation split; surface rows use the Hard Type2 partial-call subset; the remaining-one row uses the prefix-synchronized post-hoc reconstruction. These measurements locate repeated training signals; none alone demonstrates shortcut learning or generated behavior.

The pair-level and source-equal concentration values were similar. The observed skew is therefore difficult to explain solely by a small number of sources producing many pairs. In the prefix-synchronized reconstruction, remaining-one does not mean that the source trajectories lacked multi-missing states. It means the chosen and rejected behaviors usually diverged immediately before the final missing field.

Lowering rejected reward is a normal DPO optimization path [3]. High rejected-surface concentration and narrow divergence-state coverage can coexist with valid preference learning; these measurements alone do not demonstrate shortcut learning [7]. Because the original preference data were unavailable for the same measurements, these diagnostics remain specific to the local reconstruction and do not establish a difference from the original data distribution. They instead motivate a bounded sensitivity test of whether a different trajectory construction is associated with different Slot behavior.

Post-Hoc Trajectory Sensitivity

A separate post-hoc reconstruction tested whether the Hard Type2 trajectory construction was associated with the observed Slot behavior. For the 2,530 partial-call target pairs, chosen and rejected prefixes were synchronized through the turn immediately before their behaviors diverged.

The other 14,264 pairs, starting SFT model, objective, and optimizer-update budget were held fixed. Prefix synchronization nevertheless altered later trajectory content, tokenization, and branch lengths, so this condition is a sensitivity comparison rather than an isolated ablation.

The intervention did not isolate prefix equality. Under the tokenizer and chat-template rendering used for training, chosen tokens changed by -0.73% and rejected tokens by -0.09% over the full dataset. Within the changed pairs, combined chosen-plus-rejected tokens changed by -2.30%, and the mean chosen-minus-rejected length gap decreased from 41.03 to 28.36 tokens. Neither dataset contained trajectories longer than 8,192 tokens. Figure 4 summarizes the direction of the axis-level changes and the unchanged auxiliary rollout result. Table 5 retains the exact values and presents the run as a sensitivity comparison rather than an isolated ablation.

Figure 4. Deltas are prefix-synchronized DPO minus primary DPO, in percentage points. Slot increased in this single run, while Call and Completion decreased and auxiliary rollout exact success remained 15/28. Prefixes, later trajectory content, tokenization, and branch lengths changed together, so the chart does not isolate a prefix-only effect.

Model	Call n=670	Completion n=71	Slot n=442	Relevance n=123, adjusted	Macro
Primary DPO	68.81% (461)	92.96% (66)	61.09% (270)	27.64% (34)	62.62%
Prefix-synchronized DPO	66.42% (445)	91.55% (65)	68.78% (304)	28.46% (35)	63.80%
Change	-2.39 pp	-1.41 pp	+7.69 pp	+0.81 pp	+1.18 pp

Table 5. Single-run post-hoc sensitivity comparison. The prefix-synchronized reconstruction increased Slot, but also changed lengths and later trajectory content while slightly reducing Call and Completion.

Judge-based paired transitions for the sensitivity run were improvement/regression 0/1 on Completion, 45/11 on Slot, and 3/2 on Relevance. After excluding three Slot cases where identical outputs received inconsistent judgments, the output-level Slot transitions were 44 improvements and 9 regressions.

The Slot benchmark output changed in the positive direction in this run. The evidence does not identify prefix synchronization as the sole cause: both arms were reconstructed, and tokenization, sequence lengths, and later trajectory content changed together. There was no random-rewrite or placebo control. Call and Completion decreased slightly, and exact success on 28 auxiliary free-running rollouts remained 15/28 for both DPO variants. The aggregate mismatch, local diagnostics, and post-hoc sensitivity result therefore support three separate audit axes rather than a single causal explanation.

Three Audit Axes for Preference Trajectories

The evidence separates structural agreement, objective fit, and behavioral transfer. Pair counts, state paths, and objective components establish the first; preference accuracy and implicit reward margins describe the second; FunctionChat-Bench axes and paired output transitions test the third. Structural agreement and objective fit did not determine behavioral transfer: the primary reconstruction improved in aggregate without matching the paper’s Slot-dominant profile, and the post-hoc run changed Slot without uniformly improving the other axes.

Three audit axes remain useful for preference trajectories with different branch lengths, expressions, and state paths:

• Surface concentration: measure whether a small set of exact branch expressions repeatedly co-occurs with the preference label. This is a distribution diagnostic, not a shortcut test by itself.
• Divergence-state coverage: report the source count and missing-Slot distribution at the point where chosen and rejected behaviors separate. Pair count alone does not reveal the covered contrastive states.
• Objective-to-behavior transfer: report preference accuracy and reward margins separately from generated behavior and axis-level paired transitions.

These checks do not prescribe a particular reconstruction method. They make the boundary between a structurally valid preference dataset and behaviorally equivalent training evidence more explicit.

Conclusion

The central observation is that aggregate improvement and Slot behavior followed different profiles under the local and paper conditions. The local Slot gain was +7.01 points, compared with the paper’s +27.80-point change, while the relative Macro gains were numerically similar. These values come from different checkpoints, preference data, and judge backends and are not one common effect estimate.

The trajectory diagnostics describe repeated local branch surfaces and narrow divergence-state coverage, and the prefix-synchronized condition was associated with a further +7.69-point Slot change in one run. That comparison changed tokenization, lengths, and later trajectory content together, reduced Call and Completion, and left auxiliary rollout success at 15/28; it does not identify a sole cause or establish a generally superior reconstruction rule. The most stable conclusion is narrow: surface concentration, divergence-state coverage, and objective-to-behavior transfer complement pair-count and state-path checks when preference trajectories are reconstructed rather than directly available.

Limitations

• The reconstruction did not use the paper’s exact checkpoint, preference JSONL, or judge backend. On the Slot subset, the local judge agreed with an independent judge on 98.19% of cases (Cohen’s kappa 0.957). This supports stability for that subset but does not establish compatibility with the paper’s judge or scores.
• The primary DPO and post-hoc sensitivity condition were each trained once. Seed variation in the reported effect sizes was not measured.
• The post-hoc condition changed prefixes, later trajectory content, tokenization, and branch lengths together. It cannot isolate the contribution of any one component.
• The 28 auxiliary rollouts were separate from the paper’s turn-level teacher-forced FunctionChat evaluation [1] and did not show a complete-call success increase.
• The original branch-level surface and divergence-state distributions were unavailable. The local diagnostics cannot establish whether the original data had the same or different concentration patterns.

References

1. Jung, S. et al. DiaTool-DPO: Multi-Turn Direct Preference Optimization for Tool-Augmented Large Language Models. SIGDIAL 2025, 2025. ACL Anthology
2. Kakao. DiaTool-DPO Reference Implementation. GitHub repository
3. Rafailov, Rafael et al. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023, 2023. arXiv
4. Meta. Meta-Llama-3-8B-Instruct. Hugging Face model card
5. Hu, Edward J. et al. LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022, 2022. arXiv
6. Lee, Shinbok et al. FunctionChat-Bench: Comprehensive Evaluation of Language Models' Generative Capabilities in Korean Tool-use Dialogs. 2024. arXiv; evaluated repository revision
7. Geirhos, Robert et al. Shortcut Learning in Deep Neural Networks. Nature Machine Intelligence, 2, 665-673, 2020. DOI

Citation

Text citation:

Ilho Ahn, "Preference Trajectories and Slot Behavior in a DiaTool-DPO Reconstruction", Mini Research, July 29, 2026.

BibTeX:

@article{ahn2026diatooldpopublicreconstruction,
  author = {Ilho Ahn},
  title = {Preference Trajectories and Slot Behavior in a {DiaTool-DPO} Reconstruction},
  journal = {Mini Research},
  year = {2026},
  month = jul,
  url = {https://muted-color.github.io/research/2026/07/29/diatool-dpo-public-reconstruction-preference-trajectories/}
}