Odyssey: Forging Language Agents with Synthesized Environments and Mixed RL at a Humble Scale

When Your Agent Fails at Step 19

Imagine training a language agent to book a flight. The task needs a sequence: search airlines, compare prices, check seats, select options, fill forms, confirm payment. Suppose this workflow takes 19 tool calls. One bad API call at step 19 crashes everything.

With a negative trajectory-level advantage, standard policy gradient methods, such as REINFORCE with Group Baseline, GRPO, and RLOO applies the same negative advantage to all 19 steps. The update penalizes tokens from the 18 correct steps alongside those from the failed final call, even though only that last call caused the failure.

This is the credit assignment problem. It gets worse as tasks grow longer.

What Goes Wrong in Practice

Scenario 1: The environment lies to you

Agent: get_flight(airline="United", date="2025-01-15")
Environment: ❌ Error: Invalid date format
Reality: The format was correct; the database schema was wrong
RL Update: Penalize the agent for using the correct format

Scenario 2: One typo ruins 18 perfect steps

Step 1-18: ✓ Correct tool calls, valid outputs
Step 19: comfirm_booking()  # typo: should be "confirm"
Standard GRPO: advantage = -1.2 applied to all 19 steps
Result: Update penalizes 18 correct steps along with 1 typo

The Core Bottleneck: Low Learning Value of Experience

Language agents run multi-step workflows where one error cascades into failure. The bottleneck has two parts: Experience Reliability (noisy environments produce useless rollouts; if ground truth is wrong, the agent can’t learn what “correct” means) and Experience Utilization (even with reliable environments, coarse episodic rewards hurt optimization: standard GRPO applies one advantage per trajectory to every token, penalizing 18 correct steps because of 1 mistake). Training on single environment types produces specialists that fail when distributions shift. Existing work fixes these separately. Odyssey’s insight: you need both, and they must work together.

Unlike AlphaGo, which mastered a single, well-defined game environment, real-world language agents must operate across diverse and heterogeneous settings: navigating e-commerce platforms, answering research questions, debugging code, managing databases. Each domain brings different tool schemas, reward structures, and failure modes. Generalizing across this variety requires training on a broad spectrum of environments.

Mixed-Environment RL: Training Across Diverse Environments and Task Distributions

Mixed-Environment RL means training policies across multiple, structurally different environments simultaneously: RuleReasoner, MiMo-v2.6. Unlike domain randomization (varying parameters within one task family), it handles different action spaces (tool schemas vary wildly across domains), reward structures (binary success vs. partial credit), and horizon lengths (search: 3-5 steps; data analysis: 20-40 steps).

Odyssey-13K instantiates Mixed-Environment RL at a moderate scale: 13,000 environments spanning tool use, web search, knowledge work, and conversation, totally 7 macro domains and 27 micro domains. Each training batch samples from this mixture, forcing the policy to:

1. Generalize tool-calling patterns across schemas (not memorize one API)
2. Adapt credit assignment to varying trajectory lengths and failure modes
3. Transfer reasoning strategies between domains (e.g., decomposition learned in research tasks applies to e-commerce)

This is why Odyssey models show transfer to external benchmarks. They’ve seen structural diversity during training that overlaps with distribution shifts at test time.

The Odyssey Trilogy: Three Interlocking Components

Odyssey is a closed-loop system where environment quality and training signal improve together:

Odyssey-Env mines real-world data (e.g., Wikipedia) to build thousands of verifiable environments. Each environment comes with executable tools, multi-step tasks, and verification code. No human annotation, no trajectory distillation.

Odyssey-RL introduces Heuristic Credit Assignment (HCA): detect observable failures (malformed JSON, invalid tool calls, runtime errors), locate the responsible token span, and assign negative gradient only there. Clean trajectories train normally; abnormal trajectories don’t contaminate the baseline.

Odyssey-Infra enforces training correctness at scale: strict token-in-token-out (TITO) invariance, bitwise parity between training and inference, and elastic scheduling across distributed clusters.

Let’s walk through each component, starting with the environments.

Odyssey-Env: Synthesizing 13,000 Reliable Tasks for Mixed-Env RL

Garbage in, garbage out. If your environment returns wrong answers, your agent learns to be confidently wrong. If tools crash randomly, the agent can’t tell bad strategy from bad luck.

Odyssey-Env mines real-world structured data and builds 13,000 environments with built-in verification across diverse domains. Training on tool use (database queries), web search (multi-hop retrieval), and knowledge work (data analysis) forces the model to learn generalizable reasoning patterns: decomposition, constraint satisfaction, error recovery that transfer across domains.

The Odyssey-Env Synthesis Pipeline

Each environment contains: $q$ (user query), $f_{\text{submit}}$ (submission schema), $\text{solver}$ (ground-truth solution), $\text{verifier}$ (verification program).

Stage 1: Autonomous Database Seeding. Generate topic matrix $M$ across domains (travel, e-commerce, research). For each topic $m$, an LLM generates search queries and retrieves structured data into database $D_m$. This is targeted data mining: “Find Wikipedia articles about space missions with launch dates” becomes structured tables.

Stage 2: Executable Tool Synthesis. Given $D_m$ and topic $m$, generate tools $T_m$ with standardized list()/get() interfaces, data-logic separation (tools read JSON files, never hard-code data), and automated validation. Every tool is a pure function over data files.

Stage 3: Verifiable Task Builder. Create multi-step tasks with natural-language query, ground-truth solution (≥2 tool calls), and verification code that independently reconstructs ground truth. The verifier doesn’t see the solution, which catches LLM hallucinations.

Stage 4: Validation Flow. Execute in isolated sandbox. Failures trigger closed-loop feedback: crashed solution? Regenerate. Failed verifier? Check solvability. This provides self-validation with minimal human supervision.

Stage 5: Multi-Difficulty Refinement. Generate harder tasks with multi-condition constraints, cross-file analysis, longer reasoning chains (5+ tool calls).

Result: Odyssey-13K, a dataset of 13,000 tasks in 5,000 environments spanning 6 domains, 4 difficulty levels, 8-15 tools per environment.

Why This Matters: Reliability at Scale

The pipeline produces environments where tasks are executable (validated in sandbox), solutions are verifiable (programmatic ground truth), and no trajectory distillation from teacher models is needed.

This addresses the experience reliability bottleneck. But even with reliable environments, coarse credit assignment still wastes learning signal. That’s where Odyssey-RL comes in.

Cost Audit: Economics of Environment Synthesis

To provide transparency on the resource requirements of our synthesis pipeline, we conducted a detailed cost audit of 9 representative tasks from the Odyssey-Env corpus. These tasks span diverse domains, from multilingual content adaptation to smart home planning, and represent the full synthesis workflow: data collection, tool generation, task construction, and iterative verification. Note that we employ the model dpsk-v4-flash-260425 as environment designer for envs and tasks synthesis.

Summary Statistics (9 Exemplar Tasks):

Phase Breakdown (Per-Task Average):

• Data Collection: 17.0 minutes
• Tool Synthesis: 3.2 minutes
• Task Construction: 3.8 minutes
• Verification & Repair: 43.3 minutes (64% of total time)

Key Findings:

1. Verification is the bottleneck: 64% of synthesis time is spent in iterative solution-verification loops. Of the 9 tasks analyzed, only 2 passed verification on the first attempt. The remaining 7 required multiple repair cycles, with some tasks taking 15-20 verification rounds before converging to a valid solution-test pair.

2. Token distribution skews heavily toward synthesis: 84% of tokens are consumed during the synthesis stages (tool generation, task construction, verification), while FireCrawl-based data extraction accounts for 16% (avg. 43 API calls per task, ~17 minutes).

3. High variance in resource consumption: Wall-clock time ranged from 48 to 90 minutes, and token usage varied between 750K and 1.54M per task. This variance is driven primarily by the number of verification repair cycles. Tasks with complex constraint interactions or edge cases require more iteration.

4. ROI and cost-effectiveness: At 0.88 dollar per task, synthesizing 13,000 environments cost approximately 11,440 dollar. This is 2-3 orders of magnitude cheaper than hiring human annotators to design, implement, and validate equivalent environments. Each synthesized task generates hundreds of training rollouts, amortizing the upfront cost across thousands of gradient steps.

Detailed Task Breakdown:

Implications for future work: The most direct path to cost reduction is improving verification convergence. Current repair strategies use LLM-in-the-loop debugging, which scales poorly with constraint complexity. Techniques like formal verification, symbolic execution, or learned repair models could reduce the verification phase from 43 minutes to under 10 minutes per task, cutting total synthesis cost by ~50%. Additionally, caching intermediate artifacts (tool schemas, test templates) across similar tasks could reduce redundant LLM calls.

This cost audit demonstrates that automated environment synthesis is economically viable at scale, even with current inefficiencies. The per-task cost of 0.88 dollar is negligible compared to the downstream training value, and further optimization can drive this cost below 0.50 dollar per task.

Odyssey-RL: Agentic RL via Heuristic Credit Assignment

Why Heuristic Credit Assignment?

When models generate chain-of-thought reasoning or execute tool-calling sequences, they shift from latent reasoning (internal, opaque computation) to observable execution traces (structured tool calls, API responses). This observability creates an opportunity for AI4AI: detect problems in the reasoning chain via CoT Monitoring, flag violated constraints, and trace errors to their source.

The insight: We don’t need to tell the agent how to succeed. We need to tell it what not to do. Principled errors (malformed JSON, invalid tool schemas, protocol violations) should carry immediate, local penalties. This shifts learning from “here’s the right trajectory” (expensive, needs expert demonstrations) to “here’s what breaks” (cheap, automated via static analysis). Heuristic Credit Assignment operationalizes this: detect observable failures, assign blame only to the tokens that caused them.

Standard GRPO is credit-blind. For a group of $G$ trajectories sampled per question $x$, the advantage of trajectory $i$ is:

$$ \hat{A}_i = \frac{\widetilde{R}_i - \mu_x}{\sigma_x + \varepsilon} $$

where $\mu_x$ and $\sigma_x$ are the mean and standard deviation of returns in the group. The policy gradient then updates all tokens in trajectory $i$ with the same advantage $\hat{A}_i$.

This is coarse credit assignment: every action in a failed trajectory gets equal blame, even if only one malformed tool call caused the failure.

The HCA Insight: Localize Blame to Observable Errors

Heuristic Credit Assignment (HCA) introduces three mechanisms:

1. Failure Detection: Identify observable errors in rollouts
2. Localized Credit Mask: Restrict gradient to responsible token spans
3. Clean Group Baseline: Prevent abnormal trajectories from polluting advantages

Let’s walk through each.

1. Failure Detection

During rollout, flag trajectories as abnormal if they show any of these failures:

A trajectory is abnormal if $e_t \neq \varnothing$ for some turn $t$. Define the responsible step as:

$$ t^* = \min\{t : e_t \neq \varnothing\} $$

An abnormal trajectory ends at $t^*$ with training return $\widetilde{R}_i = 0$. Normal trajectories use the actual reward: $\widetilde{R}_i = R(\tau_i)$.

2. Localized Credit Mask

Let $b_{t,j}^i \in \{0,1\}$ denote the original response-loss mask (which tokens are eligible for gradient).

For normal trajectories: $m_{t,j}^i = b_{t,j}^i$ (unchanged)

For abnormal trajectories: Restrict to error span $C_i$ in the first lagged turn $t_i^*$:

$$ m_{t,j}^i = b_{t,j}^i \cdot \mathbb{1}[t = t_i^*] \cdot \mathbb{1}[j \in C_i] $$

The error span $C_i$ is the token range containing the malformed tool call or JSON. For example:

Turn 3 (normal): get_movie("tt0468569")  ✓ All tokens masked
Turn 4 (error):  get_moive("tt0137523")  ✗ Only "moive" gets gradient
Turn 5 (unreached): ...                  ✗ Zero mask

Thus $m_{t,j}^i \leq b_{t,j}^i$: HCA only removes gradient-eligible tokens. It assigns negative signal to the error span, not the entire trajectory.

3. Clean Group Baseline

Let $\mathcal{C}_x$ denote indices of clean (non-abnormal) trajectories. Define baseline set:

$$ \mathcal{B}_x = \begin{cases} \mathcal{C}_x & \text{if } |\mathcal{C}_x| \geq 1 \\ \{1, \ldots, G\} & \text{otherwise} \end{cases} $$

Compute group mean $\mu_x$ and standard deviation $\sigma_x$ over $\mathcal{B}_x$ only.

Key property: When at least one clean trajectory exists, abnormal trajectories don’t contaminate the baseline. If $\mu_x > 0$, every abnormal trajectory gets the same negative advantage:

$$ \hat{A}_i^{\mathrm{HCA}} = -\frac{\mu_x}{\sigma_x + \varepsilon} < 0 $$

regardless of how many other abnormal trajectories exist.

The HCA Objective

Let $N_i = \sum_{t,j} m_{t,j}^i$ be the masked token count. The surrogate objective is:

$$ \mathcal{J}_{\mathrm{HCA}}(\pi_\theta) = \mathbb{E}_{x \sim \mathcal{D}, \{\tau_i\}_{i=1}^G \sim \pi_\theta(\cdot|x)} \left[ \frac{1}{G} \sum_{i=1}^{G} \frac{1}{\textcolor{red}{\max(1, N_i)}} \sum_{t,j} \textcolor{red}{m_{t,j}^i} \log \pi_\theta(a_{t,j}^i \mid h_t^i, a_{t,\text{<}j}^i) \widehat{A}_i^{\mathrm{HCA}} \right] $$

Literally, It means:

• Sample $G$ trajectories per question
• If a trajectory is normal: update all its tokens with standard group-relative advantage
• If a trajectory is abnormal: update only the error span with negative advantage computed from clean baseline

When no trajectory is abnormal, $m^i = b^i$ and $\mathcal{B}_x = \{1, \ldots, G\}$, recovering standard baseline estimation. HCA is a conservative extension.

Why This Works: Preserving Signal, Blocking Noise

HCA reduces to standard group-baseline estimation when no failures occur, while:

• Preventing abnormal trajectories from diluting baselines: If 7 out of 8 rollouts crash at step 2, the baseline is computed from the 1 clean trajectory, not the mean of all 8
• Penalizing only responsible tokens: A typo at step 4 doesn’t penalize correct reasoning at steps 1-3
• Maintaining stable gradient variance: Localized masks reduce the effective token count, preventing gradient explosion from long failed rollouts

The result: sample efficiency improves on long-horizon tasks compared to standard GRPO, with gains of 0.1-0.3× observed in our experiments.

Odyssey-Infra: Towards a Correct and Stable Agentic RL System

Odyssey extends Slime to serve as the RL training infrastructure. We developed Odyssey agent harness based on R2E-Gym but with a simpler architecture and can be deemed as a basic ReAct runtime. The system enforces training correctness and stability through three key mechanisms:

1. Strict Token-In-Token-Out (TITO)

The Problem: Agent loops convert between tokens and strings. In practice, this conversion is not always faithful. Decoding a token sequence into text and re-encoding it can change the sequence, particularly for text containing abbreviations. For instance, the token for “alternatively” can split into three tokens after a decode-encode round trip. This creates inconsistency between rollout-side and training-side token sequences.

The Solution: To avoid unexpected multi-segment forking for each rollout caused by re-encoding, Odyssey enforces strict TITO following Dressage. Response tokens generated by SGLang are passed directly to the Megatron training engine as token sequences. Decoding is performed only for display or logging, never in the critical path.

2. Faster True On-Policy RL with Bitwise Parity

Faster-TOP (Faster True On-Policy) implements zero training-inference mismatch with optimized kernels:

Level 1: Deterministic Execution: deterministic attention and communication on both rollout and training paths.

Level 2: Batch-Invariant Kernels: borrowed from He et al. 2025, prevents numerical results from changing with batch composition.

Level 3: Bitwise Parity: matches tensor parallel layout, precision, operator ordering across training and inference. Uses profiling to find divergence. Result: exact agreement between trainer and rollouter log probabilities.

3. Efficient Threading and Elastic Scheduling

Similar to Mercor and Sky Lab, Odyssey coordinates separate concurrency budgets, cache-aware MCP worker reuse, and coordinated rollout cancellation to prevent stale requests from corrupting later rollouts.

Benchmarking Odyssey Agents

We evaluate Odyssey models on six benchmarks: BFCL-v4 and ACEBench (tool use), τ²-Bench and VitaBench (conversation), WorkBench and OfficeQA (knowledge work).

Held-out Benchmark Categories: BFCL-v4 and ACEBench (tool use), τ²-Bench and VitaBench (conversation), WorkBench and OfficeQA (knowledge work).

Odyssey improves over base models across most benchmarks. The 8B model achieves +6.1% average gain, with strong improvements in tool use (+11.4 on BFCL-v4) and conversation (+7.9 on τ²-Bench, +7.0 on VitaBench). The 14B model reaches 43.2% overall.

Generalization to External Benchmarks

Models trained on Odyssey-13K generalize to external benchmarks without fine-tuning.

WikiQA: Single-Hop and Multi-Hop Question Answering

Odyssey improves over ReAct at all three sizes: 60.8→69.4 at 4B (+8.6), 66.0→71.9 at 8B (+5.9), 66.0→73.0 at 14B (+7.0). Odyssey-Nano (4B) exceeds ReAct 8B and 14B by 3.4 points. Individual benchmarks don’t improve uniformly; 14B scores slightly lower than 8B on Natural Questions and 2WikiMultiHopQA. These averages show an advantage over reported baselines but don’t isolate which training components produced gains or establish statistical significance.

Deep Search: GAIA, BrowseComp-Plus, xbench-DeepSearch

On challenging deep search benchmarks that require extended search, analysis, and synthesis:

Compared to the associated base models as start points, Odyssey-14B matches or exceeds specialized 8B agents (ASearcher-32B: 28.9% BrowseComp-Plus, 58.7% GAIA; CutBill-8B: 35.1%, 45.6%), despite being a general-purpose model trained on mixed environments.

Tool Usage: BFCL-v4 and ACEBench

BFCL-v4 scores: 35.2→39.7 (4B), 36.3→47.7 (8B), 39.1→49.5 (14B). ACEBench: 51.6→65.0, 59.1→68.3, 69.0→78.3. Odyssey-Mini (8B) shows the largest BFCL-v4 gain (+11.4); Odyssey-Nano (4B) the largest ACEBench gain (+13.4). Odyssey-Nano (4B) slightly exceeds ReAct-14B on BFCL-v4 (39.7 vs. 39.1) but remains below on ACEBench (65.0 vs. 69.0).

Conversation: τ²-Bench and VitaBench

Key Observations:

• Gains at 4B and 8B: τ²-Bench improves by 7.5 and 7.9 points; VitaBench by 1.1 and 7.0 points.
• Declines at 14B: τ²-Bench drops 2.2 points (40.1→37.9); VitaBench drops 3.0 points (9.0→6.0).

Knowledge Work: WorkBench and OfficeQA

WorkBench gains: 0.9 (4B), 1.0 (8B), 1.8 (14B). OfficeQA: +0.8 (4B), 0.0 (8B), +2.3 (14B). Gains are smaller than on tool-use benchmarks.

Training on mixed environments teaches invariant structures: decompose queries into sub-goals, recover from tool errors, verify partial progress. These patterns transfer when tools, schemas, and rewards change.

Training Dynamics: Learning Behavior Over Time

The following curves summarize reward, completion, interaction length, search activity, and action validity for Odyssey-8B and Odyssey-14B. Logging ranges differ across metrics.

Key observations (late-training values average the last 20 logged points):

• Reward improves, but completion ≠ correctness. Reward rises from 0.14 to 0.38 (8B) and 0.22 to 0.39 (14B). Late-training completion rates are higher (0.91 and 0.83), but reaching environment termination doesn’t guarantee task success.
• Trajectories lengthen. Average steps rise from 4.1 to 8.8 (8B) and 4.3 to 14.1 (14B). More interaction doesn’t necessarily mean better reasoning.
• Malformed-action rates decline. Falls from 37.5% to 1.25% (8B) and 21.1% to 0.02% (14B). This supports improved action validity; attributing it specifically to HCA requires ablation.

Training-Inference Consistency: Logprob Stability

A critical aspect of reliable RL training is maintaining consistency between the training trajectory log probabilities and those computed during rollout. The rollout logprob absolute difference measures this divergence: when training and rollout produce significantly different log probabilities for the same tokens, it signals potential issues such as stale gradients, numerical instability, or train-inference mismatch.

Key observations:

• Natural convergence without Faster-TOP: The orange curve shows the logprob difference decreases naturally from 0.0117 to 0.0091 over the training run, a natural 22.6% improvement driven by training dynamics. This intuitively suggests that partial rounding errors diminish as training progresses; however, the underlying mechanisms behind this trend warrant further investigation and ablation.

• Faster-TOP target consistency: The green dashed line at 0.0 represents the perfect consistency achievable with Odyssey-Infra’s Faster-TOP optimization. Faster-TOP enforces bitwise parity between training and inference through strict TITO token passing and deterministic kernel execution. The gap between the orange curve and the green line illustrates the remaining consistency overhead that Faster-TOP eliminates. In contrast, systems with re-tokenization or non-deterministic floating-point operations can exhibit logprob drifts of 0.05 or higher.

• Impact on credit assignment: Even the naturally improving consistency (orange curve) ensures that advantage estimates computed during rollout reasonably reflect the policy’s behavior during training. When rollout logprobs drift significantly from training logprobs, the policy gradient estimator becomes biased, leading to noisy updates. The observed stability supports HCA’s ability to localize credit, as the underlying probability computations remain coherent across training and rollout phases. Faster-TOP would push this to perfect coherence.

• No catastrophic divergence: The metric exhibits natural variance (peak at 0.013, trough at 0.0085) but never undergoes sudden spikes or divergence, which would indicate gradient staleness or numerical overflow. This stability allows Odyssey to safely train for extended horizons without encountering the train-inference mismatch issues that plague many RL systems at scale.

This metric complements the reward and completion curves: while those measure task-level outcomes, logprob consistency measures the infrastructure-level reliability that enables those outcomes.

Findings & Insights

Environment quality and algorithm design must work together. Clean environments allow vanilla policy gradient with group baseline methods like GRPO to work; noisy environments need specialized methods. Odyssey uses three mechanisms:

1. Heuristic Credit Assignment: Localizes gradient updates to token spans responsible for observable failures, preventing a single mistake from contaminating gradients for correct reasoning.

2. Task-Family Balance: Maintains proportional sampling across domains. Without balancing, high-reward tasks dominate and the policy overfits.

3. Online Biased Task Optimization: Upweights tasks where the policy shows improvement variance, focusing compute on the learning frontier.

The key lesson: HCA exists because Odyssey-Env generates diverse structures where coarse credit fails. Odyssey-Env’s validation exists because HCA requires observable failure signals. They must co-evolve.

Qualitative Analyses of Traces

To complement the quantitative benchmarks, we provide interactive trace visualizations that reveal how Odyssey agents reason, search, and recover from errors. Each trace shows the complete conversation flow: system prompts, user questions, assistant reasoning (thinking), tool calls, and observations.

The trace viewer above provides eight representative examples:

• Correct Traces: Show successful reasoning patterns where the agent correctly decomposes queries, issues appropriate tool calls, and synthesizes accurate answers
• Error Traces: Reveal failure modes such as incorrect tool selection, malformed queries, or premature termination
• Anomaly Traces: Highlight edge cases detected by HCA including repeated queries, parse errors, and direct submission without tool use

Each trace displays the complete interaction flow with color-coded components (purple for system, blue for user, green for assistant, yellow for tools, orange for observations). The thinking blocks reveal the agent’s internal reasoning process, while tool calls and observations show the external interaction loop.

Promising Future

On AI4AI: The real bottleneck isn’t any single component. Sometimes the model isn’t wrong, the harness is, or the environment design, or the task spec. We’re experimenting with coding agents that close the loop on case analysis, error attribution, debugging, and validation across environment evolution, tool repair, verifier iteration, and infrastructure optimization. The model doesn’t need to be smarter than humans, it needs to be faster at iteration and willing to test variations humans wouldn’t bother with.

Multi-Agent Environments: Verification shifts from “did the agent reach state $s$?” to “did the interaction converge to equilibrium?”

Cross-Environment Composition: Can an agent trained on movie databases and travel booking handle “find a theater near my hotel showing a Nolan film”? Tool schemas must align, verifiers must compose, and credit assignment must trace failures across domain boundaries.

Real-Time RL: Production agents face queries that don’t wait for nightly training runs. Requires streaming RL algorithms that handle concept drift, plus online environment synthesis.

Harness Meta-Optimization: As the harness grows to thousands of environments, optimizing the composition becomes its own problem: curriculum learning, active task selection, automated verifier synthesis.

Open Source

We will release the synthesized environments and tasks with detailed audit records in the near future, along with the trained models and key infrastructure implementations. Hope to enable the community to reproduce our results, and contribute to the development of more capable language agents.

Acknowledgements

This work builds upon and benefits from several outstanding open-source projects:

• Slime: The foundational RL training infrastructure that Odyssey-Infra extends for agentic workloads.
• Asearcher: Web search capabilities that influenced our WebQA/Retrieval environment design.
• SearchAgent-Zero: Web search agentic RL framework that influenced our training recipe design.
• R2E-Gym: Reinforcement learning environment suite for SWE that inspired our agent scaffolds.
• DeepSeek-V3.2: Prior environment synthesis methods that informed our training recipes.
• Defending Against the Training-Inference Numeric Mismatch in RL: Inspired us on the implications of zero train-inference mismatch in modern RL, especially for large-scale fully async runs.

We are grateful to the maintainers and contributors of these projects for making their work openly available. Their efforts have significantly accelerated our research and development.

Contributors

Core Contributors: Shuhan Qin, Yang Liu, Jiaqi Li, Jun Bai, Xiaobo Wang
Project Leads: Yang Liu, Zilong Zheng
Environment Synthesis Pipeline: Xiaobo Wang, Jiaqi Li, Tong Wu, Jun Bai, Zhe Li
RL Recipe & RL Infra: Yang Liu, Shuhan Qin
Experiments and Evaluation: Yang Liu, Shuhan Qin
Corresponding Author: Zilong Zheng

Citation

If you find Odyssey useful in your research, please cite our work:

@article{qin2026odyssey,
  title={Odyssey: Forging Language Agents with Synthesized Environments and Mixed RL at a Humble Scale},
  author={Qin, Shuhan and Liu, Yang and Li, Jiaqi and Bai, Jun and Wang, Xiaobo and Wu, Tong and Wang, Yanting and Yao, Gang and Chen, Hao and Jia, Zixia and Zheng, Zilong},
  year={2026},
  month={September},
  url={https://bigai-nlco.github.io/Odyssey}
}