| Test / Benchmark | Apollo‑1 | Best LLM Agent | Δ |
|---|---|---|---|
| τ‑Bench‑Airline | 90.8–92.5 % | Claude‑4 60 % | +51% |
| Google Flights** – 111 live booking chats | 83 % | Gemini 2.5‑Flash 22 % | +277% |
| Amazon Retail** – 120 live shopping chats | 90.8 % | Rufus 16.7 % | +444% |
90.8 – 92.5 % Pass^¹ and a 30-point lead show that, while Generative AI serves individual users well, Neuro-Symbolic AI is required for conversational agents that operate on behalf of companies and other entities.
τ-Bench-Airline is the toughest public benchmark for tool-using conversational agents, measuring complex, multi-turn user-agent interactions and advanced tool use. On the headline Pass^¹ metric, preliminary results place Apollo-1 in the 90.8–92.5 % range—over 30 percentage points ahead of Claude-4’s 60 %.
| Model | Pass^1 | Δ vs Apollo‑1 |
|---|---|---|
| Apollo‑1 (Neuro‑symbolic) | 90–92.5 % | — |
| Claude 4 Sonnet (LLM) | 60 % | −30 pt |
| Claude 4 Opus (LLM) | 59.6 % | −31 pt |
| OpenAI O3 (LLM) | 54.2 % | −36 pt |
Final Pass^¹, Pass^², Pass^³, and Pass^⁴ numbers will be published in our official τ-Bench report in December 2025.
Task #12: Reservation Update & Booking.
| Time | Speaker | Message |
|---|---|---|
| 12:18 PM | Agent | Welcome! This is a simulation environment for the TAU Benchmark – Airlines. How can I help you today? |
| 12:18 PM | User | Hi, I want to book the same flights as my current reservation, but this time for my friend Ivan Smith. His date of birth should be in my profile. |
| 12:18 PM | Agent | Thank you for your request! To get started with booking the same flights for Ivan Smith, could you please provide your user ID? |
| 12:19 PM | User | My user ID is ivan_muller_7015. |
| 12:19 PM | Agent | Agent Options: |
| HAT097, HAT251 | ||
| 07:00 DTW PHX 1 stop 15:00 SEA | ||
| "Shortest total duration, minimal layover, two segments only" | ||
| $186 Basic Economy | ||
| $375 Economy | ||
| $745 Business | ||
| HAT275, HAT156 | ||
| 00:00 DTW PHX 1 stop 10:00 SEA | ||
| "Early arrival, short layover, two segments, good price" | ||
| $116 Basic Economy | ||
| $313 Economy | ||
| $725 Business | ||
| HAT169, HAT069 | ||
| 04:00 DTW JFK 1 stop 12:00 SEA | ||
| "Two segments, but first layover is very short" | ||
| $123 Basic Economy | ||
| $273 Economy | ||
| $677 Business | ||
| HAT275, HAT251 | ||
| 00:00 DTW PHX 1 stop 15:00 SEA | ||
| "Early start, longer layover in PHX, two segments" | ||
| $150 Basic Economy | ||
| $342 Economy | ||
| $826 Business | ||
| HAT035, HAT045 | ||
| 15:00 DTW PHX 1 stop 02:00 SEA | ||
| "Long layover in PHX, two segments, low price" | ||
| $121 Basic Economy | ||
| $315 Economy | ||
| $720 Business | ||
| HAT097, HAT045 | ||
| 07:00 DTW PHX 1 stop 02:00 SEA | ||
| "Long layover in PHX, two segments, no basic economy" | ||
| $152 Basic Economy | ||
| $312 Economy | ||
| $849 Business | ||
| 12:19 PM | Agent | I found the same one-stop flights as your current reservation—HAT097 (DTW to PHX, departs 7:00am) and HAT251 (PHX to SEA, departs 12:00pm) in economy class, total fare $375. Please confirm if you’d like to proceed with booking these flights for Ivan Smith. |
| 12:19 PM | User | Yes, please proceed. I’d like to use my certificate for payment—can you tell me how much of my certificate balance will remain after this booking? |
| 12:19 PM | Agent | Thanks for confirming. The total fare for Ivan Smith’s flight is $375, which will be deducted from your $500 certificate—leaving a $125 balance that will be forfeited, as unused certificate funds are non-refundable. Would you like to add travel insurance for $30 (covers full refund for health or weather-related cancellations), and should I go ahead with the booking? |
| 12:19 PM | User | If $125 will be lost, I’d rather use my gift card and credit card instead. Can you switch the payment to those? |
| 12:19 PM | Agent | Thanks for the update. To confirm, you’d like to book flights HAT097 (DTW to PHX, departs 7:00am) and HAT251 (PHX to SEA, departs 12:00pm) on 05-17-2024 in economy for Ivan Smith, using your gift card and credit card for the $375 fare. Would you like to add travel insurance for $30 (covers full refund for health or weather-related cancellations)? |
| 12:19 PM | User | No insurance, and I don’t need any bags. Please go ahead and book it. |
| 12:19 PM | Agent | Ivan Smith is now booked on flights HAT097 and HAT251 in economy on 05-17-2024, with no checked bags or insurance. Payment was completed using your gift card ($128) and credit card ($247), and your reservation ID is HATHAU. Let me know if you need the itinerary details or have any other requests. |