Tool calling in open-source models: who can do it and who just pretends

The article compares the tool-calling capabilities of open-source language models – that is, an LLM's ability to call external APIs and tools via structured requests. In production deployment, Llama 3.3 70B comes out on top with a zero parsing error rate, while Gemma is unsuitable for tool calling and Llama 4's reputation is tarnished by a benchmark scandal. The article explains the principles of tool calling, parallel calls, the formats (OpenAI JSON, Hermes XML, Pythonic) and the MCP standard, and evaluates the models based on the BFCL benchmark.

Imagine a customer types into a search box: "I want a red cabinet for the bedroom, wooden, up to CZK 5,000, with free shipping." A classic search engine handles this only approximately — it returns hundreds of results, most of which fail to meet all the criteria at once. But what if a language model could parse this sentence, call the e-shop's API with precise filters (category: cabinets, material: wood, color: red, price_max: 5000, shipping: free) and return five relevant products straight away?

That is exactly what tool calling is about — the ability of large language models (LLMs) not only to generate text, but also to call external tools, databases and services. And this is precisely where it is decided whether AI stops being a sophisticated chatbot and becomes a genuine assistant that actually gets things done for you.

This article maps which of the freely available (open-source) models truly master this ability — and which only pretend to. It focuses on models relevant for production deployment: the Llama 3.1 family, Llama 3.3 70B, Google Gemma and the controversial Llama 4. It draws on benchmark data from BFCL (Berkeley Function-Calling Leaderboard), the documentation of the vLLM and llama.cpp inference frameworks, GitHub issues and community experience as of February 2026.

Summary for those in a hurry: For production tool calling, the best choice today is Llama 3.3 70B — verified quality, zero parsing error rate, a drop-in replacement for the older Llama 3.1 70B. Gemma is not suitable for tool calling. Llama 4 is unverified and reputationally damaged. If you need parallel tool calling in vLLM, reach for Qwen 2.5 72B (or the newer Qwen 3) or xLAM-2 70B.

A language model without tool calling is like an encyclopedist locked in a library — it knows a great deal, but it cannot look out the window to find out whether it is raining outside. It can answer the question "What will the weather be like tomorrow?", but only in general terms. It cannot connect to a meteorological service and return the current forecast.

Tool calling (sometimes called "function calling") is the model's ability to recognize that answering a question requires external information or an action, and to generate a structured request — typically in JSON format — that the system processes automatically. The model does not execute code directly. Instead, it says: "I need to call the function get_weather with the parameter city='Prague'", and the serving system (the orchestrator) performs this call and returns the result to the model.

In practice, this means the model can search databases, call APIs, run computations, send emails or drive entire workflows — but only if we give it this capability and only if it can handle it reliably.

For companies like Seznam.cz, Alza or Heureka, tool calling represents a fundamental technological leap. Instead of the user entering keywords and browsing results, they can describe their intent in natural language — and based on that intent, the model calls the right API with precise parameters.

A typical scenario: the user enters "a cheap hotel in the Krkonoše mountains for a weekend, for a family with children, with a pool". The model extracts structured parameters from this (location: Krkonoše, dates: nearest weekend, type: family, amenities: pool, sorting: price ascending) and calls the accommodation portal's API. The result is a precise list instead of hundreds of imprecise links.

In practice, the most common use case for tool calling is the RAG (Retrieval-Augmented Generation) pattern — the model calls a search or database tool, obtains relevant data and builds its answer on that basis. Instead of relying solely on knowledge from training, it works with current data from your systems. For e-commerce and vertical search, RAG via tool calling is the de facto standard architecture — the model acting as an intelligent "dispatcher" that knows which API to call and with what parameters.

Parallel tool calling occurs when the model generates multiple calls at once in a single step — that is, it says: "Call the weather API, the restaurant API and the transport API simultaneously." The orchestrator then runs all the calls in parallel rather than sequentially.

The difference in speed is dramatic. If each API call takes 200 milliseconds and you need results from three services, the sequential approach takes 600 ms. The parallel approach still takes only ~200 ms — because all the calls run at the same time. For complex queries that require data from five to ten sources, the parallel approach can cut the response time from several seconds down to a fraction of a second.

Not every query requires parallel calls. Simple questions like "What time is it in Tokyo?" need only a single call. But for complex queries — "Compare flight prices to Barcelona, recommend a hotel near the beach, and find restaurants with a vegetarian menu" — the parallel approach is essential for an acceptable response speed.

The whole process has four steps:

1. Defining the tools. Before the conversation begins, the system tells the model which tools it has available. Each tool has a name, a description and a definition of its parameters — much like API documentation. For example:

Tool: search_products

Description: Searches for products in the e-shop Parameters: - category (text): product category - max_price (number): maximum price in CZK - color (text, optional): color

2. The model's decision. The model receives the user's question and the list of tools. Based on these, it decides: either it answers directly (if it needs no tool), or it generates one or more tool calls with specific parameters.

3. Executing the call. The orchestrator (the software layer between the model and the API) takes the generated call, executes it against the real API and obtains the result.

4. Processing the result. The model receives the results of the call and formulates an answer for the user on that basis — or decides to call another tool (a multi-step workflow).

Different models use different formats for expressing calls. There are three main approaches:

The OpenAI format (the de facto industry standard) uses an array of JSON objects. Each call has a unique identifier, a function name and arguments in JSON:

{ "tool_calls": [ {"id": "call_1", "function": {"name": "get_weather", "arguments": "{\"city\": \"Prague\"}"}}, {"id": "call_2", "function": {"name": "get_weather", "arguments": "{\"city\": \"Brno\"}"}} ] }

The Hermes format (widespread in the open-source world) uses XML tags, where each <tool_call> contains JSON with the function name and arguments. It is used by the Qwen and Functionary models and many others.

The Pythonic format (growing in popularity) writes calls as Python code — [get_weather(city='Prague'), get_weather(city='Brno')]. This format naturally represents parallel calls as a list and is used by Llama 4.

The format itself does not affect quality — what matters is whether the model generates the correct function names, the correct parameters and decides correctly when to call and when not to.

On top of these formats a standardization layer is emerging: MCP (Model Context Protocol), an open standard from Anthropic (November 2024) that defines a uniform way for an LLM to communicate with external tools — from defining tools through calling them to processing results. MCP is spreading rapidly through the ecosystem (it is supported by LangChain, a number of IDEs and, gradually, inference frameworks too) and is heading toward the role of a universal "USB connector" between models and tools. For developers this means: if you expose your tools today via an MCP server, they will work with any model that supports MCP — without having to rewrite the integration for each model separately.

Before we dive into specific models, it is necessary to explain how tool calling quality is measured — because this is precisely where most of the confusion arises.

BFCL (Berkeley Function-Calling Leaderboard) is the most respected benchmark for evaluating tool calling. It tests models across four categories: simple calling (single function), parallel calling of several different functions (parallel multiple), parallel calling of the same function with different parameters (parallel) and calling with missing information, where the model must recognize that it does not have enough data (missing parameters). BFCL has gone through four versions since 2024: V1 and V2 test single-turn scenarios, V3 (September 2024) added multi-turn conversations, and V4 (July 2025) added complex agentic evaluations including web search and memory.

Beware of the versions: Scores from different BFCL versions are not mutually comparable. V3 and V4 are significantly harder than V1/V2 — a model that scores 85% on V1 may drop below 60% on V4. In this article we state, for each score, the BFCL version it comes from.

For orientation: on BFCL V4 (the most recent version) Claude Opus 4.1 scores roughly 70%, Claude Sonnet 4 likewise around 70% and GPT-5 approximately 59%. The best open-source models range between 75–88% on V2, but considerably lower on V4. Anything below 60% on V2 is problematic for production.

The exact sizes:

All three variants share the same architecture (a dense Transformer); they differ in the number of layers and the width of the "memory" (hidden size). The context is 128,000 tokens for all versions — roughly 96,000 words, i.e. an entire book.

What Llama 3.1 brought: Before Llama 3.1 (July 2024), open-source models either did not support tool calling at all, or only experimentally. Llama 3.1 was the first widely available family with native support — Meta added special tokens (markers in the model's "alphabet") for recognizing and formatting tool calls. The model learned when to call, which parameters to use and how to process the response.

Practical performance by size:

Independent testing by the company Braintrust revealed surprising differences between the sizes:

The 8B model failed in 20% of test cases when parsing responses — in other words, every fifth call was syntactically incorrect (missing brackets, bad JSON formatting, incomplete parameters). Meta itself warns: "The 8B model cannot reliably maintain a conversation alongside tool definitions at the same time." It recommends it only for isolated, single-purpose calls — calling one function for one query, without conversation history.

The 70B model achieved a zero parsing error rate in the Braintrust tests — a better result than the 405B and even than the commercial GPT-4o. On BFCL V1 it scores approximately 84.8%, and 77.5% on V2. Based on its own testing, Databricks confirmed that Llama 3.1 70B is surprisingly strong and performs competitively against commercial models.

The 405B model paradoxically had a higher parsing error rate than the 70B — several failures in tests where the 70B succeeded flawlessly. On BFCL V1 it reached 88.5% (the highest of the three), but in practice it turned out that the 70B offers a better quality-to-price ratio.

Key conclusion: For production tool calling, Llama 3.1 70B is a significantly better choice than the 8B (too unreliable) and the 405B (more expensive inference, marginally better).

Parallel tool calling: Llama 3.1 formally supports parallel calls, but with limitations. The vLLM inference framework has no dedicated parser for parallel outputs for the Llama 3.x family. In llama.cpp parallel calling is available, but disabled by default — it must be explicitly enabled with the flag parallel_tool_calls: true. The practical reliability of parallel calling on Llama 3.1 is lower than that of sequential (single) calling.

Llama 3.3 exists exclusively as a 70B model — released in December 2024, no 8B version was ever created. The architecture is identical to Llama 3.1 70B (the same number of layers, the same width), but it underwent significantly better "post-training" — Meta used more advanced techniques of supervised fine-tuning (SFT, learning from examples of correct answers) and RLHF (reinforcement learning from human feedback, learning on the basis of human feedback).

Meta explicitly states: "This model replaces Llama 3.1 70B. Developers should use the new model everywhere they would otherwise use Llama 3.1 70B."

Why it is better for tool calling:

On BFCL V2, Llama 3.3 70B scores 77.3% against 77.5% for Llama 3.1 70B — a difference of 0.2 points is statistically negligible. So why do we claim it is better?

Because BFCL V2 measures only the formal correctness of calls in single-turn scenarios. The real quality of tool calling depends on other factors — and this is where Llama 3.3 dramatically excels:

The improvement in IFEval (instruction following, +4.6 points) directly affects tool calling: the model better adheres to the required output format, fills in parameters more accurately, and decides more reliably when to call a tool and when to answer directly. The improvement in MATH (+9.2 points) means more accurate extraction and processing of numerical parameters — prices, dates, quantities. The improvement in HumanEval (+7.9 points) shows up in the quality of the generated JSON.

The company Groq (an operator of inference infrastructure) confirms that Llama 3.3 70B comes surprisingly close in quality to the larger Llama 3.1 405B — a model one-fifth the size with similar quality.

Framework support: Llama 3.3 70B uses the same tool calling format as Llama 3.1 — the llama3_json parser in vLLM, llama3 in SGLang, native support in llama.cpp. Migrating from Llama 3.1 70B to 3.3 70B requires no changes to code or configuration — it is a "drop-in replacement" (a replacement without modifications).

Parallel tool calling has the same limitations as in Llama 3.1 — unsupported in vLLM for the 3.x family, available in llama.cpp via an explicit flag.

Llama 4 is a story about what happens when marketing pressure gets ahead of engineering reality.

Architecture — MoE instead of dense:

Llama 4 brought a fundamental architectural change. Whereas Llama 3.x are "dense" models (all parameters are activated for every token), Llama 4 uses Mixture of Experts (MoE) — the model contains many "expert" modules, but when processing each token it activates only a fraction of them:

The advantage of MoE: inference (generating responses) is fast and cheap, because only the active parameters are computed. The disadvantage: training is more complex and the model may behave unpredictably — different experts can "specialize" in different tasks unevenly.

What actually happened — the benchmark scandal:

Meta released Llama 4 Scout and Maverick on 5 April 2025 — unusually, on a Saturday. On LMArena (formerly Chatbot Arena, the most prestigious community ranking of models) it uploaded a special "experimental" version of Maverick that had been optimized for human evaluators — it was chattier, used emoji and was designed to "charm" users in blind voting. This version finished in 2nd place on the leaderboard.

When LMArena added the standard public version of Maverick (the one anyone could download from HuggingFace), it finished in 32nd place — behind GPT-4o, Claude 3.5 Sonnet and Gemini 1.5 Pro.

An anonymous post on a Chinese forum from a supposed ex-employee of Meta claimed that management had mixed test data from the benchmarks into the model's training. Ahmad Al-Dahle, vice president of Meta GenAI, denied it.

Definitive confirmation came from Yann LeCun himself — Meta's chief AI scientist, a pioneer of convolutional neural networks and a Turing Award laureate. In November 2025 LeCun announced his departure from Meta and the founding of his own startup, Advanced Machine Intelligence Labs (AMI Labs), focused on "world models". In an interview for the Financial Times, published in early January 2026, he then said that the benchmark results of Llama 4 "were a little bit falsified" — the team used different models for different benchmarks in order to achieve better results. LeCun added that Mark Zuckerberg had lost confidence in the team responsible and subsequently carried out a fundamental reorganization of the AI division — including the hiring of Alexandr Wang, then head of Scale AI, as Meta's new Chief AI Officer as part of an investment of roughly 14 billion dollars in Scale AI.

Real performance:

Community tests showed a yawning gap between the declared and the actual performance. On the Aider polyglot coding benchmark, Maverick reached a mere 16%. Rootly AI Labs measured Maverick's accuracy as the lowest of all tested models (69.5%) on the GMCQ benchmark, based on real GitHub issues — a worse result than older models including Llama 3.3 70B. The declared 10-million-token context of Scout turned out to be "virtual" — community tests show significant degradation in quality well below this threshold. One of the independent tests recorded only 15.6% accuracy for Scout at 128K tokens, while Gemini 2.5 Pro reaches over 90% at the same length.

Tool calling in Llama 4:

For Llama 4, Meta switched to the pythonic call format (instead of JSON in Llama 3.x). In vLLM there is a llama4_pythonic parser, in SGLang a llama4 parser. Parallel tool calling is natively supported. However, on the BFCL leaderboard there are no scores for Llama 4 Scout or Maverick — the models were either not tested or the results were not published. The practical quality of tool calling thus remains unverified.

Recommendation: Llama 4 is not unusable, but for tool calling it represents an unproven choice. If you are considering it, test thoroughly on your own use cases and compare it with Llama 3.3 70B, which has a verified track record.

Google Gemma is a family of open models derived from the commercial Gemini. Gemma 3 27B, released in March 2025, added multimodality (image processing) and expanded the context to 128,000 tokens.

Why Gemma fails at tool calling:

The core of the problem is simple: Gemma models have no dedicated tokens for tool calling. What does that mean?

Models like Llama 3.1 were trained with an expanded "alphabet" — in addition to ordinary words and characters, they contain special markers (tokens) such as <|python_tag|> or <|eom_id|>, which signal to the model: "Now I am generating a tool call" and "I have finished the call." Thanks to these markers, the model unambiguously distinguishes ordinary text from a structured call — and the inference framework knows exactly where the call begins and ends.

Gemma has nothing of the sort. Google officially states: "Gemma does not output any special token for a tool call. Your framework must detect the call by checking the structure of the output." In practice this means that tool calling works only via prompt engineering — you insert the function definitions and the required output format into the prompt, hope the model generates the correct JSON, and then try to parse the output.

The consequences are fundamental:

The absence of a native parser in frameworks. In vLLM there is no official parser for Gemma — feature requests on GitHub have remained without an official implementation. In SGLang there is likewise no Gemma parser.

Unreliable parallel calling. Without native tokens and training on a tool calling format, the model cannot reliably generate multiple calls at once.

Low BFCL score. Gemma 3 27B achieves significantly lower scores on BFCL than models with native support — about 18 points less than Llama 3.3 70B.

FunctionGemma — a specialized variant. In December 2025 Google released FunctionGemma — a model of 270 million parameters derived from Gemma 3, with its own tokens for tool calling. On the "Mobile Actions" evaluation it reaches 58% accuracy in the base version and up to 85% after domain-specific fine-tuning, but its size limits it to edge scenarios (mobile devices, IoT). For server deployment it is too small.

Conclusion: Gemma models are not recommended for tool calling in production. They are excellent models for other tasks (text generation, image analysis in Gemma 3), but without native tool calling support and framework integration they represent an unnecessary risk.

A model on its own is just a set of numbers (the weights of a neural network). In order to answer queries and call tools, it needs an inference framework — software that loads the model into GPU memory, processes queries and formats the outputs. The choice of framework fundamentally affects whether tool calling works correctly.

vLLM (Virtual Large Language Model) is the most widespread framework for production deployment. It offers high performance thanks to techniques such as PagedAttention (efficient GPU memory management) and continuous batching (simultaneous processing of multiple queries). For tool calling it provides dedicated parsers for various model families:

Launching with tool calling requires specific flags:

vllm serve meta-llama/Llama-3.3-70B-Instruct --enable-auto-tool-choice --tool-call-parser llama3_json --chat-template tool_chat_template_llama3.1_json.jinja

A crucial detail: vLLM does not support parallel tool calling for the Llama 3.x family — even though the model is capable of generating multiple calls, the parser cannot process them. For Llama 4 and Qwen, parallel calling works.

llama.cpp is a framework optimized for running on ordinary hardware — it supports CPU inference, smaller GPUs and quantized models (shrunken models with lower precision but dramatically smaller memory requirements). For tool calling you need to run the server with specific flags:

llama-server --model model.gguf --jinja -fa

Parallel tool calling is disabled by default. It is enabled by adding "parallel_tool_calls": true to the request. Quality depends on the model — llama.cpp supports Llama 3.1/3.3, Qwen, Mistral and Llama 4.

A known problem: regressions occur between versions of llama.cpp — one user reported a drop in tool calling success rate from 10/10 to 2/10 after an update. For production deployment it is advisable to pin a specific version.

SGLang offers high performance and tool calling support for Qwen, Llama 3.x, Llama 4, Mistral and DeepSeek. The parsers are llama3 for the Llama family and llama4 for Llama 4. Known problems include occasional errors in tool indexing and instability under high load.

If an inference framework does not have a dedicated parser for a given model, tool calling either does not work at all (the framework does not recognize that the model wants to call a tool) or works unreliably (a generic parser misidentifies the boundaries of calls, the parameters or the function names). This is why the combination of model + framework + parser is crucial — it is not enough for the model to "know how to do" tool calling; the framework must also know how to process it.

It is precisely this fragmentation that MCP addresses — if your inference stack supports MCP, you define the tools once and switch models without changes on the tool side. In practice, this so far works mainly with commercial APIs (Claude, GPT); for self-hosted open-source models, MCP support is still in its early stages.

What do these numbers mean in practice?

A BFCL score of 77.3% (V2) does not mean that every fourth call fails. The benchmark tests a broad spectrum of scenarios, from trivial to extremely complex. In production, where the tool definitions are consistent and the queries predictable, the success rate tends to be higher. On the other hand, benchmarks do not measure edge cases specific to your domain — which is why your own testing is essential.

It is important to distinguish the benchmark versions: on BFCL V4, which includes multi-turn and agentic scenarios, the scores are significantly lower for all models — even commercial ones. Claude Opus 4.1 reaches roughly 70% on V4, GPT-5 around 59%. Open-source models will inevitably be even lower on V4.

More sophisticated benchmarks like τ-bench (tau-bench), which test entire agentic workflows (the model must carry out a series of steps to reach a goal in a simulated customer-support environment), show considerably lower numbers. GPT-4o reaches below 50% success on τ-bench (specifically ~61% on the retail domain and ~35% on the airline domain). The pass^8 metric (success across 8 consecutive attempts at the same task) falls below 25% in the retail domain — which means that only in a quarter of cases does the model successfully and consistently complete the entire workflow.

For tool calling, this implies: a simple, isolated call (call one function with clear parameters) works reliably. Complex, multi-step workflows (call function A, based on the result call B or C, combine the results and call D) are still unreliable across all models.

Gemma for tool calling — the structural handicap (the absence of FC tokens, the missing framework support) cannot be compensated by prompt engineering at a production level of reliability.

Llama 3.1 8B for anything more complex — a 20% parsing failure rate and a Meta-confirmed inability to combine conversation with tool calling instructions.

Llama 4 without your own testing — the missing BFCL data and the benchmark scandal create too much uncertainty.

For most production scenarios this is the safest starting combination. If you do not need parallel tool calling (most simple use cases do not require it), you get verified quality with minimal risk.

If your use case requires parallel calling of multiple tools at once (e.g. multi-vertical search) and you use vLLM, consider Qwen 2.5 72B (the hermes parser) or Salesforce xLAM-2 70B (the xlam parser). Both models have verified support for parallel TC in vLLM.

No benchmark replaces testing on your specific tools, parameters and user queries. Create a test set of at least 100 examples covering common as well as edge-case scenarios, and measure the success rate on it. If you are building a RAG pipeline, test the entire chain end-to-end — from the user query through parameter extraction and the call to the retrieval API, all the way to the quality of the final answer. An error in any step degrades the overall result.

Even the best model occasionally fails. Design the system with fallback logic — if the model generates an invalid call, fall back to simple keyword search or ask the user to clarify.

The model, the inference framework and the chat template — pin specific versions in production. An update to any of them can cause a regression in tool calling quality (a documented problem with llama.cpp).

The market for open-source tool calling models is evolving rapidly. Salesforce xLAM-2 70B (a model specialized for function calling) reached a BFCL score comparable to commercial models. Qwen 3 from Alibaba strengthens parallel tool calling. DeepSeek V3 brings an MoE architecture with good tool calling support.

Meta is developing successors to its current models: a text LLM with the code name Avocado (a successor to the Llama family, focused on coding and tool orchestration) and a multimodal model Mango (image and video generation). Both are planned for the first half of 2026 and are being created within the new division Meta Superintelligence Labs under the leadership of Alexandr Wang. If the speculation about abandoning the open-weight approach is confirmed — Avocado could be the first proprietary model from Meta — the open-source ecosystem will lose one of its main contributors. This may accelerate fragmentation: instead of one dominant family (Llama), the market will be divided among Qwen, xLAM, Mistral and others.

For organizations planning to deploy tool calling, this means: invest in abstraction layers (LiteLLM, LangChain) and standards such as MCP, which allow easy switching between models, and do not plan a long-term dependence on a single model family. Moreover, MCP is becoming the natural interface for RAG architectures — tools for search, database queries and calls to external APIs can be exposed as MCP servers and dynamically offered to any model.

Methodological note

This article draws on the BFCL leaderboard (V1–V4), the official documentation of vLLM and SGLang, GitHub issues and discussions, HuggingFace model cards, independent evaluations (Braintrust, Databricks, Rootly AI Labs, Groq) and media reports (Financial Times, TechCrunch, The Register, VentureBeat). The data is current as of February 2026. Benchmark scores may change with new versions of models and evaluation sets. Practical reliability depends on the specific use case, configuration and framework version.

The BFCL scores given in the article come from different versions of the benchmark — for each number the relevant version is stated. Scores from different versions are not directly comparable, because newer versions added more demanding test categories (multi-turn, agentic).

Transparency of creation:

The conception, structure and editorial line of the article are the work of the author, who prepared the content outline, established the key theses and directed the entire creative process. Generative AI (Claude, Anthropic) was used as a technical tool for research, fact-checking and fleshing out the author's draft.

The author edited the outputs throughout, verified the key findings and approved the final wording. No part of the text was published without human review. All factual data were verified against the publicly available sources cited in the text.

The procedure complies with the requirements of Art. 50 of EU Regulation 2024/1689 (the AI Act) on the transparency of AI-generated content. #poweredByAI