A Colombian road-freight benchmark you can download and recompute.
Forty-three questions in Colombian Spanish about how road freight actually operates: vehicle configurations, DANE municipality codes, loading and unloading times by corridor. Every answer is computed from public Ministerio de Transporte records. No model wrote an answer.
Two numbers, never one.
Averaging knowledge and calibration produces a flattering score in which a model that knows nothing but politely abstains looks fine. They are reported separately.
| Track | What it measures | How it is scored | Items |
|---|---|---|---|
CLOSED_BOOK | Knowledge of the Colombian sector: DANE codes, vehicle configurations, transport operations. | Correct answer. | 15 |
GROUNDED | Statistics over the records. Without the rows in hand it measures calibration: abstaining is correct, inventing a plausible figure is the error. | Abstention, or correctness when rows are provided. | 28 |
1 / 15
correct on closed-book questions
28 / 28
correctly abstained on grounded questions, no rows provided
0
fabricated figures
qwen3.8:27b-mlx, temperature 0, no access to the data. Fluent Spanish, one of fifteen on Colombian freight. It did not invent the other fourteen. Perfect calibration here means the model does not fabricate, not that it knows when it could answer.
Eight of the forty-three, as published.
The first two of each kind in file order, with their answer keys. Not the best-looking ones.
¿Cuál es el municipio de cargue asociado al código DANE 13001000?
- Answer
- DERIVED · CARTAGENA BOLIVAR
- Derivation
codigo_cargue='13001000' maps to cargue='CARTAGENA BOLIVAR' in all 9 rows- Question
- MODEL_INFERRED · not yet confirmed by an expert
¿Cuál es el municipio de cargue asociado al código DANE 5088000?
- Answer
- DERIVED · BELLO ANTIOQUIA
- Derivation
codigo_cargue='5088000' maps to cargue='BELLO ANTIOQUIA' in all 21 rows- Question
- MODEL_INFERRED · not yet confirmed by an expert
¿Qué significa el código de configuración vehicular 2?
- Answer
- DERIVED · Camión Rígido de 2 ejes
- Derivation
cod_config_vehiculo='2' is glossed 'Camión Rígido de 2 ejes' in all 239 rows- Question
- MODEL_INFERRED · not yet confirmed by an expert
¿Qué operación de transporte corresponde al código G en el código de operación de transporte?
- Answer
- DERIVED · General
- Derivation
codoperaciontransporte='G' is glossed 'General' in all 238 rows- Question
- MODEL_INFERRED · not yet confirmed by an expert
¿Cuál es la mediana de las horas de viaje para la configuración vehicular 2?
- Answer
- DERIVED · 46 horas
- Derivation
median(horas_viaje) over 226 rows where configuracion='2'- Question
- MODEL_INFERRED · not yet confirmed by an expert
¿Cuál es la mediana de las horas de espera en cargue para la configuración vehicular 2S2?
- Answer
- DERIVED · 2.92 horas
- Derivation
median(horas_espera_cargue) over 10 rows where configuracion='2S2'- Question
- MODEL_INFERRED · not yet confirmed by an expert
¿En cuál de los dos municipios de cargue, MEDELLIN ANTIOQUIA o YUMBO VALLE DEL CAUCA, se registran las horas de espera en descargue con el valor mayor?
- Answer
- DERIVED · YUMBO VALLE DEL CAUCA
- Derivation
median(horas_espera_descargue) MEDELLIN ANTIOQUIA=0.98 vs YUMBO VALLE DEL CAUCA=3.02- Question
- MODEL_INFERRED · not yet confirmed by an expert
¿En cuál de los dos municipios de cargue, CARTAGENA BOLIVAR o ITAGUI ANTIOQUIA, se registran mayores horas de espera en descargue?
- Answer
- DERIVED · CARTAGENA BOLIVAR
- Derivation
median(horas_espera_descargue) CARTAGENA BOLIVAR=2.85 vs ITAGUI ANTIOQUIA=2.33- Question
- MODEL_INFERRED · not yet confirmed by an expert
The design decision that matters: no model produced any answer.
An evaluation whose answer key was written by a language model measures agreement with that model, not competence. A model was used for exactly one thing here, and it is labelled.
- 01Answers are DERIVED: computed deterministically from the published rows, with the derivation shipped beside each one.
- 02Questions are MODEL_INFERRED: phrased in es-CO by a model that saw the item specification without the answer, then guard-checked for answer leakage, invented vocabulary and form.
- 03No question has been confirmed by a domain expert yet. This is a preprint of the set, not a validated publication, and the manifest records prompts_confirmed_by as null.
- 04Rejections are published. One comparison item was rejected for naming only some of its options and is in rejected.jsonl with the reason.
Said before you find them.
- The answer key is public. Any model trained after this release may have seen it, which is why answers.jsonl is a separate file that can be withheld in future versions.
- One sector, one country. Road freight, Colombia. It does not generalise to others.
- Small and not diverse: forty-three items over four question templates. It measures four competences well, not forty. A model that learns the templates scores well without understanding more.
- Lookup items are DANE municipality codes. Real Colombian knowledge, but memorisation rather than reasoning, and to be read as such.
- The grounded track with rows in hand is not built yet. Today it can only run without data, as a calibration test.
- Source rows are already pseudonymised by MinTransporte: driver and plate are numeric identifiers, not names or plates.
Download and verify.
| File | Bytes | SHA-256 |
|---|---|---|
| README.md | 8,107 | 88b98391a76ba84f7f022353f121fd6aa911252f81489ded56c79c20ad9cbeb2 |
| manifest.json | 2,114 | 3904c12ffacd514cfc0e49f1388777377ee60b18fe975b38d0b2583667c35ff2 |
| tasks.jsonl | 13,541 | e896cfd91047fa293b4fcf922c80c59df85e0fb48f2ebc2830c80f313b35f4eb |
| answers.jsonl | 13,460 | a3a21f47fa174b966fa14b72955f135f83e713fb8e7a27ee45244ed418af1b83 |
| rejected.jsonl | 168 | 1fb9b48c03e54978ef9acd6e8ace68fa8846005b00812d1db508694dee9fbf54 |
| baseline-qwen3.8-27b-mlx.json | 9,296 | edf5e9f53a0f1ecbd8f278b53446b89cd7d25f88e27969acc5f07aba0beb4073 |
Sources and attribution
- Tiempos Logísticos de cada viaje de vehículos de carga · Ministerio de Transporte - MinTransporte, Bogotá D.C. · Creative Commons Attribution | Share Alike 4.0 International · updated 2026-05-18
- Registro Nacional de Despachos de Carga por Carretera · Ministerio de Transporte - MinTransporte, Bogotá D.C. · Creative Commons Attribution | Share Alike 4.0 International · updated 2026-05-18
The source datasets are published under CC BY-SA 4.0. This release is distributed under the same licence. Reuse it, cite MinTransporte and Savia, and share alike.
Generated 2026-08-19 · Cite as: Savia (2026). savia-carga-co v0.1.0. https://savia.icu/en/research/savia-carga