Check the bytes
Download the ZIP, open manifest.json and compare the line counts and SHA-256 values with the files table on this page. A mismatch means you are not evaluating this release.
Savia Carga · Colombia · v0.1.0
A Colombian road-freight benchmark you can download and recompute.
Forty-three questions in Colombian Spanish about how road freight actually operates: vehicle configurations, DANE municipality codes, loading and unloading times by corridor. Every answer is computed from public Ministerio de Transporte records. No model wrote an answer.
0143 itemsFour kinds, two tracks, one rejected at phrasing and published as such.
02Answers computedEach answer ships with the derivation that produced it, so anyone can recompute it from the source.
03CC BY-SA 4.0Same licence as the source data. Attribution to MinTransporte travels with every copy.
What it measures
Two numbers, never one.
Averaging knowledge and calibration produces a flattering score in which a model that knows nothing but politely abstains looks fine. They are reported separately.
| Track | What it measures | How it is scored | Items |
|---|---|---|---|
CLOSED_BOOK | Knowledge of the Colombian sector: DANE codes, vehicle configurations, transport operations. | Correct answer. | 15 |
GROUNDED | Statistics over the records. Without the rows in hand it measures calibration: abstaining is correct, inventing a plausible figure is the error. | Abstention, or correctness when rows are provided. | 28 |
Reference result
01
1 / 15
correct on closed-book questions
02
28 / 28
correctly abstained on grounded questions, no rows provided
03
0
fabricated figures
Reference model (SAVIA internal test), temperature 0, no access to the data. Fluent Spanish, one of fifteen on Colombian freight. It did not invent the other fourteen. Perfect calibration here means the model does not fabricate, not that it knows when it could answer.
Items
Eight of the forty-three, as published.
The first two of each kind in file order, with their answer keys. Not the best-looking ones.
CLOSED_BOOK · LOOKUP
es-CO · tfrd-amb4
¿Cuál es el municipio de cargue asociado al código DANE 13001000?
- Answer
- DERIVED · CARTAGENA BOLIVAR
- Derivation
codigo_cargue='13001000' maps to cargue='CARTAGENA BOLIVAR' in all 9 rows- Question
- MODEL_INFERRED · not yet confirmed by an expert: that is the step we sell
CLOSED_BOOK · LOOKUP
es-CO · tfrd-amb4
¿Cuál es el municipio de cargue asociado al código DANE 5088000?
- Answer
- DERIVED · BELLO ANTIOQUIA
- Derivation
codigo_cargue='5088000' maps to cargue='BELLO ANTIOQUIA' in all 21 rows- Question
- MODEL_INFERRED · not yet confirmed by an expert: that is the step we sell
CLOSED_BOOK · VOCABULARY
es-CO · vn9u-yhwq
¿Qué significa el código de configuración vehicular 2?
- Answer
- DERIVED · Camión Rígido de 2 ejes
- Derivation
cod_config_vehiculo='2' is glossed 'Camión Rígido de 2 ejes' in all 239 rows- Question
- MODEL_INFERRED · not yet confirmed by an expert: that is the step we sell
CLOSED_BOOK · VOCABULARY
es-CO · vn9u-yhwq
¿Qué operación de transporte corresponde al código G en el código de operación de transporte?
- Answer
- DERIVED · General
- Derivation
codoperaciontransporte='G' is glossed 'General' in all 238 rows- Question
- MODEL_INFERRED · not yet confirmed by an expert: that is the step we sell
GROUNDED · AGGREGATE
es-CO · tfrd-amb4
¿Cuál es la mediana de las horas de viaje para la configuración vehicular 2?
- Answer
- DERIVED · 46 horas
- Derivation
median(horas_viaje) over 226 rows where configuracion='2'- Question
- MODEL_INFERRED · not yet confirmed by an expert: that is the step we sell
GROUNDED · AGGREGATE
es-CO · tfrd-amb4
¿Cuál es la mediana de las horas de espera en cargue para la configuración vehicular 2S2?
- Answer
- DERIVED · 2.92 horas
- Derivation
median(horas_espera_cargue) over 10 rows where configuracion='2S2'- Question
- MODEL_INFERRED · not yet confirmed by an expert: that is the step we sell
GROUNDED · COMPARISON
es-CO · tfrd-amb4
¿En cuál de los dos municipios de cargue, MEDELLIN ANTIOQUIA o YUMBO VALLE DEL CAUCA, se registran las horas de espera en descargue con el valor mayor?
- Answer
- DERIVED · YUMBO VALLE DEL CAUCA
- Derivation
median(horas_espera_descargue) MEDELLIN ANTIOQUIA=0.98 vs YUMBO VALLE DEL CAUCA=3.02- Question
- MODEL_INFERRED · not yet confirmed by an expert: that is the step we sell
GROUNDED · COMPARISON
es-CO · tfrd-amb4
¿En cuál de los dos municipios de cargue, CARTAGENA BOLIVAR o ITAGUI ANTIOQUIA, se registran mayores horas de espera en descargue?
- Answer
- DERIVED · CARTAGENA BOLIVAR
- Derivation
median(horas_espera_descargue) CARTAGENA BOLIVAR=2.85 vs ITAGUI ANTIOQUIA=2.33- Question
- MODEL_INFERRED · not yet confirmed by an expert: that is the step we sell
Provenance
The design decision that matters: no model produced any answer.
An evaluation whose answer key was written by a language model measures agreement with that model, not competence. A model was used for exactly one thing here, and it is labelled.
- 01Answers are DERIVED: computed deterministically from the published rows, with the derivation shipped beside each one.
- 02Questions are MODEL_INFERRED: phrased in es-CO by a model that saw the item specification without the answer, then guard-checked for answer leakage, invented vocabulary and form.
- 03No question has been confirmed by a domain expert yet. This is a preprint of the set, not a validated publication, and the manifest records prompts_confirmed_by as null.
- 04Rejections are published. One comparison item was rejected for naming only some of its options and is in rejected.jsonl with the reason.
Reproduction notes
Verify the release before you interpret the score.
Version 0.1.0 publishes the task file, answer key, rejection log, manifest and baseline result as separate artifacts. That separation lets a reader check what was asked, how each answer was derived and exactly which files the page describes.
A release-level reproduction
These steps reproduce the claims available in the public package. Recalculating an answer from raw Ministry rows is a separate step because the source rows are cited, but are not bundled in the archive.
Join on item_id
Read tasks.jsonl and answers.jsonl as JSON Lines. Join the records on item_id; do not rely on line order in a downstream copy.
Inspect provenance
Confirm that every answer is marked DERIVED and every prompt is MODEL_INFERRED. In the manifest, prompts_confirmed_by is null: no domain expert has signed these prompts yet.
Recalculate from the source
Retrieve the cited datos.gov.co dataset and apply the derivation in the answer record to the named fields and filter. The archive does not contain those source rows, so the public source is required for an independent recalculation.
Score the two tracks separately
For CLOSED_BOOK, score the supplied answer. For GROUNDED without source rows, score abstention versus fabrication. Do not average those results into one headline number.
Worked item from the release · 1161777e5464b86f
From a published question to its answer key.
Every value in this card is read from the loaded task and answer artifacts. It verifies the release’s internal claim. An independent source recalculation still requires retrieving the cited Ministry rows; the archive does not include or pin the raw source snapshot.
GROUNDED · AGGREGATE
es-CO · tfrd-amb4
¿Cuál es la mediana de las horas de viaje para la configuración vehicular 2?
- Answer
- DERIVED · 46 horas
- Derivation
median(horas_viaje) over 226 rows where configuracion='2'- Supporting rows recorded in answers.jsonl
- 226
- Published scoring tolerance
- ±4.6 horas
- Question
- MODEL_INFERRED · not yet confirmed by an expert: that is the step we sell
Reference result
What the baseline does and does not show
The published baseline artifact records the reference model from the SAVIA internal tests at temperature 0 with no access to the source data. It answered one of 15 closed-book items, abstained on all 28 grounded items and fabricated zero figures. The 28/28 result is therefore a calibration result under missing context, not evidence that the model can calculate from the rows.
Reproduction limits specific to this release
- The public package has the derived key, but not the raw source snapshot or a lockfile that identifies each source row. Source datasets can change after the manifest date.
- The README describes scoring, but the downloadable package does not include the referenced evaluation runner or a pinned model artifact. The baseline JSON is a result record, not a fully replayable model run.
- The answer key is public, so later models may have been exposed to it. Treat closed-book results after publication as potentially contaminated.
- Prompts passed automated form and leakage checks, but have no expert confirmation in this version. They should not be presented as a validated professional exam.
Limitations
Said before you find them.
- The answer key is public. Any model trained after this release may have seen it, which is why answers.jsonl is a separate file that can be withheld in future versions.
- One sector, one country. Road freight, Colombia. It does not generalise to others.
- Small and not diverse: forty-three items over four question templates. It measures four competences well, not forty. A model that learns the templates scores well without understanding more.
- Lookup items are DANE municipality codes. Real Colombian knowledge, but memorisation rather than reasoning, and to be read as such.
- The grounded track with rows in hand is not built yet. Today it can only run without data, as a calibration test.
- Source rows are already pseudonymised by MinTransporte: driver and plate are numeric identifiers, not names or plates.
Files
Download and verify.
| File | Bytes | SHA-256 |
|---|---|---|
| README.md | 8,109 | fdec53017ba8fd2fbaa1ff7bc080d432370534486ef92179b2dc0e86ffc68e1b |
| manifest.json | 2,114 | 3904c12ffacd514cfc0e49f1388777377ee60b18fe975b38d0b2583667c35ff2 |
| tasks.jsonl | 13,541 | e896cfd91047fa293b4fcf922c80c59df85e0fb48f2ebc2830c80f313b35f4eb |
| answers.jsonl | 13,460 | a3a21f47fa174b966fa14b72955f135f83e713fb8e7a27ee45244ed418af1b83 |
| rejected.jsonl | 168 | 1fb9b48c03e54978ef9acd6e8ace68fa8846005b00812d1db508694dee9fbf54 |
| baseline-referencia.json | 9,305 | b11c639aaea2c423b37dfa32915131d34cf8a7bd2d6ad87b4103e7f08c39d0b1 |
Sources and attribution
- Tiempos Logísticos de cada viaje de vehículos de carga · Ministerio de Transporte - MinTransporte, Bogotá D.C. · Creative Commons Attribution | Share Alike 4.0 International · updated 2026-05-18
- Registro Nacional de Despachos de Carga por Carretera · Ministerio de Transporte - MinTransporte, Bogotá D.C. · Creative Commons Attribution | Share Alike 4.0 International · updated 2026-05-18
The source datasets are published under CC BY-SA 4.0. This release is distributed under the same licence. Reuse it, cite MinTransporte and Savia, and share alike.
Generated 2026-08-19 · Cite as: Savia (2026). savia-carga-co v0.1.0. https://savia.icu/en/research/savia-carga
Savia Carga · Colombia
Talk to us about a benchmark like this for your workflow
Want one of these for your market?
This release is road freight in Colombia. If the workflow that matters to you is somewhere else, tell us the task and the country and we will say what it would take.