Skip to content

Run on Grid'5000

WARNING: Use Grid'5000 only for approved research and education. Read the usage policy first.

One-time setup on a site frontend

ssh nancy
git clone https://github.com/NoeFlandre/benchmark-llms-landuse-relevance.git
cd benchmark-llms-landuse-relevance
curl -LsSf https://astral.sh/uv/install.sh | sh

Use the frontend only for checkout, submission, and monitoring. Do all heavy work on a reserved node.

Submit

Before every submission, check current usage and the remaining daily allowance:

usagepolicycheck -t
usagepolicycheck -l --sites lille,nancy,lyon

Also inspect pending and running jobs with oarstat -u on every site you plan to use. Count allocated cores times walltime across all sites, then compare that total with the current allowance and your reservation limits. The default queue limits daytime use to two hours of all allocated cores per cluster on workdays, and weekday jobs must not cross 09:00 or 19:00 Europe/Paris. Large jobs belong at night or on weekends. Refresh these checks before each submission or retry. If the allowance, another job, or the allowed window is unclear, wait. Do not resubmit automatically after a failure. Use a special permission only when the official policy documents it and the permission is already granted.

usagepolicycheck -t
scripts/g5k_submit.sh 1:00     # one GPU, one hour
oarstat -u

For a weighted multi-site run, review every remote command first. Then submit the same fixed shard plan:

scripts/g5k_submit_multisite.sh --sites-config scripts/g5k_sites.json --dry-run
scripts/g5k_submit_multisite.sh --sites-config scripts/g5k_sites.json --walltime 1:00

By default, the submission wrappers request the Grid'5000 job type night. This keeps the large sweep in the night and weekend window that the policy permits. The wrappers request a GPU with compute capability 7 or higher and at least 23,040 MiB of memory. This excludes the older GTX 1080 Ti and P100 class. It leaves room for the largest models on the roster.

WARNING: Do not change LRB_JOB_TYPE or LRB_GPU_FILTER before you check the policy, the site inventory, and the runtime compatibility.

The site list is explicit. Each configured weight becomes the share of the deterministic shard slots of that site. The script runs usagepolicycheck -t on every site before it submits. In dry-run mode, the script does not use SSH and does not change OAR.

scripts/g5k_node_run.sh runs on the node. It does these steps:

  1. It syncs the pinned environment.
  2. It validates the 85-language inventory.
  3. It benchmarks each model-language pair of the roster with the active budget of 4096 tokens.

The submission wrappers put the live checkpoints in the separate results-live/ tree. Therefore, the archive-only results of the repository never mix with a collection. The script saves the results as checkpoints at results-live/<language>/<namespace>__<model>.json. If you run the job again, it skips the valid pairs that exist. Therefore, a job that exceeded its walltime resumes. It does not repeat.

To finish one model before the next model, set LRB_MODEL_ID when you submit a wave. The same shard plan then applies only to the languages of that model. The script still skips the valid checkpoints that exist. This works for generative models and for scoring models. The node wrapper sends scoring ids to lrb score. It records the throughput and the peak CUDA VRAM:

LRB_MODEL_ID=Alibaba-NLP/gte-multilingual-reranker-base \
  scripts/g5k_submit_multisite.sh --sites-config scripts/g5k_sites.json --walltime 4:00

GTE and LiquidAI/LFM2.5-Encoder-350M are refused until a decision on their remote code is made (issue #120), so this submission cannot load GTE yet. LiquidAI/d1-3B uses reviewed custom code from its pinned model snapshot and runs through the scoring command:

LRB_MODEL_ID=LiquidAI/d1-3B \
LRB_RESULTS=results-live \
scripts/g5k_submit.sh 2:00

That command selects the full 85-language inventory, or 25,500 sentences. d1's model card lists 16 supported languages, so report scores for the other 69 as outside the stated language coverage. The model needs one GPU with at least 23,040 MiB of memory under the default filter. It uses the isolated Transformers 5.17.0 environment and the pinned model snapshot recorded in each result.

GLiClass Ultra uses model revision 33c3ffec64056b0d37173d2dd90b885ac29b72c3, the data/translations/manifest.json revision d431a6c522b3c3a616e4f3d2814906c3011843e7, and the locked gliclass 0.1.20 and Transformers 5.17.0 packages in a job-isolated environment (Python 3.12, Torch 2.14.0, bfloat16). This command scores all 85 languages (300 rows each, 25,500 cases) with batch size 16 and a 512-token sequence cap:

LRB_MODEL_ID=knowledgator/gliclass-multilang-ultra \
LRB_RESULTS=results-live-gliclass-ultra \
scripts/g5k_submit.sh 2:00

The default GPU filter requires at least 23,040 MiB. At the default bfloat16 dtype, the 1.72B weights occupy about 3.2 GiB before runtime and batch memory. The model card reports 308.2 samples/second for one label at batch size 8 on an RTX PRO 6000 Blackwell, or about 83 seconds of inference for 25,500 cases at that reference rate. That figure is a planning estimate; model download, environment setup, and actual GPU speed add time.

The tokenizer audit at data/audits/gliclass-ultra-token-lengths.csv records the formatted sequence length for each of the 25,500 frozen rows. At the pinned model and prompt revisions, lengths range from 29 to 212 tokens; none exceeds 512. The JSON sidecar records both revisions and the CSV digest. The scorer checks each formatted batch with the same tokenizer and refuses to call the inference pipeline if any sequence would be truncated.

Reproduce the audit after syncing the inference and scoring extras:

uv run --no-sync python scripts/audit_gliclass_ultra_lengths.py

It fetches only the pinned model config and tokenizer files. It does not load model weights or run inference.

Measure the reservation before you start a GPU run:

scripts/g5k_node_run.sh --dry-run

The dry run prints the language count, the total rows, the model-language pair count, the estimated prompts, the batch size, and the token budget. It does not load a model.

The number of pairs is 85 languages times the size of the roster. Each pair has 300 prompts. A focused run of one model is 85 pairs and 25,500 prompts, before the script skips the completed checkpoints. The pair checkpoints are the evidence of completion. A single model file is not.

For many nodes, assign disjoint deterministic shards:

LRB_SHARD_INDEX=0 LRB_SHARD_COUNT=4 scripts/g5k_node_run.sh
LRB_SHARD_INDEX=1 LRB_SHARD_COUNT=4 scripts/g5k_node_run.sh

The node script reports the total pair count and the assigned pair count. Each node uses the same sorted planner. Therefore, the union of the shards is complete and the shards do not overlap. Before you collect a release, inspect the status with uv run lrb status --results-dir results-live.

For arrays on one site, g5k_submit.sh --array COUNT tells OAR to schedule the shards as separate subjobs. Each subjob uses the array index of OAR as its deterministic shard index. Set LRB_MODEL_ID to one model. Use --array 85 for its 85 language checkpoints:

LRB_MODEL_ID=LiquidAI/LFM2.5-2.6B scripts/g5k_submit.sh 2:00 --array 85

Before it loads a model, the CLI prints the selected languages, the prompt count, the assigned pair count, and the state of the checkpoints that it can reuse. Size the walltime from three values: the observed 95th percentile for each language, the model load time, and a safety margin. Refer to Results. Do not reserve a full multi-model sweep on one node.

To see the completion and the OAR state across the configured sites, run:

scripts/g5k_status_multisite.py \
  --sites-config scripts/g5k_sites.json \
  --remote-root /home/nflandre/benchmark-llms-landuse-relevance

The status report gives each planned pair one label: complete, running, queued, failed, or unclaimed. It matches jobs by the configured result path and the deterministic shard. Therefore, other jobs on the same account do not count as benchmark work.

The node wrapper exports USE_TF=0 for the PyTorch loader of Laya. The wrapper loads Laya through its official SDK from the pinned Hub snapshot. It groups the typed noul calls in small shared-state batches. It reports the checkpoint context of 1,024 tokens.

d1 uses its official AutoModel and system_one_batch API, with one typed noul question per sentence. Its custom code loads the tokenizer by the model's local path, so the runner downloads the exact pinned Hub snapshot and passes that local directory to Transformers. The custom code runs with the benchmark process's permissions.

Four models run in an isolated environment for each job (.venv-<name>-$OAR_JOB_ID). An EXIT trap removes it when the job ends. Every other model uses the locked environment:

  • GTE: Transformers 5.11.0. Its remote model code calls an API that the locked Transformers 5.17.0 does not have. The script checks the installed version before it scores.
  • d1: Transformers 5.17.0, which meets the model's >=5.14 requirement. Its code is pinned with the model at 051bcc464b01b9f92942b364d9586b0ef5912432.
  • GLiClass: an extra that the default sync removes. A concurrent job on the shared environment can uninstall it in the middle of a run.
  • GLiNER2: it pins Transformers<5.
  • The GGUF quant: the script builds llama-cpp-python from source with CUDA. The node loads a site cuda-toolkit module. It fails if nvcc is not available. It adds the libraries of the toolkit to LD_LIBRARY_PATH.

On Lyon, the A100 cluster sirius needs -t exotic and -q default. The job type night is not valid there. The wrappers pass -t "$LRB_JOB_TYPE" and do not pass a queue. Therefore, submit there with oarsub directly.

To make a validation run, set LRB_LANGUAGES to a comma-separated subset of languages.

scripts/g5k_both_configs.sh is a compatibility wrapper. It runs g5k_node_run.sh with LRB_MAX_NEW_TOKENS=${HEADLINE_TOKENS:-4096} and LRB_RESULTS=$LRB_ROOT/results. The tests do not exercise it. Use the submission wrappers.

The Hugging Face cache (HF_HOME) is on the node-local scratch /tmp. Grid'5000 wipes it between jobs. The job fetches the weights again. It does not lose the results.

Size the jobs

The table shows the measured cost of one model-language pair. The pair has 300 prompts and a token budget of 4096. SGLang runs in throughput mode with 16 concurrent requests. The values are medians from the sweep of 2026-09-27.

run GPU seconds per language (median / max) 85 languages
LFM2.5-1.2B-Instruct @sglang / +DSpark L40S 1 / 11 ~2 min
LFM2.5-VL-3B (Transformers), @sglang, +DSpark L40S, RTX 6000 Ada 3-4 / 10 ~6 min
LFM2.5-8B-A1B +DSpark RTX 6000 Ada, A6000 110-132 / 214 ~3 h
LFM2.5-8B-A1B @sglang RTX 6000 Ada, L40S 161-192 / 476 ~4.5 h
LFM2.5-2.6B +DSpark L40S, RTX 6000 Ada 208-212 / 421 ~5 h
LFM2.5-2.6B @sglang RTX 6000 Ada, L40S 445-540 / 606 ~11-13 h

The reasoning models use the most time. They think at length before they answer. A pair costs minutes, not seconds. With batch size 1, the 2.6B @sglang run took 966 s for English on an H100 NVL. For this reason, the sweep uses throughput mode (ADR-0011).

The setup costs 3-8 minutes for each job (environment sync, weights, CUDA graphs). Therefore, use walltimes of 1-1.5 h. Split a long run into disjoint LRB_LANGUAGES lists across jobs. The job saves each finished language as a checkpoint. A job that stops at its walltime loses at most the pair in flight.

Keep every environment for each job and every uv cache on the node-local /tmp. Use UV_PROJECT_ENVIRONMENT and UV_CACHE_DIR for each $OAR_JOB_ID. A job that is killed skips its cleanup trap. Two jobs on one node that share a uv cache can block the sync of each other.

Collect and publish

/home is local to the site, has a quota, and has no backup. Copy the results from Grid'5000 as soon as the job ends:

scp -r nancy:benchmark-llms-landuse-relevance/results-live ./results-nancy
scp -r grenoble:benchmark-llms-landuse-relevance/results-live ./results-grenoble
uv run python scripts/g5k_collect.py \
  --site nancy=results-nancy \
  --site grenoble=results-grenoble \
  --output benchmark-runs/merged
uv run lrb report --results-dir benchmark-runs/merged
uv run python scripts/snapshot_status.py benchmark-runs/merged \
  --model-id ... # repeat for every model in the release roster
uv run lrb publish NoeFlandre/benchmark-llms-landuse-relevance \
  --results-dir benchmark-runs/merged

snapshot_status.py rewrites SNAPSHOT_STATUS.md from the merged tree. lrb publish does not change that file. If you skip this step, you publish the previous coverage claim again.

For a scoring wave, pass explicit model ids. You can also pass --scoring-only for the complete scoring roster. The default of the collector is the generative roster:

uv run python scripts/g5k_collect.py \
  --scoring-only \
  --site nancy=results-nancy-scoring \
  --site grenoble=results-grenoble-scoring \
  --output benchmark-runs/scoring

Then release the jobs that are still held:

oardel JOB_ID     # only if the job is still running
usagepolicycheck -t