Tutorial 08 · 25 min
Scenarios
A scenario is a small file that sets up a situation and says what must happen: where the bots start, where they’re sent, and what counts as success. The runner plays it out on the real simulator and station code. Behaviour like “two bots mustn’t run into each other” is written down this way and tested on every change.
Do 07. Send a bot somewhere first.
1. What’s there
scripts/scenario list
around-two-parked Around two parked
sim-3 is sent straight between two parked bots too close together to pass between. It has to go round the outside of them. First found by the stress test (seed 1, layout 17), and laid out again for gl_zero's body.
at-the-edge At the edge
A bot sent to a point 12 cm from the desk's edge, facing it. Head on, its sensor would see the desk end first and stop it short, so the station lines it up and it comes in at a slant.
between-two-parked Between two parked
sim-3 is sent between two bots parked 32 cm apart, both facing the way it's going. It fits between them, with room to spare, if it goes straight through.
box-beside Box beside the path
A box just clear of a bot's path. The sensor mustn't take it for an obstacle.
box-in-the-way Box in the way
A box the station knows nothing about, square in a bot's path. The bot's own sensor finds it, and the station goes round.
crossing Crossing
One bot crossing the desk left to right, another bottom to top, timed to reach the crossing together.
detour-by-the-edge Detour by the edge
sim-2's straight line passes just on the far side of a parked bot. Going round that side is shorter, but it's close to the desk's edge, so sim-2 goes round the near side. First found by the stress test (seed 2, layout 93), and laid out again for gl_zero's body.
four-corners Four corners
Four bots swap corners diagonally, all at once. Every path crosses the middle.
head-on Head-on
Two bots on the same line, sent towards each other's start. They meet in the middle.
jobs-a-bot-joins A bot joins
One bot and two jobs, the second coming in a second after the first. 3 seconds in, a second bot is put on the desk.
jobs-a-bot-leaves A bot leaves mid-job
Two bots, one job. sim-1, the nearer, takes it, and is taken off the desk 3 seconds in, before it gets there.
jobs-six-for-three Six jobs, three bots
Three bots parked along one side of a bigger desk, and six jobs coming in over 15 seconds, spread across it.
jobs-while-busy A job while busy
Two bots and two jobs. The first job comes in nearest sim-2; the second comes in a second later, nearer sim-2's start than sim-1, while sim-2 is on its way to the first.
low-box-beside-the-path Low box beside the path (expected to fail: the sensor can't see a low box at the side of the bot's path: a hardware limit)
A 2 cm box whose corner reaches about 1 cm into a bot's path, at the flares.
low-boxes-in-a-row Low boxes in a row
Two 2 cm boxes side by side. A bot goes round the first and passes close by the second, which it sees only at the edge of its view.
one-bot One bot, one trip
A single bot sent across the desk. The baseline. If this fails, nothing else means anything.
parked-in-the-way Parked in the way
A bot sitting still in the middle of another's path. The moving one has to go round it.
round-a-wall Round a wall
Three boxes side by side across a bot's way, 30 cm of wall the station knows nothing about, ending 35 cm short of the desk's near edge: room for gl_zero to go round.
side-by-side Side by side
Two bots in lanes 22 cm apart, both crossing the desk. They never meet, so this should pass today.
someone-crosses-the-way-round Someone crosses the way round (expected to fail: the station treats what a sensor finds as standing still: it goes round the hand on the side the hand is moving to)
As Someone crosses, but the hand moves the other way, from the far side to the near side: towards the side the station goes round on.
someone-crosses Someone crosses
A bot sent across the desk while a hand, 8 cm square and 20 cm tall, moves across its path from the near side to the far side.
three-boxes-staggered Three boxes staggered
A bot sent diagonally across the desk, past two boxes stacked in its way and a third further on.
three-way Three-way
Three bots sent through the middle of the desk from three sides, one a moment after the others.
too-near-the-edge Too near the edge
A bot sent to a point 6 cm from the desk's edge. Its body won't fit there, so the station refuses at once.
turn-beside-a-low-box Turn beside a low box
A bot starts with a 2 cm box just ahead and to its left, and has to turn to go round it.
Before the station kept bots out of each other’s way (09. Keeping out of the way), crossing,
head-on and parked-in-the-way were marked as expected to fail. The station drove each bot straight at its goal,
knowing nothing about the others, and they ran into each other. They were the problem written down, ready for the
fix. three-way and four-corners came with the fix, to test it on more than the cases it was built for.
box-in-the-way, box-beside and at-the-edge came with the bots’ distance sensor
(10. The distance sensor), and round-a-wall with the sensor read as five regions.
too-near-the-edge came when the station learned to bring a bot in to a point near the edge at a slant, and
at-the-edge changed with it: it used to end blocked.
detour-by-the-edge and around-two-parked came from the stress test (section 5): failures it found, kept once
they were fixed, and laid out again when the bots grew to gl_zero’s size. The jobs- scenarios came with jobs, and
someone-crosses and someone-crosses-the-way-round with things that move (section 4).
Scenarios has each one with a picture of where the bots start, what it teaches and what it checks.
2. Run them
scripts/scenario run
around-two-parked PASS (15.9 s simulated in 0.33 s)
score: finished in 15.9 s, 0.98 m driven, 1 near miss, stops: none, 0.1 s standing still on the way
closest sim-1 / sim-2: 0.220 m between centres
closest sim-1 / sim-3: 0.334 m between centres
closest sim-2 / sim-3: 0.144 m between centres
at-the-edge PASS (13.3 s simulated in 0.05 s)
score: finished in 13.3 s, 0.58 m driven, 0 near misses, stops: none, 0.0 s standing still on the way
between-two-parked PASS (11.5 s simulated in 0.12 s)
score: finished in 11.5 s, 0.77 m driven, 0 near misses, stops: none, 0.0 s standing still on the way
closest sim-1 / sim-2: 0.320 m between centres
closest sim-1 / sim-3: 0.160 m between centres
closest sim-2 / sim-3: 0.160 m between centres
box-beside PASS (11.5 s simulated in 0.04 s)
score: finished in 11.5 s, 0.77 m driven, 1 near miss, stops: none, 0.0 s standing still on the way
box-in-the-way PASS (16.2 s simulated in 0.22 s)
score: finished in 16.2 s, 0.97 m driven, 0 near misses, stops: none, 0.0 s standing still on the way
crossing PASS (15.1 s simulated in 0.13 s)
score: finished in 15.1 s, 1.22 m driven, 0 near misses, stops: none, 7.3 s standing still on the way
closest sim-1 / sim-2: 0.300 m between centres
detour-by-the-edge PASS (15.5 s simulated in 0.16 s)
score: finished in 15.5 s, 0.92 m driven, 1 near miss, stops: none, 0.1 s standing still on the way
closest sim-1 / sim-2: 0.121 m between centres
four-corners PASS (33.0 s simulated in 5.60 s)
score: finished in 33.0 s, 3.81 m driven, 0 near misses, stops: 1 obstacle, 39.2 s standing still on the way
closest sim-1 / sim-2: 0.211 m between centres
closest sim-1 / sim-3: 0.249 m between centres
closest sim-1 / sim-4: 0.403 m between centres
closest sim-2 / sim-3: 0.282 m between centres
closest sim-2 / sim-4: 0.269 m between centres
closest sim-3 / sim-4: 0.216 m between centres
head-on PASS (15.6 s simulated in 0.28 s)
score: finished in 15.6 s, 1.72 m driven, 0 near misses, stops: none, 4.1 s standing still on the way
closest sim-1 / sim-2: 0.215 m between centres
jobs-a-bot-joins PASS (10.4 s simulated in 0.07 s)
score: finished in 10.4 s, 2 of 2 jobs done, 2.0 s waiting for a bot, 1.00 m driven, 0 near misses, stops: none, 0.0 s standing still on the way
closest sim-1 / sim-2: 0.300 m between centres
jobs-a-bot-leaves PASS (14.1 s simulated in 0.06 s)
score: finished in 14.1 s, 1 of 1 jobs done, 0.0 s waiting for a bot, 0.99 m driven, 0 near misses, stops: none, 0.0 s standing still on the way
closest sim-1 / sim-2: 0.300 m between centres
jobs-six-for-three PASS (31.7 s simulated in 0.40 s)
score: finished in 31.7 s, 6 of 6 jobs done, 4.8 s waiting for a bot, 3.64 m driven, 0 near misses, stops: none, 3.9 s standing still on the way
closest sim-1 / sim-2: 0.250 m between centres
closest sim-1 / sim-3: 0.221 m between centres
closest sim-2 / sim-3: 0.250 m between centres
jobs-while-busy PASS (8.6 s simulated in 0.07 s)
score: finished in 8.6 s, 2 of 2 jobs done, 0.0 s waiting for a bot, 0.68 m driven, 0 near misses, stops: none, 0.0 s standing still on the way
closest sim-1 / sim-2: 0.377 m between centres
low-box-beside-the-path FAIL, as expected (30.0 s simulated in 0.73 s)
score: didn't finish (sim-1 still going), 0.27 m driven, 1 near miss, stops: none, 26.8 s standing still on the way
sim-1 didn't arrive (still going at the time limit)
sim-1 and box-1 collided at 3.2 s, near x 0.47 y 0.20
known: the sensor can't see a low box at the side of the bot's path: a hardware limit
low-boxes-in-a-row PASS (11.6 s simulated in 0.06 s)
score: finished in 11.6 s, 0.74 m driven, 1 near miss, stops: none, 0.0 s standing still on the way
one-bot PASS (12.6 s simulated in 0.05 s)
score: finished in 12.6 s, 0.87 m driven, 0 near misses, stops: none, 0.0 s standing still on the way
parked-in-the-way PASS (13.4 s simulated in 0.12 s)
score: finished in 13.4 s, 0.85 m driven, 1 near miss, stops: none, 0.0 s standing still on the way
closest sim-1 / sim-2: 0.154 m between centres
round-a-wall PASS (14.4 s simulated in 0.20 s)
score: finished in 14.4 s, 0.89 m driven, 0 near misses, stops: none, 0.0 s standing still on the way
side-by-side PASS (11.5 s simulated in 0.09 s)
score: finished in 11.5 s, 1.54 m driven, 0 near misses, stops: none, 0.0 s standing still on the way
closest sim-1 / sim-2: 0.220 m between centres
someone-crosses-the-way-round FAIL, as expected (18.0 s simulated in 0.28 s)
score: finished in 18.0 s, 0.90 m driven, 1 near miss, stops: 1 obstacle, 0.3 s standing still on the way
sim-1 and hand collided at 5.8 s, near x 0.47 y 0.39
known: the station treats what a sensor finds as standing still: it goes round the hand on the side the hand is moving to
someone-crosses PASS (18.3 s simulated in 0.15 s)
score: finished in 18.3 s, 0.99 m driven, 1 near miss, stops: 1 obstacle, 0.0 s standing still on the way
three-boxes-staggered PASS (8.2 s simulated in 0.16 s)
score: didn't finish (sim-1's trip ended: blocked), 0.34 m driven, 0 near misses, stops: none, 0.1 s standing still on the way
three-way PASS (20.6 s simulated in 0.28 s)
score: finished in 20.6 s, 2.08 m driven, 0 near misses, stops: 1 obstacle, 6.8 s standing still on the way
closest sim-1 / sim-2: 0.214 m between centres
closest sim-1 / sim-3: 0.227 m between centres
closest sim-2 / sim-3: 0.177 m between centres
too-near-the-edge PASS (0.0 s simulated in 0.00 s)
score: didn't finish (sim-1's trip ended: edge), 0.00 m driven, 0 near misses, stops: none, 0.0 s standing still on the way
turn-beside-a-low-box PASS (15.3 s simulated in 0.12 s)
score: finished in 15.3 s, 0.77 m driven, 0 near misses, stops: 1 obstacle, 0.2 s standing still on the way
- 15.9 seconds of driving in 0.33 seconds. The runner doesn’t use the broker or the clock. It wires the simulator’s bots and the station together in one process and steps time forward itself. So the same scenario plays out exactly the same way every time.
- A collision is two bodies touching, using each bot’s real outline (147.5 × 107.5 mm for gl_zero). A
simulated bot stops against a box or another bot, as against the desk’s edge, and pushes nothing. It can still
back off, and turn once its corners have room. Nothing on it feels the bump, so it keeps trying: in
low-box-beside-the-pathit sits against the box it can’t see until the time runs out, its wheels turning. A hand or a toy that moves onto a bot doesn’t hold it, so it can drive out. - The score line puts numbers on how the run went, pass or fail. How long until every bot sent somewhere had arrived: a trip that ended blocked, refused or timed out isn’t finished, and the line says how it ended. How far the bots drove between them. Near misses: two bodies coming within 3 cm without touching, the room the station tries to keep, counted once each time it happens. Stops: a bot stopping itself, at an edge, at an obstacle, at an E-STOP or on a flat battery, counted as the lab’s Reports count them. And the seconds bots spent standing still between being sent and finishing, mostly giving way to each other. A change to the station that still passes can still make these worse, and the score shows it.
- Every message between the bots and the station is checked against the protocol on the way. This caught a real bug while the runner was being built. A bot facing exactly backwards reported a heading of 3.1416, just past the protocol’s limit of π.
To run only one or two: scripts/scenario run head-on one-bot. For a program to read the results, add --json:
every scenario’s result and score, bot by bot, in place of the text (scripts/scenario run --json > results.json).
3. Watch one
The runner can also play a scenario at real speed and publish it to the broker, so the page shows it. Start the broker and the page, then:
scripts/broker up
scripts/viewer up
scripts/scenario watch crossing
watching crossing: open http://127.0.0.1:8080/?fleet=scenario&desk=1.2x0.8
Open that address within 3 seconds. The runner waits that long before starting. You’ll see each bot with its goal cross and a dashed line to it. One waits while the other goes by:
09 shows what the page draws for that.
It runs as its own fleet, scenario, so it doesn’t disturb the desk fleet. Watching is view-only. The page’s
drive panel can’t reach the runner’s station. When it finishes, it prints the result and clears everything it left
on the broker.
4. Write your own
Make a file in scenarios/, say scenarios/my-three-way.toml (three-way.toml is already there, and this
is a simpler version of it):
name = "My three-way"
about = "Three bots sent through the middle of the desk from three sides."
[[bot]]
id = "sim-1"
start = [0.2, 0.25, 0] # x, y in metres; heading in degrees, anticlockwise from +x (0 faces right)
goto = [1.0, 0.25]
[[bot]]
id = "sim-2"
start = [0.6, 0.1, 90] # 90 faces up the desk
goto = [0.6, 0.57]
[[bot]]
id = "sim-3"
start = [1.0, 0.6, 180] # 180 faces left
goto = [0.2, 0.25]
at_s = 1.5 # sent 1.5 s after the others
Then:
scripts/scenario run my-three-way
What a file can say:
| Field | Meaning | Default |
|---|---|---|
name, about |
What it is, for list |
the file’s name |
desk |
[width, depth] in metres |
[1.2, 0.8] |
time_limit_s |
When to give up | 30 |
expect_fail |
Known to fail, and why. Take it out once it passes | none |
[[bot]] id, start |
A bot, and where it starts: [x, y, heading in degrees] |
|
goto, at_s |
Where it’s sent, and when. Leave goto out for a bot that sits still |
sent at 0 s |
join_s, leave_s |
When it’s put on the desk and connects, and when it’s taken off, its link dropped | there from the start, and stays |
[[box]] at, size, height, id |
A box on the desk: its centre [x, y], [width, depth], its height, a name |
5 cm tall, box-1, … |
[[mover]] path, size, height, speed, at_s, stay, id |
Something crossing the desk: its path [[x, y], ...] (it may start and end off the desk), [width, depth], its height, metres a second, when it sets off, whether it stays at the end, a name |
20 cm tall, 0.1 m/s, sets off at 0 s, gone at the end, mover-1, … |
[[job]] at, at_s, id |
A job: a point [x, y] some bot must go to, when it comes in, a name |
comes in at 0 s, job-1, … |
[expect] arrive |
Which bots must arrive | every bot with a goto and no end |
[expect] end |
Gotos that must end another way, e.g. { sim-1 = "blocked" } |
none |
[expect] done |
Which jobs must be done | every job |
[expect] no_collisions |
Whether bodies may touch, bots or boxes | true: they may not |
[expect] on_desk |
Whether a bot may reach the desk’s edge (on a real desk it would go over) | true: it may not |
A mistake in a file stops the run and says which file and what, for example
scenario: t.toml: bot sim-1: start is [x, y, heading in degrees].
Jobs
A bot’s goto says which bot goes where. A job only says where: some bot must go there, and which one is chosen
while the scenario runs. Give bots no goto and the scenario jobs instead:
[[bot]]
id = "sim-1"
start = [0.25, 0.3, 0]
[[bot]]
id = "sim-2"
start = [0.95, 0.3, 90]
[[job]]
at = [0.95, 0.58]
[[job]]
at = [0.7, 0.3]
at_s = 1 # comes in a second after the first
That’s jobs-while-busy. The runner hands jobs out by a plain rule: the oldest job waiting goes to the nearest
bot with nothing to do. Here the first job goes to sim-2, the nearer. The second is nearer sim-2’s start too, but
sim-2 is busy, so sim-1 takes it rather than it waiting. A bot that has done a job is free for the next one. A
job whose trip ends any other way than arriving (blocked, refused at an edge, timed out) isn’t done, and isn’t
tried again.
The rule is one function, nearest_free in scenarios/gunkscen/world.py. world.run(scenario, dispatch=...)
takes any other function shaped like it: given the jobs waiting, the bots that are free and where every bot is,
it says which job goes to which bot. The station still plans every path and keeps the bots’ reflexes either
way. Only who goes where changes. Two rules run on the same scenarios can then be compared by their scores: with
jobs, the score line also says how many were done and how long they waited for a bot.
Bots joining and leaving, and things that move
A fleet changes while it works. A bot with join_s isn’t on the desk until then: it connects and the camera
sees it, and it’s free for jobs from that moment. A bot with leave_s is taken off the desk then: its link drops,
and the station hears it go offline, as the broker would tell it. Its trip ends offline, and if it was doing a
job, the job goes back to waiting for another bot (jobs-a-bot-joins, jobs-a-bot-leaves).
A [[mover]] is something crossing the desk: a hand, a toy. Like a box, the station knows nothing about it and
only the bots’ sensors can find it. Unlike a box, it moves along its path at its speed, and is gone once it gets
to the end. Touching one counts as a collision.
someone-crosses and someone-crosses-the-way-round are the same but for which way the hand goes. The first
passes. The second is expected to fail, and says why. The station plans round what a sensor found as if it stood
still, so it can turn a bot to go round on the side the hand is heading to. That’s a limit of today’s station,
kept as a scenario, so a change that fixes it shows up as PASSES NOW.
5. The stress test
Scenarios are cases someone thought of. The stress test makes up the rest: two to four bots, random starts, random goals, some sent a moment after the others, 200 layouts at a time.
scripts/scenario stress
200 layouts, 458 bots (seed 1): 0 collided or reached the edge; 2 bots didn't finish (0.4%, limit 5%)
- It fails on any collision, any bot reaching the desk’s edge, or a protocol error. Those must never happen.
- A bot that doesn’t finish is counted, not failed, up to 5% of them (it’s been under 3%). In a crowd, a bot that times out or is held back by its reflex has done the cautious thing. The count shows whether a change made things better or worse.
- The same seed makes the same layouts, and each plays out the same way.
--seed 2is 200 different ones.--layouts 1000runs 1000. - A failure prints its layout as a scenario file. Save it in
scenarios/, and from then on it’s a scenario like the others.
Goals and starts stay 22 cm in from the edges, and 24 cm apart. A bot facing an edge stops about 20 cm from it (10. The distance sensor).
6. How CI treats them
Every merge request runs every scenario.
- PASS: fine.
- FAIL, as expected: fine. It’s marked
expect_fail, and the reason is printed. - FAIL: CI fails. A scenario that used to pass has broken.
- The stress test runs too, as its own job.
- PASSES NOW: CI fails too, on purpose. Something fixed it, so take
expect_failout of its file and make the pass permanent. - Protocol errors fail a scenario whatever it expected.
scripts/check runs the same, on your machine.
How it’s built: the codebase.
Check yourself
- Why can a scenario take 15 seconds of driving and run in a tenth of a second? (2. Run them)
- If two bots did collide, would their trips stop there? Why? (2)
- A scenario marked
expect_failstarts passing. What does CI say, and what do you do? (6. How CI treats them) - Write a scenario where a bot is sent to exactly where another is parked. Should it be
expect_fail? (4) - The stress test says 4 bots didn’t finish. Does CI fail? What would make it fail? (5. The stress test)
Stuck?
- Run
./gunkslab doctor. It checks the container runtime and the ports, and printsFIXwith what to do. - Troubleshooting has the common problems and their fixes.
- Known issues says what’s untested or doesn’t work yet.
- Still stuck? Open an issue in desk-issues with what you typed,
what you saw, and the output of
./gunkslab doctor.
