Skip to content
gunkslab

Tutorial 08 · 25 min

Scenarios

A scenario is a small file that sets up a situation and says what must happen: where the bots start, where they’re sent, and what counts as success. The runner plays it out on the real simulator and station code. Behaviour like “two bots mustn’t run into each other” is written down this way and tested on every change.

Do 07. Send a bot somewhere first.

1. What’s there

scripts/scenario list
around-two-parked      Around two parked
                       sim-3 is sent straight between two parked bots too close together to pass between. It has to go round the outside of them. First found by the stress test (seed 1, layout 17), and laid out again for gl_zero's body.
at-the-edge            At the edge
                       A bot sent to a point 12 cm from the desk's edge, facing it. Head on, its sensor would see the desk end first and stop it short, so the station lines it up and it comes in at a slant.
between-two-parked     Between two parked
                       sim-3 is sent between two bots parked 32 cm apart, both facing the way it's going. It fits between them, with room to spare, if it goes straight through.
box-beside             Box beside the path
                       A box just clear of a bot's path. The sensor mustn't take it for an obstacle.
box-in-the-way         Box in the way
                       A box the station knows nothing about, square in a bot's path. The bot's own sensor finds it, and the station goes round.
crossing               Crossing
                       One bot crossing the desk left to right, another bottom to top, timed to reach the crossing together.
detour-by-the-edge     Detour by the edge
                       sim-2's straight line passes just on the far side of a parked bot. Going round that side is shorter, but it's close to the desk's edge, so sim-2 goes round the near side. First found by the stress test (seed 2, layout 93), and laid out again for gl_zero's body.
four-corners           Four corners
                       Four bots swap corners diagonally, all at once. Every path crosses the middle.
head-on                Head-on
                       Two bots on the same line, sent towards each other's start. They meet in the middle.
jobs-a-bot-joins       A bot joins
                       One bot and two jobs, the second coming in a second after the first. 3 seconds in, a second bot is put on the desk.
jobs-a-bot-leaves      A bot leaves mid-job
                       Two bots, one job. sim-1, the nearer, takes it, and is taken off the desk 3 seconds in, before it gets there.
jobs-six-for-three     Six jobs, three bots
                       Three bots parked along one side of a bigger desk, and six jobs coming in over 15 seconds, spread across it.
jobs-while-busy        A job while busy
                       Two bots and two jobs. The first job comes in nearest sim-2; the second comes in a second later, nearer sim-2's start than sim-1, while sim-2 is on its way to the first.
low-box-beside-the-path Low box beside the path  (expected to fail: the sensor can't see a low box at the side of the bot's path: a hardware limit)
                       A 2 cm box whose corner reaches about 1 cm into a bot's path, at the flares.
low-boxes-in-a-row     Low boxes in a row
                       Two 2 cm boxes side by side. A bot goes round the first and passes close by the second, which it sees only at the edge of its view.
one-bot                One bot, one trip
                       A single bot sent across the desk. The baseline. If this fails, nothing else means anything.
parked-in-the-way      Parked in the way
                       A bot sitting still in the middle of another's path. The moving one has to go round it.
round-a-wall           Round a wall
                       Three boxes side by side across a bot's way, 30 cm of wall the station knows nothing about, ending 35 cm short of the desk's near edge: room for gl_zero to go round.
side-by-side           Side by side
                       Two bots in lanes 22 cm apart, both crossing the desk. They never meet, so this should pass today.
someone-crosses-the-way-round Someone crosses the way round  (expected to fail: the station treats what a sensor finds as standing still: it goes round the hand on the side the hand is moving to)
                       As Someone crosses, but the hand moves the other way, from the far side to the near side: towards the side the station goes round on.
someone-crosses        Someone crosses
                       A bot sent across the desk while a hand, 8 cm square and 20 cm tall, moves across its path from the near side to the far side.
three-boxes-staggered  Three boxes staggered
                       A bot sent diagonally across the desk, past two boxes stacked in its way and a third further on.
three-way              Three-way
                       Three bots sent through the middle of the desk from three sides, one a moment after the others.
too-near-the-edge      Too near the edge
                       A bot sent to a point 6 cm from the desk's edge. Its body won't fit there, so the station refuses at once.
turn-beside-a-low-box  Turn beside a low box
                       A bot starts with a 2 cm box just ahead and to its left, and has to turn to go round it.

Before the station kept bots out of each other’s way (09. Keeping out of the way), crossing, head-on and parked-in-the-way were marked as expected to fail. The station drove each bot straight at its goal, knowing nothing about the others, and they ran into each other. They were the problem written down, ready for the fix. three-way and four-corners came with the fix, to test it on more than the cases it was built for. box-in-the-way, box-beside and at-the-edge came with the bots’ distance sensor (10. The distance sensor), and round-a-wall with the sensor read as five regions. too-near-the-edge came when the station learned to bring a bot in to a point near the edge at a slant, and at-the-edge changed with it: it used to end blocked. detour-by-the-edge and around-two-parked came from the stress test (section 5): failures it found, kept once they were fixed, and laid out again when the bots grew to gl_zero’s size. The jobs- scenarios came with jobs, and someone-crosses and someone-crosses-the-way-round with things that move (section 4).

Scenarios has each one with a picture of where the bots start, what it teaches and what it checks.

2. Run them

scripts/scenario run
around-two-parked      PASS  (15.9 s simulated in 0.33 s)
    score: finished in 15.9 s, 0.98 m driven, 1 near miss, stops: none, 0.1 s standing still on the way
    closest sim-1 / sim-2: 0.220 m between centres
    closest sim-1 / sim-3: 0.334 m between centres
    closest sim-2 / sim-3: 0.144 m between centres
at-the-edge            PASS  (13.3 s simulated in 0.05 s)
    score: finished in 13.3 s, 0.58 m driven, 0 near misses, stops: none, 0.0 s standing still on the way
between-two-parked     PASS  (11.5 s simulated in 0.12 s)
    score: finished in 11.5 s, 0.77 m driven, 0 near misses, stops: none, 0.0 s standing still on the way
    closest sim-1 / sim-2: 0.320 m between centres
    closest sim-1 / sim-3: 0.160 m between centres
    closest sim-2 / sim-3: 0.160 m between centres
box-beside             PASS  (11.5 s simulated in 0.04 s)
    score: finished in 11.5 s, 0.77 m driven, 1 near miss, stops: none, 0.0 s standing still on the way
box-in-the-way         PASS  (16.2 s simulated in 0.22 s)
    score: finished in 16.2 s, 0.97 m driven, 0 near misses, stops: none, 0.0 s standing still on the way
crossing               PASS  (15.1 s simulated in 0.13 s)
    score: finished in 15.1 s, 1.22 m driven, 0 near misses, stops: none, 7.3 s standing still on the way
    closest sim-1 / sim-2: 0.300 m between centres
detour-by-the-edge     PASS  (15.5 s simulated in 0.16 s)
    score: finished in 15.5 s, 0.92 m driven, 1 near miss, stops: none, 0.1 s standing still on the way
    closest sim-1 / sim-2: 0.121 m between centres
four-corners           PASS  (33.0 s simulated in 5.60 s)
    score: finished in 33.0 s, 3.81 m driven, 0 near misses, stops: 1 obstacle, 39.2 s standing still on the way
    closest sim-1 / sim-2: 0.211 m between centres
    closest sim-1 / sim-3: 0.249 m between centres
    closest sim-1 / sim-4: 0.403 m between centres
    closest sim-2 / sim-3: 0.282 m between centres
    closest sim-2 / sim-4: 0.269 m between centres
    closest sim-3 / sim-4: 0.216 m between centres
head-on                PASS  (15.6 s simulated in 0.28 s)
    score: finished in 15.6 s, 1.72 m driven, 0 near misses, stops: none, 4.1 s standing still on the way
    closest sim-1 / sim-2: 0.215 m between centres
jobs-a-bot-joins       PASS  (10.4 s simulated in 0.07 s)
    score: finished in 10.4 s, 2 of 2 jobs done, 2.0 s waiting for a bot, 1.00 m driven, 0 near misses, stops: none, 0.0 s standing still on the way
    closest sim-1 / sim-2: 0.300 m between centres
jobs-a-bot-leaves      PASS  (14.1 s simulated in 0.06 s)
    score: finished in 14.1 s, 1 of 1 jobs done, 0.0 s waiting for a bot, 0.99 m driven, 0 near misses, stops: none, 0.0 s standing still on the way
    closest sim-1 / sim-2: 0.300 m between centres
jobs-six-for-three     PASS  (31.7 s simulated in 0.40 s)
    score: finished in 31.7 s, 6 of 6 jobs done, 4.8 s waiting for a bot, 3.64 m driven, 0 near misses, stops: none, 3.9 s standing still on the way
    closest sim-1 / sim-2: 0.250 m between centres
    closest sim-1 / sim-3: 0.221 m between centres
    closest sim-2 / sim-3: 0.250 m between centres
jobs-while-busy        PASS  (8.6 s simulated in 0.07 s)
    score: finished in 8.6 s, 2 of 2 jobs done, 0.0 s waiting for a bot, 0.68 m driven, 0 near misses, stops: none, 0.0 s standing still on the way
    closest sim-1 / sim-2: 0.377 m between centres
low-box-beside-the-path FAIL, as expected  (30.0 s simulated in 0.73 s)
    score: didn't finish (sim-1 still going), 0.27 m driven, 1 near miss, stops: none, 26.8 s standing still on the way
    sim-1 didn't arrive (still going at the time limit)
    sim-1 and box-1 collided at 3.2 s, near x 0.47 y 0.20
    known: the sensor can't see a low box at the side of the bot's path: a hardware limit
low-boxes-in-a-row     PASS  (11.6 s simulated in 0.06 s)
    score: finished in 11.6 s, 0.74 m driven, 1 near miss, stops: none, 0.0 s standing still on the way
one-bot                PASS  (12.6 s simulated in 0.05 s)
    score: finished in 12.6 s, 0.87 m driven, 0 near misses, stops: none, 0.0 s standing still on the way
parked-in-the-way      PASS  (13.4 s simulated in 0.12 s)
    score: finished in 13.4 s, 0.85 m driven, 1 near miss, stops: none, 0.0 s standing still on the way
    closest sim-1 / sim-2: 0.154 m between centres
round-a-wall           PASS  (14.4 s simulated in 0.20 s)
    score: finished in 14.4 s, 0.89 m driven, 0 near misses, stops: none, 0.0 s standing still on the way
side-by-side           PASS  (11.5 s simulated in 0.09 s)
    score: finished in 11.5 s, 1.54 m driven, 0 near misses, stops: none, 0.0 s standing still on the way
    closest sim-1 / sim-2: 0.220 m between centres
someone-crosses-the-way-round FAIL, as expected  (18.0 s simulated in 0.28 s)
    score: finished in 18.0 s, 0.90 m driven, 1 near miss, stops: 1 obstacle, 0.3 s standing still on the way
    sim-1 and hand collided at 5.8 s, near x 0.47 y 0.39
    known: the station treats what a sensor finds as standing still: it goes round the hand on the side the hand is moving to
someone-crosses        PASS  (18.3 s simulated in 0.15 s)
    score: finished in 18.3 s, 0.99 m driven, 1 near miss, stops: 1 obstacle, 0.0 s standing still on the way
three-boxes-staggered  PASS  (8.2 s simulated in 0.16 s)
    score: didn't finish (sim-1's trip ended: blocked), 0.34 m driven, 0 near misses, stops: none, 0.1 s standing still on the way
three-way              PASS  (20.6 s simulated in 0.28 s)
    score: finished in 20.6 s, 2.08 m driven, 0 near misses, stops: 1 obstacle, 6.8 s standing still on the way
    closest sim-1 / sim-2: 0.214 m between centres
    closest sim-1 / sim-3: 0.227 m between centres
    closest sim-2 / sim-3: 0.177 m between centres
too-near-the-edge      PASS  (0.0 s simulated in 0.00 s)
    score: didn't finish (sim-1's trip ended: edge), 0.00 m driven, 0 near misses, stops: none, 0.0 s standing still on the way
turn-beside-a-low-box  PASS  (15.3 s simulated in 0.12 s)
    score: finished in 15.3 s, 0.77 m driven, 0 near misses, stops: 1 obstacle, 0.2 s standing still on the way
  • 15.9 seconds of driving in 0.33 seconds. The runner doesn’t use the broker or the clock. It wires the simulator’s bots and the station together in one process and steps time forward itself. So the same scenario plays out exactly the same way every time.
  • A collision is two bodies touching, using each bot’s real outline (147.5 × 107.5 mm for gl_zero). A simulated bot stops against a box or another bot, as against the desk’s edge, and pushes nothing. It can still back off, and turn once its corners have room. Nothing on it feels the bump, so it keeps trying: in low-box-beside-the-path it sits against the box it can’t see until the time runs out, its wheels turning. A hand or a toy that moves onto a bot doesn’t hold it, so it can drive out.
  • The score line puts numbers on how the run went, pass or fail. How long until every bot sent somewhere had arrived: a trip that ended blocked, refused or timed out isn’t finished, and the line says how it ended. How far the bots drove between them. Near misses: two bodies coming within 3 cm without touching, the room the station tries to keep, counted once each time it happens. Stops: a bot stopping itself, at an edge, at an obstacle, at an E-STOP or on a flat battery, counted as the lab’s Reports count them. And the seconds bots spent standing still between being sent and finishing, mostly giving way to each other. A change to the station that still passes can still make these worse, and the score shows it.
  • Every message between the bots and the station is checked against the protocol on the way. This caught a real bug while the runner was being built. A bot facing exactly backwards reported a heading of 3.1416, just past the protocol’s limit of π.

To run only one or two: scripts/scenario run head-on one-bot. For a program to read the results, add --json: every scenario’s result and score, bot by bot, in place of the text (scripts/scenario run --json > results.json).

3. Watch one

The runner can also play a scenario at real speed and publish it to the broker, so the page shows it. Start the broker and the page, then:

scripts/broker up
scripts/viewer up
scripts/scenario watch crossing
watching crossing: open http://127.0.0.1:8080/?fleet=scenario&desk=1.2x0.8

Open that address within 3 seconds. The runner waits that long before starting. You’ll see each bot with its goal cross and a dashed line to it. One waits while the other goes by:

The crossing scenario: sim-2 waits while sim-1 crosses in front of it

09 shows what the page draws for that.

It runs as its own fleet, scenario, so it doesn’t disturb the desk fleet. Watching is view-only. The page’s drive panel can’t reach the runner’s station. When it finishes, it prints the result and clears everything it left on the broker.

4. Write your own

Make a file in scenarios/, say scenarios/my-three-way.toml (three-way.toml is already there, and this is a simpler version of it):

name = "My three-way"
about = "Three bots sent through the middle of the desk from three sides."

[[bot]]
id = "sim-1"
start = [0.2, 0.25, 0]      # x, y in metres; heading in degrees, anticlockwise from +x (0 faces right)
goto = [1.0, 0.25]

[[bot]]
id = "sim-2"
start = [0.6, 0.1, 90]      # 90 faces up the desk
goto = [0.6, 0.57]

[[bot]]
id = "sim-3"
start = [1.0, 0.6, 180]     # 180 faces left
goto = [0.2, 0.25]
at_s = 1.5                  # sent 1.5 s after the others

Then:

scripts/scenario run my-three-way

What a file can say:

Field Meaning Default
name, about What it is, for list the file’s name
desk [width, depth] in metres [1.2, 0.8]
time_limit_s When to give up 30
expect_fail Known to fail, and why. Take it out once it passes none
[[bot]] id, start A bot, and where it starts: [x, y, heading in degrees]
goto, at_s Where it’s sent, and when. Leave goto out for a bot that sits still sent at 0 s
join_s, leave_s When it’s put on the desk and connects, and when it’s taken off, its link dropped there from the start, and stays
[[box]] at, size, height, id A box on the desk: its centre [x, y], [width, depth], its height, a name 5 cm tall, box-1, …
[[mover]] path, size, height, speed, at_s, stay, id Something crossing the desk: its path [[x, y], ...] (it may start and end off the desk), [width, depth], its height, metres a second, when it sets off, whether it stays at the end, a name 20 cm tall, 0.1 m/s, sets off at 0 s, gone at the end, mover-1, …
[[job]] at, at_s, id A job: a point [x, y] some bot must go to, when it comes in, a name comes in at 0 s, job-1, …
[expect] arrive Which bots must arrive every bot with a goto and no end
[expect] end Gotos that must end another way, e.g. { sim-1 = "blocked" } none
[expect] done Which jobs must be done every job
[expect] no_collisions Whether bodies may touch, bots or boxes true: they may not
[expect] on_desk Whether a bot may reach the desk’s edge (on a real desk it would go over) true: it may not

A mistake in a file stops the run and says which file and what, for example scenario: t.toml: bot sim-1: start is [x, y, heading in degrees].

Jobs

A bot’s goto says which bot goes where. A job only says where: some bot must go there, and which one is chosen while the scenario runs. Give bots no goto and the scenario jobs instead:

[[bot]]
id = "sim-1"
start = [0.25, 0.3, 0]

[[bot]]
id = "sim-2"
start = [0.95, 0.3, 90]

[[job]]
at = [0.95, 0.58]

[[job]]
at = [0.7, 0.3]
at_s = 1                    # comes in a second after the first

That’s jobs-while-busy. The runner hands jobs out by a plain rule: the oldest job waiting goes to the nearest bot with nothing to do. Here the first job goes to sim-2, the nearer. The second is nearer sim-2’s start too, but sim-2 is busy, so sim-1 takes it rather than it waiting. A bot that has done a job is free for the next one. A job whose trip ends any other way than arriving (blocked, refused at an edge, timed out) isn’t done, and isn’t tried again.

The rule is one function, nearest_free in scenarios/gunkscen/world.py. world.run(scenario, dispatch=...) takes any other function shaped like it: given the jobs waiting, the bots that are free and where every bot is, it says which job goes to which bot. The station still plans every path and keeps the bots’ reflexes either way. Only who goes where changes. Two rules run on the same scenarios can then be compared by their scores: with jobs, the score line also says how many were done and how long they waited for a bot.

Bots joining and leaving, and things that move

A fleet changes while it works. A bot with join_s isn’t on the desk until then: it connects and the camera sees it, and it’s free for jobs from that moment. A bot with leave_s is taken off the desk then: its link drops, and the station hears it go offline, as the broker would tell it. Its trip ends offline, and if it was doing a job, the job goes back to waiting for another bot (jobs-a-bot-joins, jobs-a-bot-leaves).

A [[mover]] is something crossing the desk: a hand, a toy. Like a box, the station knows nothing about it and only the bots’ sensors can find it. Unlike a box, it moves along its path at its speed, and is gone once it gets to the end. Touching one counts as a collision.

someone-crosses and someone-crosses-the-way-round are the same but for which way the hand goes. The first passes. The second is expected to fail, and says why. The station plans round what a sensor found as if it stood still, so it can turn a bot to go round on the side the hand is heading to. That’s a limit of today’s station, kept as a scenario, so a change that fixes it shows up as PASSES NOW.

5. The stress test

Scenarios are cases someone thought of. The stress test makes up the rest: two to four bots, random starts, random goals, some sent a moment after the others, 200 layouts at a time.

scripts/scenario stress
200 layouts, 458 bots (seed 1): 0 collided or reached the edge; 2 bots didn't finish (0.4%, limit 5%)
  • It fails on any collision, any bot reaching the desk’s edge, or a protocol error. Those must never happen.
  • A bot that doesn’t finish is counted, not failed, up to 5% of them (it’s been under 3%). In a crowd, a bot that times out or is held back by its reflex has done the cautious thing. The count shows whether a change made things better or worse.
  • The same seed makes the same layouts, and each plays out the same way. --seed 2 is 200 different ones. --layouts 1000 runs 1000.
  • A failure prints its layout as a scenario file. Save it in scenarios/, and from then on it’s a scenario like the others.

Goals and starts stay 22 cm in from the edges, and 24 cm apart. A bot facing an edge stops about 20 cm from it (10. The distance sensor).

6. How CI treats them

Every merge request runs every scenario.

  • PASS: fine.
  • FAIL, as expected: fine. It’s marked expect_fail, and the reason is printed.
  • FAIL: CI fails. A scenario that used to pass has broken.
  • The stress test runs too, as its own job.
  • PASSES NOW: CI fails too, on purpose. Something fixed it, so take expect_fail out of its file and make the pass permanent.
  • Protocol errors fail a scenario whatever it expected.

scripts/check runs the same, on your machine.

How it’s built: the codebase.

Check yourself

  1. Why can a scenario take 15 seconds of driving and run in a tenth of a second? (2. Run them)
  2. If two bots did collide, would their trips stop there? Why? (2)
  3. A scenario marked expect_fail starts passing. What does CI say, and what do you do? (6. How CI treats them)
  4. Write a scenario where a bot is sent to exactly where another is parked. Should it be expect_fail? (4)
  5. The stress test says 4 bots didn’t finish. Does CI fail? What would make it fail? (5. The stress test)

Stuck?

  • Run ./gunkslab doctor. It checks the container runtime and the ports, and prints FIX with what to do.
  • Troubleshooting has the common problems and their fixes.
  • Known issues says what’s untested or doesn’t work yet.
  • Still stuck? Open an issue in desk-issues with what you typed, what you saw, and the output of ./gunkslab doctor.