Modernisation of the Squirrel Script API

[victor_18993]

Simutrans already exposes a substantial part of the engine through the Squirrel Script API, but while working with it I have found some places where information already available in the C++ engine is either lost or only partially exposed to scripts.
Rather than redesigning the Script API, I would like to use this thread for small, incremental and backwards-compatible improvements where such gaps can be clearly demonstrated.
First change:

Code Select Expand

way_planner_x.get_step_cost()

Code Select Expand

way_builder_t::is_allowed_step() already does two things internally:

• determines whether a step is allowed;
• calculates the pathfinding weight for that step.

However, the existing

Code Select Expand

way_planner_x.is_allowed_step() binding only exposes the boolean result and discards the calculated weight.
As a result,

Code Select Expand

sqai and

Code Select Expand

sqai_rail currently have to estimate these weights again in

Code Select Expand

astar.nut.
This patch adds:

Code Select Expand

way_planner_x.get_step_cost(from, to)It returns:

• an integer containing the pathfinding weight already calculated by
  
  Code Select Expand
  way_builder_t;
• Code Select Expand
  null when the step is rejected or cannot be resolved.

This is a pathfinding weight, not a construction price.
The change is completely additive.

Code Select Expand

is_allowed_step() is unchanged, existing scripts do not need any modification, and this patch does not modify

Code Select Expand

sqai,

Code Select Expand

sqai_rail or

Code Select Expand

astar.nut.
Five new

Code Select Expand

way_planner_x tests have been added, covering:

• untouched terrain versus an existing way;
• slopes;
• rejected steps returning
  
  Code Select Expand
  null;
• unusual/invalid inputs;
• consistency between
  
  Code Select Expand
  get_step_cost() and
  
  Code Select Expand
  is_allowed_step().

Test results:

• 5/5 new tests passed
• previous suite: 238/238
• total: 243/243
• Code Select Expand
  none and SDL2 builds successful
• no new compiler warnings

Patch attached:

Code Select Expand

squirrel-api-01-step-cost.diffI would like to keep any further changes in this area similarly small and independent, so each improvement can be reviewed on its own merits.

[victor_18993]

The descriptor initialization fix is now in trunk as r12144.

Code Select Expand

way_builder_t now initializes

Code Select Expand

desc,

Code Select Expand

bridge_desc and

Code Select Expand

tunnel_desc to

Code Select Expand

NULL, so an unconfigured

Code Select Expand

way_planner_x now behaves deterministically instead of depending on uninitialized memory.
For an unconfigured planner:

• Code Select Expand
  is_allowed_step() returns
  
  Code Select Expand
  false
• Code Select Expand
  get_step_cost() returns
  
  Code Select Expand
  null

Configured builders are unchanged.
A regression test was added for this case.
Final validation:

• 244/244 tests passed
• Code Select Expand
  none and SDL2 builds successful
• no new compiler warnings

This fix is now complete in trunk.

[victor_18993]

The tram/elevated planner fix is now in trunk as r12145.

Code Select Expand

way_planner_x.set_build_types() now preserves the same tram and elevated way-type semantics as the normal way tool:

• tram descriptors use
  
  Code Select Expand
  schiene_tram;
• elevated descriptors use
  
  Code Select Expand
  elevated_flag;
• air ways and runways remain non-elevated.

This does not change the Script API signature, and existing road and rail scripts are unaffected.
Final validation:

• 247/247 tests passed, including three new
  
  Code Select Expand
  way_planner_x regression tests
• Code Select Expand
  none and SDL2 builds successful
• no new compiler warnings

This change is now complete in trunk.

[victor_18993]

The tunnel planner addition is now in trunk as r12146.
A new

Code Select Expand

tunnel_planner_x.find_end() method is now available to Squirrel scripts. It exposes the existing native tunnel endpoint search without building anything.

• returns the far tunnel portal when one can be found;
• returns an invalid coordinate on failure;
• accepts only a single direction;
• uses the same unrestricted search length as the native tunnel endpoint search;
• does not expose
  
  Code Select Expand
  full_tunnel or the currently incomplete tunnel error messages.

This is an additive Script API change. It does not modify the existing tunnel tool or the AI implementations.
Final validation:

• 252/252 tests passed, including 5 new tunnel planner regression tests;
• endpoint parity with the normal tunnel tool tested;
• rotated-map behaviour tested;
• Code Select Expand
  none and SDL2 builds successful;
• no new compiler warnings.

This gives scripts a native way to query a tunnel endpoint before deciding whether to build it.

[victor_18993]

API-04A is now in trunk as r12149.

Code Select Expand

command_x.build_bridge_at() now accepts an optional

Code Select Expand

max_length parameter:

Code Select Expand

command_x.build_bridge_at(pl, pos, bridge_desc [, max_length])

The existing 3-argument behaviour is unchanged and still defaults to a maximum search length of 10.

• Code Select Expand
  max_length > 0 uses the supplied limit;
• Code Select Expand
  max_length <= 0 searches as far as the bridge descriptor and
  
  Code Select Expand
  way_max_bridge_len allow;
• Code Select Expand
  min_length remains internal;
• flat ends are not enabled by this change.

The Script API search is capped at 254 tiles because the native bridge endpoint search uses an 8-bit counter internally.
Final validation:

• 256/256 tests passed;
• 4 new API-04A regression tests;
• 17/17 tests in
  
  Code Select Expand
  test_way_bridge.nut;
• 27/27 across the bridge test family;
• API-03 tunnel and way planner regressions still pass;
• headless and SDL2 builds successful;
• no new compiler warnings.

This is an additive API change. Existing scripts using the 3-argument form retain the previous behaviour.

[victor_18993]

EXEC-05 has now been integrated in r12150.

Code Select Expand

command_x.build_way() and

Code Select Expand

command_x.build_road() now accept an optional trailing

Code Select Expand

terraform argument:

command_x.build_way(pl, start, end, way, straight [, terraform])
command_x.build_road(pl, start, end, way, straight, keep_city_roads [, terraform])

The default remains

Code Select Expand

false, so existing scripts keep exactly the previous behaviour.
When

Code Select Expand

terraform=true, the command may use the native way-builder slope adjustment while constructing the way. This does not enable automatic bridges or tunnels.
The change also preserves the terraform policy through network tool execution, so the same command semantics are used on the authoritative side in multiplayer.
Regression coverage was added for the legacy/default behaviour, slope crossing,

Code Select Expand

build_road, adjacent-step construction, scenario restrictions, no automatic bridge construction, and tool-state reuse.
Full automated suite after the change: 264/264 passed.
This completes the executor side of the terraforming work. The next step will be the corresponding

Code Select Expand

way_planner_x change, so scripts can plan and execute using the same policy.

[victor_18993]

API-05 planner terraforming has now been integrated in r12151.

Code Select Expand

way_planner_x.set_build_types() now accepts an optional trailing

Code Select Expand

terraform parameter:

set_build_types(way [, terraform])

The default remains

Code Select Expand

false, so existing scripts keep the previous behaviour unchanged.
When enabled, the planner considers the same slope-changing capability exposed to

Code Select Expand

command_x.build_way() /

Code Select Expand

build_road() in r12150. This means scripts and AIs can now plan across certain slopes that the native way builder can handle through terraforming, instead of discovering that capability only when attempting construction.
The existing way-type composition is preserved, including tram and elevated ways, and enabling terraforming does not enable bridges or tunnels.
The planner remains a planning/query interface: an accepted step is not a guarantee that a later construction command will succeed, since the executor may still reject the operation for other reasons.
Regression coverage was added for:

• legacy behaviour with the parameter omitted or
  
  Code Select Expand
  false;
• slope cases requiring terraforming;
• planner/executor parity on the same geometry;
• ensuring terraforming does not bypass unrelated restrictions;
• non-mutating planner queries;
• clearing the flag when the planner is reconfigured;
• preserving tram/elevated way types.

The complete automated suite passes 271/271.
This closes the planner/executor terraforming work. The next Script API work will focus on auditing how failures and error results are currently exposed to scripts before proposing any further API changes.

[Andarix]

I believe the function 'command_x::build_bridge_at()' is no longer usable in its current form, since Simutrans no longer builds bridges with just a single click.

Are you also incorporating your changes into the Doxygen documentation?

[victor_18993]

Yes, I understand what you mean now.

build_bridge_at() still works by taking a starting tile, finding a suitable end and then building the bridge, but your point is whether that is still the right API for a Script AI now that bridge construction is naturally defined by two endpoints.

We already have bridge_planner_x.find_end() and command_x.build_bridge(start, end, ...), so I will review this part specifically and check which interface should be considered the proper one for scripts rather than assuming that the old one-click abstraction is still the best choice.

And yes, I am updating the Doxygen documentation together with the API changes. I want the final API behaviour and its documentation to stay in sync.

[victor_18993]

A small update on the Script API work.
Two more changes have now been integrated into trunk:

• r12152 clarifies the Doxygen documentation for
  
  Code Select Expand
  command_x return values and the planner/executor boundary. In particular, tool calls return
  
  Code Select Expand
  null on success and a string on failure, including the possibility of an empty string when no useful message is available.
• r12153 fixes
  
  Code Select Expand
  get_tool_key(), which was returning Ctrl and Shift prefixes reversed for some keys. This was reproduced at runtime before changing the code and then verified again after the fix.

I am now looking at the next part from the point of view of actual Script AI usage, especially the terrain/route-building problems mentioned by Andarix.
The terraform support added in r12150/r12151 gives scripts the ability to ask the planner to consider slope changes and to let the executor perform them, but the current Script AIs do not use those new arguments yet.
One interesting finding from the current audit is that the C++

Code Select Expand

way_builder_t already has a fairly clean separation between calculating a route, calculating its monetary cost and actually building it. The native AI already uses this to compare alternatives before changing the map.
Script AI, however, currently has much less information available before construction. It can ask whether individual steps are possible, but it cannot yet inspect a complete engine-calculated route together with its real construction cost in the same way.
I am still investigating this before proposing any new API. I want to distinguish carefully between limitations of the Script API and things that are simply AI strategy problems, rather than adding functions that may not actually be needed.
I am also checking the existing automatic bridge/tunnel behaviour of the way tool, since there may already be useful engine functionality available through a less obvious path.
As before, the idea is to keep the changes small and incremental: first understand and reproduce the limitation, then decide whether an API change is actually justified.

[Andarix]

... already has a fairly clean separation between calculating a route, calculating its monetary cost and actually building it. The native AI already uses this to compare alternatives before changing the map.
Script AI, however, currently has much less information available before construction. It can ask whether individual steps are possible, but it cannot yet inspect a complete engine-calculated route together with its real construction cost in the same way.
I am still investigating this before proposing any new API. I want to distinguish carefully between limitations of the Script API and things that are simply AI strategy problems, rather than adding functions that may not actually be needed.
I am also checking the existing automatic bridge/tunnel behaviour of the way tool, since there may already be useful engine functionality available through a less obvious path.
As before, the idea is to keep the changes small and incremental: first understand and reproduce the limitation, then decide whether an API change is actually justified.

sqai_rail already determines construction costs with a high degree of accuracy.

Check for 'build_cost' in the industry connection planner. I've linked two places here.

industry_connection_planner.nut#L356
industry_connection_planner.nut#L480

[victor_18993]

Thanks, that is very useful.
I had not yet followed

Code Select Expand

build_cost through the industrial connection planner in enough detail. Looking at it now,

Code Select Expand

sqai_rail is already doing a much more complete pre-build estimate than I had assumed: route length, bridge cost, tree removal, stations, depot and an estimate for terraforming are all taken into account before construction.
So I think the question is narrower than "the Script AI cannot estimate construction cost".
What I still want to determine is whether this estimate is close enough to the result produced by the engine during the real build, or whether there are specific cases where the AI has to duplicate or approximate internal builder logic and starts to diverge.
I will compare the planned

Code Select Expand

build_cost against the actual construction cost in a few controlled cases — flat terrain, slopes, trees, bridges and combinations of them.
If the existing calculation is already sufficiently accurate, then there may be no reason to add a new API at all. If there is a systematic difference, we will at least have a concrete case showing exactly what information is missing.
Thanks for pointing me to those two places in the planner.

[Andarix]

For many functions, the option to display notifications is available at the beginning.

industry_connection_planner.nut#L44

The current method for calculating construction and maintenance costs is actually quite good. This is evident from the fact that sqai_rail relatively rarely goes bankrupt (pak64, pak64.german).

It runs into difficulties when routes are too long and profits are too low such as in pak128, where industrial profits are very low.

Whether a route is built depends on more factors than just construction costs.

And once the route is known, construction costs can be determined based on the object properties (which, in my view, are complete here).

However, the credit refunds vary; in some cases, a full credit is returned, while in others, one-hundredth of a credit is returned.

[victor_18993]

Squirrel Script API modernisation — current roadmap
A short roadmap update, now that the preparation work has progressed enough to give a clearer picture of the next steps.
The dates below are target windows, not fixed deadlines. If a regression, compatibility issue or unclear contract appears, I would rather stop and investigate it than force the migration to fit a date.
15–16 August — bundled Script AI fixes
We are currently closing two pre-existing consumer-side issues found while establishing the pre-3.2 baseline:

• Code Select Expand
  sqai: failure handling when the initial convoy cannot be created.
• Code Select Expand
  sqai_rail: use of the currently undefined
  
  Code Select Expand
  debug.messages flag.

These are being handled as independent fixes, before the runtime migration, so they do not become mixed with Squirrel 3.2 compatibility work.
16–18 August — pre-migration tooling and regression baseline
The next target is to finish and integrate the preparation work that will be used to validate the runtime update:

• generated Script API signature baseline;
• refreshed
  
  Code Select Expand
  squirrel_modifications.diff;
• repaired Squirrel update workflow;
• targeted regression tests for Simutrans-specific VM behaviour;
• removal of a path-length dependency found in one of the new arity tests.

The goal is to have these changes reviewable independently from the actual Squirrel update.
18–20 August — final pre-3.2 consumer baseline

Code Select Expand

sqai and

Code Select Expand

sqai_rail are being used as real consumers of the Script API in addition to the normal automated tests.
The remaining baseline work is focused on behaviour such as:

• route/build-cost calculations;
• Code Select Expand
  sleep() suspend/resume behaviour;
• generator yield/resume behaviour;
• opcode accounting;
• save/load compatibility.

Once this is complete, we will freeze a final pre-3.2 reference consisting of API signatures, VM regressions and real consumer behaviour.
Around 20–24 August — Squirrel 3.2 migration
If the pre-migration gates are clean, the next major step will be the actual update from the current Squirrel 3.1.1-based snapshot to Squirrel 3.2.
This will not be a blind source replacement. Simutrans has several local VM customisations that need to be reconciled deliberately, including:

• operation-budget handling;
• suspend/resume behaviour;
• Code Select Expand
  sq_call /
  
  Code Select Expand
  sq_wakeupvm;
• the native
  
  Code Select Expand
  sq_tailcall() decision;
• error and arity diagnostics;
• compiler-related changes;
• the local
  
  Code Select Expand
  _OP_LOADFLOAT implementation.

After the import, the same pre-3.2 API, VM and consumer baselines will be rerun against 3.2.
Late August — stabilisation and follow-up
Any issue found during the migration will be classified separately as:

• an upstream Squirrel 3.2 behavioural change;
• a regression introduced by our integration;
• or a pre-existing Simutrans consumer/API issue.

Only the changes required for a safe migration will be included in this phase. Broader cleanup — for example further standardisation between

Code Select Expand

sqai and

Code Select Expand

sqai_rail — should remain a separate follow-up project rather than expanding the scope of the runtime update.
So the current sequence is essentially:

Code Select Expand

consumer fixes → preparation/tooling → final pre-3.2 baseline → Squirrel 3.2 → stabilisationIf everything continues to test cleanly, the aim is to have the Squirrel 3.2 work in active integration during the second half of August, but the validation gates remain more important than the dates.

[victor_18993]

Squirrel 3.2 migration — progress and updated roadmap
The pre-3.2 preparation has progressed faster than expected, so I can bring the roadmap forward a little.
The preparation stack is now integrated in trunk as r12158–r12162. This includes the generated API baseline, the refreshed Squirrel modification inventory, the repaired update workflow, the small upstream error-message correction, and the VM regression tests.
The final pre-migration baseline is also frozen: the Script API, customised VM behaviour, and the real

Code Select Expand

sqai /

Code Select Expand

sqai_rail consumers now have independent reference points that can be compared against the 3.2 result.
The actual Squirrel 3.2 runtime migration has now started in a separate development branch, based on r12162. It is not being developed directly in trunk, so intermediate migration errors or incomplete states should not affect the official builds while this work is in progress.
The upstream delta turned out to be relatively small and well bounded. The main areas requiring careful reconciliation are the

Code Select Expand

sq_getinstanceup API change, the new

Code Select Expand

bindenv compiler/VM behaviour, a string hashing change, and a few remaining library/runtime changes. Several Simutrans-specific VM areas we were concerned about are untouched by upstream 3.2, which has reduced the expected migration work.
Updated tentative roadmap

• 16–17 August: runtime/API migration cuts (
  
  Code Select Expand
  sq_getinstanceup, compiler/VM
  
  Code Select Expand
  bindenv, remaining upstream reconciliation)
• 17–18 August: Script API and VM regression comparison
• 18–19 August:
  
  Code Select Expand
  sqai /
  
  Code Select Expand
  sqai_rail consumer comparison and final pre-integration freeze
• around 19–20 August: integration candidate, if all regression gates remain clean
• afterwards: nightly/stabilisation period and fixes for any regressions reported by testers

These are still target windows rather than promises. If a migration cut exposes a real semantic regression, I will prefer to stop and investigate it rather than keep the date.
So far, however, the work has been less complicated than the original conservative schedule assumed, and the Squirrel 3.2 migration is now actively underway.