AI bug fixing experiment - early stages

[jamespetts]

Hello. I have not been around here much recently, but I have been spending some time working with AI for coding lately and wondered how capable that it would be for a difficult and complex code-base such as Simutrans-Extended. I have been experimenting with the open weights model GLM 4.7. I asked it to identify and fix the bug causing the occasional thread deadlock on the server that I've been aware of but unable to track down for a long time. Here is a transcript of the complete conversation. 

It identified the problem as an old bug from Standard (albeit more likely to affect Extended because it is more computationally intensive) that has alredy been fixed in Standard, and proposed the following patch based on the Standard fix:

Code Select Expand

--- a/src/network/network.cc
+++ b/src/network/network.cc
@@ -XXX,Y +XXX,Y @@
    // Processing loop
    while(running) {

- // Old Extended Style: Hold lock during execution
- s->lock();
- network_command_t *nwc = network_read_command(s);
- if (nwc) {
- nwc->execute(welt);
- }
- s->unlock();
+ // Standard Style: Unlock immediately after read
+ s->lock();
+ network_command_t *nwc = network_read_command(s);
+ s->unlock();
+
+ // Execute command (World Lock only)
+ if (nwc) {
+ nwc->execute(welt);
+ // Cleanup command object immediately after execution
+ delete nwc;
+ }
    }
I have not had time to see whether this works yet, but, if it does, it is promising. The difficulty that I have had recently in engaging with Simutrans-Extended is the amount of time required to complete the next features as against the amount of time that I now have available. If current AI can reduce the former, it may be more feasible to work on the codebase again. I shall have to apply this patch when I get home from my Christmas holiday and see whether this really does fix the deadlocking issue.

[Isaac Eiland-Hall]

In my experience messing about with AI, there is a real limit to the amount of context you can give it to work with. You also need to know what you're doing. For example, I've had it create simple one-page CRUD applications (simple database stuff to track things, like a to-do or medication tracker or things like that). It puts in absolutely zero security.

I trusted it to make my holiday music player, but that didn't involve a database.

If it proves to be helpful and find problems, that's awesome, and I wouldn't downplay that help. I'd just be extremely cautious about using code it generates. I personally feel it works better when it identifies problems, tells me what they are, and then proposes a fix - in general English terms, rather than writing code. Let me write the code - because it's hard enough to understand the code I've written in the first place.

I don't want to be discouraging about AI in the slightest - especially if it proves useful and helpful. Just slightly cautious and sharing my actual experience with it.

[prissi]

In my experience AI can find glaring mistakes but with the typical context window sizes will introduce more bugs on codebases the size of simutrans (or even 10th of it). If a "bug fix" compiles at all. Taking a look at the reported code is a good idea, but that would be as far as I would trust it.

Especially things like deadlocks (which are a variant of the halt problem) cannot be solved by AI.

But it said all that.

Also, standard has no threading for network. However, if you assume a deadlock, both threads waiting for freelist seems much more likely, simply because that is called a lot all over the place and extension is pretty normal.

[TransshipmentEnvoy]

Will you try proprietary models as well? e.g, Claude Sonnet/Opus 4.5 or GPT-5.2? Maybe also coding agents like Claude Code / OpenCode.

[prissi]

These all have too small context windows to really fix a bug which may lurk somewhere. What the AI can do is add new functionality, with a related tool and probably even GUI code. A lot of it are just repeating patterns with different layouts, something the AIs should be very good at understanding.

[jamespetts]

Unfortunately, on further investigation, it transpires that the AI had hallucinated the code and the patch cannot apply because the codebase is entirely different from what it was patching. This particular AI is very inconsistent in what it can search. I will have to investigate Copilot when I have some more time.

I have had some significant success with AI assisted coding with writing a complex set of scripts for JMRI and actually modifying JMRI itself, but I was using Copilot for that. 

[jamespetts]

It is time to revive this thread.

AI has come on a very long way in the past 9 months. I have now set up OpenCode and a knowledgebase of cross-referenced code documentation, itself created by AI, to allow AI agents effectively to work on this codebase without having to read the entire enormous codebase or guess how things work. The knowledgebase is now public on the Git repository.

The first thing that the AI was able to do was to get continuous integration (CI) on Github running, which does two useful things:

• creates new builds of Simutrans-Extended (Windows, Linux and (new) Mac) whenever a change is pushed; and
• runs automated tests, including address sanitiser, thread sanitiser and undefined behaviour sanitiser, which alert me if there is some error in the code.

Using the outputs of those tests, together with the AI's own initiative in finding bugs while creating the documentation and running its own tests, the AI has fixed a large number of bugs, many of which had been latent for many years.

Summary of bug fixes since AI-assisted work began (5–13 September 2026):
Threading races (ex-15, found via ThreadSanitizer) — the largest family:

• Convoy route-finding workers racing the main-thread sync step; convoy workers writing shared lazy summary caches
• Private-car/city queue bookkeeping at the head of
  
  
  Code Select Expand
  step/
  
  
  Code Select Expand
  pause_step racing with workers
• Code Select Expand
  resort_freight_info written concurrently by passenger workers
• Code Select Expand
  leitung_t::net raced between map-loop workers during threaded load; way-length finance booking raced during load
• Factory in-transit gate reading in-flight path-explorer state
• Misaligned freelist nodes on 64-bit; unsequenced
  
  
  Code Select Expand
  koord::get_str buffers; objlist publication ordering
• Stray mutex in route unreservation; invalid
  
  
  Code Select Expand
  working_method written when saving depot-stored rail vehicles (UBSan)

Network games / determinism:

• Newly created town/stop names could diverge between server and clients (name generation not world-scoped); plus a wrong-variable bug in town name suffixes
• Non-deterministic saves on the ex-15 upgrade path from uninitialised ex-15 fields when loading 14.x saves
• Monthly autosaves now deferred to the frame boundary

Crashes:

• Headless MSVC server crash: uninitialised
  
  
  Code Select Expand
  CRITICAL_SECTION in
  
  
  Code Select Expand
  simsys_w.cc
• Crash loading savegames referencing a building missing from the pakset (NULL tile dereference in the signalbox check)
• Crash from unbound halt handles left in tile haltlists after loading

Other:

• Off-by-one in the road-type count limit check in
  
  
  Code Select Expand
  settings_t::rdwr
• fluidsynth MIDI broken with the SDL3 backend; broken non-multithreaded builds; nettools MinGW build taking the wrong branch
• Multiple macOS nightly-build fixes (recursive dylib bundling, install names, re-signing)

Most are on ex-15 (threading/determinism); the crash, savegame-load and name-generation fixes apply to both branches.

For the first time, all of the CI tests on Github now pass.

It does look as though AI assisted coding is finally advanced enough to work effectively on a codebase such as that of Simutrans and its forks, which should mean that development can progress significantly more quickly than had been possible hitherto.

I should be interested if anyone who had reported crashes or losses of synchronisation in network mode might re-test to see whether the fixes have addressed those issues. 

[victor_18993]

To avoid running out of context and make sure the AI always knows what it's working on, I recommend using Obsidian; if you set it up in an organized and well-orchestrated way, you'll see the difference in the results it delivers.


https://obsidian.md/

Best regards.