Topic: slow midi initialisation

[Kuk]

Startup of simutrans is slow on debian stable/stretch if the package "fluid-soundfont-gm" is installed. Moving the midi files so simutrans does not find them speeds up the startup again. My profiler says that, if the soundfonts and the midifiles are present, almost all the startup time is spent in Mix_LoadMUS(). It seems as if sdl reinitialises fluidaudio for every call to Mix_LoadMUS() and fluidaudio then loads every foundfont installed on the system.

[DrSuperGood]

First off does this problem exist in SDL1 or SDL2 builds or both?

The function was probably intended to only load 1-2 pieces of music at any time. Possibly intended to be used asynchronously? Maybe its a bug with fluid-soundfont-gm if that adds some sort of sound system which Mix_LoadMUS uses.

[Ters]

I have no idea what fluid-soundfont-gm does, but if it does lengthy initialization before every song played, it sound like a problem with it. I have a software MIDI synthesizer, and it takes a few seconds to start up, but once it has, it stays loaded so that the subsequent songs don't have to initialize it. Since I don't play Simutrans with sound, I don't know how Simutrans works with it, but it would not have been with SDL anyway.

[Kuk]

I am not sure which sdl version is affected. The debian package has a dependency to "libsdl1.2debian", so it is probably using sdl1.

It is possible Mix_LoadMUS was intended to be used async or on only 1-2 pieces of music. Simutrans loads every midi file in the music directory on startup. Disabling music or specifying "-nomidi" on the command line does not help.

"fluid-soundfont-gm" only adds soundfonts (=midi instruments) to the system. I do not think there is a bug in "fluid-soundfont-gm". It probably only slows startup because there are more soundfonts which need to be initialised (without ever being used).

[Ters]

According to the SDL 1.2.10 documentation, Mix_LoadMUS only loads a music file into memory. It doesn't start playing it. That this triggers time consuming initialization outside the process seems to me like an issue in SDL.

There is however the issue that Simutrans loads all MIDI files into memory, only to unload them again at start-up, just to see if they are valid. Then we get back to the recent discussion that Simutrans loads sound into memory even if you use the -no... command line parameter.

[DrSuperGood]

Maybe a good solution would be to not test if all midi files are valid at startup but rather remove any invalid ones from the list of available midi files as they are encountered during runtime. This would prevent startup time having linear complexity with the number of midi files available.

[Ters]

Only with SDL does Simutrans do such a load/unload check on start-up. The Win32 backend doesn't actually load the MIDI files at all. Windows does it internally when told to play it. All other backends load the MIDI files into memory in some form at start-up, and keeps them there. Error handling seems sketchy in all of them. If there is a problem loading a MIDI file, it simply returns the id of the previously load one.

[Kuk]

The sdl api has a load and a play function. It seems logical that the slow initialisation is done in the load function since the play function is probably more time critical.

I think this sdl was meant to be compiled or used differently. The sdl source has a function Mix_GetSoundFonts(). It returns the path of the soundfont to use. It seems weird to me that this points to the systemwide soundfont directory. I would expect it to point to one specific soundfont, not only because of performance but also to make the selected instrument deterministic. It is probably either a debian packaging issue or simutrans is supposed to specify the soundfont with the environment variables.

Skipping the midi validity test on startup sound good, but I am not sure if it solves the problem completely. I can imagine that there will be issues with desync in multiplayer games when a client freezes when it starts to play a new track.

[Ters]

The sdl api has a load and a play function. It seems logical that the slow initialisation is done in the load function since the play function is probably more time critical.

Neither load nor play should ideally cause this. It should be triggered at most once per process, but preferably once per logon or once per boot, when the application sets up to play MIDI.

A contributing factor to the mess might be that SDL doesn't have a MIDI API. MIDI playback is tossed into a generic music API, despite being a completely different thing from waveform playback. It is therefore no explicit way for Simutrans to tell SDL to set up for MIDI playback, or for it to say when it is done. SDL seems to initialize MIDI when the application first shows interest in MIDI, which is Mix_LoadMUS. It might be that SDL also shuts down the underlying platform MIDI API when the last MIDI file is unloaded, rather than keeping it alive until the process exits. This causes everything to have to be set up anew on the nex Mix_LoadMUS.

I still don't get why the platform MIDI implementation needs to load soundfonts again. It should know that they have been loaded, and I don't see why it needs to unload them just because the current MIDI session has ended.

[Kuk]

Neither load nor play should ideally cause this. It should be triggered at most once per process, but preferably once per logon or once per boot, when the application sets up to play MIDI.

Yes, I agree.

I still don't get why the platform MIDI implementation needs to load soundfonts again. It should know that they have been loaded, and I don't see why it needs to unload them just because the current MIDI session has ended.

I do not think sdl really needs to reinit fluidsynth.

[DrSuperGood]

I can imagine that there will be issues with desync in multiplayer games when a client freezes when it starts to play a new track.

There will be no such issues. The lockstep system being used will mean the client falls behind and then has to speed up and catch up to the server.

It would be nice if one could check if this problem exists for SDL2. As far as I am aware SDL1 was deprecated by SDL2 so might not be as well maintained outside the odd critical bug fix.

[Ters]

It would be nice if one could check if this problem exists for SDL2. As far as I am aware SDL1 was deprecated by SDL2 so might not be as well maintained outside the odd critical bug fix.

I don't think SDL2 supports MIDI at all.

[DrSuperGood]

Where is Mix_LoadMUS being called from? I checked sdl_sound.cc and sdl_mixer_sound.cc but none of them call it directly.

[Ters]

sdl_midi.cc

[DrSuperGood]

Why is sound and music separate folders...

From what I can tell it is SDL Mixer that is providing Mix_LoadMUS and SDL Mixer is SDL2.0 compatible according to their site so should still load MIDI files with SDL 2.0.

Test if the problem persists when Simutrans is built with the latest version of SDL Mixer as there may have been bug fixes.

[Ters]

Why is sound and music separate folders...

Well, the handling of MIDI, used for background music, is very different from waveform audio, used for incidental sounds. The file formats are different, how the game interacts with them are different, the underlying API may be different. All they have in common is that they are delivered through speakers to the player's ears. Many other things are delivered through the screen to the player's eyes.

From what I can tell it is SDL Mixer that is providing Mix_LoadMUS and SDL Mixer is SDL2.0 compatible according to their site so should still load MIDI files with SDL 2.0.

I couldn't find anything about SDL Mixer on https://www.libsdl.org/. I see now that Google apparently can.

[DrSuperGood]

Well, the handling of MIDI, used for background music, is very different from waveform audio, used for incidental sounds. The file formats are different, how the game interacts with them are different, the underlying API may be different. All they have in common is that they are delivered through speakers to the player's ears. Many other things are delivered through the screen to the player's eyes.

It would have made more sense if music was nested in sound. I understand that MIDI requires a synthesizer library with instrument samples to turn it into sampled audio for blending and playback however ultimately this still has to do with sound. MIDI might not even be used for music, as was the case with Warcraft III where they used MIDI for environmental sound effects for some reason.

[prissi]

MIDI can be supported by hardware midi synthesizers. In that case the MIDI goes (as originally intended) via a serial port to the synthesizer (as it could have been done when simutrans was convieved using a virtual com port on a sound card), and may even end up on different speakers. There is absolutely zero overlap with sampled data for sounds, which can also carry stereo informations (MIDI by default does not, because it is a single instrument). Also on windows the MIDI API is not the same as the one used for sound, since there is no newer proper MIDI API.

Why should different data be handled the same way? Saying everything is noise, is like everything ends up on a monitor, so why should there be different object if everything ends on a screen ...

[DrSuperGood]

MIDI can be supported by hardware midi synthesizers.

Yes except does the audio component support doing this?

Also on windows the MIDI API is not the same as the one used for sound, since there is no newer proper MIDI API.

On Windows, starting with Windows Vista all sound is now done by software. Sampled audio at different sample rate from the output rate is resampled to the output sample rate and then mixed together with all other sound channels. The reasoning behind this is that modern processors are so fast relative to the sample rate of human audible sound there is little to no gains from using dedicated sound processing hardware while there is a lot to lose due to the complexity and potential implementation differences.

Since the concept of a dedicated sound card which does sound processing acceleration is now reserved for the odd specialist using specialist applications there is no longer a need to consider native support for MIDI and hence "no new proper MIDI API". On Windows a basic MIDI synthesizer is included for compatibility with legacy APIs but ultimately if one wants to play midi on a modern computer one needs a software synthesizer to convert the midi stream into one or more sampled sound channels for playback.

Why should different data be handled the same way? Saying everything is noise, is like everything ends up on a monitor, so why should there be different object if everything ends on a screen ...

Except from what I can tell is happening in the files it basically is being treated the same. In fact according to the documentation for Mix_LoadMUS…

Load music file to use. This can load WAVE, MOD, MIDI, OGG, MP3, FLAC, and any file that you use a command to play with.

Of which most of those formats are or at least support sampled audio.

Anyway this is deviating from the problem at hand. I do apologize for not originally noticing the code was in a separate folder.

[Ters]

Also on windows the MIDI API is not the same as the one used for sound, since there is no newer proper MIDI API.

Simutrans doesn't use the newer APIs for other sound either. Furthermore, Simutrans doesn't really use the MIDI API either, just a generic multimedia API. There isn't really any need for a newer MIDI API either, since the one that is covers pretty much what MIDI is. The only thing would be turning it into one or more COM interface just because. Microsoft has apparently however removed the MIDI mapper functionality, which as I understand it means that applications must themselves open the MIDI device the user wants. They can no longer just open the default port knowing that the user has mapped this to the actual device in the Control Panel.

Except from what I can tell is happening in the files it basically is being treated the same. In fact according to the documentation for Mix_LoadMUS…

That is a quirk from one of the backends. Well, arguably the Windows backend as well, although that one has the option of doing it in a more low-level and very MIDI specific way if it wants to, and still be a Windows backend. That is likely true for MacOS as well. Allegro has clear separation between sound and MIDI.

As has been discussed earlier, MIDI isn't really a good choice for music playback on modern systems, since hardware support has pretty much disappeared from ordinary consumer products. Windows at least comes with a bundled software synthesizer, which I don't find particularly good. Linux has nothing by default. But switching to MP3 or other more modern compressed waveform audio formats would multiply the size of the game. Not everyone has broadband.

[kierongreen]

Indeed - think of the different directories more as a sound effect / music separation than one to do with particular format. I still remember being amazed when computers became fast enough to just about playback MIDI in software (using 100% CPU and rather low quality sound libraries)...

[Ters]

I remember being amazed learning that my computer could play music from notes, not just replaying recorded sound like my cassette player. That was in hardware, a Sound Blaster 16 I think. Not so much amazed when software synthesizer took over, because it sounded like crap compared to the Sound Blaster AWE32 I had just prior. (I still use my AWE32 sound fonts when I in a moment of nostalgia dig out games from that period, or just the music from them. The sound won't be right otherwise.)