New listings server (beta!)

[Ashley]

So I've been working on a new game listing server for the past few weeks. I'm getting to the point where it's nearly ready for use.

The listings server can be accessed here (which will be its permanent home):

http://servers.simutrans.org/list

(Stylesheet still needs some work, and the help link doesn't work, the form on the /announce page can be used for testing and will be removed later).

This obviously requires a patch for the game as well. The listings system has been completely changed. Instead of requesting a server ID and using this for announcing the server the listings are now based on the primary key of IP address/DNS name combined with port number (since this is unique). You simply turn on announcement in the game configuration and the game does the rest.

The server will only accept announces from the same IP address as you are attempting to announce for (e.g. if your server has IP address 1.2.3.4 the announce server will only accept announce requests for that server which have the client IP address 1.2.3.4) in the case of DNS aliases it'll do a DNS lookup and only accept if one of the results matches the source IP address. This is basic security to prevent other people from altering your server's listing.

An announcement interval is also specified on the server (default 15 minutes), if the listing server doesn't hear from that server within double the announcement interval it'll mark the server as offline. Offline servers then won't appear in the listing sent to clients.

The patch modifies the server (so that it announces using the new format, and has the new settings etc.) and the client (which now downloads the listings in the common CSV format). Work left to do includes some code cleanup and testing.

I have yet to implement uploading of the minimap graphics to the server (this I'll probably do as a later patch since it's quite a bit more complexity).

This is still a work in progress, but feedback would be appreciated!

[Dwachs]

Impressive !

[jamespetts]

Interesting. Since the default server doesn't have a graphics backend, perhaps the system could press-gang the first client to connect to generate a minimap screenshot and upload it to the server?

[prissi]

The server can generate a minimap without any grphics installed, as it draw for this purpose a 8 bit bitmap in karte.cc. This can be either compressed via bzip (which is always on board of a server, since it is the format f savegames) or send as a png (so no further work of the list server is required).

EDIT:
Why actually not using & since this cannot be contained in any announcement string and makes parsing of the list very easy (by strtok ... )?

And why is the header so bloated up with stuff? The list server communicates exclusively with simutrans, thus would it not make sense to keep it as small as possible?

And there is announce_server( 1 ) and annouce_server( true ); imho there should be only one. (Some not so relevant comments: simutrans coding rules liek to have two spaces after an opening bracket or between conditions and allow only a single curly braket per line (this mean the closing in front of else is usually a single line) Space between functions are two lines and the commetn above a function should by rather /* so it is automatically include in doxygen.

[Ashley]

If/when I implement the minimap it'll be done as a .png file uploaded as part of a multipart/form-data content type document. I think that's the easiest way to go about it and means the server doesn't have to have to do any image format conversion itself. This would add a dependency on libpng for the game though of course (but why not? It's not a big dependency and Makeobj already uses it).

I went with CSV formatting because it's a nice standard format. There's no reason to think that someone might not name their server "Bob & James's awesome server" for example. I am thinking ahead to changing the in-game server list to a display format like:

Bob & James's awesome server (bob.mysite.com:13357)

Which I think is more descriptive for the player. I haven't put that into the patch yet because of the way the current code stores/uses the entries directly through the list control.

The header isn't bloated is it? Current fields are:

User-Agent (useful for statistics tracking on the server), Host (to conform with HTTP 1.1 spec), Content-Type (sending this is good practice) and Content-Length (again good practice). Yes the list server only communicates with the game but we have chosen to use HTTP 1.1 and so our implementation should be standards compliant (makes it easier for someone in future to implement another listing server if need be). The overhead of a few extra characters in a message which is sent by default once every 15 minutes is I think not so bad.

It should only be announce_server(0/1/2) so I'll check that. As for coding style I'll clean that up for the final patch. This is definitely still a work in progress

[prissi]

If & is allowed in comments send to the server (and similar applies to spaces too) then the handling of the "...&name=...&commetn=..." string needs to be changed.

[Ashley]

That bit I modified so that all fields get put through the EncodeURI function before being appended to the string used as data for the POST request.

Edit: Just as a note, I am still working on this although I've been very busy recently. There isn't too much left to finish up now I don't think.

[Dwachs]

Do you have any new development snapshot for testing?

Edit: testing looks good.

I noticed just one thing: There is a stack corruption going on while reading csv strings. The null-character is written at the position after the string. You can use the method tstrncpy fom utils/simstrign. Also the field c was not updated if the last cell was read. Attached the patched server_frame.cc; I modified only the read csv routine.

And one very minor detail: spaces in server names are displayed as '20' (ie as number).

[Ashley]

I had noticed the space->20 thing too, I think it has to do with url encoding but haven't had a chance to look into it. Thanks for the patch for the CSV parsing, I was a little unsure about that bit

I should have some free time this weekend so hopefully get this finished off then.

[Dwachs]

Maybe changing one line in encode_URI to

Code Select Expand


buf.printf("%%%02X", (unsigned char)c);

should translate spaces to "%20". Cannot test at the moment.

Edit: commited this small fix.

[Ashley]

Great, I am trying to work on this again and I'm now getting compiler errors:

Code Select Expand

===> CXX player/ai_passenger.cc
player/../dataobj/../tpl/slist_tpl.h: In member function 'bool slist_iterator_tpl<T>::next() [with T = quickstone_tpl<haltestelle_t>]':
player/../dataobj/../tpl/slist_tpl.h:434: internal compiler error: Segmentation fault: 11
Please submit a full bug report,
with preprocessed source if appropriate.
See <URL:http://developer.apple.com/bugreporter> for instructions.
make: *** [build/default/player/ai_passenger.o] Error 1

I don't even touch this file, and without my patch it compiles :(

[jamespetts]

Looks like a compiler bug - try a different version of GCC.

[Ashley]

If I move the #defines into simversion.h from simconst.h it works. Definitely a compiler bug though!

[prissi]

This would be easier to find out with a patch file though ...

[Ashley]

Same patch file as above will cause the segfault Prissi, I haven't changed much since then (except adding in Dwachs changes to gui/server_frame.cc)

[prissi]

You should submit this to the GCC list. However, I still use 3.4.3 which compiled nicely for me.

[Ashley]

This patch should be all that is needed for Simutrans code, it implements:

HTTP POST for announce to server
Announce on player leave/join
Detailed announce on startup
Announce server going offline
Periodic announce (real-time, configurable)
CSV format download of server listing and working filter for matching pakset/game versions
Nettool forced-announce causes detailed announce

Yet to do:
Upload of minimap (I think this should be a separate patch later since it is a complex job in itself)
More robust HTTP handling
Announcements for game when paused (no idea how this will work yet)
Display format for entries in the server_frame GUI (again, I think this GUI should be re-worked entirely but that would be another patch)

I've cleaned up the code style so it should conform to the rules you mentioned Prissi.

The listings server itself still needs some work (primarily work on the layout/stylesheet, documentation for the communications spec and implementing the translation system) but it's pretty much working and stable for now and I don't believe any further changes are needed to the game.


The following config file parameters are added:

Code Select Expand


# Automatically announce server on the central server directory (http://servers.simutrans.org/)
# 0 (default) = off, 1 = on
#server_announce = 0

# Interval of server announcement (if enabled)
# Value is number of seconds between server announcements, default is 900 (15 minutes)
# Minimum value is 60 (1 minute), for accurate listing it is recommended not to increase
# this value to greater than 3600 (1 hour)
# To disable announcements set server_announce to 0
#server_announce_interval = 900

# Fully Qualified Domain Name (FQDN) or IP address of your server (IPv6 or IPv4)
#server_dns = 127.0.0.1

# Name of server in server listing
#server_name = My Simutrans Server

# Additional information about your server (for display on the list server)
#server_comments = Comments about my server

# Email address of server maintainer (for display on the list server)
#server_email = maintainer@your.server

# Pakset download URL (for display on the list server)
#server_pakurl = http://your.domain/pakset.zip

# Server info URL (for display on the list server)
#server_infurl = http://your.domain/server-info.html

The following command line parameters are added:

Code Select Expand


-announce           Enable server announcements
-server_dns FQDN/IP FQDN or IP address of server for announcements
-server_name NAME   Name of server for announcements


[Dwachs]

Announcements for game when paused (no idea how this will work yet)

As you do the announcement based on real time, there should be no problem here. The game loop is running in paused mode as well (karte_t::interactive)

[Ashley]

Oh good, I thought that might be the case but good to confirm it

[prissi]

Ok, I submitted it to the nightlies. Please test it.

[Ashley]

I'll upgrade my servers tomorrow once the nightly has been generated.

[wernieman]

Do you need a nightly Server build?   

[Ashley]

That might be useful, though I compile the versions I run myself on a debian VM. I need the nightly versions so that people can easily obtain a version to use as client to connect to the server

[wernieman]

When you compile by yourselft ... you do not deen an nightly server ... but I look (and nett a little time for it)