The International Simutrans Forum

 

Author Topic: Help needed! Future development structure for our pakset  (Read 2056 times)

0 Members and 1 Guest are viewing this topic.

Offline Flemmbrav

  • Moderator
  • *
  • Posts: 111
  • Languages: DE, EN
Help needed! Future development structure for our pakset
« on: January 01, 2018, 07:20:55 PM »
I just tried to get a feeling about what is actually needed to convert our pakset to simutrans extended. Turns out graphics can stay the same in 9 out of 10 cases. But there a tons of things to be done that I could not imagine to do by hand. Also we're kind of running on the limit with what we have currently. I'll try to discribe our situation and what I think would be best to handle an extended tree and how to improve our infrastructure so we don't have that much work with .dat files we could spend on fancy graphics. I'ld appreciate any kind of thoughts how you would handle the challenge and what you do think about my ideas to do so!

Currently managing our pakset is a huge mess. Every part of it feels like it's kind of hacked for it's purpose. There is no system behind it, it's just the way it is. We have pricing for vehicles and some kind of idea how to name things, but that's about it.

At least I can't see a way to manage to get our pakset in extended without some changes to how we handle all our sources.
While we still can keep doing stuff the way it is nowadays, for me the time has come to start thinking about changing our "backbone" to something more powerfull than some folders with dat and png files in and an additional excel sheet for pricing.

So here are my first thoughts:

.dat files out of excel sheets (or similiar)
   I want to have a document which clearly is the single source for for information given in .dat files.
   That document has to be able to be changed by multiple people at the same time.
   There shall be a way to create .dat files out of such a sheet.
   maybe it's even possible to create diagrams out of it? like industry chains and timelines?
   Is there a way to automaticly update the text to the loading screen with each revision we upload?
   Could we do pricing for at least 2 versions? one for standard and one for extended? This should be something as easy as checking one checkbox when exporting the excel to .dat files. A thing to add here are how to handly liveries on a build for simutrans standard.

pak-ing the whole pakset from windows and linux machines
   basicly a no brainer, but not standard yet. really has to be done, our most invested developers all use windows

Gameplay
   Basicly our current balancing is all about runningcosts vs income. We do not consider different aspects nor do we have an idea what to actually could archive by balancing this game.
   We need something new from scratch.
   Pricing shall only be one part of it. There has to be a solution how to create challenges for every kind of game.
   Think about absolute untouched basics like "what's the value of a tile".

Graphics
   Where do we put the real sources? Is there a file format that can be used with at least gimp and photoshop?

So uhm, that's it for now, I'll try to keep this thread up to date when I have new thoughts about it. Please share your opinions and ideas around this topic! Also every kind of help is appreciated!

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Devotee
  • *
  • Posts: 17469
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Help needed! Future development structure for our pakset
« Reply #1 on: January 03, 2018, 06:29:24 PM »
A large but worthwhile task. I do recommend Github, incidentally, for pakset sources (and, if you use rendered 3d models, the meta-sources, too), and also for all of the balancing spreadsheets.

Nightly builds are also worthwhile. Indeed, if the sources (and a makefile) were on Github, I could produce automated nightly builds using the latest makeobj and automate the distribution of this pakset with the Simutrans-Extended complete .zip file in the same way as I currently do for the Swedish pakset.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Offline Leartin

  • Moderator
  • *
  • Posts: 1017
  • PAK-DEV P192C
  • Languages: DE, EN
Re: Help needed! Future development structure for our pakset
« Reply #2 on: January 03, 2018, 08:47:37 PM »
(Wall of text incoming in 3... 2... 1...)

We are actually using a repository already, though it's on Assembla rather than Github, but there shouldn't be much difference (we certainly don't use everything we could on Assembla, so chances are we wouldn't use more tools on Github, and just treat it like a community dropbox anyway). Nightlys were once done automatically from there (before my time), and a makefile exists, so that should be fine.

Personally, I don't think the repository choice is an issue. The real problem is that this pakset was slowly growing without enough guidelines, so the sources are kind of spread.
Objects have partly german, partly english names, often including the author in the name as well. Sometimes, we replace graphics, such that the original author wouldn't even be the author anymore. Such inconsequent naming decisions may even cause objects of different types having the same name and overwriting one another. While I went through all citybuildings for 0.5 and made sure they all reached a certain quality standard, I renamed all of them following the same simple formula, and I'd like to do the same for every other type of object - but that's a lot of work, and I don't feel comfortable doing it for vehicles.
Next thing is folder structure. It's not very well thought out, as seen by things like "buildings/station/buildings". Largest issues is the half-height-folder - initially trying to provide both a full-height and a half-height pak, all new ways and similar structures were put in a seperate folder. But since then, we pretty much gave up on the full-height version, hence it should be integrated in the main body again. Then there are objects which were too large even for 192 size, which pretty much all got their own larger size, rather then just going for a certainly big enough 384-size for ALL of them - causing at least 6 folders named after their size and specifically mentioned in the makefile. Lastly, opposite to half-height folder, it seems many people don't enjoy the "infrastructure" chains (prison-chain and graveyard-chain), hence it would make sense to create one or two "official addons" out of them, which requires them to be in their own folder structure. Nothing of that is particularly difficult to do, but it requires time and, since it's a large change, should be done right, not just on a whim.
But thats, in the end, just a cleanup task. A lot of work, sure, but really anyone could do it.

To elaborate on the difficulties Flemm mentioned:
We can't pak under Windows! In preparation for the release, I set up a virtual machine with Ubuntu so I could use make, after "Make for Windows" just wouldn't work. In the meantime, that virtual machine broke once already, and since a makeobj for linux is rarely released I'm not sure I will always have the latest version - unless I self compile, which I already asked someone to teach me - but it seems needlessly complicated. Given the majority uses windows, we really need a solution for windows rather than linux/mac

We have no idea how to balance the pakset! I enjoy creating graphics, but I usually put them in the repository without the slightest idea about their balancing. Which does not matter for citybuildings, trees and other fancy eye candy, but since I do the same with factories... yeah... Pretty much everything is just leaning on what was there before, which was probably never balanced in the first place. We do have basic ideas about how it should work out, but nobody ever crunched the numbers to decide how they must be to achieve the vision we have.

We don't use the same graphics program. I use Photoshop and have PSD files with a few hundred layers for some buildings. But even if I would put them in the repository, Flemmbrav could not open them, since he uses GIMP. Neither could PTM, who uses Paint Shop Pro 5, nor Alex who uses MS Paint. Those two can't even open some of the PNGs we create since their programs don't support a full alphachannel. Since it's Pixel Art, not having those sources isn't quite as bad as with 3D graphics, but it's still much more difficult to form something new from an object half covered by a tree than if you had  that object in it's own layer. But this probably can't be helped, and we just need to decide how and where to save single objects from a larger composition for reuse purposes.


Really, it's just something grown organically without structure, and we just piled up on top of it instead of checking the basement, mostly because we started out without knowing how the game even works, and I would happily forget about that and just continue creating graphics, as that's the most fun for me. But if Flemmbrav does the same and nobody else comes along to do the grindy stuff, in 10 years this will be the best-looking worst-to-play pakset ever :(

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Devotee
  • *
  • Posts: 17469
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Help needed! Future development structure for our pakset
« Reply #3 on: January 03, 2018, 09:59:28 PM »
I notice that Assembla hosts Git, SVN and another type of repository; is Pak192.Comic using a Git or a different type of repository? It would be easy for me to automate nightly builds using Git, but I am not sure how feasible that it would be with another type of repository (it might be possible with SVN; I have no idea about the other one).

As for balancing, when you eventually get it right, that's the most rewarding part of game creation.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Offline An_dz

  • Web Admin
  • Administrator
  • *
  • Posts: 2805
  • D'oh
    • by An_dz
  • Languages: pt, en, it, (de)
Re: Help needed! Future development structure for our pakset
« Reply #4 on: May 03, 2018, 08:00:00 PM »
(Wall of text incoming in 3... 2... 1...)

Now get ready for a REAL wall of text ;D

.dat files out of excel sheets (or similiar)
   I want to have a document which clearly is the single source for for information given in .dat files.
   There shall be a way to create .dat files out of such a sheet.
   That document has to be able to be changed by multiple people at the same time.
   maybe it's even possible to create diagrams out of it? like industry chains and timelines?

My suggestion is to use Excel Online with a xlsx file.

Why you can't put a sheet file in your repository? Because there's no format that is powerful enough to do what we want and that works with standard version control systems. Most files are compressed, others are one-liners, and each software (including versions of them) will save the file slightly different. Also Excel Online is truly online, when someone is editing something you can see in real time what they are doing.

Why not Google Sheets? Many reasons, let's list some.
  • On Excel only one person needs a Microsoft account.
    Others can edit with a special link, though there are advantages if everybody has an account as those with a link are seen as "Guests", on Google you need to mess with permissions to achieve something similar.
  • Excel has the familiar interface of the desktop version.
    Everything, including more powerful tools are easily accessible. Google Sheets only shows the most simple things, doesn't have some features and hides others very well. Google sheets is also a crap when scrolling as it always scrolls by cells rather than a normal page.
  • You can edit online with the real time multiple people feature on the desktop version (Office 2013+) or with the mobile apps.
    If a feature is not available on the online version you can use your unlimited desktop version and add it.
  • Excel online works out-of-the-box with xlsx files.
    Google Docs needs to convert the files and that sucks because with Excel you can download the file, edit on your computer to add features only available on the desktop version and upload back to OneDrive and everybody can enjoy.

And why xlsx? Because it's a file that is not hard to parse. So a script can extract information and generate/edit the files with the values. Also xlsx is a powerful format, you can have multiple sheets in a single document, link data between them and generate multiple graphics and calculations to analyse the pakset balancing.

   Could we do pricing for at least 2 versions? one for standard and one for extended? This should be something as easy as checking one checkbox when exporting the excel to .dat files. A thing to add here are how to handly liveries on a build for simutrans standard.

Certainly, as I said just above, xlsx is not hard to parse. But one must build the sheet in a way that makes parsing easier or predictable so the parser is easier to code.

   Is there a way to automaticly update the text to the loading screen with each revision we upload?

Yes, if using Git (which I believe you don't) you'll use the command git rev-parse --short HEAD to get a short revision string, if you are using SVN then you call svnversion.



Gameplay
   Basicly our current balancing is all about runningcosts vs income. We do not consider different aspects nor do we have an idea what to actually could archive by balancing this game.
   We need something new from scratch.
   Pricing shall only be one part of it. There has to be a solution how to create challenges for every kind of game.
   Think about absolute untouched basics like "what's the value of a tile".
We have no idea how to balance the pakset!

I believe this would need to be put in Excel and then as we see the whole data we can think about what connections can be made.



Graphics
   Where do we put the real sources? Is there a file format that can be used with at least gimp and photoshop?
We don't use the same graphics program.
Photoshop
GIMP
Paint Shop Pro 5
MS Paint.

GIMP can open and save (export) PSD files, GIMP is open-source and freely available for the three main desktop OSes. I can't see the problem here as long everybody has at least GIMP installed. Make sure to always have the latest version though as some PSD effects might not import/export correctly, but all my tries always worked fine.



pak-ing the whole pakset from windows and linux machines
We can't pak under Windows!
[...]
"Make for Windows" just wouldn't work.
I don't know which "Make for Windows" you got, but the one from MSYS2 should work flawlessly, and I have a tutorial on installing it here on our forum.

Given the majority uses windows, we really need a solution for windows rather than linux/mac
Why use makefile at all? Yeah, yeah, makefile is OS independent, blah, blah, blah. But it needs you to install make, while this is a given on Linux or Mac it's pretty frustrating on Windows, you generally need to install a wider package that contains make.

Instead, you can use batch/bash. Check pak96 source, I just updated them with easier commands that make stuff recursive.

In preparation for the release, I set up a virtual machine with Ubuntu so I could use make
[...]
In the meantime, that virtual machine broke once already

In my ultra honest opinion Ubuntu sucks. I won't get into details as I don't want to go off topic, but use some other distro, if you want something that is as easy to install as Ubuntu you can try Fedora.

and since a makeobj for linux is rarely released I'm not sure I will always have the latest version - unless I self compile, which I already asked someone to teach me - but it seems needlessly complicated.

Self compiling on Windows is easy with MSYS2 and, again, my tutorial.

On Linux it's also easy, at least on Arch it's a joke, on Fedora I don't remember but I think GCC and Make come pre-installed, but if not just need to install them, and you will also need to install libpng and pkg-config. Then do the same config.default file as explained in my MSYS2 tutorial and done.



Really, it's just something grown organically without structure

If it interests you this is the structure I have made for pak96, I believe it makes the most sense
Code: [Select]
- city-and-landscape
 └ buildings
  └ com
  └ cur
  └ ind
  └ mon
  └ res
  └ tow
 └ cars
 └ ground
 └ pedestrians
 └ rivers
 └ trees

- factories
 └ goods.dat

- other (misc)
 └ groundobj
 └ hq
 └ powerlines
 └ smoke
 └ waycrossings

- ui
 └ cursors
 └ menu
 └ symbols

- air
- rail
- rail-narrow
- road
- maglev
- monorail
- tram
- water

INSIDE EACH WAYTYPE FOLDER:

- buildings
- signs
- vehicles
- wayobjs
- ways
 └ tunnels
 └ bridges
 └ elevated

Offline Leartin

  • Moderator
  • *
  • Posts: 1017
  • PAK-DEV P192C
  • Languages: DE, EN
Re: Help needed! Future development structure for our pakset
« Reply #5 on: May 04, 2018, 09:50:28 AM »
Now get ready for a REAL wall of text ;D
Lies, it's all quotes, also it only looks more because there is more space between the lines  ;D

My suggestion is to use Excel Online with a xlsx file. [...] And why xlsx? Because it's a file that is not hard to parse. So a script can extract information and generate/edit the files with the values.
Sounds good. I'll check that out once the development wakes up again.

GIMP can open and save (export) PSD files, GIMP is open-source and freely available for the three main desktop OSes. I can't see the problem here as long everybody has at least GIMP installed. Make sure to always have the latest version though as some PSD effects might not import/export correctly, but all my tries always worked fine.
You can certainly export GIMP-files as PSD and open them in Photoshop without a problem, but the other way around is sadly not true. This is due to Photoshop having far more features which are not supported in GIMP, hence it can't handle them. For example Adjustment Layers or Smart Objects. I don't remember exactly what Gimp does with them, I think in most cases it just loses editability, though thats why you'd keep the source in the first place.

Instead, you can use batch/bash. Check pak96 source, I just updated them with easier commands that make stuff recursive.
Why Make? IDK. Why not Batch? IDK. I suppose the main reason would be that the only one who actually understands that stuff and set it up was Cruzer, who uses a Mac, so cross-platform support would be nice. Though, I suppose if both Make and Batch were available, that would be the best solution.



If it interests you this is the structure I have made for pak96, I believe it makes the most sense
It's interesting for sure. Ours for comparison:
Code: [Select]

- buildings
 └ 240  #oversized buildings in seperate folders for automated packing
 └ 384  #though "240" should be incorporated in "384"
 └ city #contains all dat files for citybuildings. Each file contains all buildings of one type in one climate, so the number of dat files is limited. Images are seperated in Sub-Folders
  └ images #shared graphics, like smoke or front-image-greenery
   └ com #images used in multiple climates
    └ alpine
    └ desert
    └ [each other climate]
   └ [ind, res, mon, lot, park, plaza]
    └ [each climate]
 └ cur #contains all dat files for curiosities, each file contains buildings of a vague type (architecture [eg. atomium], infrastructure [eg. radio tower], sacred [eg. church], intellectual [eg. museum], parks
  └ uncut #contains full graphics of multitile-buildings
  └ images
 └ depots #contains one depot dat and all images. images should be moved to subfolder
 └ factories #contains all dat files sorted in chains, eg. alcohol_chain contains hopfarm, maltworks, brewery, vineyard and pub as well as hop fields.
  └ uncut #contains full graphics of multitile-buildings
  └ images
 └ stations #contains all station dats, each file groups a bunch of stations (usually same waytype, same time period, same use (eg. stations for use on elevated rails together)
  └ [air, maglev, narrowgauge, road, track, water] #contains images of the corresponding waytype
  └ buildings #yep, subfolder buildings/stations/buildings, these are extension buildings for stations
  └ headquarter #don't ask, I have no idea why
 └ tows #contains dats only
  └ image #not imageS for no reason

- goods # contains a txt file listing the goods in an overview and a goods.dat with all goods.
 └ [easy, normal, hard, harder, damn_hard, dev] #different folders, all just containing one goods.dat with different values.
- ground #contains all you need for old ground (no halfheight) - texture, marker, sidewalk,...
 └ ground_objects #dats, batched in groups (ponds, stones, moving,..)
  └ image
- gui #(old) menu, skins, symbols, compass
 └ 128 #for the pakset logo
- pedestrians
- smoke
- tree #one dat, all graphics, no subfolder
- trunk #everything that gets copied not packed
- vehicles #contains an xls file and an ods file for planning/balancing
 └ 205 #oversized plane
 └ [air, citycar, maglev, road,...] #all the different types, containing dats, but this time, almost each vehicle has it's own dat
  └ image

-ways #worst part! most of this isn't even in use anymore!
 └ bridge #mostly everything randomly
  └ track #only graphics
 └ crossings
 └ roadsign #mostly dats
  └ [air, narrow, road, track] #graphics
 └ tunnel #mostly dats
  └ [narrowgauge, road, track] #graphics
 └ way #a mess I refuse to list in detail
 └ way_object #see above
 └ way250 #for an oversized elevated way

- halfheight #this was created for conversion, basically as an integrated fork, which replaces much of the stuff already listed
                   #contains dat-files, each waytype has at least one, some are split (eg. "road.dat" and "road_tunnel.dat")
 └[32,64,250] #for everything with larger/smaller graphics, eg. menu and an elevated road
 └aviation #all images related to planes. There are also some buildings in here, making it even more confusing
 └[tram, road, wall, electro,...] #contain images

I think a good solution would be to work toward this:
Code: [Select]
- buildings
 └ 384  #for everything oversized, just use 384, even if the image is actually smaller. The empty space barely matters in the grand scheme of things.
 └ city #no change
 └ cur #no change
 └ factories #almost no change except factory-smoke goes here and...
  └ goods #becomes a subfolder of factories.
 └ tows #no change (except imageS)
 └ headquarter

- landscape #new collection folder to include...
 └ ground #textures, transistions, heightmap, basement, sidewalk,...
 └ ground_objects
 └ tree
 └ pedestrians

-vehicles
 └ 384 #everything oversized
 └ smokes (for vehicles)
 └ [air, maglev, narrowgauge,...] #combine dats where it makes sense
  └ imageS
 └ train #for better overview (since there are too many):
  └ locomotives
  └ goods_wagons
  └ pax_wagons
  └ sets #eg. ICE should not be seperated, so all parts of it use one ICE.dat in the sets folder.
 └ road
  └ bus?
  └ truck?
  └ citycar

- infrastructure
 └ 384 #everything oversized
  └ images
 └ buildings #all extension buildings
  └ images
 └ crossings
 └ aviation #contains all dats, grouped by type (stations, depot, signs, ways, ...)
  └ images #contains all graphics
 └[electro, tram, track,...] #all other waytypes
  └ images

- UI #cursors, symbols, marker,...
 └ 32 #menu icons
 └ theme

Offline DrSuperGood

  • Dev Team
  • Devotee
  • *
  • Posts: 2376
  • Languages: EN
Re: Help needed! Future development structure for our pakset
« Reply #6 on: May 04, 2018, 10:48:36 AM »
As part of my balance work on extended I have kind of created a java tool which can extract mappings from .dat files and place them inside a UK/US formated CSV spreadsheet. This CSV spreadsheet can then be promoted to a normal Microsoft Spreadsheet inside Excel for modification. One can then flatten this normal Excel spreadsheet and save it as a UK/US formatted CSV. The program will then parse this CSV file and automatically insert, replace or remove mappings that thave changed from the various .dat files. It cannot automatically create new entires, it was hacked together for modifying the balance of an extremely complex existing pakset (Pak128Britain Extended) rather than making a new pakset. It also features some kinds of error detection, such as duplicate mapping detection, which makeobj does not.

Why CSV? Because UK/US formated CSV is so easy to parse I did not even need to write a separate function for it (although I probably should have...). Of course it could probably be adapted to support other spreadsheet formats with the help of some third party libraries such as apache commons. I just did not want to spend 99% of the time writing a spreadsheet API.

Offline Leartin

  • Moderator
  • *
  • Posts: 1017
  • PAK-DEV P192C
  • Languages: DE, EN
Re: Help needed! Future development structure for our pakset
« Reply #7 on: May 04, 2018, 11:05:37 AM »
So I read that correctly, you can create a spreadsheet from the contents of dat files, edit values, and place them back into dat files; but no new dat files are created, only existing ones changed?

Offline An_dz

  • Web Admin
  • Administrator
  • *
  • Posts: 2805
  • D'oh
    • by An_dz
  • Languages: pt, en, it, (de)
Re: Help needed! Future development structure for our pakset
« Reply #8 on: May 04, 2018, 05:32:56 PM »
That's what @DrSuperGood script does, but you can create one that generates dats.

Lies, it's all quotes, also it only looks more because there is more space between the lines  ;D
Lies, Deceptions! ;D

Offline DrSuperGood

  • Dev Team
  • Devotee
  • *
  • Posts: 2376
  • Languages: EN
Re: Help needed! Future development structure for our pakset
« Reply #9 on: May 04, 2018, 08:52:41 PM »
Quote
So I read that correctly, you can create a spreadsheet from the contents of dat files, edit values, and place them back into dat files; but no new dat files are created, only existing ones changed?
Yes that is how I am modifying the balance of pak128Britain Extended.

Pak128Britain Extended has extensive commentary in its hand made dat files which has to be preserved and kept in the correct order, as such I needed a system to interact with existing dat files in a sensible way, rather than out right making new ones. Obviously if you build the entire pakset dat from a spreadsheet this is not a problem, and in such a case the writer part could be modified to create entirely new dat files in I recon 15-45 minutes of programming.

One needs to define a clear ordering of records within dat files. As such one would need to either automatically order them based on the value of a field or explicitly order them with an index. I went with the index approach as I had to locate specific records to modify and debug, but one might as well go with an automatic ordering approach if one is generating all dat files.
« Last Edit: May 05, 2018, 04:03:04 AM by DrSuperGood »