The International Simutrans Forum

 

Author Topic: Help needed! Future development structure for our pakset  (Read 4869 times)

0 Members and 1 Guest are viewing this topic.

Offline Flemmbrav

  • Moderator
  • *
  • Posts: 222
  • PAK-DEV P192C
  • Languages: DE, EN
Help needed! Future development structure for our pakset
« on: January 01, 2018, 07:20:55 PM »
I just tried to get a feeling about what is actually needed to convert our pakset to simutrans extended. Turns out graphics can stay the same in 9 out of 10 cases. But there a tons of things to be done that I could not imagine to do by hand. Also we're kind of running on the limit with what we have currently. I'll try to discribe our situation and what I think would be best to handle an extended tree and how to improve our infrastructure so we don't have that much work with .dat files we could spend on fancy graphics. I'ld appreciate any kind of thoughts how you would handle the challenge and what you do think about my ideas to do so!

Currently managing our pakset is a huge mess. Every part of it feels like it's kind of hacked for it's purpose. There is no system behind it, it's just the way it is. We have pricing for vehicles and some kind of idea how to name things, but that's about it.

At least I can't see a way to manage to get our pakset in extended without some changes to how we handle all our sources.
While we still can keep doing stuff the way it is nowadays, for me the time has come to start thinking about changing our "backbone" to something more powerfull than some folders with dat and png files in and an additional excel sheet for pricing.

So here are my first thoughts:

.dat files out of excel sheets (or similiar)
   I want to have a document which clearly is the single source for for information given in .dat files.
   That document has to be able to be changed by multiple people at the same time.
   There shall be a way to create .dat files out of such a sheet.
   maybe it's even possible to create diagrams out of it? like industry chains and timelines?
   Is there a way to automaticly update the text to the loading screen with each revision we upload?
   Could we do pricing for at least 2 versions? one for standard and one for extended? This should be something as easy as checking one checkbox when exporting the excel to .dat files. A thing to add here are how to handly liveries on a build for simutrans standard.

pak-ing the whole pakset from windows and linux machines
   basicly a no brainer, but not standard yet. really has to be done, our most invested developers all use windows

Gameplay
   Basicly our current balancing is all about runningcosts vs income. We do not consider different aspects nor do we have an idea what to actually could archive by balancing this game.
   We need something new from scratch.
   Pricing shall only be one part of it. There has to be a solution how to create challenges for every kind of game.
   Think about absolute untouched basics like "what's the value of a tile".

Graphics
   Where do we put the real sources? Is there a file format that can be used with at least gimp and photoshop?

So uhm, that's it for now, I'll try to keep this thread up to date when I have new thoughts about it. Please share your opinions and ideas around this topic! Also every kind of help is appreciated!

Offline jamespetts

  • Simutrans-Extended project coordinator
  • Administrator
  • *
  • Posts: 20346
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Help needed! Future development structure for our pakset
« Reply #1 on: January 03, 2018, 06:29:24 PM »
A large but worthwhile task. I do recommend Github, incidentally, for pakset sources (and, if you use rendered 3d models, the meta-sources, too), and also for all of the balancing spreadsheets.

Nightly builds are also worthwhile. Indeed, if the sources (and a makefile) were on Github, I could produce automated nightly builds using the latest makeobj and automate the distribution of this pakset with the Simutrans-Extended complete .zip file in the same way as I currently do for the Swedish pakset.

Offline Leartin

  • Heir-Benevolent-Dictator-Apparent
  • Moderator
  • *
  • Posts: 1473
  • PAK-DEV P192C
  • Languages: DE, EN
Re: Help needed! Future development structure for our pakset
« Reply #2 on: January 03, 2018, 08:47:37 PM »
(Wall of text incoming in 3... 2... 1...)

We are actually using a repository already, though it's on Assembla rather than Github, but there shouldn't be much difference (we certainly don't use everything we could on Assembla, so chances are we wouldn't use more tools on Github, and just treat it like a community dropbox anyway). Nightlys were once done automatically from there (before my time), and a makefile exists, so that should be fine.

Personally, I don't think the repository choice is an issue. The real problem is that this pakset was slowly growing without enough guidelines, so the sources are kind of spread.
Objects have partly german, partly english names, often including the author in the name as well. Sometimes, we replace graphics, such that the original author wouldn't even be the author anymore. Such inconsequent naming decisions may even cause objects of different types having the same name and overwriting one another. While I went through all citybuildings for 0.5 and made sure they all reached a certain quality standard, I renamed all of them following the same simple formula, and I'd like to do the same for every other type of object - but that's a lot of work, and I don't feel comfortable doing it for vehicles.
Next thing is folder structure. It's not very well thought out, as seen by things like "buildings/station/buildings". Largest issues is the half-height-folder - initially trying to provide both a full-height and a half-height pak, all new ways and similar structures were put in a seperate folder. But since then, we pretty much gave up on the full-height version, hence it should be integrated in the main body again. Then there are objects which were too large even for 192 size, which pretty much all got their own larger size, rather then just going for a certainly big enough 384-size for ALL of them - causing at least 6 folders named after their size and specifically mentioned in the makefile. Lastly, opposite to half-height folder, it seems many people don't enjoy the "infrastructure" chains (prison-chain and graveyard-chain), hence it would make sense to create one or two "official addons" out of them, which requires them to be in their own folder structure. Nothing of that is particularly difficult to do, but it requires time and, since it's a large change, should be done right, not just on a whim.
But thats, in the end, just a cleanup task. A lot of work, sure, but really anyone could do it.

To elaborate on the difficulties Flemm mentioned:
We can't pak under Windows! In preparation for the release, I set up a virtual machine with Ubuntu so I could use make, after "Make for Windows" just wouldn't work. In the meantime, that virtual machine broke once already, and since a makeobj for linux is rarely released I'm not sure I will always have the latest version - unless I self compile, which I already asked someone to teach me - but it seems needlessly complicated. Given the majority uses windows, we really need a solution for windows rather than linux/mac

We have no idea how to balance the pakset! I enjoy creating graphics, but I usually put them in the repository without the slightest idea about their balancing. Which does not matter for citybuildings, trees and other fancy eye candy, but since I do the same with factories... yeah... Pretty much everything is just leaning on what was there before, which was probably never balanced in the first place. We do have basic ideas about how it should work out, but nobody ever crunched the numbers to decide how they must be to achieve the vision we have.

We don't use the same graphics program. I use Photoshop and have PSD files with a few hundred layers for some buildings. But even if I would put them in the repository, Flemmbrav could not open them, since he uses GIMP. Neither could PTM, who uses Paint Shop Pro 5, nor Alex who uses MS Paint. Those two can't even open some of the PNGs we create since their programs don't support a full alphachannel. Since it's Pixel Art, not having those sources isn't quite as bad as with 3D graphics, but it's still much more difficult to form something new from an object half covered by a tree than if you had  that object in it's own layer. But this probably can't be helped, and we just need to decide how and where to save single objects from a larger composition for reuse purposes.


Really, it's just something grown organically without structure, and we just piled up on top of it instead of checking the basement, mostly because we started out without knowing how the game even works, and I would happily forget about that and just continue creating graphics, as that's the most fun for me. But if Flemmbrav does the same and nobody else comes along to do the grindy stuff, in 10 years this will be the best-looking worst-to-play pakset ever :(

Offline jamespetts

  • Simutrans-Extended project coordinator
  • Administrator
  • *
  • Posts: 20346
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Help needed! Future development structure for our pakset
« Reply #3 on: January 03, 2018, 09:59:28 PM »
I notice that Assembla hosts Git, SVN and another type of repository; is Pak192.Comic using a Git or a different type of repository? It would be easy for me to automate nightly builds using Git, but I am not sure how feasible that it would be with another type of repository (it might be possible with SVN; I have no idea about the other one).

As for balancing, when you eventually get it right, that's the most rewarding part of game creation.

Offline An_dz

  • Web Admin
  • Administrator
  • *
  • Posts: 2920
  • D'oh
    • by An_dz
  • Languages: pt, en, it, (de)
Re: Help needed! Future development structure for our pakset
« Reply #4 on: May 03, 2018, 08:00:00 PM »
(Wall of text incoming in 3... 2... 1...)

Now get ready for a REAL wall of text ;D

.dat files out of excel sheets (or similiar)
   I want to have a document which clearly is the single source for for information given in .dat files.
   There shall be a way to create .dat files out of such a sheet.
   That document has to be able to be changed by multiple people at the same time.
   maybe it's even possible to create diagrams out of it? like industry chains and timelines?

My suggestion is to use Excel Online with a xlsx file.

Why you can't put a sheet file in your repository? Because there's no format that is powerful enough to do what we want and that works with standard version control systems. Most files are compressed, others are one-liners, and each software (including versions of them) will save the file slightly different. Also Excel Online is truly online, when someone is editing something you can see in real time what they are doing.

Why not Google Sheets? Many reasons, let's list some.
  • On Excel only one person needs a Microsoft account.
    Others can edit with a special link, though there are advantages if everybody has an account as those with a link are seen as "Guests", on Google you need to mess with permissions to achieve something similar.
  • Excel has the familiar interface of the desktop version.
    Everything, including more powerful tools are easily accessible. Google Sheets only shows the most simple things, doesn't have some features and hides others very well. Google sheets is also a crap when scrolling as it always scrolls by cells rather than a normal page.
  • You can edit online with the real time multiple people feature on the desktop version (Office 2013+) or with the mobile apps.
    If a feature is not available on the online version you can use your unlimited desktop version and add it.
  • Excel online works out-of-the-box with xlsx files.
    Google Docs needs to convert the files and that sucks because with Excel you can download the file, edit on your computer to add features only available on the desktop version and upload back to OneDrive and everybody can enjoy.

And why xlsx? Because it's a file that is not hard to parse. So a script can extract information and generate/edit the files with the values. Also xlsx is a powerful format, you can have multiple sheets in a single document, link data between them and generate multiple graphics and calculations to analyse the pakset balancing.

   Could we do pricing for at least 2 versions? one for standard and one for extended? This should be something as easy as checking one checkbox when exporting the excel to .dat files. A thing to add here are how to handly liveries on a build for simutrans standard.

Certainly, as I said just above, xlsx is not hard to parse. But one must build the sheet in a way that makes parsing easier or predictable so the parser is easier to code.

   Is there a way to automaticly update the text to the loading screen with each revision we upload?

Yes, if using Git (which I believe you don't) you'll use the command git rev-parse --short HEAD to get a short revision string, if you are using SVN then you call svnversion.



Gameplay
   Basicly our current balancing is all about runningcosts vs income. We do not consider different aspects nor do we have an idea what to actually could archive by balancing this game.
   We need something new from scratch.
   Pricing shall only be one part of it. There has to be a solution how to create challenges for every kind of game.
   Think about absolute untouched basics like "what's the value of a tile".
We have no idea how to balance the pakset!

I believe this would need to be put in Excel and then as we see the whole data we can think about what connections can be made.



Graphics
   Where do we put the real sources? Is there a file format that can be used with at least gimp and photoshop?
We don't use the same graphics program.
Photoshop
GIMP
Paint Shop Pro 5
MS Paint.

GIMP can open and save (export) PSD files, GIMP is open-source and freely available for the three main desktop OSes. I can't see the problem here as long everybody has at least GIMP installed. Make sure to always have the latest version though as some PSD effects might not import/export correctly, but all my tries always worked fine.



pak-ing the whole pakset from windows and linux machines
We can't pak under Windows!
[...]
"Make for Windows" just wouldn't work.
I don't know which "Make for Windows" you got, but the one from MSYS2 should work flawlessly, and I have a tutorial on installing it here on our forum.

Given the majority uses windows, we really need a solution for windows rather than linux/mac
Why use makefile at all? Yeah, yeah, makefile is OS independent, blah, blah, blah. But it needs you to install make, while this is a given on Linux or Mac it's pretty frustrating on Windows, you generally need to install a wider package that contains make.

Instead, you can use batch/bash. Check pak96 source, I just updated them with easier commands that make stuff recursive.

In preparation for the release, I set up a virtual machine with Ubuntu so I could use make
[...]
In the meantime, that virtual machine broke once already

In my ultra honest opinion Ubuntu sucks. I won't get into details as I don't want to go off topic, but use some other distro, if you want something that is as easy to install as Ubuntu you can try Fedora.

and since a makeobj for linux is rarely released I'm not sure I will always have the latest version - unless I self compile, which I already asked someone to teach me - but it seems needlessly complicated.

Self compiling on Windows is easy with MSYS2 and, again, my tutorial.

On Linux it's also easy, at least on Arch it's a joke, on Fedora I don't remember but I think GCC and Make come pre-installed, but if not just need to install them, and you will also need to install libpng and pkg-config. Then do the same config.default file as explained in my MSYS2 tutorial and done.



Really, it's just something grown organically without structure

If it interests you this is the structure I have made for pak96, I believe it makes the most sense
Code: [Select]
- city-and-landscape
 └ buildings
  └ com
  └ cur
  └ ind
  └ mon
  └ res
  └ tow
 └ cars
 └ ground
 └ pedestrians
 └ rivers
 └ trees

- factories
 └ goods.dat

- other (misc)
 └ groundobj
 └ hq
 └ powerlines
 └ smoke
 └ waycrossings

- ui
 └ cursors
 └ menu
 └ symbols

- air
- rail
- rail-narrow
- road
- maglev
- monorail
- tram
- water

INSIDE EACH WAYTYPE FOLDER:

- buildings
- signs
- vehicles
- wayobjs
- ways
 └ tunnels
 └ bridges
 └ elevated

Offline Leartin

  • Heir-Benevolent-Dictator-Apparent
  • Moderator
  • *
  • Posts: 1473
  • PAK-DEV P192C
  • Languages: DE, EN
Re: Help needed! Future development structure for our pakset
« Reply #5 on: May 04, 2018, 09:50:28 AM »
Now get ready for a REAL wall of text ;D
Lies, it's all quotes, also it only looks more because there is more space between the lines  ;D

My suggestion is to use Excel Online with a xlsx file. [...] And why xlsx? Because it's a file that is not hard to parse. So a script can extract information and generate/edit the files with the values.
Sounds good. I'll check that out once the development wakes up again.

GIMP can open and save (export) PSD files, GIMP is open-source and freely available for the three main desktop OSes. I can't see the problem here as long everybody has at least GIMP installed. Make sure to always have the latest version though as some PSD effects might not import/export correctly, but all my tries always worked fine.
You can certainly export GIMP-files as PSD and open them in Photoshop without a problem, but the other way around is sadly not true. This is due to Photoshop having far more features which are not supported in GIMP, hence it can't handle them. For example Adjustment Layers or Smart Objects. I don't remember exactly what Gimp does with them, I think in most cases it just loses editability, though thats why you'd keep the source in the first place.

Instead, you can use batch/bash. Check pak96 source, I just updated them with easier commands that make stuff recursive.
Why Make? IDK. Why not Batch? IDK. I suppose the main reason would be that the only one who actually understands that stuff and set it up was Cruzer, who uses a Mac, so cross-platform support would be nice. Though, I suppose if both Make and Batch were available, that would be the best solution.



If it interests you this is the structure I have made for pak96, I believe it makes the most sense
It's interesting for sure. Ours for comparison:
Code: [Select]

- buildings
 └ 240  #oversized buildings in seperate folders for automated packing
 └ 384  #though "240" should be incorporated in "384"
 └ city #contains all dat files for citybuildings. Each file contains all buildings of one type in one climate, so the number of dat files is limited. Images are seperated in Sub-Folders
  └ images #shared graphics, like smoke or front-image-greenery
   └ com #images used in multiple climates
    └ alpine
    └ desert
    └ [each other climate]
   └ [ind, res, mon, lot, park, plaza]
    └ [each climate]
 └ cur #contains all dat files for curiosities, each file contains buildings of a vague type (architecture [eg. atomium], infrastructure [eg. radio tower], sacred [eg. church], intellectual [eg. museum], parks
  └ uncut #contains full graphics of multitile-buildings
  └ images
 └ depots #contains one depot dat and all images. images should be moved to subfolder
 └ factories #contains all dat files sorted in chains, eg. alcohol_chain contains hopfarm, maltworks, brewery, vineyard and pub as well as hop fields.
  └ uncut #contains full graphics of multitile-buildings
  └ images
 └ stations #contains all station dats, each file groups a bunch of stations (usually same waytype, same time period, same use (eg. stations for use on elevated rails together)
  └ [air, maglev, narrowgauge, road, track, water] #contains images of the corresponding waytype
  └ buildings #yep, subfolder buildings/stations/buildings, these are extension buildings for stations
  └ headquarter #don't ask, I have no idea why
 └ tows #contains dats only
  └ image #not imageS for no reason

- goods # contains a txt file listing the goods in an overview and a goods.dat with all goods.
 └ [easy, normal, hard, harder, damn_hard, dev] #different folders, all just containing one goods.dat with different values.
- ground #contains all you need for old ground (no halfheight) - texture, marker, sidewalk,...
 └ ground_objects #dats, batched in groups (ponds, stones, moving,..)
  └ image
- gui #(old) menu, skins, symbols, compass
 └ 128 #for the pakset logo
- pedestrians
- smoke
- tree #one dat, all graphics, no subfolder
- trunk #everything that gets copied not packed
- vehicles #contains an xls file and an ods file for planning/balancing
 └ 205 #oversized plane
 └ [air, citycar, maglev, road,...] #all the different types, containing dats, but this time, almost each vehicle has it's own dat
  └ image

-ways #worst part! most of this isn't even in use anymore!
 └ bridge #mostly everything randomly
  └ track #only graphics
 └ crossings
 └ roadsign #mostly dats
  └ [air, narrow, road, track] #graphics
 └ tunnel #mostly dats
  └ [narrowgauge, road, track] #graphics
 └ way #a mess I refuse to list in detail
 └ way_object #see above
 └ way250 #for an oversized elevated way

- halfheight #this was created for conversion, basically as an integrated fork, which replaces much of the stuff already listed
                   #contains dat-files, each waytype has at least one, some are split (eg. "road.dat" and "road_tunnel.dat")
 └[32,64,250] #for everything with larger/smaller graphics, eg. menu and an elevated road
 └aviation #all images related to planes. There are also some buildings in here, making it even more confusing
 └[tram, road, wall, electro,...] #contain images

I think a good solution would be to work toward this:
Code: [Select]
- buildings
 └ 384  #for everything oversized, just use 384, even if the image is actually smaller. The empty space barely matters in the grand scheme of things.
 └ city #no change
 └ cur #no change
 └ factories #almost no change except factory-smoke goes here and...
  └ goods #becomes a subfolder of factories.
 └ tows #no change (except imageS)
 └ headquarter

- landscape #new collection folder to include...
 └ ground #textures, transistions, heightmap, basement, sidewalk,...
 └ ground_objects
 └ tree
 └ pedestrians

-vehicles
 └ 384 #everything oversized
 └ smokes (for vehicles)
 └ [air, maglev, narrowgauge,...] #combine dats where it makes sense
  └ imageS
 └ train #for better overview (since there are too many):
  └ locomotives
  └ goods_wagons
  └ pax_wagons
  └ sets #eg. ICE should not be seperated, so all parts of it use one ICE.dat in the sets folder.
 └ road
  └ bus?
  └ truck?
  └ citycar

- infrastructure
 └ 384 #everything oversized
  └ images
 └ buildings #all extension buildings
  └ images
 └ crossings
 └ aviation #contains all dats, grouped by type (stations, depot, signs, ways, ...)
  └ images #contains all graphics
 └[electro, tram, track,...] #all other waytypes
  └ images

- UI #cursors, symbols, marker,...
 └ 32 #menu icons
 └ theme

Offline DrSuperGood

  • Dev Team
  • Devotee
  • *
  • Posts: 2835
  • Languages: EN
Re: Help needed! Future development structure for our pakset
« Reply #6 on: May 04, 2018, 10:48:36 AM »
As part of my balance work on extended I have kind of created a java tool which can extract mappings from .dat files and place them inside a UK/US formated CSV spreadsheet. This CSV spreadsheet can then be promoted to a normal Microsoft Spreadsheet inside Excel for modification. One can then flatten this normal Excel spreadsheet and save it as a UK/US formatted CSV. The program will then parse this CSV file and automatically insert, replace or remove mappings that thave changed from the various .dat files. It cannot automatically create new entires, it was hacked together for modifying the balance of an extremely complex existing pakset (Pak128Britain Extended) rather than making a new pakset. It also features some kinds of error detection, such as duplicate mapping detection, which makeobj does not.

Why CSV? Because UK/US formated CSV is so easy to parse I did not even need to write a separate function for it (although I probably should have...). Of course it could probably be adapted to support other spreadsheet formats with the help of some third party libraries such as apache commons. I just did not want to spend 99% of the time writing a spreadsheet API.

Offline Leartin

  • Heir-Benevolent-Dictator-Apparent
  • Moderator
  • *
  • Posts: 1473
  • PAK-DEV P192C
  • Languages: DE, EN
Re: Help needed! Future development structure for our pakset
« Reply #7 on: May 04, 2018, 11:05:37 AM »
So I read that correctly, you can create a spreadsheet from the contents of dat files, edit values, and place them back into dat files; but no new dat files are created, only existing ones changed?

Offline An_dz

  • Web Admin
  • Administrator
  • *
  • Posts: 2920
  • D'oh
    • by An_dz
  • Languages: pt, en, it, (de)
Re: Help needed! Future development structure for our pakset
« Reply #8 on: May 04, 2018, 05:32:56 PM »
That's what @DrSuperGood script does, but you can create one that generates dats.

Lies, it's all quotes, also it only looks more because there is more space between the lines  ;D
Lies, Deceptions! ;D

Offline DrSuperGood

  • Dev Team
  • Devotee
  • *
  • Posts: 2835
  • Languages: EN
Re: Help needed! Future development structure for our pakset
« Reply #9 on: May 04, 2018, 08:52:41 PM »
Quote
So I read that correctly, you can create a spreadsheet from the contents of dat files, edit values, and place them back into dat files; but no new dat files are created, only existing ones changed?
Yes that is how I am modifying the balance of pak128Britain Extended.

Pak128Britain Extended has extensive commentary in its hand made dat files which has to be preserved and kept in the correct order, as such I needed a system to interact with existing dat files in a sensible way, rather than out right making new ones. Obviously if you build the entire pakset dat from a spreadsheet this is not a problem, and in such a case the writer part could be modified to create entirely new dat files in I recon 15-45 minutes of programming.

One needs to define a clear ordering of records within dat files. As such one would need to either automatically order them based on the value of a field or explicitly order them with an index. I went with the index approach as I had to locate specific records to modify and debug, but one might as well go with an automatic ordering approach if one is generating all dat files.
« Last Edit: May 05, 2018, 04:03:04 AM by DrSuperGood »

Offline Flemmbrav

  • Moderator
  • *
  • Posts: 222
  • PAK-DEV P192C
  • Languages: DE, EN
Re: Help needed! Future development structure for our pakset
« Reply #10 on: January 02, 2020, 09:07:29 PM »
I want a wall of text too, but i'm late af since i've missed all of this sadly.
But we'll get there! And it's a bit short to call it a wall, but i wanted to give my 2cents too...

First and foremost, an update on the current state of dev:
Ve've moved the repo to github. https://github.com/Flemmbrav/Pak192.Comic
Additionally, there are shell scribts to compile the pakset for windows and unix. These are powershell? based and seem to work just fine.
Compiling speed is heavily affected by the time it takes to generate the image locations out of shortened .dat files.
The scribts are 99% the same for both systems, the only difference is that the windows one adresses "makeobj.exe" while the unix one adresses "makeobj".

Makobj itself is in the repo. And yes, it's there twice. I do not like this at all, there has to be a way to link this in there instead. Kind of like how I imagine dependancies work on unix. If you want to compile the pakset, and don't have it there, you should get it there from the makeobj repo.

I currently try to get DatSheet working in our repo. Same dependancy issue here, I feel bad about just copying and uploading the working program in there. In plus i'm a bit clueless on how to make good use of it by now, as I'd love to have all at one place rather than depending on the Google/Microsoft cloud in plus.
While I like the idea of having it all in one sheet, I don't believe that's what we should aim to have.
I favor to use the current .dat files as data source, since Git can handle them easily. That would also mean, that you could easily add new stuff instead of manually converting it into the .xlsx file. The .xlsx file would still be used to calculate values, or do mass changes. I prefer .xlsx here, since we have the option, and it can store complicated formulas quite well, even thoguht i do have more experience just dumping all the data in .csv files ^^. It jsut should not be part of the reposity itself, and rather be generated, used and converted to .dat files after.
This might result in the need of a master .xlsx file, that contains formulas, and a small piece of code, that overwrites the generated .xlsx file with the formulas from the formula sheet.
I do promise myself to have an easy time figuring a nice pricing with this, thus I really do want to have it in there.
In plus, there is some talk on how to figure values that are not prices. You may have noticed that we didn't really care that much before, that's subject to change. I do want to implement this with the auto generated values DatSheet can provide us.

This also brings me to the extended stuff.
As far as I understood the way DatSheet works, the tool doesn't give a **** about the actual commands, and just copies the .dat files in a sheet. We could make use out of that, and store the information we need for both versions of the game. Would have the advantage of having all the data at one place and being able to use all of them for calculations.
I'm not aware if there are any values that really do conflict between them. If there would be a chance to keep all the needed data in the .dat files, the defnining data could be stored in the .dat files, and the other ones could be calculated with different .xlsx templetes. After that, new .dat files are generated, that use the correct commands for the version of simutrans to use.

On the source files for images:
Simutrans uses transparencies by now, thus we have them too. I've started to make use out of the alpha channel support by adding shades of the light blue as invisible markers. This means, that Alex can't see the files anyways. In plus, I've started to put my new .xcf (GIMP) files in a different folder within the Git, so people could work with them.
Last time I've checked, layers within layer groups disappeared during the .psd (Photoshop) export, thus I won't do that. On the other hand, Leartin uses some Photoshop exclusive features resulting in me not being able t make use out of his files. This whole thing is still a huge mess.


Offline jamespetts

  • Simutrans-Extended project coordinator
  • Administrator
  • *
  • Posts: 20346
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Help needed! Future development structure for our pakset
« Reply #11 on: January 02, 2020, 10:09:04 PM »
It is lovely to see work resume on this pakset again - it offers a delightful and refreshing change to the more realism focussed paksets. It would be splendid to see a Simutrans-Extended version of this, too.

Offline Qayyum

  • *
  • Posts: 163
Re: Help needed! Future development structure for our pakset
« Reply #12 on: January 03, 2020, 10:04:20 AM »
Me thought that in order to know what to balance the prices, first you have to know what is the minimum living wage, with rent, second, you have to know the minimum wage, without rent, third, you have to know that value of land rises as nmore land is used.

Offline Flemmbrav

  • Moderator
  • *
  • Posts: 222
  • PAK-DEV P192C
  • Languages: DE, EN
Re: Help needed! Future development structure for our pakset
« Reply #13 on: January 03, 2020, 10:40:42 AM »
Honeslty I doubt prices itself really are an issue with what excel can provide us there.
It's also not like the current pricing is that horrible, sure, there are some things, that didn't see a proper pricing, but most vehicles at least are priced in a way that makes sense to some extend.
Further, we did some theory crafting on it here https://www.simutrans-forum.de/mybb/showthread.php?tid=9041 (sadly German).
More interesting are things like loading_time, industry out and input and the likes of that.

I personally would love to see the pakset in extended too! I just have no idea on how to get it working there. Like, I can't get it running at all, nor do I really set the prio on fixing that, as it seems to be quite time consuming and not as essential as drawing cars for goods that are in the game but can't be transported yet... In plus we use shortened .dat files and multi-tile-city buildings, these will have to be converted or supported.

@James
you did offer to do a nightly compiler before, is that still a thing? What do you need to get that running?

Offline jamespetts

  • Simutrans-Extended project coordinator
  • Administrator
  • *
  • Posts: 20346
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Help needed! Future development structure for our pakset
« Reply #14 on: January 03, 2020, 11:27:46 AM »
@James
you did offer to do a nightly compiler before, is that still a thing? What do you need to get that running?

Yes, indeed - what we need to set up nightly compilation for an Extended version on the Bridgewater-Brunel server is the full set of sources on a Github repository and a working Linux makefile for them.

Offline jamespetts

  • Simutrans-Extended project coordinator
  • Administrator
  • *
  • Posts: 20346
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Help needed! Future development structure for our pakset
« Reply #15 on: January 11, 2020, 11:33:22 AM »
I notice that the sources are available on Github, but there does not appear to be a makefile. Here is the Pak128.Britain-Ex makefile; this should give some idea of what is involved in creating a makefile so that you can create one for this pakset so that I can create automatic nightly builds of the Extended version of this pakset.

It would be a splendid thing for Pak192.Comic to join the ranks of the Extended paksets.

Edit: Also - I have added reference to Pak192.Comic in the list of Simutrans-Extended compatible paksets: I should be grateful if you could let me know if there is any information there that is incorrect. Incidentally, may I ask under which licence that that the sources are made available?

Edit 2: I notice the reference above to multi-tile city buildings: I should note that these are already supported in Extended.

Offline Leartin

  • Heir-Benevolent-Dictator-Apparent
  • Moderator
  • *
  • Posts: 1473
  • PAK-DEV P192C
  • Languages: DE, EN
Re: Help needed! Future development structure for our pakset
« Reply #16 on: January 13, 2020, 05:52:31 AM »
We use scripts instead of a makefile, since we are dumb windows users who can't even install make.
The license is CC BY-SA 3.0

Offline jamespetts

  • Simutrans-Extended project coordinator
  • Administrator
  • *
  • Posts: 20346
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Help needed! Future development structure for our pakset
« Reply #17 on: January 13, 2020, 11:11:21 AM »
We use scripts instead of a makefile, since we are dumb windows users who can't even install make.
The license is CC BY-SA 3.0

Excellent - I approve of open licences.

You can use make in Windows, and this is what I now use at home to compile the pakset on my new Windows computer. All that you have to do is use the Git command line shell and install make there. It is quite straightforward, and there are some simple online tutorials about this easily revealed by searching.

Once you have a working makefile, I can set up nightly compiles of the Extended version.

Offline Freahk

  • Devotee
  • *
  • Posts: 1357
  • Languages: DE, EN
Re: Help needed! Future development structure for our pakset
« Reply #18 on: January 13, 2020, 11:44:26 AM »
If the compile.sh is up-to-date it should not be an issue to call the script instead of make from the linux build server.
However, maintaining different build scripts for different platforms is quite doubled amount of work to maintain and can lead to hard to locate network incompatibilies in between platforms especially when addons are used.

Offline jamespetts

  • Simutrans-Extended project coordinator
  • Administrator
  • *
  • Posts: 20346
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Help needed! Future development structure for our pakset
« Reply #19 on: January 13, 2020, 12:23:47 PM »
I did not realise that the compilation script was a Linux shell script - is this so? If one can have a "compile.sh" script, surely one can simply create a makefile?

Offline Freahk

  • Devotee
  • *
  • Posts: 1357
  • Languages: DE, EN
Re: Help needed! Future development structure for our pakset
« Reply #20 on: January 13, 2020, 01:25:38 PM »
I didn't try it out but there is a COMPILE.sh, which is a bash script, thus linux. However, there does not seem to be bash script for extended compile but I suspect adding that should not be a huge thing.

Offline prissi

  • Developer
  • Administrator
  • *
  • Posts: 10306
  • Languages: De,EN,JP
Re: Help needed! Future development structure for our pakset
« Reply #21 on: January 13, 2020, 01:39:04 PM »
It is very easy to have script directly generate nightlies on github usig actions. If you have a linux script that calls compile.sh the following file (save as .github/workflow/build_nightly.yml) will do the trick: (Compilation of makeobj can be removed, and of course not tested)

Code: [Select]
name: Nightly build Ubuntu

on: [push]

jobs:
  build:

    runs-on: ubuntu-latest
   
    steps:
    - uses: actions/checkout@v1

    - name: install_dependencies
      run: |
        sudo apt-get -y update
        sudo apt-get -ym install libpng-dev
        sudo apt-get -ym install libsdl2-dev
        sudo apt-get -ym install libbz2-dev
        sudo apt-get -ym install autoconf
        svn checkout svn://servers.simutrans.org/simutrans simutrans

    - name: setup
      run: |
        cd simutrans/trunk
        autoconf
        ./configure
        cat config.default >>/dev/stderr

    - name: make makeobj
      run: |
          cd makobj
          make
          cd ..
          cd build
          cp makeobj ~

    - name: make pak192
      run: |
        cd ~
        sh ./compile.sh

    - name: Rename result
      run:  mv RESULT.zip pak192-nightly.zip

    - name: Update binaries of Nightly Release
      uses: svenstaro/upload-release-action@v1-release
      with:
        repo_token: ${{ secrets.GITHUB_TOKEN }}
        file: ./pak192-nightly.zip
        asset_name: pak192-nightly.zip
        tag: Nightly
        overwrite: true
« Last Edit: January 13, 2020, 02:03:29 PM by prissi »

Offline Leartin

  • Heir-Benevolent-Dictator-Apparent
  • Moderator
  • *
  • Posts: 1473
  • PAK-DEV P192C
  • Languages: DE, EN
Re: Help needed! Future development structure for our pakset
« Reply #22 on: January 13, 2020, 09:43:29 PM »
Code: [Select]
      run: |
          cd makobj
          make
          cd ..
          cd build
          cp makeobj ~
This is the part were I get stuck. I was able to fix a few issues - well, this is what I got:
Code: [Select]
      run: |
          cd simutrans/trunk/makeobj
          #typo, and since it's a new shell instance it's back outside.
          make
          cd ..
          cd build/default
          #makeobj is put in the default subfolder according to it's makefile
          cp -r -v makeobj ../../../../
          # ~ does not put me in the folder I start a shell instance from, so I manually "walk back".
But it doesn't matter, since I don't get an actual makeobj in the first place - instead, I get a folder "makeobj" which contains "makeobj-makeobj.o" and "makeobj-makeobj.d" and I have no idea what to do with them.  :-[


Oh yeah, in case that matter, the last few lines of the makefile execution:
Code: [Select]
===> CXX ../utils/log.cc
../utils/log.cc: In member function ‘void log_t::fatal(const char*, const char*, ...)’:
../utils/log.cc:266:6: warning: unused variable ‘n’ [-Wunused-variable]
  int n = vsprintf( buffer, formatbuffer, argptr );
      ^
===> CXX makeobj.cc
===> LD  /home/runner/work/Pak192.Comic/Pak192.Comic/simutrans/trunk/makeobj/makeobj


Offline prissi

  • Developer
  • Administrator
  • *
  • Posts: 10306
  • Languages: De,EN,JP
Re: Help needed! Future development structure for our pakset
« Reply #23 on: January 17, 2020, 07:05:02 AM »
This output looks already very good. Just the makefile is in the makeobj folder.
Code: [Select]
name: Nightly build Ubuntu

on: [push]

jobs:
  build:

    runs-on: ubuntu-latest
   
    steps:
    - uses: actions/checkout@v1

    - name: install_dependencies
      run: |
        sudo apt-get -y update
        sudo apt-get -ym install libpng-dev
        sudo apt-get -ym install libsdl2-dev
        sudo apt-get -ym install libbz2-dev
        sudo apt-get -ym install autoconf
        svn checkout svn://servers.simutrans.org/simutrans simutrans

    - name: setup
      run: |
        cd simutrans/trunk
        autoconf
        ./configure
        cat config.default >>/dev/stderr

    - name: make makeobj
      run: |
          cd makobj
          make
          mv makeobj ../../..
          cd ../../..
          rm -rf simutrans

    - name: make pak192
      run: |
        sh ./compile.sh

    - name: Rename result
      run:  mv RESULT.zip pak192-nightly.zip

    - name: Update binaries of Nightly Release
      uses: svenstaro/upload-release-action@v1-release
      with:
        repo_token: ${{ secrets.GITHUB_TOKEN }}
        file: ./pak192-nightly.zip
        asset_name: pak192-nightly.zip
        tag: Nightly
        overwrite: true


Offline Leartin

  • Heir-Benevolent-Dictator-Apparent
  • Moderator
  • *
  • Posts: 1473
  • PAK-DEV P192C
  • Languages: DE, EN
Re: Help needed! Future development structure for our pakset
« Reply #24 on: January 17, 2020, 09:06:02 AM »
W00t! Thank you, it works <3

Just to confirm:
  • The build-folder is just residue from the compilation process, there is nothing operational there
  • These commands run in a sandbox - nothing inside the repository changes through the action, except for the explicity released zip file, eg. removing the simutrans folder after you got the makeobj is just style, but it would be gone anyway.
  • In my test-branch, the release states '@github-actions github-actions released this 3 days ago'. This date is just when the action was first run and will stay, the end consumer won't know how old the files are.

Offline prissi

  • Developer
  • Administrator
  • *
  • Posts: 10306
  • Languages: De,EN,JP
Re: Help needed! Future development structure for our pakset
« Reply #25 on: January 17, 2020, 03:36:15 PM »
Sometimes something inside the runner remains, but outside of the home directory.

I removed simutrans, since this would be the directory pak64 would be packed to. I did not know your packing, so I just played safe.

The "artefact" i.e. the zip file will be updated, but the tag will stay with the strage name, I think. (For testing I used empty submits, so there was no further string). I am new to this as well ...

Offline Flemmbrav

  • Moderator
  • *
  • Posts: 222
  • PAK-DEV P192C
  • Languages: DE, EN
Re: Help needed! Future development structure for our pakset
« Reply #26 on: February 09, 2020, 02:48:25 PM »
Hello again,

I wanted to give a brief update on what we did so far, what I'm at right now and wanted to talk a bit on how to store data for the game.

Creating nightlies works fine for us.
We have seperated zips for the pakset and the addons we have in the Git too. It takes a while to create them, that's for mutliple reasons. First we compile makeobj every time we create the pakset. Second we rebuild the whole pakset with every change. I'm not sure if we really should be doing that - on the other hand, not doing so would require the scribt to be able to permanently write in the Git. I'm not sure if that's a thing sadly.

Further, I've started writing a bash scribt, that searches for dat files, opens them, adds or replaces certain values, and re-writes the dat in a different folder. my idea of the final product is to have a scribt that adds all the stuff, we did not know yet as we made a dat. Like if I don't know how much it should cost, it will add estimated values to the game, without modifying the old dat. It could also be used to have constraint classes internally, that keep track of all the vehicles in the class, and copies them in the constraints after.
That's also what I want to talk about next. With this tool I have the advantage of haveing the pricing automated, while also being able to have multiple people working on the Git at the same time. Every object has their own small file with all the stuff I know about the object in there. Ideally the scribt builds me a dat for standard and another dat for extended out of that. But that file can't be compiled by makobj itself. Leartin mentioned having issues with files being stored like that, without storing the working dat files in plus, and I'd love to know a few more opinions on that.
To summerize it a bit;
We have right now: Git stores: dat + png ; Building it by iterating standard makobj over every dat
What my new scribt does: Git stores: notworking.dat + png ; Building it by copying the png files, iterating the scribt over the not working dat files, and iterating makeobj over the freshly generated dats after.

I'm not really sure on how to use it in future. Do you think storing working dat files in the Git should be the way? Is there a way to generate these with every commit? Is it enough to provide a scribt that creates all the working dat files?