The International Simutrans Forum

 

Author Topic: Transparent Workflow - Tidying up and moving on the work  (Read 1208 times)

0 Members and 1 Guest are viewing this topic.

Online Spenk009

  • *
  • Posts: 280
Transparent Workflow - Tidying up and moving on the work
« on: November 05, 2019, 09:34:25 PM »
I've been trying to continue the conversion to transparency, but got stuck on the SR3-Subs. Since then I've been doing work here and there, focusing efforts on fixing bugs in the pakset mentioned in the forum.

The trains images folder contains 17000 images. I'm working from a SATA SSD and James soon from an M.2 SSD, but the loading takes quite some time to complete. I have removed many (hopefully all) unused images from "trains/images", almost 800 images, branch here. Would it make sense to split these into the same categories as the Blends repository has? I know there are problems with applying this (especially DMUs/EMUs, fast freight, etc.), but it would improve working in the repo.

I had a lot of trouble due to a lack of resources and knowledge regarding the LBSCR 48ft, balloon and further carriages. I'll try to find other vehicles to convert instead. I've seen many steam locomotives waiting for their update, so I'll try focusing on that for the moment.


Offline DrSuperGood

  • Dev Team
  • Devotee
  • *
  • Posts: 2748
  • Languages: EN
Re: Transparent Workflow - Tidying up and moving on the work
« Reply #1 on: November 06, 2019, 02:37:27 AM »
The trains images folder contains 17000 images. I'm working from a SATA SSD and James soon from an M.2 SSD, but the loading takes quite some time to complete.
The loading? Access to a file within a folder should be O(log2(n))) so even if the folder has 17,000 files loading each specific file should be trivial, especially with a SSD.

That said I do agree that assets should be separated into folders on a per vehicle or vehicle set basis rather than in a huge folder. One could also include the data files for the vehicles with it if the build process allows.

Online Spenk009

  • *
  • Posts: 280
Re: Transparent Workflow - Tidying up and moving on the work
« Reply #2 on: November 11, 2019, 11:50:10 AM »
I know it should be trivial, but it's also a question of updating the folder when files are pasted into it. I've started adding some more transparent vehicles and am slowly incorporating the solution. The Gresley Express vehicles consist of 560 individual images alone. The folder would balloon in due course, so I'm setting up folders as I go along.

I would like to note a completion of the LNER vehicles aside from the coronation sets. Branch here.

Offline Freahk

  • *
  • Posts: 228
  • Languages: DE, EN
Re: Transparent Workflow - Tidying up and moving on the work
« Reply #3 on: November 11, 2019, 12:00:17 PM »
Just for curiosity, may I ask how transarency can improve vehicle images?
I expected transparency to be pretty useful for things that are transparent in reality e.g. glass station roofs but I don't see such things at trains. Given the viewing angle one can not look through the windows to the other side of the train.

Online Spenk009

  • *
  • Posts: 280
Re: Transparent Workflow - Tidying up and moving on the work
« Reply #4 on: November 11, 2019, 01:48:29 PM »
Mainly smaller details can now be rendered, e.g. pantographs and masts, while not occupying a whole pixel but instead blending in better. Old horse drawn carriages, trees and thin masts really benefit from this.

Have a look at this thread:
https://forum.simutrans.com/index.php/topic,15314.msg157307.html#msg157307

Offline Freahk

  • *
  • Posts: 228
  • Languages: DE, EN
Re: Transparent Workflow - Tidying up and moving on the work
« Reply #5 on: November 11, 2019, 02:18:01 PM »
Thanks,
Can't see a difference for the bus wheel example to be honest. I guess that's why I am not an artist.
Maybe one can better see this on moving objects.

Offline DrSuperGood

  • Dev Team
  • Devotee
  • *
  • Posts: 2748
  • Languages: EN
Re: Transparent Workflow - Tidying up and moving on the work
« Reply #6 on: November 11, 2019, 06:13:43 PM »
I know it should be trivial, but it's also a question of updating the folder when files are p
Should still be trivial for computers. Modern file systems were designed with this sort of extreme usage in mind and so scale very well.

Of course if the file explorer used to manually view and manipulate the folder scales as well is another question...
Just for curiosity, may I ask how transarency can improve vehicle images?I expected transparency to be pretty useful for things that are transparent in reality e.g. glass station roofs but I don't see such things at trains. Given the viewing angle one can not look through the windows to the other side of the train.
Mostly due to anti-aliasing around the edges of the object. Allows the edges to appear smoother or smaller edge details captured.

Online Spenk009

  • *
  • Posts: 280
Re: Transparent Workflow - Tidying up and moving on the work
« Reply #7 on: November 11, 2019, 07:48:59 PM »
Several questions for both the Blends repo and the pak repo:

The LNER Coronation carriages are missing, these were added by the Hood back in 2009. Is there any chance we can get these? I remember reading about more problems with getting old sources of files due to his departure. commit
The LMS Coronation Scots are written as corination, is this a mishap from the Christmas pakwork frenzy? commit
A while back, quite a few source images were added into the root of the pakset. May I move these into a separate sources folder? They are valuable drawings and should probably be preserved. commit


Should still be trivial for computers. Modern file systems were designed with this sort of extreme usage in mind and so scale very well.
They have improved significantly. I reckon that the pak building time is reduced too, as splicing the images and creating a new transparent layer is omitted for a large portion of the pak.
« Last Edit: November 11, 2019, 08:10:33 PM by Spenk009 »

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18763
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Transparent Workflow - Tidying up and moving on the work
« Reply #8 on: November 12, 2019, 01:07:29 AM »
Thank you for your work on this. I should be able to look at this properly once I have my new computer and can sensibly test things.

In the meantime, if you have identified any .png files of the old workflow that are no longer required by the current .dat files, feel free to delete these from your repository (the deletions should be in their own commit, however) so that these deltions occur when I merge with your repository.

However, as Dr. Supergood says, there may be something a little wrong if there just being that many images makes your computer that slow, although possibly this may be caused by a setting to give a preview image of each one; you might change to a detailed view without a thumbnail to check whether this improves the folder response times.

As for missing Blender files, I am afraid that I cannot assist with these; some may have been lost entirely. There may be some cases where it is necessary in due course to re-create the Blender files from scratch if it is desired to render these with the transparent workflow.

As to the 48ft Brighton carraiges, I may well be able to find research material on these if you let me know what information that you need.

Online Spenk009

  • *
  • Posts: 280
Re: Transparent Workflow - Tidying up and moving on the work
« Reply #9 on: November 12, 2019, 08:08:09 PM »
Thank you for your work on this. I should be able to look at this properly once I have my new computer and can sensibly test things.
In the meantime, if you have identified any .png files of the old workflow that are no longer required by the current .dat files, feel free to delete these from your repository (the deletions should be in their own commit, however) so that these deltions occur when I merge with your repository.
However, as Dr. Supergood says, there may be something a little wrong if there just being that many images makes your computer that slow, although possibly this may be caused by a setting to give a preview image of each one; you might change to a detailed view without a thumbnail to check whether this improves the folder response times.
As to the 48ft Brighton carraiges, I may well be able to find research material on these if you let me know what information that you need.

You're most welcome. No need to hurry anything that is a hobby.

Old .png files are removed in the commit that they are replaced in. If you're referencing removing all unused images in one step/commit, the relevant commit is here

I don't think there is anything wrong with my machine, loading only initially takes around 15 seconds. Any subsequent opening of the folder below 2 seconds. It is just looking forwards into when we end up with many more images in the folder.

The issue is the selection of windows from their respective layer in order to produce the correct images. It will take some time to get to, so instead I hope to focus efforts on easier tasks to reduce the existing backlog.


I've completed the LNWR Locomotives for transparent vehicles. Testing is completed. I may have missed one or two, but these are best reported as bugs
« Last Edit: November 21, 2019, 09:13:11 PM by Spenk009 »

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18763
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Transparent Workflow - Tidying up and moving on the work
« Reply #10 on: November 12, 2019, 08:30:58 PM »
Splendid, that is very helpful: thank you.

Online Spenk009

  • *
  • Posts: 280
Re: Transparent Workflow - Tidying up and moving on the work
« Reply #11 on: November 23, 2019, 08:47:07 PM »
Hi everyone, less of a progress update but a tools one.

I've created a script to relocate the images of vehicles (called relocator). It changes the images folder in the dat file and moves the images accordingly. It's simple, but should help make the move to separate folders.



I have also included the script to easily rebuild the pakset and move it to the appropriate folder (called pakupdater). It's not perfect, but problems during the compilation surface recognizably. It takes my machine around 65-70 seconds to complete.

The third script renames the images to the correct type. I will need to add handling for cases where the old images were only one level tall ( eg. .9 rather than 1.9 for the image definition).

Have a gander at these, maybe they can help you in the pak maintainance of your own pak. Feel free to share how long the pak building script takes on your machine. Please keep in mind that these are first advances into dabbling with bash scripts. They are a bit crap and include bad writing, but do the job for now.

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18763
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Transparent Workflow - Tidying up and moving on the work
« Reply #12 on: November 24, 2019, 12:24:26 PM »
Interesting. Do I understand that it is the .dat and .png files that you are moving? Does this script update all the necessary matters both in the makefile and the Python script used for building the pakset (the former for Linux the latter for Windows)?
Edit: Incidentally, I see that you have two transparent vehicles branches on your Github. Are these ready to be pulled yet, or do they need further testing/refinement?

Online Spenk009

  • *
  • Posts: 280
Re: Transparent Workflow - Tidying up and moving on the work
« Reply #13 on: November 24, 2019, 03:55:27 PM »
Interesting. Do I understand that it is the .dat and .png files that you are moving? Does this script update all the necessary matters both in the makefile and the Python script used for building the pakset (the former for Linux the latter for Windows)?
I only intend to separate the images, not create further work by editing the compilation (and/or scripts). The dat files only have the image location changed within themselves. The images are moved to the appropriate folder.

Incidentally, I see that you have two transparent vehicles branches on your Github. Are these ready to be pulled yet, or do they need further testing/refinement?
No, these are still a work in progress. The main branch for the changes is the transparent-vehicles branch, with these sorting changes applied in there. Once presentable, I will create a pull request and mention it here. The SR-transparent branch was an attempt at unnecessarily splitting work up, I'll remove it in due course.

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18763
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Transparent Workflow - Tidying up and moving on the work
« Reply #14 on: November 25, 2019, 02:03:00 AM »
Thank you for the clarification - that is most helpful.

Online Spenk009

  • *
  • Posts: 280
Re: Transparent Workflow - Tidying up and moving on the work
« Reply #15 on: November 25, 2019, 09:53:23 PM »
I am making several commits to stage the work, but will try to limit these a little in order to simplify the process. I try to push working (i.e. compiling) stages to the repository, but this may not always be the case. All BR, LMS and goods vehicles have been moved, around 6000 images so far.

The lms-d1904 and lms-d2007 are parcel vans. They have not been implemented as vehicles yet (I can't find any trace of their usage). Would we want to implement these as brake vans or just empty carriages that are there for visuals/realism?

I envision the creation of a script that checks whether an image is actually used. When this is around, we can check for other cases like this.

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18763
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Transparent Workflow - Tidying up and moving on the work
« Reply #16 on: November 26, 2019, 11:50:48 AM »
The general rule that I follow for Simutrans-Extended is that everything that has an economic function in reality must have an economic function in the game. Thus, there should be no vehicles that are for cosmetic effect only. The parcels vans should be used for mail in Simutrans. However, you may want to check to see whether these have been superseded by something else, as vehicles that are simply not used are often not used for good reason (usually that developments have rendered them redundant, e.g. by providing a more accurate alternaive).

Online Spenk009

  • *
  • Posts: 280
Re: Transparent Workflow - Tidying up and moving on the work
« Reply #17 on: November 26, 2019, 09:25:17 PM »
The parcels vans should be used for mail in Simutrans. However, you may want to check to see whether these have been superseded by something else, as vehicles that are simply not used are often not used for good reason (usually that developments have rendered them redundant, e.g. by providing a more accurate alternaive).
Thanks, I wasn't sure whether they were additional mail carriages or carriages to help luggage requirements on longer routes.

I have finished the moving of almost all railway vehicles, found here. The only remaining ones are horses , their stagecoaches and alignment templates. It ended up as one big commit in the very end, because I am no master of managing inclusions and exclusions in a commit. The pakset compiles and the folders are reasonably fast to navigate.

as a side note:
Should still be trivial for computers. Modern file systems were designed with this sort of extreme usage in mind and so scale very well.
I had to switch to a laptop and was recently looking to expand the storage. While there are two M.2 slots in the machine, they are limited to 2242 dimensions. I ran a speed test on the internal SSD, with the result of 500mb/s. Hoping to break 1GB/s I looked into expansion via M.2, but the shorter cards tend to have lower speeds. I've taken the route of a bay adaptor that will be filled with a hdd for media. You're right, they do scale extremely well. I just think something runs out of cached folder content at 18000 files.

Offline jamespetts gb

  • Simutrans-Extended project coordinator
  • Moderator
  • *
  • Posts: 18763
  • Cake baker
    • Bridgewater-Brunel
  • Languages: EN
Re: Transparent Workflow - Tidying up and moving on the work
« Reply #18 on: November 26, 2019, 10:58:53 PM »
Thank you. Can I confirm whether these are ready for testing/integration yet? Your work in this regard is much appreciated.

Online Spenk009

  • *
  • Posts: 280
Re: Transparent Workflow - Tidying up and moving on the work
« Reply #19 on: November 27, 2019, 10:15:22 PM »
These I confirm to be working. Would you like to merge them in yourself or should I create a pull request?

I'll continue by converting further vehicles to the transparent workflow. Just out of curiosity, when do you see yourself working on the pakset if you find time over the holidays next month? I'd need your assistance in the matter of a few vehicles (e.g. SR-EMUs and their varying files & configurations), but would prefer to have completed further work by then and have compiled a small list to give an overview.

Online Spenk009

  • *
  • Posts: 280
Re: Transparent Workflow - Tidying up and moving on the work
« Reply #20 on: December 06, 2019, 01:33:01 PM »
I have just added further BR Locomotives to the transparent images. Some are missing due to missing blend files and the Class 43 I have not done yet. The Class 33 received a retouch, moving looks closer to what I have found from images. Currently I'm working to update the early 20th century locos as far as blend files allow. After or during the holidays, I'll move into recreating missing/lost liveries and incorporating these.

James, I would like to have completed and committed most of this work before the holidays, in order to allow for easier focus on cases that I'm not sure of and avoid us running into conflicting changes. May I ask if/when you will be using the holidays to work on the pakset?

edit:
I have updated many locomotives except for the LBSCR, LMS, and MR. Some cases of missing .blend files mean some images have been left as is, but these are mostly livery tasks. We're approaching 20,000 image files for trains by now.
« Last Edit: Today at 11:41:16 AM by Spenk009 »