News:

Simutrans Tools
Know our tools that can help you to create add-ons, install and customize Simutrans.

Transparent Workflow - Tidying up and moving on the work

Started by Spenk009, November 05, 2019, 09:34:25 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

Spenk009

I've been trying to continue the conversion to transparency, but got stuck on the SR3-Subs. Since then I've been doing work here and there, focusing efforts on fixing bugs in the pakset mentioned in the forum.

The trains images folder contains 17000 images. I'm working from a SATA SSD and James soon from an M.2 SSD, but the loading takes quite some time to complete. I have removed many (hopefully all) unused images from "trains/images", almost 800 images, branch here. Would it make sense to split these into the same categories as the Blends repository has? I know there are problems with applying this (especially DMUs/EMUs, fast freight, etc.), but it would improve working in the repo.

I had a lot of trouble due to a lack of resources and knowledge regarding the LBSCR 48ft, balloon and further carriages. I'll try to find other vehicles to convert instead. I've seen many steam locomotives waiting for their update, so I'll try focusing on that for the moment.


DrSuperGood

Quote from: Spenk009 on November 05, 2019, 09:34:25 PMThe trains images folder contains 17000 images. I'm working from a SATA SSD and James soon from an M.2 SSD, but the loading takes quite some time to complete.
The loading? Access to a file within a folder should be O(log2(n))) so even if the folder has 17,000 files loading each specific file should be trivial, especially with a SSD.

That said I do agree that assets should be separated into folders on a per vehicle or vehicle set basis rather than in a huge folder. One could also include the data files for the vehicles with it if the build process allows.

Spenk009

I know it should be trivial, but it's also a question of updating the folder when files are pasted into it. I've started adding some more transparent vehicles and am slowly incorporating the solution. The Gresley Express vehicles consist of 560 individual images alone. The folder would balloon in due course, so I'm setting up folders as I go along.

I would like to note a completion of the LNER vehicles aside from the coronation sets. Branch here.

Mariculous

Just for curiosity, may I ask how transarency can improve vehicle images?
I expected transparency to be pretty useful for things that are transparent in reality e.g. glass station roofs but I don't see such things at trains. Given the viewing angle one can not look through the windows to the other side of the train.

Spenk009

Mainly smaller details can now be rendered, e.g. pantographs and masts, while not occupying a whole pixel but instead blending in better. Old horse drawn carriages, trees and thin masts really benefit from this.

Have a look at this thread:
https://forum.simutrans.com/index.php/topic,15314.msg157307.html#msg157307

Mariculous

Thanks,
Can't see a difference for the bus wheel example to be honest. I guess that's why I am not an artist.
Maybe one can better see this on moving objects.

DrSuperGood

Quote from: Spenk009 on November 11, 2019, 11:50:10 AMI know it should be trivial, but it's also a question of updating the folder when files are p
Should still be trivial for computers. Modern file systems were designed with this sort of extreme usage in mind and so scale very well.

Of course if the file explorer used to manually view and manipulate the folder scales as well is another question...
Quote from: Freahk on November 11, 2019, 12:00:17 PMJust for curiosity, may I ask how transarency can improve vehicle images?I expected transparency to be pretty useful for things that are transparent in reality e.g. glass station roofs but I don't see such things at trains. Given the viewing angle one can not look through the windows to the other side of the train.
Mostly due to anti-aliasing around the edges of the object. Allows the edges to appear smoother or smaller edge details captured.

Spenk009

#7
Several questions for both the Blends repo and the pak repo:

The LNER Coronation carriages are missing, these were added by the Hood back in 2009. Is there any chance we can get these? I remember reading about more problems with getting old sources of files due to his departure. commit
The LMS Coronation Scots are written as corination, is this a mishap from the Christmas pakwork frenzy? commit
A while back, quite a few source images were added into the root of the pakset. May I move these into a separate sources folder? They are valuable drawings and should probably be preserved. commit



Quote from: DrSuperGood on November 11, 2019, 06:13:43 PMShould still be trivial for computers. Modern file systems were designed with this sort of extreme usage in mind and so scale very well.
They have improved significantly. I reckon that the pak building time is reduced too, as splicing the images and creating a new transparent layer is omitted for a large portion of the pak.

jamespetts

Thank you for your work on this. I should be able to look at this properly once I have my new computer and can sensibly test things.

In the meantime, if you have identified any .png files of the old workflow that are no longer required by the current .dat files, feel free to delete these from your repository (the deletions should be in their own commit, however) so that these deltions occur when I merge with your repository.

However, as Dr. Supergood says, there may be something a little wrong if there just being that many images makes your computer that slow, although possibly this may be caused by a setting to give a preview image of each one; you might change to a detailed view without a thumbnail to check whether this improves the folder response times.

As for missing Blender files, I am afraid that I cannot assist with these; some may have been lost entirely. There may be some cases where it is necessary in due course to re-create the Blender files from scratch if it is desired to render these with the transparent workflow.

As to the 48ft Brighton carraiges, I may well be able to find research material on these if you let me know what information that you need.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Spenk009

#9
Quote from: jamespetts on November 12, 2019, 01:07:29 AM
Thank you for your work on this. I should be able to look at this properly once I have my new computer and can sensibly test things.
In the meantime, if you have identified any .png files of the old workflow that are no longer required by the current .dat files, feel free to delete these from your repository (the deletions should be in their own commit, however) so that these deltions occur when I merge with your repository.
However, as Dr. Supergood says, there may be something a little wrong if there just being that many images makes your computer that slow, although possibly this may be caused by a setting to give a preview image of each one; you might change to a detailed view without a thumbnail to check whether this improves the folder response times.
As to the 48ft Brighton carraiges, I may well be able to find research material on these if you let me know what information that you need.

You're most welcome. No need to hurry anything that is a hobby.

Old .png files are removed in the commit that they are replaced in. If you're referencing removing all unused images in one step/commit, the relevant commit is here

I don't think there is anything wrong with my machine, loading only initially takes around 15 seconds. Any subsequent opening of the folder below 2 seconds. It is just looking forwards into when we end up with many more images in the folder.

The issue is the selection of windows from their respective layer in order to produce the correct images. It will take some time to get to, so instead I hope to focus efforts on easier tasks to reduce the existing backlog.



I've completed the LNWR Locomotives for transparent vehicles. Testing is completed. I may have missed one or two, but these are best reported as bugs

jamespetts

Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Spenk009

Hi everyone, less of a progress update but a tools one.

I've created a script to relocate the images of vehicles (called relocator). It changes the images folder in the dat file and moves the images accordingly. It's simple, but should help make the move to separate folders.



I have also included the script to easily rebuild the pakset and move it to the appropriate folder (called pakupdater). It's not perfect, but problems during the compilation surface recognizably. It takes my machine around 65-70 seconds to complete.

The third script renames the images to the correct type. I will need to add handling for cases where the old images were only one level tall ( eg. .9 rather than 1.9 for the image definition).

Have a gander at these, maybe they can help you in the pak maintainance of your own pak. Feel free to share how long the pak building script takes on your machine. Please keep in mind that these are first advances into dabbling with bash scripts. They are a bit crap and include bad writing, but do the job for now.

jamespetts

Interesting. Do I understand that it is the .dat and .png files that you are moving? Does this script update all the necessary matters both in the makefile and the Python script used for building the pakset (the former for Linux the latter for Windows)?
Edit: Incidentally, I see that you have two transparent vehicles branches on your Github. Are these ready to be pulled yet, or do they need further testing/refinement?
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Spenk009

Quote from: jamespetts on November 24, 2019, 12:24:26 PM
Interesting. Do I understand that it is the .dat and .png files that you are moving? Does this script update all the necessary matters both in the makefile and the Python script used for building the pakset (the former for Linux the latter for Windows)?
I only intend to separate the images, not create further work by editing the compilation (and/or scripts). The dat files only have the image location changed within themselves. The images are moved to the appropriate folder.

Quote from: jamespetts on November 24, 2019, 12:24:26 PM
Incidentally, I see that you have two transparent vehicles branches on your Github. Are these ready to be pulled yet, or do they need further testing/refinement?
No, these are still a work in progress. The main branch for the changes is the transparent-vehicles branch, with these sorting changes applied in there. Once presentable, I will create a pull request and mention it here. The SR-transparent branch was an attempt at unnecessarily splitting work up, I'll remove it in due course.

jamespetts

Thank you for the clarification - that is most helpful.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Spenk009

I am making several commits to stage the work, but will try to limit these a little in order to simplify the process. I try to push working (i.e. compiling) stages to the repository, but this may not always be the case. All BR, LMS and goods vehicles have been moved, around 6000 images so far.

The lms-d1904 and lms-d2007 are parcel vans. They have not been implemented as vehicles yet (I can't find any trace of their usage). Would we want to implement these as brake vans or just empty carriages that are there for visuals/realism?

I envision the creation of a script that checks whether an image is actually used. When this is around, we can check for other cases like this.

jamespetts

The general rule that I follow for Simutrans-Extended is that everything that has an economic function in reality must have an economic function in the game. Thus, there should be no vehicles that are for cosmetic effect only. The parcels vans should be used for mail in Simutrans. However, you may want to check to see whether these have been superseded by something else, as vehicles that are simply not used are often not used for good reason (usually that developments have rendered them redundant, e.g. by providing a more accurate alternaive).
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Spenk009

Quote from: jamespetts on November 26, 2019, 11:50:48 AM
The parcels vans should be used for mail in Simutrans. However, you may want to check to see whether these have been superseded by something else, as vehicles that are simply not used are often not used for good reason (usually that developments have rendered them redundant, e.g. by providing a more accurate alternaive).
Thanks, I wasn't sure whether they were additional mail carriages or carriages to help luggage requirements on longer routes.

I have finished the moving of almost all railway vehicles, found here. The only remaining ones are horses , their stagecoaches and alignment templates. It ended up as one big commit in the very end, because I am no master of managing inclusions and exclusions in a commit. The pakset compiles and the folders are reasonably fast to navigate.

as a side note:
Quote from: DrSuperGood on November 11, 2019, 06:13:43 PM
Should still be trivial for computers. Modern file systems were designed with this sort of extreme usage in mind and so scale very well.
I had to switch to a laptop and was recently looking to expand the storage. While there are two M.2 slots in the machine, they are limited to 2242 dimensions. I ran a speed test on the internal SSD, with the result of 500mb/s. Hoping to break 1GB/s I looked into expansion via M.2, but the shorter cards tend to have lower speeds. I've taken the route of a bay adaptor that will be filled with a hdd for media. You're right, they do scale extremely well. I just think something runs out of cached folder content at 18000 files.

jamespetts

Thank you. Can I confirm whether these are ready for testing/integration yet? Your work in this regard is much appreciated.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Spenk009

#19
These I confirm to be working. Would you like to merge them in yourself or should I create a pull request?

I'll continue by converting further vehicles to the transparent workflow. Just out of curiosity, when do you see yourself working on the pakset if you find time over the holidays next month? I'd need your assistance in the matter of a few vehicles (e.g. SR-EMUs and their varying files & configurations), but would prefer to have completed further work by then and have compiled a small list to give an overview.




I have just added further BR Locomotives to the transparent images. Some are missing due to missing blend files and the Class 43 I have not done yet. The Class 33 received a retouch, moving looks closer to what I have found from images. Currently I'm working to update the early 20th century locos as far as blend files allow. After or during the holidays, I'll move into recreating missing/lost liveries and incorporating these.

James, I would like to have completed and committed most of this work before the holidays, in order to allow for easier focus on cases that I'm not sure of and avoid us running into conflicting changes. May I ask if/when you will be using the holidays to work on the pakset?

edit:
I have updated many locomotives except for the LBSCR, LMS, and MR. Some cases of missing .blend files mean some images have been left as is, but these are mostly livery tasks. We're approaching 20,000 image files for trains by now.

Spenk009

#20
Here are two screenshots of my workflow on Linux. I make heavy use of workspaces (also natively available on Windows 10).

The first workspace is my Blender/Compiling workspace. I keep a terminal open with one tab ready to update the pak, one tab to call the image transcoding scripts and work the pak's git, and one tab with commands ready to clean up the Blends repository and issue git commands. Next, is a file browser with the Blend files in view for easy selection and opening of files.



The second workspace is for pak imaging and individual editing/correcting work. I keep a file browser with three (sometimes four) tabs open, which consist of the base .dat files folder, a search for image files in the respective Blends folder and one tab of the images folder I'm inserting into. There is also a terminal open, usually with some form of " grep -l 'fragment of file' ./* | xargs xed " open.



The third workspace is for Simutrans testing and a Firefox window (or five, with tabs) for research.

The progression is to look up the desired/old images, parse the respective blender files (edit & render views), update the relevant .dat file and replace the single image with transparent images. After sufficient files have been edited, I switch the tab in workspace 1 and compile the pak. If there are errors, I look into the source of these using the terminal (2nd workspace) that automatically opens matching files in a convenient text editor. Once the build compiles, the changes can be checked on in Simutrans (3rd workspace). If everything works out, it is committed to Github.

Spenk009

#21
In the spirit of summarising work for the coming weeks, here's a general list:

Carriages:
Many carriages remain to be converted. My estimate is 500+ Carriages, resulting in up to 4000 more images.
BR: Mk1 to Mk4, GNR, GWR, LBSCR, LNER Coronation Set, LSWR, SECR Utility Vans, and SR
edit: I'm working on converting BR Mk1-Mk4 carriages
Locos:
BR: several, mainly Cl43
GER Clauds
LBSCR C2/C2X have more liveries in blends that are absent/mismatched in dat. Remaining are the 6ft6', c-tender, and D2 Class Locomotives.
LMS 10000 series, garratt, ivatt, stanier
LSWR 6 Wheels
MR-3538 tender <-- this is probably just a little lost
SECR N-Class
SR Lord Nelson, MNß, Q1 and one livery of WC

Railcars:
Around 150 out of approx. 750 have been converted so far. I have not yet taken stock of these.

Wagons:
Are almost complete. I would really like honest feedback to the new colours for materials in the hta and haa bulk wagons (changing them takes really no time at all).
The container wagon is missing a variety. We could implement different ones for different freights or leave it as a single type of container for now.
The parliamentary milk van, which I have not looked for.

All work is committed and pushed to Github. All commits are made after the pakset compiles successfully. James, if you would like a Pull Request I can make one.




I would also like to thank everyone who worked on these models. Almost all of them are very detailed, complex and have required a substantial amount of work, making them a shame to be hidden away in small 2D sprites. The change to transparent workflow has also reduced compile times to less than a minute (on my machine), the marginally improved speeds are much appreciated.

jamespetts

Thank you very much for all of your work on this: this is most helpful. This really is a lot of work and is most impressive and an excellent contribution.

I have now incorporated these changes, which will be available generally from to-morrow's nightly build. I notice that the articulated car transporter wagons especially benefit from transparency, but other graphics are also noticeably better, with the loss of harsh borders and black fringes and greater subtlety generally.

The modern bulk wagons look good, although for even greater improvement, you might update the colours of the HTA bulk wagons to match the colours of the class 66 locomotive, which is intended to have the same livery; the red looks fairly similar, but the yellow on the wagons is brighter. This is a relatively minor point, however. (Ideally, one would also make the logo look a little subtler by using actual "EWS" letters rather than a plain box - one would not be able to read the letters, but one could at least tell that they are letters rather than a rectangle; but this is an even more minor point).

To answer a question that you asked some time ago on this thread: yes, I will be spending the Christmas holidays working on the pakset. I plan to focus on the GWR, filling out the range of carriages and locomotives, and adding a full set of passenger classes to the carriages, as well as a full set of the GWR's various different liveries. I may also do some other miscellaneous things.

In relation to the .blends, may I ask what the address for your Github repository with those on it is? I will need to merge your version of the .blends into my local repository so that I can keep up to date and so that any changes that I make can be applied to your .blends rather than the earlier versions.

Thank you again for your excellent work on this: this really does help to progress the pakset.

Edit: Incidentally - you wrote above that you needed some help with the SR 3-SUB EMUs - may I ask what the issue is? Have those not already been converted to the transparent workflow in any event? I thought that I had finished all of the EMUs when adding the classes, but I may have mis-remembered.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

CK

I've spotted a small mistake: the classes 507 and 313 in Rail Blue seem to have traded their rear cabs for pantographs. To be a bit more technical, it appears that br_313-c-bg_NE.png is the same as br_313-b-bg_NE.png.

Spenk009

#24
Apologies for the slow reply. I've been caught up in going home and all the Christmas madness myself. Thank you very much for the appreciation, I'm happy to be contributing.

Quote from: jamespetts on December 22, 2019, 03:46:24 PM
In relation to the .blends, may I ask what the address for your Github repository with those on it is? I will need to merge your version of the .blends into my local repository so that I can keep up to date and so that any changes that I make can be applied to your .blends rather than the earlier versions.
The Blends repository is here. I commit my work to the master branch as the changes are ready, since these don't effect any change in content or type. I'm a bit slower with these, but will likely manage to push an update early 25th.

Quote from: jamespetts on December 22, 2019, 03:46:24 PM
Edit: Incidentally - you wrote above that you needed some help with the SR 3-SUB EMUs - may I ask what the issue is? Have those not already been converted to the transparent workflow in any event? I thought that I had finished all of the EMUs when adding the classes, but I may have mis-remembered.

They are differently matching Blend models and names for what should be produced. It may be best if you approach these afresh, without my confusion.

Quote from: jamespetts on December 22, 2019, 03:46:24 PM
The modern bulk wagons look good, although for even greater improvement, you might update the colours of the HTA bulk wagons to match the colours of the class 66 locomotive, which is intended to have the same livery; the red looks fairly similar, but the yellow on the wagons is brighter. This is a relatively minor point, however. (Ideally, one would also make the logo look a little subtler by using actual "EWS" letters rather than a plain box - one would not be able to read the letters, but one could at least tell that they are letters rather than a rectangle; but this is an even more minor point).
Noted and fixed. Pull request included the following find is here

Quote from: CK on December 24, 2019, 01:01:13 PM
I've spotted a small mistake: the classes 507 and 313 in Rail Blue seem to have traded their rear cabs for pantographs. To be a bit more technical, it appears that br_313-c-bg_NE.png is the same as br_313-b-bg_NE.png.
Thank you CK for catching that. It was caused by the c vehicle (third railcar) being mislabelled as c-nse in the blend file. There was a livery issue with the c-nse coach too which I've fixed.

jamespetts

Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

Spenk009

Great! Good progress on the GWR Locos too.

I have completed the LBSCR locomotives. Remaining are the 6ft6', c-tender, and D2 Class Locomotives. Pull Request here.

The Blends repository is also updated with a pull request here

jamespetts

Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.

jamespetts

I note that you had got stuck with the SR 3-SUB. Some of them had been saved with the wrong rotation, which I have fixed this evening. I have now re-rendered the 3-SUBs, but I note that the other SR EMUs still have the old graphics; I will leave these to you.

Thank you again for your work on this.
Download Simutrans-Extended.

Want to help with development? See here for things to do for coding, and here for information on how to make graphics/objects.

Follow Simutrans-Extended on Facebook.