[bug] makeobj r3354 - "random" crashes when merging

[VS]

It seems there is some problem when merging files with wildcard, like this: makeobj merge target.pak *.pak The effects vary.

See first few posts for original description of problem:
http://forum.simutrans.com/index.php?topic=5142.0

After compiling makeobj and some experiments, it seems that there is some problem when merging:
(reply #22) http://forum.simutrans.com/index.php?topic=5142.msg50990#msg50990

I was able to get the Python caller to receive SIGSEGV, as it happens to Werner. Additionally, I had one case where makeobj started copying the file recursively to itself or something - the pak file grew at 100 MB/s.

So far I couldn't get anything to happen on Windows.

edit: This could have something to do with timing or environment, since I wasn't able to reproduce this outside the python script.

[wernieman]

Sorry, but did somebody work on this Problem??

[Dwachs]

No / not yet.

If this problem happens only with specific paks, then please upload an archive of these paks somewhere.

Edit: small change in rev 3396 (use "delete []"). Maybe this lead to memory corruption?

If the problem is still there, please let me know. Extracting and merging the GUI.128.pak (from pak128 standard) worked fine for me (and valgrind, too).

[VS]

Werner: I would say it's a bit hard to debug a problem that is vaguely described and probably exists only outside debugger

[wernieman]

But .. how we solf the problem?

At now I could use for pak128 an offizial makeobj, but there will be a new version and then .. whe have the problem (

[VS]

I looked through documentation for the python library and tried to find something to alter the execution... at least trying, you know

After setting this on, I consistently get only the growing file, no signals.

If close_fds is true, all file descriptors except 0, 1 and 2 will be closed before the child process is executed. (Unix only). Or, on Windows, if close_fds is true then no handles will be inherited by the child process. Note that on Windows, you cannot set close_fds to true and also redirect the standard handles by setting stdin, stdout  or stderr.

[wernieman]

So .. there will be a new phyton script?

It would be on the svn too??

[VS]

It does not help... only the problem becomes consistent (for me, anyway).



edit: I can reliably reproduce the hang+infinite file this way:

1) pak128 r974
2) self-compiled makeobj r3397
3) #comment out all FOLDERS in top-level pakmak.tab except for "base" (no need to compile the whole set before)
4) when merging hangars, IT happens

[wernieman]

sch ....

I test to use a spezial makeobj for PAK128...

At now I generate all Pnightly-PAKs with the same makeobj .. then PAK18 use an other makeobj than the other ....

[Dwachs]

@VS: can you upload the offending paks that dont want to be merged?

[VS]

Here goes...

http://anyhub.net/file/premerge.tar.gz

[prissi]

Sounds to me rather that the created pak is found even while enumerating through all other paks. But then the enumeration of paks is done via the shell:

Code Select Expand


if (!STRICMP(argv[0], "merge")) {
argv++, argc--;
try {
const char* dest = argv[0];
argv++, argc--;
root_writer_t::instance()->copy(dest, argc, argv);
}
...
return 0;


This means that there is already a pak with the tarbet name while adding with a wildcard?

(But a check for identical names would not be too hard to implement.)

[VS]

Well... this is certainly one possibility. But on Windows it does not fail for some reason!



Here's the patch... it seems it fixes that.

Code Select Expand

Index: besch/writer/root_writer.cc
===================================================================
--- besch/writer/root_writer.cc (revision 3397)
+++ besch/writer/root_writer.cc (working copy)
@@ -301,7 +301,11 @@
} else {
find.search(argv[i], "pak");
for (searchfolder_t::const_iterator i = find.begin(), end = find.end(); i != end; ++i) {
- any |= do_copy(outfp, root, *i);
+ if (strcmp(*i, name) != 0) {
+ any |= do_copy(outfp, root, *i);
+ } else {
+ printf("WARNING: skipping reading from output file\n");
+ }
}
}


[prissi]

Windows lock open file for writing. Maybe it just fails to be opened?

[VS]

You're right. But that still leaves the question "how could this work before"

Werner: does the patch fix this problem for you?

[wernieman]

I must test it tomorow ... at now I must sleep ;o)

[wernieman]

It work!

Now the Output is (at example):

Code Select Expand

Executing command: lift symbol.BigLogo.pak
Merging: makeobj QUIET merge GUI.128.pak *.pak
writing file GUI.128.pak
  copying file cursor.Mouse.pak
  copying file misc.Construction.pak
  copying file cursor.Marked.pak
  copying file cursor.GeneralTools.pak
  copying file symbol.Builder.pak
  copying file cursor.Builder.pak
WARNING: skipping reading from output file
My folders:  **none**
At this Part the makeobj without the Patch make an error ....
(But I don´t test the PAK, only the generation)

Edit:
When I count the quantity of the warnung I get 31, so it is Important for PAK128

Code Select Expand

simutrans@hedwig ~ $ grep "WARNING: skipping reading from output file" /home/simutrans/PAK/PAK128/info.txt | wc -l
31


[wernieman]

Dit it go into trunk??

[VS]

Warning will be generated for all folders since it's always merging against wildcard.

[wernieman]

Testet and Solved in 3405

Thanks!!