www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - OPTLINK needs to die.

reply Jarrett Billingsley <jarrett.billingsley gmail.com> writes:
That's it -- I'm finished.  I'm tired of constantly rearranging code
to appease OPTLINK and its outdated object format.

This is, for me, the number one blocker of most of my more complex
code on Windows.  More than any DMD bug.  Tom S can vouch for it too,
and Eldar sounds like he s about to give up on D after dealing with
OPTLINK's crashes.

OpenWatcom is not an option.  It seems to be buggy with regards to
weak symbols.  Furthermore its documentation is sparse at best and its
option syntax is completely different from OPTLINK's.

I *do not care* how fast OPTLINK is.  I *do not care* how many hours
of hand-coded assembly went into it.  I want a linker that *works*,
and preferably one that doesn't use an object format that *no other
modern compilers use*.

Please do not underestimate the importance of this, Walter.  OMF and
OPTLINK will continue to be a thorn in DMD's side unless you do
something about it now.
Feb 16 2009
next sibling parent reply Tom S <h3r3tic remove.mat.uni.torun.pl> writes:
Jarrett Billingsley wrote:
 That's it -- I'm finished.  I'm tired of constantly rearranging code
 to appease OPTLINK and its outdated object format.
 
 This is, for me, the number one blocker of most of my more complex
 code on Windows.  More than any DMD bug.  Tom S can vouch for it too,
 and Eldar sounds like he s about to give up on D after dealing with
 OPTLINK's crashes.

Hehehe, agreed :P Out of desperation I've even tried not using OPTLINK for one of my programs and loading it at runtime from .obj files using DDL instead :P It took 3 seconds to link since DDL needs many optimizations, but at least it didn't crash like OPTLINK would :P
 OpenWatcom is not an option.  It seems to be buggy with regards to
 weak symbols.  Furthermore its documentation is sparse at best and its
 option syntax is completely different from OPTLINK's.

Indeed, I've tried Open Watcom's wlink two days ago and it borks on some weak symbols from snn.lib, claiming that they are unresolved. The old version on its author's site doesn't like long symbol names (since DMD and OPTLINK use an extension for the OMF format to allow longer names). As for the documentation, you probably haven't found it :P There's a huge PDF file with examples :)
 I *do not care* how fast OPTLINK is.  I *do not care* how many hours
 of hand-coded assembly went into it.  I want a linker that *works*,
 and preferably one that doesn't use an object format that *no other
 modern compilers use*.

Agreed :P
 Please do not underestimate the importance of this, Walter.  OMF and
 OPTLINK will continue to be a thorn in DMD's side unless you do
 something about it now.

Perhaps our best bet would be fixing the bug in Open Watcom's linker? Or rolling something on our own using the code from DDL? I wouldn't mind DMD emitting COFF obj files, that the MS linker could process, either :P -- Tomasz Stachowiak http://h3.team0xf.com/ h3/h3r3tic on #D freenode
Feb 16 2009
parent BCS <none anon.com> writes:
Hello Tom,

 Perhaps our best bet would be fixing the bug in Open Watcom's linker?
 Or rolling something on our own using the code from DDL? I wouldn't
 mind DMD emitting COFF obj files, that the MS linker could process,
 either :P
 

Well Walter is up to speed on object file generators right now. I second the motion for DMD's next target to be a better object file format on windows. Also, if I understand the situation correctly, it wouldn't be hard to make a DMC release to match.
Feb 16 2009
prev sibling next sibling parent reply "Tim M" <a b.com> writes:
On Tue, 17 Feb 2009 13:27:14 +1300, Jarrett Billingsley  
<jarrett.billingsley gmail.com> wrote:

 That's it -- I'm finished.  I'm tired of constantly rearranging code
 to appease OPTLINK and its outdated object format.

 This is, for me, the number one blocker of most of my more complex
 code on Windows.  More than any DMD bug.  Tom S can vouch for it too,
 and Eldar sounds like he s about to give up on D after dealing with
 OPTLINK's crashes.

 OpenWatcom is not an option.  It seems to be buggy with regards to
 weak symbols.  Furthermore its documentation is sparse at best and its
 option syntax is completely different from OPTLINK's.

 I *do not care* how fast OPTLINK is.  I *do not care* how many hours
 of hand-coded assembly went into it.  I want a linker that *works*,
 and preferably one that doesn't use an object format that *no other
 modern compilers use*.

 Please do not underestimate the importance of this, Walter.  OMF and
 OPTLINK will continue to be a thorn in DMD's side unless you do
 something about it now.

Just checking you are aware of objconv which can be found here http://www.agner.org/optimize/.
Feb 16 2009
parent Tom S <h3r3tic remove.mat.uni.torun.pl> writes:
Tim M wrote:
 Just checking you are aware of objconv which can be found here 
 http://www.agner.org/optimize/.

Have you successfully used it to convert all OMF obj files of some application to another format and link them? It gives me a lot of: "Error 2316: Incompatible relocation method: 16+32 bit far". -- Tomasz Stachowiak http://h3.team0xf.com/ h3/h3r3tic on #D freenode
Feb 16 2009
prev sibling next sibling parent Jarrett Billingsley <jarrett.billingsley gmail.com> writes:
On Mon, Feb 16, 2009 at 8:19 PM, Tim M <a b.com> wrote:
 On Tue, 17 Feb 2009 13:27:14 +1300, Jarrett Billingsley
 <jarrett.billingsley gmail.com> wrote:

 That's it -- I'm finished.  I'm tired of constantly rearranging code
 to appease OPTLINK and its outdated object format.

 This is, for me, the number one blocker of most of my more complex
 code on Windows.  More than any DMD bug.  Tom S can vouch for it too,
 and Eldar sounds like he s about to give up on D after dealing with
 OPTLINK's crashes.

 OpenWatcom is not an option.  It seems to be buggy with regards to
 weak symbols.  Furthermore its documentation is sparse at best and its
 option syntax is completely different from OPTLINK's.

 I *do not care* how fast OPTLINK is.  I *do not care* how many hours
 of hand-coded assembly went into it.  I want a linker that *works*,
 and preferably one that doesn't use an object format that *no other
 modern compilers use*.

 Please do not underestimate the importance of this, Walter.  OMF and
 OPTLINK will continue to be a thorn in DMD's side unless you do
 something about it now.

Just checking you are aware of objconv which can be found here http://www.agner.org/optimize/.

I'd rather not have to add in an _object conversion_ step to my build process. And I wouldn't want to have to force users of my libraries to do so either.
Feb 16 2009
prev sibling next sibling parent "Tim M" <a b.com> writes:
On Tue, 17 Feb 2009 14:35:55 +1300, Tom S  
<h3r3tic remove.mat.uni.torun.pl> wrote:

 Tim M wrote:
 Just checking you are aware of objconv which can be found here  
 http://www.agner.org/optimize/.

Have you successfully used it to convert all OMF obj files of some application to another format and link them? It gives me a lot of: "Error 2316: Incompatible relocation method: 16+32 bit far".

No I convert MS COFF files to OMF and I totally agree with the multiple formats problems. I don't think making another converter or improving exisitng converters is the sollution though. Native COFF support is the way.
Feb 16 2009
prev sibling next sibling parent Jarrett Billingsley <jarrett.billingsley gmail.com> writes:
On Mon, Feb 16, 2009 at 8:48 PM, Tim M <a b.com> wrote:
 No I convert MS COFF files to OMF and I totally agree with the multiple
 formats problems. I don't think making another converter or improving
 exisitng converters is the sollution though. Native COFF support is the way.

Oh, that might be useful for linking to COFF libraries, but it certainly doesn't help with the other major problems with OPTLINK and the OMF format.
Feb 16 2009
prev sibling next sibling parent reply Walter Bright <newshound1 digitalmars.com> writes:
One thing you can try is:

1. download imagecfg.exe from http://www.robpol86.com/pages/imagecfg.php

2. run the command:

	imagecfg -a 0x1 link.exe

This will set optlink to use only one core. I've suspected for a while 
that the multithreading locks in it don't work right for multicore. I 
have a single core machine and cannot test multicore behavior.
Feb 16 2009
parent reply Walter Bright <newshound1 digitalmars.com> writes:
Jarrett Billingsley wrote:
 On Mon, Feb 16, 2009 at 9:54 PM, Walter Bright
 <newshound1 digitalmars.com> wrote:
 One thing you can try is:

 1. download imagecfg.exe from http://www.robpol86.com/pages/imagecfg.php

 2. run the command:

        imagecfg -a 0x1 link.exe

 This will set optlink to use only one core. I've suspected for a while that
 the multithreading locks in it don't work right for multicore. I have a
 single core machine and cannot test multicore behavior.

Tom S does all of his development on a single-core machine, and I have this problem on my single-core laptop as well. It is not multithreading.

But I downloaded the offending obj file from bugzilla, and it links and runs without error: http://d.puremagic.com/issues/show_bug.cgi?id=2436
Feb 16 2009
parent reply Walter Bright <newshound1 digitalmars.com> writes:
Jarrett Billingsley wrote:
 http://d.puremagic.com/issues/show_bug.cgi?id=424

I just added this to the bug report: ====================== The compiler already has a switch to generate multiple obj files from one source file: -multiobj. I use it for debugging. But it would be a royal pain to use if there's a lot. A better way is to compile to a library with the -lib switch. That splits the module into a zillion obj files, and stuffs them all into the library. Then, link with the library (the main() program will still have to be separate). ======================= Can you give it a try?
Feb 16 2009
next sibling parent reply Tom S <h3r3tic remove.mat.uni.torun.pl> writes:
Walter Bright wrote:
 Jarrett Billingsley wrote:
 http://d.puremagic.com/issues/show_bug.cgi?id=424

I just added this to the bug report: ====================== The compiler already has a switch to generate multiple obj files from one source file: -multiobj. I use it for debugging. But it would be a royal pain to use if there's a lot. A better way is to compile to a library with the -lib switch. That splits the module into a zillion obj files, and stuffs them all into the library. Then, link with the library (the main() program will still have to be separate). ======================= Can you give it a try?

I've tried it before and unfortunately there's a problem with this approach: static ctors from modules linked from a .lib don't get executed:
cat Main.d

}
cat Mod.d

static this() { printf("Running a static ctor for Mod"\n); } // compiling Mod into an .obj
dmd -c Main.d
dmd -c Mod.d
dmd Main.obj Mod.obj && Main.exe

// compiling Mod into a .lib
dmd -lib Mod.d
dmd Main.obj Mod.lib && Main.exe

-- Tomasz Stachowiak http://h3.team0xf.com/ h3/h3r3tic on #D freenode
Feb 16 2009
parent reply Walter Bright <newshound1 digitalmars.com> writes:
Tom S wrote:
 I've tried it before and unfortunately there's a problem with this 
 approach: static ctors from modules linked from a .lib don't get executed:

Not everything is linked in from a library, just referenced things (that's the whole point of a library). There is nothing referenced in mod.d. But rewriting as: --------------- import mod; void main() { foo(); } --------------- import std.c.stdio; void foo() { } static this() { printf("Running a static ctor for Mod"\n); } --------------- Compiling and running: C:>dmd -lib mod C:>dmd test5 mod.lib C:>test5 Running a static ctor for Mod Or, compile the module with just the static constructor in it separately and add it explicitly to the link list.
Feb 16 2009
parent reply Tom S <h3r3tic remove.mat.uni.torun.pl> writes:
Walter Bright wrote:
 Tom S wrote:
 I've tried it before and unfortunately there's a problem with this 
 approach: static ctors from modules linked from a .lib don't get 
 executed:

Not everything is linked in from a library, just referenced things (that's the whole point of a library). There is nothing referenced in mod.d. But rewriting as: --------------- import mod; void main() { foo(); } --------------- import std.c.stdio; void foo() { } static this() { printf("Running a static ctor for Mod"\n); } --------------- Compiling and running: C:>dmd -lib mod C:>dmd test5 mod.lib C:>test5 Running a static ctor for Mod Or, compile the module with just the static constructor in it separately and add it explicitly to the link list.

This one still has problems. ---- Main.d: import Mod; void main() { foo(); } ---- Mod.d: import Mod2; extern (C) int printf(char*, ...); void foo() { foo2(); } static this() { printf("Running a static ctor for Mod"\n); } ---- Mod2.d: extern (C) int printf(char*, ...); void foo2() { } static this() { printf("Running a static ctor for Mod2"\n); } ---- Compiling these modules as: dmd -c Main.d dmd -lib Mod.d Mod2.d dmd Main.obj Mod.lib ... and running Main.exe yields: Running a static ctor for Mod So it seems that Main.d would have to reference symbols from each module in the program. Ouch. Additionally, using such an approach breaks any possibility to use incremental compilation. So there's also the option of having a .lib per .d file and linking that all together. So I've just tried it: dmd -c Main.d dmd -lib Mod.d dmd -lib Mod2.d dmd Main.obj Mod.lib Mod2.d ... and OPTLINK crashed with the familiar MsgBox containing register values. While we're at incremental compilation, this bug: http://d.puremagic.com/issues/show_bug.cgi?id=1629 often kills it for me and it's been around for a few whiles already. -- Tomasz Stachowiak http://h3.team0xf.com/ h3/h3r3tic on #D freenode
Feb 17 2009
next sibling parent reply Walter Bright <newshound1 digitalmars.com> writes:
Tom S wrote:
 Additionally, using such an approach breaks any possibility to use 
 incremental compilation. So there's also the option of having a .lib per 
 .d file and linking that all together. So I've just tried it:
 dmd -c Main.d
 dmd -lib Mod.d
 dmd -lib Mod2.d
 dmd Main.obj Mod.lib Mod2.d
 ... and OPTLINK crashed with the familiar MsgBox containing register 
 values.

It does not crash when I try it. Are you sure you're not running a multicore computer?
Feb 17 2009
next sibling parent reply Walter Bright <newshound1 digitalmars.com> writes:
Danny Wilson wrote:
 
 Op Tue, 17 Feb 2009 11:06:43 +0100 schreef Walter Bright 
 <newshound1 digitalmars.com>:
 
 It does not crash when I try it. Are you sure you're not running a 
 multicore computer?

It's pretty hard to _NOT_ get a multicore computer these days. Please kill OPTLINK.

What I'm asking is for someone with a multicore computer who is experiencing these crashes to try the patch I outlined to restrict it to one core.
Feb 17 2009
parent reply Walter Bright <newshound1 digitalmars.com> writes:
Denis Koroskin wrote:
 On Tue, 17 Feb 2009 14:24:10 +0300, Walter Bright 
 <newshound1 digitalmars.com> wrote:
 
 Danny Wilson wrote:
  Op Tue, 17 Feb 2009 11:06:43 +0100 schreef Walter Bright 
 <newshound1 digitalmars.com>:

 It does not crash when I try it. Are you sure you're not running a 
 multicore computer?

Please kill OPTLINK.

What I'm asking is for someone with a multicore computer who is experiencing these crashes to try the patch I outlined to restrict it to one core.

It is responded that three people (Tom S, Jarrett, Max Samukha) are running single-core machines and still experiencing constant crashes.

It doesn't crash on my machine on *any* of the examples that produce a crash on your machines. Can you please just try the patch anyway? It may also be related to the hyperthreading on a single core machine.
Feb 17 2009
next sibling parent reply HOSOKAWA Kenchi <hskwk inter7.jp> writes:
It's not sure but some processors which have SMT act as pseudo-multicore
manner, like Pentium 4 and Atom.
Unfortunately I don't have them.


Walter Bright Wrote:
 Denis Koroskin wrote:
 On Tue, 17 Feb 2009 14:24:10 +0300, Walter Bright 
 <newshound1 digitalmars.com> wrote:
 
 Danny Wilson wrote:
  Op Tue, 17 Feb 2009 11:06:43 +0100 schreef Walter Bright 
 <newshound1 digitalmars.com>:

 It does not crash when I try it. Are you sure you're not running a 
 multicore computer?

Please kill OPTLINK.

What I'm asking is for someone with a multicore computer who is experiencing these crashes to try the patch I outlined to restrict it to one core.

It is responded that three people (Tom S, Jarrett, Max Samukha) are running single-core machines and still experiencing constant crashes.

It doesn't crash on my machine on *any* of the examples that produce a crash on your machines. Can you please just try the patch anyway? It may also be related to the hyperthreading on a single core machine.

Feb 17 2009
parent HOSOKAWA Kenchi <hskwk inter7.jp> writes:
I'm sorry that my eyes leaped over the word "hyperthreading".

I just tested OPTLINK bug I reported on the Issue Tracking System,
http://d.puremagic.com/issues/show_bug.cgi?id=2378
with fixed link.exe by imagecfg.

The issue is not solved.


HOSOKAWA Kenchi Wrote:

 It's not sure but some processors which have SMT act as pseudo-multicore
manner, like Pentium 4 and Atom.
 Unfortunately I don't have them.
 
 
 Walter Bright Wrote:
 Denis Koroskin wrote:
 On Tue, 17 Feb 2009 14:24:10 +0300, Walter Bright 
 <newshound1 digitalmars.com> wrote:
 
 Danny Wilson wrote:
  Op Tue, 17 Feb 2009 11:06:43 +0100 schreef Walter Bright 
 <newshound1 digitalmars.com>:

 It does not crash when I try it. Are you sure you're not running a 
 multicore computer?

Please kill OPTLINK.

What I'm asking is for someone with a multicore computer who is experiencing these crashes to try the patch I outlined to restrict it to one core.

It is responded that three people (Tom S, Jarrett, Max Samukha) are running single-core machines and still experiencing constant crashes.

It doesn't crash on my machine on *any* of the examples that produce a crash on your machines. Can you please just try the patch anyway? It may also be related to the hyperthreading on a single core machine.


Feb 17 2009
prev sibling parent reply Max Samukha <samukha voliacable.com.removethis> writes:
On Tue, 17 Feb 2009 04:04:18 -0800, Walter Bright
<newshound1 digitalmars.com> wrote:

Denis Koroskin wrote:
 On Tue, 17 Feb 2009 14:24:10 +0300, Walter Bright 
 <newshound1 digitalmars.com> wrote:
 
 Danny Wilson wrote:
  Op Tue, 17 Feb 2009 11:06:43 +0100 schreef Walter Bright 
 <newshound1 digitalmars.com>:

 It does not crash when I try it. Are you sure you're not running a 
 multicore computer?

Please kill OPTLINK.

What I'm asking is for someone with a multicore computer who is experiencing these crashes to try the patch I outlined to restrict it to one core.

It is responded that three people (Tom S, Jarrett, Max Samukha) are running single-core machines and still experiencing constant crashes.

It doesn't crash on my machine on *any* of the examples that produce a crash on your machines. Can you please just try the patch anyway? It may also be related to the hyperthreading on a single core machine.

Have you tried to link my example with /co as I asked you in my last post here http://d.puremagic.com/issues/show_bug.cgi?id=2436#c4 ? It fails on AMD Athlon 1800, which is rather old. Changing the afinity mask doesn't help
Feb 17 2009
next sibling parent Max Samukha <samukha voliacable.com.removethis> writes:
On Tue, 17 Feb 2009 14:54:09 +0200, Max Samukha
<samukha voliacable.com.removethis> wrote:

On Tue, 17 Feb 2009 04:04:18 -0800, Walter Bright
<newshound1 digitalmars.com> wrote:

Denis Koroskin wrote:
 On Tue, 17 Feb 2009 14:24:10 +0300, Walter Bright 
 <newshound1 digitalmars.com> wrote:
 
 Danny Wilson wrote:
  Op Tue, 17 Feb 2009 11:06:43 +0100 schreef Walter Bright 
 <newshound1 digitalmars.com>:

 It does not crash when I try it. Are you sure you're not running a 
 multicore computer?

Please kill OPTLINK.

What I'm asking is for someone with a multicore computer who is experiencing these crashes to try the patch I outlined to restrict it to one core.

It is responded that three people (Tom S, Jarrett, Max Samukha) are running single-core machines and still experiencing constant crashes.

It doesn't crash on my machine on *any* of the examples that produce a crash on your machines. Can you please just try the patch anyway? It may also be related to the hyperthreading on a single core machine.

Have you tried to link my example with /co as I asked you in my last post here http://d.puremagic.com/issues/show_bug.cgi?id=2436#c4 ? It fails on AMD Athlon 1800, which is rather old. Changing the afinity mask doesn't help

"affinity" it should have been. Please don't ignore me because of my grammer :)
Feb 17 2009
prev sibling parent Walter Bright <newshound1 digitalmars.com> writes:
Max Samukha wrote:
 Have you tried to link my example with /co as I asked you in my last
 post here http://d.puremagic.com/issues/show_bug.cgi?id=2436#c4 ?

No, I hadn't tried it with /co, and yes, I can reproduce the failure with /co. Thanks!
Feb 17 2009
prev sibling parent "Nick Sabalausky" <a a.a> writes:
"Danny Wilson" <bluezenix gmail.com> wrote in message 
news:op.uphrgb2pbmuhq1 danny-wilsons-ep45-ds4.local...
 Op Tue, 17 Feb 2009 11:06:43 +0100 schreef Walter Bright 
 <newshound1 digitalmars.com>:

 It does not crash when I try it. Are you sure you're not running a 
 multicore computer?

It's pretty hard to _NOT_ get a multicore computer these days. Please kill OPTLINK.

Minor pet peeve: Just because something stops being offered in stores does *not* mean it isn't in heavy use.
Feb 17 2009
prev sibling next sibling parent reply Walter Bright <newshound1 digitalmars.com> writes:
Tom S wrote:
 So it seems that Main.d would have to reference symbols from each module 
 in the program. Ouch.

I found that problem, and fixed it.
 Additionally, using such an approach breaks any possibility to use 
 incremental compilation.

It depends. If instead of a.obj, b.obj, and c.obj, think a.lib, b.lib, c.lib, you can incrementally compile.
 So there's also the option of having a .lib per 
 .d file and linking that all together. So I've just tried it:
 dmd -c Main.d
 dmd -lib Mod.d
 dmd -lib Mod2.d
 dmd Main.obj Mod.lib Mod2.d
 ... and OPTLINK crashed with the familiar MsgBox containing register 
 values.

Can't repro it. I'm pretty sure it has something to do with how threading works on newer processors (mine is old). Vendor string: GenuineIntel Processor string: Intel(R) Pentium(R) 4 CPU 1.60GHz Signature: Family=15 Model=1 Stepping=2 Features: MMX FXSR SSE SSE2 HTT Multithreading: 1 threads / 1 cores What's your CPU?
 While we're at incremental compilation, this bug: 
 http://d.puremagic.com/issues/show_bug.cgi?id=1629 often kills it for me 
 and it's been around for a few whiles already.

Ok.
Feb 17 2009
next sibling parent reply Max Samukha <samukha voliacable.com.removethis> writes:
On Tue, 17 Feb 2009 04:16:35 -0800, Walter Bright
<newshound1 digitalmars.com> wrote:

Tom S wrote:
 So it seems that Main.d would have to reference symbols from each module 
 in the program. Ouch.

I found that problem, and fixed it.
 Additionally, using such an approach breaks any possibility to use 
 incremental compilation.

It depends. If instead of a.obj, b.obj, and c.obj, think a.lib, b.lib, c.lib, you can incrementally compile.
 So there's also the option of having a .lib per 
 .d file and linking that all together. So I've just tried it:
 dmd -c Main.d
 dmd -lib Mod.d
 dmd -lib Mod2.d
 dmd Main.obj Mod.lib Mod2.d
 ... and OPTLINK crashed with the familiar MsgBox containing register 
 values.

Can't repro it. I'm pretty sure it has something to do with how threading works on newer processors (mine is old).

Probably you typed literally. The last one should be 'dmd Main.obj Mod.lib Mod2.lib'
Vendor string:    GenuineIntel
Processor string: Intel(R) Pentium(R) 4 CPU 1.60GHz
Signature:        Family=15 Model=1 Stepping=2
Features:         MMX FXSR SSE SSE2 HTT
Multithreading:   1 threads / 1 cores

What's your CPU?

 While we're at incremental compilation, this bug: 
 http://d.puremagic.com/issues/show_bug.cgi?id=1629 often kills it for me 
 and it's been around for a few whiles already.

Ok.

Checked on an old Celeron and Athlon. Both fail
Feb 17 2009
next sibling parent Walter Bright <newshound1 digitalmars.com> writes:
Max Samukha wrote:
 Probably you typed literally.

I cut & pasted to be sure!
 The last one should be 'dmd Main.obj
 Mod.lib Mod2.lib'

Thank you. That does repro it.
Feb 17 2009
prev sibling parent Tom S <h3r3tic remove.mat.uni.torun.pl> writes:
Max Samukha wrote:
 On Tue, 17 Feb 2009 04:16:35 -0800, Walter Bright
 <newshound1 digitalmars.com> wrote:
 
 Tom S wrote:
 So there's also the option of having a .lib per 
 .d file and linking that all together. So I've just tried it:
 dmd -c Main.d
 dmd -lib Mod.d
 dmd -lib Mod2.d
 dmd Main.obj Mod.lib Mod2.d
 ... and OPTLINK crashed with the familiar MsgBox containing register 
 values.

threading works on newer processors (mine is old).

Probably you typed literally. The last one should be 'dmd Main.obj Mod.lib Mod2.lib'

Argh! Sorry, I knew I'd make a typo there since I didn't copy-paste the console commands :( Yea, it should be 'dmd Main.obj Mod.lib Mod2.lib' -- Tomasz Stachowiak http://h3.team0xf.com/ h3/h3r3tic on #D freenode
Feb 17 2009
prev sibling parent reply Tom S <h3r3tic remove.mat.uni.torun.pl> writes:
Walter Bright wrote:
 Tom S wrote:
 So it seems that Main.d would have to reference symbols from each 
 module in the program. Ouch.

I found that problem, and fixed it.

Cool, thanks :)
 Additionally, using such an approach breaks any possibility to use 
 incremental compilation.

It depends. If instead of a.obj, b.obj, and c.obj, think a.lib, b.lib, c.lib, you can incrementally compile.

Yea, that's why I tried it next ;)
 So there's also the option of having a .lib per .d file and linking 
 that all together. So I've just tried it:
 dmd -c Main.d
 dmd -lib Mod.d
 dmd -lib Mod2.d
 dmd Main.obj Mod.lib Mod2.d
 ... and OPTLINK crashed with the familiar MsgBox containing register 
 values.

Can't repro it. I'm pretty sure it has something to do with how threading works on newer processors (mine is old).

I'll try DMD 1.040 and different OPTLINK versions then. I'm currently using: Digital Mars D Compiler v1.039 OPTLINK (R) for Win32 Release 8.00.1 Digital Mars Librarian Version 8.02n ( I don't think it's used in this case :P )
 Vendor string:    GenuineIntel
 Processor string: Intel(R) Pentium(R) 4 CPU 1.60GHz
 Signature:        Family=15 Model=1 Stepping=2
 Features:         MMX FXSR SSE SSE2 HTT
 Multithreading:   1 threads / 1 cores
 
 What's your CPU?

Vendor string: GenuineIntel Processor string: Intel(R) Pentium(R) M processor 1.70GHz Signature: Family=6 Model=13 Stepping=6 Features: MMX FXSR SSE SSE2 Multithreading: 1 threads / 1 cores -- Tomasz Stachowiak http://h3.team0xf.com/ h3/h3r3tic on #D freenode
Feb 17 2009
parent reply Walter Bright <newshound1 digitalmars.com> writes:
The problem appears to be that optlink cannot deal correctly with weak 
references. I've modified the compiler to not emit weak references, and 
the problem goes away. It's not a perfect fix, but it works.
Feb 19 2009
parent reply Tom S <h3r3tic remove.mat.uni.torun.pl> writes:
Walter Bright wrote:
 The problem appears to be that optlink cannot deal correctly with weak 
 references. I've modified the compiler to not emit weak references, and 
 the problem goes away. It's not a perfect fix, but it works.

With the modified compiler, does the following link? If so, is the output correct? ---- module a; import c; alias C!(int) aC; import b; extern(C) int printf(char*, ...); void main() { printf("&aC.x == %x"\n, &aC.x); printf("&bC.x == %x"\n, &bC.x); assert (&aC.x is &bC.x); } ---- module b; import c; alias C!(int) bC; ---- module c; template C(T) { int x; } ---- dmd -c a.d dmd -c b.d dmd -c c.d dmd a.obj b.obj c.obj a.exe ---- I'm getting: &aC.x == 419c30 &bC.x == 419c30 I'm afraid that it might refuse to link and then even if OPTLINK allowed multiple strong symbols of the same name, I'd expect the assertion to fail. Aren't weak symbols essential to properly handling templates? Of course compiling the modules together causes only one symbol to be emitted for the template instantiations. Yet this is not an option for a larger project, since compilation times and memory usage can get rather high. -- Tomasz Stachowiak http://h3.team0xf.com/ h3/h3r3tic on #D freenode
Feb 19 2009
parent reply Walter Bright <newshound1 digitalmars.com> writes:
Tom S wrote:
 Walter Bright wrote:
 The problem appears to be that optlink cannot deal correctly with weak 
 references. I've modified the compiler to not emit weak references, 
 and the problem goes away. It's not a perfect fix, but it works.

With the modified compiler, does the following link?

Yes.
 If so, is the output correct?

 I'm getting:
 &aC.x == 419c30
 &bC.x == 419c30

Yes, that is correct (there should be only one instance in the executable).
 I'm afraid that it might refuse to link and then even if OPTLINK allowed 
 multiple strong symbols of the same name, I'd expect the assertion to 
 fail. Aren't weak symbols essential to properly handling templates?

No. Weak references are: "if this symbol does not get resolved, then default initialize it to this other symbol". In other words, weak references do not cause things to be pulled in from the library. Multiple symbols with the same name and same contents, across different object files, are called COMDATs and the linker picks one and discards the duplicates. Where the compiler was emitting weak references was in the list of which modules are imported by the object file, so that if nobody actually referred to the import's contents beyond the import statement, the import would not get linked in.
 Of course compiling the modules together causes only one symbol to be 
 emitted for the template instantiations. Yet this is not an option for a 
 larger project, since compilation times and memory usage can get rather 
 high.

I know. This is why COMDATs were invented (for C++ templates). C++ drove a number of improvements in obj files.
Feb 19 2009
parent reply Tom S <h3r3tic remove.mat.uni.torun.pl> writes:
Walter Bright wrote:
 Tom S wrote:
 Walter Bright wrote:
 The problem appears to be that optlink cannot deal correctly with 
 weak references. I've modified the compiler to not emit weak 
 references, and the problem goes away. It's not a perfect fix, but it 
 works.

With the modified compiler, does the following link?

Yes.
 If so, is the output correct?

 I'm getting:
 &aC.x == 419c30
 &bC.x == 419c30

Yes, that is correct (there should be only one instance in the executable).
 I'm afraid that it might refuse to link and then even if OPTLINK 
 allowed multiple strong symbols of the same name, I'd expect the 
 assertion to fail. Aren't weak symbols essential to properly handling 
 templates?

No. Weak references are: "if this symbol does not get resolved, then default initialize it to this other symbol". In other words, weak references do not cause things to be pulled in from the library. Multiple symbols with the same name and same contents, across different object files, are called COMDATs and the linker picks one and discards the duplicates. Where the compiler was emitting weak references was in the list of which modules are imported by the object file, so that if nobody actually referred to the import's contents beyond the import statement, the import would not get linked in.
 Of course compiling the modules together causes only one symbol to be 
 emitted for the template instantiations. Yet this is not an option for 
 a larger project, since compilation times and memory usage can get 
 rather high.

I know. This is why COMDATs were invented (for C++ templates). C++ drove a number of improvements in obj files.

I'm glad to be proven wrong :) Thanks for the info! I'm sure it will come in handy for the dynamic linking I'm doing with DDL. I got the silly idea that you'd disable COMDATs... Another case of "Too much blood in my caffeine". -- Tomasz Stachowiak http://h3.team0xf.com/ h3/h3r3tic on #D freenode
Feb 19 2009
parent Walter Bright <newshound1 digitalmars.com> writes:
Tom S wrote:
 I'm glad to be proven wrong :) Thanks for the info! I'm sure it will 
 come in handy for the dynamic linking I'm doing with DDL. I got the 
 silly idea that you'd disable COMDATs... Another case of "Too much blood 
 in my caffeine".

dmd would be quite dead if I had to disable COMDATs <g>.
Feb 19 2009
prev sibling parent Walter Bright <newshound1 digitalmars.com> writes:
Jarrett Billingsley wrote:
 Using imagecfg changes nothing.

Thanks for giving it a try. It saves me a lot of wasted effort looking under the wrong rocks :-)
Feb 17 2009
prev sibling parent Walter Bright <newshound1 digitalmars.com> writes:
Jarrett Billingsley wrote:
 On Mon, Feb 16, 2009 at 11:32 PM, Walter Bright
 <newshound1 digitalmars.com> wrote:
 Jarrett Billingsley wrote:
 http://d.puremagic.com/issues/show_bug.cgi?id=424

I just added this to the bug report: ====================== The compiler already has a switch to generate multiple obj files from one source file: -multiobj. I use it for debugging. But it would be a royal pain to use if there's a lot.

Why isn't it documented? And you're damn right it would be a pain. Especially when I'm compiling three external libraries and a large library of my own.

Because it's pretty useless.
 Can you give it a try?

library and I have no idea how well bud/rebuild will cope with a zillion object files.

The point of building a library directly is that you never see the zillion object files. dmd generates the library directly.
 In any case the -lib switch is just sidestepping the problems with the
 linker and object format by pushing some of the linker's
 responsibilities into the compiler.  DMD (and the D language itself,
 if you want to count the 16MB limit on static data, which has no place
 in a portable language specification!) already makes far too many
 concessions to deal with the ridiculous constraints of the linker.
 Please, stop designing around the problem and fix it.

The -lib option has nothing to do with linker problems. It exists and is highly useful on linux and osx. It enables building libraries in one operation without having to dink around with managing the object files.
Feb 16 2009
prev sibling next sibling parent Jarrett Billingsley <jarrett.billingsley gmail.com> writes:
On Mon, Feb 16, 2009 at 9:54 PM, Walter Bright
<newshound1 digitalmars.com> wrote:
 One thing you can try is:

 1. download imagecfg.exe from http://www.robpol86.com/pages/imagecfg.php

 2. run the command:

        imagecfg -a 0x1 link.exe

 This will set optlink to use only one core. I've suspected for a while that
 the multithreading locks in it don't work right for multicore. I have a
 single core machine and cannot test multicore behavior.

Tom S does all of his development on a single-core machine, and I have this problem on my single-core laptop as well. It is not multithreading.
Feb 16 2009
prev sibling next sibling parent Jarrett Billingsley <jarrett.billingsley gmail.com> writes:
On Mon, Feb 16, 2009 at 11:08 PM, Walter Bright
<newshound1 digitalmars.com> wrote:
 But I downloaded the offending obj file from bugzilla, and it links and runs
 without error:

 http://d.puremagic.com/issues/show_bug.cgi?id=2436

http://d.puremagic.com/issues/show_bug.cgi?id=424 is *much* older, and you already seem to have classed it as unfixable. It's the "too many fixups" bug (although instead of popping up a message to that effect OPTLINK just unhelpfully crashes and dumps its registers). This is the one that bites me all the time. I'm not doing terribly uncommon or large things. Even something like Pyd is liable to kill OPTLINK with relatively small wrappings. Splitting code across multiple modules _sometimes_ helps and _sometimes_ does not. Besides being a crude workaround it's sometimes much more difficult than it sounds (because as we all know, DMD does *not* like circular imports, and sometimes massive modules are the result of working around _that_ bug). -g affects it as it's now generating lots of extra debug symbols, pushing the number of fixups up. There's no way to tell how many symbols I'm generating and I shouldn't _have_ to worry about that kind of crap. Worst of all, DMDWin is the *only* compiler with these problems. So I'm doing all this work to get around bugs in a single compiler on a single platform. It makes me want to just abandon it. Do you think that with the new uber-templated Phobos2 that the problem is going to be getting any better?
Feb 16 2009
prev sibling next sibling parent Jarrett Billingsley <jarrett.billingsley gmail.com> writes:
On Mon, Feb 16, 2009 at 11:32 PM, Walter Bright
<newshound1 digitalmars.com> wrote:
 Jarrett Billingsley wrote:
 http://d.puremagic.com/issues/show_bug.cgi?id=424

I just added this to the bug report: ====================== The compiler already has a switch to generate multiple obj files from one source file: -multiobj. I use it for debugging. But it would be a royal pain to use if there's a lot.

Why isn't it documented? And you're damn right it would be a pain. Especially when I'm compiling three external libraries and a large library of my own.
 A better way is to compile to a library with the -lib switch. That splits
 the module into a zillion obj files, and stuffs them all into the library.
 Then, link with the library (the main() program will still have to be
 separate).
 =======================

 Can you give it a try?

Not easily. Under my normal compilation scheme I'm not building a library and I have no idea how well bud/rebuild will cope with a zillion object files. In any case the -lib switch is just sidestepping the problems with the linker and object format by pushing some of the linker's responsibilities into the compiler. DMD (and the D language itself, if you want to count the 16MB limit on static data, which has no place in a portable language specification!) already makes far too many concessions to deal with the ridiculous constraints of the linker. Please, stop designing around the problem and fix it.
Feb 16 2009
prev sibling next sibling parent grauzone <none example.net> writes:
Jarrett Billingsley wrote:
 This is, for me, the number one blocker of most of my more complex
 code on Windows.  More than any DMD bug.  Tom S can vouch for it too,
 and Eldar sounds like he s about to give up on D after dealing with
 OPTLINK's crashes.

He's not the only one. I hope Walter is aware how serious the problem is. Although D is a really nice language, it becomes unusable with its shitty toolchain support. Same with "popular" bugs like forward referencing / circular module bugs. If I tell someone about D, I always add a warning about how shitty the toolchain is. Then I hope they won't just ditch D if they hit compiler and linker bugs... Even on Linux, where the GNU linker is used, there are some problems with linking. Sometimes, the linker will complain about either multiply defined symbols or missing ones. Or the compiled program segfaults... then you completely recompile it, and it works.
Feb 16 2009
prev sibling next sibling parent "Danny Wilson" <bluezenix gmail.com> writes:
Op Tue, 17 Feb 2009 11:06:43 +0100 schreef Walter Bright  
<newshound1 digitalmars.com>:

 It does not crash when I try it. Are you sure you're not running a  
 multicore computer?

It's pretty hard to _NOT_ get a multicore computer these days. Please kill OPTLINK.
Feb 17 2009
prev sibling next sibling parent "Denis Koroskin" <2korden gmail.com> writes:
On Tue, 17 Feb 2009 14:24:10 +0300, Walter Bright <newshound1 digitalmars.com>
wrote:

 Danny Wilson wrote:
  Op Tue, 17 Feb 2009 11:06:43 +0100 schreef Walter Bright  
 <newshound1 digitalmars.com>:

 It does not crash when I try it. Are you sure you're not running a  
 multicore computer?

kill OPTLINK.

What I'm asking is for someone with a multicore computer who is experiencing these crashes to try the patch I outlined to restrict it to one core.

It is responded that three people (Tom S, Jarrett, Max Samukha) are running single-core machines and still experiencing constant crashes.
Feb 17 2009
prev sibling next sibling parent Jarrett Billingsley <jarrett.billingsley gmail.com> writes:
On Tue, Feb 17, 2009 at 4:49 AM, Tom S <h3r3tic remove.mat.uni.torun.pl> wrote:

 Additionally, using such an approach breaks any possibility to use
 incremental compilation. So there's also the option of having a .lib per .d
 file and linking that all together. So I've just tried it:
 dmd -c Main.d
 dmd -lib Mod.d
 dmd -lib Mod2.d
 dmd Main.obj Mod.lib Mod2.d
 ... and OPTLINK crashed with the familiar MsgBox containing register values.

I can confirm that this happens on both my laptop (single-core, single-threaded Pentium M) and my desktop. Using imagecfg changes nothing.
Feb 17 2009
prev sibling parent Jarrett Billingsley <jarrett.billingsley gmail.com> writes:
On Tue, Feb 17, 2009 at 9:06 AM, Jarrett Billingsley
<jarrett.billingsley gmail.com> wrote:
 On Tue, Feb 17, 2009 at 4:49 AM, Tom S <h3r3tic remove.mat.uni.torun.pl> wrote:

 Additionally, using such an approach breaks any possibility to use
 incremental compilation. So there's also the option of having a .lib per .d
 file and linking that all together. So I've just tried it:
 dmd -c Main.d
 dmd -lib Mod.d
 dmd -lib Mod2.d
 dmd Main.obj Mod.lib Mod2.d
 ... and OPTLINK crashed with the familiar MsgBox containing register values.

I can confirm that this happens on both my laptop (single-core, single-threaded Pentium M) and my desktop. Using imagecfg changes nothing.

Also, I'm using DMD 1.039 and OPTLINK 8.00.1 on both machines.
Feb 17 2009