www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Unilink - alternative linker for win32/64, DMD OMF extensions?

reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

After hitting some troubles with optlink in the past (though my problems 
got solved others may be not)
I was in constant search for alternatives.
The good news:
there *do* happen to be very versatile linker for windows able to 
produce  32/64bit PE, that supports OMF format and so on. Another cool 
thing that has is so called mix-mode linking i.e. linking objects 
produced by different compilers with different RTLs (tricky but might 
worth the effort).
Link to ftp with docs and binary: ftp://ftp.styx.cabel.net/pub/UniLink/
According to author it does fully support DMC++ along with a huge pack 
of other C++ compilers.

Now the bad news:
it doesn't quite work with DMD for the moment. It's able to link test.d, 
but fails to link test2.d. (see attach) Stating that the resulting OMF 
is invalid. I asked the developer about it,  his reply:
/reply

DM is in list of supported compilers (with DM C extensions). But D
extension in sample is suspicious. There it attempt to define "select
any" comdat that referenced to TLS-type segment. This is similar to common
backend error. Another possibility is that it is an D "extension". But
we don't possess documentation of D object format extensions and only
can gess what it is.

If the D becomes common used language we can "examine" its new
features and take in into consideration. But... Maybe is's easier to
contact with D developer and ask for new features in objects.
/end reply

QUESTION
--->>>
So here I am asking what kind of OMF extensions DMD uses, and what's difference
between test.d and test2.d in terms of object file output?
<<<---

Any hints on where to look for them are also welcome.
If they are minor then we can convince the developer
and have 64bit linker to experiment with on windows.

NOTES

I used dmd 2.051, though with 2.050 it's the same issue;
command line:
dmd -c<xxx>.d

For those interested in experimenting with unilink themselves,
use the following config (ulink.cfg) adjusting the path to dmd as needed:
-zkernel32;advapi32;user32;wsock32;shell32;snn.lib
-LC:\dmd2\windows\lib
-Go

Then linking is just: ulink <xxx>.obj

-- 
Dmitry Olshansky
Jan 23 2011
next sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
Yeah I can reproduce the same thing as you have.

I did manage to convert an OMF to COFF file format, and link it with
GCC (MinGW). See my thread here:

http://www.digitalmars.com/pnews/read.php?server=news.digitalmars.com&group=digitalmars.D&artnum=127079

I have yet to try it on more complex examples though. Give it a try
and let me know if it works? :)
Jan 23 2011
parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 23.01.2011 20:43, Andrej Mitrovic wrote:
 Yeah I can reproduce the same thing as you have.

 I did manage to convert an OMF to COFF file format, and link it with
 GCC (MinGW). See my thread here:

 http://www.digitalmars.com/pnews/read.php?server=news.digitalmars.com&group=digitalmars.D&artnum=127079

 I have yet to try it on more complex examples though. Give it a try
 and let me know if it works? :)

Fatal: Entry point must be specified So, yes it strips something, yet it's no longer an invalid OMF ;) Anyhow after jumping though some hoops with objconv I got ld to linktime, but it gets a pack of unresolved externals from D's runtime/phobos and I don't think converting them all is wise idea. -- Dmitry Olshansky
Jan 23 2011
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
Perhaps this page would be helpful for the linker guys:
http://www.digitalmars.com/d/2.0/abi.html
Jan 23 2011
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
Yeah, we're kind of stuck with Optlink for now.
Jan 23 2011
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
Dmitry Olshansky wrote:
 If the D becomes common used language we can "examine" its new
 features and take in into consideration. But... Maybe is's easier to
 contact with D developer and ask for new features in objects.

If the author wants to email me with any specific questions, I'd be happy to help him.
Jan 23 2011
next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 23.01.2011 23:55, Walter Bright wrote:
 Dmitry Olshansky wrote:
 If the D becomes common used language we can "examine" its new
 features and take in into consideration. But... Maybe is's easier to
 contact with D developer and ask for new features in objects.

If the author wants to email me with any specific questions, I'd be happy to help him.

himself unless someone proves it's usefulness. It's more like we need a little help not the other way around. Well, anyway I can mail him and hope that he will do it just out of curiosity, what's yours relevant email then? May I just as well tell him that you are interested in it or anything? -- Dmitry Olshansky
Jan 23 2011
parent reply Walter Bright <newshound2 digitalmars.com> writes:
Dmitry Olshansky wrote:
 On 23.01.2011 23:55, Walter Bright wrote:
 Dmitry Olshansky wrote:
 If the D becomes common used language we can "examine" its new
 features and take in into consideration. But... Maybe is's easier to
 contact with D developer and ask for new features in objects.

If the author wants to email me with any specific questions, I'd be happy to help him.

himself unless someone proves it's usefulness. It's more like we need a little help not the other way around. Well, anyway I can mail him and hope that he will do it just out of curiosity, what's yours relevant email then? May I just as well tell him that you are interested in it or anything?

First off, there are no D extensions to the OMF. There was an extension added around 1993 or so to support long identifier names. There were also a couple extensions to the codeview debug format, which are documented in the ABI. My offer is if there is something specific about the OMF files generated that I can explain to him, I'd be happy to do so. My mail is walter followed by digitalmars.com.
Jan 23 2011
parent reply Trass3r <un known.com> writes:
Walter Bright Wrote:
 Dmitry Olshansky wrote:
 Well, anyway I can mail him and hope that he will do it just out of 
 curiosity, what's yours relevant email then?
 May I just as well tell him that you are interested in it or anything?
 

My offer is if there is something specific about the OMF files generated that I can explain to him, I'd be happy to do so. My mail is walter followed by digitalmars.com.

Any news here?
Jan 26 2011
parent reply Walter Bright <newshound2 digitalmars.com> writes:
Trass3r wrote:
 Walter Bright Wrote:
 Dmitry Olshansky wrote:
 Well, anyway I can mail him and hope that he will do it just out of 
 curiosity, what's yours relevant email then?
 May I just as well tell him that you are interested in it or anything?

can explain to him, I'd be happy to do so. My mail is walter followed by digitalmars.com.

Any news here?

No, I have not heard anything.
Jan 26 2011
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 26.01.2011 21:52, Walter Bright wrote:
 Trass3r wrote:
 Walter Bright Wrote:
 Dmitry Olshansky wrote:
 Well, anyway I can mail him and hope that he will do it just out of 
 curiosity, what's yours relevant email then?
 May I just as well tell him that you are interested in it or anything?

generated that I can explain to him, I'd be happy to do so. My mail is walter followed by digitalmars.com.

Any news here?

No, I have not heard anything.

I've got their detailed report on the sample object file, and relayed it to Walter. Something related to D's stance on TLS it seems. With me being middle man all this can take some extra time, but I don't mind. The end result may still well worth it. -- Dmitry Olshansky
Jan 27 2011
parent reply Trass3r <un known.com> writes:
I wonder why this tool isn't promoted in any way, no website etc.
I also wonder if he was willing to make it open-source. Then we could help
support D and if it works someday, we can even include D symbol demangling :)
Jan 28 2011
parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 28.01.2011 13:21, Trass3r wrote:
 I wonder why this tool isn't promoted in any way, no website etc.

I got to this ftp only because I was 100% sure that tool existed, as a friend of mine suggested to use it instead of Borland's crappy linker few years ago.
 I also wonder if he was willing to make it open-source. Then we could help
support D and if it works someday, we can even include D symbol demangling :)

-- Dmitry Olshansky
Jan 28 2011
prev sibling parent Trass3r <un known.com> writes:
 I've got their detailed report on the sample object file, and relayed it  
 to Walter.
 Something related to D's stance on  TLS it seems.
 With me being middle man all this can take some extra time, but I don't  
 mind.
 The end result may still well worth it.

Hope so too.
Jan 27 2011
prev sibling next sibling parent Trass3r <un known.com> writes:
Hopefully we can get the author to support dmd.
I can't wait years until optlink might support x64.
Jan 23 2011
prev sibling next sibling parent reply Trass3r <un known.com> writes:
Just another thought:
dmd uses ld on linux, couldn't it use MinGW's ld on Windows?
Feb 06 2011
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2011-02-06 23:26, Trass3r wrote:
 Just another thought:
 dmd uses ld on linux, couldn't it use MinGW's ld on Windows?

MinGW's ld doesn't use the same object format as DMD/DMC -- /Jacob Carlborg
Feb 07 2011
prev sibling next sibling parent Trass3r <un known.com> writes:
 Just another thought:
 dmd uses ld on linux, couldn't it use MinGW's ld on Windows?

MinGW's ld doesn't use the same object format as DMD/DMC

I know, of course this would include discarding OMF.
Feb 07 2011
prev sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
I found this gem in digitalmars' bookshelf:

http://www.digitalmars.com/bibliography.html#fileformats
Feb 07 2011
prev sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
Ok, bumping this up with the latest news from UniLink developers:
<quote>

Ok, we release it's as D extension in next release.

Best regards,
  UniLink

</quote>

That's just plain awesome ;)

-- 
Dmitry Olshansky
Feb 11 2011
next sibling parent Daniel Gibson <metalcaedes gmail.com> writes:
Am 11.02.2011 19:56, schrieb Dmitry Olshansky:
 Ok, bumping this up with the latest news from UniLink developers:
 <quote>
 
 Ok, we release it's as D extension in next release.
 
 Best regards,
  UniLink
 
 </quote>
 
 That's just plain awesome ;)
 

Great :)
Feb 11 2011
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/11/11 12:56 PM, Dmitry Olshansky wrote:
 Ok, bumping this up with the latest news from UniLink developers:
 <quote>

 Ok, we release it's as D extension in next release.

 Best regards,
 UniLink

 </quote>

 That's just plain awesome ;)

Is UniLink working with 64 bits? Andrei
Feb 11 2011
parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 12.02.2011 3:13, Andrei Alexandrescu wrote:
 On 2/11/11 12:56 PM, Dmitry Olshansky wrote:
 Ok, bumping this up with the latest news from UniLink developers:
 <quote>

 Ok, we release it's as D extension in next release.

 Best regards,
 UniLink

 </quote>

 That's just plain awesome ;)

Is UniLink working with 64 bits? Andrei

Yes, that's actually one of it's benefits, as I told somewhere along the thread. To be 101% sure I just linked a simple 'hello x64' program. -- Dmitry Olshansky
Feb 11 2011
prev sibling next sibling parent Trass3r <un known.com> writes:
Hopefully that release will come soon ;)
Feb 12 2011
prev sibling next sibling parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
Dmitry Olshansky wrote:
 Ok, bumping this up with the latest news from UniLink developers:
 <quote>
 
 Ok, we release it's as D extension in next release.
 
 Best regards,
  UniLink
 
 </quote>
 
 That's just plain awesome ;)
 

yeah, it would be great to be able to link against anything else in the windows world. I just tried unilink on a small program compiled with dmd-2.051 and it worked alright. Surprisingly, the executable is less than half the size of the optlink generated file. Comparing the map files, it does not seem to reference less symbols (the corrupted map files written by optlink do not allow perfect comparison), so maybe there is a rather large alignment used by optlink? Unfortunately, unilink does not support the debug information emitted by dmd: UniLink v1.07 [beta] (build 1.19) Warning: CV(1)-style debug-info not supported Warning: Linked modules don't contain source-level debug information BTW: The authors seem to plan to make unilink a commercial product. Here's a quote from the help file: "If current version of UniLink is beta (all other versions are commercial and can't be used w/o license), keep in mind, any beta will "yaff" after ~1.5 month of expired."
Feb 12 2011
next sibling parent David Nadlinger <see klickverbot.at> writes:
On 2/12/11 3:30 PM, Andrej Mitrovic wrote:
 What linker does GDC use on Windows? And has anyone had any success
 with using it on Windows?

IIRC, it emits COFF and uses the ld from MinGW to link. David
Feb 12 2011
prev sibling next sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 12.02.2011 16:32, Rainer Schuetze wrote:
 Dmitry Olshansky wrote:
 Ok, bumping this up with the latest news from UniLink developers:
 <quote>

 Ok, we release it's as D extension in next release.

 Best regards,
  UniLink

 </quote>

 That's just plain awesome ;)

yeah, it would be great to be able to link against anything else in the windows world. I just tried unilink on a small program compiled with dmd-2.051 and it worked alright. Surprisingly, the executable is less than half the size of the optlink generated file. Comparing the map files, it does not seem to reference less symbols (the corrupted map files written by optlink do not allow perfect comparison), so maybe there is a rather large alignment used by optlink?

Same things here, as to the executable size it might be some smart technology they plan to sell ;)
 Unfortunately, unilink does not support the debug information emitted 
 by dmd:

 UniLink v1.07 [beta] (build 1.19)
 Warning: CV(1)-style debug-info not supported
 Warning: Linked modules don't contain source-level debug information

Yeah, that spoils the party...
 BTW: The authors seem to plan to make unilink a commercial product. 
 Here's a quote from the help file: "If current version of UniLink is 
 beta (all other versions are commercial and can't be used w/o 
 license), keep in mind, any beta will "yaff" after ~1.5 month of 
 expired."

I've seen that "yaff" :) Merely a message in console that states that your beta linker is out of date, functionality not crippled. More interestingly I haven't seen any commercial versions yet.. -- Dmitry Olshansky
Feb 12 2011
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
Andrej Mitrovic wrote:
 Great, from one closed-source linker to another.

Making optlink open source won't make any difference. Few are skilled at asm anymore, and likely none of them would want to work on optlink for free.
Feb 12 2011
next sibling parent reply "Akakima" <akakima33 gmail.com> writes:
 Making optlink open source won't make any difference. Few are skilled at 
 asm anymore, and likely none of them would want to work on optlink for 
 free.

That's true. But the real problem is not optlink. Optlink is a very good linker. The problem is OMF. 11 years ago OMF was a good choice. But not anymore. I know you are a competent (probably very competent) compiler writer. You modified D on linux, so it produce ELF. How much time would that take to modify DMD so it produce COFF ? Given all the bad publicity OMF gives to D, it should be viewed as a good choice. There are many (not much), but there are open source linkers. Of course ld is not as fast as optlink, but it's good. And there is a faster version made by the project Ultimate++ IDE. Going to COFF would have a lot of advantages for everybody and for D. Do you agree ?
Feb 12 2011
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
Akakima wrote:
 Making optlink open source won't make any difference. Few are skilled at 
 asm anymore, and likely none of them would want to work on optlink for 
 free.

That's true. But the real problem is not optlink. Optlink is a very good linker. The problem is OMF. 11 years ago OMF was a good choice. But not anymore. I know you are a competent (probably very competent) compiler writer. You modified D on linux, so it produce ELF. How much time would that take to modify DMD so it produce COFF ? Given all the bad publicity OMF gives to D, it should be viewed as a good choice. There are many (not much), but there are open source linkers. Of course ld is not as fast as optlink, but it's good. And there is a faster version made by the project Ultimate++ IDE. Going to COFF would have a lot of advantages for everybody and for D. Do you agree ?

Changing the object module format is not sufficient. The symbolic debug info would have to be changed (and Microsoft's is undocumented) and then there's the dependency on Microsoft's C runtime library if linking with VC generated object files.
Feb 12 2011
next sibling parent bearophile <bearophileHUGS lycos.com> writes:
Walter:

 Changing the object module format is not sufficient. The symbolic debug info 
 would have to be changed (and Microsoft's is undocumented) and then there's
the 
 dependency on Microsoft's C runtime library if linking with VC generated
object 
 files.

What about linking with MinGW (GCC 4.5+) object files? Bye, bearophile
Feb 12 2011
prev sibling parent reply "Akakima" <akakima33 gmail.com> writes:
 Changing the object module format is not sufficient. The symbolic debug 
 info would have to be changed (and Microsoft's is undocumented) and then 
 there's the dependency on Microsoft's C runtime library if linking with VC 
 generated object files.

I found some doc there: http://pierrelib.pagesperso-orange.fr/exec_formats/index.html Microsoft Symbol and Type Information By TIS / Microsoft. Entry added 12/28/2004. Keywords: ms, symbol, type, info File: MS_Symbol_Type_v1.0.pdf This document describes Microsoft Symbol and Type Information, a debugging information format fromMicrosoft Corporation for the 32-bit Windows environment. There is also some doc on the MSDN CD that comes with Visual C++ 6.0.
Feb 13 2011
parent reply Walter Bright <newshound2 digitalmars.com> writes:
Akakima wrote:
 Changing the object module format is not sufficient. The symbolic debug 
 info would have to be changed (and Microsoft's is undocumented) and then 
 there's the dependency on Microsoft's C runtime library if linking with VC 
 generated object files.

I found some doc there: http://pierrelib.pagesperso-orange.fr/exec_formats/index.html Microsoft Symbol and Type Information By TIS / Microsoft. Entry added 12/28/2004. Keywords: ms, symbol, type, info File: MS_Symbol_Type_v1.0.pdf This document describes Microsoft Symbol and Type Information, a debugging information format fromMicrosoft Corporation for the 32-bit Windows environment.

That document describes the Codeview symbol debug format, which Microsoft abandoned 15 years ago in favor of a proprietary format. Dmd generates that older format :-)
Feb 13 2011
parent reply Lutger Blijdestijn <lutger.blijdestijn gmail.com> writes:
Walter Bright wrote:

 Akakima wrote:
 Changing the object module format is not sufficient. The symbolic debug
 info would have to be changed (and Microsoft's is undocumented) and then
 there's the dependency on Microsoft's C runtime library if linking with
 VC generated object files.

I found some doc there: http://pierrelib.pagesperso-orange.fr/exec_formats/index.html Microsoft Symbol and Type Information By TIS / Microsoft. Entry added 12/28/2004. Keywords: ms, symbol, type, info File: MS_Symbol_Type_v1.0.pdf « This document describes Microsoft Symbol and Type Information, a debugging information format fromMicrosoft Corporation for the 32-bit Windows environment. »

That document describes the Codeview symbol debug format, which Microsoft abandoned 15 years ago in favor of a proprietary format. Dmd generates that older format :-)

Are you going to do Elf? With optlink in D? (Does that even make sense?)
Feb 13 2011
parent Walter Bright <newshound2 digitalmars.com> writes:
Lutger Blijdestijn wrote:
 Walter Bright wrote:
 
 Akakima wrote:
 Changing the object module format is not sufficient. The symbolic debug
 info would have to be changed (and Microsoft's is undocumented) and then
 there's the dependency on Microsoft's C runtime library if linking with
 VC generated object files.

http://pierrelib.pagesperso-orange.fr/exec_formats/index.html Microsoft Symbol and Type Information By TIS / Microsoft. Entry added 12/28/2004. Keywords: ms, symbol, type, info File: MS_Symbol_Type_v1.0.pdf « This document describes Microsoft Symbol and Type Information, a debugging information format fromMicrosoft Corporation for the 32-bit Windows environment. »

That document describes the Codeview symbol debug format, which Microsoft abandoned 15 years ago in favor of a proprietary format. Dmd generates that older format :-)

Are you going to do Elf? With optlink in D? (Does that even make sense?)

Elf is not a symbolic debug format. It is possible to use Codeview with Elf. In any case, such decisions will have to revolve around the availability of things like debuggers.
Feb 13 2011
prev sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday 12 February 2011 17:09:36 Walter Bright wrote:
 Akakima wrote:
 Making optlink open source won't make any difference. Few are skilled at
 asm anymore, and likely none of them would want to work on optlink for
 free.

That's true. But the real problem is not optlink. Optlink is a very good linker. The problem is OMF. 11 years ago OMF was a good choice. But not anymore. I know you are a competent (probably very competent) compiler writer. You modified D on linux, so it produce ELF. How much time would that take to modify DMD so it produce COFF ? Given all the bad publicity OMF gives to D, it should be viewed as a good choice. There are many (not much), but there are open source linkers. Of course ld is not as fast as optlink, but it's good. And there is a faster version made by the project Ultimate++ IDE. Going to COFF would have a lot of advantages for everybody and for D. Do you agree ?

Changing the object module format is not sufficient. The symbolic debug info would have to be changed (and Microsoft's is undocumented) and then there's the dependency on Microsoft's C runtime library if linking with VC generated object files.

That's a problem for some code and pretty much an absolute necessity for other code. Any project that I'd do at work would _have_ to use Microsoft's runtime library. In the long run, we really do need a solution which is compatible with Microsoft's compiler. Without that, there are a lot of programmers who just won't be able to use D in any project which would be using C or C++ code, and not being able to use any C or C++ code in D can be a major problem - especially if you're trying to use D with an existing code base. So, I really don't know what the best way to handle all of this is, and I really don't know all that much about linkers, but the fact that you can't link D code with C code generated by Microsoft's compiler is crippling for Windows development. - Jonathan M Davis
Feb 12 2011
prev sibling parent %u <u pathlink.org> writes:
Andrej Mitrovic Wrote:

 On 2/12/11, Walter Bright <newshound2 digitalmars.com> wrote:
 Andrej Mitrovic wrote:
 Great, from one closed-source linker to another.

Making optlink open source won't make any difference. Few are skilled at asm anymore, and likely none of them would want to work on optlink for free.

Ok, but what about your upcoming C version? Don't get me wrong, I appreciate that we even have a linker! But it's kind of sad when one of the most active D developers quits because of linker issues.

We should rename Optlink to Vuvulink, it's the vuvuzela of D's world. We can't seriously use D with any commercial technologies since the code isn't binary compatible. This is getting ridiculous. 12 years of fail. We could even pay to get a proper linker. But how much would it cost? Around $200000 - $500000. That's too much. We could outsource our whole software development at that price and focus on profits.
Feb 12 2011
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 2/12/11, Rainer Schuetze <r.sagitario gmx.de> wrote:
 BTW: The authors seem to plan to make unilink a commercial product.
 Here's a quote from the help file: "If current version of UniLink is
 beta (all other versions are commercial and can't be used w/o license),
 keep in mind, any beta will "yaff" after ~1.5 month of expired."

Great, from one closed-source linker to another. What linker does GDC use on Windows? And has anyone had any success with using it on Windows?
Feb 12 2011
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 2/12/11, David Nadlinger <see klickverbot.at> wrote:
 On 2/12/11 3:30 PM, Andrej Mitrovic wrote:
 What linker does GDC use on Windows? And has anyone had any success
 with using it on Windows?

IIRC, it emits COFF and uses the ld from MinGW to link. David

That's nice to know. I'll give GDC a spin and see how it's working out.
Feb 12 2011
prev sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 2/12/11, Walter Bright <newshound2 digitalmars.com> wrote:
 Andrej Mitrovic wrote:
 Great, from one closed-source linker to another.

Making optlink open source won't make any difference. Few are skilled at asm anymore, and likely none of them would want to work on optlink for free.

Ok, but what about your upcoming C version? Don't get me wrong, I appreciate that we even have a linker! But it's kind of sad when one of the most active D developers quits because of linker issues.
Feb 12 2011