www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - Alternative linker win32/64

reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
The fresh beta of universal linker UniLink has arrived, with exciting 
new features.

Quoting the changelog:
+  Preliminary support of Digital Mars D language compiler.

+  The debug info processing in old CV format added (dmc, dmd, watcom, 
old ml).

And it works! At least for the first part.
I actually linked my port of DMDscript and run it through some stress 
tests, no problems so far.
The nice thing is reduction in half of the resulting binary size.

As to debug info I think it may need another round of polishing... but 
the devs are willing to support it ;)
C:\ulink>ulink -v test.obj
UniLink v1.07 [beta] (build 3.01)
Error: Invalid debug information in 'test.obj'

Nevertheless the grim rule of Optlink in windows lands is at end :)

Some tips on how to get started.

Grab the latest beta here (for now it's ulnb0301.zip):
ftp://ftp.styx.cabel.net/pub/UniLink/
Installation is extract + add to PATH

Also to save you a bit of investigation, the relevant ulink.cfg config 
for dmd is along the lines:
-zkernel32;advapi32;user32;wsock32;shell32;snn
-LC:\dmd2\windows\lib
-Go


-- 
Dmitry Olshansky
Feb 18 2011
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/18/11 4:00 AM, Dmitry Olshansky wrote:

 The fresh beta of universal linker UniLink has arrived, with exciting
 new features.


Wow. If this works out, it is the closest thing in real life to "Deux ex 
machina". We were pretty much out of options.

Andrei
Feb 18 2011
parent reply dsimcha <dsimcha yahoo.com> writes:
== Quote from Andrei Alexandrescu (SeeWebsiteForEmail erdani.org)'s article

 On 2/18/11 4:00 AM, Dmitry Olshansky wrote:

 The fresh beta of universal linker UniLink has arrived, with exciting
 new features.


 Wow. If this works out, it is the closest thing in real life to "Deux ex
 machina". We were pretty much out of options.
 Andrei


Let's say for the sake of discussion that UniLink does work.  How close does
that
put us to having 64 support on Windows, too, if OptLink doesn't need to be
translated/rewritten/whatever?
Feb 18 2011
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/18/11 9:42 AM, dsimcha wrote:

 == Quote from Andrei Alexandrescu (SeeWebsiteForEmail erdani.org)'s article

 On 2/18/11 4:00 AM, Dmitry Olshansky wrote:

 The fresh beta of universal linker UniLink has arrived, with exciting
 new features.


 Wow. If this works out, it is the closest thing in real life to "Deux ex
 machina". We were pretty much out of options.
 Andrei


 Let's say for the sake of discussion that UniLink does work.  How close does
that
 put us to having 64 support on Windows, too, if OptLink doesn't need to be
 translated/rewritten/whatever?


AFAIK the linker is the largest roadblock, but Walter should really 
answer this question.

Andrei
Feb 18 2011
parent reply Daniel Gibson <metalcaedes gmail.com> writes:
Am 18.02.2011 16:41, schrieb Andrei Alexandrescu:

 On 2/18/11 9:42 AM, dsimcha wrote:

 == Quote from Andrei Alexandrescu (SeeWebsiteForEmail erdani.org)'s
 article

 On 2/18/11 4:00 AM, Dmitry Olshansky wrote:

 The fresh beta of universal linker UniLink has arrived, with exciting
 new features.


 Wow. If this works out, it is the closest thing in real life to "Deux ex
 machina". We were pretty much out of options.
 Andrei


 Let's say for the sake of discussion that UniLink does work. How close
 does that
 put us to having 64 support on Windows, too, if OptLink doesn't need
 to be
 translated/rewritten/whatever?


 AFAIK the linker is the largest roadblock, but Walter should really
 answer this question.

 Andrei


I've got this message (and the other one from you in this thread) twice, 
one sent to the poster you were replying to and one without an explicit 
to, but both in this news group.
Was that intended?

Cheers,
- Daniel
Feb 18 2011
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/18/11 9:48 AM, Daniel Gibson wrote:

 Am 18.02.2011 16:41, schrieb Andrei Alexandrescu:

 On 2/18/11 9:42 AM, dsimcha wrote:

 == Quote from Andrei Alexandrescu (SeeWebsiteForEmail erdani.org)'s
 article

 On 2/18/11 4:00 AM, Dmitry Olshansky wrote:

 The fresh beta of universal linker UniLink has arrived, with exciting
 new features.


 Wow. If this works out, it is the closest thing in real life to
 "Deux ex
 machina". We were pretty much out of options.
 Andrei


 Let's say for the sake of discussion that UniLink does work. How close
 does that
 put us to having 64 support on Windows, too, if OptLink doesn't need
 to be
 translated/rewritten/whatever?


 AFAIK the linker is the largest roadblock, but Walter should really
 answer this question.

 Andrei


 I've got this message (and the other one from you in this thread) twice,
 one sent to the poster you were replying to and one without an explicit
 to, but both in this news group.
 Was that intended?

 Cheers,
 - Daniel


Yes, it's a common mistake I make. Sorry.

Andrei
Feb 18 2011
prev sibling next sibling parent reply Trass3r <un known.com> writes:
Sounds fantastic, but what about those rumors that Unilink will become  
commercial sometime?
Feb 18 2011
parent reply dsimcha <dsimcha yahoo.com> writes:
== Quote from Trass3r (un known.com)'s article

 Sounds fantastic, but what about those rumors that Unilink will become
 commercial sometime?


Well, IMHO the intermediate solution would be to just get DMD to output Win64
OMF
object files for now, and cross the linking bridge when we come to it.  Unilink
is
good enough for at least testing that the object files output by DMD are
correct,
and should be at least better than nothing until Optlink gets rewritten.
Feb 18 2011
parent reply Trass3r <un known.com> writes:
 Well, IMHO the intermediate solution would be to just get DMD to output  
 Win64 OMF object files for now


Hopefully we won't have to wait too long!
Feb 18 2011
parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
Is there even such a thing as Win64 OMF?
Feb 18 2011
parent dsimcha <dsimcha yahoo.com> writes:
On 2/18/2011 2:28 PM, Andrej Mitrovic wrote:

 Is there even such a thing as Win64 OMF?


Ok, maybe there isn't.  I just assumed there was.  The bottom line is 
that it makes sense to get DMD to generate the 64-bit object files first 
so that people with alternative linkers can use it in 64 mode, and then 
and only then worry about the linker.
Feb 18 2011
prev sibling next sibling parent reply notna <notna.remove.this ist-einmalig.de> writes:
Am I the only one getting a virus warning as soon as I try to unpack the 
ZIP file?

This one is found within the binary:
TR/Crypt.XPACK.Gen

Regards
  notna

On 18.02.2011 11:00, Dmitry Olshansky wrote:

 The fresh beta of universal linker UniLink has arrived, with exciting
 new features.

 Quoting the changelog:
 + Preliminary support of Digital Mars D language compiler.

 + The debug info processing in old CV format added (dmc, dmd, watcom,
 old ml).

 And it works! At least for the first part.
 I actually linked my port of DMDscript and run it through some stress
 tests, no problems so far.
 The nice thing is reduction in half of the resulting binary size.

 As to debug info I think it may need another round of polishing... but
 the devs are willing to support it ;)
 C:\ulink>ulink -v test.obj
 UniLink v1.07 [beta] (build 3.01)
 Error: Invalid debug information in 'test.obj'

 Nevertheless the grim rule of Optlink in windows lands is at end :)

 Some tips on how to get started.

 Grab the latest beta here (for now it's ulnb0301.zip):
 ftp://ftp.styx.cabel.net/pub/UniLink/
 Installation is extract + add to PATH

 Also to save you a bit of investigation, the relevant ulink.cfg config
 for dmd is along the lines:
 -zkernel32;advapi32;user32;wsock32;shell32;snn
 -LC:\dmd2\windows\lib
 -Go
Feb 18 2011
next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 18.02.2011 23:15, notna wrote:

 Am I the only one getting a virus warning as soon as I try to unpack 
 the ZIP file?

 This one is found within the binary:
 TR/Crypt.XPACK.Gen


Ouch, that creepy. My McAfee doesn't detect anything. Being suspicious  
by nature I run it through http://www.virustotal.com 
<http://www.virustotal.com/>
1/42 positives which makes my think you have Avira AntiVir.
Well, I'm aware that UniLink uses some weird protection techs to prevent 
revers engineering, and digging through Avira database reveals it's 
heuristic entry in nature:

*TR/Crypt.XPACK.Gen*

*Description:*
A generic detection routine designed to detect common family 
characteristics shared in several variants.

This special detection routine was developed in order to detect unknown 
variants and will be enhanced continuously.

IMHO it's misfire, but it might be a good idea to send them the file for 
analysis.

-- 
Dmitry Olshansky
Feb 18 2011
next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 18 Feb 2011 15:35:17 -0500, Dmitry Olshansky  
<dmitry.olsh gmail.com> wrote:


 On 18.02.2011 23:15, notna wrote:

 Am I the only one getting a virus warning as soon as I try to unpack  
 the ZIP file?

 This one is found within the binary:
 TR/Crypt.XPACK.Gen


 Ouch, that creepy. My McAfee doesn't detect anything. Being suspicious   
 by nature I run it through http://www.virustotal.com  
 <http://www.virustotal.com/>
 1/42 positives which makes my think you have Avira AntiVir.
 Well, I'm aware that UniLink uses some weird protection techs to prevent  
 revers engineering, and digging through Avira database reveals it's  
 heuristic entry in nature:

 *TR/Crypt.XPACK.Gen*

 *Description:*
 A generic detection routine designed to detect common family  
 characteristics shared in several variants.

 This special detection routine was developed in order to detect unknown  
 variants and will be enhanced continuously.

 IMHO it's misfire, but it might be a good idea to send them the file for  
 analysis.


Yeah, I dropped Avira in favor of Avast because they have some big  
problems, kind of like this.

-Steve
Feb 18 2011
prev sibling parent notna <notna.remove.this ist-einmalig.de> writes:
Yes, I've Avira AntiVir. I'm quite happy with it since years and it's 
kind od common here in Germany...

Thanks a lot for your help, I'll send them the file and the log...

Btw, also thanks for the tool. It really sounds great!


On 18.02.2011 21:35, Dmitry Olshansky wrote:

 On 18.02.2011 23:15, notna wrote:

 Am I the only one getting a virus warning as soon as I try to unpack
 the ZIP file?

 This one is found within the binary:
 TR/Crypt.XPACK.Gen


 Ouch, that creepy. My McAfee doesn't detect anything. Being suspicious
 by nature I run it through http://www.virustotal.com
 <http://www.virustotal.com/>
 1/42 positives which makes my think you have Avira AntiVir.
 Well, I'm aware that UniLink uses some weird protection techs to prevent
 revers engineering, and digging through Avira database reveals it's
 heuristic entry in nature:

 *TR/Crypt.XPACK.Gen*

 *Description:*
 A generic detection routine designed to detect common family
 characteristics shared in several variants.

 This special detection routine was developed in order to detect unknown
 variants and will be enhanced continuously.

 IMHO it's misfire, but it might be a good idea to send them the file for
 analysis.
Feb 18 2011
prev sibling parent Trass3r <un known.com> writes:
 Am I the only one getting a virus warning as soon as I try to unpack the  
 ZIP file?

 This one is found within the binary:
 TR/Crypt.XPACK.Gen


Yep, Antivir is pretty paranoid and fires up even if an exe is only  
compressed.
Feb 18 2011
prev sibling parent reply Sebastian Schuberth <sschuberth gmail.com> writes:
On 18.02.2011 11:00, Dmitry Olshansky wrote:


 The nice thing is reduction in half of the resulting binary size.


That's indeed nice! The unnecessarily huge size of binaries created with 
D / Optlink was in fact something hindering me to use D at all!

-- 
Sebastian Schuberth
Feb 24 2011
parent reply Walter Bright <newshound2 digitalmars.com> writes:
Sebastian Schuberth wrote:

 On 18.02.2011 11:00, Dmitry Olshansky wrote:
 

 The nice thing is reduction in half of the resulting binary size.


 
 That's indeed nice! The unnecessarily huge size of binaries created with 
 D / Optlink was in fact something hindering me to use D at all!


I'm sure that linker is doing it by writing compressed exe's. This means that
it 
has the same memory footprint, and it loads slower because it must be 
decompressed. Also, if you store it in a zip file, the zip file won't be any 
smaller because recompressing compressed data doesn't make it smaller.
Feb 24 2011
next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 24.02.2011 21:41, Walter Bright wrote:

 Sebastian Schuberth wrote:

 On 18.02.2011 11:00, Dmitry Olshansky wrote:


 The nice thing is reduction in half of the resulting binary size.


 That's indeed nice! The unnecessarily huge size of binaries created 
 with D / Optlink was in fact something hindering me to use D at all!


 I'm sure that linker is doing it by writing compressed exe's. This 
 means that it has the same memory footprint, and it loads slower 
 because it must be decompressed. Also, if you store it in a zip file, 
 the zip file won't be any smaller because recompressing compressed 
 data doesn't make it smaller.


Tried that, it's still compresses to roughly 50% of it's size. Well, I 
no expert but a glance at disassembly shows that at least strings  / 
imports are intact.
For what it's worth the IDA control flow analysis shows a bulk of 
unreferenced procedures in optlink's one.
So maybe the case is that they also included a decent stripper.

-- 
Dmitry Olshansky
Feb 24 2011
parent Walter Bright <newshound2 digitalmars.com> writes:
Dmitry Olshansky wrote:

 On 24.02.2011 21:41, Walter Bright wrote:

 Sebastian Schuberth wrote:

 On 18.02.2011 11:00, Dmitry Olshansky wrote:


 The nice thing is reduction in half of the resulting binary size.


 That's indeed nice! The unnecessarily huge size of binaries created 
 with D / Optlink was in fact something hindering me to use D at all!


 I'm sure that linker is doing it by writing compressed exe's. This 
 means that it has the same memory footprint, and it loads slower 
 because it must be decompressed. Also, if you store it in a zip file, 
 the zip file won't be any smaller because recompressing compressed 
 data doesn't make it smaller.


 
 Tried that, it's still compresses to roughly 50% of it's size. Well, I 
 no expert but a glance at disassembly shows that at least strings  / 
 imports are intact.
 For what it's worth the IDA control flow analysis shows a bulk of 
 unreferenced procedures in optlink's one.
 So maybe the case is that they also included a decent stripper.


If that's so, that's good!
Feb 24 2011
prev sibling parent reply Sebastian Schuberth <sschuberth gmail.com> writes:
On 24.02.2011 19:41, Walter Bright wrote:


 The nice thing is reduction in half of the resulting binary size.


 That's indeed nice! The unnecessarily huge size of binaries created
 with D / Optlink was in fact something hindering me to use D at all!


 I'm sure that linker is doing it by writing compressed exe's. This means
 that it has the same memory footprint, and it loads slower because it
 must be decompressed. Also, if you store it in a zip file, the zip file


IMHO, that is a common misbelief when it comes to executable 
compressors. AFAIK, the time required for decompression is 
overcompensated by the time required to read less data from disk, even 
still nowadays.


 won't be any smaller because recompressing compressed data doesn't make
 it smaller.


There really needs to be no compression or back magic involved to make 
the executable size for a simple program like

---8<---

import std.stdio;

void main(string[] args)
{
     writeln("Hello World, Reloaded");
}

---8<---

smaller than the current 286 KiB! Some dead code elemination would 
already do, I guess.

-- 
Sebastian Schuberth
Feb 26 2011
next sibling parent Michel Fortin <michel.fortin michelf.com> writes:
On 2011-02-26 06:10:19 -0500, Sebastian Schuberth <sschuberth gmail.com> said:


 On 24.02.2011 19:41, Walter Bright wrote:
 

 The nice thing is reduction in half of the resulting binary size.


 
 That's indeed nice! The unnecessarily huge size of binaries created
 with D / Optlink was in fact something hindering me to use D at all!


 
 I'm sure that linker is doing it by writing compressed exe's. This means
 that it has the same memory footprint, and it loads slower because it
 must be decompressed. Also, if you store it in a zip file, the zip file


 
 IMHO, that is a common misbelief when it comes to executable 
 compressors. AFAIK, the time required for decompression is 
 overcompensated by the time required to read less data from disk, even 
 still nowadays.


True in general. But in the case of an executable file, how does it 
interacts with memory mapping in the virtual memory subsystem? If you 
need to actually load and decompress the whole executable upfront 
instead of relying on the virtual memory subsystem to load pages from 
disk as needed, it might very well increase your startup time. Also, if 
some parts of the executable needs to be paged out to disk because free 
physical memory is running low, this is almost free if the memory is 
directly mapped to a read-only file as the data is already on disk.

So there's certainly some scenarios where a compressed executable will 
fare better, but there's surely plenty others where it's the reverse.

-- 
Michel Fortin
michel.fortin michelf.com
http://michelf.com/
Feb 26 2011
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2011-02-26 12:10, Sebastian Schuberth wrote:

 On 24.02.2011 19:41, Walter Bright wrote:


 The nice thing is reduction in half of the resulting binary size.


 That's indeed nice! The unnecessarily huge size of binaries created
 with D / Optlink was in fact something hindering me to use D at all!


 I'm sure that linker is doing it by writing compressed exe's. This means
 that it has the same memory footprint, and it loads slower because it
 must be decompressed. Also, if you store it in a zip file, the zip file


 IMHO, that is a common misbelief when it comes to executable
 compressors. AFAIK, the time required for decompression is
 overcompensated by the time required to read less data from disk, even
 still nowadays.


 won't be any smaller because recompressing compressed data doesn't make
 it smaller.


 There really needs to be no compression or back magic involved to make
 the executable size for a simple program like

 ---8<---

 import std.stdio;

 void main(string[] args)
 {
 writeln("Hello World, Reloaded");
 }

 ---8<---

 smaller than the current 286 KiB! Some dead code elemination would
 already do, I guess.


On Mac OS X linking a Hello World application dynamically to Tango 
results in a 16Kb executable, the same size as for a Hello World written 
in C.

-- 
/Jacob Carlborg
Feb 27 2011