www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - Alternative linker win32/64

reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
The fresh beta of universal linker UniLink has arrived, with exciting 
new features.

Quoting the changelog:
+  Preliminary support of Digital Mars D language compiler.

+  The debug info processing in old CV format added (dmc, dmd, watcom, 
old ml).

And it works! At least for the first part.
I actually linked my port of DMDscript and run it through some stress 
tests, no problems so far.
The nice thing is reduction in half of the resulting binary size.

As to debug info I think it may need another round of polishing... but 
the devs are willing to support it ;)
C:\ulink>ulink -v test.obj
UniLink v1.07 [beta] (build 3.01)
Error: Invalid debug information in 'test.obj'

Nevertheless the grim rule of Optlink in windows lands is at end :)

Some tips on how to get started.

Grab the latest beta here (for now it's ulnb0301.zip):
ftp://ftp.styx.cabel.net/pub/UniLink/
Installation is extract + add to PATH

Also to save you a bit of investigation, the relevant ulink.cfg config 
for dmd is along the lines:
-zkernel32;advapi32;user32;wsock32;shell32;snn
-LC:\dmd2\windows\lib
-Go


-- 
Dmitry Olshansky
Feb 18 2011
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/18/11 4:00 AM, Dmitry Olshansky wrote:
 The fresh beta of universal linker UniLink has arrived, with exciting
 new features.

Wow. If this works out, it is the closest thing in real life to "Deux ex machina". We were pretty much out of options. Andrei
Feb 18 2011
parent reply dsimcha <dsimcha yahoo.com> writes:
== Quote from Andrei Alexandrescu (SeeWebsiteForEmail erdani.org)'s article
 On 2/18/11 4:00 AM, Dmitry Olshansky wrote:
 The fresh beta of universal linker UniLink has arrived, with exciting
 new features.

machina". We were pretty much out of options. Andrei

Let's say for the sake of discussion that UniLink does work. How close does that put us to having 64 support on Windows, too, if OptLink doesn't need to be translated/rewritten/whatever?
Feb 18 2011
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/18/11 9:42 AM, dsimcha wrote:
 == Quote from Andrei Alexandrescu (SeeWebsiteForEmail erdani.org)'s article
 On 2/18/11 4:00 AM, Dmitry Olshansky wrote:
 The fresh beta of universal linker UniLink has arrived, with exciting
 new features.

machina". We were pretty much out of options. Andrei

Let's say for the sake of discussion that UniLink does work. How close does that put us to having 64 support on Windows, too, if OptLink doesn't need to be translated/rewritten/whatever?

AFAIK the linker is the largest roadblock, but Walter should really answer this question. Andrei
Feb 18 2011
parent reply Daniel Gibson <metalcaedes gmail.com> writes:
Am 18.02.2011 16:41, schrieb Andrei Alexandrescu:
 On 2/18/11 9:42 AM, dsimcha wrote:
 == Quote from Andrei Alexandrescu (SeeWebsiteForEmail erdani.org)'s
 article
 On 2/18/11 4:00 AM, Dmitry Olshansky wrote:
 The fresh beta of universal linker UniLink has arrived, with exciting
 new features.

machina". We were pretty much out of options. Andrei

Let's say for the sake of discussion that UniLink does work. How close does that put us to having 64 support on Windows, too, if OptLink doesn't need to be translated/rewritten/whatever?

AFAIK the linker is the largest roadblock, but Walter should really answer this question. Andrei

I've got this message (and the other one from you in this thread) twice, one sent to the poster you were replying to and one without an explicit to, but both in this news group. Was that intended? Cheers, - Daniel
Feb 18 2011
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/18/11 9:48 AM, Daniel Gibson wrote:
 Am 18.02.2011 16:41, schrieb Andrei Alexandrescu:
 On 2/18/11 9:42 AM, dsimcha wrote:
 == Quote from Andrei Alexandrescu (SeeWebsiteForEmail erdani.org)'s
 article
 On 2/18/11 4:00 AM, Dmitry Olshansky wrote:
 The fresh beta of universal linker UniLink has arrived, with exciting
 new features.

"Deux ex machina". We were pretty much out of options. Andrei

Let's say for the sake of discussion that UniLink does work. How close does that put us to having 64 support on Windows, too, if OptLink doesn't need to be translated/rewritten/whatever?

AFAIK the linker is the largest roadblock, but Walter should really answer this question. Andrei

I've got this message (and the other one from you in this thread) twice, one sent to the poster you were replying to and one without an explicit to, but both in this news group. Was that intended? Cheers, - Daniel

Yes, it's a common mistake I make. Sorry. Andrei
Feb 18 2011
prev sibling next sibling parent reply Trass3r <un known.com> writes:
Sounds fantastic, but what about those rumors that Unilink will become  
commercial sometime?
Feb 18 2011
next sibling parent reply dsimcha <dsimcha yahoo.com> writes:
== Quote from Trass3r (un known.com)'s article
 Sounds fantastic, but what about those rumors that Unilink will become
 commercial sometime?

Well, IMHO the intermediate solution would be to just get DMD to output Win64 OMF object files for now, and cross the linking bridge when we come to it. Unilink is good enough for at least testing that the object files output by DMD are correct, and should be at least better than nothing until Optlink gets rewritten.
Feb 18 2011
parent dsimcha <dsimcha yahoo.com> writes:
On 2/18/2011 2:28 PM, Andrej Mitrovic wrote:
 Is there even such a thing as Win64 OMF?

Ok, maybe there isn't. I just assumed there was. The bottom line is that it makes sense to get DMD to generate the 64-bit object files first so that people with alternative linkers can use it in 64 mode, and then and only then worry about the linker.
Feb 18 2011
prev sibling next sibling parent Trass3r <un known.com> writes:
 Well, IMHO the intermediate solution would be to just get DMD to output  
 Win64 OMF object files for now

Hopefully we won't have to wait too long!
Feb 18 2011
prev sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
Is there even such a thing as Win64 OMF?
Feb 18 2011
prev sibling next sibling parent reply notna <notna.remove.this ist-einmalig.de> writes:
Am I the only one getting a virus warning as soon as I try to unpack the 
ZIP file?

This one is found within the binary:
TR/Crypt.XPACK.Gen

Regards
  notna

On 18.02.2011 11:00, Dmitry Olshansky wrote:
 The fresh beta of universal linker UniLink has arrived, with exciting
 new features.

 Quoting the changelog:
 + Preliminary support of Digital Mars D language compiler.

 + The debug info processing in old CV format added (dmc, dmd, watcom,
 old ml).

 And it works! At least for the first part.
 I actually linked my port of DMDscript and run it through some stress
 tests, no problems so far.
 The nice thing is reduction in half of the resulting binary size.

 As to debug info I think it may need another round of polishing... but
 the devs are willing to support it ;)
 C:\ulink>ulink -v test.obj
 UniLink v1.07 [beta] (build 3.01)
 Error: Invalid debug information in 'test.obj'

 Nevertheless the grim rule of Optlink in windows lands is at end :)

 Some tips on how to get started.

 Grab the latest beta here (for now it's ulnb0301.zip):
 ftp://ftp.styx.cabel.net/pub/UniLink/
 Installation is extract + add to PATH

 Also to save you a bit of investigation, the relevant ulink.cfg config
 for dmd is along the lines:
 -zkernel32;advapi32;user32;wsock32;shell32;snn
 -LC:\dmd2\windows\lib
 -Go

Feb 18 2011
next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 18.02.2011 23:15, notna wrote:
 Am I the only one getting a virus warning as soon as I try to unpack 
 the ZIP file?

 This one is found within the binary:
 TR/Crypt.XPACK.Gen

Ouch, that creepy. My McAfee doesn't detect anything. Being suspicious by nature I run it through http://www.virustotal.com <http://www.virustotal.com/> 1/42 positives which makes my think you have Avira AntiVir. Well, I'm aware that UniLink uses some weird protection techs to prevent revers engineering, and digging through Avira database reveals it's heuristic entry in nature: *TR/Crypt.XPACK.Gen* *Description:* A generic detection routine designed to detect common family characteristics shared in several variants. This special detection routine was developed in order to detect unknown variants and will be enhanced continuously. IMHO it's misfire, but it might be a good idea to send them the file for analysis. -- Dmitry Olshansky
Feb 18 2011
parent notna <notna.remove.this ist-einmalig.de> writes:
Yes, I've Avira AntiVir. I'm quite happy with it since years and it's 
kind od common here in Germany...

Thanks a lot for your help, I'll send them the file and the log...

Btw, also thanks for the tool. It really sounds great!


On 18.02.2011 21:35, Dmitry Olshansky wrote:
 On 18.02.2011 23:15, notna wrote:
 Am I the only one getting a virus warning as soon as I try to unpack
 the ZIP file?

 This one is found within the binary:
 TR/Crypt.XPACK.Gen

Ouch, that creepy. My McAfee doesn't detect anything. Being suspicious by nature I run it through http://www.virustotal.com <http://www.virustotal.com/> 1/42 positives which makes my think you have Avira AntiVir. Well, I'm aware that UniLink uses some weird protection techs to prevent revers engineering, and digging through Avira database reveals it's heuristic entry in nature: *TR/Crypt.XPACK.Gen* *Description:* A generic detection routine designed to detect common family characteristics shared in several variants. This special detection routine was developed in order to detect unknown variants and will be enhanced continuously. IMHO it's misfire, but it might be a good idea to send them the file for analysis.

Feb 18 2011
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 18 Feb 2011 15:35:17 -0500, Dmitry Olshansky  
<dmitry.olsh gmail.com> wrote:

 On 18.02.2011 23:15, notna wrote:
 Am I the only one getting a virus warning as soon as I try to unpack  
 the ZIP file?

 This one is found within the binary:
 TR/Crypt.XPACK.Gen

Ouch, that creepy. My McAfee doesn't detect anything. Being suspicious by nature I run it through http://www.virustotal.com <http://www.virustotal.com/> 1/42 positives which makes my think you have Avira AntiVir. Well, I'm aware that UniLink uses some weird protection techs to prevent revers engineering, and digging through Avira database reveals it's heuristic entry in nature: *TR/Crypt.XPACK.Gen* *Description:* A generic detection routine designed to detect common family characteristics shared in several variants. This special detection routine was developed in order to detect unknown variants and will be enhanced continuously. IMHO it's misfire, but it might be a good idea to send them the file for analysis.

Yeah, I dropped Avira in favor of Avast because they have some big problems, kind of like this. -Steve
Feb 18 2011
prev sibling parent Trass3r <un known.com> writes:
 Am I the only one getting a virus warning as soon as I try to unpack the  
 ZIP file?

 This one is found within the binary:
 TR/Crypt.XPACK.Gen

Yep, Antivir is pretty paranoid and fires up even if an exe is only compressed.
Feb 18 2011
prev sibling parent reply Sebastian Schuberth <sschuberth gmail.com> writes:
On 18.02.2011 11:00, Dmitry Olshansky wrote:

 The nice thing is reduction in half of the resulting binary size.

That's indeed nice! The unnecessarily huge size of binaries created with D / Optlink was in fact something hindering me to use D at all! -- Sebastian Schuberth
Feb 24 2011
parent reply Walter Bright <newshound2 digitalmars.com> writes:
Sebastian Schuberth wrote:
 On 18.02.2011 11:00, Dmitry Olshansky wrote:
 
 The nice thing is reduction in half of the resulting binary size.

That's indeed nice! The unnecessarily huge size of binaries created with D / Optlink was in fact something hindering me to use D at all!

I'm sure that linker is doing it by writing compressed exe's. This means that it has the same memory footprint, and it loads slower because it must be decompressed. Also, if you store it in a zip file, the zip file won't be any smaller because recompressing compressed data doesn't make it smaller.
Feb 24 2011
next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 24.02.2011 21:41, Walter Bright wrote:
 Sebastian Schuberth wrote:
 On 18.02.2011 11:00, Dmitry Olshansky wrote:

 The nice thing is reduction in half of the resulting binary size.

That's indeed nice! The unnecessarily huge size of binaries created with D / Optlink was in fact something hindering me to use D at all!

I'm sure that linker is doing it by writing compressed exe's. This means that it has the same memory footprint, and it loads slower because it must be decompressed. Also, if you store it in a zip file, the zip file won't be any smaller because recompressing compressed data doesn't make it smaller.

Tried that, it's still compresses to roughly 50% of it's size. Well, I no expert but a glance at disassembly shows that at least strings / imports are intact. For what it's worth the IDA control flow analysis shows a bulk of unreferenced procedures in optlink's one. So maybe the case is that they also included a decent stripper. -- Dmitry Olshansky
Feb 24 2011
parent Walter Bright <newshound2 digitalmars.com> writes:
Dmitry Olshansky wrote:
 On 24.02.2011 21:41, Walter Bright wrote:
 Sebastian Schuberth wrote:
 On 18.02.2011 11:00, Dmitry Olshansky wrote:

 The nice thing is reduction in half of the resulting binary size.

That's indeed nice! The unnecessarily huge size of binaries created with D / Optlink was in fact something hindering me to use D at all!

I'm sure that linker is doing it by writing compressed exe's. This means that it has the same memory footprint, and it loads slower because it must be decompressed. Also, if you store it in a zip file, the zip file won't be any smaller because recompressing compressed data doesn't make it smaller.

Tried that, it's still compresses to roughly 50% of it's size. Well, I no expert but a glance at disassembly shows that at least strings / imports are intact. For what it's worth the IDA control flow analysis shows a bulk of unreferenced procedures in optlink's one. So maybe the case is that they also included a decent stripper.

If that's so, that's good!
Feb 24 2011
prev sibling parent reply Sebastian Schuberth <sschuberth gmail.com> writes:
On 24.02.2011 19:41, Walter Bright wrote:

 The nice thing is reduction in half of the resulting binary size.

That's indeed nice! The unnecessarily huge size of binaries created with D / Optlink was in fact something hindering me to use D at all!

I'm sure that linker is doing it by writing compressed exe's. This means that it has the same memory footprint, and it loads slower because it must be decompressed. Also, if you store it in a zip file, the zip file

IMHO, that is a common misbelief when it comes to executable compressors. AFAIK, the time required for decompression is overcompensated by the time required to read less data from disk, even still nowadays.
 won't be any smaller because recompressing compressed data doesn't make
 it smaller.

There really needs to be no compression or back magic involved to make the executable size for a simple program like ---8<--- import std.stdio; void main(string[] args) { writeln("Hello World, Reloaded"); } ---8<--- smaller than the current 286 KiB! Some dead code elemination would already do, I guess. -- Sebastian Schuberth
Feb 26 2011
next sibling parent Michel Fortin <michel.fortin michelf.com> writes:
On 2011-02-26 06:10:19 -0500, Sebastian Schuberth <sschuberth gmail.com> said:

 On 24.02.2011 19:41, Walter Bright wrote:
 
 The nice thing is reduction in half of the resulting binary size.

That's indeed nice! The unnecessarily huge size of binaries created with D / Optlink was in fact something hindering me to use D at all!

I'm sure that linker is doing it by writing compressed exe's. This means that it has the same memory footprint, and it loads slower because it must be decompressed. Also, if you store it in a zip file, the zip file

IMHO, that is a common misbelief when it comes to executable compressors. AFAIK, the time required for decompression is overcompensated by the time required to read less data from disk, even still nowadays.

True in general. But in the case of an executable file, how does it interacts with memory mapping in the virtual memory subsystem? If you need to actually load and decompress the whole executable upfront instead of relying on the virtual memory subsystem to load pages from disk as needed, it might very well increase your startup time. Also, if some parts of the executable needs to be paged out to disk because free physical memory is running low, this is almost free if the memory is directly mapped to a read-only file as the data is already on disk. So there's certainly some scenarios where a compressed executable will fare better, but there's surely plenty others where it's the reverse. -- Michel Fortin michel.fortin michelf.com http://michelf.com/
Feb 26 2011
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2011-02-26 12:10, Sebastian Schuberth wrote:
 On 24.02.2011 19:41, Walter Bright wrote:

 The nice thing is reduction in half of the resulting binary size.

That's indeed nice! The unnecessarily huge size of binaries created with D / Optlink was in fact something hindering me to use D at all!

I'm sure that linker is doing it by writing compressed exe's. This means that it has the same memory footprint, and it loads slower because it must be decompressed. Also, if you store it in a zip file, the zip file

IMHO, that is a common misbelief when it comes to executable compressors. AFAIK, the time required for decompression is overcompensated by the time required to read less data from disk, even still nowadays.
 won't be any smaller because recompressing compressed data doesn't make
 it smaller.

There really needs to be no compression or back magic involved to make the executable size for a simple program like ---8<--- import std.stdio; void main(string[] args) { writeln("Hello World, Reloaded"); } ---8<--- smaller than the current 286 KiB! Some dead code elemination would already do, I guess.

On Mac OS X linking a Hello World application dynamically to Tango results in a 16Kb executable, the same size as for a Hello World written in C. -- /Jacob Carlborg
Feb 27 2011