www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Optlink is on github

reply Walter Bright <newshound2 digitalmars.com> writes:
Some months ago, I did make the source to optlink available on github:

https://github.com/DigitalMars/optlink

Rainer Schuetze has improved it where it can be built with modern tools (the 
older tools would not run on Win7). I know some of you are frustrated by
optlink 
problems. Well, now you can do something about it!

Note that the optlink source is quite a challenge to work on. It's very old 
skool (80's style).

Here's an approach that I've used successfully to fix optlink seg faults:

1. Find out where in the source code it is. This is not so easy, even if using
a 
debugger. The trouble is the asm functions do not have standard stack frames,
so 
the debugger cannot do a stack trace. Even where there is a standard stack 
frame, the modern Microsoft debuggers fail to recognize them. You gotta use an 
older debugger under Windows XP.

So, what I do, is look at the asm code where the seg fault occurs, and then
grep 
through the asm source till I find it.

2. Convert the source file where the seg fault occurs to C. I do this one 
function at a time, running the full set of tests at each step. Otherwise, it's 
just impossible to figure out where you made a mistake in the conversion. 
Sometimes, you gotta go even finer grained - using the inline assembler feature 
of dmc to convert asm code line by line.

You gotta pay very, very close attention to which registers have parameters 
passed through them, which registers are saved, and which registers have 
parameters that are silently passed through a function to callees of that 
function (!). This information is all pretty much utterly lacking in the 
comments, and the comments are often dead wrong as they refer to much older 
versions of the code.

3. Once it is in C, things get a lot easier. You can, for example, insert 
printf's to figure out where things go wrong, and *then* fix it.


The full test suite for optlink has a lot of stuff I cannot publish on github. 
However, what you can do is run the win32 tests for dmd. I, of course, will run 
the full one when verifying pulls.

Happy hacking!
Mar 06 2013
next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Walter Bright:


 Happy hacking!


Extra karma points if done blindfolded, using a Braille tablet :-)

Bye,
bearophile
Mar 06 2013
parent "Alex Ogheri" <ogheri alessandroogheri.com> writes:
On Thursday, 7 March 2013 at 03:06:48 UTC, bearophile wrote:

 Walter Bright:


 Happy hacking!


 Extra karma points if done blindfolded, using a Braille tablet 
 :-)

 Bye,
 bearophile


As far as I know, Walter DOES IT blindfolded, using a Braille 
tablet and while he writes 3 other compilers at the same time...

(but the same has been said of Linus Torvalds, or Richard 
Stallman, so perhaps its a myth...)
Apr 09 2014
prev sibling next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 7 March 2013 at 01:25:02 UTC, Walter Bright wrote:

 Some months ago, I did make the source to optlink available on 
 github:

 https://github.com/DigitalMars/optlink

 Rainer Schuetze has improved it where it can be built with 
 modern tools (the older tools would not run on Win7). I know 
 some of you are frustrated by optlink problems. Well, now you 
 can do something about it!

 Note that the optlink source is quite a challenge to work on. 
 It's very old skool (80's style).

 Here's an approach that I've used successfully to fix optlink 
 seg faults:

 1. Find out where in the source code it is. This is not so 
 easy, even if using a debugger. The trouble is the asm 
 functions do not have standard stack frames, so the debugger 
 cannot do a stack trace. Even where there is a standard stack 
 frame, the modern Microsoft debuggers fail to recognize them. 
 You gotta use an older debugger under Windows XP.

 So, what I do, is look at the asm code where the seg fault 
 occurs, and then grep through the asm source till I find it.

 2. Convert the source file where the seg fault occurs to C. I 
 do this one function at a time, running the full set of tests 
 at each step. Otherwise, it's just impossible to figure out 
 where you made a mistake in the conversion. Sometimes, you 
 gotta go even finer grained - using the inline assembler 
 feature of dmc to convert asm code line by line.

 You gotta pay very, very close attention to which registers 
 have parameters passed through them, which registers are saved, 
 and which registers have parameters that are silently passed 
 through a function to callees of that function (!). This 
 information is all pretty much utterly lacking in the comments, 
 and the comments are often dead wrong as they refer to much 
 older versions of the code.

 3. Once it is in C, things get a lot easier. You can, for 
 example, insert printf's to figure out where things go wrong, 
 and *then* fix it.


 The full test suite for optlink has a lot of stuff I cannot 
 publish on github. However, what you can do is run the win32 
 tests for dmd. I, of course, will run the full one when 
 verifying pulls.

 Happy hacking!


It sound like a nightmare. What is the reason in the first place 
to use optlink ? Isn't it possible to use another linker ?
Mar 06 2013
parent =?iso-8859-15?Q?Simen_Kj=E6r=E5s?= <simen.kjaras gmail.com> writes:
On Thu, 07 Mar 2013 06:04:29 +0100, deadalnix <deadalnix gmail.com> wrote:


 It sound like a nightmare. What is the reason in the first place to use  
 optlink ? Isn't it possible to use another linker ?


It's fast, it's pretty well tested, Walter's written it himself, and
changing to a different linker is likely to have its own set of challenges.

-- 
Simen
Mar 06 2013
prev sibling next sibling parent reply "Lars T. Kyllingstad" <public kyllingen.net> writes:
On Thursday, 7 March 2013 at 01:25:02 UTC, Walter Bright wrote:

 Some months ago, I did make the source to optlink available on 
 github:

 [...]

 Happy hacking!


This reminds me of 
http://www.catb.org/jargon/html/story-of-mel.html

:)

Lars
Mar 06 2013
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Mar 07, 2013 at 07:09:22AM +0100, Lars T. Kyllingstad wrote:

 On Thursday, 7 March 2013 at 01:25:02 UTC, Walter Bright wrote:

Some months ago, I did make the source to optlink available on
github:

[...]

Happy hacking!


 
 This reminds me of http://www.catb.org/jargon/html/story-of-mel.html


[...]

Man, that brings back the memories. I remember doing those kinds of
stunts when coding for the good ole Motorola 6502. Back in those days,
Apple actually published the full listing of their ROM code along with
the instruction specs, including the number of cycles per instruction,
etc.. There were many a fun night plotting out the optimal instruction
sequences and spacing out instructions just right so the exact timing is
achieved. Lots of fun.


T

-- 
What doesn't kill me makes me stranger.
Mar 06 2013
next sibling parent reply Marco Leise <Marco.Leise gmx.de> writes:
On Thu, Mar 07, 2013 at 07:09:22AM +0100, Lars T. Kyllingstad
wrote:


 This reminds me of http://www.catb.org/jargon/html/story-of-mel.html


Good story! Imagine how Mel must have known the entire program
by heart and he never got any change requests that are so common today.
I would have been addicted to that, too. But luckily 1975 was 8
years before my birth and so I started "hacking" with QBasic
Gorillaz, a turn based strategy game.
I guess with SSE it still today makes sense to count
instruction delays and throughput. Also take a look at this:
http://research.scee.net/files/presentations/gcapaustralia09/Pitfalls_of_Object_Oriented_Programming_GCAP_09.pdf
I don't think people like this programmer Mel are job-less
today.

-- 
Marco
Mar 07 2013
parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Thursday, 7 March 2013 at 14:42:25 UTC, Marco Leise wrote:

 On Thu, Mar 07, 2013 at 07:09:22AM +0100, Lars T. Kyllingstad
 wrote:


 This reminds me of 
 http://www.catb.org/jargon/html/story-of-mel.html


 Good story! Imagine how Mel must have known the entire program
 by heart and he never got any change requests that are so 
 common today.
 I would have been addicted to that, too. But luckily 1975 was 8
 years before my birth and so I started "hacking" with QBasic
 Gorillaz, a turn based strategy game.



 I guess with SSE it still today makes sense to count
 instruction delays and throughput. Also take a look at this:
 http://research.scee.net/files/presentations/gcapaustralia09/Pitfalls_of_Object_Oriented_Programming_GCAP_09.pdf
 I don't think people like this programmer Mel are job-less
 today.


True, although I think this type of work is very complicated to 
do nowadays given how modern architectures work.

Only in simple CPU architectures it is possible to know the 
effect each instruction still has, whereas with modern ones you 
have to use some kind of profiler/code analyzer even for Assembly 
code, given the multiple level caches, parallel execution of 
instructions, instruction re-ordering and so on.

--
Paulo
Mar 14 2013
prev sibling parent "Ralf" <r.a.quint gmail.com> writes:
On Thursday, 7 March 2013 at 06:34:05 UTC, H. S. Teoh wrote:

 Man, that brings back the memories. I remember doing those 
 kinds of
 stunts when coding for the good ole Motorola 6502. Back in 
 those days,


Well, there never was a 'Motorola' 6502, in fact the manufacturer 
MOS Technologies (though founded by former Motorola employees) 
was a competitor to Motorola (which had released the 6800, which 
was similar but 10x as expensive) and was sued by Motorola at 
some point for patent infringements just before being bought up 
by Commodore...

Ralf
Mar 13 2013
prev sibling next sibling parent reply "Moritz Maxeiner" <moritz ucworks.org> writes:
On Thursday, 7 March 2013 at 01:25:02 UTC, Walter Bright wrote:

 https://github.com/DigitalMars/optlink

 Rainer Schuetze has improved it where it can be built with 
 modern tools (the older tools would not run on Win7). I know 
 some of you are frustrated by optlink problems. Well, now you 
 can do something about it!


Sorry if this has been answered before/is common knowledge, but 
is porting functions at a time to C wanted for optlink in 
general, or only
for finding segfaults? (e.g. are pull-requests for that welcome)
Mar 07 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-03-07 11:12, Moritz Maxeiner wrote:


 Sorry if this has been answered before/is common knowledge, but is
 porting functions at a time to C wanted for optlink in general, or only
 for finding segfaults? (e.g. are pull-requests for that welcome)


Yes, in general. I think the idea is to port the whole Optlink to C and 
then to D. It's easier to port from C to D than from assembly to D. This 
is because you can use a C version that doesn't use the runtime or 
standard library to get the generate assembly as close as possible to 
the original one. I think it was something like that.

-- 
/Jacob Carlborg
Mar 07 2013
parent reply Denis Shelomovskij <verylonglogin.reg gmail.com> writes:
07.03.2013 14:28, Jacob Carlborg пишет:

 On 2013-03-07 11:12, Moritz Maxeiner wrote:


 Sorry if this has been answered before/is common knowledge, but is
 porting functions at a time to C wanted for optlink in general, or only
 for finding segfaults? (e.g. are pull-requests for that welcome)


 Yes, in general. I think the idea is to port the whole Optlink to C and
 then to D. It's easier to port from C to D than from assembly to D. This
 is because you can use a C version that doesn't use the runtime or
 standard library to get the generate assembly as close as possible to
 the original one. I think it was something like that.


Didn't get. You don't have to use D with druntime. Just don't link it 
and everything will be OK - you will just get "better C" (i.e. with D 
structs and other good stuff).

-- 
Денис В. Шеломовский
Denis V. Shelomovskij
Mar 07 2013
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-03-07 16:53, Denis Shelomovskij wrote:


 Didn't get. You don't have to use D with druntime. Just don't link it
 and everything will be OK - you will just get "better C" (i.e. with D
 structs and other good stuff).


I was referring to using C with out the C runtime or C standard library. 
Walter talked about this is some thread. He basically reimplemented printf.

-- 
/Jacob Carlborg
Mar 07 2013
prev sibling next sibling parent reply "Jesse Phillips" <Jesse.K.Phillips+D gmail.com> writes:
On Thursday, 7 March 2013 at 15:53:09 UTC, Denis Shelomovskij 
wrote:

 Didn't get. You don't have to use D with druntime. Just don't 
 link it and everything will be OK - you will just get "better 
 C" (i.e. with D structs and other good stuff).


Walter wrote all about it: 
http://www.drdobbs.com/architecture-and-design/assembler-to-c/228701319
Mar 08 2013
parent "Asman01" <jckj33 gmail.com> writes:
On Saturday, 9 March 2013 at 05:22:31 UTC, Jesse Phillips wrote:

 On Thursday, 7 March 2013 at 15:53:09 UTC, Denis Shelomovskij 
 wrote:

 Didn't get. You don't have to use D with druntime. Just don't 
 link it and everything will be OK - you will just get "better 
 C" (i.e. with D structs and other good stuff).


 Walter wrote all about it: 
 http://www.drdobbs.com/architecture-and-design/assembler-to-c/228701319


Why isn't this link 
here(http://www.drdobbs.com/author/Walter-Bright) ? are there 
more Walter's articles not listed in this page? if so, I want to 
know. Please :)
Mar 23 2014
prev sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Thu, 07 Mar 2013 19:53:29 +0400
schrieb Denis Shelomovskij <verylonglogin.reg gmail.com>:


 07.03.2013 14:28, Jacob Carlborg =D0=BF=D0=B8=D1=88=D0=B5=D1=82:

 On 2013-03-07 11:12, Moritz Maxeiner wrote:


 Sorry if this has been answered before/is common knowledge, but is
 porting functions at a time to C wanted for optlink in general, or
 only for finding segfaults? (e.g. are pull-requests for that
 welcome)


 Yes, in general. I think the idea is to port the whole Optlink to C
 and then to D. It's easier to port from C to D than from assembly
 to D. This is because you can use a C version that doesn't use the
 runtime or standard library to get the generate assembly as close
 as possible to the original one. I think it was something like that.


=20
 Didn't get. You don't have to use D with druntime. Just don't link it=20
 and everything will be OK - you will just get "better C" (i.e. with D=20
 structs and other good stuff).
=20


That's a little bit oversimplified. Even a simple POD struct references
the TypeInfo_Struct declaration in druntime. There's still some
compiler work needed to really make D usable without druntime. (Think
of issues like these: If you don't have druntime -> you don't have
typeinfo -> no D style varargs -> no associative arrays; you can't even
compare normal arrays (https://github.com/D-Programming-Language/druntime/b=
lob/e47a00bff935c3f079bb567a6ec97663ba384487/src/rt/adi.d#L368))
Mar 09 2013
parent reply "Dicebot" <m.strashun gmail.com> writes:
On Saturday, 9 March 2013 at 09:11:54 UTC, Johannes Pfau wrote:

 Am Thu, 07 Mar 2013 19:53:29 +0400
 schrieb Denis Shelomovskij <verylonglogin.reg gmail.com>:


 07.03.2013 14:28, Jacob Carlborg пишет:

 On 2013-03-07 11:12, Moritz Maxeiner wrote:


 Sorry if this has been answered before/is common knowledge, 
 but is
 porting functions at a time to C wanted for optlink in 
 general, or
 only for finding segfaults? (e.g. are pull-requests for that
 welcome)


 Yes, in general. I think the idea is to port the whole 
 Optlink to C
 and then to D. It's easier to port from C to D than from 
 assembly
 to D. This is because you can use a C version that doesn't 
 use the
 runtime or standard library to get the generate assembly as 
 close
 as possible to the original one. I think it was something 
 like that.


 
 Didn't get. You don't have to use D with druntime. Just don't 
 link it and everything will be OK - you will just get "better 
 C" (i.e. with D structs and other good stuff).
 


 That's a little bit oversimplified. Even a simple POD struct 
 references
 the TypeInfo_Struct declaration in druntime. There's still some
 compiler work needed to really make D usable without druntime. 
 (Think
 of issues like these: If you don't have druntime -> you don't 
 have
 typeinfo -> no D style varargs -> no associative arrays; you 
 can't even
 compare normal arrays 
 (https://github.com/D-Programming-Language/druntime/blob/e47a00bff935c3f079bb567a6ec97663ba384487/src/rt/adi.d#L368))


It can be easily checked with "-betterC -defaultlib=" combo. You 
can skip phobos, druntime, use only C stdlib and look what will 
remain working in the language. Not much.
Mar 09 2013
parent reply Johannes Pfau <nospam example.com> writes:
Am Sat, 09 Mar 2013 10:32:18 +0100
schrieb "Dicebot" <m.strashun gmail.com>:


 On Saturday, 9 March 2013 at 09:11:54 UTC, Johannes Pfau wrote:

 Am Thu, 07 Mar 2013 19:53:29 +0400
 schrieb Denis Shelomovskij <verylonglogin.reg gmail.com>:


 07.03.2013 14:28, Jacob Carlborg =D0=BF=D0=B8=D1=88=D0=B5=D1=82:

 On 2013-03-07 11:12, Moritz Maxeiner wrote:


 Sorry if this has been answered before/is common knowledge,=20
 but is
 porting functions at a time to C wanted for optlink in=20
 general, or
 only for finding segfaults? (e.g. are pull-requests for that
 welcome)


 Yes, in general. I think the idea is to port the whole=20
 Optlink to C
 and then to D. It's easier to port from C to D than from=20
 assembly
 to D. This is because you can use a C version that doesn't=20
 use the
 runtime or standard library to get the generate assembly as=20
 close
 as possible to the original one. I think it was something=20
 like that.


=20
 Didn't get. You don't have to use D with druntime. Just don't=20
 link it and everything will be OK - you will just get "better=20
 C" (i.e. with D structs and other good stuff).
=20


 That's a little bit oversimplified. Even a simple POD struct=20
 references
 the TypeInfo_Struct declaration in druntime. There's still some
 compiler work needed to really make D usable without druntime.=20
 (Think
 of issues like these: If you don't have druntime -> you don't=20
 have
 typeinfo -> no D style varargs -> no associative arrays; you=20
 can't even
 compare normal arrays=20
 (https://github.com/D-Programming-Language/druntime/blob/e47a00bff935c3=




f079bb567a6ec97663ba384487/src/rt/adi.d#L368))

=20
 It can be easily checked with "-betterC -defaultlib=3D" combo. You=20
 can skip phobos, druntime, use only C stdlib and look what will=20
 remain working in the language. Not much.


I know that, I've run D code on the DS without druntime and on
Android as well, but betterC only disables moduleinfo, not typeinfo. So
you don't even get structs without druntime and that is certainly not
better than C.

test.d:
-------
struct A
{
}
-------

dmd -betterC -defaultlib=3D -c test.d
nm test.o
U _D15TypeInfo_Struct6__vtblZ
Mar 09 2013
parent reply "Dicebot" <m.strashun gmail.com> writes:
Ye, sure, I wanted to support your point with practical example, 
not object it by any means. By the way, templates and array 
literals I have tried were doomed, too :)
Mar 09 2013
next sibling parent Johannes Pfau <nospam example.com> writes:
Am Sat, 09 Mar 2013 11:35:42 +0100
schrieb "Dicebot" <m.strashun gmail.com>:


 Ye, sure, I wanted to support your point with practical example, 
 not object it by any means. By the way, templates and array 
 literals I have tried were doomed, too :)


Really, templates also don't work? There's a lot of work required till
betterC is finished...
Mar 09 2013
prev sibling parent Johannes Pfau <nospam example.com> writes:
Am Sat, 09 Mar 2013 11:35:42 +0100
schrieb "Dicebot" <m.strashun gmail.com>:


 Ye, sure, I wanted to support your point with practical example, 
 not object it by any means. By the way, templates and array 
 literals I have tried were doomed, too :)


Sorry, I misunderstood that then.
Mar 09 2013
prev sibling next sibling parent reply "Daniel Murphy" <yebblies nospamgmail.com> writes:
"Walter Bright" <newshound2 digitalmars.com> wrote in message 
news:kh8q9e$2j02$1 digitalmars.com...

 Some months ago, I did make the source to optlink available on github:

 https://github.com/DigitalMars/optlink


[snip]

 So, what I do, is look at the asm code where the seg fault occurs, and 
 then grep through the asm source till I find it.


Maybe I'm missing something, but I don't understand why this is necessary. 
It should be trivial to look at the map file for the closest preceding 
label.  On modern toolchains it should even be possible to compile the asm 
with debug line information.  Or is grepping more effective than it sounds?


 2. Convert the source file where the seg fault occurs to C. I do this one 
 function at a time, running the full set of tests at each step. Otherwise, 
 it's just impossible to figure out where you made a mistake in the 
 conversion. Sometimes, you gotta go even finer grained - using the inline 
 assembler feature of dmc to convert asm code line by line.

 You gotta pay very, very close attention to which registers have 
 parameters passed through them, which registers are saved, and which 
 registers have parameters that are silently passed through a function to 
 callees of that function (!). This information is all pretty much utterly 
 lacking in the comments, and the comments are often dead wrong as they 
 refer to much older versions of the code.


This sounds completely insane. (But also fun, in an extremely frustrating 
way)
I suspect it would be easier to reverse-engineer microsoft's (or borland's?) 
old linker from the disassembly, if they wrote it in C.


 3. Once it is in C, things get a lot easier. You can, for example, insert 
 printf's to figure out where things go wrong, and *then* fix it.


 The full test suite for optlink has a lot of stuff I cannot publish on 
 github. However, what you can do is run the win32 tests for dmd. I, of 
 course, will run the full one when verifying pulls.

 Happy hacking!


As a side note I actually wrote an omf (dmd-emitted subset) linker last year 
that gets through almost all of the dmd test suite.  It has no debug info 
support, which sucks, but it may work in some cases where optlink can't 
cope.  It is fairly naive performance-wise, but is written in a much 
friendlier language.

If anyone is interested I'll put it up on github.
Mar 07 2013
next sibling parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Thu, 7 Mar 2013 23:35:52 +1100
schrieb "Daniel Murphy" <yebblies nospamgmail.com>:


 As a side note I actually wrote an omf (dmd-emitted subset) linker last year 
 that gets through almost all of the dmd test suite.  It has no debug info 
 support, which sucks, but it may work in some cases where optlink can't 
 cope.  It is fairly naive performance-wise, but is written in a much 
 friendlier language.


I wonder what would happen if someone plugged that linker
right into the dmd D port to skip the external linking phase
alltogether...

-- 
Marco
Mar 07 2013
parent "Daniel Murphy" <yebblies nospamgmail.com> writes:
"Marco Leise" <Marco.Leise gmx.de> wrote in message 
news:20130307154744.206d9bb0 marco-leise...

 Am Thu, 7 Mar 2013 23:35:52 +1100
 schrieb "Daniel Murphy" <yebblies nospamgmail.com>:


 As a side note I actually wrote an omf (dmd-emitted subset) linker last 
 year
 that gets through almost all of the dmd test suite.  It has no debug info
 support, which sucks, but it may work in some cases where optlink can't
 cope.  It is fairly naive performance-wise, but is written in a much
 friendlier language.


 I wonder what would happen if someone plugged that linker
 right into the dmd D port to skip the external linking phase
 alltogether...

 -- 
 Marco


Performance++
Memory_Usage++

Even optlink has/once had a library version.

llvm's lld is an interesting project, and adding an omf reader could solve 
our linker problems once it's more mature.

Linkers are fun, corrupt executables are really in their own special class 
when it comes to nasty bugs.
Mar 07 2013
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 3/7/2013 4:35 AM, Daniel Murphy wrote:

 Maybe I'm missing something, but I don't understand why this is necessary.
 It should be trivial to look at the map file for the closest preceding
 label.  On modern toolchains it should even be possible to compile the asm
 with debug line information.  Or is grepping more effective than it sounds?


Modern operating systems load programs at different base addresses each time,
so 
the addresses in the map file don't match up with the addresses at run time.

The older assembler I was using didn't emit symbolic debug info. I haven't
tried 
out ML that way.



 This sounds completely insane. (But also fun, in an extremely frustrating
 way)
 I suspect it would be easier to reverse-engineer microsoft's (or borland's?)
 old linker from the disassembly, if they wrote it in C.


You'd be incorrect if you thought MS's or Borland's linker was better. Optlink 
sold very, very well against those linkers back in the day, because Optlink was 
far better.



 As a side note I actually wrote an omf (dmd-emitted subset) linker last year
 that gets through almost all of the dmd test suite.  It has no debug info
 support, which sucks, but it may work in some cases where optlink can't
 cope.  It is fairly naive performance-wise, but is written in a much
 friendlier language.

 If anyone is interested I'll put it up on github.


There's much more to making a linker. There's the debug info (which you 
mentioned), then there are the resource files, the module definition files, and 
lastly the myriad of rather complex switches that are expected. Almost none of 
that is tested by the dmd test suite.
Mar 07 2013
next sibling parent reply "Andrej Mitrovic" <andrej.mitrovich gmail.com> writes:
On Thursday, 7 March 2013 at 19:26:49 UTC, Walter Bright wrote:

 If anyone is interested I'll put it up on github.


 There's much more to making a linker. There's the debug info 
 (which you mentioned), then there are the resource files, the 
 module definition files, and lastly the myriad of rather 
 complex switches that are expected. Almost none of that is 
 tested by the dmd test suite.


Nevertheless having a linker written in D goes hand-in-hand with 
having a compiler written in D. I'd be interested in seeing his 
work. I'd also be interested in having something that's 
documented and that isn't bound by licensing issues.

We also have the abandoned DDL[1] project which had a linker that 
supports OMF, and Eric Anderton was nice enough to document his 
process of implementing it: 
http://www.dsource.org/forums/viewtopic.php?t=959

[1] http://dsource.org/projects/ddl/
Mar 07 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 3/7/2013 11:32 AM, Andrej Mitrovic wrote:

 On Thursday, 7 March 2013 at 19:26:49 UTC, Walter Bright wrote:

 If anyone is interested I'll put it up on github.


 There's much more to making a linker. There's the debug info (which you
 mentioned), then there are the resource files, the module definition files,
 and lastly the myriad of rather complex switches that are expected. Almost
 none of that is tested by the dmd test suite.


 Nevertheless having a linker written in D goes hand-in-hand with having a
 compiler written in D. I'd be interested in seeing his work. I'd also be
 interested in having something that's documented and that isn't bound by
 licensing issues.

 We also have the abandoned DDL[1] project which had a linker that supports OMF,
 and Eric Anderton was nice enough to document his process of implementing it:
 http://www.dsource.org/forums/viewtopic.php?t=959

 [1] http://dsource.org/projects/ddl/


Sure, having a complete linker written in D would be preferable. I also began a 
project to write such. However, getting such to be a complete, viable 
replacement for optlink will require a determined, sustained effort.

I also believe that even if Optlink were completely converted to C, it will 
still be intractably difficult to refactor it to do things like support other 
object file formats. This is why I had started a new linker project.

However, Optlink does bring considerable value to any linker replacement 
project, in that embedded in it is an enormous amount of lore about all those 
twinkie little things that matter, and how things really need to work.

And, of course, consider that we don't have a thorough test suite for a linker. 
This is a serious problem. Trying to make symbolic debug info work without such 
is a big issue. Heck, I spent quite a bit of time getting dmd to generate 
correct CV8 info, given that what MS's debugger accepts does not always match 
the (sparse and confusing) documentation.

What we *can* do with Optlink is fix the occasional bug that crops up in it.
Mar 07 2013
parent reply "Andrej Mitrovic" <andrej.mitrovich gmail.com> writes:
On Thursday, 7 March 2013 at 20:01:58 UTC, Walter Bright wrote:

 What we *can* do with Optlink is fix the occasional bug that 
 crops up in it.


I hope so. :)

On Thursday, 7 March 2013 at 20:01:58 UTC, Walter Bright wrote:

 I also believe that even if Optlink were completely converted 
 to C, it will still be intractably difficult to refactor it to 
 do things like support other object file formats.


I really wish Unilink[1] was open-source. It's apparently a 
student project, and it's closed source but freeware. It works 
with OMF but can also link OMF and COFF objects together, which 
is a boon for win32 programmers who don't want to go through 
OBJCONV (object file conversion, doesn't always work) or COFF2OMF 
(import libs). It also has pretty good error messages.

Of course there's no telling how big its test-suite is. It's 
fairly easy to set up though[2].

[1] ftp://ftp.styx.cabel.net/pub/UniLink/
[2] 
http://forum.dlang.org/thread/jbvugp$8l3$1 digitalmars.com#post-mailman.1414.1323538292.24802.digitalmars-d:40puremagic.com

On Thursday, 7 March 2013 at 20:01:58 UTC, Walter Bright wrote:

 This is why I had started a new linker project.


Interesting. But if it's made behind-the-scenes with no input or 
work from the community I won't be holding my hopes up too much, 
I know you have very limited time for side-projects like that.
Mar 07 2013
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 3/7/2013 12:15 PM, Andrej Mitrovic wrote:

 Interesting. But if it's made behind-the-scenes with no input or work from the
 community I won't be holding my hopes up too much, I know you have very limited
 time for side-projects like that.


It didn't get much past the conceptual part. I wanted to make a linker where:

1. all the supported file formats were completely encapsulated and modularized

For some idea of how that can work, see libmscoff.c and scanmscoff.c, where the 
object file format and library file format is completely separated.

2. used ranges to their fullest extend
Mar 07 2013
parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 3/7/13, Walter Bright <newshound2 digitalmars.com> wrote:

 1. all the supported file formats were completely encapsulated and
 modularized
 For some idea of how that can work, see libmscoff.c and scanmscoff.c


DDL seems to do a decent job of abstracting things, e.g.:

OMF loader: http://dsource.org/projects/ddl/browser/trunk/ddl/omf
COFF loader: http://dsource.org/projects/ddl/browser/trunk/ddl/coff
And the root dir: http://dsource.org/projects/ddl/browser/trunk/ddl

Anyway I'm not suggesting it's a complete implementation, but it might
be worth taking a look. It's BSD licensed, hopefully that means it's
ok to look at and maybe extract some useful code or ideas from it.

From the source comments it appears Don Clugston also worked on this


for some time.
Mar 07 2013
prev sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
08-Mar-2013 00:15, Andrej Mitrovic пишет:

 On Thursday, 7 March 2013 at 20:01:58 UTC, Walter Bright wrote:

 What we *can* do with Optlink is fix the occasional bug that crops up
 in it.


 I hope so. :)

 On Thursday, 7 March 2013 at 20:01:58 UTC, Walter Bright wrote:

 I also believe that even if Optlink were completely converted to C, it
 will still be intractably difficult to refactor it to do things like
 support other object file formats.


 I really wish Unilink[1] was open-source.


Same here. Maybe we should try and convince him to share it with some 
OpenSource license?


 It's apparently a student
 project, and it's closed source but freeware. It works with OMF but can
 also link OMF and COFF objects together, which is a boon for win32
 programmers who don't want to go through OBJCONV (object file
 conversion, doesn't always work) or COFF2OMF (import libs). It also has
 pretty good error messages.

 Of course there's no telling how big its test-suite is. It's fairly easy
 to set up though[2].

 [1] ftp://ftp.styx.cabel.net/pub/UniLink/
 [2]
 http://forum.dlang.org/thread/jbvugp$8l3$1 digitalmars.com#post-mailman.1414.1323538292.24802.digitalmars-d:40puremagic.com


 On Thursday, 7 March 2013 at 20:01:58 UTC, Walter Bright wrote:

 This is why I had started a new linker project.


 Interesting. But if it's made behind-the-scenes with no input or work
 from the community I won't be holding my hopes up too much, I know you
 have very limited time for side-projects like that.



-- 
Dmitry Olshansky
Mar 08 2013
prev sibling parent reply "Daniel Murphy" <yebblies nospamgmail.com> writes:
"Walter Bright" <newshound2 digitalmars.com> wrote in message 
news:khaplp$nn5$1 digitalmars.com...

 On 3/7/2013 4:35 AM, Daniel Murphy wrote:

 Maybe I'm missing something, but I don't understand why this is 
 necessary.
 It should be trivial to look at the map file for the closest preceding
 label.  On modern toolchains it should even be possible to compile the 
 asm
 with debug line information.  Or is grepping more effective than it 
 sounds?


 Modern operating systems load programs at different base addresses each 
 time, so the addresses in the map file don't match up with the addresses 
 at run time.


You can turn windows 7's aslr off.


 The older assembler I was using didn't emit symbolic debug info. I haven't 
 tried out ML that way.


Looks like the /Zd swich gives you line numbers.


 This sounds completely insane. (But also fun, in an extremely frustrating
 way)
 I suspect it would be easier to reverse-engineer microsoft's (or 
 borland's?)
 old linker from the disassembly, if they wrote it in C.


 You'd be incorrect if you thought MS's or Borland's linker was better. 
 Optlink sold very, very well against those linkers back in the day, 
 because Optlink was far better.


That's not what I meant.  I mean that when trying to work out what the 
assembly does, having real stack frames and a consistent calling convention 
could be more useful than knowing the label names and having out of date 
comments.

Optlink is lightning fast and memory efficient, but honestly most of the 
time you don't care.  Computers these days are ridiculously overpowered for 
something as simple as linking.

In huge projects this matters more, but then you start running into 
optlink's limitations.


 There's much more to making a linker. There's the debug info (which you 
 mentioned), then there are the resource files, the module definition 
 files, and lastly the myriad of rather complex switches that are expected. 
 Almost none of that is tested by the dmd test suite.


Yeah, debug information is fairly nasty.  It is at least as hard to 
implement as the actual linking.  (I have done some work on this for CV4, 
just not finished)

I don't expect res and def support are anywhere near that complicated.  Same 
goes for most of the command line switches.  (I have read through the list, 
but it's possible I'm being overconfident here)


 However, Optlink does bring considerable value to any linker replacement 
 project, in that embedded in it is an enormous amount of lore about all 
 those twinkie little things that matter, and how things really need to 
 work.


Optlink also has a lot of support for dead file formats/memory models and an 
insanely convoluted code base.  Like you said, porting to C doesn't escape 
this.
Mar 07 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 3/7/2013 2:25 PM, Daniel Murphy wrote:

 That's not what I meant.  I mean that when trying to work out what the
 assembly does, having real stack frames and a consistent calling convention
 could be more useful than knowing the label names and having out of date
 comments.


I've done some disassembly of real programs. It is a LOT harder than dealing 
with Optlink. Really, it's terribly difficult. Having the symbolic names, a 
logical organization, and even out of date comments, are a huge help.



 I don't expect res and def support are anywhere near that complicated.  Same
 goes for most of the command line switches.  (I have read through the list,
 but it's possible I'm being overconfident here)


The major problem is the lack of a test suite for those things.



 However, Optlink does bring considerable value to any linker replacement
 project, in that embedded in it is an enormous amount of lore about all
 those twinkie little things that matter, and how things really need to
 work.


 Optlink also has a lot of support for dead file formats/memory models and an
 insanely convoluted code base.  Like you said, porting to C doesn't escape
 this.


In my work converting Optlink to C, I dropped support for the false compile
time 
conditionals. There's no rationale for building a real mode optlink executable. 
There's also no reason to support building OS/2 executables anymore, etc.
Mar 07 2013
parent reply "Daniel Murphy" <yebblies nospamgmail.com> writes:
"Walter Bright" <newshound2 digitalmars.com> wrote in message 
news:khb5d7$1cfd$1 digitalmars.com...

 On 3/7/2013 2:25 PM, Daniel Murphy wrote:

 That's not what I meant.  I mean that when trying to work out what the
 assembly does, having real stack frames and a consistent calling 
 convention
 could be more useful than knowing the label names and having out of date
 comments.


 I've done some disassembly of real programs. It is a LOT harder than 
 dealing with Optlink. Really, it's terribly difficult. Having the symbolic 
 names, a logical organization, and even out of date comments, are a huge 
 help.


I've never worked on assembly that didn't use standard calling conventions, 
so I don't know how difficult it is.


 I don't expect res and def support are anywhere near that complicated. 
 Same
 goes for most of the command line switches.  (I have read through the 
 list,
 but it's possible I'm being overconfident here)


 The major problem is the lack of a test suite for those things.


Makes sense.  I have no idea how big the dmd/dmc/msc subset that would need 
to be supported is.


 However, Optlink does bring considerable value to any linker replacement
 project, in that embedded in it is an enormous amount of lore about all
 those twinkie little things that matter, and how things really need to
 work.


 Optlink also has a lot of support for dead file formats/memory models and 
 an
 insanely convoluted code base.  Like you said, porting to C doesn't 
 escape
 this.


 In my work converting Optlink to C, I dropped support for the false 
 compile time conditionals. There's no rationale for building a real mode 
 optlink executable. There's also no reason to support building OS/2 
 executables anymore, etc.


Good, but does the code still increase the difficulty in porting?

And even once it's in C, optlink will probably never be more than a 
win32/omf linker.
Mar 07 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 3/7/2013 7:09 PM, Daniel Murphy wrote:

 Good, but does the code still increase the difficulty in porting?


I don't understand your question.



 And even once it's in C, optlink will probably never be more than a
 win32/omf linker.


That's correct. However, it'll be much more maintainable, and it'll be a gold 
mine of information about linker trivia on how to do things with 
obscure/undocumented/bizarre file formats.
Mar 07 2013
parent reply "Daniel Murphy" <yebblies nospamgmail.com> writes:
"Walter Bright" <newshound2 digitalmars.com> wrote in message 
news:khblbe$27f5$1 digitalmars.com...

 On 3/7/2013 7:09 PM, Daniel Murphy wrote:

 Good, but does the code still increase the difficulty in porting?


 I don't understand your question.


Does the presence of support for eg. linking OS2 executables make the 
codebase harder to understand?


 And even once it's in C, optlink will probably never be more than a
 win32/omf linker.


 That's correct. However, it'll be much more maintainable,


I don't know how much redesign you're planning, but I can't imagine it ever 
being as maintainable as a pure d codebase.  A less stable/complete linker 
that attracts more contributors should overtake a more stable linker with 
only a couple of developers that grok it.


 and it'll be a gold mine of information about linker trivia on how to do 
 things with obscure/undocumented/bizarre file formats.


What is the license on optlink?  Can other linkers actually use this 
information?
Mar 07 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 3/7/2013 7:27 PM, Daniel Murphy wrote:

 "Walter Bright" <newshound2 digitalmars.com> wrote in message
 news:khblbe$27f5$1 digitalmars.com...

 On 3/7/2013 7:09 PM, Daniel Murphy wrote:

 Good, but does the code still increase the difficulty in porting?


 I don't understand your question.


 Does the presence of support for eg. linking OS2 executables make the
 codebase harder to understand?


Yes.



 That's correct. However, it'll be much more maintainable,


 I don't know how much redesign you're planning, but I can't imagine it ever
 being as maintainable as a pure d codebase.  A less stable/complete linker
 that attracts more contributors should overtake a more stable linker with
 only a couple of developers that grok it.


That's true, but we don't have that other linker yet.

The other thing is, we just don't have a need for our own linker for any 
platform other than win32. So what's the cost benefit moving forward? I think 
it's easier to just fix optlink's bugs.

I don't want to discourage people from trying to come up with a replacement 
linker for win32 written in D. I think that is a great project. But while a 
linker is a conceptually simple program, the awful file formats involved make
it 
unnecessarily difficult and there are simply a lot of details and other things 
one has to do.

Like I said before, it'll take a sustained and determined effort to come up
with 
a viable replacement for optlink.



 What is the license on optlink?


Same as the dmd back end.



 Can other linkers actually use this information?


They can use the information, yes, but not the code.
Mar 07 2013
next sibling parent "Daniel Murphy" <yebblies nospamgmail.com> writes:
"Walter Bright" <newshound2 digitalmars.com> wrote in message 
news:khbmkn$298s$1 digitalmars.com...

 That's correct. However, it'll be much more maintainable,


 I don't know how much redesign you're planning, but I can't imagine it 
 ever
 being as maintainable as a pure d codebase.  A less stable/complete 
 linker
 that attracts more contributors should overtake a more stable linker with
 only a couple of developers that grok it.


 That's true, but we don't have that other linker yet.

 The other thing is, we just don't have a need for our own linker for any 
 platform other than win32. So what's the cost benefit moving forward? I 
 think it's easier to just fix optlink's bugs.


You're probably right, but it would still be awesome to have our own modern 
linker.  Just being able to use coff import libraries directly on win32 
would be very nice.


 I don't want to discourage people from trying to come up with a 
 replacement linker for win32 written in D. I think that is a great 
 project. But while a linker is a conceptually simple program, the awful 
 file formats involved make it unnecessarily difficult and there are simply 
 a lot of details and other things one has to do.

 Like I said before, it'll take a sustained and determined effort to come 
 up with a viable replacement for optlink.


Agreed.


 What is the license on optlink?


 Same as the dmd back end.



 Can other linkers actually use this information?


 They can use the information, yes, but not the code.


That's interesting, I didn't realise this.
Mar 07 2013
prev sibling next sibling parent reply Brad Roberts <braddr puremagic.com> writes:
On 3/7/2013 7:41 PM, Walter Bright wrote:

 On 3/7/2013 7:27 PM, Daniel Murphy wrote:

 "Walter Bright" <newshound2 digitalmars.com> wrote in message
 news:khblbe$27f5$1 digitalmars.com...

 On 3/7/2013 7:09 PM, Daniel Murphy wrote:
 That's correct. However, it'll be much more maintainable,


 I don't know how much redesign you're planning, but I can't imagine it ever
 being as maintainable as a pure d codebase.  A less stable/complete linker
 that attracts more contributors should overtake a more stable linker with
 only a couple of developers that grok it.


 
 That's true, but we don't have that other linker yet.
 
 The other thing is, we just don't have a need for our own linker for any
platform other than win32. So what's the cost
 benefit moving forward? I think it's easier to just fix optlink's bugs.
 
 I don't want to discourage people from trying to come up with a replacement
linker for win32 written in D. I think that
 is a great project. But while a linker is a conceptually simple program, the
awful file formats involved make it
 unnecessarily difficult and there are simply a lot of details and other things
one has to do.
 
 Like I said before, it'll take a sustained and determined effort to come up
with a viable replacement for optlink.


Personally, even though I don't use win32, I believe that moving it over to use
the VS toolchain and runtime is the
right path forward.
Mar 07 2013
next sibling parent reply "Rikki Cattermole" <alphaglosined gmail.com> writes:
On Friday, 8 March 2013 at 04:17:03 UTC, Brad Roberts wrote:

 On 3/7/2013 7:41 PM, Walter Bright wrote:

 On 3/7/2013 7:27 PM, Daniel Murphy wrote:

 "Walter Bright" <newshound2 digitalmars.com> wrote in message
 news:khblbe$27f5$1 digitalmars.com...

 On 3/7/2013 7:09 PM, Daniel Murphy wrote:
 That's correct. However, it'll be much more maintainable,


 I don't know how much redesign you're planning, but I can't 
 imagine it ever
 being as maintainable as a pure d codebase.  A less 
 stable/complete linker
 that attracts more contributors should overtake a more stable 
 linker with
 only a couple of developers that grok it.


 
 That's true, but we don't have that other linker yet.
 
 The other thing is, we just don't have a need for our own 
 linker for any platform other than win32. So what's the cost
 benefit moving forward? I think it's easier to just fix 
 optlink's bugs.
 
 I don't want to discourage people from trying to come up with 
 a replacement linker for win32 written in D. I think that
 is a great project. But while a linker is a conceptually 
 simple program, the awful file formats involved make it
 unnecessarily difficult and there are simply a lot of details 
 and other things one has to do.
 
 Like I said before, it'll take a sustained and determined 
 effort to come up with a viable replacement for optlink.


 Personally, even though I don't use win32, I believe that 
 moving it over to use the VS toolchain and runtime is the
 right path forward.


I've actually modified dmd's frontend to support using the VS 
tool chain for linking 32 bit executables. But I didn't 
understand the backend to seperate the 32bit binary generation vs 
64bit. I believe it shouldn't be too much. But I really don't 
understand it so if anyone is willing to help on this.
I haven't broken 64bit generation as far as I know. Because I 
added a cli argument (-msvc) which is enabled for -m64 
automatically on Windows.
Mar 07 2013
parent reply "Jakob Ovrum" <jakobovrum gmail.com> writes:
On Friday, 8 March 2013 at 05:29:14 UTC, Rikki Cattermole wrote:

 I've actually modified dmd's frontend to support using the VS 
 tool chain for linking 32 bit executables. But I didn't 
 understand the backend to seperate the 32bit binary generation 
 vs 64bit. I believe it shouldn't be too much. But I really 
 don't understand it so if anyone is willing to help on this.
 I haven't broken 64bit generation as far as I know. Because I 
 added a cli argument (-msvc) which is enabled for -m64 
 automatically on Windows.


I'd love to have an option like that. Win32 is at the very least 
going to be a target for many years to come (for some also a 
development platform) and the OPTLINK/OMF business is a real 
impediment.
Mar 08 2013
parent "Rikki Cattermole" <alphaglosined gmail.com> writes:
On Friday, 8 March 2013 at 13:02:06 UTC, Jakob Ovrum wrote:

 On Friday, 8 March 2013 at 05:29:14 UTC, Rikki Cattermole wrote:

 I've actually modified dmd's frontend to support using the VS 
 tool chain for linking 32 bit executables. But I didn't 
 understand the backend to seperate the 32bit binary generation 
 vs 64bit. I believe it shouldn't be too much. But I really 
 don't understand it so if anyone is willing to help on this.
 I haven't broken 64bit generation as far as I know. Because I 
 added a cli argument (-msvc) which is enabled for -m64 
 automatically on Windows.


 I'd love to have an option like that. Win32 is at the very 
 least going to be a target for many years to come (for some 
 also a development platform) and the OPTLINK/OMF business is a 
 real impediment.


Personally I think of it as a barrier for adoption. A real BIG 
one for casual devs, which is what we need to grow. It also 
limits us from getting it into e.g. high schools. Which may at 
best be using core 2 duo's. Hence why I started working on it. 
Got so sick of not having it.
Just wish my knowledge extended to how the backend worked because 
then I could have finished this weeks ago.
Mar 08 2013
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 3/7/2013 8:16 PM, Brad Roberts wrote:

 Personally, even though I don't use win32, I believe that moving it over to
use the VS toolchain and runtime is the
 right path forward.


I like being able to provide a completely free D toolchain for win32.
Mar 07 2013
next sibling parent reply "Rikki Cattermole" <alphaglosined gmail.com> writes:
On Friday, 8 March 2013 at 07:17:24 UTC, Walter Bright wrote:

 On 3/7/2013 8:16 PM, Brad Roberts wrote:

 Personally, even though I don't use win32, I believe that 
 moving it over to use the VS toolchain and runtime is the
 right path forward.


 I like being able to provide a completely free D toolchain for 
 win32.


The express edition of Visual Studio should be enough to provide 
e.g. the linker.
http://www.microsoft.com/visualstudio/eng/downloads#d-express-windows-desktop
It is free. However you do need to register after 30 days.
Mar 07 2013
parent Rainer Schuetze <r.sagitario gmx.de> writes:
On 08.03.2013 08:25, Rikki Cattermole wrote:

 On Friday, 8 March 2013 at 07:17:24 UTC, Walter Bright wrote:

 On 3/7/2013 8:16 PM, Brad Roberts wrote:

 Personally, even though I don't use win32, I believe that moving it
 over to use the VS toolchain and runtime is the
 right path forward.


 I like being able to provide a completely free D toolchain for win32.


 The express edition of Visual Studio should be enough to provide e.g.
 the linker.
 http://www.microsoft.com/visualstudio/eng/downloads#d-express-windows-desktop

 It is free. However you do need to register after 30 days.


There are also C++ compiler and linker in the Windows 7 SDK that you can 
download and install without registering 
(http://www.microsoft.com/en-us/download/details.aspx?id=8442).

It might be fun, but I think that building your own COFF capable linker 
will be a huge undertaking if you want to link against C/C++ code built 
with the MS compiler without stripping a whole lot of information (debug 
information, FPO records, etc).
Mar 08 2013
prev sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 3/8/13, Walter Bright <newshound2 digitalmars.com> wrote:

 On 3/7/2013 8:16 PM, Brad Roberts wrote:

 Personally, even though I don't use win32, I believe that moving it over
 to use the VS toolchain and runtime is the
 right path forward.


 I like being able to provide a completely free D toolchain for win32.


s/for win32/./

:)

Honestly D has huge potential for tools like compilers/linkers/etc.
They're the types of programs where you practically know all your
requirements at compile-time, so e.g. generics come into play really
nicely.

We should keep the idea of having a free D toolchain for all OSes in
the back of our heads, even if we don't work on it any time soon.
Mar 08 2013
next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Friday, 8 March 2013 at 17:40:58 UTC, Andrej Mitrovic wrote:

 Honestly D has huge potential for tools like 
 compilers/linkers/etc.
 They're the types of programs where you practically know all 
 your
 requirements at compile-time, so e.g. generics come into play 
 really
 nicely.


That is the theory. In practice you'll run in MANY problems.
Mar 08 2013
parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 3/9/13, deadalnix <deadalnix gmail.com> wrote:

 On Friday, 8 March 2013 at 17:40:58 UTC, Andrej Mitrovic wrote:

 Honestly D has huge potential for tools like
 compilers/linkers/etc.
 They're the types of programs where you practically know all
 your
 requirements at compile-time, so e.g. generics come into play
 really
 nicely.


 That is the theory. In practice you'll run in MANY problems.


In practice I'm getting quite fed up with linker bugs, or DMD object
gen bugs which cause the linker to choke. :)
Mar 09 2013
prev sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Friday, 8 March 2013 at 17:40:58 UTC, Andrej Mitrovic wrote:

 On 3/8/13, Walter Bright <newshound2 digitalmars.com> wrote:

 On 3/7/2013 8:16 PM, Brad Roberts wrote:

 Personally, even though I don't use win32, I believe that 
 moving it over
 to use the VS toolchain and runtime is the
 right path forward.


 I like being able to provide a completely free D toolchain for 
 win32.


 s/for win32/./

 :)

 Honestly D has huge potential for tools like 
 compilers/linkers/etc.
 They're the types of programs where you practically know all 
 your
 requirements at compile-time, so e.g. generics come into play 
 really
 nicely.



Personally I prefer FP based languages for this type of work, 
given how easy it is to manipulate ASTs.

--
Paulo
Mar 14 2013
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-03-08 04:41, Walter Bright wrote:


 The other thing is, we just don't have a need for our own linker for any
 platform other than win32. So what's the cost benefit moving forward? I
 think it's easier to just fix optlink's bugs.


If we have our own linker for every supported platform and format I 
pretty sure that we can take advantage of that and come up with features 
not possible using other linkers.

For example, storing .di files directly in the object file. This has 
also been suggested before.

-- 
/Jacob Carlborg
Mar 08 2013
next sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
08-Mar-2013 12:50, Jacob Carlborg пишет:

 On 2013-03-08 04:41, Walter Bright wrote:


 The other thing is, we just don't have a need for our own linker for any
 platform other than win32. So what's the cost benefit moving forward? I
 think it's easier to just fix optlink's bugs.


 If we have our own linker for every supported platform and format I
 pretty sure that we can take advantage of that and come up with features
 not possible using other linkers.


Yes, along with storing meta-data such as pure, nothrow etc. Mangling 
does it but in a such a backward way...
But it would always have to maintain backwards compatibility or simple 
conversion to this format.


 For example, storing .di files directly in the object file. This has
 also been suggested before.



-- 
Dmitry Olshansky
Mar 08 2013
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 3/8/2013 12:50 AM, Jacob Carlborg wrote:

 If we have our own linker for every supported platform and format I pretty sure
 that we can take advantage of that and come up with features not possible using
 other linkers.

 For example, storing .di files directly in the object file. This has also been
 suggested before.



Sure, but who is going to do the work?

BTW, we did come up with our librarian (it's built in to dmd) and it works
great 
- enabling simple, smooth handling of libraries in a way that no other similar 
language does. Notice, though, there are 4 completely distinct code bases to 
support it, for the 4 main platforms. The librarian is a simple program, but 
linkers are much more complex.
Mar 08 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-03-08 09:58, Walter Bright wrote:


 Sure, but who is going to do the work?


Anyone that's interested in doing the work. But we don't have to invest 
all our time doing it. Daniel Murphy seems to be working on a linker, so 
apparently there is at least some interest.


 BTW, we did come up with our librarian (it's built in to dmd) and it
 works great - enabling simple, smooth handling of libraries in a way
 that no other similar language does. Notice, though, there are 4
 completely distinct code bases to support it, for the 4 main platforms.
 The librarian is a simple program, but linkers are much more complex.


Yeah, I didn't say it was going to be easy.

-- 
/Jacob Carlborg
Mar 08 2013
prev sibling parent reply Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On Thu, 07 Mar 2013 19:41:10 -0800
Walter Bright <newshound2 digitalmars.com> wrote:

 
 I don't want to discourage people from trying to come up with a
 replacement linker for win32 written in D. I think that is a great
 project. But while a linker is a conceptually simple program, the
 awful file formats involved make it unnecessarily difficult and there
 are simply a lot of details and other things one has to do.
 
 Like I said before, it'll take a sustained and determined effort to
 come up with a viable replacement for optlink.
 


Interesting thing about this, is that if/when we did have this "new
linker" as discussed (including separation of individual object file
formats) is that we could also add in our own object file format
designed (and proven via a good test suite) to be much less quirky and
badly documented. Or, you know, not quirky and not badly
documented ;) Thus, at the very least, opening a door for others to
migrate, and potentially/finally/eventually get rid of the crufty
current formats. I mean, like you said, linkers are very simple
conceptually. So if they *can* be so simple, why should everyone have
to continue to stay stuck with a crappy one?

Just my random thoughts on it anyway.
Mar 08 2013
parent Walter Bright <newshound2 digitalmars.com> writes:
On 3/8/2013 11:29 PM, Nick Sabalausky wrote:

 So if they *can* be so simple, why should everyone have
 to continue to stay stuck with a crappy one?


That's a good question. It's surprising how excessively complicated most
designs 
are.
Mar 08 2013
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-03-07 13:35, Daniel Murphy wrote:


 If anyone is interested I'll put it up on github.


I would say put it there to see how much interest there is.

-- 
/Jacob Carlborg
Mar 07 2013
parent "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Jacob Carlborg"  wrote in message news:khaqhb$on0$1 digitalmars.com... 

On 2013-03-07 13:35, Daniel Murphy wrote:


 If anyone is interested I'll put it up on github.


 
 I would say put it there to see how much interest there is.


Better late than never...

https://github.com/yebblies/ylink
Mar 23 2014
prev sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 3/7/13, Daniel Murphy <yebblies nospamgmail.com> wrote:

 If anyone is interested I'll put it up on github.


I'd be interested, please do put it online when you get the time.
Mar 09 2013
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 7 March 2013 at 01:25:02 UTC, Walter Bright wrote:

 Some months ago, I did make the source to optlink available on 
 github:

 https://github.com/DigitalMars/optlink

 Rainer Schuetze has improved it where it can be built with 
 modern tools (the older tools would not run on Win7). I know 
 some of you are frustrated by optlink problems. Well, now you 
 can do something about it!

 Note that the optlink source is quite a challenge to work on. 
 It's very old skool (80's style).

 Here's an approach that I've used successfully to fix optlink 
 seg faults:

 1. Find out where in the source code it is. This is not so 
 easy, even if using a debugger. The trouble is the asm 
 functions do not have standard stack frames, so the debugger 
 cannot do a stack trace. Even where there is a standard stack 
 frame, the modern Microsoft debuggers fail to recognize them. 
 You gotta use an older debugger under Windows XP.

 So, what I do, is look at the asm code where the seg fault 
 occurs, and then grep through the asm source till I find it.

 2. Convert the source file where the seg fault occurs to C. I 
 do this one function at a time, running the full set of tests 
 at each step. Otherwise, it's just impossible to figure out 
 where you made a mistake in the conversion. Sometimes, you 
 gotta go even finer grained - using the inline assembler 
 feature of dmc to convert asm code line by line.

 You gotta pay very, very close attention to which registers 
 have parameters passed through them, which registers are saved, 
 and which registers have parameters that are silently passed 
 through a function to callees of that function (!). This 
 information is all pretty much utterly lacking in the comments, 
 and the comments are often dead wrong as they refer to much 
 older versions of the code.

 3. Once it is in C, things get a lot easier. You can, for 
 example, insert printf's to figure out where things go wrong, 
 and *then* fix it.


 The full test suite for optlink has a lot of stuff I cannot 
 publish on github. However, what you can do is run the win32 
 tests for dmd. I, of course, will run the full one when 
 verifying pulls.

 Happy hacking!


Most people would consider this an exercise in self-flagellation, 
but I enjoy doing it.

http://www.drdobbs.com/architecture-and-design/assembler-to-c/228701319
Mar 08 2013
prev sibling next sibling parent reply "Asman01" <jckj33 gmail.com> writes:
On Thursday, 7 March 2013 at 01:25:02 UTC, Walter Bright wrote:

 Some months ago, I did make the source to optlink available on 
 github:

 https://github.com/DigitalMars/optlink

 Rainer Schuetze has improved it where it can be built with 
 modern tools (the older tools would not run on Win7). I know 
 some of you are frustrated by optlink problems. Well, now you 
 can do something about it!

 Note that the optlink source is quite a challenge to work on. 
 It's very old skool (80's style).

 Here's an approach that I've used successfully to fix optlink 
 seg faults:

 1. Find out where in the source code it is. This is not so 
 easy, even if using a debugger. The trouble is the asm 
 functions do not have standard stack frames, so the debugger 
 cannot do a stack trace. Even where there is a standard stack 
 frame, the modern Microsoft debuggers fail to recognize them. 
 You gotta use an older debugger under Windows XP.

 So, what I do, is look at the asm code where the seg fault 
 occurs, and then grep through the asm source till I find it.

 2. Convert the source file where the seg fault occurs to C. I 
 do this one function at a time, running the full set of tests 
 at each step. Otherwise, it's just impossible to figure out 
 where you made a mistake in the conversion. Sometimes, you 
 gotta go even finer grained - using the inline assembler 
 feature of dmc to convert asm code line by line.

 You gotta pay very, very close attention to which registers 
 have parameters passed through them, which registers are saved, 
 and which registers have parameters that are silently passed 
 through a function to callees of that function (!). This 
 information is all pretty much utterly lacking in the comments, 
 and the comments are often dead wrong as they refer to much 
 older versions of the code.

 3. Once it is in C, things get a lot easier. You can, for 
 example, insert printf's to figure out where things go wrong, 
 and *then* fix it.


 The full test suite for optlink has a lot of stuff I cannot 
 publish on github. However, what you can do is run the win32 
 tests for dmd. I, of course, will run the full one when 
 verifying pulls.

 Happy hacking!


Is this the current linker used by DMD on all platforms?
Mar 23 2014
parent "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Asman01"  wrote in message news:klvtbihnvoxfarsjdlhv forum.dlang.org... 


 Is this the current linker used by DMD on all platforms?


No, only when building win32 executables.
Mar 23 2014
prev sibling parent reply "Asman01" <jckj33 gmail.com> writes:
Very noob question about binary files. What else also put the 
code to load at right address (say, 0x08048000 on linux) of 
operating system is needed to a program run?
Mar 23 2014
parent reply "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Asman01"  wrote in message news:ucqujzetvkkxzelvjleq forum.dlang.org...


 Very noob question about binary files. What else also put the code to load 
 at right address (say, 0x08048000 on linux) of operating system is needed 
 to a program run?


Not really sure what you're asking, but the executable file usually contains 
an address for each section (code, data, bss, etc) and the runtime loader 
will choose to load it there or somewhere else (eg aslr/shared libraries) 
and map the data into virtual memory.
Mar 23 2014
parent "Asman01" <jckj33 gmail.com> writes:
On Sunday, 23 March 2014 at 16:10:48 UTC, Daniel Murphy wrote:

 "Asman01"  wrote in message 
 news:ucqujzetvkkxzelvjleq forum.dlang.org...


 Very noob question about binary files. What else also put the 
 code to load at right address (say, 0x08048000 on linux) of 
 operating system is needed to a program run?


 Not really sure what you're asking, but the executable file 
 usually contains an address for each section (code, data, bss, 
 etc) and the runtime loader will choose to load it there or 
 somewhere else (eg aslr/shared libraries) and map the data into 
 virtual memory.


I think I was confusing this to static executables which load at 
0x08048000 address and start running from value your set in entry 
point.
Mar 23 2014