www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Optlink is on github

reply Walter Bright <newshound2 digitalmars.com> writes:
Some months ago, I did make the source to optlink available on github:

https://github.com/DigitalMars/optlink

Rainer Schuetze has improved it where it can be built with modern tools (the 
older tools would not run on Win7). I know some of you are frustrated by
optlink 
problems. Well, now you can do something about it!

Note that the optlink source is quite a challenge to work on. It's very old 
skool (80's style).

Here's an approach that I've used successfully to fix optlink seg faults:

1. Find out where in the source code it is. This is not so easy, even if using
a 
debugger. The trouble is the asm functions do not have standard stack frames,
so 
the debugger cannot do a stack trace. Even where there is a standard stack 
frame, the modern Microsoft debuggers fail to recognize them. You gotta use an 
older debugger under Windows XP.

So, what I do, is look at the asm code where the seg fault occurs, and then
grep 
through the asm source till I find it.

2. Convert the source file where the seg fault occurs to C. I do this one 
function at a time, running the full set of tests at each step. Otherwise, it's 
just impossible to figure out where you made a mistake in the conversion. 
Sometimes, you gotta go even finer grained - using the inline assembler feature 
of dmc to convert asm code line by line.

You gotta pay very, very close attention to which registers have parameters 
passed through them, which registers are saved, and which registers have 
parameters that are silently passed through a function to callees of that 
function (!). This information is all pretty much utterly lacking in the 
comments, and the comments are often dead wrong as they refer to much older 
versions of the code.

3. Once it is in C, things get a lot easier. You can, for example, insert 
printf's to figure out where things go wrong, and *then* fix it.


The full test suite for optlink has a lot of stuff I cannot publish on github. 
However, what you can do is run the win32 tests for dmd. I, of course, will run 
the full one when verifying pulls.

Happy hacking!
Mar 06 2013
next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Walter Bright:

 Happy hacking!

Extra karma points if done blindfolded, using a Braille tablet :-) Bye, bearophile
Mar 06 2013
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 7 March 2013 at 01:25:02 UTC, Walter Bright wrote:
 Some months ago, I did make the source to optlink available on 
 github:

 https://github.com/DigitalMars/optlink

 Rainer Schuetze has improved it where it can be built with 
 modern tools (the older tools would not run on Win7). I know 
 some of you are frustrated by optlink problems. Well, now you 
 can do something about it!

 Note that the optlink source is quite a challenge to work on. 
 It's very old skool (80's style).

 Here's an approach that I've used successfully to fix optlink 
 seg faults:

 1. Find out where in the source code it is. This is not so 
 easy, even if using a debugger. The trouble is the asm 
 functions do not have standard stack frames, so the debugger 
 cannot do a stack trace. Even where there is a standard stack 
 frame, the modern Microsoft debuggers fail to recognize them. 
 You gotta use an older debugger under Windows XP.

 So, what I do, is look at the asm code where the seg fault 
 occurs, and then grep through the asm source till I find it.

 2. Convert the source file where the seg fault occurs to C. I 
 do this one function at a time, running the full set of tests 
 at each step. Otherwise, it's just impossible to figure out 
 where you made a mistake in the conversion. Sometimes, you 
 gotta go even finer grained - using the inline assembler 
 feature of dmc to convert asm code line by line.

 You gotta pay very, very close attention to which registers 
 have parameters passed through them, which registers are saved, 
 and which registers have parameters that are silently passed 
 through a function to callees of that function (!). This 
 information is all pretty much utterly lacking in the comments, 
 and the comments are often dead wrong as they refer to much 
 older versions of the code.

 3. Once it is in C, things get a lot easier. You can, for 
 example, insert printf's to figure out where things go wrong, 
 and *then* fix it.


 The full test suite for optlink has a lot of stuff I cannot 
 publish on github. However, what you can do is run the win32 
 tests for dmd. I, of course, will run the full one when 
 verifying pulls.

 Happy hacking!

It sound like a nightmare. What is the reason in the first place to use optlink ? Isn't it possible to use another linker ?
Mar 06 2013
prev sibling next sibling parent "Lars T. Kyllingstad" <public kyllingen.net> writes:
On Thursday, 7 March 2013 at 01:25:02 UTC, Walter Bright wrote:
 Some months ago, I did make the source to optlink available on 
 github:

 [...]

 Happy hacking!

This reminds me of http://www.catb.org/jargon/html/story-of-mel.html :) Lars
Mar 06 2013
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Mar 07, 2013 at 07:09:22AM +0100, Lars T. Kyllingstad wrote:
 On Thursday, 7 March 2013 at 01:25:02 UTC, Walter Bright wrote:
Some months ago, I did make the source to optlink available on
github:

[...]

Happy hacking!

This reminds me of http://www.catb.org/jargon/html/story-of-mel.html

Man, that brings back the memories. I remember doing those kinds of stunts when coding for the good ole Motorola 6502. Back in those days, Apple actually published the full listing of their ROM code along with the instruction specs, including the number of cycles per instruction, etc.. There were many a fun night plotting out the optimal instruction sequences and spacing out instructions just right so the exact timing is achieved. Lots of fun. T -- What doesn't kill me makes me stranger.
Mar 06 2013
prev sibling next sibling parent =?iso-8859-15?Q?Simen_Kj=E6r=E5s?= <simen.kjaras gmail.com> writes:
On Thu, 07 Mar 2013 06:04:29 +0100, deadalnix <deadalnix gmail.com> wrote:

 It sound like a nightmare. What is the reason in the first place to use  
 optlink ? Isn't it possible to use another linker ?

It's fast, it's pretty well tested, Walter's written it himself, and changing to a different linker is likely to have its own set of challenges. -- Simen
Mar 06 2013
prev sibling next sibling parent reply "Moritz Maxeiner" <moritz ucworks.org> writes:
On Thursday, 7 March 2013 at 01:25:02 UTC, Walter Bright wrote:
 https://github.com/DigitalMars/optlink

 Rainer Schuetze has improved it where it can be built with 
 modern tools (the older tools would not run on Win7). I know 
 some of you are frustrated by optlink problems. Well, now you 
 can do something about it!

Sorry if this has been answered before/is common knowledge, but is porting functions at a time to C wanted for optlink in general, or only for finding segfaults? (e.g. are pull-requests for that welcome)
Mar 07 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-03-07 11:12, Moritz Maxeiner wrote:

 Sorry if this has been answered before/is common knowledge, but is
 porting functions at a time to C wanted for optlink in general, or only
 for finding segfaults? (e.g. are pull-requests for that welcome)

Yes, in general. I think the idea is to port the whole Optlink to C and then to D. It's easier to port from C to D than from assembly to D. This is because you can use a C version that doesn't use the runtime or standard library to get the generate assembly as close as possible to the original one. I think it was something like that. -- /Jacob Carlborg
Mar 07 2013
parent reply Denis Shelomovskij <verylonglogin.reg gmail.com> writes:
07.03.2013 14:28, Jacob Carlborg пишет:
 On 2013-03-07 11:12, Moritz Maxeiner wrote:

 Sorry if this has been answered before/is common knowledge, but is
 porting functions at a time to C wanted for optlink in general, or only
 for finding segfaults? (e.g. are pull-requests for that welcome)

Yes, in general. I think the idea is to port the whole Optlink to C and then to D. It's easier to port from C to D than from assembly to D. This is because you can use a C version that doesn't use the runtime or standard library to get the generate assembly as close as possible to the original one. I think it was something like that.

Didn't get. You don't have to use D with druntime. Just don't link it and everything will be OK - you will just get "better C" (i.e. with D structs and other good stuff). -- Денис В. Шеломовский Denis V. Shelomovskij
Mar 07 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-03-07 16:53, Denis Shelomovskij wrote:

 Didn't get. You don't have to use D with druntime. Just don't link it
 and everything will be OK - you will just get "better C" (i.e. with D
 structs and other good stuff).

I was referring to using C with out the C runtime or C standard library. Walter talked about this is some thread. He basically reimplemented printf. -- /Jacob Carlborg
Mar 07 2013
prev sibling next sibling parent reply "Daniel Murphy" <yebblies nospamgmail.com> writes:
"Walter Bright" <newshound2 digitalmars.com> wrote in message 
news:kh8q9e$2j02$1 digitalmars.com...
 Some months ago, I did make the source to optlink available on github:

 https://github.com/DigitalMars/optlink

 So, what I do, is look at the asm code where the seg fault occurs, and 
 then grep through the asm source till I find it.

Maybe I'm missing something, but I don't understand why this is necessary. It should be trivial to look at the map file for the closest preceding label. On modern toolchains it should even be possible to compile the asm with debug line information. Or is grepping more effective than it sounds?
 2. Convert the source file where the seg fault occurs to C. I do this one 
 function at a time, running the full set of tests at each step. Otherwise, 
 it's just impossible to figure out where you made a mistake in the 
 conversion. Sometimes, you gotta go even finer grained - using the inline 
 assembler feature of dmc to convert asm code line by line.

 You gotta pay very, very close attention to which registers have 
 parameters passed through them, which registers are saved, and which 
 registers have parameters that are silently passed through a function to 
 callees of that function (!). This information is all pretty much utterly 
 lacking in the comments, and the comments are often dead wrong as they 
 refer to much older versions of the code.

This sounds completely insane. (But also fun, in an extremely frustrating way) I suspect it would be easier to reverse-engineer microsoft's (or borland's?) old linker from the disassembly, if they wrote it in C.
 3. Once it is in C, things get a lot easier. You can, for example, insert 
 printf's to figure out where things go wrong, and *then* fix it.


 The full test suite for optlink has a lot of stuff I cannot publish on 
 github. However, what you can do is run the win32 tests for dmd. I, of 
 course, will run the full one when verifying pulls.

 Happy hacking!

As a side note I actually wrote an omf (dmd-emitted subset) linker last year that gets through almost all of the dmd test suite. It has no debug info support, which sucks, but it may work in some cases where optlink can't cope. It is fairly naive performance-wise, but is written in a much friendlier language. If anyone is interested I'll put it up on github.
Mar 07 2013
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 3/7/2013 4:35 AM, Daniel Murphy wrote:
 Maybe I'm missing something, but I don't understand why this is necessary.
 It should be trivial to look at the map file for the closest preceding
 label.  On modern toolchains it should even be possible to compile the asm
 with debug line information.  Or is grepping more effective than it sounds?

Modern operating systems load programs at different base addresses each time, so the addresses in the map file don't match up with the addresses at run time. The older assembler I was using didn't emit symbolic debug info. I haven't tried out ML that way.
 This sounds completely insane. (But also fun, in an extremely frustrating
 way)
 I suspect it would be easier to reverse-engineer microsoft's (or borland's?)
 old linker from the disassembly, if they wrote it in C.

You'd be incorrect if you thought MS's or Borland's linker was better. Optlink sold very, very well against those linkers back in the day, because Optlink was far better.
 As a side note I actually wrote an omf (dmd-emitted subset) linker last year
 that gets through almost all of the dmd test suite.  It has no debug info
 support, which sucks, but it may work in some cases where optlink can't
 cope.  It is fairly naive performance-wise, but is written in a much
 friendlier language.

 If anyone is interested I'll put it up on github.

There's much more to making a linker. There's the debug info (which you mentioned), then there are the resource files, the module definition files, and lastly the myriad of rather complex switches that are expected. Almost none of that is tested by the dmd test suite.
Mar 07 2013
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 3/7/2013 11:32 AM, Andrej Mitrovic wrote:
 On Thursday, 7 March 2013 at 19:26:49 UTC, Walter Bright wrote:
 If anyone is interested I'll put it up on github.

There's much more to making a linker. There's the debug info (which you mentioned), then there are the resource files, the module definition files, and lastly the myriad of rather complex switches that are expected. Almost none of that is tested by the dmd test suite.

Nevertheless having a linker written in D goes hand-in-hand with having a compiler written in D. I'd be interested in seeing his work. I'd also be interested in having something that's documented and that isn't bound by licensing issues. We also have the abandoned DDL[1] project which had a linker that supports OMF, and Eric Anderton was nice enough to document his process of implementing it: http://www.dsource.org/forums/viewtopic.php?t=959 [1] http://dsource.org/projects/ddl/

Sure, having a complete linker written in D would be preferable. I also began a project to write such. However, getting such to be a complete, viable replacement for optlink will require a determined, sustained effort. I also believe that even if Optlink were completely converted to C, it will still be intractably difficult to refactor it to do things like support other object file formats. This is why I had started a new linker project. However, Optlink does bring considerable value to any linker replacement project, in that embedded in it is an enormous amount of lore about all those twinkie little things that matter, and how things really need to work. And, of course, consider that we don't have a thorough test suite for a linker. This is a serious problem. Trying to make symbolic debug info work without such is a big issue. Heck, I spent quite a bit of time getting dmd to generate correct CV8 info, given that what MS's debugger accepts does not always match the (sparse and confusing) documentation. What we *can* do with Optlink is fix the occasional bug that crops up in it.
Mar 07 2013
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 3/7/2013 12:15 PM, Andrej Mitrovic wrote:
 Interesting. But if it's made behind-the-scenes with no input or work from the
 community I won't be holding my hopes up too much, I know you have very limited
 time for side-projects like that.

It didn't get much past the conceptual part. I wanted to make a linker where: 1. all the supported file formats were completely encapsulated and modularized For some idea of how that can work, see libmscoff.c and scanmscoff.c, where the object file format and library file format is completely separated. 2. used ranges to their fullest extend
Mar 07 2013
prev sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
08-Mar-2013 00:15, Andrej Mitrovic пишет:
 On Thursday, 7 March 2013 at 20:01:58 UTC, Walter Bright wrote:
 What we *can* do with Optlink is fix the occasional bug that crops up
 in it.

I hope so. :) On Thursday, 7 March 2013 at 20:01:58 UTC, Walter Bright wrote:
 I also believe that even if Optlink were completely converted to C, it
 will still be intractably difficult to refactor it to do things like
 support other object file formats.

I really wish Unilink[1] was open-source.

Same here. Maybe we should try and convince him to share it with some OpenSource license?
 It's apparently a student
 project, and it's closed source but freeware. It works with OMF but can
 also link OMF and COFF objects together, which is a boon for win32
 programmers who don't want to go through OBJCONV (object file
 conversion, doesn't always work) or COFF2OMF (import libs). It also has
 pretty good error messages.

 Of course there's no telling how big its test-suite is. It's fairly easy
 to set up though[2].

 [1] ftp://ftp.styx.cabel.net/pub/UniLink/
 [2]
 http://forum.dlang.org/thread/jbvugp$8l3$1 digitalmars.com#post-mailman.1414.1323538292.24802.digitalmars-d:40puremagic.com


 On Thursday, 7 March 2013 at 20:01:58 UTC, Walter Bright wrote:
 This is why I had started a new linker project.

Interesting. But if it's made behind-the-scenes with no input or work from the community I won't be holding my hopes up too much, I know you have very limited time for side-projects like that.

-- Dmitry Olshansky
Mar 08 2013
prev sibling parent reply "Daniel Murphy" <yebblies nospamgmail.com> writes:
"Walter Bright" <newshound2 digitalmars.com> wrote in message 
news:khaplp$nn5$1 digitalmars.com...
 On 3/7/2013 4:35 AM, Daniel Murphy wrote:
 Maybe I'm missing something, but I don't understand why this is 
 necessary.
 It should be trivial to look at the map file for the closest preceding
 label.  On modern toolchains it should even be possible to compile the 
 asm
 with debug line information.  Or is grepping more effective than it 
 sounds?

Modern operating systems load programs at different base addresses each time, so the addresses in the map file don't match up with the addresses at run time.

You can turn windows 7's aslr off.
 The older assembler I was using didn't emit symbolic debug info. I haven't 
 tried out ML that way.

Looks like the /Zd swich gives you line numbers.
 This sounds completely insane. (But also fun, in an extremely frustrating
 way)
 I suspect it would be easier to reverse-engineer microsoft's (or 
 borland's?)
 old linker from the disassembly, if they wrote it in C.

You'd be incorrect if you thought MS's or Borland's linker was better. Optlink sold very, very well against those linkers back in the day, because Optlink was far better.

That's not what I meant. I mean that when trying to work out what the assembly does, having real stack frames and a consistent calling convention could be more useful than knowing the label names and having out of date comments. Optlink is lightning fast and memory efficient, but honestly most of the time you don't care. Computers these days are ridiculously overpowered for something as simple as linking. In huge projects this matters more, but then you start running into optlink's limitations.
 There's much more to making a linker. There's the debug info (which you 
 mentioned), then there are the resource files, the module definition 
 files, and lastly the myriad of rather complex switches that are expected. 
 Almost none of that is tested by the dmd test suite.

Yeah, debug information is fairly nasty. It is at least as hard to implement as the actual linking. (I have done some work on this for CV4, just not finished) I don't expect res and def support are anywhere near that complicated. Same goes for most of the command line switches. (I have read through the list, but it's possible I'm being overconfident here)
 However, Optlink does bring considerable value to any linker replacement 
 project, in that embedded in it is an enormous amount of lore about all 
 those twinkie little things that matter, and how things really need to 
 work.

Optlink also has a lot of support for dead file formats/memory models and an insanely convoluted code base. Like you said, porting to C doesn't escape this.
Mar 07 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 3/7/2013 2:25 PM, Daniel Murphy wrote:
 That's not what I meant.  I mean that when trying to work out what the
 assembly does, having real stack frames and a consistent calling convention
 could be more useful than knowing the label names and having out of date
 comments.

I've done some disassembly of real programs. It is a LOT harder than dealing with Optlink. Really, it's terribly difficult. Having the symbolic names, a logical organization, and even out of date comments, are a huge help.
 I don't expect res and def support are anywhere near that complicated.  Same
 goes for most of the command line switches.  (I have read through the list,
 but it's possible I'm being overconfident here)

The major problem is the lack of a test suite for those things.
 However, Optlink does bring considerable value to any linker replacement
 project, in that embedded in it is an enormous amount of lore about all
 those twinkie little things that matter, and how things really need to
 work.

Optlink also has a lot of support for dead file formats/memory models and an insanely convoluted code base. Like you said, porting to C doesn't escape this.

In my work converting Optlink to C, I dropped support for the false compile time conditionals. There's no rationale for building a real mode optlink executable. There's also no reason to support building OS/2 executables anymore, etc.
Mar 07 2013
parent reply "Daniel Murphy" <yebblies nospamgmail.com> writes:
"Walter Bright" <newshound2 digitalmars.com> wrote in message 
news:khb5d7$1cfd$1 digitalmars.com...
 On 3/7/2013 2:25 PM, Daniel Murphy wrote:
 That's not what I meant.  I mean that when trying to work out what the
 assembly does, having real stack frames and a consistent calling 
 convention
 could be more useful than knowing the label names and having out of date
 comments.

I've done some disassembly of real programs. It is a LOT harder than dealing with Optlink. Really, it's terribly difficult. Having the symbolic names, a logical organization, and even out of date comments, are a huge help.

I've never worked on assembly that didn't use standard calling conventions, so I don't know how difficult it is.
 I don't expect res and def support are anywhere near that complicated. 
 Same
 goes for most of the command line switches.  (I have read through the 
 list,
 but it's possible I'm being overconfident here)

The major problem is the lack of a test suite for those things.

Makes sense. I have no idea how big the dmd/dmc/msc subset that would need to be supported is.
 However, Optlink does bring considerable value to any linker replacement
 project, in that embedded in it is an enormous amount of lore about all
 those twinkie little things that matter, and how things really need to
 work.

Optlink also has a lot of support for dead file formats/memory models and an insanely convoluted code base. Like you said, porting to C doesn't escape this.

In my work converting Optlink to C, I dropped support for the false compile time conditionals. There's no rationale for building a real mode optlink executable. There's also no reason to support building OS/2 executables anymore, etc.

Good, but does the code still increase the difficulty in porting? And even once it's in C, optlink will probably never be more than a win32/omf linker.
Mar 07 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 3/7/2013 7:09 PM, Daniel Murphy wrote:
 Good, but does the code still increase the difficulty in porting?

I don't understand your question.
 And even once it's in C, optlink will probably never be more than a
 win32/omf linker.

That's correct. However, it'll be much more maintainable, and it'll be a gold mine of information about linker trivia on how to do things with obscure/undocumented/bizarre file formats.
Mar 07 2013
parent reply "Daniel Murphy" <yebblies nospamgmail.com> writes:
"Walter Bright" <newshound2 digitalmars.com> wrote in message 
news:khblbe$27f5$1 digitalmars.com...
 On 3/7/2013 7:09 PM, Daniel Murphy wrote:
 Good, but does the code still increase the difficulty in porting?

I don't understand your question.

Does the presence of support for eg. linking OS2 executables make the codebase harder to understand?
 And even once it's in C, optlink will probably never be more than a
 win32/omf linker.

That's correct. However, it'll be much more maintainable,

I don't know how much redesign you're planning, but I can't imagine it ever being as maintainable as a pure d codebase. A less stable/complete linker that attracts more contributors should overtake a more stable linker with only a couple of developers that grok it.
 and it'll be a gold mine of information about linker trivia on how to do 
 things with obscure/undocumented/bizarre file formats.

What is the license on optlink? Can other linkers actually use this information?
Mar 07 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 3/7/2013 7:27 PM, Daniel Murphy wrote:
 "Walter Bright" <newshound2 digitalmars.com> wrote in message
 news:khblbe$27f5$1 digitalmars.com...
 On 3/7/2013 7:09 PM, Daniel Murphy wrote:
 Good, but does the code still increase the difficulty in porting?

I don't understand your question.

Does the presence of support for eg. linking OS2 executables make the codebase harder to understand?

Yes.
 That's correct. However, it'll be much more maintainable,

I don't know how much redesign you're planning, but I can't imagine it ever being as maintainable as a pure d codebase. A less stable/complete linker that attracts more contributors should overtake a more stable linker with only a couple of developers that grok it.

That's true, but we don't have that other linker yet. The other thing is, we just don't have a need for our own linker for any platform other than win32. So what's the cost benefit moving forward? I think it's easier to just fix optlink's bugs. I don't want to discourage people from trying to come up with a replacement linker for win32 written in D. I think that is a great project. But while a linker is a conceptually simple program, the awful file formats involved make it unnecessarily difficult and there are simply a lot of details and other things one has to do. Like I said before, it'll take a sustained and determined effort to come up with a viable replacement for optlink.
 What is the license on optlink?

Same as the dmd back end.
 Can other linkers actually use this information?

They can use the information, yes, but not the code.
Mar 07 2013
next sibling parent "Daniel Murphy" <yebblies nospamgmail.com> writes:
"Walter Bright" <newshound2 digitalmars.com> wrote in message 
news:khbmkn$298s$1 digitalmars.com...
 That's correct. However, it'll be much more maintainable,

I don't know how much redesign you're planning, but I can't imagine it ever being as maintainable as a pure d codebase. A less stable/complete linker that attracts more contributors should overtake a more stable linker with only a couple of developers that grok it.

That's true, but we don't have that other linker yet. The other thing is, we just don't have a need for our own linker for any platform other than win32. So what's the cost benefit moving forward? I think it's easier to just fix optlink's bugs.

You're probably right, but it would still be awesome to have our own modern linker. Just being able to use coff import libraries directly on win32 would be very nice.
 I don't want to discourage people from trying to come up with a 
 replacement linker for win32 written in D. I think that is a great 
 project. But while a linker is a conceptually simple program, the awful 
 file formats involved make it unnecessarily difficult and there are simply 
 a lot of details and other things one has to do.

 Like I said before, it'll take a sustained and determined effort to come 
 up with a viable replacement for optlink.

Agreed.
 What is the license on optlink?

Same as the dmd back end.
 Can other linkers actually use this information?

They can use the information, yes, but not the code.

That's interesting, I didn't realise this.
Mar 07 2013
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 3/7/2013 8:16 PM, Brad Roberts wrote:
 Personally, even though I don't use win32, I believe that moving it over to
use the VS toolchain and runtime is the
 right path forward.

I like being able to provide a completely free D toolchain for win32.
Mar 07 2013
parent Rainer Schuetze <r.sagitario gmx.de> writes:
On 08.03.2013 08:25, Rikki Cattermole wrote:
 On Friday, 8 March 2013 at 07:17:24 UTC, Walter Bright wrote:
 On 3/7/2013 8:16 PM, Brad Roberts wrote:
 Personally, even though I don't use win32, I believe that moving it
 over to use the VS toolchain and runtime is the
 right path forward.

I like being able to provide a completely free D toolchain for win32.

The express edition of Visual Studio should be enough to provide e.g. the linker. http://www.microsoft.com/visualstudio/eng/downloads#d-express-windows-desktop It is free. However you do need to register after 30 days.

There are also C++ compiler and linker in the Windows 7 SDK that you can download and install without registering (http://www.microsoft.com/en-us/download/details.aspx?id=8442). It might be fun, but I think that building your own COFF capable linker will be a huge undertaking if you want to link against C/C++ code built with the MS compiler without stripping a whole lot of information (debug information, FPO records, etc).
Mar 08 2013
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-03-08 04:41, Walter Bright wrote:

 The other thing is, we just don't have a need for our own linker for any
 platform other than win32. So what's the cost benefit moving forward? I
 think it's easier to just fix optlink's bugs.

If we have our own linker for every supported platform and format I pretty sure that we can take advantage of that and come up with features not possible using other linkers. For example, storing .di files directly in the object file. This has also been suggested before. -- /Jacob Carlborg
Mar 08 2013
next sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
08-Mar-2013 12:50, Jacob Carlborg пишет:
 On 2013-03-08 04:41, Walter Bright wrote:

 The other thing is, we just don't have a need for our own linker for any
 platform other than win32. So what's the cost benefit moving forward? I
 think it's easier to just fix optlink's bugs.

If we have our own linker for every supported platform and format I pretty sure that we can take advantage of that and come up with features not possible using other linkers.

Yes, along with storing meta-data such as pure, nothrow etc. Mangling does it but in a such a backward way... But it would always have to maintain backwards compatibility or simple conversion to this format.
 For example, storing .di files directly in the object file. This has
 also been suggested before.

-- Dmitry Olshansky
Mar 08 2013
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 3/8/2013 12:50 AM, Jacob Carlborg wrote:
 If we have our own linker for every supported platform and format I pretty sure
 that we can take advantage of that and come up with features not possible using
 other linkers.

 For example, storing .di files directly in the object file. This has also been
 suggested before.

Sure, but who is going to do the work? BTW, we did come up with our librarian (it's built in to dmd) and it works great - enabling simple, smooth handling of libraries in a way that no other similar language does. Notice, though, there are 4 completely distinct code bases to support it, for the 4 main platforms. The librarian is a simple program, but linkers are much more complex.
Mar 08 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-03-08 09:58, Walter Bright wrote:

 Sure, but who is going to do the work?

Anyone that's interested in doing the work. But we don't have to invest all our time doing it. Daniel Murphy seems to be working on a linker, so apparently there is at least some interest.
 BTW, we did come up with our librarian (it's built in to dmd) and it
 works great - enabling simple, smooth handling of libraries in a way
 that no other similar language does. Notice, though, there are 4
 completely distinct code bases to support it, for the 4 main platforms.
 The librarian is a simple program, but linkers are much more complex.

Yeah, I didn't say it was going to be easy. -- /Jacob Carlborg
Mar 08 2013
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 3/8/2013 11:29 PM, Nick Sabalausky wrote:
 So if they *can* be so simple, why should everyone have
 to continue to stay stuck with a crappy one?

That's a good question. It's surprising how excessively complicated most designs are.
Mar 08 2013
prev sibling next sibling parent "Andrej Mitrovic" <andrej.mitrovich gmail.com> writes:
On Thursday, 7 March 2013 at 19:26:49 UTC, Walter Bright wrote:
 If anyone is interested I'll put it up on github.

There's much more to making a linker. There's the debug info (which you mentioned), then there are the resource files, the module definition files, and lastly the myriad of rather complex switches that are expected. Almost none of that is tested by the dmd test suite.

Nevertheless having a linker written in D goes hand-in-hand with having a compiler written in D. I'd be interested in seeing his work. I'd also be interested in having something that's documented and that isn't bound by licensing issues. We also have the abandoned DDL[1] project which had a linker that supports OMF, and Eric Anderton was nice enough to document his process of implementing it: http://www.dsource.org/forums/viewtopic.php?t=959 [1] http://dsource.org/projects/ddl/
Mar 07 2013
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-03-07 13:35, Daniel Murphy wrote:

 If anyone is interested I'll put it up on github.

I would say put it there to see how much interest there is. -- /Jacob Carlborg
Mar 07 2013
parent "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Jacob Carlborg"  wrote in message news:khaqhb$on0$1 digitalmars.com... 

On 2013-03-07 13:35, Daniel Murphy wrote:

 If anyone is interested I'll put it up on github.

I would say put it there to see how much interest there is.

Better late than never... https://github.com/yebblies/ylink
Mar 23 2014
prev sibling next sibling parent "Andrej Mitrovic" <andrej.mitrovich gmail.com> writes:
On Thursday, 7 March 2013 at 20:01:58 UTC, Walter Bright wrote:
 What we *can* do with Optlink is fix the occasional bug that 
 crops up in it.

I hope so. :) On Thursday, 7 March 2013 at 20:01:58 UTC, Walter Bright wrote:
 I also believe that even if Optlink were completely converted 
 to C, it will still be intractably difficult to refactor it to 
 do things like support other object file formats.

I really wish Unilink[1] was open-source. It's apparently a student project, and it's closed source but freeware. It works with OMF but can also link OMF and COFF objects together, which is a boon for win32 programmers who don't want to go through OBJCONV (object file conversion, doesn't always work) or COFF2OMF (import libs). It also has pretty good error messages. Of course there's no telling how big its test-suite is. It's fairly easy to set up though[2]. [1] ftp://ftp.styx.cabel.net/pub/UniLink/ [2] http://forum.dlang.org/thread/jbvugp$8l3$1 digitalmars.com#post-mailman.1414.1323538292.24802.digitalmars-d:40puremagic.com On Thursday, 7 March 2013 at 20:01:58 UTC, Walter Bright wrote:
 This is why I had started a new linker project.

Interesting. But if it's made behind-the-scenes with no input or work from the community I won't be holding my hopes up too much, I know you have very limited time for side-projects like that.
Mar 07 2013
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 3/7/13, Walter Bright <newshound2 digitalmars.com> wrote:
 1. all the supported file formats were completely encapsulated and
 modularized
 For some idea of how that can work, see libmscoff.c and scanmscoff.c

DDL seems to do a decent job of abstracting things, e.g.: OMF loader: http://dsource.org/projects/ddl/browser/trunk/ddl/omf COFF loader: http://dsource.org/projects/ddl/browser/trunk/ddl/coff And the root dir: http://dsource.org/projects/ddl/browser/trunk/ddl Anyway I'm not suggesting it's a complete implementation, but it might be worth taking a look. It's BSD licensed, hopefully that means it's ok to look at and maybe extract some useful code or ideas from it.
From the source comments it appears Don Clugston also worked on this

Mar 07 2013
prev sibling next sibling parent Brad Roberts <braddr puremagic.com> writes:
On 3/7/2013 7:41 PM, Walter Bright wrote:
 On 3/7/2013 7:27 PM, Daniel Murphy wrote:
 "Walter Bright" <newshound2 digitalmars.com> wrote in message
 news:khblbe$27f5$1 digitalmars.com...
 On 3/7/2013 7:09 PM, Daniel Murphy wrote:
 That's correct. However, it'll be much more maintainable,

I don't know how much redesign you're planning, but I can't imagine it ever being as maintainable as a pure d codebase. A less stable/complete linker that attracts more contributors should overtake a more stable linker with only a couple of developers that grok it.

That's true, but we don't have that other linker yet. The other thing is, we just don't have a need for our own linker for any platform other than win32. So what's the cost benefit moving forward? I think it's easier to just fix optlink's bugs. I don't want to discourage people from trying to come up with a replacement linker for win32 written in D. I think that is a great project. But while a linker is a conceptually simple program, the awful file formats involved make it unnecessarily difficult and there are simply a lot of details and other things one has to do. Like I said before, it'll take a sustained and determined effort to come up with a viable replacement for optlink.

Personally, even though I don't use win32, I believe that moving it over to use the VS toolchain and runtime is the right path forward.
Mar 07 2013
prev sibling next sibling parent "Rikki Cattermole" <alphaglosined gmail.com> writes:
On Friday, 8 March 2013 at 04:17:03 UTC, Brad Roberts wrote:
 On 3/7/2013 7:41 PM, Walter Bright wrote:
 On 3/7/2013 7:27 PM, Daniel Murphy wrote:
 "Walter Bright" <newshound2 digitalmars.com> wrote in message
 news:khblbe$27f5$1 digitalmars.com...
 On 3/7/2013 7:09 PM, Daniel Murphy wrote:
 That's correct. However, it'll be much more maintainable,

I don't know how much redesign you're planning, but I can't imagine it ever being as maintainable as a pure d codebase. A less stable/complete linker that attracts more contributors should overtake a more stable linker with only a couple of developers that grok it.

That's true, but we don't have that other linker yet. The other thing is, we just don't have a need for our own linker for any platform other than win32. So what's the cost benefit moving forward? I think it's easier to just fix optlink's bugs. I don't want to discourage people from trying to come up with a replacement linker for win32 written in D. I think that is a great project. But while a linker is a conceptually simple program, the awful file formats involved make it unnecessarily difficult and there are simply a lot of details and other things one has to do. Like I said before, it'll take a sustained and determined effort to come up with a viable replacement for optlink.

Personally, even though I don't use win32, I believe that moving it over to use the VS toolchain and runtime is the right path forward.

I've actually modified dmd's frontend to support using the VS tool chain for linking 32 bit executables. But I didn't understand the backend to seperate the 32bit binary generation vs 64bit. I believe it shouldn't be too much. But I really don't understand it so if anyone is willing to help on this. I haven't broken 64bit generation as far as I know. Because I added a cli argument (-msvc) which is enabled for -m64 automatically on Windows.
Mar 07 2013
prev sibling next sibling parent "Rikki Cattermole" <alphaglosined gmail.com> writes:
On Friday, 8 March 2013 at 07:17:24 UTC, Walter Bright wrote:
 On 3/7/2013 8:16 PM, Brad Roberts wrote:
 Personally, even though I don't use win32, I believe that 
 moving it over to use the VS toolchain and runtime is the
 right path forward.

I like being able to provide a completely free D toolchain for win32.

The express edition of Visual Studio should be enough to provide e.g. the linker. http://www.microsoft.com/visualstudio/eng/downloads#d-express-windows-desktop It is free. However you do need to register after 30 days.
Mar 07 2013
prev sibling next sibling parent "Jakob Ovrum" <jakobovrum gmail.com> writes:
On Friday, 8 March 2013 at 05:29:14 UTC, Rikki Cattermole wrote:
 I've actually modified dmd's frontend to support using the VS 
 tool chain for linking 32 bit executables. But I didn't 
 understand the backend to seperate the 32bit binary generation 
 vs 64bit. I believe it shouldn't be too much. But I really 
 don't understand it so if anyone is willing to help on this.
 I haven't broken 64bit generation as far as I know. Because I 
 added a cli argument (-msvc) which is enabled for -m64 
 automatically on Windows.

I'd love to have an option like that. Win32 is at the very least going to be a target for many years to come (for some also a development platform) and the OPTLINK/OMF business is a real impediment.
Mar 08 2013
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 3/8/13, Walter Bright <newshound2 digitalmars.com> wrote:
 On 3/7/2013 8:16 PM, Brad Roberts wrote:
 Personally, even though I don't use win32, I believe that moving it over
 to use the VS toolchain and runtime is the
 right path forward.

I like being able to provide a completely free D toolchain for win32.

s/for win32/./ :) Honestly D has huge potential for tools like compilers/linkers/etc. They're the types of programs where you practically know all your requirements at compile-time, so e.g. generics come into play really nicely. We should keep the idea of having a free D toolchain for all OSes in the back of our heads, even if we don't work on it any time soon.
Mar 08 2013
prev sibling next sibling parent "Rikki Cattermole" <alphaglosined gmail.com> writes:
On Friday, 8 March 2013 at 13:02:06 UTC, Jakob Ovrum wrote:
 On Friday, 8 March 2013 at 05:29:14 UTC, Rikki Cattermole wrote:
 I've actually modified dmd's frontend to support using the VS 
 tool chain for linking 32 bit executables. But I didn't 
 understand the backend to seperate the 32bit binary generation 
 vs 64bit. I believe it shouldn't be too much. But I really 
 don't understand it so if anyone is willing to help on this.
 I haven't broken 64bit generation as far as I know. Because I 
 added a cli argument (-msvc) which is enabled for -m64 
 automatically on Windows.

I'd love to have an option like that. Win32 is at the very least going to be a target for many years to come (for some also a development platform) and the OPTLINK/OMF business is a real impediment.

Personally I think of it as a barrier for adoption. A real BIG one for casual devs, which is what we need to grow. It also limits us from getting it into e.g. high schools. Which may at best be using core 2 duo's. Hence why I started working on it. Got so sick of not having it. Just wish my knowledge extended to how the backend worked because then I could have finished this weeks ago.
Mar 08 2013
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Friday, 8 March 2013 at 17:40:58 UTC, Andrej Mitrovic wrote:
 Honestly D has huge potential for tools like 
 compilers/linkers/etc.
 They're the types of programs where you practically know all 
 your
 requirements at compile-time, so e.g. generics come into play 
 really
 nicely.

That is the theory. In practice you'll run in MANY problems.
Mar 08 2013
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 3/9/13, deadalnix <deadalnix gmail.com> wrote:
 On Friday, 8 March 2013 at 17:40:58 UTC, Andrej Mitrovic wrote:
 Honestly D has huge potential for tools like
 compilers/linkers/etc.
 They're the types of programs where you practically know all
 your
 requirements at compile-time, so e.g. generics come into play
 really
 nicely.

That is the theory. In practice you'll run in MANY problems.

In practice I'm getting quite fed up with linker bugs, or DMD object gen bugs which cause the linker to choke. :)
Mar 09 2013
prev sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Friday, 8 March 2013 at 17:40:58 UTC, Andrej Mitrovic wrote:
 On 3/8/13, Walter Bright <newshound2 digitalmars.com> wrote:
 On 3/7/2013 8:16 PM, Brad Roberts wrote:
 Personally, even though I don't use win32, I believe that 
 moving it over
 to use the VS toolchain and runtime is the
 right path forward.

I like being able to provide a completely free D toolchain for win32.

s/for win32/./ :) Honestly D has huge potential for tools like compilers/linkers/etc. They're the types of programs where you practically know all your requirements at compile-time, so e.g. generics come into play really nicely.

Personally I prefer FP based languages for this type of work, given how easy it is to manipulate ASTs. -- Paulo
Mar 14 2013
prev sibling next sibling parent Marco Leise <Marco.Leise gmx.de> writes:
On Thu, Mar 07, 2013 at 07:09:22AM +0100, Lars T. Kyllingstad
wrote:

 This reminds me of http://www.catb.org/jargon/html/story-of-mel.html

Good story! Imagine how Mel must have known the entire program by heart and he never got any change requests that are so common today. I would have been addicted to that, too. But luckily 1975 was 8 years before my birth and so I started "hacking" with QBasic Gorillaz, a turn based strategy game. I guess with SSE it still today makes sense to count instruction delays and throughput. Also take a look at this: http://research.scee.net/files/presentations/gcapaustralia09/Pitfalls_of_Object_Oriented_Programming_GCAP_09.pdf I don't think people like this programmer Mel are job-less today. -- Marco
Mar 07 2013
prev sibling next sibling parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Thu, 7 Mar 2013 23:35:52 +1100
schrieb "Daniel Murphy" <yebblies nospamgmail.com>:

 As a side note I actually wrote an omf (dmd-emitted subset) linker last year 
 that gets through almost all of the dmd test suite.  It has no debug info 
 support, which sucks, but it may work in some cases where optlink can't 
 cope.  It is fairly naive performance-wise, but is written in a much 
 friendlier language.

I wonder what would happen if someone plugged that linker right into the dmd D port to skip the external linking phase alltogether... -- Marco
Mar 07 2013
parent "Daniel Murphy" <yebblies nospamgmail.com> writes:
"Marco Leise" <Marco.Leise gmx.de> wrote in message 
news:20130307154744.206d9bb0 marco-leise...
 Am Thu, 7 Mar 2013 23:35:52 +1100
 schrieb "Daniel Murphy" <yebblies nospamgmail.com>:

 As a side note I actually wrote an omf (dmd-emitted subset) linker last 
 year
 that gets through almost all of the dmd test suite.  It has no debug info
 support, which sucks, but it may work in some cases where optlink can't
 cope.  It is fairly naive performance-wise, but is written in a much
 friendlier language.

I wonder what would happen if someone plugged that linker right into the dmd D port to skip the external linking phase alltogether... -- Marco

Performance++ Memory_Usage++ Even optlink has/once had a library version. llvm's lld is an interesting project, and adding an omf reader could solve our linker problems once it's more mature. Linkers are fun, corrupt executables are really in their own special class when it comes to nasty bugs.
Mar 07 2013
prev sibling next sibling parent "Jesse Phillips" <Jesse.K.Phillips+D gmail.com> writes:
On Thursday, 7 March 2013 at 15:53:09 UTC, Denis Shelomovskij 
wrote:
 Didn't get. You don't have to use D with druntime. Just don't 
 link it and everything will be OK - you will just get "better 
 C" (i.e. with D structs and other good stuff).

Walter wrote all about it: http://www.drdobbs.com/architecture-and-design/assembler-to-c/228701319
Mar 08 2013
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 7 March 2013 at 01:25:02 UTC, Walter Bright wrote:
 Some months ago, I did make the source to optlink available on 
 github:

 https://github.com/DigitalMars/optlink

 Rainer Schuetze has improved it where it can be built with 
 modern tools (the older tools would not run on Win7). I know 
 some of you are frustrated by optlink problems. Well, now you 
 can do something about it!

 Note that the optlink source is quite a challenge to work on. 
 It's very old skool (80's style).

 Here's an approach that I've used successfully to fix optlink 
 seg faults:

 1. Find out where in the source code it is. This is not so 
 easy, even if using a debugger. The trouble is the asm 
 functions do not have standard stack frames, so the debugger 
 cannot do a stack trace. Even where there is a standard stack 
 frame, the modern Microsoft debuggers fail to recognize them. 
 You gotta use an older debugger under Windows XP.

 So, what I do, is look at the asm code where the seg fault 
 occurs, and then grep through the asm source till I find it.

 2. Convert the source file where the seg fault occurs to C. I 
 do this one function at a time, running the full set of tests 
 at each step. Otherwise, it's just impossible to figure out 
 where you made a mistake in the conversion. Sometimes, you 
 gotta go even finer grained - using the inline assembler 
 feature of dmc to convert asm code line by line.

 You gotta pay very, very close attention to which registers 
 have parameters passed through them, which registers are saved, 
 and which registers have parameters that are silently passed 
 through a function to callees of that function (!). This 
 information is all pretty much utterly lacking in the comments, 
 and the comments are often dead wrong as they refer to much 
 older versions of the code.

 3. Once it is in C, things get a lot easier. You can, for 
 example, insert printf's to figure out where things go wrong, 
 and *then* fix it.


 The full test suite for optlink has a lot of stuff I cannot 
 publish on github. However, what you can do is run the win32 
 tests for dmd. I, of course, will run the full one when 
 verifying pulls.

 Happy hacking!

Most people would consider this an exercise in self-flagellation, but I enjoy doing it. http://www.drdobbs.com/architecture-and-design/assembler-to-c/228701319
Mar 08 2013
prev sibling next sibling parent Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On Thu, 07 Mar 2013 19:41:10 -0800
Walter Bright <newshound2 digitalmars.com> wrote:
 
 I don't want to discourage people from trying to come up with a
 replacement linker for win32 written in D. I think that is a great
 project. But while a linker is a conceptually simple program, the
 awful file formats involved make it unnecessarily difficult and there
 are simply a lot of details and other things one has to do.
 
 Like I said before, it'll take a sustained and determined effort to
 come up with a viable replacement for optlink.
 

Interesting thing about this, is that if/when we did have this "new linker" as discussed (including separation of individual object file formats) is that we could also add in our own object file format designed (and proven via a good test suite) to be much less quirky and badly documented. Or, you know, not quirky and not badly documented ;) Thus, at the very least, opening a door for others to migrate, and potentially/finally/eventually get rid of the crufty current formats. I mean, like you said, linkers are very simple conceptually. So if they *can* be so simple, why should everyone have to continue to stay stuck with a crappy one? Just my random thoughts on it anyway.
Mar 08 2013
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Thu, 07 Mar 2013 19:53:29 +0400
schrieb Denis Shelomovskij <verylonglogin.reg gmail.com>:

 07.03.2013 14:28, Jacob Carlborg =D0=BF=D0=B8=D1=88=D0=B5=D1=82:
 On 2013-03-07 11:12, Moritz Maxeiner wrote:

 Sorry if this has been answered before/is common knowledge, but is
 porting functions at a time to C wanted for optlink in general, or
 only for finding segfaults? (e.g. are pull-requests for that
 welcome)

Yes, in general. I think the idea is to port the whole Optlink to C and then to D. It's easier to port from C to D than from assembly to D. This is because you can use a C version that doesn't use the runtime or standard library to get the generate assembly as close as possible to the original one. I think it was something like that.

Didn't get. You don't have to use D with druntime. Just don't link it=20 and everything will be OK - you will just get "better C" (i.e. with D=20 structs and other good stuff). =20

That's a little bit oversimplified. Even a simple POD struct references the TypeInfo_Struct declaration in druntime. There's still some compiler work needed to really make D usable without druntime. (Think of issues like these: If you don't have druntime -> you don't have typeinfo -> no D style varargs -> no associative arrays; you can't even compare normal arrays (https://github.com/D-Programming-Language/druntime/b= lob/e47a00bff935c3f079bb567a6ec97663ba384487/src/rt/adi.d#L368))
Mar 09 2013
prev sibling next sibling parent "Dicebot" <m.strashun gmail.com> writes:
On Saturday, 9 March 2013 at 09:11:54 UTC, Johannes Pfau wrote:
 Am Thu, 07 Mar 2013 19:53:29 +0400
 schrieb Denis Shelomovskij <verylonglogin.reg gmail.com>:

 07.03.2013 14:28, Jacob Carlborg пишет:
 On 2013-03-07 11:12, Moritz Maxeiner wrote:

 Sorry if this has been answered before/is common knowledge, 
 but is
 porting functions at a time to C wanted for optlink in 
 general, or
 only for finding segfaults? (e.g. are pull-requests for that
 welcome)

Yes, in general. I think the idea is to port the whole Optlink to C and then to D. It's easier to port from C to D than from assembly to D. This is because you can use a C version that doesn't use the runtime or standard library to get the generate assembly as close as possible to the original one. I think it was something like that.

Didn't get. You don't have to use D with druntime. Just don't link it and everything will be OK - you will just get "better C" (i.e. with D structs and other good stuff).

That's a little bit oversimplified. Even a simple POD struct references the TypeInfo_Struct declaration in druntime. There's still some compiler work needed to really make D usable without druntime. (Think of issues like these: If you don't have druntime -> you don't have typeinfo -> no D style varargs -> no associative arrays; you can't even compare normal arrays (https://github.com/D-Programming-Language/druntime/blob/e47a00bff935c3f079bb567a6ec97663ba384487/src/rt/adi.d#L368))

It can be easily checked with "-betterC -defaultlib=" combo. You can skip phobos, druntime, use only C stdlib and look what will remain working in the language. Not much.
Mar 09 2013
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Sat, 09 Mar 2013 10:32:18 +0100
schrieb "Dicebot" <m.strashun gmail.com>:

 On Saturday, 9 March 2013 at 09:11:54 UTC, Johannes Pfau wrote:
 Am Thu, 07 Mar 2013 19:53:29 +0400
 schrieb Denis Shelomovskij <verylonglogin.reg gmail.com>:

 07.03.2013 14:28, Jacob Carlborg =D0=BF=D0=B8=D1=88=D0=B5=D1=82:
 On 2013-03-07 11:12, Moritz Maxeiner wrote:

 Sorry if this has been answered before/is common knowledge,=20
 but is
 porting functions at a time to C wanted for optlink in=20
 general, or
 only for finding segfaults? (e.g. are pull-requests for that
 welcome)

Yes, in general. I think the idea is to port the whole=20 Optlink to C and then to D. It's easier to port from C to D than from=20 assembly to D. This is because you can use a C version that doesn't=20 use the runtime or standard library to get the generate assembly as=20 close as possible to the original one. I think it was something=20 like that.

Didn't get. You don't have to use D with druntime. Just don't=20 link it and everything will be OK - you will just get "better=20 C" (i.e. with D structs and other good stuff). =20

That's a little bit oversimplified. Even a simple POD struct=20 references the TypeInfo_Struct declaration in druntime. There's still some compiler work needed to really make D usable without druntime.=20 (Think of issues like these: If you don't have druntime -> you don't=20 have typeinfo -> no D style varargs -> no associative arrays; you=20 can't even compare normal arrays=20 (https://github.com/D-Programming-Language/druntime/blob/e47a00bff935c3=


=20
 It can be easily checked with "-betterC -defaultlib=3D" combo. You=20
 can skip phobos, druntime, use only C stdlib and look what will=20
 remain working in the language. Not much.

I know that, I've run D code on the DS without druntime and on Android as well, but betterC only disables moduleinfo, not typeinfo. So you don't even get structs without druntime and that is certainly not better than C. test.d: ------- struct A { } ------- dmd -betterC -defaultlib=3D -c test.d nm test.o U _D15TypeInfo_Struct6__vtblZ
Mar 09 2013
prev sibling next sibling parent "Dicebot" <m.strashun gmail.com> writes:
Ye, sure, I wanted to support your point with practical example, 
not object it by any means. By the way, templates and array 
literals I have tried were doomed, too :)
Mar 09 2013
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Sat, 09 Mar 2013 11:35:42 +0100
schrieb "Dicebot" <m.strashun gmail.com>:

 Ye, sure, I wanted to support your point with practical example, 
 not object it by any means. By the way, templates and array 
 literals I have tried were doomed, too :)

Really, templates also don't work? There's a lot of work required till betterC is finished...
Mar 09 2013
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Sat, 09 Mar 2013 11:35:42 +0100
schrieb "Dicebot" <m.strashun gmail.com>:

 Ye, sure, I wanted to support your point with practical example, 
 not object it by any means. By the way, templates and array 
 literals I have tried were doomed, too :)

Sorry, I misunderstood that then.
Mar 09 2013
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 3/7/13, Daniel Murphy <yebblies nospamgmail.com> wrote:
 If anyone is interested I'll put it up on github.

I'd be interested, please do put it online when you get the time.
Mar 09 2013
prev sibling next sibling parent "Ralf" <r.a.quint gmail.com> writes:
On Thursday, 7 March 2013 at 06:34:05 UTC, H. S. Teoh wrote:
 Man, that brings back the memories. I remember doing those 
 kinds of
 stunts when coding for the good ole Motorola 6502. Back in 
 those days,

MOS Technologies (though founded by former Motorola employees) was a competitor to Motorola (which had released the 6800, which was similar but 10x as expensive) and was sued by Motorola at some point for patent infringements just before being bought up by Commodore... Ralf
Mar 13 2013
prev sibling next sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Thursday, 7 March 2013 at 14:42:25 UTC, Marco Leise wrote:
 On Thu, Mar 07, 2013 at 07:09:22AM +0100, Lars T. Kyllingstad
 wrote:

 This reminds me of 
 http://www.catb.org/jargon/html/story-of-mel.html

Good story! Imagine how Mel must have known the entire program by heart and he never got any change requests that are so common today. I would have been addicted to that, too. But luckily 1975 was 8 years before my birth and so I started "hacking" with QBasic Gorillaz, a turn based strategy game.

 I guess with SSE it still today makes sense to count
 instruction delays and throughput. Also take a look at this:
 http://research.scee.net/files/presentations/gcapaustralia09/Pitfalls_of_Object_Oriented_Programming_GCAP_09.pdf
 I don't think people like this programmer Mel are job-less
 today.

True, although I think this type of work is very complicated to do nowadays given how modern architectures work. Only in simple CPU architectures it is possible to know the effect each instruction still has, whereas with modern ones you have to use some kind of profiler/code analyzer even for Assembly code, given the multiple level caches, parallel execution of instructions, instruction re-ordering and so on. -- Paulo
Mar 14 2013
prev sibling next sibling parent reply "Asman01" <jckj33 gmail.com> writes:
On Thursday, 7 March 2013 at 01:25:02 UTC, Walter Bright wrote:
 Some months ago, I did make the source to optlink available on 
 github:

 https://github.com/DigitalMars/optlink

 Rainer Schuetze has improved it where it can be built with 
 modern tools (the older tools would not run on Win7). I know 
 some of you are frustrated by optlink problems. Well, now you 
 can do something about it!

 Note that the optlink source is quite a challenge to work on. 
 It's very old skool (80's style).

 Here's an approach that I've used successfully to fix optlink 
 seg faults:

 1. Find out where in the source code it is. This is not so 
 easy, even if using a debugger. The trouble is the asm 
 functions do not have standard stack frames, so the debugger 
 cannot do a stack trace. Even where there is a standard stack 
 frame, the modern Microsoft debuggers fail to recognize them. 
 You gotta use an older debugger under Windows XP.

 So, what I do, is look at the asm code where the seg fault 
 occurs, and then grep through the asm source till I find it.

 2. Convert the source file where the seg fault occurs to C. I 
 do this one function at a time, running the full set of tests 
 at each step. Otherwise, it's just impossible to figure out 
 where you made a mistake in the conversion. Sometimes, you 
 gotta go even finer grained - using the inline assembler 
 feature of dmc to convert asm code line by line.

 You gotta pay very, very close attention to which registers 
 have parameters passed through them, which registers are saved, 
 and which registers have parameters that are silently passed 
 through a function to callees of that function (!). This 
 information is all pretty much utterly lacking in the comments, 
 and the comments are often dead wrong as they refer to much 
 older versions of the code.

 3. Once it is in C, things get a lot easier. You can, for 
 example, insert printf's to figure out where things go wrong, 
 and *then* fix it.


 The full test suite for optlink has a lot of stuff I cannot 
 publish on github. However, what you can do is run the win32 
 tests for dmd. I, of course, will run the full one when 
 verifying pulls.

 Happy hacking!

Is this the current linker used by DMD on all platforms?
Mar 23 2014
parent "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Asman01"  wrote in message news:klvtbihnvoxfarsjdlhv forum.dlang.org... 

 Is this the current linker used by DMD on all platforms?

No, only when building win32 executables.
Mar 23 2014
prev sibling next sibling parent reply "Asman01" <jckj33 gmail.com> writes:
Very noob question about binary files. What else also put the 
code to load at right address (say, 0x08048000 on linux) of 
operating system is needed to a program run?
Mar 23 2014
parent "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Asman01"  wrote in message news:ucqujzetvkkxzelvjleq forum.dlang.org...

 Very noob question about binary files. What else also put the code to load 
 at right address (say, 0x08048000 on linux) of operating system is needed 
 to a program run?

Not really sure what you're asking, but the executable file usually contains an address for each section (code, data, bss, etc) and the runtime loader will choose to load it there or somewhere else (eg aslr/shared libraries) and map the data into virtual memory.
Mar 23 2014
prev sibling next sibling parent "Asman01" <jckj33 gmail.com> writes:
On Saturday, 9 March 2013 at 05:22:31 UTC, Jesse Phillips wrote:
 On Thursday, 7 March 2013 at 15:53:09 UTC, Denis Shelomovskij 
 wrote:
 Didn't get. You don't have to use D with druntime. Just don't 
 link it and everything will be OK - you will just get "better 
 C" (i.e. with D structs and other good stuff).

Walter wrote all about it: http://www.drdobbs.com/architecture-and-design/assembler-to-c/228701319

Why isn't this link here(http://www.drdobbs.com/author/Walter-Bright) ? are there more Walter's articles not listed in this page? if so, I want to know. Please :)
Mar 23 2014
prev sibling next sibling parent "Asman01" <jckj33 gmail.com> writes:
On Sunday, 23 March 2014 at 16:10:48 UTC, Daniel Murphy wrote:
 "Asman01"  wrote in message 
 news:ucqujzetvkkxzelvjleq forum.dlang.org...

 Very noob question about binary files. What else also put the 
 code to load at right address (say, 0x08048000 on linux) of 
 operating system is needed to a program run?

Not really sure what you're asking, but the executable file usually contains an address for each section (code, data, bss, etc) and the runtime loader will choose to load it there or somewhere else (eg aslr/shared libraries) and map the data into virtual memory.

I think I was confusing this to static executables which load at 0x08048000 address and start running from value your set in entry point.
Mar 23 2014
prev sibling parent "Alex Ogheri" <ogheri alessandroogheri.com> writes:
On Thursday, 7 March 2013 at 03:06:48 UTC, bearophile wrote:
 Walter Bright:

 Happy hacking!

Extra karma points if done blindfolded, using a Braille tablet :-) Bye, bearophile

As far as I know, Walter DOES IT blindfolded, using a Braille tablet and while he writes 3 other compilers at the same time... (but the same has been said of Linus Torvalds, or Richard Stallman, so perhaps its a myth...)
Apr 09 2014