www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Doesn't DMD as a library already exist without an external dub

reply Andrew Edwards <edwards.ac gmail.com> writes:
My naïveté often gets the best of me, so if I seem to be jumping 
off the deep end, please pull me back before I hurt myself.

I've been wondering why so much effort is necessary to make DMD 
available as a library. Isn't DMD already a library? If you 
remove the module containing main(), all you've got left are 
library files. So why not merely expose them to the masses and 
call it a day?

Thanks to  alphaglosined on IRC, I started thinking that to make 
dmd accessible as a library, all that's required is to provide a 
libdmd.a file in the appropriate directory and expose the dmd 
directory for use with import statements.
Today I did just that to gain access to the lexer and can use it 
without any problems.

Here's what I did:

1) modify dmd.conf to include -I% P%/../../src/dmd

2) navigate to src/dmd/dmd and execute the command
    dmd -lib -J../../../src -of=libdmd.a {root/*, console, \
entity, errors, filecache, globals, id, identifier, \
lexer, tokens, utf, utils, astbase, parse, transitivevisitor, \
permissivevisitor, strictvisitor}.d

2) move the file to the lib folder: mv libdmd.a 
~/dlang/dmd-2.093.0/osx/lib/

The result? You be the judge:

(dmd-2.093.0)edwarac rukongai ~> dmd diced         //[1]
(dmd-2.093.0)edwarac rukongai ~> ./diced diced.d
pragma
(
lib
,
"dmd"
)
;
import
dmd
.
lexer
;
import
dmd
.
tokens
;
import
std
.
file
;
import
std
.
stdio
;
import
std
.
conv
:
to
;
void
main
(
string
[
]
args
)
{
args
=
args
[
1
..
$
]
;
if
(
args
.
length


1
)
{
writeln
(
"Usage: diced [script]"
)
;
}
else
if
(
args
.
length
==
1
)
{
runFile
(
args
[
0
]
)
;
}
else
{
runPrompt
(
)
;
}
}
void
runFile
(
string
path
)
{
path
.
readText
.
run
;
}
void
runPrompt
(
)
{
for
(
;
;
)
{
write
(
"> "
)
;
auto
line
=
stdin
.
readln
(
)
;
if
(
line
is
null
)
break
;
run
(
line
)
;
}
}
void
run
(
string
source
)
{
scope
lex1
=
new
Lexer
(
null
,
source
.
ptr
,
0
,
source
.
length
,
0
,
0
)
;
TOK
tok
=
lex1
.
nextToken
(
)
;
while
(
tok
!=
TOK
.
endOfFile
)
{
writeln
(
lex1
.
token
.
toChars
(
)
.
to
!
string
)
;
tok
=
lex1
.
nextToken
(
)
;
}
}

Could it be that simple, or have I completely missed something? 
The compiler frontend is missing from this demo, but it shouldn't 
be too difficult to fix. So please let me know, other than that, 
what have I missed/misunderstood?

--Andrew

[1] I could have used -L-ldmd here but chose to use a pragma 
instead (both works)



 Sep 06 2020



 next sibling parent reply Andre Pany <andre s-e-a-p.de>  writes:


On Sunday, 6 September 2020 at 11:55:17 UTC, Andrew Edwards wrote:

 My naïveté often gets the best of me, so if I seem to be 
 jumping off the deep end, please pull me back before I hurt 
 myself.

 [...]


As far as I remember, the current version schema of dmd (2.0xx.x) 
isn't semver compatible but that will be solved soon with dmd 
2.100.0.

Kind regards
Andre



 Sep 06 2020



  parent reply Andrew Edwards <edwards.ac gmail.com>  writes:


On Sunday, 6 September 2020 at 12:48:30 UTC, Andre Pany wrote:

 On Sunday, 6 September 2020 at 11:55:17 UTC, Andrew Edwards 
 wrote:

 My naïveté often gets the best of me, so if I seem to be 
 jumping off the deep end, please pull me back before I hurt 
 myself.

 [...]


 As far as I remember, the current version schema of dmd 
 (2.0xx.x) isn't semver compatible but that will be solved soon 
 with dmd 2.100.0.

 Kind regards
 Andre


It's been a running complaint in the forum for over a decade now. 
I wasn't aware there were efforts underway to resolve the issue, 
though. Thanks for pointing that out. Just curious, how does 
semver (non-)compatibility prevent/aid in using the compiler as a 
library?

Thanks,
Andrew



 Sep 06 2020



  parent  Andre Pany <andre s-e-a-p.de>  writes:


On Sunday, 6 September 2020 at 13:13:43 UTC, Andrew Edwards wrote:

 On Sunday, 6 September 2020 at 12:48:30 UTC, Andre Pany wrote:

 On Sunday, 6 September 2020 at 11:55:17 UTC, Andrew Edwards 
 wrote:

 My naïveté often gets the best of me, so if I seem to be 
 jumping off the deep end, please pull me back before I hurt 
 myself.

 [...]


 As far as I remember, the current version schema of dmd 
 (2.0xx.x) isn't semver compatible but that will be solved soon 
 with dmd 2.100.0.

 Kind regards
 Andre


 It's been a running complaint in the forum for over a decade 
 now. I wasn't aware there were efforts underway to resolve the 
 issue, though. Thanks for pointing that out. Just curious, how 
 does semver (non-)compatibility prevent/aid in using the 
 compiler as a library?

 Thanks,
 Andrew


Afaik there is no active development regarding semver compatible 
of dmd. It is only a technical detail that 2.099.0 is not a valid 
semver version while 2.100.0 is. Therefore it could be added as 
dub package starting with 2.100.0.

Kind regards
Andre



 Sep 06 2020



prev sibling  parent reply Jacob Carlborg <doob me.com>  writes:


On 2020-09-06 13:55, Andrew Edwards wrote:

 My naïveté often gets the best of me, so if I seem to be jumping off the 
 deep end, please pull me back before I hurt myself.
 
 I've been wondering why so much effort is necessary to make DMD 
 available as a library. Isn't DMD already a library? If you remove the 
 module containing main(), all you've got left are library files. So why 
 not merely expose them to the masses and call it a day?


That's correct. And it's already available to the masses through Dub 
[1][2]. Or is your goal to avoiding using Dub?

If you're referring to the conversations you can see here from time to 
time about DMD as a library, it's usually about make it easier and more 
usable as a library. There are a lot of things in DMD that makes it 
difficult to use as a library.

[1] https://code.dlang.org/packages/dmd
[2] https://github.com/dlang/dmd/blob/master/dub.sdl

-- 
/Jacob Carlborg



 Sep 06 2020



  parent reply Andrew Edwards <edwards.ac gmail.com>  writes:


On Sunday, 6 September 2020 at 13:15:10 UTC, Jacob Carlborg wrote:

 On 2020-09-06 13:55, Andrew Edwards wrote:

 My naïveté often gets the best of me, so if I seem to be 
 jumping off the deep end, please pull me back before I hurt 
 myself.
 
 I've been wondering why so much effort is necessary to make 
 DMD available as a library. Isn't DMD already a library? If 
 you remove the module containing main(), all you've got left 
 are library files. So why not merely expose them to the masses 
 and call it a day?


 That's correct. And it's already available to the masses 
 through Dub [1][2]. Or is your goal to avoiding using Dub?


That is indeed the goal. I just want to install the compiler and 
automatically have this capability. I shouldn't need to use dub 
for this because the source code included with every compiler 
install.


 If you're referring to the conversations you can see here from 
 time to time about DMD as a library, it's usually about make it 
 easier and more usable as a library. There are a lot of things 
 in DMD that makes it difficult to use as a library.


Yes. Those are the conversations I am referring to. Ok got you, 
thanks for the explanation. I always thought it was simply that 
the user was not able to access the compiler for use in their 
code. Is there a list somewhere capturing the improvements 
desired or problems encountered in the as-a-library experience?



 Sep 06 2020



  parent  Jacob Carlborg <doob me.com>  writes:


On 2020-09-06 15:47, Andrew Edwards wrote:


 That is indeed the goal. I just want to install the compiler and 
 automatically have this capability. I shouldn't need to use dub for this 
 because the source code included with every compiler install.


You can have a look at the Dub file to see how it's built.


 Is there a list 
 somewhere capturing the improvements desired or problems encountered in 
 the as-a-library experience?


Not really. It's kind of spread out over newsgroup posts. Here are some 
of the problems:

* There's a lot of global state in the compiler. This might need to be 
reset when using the compiler in an IDE/editor LSP server

* There's no end location for tokens or AST nodes. This will make source 
code refactorings difficult/impossible

* Some AST nodes don't have location information at all

* Location information does not contain a buffer position/offset

* It's not possible to analyze source code that is only in memory. 
Imports and string imports will be looked up on disk

* The error handling/reporting is difficult to customize

* Information is lost between the various compilation phases

* The semantic analyzer is destructive

[1] https://github.com/dlang/dmd/blob/master/dub.sdl

-- 
/Jacob Carlborg



 Sep 10 2020