www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Type safety could prevent nuclear war

reply tsbockman <thomas.bockman gmail.com> writes:
The annual Underhanded C Contest announced their winners today.

As always, the results are very entertaining, and also an 
excellent advertisement for languages-that-are-not-C.

The first place entry is particularly ridiculous; is there any 
modern language that would make it so easy to commit such an 
awful "mistake"?

http://www.underhanded-c.org/#winner

Actually, I'm surprised that this works even in C - I would have 
expected at least a compiler (or linker?) warning; this seems 
like it should be easy to detect automatically.
Feb 04 2016
next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, Feb 04, 2016 at 10:57:00PM +0000, tsbockman via Digitalmars-d wrote:

 The annual Underhanded C Contest announced their winners today.
 
 As always, the results are very entertaining, and also an excellent
 advertisement for languages-that-are-not-C.
 
 The first place entry is particularly ridiculous; is there any modern
 language that would make it so easy to commit such an awful "mistake"?
 
 http://www.underhanded-c.org/#winner
 
 Actually, I'm surprised that this works even in C - I would have
 expected at least a compiler (or linker?) warning; this seems like it
 should be easy to detect automatically.


The C preprocessor accepts all sorts of nasty, nonsensical things. For
example, the following code compiles and runs (without any warning(!) on
my Linux box's standard gcc installation), and prints "No":

	#include <stdio.h>
	#define if(a) if(!(a))
	int main() {
		int i = 1;
		if (i == 1)
			printf("Yes\n");
		else
			printf("No\n");
	}

Imagine if this nasty #define is buried somewhere under several layers
of #include's.

I'm pretty sure somebody can also concoct some nasty #define that will
break the standard #include headers in horrible ways by changing the
semantics of certain supposedly-built-in constructs.


T

-- 
Mediocrity has been pushed to extremes.
Feb 04 2016
next sibling parent reply tsbockman <thomas.bockman gmail.com> writes:
On Thursday, 4 February 2016 at 23:10:23 UTC, H. S. Teoh wrote:

 On Thu, Feb 04, 2016 at 10:57:00PM +0000, tsbockman via 
 Digitalmars-d wrote:

 The annual Underhanded C Contest announced their winners today.
 
 As always, the results are very entertaining, and also an 
 excellent advertisement for languages-that-are-not-C.
 
 The first place entry is particularly ridiculous; is there any 
 modern language that would make it so easy to commit such an 
 awful "mistake"?
 
 http://www.underhanded-c.org/#winner
 
 Actually, I'm surprised that this works even in C - I would 
 have expected at least a compiler (or linker?) warning; this 
 seems like it should be easy to detect automatically.


 The C preprocessor accepts all sorts of nasty, nonsensical 
 things.


Definitely. What puzzles me about the winning entry, though, is 
that the compiler and/or linker should be able to trivially 
detect the type mismatch *after* the preprocessor pass(es) are 
already done.

It should just see that the post-preprocessor signatures of 
`spectral_contrast()` in match.c and spectral_contrast.c are in 
conflict, and either issue a warning, or refuse to link them at 
all.
Feb 04 2016
next sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Thursday, 4 February 2016 at 23:21:54 UTC, tsbockman wrote:

 Definitely. What puzzles me about the winning entry, though, is 
 that the compiler and/or linker should be able to trivially 
 detect the type mismatch *after* the preprocessor pass(es) are 
 already done.


Linkers don't know anything about types. A type is a language 
feature.


 It should just see that the post-preprocessor signatures of 
 `spectral_contrast()` in match.c and spectral_contrast.c are in 
 conflict, and either issue a warning, or refuse to link them at 
 all.


Has nothing to do with the preprocessor.

He defined float_t to be an alias for double in one compilation 
unit, and float_t to be an alias for float in another compilation 
unit.

In C, compilation units are completely independent, and can in 
fact come from different compilers and different languages. C is 
very much a system level programming language.
Feb 04 2016
next sibling parent reply tsbockman <thomas.bockman gmail.com> writes:
On Thursday, 4 February 2016 at 23:25:58 UTC, Ola Fosheim Grøstad 
wrote:

 On Thursday, 4 February 2016 at 23:21:54 UTC, tsbockman wrote:

 It should just see that the post-preprocessor signatures of 
 `spectral_contrast()` in match.c and spectral_contrast.c are 
 in conflict, and either issue a warning, or refuse to link 
 them at all.


 Has nothing to do with the preprocessor.


Yes, that was my point...


 He defined float_t to be an alias for double in one compilation 
 unit, and float_t to be an alias for float in another 
 compilation unit.

 In C, compilation units are completely independent, and can in 
 fact come from different compilers and different languages. C 
 is very much a system level programming language.


Just because *sometimes* the source code of the other module must 
be compiled independently, is a poor excuse to skip obvious, 
useful safety checks *all* the time.
Feb 04 2016
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Thursday, 4 February 2016 at 23:35:46 UTC, tsbockman wrote:

 Just because *sometimes* the source code of the other module 
 must be compiled independently, is a poor excuse to skip 
 obvious, useful safety checks *all* the time.


The context is a compilation system for building big software on 
very slow CPUs with kilobytes of RAM.

C was designed for always compiling independently and compiling 
source files that are bigger than what can be held in RAM, and 
also for building executables that can fill most of system RAM. 
So the compilation system was designed for using external memory 
(disk) and that affects C a lot. The forerunner for C, BCPL was a 
bootstrap language for writing compilers. So C is minimal by 
design.

BTW, C++ programmers sometimes use similar unsafe hacks of 
"pruned header files" to break dependencies and speed up 
compilation. So this is not unique to C, but C++ introduced the 
mangling of types into names to support overloading of functions 
on parameter types, which is why C++ detects (some) type issues 
at link time.
Feb 04 2016
parent reply tsbockman <thomas.bockman gmail.com> writes:
On Thursday, 4 February 2016 at 23:53:58 UTC, Ola Fosheim Grøstad 
wrote:

 On Thursday, 4 February 2016 at 23:35:46 UTC, tsbockman wrote:

 Just because *sometimes* the source code of the other module 
 must be compiled independently, is a poor excuse to skip 
 obvious, useful safety checks *all* the time.


 The context is a compilation system for building big software 
 on very slow CPUs with kilobytes of RAM.

 C was designed for always compiling independently and compiling 
 source files that are bigger than what can be held in RAM, and 
 also for building executables that can fill most of system RAM. 
 So the compilation system was designed for using external 
 memory (disk) and that affects C a lot. The forerunner for C, 
 BCPL was a bootstrap language for writing compilers. So C is 
 minimal by design.


OK. That's a good reason for C's original design.

But it's 2016 and my PC has 32GiB of RAM. Why should a C compiler 
running on such a system skip safety checks just because they 
would be too expensive to run on some *other* computer?

This isn't even a particularly expensive (in compile-time costs) 
check to perform anyway; all that is necessary is to store a 
temporary table of symbol signatures somewhere (it doesn't need 
to be in RAM), and check that any duplicate entries are 
consistent with each other before linking.

This is already a solved problem in most other programming 
languages; there is no fundamental reason that the solutions used 
in D, C++, or Java could not be applied to C - without even 
changing any of the language semantics.
Feb 04 2016
next sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Friday, 5 February 2016 at 00:14:11 UTC, tsbockman wrote:

 But it's 2016 and my PC has 32GiB of RAM. Why should a C 
 compiler running on such a system skip safety checks just 
 because they would be too expensive to run on some *other* 
 computer?


C has to be backwards compatible, but I don't know why people do 
larger projects in C in 2016.

Libraries are done in C for portability and because it provides a 
FFI interface defined as the ABI by hardware and OS vendors. BeOS 
tried to define a specific C++ compiler as their ABI, but it was 
problematic.

C++ does not have an ABI, you cannot link object files from 

So, basically, there is no suitable industry standard other than 
C.


 This is already a solved problem in most other programming 
 languages; there is no fundamental reason that the solutions 
 used in D, C++, or Java could not be applied to C - without 
 even changing any of the language semantics.


D and C++ change.  C uses the ABI defined by the hardware/OS 
vendor.  It is locked in stone, frozen, beyond discussion.

As mentioned BeOS adopted C++. Apple has adopted Objective-C and 
Swift. But how can you make _all_ the other vendors (Microsoft, 
Google, IBM etc) standardize on something that isn't C?


 Aliasing types like that can be useful sometimes, but only 
 within certain limits. In particular, the size (with alignment 
 padding) of the types in question must match, otherwise you 
 will corrupt the stack.


I see where you are coming from, but I meant what I said 
literally. Machine language only deals with bitpatterns. When we 
interface with machine language we just add lots of constraints 
on what we hand over to it. Adding _more_ constraints the the 
creator of the machine language code intended is never wrong. Not 
adding enough constraints is not ideal, but often difficult to 
avoid if we care about performance.

So if I write a piece of machine language code and give you the 
object file you only have my words for what the input is supposed 
to be. And then you have to make a formulation of the constraints 
that fits your use case and is expressible in your language. 
Different languages have different levels of expressiveness for 
describing and enforcing type constraints.
Feb 04 2016
parent reply tsbockman <thomas.bockman gmail.com> writes:
On Friday, 5 February 2016 at 00:41:52 UTC, Ola Fosheim Grøstad 
wrote:

 On Friday, 5 February 2016 at 00:14:11 UTC, tsbockman wrote:

 But it's 2016 and my PC has 32GiB of RAM. Why should a C 
 compiler running on such a system skip safety checks just 
 because they would be too expensive to run on some *other* 
 computer?


 C has to be backwards compatible, but I don't know why people 
 do larger projects in C in 2016.
 [...]


Why would simply adding a warning change any of that?

No ABI changes are required. Backwards compatibility is not 
broken.
Feb 04 2016
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Friday, 5 February 2016 at 00:50:32 UTC, tsbockman wrote:

 On Friday, 5 February 2016 at 00:41:52 UTC, Ola Fosheim Grøstad 
 wrote:

 On Friday, 5 February 2016 at 00:14:11 UTC, tsbockman wrote:

 But it's 2016 and my PC has 32GiB of RAM. Why should a C 
 compiler running on such a system skip safety checks just 
 because they would be too expensive to run on some *other* 
 computer?


 C has to be backwards compatible, but I don't know why people 
 do larger projects in C in 2016.
 [...]


 Why would simply adding a warning change any of that?

 No ABI changes are required. Backwards compatibility is not 
 broken.


Not sure what you mean by adding a warning. You can probably find 
sanitizers that do it, but the standard does not require warnings 
for anything (AFAIK). That is up to compiler vendors.

As for why C isn't displaced by something better, maybe the right 
question is: why don't new languages stick to the C ABI and 
provide sensible C code gen.

Well, they want more features... and features... and features...

There is probably a market for it, but nobody can be bothered to 
create and maintain a simple modern system level language.
Feb 04 2016
prev sibling parent "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Fri, Feb 05, 2016 at 12:14:11AM +0000, tsbockman via Digitalmars-d wrote:
[...]

 This isn't even a particularly expensive (in compile-time costs) check
 to perform anyway; all that is necessary is to store a temporary table
 of symbol signatures somewhere (it doesn't need to be in RAM), and
 check that any duplicate entries are consistent with each other before
 linking.


That's a lot more expensive than you think. There's a reason most modern
linkers do not do full cross-referencing of symbols -- because doing so
would be excruciatingly slow and consume gobs of memory. Even a 32GB
machine would not be able to hold *all* the symbols in some very large
software projects, and looking things up on disk is unacceptably slow
for software of those sizes. Most modern linkers instead use faster
algorithms that rely on clever scheduling of the order of symbol
resolution, just so they *don't* have to cross-reference all symbols at
once.

Besides, all this is unnecessary work. All you need to do is to have C
compilers mangle function names.  Mission accomplished.

(However, this *will* break a lot of existing inter-language code that
rely on being able to spell out symbols explicitly. So it probably will
not fly.  But, in theory, it *is* possible...)

And to paraphrase one of my favorite Walter quotes: fixing inconsistent
function signatures is only plugging one hole in a cheese grater. C has
far more dangerous gotchas than just function signature mismatches.


T

-- 
They say that "guns don't kill people, people kill people." Well I think
the gun helps. If you just stood there and yelled BANG, I don't think
you'd kill too many people. -- Eddie Izzard, Dressed to Kill
Feb 04 2016
prev sibling parent tsbockman <thomas.bockman gmail.com> writes:
On Thursday, 4 February 2016 at 23:25:58 UTC, Ola Fosheim Grøstad 
wrote:

 In C, compilation units are completely independent, and can in 
 fact come from different compilers and different languages. C 
 is very much a system level programming language.


I should also point out that D can link to (more or less) 
anything that C can, and yet does not have the weakness exploited 
by the winning entry.

The only real reason that D is one wit less of a "system level 
programming language" than C, is the heavyweight runtime library 
- but that is irrelevant to the problem of type-checking 
cross-module references within the same code base.
Feb 04 2016
prev sibling parent "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, Feb 04, 2016 at 11:21:54PM +0000, tsbockman via Digitalmars-d wrote:
[...]

 Definitely. What puzzles me about the winning entry, though, is that the
 compiler and/or linker should be able to trivially detect the type mismatch
 *after* the preprocessor pass(es) are already done.


It cannot, because C symbols are not mangled. The function name uniquely
identifies the function, and the signature is not encoded anywhere.

The linker knows nothing about types or parameters; all it knows is that
within offset X of binary blob B, there's a binary number (usually a 32-
or 64-bit address) associated with a symbol that it needs to replace
with the value (i.e., address) of that symbol, which it obtains from the
object file that defines that symbol.

So as far as the linker is concerned, the function names match up, and
that's all there is to it.

C provides zero protection against calling functions with mismatched
parameters if the caller is not in the same file, and does not have the
right declaration. E.g.:

	/* module1.c */
	void func(int a, int b) { ... }

	/* module2.c */
	extern int func(double x); /* I'm too lazy to #include a header */
	int main() {
		int x = func(1.0); /* kaboom */
	}

In theory, this problem is solved by #include'ing the appropriate header
file, but even that isn't free from accidents like forgetting to update
the header after you change the function signature.  Of course, most
sane C projects will also #include the header in the file that defines
the function, in which case, finally, the compiler will catch the
mistake. But you can see just how fragile this is, and how many points
of failure it has, and, believe it or not, there *are* still C projects
out there that don't follow the convention of one header per .c file,
and of those that do, a frightening number do not #include the header in
the .c file.

This isn't the whole story, either. Even if you follow said conventions
to prevent function signature mismatches, problems can still occur. For
instance, once I've had to debug a mysterious crash problem in an
enterprise project that, seemingly, cannot be found in the code.  Turns
out, that it was caused by two shared libraries that defined two
different functions under the same name. Since the conflicting functions
are in separately-compiled libraries, the compiler is oblivious to the
conflict. Furthermore, the linker doesn't detect it either, because,
being shared libraries, all the linker knows is that it found symbol X
in library1, so it didn't bother looking for symbol X again in library2
which is processed afterward. An unrelated code change caused the order
of libraries linked to change, and suddenly now the linker finds symbol
X in library2 first, leading to the function call being linked to the
wrong implementation.  So at runtime, kaboom.

Name mangling singlehandedly solves all of the above problems.


T

-- 
Beware of bugs in the above code; I have only proved it correct, not tried it.
-- Donald Knuth
Feb 04 2016
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 2/4/2016 3:10 PM, H. S. Teoh via Digitalmars-d wrote:

 The C preprocessor accepts all sorts of nasty, nonsensical things.


The preprocessor makes C++ into an inherently unreliable, unsafe programming 
language. I've talked to some C++ committee members about this, about why there 
is no push to rid (at least deprecate) all use of the preprocessor. The general 
reaction I get is it is unimportant to do so.
Feb 04 2016
prev sibling next sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Thursday, 4 February 2016 at 22:57:00 UTC, tsbockman wrote:

 Actually, I'm surprised that this works even in C - I would 
 have expected at least a compiler (or linker?) warning; this 
 seems like it should be easy to detect automatically.


AFAICT C would have complained if he had included <math.h>. This 
is a rather unlikely mistake...

Anyway, in C being able to work around restrictions is sometimes 
desired, so... if you don't want the ability to do it, don't use 
C.
Feb 04 2016
parent tsbockman <thomas.bockman gmail.com> writes:
On Thursday, 4 February 2016 at 23:19:20 UTC, Ola Fosheim Grøstad 
wrote:

 On Thursday, 4 February 2016 at 22:57:00 UTC, tsbockman wrote:

 Actually, I'm surprised that this works even in C - I would 
 have expected at least a compiler (or linker?) warning; this 
 seems like it should be easy to detect automatically.


 AFAICT C would have complained if he had included <math.h>. 
 This is a rather unlikely mistake...

 Anyway, in C being able to work around restrictions is 
 sometimes desired, so... if you don't want the ability to do 
 it, don't use C.


What restriction does not checking, by default, that the 
parameter types match allow one to work around, though?

C already has `void*` and explicit casts, either of which would 
allow one to explicitly indicate that type checking is not 
desired.
Feb 04 2016
prev sibling next sibling parent reply Chris Wright <dhasenan gmail.com> writes:
On Thu, 04 Feb 2016 22:57:00 +0000, tsbockman wrote:


 The annual Underhanded C Contest announced their winners today.
 
 As always, the results are very entertaining, and also an excellent
 advertisement for languages-that-are-not-C.
 
 The first place entry is particularly ridiculous; is there any modern
 language that would make it so easy to commit such an awful "mistake"?
 
 http://www.underhanded-c.org/#winner
 
 Actually, I'm surprised that this works even in C - I would have
 expected at least a compiler (or linker?) warning; this seems like it
 should be easy to detect automatically.


C linkage does zero name mangling, which is the problem. C++ introduced 
name mangling, so compiling with g++ would show the error rather quickly. 
C99 is pretty close to C++98, but there are enough differences that that 
isn't a reliable diagnostic. (Though if you're familiar with the 
differences, you could use it as a quick way to show potential problem 
areas.)

I suppose a compiler could produce two symbol tables, one featuring 
mangled names and one with unmangled names. The linker would prefer 
matching mangled names and issue a warning if it only had an unmangled 
match with a mangled false match.
Feb 04 2016
parent reply tsbockman <thomas.bockman gmail.com> writes:
On Thursday, 4 February 2016 at 23:24:21 UTC, Chris Wright wrote:

 C linkage does zero name mangling, which is the problem. C++ 
 introduced name mangling, so compiling with g++ would show the 
 error rather quickly. C99 is pretty close to C++98, but there 
 are enough differences that that isn't a reliable diagnostic. 
 (Though if you're familiar with the differences, you could use 
 it as a quick way to show potential problem areas.)

 I suppose a compiler could produce two symbol tables, one 
 featuring mangled names and one with unmangled names. The 
 linker would prefer matching mangled names and issue a warning 
 if it only had an unmangled match with a mangled false match.


That explains why the linker doesn't catch it. I still don't see 
much excuse for the compiler allowing it though, beyond a desire 
to allow each module to be compiled independently.
Feb 04 2016
next sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Thursday, 4 February 2016 at 23:29:10 UTC, tsbockman wrote:

 That explains why the linker doesn't catch it. I still don't 
 see much excuse for the compiler allowing it though, beyond a 
 desire to allow each module to be compiled independently.


The excuse is that C use the same mechanism for creating bindings 
to C and non-C code. It is actually very handy. IF you want a 
system level language and full separation of compilation units 
(which allows for very fast compilation).
Feb 04 2016
prev sibling parent reply Chris Wright <dhasenan gmail.com> writes:
On Thu, 04 Feb 2016 23:29:10 +0000, tsbockman wrote:


 That explains why the linker doesn't catch it. I still don't see much
 excuse for the compiler allowing it though, beyond a desire to allow
 each module to be compiled independently.


Doing this sort of validation requires build system integration (track 
the command line arguments that went into producing this object file; 
find which object files are combined into which targets; run the analysis 
on that) and costs as much time as compiling the whole project from 
scratch. Developing such a system is nontrivial, so it's not a matter of 
conjuring excuses; rather, someone would have to put in considerable 
effort to make it work.

I'm betting some of the commercial static analyzers for C do this, but 
they're not the sort of things you install on every dev machine and run 
on every build. Generally they're the sort of thing that you send off to 
the security company anda they send you a report some weeks later.
Feb 04 2016
parent reply tsbockman <thomas.bockman gmail.com> writes:
On Friday, 5 February 2016 at 00:03:56 UTC, Chris Wright wrote:

 Doing this sort of validation requires build system integration 
 (track the command line arguments that went into producing this 
 object file; find which object files are combined into which 
 targets; run the analysis on that) and costs as much time as 
 compiling the whole project from scratch.


There is no need to take "as much time as compiling the whole 
project from scratch".

The necessary information is already gathered during the normal 
course of compilation; all that is required is to actually save 
it somewhere until link-time, instead of throwing it away.

The time required for the check should be at most O(N log(N)), 
where N is the number of function and global variable 
declarations in the project. The space required for the table is 
O(N). In both cases the constant factors should be quite small.


 Developing such a system is nontrivial, so it's not a matter of
 conjuring excuses; rather, someone would have to put in
 considerable effort to make it work.


Adding any interesting feature to a build system is usually 
nontrivial, but I still think you're overestimating the cost of 
this one.

Again, the hard part (finding all the signatures and processing 
them into a semantically meaningful form) is already being done 
by the compiler. The results just need to be saved, sorted, and 
scanned for conflicts.
Feb 04 2016
parent reply Chris Wright <dhasenan gmail.com> writes:
On Fri, 05 Feb 2016 00:38:16 +0000, tsbockman wrote:


 On Friday, 5 February 2016 at 00:03:56 UTC, Chris Wright wrote:

 Doing this sort of validation requires build system integration (track
 the command line arguments that went into producing this object file;
 find which object files are combined into which targets; run the
 analysis on that) and costs as much time as compiling the whole project
 from scratch.


 
 There is no need to take "as much time as compiling the whole project
 from scratch".
 
 The necessary information is already gathered during the normal course
 of compilation; all that is required is to actually save it somewhere
 until link-time, instead of throwing it away.


True. That works if this is baked into your compiler, or if your compiler 
has plugin support. And you'd have to compile with this plugin or the 
relevant options turned on by default in order for you not to duplicate 
work.

That's partly an engineering issue (build this thing in this particular 
way) and partly a social issue (get people to run it by default; have 
them add the extra flag to the makefile to specify to create the relevant 
output; possibly get your compiler vendor to build it in, depending on 
what compiler your devs are using).

I imagine Google, to take a random example where I have experience, would 
add this as a presubmit step rather than requiring it on every build.
Feb 04 2016
parent reply tsbockman <thomas.bockman gmail.com> writes:
On Friday, 5 February 2016 at 00:56:16 UTC, Chris Wright wrote:

 True. That works if this is baked into your compiler, or if 
 your compiler has plugin support. And you'd have to compile 
 with this plugin or the relevant options turned on by default 
 in order for you not to duplicate work.


On Friday, 5 February 2016 at 00:56:28 UTC, Ola Fosheim Grøstad 
wrote:

 Not sure what you mean by adding a warning. You can probably 
 find sanitizers that do it, but the standard does not require 
 warnings for anything (AFAIK). That is up to compiler vendors.


Quoting myself (emphasis added):

On Thursday, 4 February 2016 at 22:57:00 UTC, tsbockman wrote:

 Actually, I'm surprised that this works even in C - I would 
 have expected at least a COMPILER (or linker?) warning; this 
 seems like it should be easy to detect automatically.


All along I have been saying this is something that *compilers* 
should warn about. As far as I can recall, I never suggested 
using linters, sanitizers, changing the C standard - or even 
compiler plugins.

(I did suggest the linker as an alternative, but you all have 
already explained why that can't work for C.)
Feb 04 2016
next sibling parent reply Chris Wright <dhasenan gmail.com> writes:
On Fri, 05 Feb 2016 01:10:53 +0000, tsbockman wrote:


 All along I have been saying this is something that *compilers* should
 warn about.


The compiler doesn't have all the information you need. You could add it 
to the build system or the linker as well as the compiler. Adding it to 
the linker is almost identical to my previous suggestion of adding 
optional name mangling to C.
Feb 04 2016
parent reply tsbockman <thomas.bockman gmail.com> writes:
On Friday, 5 February 2016 at 03:46:37 UTC, Chris Wright wrote:

 On Fri, 05 Feb 2016 01:10:53 +0000, tsbockman wrote:
 The compiler doesn't have all the information you need. You 
 could add it to the build system or the linker as well as the 
 compiler. Adding it to the linker is almost identical to my 
 previous suggestion of adding optional name mangling to C.


What information, specifically, is the compiler missing?

The compiler already computes the name and type signature of each 
function. As far as I can see, all that is necessary is to:

1) Insert that information (together with what file and line 
number it came from) into a big list in a temporary file.
2) After all modules have been compiled, go back and sort the 
list by function name.
3) Finally, scan the list for entries that share the same name, 
but have incompatible type signatures. Emit warning messages as 
needed. (The compiler should be used for this step, because it 
already has a lot of information about C's type system built into 
it that can help define "incompatible" sensibly.)

As far as I can see, this requires an extra pass, but no 
additional information. What am I missing?
Feb 04 2016
next sibling parent reply Chris Wright <dhasenan gmail.com> writes:
On Fri, 05 Feb 2016 04:02:41 +0000, tsbockman wrote:


 On Friday, 5 February 2016 at 03:46:37 UTC, Chris Wright wrote:

 On Fri, 05 Feb 2016 01:10:53 +0000, tsbockman wrote:
 The compiler doesn't have all the information you need. You could add
 it to the build system or the linker as well as the compiler. Adding it
 to the linker is almost identical to my previous suggestion of adding
 optional name mangling to C.


 
 What information, specifically, is the compiler missing?


It doesn't know what targets I'm ultimately creating, and it doesn't know 
what files have been modified that I'm about to compile (but haven't 
compiled yet).

Example 1:

I compile one .c file referencing a function:
void foo(int);

That's going to end up in libfoo.so.

I compile another .c file in the same directory defining a function:
void foo(float);

That's going to end up in libbar.so.

No bug here. (The linker should tell us if someone depends on foo from 
libbar and foo from libfoo in the same executable.)

How does your putative compiler plugin handle it? Either I have to define 
a build rule for every source file to specify where to put this symbol 
cache (and you need to add parameters for the plugin to look for multiple 
caches, because libfoo and libbar share a lot of source files), or the 
plugin gives me false positives.

Example 2:

I compile a.c:
int foo(int i) { return i + 1; }

In the course of refactoring, I delete that function from a.c and add it 
to b.c with modifications:
int foo(int i, int increment) { return i + increment; }

My build script recompiles b.c before it recompiles a.c. Your compiler 
plugin produces a build error, halting my build. I have to make clean &&  
make in order to proceed -- and that's assuming I know your tool doesn't 
work well with incremental compilation.

The first problem might be uncommon, but the second would crop up 
constantly. They have the same fix: collect the information when you 
compile, evaluate it when you link.
Feb 04 2016
parent tsbockman <thomas.bockman gmail.com> writes:
On Friday, 5 February 2016 at 06:05:49 UTC, Chris Wright wrote:

 It doesn't know what targets I'm ultimately creating, and it 
 doesn't know what files have been modified that I'm about to 
 compile (but haven't compiled yet).

 Example 1:

 I compile one .c file referencing a function:
 void foo(int);

 That's going to end up in libfoo.so.

 I compile another .c file in the same directory defining a 
 function:
 void foo(float);

 That's going to end up in libbar.so.

 No bug here. (The linker should tell us if someone depends on 
 foo from libbar and foo from libfoo in the same executable.)

 How does your putative compiler plugin handle it? Either I have 
 to define a build rule for every source file to specify where 
 to put this symbol cache (and you need to add parameters for 
 the plugin to look for multiple caches, because libfoo and 
 libbar share a lot of source files), or the plugin gives me 
 false positives.

 Example 2:

 I compile a.c:
 int foo(int i) { return i + 1; }

 In the course of refactoring, I delete that function from a.c 
 and add it
 to b.c with modifications:
 int foo(int i, int increment) { return i + increment; }

 My build script recompiles b.c before it recompiles a.c. Your 
 compiler plugin produces a build error, halting my build. I 
 have to make clean && make in order to proceed -- and that's 
 assuming I know your tool doesn't work well with incremental 
 compilation.

 The first problem might be uncommon, but the second would crop 
 up constantly. They have the same fix: collect the information 
 when you compile, evaluate it when you link.


No spurious error is generated by my proposal in your example 2, 
because I specifically stated that the extra pass must be done 
once, after *all* modules have been compiled.

I see, however, that this would require one of:

1) Modifying build scripts to pass the complete list of .c files 
to the compiler in a single command, or
2) Modifying build scripts to run the compiler one extra time 
after processing all the .c files, or
3) Run the final check at link-time.

For a C tool chain with a clean-sheet design, any of those would 
handle example 2 fine. (1) or (3) could also handle example 1 
without issue.

However, as you say, only (3) is backwards compatible with 
existing make files and what-not. (This is not a limitation of 
the C language or ABI, though.)
Feb 04 2016
prev sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Fri, Feb 05, 2016 at 04:02:41AM +0000, tsbockman via Digitalmars-d wrote:

 On Friday, 5 February 2016 at 03:46:37 UTC, Chris Wright wrote:

On Fri, 05 Feb 2016 01:10:53 +0000, tsbockman wrote:
The compiler doesn't have all the information you need. You could add it
to the build system or the linker as well as the compiler. Adding it to
the linker is almost identical to my previous suggestion of adding
optional name mangling to C.


 
 What information, specifically, is the compiler missing?
 
 The compiler already computes the name and type signature of each function.
 As far as I can see, all that is necessary is to:
 
 1) Insert that information (together with what file and line number it
 came from) into a big list in a temporary file.
 2) After all modules have been compiled, go back and sort the list by
 function name.


This would make compilation of large projects excruciatingly slow.



 3) Finally, scan the list for entries that share the same name, but
 have incompatible type signatures. Emit warning messages as needed.
 (The compiler should be used for this step, because it already has a
 lot of information about C's type system built into it that can help
 define "incompatible" sensibly.)


This fails for multi-executable projects, which may legally have
different functions under the same name. (Even though that's arguably a
very bad idea.)



 As far as I can see, this requires an extra pass, but no additional
 information. What am I missing?


The fact that the C compiler only sees one file at a time, and has no
idea which one, if any, of them will even end up in the final
executable. Many projects produce multiple executables with some shared
sources between them, and only the build system knows which file(s) go
with which executables.

So as others have said, this can only work for compilers that are aware
of the larger picture than just the single source file it's currently
compiling. Even in D, for a sufficiently large project the compiler
can't see everything at once either, because it won't fit into your RAM.
Thankfully, D doesn't suffer from this particular problem because of
name mangling.

Which is why I said, adding name mangling to the C compiler will solve
this problem. Except that it breaks existing inter-language code, so it
won't work for *all* C programs. And it will also break linkage with
existing shared libraries, which are *not* name-mangled. (Recompiling
said libraries may not be an option if they are OEM, binary-only blobs.)
So it can only work for self-contained, independent projects with no
inter-language linkage, which would be a very restricted subset of C
codebases.


T

-- 
Nobody is perfect.  I am Nobody. -- pepoluan, GKC forum
Feb 04 2016
next sibling parent reply tsbockman <thomas.bockman gmail.com> writes:
On Friday, 5 February 2016 at 07:05:06 UTC, H. S. Teoh wrote:

 On Fri, Feb 05, 2016 at 04:02:41AM +0000, tsbockman via 
 Digitalmars-d wrote:

 On Friday, 5 February 2016 at 03:46:37 UTC, Chris Wright wrote:

On Fri, 05 Feb 2016 01:10:53 +0000, tsbockman wrote:


 What information, specifically, is the compiler missing?
 
 The compiler already computes the name and type signature of 
 each function. As far as I can see, all that is necessary is 
 to:
 
 1) Insert that information (together with what file and line 
 number it
 came from) into a big list in a temporary file.
 2) After all modules have been compiled, go back and sort the 
 list by
 function name.


 This would make compilation of large projects excruciatingly 
 slow.


It's a small fraction of the total data being handled by the 
compiler (smaller than the source code), and the list could 
probably be directly generated in a partially sorted state. 
Little-to-no random access to the list is required at any point 
in the process. It does not ever need to all be in RAM at the 
same time.

I can see it may cost more than it's actually worth, but where 
does the "excruciatingly slow" part come from?


 3) Finally, scan the list for entries that share the same 
 name, but have incompatible type signatures. Emit warning 
 messages as needed. (The compiler should be used for this 
 step, because it already has a lot of information about C's 
 type system built into it that can help define "incompatible" 
 sensibly.)


 This fails for multi-executable projects, which may legally 
 have different functions under the same name. (Even though 
 that's arguably a very bad idea.)


Chris Wright pointed this out, as well. This just means the final 
pass should be done at link-time, though. It's not a fundamental 
problem with generating the warning.


 As far as I can see, this requires an extra pass, but no 
 additional information. What am I missing?


 The fact that the C compiler only sees one file at a time, and 
 has no idea which one, if any, of them will even end up in the 
 final executable. Many projects produce multiple executables 
 with some shared sources between them, and only the build 
 system knows which file(s) go with which executables.


This could be worked around with a little cooperation between the 
compiler and the linker. It's not even a feature of C the 
language - it's just the way current tool chains happen to work.
Feb 04 2016
parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Fri, Feb 05, 2016 at 07:31:34AM +0000, tsbockman via Digitalmars-d wrote:

 On Friday, 5 February 2016 at 07:05:06 UTC, H. S. Teoh wrote:

On Fri, Feb 05, 2016 at 04:02:41AM +0000, tsbockman via Digitalmars-d
wrote:

On Friday, 5 February 2016 at 03:46:37 UTC, Chris Wright wrote:

On Fri, 05 Feb 2016 01:10:53 +0000, tsbockman wrote:


What information, specifically, is the compiler missing?

The compiler already computes the name and type signature of each
function. As far as I can see, all that is necessary is to:

1) Insert that information (together with what file and line number
it came from) into a big list in a temporary file.
2) After all modules have been compiled, go back and sort the list
by function name.


This would make compilation of large projects excruciatingly slow.


 
 It's a small fraction of the total data being handled by the compiler
 (smaller than the source code), and the list could probably be
 directly generated in a partially sorted state. Little-to-no random
 access to the list is required at any point in the process. It does
 not ever need to all be in RAM at the same time.
 
 I can see it may cost more than it's actually worth, but where does
 the "excruciatingly slow" part come from?


OK, probably I'm misunderstanding something here. :-P



3) Finally, scan the list for entries that share the same name, but
have incompatible type signatures. Emit warning messages as needed.
(The compiler should be used for this step, because it already has a
lot of information about C's type system built into it that can help
define "incompatible" sensibly.)


This fails for multi-executable projects, which may legally have
different functions under the same name. (Even though that's arguably
a very bad idea.)


 
 Chris Wright pointed this out, as well. This just means the final pass
 should be done at link-time, though. It's not a fundamental problem
 with generating the warning.


The problem is, the linker knows nothing about the language. Arguably it
should, but as things stand currently, it doesn't, and can't, because
usually linkers are shipped with the OS, and are expected to link object
files of *any* pedigree without needing to code for language-explicit
checks.

Perhaps this is slowly starting to change, as LTO and other recent
innovations are pushing the envelope of what the linker can do.  Maybe
one day there will emerge a language-agnostic way for the linker to
check for such errors... but I really don't see it happening, because
languages *other* than C have already solved the problem with name
mangling. There isn't much motivation for linkers to change just because
C has some language design issues.

(And note that I'm not trying to disagree with you -- I'm totally in
agreement that what C allows is oftentimes extremely dangerous and
rather unwise. But the way things are is just so entrenched that it's
unlikely to change in the near (or even distant) future.)



As far as I can see, this requires an extra pass, but no additional
information. What am I missing?


The fact that the C compiler only sees one file at a time, and has no
idea which one, if any, of them will even end up in the final
executable. Many projects produce multiple executables with some
shared sources between them, and only the build system knows which
file(s) go with which executables.


 
 This could be worked around with a little cooperation between the
 compiler and the linker. It's not even a feature of C the language -
 it's just the way current tool chains happen to work.


And that's where the sticky part lies. Current toolchains work in this,
arguably suboptimal, way mainly because of historical baggage, but more
because doing otherwise will make the toolchain incompatible with
existing other toolchains and systems. The current divide between
compiler and linker is actually IMO not in the best place it could be,
as it hampers a lot of what, arguably, should be the compiler's job, not
the linker's. Nevertheless, changing this means you become incompatible
with much of the ecosystem and become a walled garden -- like Java (JNI
was an afterthought, and requires a very specific setup to even work --
there's definitely no way to link Java objects with OS-level object
files without jumping through lots of hoops with lots of caveats). I
just don't see this ever happening, especially not for something that,
in the big picture, really isn't *that* big of a deal. After all, C
coders have gotten used to working with far more dangerous things in C
than merely mismatched prototypes; it would take a LOT more than that
for people to accept changing the way things work.


T

-- 
Skill without imagination is craftsmanship and gives us many useful objects
such as wickerwork picnic baskets.  Imagination without skill gives us modern
art. -- Tom Stoppard
Feb 05 2016
parent reply Chris Wright <dhasenan gmail.com> writes:
On Fri, 05 Feb 2016 10:04:01 -0800, H. S. Teoh via Digitalmars-d wrote:


 On Fri, Feb 05, 2016 at 07:31:34AM +0000, tsbockman via Digitalmars-d
 wrote:

 On Friday, 5 February 2016 at 07:05:06 UTC, H. S. Teoh wrote:

On Fri, Feb 05, 2016 at 04:02:41AM +0000, tsbockman via Digitalmars-d
wrote:

On Friday, 5 February 2016 at 03:46:37 UTC, Chris Wright wrote:

On Fri, 05 Feb 2016 01:10:53 +0000, tsbockman wrote:


What information, specifically, is the compiler missing?

The compiler already computes the name and type signature of each
function. As far as I can see, all that is necessary is to:

1) Insert that information (together with what file and line number
it came from) into a big list in a temporary file.
2) After all modules have been compiled, go back and sort the list by
function name.


This would make compilation of large projects excruciatingly slow.


 
 It's a small fraction of the total data being handled by the compiler
 (smaller than the source code), and the list could probably be directly
 generated in a partially sorted state. Little-to-no random access to
 the list is required at any point in the process. It does not ever need
 to all be in RAM at the same time.
 
 I can see it may cost more than it's actually worth, but where does the
 "excruciatingly slow" part come from?


 
 OK, probably I'm misunderstanding something here. :-P


I think you're talking about maintaining an in-memory, modifiable data 
structure, doing one insert per operation and one point query per use. 
That's useful for incremental compilation, but it's going to be pretty 
slow.

tsbockman is thinking of a single pass at link time that checks 
everything at once. You append an entry to a list for each prototype and 
definition, then later sort all those lists together by name. Error on 
duplicate names with mismatched signatures.

This is faster for fresh builds than it is for incremental compilation -- 
tsbockman mentioned a brief benchmark, and that cost would crop up on 
every build, even if you'd only changed one line of code. (Granted, that 
example was pretty huge.) But this might typically be faster than a bunch 
of point queries even with incremental compilation.

Anyway, that's why I'm thinking most people who used such a feature would 
turn it on in their continuous integration server or as a presubmit step 
rather than every build.


 The problem is, the linker knows nothing about the language.


We're only talking about a linker because we need to run this tool after 
compiling all your files, and it has to know what input files you're 
putting into the linker.

So this "linker" is really just a shell script that invokes our checker 
and then calls the system linker.
Feb 05 2016
parent tsbockman <thomas.bockman gmail.com> writes:
On Friday, 5 February 2016 at 20:35:16 UTC, Chris Wright wrote:

 On Fri, 05 Feb 2016 10:04:01 -0800, H. S. Teoh via 
 Digitalmars-d wrote:


 On Fri, Feb 05, 2016 at 07:31:34AM +0000, tsbockman via 
 Digitalmars-d wrote:

 On Friday, 5 February 2016 at 07:05:06 UTC, H. S. Teoh wrote:


 OK, probably I'm misunderstanding something here. :-P


 I think you're talking about maintaining an in-memory, 
 modifiable data structure, doing one insert per operation and 
 one point query per use. That's useful for incremental 
 compilation, but it's going to be pretty slow.

 tsbockman is thinking of a single pass at link time that checks 
 everything at once. You append an entry to a list for each 
 prototype and definition, then later sort all those lists 
 together by name. Error on duplicate names with mismatched 
 signatures.


Yes.


 This is faster for fresh builds than it is for incremental 
 compilation -- tsbockman mentioned a brief benchmark, and that 
 cost would crop up on every build, even if you'd only changed 
 one line of code. (Granted, that example was pretty huge.) But 
 this might typically be faster than a bunch of point queries 
 even with incremental compilation.

 Anyway, that's why I'm thinking most people who used such a 
 feature would turn it on in their continuous integration server 
 or as a presubmit step rather than every build.


It doesn't necessarily have to be slow when you only changed one 
line:

* The list from the previous compilation could be re-used to 
speed things up considerably, although retaining it would cost 
some disk space.

* If that's still too expensive, just don't cross-check files 
that aren't being recompiled. The check will be less useful on 
incremental builds, but not *useless*. The CI server can still do 
the full check (using the compiler), as you suggest.


 The problem is, the linker knows nothing about the language.


 We're only talking about a linker because we need to run this 
 tool after compiling all your files, and it has to know what 
 input files you're putting into the linker.

 So this "linker" is really just a shell script that invokes our 
 checker and then calls the system linker.


Yes. (Or, it's the compiler with a special option set, which then 
calls the linker after it finishes its global pre-link tasks.)
Feb 05 2016
prev sibling parent tsbockman <thomas.bockman gmail.com> writes:
On Friday, 5 February 2016 at 07:05:06 UTC, H. S. Teoh wrote:

 On Fri, Feb 05, 2016 at 04:02:41AM +0000, tsbockman via 
 Digitalmars-d wrote:

 1) Insert that information (together with what file and line 
 number it
 came from) into a big list in a temporary file.
 2) After all modules have been compiled, go back and sort the 
 list by
 function name.


 This would make compilation of large projects excruciatingly 
 slow.


I did some quick tests on my system, and even with 100,000,000 
names (more names than there are lines of code in the Linux 
kernel...) this can be done in less than three minutes. Smaller 
projects take seconds or less.

I suspect there is a major disconnect between what I meant, and 
what you think I meant.
Feb 05 2016
prev sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Friday, 5 February 2016 at 01:10:53 UTC, tsbockman wrote:

 All along I have been saying this is something that *compilers* 
 should warn about. As far as I can recall, I never suggested 
 using linters, sanitizers, changing the C standard - or even 
 compiler plugins.


Well, compilers "should" only implement the standard, then they 
"may" add extra static analysis.

The direction C and C++ takes is that increasing compilation 
times by doing extra static analysis on every build isn't 
desirable. Therefore compilers should focus on what is necessary 
for code gen and optimization and sanitizers should focus on 
correctness.

This is different from Rust, who do sanitization as part of their 
compilation, but that makes the compiler more complicated and/or 
much _slower_.


 (I did suggest the linker as an alternative, but you all have 
 already explained why that can't work for C.)


It can work if you compile all source files with the same 
compiler, that has historically not been the case as commercial 
libraries would be compiled with other compilers or be 
handwritten assembly.

C compilers that do Whole Program Analysis have dedicated linkers 
that should be able to do extended type checking if the IR used 
in the object file provides typing info. I don't know if Clang or 
GCC does emit typing info though, but they _could_. Yes.
Feb 05 2016
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
Let me add to this that the superior approach is to compile to an 
intermediated high level format that retains type information. I 
guess this is where Rust is heading.

It just isn't possible with C semantics to make a reasonable 
version of that, since the language itself is 90% unsafe and just 
a small step up from assembly (for good and bad).
Feb 05 2016
prev sibling next sibling parent reply anonymous <anonymous example.com> writes:
On 04.02.2016 23:57, tsbockman wrote:

 http://www.underhanded-c.org/#winner

 Actually, I'm surprised that this works even in C - I would have
 expected at least a compiler (or linker?) warning; this seems like it
 should be easy to detect automatically.


You can do the same thing in D, using extern(C) to get no mangling:

main.d:
----
alias float_t = double;
extern(C) float_t deref(float_t* a);
void main()
{
     import std.stdio: writeln;
     float_t d = 1.23;
     writeln(deref(&d)); /* prints "1.01856e-314" */
}
----

deref.d:
----
alias float_t = float;
extern(C) float_t deref(float_t* a) {return *a;}
----

Command to build and run:
----
dmd main.d deref.d && ./main
----
Feb 04 2016
parent reply tsbockman <thomas.bockman gmail.com> writes:
On Thursday, 4 February 2016 at 23:40:13 UTC, anonymous wrote:

 You can do the same thing in D, using extern(C) to get no 
 mangling:

 main.d:
 ----
 alias float_t = double;
 extern(C) float_t deref(float_t* a);
 void main()
 {
     import std.stdio: writeln;
     float_t d = 1.23;
     writeln(deref(&d)); /* prints "1.01856e-314" */
 }
 ----

 deref.d:
 ----
 alias float_t = float;
 extern(C) float_t deref(float_t* a) {return *a;}
 ----

 Command to build and run:
 ----
 dmd main.d deref.d && ./main
 ----


You can do the same thing in D if you try, but it's not natural 
at all to use `extern(C)` for *internal* linkage of an all-D 
program like that.

Any competent reviewer would certainly question why you were 
using `extern(C)`; this scores much lower in "underhanded-ness" 
than the original C program.

Even so, I think that qualifies as a compiler bug or a hole in 
the D spec.
Feb 04 2016
next sibling parent reply anonymous <anonymous example.com> writes:
On 05.02.2016 00:47, tsbockman wrote:

 You can do the same thing in D if you try, but it's not natural at all
 to use `extern(C)` for *internal* linkage of an all-D program like that.

 Any competent reviewer would certainly question why you were using
 `extern(C)`; this scores much lower in "underhanded-ness" than the
 original C program.


We do have a lot of bindings to C libraries, though. When there's a 
wrong alias in one of them, you have the same scenario.


 Even so, I think that qualifies as a compiler bug or a hole in the D spec.


Can anything be done about it? The compiler simply has no way to verify 
declarations, has it?
Feb 04 2016
parent reply tsbockman <thomas.bockman gmail.com> writes:
On Thursday, 4 February 2016 at 23:51:57 UTC, anonymous wrote:

 We do have a lot of bindings to C libraries, though. When 
 there's a wrong alias in one of them, you have the same 
 scenario.

 On 05.02.2016 00:47, tsbockman wrote:

 Even so, I think that qualifies as a compiler bug or a hole in 
 the D spec.


 Can anything be done about it? The compiler simply has no way 
 to verify declarations, has it?


The compiler cannot (in the general case) verify that `extern(C)` 
declarations are *correct*. What it could do, though, is verify 
that they are *consistent*.

If the same `extern(C)` symbol is declared multiple places in the 
D source code for a program, the compiler should issue at least a 
warning if the D signatures don't agree with each other.
Feb 04 2016
next sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Friday, 5 February 2016 at 00:03:20 UTC, tsbockman wrote:

 If the same `extern(C)` symbol is declared multiple places in 
 the D source code for a program, the compiler should issue at 
 least a warning if the D signatures don't agree with each other.


I guess D could do it, although this is a rather unlikely source 
for bugs.

C cannot do it. It would be annoying as declarations are file 
local.

C doesn't really build programs, it builds object files that are 
linked into a program.

It makes perfect sense for one compilation unit to type a 
parameter pointer to float  and another unit to type the same 
parameter as a simd-array of floats. The underlying code could be 
machine language. And in machine language there are no types (on 
current CPUs), only bit patterns. So you can have multiple 
reasonable interpretations of the same machine language entry.

A type is a constraint, but it isn't a property of the actual 
bits, it is a language specific interpretation.
Feb 04 2016
parent tsbockman <thomas.bockman gmail.com> writes:
On Friday, 5 February 2016 at 00:12:07 UTC, Ola Fosheim Grøstad 
wrote:

 It makes perfect sense for one compilation unit to type a 
 parameter pointer to float  and another unit to type the same 
 parameter as a simd-array of floats. The underlying code could 
 be machine language. And in machine language there are no types 
 (on current CPUs), only bit patterns. So you can have multiple 
 reasonable interpretations of the same machine language entry.

 A type is a constraint, but it isn't a property of the actual 
 bits, it is a language specific interpretation.


Aliasing types like that can be useful sometimes, but only within 
certain limits. In particular, the size (with alignment padding) 
of the types in question must match, otherwise you will corrupt 
the stack.

It is often useful to cast from one pointer type to another, but 
that is why C has void* and explicit casts - so that one may 
document that the reinterpretation is intentional.
Feb 04 2016
prev sibling parent reply Daniel Murphy <yebbliesnospam gmail.com> writes:
On 5/02/2016 11:03 AM, tsbockman wrote:

 The compiler cannot (in the general case) verify that `extern(C)`
 declarations are *correct*. What it could do, though, is verify that
 they are *consistent*.

 If the same `extern(C)` symbol is declared multiple places in the D
 source code for a program, the compiler should issue at least a warning
 if the D signatures don't agree with each other.


Currently D allows overloading extern(C) declarations, see
https://issues.dlang.org/show_bug.cgi?id=15217

Checking for invalid overloads with non-D linkage is covered here:
https://issues.dlang.org/show_bug.cgi?id=2789

But neither of these cover overloads that aren't simultaneously visible.
15217 shows us that this lack of checking, when combined with D's 
abundant binary-compatible-but-distinct types, is somewhat useful.

Apart from some scary ABI hacks there is nothing really stopping us from 
enforcing that all non-D function in all modules included in a single 
compilation have distinct symbol names or (at least binary-compatible) 
matching D parameters.
Feb 05 2016
parent reply tsbockman <thomas.bockman gmail.com> writes:
On Friday, 5 February 2016 at 10:49:50 UTC, Daniel Murphy wrote:

 Currently D allows overloading extern(C) declarations, see
 https://issues.dlang.org/show_bug.cgi?id=15217

 Checking for invalid overloads with non-D linkage is covered 
 here:
 https://issues.dlang.org/show_bug.cgi?id=2789

 But neither of these cover overloads that aren't simultaneously 
 visible.
 15217 shows us that this lack of checking, when combined with 
 D's abundant binary-compatible-but-distinct types, is somewhat 
 useful.

 Apart from some scary ABI hacks there is nothing really 
 stopping us from enforcing that all non-D function in all 
 modules included in a single compilation have distinct symbol 
 names or (at least binary-compatible) matching D parameters.


I think it makes sense (when actually linking to C) to allow 
stuff like druntime's creative use of overloads. The signatures 
of the two bsd_signal() overloads are compatible (from C's 
perspective), so why not?

However, multiple `extern(C)` overloads that differ in the number 
or size of arguments should trigger a warning. Signed versus 
unsigned or even int versus floating point is more of a gray area.

Overloads with conflicting pointer types should definitely be 
allowed, but ideally the compiler would force them to be marked 
 system or  trusted, since there is an implied unsafe cast in 
there somewhere.
Feb 05 2016
parent Daniel Murphy <yebbliesnospam gmail.com> writes:
On 5/02/2016 10:07 PM, tsbockman wrote:

 I think it makes sense (when actually linking to C) to allow stuff like
 druntime's creative use of overloads. The signatures of the two
 bsd_signal() overloads are compatible (from C's perspective), so why not?

 However, multiple `extern(C)` overloads that differ in the number or
 size of arguments should trigger a warning. Signed versus unsigned or
 even int versus floating point is more of a gray area.


That's what I meant by binary compatible.


 Overloads with conflicting pointer types should definitely be allowed,
 but ideally the compiler would force them to be marked  system or
  trusted, since there is an implied unsafe cast in there somewhere.


Safety on C functions is always going to need to be hand verified, the 
presence of overloads doesn't change that.  Conflicting pointer types 
are pretty much the same as a function taking void* - all the unsafe 
stuff is on the other side and invisible to the D compiler.
Feb 05 2016
prev sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, Feb 04, 2016 at 11:47:53PM +0000, tsbockman via Digitalmars-d wrote:
[...]

 You can do the same thing in D if you try, but it's not natural at all
 to use `extern(C)` for *internal* linkage of an all-D program like
 that.
 
 Any competent reviewer would certainly question why you were using
 `extern(C)`; this scores much lower in "underhanded-ness" than the
 original C program.
 
 Even so, I think that qualifies as a compiler bug or a hole in the D
 spec.


Nah... while D, by default, tries to be type-safe and prevent guffaws
like the above, it *is* also a systems programming language (or at
least, that's one of the stated goals), so it does allow you to go under
the hood to do things that you normally aren't allowed to do.

Linking to foreign languages is a use case for allowing extern(C)
function names: if you know the mangling scheme of the target language,
you can declare the mangled name under extern(C) and that will allow D
code to call functions written in the target language directly.
Otherwise you'd have to change the compiler (and wait for the next
release, etc.) before you could do that.


T

-- 
Do not reason with the unreasonable; you lose by definition.
Feb 04 2016
next sibling parent reply Chris Wright <dhasenan gmail.com> writes:
On Thu, 04 Feb 2016 15:59:06 -0800, H. S. Teoh via Digitalmars-d wrote:


 Nah... while D, by default, tries to be type-safe and prevent guffaws
 like the above, it *is* also a systems programming language (or at
 least, that's one of the stated goals), so it does allow you to go under
 the hood to do things that you normally aren't allowed to do.


Which suggests a check of this sort should be a warning rather than an 
error, or perhaps that a pragma or attribute could be offered to ignore 
it.

Systems languages let you go into "Here Be Dragons" territory, but it 
would be nice if they still pointed out the signs to you.
Feb 04 2016
parent tsbockman <thomas.bockman gmail.com> writes:
On Friday, 5 February 2016 at 00:07:45 UTC, Chris Wright wrote:

 Which suggests a check of this sort should be a warning rather 
 than an error, or perhaps that a pragma or attribute could be 
 offered to ignore it.

 Systems languages let you go into "Here Be Dragons" territory, 
 but it would be nice if they still pointed out the signs to you.


Yes.
Feb 04 2016
prev sibling parent tsbockman <thomas.bockman gmail.com> writes:
On Thursday, 4 February 2016 at 23:59:06 UTC, H. S. Teoh wrote:

 On Thu, Feb 04, 2016 at 11:47:53PM +0000, tsbockman via 
 Digitalmars-d wrote: [...]

 Even so, I think that qualifies as a compiler bug or a hole in 
 the D spec.


 Nah... while D, by default, tries to be type-safe and prevent 
 guffaws like the above, it *is* also a systems programming 
 language (or at least, that's one of the stated goals), so it 
 does allow you to go under the hood to do things that you 
 normally aren't allowed to do.

 Linking to foreign languages is a use case for allowing 
 extern(C) function names: if you know the mangling scheme of 
 the target language, you can declare the mangled name under 
 extern(C) and that will allow D code to call functions written 
 in the target language directly. Otherwise you'd have to change 
 the compiler (and wait for the next release, etc.) before you 
 could do that.


 T


I'm not saying that `extern(C)` is bad in general; I understand 
why it's necessary.

I'm saying that anonymous' example 
(http://forum.dlang.org/post/n90ngu$1r6v$1 digitalmars.com) 
showcases a hole in the spec, because in it the D compiler has 
access to the full source code of the function being linked to, 
and doesn't bother to verify that its signature in main.d is 
compatible with the definition in deref.d.

If the D compiler does *not* have access to the function's 
definition, then obviously it cannot perform this verification.
Feb 04 2016
prev sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Thursday, 4 February 2016 at 22:57:00 UTC, tsbockman wrote:

 The first place entry is particularly ridiculous; is there any 
 modern language that would make it so easy to commit such an 
 awful "mistake"?


D allows that. This is why I recommend putting `static 
assert(foo.sizeof == expectation);` in code that interfaces with 
external things, like C code, or D .di stuff.

#include <math.h> /* sqrt */

that line is an interesting one too: the trick is depending on 
namespace pollution by the include. In D, you might write `import 
core.stdc.math : sqrt;` and make that misleading comment part of 
the code.... though then you could perhaps exploit that module 
bug (314?).
Feb 04 2016
parent reply tsbockman <thomas.bockman gmail.com> writes:
On Friday, 5 February 2016 at 01:14:05 UTC, Adam D. Ruppe wrote:

 D allows that. This is why I recommend putting `static 
 assert(foo.sizeof == expectation);` in code that interfaces 
 with external things, like C code, or D .di stuff.

 #include <math.h> /* sqrt */


D *doesn't* allow that though - at least, not in a monolithic, 
idiomatic D program: there wouldn't be any duplicate declaration 
of `spectral_contrast()` to mess up.

Yes, you can force the matter using `extern(C)` like anonymous 
demonstrated earlier - but using `extern(C)` for internal linkage 
in an all-D program would certainly attract scrutiny from 
reviewers; it would score poorly on the "underhanded-ness" test.

As to the ".di" stuff - I've not used them. Care to educate me? 
How can they cause similar problems?


 that line is an interesting one too: the trick is depending on 
 namespace pollution by the include. In D, you might write 
 `import core.stdc.math : sqrt;` and make that misleading 
 comment part of the code.... though then you could perhaps 
 exploit that module bug (314?).


314 definitely has potential. Should we start an "Underhanded D" 
contest? Sounds like bad marketing, but a lot of fun :-P
Feb 04 2016
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Friday, 5 February 2016 at 01:33:14 UTC, tsbockman wrote:

 As to the ".di" stuff - I've not used them. Care to educate me? 
 How can they cause similar problems?


Well, technically, a .di file is just a .d file renamed, but it 
tends to have the bodies stripped out. Separate compliation is a 
supported feature of D.

The way you'd do it is something like this:

struct Foo {
    float a;
    float b;
}

void bar(Foo* f) {
    f.b = whatever;
}


Then compile it with -lib and make a "header" file manually:

struct Foo {
    double a;
    double b;
}
void bar(Foo*);


You can now create D modules that import this and link against 
the compiled library. Very similar to C's model...

But I redefined Foo! The name mangling won't catch this. bar will 
be mangled to take `Foo` as an argument and the linker will catch 
if we change that, but it doesn't know what Foo actually is.

By changing that, we introduce the problem.


 314 definitely has potential. Should we start an "Underhanded 
 D" contest? Sounds like bad marketing, but a lot of fun :-P


it might be :)
Feb 04 2016
parent reply tsbockman <thomas.bockman gmail.com> writes:
On Friday, 5 February 2016 at 04:25:09 UTC, Adam D. Ruppe wrote:

 On Friday, 5 February 2016 at 01:33:14 UTC, tsbockman wrote:

 As to the ".di" stuff - I've not used them. Care to educate 
 me? How can they cause similar problems?


 Well, technically, a .di file is just a .d file renamed, but it 
 tends to have the bodies stripped out. Separate compliation is 
 a supported feature of D.

 The way you'd do it is something like this:

 struct Foo {
    float a;
    float b;
 }

 void bar(Foo* f) {
    f.b = whatever;
 }


 Then compile it with -lib and make a "header" file manually:

 struct Foo {
    double a;
    double b;
 }
 void bar(Foo*);


 You can now create D modules that import this and link against 
 the compiled library. Very similar to C's model...

 But I redefined Foo! The name mangling won't catch this. bar 
 will be mangled to take `Foo` as an argument and the linker 
 will catch if we change that, but it doesn't know what Foo 
 actually is.

 By changing that, we introduce the problem.


 314 definitely has potential. Should we start an "Underhanded 
 D" contest? Sounds like bad marketing, but a lot of fun :-P


 it might be :)


Thanks for the explanation. That does sound basically the same as 
the C issue.

Since .di files are normally generated automatically, this seems 
like an easily solvable problem:

1) When compiling a library and its attendant .di file(s), 
generate a unique version identifier (such as a UUID or a hash of 
the completed binary) and append it to both the library and each 
.di file.

2) Whenever someone tries to link against the library, verify 
that the version ID matches. If it does not, issue a prominent 
warning.

Problem solved? Or is this harder than it looks?

(Of course there are various details to consider, such as how to 
efficiently share one set of .di files across many 
platforms/compiler settings; this is just a rough sketch.)
Feb 04 2016
parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Fri, Feb 05, 2016 at 04:39:13AM +0000, tsbockman via Digitalmars-d wrote:
[...]

 Thanks for the explanation. That does sound basically the same as the
 C issue.
 
 Since .di files are normally generated automatically, this seems like
 an easily solvable problem:
 
 1) When compiling a library and its attendant .di file(s), generate a
 unique version identifier (such as a UUID or a hash of the completed
 binary) and append it to both the library and each .di file.
 
 2) Whenever someone tries to link against the library, verify that the
 version ID matches. If it does not, issue a prominent warning.


[...]

This would break shared library upgrades that do not change the ABI.

Plus, it doesn't fix wrong linkage at runtime, because the dynamic
linker is part of the OS and the D compiler has no control over what it
does beyond the standard symbol matching and relocation mechanisms. If
you compile against libfoo, but at runtime the user happens to have a
stale, ABI-incompatible version of libfoo hanging around that gets
picked up by the dynamic linker, you'll have the same problem.


T

-- 
VI = Visual Irritation
Feb 04 2016
parent tsbockman <thomas.bockman gmail.com> writes:
On Friday, 5 February 2016 at 07:15:56 UTC, H. S. Teoh wrote:

 This would break shared library upgrades that do not change the 
 ABI.

 Plus, it doesn't fix wrong linkage at runtime, because the 
 dynamic linker is part of the OS and the D compiler has no 
 control over what it does beyond the standard symbol matching 
 and relocation mechanisms. If you compile against libfoo, but 
 at runtime the user happens to have a stale, ABI-incompatible 
 version of libfoo hanging around that gets picked up by the 
 dynamic linker, you'll have the same problem.


I should have clarified that I was considering static libraries, 
only. (I thought D's dynamic library support was kind of broken 
right at the moment, anyway?)

Dynamic libraries are definitely a harder problem. I think useful 
automated protection against bad .di files could be developed for 
dynamic libraries as well, but the scheme wouldn't be anywhere 
near as simple and it might require the maintainer to actually 
follow SemVer to be useful.
Feb 04 2016