www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - memcpy, memset, memcmp and friends in the D Runtime

reply "Mike" <none none.com> writes:
I'm trying to port the D Runtime to a bare metal platform.  In 
the short time I've been doing bare metal work, I've come to rely 
on two very fundamental methods: memcpy and memset.  In my D 
runtime, however, I'm not using the C standard library; I'm doing 
it all in D.

It seems the D Runtime and Phobos also often make use of memcpy 
and memset, but they do so through core.stdc which doesn't appear 
to be part of the D Runtime.  Was this just a matter of 
convenience or thoughtful design?  I'm guessing the former based 
on the fact that core.stdc is not in the official language 
reference.

So, I would like to know what the design philosophy is for 
fundamental memory utilities like memcpy, memset, memcmp etc..., 
because I need them and would like to implement harmony with D's 
design.

* Were these methods intentionally omitted from the D Runtime, or 
just taken for granted since they were so conveniently available 
in the C standard library?

* Should they be added to core.memory or some other module and 
exposed as official, supported, methods of the D Runtime 
(hopefully changing their names)?

Please advise.

Thanks,
Mike
Apr 07 2014
next sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Mon, 07 Apr 2014 07:51:38 -0400, Mike <none none.com> wrote:

 * Were these methods intentionally omitted from the D Runtime, or just  
 taken for granted since they were so conveniently available in the C  
 standard library?

I don't think it's convenience so much as being very experienced and tuned functions.
 * Should they be added to core.memory or some other module and exposed  
 as official, supported, methods of the D Runtime (hopefully changing  
 their names)?

I think using using D's safer syntax of a[] = b[] and a[] = 0; Is much preferable to using memcpy/memset directly. They should do the "Right Thing", including calling memcpy/memset when correct. -Steve
Apr 07 2014
parent reply "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Mike"  wrote in message news:iflykgpsrmdbfhuwzqtp forum.dlang.org...

 So, my question remains:  If I'm porting D to a platform that has no C 
 library, shouldn't a public, supported function that copies memory be 
 added in the D Runtime?

What platform doesn't have a C library? And you can always cast arrays to ubyte[] before assigning to avoid postblit.
Apr 09 2014
next sibling parent reply "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Mike"  wrote in message news:nxretodrkqizyiukwpze forum.dlang.org...

 My custom, bare-metal, D-only platform has no C library, nor will it ever. 
 And, Yes, I know I can cast, but should I?

Ah.
 So far, it seems D has only been used on one subset of computer systems 
 that happen to have a C library, but some of us our trying to break some 
 new ground with D.  I'm challenging the "D is only useful on PCs with a C 
 library" bias and looking to see if D has the will to stand on its own two 
 feet instead of leaning on C.

druntime and phobos both assume a c library is present, and this is a very safe assumption to make. It's like assuming that bytes are 8 bits - you can certainly create platforms where this is true but those uses are not worth worrying about. For a bare metal runtime that doesn't support C, it makes sense to only have a single runtime and add all the needed D and C runtime functions there. So:
 It seems the D Runtime and Phobos also often make use of memcpy and 
 memset, but they do so through core.stdc which doesn't appear to be part 
 of the D Runtime.  Was this just a matter of convenience or thoughtful 
 design?  I'm guessing the former based on the fact that core.stdc is not 
 in the official language reference.

I don't think you can read that much into the lack of documentation. I think it's a combination of "these are already documented on other sites" + "these are discourages" + "copying documentation has licensing issues". The interface to the c library is a part of druntime and here to stay.
 * Were these methods intentionally omitted from the D Runtime, or just 
 taken for granted since they were so conveniently available in the C 
 standard library?

They were _not_ omitted from druntime, they were exposed though core.stdc. (and other places before that) An implementation in druntime was not necessary because all platforms druntime targets provide a c runtime.
 * Should they be added to core.memory or some other module and exposed as 
 official, supported, methods of the D Runtime (hopefully changing their 
 names)?

No, they are already exposed via core.stdc. So, if you want to support the full interface that druntime does, you will need to implement these functions somewhere. There is no point in adding a second interface to them in upstream druntime.
Apr 09 2014
parent "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Mike"  wrote in message news:ycvyulqzunbsuweipais forum.dlang.org...

 Let me ask you this:  From a principled D design point of view, is 
 core.stdc an interface only for ports with a C library, an implementation 
 detail of the current druntime that phobos is circumventing, or an 
 official, supported interface that all complying ports must implement?

From a principled D design point of view, only targets with a C library are supported. core.stdc is part of the public interface of druntime (as is everything in core) and not an implementation. There is no real definition in D for what a 'complying port' is, but many D applications and libraries will not run if you do not implement the commonly-used parts of core.stdc. Full coverage is likely unrealistic for a bare-metal implementation.
 If the latter, why is it non-existing in the library reference?

Because there is no ddoc for these files, and nobody has gone to the effort to make them show up on the website. Whatever the original intention, enough code using them exists that removing them will never be an option. They probably should just link to the official c-lib documentation.
Apr 09 2014
prev sibling next sibling parent reply "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"monarch_dodra"  wrote in message 
news:pdzhmmnjxclrjtkgupoz forum.dlang.org...

 I think arguably, there should be D equivalents of the C runtime 
 functions, if only for the added safety of slices (for example, memchr 
 could be 100% certifiably safe), and to avoid the rampant deduplication of 
 things as in the above, because of CTFE.

The utility of a 'slightly safer' layer between the C functions and the phobos range-based functions is questionable IMO. eg in cases where memcpy on arrays is safe, phobos' copy would likely optimize to the exact same code.
Apr 09 2014
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
09-Apr-2014 16:56, Daniel Murphy пишет:
 "monarch_dodra"  wrote in message
 news:pdzhmmnjxclrjtkgupoz forum.dlang.org...

 I think arguably, there should be D equivalents of the C runtime
 functions, if only for the added safety of slices (for example, memchr
 could be 100% certifiably safe), and to avoid the rampant
 deduplication of things as in the above, because of CTFE.

The utility of a 'slightly safer' layer between the C functions and the phobos range-based functions is questionable IMO. eg in cases where memcpy on arrays is safe, phobos' copy would likely optimize to the exact same code.

Pros: 1. No extra scuffolding for CTFE. 2. Type-safe 3. Optional (non-release, etc.) bounds-checking. 4. Less error prone interface, e.g. memset is a known source of bugs in C, even for tried and true code bases. 5. More flexible, see e.g. memchr as an example. Cons: A set of primitives to learn. -- Dmitry Olshansky
Apr 10 2014
parent reply "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Dmitry Olshansky"  wrote in message news:li5lfd$203s$1 digitalmars.com... 

 Pros:
 
 1. No extra scuffolding for CTFE.
 2. Type-safe
 3. Optional (non-release, etc.) bounds-checking.
 4. Less error prone interface, e.g. memset is a known source of bugs in 
 C, even for tried and true code bases.
 5. More flexible, see e.g. memchr as an example.
 
 Cons:
 A set of primitives to learn.

I don't know why you'd ever use memcpy/memset when you have: arr[] = 0; foreach(ref v; arr) v = 0; etc
Apr 10 2014
next sibling parent "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"monarch_dodra"  wrote in message 
news:obksandhskaldxplghme forum.dlang.org...

 Because those are assignments.

You mean they might call postblit etc? I'd still rather cast to ubyte[] before assigning than call memset.
Apr 10 2014
prev sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
10-Apr-2014 16:22, Daniel Murphy пишет:
 "Dmitry Olshansky"  wrote in message news:li5lfd$203s$1 digitalmars.com...
 Pros:

 1. No extra scuffolding for CTFE.
 2. Type-safe
 3. Optional (non-release, etc.) bounds-checking.
 4. Less error prone interface, e.g. memset is a known source of bugs
 in C, even for tried and true code bases.
 5. More flexible, see e.g. memchr as an example.

 Cons:
 A set of primitives to learn.

I don't know why you'd ever use memcpy/memset when you have: arr[] = 0;

Calls postblit/dtor etc.? What I if I need bitwise assignment? (which is what memcpy is all about) Yes, and with CTFE please. See even std.algo.move and std.algo.swap...
 foreach(ref v; arr)
     v = 0;

Same but also sucks performance-wise. There is a reason Phobos is littered with carefully laid out special cases around C-runtime stuff. -- Dmitry Olshansky
Apr 10 2014
prev sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
12-Apr-2014 00:25, monarch_dodra пишет:
 On Friday, 11 April 2014 at 20:17:35 UTC, Brad Anderson wrote:
 This list of secure alternatives to C functions Microsoft has would
 probably be a good source of inspiration when deciding what functions
 should be considered for reimplementation:

 http://msdn.microsoft.com/en-us/library/wd3wzwts.aspx%7C%7Clist

I just wrote this as a rough draft. I'll make "formal" announcement for it sometime soon, if initial feedback is positive. http://wiki.dlang.org/DIP59

Some thoughts. Provide only bitwise comparison, search etc. It's the only case that can reliably benefit from SIMD or hand-written ASM, has problem with CTFE support, etc. Then requiring that T be a POD or built-in type is fine IMHO. Maybe go for copy ---> rawCopy, find --> rawFind (i.e. model after std.algorithm) rather then mimicking C functions. This way we are open to a superset of C-runtime ;) -- Dmitry Olshansky
Apr 12 2014
prev sibling next sibling parent "Mike" <none none.com> writes:
On Monday, 7 April 2014 at 12:38:55 UTC, Steven Schveighoffer 
wrote:
 I think using using D's safer syntax of

 a[] = b[]

 and

 a[] = 0;

 Is much preferable to using memcpy/memset directly.

 They should do the "Right Thing", including calling 
 memcpy/memset when correct.

Oh! that is SO much better. I should have known. Thank you.
Apr 07 2014
prev sibling next sibling parent Orvid King <blah38621 gmail.com> writes:
I know that with DMD at least, memcpy, memset, and memcmp are all
intrinsics, meaning the compiler emits their implementations directly,
thus why there isn't a body for them in druntime.

On 4/7/14, Mike <none none.com> wrote:
 On Monday, 7 April 2014 at 12:38:55 UTC, Steven Schveighoffer
 wrote:
 I think using using D's safer syntax of

 a[] = b[]

 and

 a[] = 0;

 Is much preferable to using memcpy/memset directly.

 They should do the "Right Thing", including calling
 memcpy/memset when correct.

Oh! that is SO much better. I should have known. Thank you.

Apr 07 2014
prev sibling next sibling parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Monday, 7 April 2014 at 13:17:37 UTC, Orvid King wrote:
 I know that with DMD at least, memcpy, memset, and memcmp are 
 all
 intrinsics, meaning the compiler emits their implementations 
 directly,
 thus why there isn't a body for them in druntime.

 On 4/7/14, Mike <none none.com> wrote:
 On Monday, 7 April 2014 at 12:38:55 UTC, Steven Schveighoffer
 wrote:
 I think using using D's safer syntax of

 a[] = b[]

 and

 a[] = 0;

 Is much preferable to using memcpy/memset directly.

 They should do the "Right Thing", including calling
 memcpy/memset when correct.

Oh! that is SO much better. I should have known. Thank you.


What's this for then? https://github.com/D-Programming-Language/druntime/blob/master/src/rt/memset.d
Apr 07 2014
prev sibling next sibling parent "Mike" <none none.com> writes:
On Monday, 7 April 2014 at 14:05:38 UTC, John Colvin wrote:
 What's this for then? 
 https://github.com/D-Programming-Language/druntime/blob/master/src/rt/memset.d

Those appear to be the implementations of the a[] = x syntax as compiler builtins and provide even more of a reason to it. They are internal to the runtime. I was wondering why such methods aren't part of the runtime's interface and I believe I received the answer. I'm still wondering if the usage of memcpy in phobos could be replaced with something more 'D'. a[] = b[] could always be translated to a call to memcpy in the platform's runtime port. Maybe I'll do some tests and submit a pull request.
Apr 07 2014
prev sibling next sibling parent "Mike" <none none.com> writes:
On Monday, 7 April 2014 at 22:42:54 UTC, Mike wrote:
 On Monday, 7 April 2014 at 14:05:38 UTC, John Colvin wrote:
 What's this for then? 
 https://github.com/D-Programming-Language/druntime/blob/master/src/rt/memset.d

Those appear to be the implementations of the a[] = x syntax as compiler builtins and provide even more of a reason to it. They are internal to the runtime. I was wondering why such methods aren't part of the runtime's interface and I believe I received the answer. I'm still wondering if the usage of memcpy in phobos could be replaced with something more 'D'. a[] = b[] could always be translated to a call to memcpy in the platform's runtime port. Maybe I'll do some tests and submit a pull request.

Apologies, John. I see now you were responding to Orvid, not me.
Apr 07 2014
prev sibling next sibling parent "Mike" <none none.com> writes:
On Monday, 7 April 2014 at 12:38:55 UTC, Steven Schveighoffer 
wrote:
 I think using using D's safer syntax of

 a[] = b[]

 and

 a[] = 0;

 Is much preferable to using memcpy/memset directly.

 They should do the "Right Thing", including calling 
 memcpy/memset when correct.

 -Steve

I received a very informative answer on D.learn (http://forum.dlang.org/post/xdnbfbdvruevjgqvbhfb forum.dlang.org) that describes why a function that copies memory still may be needed: "Depending on the type being copied, "a[] = b[]" is not straight up "memory copy" it's the actual full assignment, with postblit and all. That's why the call we want is *really* memcpy." So, my question remains: If I'm porting D to a platform that has no C library, shouldn't a public, supported function that copies memory be added in the D Runtime?
Apr 09 2014
prev sibling next sibling parent "Mike" <none none.com> writes:
On Wednesday, 9 April 2014 at 11:50:13 UTC, Daniel Murphy wrote:
 "Mike"  wrote in message 
 news:iflykgpsrmdbfhuwzqtp forum.dlang.org...

 So, my question remains:  If I'm porting D to a platform that 
 has no C library, shouldn't a public, supported function that 
 copies memory be added in the D Runtime?

What platform doesn't have a C library? And you can always cast arrays to ubyte[] before assigning to avoid postblit.

My custom, bare-metal, D-only platform has no C library, nor will it ever. And, Yes, I know I can cast, but should I? memcpy is used quite often in Phobos. Here's a small sample from std.array: void trustedMemcopy(T[] dest, T[] src) trusted { assert(src.length == dest.length); if (!__ctfe) memcpy(dest.ptr, src.ptr, src.length * T.sizeof); else { dest[] = src[]; } } You could ask the same question there, "Why isn't a cast used instead?" and remove the dependency on the C library. So far, it seems D has only been used on one subset of computer systems that happen to have a C library, but some of us our trying to break some new ground with D. I'm challenging the "D is only useful on PCs with a C library" bias and looking to see if D has the will to stand on its own two feet instead of leaning on C.
Apr 09 2014
prev sibling next sibling parent "monarch_dodra" <monarchdodra gmail.com> writes:
On Wednesday, 9 April 2014 at 12:12:35 UTC, Mike wrote:
 On Wednesday, 9 April 2014 at 11:50:13 UTC, Daniel Murphy wrote:
 "Mike"  wrote in message 
 news:iflykgpsrmdbfhuwzqtp forum.dlang.org...

 So, my question remains:  If I'm porting D to a platform that 
 has no C library, shouldn't a public, supported function that 
 copies memory be added in the D Runtime?

What platform doesn't have a C library? And you can always cast arrays to ubyte[] before assigning to avoid postblit.

My custom, bare-metal, D-only platform has no C library, nor will it ever. And, Yes, I know I can cast, but should I? memcpy is used quite often in Phobos. Here's a small sample from std.array: void trustedMemcopy(T[] dest, T[] src) trusted { assert(src.length == dest.length); if (!__ctfe) memcpy(dest.ptr, src.ptr, src.length * T.sizeof); else { dest[] = src[]; } } You could ask the same question there, "Why isn't a cast used instead?" and remove the dependency on the C library. So far, it seems D has only been used on one subset of computer systems that happen to have a C library, but some of us our trying to break some new ground with D. I'm challenging the "D is only useful on PCs with a C library" bias and looking to see if D has the will to stand on its own two feet instead of leaning on C.

I think arguably, there should be D equivalents of the C runtime functions, if only for the added safety of slices (for example, memchr could be 100% certifiably safe), and to avoid the rampant deduplication of things as in the above, because of CTFE. These may or may not use the C-runtime library. But at that point, it would become a "druntime implementation detail", rather than rampant usage throughout phobos.
Apr 09 2014
prev sibling next sibling parent "Mike" <none none.com> writes:
On Wednesday, 9 April 2014 at 12:50:23 UTC, Daniel Murphy wrote:
 * Were these methods intentionally omitted from the D Runtime, 
 or just taken for granted since they were so conveniently 
 available in the C standard library?

They were _not_ omitted from druntime, they were exposed though core.stdc. (and other places before that) An implementation in druntime was not necessary because all platforms druntime targets provide a c runtime.
 * Should they be added to core.memory or some other module and 
 exposed as official, supported, methods of the D Runtime 
 (hopefully changing their names)?

No, they are already exposed via core.stdc.

Thank you for your thoughtful answer (although I'm not ready to accept it just yet :-) Let me ask you this: From a principled D design point of view, is core.stdc an interface only for ports with a C library, an implementation detail of the current druntime that phobos is circumventing, or an official, supported interface that all complying ports must implement? If the latter, why is it non-existing in the library reference?
Apr 09 2014
prev sibling next sibling parent "monarch_dodra" <monarchdodra gmail.com> writes:
On Thursday, 10 April 2014 at 12:22:37 UTC, Daniel Murphy wrote:
 "Dmitry Olshansky"  wrote in message 
 news:li5lfd$203s$1 digitalmars.com...

 Pros:
 
 1. No extra scuffolding for CTFE.
 2. Type-safe
 3. Optional (non-release, etc.) bounds-checking.
 4. Less error prone interface, e.g. memset is a known source 
 of bugs in C, even for tried and true code bases.
 5. More flexible, see e.g. memchr as an example.
 
 Cons:
 A set of primitives to learn.

I don't know why you'd ever use memcpy/memset when you have: arr[] = 0; foreach(ref v; arr) v = 0; etc

Because those are assignments.
Apr 10 2014
prev sibling next sibling parent "monarch_dodra" <monarchdodra gmail.com> writes:
On Thursday, 10 April 2014 at 16:59:27 UTC, Daniel Murphy wrote:
 "monarch_dodra"  wrote in message 
 news:obksandhskaldxplghme forum.dlang.org...

 Because those are assignments.

You mean they might call postblit etc? I'd still rather cast to ubyte[] before assigning than call memset.

Yeah, but you can't cast to ubyte with CTFE. So you still end up pulling out boilerplate code "at call site", when you could just "deal with it" it a generic way in a named function.
Apr 10 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 10 Apr 2014 16:13:31 -0400, Dmitry Olshansky  =

<dmitry.olsh gmail.com> wrote:

 10-Apr-2014 16:22, Daniel Murphy =D0=BF=D0=B8=D1=88=D0=B5=D1=82:
 "Dmitry Olshansky"  wrote in message  =


 news:li5lfd$203s$1 digitalmars.com...
 Pros:

 1. No extra scuffolding for CTFE.
 2. Type-safe
 3. Optional (non-release, etc.) bounds-checking.
 4. Less error prone interface, e.g. memset is a known source of bugs=



 in C, even for tried and true code bases.
 5. More flexible, see e.g. memchr as an example.

 Cons:
 A set of primitives to learn.

I don't know why you'd ever use memcpy/memset when you have: arr[] =3D 0;

Calls postblit/dtor etc.? What I if I need bitwise assignment? (which =

 what memcpy is all about)
 Yes, and with CTFE please.
 See even std.algo.move and std.algo.swap...

 foreach(ref v; arr)
     v =3D 0;

Same but also sucks performance-wise. There is a reason Phobos is littered with carefully laid out special =

 cases around C-runtime stuff.

I think a simple set of functions would be sufficient. It wouldn't even = be = that complex. core.memops -Steve
Apr 10 2014
prev sibling next sibling parent "Brad Anderson" <eco gnuk.net> writes:
On Wednesday, 9 April 2014 at 12:38:20 UTC, monarch_dodra wrote:
 On Wednesday, 9 April 2014 at 12:12:35 UTC, Mike wrote:
 On Wednesday, 9 April 2014 at 11:50:13 UTC, Daniel Murphy 
 wrote:
 "Mike"  wrote in message 
 news:iflykgpsrmdbfhuwzqtp forum.dlang.org...

 So, my question remains:  If I'm porting D to a platform 
 that has no C library, shouldn't a public, supported 
 function that copies memory be added in the D Runtime?

What platform doesn't have a C library? And you can always cast arrays to ubyte[] before assigning to avoid postblit.

My custom, bare-metal, D-only platform has no C library, nor will it ever. And, Yes, I know I can cast, but should I? memcpy is used quite often in Phobos. Here's a small sample from std.array: void trustedMemcopy(T[] dest, T[] src) trusted { assert(src.length == dest.length); if (!__ctfe) memcpy(dest.ptr, src.ptr, src.length * T.sizeof); else { dest[] = src[]; } } You could ask the same question there, "Why isn't a cast used instead?" and remove the dependency on the C library. So far, it seems D has only been used on one subset of computer systems that happen to have a C library, but some of us our trying to break some new ground with D. I'm challenging the "D is only useful on PCs with a C library" bias and looking to see if D has the will to stand on its own two feet instead of leaning on C.

I think arguably, there should be D equivalents of the C runtime functions, if only for the added safety of slices (for example, memchr could be 100% certifiably safe), and to avoid the rampant deduplication of things as in the above, because of CTFE. These may or may not use the C-runtime library. But at that point, it would become a "druntime implementation detail", rather than rampant usage throughout phobos.

This list of secure alternatives to C functions Microsoft has would probably be a good source of inspiration when deciding what functions should be considered for reimplementation: http://msdn.microsoft.com/en-us/library/wd3wzwts.aspx%7C%7Clist
Apr 11 2014
prev sibling next sibling parent "Brad Anderson" <eco gnuk.net> writes:
On Friday, 11 April 2014 at 20:17:35 UTC, Brad Anderson wrote:
 This list of secure alternatives to C functions Microsoft has 
 would probably be a good source of inspiration when deciding 
 what functions should be considered for reimplementation:

 http://msdn.microsoft.com/en-us/library/wd3wzwts.aspx%7C%7Clist

Oops, broken link. http://msdn.microsoft.com/en-us/library/wd3wzwts.aspx
Apr 11 2014
prev sibling next sibling parent "monarch_dodra" <monarchdodra gmail.com> writes:
On Friday, 11 April 2014 at 20:17:35 UTC, Brad Anderson wrote:
 This list of secure alternatives to C functions Microsoft has 
 would probably be a good source of inspiration when deciding 
 what functions should be considered for reimplementation:

 http://msdn.microsoft.com/en-us/library/wd3wzwts.aspx%7C%7Clist

I just wrote this as a rough draft. I'll make "formal" announcement for it sometime soon, if initial feedback is positive. http://wiki.dlang.org/DIP59
Apr 11 2014
prev sibling parent "Brad Anderson" <eco gnuk.net> writes:
On Friday, 11 April 2014 at 20:25:21 UTC, monarch_dodra wrote:
 On Friday, 11 April 2014 at 20:17:35 UTC, Brad Anderson wrote:
 This list of secure alternatives to C functions Microsoft has 
 would probably be a good source of inspiration when deciding 
 what functions should be considered for reimplementation:

 http://msdn.microsoft.com/en-us/library/wd3wzwts.aspx%7C%7Clist

I just wrote this as a rough draft. I'll make "formal" announcement for it sometime soon, if initial feedback is positive. http://wiki.dlang.org/DIP59

I know :). We #d IRC users see all! (Vladimir Panteleev's DFeed bot announces basically everything going on everywhere in D).
Apr 11 2014