www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Allocating a wstring on the stack (no GC)?

reply "Maxime Chevalier-Boisvert" <maximechevalierb gmail.com> writes:
I have a very specific use case (JIT compiler) in which I have a 
pre-allocated array of wchar string data stored somewhere in 
memory. I'd like to be able to create a temporary D wstring 
object to pass this as a "regular" string to other functions. For 
performance reasons, it would be preferable not to dynamically 
allocate or copy any data. Dynamically allocating the strings 
tends to trigger the D GC which severely impacts the performance.

So, my question is, would it be possible for me to allocate a 
wstring object on the stack, and manually set its string data 
pointer and length? If so, how? Your wizardly help is much 
appreciated.
May 07 2014
next sibling parent reply "Brad Anderson" <eco gnuk.net> writes:
On Wednesday, 7 May 2014 at 18:26:08 UTC, Maxime 
Chevalier-Boisvert wrote:

 I have a very specific use case (JIT compiler) in which I have 
 a pre-allocated array of wchar string data stored somewhere in 
 memory. I'd like to be able to create a temporary D wstring 
 object to pass this as a "regular" string to other functions. 
 For performance reasons, it would be preferable not to 
 dynamically allocate or copy any data. Dynamically allocating 
 the strings tends to trigger the D GC which severely impacts 
 the performance.

 So, my question is, would it be possible for me to allocate a 
 wstring object on the stack, and manually set its string data 
 pointer and length? If so, how? Your wizardly help is much 
 appreciated.


Unless I'm misunderstanding it should be as simple as:

wchar[100] stackws; // alloca() if you need it to be dynamically 
sized.

A slice of this static array behaves just like a slice of a 
dynamic array.
May 07 2014
next sibling parent reply "Meta" <jared771 gmail.com> writes:
On Wednesday, 7 May 2014 at 18:29:23 UTC, Brad Anderson wrote:

 Unless I'm misunderstanding it should be as simple as:

 wchar[100] stackws; // alloca() if you need it to be 
 dynamically sized.

 A slice of this static array behaves just like a slice of a 
 dynamic array.


But you should avoid slicing the static array unless it's via 
arr.dup.
May 07 2014
next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Meta:


 But you should avoid slicing the static array unless it's via 
 arr.dup.


Slicing fixed size arrays is often necessary, because many Phobos 
functions don't accept fixed size arrays, they force you to lose 
the compile-time knowledge of the length.

Bye,
bearophile
May 07 2014
prev sibling parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Wednesday, 7 May 2014 at 18:34:10 UTC, Meta wrote:

 On Wednesday, 7 May 2014 at 18:29:23 UTC, Brad Anderson wrote:

 Unless I'm misunderstanding it should be as simple as:

 wchar[100] stackws; // alloca() if you need it to be 
 dynamically sized.

 A slice of this static array behaves just like a slice of a 
 dynamic array.


 But you should avoid slicing the static array unless it's via 
 arr.dup.


I use slicing of fixed size arrays a lot.

Sure, you have to keep track of the memory to make sure it 
doesn't survive past the current scope (or at least isn't used 
past that point, depending on how security critical your 
application is), but that's often trivial to do and saving the 
extra allocation can make a big difference to performance.
May 08 2014
parent "bearophile" <bearophileHUGS lycos.com> writes:
John Colvin:


 Sure, you have to keep track of the memory to make sure it 
 doesn't survive past the current scope (or at least isn't used 
 past that point, depending on how security critical your 
 application is), but that's often trivial to do and saving the 
 extra allocation can make a big difference to performance.


It's also bug-prone, so better to limit that usage to only the 
parts of the code that need the performance difference (often 
it's the profiler that tells where to use that). Rust language 
allows to perform similar things safely.

Bye,
bearophile
May 08 2014
prev sibling parent reply "Maxime Chevalier-Boisvert" <maximechevalierb gmail.com> writes:
 Unless I'm misunderstanding it should be as simple as:

 wchar[100] stackws; // alloca() if you need it to be 
 dynamically sized.

 A slice of this static array behaves just like a slice of a 
 dynamic array.


I do need it to be dynamically sized. I also want to avoid 
copying my string data if possible. Basically, I just want to 
create a wstring "view" on an existing "raw" buffer that exists 
in memory somewhere, based on a pointer to this buffer and its 
length.

Side note: wouldn't alloca just produce a wchar*, not an array 
(wouldn't have length information)?
May 07 2014
next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Maxime Chevalier-Boisvert:


 I do need it to be dynamically sized.


But often you can determine statically a maximum length of the 
string, so you can use a fixed size stack buffer and slice it 
with a dynamic length. If this is not acceptable, then use alloca.



 Basically, I just want to create a wstring "view" on an 
 existing "raw" buffer that exists in memory somewhere, based on 
 a pointer to this buffer and its length.


This is named slicing. You can also slice a 
global/static/__gshared buffer.



 Side note: wouldn't alloca just produce a wchar*, not an array 
 (wouldn't have length information)?


alloca returns a void*, then you can cast it to the pointer type 
you want, and then you slice the pointer:

auto ptr = cast(wchar*)alloca(wchar.sizeof * len);
if (ptr == null) throw new Error("...");
auto mySlice = ptr[0 .. len];

Bye,
bearophile
May 07 2014
parent reply "Maxime Chevalier-Boisvert" <maximechevalierb gmail.com> writes:
 This is named slicing. You can also slice a 
 global/static/__gshared buffer.



 alloca returns a void*, then you can cast it to the pointer 
 type you want, and then you slice the pointer:

 auto ptr = cast(wchar*)alloca(wchar.sizeof * len);
 if (ptr == null) throw new Error("...");
 auto mySlice = ptr[0 .. len];


Is the slice going to be allocated on the stack? (I imagine the 
answer is yes)

Thanks for your help Mr Bear.
May 07 2014
next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Maxime Chevalier-Boisvert:


 Is the slice going to be allocated on the stack? (I imagine the 
 answer is yes)


Slicing doesn't change where the data is allocated. Slicing means 
just creating a new struct that contains a length and pointer to 
the data (and the struct itself is allocated in-place. So it's 
allocated on the stack, or inside a struct instance, or inside a 
class instance, or on the data segment, etc).



 Thanks for your help Mr Bear.


You are welcome :-)

Bye,
bearophile
May 07 2014
parent reply "Maxime Chevalier-Boisvert" <maximechevalierb gmail.com> writes:
 Is the slice going to be allocated on the stack? (I imagine 
 the answer is yes)


 Slicing doesn't change where the data is allocated. Slicing 
 means just creating a new struct that contains a length and 
 pointer to the data (and the struct itself is allocated 
 in-place. So it's allocated on the stack, or inside a struct 
 instance, or inside a class instance, or on the data segment, 
 etc).


Indeed. It's the struct representing the slice I was asking 
about. Off to test out the performance impact :)
May 07 2014
parent "monarch_dodra" <monarchdodra gmail.com> writes:
On Wednesday, 7 May 2014 at 19:18:09 UTC, Maxime 
Chevalier-Boisvert wrote:

 Is the slice going to be allocated on the stack? (I imagine 
 the answer is yes)


 Slicing doesn't change where the data is allocated. Slicing 
 means just creating a new struct that contains a length and 
 pointer to the data (and the struct itself is allocated 
 in-place. So it's allocated on the stack, or inside a struct 
 instance, or inside a class instance, or on the data segment, 
 etc).


 Indeed. It's the struct representing the slice I was asking 
 about. Off to test out the performance impact :)


I slice is really nothing more than a fat pointer (a pointer + a 
size_t). You don't really allocate it any more than you allocate 
a pointer when you do:
int* p = &someInt;
May 07 2014
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/7/2014 12:03 PM, Maxime Chevalier-Boisvert wrote:

 auto ptr = cast(wchar*)alloca(wchar.sizeof * len);
 if (ptr == null) throw new Error("...");
 auto mySlice = ptr[0 .. len];


 Is the slice going to be allocated on the stack? (I imagine the answer is yes)


Yes (slices do not copy).
May 18 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Wednesday, 7 May 2014 at 18:41:17 UTC, Maxime 
Chevalier-Boisvert wrote:

 Basically, I just want to create a wstring "view" on an 
 existing "raw" buffer that exists in memory somewhere, based on 
 a pointer to this buffer and its length.


Looks like you actually don't need to allocate anything at all. 
Any slice is just a struct with 2 fields, it does not own data 
and can be used as a view to anything:

void* some_external_data;
size_t external_length;

void foo()
{
     auto str = 
cast(wstring[])(some_external_data[0..external_length]);
     // this will create a stack instance of slice struct with 
`.ptr` field pointing to same address as `some_external_data`
}
May 07 2014
prev sibling next sibling parent Benjamin Thaut <code benjamin-thaut.de> writes:
Am 07.05.2014 20:41, schrieb Maxime Chevalier-Boisvert:

 I do need it to be dynamically sized. I also want to avoid copying my
 string data if possible. Basically, I just want to create a wstring
 "view" on an existing "raw" buffer that exists in memory somewhere,
 based on a pointer to this buffer and its length.


If you just need a view of the raw buffer that already exists, why don't 
you slice it directly? Is it neccessary that you make a copy of it?

void[] rawBuffer = ...;
size_t offset = ...;

assert(rawBuffer.length >= offset + stringLength * wchar.sizeof, "out of 
bounds access");
const(wchar)[] stringView = (cast(const(wchar)*)rawBuffer.ptr + 
offset)[0..stringLength];

Kind Regards
Benjamin Thaut
May 07 2014
prev sibling parent "Regan Heath" <regan netmail.co.nz> writes:
On Wed, 07 May 2014 19:41:16 +0100, Maxime Chevalier-Boisvert  
<maximechevalierb gmail.com> wrote:


 Unless I'm misunderstanding it should be as simple as:

 wchar[100] stackws; // alloca() if you need it to be dynamically sized.

 A slice of this static array behaves just like a slice of a dynamic  
 array.


 I do need it to be dynamically sized. I also want to avoid copying my  
 string data if possible. Basically, I just want to create a wstring  
 "view" on an existing "raw" buffer that exists in memory somewhere,  
 based on a pointer to this buffer and its length.


import std.stdio;
import core.stdc.stdlib : malloc;
import core.stdc.wchar_ : wcscpy;

wchar[] toWChar(const void *ptr, int len)
{
	// Cast pointer to wchar*, create slice (on the heap?) from it (copies no  
data)
	return (cast(wchar*)ptr)[0..len];
}

void main()
{
	// Pre-existing data
	int len = 12;
	wchar *ptr = cast(wchar*)malloc(len * wchar.sizeof);
	wcscpy(ptr, "Hello World");
	
	// Create slice of data
	wchar[] slice = toWChar(ptr, len);
	writefln("%s", slice);
}

R

-- 
Using Opera's revolutionary email client: http://www.opera.com/mail/
May 09 2014
prev sibling next sibling parent "Jesse Phillips" <Jesse.K.Phillips+D gmail.com> writes:
On Wednesday, 7 May 2014 at 18:26:08 UTC, Maxime 
Chevalier-Boisvert wrote:

 I have a very specific use case (JIT compiler) in which I have 
 a pre-allocated array of wchar string data stored somewhere in 
 memory. I'd like to be able to create a temporary D wstring 
 object to pass this as a "regular" string to other functions. 
 For performance reasons, it would be preferable not to 
 dynamically allocate or copy any data. Dynamically allocating 
 the strings tends to trigger the D GC which severely impacts 
 the performance.

 So, my question is, would it be possible for me to allocate a 
 wstring object on the stack, and manually set its string data 
 pointer and length? If so, how? Your wizardly help is much 
 appreciated.


As mentioned a slice is the stack allocated view which you 
desire, however your selection of terms, "wchar string" and 
"wstring" suggest to me that you'll also be dealing with a 
conversion from mutable to immutable. To put my expectations of 
these terms into types:

wchar string : wchar[]
wstring : immutable(wchar)[]

If I have understood this correctly then you'll also want to look 
toward std.exception.assumeUnique() which you can call on a slice 
(wchar[]). However, you should not use this if you will maintain 
a mutable reference to your data.
May 07 2014
prev sibling next sibling parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Wednesday, 7 May 2014 at 18:26:08 UTC, Maxime 
Chevalier-Boisvert wrote:

 I have a very specific use case (JIT compiler) in which I have 
 a pre-allocated array of wchar string data stored somewhere in 
 memory. I'd like to be able to create a temporary D wstring 
 object to pass this as a "regular" string to other functions. 
 For performance reasons, it would be preferable not to 
 dynamically allocate or copy any data. Dynamically allocating 
 the strings tends to trigger the D GC which severely impacts 
 the performance.

 So, my question is, would it be possible for me to allocate a 
 wstring object on the stack, and manually set its string data 
 pointer and length? If so, how? Your wizardly help is much 
 appreciated.


If I read your question correctly, you just want to use wstring; 
slices in D already work how you want.

auto len = 100;
auto a = new char[len];
// a (the pointer and length) is on the stack, the new memory of 
length len is on the GC heap

auto b = a;
// b is also on the stack, with the same pointer and length as a

auto c = b[3 .. 40];
// c is also on the stack, with pointer equal to b.ptr + 3 and 
length 37
May 08 2014
prev sibling next sibling parent Denis Shelomovskij <verylonglogin.reg gmail.com> writes:
07.05.2014 22:26, Maxime Chevalier-Boisvert пишет:

 I have a very specific use case (JIT compiler) in which I have a
 pre-allocated array of wchar string data stored somewhere in memory. I'd
 like to be able to create a temporary D wstring object to pass this as a
 "regular" string to other functions. For performance reasons, it would
 be preferable not to dynamically allocate or copy any data. Dynamically
 allocating the strings tends to trigger the D GC which severely impacts
 the performance.

 So, my question is, would it be possible for me to allocate a wstring
 object on the stack, and manually set its string data pointer and
 length? If so, how? Your wizardly help is much appreciated.


If you have a preallocated data, just using slicing will be enough. A 
result will be `wstring` for immutable data or `const(wchar)[]` for 
non-immutable, if you respect a typesystem.

If a new data is generated but you want it to be put on stack if it's 
small enough (you can't put big data on stack anyway) you need some 
allocation facility. E.g. `unstd.memory.allocation.tempAlloc` [1] will 
do the work.

[1] 
http://denis-sh.bitbucket.org/unstandard/unstd.memory.allocation.html#tempAlloc

-- 
Денис В. Шеломовский
Denis V. Shelomovskij
May 18 2014
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/7/2014 11:26 AM, Maxime Chevalier-Boisvert wrote:

 I have a very specific use case (JIT compiler) in which I have a pre-allocated
 array of wchar string data stored somewhere in memory. I'd like to be able to
 create a temporary D wstring object to pass this as a "regular" string to other
 functions. For performance reasons, it would be preferable not to dynamically
 allocate or copy any data. Dynamically allocating the strings tends to trigger
 the D GC which severely impacts the performance.

 So, my question is, would it be possible for me to allocate a wstring object on
 the stack, and manually set its string data pointer and length? If so, how?
Your
 wizardly help is much appreciated.


Checkout std.internal.scopebuffer (new for 2.066)
May 18 2014