www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Shared pain

reply Steve Teale <steve.teale britseyeview.com> writes:
I had D code that provided a basis for creation of Windows services, 
which I have just tried to get working with the latest D2.

No dice.

The point of failure was in this method

static void StartService()
{
   if (!StartServiceCtrlDispatcherA(cast(SERVICE_TABLE_ENTRY *) &_sta[0]))
   {
      ...
   }
}

Where _sta is an array of SERVICE_TABLE_ENTRY structs. The bits of code 
that install and remove the service worked fine, but the service control 
dispatcher was clearly not happy about these structs being in TLS, and 
the service crashed immediately when I tried to start it.

The class also has a couple of methods with signatures like:

extern (Windows) static export void service_ctrl(uint dwCtrlCode) {...}

That's probably not relevant, but I think it contributed to a host of 
errors that finally made be take a big step back.

I re-implemented the thing in the style used for OOP in C.

struct ServiceBase
{
   SERVICE_TABLE_ENTRY[2] _sta;
   ...
}

__gshared ServiceBase __sb;

void initialize(ServiceBase* sb) { ... }
void whatever(ServiceBase* sb ...) { ... }

and used the __sb global within the functions with the Windows signature, 
which just became

extern (Windows) export void service_ctrl(uint dwCtrlCode) {...}

Now it's working again, but I want to change it into idiomatically sound 
D code.

Style suggestions please!

Steve
Nov 18 2010
next sibling parent Jason House <jason.james.house gmail.com> writes:
I'm not familiar with the API, but are you able to declare your original _sta
as immutable and call it a day? Immutable data should have the same protection
as __gshared but with no implications of bypassing the type system.

If that's not helpful, can you give more details like calling patterns? Is the
function called from one thread or from many? My quick read of the windows
documentation makes it seem like that's true. Probably unimportant, but I'd use
_sta.ptr instead of &_sta[0] because the former more clearly implies array
pointer to me, but that may be my own quirky style. 

Steve Teale Wrote:

 I had D code that provided a basis for creation of Windows services, 
 which I have just tried to get working with the latest D2.
 
 No dice.
 
 The point of failure was in this method
 
 static void StartService()

 {

    if (!StartServiceCtrlDispatcherA(cast(SERVICE_TABLE_ENTRY *) &_sta[0]))

    {
       ...

    }

 }
 
 Where _sta is an array of SERVICE_TABLE_ENTRY structs. The bits of code 
 that install and remove the service worked fine, but the service control 
 dispatcher was clearly not happy about these structs being in TLS, and 
 the service crashed immediately when I tried to start it.
 
 The class also has a couple of methods with signatures like:
 
 extern (Windows) static export void service_ctrl(uint dwCtrlCode) {...}
 
 That's probably not relevant, but I think it contributed to a host of 
 errors that finally made be take a big step back.
 
 I re-implemented the thing in the style used for OOP in C.
 
 struct ServiceBase
 {
    SERVICE_TABLE_ENTRY[2] _sta;
    ...
 }
 
 __gshared ServiceBase __sb;
 
 void initialize(ServiceBase* sb) { ... }
 void whatever(ServiceBase* sb ...) { ... }
 
 and used the __sb global within the functions with the Windows signature, 
 which just became
 
 extern (Windows) export void service_ctrl(uint dwCtrlCode) {...}
 
 Now it's working again, but I want to change it into idiomatically sound 
 D code.
 
 Style suggestions please!
 
 Steve

Nov 18 2010
prev sibling next sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 18 Nov 2010 06:26:39 -0500, Steve Teale  
<steve.teale britseyeview.com> wrote:

 I had D code that provided a basis for creation of Windows services,
 which I have just tried to get working with the latest D2.

 No dice.

 The point of failure was in this method

 static void StartService()

 {

    if (!StartServiceCtrlDispatcherA(cast(SERVICE_TABLE_ENTRY *)  
 &_sta[0]))

    {
       ...

    }

 }

 Where _sta is an array of SERVICE_TABLE_ENTRY structs. The bits of code
 that install and remove the service worked fine, but the service control
 dispatcher was clearly not happy about these structs being in TLS, and
 the service crashed immediately when I tried to start it.

I think this has something to do with how services start. From this page: http://msdn.microsoft.com/en-us/library/ms685990(v=VS.85).aspx I see that StartServiceCtrlDispatcher creates its own thread to run ServiceMain, which most likely does not call the D thread initialization routines (not sure). TLS may not be properly set up inside your thread that is running ServiceMain, and probably the static this() module functions have not been called.
 The class also has a couple of methods with signatures like:

 extern (Windows) static export void service_ctrl(uint dwCtrlCode) {...}

 That's probably not relevant, but I think it contributed to a host of
 errors that finally made be take a big step back.

 I re-implemented the thing in the style used for OOP in C.

 struct ServiceBase
 {
    SERVICE_TABLE_ENTRY[2] _sta;
    ...
 }

 __gshared ServiceBase __sb;

 void initialize(ServiceBase* sb) { ... }
 void whatever(ServiceBase* sb ...) { ... }

 and used the __sb global within the functions with the Windows signature,
 which just became

 extern (Windows) export void service_ctrl(uint dwCtrlCode) {...}

 Now it's working again, but I want to change it into idiomatically sound
 D code.

 Style suggestions please!

I think using a class in D is a good idea. But what you may need is global functions which forward to your class methods. This is how I would probably do it. -Steve
Nov 18 2010
parent reply Stanislav Blinov <blinov loniir.ru> writes:
18.11.2010 21:53, Steve Teale пишет:
 As you can imagine, this is terrible to debug - printf or similar is bad
 enough, but for the service I'm pushing entries into the Windows event
 logger system. [...]

 But first I think I'll stop and do some carpentry for a couple of days.
 Then eventually something might dawn on me.

strings to the 'debugger'. DebugView application from SysInternals suite nicely prints those strings, and also times them.
Nov 19 2010
parent Stanislav Blinov <blinov loniir.ru> writes:
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

19.11.2010 19:08, Steve Teale пишет:
 On Fri, 19 Nov 2010 16:58:26 +0300, Stanislav Blinov wrote:

 Have you tried OutputDebugString()? It's WinAPI function that sends
 strings to the 'debugger'. DebugView application from SysInternals suite
 nicely prints those strings, and also times them.

but it's still a pain in the arse.

It's better precisely in that it doesn't need event log. Cluttering system log with debug messages is not a thing I personally like :)
 In any case, if it's a service, where does it output the debug string to?

It's not a console output. That function generates special Windows debug event that can be captured by external application (i.e. debugger, but not necessarily). It is very useful with things like services when console is not available and you don't feel like creating log file. SysInternals suite has a DebugView application that captures these debug strings from currently running processes and displays them as a kind of log: they are numbered and timed (see attached screenshot).
Nov 19 2010
prev sibling next sibling parent Steve Teale <steve.teale britseyeview.com> writes:
On Thu, 18 Nov 2010 08:33:12 -0500, Steven Schveighoffer wrote:

 
 I think using a class in D is a good idea.  But what you may need is
 global functions which forward to your class methods.  This is how I
 would probably do it.
 
 -Steve

Steve, Thanks. I kind of went that way. Having done the laborious C OOP style thing, I put most of the methods back into the struct, just leaving the extern(Windows) functions at global scope - they referred to the __gshare'd object explicitly - I hate that. Then I hid all of the crap behind a class that sneakily referred to the __gshared struct, but at least now to an external user of the module the interface looks much the same as it did before. Now it works OK with the test case example - a service that just writes entries in the event log. However, the real service I'm trying to update works fine if I run it in debug mode (i.e. basically as just an executable), but if I run it as a service it seems to crash the first time I use a RegExp.find(). As you can imagine, this is terrible to debug - printf or similar is bad enough, but for the service I'm pushing entries into the Windows event logger system. Fortunately my D class for doing that has survived the D2 changes. I'm going to try tweaking the service so it will interact with the desktop, then at least I can revert to printf. But first I think I'll stop and do some carpentry for a couple of days. Then eventually something might dawn on me. Steve
Nov 18 2010
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 18 Nov 2010 13:53:53 -0500, Steve Teale  
<steve.teale britseyeview.com> wrote:

 On Thu, 18 Nov 2010 08:33:12 -0500, Steven Schveighoffer wrote:

 I think using a class in D is a good idea.  But what you may need is
 global functions which forward to your class methods.  This is how I
 would probably do it.

 -Steve

Steve, Thanks. I kind of went that way. Having done the laborious C OOP style thing, I put most of the methods back into the struct, just leaving the extern(Windows) functions at global scope - they referred to the __gshare'd object explicitly - I hate that. Then I hid all of the crap behind a class that sneakily referred to the __gshared struct, but at least now to an external user of the module the interface looks much the same as it did before. Now it works OK with the test case example - a service that just writes entries in the event log. However, the real service I'm trying to update works fine if I run it in debug mode (i.e. basically as just an executable), but if I run it as a service it seems to crash the first time I use a RegExp.find().

As I said before, I don't know if the thread being created by the windows service procedure is properly initializing the D modules of the library/runtime. Try as the first line of ServiceMain to initialize the current thread: auto mythread = thread_attachThis(); see http://www.digitalmars.com/d/2.0/phobos/core_thread.html -Steve
Nov 18 2010
prev sibling next sibling parent Steve Teale <steve.teale britseyeview.com> writes:
 As I said before, I don't know if the thread being created by the
 windows service procedure is properly initializing the D modules of the
 library/runtime.  Try as the first line of ServiceMain to initialize the
 current thread:
 
 auto mythread = thread_attachThis();
 
 see http://www.digitalmars.com/d/2.0/phobos/core_thread.html
 
 -Steve

Steve, I don't think it gets as far as ServiceMain. Anyway, I tried it there, and at the point just before the first WINAPI call, but it still crashes. I can fix it, by making my statically initialized RegExp objects __gshared or immutable, but the latter involves inserting a slew of tedious casts into my XML parser, since RexExp calls are used all over the place. See my separate post for my thoughts on that. Thanks Steve
Nov 18 2010
prev sibling next sibling parent reply Steve Teale <steve.teale britseyeview.com> writes:
On Thu, 18 Nov 2010 11:26:39 +0000, Steve Teale wrote:

 I had D code that provided a basis for creation of Windows services,
 which I have just tried to get working with the latest D2.
 
 No dice.
 

static data structures containing function pointers will now cause a Windows service to crash. For happiness, these must be forced into the global data segment. I have demonstrated this within a vestigial service for a couple of cases - plain old struct containing an int and a function pointer, and for Regexp objects by using and not using __gshared on the offending objects. Now that I'm reasonably sure what's happening, I ought to be able to make these things work by making such static members immutable, and initializing them in static this(). However, I then run into a slew of complier errors. Lets start with this simple framework: import std.stdio; import std.regexp; class ThingWithStatic { static RegExp rex; string ns; static this() { rex = RegExp("ab+a"); } this() { ns = "abbaabba"; } int getFromStatic() { return rex.find(ns); } } void main() { ThingWithStatic tws = new ThingWithStatic(); writefln("%d", tws.getFromStatic()); } This compiles fine, but if I include this class in a service, and call getFromStatic(), the service will crash. If I change to: __gshared RegExp rex; It compiles fine, and the service does not crash. If I change to: static immutable RegExp rex; Then I get errors: triv.d(11): Error: cannot implicitly convert expression (opCall("ab +a",null)) of type std.regexp.RegExp to immutable(RegExp) triv.d(19): Error: function std.regexp.RegExp.find (string string) is not callable using argument types (string) immutable steve Ubuntu:~/scratch$ So then I use a cast: rex = cast(immutable(RegExp)) RegExp("ab+a"); and get down to: triv.d(19): Error: function std.regexp.RegExp.find (string string) is not callable using argument types (string) immutable as somewhat wierd message in itself. So I use a cast in the call to find (): return (cast(RegExp) rex).find(ns); At this point it compiles, and if it's used in the service, the service does not crash. BUT - all these complicated casts will be extremely tedious to administer to code that uses RexExp methods all over the place, so __gshared is a great temptation. The compiler should try to eliminate such temptations in the interests of safety. Also why are these casts necessary. If I am assigning to something that is the same type and was declared as immutable should not at least the first cast be done implicitly. In the second case, I'm using a Regexp object, wherever it may be stored, so why the cast? If find was pure would the cast still be required? Thanks Steve
Nov 18 2010
next sibling parent reply Jason House <jason.james.house gmail.com> writes:
A cast to immutable is required when constructing immutable objects. The type
system can't prove that's safe. It's a design limitation to keep complexity low.

http://www.digitalmars.com/d/2.0/phobos/std_regexp.html
Looking at the Regexp docs, find isn't marked const or pure. If that's an
oversite, file it in bugzilla. If it can't be const, then you should find
another approach.

When this stuff crashes for you, is it accessing a different thread's TLS or is
it simply unable to handle TLS at all?

Steve Teale Wrote:

 On Thu, 18 Nov 2010 11:26:39 +0000, Steve Teale wrote:
 
 I had D code that provided a basis for creation of Windows services,
 which I have just tried to get working with the latest D2.
 
 No dice.
 

static data structures containing function pointers will now cause a Windows service to crash. For happiness, these must be forced into the global data segment. I have demonstrated this within a vestigial service for a couple of cases - plain old struct containing an int and a function pointer, and for Regexp objects by using and not using __gshared on the offending objects. Now that I'm reasonably sure what's happening, I ought to be able to make these things work by making such static members immutable, and initializing them in static this(). However, I then run into a slew of complier errors. Lets start with this simple framework: import std.stdio;

 import std.regexp;

 

 class ThingWithStatic

 {

    static RegExp rex;

    string ns;

 

    static this()

    {

       rex = RegExp("ab+a");

    }

 

    this()

    {

       ns = "abbaabba";

    }

 

    int getFromStatic() { return rex.find(ns); }

 }

 

 void main()

 {

    ThingWithStatic tws = new ThingWithStatic();

    writefln("%d", tws.getFromStatic());

 }
 
 This compiles fine, but if I include this class in a service, and call 
 getFromStatic(), the service will crash. If I change to:
 
 __gshared RegExp rex;
 
 It compiles fine, and the service does not crash.
 
 If I change to:
 
    static immutable RegExp rex;
 
 Then I get errors:
 
 triv.d(11): Error: cannot implicitly convert expression (opCall("ab
 +a",null)) of type std.regexp.RegExp to immutable(RegExp)
 triv.d(19): Error: function std.regexp.RegExp.find (string string) is not 
 callable using argument types (string) immutable
 steve Ubuntu:~/scratch$
 
 So then I use a cast:
 
 rex = cast(immutable(RegExp)) RegExp("ab+a");
 
 and get down to:
 
 triv.d(19): Error: function std.regexp.RegExp.find (string string) is not 
 callable using argument types (string) immutable
 
 as somewhat wierd message in itself. So I use a cast in the call to find
 ():
 
  return (cast(RegExp) rex).find(ns);
  
  At this point it compiles, and if it's used in the service, the service 
 does not crash.
  
  BUT - all these complicated casts will be extremely tedious to 
 administer to code that uses RexExp methods all  over the place, so 
 __gshared is a great temptation. The compiler should try to eliminate 
 such temptations in the interests of safety.
  
 Also why are these casts necessary. If I am assigning to something that 
 is the same type and was declared as immutable should not at least the 
 first cast be done implicitly. In the second case, I'm using a Regexp 
 object, wherever it may be stored, so why the cast? If find was pure 
 would the cast still be required?
 
 Thanks
 Steve

Nov 19 2010
next sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
Steve Teale:

 Languages can't be designed just on theory - some recognition of 
 practicality is also required.

In the case of creation of immutable data structures it looks the opposite to me: DMD is too much pragmatic and there isn't enough theory behind it :-)
 But design limitations like that will force programmers who are on a 
 tight schedule to just say __gshared, and to hell with it.

Regarding the creation of immutable data structures, there is a proposal that is probably able to remove some of the pain: the result of strongly pure functions may become implicitly castable to immutable. Bye, bearophile
Nov 19 2010
parent bearophile <bearophileHUGS lycos.com> writes:
Steve Teale:

 I admire your dedication to language theory and purity, but there are 
 many who'd translate that to impracticality and obscurity.

I like that idea about immutables & strong pure functions, but it was not an idea of mine :-) That enhancement request in Bugzilla is not written by me.
 I came to D in the first place because I found it refreshingly clear and 
 easy to use after C++ and Java. But now it's getting painful.

But strongly pure functions are already present in D. So it's morally good to push its usages and implications as far as possible, otherwise we are wasting possibilities. That implicit cast idea is another implication of purity. It's a clean thing, it's not a hack or workaround or a dangerous thing or something like that, it comes from a clean & logical reasoning about consequences of purity. So it can't be compared to many other dirty things present in D2. It actually helps to clean the programs a little, because removes the need for one important use case of casts. So I can't agree. There is a large complexity difference between extending the logical implications of an already present feature, and adding a true special case.
 What's your estimate of how long it will be before D is a stable language?

I have no idea, I am not that expert, and probably no one knows. D is developed by a small group of devs, and despite in life you seem to "need" to be fast everything you do (for example because otherwise others do it before you and you miss the train), D is not a commercial project, so it has less timing pressures. Often commercial software projects produce a worse result, despite having more resources, because of the strict development timing requirements. If you look at the PyPy Python project it's may years out of schedule, it was started by lot of money coming from European Union, but in the end it may deliver something good any way. Open Source projects too need to respect some timing schedules, but in OS you are sometimes able to say "it will be done when it will be done," while in most commercial software projects you can't say that. The "feature" we are describing here probably needs only few lines of code to be implemented in the compiler (but I can't be sure) and it's backward-compatible (because it removes the need of a cast in some cases), so it's a kind of additive change. Bye, bearophile
Nov 19 2010
prev sibling parent Jason House <jason.james.house gmail.com> writes:
Steve Teale Wrote:

 On Fri, 19 Nov 2010 08:25:17 -0500, Jason House wrote:
 
 A cast to immutable is required when constructing immutable objects. The
 type system can't prove that's safe. It's a design limitation to keep
 complexity low.
 

But design limitations like that will force programmers who are on a tight schedule to just say __gshared, and to hell with it. Languages can't be designed just on theory - some recognition of practicality is also required.

I didn't say I liked the need for the cast or even that I think D has the proper design! IIRC, the cast with object creation was in lieu of adding "unique" tepe qualifier and the extra complexity to get that right. In your example, I believe the object was created once, so a cast really isn't that bad. The bad part to me was the need to cast away immutability with every use. That's why I encouraged a bugzilla entry. I did't check this case, but Phobos had a bad track record with adopting type qualifiers after they were added to D2.
Nov 19 2010
prev sibling next sibling parent Steve Teale <steve.teale britseyeview.com> writes:
On Fri, 19 Nov 2010 08:25:17 -0500, Jason House wrote:

 A cast to immutable is required when constructing immutable objects. The
 type system can't prove that's safe. It's a design limitation to keep
 complexity low.
 

But design limitations like that will force programmers who are on a tight schedule to just say __gshared, and to hell with it. Languages can't be designed just on theory - some recognition of practicality is also required. As to the thread - I don't think that is under my control. I believe the Windows service dispatcher does its own thing. Thanks for taking the trouble Steve
Nov 19 2010
prev sibling next sibling parent Steve Teale <steve.teale britseyeview.com> writes:
On Fri, 19 Nov 2010 11:23:44 -0500, bearophile wrote:

 Regarding the creation of immutable data structures, there is a proposal
 that is probably able to remove some of the pain: the result of strongly
 pure functions may become implicitly castable to immutable.
 
 Bye,
 bearophile

BP, I admire your dedication to language theory and purity, but there are many who'd translate that to impracticality and obscurity. I came to D in the first place because I found it refreshingly clear and easy to use after C++ and Java. But now it's getting painful. I've bailed out several times, just keep coming back to see how it is doing. I have a bunch of code that worked with D1 and D2 at one time. I've given up on D1, since it is now obviously legacy, but even without the complexity of supporting both, It's been real hard this visit to get things working again. What's your estimate of how long it will be before D is a stable language? Thanks Steve
Nov 19 2010
prev sibling next sibling parent Fawzi Mohamed <fawzi gmx.ch> writes:
On 19-nov-10, at 17:42, Steve Teale wrote:

 On Fri, 19 Nov 2010 11:23:44 -0500, bearophile wrote:

 Regarding the creation of immutable data structures, there is a  
 proposal
 that is probably able to remove some of the pain: the result of  
 strongly
 pure functions may become implicitly castable to immutable.

 Bye,
 bearophile

BP, I admire your dedication to language theory and purity, but there are many who'd translate that to impracticality and obscurity. I came to D in the first place because I found it refreshingly clear and easy to use after C++ and Java. But now it's getting painful. I've bailed out several times, just keep coming back to see how it is doing. I have a bunch of code that worked with D1 and D2 at one time. I've given up on D1, since it is now obviously legacy, but even without the complexity of supporting both, It's been real hard this visit to get things working again. What's your estimate of how long it will be before D is a stable language?

well D1 is pretty stable I think, if you are interested in stability that is a good choice, has worked well for me. This does not mean that I will not consider D2, but D1 is my main workhorse. Fwzi
 Thanks
 Steve

Nov 19 2010
prev sibling parent Steve Teale <steve.teale britseyeview.com> writes:
On Fri, 19 Nov 2010 16:21:41 -0500, Jason House wrote:

 Steve Teale Wrote:
 
 On Fri, 19 Nov 2010 08:25:17 -0500, Jason House wrote:
 
 A cast to immutable is required when constructing immutable objects.
 The type system can't prove that's safe. It's a design limitation to
 keep complexity low.
 

But design limitations like that will force programmers who are on a tight schedule to just say __gshared, and to hell with it. Languages can't be designed just on theory - some recognition of practicality is also required.

I didn't say I liked the need for the cast or even that I think D has the proper design! IIRC, the cast with object creation was in lieu of adding "unique" tepe qualifier and the extra complexity to get that right. In your example, I believe the object was created once, so a cast really isn't that bad. The bad part to me was the need to cast away immutability with every use. That's why I encouraged a bugzilla entry. I did't check this case, but Phobos had a bad track record with adopting type qualifiers after they were added to D2.

I'm a bit calmer now I have got most of my utility modules working again. There were two circumstances where I found the casts particularly irksome. I had at one point an immutable char* that I would be passing to Win32 functions. I was assigning to it the address of a plain static character array, and I had to use a cast to do it. I could not see what harm such an assignment could do. Then I had a slew of cases where I had created immutable RegExp objects, and I had to do (cast(RegExp) rexobj).find(...). I could not see what the cast was protecting here either. I think these are bugs, if you agree I will put them in bugzilla.
Nov 19 2010
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 19 Nov 2010 01:49:30 -0500, Steve Teale  
<steve.teale britseyeview.com> wrote:

 As I said before, I don't know if the thread being created by the
 windows service procedure is properly initializing the D modules of the
 library/runtime.  Try as the first line of ServiceMain to initialize the
 current thread:

 auto mythread = thread_attachThis();

 see http://www.digitalmars.com/d/2.0/phobos/core_thread.html

 -Steve

Steve, I don't think it gets as far as ServiceMain. Anyway, I tried it there, and at the point just before the first WINAPI call, but it still crashes. I can fix it, by making my statically initialized RegExp objects __gshared or immutable, but the latter involves inserting a slew of tedious casts into my XML parser, since RexExp calls are used all over the place. See my separate post for my thoughts on that.

OK, I'm out of ideas, sorry :( It certainly looks like there is an issue with TLS. I wouldn't suggest band-aiding things how you are doing, because some modules in phobos/druntime use TLS also. It's reasonable to assume that if your uses of TLS are failing, those may fail as well. -Steve
Nov 19 2010
prev sibling parent Steve Teale <steve.teale britseyeview.com> writes:
On Fri, 19 Nov 2010 16:58:26 +0300, Stanislav Blinov wrote:

 Have you tried OutputDebugString()? It's WinAPI function that sends
 strings to the 'debugger'. DebugView application from SysInternals suite
 nicely prints those strings, and also times them.

How is this better than the event log. It only takes one line of code, but it's still a pain in the arse. In any case, if it's a service, where does it output the debug string to? Thanks for you suggestion Stanislav, but I'm pretty much worked through it by now - albeit somewhat pissed off ;=) Steve
Nov 19 2010