www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Passing C++ class to DLL for callbacks from D (Steam)

reply cc <cc nevernet.com> writes:
Hello, I'm attempting to interface with the Steam API DLL in D 
and running into some trouble working with callbacks.  I'm aware 
there's already a project here http://derelict-steamworks.dub.pm/ 
but it seems to have not yet addressed the same issue.  Steam 
provides ways to poll for whether an asynchronous request has 
completed yet and retrieve the results, but what I am trying to 
implement is receiving actual callbacks from the API.

Just for reference, I've gotten simple callbacks working in other 
APIs (passing a D function pointer to a C function and having it 
be called back), and the other basic Steam functions that don't 
deal with callbacks all work fine (defined in D as extern(C)).  
Here's where I'm running into trouble:

Definitions in C++ headers provided by Steam:

typedef uint64 SteamAPICall_t;
S_API void S_CALLTYPE SteamAPI_RegisterCallResult( class 
CCallbackBase *pCallback, SteamAPICall_t hAPICall );
S_API void S_CALLTYPE SteamAPI_UnregisterCallResult( class 
CCallbackBase *pCallback, SteamAPICall_t hAPICall );

class CCallbackBase
{
public:
	CCallbackBase() { m_nCallbackFlags = 0; m_iCallback = 0; }
	// don't add a virtual destructor because we export this binary 
interface across dll's
	virtual void Run( void *pvParam ) = 0;
	virtual void Run( void *pvParam, bool bIOFailure, SteamAPICall_t 
hSteamAPICall ) = 0;
	int GetICallback() { return m_iCallback; }
	virtual int GetCallbackSizeBytes() = 0;

protected:
	enum { k_ECallbackFlagsRegistered = 0x01, 
k_ECallbackFlagsGameServer = 0x02 };
	uint8 m_nCallbackFlags;
	int m_iCallback;
	friend class CCallbackMgr;

private:
	CCallbackBase( const CCallbackBase& );
	CCallbackBase& operator=( const CCallbackBase& );
};


It's actually more complex than that as it has templated classes 
that inherit from that base so that there is a different template 
for each expected struct that gets returned from various async 
requests.  I'm trying to get the simplest possible scenario 
working first and hoping I'm not missing anything, but I don't 
actually understand how I'm supposed to work with the C++ objects 
enough really to be sure.  Here's how I've defined it in D, as 
per https://dlang.org/spec/cpp_interface.html#classes :

alias ulong SteamAPICall_t;
extern(C) {
	void SteamAPI_RegisterCallResult(CCallbackBase pCallback, 
SteamAPICall_t hAPICall);
	void SteamAPI_UnregisterCallResult(CCallbackBase pCallback, 
SteamAPICall_t hAPICall);
}

extern(C++) {
	interface CCallbackBase {
		//this() { m_nCallbackFlags = 0; m_iCallback = 0; }
		void Run( void *pvParam );
		void Run( void *pvParam, bool bIOFailure, SteamAPICall_t 
hSteamAPICall );
		int GetICallback();
		int GetCallbackSizeBytes();

		enum { k_ECallbackFlagsRegistered = 0x01, 
k_ECallbackFlagsGameServer = 0x02 }
		//uint8 m_nCallbackFlags;
		//int m_iCallback;
		//friend class CCallbackMgr;

		//CCallbackBase( const CCallbackBase& );
		//CCallbackBase& operator=( const CCallbackBase& );
	}
}

class CImpl : CCallbackBase {
	extern(C++) {
		this() { m_nCallbackFlags = 0; m_iCallback = 0; }
		void Run( void *pvParam ) { writeln("Run1"); }
		void Run( void *pvParam, bool bIOFailure, SteamAPICall_t 
hSteamAPICall ) { writeln("Run2"); }
		int GetICallback() { return m_iCallback; }
		int GetCallbackSizeBytes() { return 
NumberOfCurrentPlayers_t.sizeof; } // ordinarily use templates to 
determine what type struct ptr to return
	}
	uint8 m_nCallbackFlags;
	int m_iCallback;
}


Which then gets called later on as:

auto cbk = new CImpl();
cbk.m_iCallback = NumberOfCurrentPlayers_t.k_iCallback;
auto hid = 
SteamAPI_ISteamUserStats_GetNumberOfCurrentPlayers(...);
SteamAPI_RegisterCallResult(cbk, hid);

And then every frame SteamAPI_RunCallbacks(); runs, which is 
supposed to fire off the callbacks.  I've also tried using 
abstract class instead of interface.  My expectation of what's 
supposed to happen is that one of those Run() functions gets 
called at some point.  However, it never does (even though I can 
poll to determine the call has actually completed).  I'm hoping 
the problem is I'm simply not passing the right expected type of 
data to the registration function and/or haven't defined the 
interface in D properly (I admit I have very little idea how the 
C++ is working anymore, including what those private methods are 
for), and not an issue with utilizing the API itself, since I 
doubt there's anyone I could ask on the Steam end of things about 
interfacing with D.

My gut feeling is I'm doing something very incorrect/stupid on 
the definition end of things, I understand passing function 
pointers to C++ functions that intend to call back well enough 
but I'm confused about this concept of passing "class SomeClass 
*" to a function and expecting it to call a method on that class, 
I don't know where everything is supposed to "exist".  If anyone 
has any insight to provide it would be greatly appreciated, 
thanks!
Jun 07 2018
next sibling parent reply Mike Parker <aldacron gmail.com> writes:
On Friday, 8 June 2018 at 00:55:35 UTC, cc wrote:


 class CImpl : CCallbackBase {
 	extern(C++) {



 If anyone has any insight to provide it would be greatly 
 appreciated, thanks!


I've not used any of the C++ interfacing features yet, but my 
understanding is the extern(C++) has to apply to the class 
declaration itself, not only the member functions:

extern(C++)
class CImpl : CCallbackBase {

Does that make a difference?
Jun 07 2018
parent reply cc <cc nevernet.com> writes:
On Friday, 8 June 2018 at 02:52:10 UTC, Mike Parker wrote:

 On Friday, 8 June 2018 at 00:55:35 UTC, cc wrote:


 class CImpl : CCallbackBase {
 	extern(C++) {



 If anyone has any insight to provide it would be greatly 
 appreciated, thanks!


 I've not used any of the C++ interfacing features yet, but my 
 understanding is the extern(C++) has to apply to the class 
 declaration itself, not only the member functions:

 extern(C++)
 class CImpl : CCallbackBase {

 Does that make a difference?


Tried wrapping the CImpl class in extern(C++), no luck.. tried 
final keyword on the methods too.  Thanks though.
Jun 07 2018
parent reply evilrat <evilrat666 gmail.com> writes:
On Friday, 8 June 2018 at 06:59:51 UTC, cc wrote:

 On Friday, 8 June 2018 at 02:52:10 UTC, Mike Parker wrote:

 On Friday, 8 June 2018 at 00:55:35 UTC, cc wrote:


 class CImpl : CCallbackBase {
 	extern(C++) {



 If anyone has any insight to provide it would be greatly 
 appreciated, thanks!


 I've not used any of the C++ interfacing features yet, but my 
 understanding is the extern(C++) has to apply to the class 
 declaration itself, not only the member functions:

 extern(C++)
 class CImpl : CCallbackBase {

 Does that make a difference?


 Tried wrapping the CImpl class in extern(C++), no luck.. tried 
 final keyword on the methods too.  Thanks though.


Can you upload reduced example somewhere to play with?
Jun 08 2018
parent reply cc <cc nevernet.com> writes:
On Friday, 8 June 2018 at 07:32:54 UTC, evilrat wrote:

 On Friday, 8 June 2018 at 06:59:51 UTC, cc wrote:

 On Friday, 8 June 2018 at 02:52:10 UTC, Mike Parker wrote:

 On Friday, 8 June 2018 at 00:55:35 UTC, cc wrote:


 class CImpl : CCallbackBase {
 	extern(C++) {



 If anyone has any insight to provide it would be greatly 
 appreciated, thanks!


 I've not used any of the C++ interfacing features yet, but my 
 understanding is the extern(C++) has to apply to the class 
 declaration itself, not only the member functions:

 extern(C++)
 class CImpl : CCallbackBase {

 Does that make a difference?


 Tried wrapping the CImpl class in extern(C++), no luck.. tried 
 final keyword on the methods too.  Thanks though.


 Can you upload reduced example somewhere to play with?


I've put together a simplified test program here (124KB):
https://drive.google.com/uc?id=1VPxCcPShlYpGo8BggxniNlPH96iIRG8_&export=download

The official SDK can be downloaded here (including the 
steam_api.dll if you don't want to use the one in the zip for 
security):
https://partner.steamgames.com/doc/sdk
Although it may require a developer account to download.

Also I should note that Steam differentiates between "callbacks" 
(events that may be fired to the application that registers to 
listen for them at any time or multiple times) and "call results" 
(1:1 asynchronous requests, I believe).  This test so far is 
using call results.
Jun 08 2018
parent reply cc <cc nevernet.com> writes:
On Saturday, 9 June 2018 at 03:07:39 UTC, cc wrote:

 I've put together a simplified test program here (124KB):


Here is a pastebin of the D source file updated with some 
additional comments at the end with the callback class 
definitions from the original header files
https://pastebin.com/M8hDXt6L
Jun 08 2018
next sibling parent cc <cc nevernet.com> writes:
Sample output:

Initializing.
User logged on: true
Starting request.
hid: 4838393704146785693
..!!!!!!!!
Request completed: NumberOfCurrentPlayers_t(1, 5828)
Terminating.


Not present: any indication that the registered callresult was 
executed.
Jun 08 2018
prev sibling parent reply evilrat <evilrat666 gmail.com> writes:
On Saturday, 9 June 2018 at 03:14:13 UTC, cc wrote:

 On Saturday, 9 June 2018 at 03:07:39 UTC, cc wrote:

 I've put together a simplified test program here (124KB):


 Here is a pastebin of the D source file updated with some 
 additional comments at the end with the callback class 
 definitions from the original header files
 https://pastebin.com/M8hDXt6L


The proper bare minimum signature for call result callback is
------------------
extern(C++) abstract class CCallbackBase {
   abstract void Run( void *pvParam ) { writeln("Run()"); }
   abstract void Run( void *pvParam, bool bIOFailure, 
SteamAPICall_t hSteamAPICall );
   final int GetICallback() { return m_iCallback; } // this is 
actually optional
   abstract int GetCallbackSizeBytes();
  protected:
   uint8 m_nCallbackFlags; // probably can be left for some 
specific use cases
   int m_iCallback;
}
------------------
and the rest templates and stuff is purely for C++ convenience. 
Run(void*, bool, SteamAPICall_t) is what actually called for call 
result, then CallResult<> template overrides it to call 
appropriate pointer member in class with void (T::fun)(Data*, 
bool bIOFailure) signature.  You can mimic it on D side, or just 
ignore and make own.

There is one catch however, for it to be called flags should be 
equal 1, and iCallback must match struct enum, like 
NumberOfCurrentPlayers_t.k_iCallback for 
GetNumberOfCurrentPlayers()


However steam devs decided to shield actual pointer and return 
pointer sized integer when C API is used(or they just screw up?). 
Anyway, the pointers for subsystems returned by context calls on 
C++ API and mirrored C API calls are different, but they also 
have some mechanism for filtering this stuff, that way both 
integer handle and pointer calls the same underlying 
implementation, but C API call again is shielded so setting up 
CallResult and CCallback are ignored.


So my solution was just to make simple wrapper around C++ context 
calls and pass that real pointer to D side.

------------------- C++ code
// build as static library & link with your project
#ifdef _WIN32
// sorry, I just hardcoded it...
#pragma comment (lib, "steam_api64.lib")
#define _CRT_SECURE_NO_WARNINGS 1
#endif
   #include "public/steam/steam_api.h"
#ifdef _WIN32
#define _CRT_SECURE_NO_WARNINGS 1
#endif

extern "C" void* D_GetSteamUserStats()
{
   return SteamUserStats();
}

------------------- D code
// yes, I messed up with the signature to avoid casting, this may 
or may not be dangerous
extern(C) ISteamUserStats D_GetSteamUserStats();


This of course involves making C++ wrapper, but I don't really 
see other ways because they inlined everything ...
Jun 09 2018
parent reply cc <cc nevernet.com> writes:
On Saturday, 9 June 2018 at 14:11:13 UTC, evilrat wrote:

 However steam devs decided to shield actual pointer and return 
 pointer sized integer when C API is used(or they just screw 
 up?). Anyway, the pointers for subsystems returned by context 
 calls on C++ API and mirrored C API calls are different, but 
 they also have some mechanism for filtering this stuff, that 
 way both integer handle and pointer calls the same underlying 
 implementation, but C API call again is shielded so setting up 
 CallResult and CCallback are ignored.


 So my solution was just to make simple wrapper around C++ 
 context calls and pass that real pointer to D side.


I see, thank you for checking it out.  Is it only the functions 
that return the interface pointers e.g. ISteamUserStats that need 
to be wrapped to use the class-based versions instead of the 
steam_api_flat versions?  Or does the entire callback system need 
to be handled through the wrapper?
Jun 09 2018
parent reply evilrat <evilrat666 gmail.com> writes:
On Sunday, 10 June 2018 at 01:35:40 UTC, cc wrote:

 On Saturday, 9 June 2018 at 14:11:13 UTC, evilrat wrote:

 However steam devs decided to shield actual pointer and return 
 pointer sized integer when C API is used(or they just screw 
 up?). Anyway, the pointers for subsystems returned by context 
 calls on C++ API and mirrored C API calls are different, but 
 they also have some mechanism for filtering this stuff, that 
 way both integer handle and pointer calls the same underlying 
 implementation, but C API call again is shielded so setting up 
 CallResult and CCallback are ignored.


 So my solution was just to make simple wrapper around C++ 
 context calls and pass that real pointer to D side.


 I see, thank you for checking it out.  Is it only the functions 
 that return the interface pointers e.g. ISteamUserStats that 
 need to be wrapped to use the class-based versions instead of 
 the steam_api_flat versions?  Or does the entire callback 
 system need to be handled through the wrapper?


Only subsystems getters like SteamUser() or SteamInventory() 
requires wrapping.

I really can't understand why they ever choose to silently ignore 
registering callbacks received with C API systems handles...
Jun 09 2018
parent reply cc <cc nevernet.com> writes:
On Sunday, 10 June 2018 at 02:57:34 UTC, evilrat wrote:

 Only subsystems getters like SteamUser() or SteamInventory() 
 requires wrapping.

 I really can't understand why they ever choose to silently 
 ignore registering callbacks received with C API systems 
 handles...


Thanks to the information you supplied I was able to get it 
working without a wrapper, like so:

extern(C++) abstract class ISteamClient {
	//public abstract IntPtr GetIntPtr();
	public abstract uint CreateSteamPipe();
	public abstract bool BReleaseSteamPipe(uint hSteamPipe);
	public abstract uint ConnectToGlobalUser(uint hSteamPipe);
	public abstract uint CreateLocalUser(ref uint phSteamPipe,uint 
eAccountType);
	public abstract void ReleaseUser(uint hSteamPipe,uint hUser);
	public abstract ISteamUser GetISteamUser(uint hSteamUser,uint 
hSteamPipe,const(char)* pchVersion);
	...
	public abstract ISteamUserStats GetISteamUserStats(uint 
hSteamUser,uint hSteamPipe,const(char)* pchVersion);
	...
}

HSteamUser hSteamUser = SteamAPI_GetHSteamUser();
HSteamPipe hSteamPipe = SteamAPI_GetHSteamPipe();
auto steamClient = cast(ISteamClient) 
SteamInternal_CreateInterface(STEAMCLIENT_INTERFACE_VERSION);
auto userStats = steamClient.GetISteamUserStats(hSteamUser, 
hSteamPipe, STEAMUSERSTATS_INTERFACE_VERSION);
auto hid = 
SteamAPI_ISteamUserStats_GetNumberOfCurrentPlayers(userStats);
auto cbk = new CallResult!NumberOfCurrentPlayers_t();
SteamAPI_RegisterCallResult(cbk, hid);


And it successfully fires the 3-arg Run method of the callback 
object.  However for some reason the function table of the 
ISteamClient seems to be off by one.. it kept calling the wrong 
methods until I commented one out, in this case GetIntPtr() as 
seen above, then everything seemed to line up.  Not sure what the 
proper way to ensure it matches the C++ layout here, but at least 
it seems to be mostly working for now.  Thanks again!
Jun 10 2018
parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 10/06/2018 10:29 PM, cc wrote:

 And it successfully fires the 3-arg Run method of the callback object.  
 However for some reason the function table of the ISteamClient seems to 
 be off by one.. it kept calling the wrong methods until I commented one 
 out, in this case GetIntPtr() as seen above, then everything seemed to 
 line up.  Not sure what the proper way to ensure it matches the C++ 
 layout here, but at least it seems to be mostly working for now.  Thanks 
 again!


Ugh what GetIntPtr? Unless of course this header file is wrong[0].

Make the members match exactly, order and everything and it should "just 
work".

[0] 
https://github.com/ValveSoftware/source-sdk-2013/blob/master/mp/src/public/steam/isteamclient.h#L113
Jun 10 2018
parent reply cc <cc nevernet.com> writes:
On Sunday, 10 June 2018 at 10:47:58 UTC, rikki cattermole wrote:

 On 10/06/2018 10:29 PM, cc wrote:

 And it successfully fires the 3-arg Run method of the callback 
 object.  However for some reason the function table of the 
 ISteamClient seems to be off by one.. it kept calling the 
 wrong methods until I commented one out, in this case 
 GetIntPtr() as seen above, then everything seemed to line up.  
 Not sure what the proper way to ensure it matches the C++ 
 layout here, but at least it seems to be mostly working for 
 now.  Thanks again!


 Ugh what GetIntPtr? Unless of course this header file is 
 wrong[0].

 Make the members match exactly, order and everything and it 
 should "just work".

 [0] 
 https://github.com/ValveSoftware/source-sdk-2013/blob/master/mp/src/public/steam/isteamclient.h#L113


Woops, that GetIntPtr came from the .cs header in the same folder 
as the C++ headers distributed with the SDK, that'll teach me to 
ctrl+f "class ISteamClient" in all open files and copy/paste 
before reading.

Anyway I played around with it some more and found the one single 
line that was causing the problem.  It needs this:

HSteamPipe SteamAPI_GetHSteamPipe();

rather than one of these:

HSteamPipe SteamAPI_ISteamClient_CreateSteamPipe(ISteamClient 
instancePtr);
virtual HSteamPipe CreateSteamPipe() = 0;

It looks like the pipe is already created upon initializing and 
calling CreateSteamPipe creates a secondary one that doesn't 
receive callbacks. SteamAPI_GetHSteamPipe is called inline to 
retrieve the current pipe on initialization of the the classed 
version of the API, but the documentation doesn't mention it 
exists at all, leading to confusion when using the flat version.  
Well, mystery solved.
Jun 10 2018
parent rikki cattermole <rikki cattermole.co.nz> writes:
On 10/06/2018 11:28 PM, cc wrote:

 Woops, that GetIntPtr came from the .cs header in the same folder as the 
 C++ headers distributed with the SDK, that'll teach me to ctrl+f "class 
 ISteamClient" in all open files and copy/paste before reading.


Stay with the c/c++ headers for c/c++ code. We don't generally don't go 
testing the long way ;)
Jun 10 2018
prev sibling parent evilrat <evilrat666 gmail.com> writes:
On Friday, 8 June 2018 at 00:55:35 UTC, cc wrote:

 I've defined it in D, as per 
 https://dlang.org/spec/cpp_interface.html#classes :


change this to class, or even abstract class as shown in example


 extern(C++) {
 	interface CCallbackBase {
 		//this() { m_nCallbackFlags = 0; m_iCallback = 0; }
 		void Run( void *pvParam );
 		void Run( void *pvParam, bool bIOFailure, SteamAPICall_t 
 hSteamAPICall );
 		int GetICallback();
 		int GetCallbackSizeBytes();

 		enum { k_ECallbackFlagsRegistered = 0x01, 
 k_ECallbackFlagsGameServer = 0x02 }
 		//uint8 m_nCallbackFlags;
 		//int m_iCallback;
 		//friend class CCallbackMgr;

 		//CCallbackBase( const CCallbackBase& );
 		//CCallbackBase& operator=( const CCallbackBase& );
 	}
 }


add extern(C++) to class as well to


 class CImpl : CCallbackBase {
 	extern(C++) {
 		this() { m_nCallbackFlags = 0; m_iCallback = 0; }
 		void Run( void *pvParam ) { writeln("Run1"); }
 		void Run( void *pvParam, bool bIOFailure, SteamAPICall_t 
 hSteamAPICall ) { writeln("Run2"); }
 		int GetICallback() { return m_iCallback; }
 		int GetCallbackSizeBytes() { return 
 NumberOfCurrentPlayers_t.sizeof; } // ordinarily use templates 
 to determine what type struct ptr to return
 	}
 	uint8 m_nCallbackFlags;
 	int m_iCallback;
 }


you also may or may not need to mark non-virtual C++ methods as 
final.

Of course I haven't used D for quite some time so I can be 
mistaken.

But the lesson I learned the hard way is that in D for 
extern(C++) you don't use interface(it simply has no mapping to 
C++ types?), and for defining COM-interfaces use interface or it 
will bite you.
I hope it helps.
Jun 07 2018