www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - WTF is going on! Corrupt value that is never assigned

reply FoxyBrown <Foxy Brown.IPT> writes:
The following code is pretty screwed up, even though it doesn't 
look like it. I have a buf, a simple malloc which hold the 
results of a win32 call. I am then trying to copy over the data 
in buf to a D struct.

But when copying the strings, the buf location changes, screwing 
up the copying process. It shouldn't happen, buf never changes 
value anywhere except the first malloc(which is once). Somehow it 
is getting changed, but where?

The specific win32 or style is irrelevant, I am talking either 
about a bug or some subtle D thing because the code makes sense. 
(Fill Buf, iterate through buffer copying over to D values).

I've ran in to this before, D does something fishy and it wastes 
hours upon hours trying to track done some stupid little thing it 
does.

The buf value changes when calling cstr2dstr but I've had it with 
other values to(any function call such as to!string, etc seems to 
trigger it).




struct ServiceData
{
     wstring Name;
     wstring LongName;
     int Type;
     int State;
     int ControlsAccepted;
     int Win32ExitCode;
     int SpecificExitCode;
     int CheckPoint;
     int WaitHint;
     int ProcessId;
     int Flags;
}

auto cstr2dstr(wchar* cstr)
{
	import std.array;
	auto str = appender!wstring;
	auto len = lstrlen(cstr);
	str.reserve(len);
	
	for(int i = 0; i < len; i++)
		str.put(cstr[i]);

	return str.data;
}


auto EnumServices()
{
	import core.stdc.stdlib, std.traits;
	ServiceData[] servicesList;

	auto buf = malloc(50000); // Gets changed later, even though 
never an assignment
	auto buf2 = buf; // does not change
     auto schSCManager = OpenSCManager(null, null, 
SC_MANAGER_ALL_ACCESS);
     if (NULL == schSCManager)
     {
         print("OpenSCManager failed (%d)\n", GetLastError());
         return servicesList;
     }
		
	DWORD dwBytesNeeded, dwCount, lpResumeHandle, resume, totalCount;
	auto servicesType = (SERVICE_DRIVER | SERVICE_FILE_SYSTEM_DRIVER 
| SERVICE_KERNEL_DRIVER | SERVICE_WIN32 | 
SERVICE_WIN32_OWN_PROCESS | SERVICE_WIN32_SHARE_PROCESS);

	int cnt = 0;		
	auto res = 0;
	do
     {
		// Manually copy over data, this is because EnumSErvicesStatus 
adds data at end of array for some odd ball reason making it 
difficult to build the correct array sequentially
		for(int i = 0; i < dwCount; i++)
		{
			ENUM_SERVICE_STATUS_PROCESSW x;
			ServiceData d;
			auto s = cast(ENUM_SERVICE_STATUS_PROCESSW*)(buf + 
i*ENUM_SERVICE_STATUS_PROCESSW.sizeof);
//                    before buf is of correct value
			d.Name = cstr2dstr(s.lpServiceName);
//                    after buf is invalid, yet buf is never 
assigned

		}

		
		res = EnumServicesStatusExW(schSCManager, 
SC_ENUM_TYPE.SC_ENUM_PROCESS_INFO, servicesType, 
SERVICE_STATE_ALL, cast(ubyte*)buf, 50000, &dwBytesNeeded, 
&dwCount, &resume, cast(const(char)*)null);
		if (ERROR_MORE_DATA != GetLastError()) { print("Error 
enumerating services."); break; } 				
	} while (res == 0);

	
	for(int i = 0; i < totalCount; i++)
	{
		auto s = servicesList[i];
		writeln(s.Name, " - ", s.LongName, " - ", s.Type, " = ", 
s.State);
	}


	return servicesList;
}
Jul 13
next sibling parent reply Moritz Maxeiner <moritz ucworks.org> writes:
On Thursday, 13 July 2017 at 18:22:34 UTC, FoxyBrown wrote:
 The following code is pretty screwed up, even though it doesn't 
 look like it. I have a buf, a simple malloc which hold the 
 results of a win32 call. I am then trying to copy over the data 
 in buf to a D struct.

 But when copying the strings, the buf location changes, 
 screwing up the copying process. It shouldn't happen, buf never 
 changes value anywhere except the first malloc(which is once). 
 Somehow it is getting changed, but where?

 [...]

 The buf value changes when calling cstr2dstr but I've had it 
 with other values to(any function call such as to!string, etc 
 seems to trigger it).

 [...]
- Does this happen every time, or only sometimes? - At which loop iteration does it occur? - Which compiler (+version) are you using (with what flags)? - What are the steps to reproduce (i.e. does this e.g. happen with a main that consist of one call to EnumServices) ?
Jul 13
parent reply FoxyBrown <Foxy Brown.IPT> writes:
On Thursday, 13 July 2017 at 20:35:19 UTC, Moritz Maxeiner wrote:
 On Thursday, 13 July 2017 at 18:22:34 UTC, FoxyBrown wrote:
 The following code is pretty screwed up, even though it 
 doesn't look like it. I have a buf, a simple malloc which hold 
 the results of a win32 call. I am then trying to copy over the 
 data in buf to a D struct.

 But when copying the strings, the buf location changes, 
 screwing up the copying process. It shouldn't happen, buf 
 never changes value anywhere except the first malloc(which is 
 once). Somehow it is getting changed, but where?

 [...]

 The buf value changes when calling cstr2dstr but I've had it 
 with other values to(any function call such as to!string, etc 
 seems to trigger it).

 [...]
- Does this happen every time, or only sometimes?
yes, but I've been having this problem and not sure if it was quite as consistent as before or that I just recognized it.
 - At which loop iteration does it occur?
Now it seems to occur after the first iteration, but I've add it happen after a while and in other cases it's worked.. depends on if I use malloc, or a D array, or what.
 - Which compiler (+version) are you using (with what flags)?
Latest DMD official.. whatever default flags exist in debug mode with visual D... why should it matter? buf is changing on the only source of that change could be through the winapi call or the temp pointer used to index it.. which is never assigned to, so it can't be modifying it. The c2d function, when called, clearly has no understanding of buff, yet after it returns, it is causing the problem. This seems like the stack is being corrupted by the function call.
 - What are the steps to reproduce (i.e. does this e.g. happen 
 with a main that consist of one call to EnumServices) ?
Yes, It is basically the first thing I do when I run my program. It is a rather isolated function(Just trying to get a list of current services, which has been a total PITA because D is not interfacing well with C. First, the win32 function does not simply fill in an array but adds additional junk at the end(didn't know that until after a few wasted hours trying to get it to fill in an array properly). Hence now I'm trying to convert the returned data one iteration at a time rather than all at once, but I can't get that to work because the pointer to the buffer I created is changing. I could use a temp and get it to work, but that doesn't explain what the hell is going on. The value of buff seems to be erratic, I've had it point valid stuff(other data) and then have small values in it like 8. I don't know how any stack corruption could be occurring but that is exactly what it looks like. "Return from function call and "static variables"(with respect to the call) are changed.". But that seems really hard to sell given that it's pretty simple and D should have all those basics well covered.
Jul 13
parent reply Moritz Maxeiner <moritz ucworks.org> writes:
On Thursday, 13 July 2017 at 22:53:45 UTC, FoxyBrown wrote:
 On Thursday, 13 July 2017 at 20:35:19 UTC, Moritz Maxeiner 
 wrote:
 On Thursday, 13 July 2017 at 18:22:34 UTC, FoxyBrown wrote:
 The following code is pretty screwed up, even though it 
 doesn't look like it. I have a buf, a simple malloc which 
 hold the results of a win32 call. I am then trying to copy 
 over the data in buf to a D struct.

 But when copying the strings, the buf location changes, 
 screwing up the copying process. It shouldn't happen, buf 
 never changes value anywhere except the first malloc(which is 
 once). Somehow it is getting changed, but where?

 [...]

 The buf value changes when calling cstr2dstr but I've had it 
 with other values to(any function call such as to!string, etc 
 seems to trigger it).

 [...]
- Does this happen every time, or only sometimes?
yes, but I've been having this problem and not sure if it was quite as consistent as before or that I just recognized it.
 - At which loop iteration does it occur?
Now it seems to occur after the first iteration, but I've add it happen after a while and in other cases it's worked.. depends on if I use malloc, or a D array, or what.
 - Which compiler (+version) are you using (with what flags)?
Latest DMD official.. whatever default flags exist in debug mode with visual D... why should it matter? [...]
Because it's part of the usual "Steps to reproduce" you are supposed to provide so others can verify what you're encountering.
 - What are the steps to reproduce (i.e. does this e.g. happen 
 with a main that consist of one call to EnumServices) ?
Yes, It is basically the first thing I do when I run my program. [...]
Okay, I'll setup a Windows VM when I have time and check it out (unless someone solves it beforehand).
 because D  is not interfacing well with C. First, the win32 
 function  does not simply fill in an array but adds additional 
 junk at the end(didn't know that until after a few wasted hours 
 trying to get it to fill in an array properly).
To be fair, that's neither C nor D fault; that's Microsoft providing unintuitive, horrible APIs and doing an amazing job of providing documentation (MSDN) that *appears* to be exhaustive and well written, but misses all these little important details that you actually have to know in order to program correct control logic, driving you to the edge of sanity. Been there, done that.
 I don't know how any stack corruption could be occurring but 
 that is exactly what it looks like. "Return from function call 
 and "static variables"(with respect to the call) are changed.". 
 But that seems really hard to sell given that it's pretty 
 simple and D should have all those basics well covered.
It's always possible for the D compiler to generate wrong code (though I'm not convinced that this is the case here), you should have a look at the generated assembly.
Jul 13
next sibling parent reply FoxyBrown <Foxy Brown.IPT> writes:
On Thursday, 13 July 2017 at 23:30:39 UTC, Moritz Maxeiner wrote:
 On Thursday, 13 July 2017 at 22:53:45 UTC, FoxyBrown wrote:
 On Thursday, 13 July 2017 at 20:35:19 UTC, Moritz Maxeiner 
 wrote:
 On Thursday, 13 July 2017 at 18:22:34 UTC, FoxyBrown wrote:
 The following code is pretty screwed up, even though it 
 doesn't look like it. I have a buf, a simple malloc which 
 hold the results of a win32 call. I am then trying to copy 
 over the data in buf to a D struct.

 But when copying the strings, the buf location changes, 
 screwing up the copying process. It shouldn't happen, buf 
 never changes value anywhere except the first malloc(which 
 is once). Somehow it is getting changed, but where?

 [...]

 The buf value changes when calling cstr2dstr but I've had it 
 with other values to(any function call such as to!string, 
 etc seems to trigger it).

 [...]
- Does this happen every time, or only sometimes?
yes, but I've been having this problem and not sure if it was quite as consistent as before or that I just recognized it.
 - At which loop iteration does it occur?
Now it seems to occur after the first iteration, but I've add it happen after a while and in other cases it's worked.. depends on if I use malloc, or a D array, or what.
 - Which compiler (+version) are you using (with what flags)?
Latest DMD official.. whatever default flags exist in debug mode with visual D... why should it matter? [...]
Because it's part of the usual "Steps to reproduce" you are supposed to provide so others can verify what you're encountering.
 - What are the steps to reproduce (i.e. does this e.g. happen 
 with a main that consist of one call to EnumServices) ?
Yes, It is basically the first thing I do when I run my program. [...]
Okay, I'll setup a Windows VM when I have time and check it out (unless someone solves it beforehand).
 because D  is not interfacing well with C. First, the win32 
 function  does not simply fill in an array but adds additional 
 junk at the end(didn't know that until after a few wasted 
 hours trying to get it to fill in an array properly).
To be fair, that's neither C nor D fault; that's Microsoft providing unintuitive, horrible APIs and doing an amazing job of providing documentation (MSDN) that *appears* to be exhaustive and well written, but misses all these little important details that you actually have to know in order to program correct control logic, driving you to the edge of sanity. Been there, done that.
I have generally not had that problem. Usually works as it does and pretty straight forward. Might have to fish a little for info from others but most things have been worked out by someone somewhere. What I'm trying to do is fairly straightforward but I've wasted nearly 2 days on it. I had no issues starting or stopping services but I can't get the enumerate code to work(the winapi function works fine, it's getting the info out of what it passes in to D that is the problem). There is no explanation why buf is being changed, I never change it after it is initialized, yet, it changes somehow. Either it is being overwritten because of a pointer that is invalid or it is a stack problem. It is most likely the former, but I have tried many different ways and they all lead to similar results.
Jul 13
parent Adam D. Ruppe <destructionator gmail.com> writes:
On Friday, 14 July 2017 at 00:33:12 UTC, FoxyBrown wrote:
 What I'm trying to do is fairly straightforward but I've wasted 
 nearly 2 days on it.
//added this so it would all compile import core.sys.windows.windows; import core.stdc.stdio; import core.stdc.stdlib; import std.stdio; import std.conv; import std.array; pragma(lib, "advapi32"); struct ServiceData { wstring Name; wstring LongName; int Type; int State; int ControlsAccepted; int Win32ExitCode; int SpecificExitCode; int CheckPoint; int WaitHint; int ProcessId; int Flags; } auto cstr2dstr(wchar* cstr) { import std.array; auto str = appender!wstring; auto len = lstrlen(cstr); str.reserve(len); for(int i = 0; i < len; i++) str.put(cstr[i]); return str.data; } auto EnumServices() { import core.stdc.stdlib, std.traits; ServiceData[] servicesList; auto buf = malloc(50000); // Gets changed later, even though never an assignment auto buf2 = buf; // does not change auto schSCManager = OpenSCManager(null, null, SC_MANAGER_ALL_ACCESS); if (NULL == schSCManager) { printf("OpenSCManager failed (%d)\n", GetLastError()); return servicesList; } DWORD dwBytesNeeded, dwCount, lpResumeHandle, resume, totalCount; auto servicesType = (SERVICE_DRIVER | SERVICE_FILE_SYSTEM_DRIVER | SERVICE_KERNEL_DRIVER | SERVICE_WIN32 | SERVICE_WIN32_OWN_PROCESS | SERVICE_WIN32_SHARE_PROCESS); int cnt = 0; auto res = 0; do { // Manually copy over data, this is because EnumSErvicesStatus adds data at end of array for some odd ball reason making it difficult to build the correct array sequentially for(int i = 0; i < dwCount; i++) { ENUM_SERVICE_STATUS_PROCESS x; ServiceData d; auto s = cast(ENUM_SERVICE_STATUS_PROCESS*)(buf + i*ENUM_SERVICE_STATUS_PROCESS.sizeof); // before buf is of correct value d.Name = cstr2dstr(s.lpServiceName); // added these to copy the rest of the fields d.LongName = cstr2dstr(s.lpDisplayName); d.tupleof[2 .. $] = s.ServiceStatusProcess.tupleof; // after buf is invalid, yet buf is never assigned // added this so it actually appends to array servicesList ~= d; } res = EnumServicesStatusEx(schSCManager, SC_ENUM_TYPE.SC_ENUM_PROCESS_INFO, servicesType, SERVICE_STATE_ALL, cast(ubyte*)buf, 50000, &dwBytesNeeded, &dwCount, &resume, null); if (ERROR_MORE_DATA != GetLastError()) { printf("Error enumerating services."); break; } } while (res == 0); foreach(s; servicesList) { writeln(s.Name, " - ", s.LongName, " - ", s.Type, " = ", s.State); } return servicesList; } void main() { EnumServices(); } ---- I only modified a few lines in there but it works for me on win64.
Jul 13
prev sibling parent Moritz Maxeiner <moritz ucworks.org> writes:
On Thursday, 13 July 2017 at 23:30:39 UTC, Moritz Maxeiner wrote:
 Okay, I'll setup a Windows VM when I have time and check it out 
 (unless someone solves it beforehand).
I have been unable to reproduce your reported behaviour with dmd 2.074.1 (same as Adam).
Jul 14
prev sibling parent ag0aep6g <anonymous example.com> writes:
On 07/13/2017 08:22 PM, FoxyBrown wrote:
          res = EnumServicesStatusExW(schSCManager, 
 SC_ENUM_TYPE.SC_ENUM_PROCESS_INFO, servicesType, SERVICE_STATE_ALL, 
 cast(ubyte*)buf, 50000, &dwBytesNeeded, &dwCount, &resume, 
 cast(const(char)*)null);
The cast to `char*` here looks odd. The 'W' suffix in the function name indicates that it's the UTF-16 variant. It should be taking a `wchar*`, not of a `char*`. This might hint at a wrong declaration which could lead to memory corruption. (The cast shouldn't be necessary anyway. `null` converts to all pointer types.)
Jul 13