www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - AWS API Dlang, hmac sha256 function.

reply holo <holosian gmail.com> writes:
Hello

I'm trying to contact AWS API with D according to documentation:

http://docs.aws.amazon.com/general/latest/gr/sigv4_signing.html

there is written example in python which i want to rewrite to D:

http://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html

I want to ask about such 2 things:

Is it equivalent in D:

hexDigest!SHA256("sometexttohash");

for such python method:

hashlib.sha256('sometexttohash').hexdigest()

?

Maybe is there some prettier solution in D eg:

toHexString(sha256Of("sometexttohash"));

or

SHA256 sha;
sha.start();
string data = "sometexttohash";
sha.put(data);
toHexString(sha.finish())

If those are equivalents to python method which should i choose?

Last but not least, how to write such function in D:

def sign(key, msg):
     return hmac.new(key, msg.encode("utf-8"), 
hashlib.sha256).digest()

?

I can't find in standard libraryt hmac function, is it existing?

//holo
Oct 02 2015
next sibling parent reply Rikki Cattermole <alphaglosined gmail.com> writes:
On 03/10/15 4:11 PM, holo wrote:
 Hello

 I'm trying to contact AWS API with D according to documentation:

 http://docs.aws.amazon.com/general/latest/gr/sigv4_signing.html

 there is written example in python which i want to rewrite to D:

 http://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html


 I want to ask about such 2 things:

 Is it equivalent in D:

 hexDigest!SHA256("sometexttohash");

 for such python method:

 hashlib.sha256('sometexttohash').hexdigest()

 ?

 Maybe is there some prettier solution in D eg:

 toHexString(sha256Of("sometexttohash"));

 or

 SHA256 sha;
 sha.start();
 string data = "sometexttohash";
 sha.put(data);
 toHexString(sha.finish())

 If those are equivalents to python method which should i choose?

 Last but not least, how to write such function in D:

 def sign(key, msg):
      return hmac.new(key, msg.encode("utf-8"), hashlib.sha256).digest()

 ?

 I can't find in standard libraryt hmac function, is it existing?

 //holo
You could implement it yourself, (it looks pretty easy). Or go the route of Botan: https://github.com/etcimon/botan/blob/3bdc835d5b9bad7523deaa86fe98b1fbb2f09d6b/source/botan/mac/hmac.d
Oct 02 2015
parent reply holo <holosian gmail.com> writes:
On Saturday, 3 October 2015 at 03:15:21 UTC, Rikki Cattermole 
wrote:
 You could implement it yourself, (it looks pretty easy).
 Or go the route of Botan: 
 https://github.com/etcimon/botan/blob/3bdc835d5b9bad7523deaa86fe98b1fbb2f09d6b/source/botan/mac/hmac.d
Thank you for answer. I used dub to fetch botan, after that i created project: dub init projectname botan and tried to add botan module: $ cat app.d #!/usr/bin/rdmd import std.stdio; import botan.mac.hmac; void main() { writeln("Edit source/app.d to start your project."); } but when im trying to run my simple app i get: $ ./app.d ./app.d(4): Error: module hmac is in file 'botan/mac/hmac.d' which cannot be read import path[0] = . import path[1] = /usr/include/dlang/dmd Failed: ["dmd", "-v", "-o-", "./app.d", "-I."] What am i doing wrong? How to add library with dub package manager? Im really beginner in D (basically in programming) sorry if im asking obvious questions. According to my other question are that functions equivalent to that python one? If they are not doing exact that same i could not create auth string which is needed to connect to AWS api.
Oct 02 2015
parent reply Rikki Cattermole <alphaglosined gmail.com> writes:
On 03/10/15 4:54 PM, holo wrote:
 On Saturday, 3 October 2015 at 03:15:21 UTC, Rikki Cattermole wrote:
 You could implement it yourself, (it looks pretty easy).
 Or go the route of Botan:
 https://github.com/etcimon/botan/blob/3bdc835d5b9bad7523deaa86fe98b1fbb2f09d6b/source/botan/mac/hmac.d
Thank you for answer. I used dub to fetch botan, after that i created project: dub init projectname botan and tried to add botan module: $ cat app.d #!/usr/bin/rdmd import std.stdio; import botan.mac.hmac; void main() { writeln("Edit source/app.d to start your project."); } but when im trying to run my simple app i get: $ ./app.d ./app.d(4): Error: module hmac is in file 'botan/mac/hmac.d' which cannot be read import path[0] = . import path[1] = /usr/include/dlang/dmd Failed: ["dmd", "-v", "-o-", "./app.d", "-I."] What am i doing wrong? How to add library with dub package manager? Im really beginner in D (basically in programming) sorry if im asking obvious questions.
By the looks of that error message with paths, botan isn't actually added as a dependency. Can you please paste the dub file?
 According to my other question are that functions equivalent to that
 python one? If they are not doing exact that same i could not create
 auth string which is needed to connect to AWS api.
I can't say for certain if it is not exact. The reason I did not answer it is because I am not familiar with environment or any of what you are doing. I can unfortunately only point you in the right direction.
Oct 02 2015
parent reply Rikki Cattermole <alphaglosined gmail.com> writes:
On 03/10/15 6:01 PM, Rikki Cattermole wrote:
 On 03/10/15 4:54 PM, holo wrote:
 On Saturday, 3 October 2015 at 03:15:21 UTC, Rikki Cattermole wrote:
 You could implement it yourself, (it looks pretty easy).
 Or go the route of Botan:
 https://github.com/etcimon/botan/blob/3bdc835d5b9bad7523deaa86fe98b1fbb2f09d6b/source/botan/mac/hmac.d
Thank you for answer. I used dub to fetch botan, after that i created project: dub init projectname botan and tried to add botan module: $ cat app.d #!/usr/bin/rdmd import std.stdio; import botan.mac.hmac; void main() { writeln("Edit source/app.d to start your project."); } but when im trying to run my simple app i get: $ ./app.d ./app.d(4): Error: module hmac is in file 'botan/mac/hmac.d' which cannot be read import path[0] = . import path[1] = /usr/include/dlang/dmd Failed: ["dmd", "-v", "-o-", "./app.d", "-I."] What am i doing wrong? How to add library with dub package manager? Im really beginner in D (basically in programming) sorry if im asking obvious questions.
By the looks of that error message with paths, botan isn't actually added as a dependency. Can you please paste the dub file?
Ohhhh wait nevermind. Use dub to compile/run your program. You are using the shebang line! $ dub run
 According to my other question are that functions equivalent to that
 python one? If they are not doing exact that same i could not create
 auth string which is needed to connect to AWS api.
I can't say for certain if it is not exact. The reason I did not answer it is because I am not familiar with environment or any of what you are doing. I can unfortunately only point you in the right direction.
Oct 02 2015
parent reply holo <holosian gmail.com> writes:
On Saturday, 3 October 2015 at 05:02:58 UTC, Rikki Cattermole 
wrote:
 On 03/10/15 6:01 PM, Rikki Cattermole wrote:
 On 03/10/15 4:54 PM, holo wrote:
 On Saturday, 3 October 2015 at 03:15:21 UTC, Rikki Cattermole 
 wrote:
 You could implement it yourself, (it looks pretty easy).
 Or go the route of Botan:
 https://github.com/etcimon/botan/blob/3bdc835d5b9bad7523deaa86fe98b1fbb2f09d6b/source/botan/mac/hmac.d
Thank you for answer. I used dub to fetch botan, after that i created project: dub init projectname botan and tried to add botan module: $ cat app.d #!/usr/bin/rdmd import std.stdio; import botan.mac.hmac; void main() { writeln("Edit source/app.d to start your project."); } but when im trying to run my simple app i get: $ ./app.d ./app.d(4): Error: module hmac is in file 'botan/mac/hmac.d' which cannot be read import path[0] = . import path[1] = /usr/include/dlang/dmd Failed: ["dmd", "-v", "-o-", "./app.d", "-I."] What am i doing wrong? How to add library with dub package manager? Im really beginner in D (basically in programming) sorry if im asking obvious questions.
By the looks of that error message with paths, botan isn't actually added as a dependency. Can you please paste the dub file?
Ohhhh wait nevermind. Use dub to compile/run your program. You are using the shebang line! $ dub run
 According to my other question are that functions equivalent 
 to that
 python one? If they are not doing exact that same i could not 
 create
 auth string which is needed to connect to AWS api.
I can't say for certain if it is not exact. The reason I did not answer it is because I am not familiar with environment or any of what you are doing. I can unfortunately only point you in the right direction.
Thank you for explaining how to use dub. I find out in master branch of phobos on git hub is included hmac.d module which will be added in next release of phobos so i think that will be best solution for me to stick with standard library for beginning. https://github.com/D-Programming-Language/phobos/tree/master/std I downloaded hmac.d and new version of digest.d file and place them in my phobos library. I think it exactly contains what i need but when im trying to test it i have such error: $ ./app.d ./app.d(9): Error: template std.digest.hmac.hmac cannot deduce function from argument types !(SHA!(512, 256))(immutable(ubyte)[]), candidates are: /usr/include/dlang/dmd/std/digest/hmac.d(202): std.digest.hmac.hmac(H) if (isDigest!H && hasBlockSize!H) /usr/include/dlang/dmd/std/digest/hmac.d(208): std.digest.hmac.hmac(H, ulong blockSize)(scope const(ubyte)[] secret) if (isDigest!H) Failed: ["dmd", "-v", "-o-", "./app.d", "-I."] My test code (based on unint test from module) looks like that: #!/usr/bin/rdmd import std.stdio; import std.digest.sha, std.digest.hmac; import std.string : representation; void main() { auto hmac = hmac!SHA256("secretkey".representation); auto digest = hmac.put("texttohash".representation).finish; writeln(digest); } Am i using it correctly?
Oct 03 2015
parent reply Rikki Cattermole <alphaglosined gmail.com> writes:
On 04/10/15 1:09 AM, holo wrote:
 On Saturday, 3 October 2015 at 05:02:58 UTC, Rikki Cattermole wrote:
 On 03/10/15 6:01 PM, Rikki Cattermole wrote:
 On 03/10/15 4:54 PM, holo wrote:
 On Saturday, 3 October 2015 at 03:15:21 UTC, Rikki Cattermole wrote:
 You could implement it yourself, (it looks pretty easy).
 Or go the route of Botan:
 https://github.com/etcimon/botan/blob/3bdc835d5b9bad7523deaa86fe98b1fbb2f09d6b/source/botan/mac/hmac.d
Thank you for answer. I used dub to fetch botan, after that i created project: dub init projectname botan and tried to add botan module: $ cat app.d #!/usr/bin/rdmd import std.stdio; import botan.mac.hmac; void main() { writeln("Edit source/app.d to start your project."); } but when im trying to run my simple app i get: $ ./app.d ./app.d(4): Error: module hmac is in file 'botan/mac/hmac.d' which cannot be read import path[0] = . import path[1] = /usr/include/dlang/dmd Failed: ["dmd", "-v", "-o-", "./app.d", "-I."] What am i doing wrong? How to add library with dub package manager? Im really beginner in D (basically in programming) sorry if im asking obvious questions.
By the looks of that error message with paths, botan isn't actually added as a dependency. Can you please paste the dub file?
Ohhhh wait nevermind. Use dub to compile/run your program. You are using the shebang line! $ dub run
 According to my other question are that functions equivalent to that
 python one? If they are not doing exact that same i could not create
 auth string which is needed to connect to AWS api.
I can't say for certain if it is not exact. The reason I did not answer it is because I am not familiar with environment or any of what you are doing. I can unfortunately only point you in the right direction.
Thank you for explaining how to use dub. I find out in master branch of phobos on git hub is included hmac.d module which will be added in next release of phobos so i think that will be best solution for me to stick with standard library for beginning. https://github.com/D-Programming-Language/phobos/tree/master/std I downloaded hmac.d and new version of digest.d file and place them in my phobos library. I think it exactly contains what i need but when im trying to test it i have such error: $ ./app.d ./app.d(9): Error: template std.digest.hmac.hmac cannot deduce function from argument types !(SHA!(512, 256))(immutable(ubyte)[]), candidates are: /usr/include/dlang/dmd/std/digest/hmac.d(202): std.digest.hmac.hmac(H) if (isDigest!H && hasBlockSize!H) /usr/include/dlang/dmd/std/digest/hmac.d(208): std.digest.hmac.hmac(H, ulong blockSize)(scope const(ubyte)[] secret) if (isDigest!H) Failed: ["dmd", "-v", "-o-", "./app.d", "-I."] My test code (based on unint test from module) looks like that: #!/usr/bin/rdmd import std.stdio; import std.digest.sha, std.digest.hmac; import std.string : representation; void main() { auto hmac = hmac!SHA256("secretkey".representation); auto digest = hmac.put("texttohash".representation).finish; writeln(digest); } Am i using it correctly?
By the looks of things the problem is with SHA256, I'm guessing it doesn't have its block size defined. https://github.com/D-Programming-Language/phobos/blob/master/std/digest/digest.d#L394 You'll also need to update sha.d as well. Otherwise it looks good.
Oct 03 2015
parent reply holo <holosian gmail.com> writes:
On Saturday, 3 October 2015 at 12:22:11 UTC, Rikki Cattermole 
wrote:
 On 04/10/15 1:09 AM, holo wrote:
 On Saturday, 3 October 2015 at 05:02:58 UTC, Rikki Cattermole 
 wrote:
 On 03/10/15 6:01 PM, Rikki Cattermole wrote:
 On 03/10/15 4:54 PM, holo wrote:
 [...]
By the looks of that error message with paths, botan isn't actually added as a dependency. Can you please paste the dub file?
Ohhhh wait nevermind. Use dub to compile/run your program. You are using the shebang line! $ dub run
 [...]
I can't say for certain if it is not exact. The reason I did not answer it is because I am not familiar with environment or any of what you are doing. I can unfortunately only point you in the right direction.
Thank you for explaining how to use dub. I find out in master branch of phobos on git hub is included hmac.d module which will be added in next release of phobos so i think that will be best solution for me to stick with standard library for beginning. https://github.com/D-Programming-Language/phobos/tree/master/std I downloaded hmac.d and new version of digest.d file and place them in my phobos library. I think it exactly contains what i need but when im trying to test it i have such error: $ ./app.d ./app.d(9): Error: template std.digest.hmac.hmac cannot deduce function from argument types !(SHA!(512, 256))(immutable(ubyte)[]), candidates are: /usr/include/dlang/dmd/std/digest/hmac.d(202): std.digest.hmac.hmac(H) if (isDigest!H && hasBlockSize!H) /usr/include/dlang/dmd/std/digest/hmac.d(208): std.digest.hmac.hmac(H, ulong blockSize)(scope const(ubyte)[] secret) if (isDigest!H) Failed: ["dmd", "-v", "-o-", "./app.d", "-I."] My test code (based on unint test from module) looks like that: #!/usr/bin/rdmd import std.stdio; import std.digest.sha, std.digest.hmac; import std.string : representation; void main() { auto hmac = hmac!SHA256("secretkey".representation); auto digest = hmac.put("texttohash".representation).finish; writeln(digest); } Am i using it correctly?
By the looks of things the problem is with SHA256, I'm guessing it doesn't have its block size defined. https://github.com/D-Programming-Language/phobos/blob/master/std/digest/digest.d#L394 You'll also need to update sha.d as well. Otherwise it looks good.
I updated sha.d and now im getting following error: $ ./app.d /usr/include/dlang/dmd/std/digest/sha.d(225): Error: pure function 'std.digest.sha.SHA!(512u, 160u).SHA.transform' cannot call impure function 'core.cpuid.ssse3' /usr/include/dlang/dmd/std/digest/digest.d(285): Error: template instance std.range.primitives.isOutputRange!(SHA!(512u, 160u), const(ubyte)[]) error instantiating /usr/include/dlang/dmd/std/digest/digest.d(851): instantiated from here: isDigest!(SHA!(512u, 160u)) /usr/include/dlang/dmd/std/digest/sha.d(1179): while looking for match for WrapperDigest!(SHA!(512u, 160u)) Failed: ["dmd", "-v", "-o-", "./app.d", "-I."]
Oct 03 2015
parent reply Rikki Cattermole <alphaglosined gmail.com> writes:
On 04/10/15 1:49 AM, holo wrote:
 On Saturday, 3 October 2015 at 12:22:11 UTC, Rikki Cattermole wrote:
 On 04/10/15 1:09 AM, holo wrote:
 On Saturday, 3 October 2015 at 05:02:58 UTC, Rikki Cattermole wrote:
 On 03/10/15 6:01 PM, Rikki Cattermole wrote:
 On 03/10/15 4:54 PM, holo wrote:
 [...]
By the looks of that error message with paths, botan isn't actually added as a dependency. Can you please paste the dub file?
Ohhhh wait nevermind. Use dub to compile/run your program. You are using the shebang line! $ dub run
 [...]
I can't say for certain if it is not exact. The reason I did not answer it is because I am not familiar with environment or any of what you are doing. I can unfortunately only point you in the right direction.
Thank you for explaining how to use dub. I find out in master branch of phobos on git hub is included hmac.d module which will be added in next release of phobos so i think that will be best solution for me to stick with standard library for beginning. https://github.com/D-Programming-Language/phobos/tree/master/std I downloaded hmac.d and new version of digest.d file and place them in my phobos library. I think it exactly contains what i need but when im trying to test it i have such error: $ ./app.d ./app.d(9): Error: template std.digest.hmac.hmac cannot deduce function from argument types !(SHA!(512, 256))(immutable(ubyte)[]), candidates are: /usr/include/dlang/dmd/std/digest/hmac.d(202): std.digest.hmac.hmac(H) if (isDigest!H && hasBlockSize!H) /usr/include/dlang/dmd/std/digest/hmac.d(208): std.digest.hmac.hmac(H, ulong blockSize)(scope const(ubyte)[] secret) if (isDigest!H) Failed: ["dmd", "-v", "-o-", "./app.d", "-I."] My test code (based on unint test from module) looks like that: #!/usr/bin/rdmd import std.stdio; import std.digest.sha, std.digest.hmac; import std.string : representation; void main() { auto hmac = hmac!SHA256("secretkey".representation); auto digest = hmac.put("texttohash".representation).finish; writeln(digest); } Am i using it correctly?
By the looks of things the problem is with SHA256, I'm guessing it doesn't have its block size defined. https://github.com/D-Programming-Language/phobos/blob/master/std/digest/digest.d#L394 You'll also need to update sha.d as well. Otherwise it looks good.
I updated sha.d and now im getting following error: $ ./app.d /usr/include/dlang/dmd/std/digest/sha.d(225): Error: pure function 'std.digest.sha.SHA!(512u, 160u).SHA.transform' cannot call impure function 'core.cpuid.ssse3' /usr/include/dlang/dmd/std/digest/digest.d(285): Error: template instance std.range.primitives.isOutputRange!(SHA!(512u, 160u), const(ubyte)[]) error instantiating /usr/include/dlang/dmd/std/digest/digest.d(851): instantiated from here: isDigest!(SHA!(512u, 160u)) /usr/include/dlang/dmd/std/digest/sha.d(1179): while looking for match for WrapperDigest!(SHA!(512u, 160u)) Failed: ["dmd", "-v", "-o-", "./app.d", "-I."]
And one more file (ssse3.d) needs to be updated. You are going to be playing whack a mole I think for a while.
Oct 03 2015
parent reply holo <holosian gmail.com> writes:
On Saturday, 3 October 2015 at 12:50:58 UTC, Rikki Cattermole 
wrote:
 On 04/10/15 1:49 AM, holo wrote:
 On Saturday, 3 October 2015 at 12:22:11 UTC, Rikki Cattermole 
 wrote:
 On 04/10/15 1:09 AM, holo wrote:
 [...]
By the looks of things the problem is with SHA256, I'm guessing it doesn't have its block size defined. https://github.com/D-Programming-Language/phobos/blob/master/std/digest/digest.d#L394 You'll also need to update sha.d as well. Otherwise it looks good.
I updated sha.d and now im getting following error: $ ./app.d /usr/include/dlang/dmd/std/digest/sha.d(225): Error: pure function 'std.digest.sha.SHA!(512u, 160u).SHA.transform' cannot call impure function 'core.cpuid.ssse3' /usr/include/dlang/dmd/std/digest/digest.d(285): Error: template instance std.range.primitives.isOutputRange!(SHA!(512u, 160u), const(ubyte)[]) error instantiating /usr/include/dlang/dmd/std/digest/digest.d(851): instantiated from here: isDigest!(SHA!(512u, 160u)) /usr/include/dlang/dmd/std/digest/sha.d(1179): while looking for match for WrapperDigest!(SHA!(512u, 160u)) Failed: ["dmd", "-v", "-o-", "./app.d", "-I."]
And one more file (ssse3.d) needs to be updated. You are going to be playing whack a mole I think for a while.
I downloaded whole master branch phobos and tried to use it (changed path in dmd.conf), but there are missing much more files (i think that master branch is not ready yet). So i get back to my original library and tried to updated file you mention. But i can't find it (ssse3.d), its not appearing in my stable lib and even in that master-branch from zip file.
Oct 03 2015
parent reply Rikki Cattermole <alphaglosined gmail.com> writes:
On 04/10/15 2:31 AM, holo wrote:
 On Saturday, 3 October 2015 at 12:50:58 UTC, Rikki Cattermole wrote:
 On 04/10/15 1:49 AM, holo wrote:
 On Saturday, 3 October 2015 at 12:22:11 UTC, Rikki Cattermole wrote:
 On 04/10/15 1:09 AM, holo wrote:
 [...]
By the looks of things the problem is with SHA256, I'm guessing it doesn't have its block size defined. https://github.com/D-Programming-Language/phobos/blob/master/std/digest/digest.d#L394 You'll also need to update sha.d as well. Otherwise it looks good.
I updated sha.d and now im getting following error: $ ./app.d /usr/include/dlang/dmd/std/digest/sha.d(225): Error: pure function 'std.digest.sha.SHA!(512u, 160u).SHA.transform' cannot call impure function 'core.cpuid.ssse3' /usr/include/dlang/dmd/std/digest/digest.d(285): Error: template instance std.range.primitives.isOutputRange!(SHA!(512u, 160u), const(ubyte)[]) error instantiating /usr/include/dlang/dmd/std/digest/digest.d(851): instantiated from here: isDigest!(SHA!(512u, 160u)) /usr/include/dlang/dmd/std/digest/sha.d(1179): while looking for match for WrapperDigest!(SHA!(512u, 160u)) Failed: ["dmd", "-v", "-o-", "./app.d", "-I."]
And one more file (ssse3.d) needs to be updated. You are going to be playing whack a mole I think for a while.
I downloaded whole master branch phobos and tried to use it (changed path in dmd.conf), but there are missing much more files (i think that master branch is not ready yet). So i get back to my original library and tried to updated file you mention. But i can't find it (ssse3.d), its not appearing in my stable lib and even in that master-branch from zip file.
Apologies, I didn't see this till after I got to bed. It's core.cpuid not ssse3.d. It will be in druntime, not Phobos.
Oct 03 2015
parent reply holo <holosian gmail.com> writes:
On Saturday, 3 October 2015 at 23:58:39 UTC, Rikki Cattermole 
wrote:
 On 04/10/15 2:31 AM, holo wrote:
 On Saturday, 3 October 2015 at 12:50:58 UTC, Rikki Cattermole 
 wrote:
 On 04/10/15 1:49 AM, holo wrote:
 On Saturday, 3 October 2015 at 12:22:11 UTC, Rikki 
 Cattermole wrote:
 On 04/10/15 1:09 AM, holo wrote:
 [...]
By the looks of things the problem is with SHA256, I'm guessing it doesn't have its block size defined. https://github.com/D-Programming-Language/phobos/blob/master/std/digest/digest.d#L394 You'll also need to update sha.d as well. Otherwise it looks good.
I updated sha.d and now im getting following error: $ ./app.d /usr/include/dlang/dmd/std/digest/sha.d(225): Error: pure function 'std.digest.sha.SHA!(512u, 160u).SHA.transform' cannot call impure function 'core.cpuid.ssse3' /usr/include/dlang/dmd/std/digest/digest.d(285): Error: template instance std.range.primitives.isOutputRange!(SHA!(512u, 160u), const(ubyte)[]) error instantiating /usr/include/dlang/dmd/std/digest/digest.d(851): instantiated from here: isDigest!(SHA!(512u, 160u)) /usr/include/dlang/dmd/std/digest/sha.d(1179): while looking for match for WrapperDigest!(SHA!(512u, 160u)) Failed: ["dmd", "-v", "-o-", "./app.d", "-I."]
And one more file (ssse3.d) needs to be updated. You are going to be playing whack a mole I think for a while.
I downloaded whole master branch phobos and tried to use it (changed path in dmd.conf), but there are missing much more files (i think that master branch is not ready yet). So i get back to my original library and tried to updated file you mention. But i can't find it (ssse3.d), its not appearing in my stable lib and even in that master-branch from zip file.
Apologies, I didn't see this till after I got to bed. It's core.cpuid not ssse3.d. It will be in druntime, not Phobos.
Sorry i was busy at weekend. Thank you all for answers. I will try to write some solution basing on yawniek example and will back if some troubles appears. Vladimir where can i check or do you know when next version of phobos will be available? Rikki i downloaded both master zip files for runtime and phobos librarys all dependencies looks are satisfied right now but still getting errors: [holo ultraxps workplace]$ dmd hmac.d hmac.o: In function `_D3std6digest4hmac58__T4hmacTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4hmacFNaNbNiNfMAxhZS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC': hmac.d:(.text._D3std6digest4hmac58__T4hmacTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4hmacFNaNbNiNfMAxhZS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SH Vmi512Z4HMAC+0x18): undefined reference to `_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA6__initZ' hmac.o: In function `_D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC6__ctorMFNaNbNcNiNfMAxhZS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC': hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC6__ctorMFNaNbNcNiNfMAxhZS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SH Vmi512Z4HMAC+0x1b): undefined reference to `_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA5startMFNaNbNiNfZv' hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC6__ctorMFNaNbNcNiNfMAxhZS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SH Vmi512Z4HMAC+0x31): undefined reference to `_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA3putMFNaNbNiNeMAxhXv' hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC6__ctorMFNaNbNcNiNfMAxhZS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SH Vmi512Z4HMAC+0x3e): undefined reference to `_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA6finishMFNaNbNiNeZG20h' hmac.o: In function `_D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC5startMFNaNbNcNiNjNfZS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC': hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC5startMFNaNbNcNiNjNfZS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SH Vmi512Z4HMAC+0x8d): undefined reference to `_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA5startMFNaNbNiNfZv' hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC5startMFNaNbNcNiNjNfZS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SH Vmi512Z4HMAC+0xa4): undefined reference to `_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA3putMFNaNbNiNeMAxhXv' hmac.o: In function `_D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC3putMFNaNbNcNiNjNfxAhXS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC': hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC3putMFNaNbNcNiNjNfxAhXS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SH Vmi512Z4HMAC+0x5f): undefined reference to `_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA3putMFNaNbNiNeMAxhXv' hmac.o: In function `_D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC6finishMFNaNbNiNfZG20h': hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC6finishMF aNbNiNfZG20h+0xa1): undefined reference to `_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA6finishMFNaNbNiNeZG20h' hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC6finishMF aNbNiNfZG20h+0xaa): undefined reference to `_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA5startMFNaNbNiNfZv' hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC6finishMF aNbNiNfZG20h+0xc4): undefined reference to `_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA3putMFNaNbNiNeMAxhXv' hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC6finishMF aNbNiNfZG20h+0xdb): undefined reference to `_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA3putMFNaNbNiNeMAxhXv' hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC6finishMF aNbNiNfZG20h+0xe8): undefined reference to `_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA6finishMFNaNbNiNeZG20h' hmac.o: In function `_D3std6format61__T11formatValueTS3std5array17__T8AppenderTAyaZ8AppenderThTaZ11formatValueFNaNfS3std5array17__T8AppenderTAyaZ8AppenderhKS3std6format18__T10FormatSpecTaZ10FormatSpecZv': hmac.d:(.text._D3std6format61__T11formatValueTS3std5array17__T8AppenderTAyaZ8AppenderThTaZ11formatValueFNaNfS3std5array17__T8AppenderTAyaZ8AppenderhKS3std6format18__T10FormatSpecTaZ10 ormatSpecZv+0x16a): undefined reference to `_D3std6format64__T14formatIntegralTS3std5array17__T8AppenderTAyaZ8AppenderTmTaZ14formatIntegralFNaNbNfS3std5array17__T8AppenderTAyaZ8AppenderxmKxS3std6format18__T10FormatSpecTaZ10FormatSpeckmZv' hmac.o: In function `_D3std6format62__T11formatValueTS3std5array17__T8AppenderTAyaZ8AppenderTxhTaZ11formatValueFNaNfS3std5array17__T8AppenderTAyaZ8AppenderxhKS3std6format18__T10FormatSpecTaZ10FormatSpecZv': hmac.d:(.text._D3std6format62__T11formatValueTS3std5array17__T8AppenderTAyaZ8AppenderTxhTaZ11formatValueFNaNfS3std5array17__T8AppenderTAyaZ8AppenderxhKS3std6format18__T10FormatSpecTaZ10 ormatSpecZv+0x16a): undefined reference to `_D3std6format64__T14formatIntegralTS3std5array17__T8AppenderTAyaZ8AppenderTmTaZ14formatIntegralFNaNbNfS3std5array17__T8AppenderTAyaZ8AppenderxmKxS3std6format18__T10FormatSpecTaZ10FormatSpeckmZv' hmac.o: In function `_D3std6format64__T14formatIntegralTS3std5array17__T8AppenderTAyaZ8AppenderTlTaZ14formatIntegralFNaNbNfS3std5array17__T8AppenderTAyaZ8AppenderxlKxS3std6format18__T10FormatSpecTaZ10FormatSpeckmZv': hmac.d:(.text._D3std6format64__T14formatIntegralTS3std5array17__T8AppenderTAyaZ8AppenderTlTaZ14formatIntegralFNaNbNfS3std5array17__T8AppenderTAyaZ8AppenderxlKxS3std6format18__T10FormatSpecTaZ10F rmatSpeckmZv+0x46): undefined reference to `_D3std6format64__T14formatUnsignedTS3std5array17__T8AppenderTAyaZ8AppenderTmTaZ14formatUnsignedFNaNbNfS3std5array17__T8AppenderTAyaZ8AppendermKxS3std6format18__T10FormatSpecTaZ10FormatSpeckbZv' hmac.o: In function `_D3std6format62__T11formatValueTS3std5array17__T8AppenderTAyaZ8AppenderTxmTaZ11formatValueFNaNfS3std5array17__T8AppenderTAyaZ8AppenderxmKS3std6format18__T10FormatSpecTaZ10FormatSpecZv': hmac.d:(.text._D3std6format62__T11formatValueTS3std5array17__T8AppenderTAyaZ8AppenderTxmTaZ11formatValueFNaNfS3std5array17__T8AppenderTAyaZ8AppenderxmKS3std6format18__T10FormatSpecTaZ10 ormatSpecZv+0x16a): undefined reference to `_D3std6format64__T14formatIntegralTS3std5array17__T8AppenderTAyaZ8AppenderTmTaZ14formatIntegralFNaNbNfS3std5array17__T8AppenderTAyaZ8AppenderxmKxS3std6format18__T10FormatSpecTaZ10FormatSpeckmZv' collect2: error: ld returned 1 exit status --- errorlevel 1 [holo ultraxps workplace]$ cat hmac.d #!/usr/bin/rdmd import std.stdio; import std.digest.sha, std.digest.hmac; import std.string : representation; void main() { auto hmac = hmac!SHA1("secretkey".representation); hmac.put("texttohash".representation); auto digest = hmac.finish(); writeln(digest); } [holo ultraxps workplace]$
Oct 05 2015
parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Monday, 5 October 2015 at 19:43:39 UTC, holo wrote:
  Vladimir where can i check or do you know when next version of 
 phobos will be available?
You can use Digger to get the latest version of D: https://github.com/CyberShadow/Digger
Oct 05 2015
parent reply holo <holosian gmail.com> writes:
On Monday, 5 October 2015 at 21:00:38 UTC, Vladimir Panteleev 
wrote:
 On Monday, 5 October 2015 at 19:43:39 UTC, holo wrote:
  Vladimir where can i check or do you know when next version 
 of phobos will be available?
You can use Digger to get the latest version of D: https://github.com/CyberShadow/Digger
Thank you, now it is working as expected. Hope i wont miss upgrade in my system to uninstall digger version before. [holo ultraxps workplace]$ cat ./hmac.d #!/usr/bin/rdmd import std.stdio; import std.digest.sha, std.digest.hmac; import std.string : representation; void main() { auto hmac = hmac!SHA256("secretkey".representation); hmac.put("texttohash".representation); auto digest = hmac.finish(); writeln(digest); } [holo ultraxps workplace]$ ./hmac.d [28, 169, 142, 131, 133, 104, 149, 47, 205, 215, 20, 154, 170, 148, 84, 170, 252, 36, 10, 119, 18, 25, 10, 145, 183, 133, 135, 252, 26, 124, 215, 6] [holo ultraxps workplace]$ dmd --version DMD64 D Compiler v2.069-devel-5e73c30 Copyright (c) 1999-2015 by Digital Mars written by Walter Bright [holo ultraxps workplace]$ Now need try to write some code which will connect me to AWS api based on examples.
Oct 05 2015
parent reply Rikki Cattermole <alphaglosined gmail.com> writes:
On 06/10/15 11:26 AM, holo wrote:
 On Monday, 5 October 2015 at 21:00:38 UTC, Vladimir Panteleev wrote:
 On Monday, 5 October 2015 at 19:43:39 UTC, holo wrote:
  Vladimir where can i check or do you know when next version of
 phobos will be available?
You can use Digger to get the latest version of D: https://github.com/CyberShadow/Digger
Thank you, now it is working as expected. Hope i wont miss upgrade in my system to uninstall digger version before. [holo ultraxps workplace]$ cat ./hmac.d #!/usr/bin/rdmd import std.stdio; import std.digest.sha, std.digest.hmac; import std.string : representation; void main() { auto hmac = hmac!SHA256("secretkey".representation); hmac.put("texttohash".representation); auto digest = hmac.finish(); writeln(digest); } [holo ultraxps workplace]$ ./hmac.d [28, 169, 142, 131, 133, 104, 149, 47, 205, 215, 20, 154, 170, 148, 84, 170, 252, 36, 10, 119, 18, 25, 10, 145, 183, 133, 135, 252, 26, 124, 215, 6] [holo ultraxps workplace]$ dmd --version DMD64 D Compiler v2.069-devel-5e73c30 Copyright (c) 1999-2015 by Digital Mars written by Walter Bright [holo ultraxps workplace]$ Now need try to write some code which will connect me to AWS api based on examples.
Congrats on getting it working!
Oct 05 2015
parent reply holo <holosian gmail.com> writes:
 Congrats on getting it working!
Rikki Thanks :) I was trying to write my own lib from beginning based on examples but after some time i resign from that idea (will back to it when i will have some more experience) and right now im trying to customize that one from link which yawniek paste: https://github.com/yannick/vibe-aws/blob/master/source/vibe/aws/sigv4.d I removed import from vibe.d library and copy/paste missed functions formEncode and filterURLEncode (BTW: what that "(R)" mean in it? filterURLEncode(R)(ref R dst, ..., ..) ). Next thing what i did was to replace hmac function to use hmac module from newest library. Now whole code looks like this: module sigv4; import std.array; import std.algorithm; import std.digest.sha; import std.range; import std.stdio; import std.string; import std.format; import std.digest.hmac; const algorithm = "AWS4-HMAC-SHA256"; void filterURLEncode(R)(ref R dst, string str, string allowed_chars = null, bool form_encoding = false) { while( str.length > 0 ) { switch(str[0]) { case ' ': if (form_encoding) { dst.put('+'); break; } goto default; case 'A': .. case 'Z': case 'a': .. case 'z': case '0': .. case '9': case '-': case '_': case '.': case '~': dst.put(str[0]); break; default: if (allowed_chars.canFind(str[0])) dst.put(str[0]); else formattedWrite(dst, "%%%02X", str[0]); } str = str[1 .. $]; } } string formEncode(string str, string allowed_chars = null) safe { auto dst = appender!string(); dst.reserve(str.length); filterURLEncode(dst, str, allowed_chars, true); return dst.data; } struct CanonicalRequest { string method; string uri; string[string] queryParameters; string[string] headers; ubyte[] payload; } string canonicalQueryString(string[string] queryParameters) { alias encode = formEncode; string[string] encoded; foreach (p; queryParameters.keys()) { encoded[encode(p)] = encode(queryParameters[p]); } string[] keys = encoded.keys(); sort(keys); return keys.map!(k => k ~ "=" ~ encoded[k]).join("&"); } string canonicalHeaders(string[string] headers) { string[string] trimmed; foreach (h; headers.keys()) { trimmed[h.toLower().strip()] = headers[h].strip(); } string[] keys = trimmed.keys(); sort(keys); return keys.map!(k => k ~ ":" ~ trimmed[k] ~ "\n").join(""); } string signedHeaders(string[string] headers) { string[] keys = headers.keys().map!(k => k.toLower()).array(); sort(keys); return keys.join(";"); } string hash(T)(T payload) { auto hash = sha256Of(payload); return hash.toHexString().toLower(); } string makeCRSigV4(CanonicalRequest r) { auto cr = r.method.toUpper() ~ "\n" ~ (r.uri.empty ? "/" : r.uri) ~ "\n" ~ canonicalQueryString(r.queryParameters) ~ "\n" ~ canonicalHeaders(r.headers) ~ "\n" ~ signedHeaders(r.headers) ~ "\n" ~ hash(r.payload); return hash(cr); } unittest { string[string] empty; auto r = CanonicalRequest( "POST", "/", empty, ["content-type": "application/x-www-form-urlencoded; charset=utf-8", "host": "iam.amazonaws.com", "x-amz-date": "20110909T233600Z"], cast(ubyte[])"Action=ListUsers&Version=2010-05-08"); auto sig = makeCRSigV4(r); assert(sig == "3511de7e95d28ecd39e9513b642aee07e54f4941150d8df8bf94b328ef7e55e2"); } struct SignableRequest { string dateString; string timeStringUTC; string region; string service; CanonicalRequest canonicalRequest; } string signableString(SignableRequest r) { return algorithm ~ "\n" ~ r.dateString ~ "T" ~ r.timeStringUTC ~ "Z\n" ~ r.dateString ~ "/" ~ r.region ~ "/" ~ r.service ~ "/aws4_request\n" ~ makeCRSigV4(r.canonicalRequest); } unittest { string[string] empty; SignableRequest r; r.dateString = "20110909"; r.timeStringUTC = "233600"; r.region = "us-east-1"; r.service = "iam"; r.canonicalRequest = CanonicalRequest( "POST", "/", empty, ["content-type": "application/x-www-form-urlencoded; charset=utf-8", "host": "iam.amazonaws.com", "x-amz-date": "20110909T233600Z"], cast(ubyte[])"Action=ListUsers&Version=2010-05-08"); auto sampleString = algorithm ~ "\n" ~ "20110909T233600Z\n" ~ "20110909/us-east-1/iam/aws4_request\n" ~ "3511de7e95d28ecd39e9513b642aee07e54f4941150d8df8bf94b328ef7e55e2"; assert(sampleString == signableString(r)); } ubyte[] array_xor(ubyte[] b1, ubyte[] b2) { assert(b1.length == b2.length); ubyte[] ret; for (uint i = 0; i < b1.length; i++) ret ~= b1[i] ^ b2[i]; return ret; } auto hmac_sha256(string key, string message) { auto hmac = hmac!SHA256(key.representation); hmac.put(message.representation); return hmac.finish; } unittest { string key = "key"; string message = "The quick brown fox jumps over the lazy dog"; string mac = hmac_sha256(key, message).toHexString().toLower(); assert(mac == "f7bc83f430538424b13298e6aa6fb143ef4d59a14946175997479dbc2d1a3cd8"); } auto signingKey(string secret, string dateString, string region, string service) { auto kSecret = "AWS4" ~ secret; auto kDate = hmac_sha256(kSecret, dateString); auto kRegion = hmac_sha256(kDate, region); auto kService = hmac_sha256(kRegion, service); return hmac_sha256(kService, "aws4_request"); } unittest { string secretKey = "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY"; auto signKey = signingKey(secretKey, "20110909", "us-east-1", "iam"); ubyte[] expected = [152, 241, 216, 137, 254, 196, 244, 66, 26, 220, 82, 43, 171, 12, 225, 248, 46, 105, 41, 194, 98, 237, 21, 229, 169, 76, 144, 239, 209, 227, 176, 231 ]; assert(expected == signKey); } alias sign = hmac_sha256; unittest { auto sampleString = "AWS4-HMAC-SHA256\n" ~ "20110909T233600Z\n" ~ "20110909/us-east-1/iam/aws4_request\n" ~ "3511de7e95d28ecd39e9513b642aee07e54f4941150d8df8bf94b328ef7e55e2"; auto secretKey = "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY"; auto signKey = signingKey(secretKey, "20110909", "us-east-1", "iam"); auto signature = sign(signKey, sampleString).toHexString().toLower(); auto expected = "ced6826de92d2bdeed8f846f0bf508e8559e98e4b0199114b84c54174deb456c"; assert(signature == expected); } /** * CredentialScope == date / region / service / aws4_request */ string createSignatureHeader(string accessKeyID, string credentialScope, string[string] reqHeaders, ubyte[] signature) { return algorithm ~ " Credential=" ~ accessKeyID ~ "/" ~ credentialScope ~ "/aws4_request, SignedHeaders=" ~ signedHeaders(reqHeaders) ~ ", Signature=" ~ signature.toHexString().toLower(); } string dateFromISOString(string iso) { auto i = iso.indexOf('T'); if (i == -1) throw new Exception("ISO time in wrong format: " ~ iso); return iso[0..i]; } string timeFromISOString(string iso) { auto t = iso.indexOf('T'); auto z = iso.indexOf('Z'); if (t == -1 || z == -1) throw new Exception("ISO time in wrong format: " ~ iso); return iso[t+1..z]; } unittest { assert(dateFromISOString("20110909T1203Z") == "20110909"); } void main() { auto sampleString = "AWS4-HMAC-SHA256\n" ~ "20110909T233600Z\n" ~ "20110909/us-east-1/iam/aws4_request\n" ~ "3511de7e95d28ecd39e9513b642aee07e54f4941150d8df8bf94b328ef7e55e2"; auto secretKey = "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY"; auto signKey = signingKey(secretKey, "20110909", "us-east-1", "iam"); auto signature = sign(signKey, sampleString).toHexString().toLower(); writeln(signature); } When i try to compile it im getting such error: [holo ultraxps test]$ dmd -unittest hello.d hello.d(196): Error: function sigv4.hmac_sha256 (string key, string message) is not callable using argument types (ubyte[32], string) [holo ultraxps test]$ dmd hello.d hello.d(196): Error: function sigv4.hmac_sha256 (string key, string message) is not callable using argument types (ubyte[32], string) [holo ultraxps test]$ Line 196 is: "auto kRegion = hmac_sha256(kDate, region);" I was looking for ubyte[32] variable but i cant find it anywhere. What is strange unit tests are passing (i think, maybe im wrong).
Oct 06 2015
parent reply Rikki Cattermole <alphaglosined gmail.com> writes:
On 07/10/15 3:18 PM, holo wrote:
 Congrats on getting it working!
Rikki Thanks :) I was trying to write my own lib from beginning based on examples but after some time i resign from that idea (will back to it when i will have some more experience) and right now im trying to customize that one from link which yawniek paste: https://github.com/yannick/vibe-aws/blob/master/source/vibe/aws/sigv4.d I removed import from vibe.d library and copy/paste missed functions formEncode and filterURLEncode (BTW: what that "(R)" mean in it? filterURLEncode(R)(ref R dst, ..., ..) ).
If you see a template argument (which R is) it is typically meant for a range. In this case an output range. Although there should be isOutputRange!R in an template if condition, to check this.
 Next thing what i did was to replace hmac function to use hmac module
 from newest library. Now whole code looks like this:

 module sigv4;

 import std.array;
 import std.algorithm;
 import std.digest.sha;
 import std.range;
 import std.stdio;
 import std.string;
 import std.format;
 import std.digest.hmac;


 const algorithm = "AWS4-HMAC-SHA256";


 void filterURLEncode(R)(ref R dst, string str, string allowed_chars =
 null, bool form_encoding = false)
 {
      while( str.length > 0 ) {
          switch(str[0]) {
              case ' ':
                  if (form_encoding) {
                      dst.put('+');
                      break;
                  }
                  goto default;
              case 'A': .. case 'Z':
              case 'a': .. case 'z':
              case '0': .. case '9':
              case '-': case '_': case '.': case '~':
                  dst.put(str[0]);
                  break;
              default:
                  if (allowed_chars.canFind(str[0])) dst.put(str[0]);
                  else formattedWrite(dst, "%%%02X", str[0]);
          }
          str = str[1 .. $];
      }
 }

 string formEncode(string str, string allowed_chars = null)
  safe {
      auto dst = appender!string();
      dst.reserve(str.length);
      filterURLEncode(dst, str, allowed_chars, true);
      return dst.data;
 }

 struct CanonicalRequest
 {
      string method;
      string uri;
      string[string] queryParameters;
      string[string] headers;
      ubyte[] payload;
 }

 string canonicalQueryString(string[string] queryParameters)
 {
      alias encode = formEncode;

      string[string] encoded;
      foreach (p; queryParameters.keys())
      {
          encoded[encode(p)] = encode(queryParameters[p]);
      }
      string[] keys = encoded.keys();
      sort(keys);
      return keys.map!(k => k ~ "=" ~ encoded[k]).join("&");
 }

 string canonicalHeaders(string[string] headers)
 {
      string[string] trimmed;
      foreach (h; headers.keys())
      {
          trimmed[h.toLower().strip()] = headers[h].strip();
      }
      string[] keys = trimmed.keys();
      sort(keys);
      return keys.map!(k => k ~ ":" ~ trimmed[k] ~ "\n").join("");
 }

 string signedHeaders(string[string] headers)
 {
      string[] keys = headers.keys().map!(k => k.toLower()).array();
      sort(keys);
      return keys.join(";");
 }

 string hash(T)(T payload)
 {
      auto hash = sha256Of(payload);
      return hash.toHexString().toLower();
 }

 string makeCRSigV4(CanonicalRequest r)
 {
      auto cr =
          r.method.toUpper() ~ "\n" ~
          (r.uri.empty ? "/" : r.uri) ~ "\n" ~
          canonicalQueryString(r.queryParameters) ~ "\n" ~
          canonicalHeaders(r.headers) ~ "\n" ~
          signedHeaders(r.headers) ~ "\n" ~
          hash(r.payload);

      return hash(cr);
 }

 unittest {
      string[string] empty;

      auto r = CanonicalRequest(
              "POST",
              "/",
              empty,
              ["content-type": "application/x-www-form-urlencoded;
 charset=utf-8",
               "host": "iam.amazonaws.com",
               "x-amz-date": "20110909T233600Z"],
              cast(ubyte[])"Action=ListUsers&Version=2010-05-08");

      auto sig = makeCRSigV4(r);

      assert(sig ==
 "3511de7e95d28ecd39e9513b642aee07e54f4941150d8df8bf94b328ef7e55e2");
 }

 struct SignableRequest
 {
      string dateString;
      string timeStringUTC;
      string region;
      string service;
      CanonicalRequest canonicalRequest;
 }

 string signableString(SignableRequest r) {
      return algorithm ~ "\n" ~
          r.dateString ~ "T" ~ r.timeStringUTC ~ "Z\n" ~
          r.dateString ~ "/" ~ r.region ~ "/" ~ r.service ~
 "/aws4_request\n" ~
          makeCRSigV4(r.canonicalRequest);
 }

 unittest {
      string[string] empty;

      SignableRequest r;
      r.dateString = "20110909";
      r.timeStringUTC = "233600";
      r.region = "us-east-1";
      r.service = "iam";
      r.canonicalRequest = CanonicalRequest(
              "POST",
              "/",
              empty,
              ["content-type": "application/x-www-form-urlencoded;
 charset=utf-8",
               "host": "iam.amazonaws.com",
               "x-amz-date": "20110909T233600Z"],
              cast(ubyte[])"Action=ListUsers&Version=2010-05-08");

      auto sampleString =
          algorithm ~ "\n" ~
          "20110909T233600Z\n" ~
          "20110909/us-east-1/iam/aws4_request\n" ~
 "3511de7e95d28ecd39e9513b642aee07e54f4941150d8df8bf94b328ef7e55e2";

      assert(sampleString == signableString(r));
 }

 ubyte[] array_xor(ubyte[] b1, ubyte[] b2)
 {
      assert(b1.length == b2.length);
      ubyte[] ret;
      for (uint i = 0; i < b1.length; i++)
          ret ~= b1[i] ^ b2[i];
      return ret;
 }

 auto hmac_sha256(string key, string message)
 {
      auto hmac = hmac!SHA256(key.representation);
      hmac.put(message.representation);
      return hmac.finish;
 }

 unittest {
      string key = "key";
      string message = "The quick brown fox jumps over the lazy dog";

      string mac = hmac_sha256(key, message).toHexString().toLower();
      assert(mac ==
 "f7bc83f430538424b13298e6aa6fb143ef4d59a14946175997479dbc2d1a3cd8");
 }

 auto signingKey(string secret, string dateString, string region, string
 service)
 {
      auto kSecret = "AWS4" ~ secret;
      auto kDate = hmac_sha256(kSecret, dateString);
      auto kRegion = hmac_sha256(kDate, region);
      auto kService = hmac_sha256(kRegion, service);
      return hmac_sha256(kService, "aws4_request");
 }

 unittest {
      string secretKey = "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY";
      auto signKey = signingKey(secretKey, "20110909", "us-east-1", "iam");

      ubyte[] expected = [152, 241, 216, 137, 254, 196, 244, 66, 26, 220,
 82, 43, 171, 12, 225, 248, 46, 105, 41, 194, 98, 237, 21, 229, 169, 76,
 144, 239, 209, 227, 176, 231 ];
      assert(expected == signKey);
 }

 alias sign = hmac_sha256;

 unittest {
      auto sampleString =
          "AWS4-HMAC-SHA256\n" ~
          "20110909T233600Z\n" ~
          "20110909/us-east-1/iam/aws4_request\n" ~
 "3511de7e95d28ecd39e9513b642aee07e54f4941150d8df8bf94b328ef7e55e2";

      auto secretKey = "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY";
      auto signKey = signingKey(secretKey, "20110909", "us-east-1", "iam");

      auto signature = sign(signKey, sampleString).toHexString().toLower();
      auto expected =
 "ced6826de92d2bdeed8f846f0bf508e8559e98e4b0199114b84c54174deb456c";

      assert(signature == expected);
 }

 /**
   * CredentialScope == date / region / service / aws4_request
   */
 string createSignatureHeader(string accessKeyID, string credentialScope,
 string[string] reqHeaders, ubyte[] signature)
 {
      return algorithm ~ " Credential=" ~ accessKeyID ~ "/" ~
 credentialScope ~ "/aws4_request, SignedHeaders=" ~
 signedHeaders(reqHeaders) ~ ", Signature=" ~
 signature.toHexString().toLower();
 }

 string dateFromISOString(string iso)
 {
      auto i = iso.indexOf('T');
      if (i == -1) throw new Exception("ISO time in wrong format: " ~ iso);
      return iso[0..i];
 }

 string timeFromISOString(string iso)
 {
      auto t = iso.indexOf('T');
      auto z = iso.indexOf('Z');
      if (t == -1 || z == -1) throw new Exception("ISO time in wrong
 format: " ~ iso);
      return iso[t+1..z];
 }

 unittest {
      assert(dateFromISOString("20110909T1203Z") == "20110909");
 }


 void main()
 {
      auto sampleString =
          "AWS4-HMAC-SHA256\n" ~
          "20110909T233600Z\n" ~
          "20110909/us-east-1/iam/aws4_request\n" ~
 "3511de7e95d28ecd39e9513b642aee07e54f4941150d8df8bf94b328ef7e55e2";

      auto secretKey = "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY";
      auto signKey = signingKey(secretKey, "20110909", "us-east-1", "iam");

      auto signature = sign(signKey, sampleString).toHexString().toLower();

      writeln(signature);
 }

 When i try to compile it im getting such error:

 [holo ultraxps test]$ dmd -unittest hello.d
 hello.d(196): Error: function sigv4.hmac_sha256 (string key, string
 message) is not callable using argument types (ubyte[32], string)
 [holo ultraxps test]$ dmd hello.d
 hello.d(196): Error: function sigv4.hmac_sha256 (string key, string
 message) is not callable using argument types (ubyte[32], string)
 [holo ultraxps test]$


 Line 196 is: "auto kRegion = hmac_sha256(kDate, region);"

 I was looking for ubyte[32] variable but i cant find it anywhere. What
 is strange unit tests are passing (i think, maybe im wrong).
It's in signingKey. Remember auto just means work out type based upon assigning value's type. I would really really recommend since you keep having issues with hmac, to go the Botan way. Since it works.
Oct 06 2015
parent reply holo <holosian gmail.com> writes:
I was fighting with it a little bit and after all i just leave 
original function which was in code, removed dependencies from 
vibe.d and finally tried to contact api. Here is my present code:

#!/usr/bin/rdmd -L-lcurl

import std.stdio;
import std.string;
import std.file;
import std.datetime;
import std.process;
import std.digest.sha;
import std.net.curl;
import sigv4;


auto zone = "us-east-1";
auto service = "ec2";


void main()
{
	auto accKey = environment["AWS_ACCESS_KEY"];
	auto secKey = environment["AWS_SECRET_KEY"];

	auto currentClock = Clock.currTime;

	auto currentDate = cast(Date)currentClock;
	auto curDateStr = currentDate.toISOString;

	auto currentTime = cast(TimeOfDay)currentClock;
	auto curTimeStr = currentTime.toISOString;

	auto xamztime = curDateStr ~ "T" ~ curTimeStr ~ "Z";

	string[string] empty;

	SignableRequest r;
	r.dateString = curDateStr;
	r.timeStringUTC = curTimeStr;
	r.region = zone;
	r.service = service;
	r.canonicalRequest = CanonicalRequest(
			"GET",
			"/",
			["action" : "DescribeInstances", "version" : "2013-10-15"],
			["content-type" : "application/x-www-form-urlencoded; 
charset=utf-8",
			 "host" : service ~ ".amazonaws.com",
			 "x-amz-date" : xamztime],
			 cast(ubyte[])"");
	
	auto qParm = 
canonicalQueryString(r.canonicalRequest.queryParameters);

	auto sigHead = canonicalHeaders(r.canonicalRequest.headers);

	auto sigStr = signableString(r);

	auto sigKey = signingKey(secKey, curDateStr, zone, service);
	
	auto signature = sign(sigKey, 
cast(ubyte[])sigStr).toHexString().toLower();

	writeln();	
	writeln(qParm);
	writeln();
	writeln(sigHead);
	writeln();
	writeln(sigStr);
	writeln();
	writeln(signature);
	writeln();

	auto client = HTTP();
	client.clearRequestHeaders;
	client.addRequestHeader("content-type:", 
"application/x-www-form-urlencoded; charset=utf-8");
	client.addRequestHeader("host:", service ~ ".amazonaws.com");
	client.addRequestHeader("x-amz-date:", xamztime);
	client.addRequestHeader("Authoryzation:", "AWS4-HMAC-SHA256" ~ " 
" ~ "Credential=" ~ accKey ~ "/" ~ xamztime ~ "/" ~ zone ~ "/" ~ 
service ~ "/" ~ "aws4_request" ~ ", " ~ "SignedHeaders=" ~ 
"content-type;host;x-amz-date" ~ ", " ~ "Signature=" ~ signature);

	auto url = service ~ ".amazonaws.com?" ~ 
"Action=DescribeInstances&Version=2013-10-15";
	writeln(url);
	auto content = get(url, client);
	writeln(content);
}

Everything is compiling but im getting 400 (Bad Request):

   [root ultraxps aws]# ./header.d

action=DescribeRegions&version=2013-10-15

content-type: application/x-www-form-urlencoded; charset=utf-8
host: ec2.amazonaws.com
x-amz-date: 20151009T053800Z


AWS4-HMAC-SHA256
20151009T053800Z
20151009/us-east-1/ec2/aws4_request
888595748692147ceafafcae3941ec0d83ac42c97641e4d954d7447a00c56270

69b1e4c5212cc6b485569fdfb43f7dde94413b36c50393c55d4810ced47f167b

ec2.amazonaws.com?Action=DescribeRegions&Version=2013-10-15
std.net.curl.CurlException /usr/include/dlang/dmd/std/net/curl.d(824): HTTP
request returned status code 400 (Bad Request)
----------------
/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(pure  safe
bool std.exception.enforce!(std.net.curl.CurlException, bool).enforce(bool,
lazy const(char)[], immutable(char)[], ulong)+0x65) [0x5004bd]
/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(char[]
std.net.curl._basicHTTP!(char)._basicHTTP(const(char)[], const(void)[],
std.net.curl.HTTP)+0x1a2) [0x4fc0ba]
/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(char[]
std.net.curl.get!(std.net.curl.HTTP, char).get(const(char)[],
std.net.curl.HTTP)+0x6a) [0x4fbec2]
/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/
eader(_Dmain+0x840) [0x4f8f98]
/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(_D2rt6dmain211_d_run_mainUiPPaPUAAaZiZ6runAllMFZ
__lambda1MFZv+0x1f) [0x533eab]
/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(void
rt.dmain2._d_run_main(int, char**, extern (C) int
function(char[][])*).tryExec(scope void delegate())+0x2a) [0x533e06]
/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(void
rt.dmain2._d_run_main(int, char**, extern (C) int
function(char[][])*).runAll()+0x2b) [0x533e67]
/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(void
rt.dmain2._d_run_main(int, char**, extern (C) int
function(char[][])*).tryExec(scope void delegate())+0x2a) [0x533e06]
/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/heade
(_d_run_main+0x1d2) [0x533d86]
/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(main+0x12)
[0x52cb62]
/usr/lib/libc.so.6(__libc_start_main+0xf0) [0x7f9c39f71610]
[root ultraxps aws]#

What am i doing wrong? What is strange when when i do same with 
curl its responding normally of course is not authenticated).

[root ultraxps aws]# curl 
ec2.amazonaws.com?Action=DescribeInstances&Version=2013-10-15
[1] 20390
[root ultraxps aws]# <?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>MissingParameter</Code><Message>The request must
contain the parameter
AWSAccessKeyId</Message></Error></Errors><RequestID>e1352781-c2b4-4e74-ade3-80d655efd0ac</RequestID></Response>
Oct 08 2015
parent reply Laeeth Isharc <laeethnospam nospam.laeeth.com> writes:
On Friday, 9 October 2015 at 04:04:57 UTC, holo wrote:

 	r.dateString = 	client.addRequestHeader("Authoryzation:", 
 "AWS4-HMAC-SHA256" ~ " " ~ "Credential=" ~ accKey ~ "/" ~ 
 xamztime ~ "/" ~ zone ~ "/" ~ service ~ "/" ~ "aws4_request" ~ 
 ", " ~ "SignedHeaders=" ~ "content-type;host;x-amz-date" ~ ", " 
 ~ "Signature=" ~ signature);
authorisation ??? (Or maybe authorization) ie a typo (check other fields carefully too, just in case)
 	auto url = service ~ ".amazonaws.com?" ~ 
 "Action=DescribeInstances&Version=2013-10-15";
 	writeln(url);
 	auto content = get(url, client);
 	writeln(content);
 }

 Everything is compiling but im getting 400 (Bad Request):

   [root ultraxps aws]# ./header.d

 action=DescribeRegions&version=2013-10-15

 content-type: application/x-www-form-urlencoded; charset=utf-8
 host: ec2.amazonaws.com
 x-amz-date: 20151009T053800Z


 AWS4-HMAC-SHA256
 20151009T053800Z
 20151009/us-east-1/ec2/aws4_request
 888595748692147ceafafcae3941ec0d83ac42c97641e4d954d7447a00c56270

 69b1e4c5212cc6b485569fdfb43f7dde94413b36c50393c55d4810ced47f167b

 ec2.amazonaws.com?Action=DescribeRegions&Version=2013-10-15
 std.net.curl.CurlException /usr/include/dlang/dmd/std/net/curl.d(824): HTTP
request returned status code 400 (Bad Request)
 ----------------
 /tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(pure  safe
bool std.exception.enforce!(std.net.curl.CurlException, bool).enforce(bool,
lazy const(char)[], immutable(char)[], ulong)+0x65) [0x5004bd]
 /tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(char[]
std.net.curl._basicHTTP!(char)._basicHTTP(const(char)[], const(void)[],
std.net.curl.HTTP)+0x1a2) [0x4fc0ba]
 /tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(char[]
std.net.curl.get!(std.net.curl.HTTP, char).get(const(char)[],
std.net.curl.HTTP)+0x6a) [0x4fbec2]
 /tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/
eader(_Dmain+0x840) [0x4f8f98]
 /tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(_D2rt6dmain211_d_run_mainUiPPaPUAAaZiZ6runAllMFZ
__lambda1MFZv+0x1f) [0x533eab]
 /tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(void
rt.dmain2._d_run_main(int, char**, extern (C) int
function(char[][])*).tryExec(scope void delegate())+0x2a) [0x533e06]
 /tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(void
rt.dmain2._d_run_main(int, char**, extern (C) int
function(char[][])*).runAll()+0x2b) [0x533e67]
 /tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(void
rt.dmain2._d_run_main(int, char**, extern (C) int
function(char[][])*).tryExec(scope void delegate())+0x2a) [0x533e06]
 /tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/heade
(_d_run_main+0x1d2) [0x533d86]
 /tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(main+0x12)
[0x52cb62]
 /usr/lib/libc.so.6(__libc_start_main+0xf0) [0x7f9c39f71610]
 [root ultraxps aws]#

 What am i doing wrong? What is strange when when i do same with 
 curl its responding normally of course is not authenticated).

 [root ultraxps aws]# curl 
 ec2.amazonaws.com?Action=DescribeInstances&Version=2013-10-15
 [1] 20390
 [root ultraxps aws]# <?xml version="1.0" encoding="UTF-8"?>
 <Response><Errors><Error><Code>MissingParameter</Code><Message>The request
must contain the parameter
AWSAccessKeyId</Message></Error></Errors><RequestID>e1352781-c2b4-4e74-ade3-80d655efd0ac</RequestID></Response>
Oct 08 2015
parent reply holo <holosian gmail.com> writes:
I correct typo (thats whats happening when you trying to solve 
problem after night at 6:00 AM ).But still  have same error. When 
use curl with same generated creditentials im getting "400". What 
is strange when im using same generated headers and signature 
with command line curl im getting "500". My headers right now:

auto client = HTTP();
         client.clearRequestHeaders;
         client.addRequestHeader("Content-Type:", 
"application/x-www-form-urlencoded; charset=utf-8");
         client.addRequestHeader("Host:", service ~ 
".amazonaws.com");
         client.addRequestHeader("X-Amz-Date:", xamztime);
         client.addRequestHeader("Authorization:", 
"AWS4-HMAC-SHA256" ~ " " ~ "Credential=" ~ accKey ~ "/" ~ 
xamztime ~ "/" ~ zone ~ "/" ~ service ~ "/" ~ "aws4_request" ~ ", 
" ~ "SignedHeaders=" ~ "content-type;host;x-amz-date" ~ ", " ~ 
"Signature=" ~ signature);

         auto url = service ~ ".amazonaws.com?" ~ 
"Action=DescribeInstances&Version=2013-10-15";
         writeln(url);
         auto content = get(url, client);
         writeln(content);


Did i set up it correctly?
Oct 09 2015
parent reply holo <holosian gmail.com> writes:
I doped HTTP packets with wireshark, maybe that  will  be useful:

First recived:

  Hypertext Transfer Protocol
     HTTP/1.1 400 Bad Request\r\n
         [Expert Info (Chat/Sequence): HTTP/1.1 400 Bad 
Request\r\n]
         Request Version: HTTP/1.1
         Status Code: 400
         Response Phrase: Bad Request
     Transfer-Encoding: chunked\r\n
     Date: Fri, 09 Oct 2015 15:43:37 GMT\r\n
     Cneonction: close\r\n
     Server: AmazonEC2\r\n
     \r\n
     [HTTP response 1/2]
     [Next request in frame: 7739]
     HTTP chunked response

Second sent:

Hypertext Transfer Protocol
     GET /?Action=DescribeInstances&Version=2013-10-15 HTTP/1.1\r\n
         [Expert Info (Chat/Sequence): GET 
/?Action=DescribeInstances&Version=2013-10-15 HTTP/1.1\r\n]
         Request Method: GET
         Request URI: /?Action=DescribeInstances&Version=2013-10-15
         Request Version: HTTP/1.1
     Host:: ec2.amazonaws.com\r\n
     User-Agent: Phobos-std.net.curl/2.069 (libcurl/7.44.0)\r\n
     Accept: */*\r\n
     Content-Type:: application/x-www-form-urlencoded; 
charset=utf-8\r\n
     X-Amz-Date:: 20151009T174337Z\r\n
     Authorization:: AWS4-HMAC-SHA256 
Credential=AKIAIHGPAJQ2PJ47YKKA/20151009T174337Z/us-east-1/ec2/aws4_request,
SignedHeaders=content-type;host;x-amz-date,
Signature=3049eff7d6b5084c2fee71d10d5cffb17e0df1a9fe76963859c423f637d0496e\r\n
     \r\n
     [Full request URI: http://: 
ec2.amazonaws.com/?Action=DescribeInstances&Version=2013-10-15]
     [HTTP request 2/2]

What is strange why im getting first that 400 Bad Request packet 
and after that is sent GET one?
Oct 09 2015
next sibling parent Marc =?UTF-8?B?U2Now7x0eg==?= <schuetzm gmx.net> writes:
On Friday, 9 October 2015 at 16:01:33 UTC, holo wrote:
     Host:: ec2.amazonaws.com\r\n
This, ...
     Content-Type:: application/x-www-form-urlencoded; 
 charset=utf-8\r\n
     X-Amz-Date:: 20151009T174337Z\r\n
     Authorization:: AWS4-HMAC-SHA256 
 Credential=AKIAIHGPAJQ2PJ47YKKA/20151009T174337Z/us-east-1/ec2/aws4_request,
SignedHeaders=content-type;host;x-amz-date,
Signature=3049eff7d6b5084c2fee71d10d5cffb17e0df1a9fe76963859c423f637d0496e\r\n
     \r\n
... these ones, ...
     [Full request URI: http://: 
 ec2.amazonaws.com/?Action=DescribeInstances&Version=2013-10-15]
     [HTTP request 2/2]
... and this look wrong. Where does the additional ": " come from?
Oct 09 2015
prev sibling parent reply holo <holosian gmail.com> writes:
OK i find out error, in addRequestHeader i was using ":" after 
header name what casing problem. I removed it and now im getting 
"unauthorized". Here is how it looks right now:

  HTTP/1.1 401 Unauthorized\r\n
         [Expert Info (Chat/Sequence): HTTP/1.1 401 
Unauthorized\r\n]
         Request Version: HTTP/1.1
         Status Code: 401
         Response Phrase: Unauthorized
     Transfer-Encoding: chunked\r\n
     Date: Fri, 09 Oct 2015 16:22:47 GMT\r\n
     Server: AmazonEC2\r\n
     \r\n
     [HTTP response 1/2]
     [Next request in frame: 8371]
     HTTP chunked response
         Data chunk (254 octets)
             Chunk size: 254 octets
             Data (254 bytes)

In data field i can read:

"AWS was not able to validate provided access credentials" Is my 
signing process incorrect?
Oct 09 2015
parent reply holo <holosian gmail.com> writes:
On Friday, 9 October 2015 at 16:30:26 UTC, holo wrote:
 OK i find out error, in addRequestHeader i was using ":" after 
 header name what casing problem. I removed it and now im 
 getting "unauthorized". Here is how it looks right now:

  HTTP/1.1 401 Unauthorized\r\n
         [Expert Info (Chat/Sequence): HTTP/1.1 401 
 Unauthorized\r\n]
         Request Version: HTTP/1.1
         Status Code: 401
         Response Phrase: Unauthorized
     Transfer-Encoding: chunked\r\n
     Date: Fri, 09 Oct 2015 16:22:47 GMT\r\n
     Server: AmazonEC2\r\n
     \r\n
     [HTTP response 1/2]
     [Next request in frame: 8371]
     HTTP chunked response
         Data chunk (254 octets)
             Chunk size: 254 octets
             Data (254 bytes)

 In data field i can read:

 "AWS was not able to validate provided access credentials" Is 
 my signing process incorrect?
Maybe i will put my present code again: #!/usr/bin/rdmd -L-lcurl import std.stdio; import std.string; import std.file; import std.datetime; import std.process; import std.digest.sha; import std.net.curl; import std.uri; import sigv4; auto zone = "us-east-1"; auto service = "ec2"; void main() { auto accKey = environment["AWS_ACCESS_KEY"]; auto secKey = environment["AWS_SECRET_KEY"]; auto currentClock = Clock.currTime; auto currentDate = cast(Date)currentClock; auto curDateStr = currentDate.toISOString; auto currentTime = cast(TimeOfDay)currentClock; auto curTimeStr = currentTime.toISOString; auto xamztime = curDateStr ~ "T" ~ curTimeStr ~ "Z"; string[string] empty; SignableRequest r; r.dateString = curDateStr; r.timeStringUTC = curTimeStr; r.region = zone; r.service = service; r.canonicalRequest = CanonicalRequest( "POST", "/", ["action" : "DescribeInstances", "version" : "2013-10-15"], // ["accept" : "*/*", ["content-type" : "application/x-www-form-urlencoded; charset=utf-8", "host" : service ~ ".amazonaws.com", "x-amz-date" : xamztime], cast(ubyte[])""); //cast(ubyte[])"Action=DescribeInstances&Version=2013-10-15"); auto qParm = canonicalQueryString(r.canonicalRequest.queryParameters); auto sigHead = canonicalHeaders(r.canonicalRequest.headers); auto sigStr = signableString(r); auto sigKey = signingKey(secKey, curDateStr, zone, service); auto signature = sign(sigKey, cast(ubyte[])sigStr).toHexString().toLower(); writeln(); writeln(qParm); writeln(); writeln(sigHead); writeln(); writeln(sigStr); writeln(); writeln(signature); writeln(); auto client = HTTP(); // client.clearRequestHeaders; client.addRequestHeader("content-type", "application/x-www-form-urlencoded; charset=utf-8"); client.addRequestHeader("host", service ~ ".amazonaws.com"); client.addRequestHeader("x-amz-date", xamztime); client.addRequestHeader("authorization", "AWS4-HMAC-SHA256" ~ " " ~ "Credential=" ~ accKey ~ "/" ~ xamztime ~ "/" ~ zone ~ "/" ~ service ~ "/" ~ "aws4_request" ~ ", " ~ "SignedHeaders=" ~ "content-type;host;x-amz-date" ~ ", " ~ "Signature=" ~ signature); auto url = "ec2.amazonaws.com/?" ~ "Action=DescribeInstances&Version=2013-10-15"; auto urlenc = encode(url); writeln(url); auto content = get(urlenc, client); writeln(content); } Is my signing process correct?
Oct 10 2015
parent reply holo <holosian gmail.com> writes:
After long fight with previous code i try to rewrite "one-to-one" 
python example from 
http://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html
(GET part) from begging to D with full success. Here is working code in clear
D. Im using hmac function which is available from 2.069 of phobos). It is not
too beautiful but like i said im beginner, here it is:


#!/usr/bin/rdmd -L-lcurl

module sigawsv4;

import std.stdio, std.process;
import std.digest.sha, std.digest.hmac;
import std.string;
import std.conv;
import std.datetime;
import std.net.curl;

void main()
{
	auto accessKey = environment["AWS_ACCESS_KEY"];
	auto secretKey = environment["AWS_SECRET_KEY"];

	auto currentClock = Clock.currTime(UTC());

	auto currentDate = cast(Date)currentClock;
	auto curDateStr = currentDate.toISOString;

	auto currentTime = cast(TimeOfDay)currentClock;
	auto curTimeStr = currentTime.toISOString;

	auto xamztime = curDateStr ~ "T" ~ curTimeStr ~ "Z";

	
	string method = "GET";
	string service = "ec2";
	string host = "ec2.amazonaws.com";
	string region = "us-east-1";
	string endpoint = "https://ec2.amazonaws.com";
	string request_parameters = 
"Action=DescribeInstances&Version=2013-10-15";


	auto hmac_sha256(ubyte[] key, ubyte[] msg)
	{
		auto hmac = hmac!SHA256(key);
		hmac.put(msg);
		auto digest = hmac.finish;

		return digest;
	}

	alias sign = hmac_sha256;

	auto getSignatureKey(string key, string dateStamp, string 
regionName, string serviceName)
	{
		ubyte[] kString = cast(ubyte[])("AWS4" ~ key);
		auto kDate = sign(kString, cast(ubyte[])dateStamp);
		auto kRegion = sign(kDate, cast(ubyte[])regionName);
		auto kService = sign(kRegion,  cast(ubyte[])serviceName);
		auto kSigning = sign(kService, cast(ubyte[])"aws4_request");

		return kSigning;
	}


	string canonicalURI = "/";
	string canonicalQueryString = request_parameters;
	string canonicalHeadersString =  "host:" ~ host ~ "\n" ~ 
"x-amz-date:" ~ xamztime ~ "\n";
	string signedHeaders = "host;x-amz-date";
	string payloadHash = sha256Of("").toHexString.toLower;
	string canonicalRequest = method ~ "\n" ~ canonicalURI ~ "\n" ~ 
canonicalQueryString ~ "\n" ~ canonicalHeadersString ~ "\n" ~ 
signedHeaders ~ "\n" ~ payloadHash;


	string algorithm = "AWS4-HMAC-SHA256";
	string credentialScope = curDateStr ~ "/" ~ region ~ "/" ~ 
service ~ "/" ~ "aws4_request";
	string stringToSign = algorithm ~ "\n" ~ xamztime ~ "\n" ~ 
credentialScope ~ "\n" ~ 
sha256Of(canonicalRequest).toHexString.toLower;

	auto signingKey = getSignatureKey(secretKey, curDateStr, region, 
service);

	string signature = hmac_sha256(signingKey, 
cast(ubyte[])stringToSign).toHexString.toLower;

	string authorizationHeader = algorithm ~ " " ~ "Credential=" ~ 
accessKey ~ "/" ~ credentialScope ~ ", " ~ "SignedHeaders=" ~ 
signedHeaders ~ ", " ~ "Signature=" ~ signature;

	auto client = HTTP(endpoint ~ "?" ~ canonicalQueryString);
	client.method = HTTP.Method.get;
	client.addRequestHeader("x-amz-date", xamztime);
	client.addRequestHeader("Authorization", authorizationHeader);
	auto content = client.perform();

	writeln(content);
}

Now will try to "pack" it to some class and make it more usable 
and universal. I think that thread can be closed/solved (if 
something like this exist here)

Thank you all for help.

//holo
Oct 11 2015
next sibling parent Laeeth Isharc <laeethnospam nospamlaeeth.com> writes:
On Sunday, 11 October 2015 at 23:16:51 UTC, holo wrote:
 After long fight with previous code i try to rewrite 
 "one-to-one" python example from 
 http://docs.aws.amazon.com/general/latest/gr/sigv4-signed-r
quest-examples.html (GET part) from begging to D with full success. Here is
working code in clear D. Im using hmac function which is available from 2.069
of phobos). It is not too beautiful but like i said im beginner, here it is:
Thanks for sharing holo - you saved me a bit of time, as I needed just the same thing for another api (dnsmadeeasy). Good luck with your own project. Laeeth.
Oct 11 2015
prev sibling parent reply Marc =?UTF-8?B?U2Now7x0eg==?= <schuetzm gmx.net> writes:
On Sunday, 11 October 2015 at 23:16:51 UTC, holo wrote:
 	auto hmac_sha256(ubyte[] key, ubyte[] msg)
 	{
 		auto hmac = hmac!SHA256(key);
 		hmac.put(msg);
 		auto digest = hmac.finish;

 		return digest;
 	}

 	alias sign = hmac_sha256;
std.digest.hmac already contains a helper function for this: hmac!SHA256(data, secret); The order of arguments is this way to allow chaining with UFCS: data.hmac!SHA256(secret); But it seems the documentation generator can't handle ddoc comments in templates, so unfortunately it doesn't appear in the documentation.
Oct 12 2015
parent holo <holosian gmail.com> writes:
Thank you for info, i changed my code to use that build in 
template and changed "cast(ubyte[]"  to "xxx.representation" and 
it is still working:

#!/usr/bin/rdmd -L-lcurl

module sigawsv4;

import std.stdio, std.process;
import std.digest.sha, std.digest.hmac;
import std.string;
import std.conv;
import std.datetime;
import std.net.curl;

void main()
{
	auto accessKey = environment["AWS_ACCESS_KEY"];
	auto secretKey = environment["AWS_SECRET_KEY"];

	auto currentClock = Clock.currTime(UTC());

	auto currentDate = cast(Date)currentClock;
	auto curDateStr = currentDate.toISOString;

	auto currentTime = cast(TimeOfDay)currentClock;
	auto curTimeStr = currentTime.toISOString;

	auto xamztime = curDateStr ~ "T" ~ curTimeStr ~ "Z";

	
	string method = "GET";
	string service = "ec2";
	string host = "ec2.amazonaws.com";
	string region = "us-east-1";
	string endpoint = "https://ec2.amazonaws.com";
	string request_parameters = 
"Action=DescribeInstances&Version=2013-10-15";


	alias sign = hmac!SHA256;

	auto getSignatureKey(string key, string dateStamp, string 
regionName, string serviceName)
	{
		auto kString = ("AWS4" ~ key).representation;
		auto kDate = sign(dateStamp.representation, kString);
		auto kRegion = sign(regionName.representation, kDate);
		auto kService = sign(serviceName.representation, kRegion);
		auto kSigning = sign("aws4_request".representation, kService);

		return kSigning;
	}


	string canonicalURI = "/";
	string canonicalQueryString = request_parameters;
	string canonicalHeadersString =  "host:" ~ host ~ "\n" ~ 
"x-amz-date:" ~ xamztime ~ "\n";
	string signedHeaders = "host;x-amz-date";
	string payloadHash = sha256Of("").toHexString.toLower;
	string canonicalRequest = method ~ "\n" ~ canonicalURI ~ "\n" ~ 
canonicalQueryString ~ "\n" ~ canonicalHeadersString ~ "\n" ~ 
signedHeaders ~ "\n" ~ payloadHash;


	string algorithm = "AWS4-HMAC-SHA256";
	string credentialScope = curDateStr ~ "/" ~ region ~ "/" ~ 
service ~ "/" ~ "aws4_request";
	string stringToSign = algorithm ~ "\n" ~ xamztime ~ "\n" ~ 
credentialScope ~ "\n" ~ 
sha256Of(canonicalRequest).toHexString.toLower;

	auto signingKey = getSignatureKey(secretKey, curDateStr, region, 
service);

	string signature = hmac!SHA256(stringToSign.representation, 
signingKey).toHexString.toLower;

	string authorizationHeader = algorithm ~ " " ~ "Credential=" ~ 
accessKey ~ "/" ~ credentialScope ~ ", " ~ "SignedHeaders=" ~ 
signedHeaders ~ ", " ~ "Signature=" ~ signature;

	auto client = HTTP(endpoint ~ "?" ~ canonicalQueryString);
	client.method = HTTP.Method.get;
	client.addRequestHeader("x-amz-date", xamztime);
	client.addRequestHeader("Authorization", authorizationHeader);
	auto content = client.perform();

	writeln(content);
}
Oct 12 2015
prev sibling next sibling parent yawniek <dlang srtnwz.com> writes:
On Saturday, 3 October 2015 at 03:11:06 UTC, holo wrote:
 Hello

 I'm trying to contact AWS API with D according to documentation:

 [...]
check https://github.com/yannick/vibe-aws it has v4 implemented
Oct 03 2015
prev sibling parent Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Saturday, 3 October 2015 at 03:11:06 UTC, holo wrote:
 Last but not least, how to write such function in D:

 def sign(key, msg):
     return hmac.new(key, msg.encode("utf-8"), 
 hashlib.sha256).digest()

 ?

 I can't find in standard libraryt hmac function, is it existing?
The next version of D will have a std.digest.hmac module: http://dlang.org/phobos-prerelease/std_digest_hmac.html
Oct 03 2015