digitalmars.D.learn - AWS API Dlang, hmac sha256 function.

holo (27/27) Oct 02 2015 Hello

Rikki Cattermole (4/30) Oct 02 2015 You could implement it yourself, (it looks pretty easy).

holo (27/30) Oct 02 2015 Thank you for answer. I used dub to fetch botan, after that i

Rikki Cattermole (7/38) Oct 02 2015 By the looks of that error message with paths, botan isn't actually

Rikki Cattermole (4/53) Oct 02 2015 Ohhhh wait nevermind.

holo (33/99) Oct 03 2015 Thank you for explaining how to use dub.

Rikki Cattermole (5/93) Oct 03 2015 By the looks of things the problem is with SHA256, I'm guessing it

holo (15/95) Oct 03 2015 I updated sha.d and now im getting following error:

Rikki Cattermole (3/87) Oct 03 2015 And one more file (ssse3.d) needs to be updated.

holo (8/43) Oct 03 2015 I downloaded whole master branch phobos and tried to use it

Rikki Cattermole (3/39) Oct 03 2015 Apologies, I didn't see this till after I got to bed. It's core.cpuid

holo (60/115) Oct 05 2015 Sorry i was busy at weekend. Thank you all for answers. I will

Vladimir Panteleev (3/5) Oct 05 2015 You can use Digger to get the latest version of D:

holo (26/31) Oct 05 2015 Thank you, now it is working as expected. Hope i wont miss

Rikki Cattermole (2/34) Oct 05 2015 Congrats on getting it working!

holo (274/275) Oct 06 2015 @Rikki Thanks :)

Rikki Cattermole (8/270) Oct 06 2015 If you see a template argument (which R is) it is typically meant for a

holo (106/106) Oct 08 2015 I was fighting with it a little bit and after all i just leave

Laeeth Isharc (3/47) Oct 08 2015 authorisation ??? (Or maybe authorization) ie a typo (check other

holo (23/23) Oct 09 2015 I correct typo (thats whats happening when you trying to solve

holo (39/39) Oct 09 2015 I doped HTTP packets with wireshark, maybe that will be useful:

Marc =?UTF-8?B?U2Now7x0eg==?= (4/14) Oct 09 2015 ... these ones, ...
holo (22/22) Oct 09 2015 OK i find out error, in addRequestHeader i was using ":" after

holo (77/99) Oct 10 2015 Maybe i will put my present code again:

holo (81/81) Oct 11 2015 After long fight with previous code i try to rewrite "one-to-one"

Laeeth Isharc (5/8) Oct 11 2015 Thanks for sharing holo - you saved me a bit of time, as I needed
Marc =?UTF-8?B?U2Now7x0eg==?= (8/16) Oct 12 2015 std.digest.hmac already contains a helper function for this:

holo (69/69) Oct 12 2015 Thank you for info, i changed my code to use that build in

yawniek (3/6) Oct 03 2015 check https://github.com/yannick/vibe-aws
Vladimir Panteleev (3/9) Oct 03 2015 The next version of D will have a std.digest.hmac module:

holo <holosian gmail.com> writes:

Hello

I'm trying to contact AWS API with D according to documentation:

http://docs.aws.amazon.com/general/latest/gr/sigv4_signing.html

there is written example in python which i want to rewrite to D:

http://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html

I want to ask about such 2 things:

Is it equivalent in D:

hexDigest!SHA256("sometexttohash");

for such python method:

hashlib.sha256('sometexttohash').hexdigest()

?

Maybe is there some prettier solution in D eg:

toHexString(sha256Of("sometexttohash"));

or

SHA256 sha;
sha.start();
string data = "sometexttohash";
sha.put(data);
toHexString(sha.finish())

If those are equivalents to python method which should i choose?

Last but not least, how to write such function in D:

def sign(key, msg):
     return hmac.new(key, msg.encode("utf-8"), 
hashlib.sha256).digest()

?

I can't find in standard libraryt hmac function, is it existing?

//holo

Oct 02 2015

Rikki Cattermole <alphaglosined gmail.com> writes:

On 03/10/15 4:11 PM, holo wrote:
 Hello

 I'm trying to contact AWS API with D according to documentation:

 http://docs.aws.amazon.com/general/latest/gr/sigv4_signing.html

 there is written example in python which i want to rewrite to D:

 http://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html


 I want to ask about such 2 things:

 Is it equivalent in D:

 hexDigest!SHA256("sometexttohash");

 for such python method:

 hashlib.sha256('sometexttohash').hexdigest()

 ?

 Maybe is there some prettier solution in D eg:

 toHexString(sha256Of("sometexttohash"));

 or

 SHA256 sha;
 sha.start();
 string data = "sometexttohash";
 sha.put(data);
 toHexString(sha.finish())

 If those are equivalents to python method which should i choose?

 Last but not least, how to write such function in D:

 def sign(key, msg):
      return hmac.new(key, msg.encode("utf-8"), hashlib.sha256).digest()

 ?

 I can't find in standard libraryt hmac function, is it existing?

 //holo

You could implement it yourself, (it looks pretty easy).
Or go the route of Botan: 
https://github.com/etcimon/botan/blob/3bdc835d5b9bad7523deaa86fe98b1fbb2f09d6b/source/botan/mac/hmac.d

Oct 02 2015

holo <holosian gmail.com> writes:

On Saturday, 3 October 2015 at 03:15:21 UTC, Rikki Cattermole 
wrote:
 You could implement it yourself, (it looks pretty easy).
 Or go the route of Botan: 
 https://github.com/etcimon/botan/blob/3bdc835d5b9bad7523deaa86fe98b1fbb2f09d6b/source/botan/mac/hmac.d


Thank you for answer. I used dub to fetch botan, after that i 
created project:

dub init projectname botan

and tried to add botan module:

$ cat app.d


import std.stdio;
import botan.mac.hmac;

void main()
{
	writeln("Edit source/app.d to start your project.");
}

but when im trying to run my simple app i get:

$ ./app.d
./app.d(4): Error: module hmac is in file 'botan/mac/hmac.d' 
which cannot be read
import path[0] = .
import path[1] = /usr/include/dlang/dmd
Failed: ["dmd", "-v", "-o-", "./app.d", "-I."]

What am i doing wrong? How to add library with dub package 
manager? Im really beginner in D (basically in programming) sorry 
if im asking obvious questions.

According to my other question are that functions equivalent to 
that python one? If they are not doing exact that same i could 
not create auth string which is needed to connect to AWS api.

Oct 02 2015

Rikki Cattermole <alphaglosined gmail.com> writes:

On 03/10/15 4:54 PM, holo wrote:
 On Saturday, 3 October 2015 at 03:15:21 UTC, Rikki Cattermole wrote:
 You could implement it yourself, (it looks pretty easy).
 Or go the route of Botan:
 https://github.com/etcimon/botan/blob/3bdc835d5b9bad7523deaa86fe98b1fbb2f09d6b/source/botan/mac/hmac.d


 Thank you for answer. I used dub to fetch botan, after that i created
 project:

 dub init projectname botan

 and tried to add botan module:

 $ cat app.d


 import std.stdio;
 import botan.mac.hmac;

 void main()
 {
      writeln("Edit source/app.d to start your project.");
 }

 but when im trying to run my simple app i get:

 $ ./app.d
 ./app.d(4): Error: module hmac is in file 'botan/mac/hmac.d' which
 cannot be read
 import path[0] = .
 import path[1] = /usr/include/dlang/dmd
 Failed: ["dmd", "-v", "-o-", "./app.d", "-I."]

 What am i doing wrong? How to add library with dub package manager? Im
 really beginner in D (basically in programming) sorry if im asking
 obvious questions.

By the looks of that error message with paths, botan isn't actually 
added as a dependency. Can you please paste the dub file?

 According to my other question are that functions equivalent to that
 python one? If they are not doing exact that same i could not create
 auth string which is needed to connect to AWS api.

I can't say for certain if it is not exact. The reason I did not answer 
it is because I am not familiar with environment or any of what you are 
doing.
I can unfortunately only point you in the right direction.

Oct 02 2015

Rikki Cattermole <alphaglosined gmail.com> writes:

On 03/10/15 6:01 PM, Rikki Cattermole wrote:
 On 03/10/15 4:54 PM, holo wrote:
 On Saturday, 3 October 2015 at 03:15:21 UTC, Rikki Cattermole wrote:
 You could implement it yourself, (it looks pretty easy).
 Or go the route of Botan:
 https://github.com/etcimon/botan/blob/3bdc835d5b9bad7523deaa86fe98b1fbb2f09d6b/source/botan/mac/hmac.d


 Thank you for answer. I used dub to fetch botan, after that i created
 project:

 dub init projectname botan

 and tried to add botan module:

 $ cat app.d


 import std.stdio;
 import botan.mac.hmac;

 void main()
 {
      writeln("Edit source/app.d to start your project.");
 }

 but when im trying to run my simple app i get:

 $ ./app.d
 ./app.d(4): Error: module hmac is in file 'botan/mac/hmac.d' which
 cannot be read
 import path[0] = .
 import path[1] = /usr/include/dlang/dmd
 Failed: ["dmd", "-v", "-o-", "./app.d", "-I."]

 What am i doing wrong? How to add library with dub package manager? Im
 really beginner in D (basically in programming) sorry if im asking
 obvious questions.

 By the looks of that error message with paths, botan isn't actually
 added as a dependency. Can you please paste the dub file?

Ohhhh wait nevermind.
Use dub to compile/run your program. You are using the shebang line!

$ dub run

 According to my other question are that functions equivalent to that
 python one? If they are not doing exact that same i could not create
 auth string which is needed to connect to AWS api.

 I can't say for certain if it is not exact. The reason I did not answer
 it is because I am not familiar with environment or any of what you are
 doing.
 I can unfortunately only point you in the right direction.

Oct 02 2015

holo <holosian gmail.com> writes:

On Saturday, 3 October 2015 at 05:02:58 UTC, Rikki Cattermole 
wrote:
 On 03/10/15 6:01 PM, Rikki Cattermole wrote:
 On 03/10/15 4:54 PM, holo wrote:
 On Saturday, 3 October 2015 at 03:15:21 UTC, Rikki Cattermole 
 wrote:
 You could implement it yourself, (it looks pretty easy).
 Or go the route of Botan:
 https://github.com/etcimon/botan/blob/3bdc835d5b9bad7523deaa86fe98b1fbb2f09d6b/source/botan/mac/hmac.d


 Thank you for answer. I used dub to fetch botan, after that i 
 created
 project:

 dub init projectname botan

 and tried to add botan module:

 $ cat app.d


 import std.stdio;
 import botan.mac.hmac;

 void main()
 {
      writeln("Edit source/app.d to start your project.");
 }

 but when im trying to run my simple app i get:

 $ ./app.d
 ./app.d(4): Error: module hmac is in file 'botan/mac/hmac.d' 
 which
 cannot be read
 import path[0] = .
 import path[1] = /usr/include/dlang/dmd
 Failed: ["dmd", "-v", "-o-", "./app.d", "-I."]

 What am i doing wrong? How to add library with dub package 
 manager? Im
 really beginner in D (basically in programming) sorry if im 
 asking
 obvious questions.

 By the looks of that error message with paths, botan isn't 
 actually
 added as a dependency. Can you please paste the dub file?

 Ohhhh wait nevermind.
 Use dub to compile/run your program. You are using the shebang 
 line!

 $ dub run

 According to my other question are that functions equivalent 
 to that
 python one? If they are not doing exact that same i could not 
 create
 auth string which is needed to connect to AWS api.

 I can't say for certain if it is not exact. The reason I did 
 not answer
 it is because I am not familiar with environment or any of 
 what you are
 doing.
 I can unfortunately only point you in the right direction.


Thank you for explaining how to use dub.

I find out in master branch of phobos on git hub is included 
hmac.d module which will be added in next release of phobos so i 
think that will be best solution for me to stick with standard 
library for beginning.

https://github.com/D-Programming-Language/phobos/tree/master/std

I downloaded hmac.d and new version of digest.d file and place 
them in my phobos library. I think it exactly contains what i 
need but when im trying to test it i have such error:

$ ./app.d
./app.d(9): Error: template std.digest.hmac.hmac cannot deduce 
function from argument types !(SHA!(512, 
256))(immutable(ubyte)[]), candidates are:
/usr/include/dlang/dmd/std/digest/hmac.d(202):        
std.digest.hmac.hmac(H) if (isDigest!H && hasBlockSize!H)
/usr/include/dlang/dmd/std/digest/hmac.d(208):        
std.digest.hmac.hmac(H, ulong blockSize)(scope const(ubyte)[] 
secret) if (isDigest!H)
Failed: ["dmd", "-v", "-o-", "./app.d", "-I."]



My test code (based on unint test from module) looks like that:



import std.stdio;
import std.digest.sha, std.digest.hmac;
import std.string : representation;

void main()
{
	auto hmac = hmac!SHA256("secretkey".representation);
	auto digest = hmac.put("texttohash".representation).finish;
	writeln(digest);
}

Am i using it correctly?

Oct 03 2015

Rikki Cattermole <alphaglosined gmail.com> writes:

On 04/10/15 1:09 AM, holo wrote:
 On Saturday, 3 October 2015 at 05:02:58 UTC, Rikki Cattermole wrote:
 On 03/10/15 6:01 PM, Rikki Cattermole wrote:
 On 03/10/15 4:54 PM, holo wrote:
 On Saturday, 3 October 2015 at 03:15:21 UTC, Rikki Cattermole wrote:
 You could implement it yourself, (it looks pretty easy).
 Or go the route of Botan:
 https://github.com/etcimon/botan/blob/3bdc835d5b9bad7523deaa86fe98b1fbb2f09d6b/source/botan/mac/hmac.d


 Thank you for answer. I used dub to fetch botan, after that i created
 project:

 dub init projectname botan

 and tried to add botan module:

 $ cat app.d


 import std.stdio;
 import botan.mac.hmac;

 void main()
 {
      writeln("Edit source/app.d to start your project.");
 }

 but when im trying to run my simple app i get:

 $ ./app.d
 ./app.d(4): Error: module hmac is in file 'botan/mac/hmac.d' which
 cannot be read
 import path[0] = .
 import path[1] = /usr/include/dlang/dmd
 Failed: ["dmd", "-v", "-o-", "./app.d", "-I."]

 What am i doing wrong? How to add library with dub package manager? Im
 really beginner in D (basically in programming) sorry if im asking
 obvious questions.

 By the looks of that error message with paths, botan isn't actually
 added as a dependency. Can you please paste the dub file?

 Ohhhh wait nevermind.
 Use dub to compile/run your program. You are using the shebang line!

 $ dub run

 According to my other question are that functions equivalent to that
 python one? If they are not doing exact that same i could not create
 auth string which is needed to connect to AWS api.

 I can't say for certain if it is not exact. The reason I did not answer
 it is because I am not familiar with environment or any of what you are
 doing.
 I can unfortunately only point you in the right direction.


 Thank you for explaining how to use dub.

 I find out in master branch of phobos on git hub is included hmac.d
 module which will be added in next release of phobos so i think that
 will be best solution for me to stick with standard library for beginning.

 https://github.com/D-Programming-Language/phobos/tree/master/std

 I downloaded hmac.d and new version of digest.d file and place them in
 my phobos library. I think it exactly contains what i need but when im
 trying to test it i have such error:

 $ ./app.d
 ./app.d(9): Error: template std.digest.hmac.hmac cannot deduce function
 from argument types !(SHA!(512, 256))(immutable(ubyte)[]), candidates are:
 /usr/include/dlang/dmd/std/digest/hmac.d(202): std.digest.hmac.hmac(H)
 if (isDigest!H && hasBlockSize!H)
 /usr/include/dlang/dmd/std/digest/hmac.d(208): std.digest.hmac.hmac(H,
 ulong blockSize)(scope const(ubyte)[] secret) if (isDigest!H)
 Failed: ["dmd", "-v", "-o-", "./app.d", "-I."]



 My test code (based on unint test from module) looks like that:



 import std.stdio;
 import std.digest.sha, std.digest.hmac;
 import std.string : representation;

 void main()
 {
      auto hmac = hmac!SHA256("secretkey".representation);
      auto digest = hmac.put("texttohash".representation).finish;
      writeln(digest);
 }

 Am i using it correctly?

By the looks of things the problem is with SHA256, I'm guessing it 
doesn't have its block size defined.
https://github.com/D-Programming-Language/phobos/blob/master/std/digest/digest.d#L394
You'll also need to update sha.d as well. Otherwise it looks good.

Oct 03 2015

holo <holosian gmail.com> writes:

On Saturday, 3 October 2015 at 12:22:11 UTC, Rikki Cattermole 
wrote:
 On 04/10/15 1:09 AM, holo wrote:
 On Saturday, 3 October 2015 at 05:02:58 UTC, Rikki Cattermole 
 wrote:
 On 03/10/15 6:01 PM, Rikki Cattermole wrote:
 On 03/10/15 4:54 PM, holo wrote:
 [...]

 By the looks of that error message with paths, botan isn't 
 actually
 added as a dependency. Can you please paste the dub file?

 Ohhhh wait nevermind.
 Use dub to compile/run your program. You are using the 
 shebang line!

 $ dub run

 [...]

 I can't say for certain if it is not exact. The reason I did 
 not answer
 it is because I am not familiar with environment or any of 
 what you are
 doing.
 I can unfortunately only point you in the right direction.


 Thank you for explaining how to use dub.

 I find out in master branch of phobos on git hub is included 
 hmac.d
 module which will be added in next release of phobos so i 
 think that
 will be best solution for me to stick with standard library 
 for beginning.

 https://github.com/D-Programming-Language/phobos/tree/master/std

 I downloaded hmac.d and new version of digest.d file and place 
 them in
 my phobos library. I think it exactly contains what i need but 
 when im
 trying to test it i have such error:

 $ ./app.d
 ./app.d(9): Error: template std.digest.hmac.hmac cannot deduce 
 function
 from argument types !(SHA!(512, 256))(immutable(ubyte)[]), 
 candidates are:
 /usr/include/dlang/dmd/std/digest/hmac.d(202): 
 std.digest.hmac.hmac(H)
 if (isDigest!H && hasBlockSize!H)
 /usr/include/dlang/dmd/std/digest/hmac.d(208): 
 std.digest.hmac.hmac(H,
 ulong blockSize)(scope const(ubyte)[] secret) if (isDigest!H)
 Failed: ["dmd", "-v", "-o-", "./app.d", "-I."]



 My test code (based on unint test from module) looks like that:



 import std.stdio;
 import std.digest.sha, std.digest.hmac;
 import std.string : representation;

 void main()
 {
      auto hmac = hmac!SHA256("secretkey".representation);
      auto digest = 
 hmac.put("texttohash".representation).finish;
      writeln(digest);
 }

 Am i using it correctly?

 By the looks of things the problem is with SHA256, I'm guessing 
 it doesn't have its block size defined.
 https://github.com/D-Programming-Language/phobos/blob/master/std/digest/digest.d#L394
 You'll also need to update sha.d as well. Otherwise it looks 
 good.

I updated sha.d and now im getting following error:

$ ./app.d
/usr/include/dlang/dmd/std/digest/sha.d(225): Error: pure 
function 'std.digest.sha.SHA!(512u, 160u).SHA.transform' cannot 
call impure function 'core.cpuid.ssse3'
/usr/include/dlang/dmd/std/digest/digest.d(285): Error: template 
instance std.range.primitives.isOutputRange!(SHA!(512u, 160u), 
const(ubyte)[]) error instantiating
/usr/include/dlang/dmd/std/digest/digest.d(851):        
instantiated from here: isDigest!(SHA!(512u, 160u))
/usr/include/dlang/dmd/std/digest/sha.d(1179):        while 
looking for match for WrapperDigest!(SHA!(512u, 160u))
Failed: ["dmd", "-v", "-o-", "./app.d", "-I."]

Oct 03 2015

Rikki Cattermole <alphaglosined gmail.com> writes:

On 04/10/15 1:49 AM, holo wrote:
 On Saturday, 3 October 2015 at 12:22:11 UTC, Rikki Cattermole wrote:
 On 04/10/15 1:09 AM, holo wrote:
 On Saturday, 3 October 2015 at 05:02:58 UTC, Rikki Cattermole wrote:
 On 03/10/15 6:01 PM, Rikki Cattermole wrote:
 On 03/10/15 4:54 PM, holo wrote:
 [...]

 By the looks of that error message with paths, botan isn't actually
 added as a dependency. Can you please paste the dub file?

 Ohhhh wait nevermind.
 Use dub to compile/run your program. You are using the shebang line!

 $ dub run

 [...]

 I can't say for certain if it is not exact. The reason I did not
 answer
 it is because I am not familiar with environment or any of what you
 are
 doing.
 I can unfortunately only point you in the right direction.


 Thank you for explaining how to use dub.

 I find out in master branch of phobos on git hub is included hmac.d
 module which will be added in next release of phobos so i think that
 will be best solution for me to stick with standard library for
 beginning.

 https://github.com/D-Programming-Language/phobos/tree/master/std

 I downloaded hmac.d and new version of digest.d file and place them in
 my phobos library. I think it exactly contains what i need but when im
 trying to test it i have such error:

 $ ./app.d
 ./app.d(9): Error: template std.digest.hmac.hmac cannot deduce function
 from argument types !(SHA!(512, 256))(immutable(ubyte)[]), candidates
 are:
 /usr/include/dlang/dmd/std/digest/hmac.d(202): std.digest.hmac.hmac(H)
 if (isDigest!H && hasBlockSize!H)
 /usr/include/dlang/dmd/std/digest/hmac.d(208): std.digest.hmac.hmac(H,
 ulong blockSize)(scope const(ubyte)[] secret) if (isDigest!H)
 Failed: ["dmd", "-v", "-o-", "./app.d", "-I."]



 My test code (based on unint test from module) looks like that:



 import std.stdio;
 import std.digest.sha, std.digest.hmac;
 import std.string : representation;

 void main()
 {
      auto hmac = hmac!SHA256("secretkey".representation);
      auto digest = hmac.put("texttohash".representation).finish;
      writeln(digest);
 }

 Am i using it correctly?

 By the looks of things the problem is with SHA256, I'm guessing it
 doesn't have its block size defined.
 https://github.com/D-Programming-Language/phobos/blob/master/std/digest/digest.d#L394

 You'll also need to update sha.d as well. Otherwise it looks good.

 I updated sha.d and now im getting following error:

 $ ./app.d
 /usr/include/dlang/dmd/std/digest/sha.d(225): Error: pure function
 'std.digest.sha.SHA!(512u, 160u).SHA.transform' cannot call impure
 function 'core.cpuid.ssse3'
 /usr/include/dlang/dmd/std/digest/digest.d(285): Error: template
 instance std.range.primitives.isOutputRange!(SHA!(512u, 160u),
 const(ubyte)[]) error instantiating
 /usr/include/dlang/dmd/std/digest/digest.d(851): instantiated from here:
 isDigest!(SHA!(512u, 160u))
 /usr/include/dlang/dmd/std/digest/sha.d(1179):        while looking for
 match for WrapperDigest!(SHA!(512u, 160u))
 Failed: ["dmd", "-v", "-o-", "./app.d", "-I."]

And one more file (ssse3.d) needs to be updated.
You are going to be playing whack a mole I think for a while.

Oct 03 2015

holo <holosian gmail.com> writes:

On Saturday, 3 October 2015 at 12:50:58 UTC, Rikki Cattermole 
wrote:
 On 04/10/15 1:49 AM, holo wrote:
 On Saturday, 3 October 2015 at 12:22:11 UTC, Rikki Cattermole 
 wrote:
 On 04/10/15 1:09 AM, holo wrote:
 [...]

 By the looks of things the problem is with SHA256, I'm 
 guessing it
 doesn't have its block size defined.
 https://github.com/D-Programming-Language/phobos/blob/master/std/digest/digest.d#L394

 You'll also need to update sha.d as well. Otherwise it looks 
 good.

 I updated sha.d and now im getting following error:

 $ ./app.d
 /usr/include/dlang/dmd/std/digest/sha.d(225): Error: pure 
 function
 'std.digest.sha.SHA!(512u, 160u).SHA.transform' cannot call 
 impure
 function 'core.cpuid.ssse3'
 /usr/include/dlang/dmd/std/digest/digest.d(285): Error: 
 template
 instance std.range.primitives.isOutputRange!(SHA!(512u, 160u),
 const(ubyte)[]) error instantiating
 /usr/include/dlang/dmd/std/digest/digest.d(851): instantiated 
 from here:
 isDigest!(SHA!(512u, 160u))
 /usr/include/dlang/dmd/std/digest/sha.d(1179):        while 
 looking for
 match for WrapperDigest!(SHA!(512u, 160u))
 Failed: ["dmd", "-v", "-o-", "./app.d", "-I."]

 And one more file (ssse3.d) needs to be updated.
 You are going to be playing whack a mole I think for a while.

I downloaded whole master branch phobos and tried to use it 
(changed path in dmd.conf), but there are missing much more files 
(i think that master branch is not ready yet). So i get back to 
my original library and tried to updated file you mention. But i 
can't find it (ssse3.d), its not appearing in my stable lib and 
even in that master-branch from zip file.

Oct 03 2015

Rikki Cattermole <alphaglosined gmail.com> writes:

On 04/10/15 2:31 AM, holo wrote:
 On Saturday, 3 October 2015 at 12:50:58 UTC, Rikki Cattermole wrote:
 On 04/10/15 1:49 AM, holo wrote:
 On Saturday, 3 October 2015 at 12:22:11 UTC, Rikki Cattermole wrote:
 On 04/10/15 1:09 AM, holo wrote:
 [...]

 By the looks of things the problem is with SHA256, I'm guessing it
 doesn't have its block size defined.
 https://github.com/D-Programming-Language/phobos/blob/master/std/digest/digest.d#L394


 You'll also need to update sha.d as well. Otherwise it looks good.

 I updated sha.d and now im getting following error:

 $ ./app.d
 /usr/include/dlang/dmd/std/digest/sha.d(225): Error: pure function
 'std.digest.sha.SHA!(512u, 160u).SHA.transform' cannot call impure
 function 'core.cpuid.ssse3'
 /usr/include/dlang/dmd/std/digest/digest.d(285): Error: template
 instance std.range.primitives.isOutputRange!(SHA!(512u, 160u),
 const(ubyte)[]) error instantiating
 /usr/include/dlang/dmd/std/digest/digest.d(851): instantiated from here:
 isDigest!(SHA!(512u, 160u))
 /usr/include/dlang/dmd/std/digest/sha.d(1179):        while looking for
 match for WrapperDigest!(SHA!(512u, 160u))
 Failed: ["dmd", "-v", "-o-", "./app.d", "-I."]

 And one more file (ssse3.d) needs to be updated.
 You are going to be playing whack a mole I think for a while.

 I downloaded whole master branch phobos and tried to use it (changed
 path in dmd.conf), but there are missing much more files (i think that
 master branch is not ready yet). So i get back to my original library
 and tried to updated file you mention. But i can't find it (ssse3.d),
 its not appearing in my stable lib and even in that master-branch from
 zip file.

Apologies, I didn't see this till after I got to bed. It's core.cpuid 
not ssse3.d. It will be in druntime, not Phobos.

Oct 03 2015

holo <holosian gmail.com> writes:

On Saturday, 3 October 2015 at 23:58:39 UTC, Rikki Cattermole 
wrote:
 On 04/10/15 2:31 AM, holo wrote:
 On Saturday, 3 October 2015 at 12:50:58 UTC, Rikki Cattermole 
 wrote:
 On 04/10/15 1:49 AM, holo wrote:
 On Saturday, 3 October 2015 at 12:22:11 UTC, Rikki 
 Cattermole wrote:
 On 04/10/15 1:09 AM, holo wrote:
 [...]

 By the looks of things the problem is with SHA256, I'm 
 guessing it
 doesn't have its block size defined.
 https://github.com/D-Programming-Language/phobos/blob/master/std/digest/digest.d#L394


 You'll also need to update sha.d as well. Otherwise it 
 looks good.

 I updated sha.d and now im getting following error:

 $ ./app.d
 /usr/include/dlang/dmd/std/digest/sha.d(225): Error: pure 
 function
 'std.digest.sha.SHA!(512u, 160u).SHA.transform' cannot call 
 impure
 function 'core.cpuid.ssse3'
 /usr/include/dlang/dmd/std/digest/digest.d(285): Error: 
 template
 instance std.range.primitives.isOutputRange!(SHA!(512u, 
 160u),
 const(ubyte)[]) error instantiating
 /usr/include/dlang/dmd/std/digest/digest.d(851): 
 instantiated from here:
 isDigest!(SHA!(512u, 160u))
 /usr/include/dlang/dmd/std/digest/sha.d(1179):        while 
 looking for
 match for WrapperDigest!(SHA!(512u, 160u))
 Failed: ["dmd", "-v", "-o-", "./app.d", "-I."]

 And one more file (ssse3.d) needs to be updated.
 You are going to be playing whack a mole I think for a while.

 I downloaded whole master branch phobos and tried to use it 
 (changed
 path in dmd.conf), but there are missing much more files (i 
 think that
 master branch is not ready yet). So i get back to my original 
 library
 and tried to updated file you mention. But i can't find it 
 (ssse3.d),
 its not appearing in my stable lib and even in that 
 master-branch from
 zip file.

 Apologies, I didn't see this till after I got to bed. It's 
 core.cpuid not ssse3.d. It will be in druntime, not Phobos.

Sorry i was busy at weekend. Thank you all for answers. I will 
try to write some solution basing on yawniek example and will 
back if some troubles appears.

 Vladimir where can i check or do you know when next version of 
phobos will be available?

 Rikki i downloaded both master zip files for runtime and phobos 
librarys all dependencies looks are satisfied right now but still 
getting errors:

[holo ultraxps workplace]$ dmd hmac.d
hmac.o: In function 
`_D3std6digest4hmac58__T4hmacTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4hmacFNaNbNiNfMAxhZS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC':
hmac.d:(.text._D3std6digest4hmac58__T4hmacTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4hmacFNaNbNiNfMAxhZS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SH
Vmi512Z4HMAC+0x18): undefined reference to
`_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA6__initZ'
hmac.o: In function 
`_D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC6__ctorMFNaNbNcNiNfMAxhZS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC':
hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC6__ctorMFNaNbNcNiNfMAxhZS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SH
Vmi512Z4HMAC+0x1b): undefined reference to
`_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA5startMFNaNbNiNfZv'
hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC6__ctorMFNaNbNcNiNfMAxhZS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SH
Vmi512Z4HMAC+0x31): undefined reference to
`_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA3putMFNaNbNiNeMAxhXv'
hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC6__ctorMFNaNbNcNiNfMAxhZS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SH
Vmi512Z4HMAC+0x3e): undefined reference to
`_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA6finishMFNaNbNiNeZG20h'
hmac.o: In function 
`_D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC5startMFNaNbNcNiNjNfZS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC':
hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC5startMFNaNbNcNiNjNfZS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SH
Vmi512Z4HMAC+0x8d): undefined reference to
`_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA5startMFNaNbNiNfZv'
hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC5startMFNaNbNcNiNjNfZS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SH
Vmi512Z4HMAC+0xa4): undefined reference to
`_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA3putMFNaNbNiNeMAxhXv'
hmac.o: In function 
`_D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC3putMFNaNbNcNiNjNfxAhXS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC':
hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC3putMFNaNbNcNiNjNfxAhXS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SH
Vmi512Z4HMAC+0x5f): undefined reference to
`_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA3putMFNaNbNiNeMAxhXv'
hmac.o: In function 
`_D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC6finishMFNaNbNiNfZG20h':
hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC6finishMF
aNbNiNfZG20h+0xa1): undefined reference to
`_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA6finishMFNaNbNiNeZG20h'
hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC6finishMF
aNbNiNfZG20h+0xaa): undefined reference to
`_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA5startMFNaNbNiNfZv'
hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC6finishMF
aNbNiNfZG20h+0xc4): undefined reference to
`_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA3putMFNaNbNiNeMAxhXv'
hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC6finishMF
aNbNiNfZG20h+0xdb): undefined reference to
`_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA3putMFNaNbNiNeMAxhXv'
hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC6finishMF
aNbNiNfZG20h+0xe8): undefined reference to
`_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA6finishMFNaNbNiNeZG20h'
hmac.o: In function 
`_D3std6format61__T11formatValueTS3std5array17__T8AppenderTAyaZ8AppenderThTaZ11formatValueFNaNfS3std5array17__T8AppenderTAyaZ8AppenderhKS3std6format18__T10FormatSpecTaZ10FormatSpecZv':
hmac.d:(.text._D3std6format61__T11formatValueTS3std5array17__T8AppenderTAyaZ8AppenderThTaZ11formatValueFNaNfS3std5array17__T8AppenderTAyaZ8AppenderhKS3std6format18__T10FormatSpecTaZ10
ormatSpecZv+0x16a): undefined reference to
`_D3std6format64__T14formatIntegralTS3std5array17__T8AppenderTAyaZ8AppenderTmTaZ14formatIntegralFNaNbNfS3std5array17__T8AppenderTAyaZ8AppenderxmKxS3std6format18__T10FormatSpecTaZ10FormatSpeckmZv'
hmac.o: In function 
`_D3std6format62__T11formatValueTS3std5array17__T8AppenderTAyaZ8AppenderTxhTaZ11formatValueFNaNfS3std5array17__T8AppenderTAyaZ8AppenderxhKS3std6format18__T10FormatSpecTaZ10FormatSpecZv':
hmac.d:(.text._D3std6format62__T11formatValueTS3std5array17__T8AppenderTAyaZ8AppenderTxhTaZ11formatValueFNaNfS3std5array17__T8AppenderTAyaZ8AppenderxhKS3std6format18__T10FormatSpecTaZ10
ormatSpecZv+0x16a): undefined reference to
`_D3std6format64__T14formatIntegralTS3std5array17__T8AppenderTAyaZ8AppenderTmTaZ14formatIntegralFNaNbNfS3std5array17__T8AppenderTAyaZ8AppenderxmKxS3std6format18__T10FormatSpecTaZ10FormatSpeckmZv'
hmac.o: In function 
`_D3std6format64__T14formatIntegralTS3std5array17__T8AppenderTAyaZ8AppenderTlTaZ14formatIntegralFNaNbNfS3std5array17__T8AppenderTAyaZ8AppenderxlKxS3std6format18__T10FormatSpecTaZ10FormatSpeckmZv':
hmac.d:(.text._D3std6format64__T14formatIntegralTS3std5array17__T8AppenderTAyaZ8AppenderTlTaZ14formatIntegralFNaNbNfS3std5array17__T8AppenderTAyaZ8AppenderxlKxS3std6format18__T10FormatSpecTaZ10F
rmatSpeckmZv+0x46): undefined reference to
`_D3std6format64__T14formatUnsignedTS3std5array17__T8AppenderTAyaZ8AppenderTmTaZ14formatUnsignedFNaNbNfS3std5array17__T8AppenderTAyaZ8AppendermKxS3std6format18__T10FormatSpecTaZ10FormatSpeckbZv'
hmac.o: In function 
`_D3std6format62__T11formatValueTS3std5array17__T8AppenderTAyaZ8AppenderTxmTaZ11formatValueFNaNfS3std5array17__T8AppenderTAyaZ8AppenderxmKS3std6format18__T10FormatSpecTaZ10FormatSpecZv':
hmac.d:(.text._D3std6format62__T11formatValueTS3std5array17__T8AppenderTAyaZ8AppenderTxmTaZ11formatValueFNaNfS3std5array17__T8AppenderTAyaZ8AppenderxmKS3std6format18__T10FormatSpecTaZ10
ormatSpecZv+0x16a): undefined reference to
`_D3std6format64__T14formatIntegralTS3std5array17__T8AppenderTAyaZ8AppenderTmTaZ14formatIntegralFNaNbNfS3std5array17__T8AppenderTAyaZ8AppenderxmKxS3std6format18__T10FormatSpecTaZ10FormatSpeckmZv'
collect2: error: ld returned 1 exit status
--- errorlevel 1
[holo ultraxps workplace]$ cat hmac.d


import std.stdio;
import std.digest.sha, std.digest.hmac;
import std.string : representation;

void main()
{
	auto hmac = hmac!SHA1("secretkey".representation);
	hmac.put("texttohash".representation);
	auto digest = hmac.finish();

	writeln(digest);
}

[holo ultraxps workplace]$

Oct 05 2015

Vladimir Panteleev <thecybershadow.lists gmail.com> writes:

On Monday, 5 October 2015 at 19:43:39 UTC, holo wrote:
  Vladimir where can i check or do you know when next version of 
 phobos will be available?

You can use Digger to get the latest version of D:

https://github.com/CyberShadow/Digger

Oct 05 2015

holo <holosian gmail.com> writes:

On Monday, 5 October 2015 at 21:00:38 UTC, Vladimir Panteleev 
wrote:
 On Monday, 5 October 2015 at 19:43:39 UTC, holo wrote:
  Vladimir where can i check or do you know when next version 
 of phobos will be available?

 You can use Digger to get the latest version of D:

 https://github.com/CyberShadow/Digger

Thank you, now it is working as expected. Hope i wont miss 
upgrade in my system to uninstall digger version before.

[holo ultraxps workplace]$ cat ./hmac.d


import std.stdio;
import std.digest.sha, std.digest.hmac;
import std.string : representation;

void main()
{
	auto hmac = hmac!SHA256("secretkey".representation);
	hmac.put("texttohash".representation);
	auto digest = hmac.finish();

	writeln(digest);
}

[holo ultraxps workplace]$ ./hmac.d
[28, 169, 142, 131, 133, 104, 149, 47, 205, 215, 20, 154, 170, 
148, 84, 170, 252, 36, 10, 119, 18, 25, 10, 145, 183, 133, 135, 
252, 26, 124, 215, 6]
[holo ultraxps workplace]$ dmd --version
DMD64 D Compiler v2.069-devel-5e73c30
Copyright (c) 1999-2015 by Digital Mars written by Walter Bright
[holo ultraxps workplace]$

Now need try to write some code which will connect me to AWS api 
based on examples.

Oct 05 2015

Rikki Cattermole <alphaglosined gmail.com> writes:

On 06/10/15 11:26 AM, holo wrote:
 On Monday, 5 October 2015 at 21:00:38 UTC, Vladimir Panteleev wrote:
 On Monday, 5 October 2015 at 19:43:39 UTC, holo wrote:
  Vladimir where can i check or do you know when next version of
 phobos will be available?

 You can use Digger to get the latest version of D:

 https://github.com/CyberShadow/Digger

 Thank you, now it is working as expected. Hope i wont miss upgrade in my
 system to uninstall digger version before.

 [holo ultraxps workplace]$ cat ./hmac.d


 import std.stdio;
 import std.digest.sha, std.digest.hmac;
 import std.string : representation;

 void main()
 {
      auto hmac = hmac!SHA256("secretkey".representation);
      hmac.put("texttohash".representation);
      auto digest = hmac.finish();

      writeln(digest);
 }

 [holo ultraxps workplace]$ ./hmac.d
 [28, 169, 142, 131, 133, 104, 149, 47, 205, 215, 20, 154, 170, 148, 84,
 170, 252, 36, 10, 119, 18, 25, 10, 145, 183, 133, 135, 252, 26, 124,
 215, 6]
 [holo ultraxps workplace]$ dmd --version
 DMD64 D Compiler v2.069-devel-5e73c30
 Copyright (c) 1999-2015 by Digital Mars written by Walter Bright
 [holo ultraxps workplace]$

 Now need try to write some code which will connect me to AWS api based
 on examples.

Congrats on getting it working!

Oct 05 2015

holo <holosian gmail.com> writes:

 Congrats on getting it working!

 Rikki Thanks :)

I was trying to write my own lib from beginning based on examples 
but after some time i resign from that idea (will back to it when 
i will have some more experience) and right now im trying to 
customize that one from link which yawniek paste:

https://github.com/yannick/vibe-aws/blob/master/source/vibe/aws/sigv4.d

I removed import from vibe.d library and copy/paste missed 
functions formEncode and filterURLEncode (BTW: what that "(R)" 
mean in it? filterURLEncode(R)(ref R dst, ..., ..) ).

Next thing what i did was to replace hmac function to use hmac 
module from newest library. Now whole code looks like this:

module sigv4;

import std.array;
import std.algorithm;
import std.digest.sha;
import std.range;
import std.stdio;
import std.string;
import std.format;
import std.digest.hmac;


const algorithm = "AWS4-HMAC-SHA256";


void filterURLEncode(R)(ref R dst, string str, string 
allowed_chars = null, bool form_encoding = false)
{
	while( str.length > 0 ) {
		switch(str[0]) {
			case ' ':
				if (form_encoding) {
					dst.put('+');
					break;
				}
				goto default;
			case 'A': .. case 'Z':
			case 'a': .. case 'z':
			case '0': .. case '9':
			case '-': case '_': case '.': case '~':
				dst.put(str[0]);
				break;
			default:
				if (allowed_chars.canFind(str[0])) dst.put(str[0]);
				else formattedWrite(dst, "%%%02X", str[0]);
		}
		str = str[1 .. $];
	}
}

string formEncode(string str, string allowed_chars = null)
 safe {
	auto dst = appender!string();
	dst.reserve(str.length);
	filterURLEncode(dst, str, allowed_chars, true);
	return dst.data;
}

struct CanonicalRequest
{
     string method;
     string uri;
     string[string] queryParameters;
     string[string] headers;
     ubyte[] payload;
}

string canonicalQueryString(string[string] queryParameters)
{
     alias encode = formEncode;

     string[string] encoded;
     foreach (p; queryParameters.keys())
     {
         encoded[encode(p)] = encode(queryParameters[p]);
     }
     string[] keys = encoded.keys();
     sort(keys);
     return keys.map!(k => k ~ "=" ~ encoded[k]).join("&");
}

string canonicalHeaders(string[string] headers)
{
     string[string] trimmed;
     foreach (h; headers.keys())
     {
         trimmed[h.toLower().strip()] = headers[h].strip();
     }
     string[] keys = trimmed.keys();
     sort(keys);
     return keys.map!(k => k ~ ":" ~ trimmed[k] ~ "\n").join("");
}

string signedHeaders(string[string] headers)
{
     string[] keys = headers.keys().map!(k => k.toLower()).array();
     sort(keys);
     return keys.join(";");
}

string hash(T)(T payload)
{
     auto hash = sha256Of(payload);
     return hash.toHexString().toLower();
}

string makeCRSigV4(CanonicalRequest r)
{
     auto cr =
         r.method.toUpper() ~ "\n" ~
         (r.uri.empty ? "/" : r.uri) ~ "\n" ~
         canonicalQueryString(r.queryParameters) ~ "\n" ~
         canonicalHeaders(r.headers) ~ "\n" ~
         signedHeaders(r.headers) ~ "\n" ~
         hash(r.payload);

     return hash(cr);
}

unittest {
     string[string] empty;

     auto r = CanonicalRequest(
             "POST",
             "/",
             empty,
             ["content-type": "application/x-www-form-urlencoded; 
charset=utf-8",
              "host": "iam.amazonaws.com",
              "x-amz-date": "20110909T233600Z"],
             cast(ubyte[])"Action=ListUsers&Version=2010-05-08");

     auto sig = makeCRSigV4(r);

     assert(sig == 
"3511de7e95d28ecd39e9513b642aee07e54f4941150d8df8bf94b328ef7e55e2");
}

struct SignableRequest
{
     string dateString;
     string timeStringUTC;
     string region;
     string service;
     CanonicalRequest canonicalRequest;
}

string signableString(SignableRequest r) {
     return algorithm ~ "\n" ~
         r.dateString ~ "T" ~ r.timeStringUTC ~ "Z\n" ~
         r.dateString ~ "/" ~ r.region ~ "/" ~ r.service ~ 
"/aws4_request\n" ~
         makeCRSigV4(r.canonicalRequest);
}

unittest {
     string[string] empty;

     SignableRequest r;
     r.dateString = "20110909";
     r.timeStringUTC = "233600";
     r.region = "us-east-1";
     r.service = "iam";
     r.canonicalRequest = CanonicalRequest(
             "POST",
             "/",
             empty,
             ["content-type": "application/x-www-form-urlencoded; 
charset=utf-8",
              "host": "iam.amazonaws.com",
              "x-amz-date": "20110909T233600Z"],
             cast(ubyte[])"Action=ListUsers&Version=2010-05-08");

     auto sampleString =
         algorithm ~ "\n" ~
         "20110909T233600Z\n" ~
         "20110909/us-east-1/iam/aws4_request\n" ~
         
"3511de7e95d28ecd39e9513b642aee07e54f4941150d8df8bf94b328ef7e55e2";

     assert(sampleString == signableString(r));
}

ubyte[] array_xor(ubyte[] b1, ubyte[] b2)
{
     assert(b1.length == b2.length);
     ubyte[] ret;
     for (uint i = 0; i < b1.length; i++)
         ret ~= b1[i] ^ b2[i];
     return ret;
}

auto hmac_sha256(string key, string message)
{
     auto hmac = hmac!SHA256(key.representation);
     hmac.put(message.representation);
     return hmac.finish;
}

unittest {
     string key = "key";
     string message = "The quick brown fox jumps over the lazy 
dog";

     string mac = hmac_sha256(key, 
message).toHexString().toLower();
     assert(mac == 
"f7bc83f430538424b13298e6aa6fb143ef4d59a14946175997479dbc2d1a3cd8");
}

auto signingKey(string secret, string dateString, string region, 
string service)
{
     auto kSecret = "AWS4" ~ secret;
     auto kDate = hmac_sha256(kSecret, dateString);
     auto kRegion = hmac_sha256(kDate, region);
     auto kService = hmac_sha256(kRegion, service);
     return hmac_sha256(kService, "aws4_request");
}

unittest {
     string secretKey = "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY";
     auto signKey = signingKey(secretKey, "20110909", "us-east-1", 
"iam");

     ubyte[] expected = [152, 241, 216, 137, 254, 196, 244, 66, 
26, 220, 82, 43, 171, 12, 225, 248, 46, 105, 41, 194, 98, 237, 
21, 229, 169, 76, 144, 239, 209, 227, 176, 231 ];
     assert(expected == signKey);
}

alias sign = hmac_sha256;

unittest {
     auto sampleString =
         "AWS4-HMAC-SHA256\n" ~
         "20110909T233600Z\n" ~
         "20110909/us-east-1/iam/aws4_request\n" ~
         
"3511de7e95d28ecd39e9513b642aee07e54f4941150d8df8bf94b328ef7e55e2";

     auto secretKey = "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY";
     auto signKey = signingKey(secretKey, "20110909", "us-east-1", 
"iam");

     auto signature = sign(signKey, 
sampleString).toHexString().toLower();
     auto expected = 
"ced6826de92d2bdeed8f846f0bf508e8559e98e4b0199114b84c54174deb456c";

     assert(signature == expected);
}

/**
  * CredentialScope == date / region / service / aws4_request
  */
string createSignatureHeader(string accessKeyID, string 
credentialScope, string[string] reqHeaders, ubyte[] signature)
{
     return algorithm ~ " Credential=" ~ accessKeyID ~ "/" ~ 
credentialScope ~ "/aws4_request, SignedHeaders=" ~ 
signedHeaders(reqHeaders) ~ ", Signature=" ~ 
signature.toHexString().toLower();
}

string dateFromISOString(string iso)
{
     auto i = iso.indexOf('T');
     if (i == -1) throw new Exception("ISO time in wrong format: " 
~ iso);
     return iso[0..i];
}

string timeFromISOString(string iso)
{
     auto t = iso.indexOf('T');
     auto z = iso.indexOf('Z');
     if (t == -1 || z == -1) throw new Exception("ISO time in 
wrong format: " ~ iso);
     return iso[t+1..z];
}

unittest {
     assert(dateFromISOString("20110909T1203Z") == "20110909");
}


void main()
{
     auto sampleString =
         "AWS4-HMAC-SHA256\n" ~
         "20110909T233600Z\n" ~
         "20110909/us-east-1/iam/aws4_request\n" ~
         
"3511de7e95d28ecd39e9513b642aee07e54f4941150d8df8bf94b328ef7e55e2";

     auto secretKey = "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY";
     auto signKey = signingKey(secretKey, "20110909", "us-east-1", 
"iam");

     auto signature = sign(signKey, 
sampleString).toHexString().toLower();

     writeln(signature);
}

When i try to compile it im getting such error:

[holo ultraxps test]$ dmd -unittest hello.d
hello.d(196): Error: function sigv4.hmac_sha256 (string key, 
string message) is not callable using argument types (ubyte[32], 
string)
[holo ultraxps test]$ dmd hello.d
hello.d(196): Error: function sigv4.hmac_sha256 (string key, 
string message) is not callable using argument types (ubyte[32], 
string)
[holo ultraxps test]$


Line 196 is: "auto kRegion = hmac_sha256(kDate, region);"

I was looking for ubyte[32] variable but i cant find it anywhere. 
What is strange unit tests are passing (i think, maybe im wrong).

Oct 06 2015

Rikki Cattermole <alphaglosined gmail.com> writes:

On 07/10/15 3:18 PM, holo wrote:
 Congrats on getting it working!

  Rikki Thanks :)

 I was trying to write my own lib from beginning based on examples but
 after some time i resign from that idea (will back to it when i will
 have some more experience) and right now im trying to customize that one
 from link which yawniek paste:

 https://github.com/yannick/vibe-aws/blob/master/source/vibe/aws/sigv4.d

 I removed import from vibe.d library and copy/paste missed functions
 formEncode and filterURLEncode (BTW: what that "(R)" mean in it?
 filterURLEncode(R)(ref R dst, ..., ..) ).

If you see a template argument (which R is) it is typically meant for a 
range. In this case an output range. Although there should be 
isOutputRange!R in an template if condition, to check this.

 Next thing what i did was to replace hmac function to use hmac module
 from newest library. Now whole code looks like this:

 module sigv4;

 import std.array;
 import std.algorithm;
 import std.digest.sha;
 import std.range;
 import std.stdio;
 import std.string;
 import std.format;
 import std.digest.hmac;


 const algorithm = "AWS4-HMAC-SHA256";


 void filterURLEncode(R)(ref R dst, string str, string allowed_chars =
 null, bool form_encoding = false)
 {
      while( str.length > 0 ) {
          switch(str[0]) {
              case ' ':
                  if (form_encoding) {
                      dst.put('+');
                      break;
                  }
                  goto default;
              case 'A': .. case 'Z':
              case 'a': .. case 'z':
              case '0': .. case '9':
              case '-': case '_': case '.': case '~':
                  dst.put(str[0]);
                  break;
              default:
                  if (allowed_chars.canFind(str[0])) dst.put(str[0]);
                  else formattedWrite(dst, "%%%02X", str[0]);
          }
          str = str[1 .. $];
      }
 }

 string formEncode(string str, string allowed_chars = null)
  safe {
      auto dst = appender!string();
      dst.reserve(str.length);
      filterURLEncode(dst, str, allowed_chars, true);
      return dst.data;
 }

 struct CanonicalRequest
 {
      string method;
      string uri;
      string[string] queryParameters;
      string[string] headers;
      ubyte[] payload;
 }

 string canonicalQueryString(string[string] queryParameters)
 {
      alias encode = formEncode;

      string[string] encoded;
      foreach (p; queryParameters.keys())
      {
          encoded[encode(p)] = encode(queryParameters[p]);
      }
      string[] keys = encoded.keys();
      sort(keys);
      return keys.map!(k => k ~ "=" ~ encoded[k]).join("&");
 }

 string canonicalHeaders(string[string] headers)
 {
      string[string] trimmed;
      foreach (h; headers.keys())
      {
          trimmed[h.toLower().strip()] = headers[h].strip();
      }
      string[] keys = trimmed.keys();
      sort(keys);
      return keys.map!(k => k ~ ":" ~ trimmed[k] ~ "\n").join("");
 }

 string signedHeaders(string[string] headers)
 {
      string[] keys = headers.keys().map!(k => k.toLower()).array();
      sort(keys);
      return keys.join(";");
 }

 string hash(T)(T payload)
 {
      auto hash = sha256Of(payload);
      return hash.toHexString().toLower();
 }

 string makeCRSigV4(CanonicalRequest r)
 {
      auto cr =
          r.method.toUpper() ~ "\n" ~
          (r.uri.empty ? "/" : r.uri) ~ "\n" ~
          canonicalQueryString(r.queryParameters) ~ "\n" ~
          canonicalHeaders(r.headers) ~ "\n" ~
          signedHeaders(r.headers) ~ "\n" ~
          hash(r.payload);

      return hash(cr);
 }

 unittest {
      string[string] empty;

      auto r = CanonicalRequest(
              "POST",
              "/",
              empty,
              ["content-type": "application/x-www-form-urlencoded;
 charset=utf-8",
               "host": "iam.amazonaws.com",
               "x-amz-date": "20110909T233600Z"],
              cast(ubyte[])"Action=ListUsers&Version=2010-05-08");

      auto sig = makeCRSigV4(r);

      assert(sig ==
 "3511de7e95d28ecd39e9513b642aee07e54f4941150d8df8bf94b328ef7e55e2");
 }

 struct SignableRequest
 {
      string dateString;
      string timeStringUTC;
      string region;
      string service;
      CanonicalRequest canonicalRequest;
 }

 string signableString(SignableRequest r) {
      return algorithm ~ "\n" ~
          r.dateString ~ "T" ~ r.timeStringUTC ~ "Z\n" ~
          r.dateString ~ "/" ~ r.region ~ "/" ~ r.service ~
 "/aws4_request\n" ~
          makeCRSigV4(r.canonicalRequest);
 }

 unittest {
      string[string] empty;

      SignableRequest r;
      r.dateString = "20110909";
      r.timeStringUTC = "233600";
      r.region = "us-east-1";
      r.service = "iam";
      r.canonicalRequest = CanonicalRequest(
              "POST",
              "/",
              empty,
              ["content-type": "application/x-www-form-urlencoded;
 charset=utf-8",
               "host": "iam.amazonaws.com",
               "x-amz-date": "20110909T233600Z"],
              cast(ubyte[])"Action=ListUsers&Version=2010-05-08");

      auto sampleString =
          algorithm ~ "\n" ~
          "20110909T233600Z\n" ~
          "20110909/us-east-1/iam/aws4_request\n" ~
 "3511de7e95d28ecd39e9513b642aee07e54f4941150d8df8bf94b328ef7e55e2";

      assert(sampleString == signableString(r));
 }

 ubyte[] array_xor(ubyte[] b1, ubyte[] b2)
 {
      assert(b1.length == b2.length);
      ubyte[] ret;
      for (uint i = 0; i < b1.length; i++)
          ret ~= b1[i] ^ b2[i];
      return ret;
 }

 auto hmac_sha256(string key, string message)
 {
      auto hmac = hmac!SHA256(key.representation);
      hmac.put(message.representation);
      return hmac.finish;
 }

 unittest {
      string key = "key";
      string message = "The quick brown fox jumps over the lazy dog";

      string mac = hmac_sha256(key, message).toHexString().toLower();
      assert(mac ==
 "f7bc83f430538424b13298e6aa6fb143ef4d59a14946175997479dbc2d1a3cd8");
 }

 auto signingKey(string secret, string dateString, string region, string
 service)
 {
      auto kSecret = "AWS4" ~ secret;
      auto kDate = hmac_sha256(kSecret, dateString);
      auto kRegion = hmac_sha256(kDate, region);
      auto kService = hmac_sha256(kRegion, service);
      return hmac_sha256(kService, "aws4_request");
 }

 unittest {
      string secretKey = "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY";
      auto signKey = signingKey(secretKey, "20110909", "us-east-1", "iam");

      ubyte[] expected = [152, 241, 216, 137, 254, 196, 244, 66, 26, 220,
 82, 43, 171, 12, 225, 248, 46, 105, 41, 194, 98, 237, 21, 229, 169, 76,
 144, 239, 209, 227, 176, 231 ];
      assert(expected == signKey);
 }

 alias sign = hmac_sha256;

 unittest {
      auto sampleString =
          "AWS4-HMAC-SHA256\n" ~
          "20110909T233600Z\n" ~
          "20110909/us-east-1/iam/aws4_request\n" ~
 "3511de7e95d28ecd39e9513b642aee07e54f4941150d8df8bf94b328ef7e55e2";

      auto secretKey = "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY";
      auto signKey = signingKey(secretKey, "20110909", "us-east-1", "iam");

      auto signature = sign(signKey, sampleString).toHexString().toLower();
      auto expected =
 "ced6826de92d2bdeed8f846f0bf508e8559e98e4b0199114b84c54174deb456c";

      assert(signature == expected);
 }

 /**
   * CredentialScope == date / region / service / aws4_request
   */
 string createSignatureHeader(string accessKeyID, string credentialScope,
 string[string] reqHeaders, ubyte[] signature)
 {
      return algorithm ~ " Credential=" ~ accessKeyID ~ "/" ~
 credentialScope ~ "/aws4_request, SignedHeaders=" ~
 signedHeaders(reqHeaders) ~ ", Signature=" ~
 signature.toHexString().toLower();
 }

 string dateFromISOString(string iso)
 {
      auto i = iso.indexOf('T');
      if (i == -1) throw new Exception("ISO time in wrong format: " ~ iso);
      return iso[0..i];
 }

 string timeFromISOString(string iso)
 {
      auto t = iso.indexOf('T');
      auto z = iso.indexOf('Z');
      if (t == -1 || z == -1) throw new Exception("ISO time in wrong
 format: " ~ iso);
      return iso[t+1..z];
 }

 unittest {
      assert(dateFromISOString("20110909T1203Z") == "20110909");
 }


 void main()
 {
      auto sampleString =
          "AWS4-HMAC-SHA256\n" ~
          "20110909T233600Z\n" ~
          "20110909/us-east-1/iam/aws4_request\n" ~
 "3511de7e95d28ecd39e9513b642aee07e54f4941150d8df8bf94b328ef7e55e2";

      auto secretKey = "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY";
      auto signKey = signingKey(secretKey, "20110909", "us-east-1", "iam");

      auto signature = sign(signKey, sampleString).toHexString().toLower();

      writeln(signature);
 }

 When i try to compile it im getting such error:

 [holo ultraxps test]$ dmd -unittest hello.d
 hello.d(196): Error: function sigv4.hmac_sha256 (string key, string
 message) is not callable using argument types (ubyte[32], string)
 [holo ultraxps test]$ dmd hello.d
 hello.d(196): Error: function sigv4.hmac_sha256 (string key, string
 message) is not callable using argument types (ubyte[32], string)
 [holo ultraxps test]$


 Line 196 is: "auto kRegion = hmac_sha256(kDate, region);"

 I was looking for ubyte[32] variable but i cant find it anywhere. What
 is strange unit tests are passing (i think, maybe im wrong).

It's in signingKey. Remember auto just means work out type based upon 
assigning value's type.
I would really really recommend since you keep having issues with hmac, 
to go the Botan way. Since it works.

Oct 06 2015

holo <holosian gmail.com> writes:

I was fighting with it a little bit and after all i just leave 
original function which was in code, removed dependencies from 
vibe.d and finally tried to contact api. Here is my present code:



import std.stdio;
import std.string;
import std.file;
import std.datetime;
import std.process;
import std.digest.sha;
import std.net.curl;
import sigv4;


auto zone = "us-east-1";
auto service = "ec2";


void main()
{
	auto accKey = environment["AWS_ACCESS_KEY"];
	auto secKey = environment["AWS_SECRET_KEY"];

	auto currentClock = Clock.currTime;

	auto currentDate = cast(Date)currentClock;
	auto curDateStr = currentDate.toISOString;

	auto currentTime = cast(TimeOfDay)currentClock;
	auto curTimeStr = currentTime.toISOString;

	auto xamztime = curDateStr ~ "T" ~ curTimeStr ~ "Z";

	string[string] empty;

	SignableRequest r;
	r.dateString = curDateStr;
	r.timeStringUTC = curTimeStr;
	r.region = zone;
	r.service = service;
	r.canonicalRequest = CanonicalRequest(
			"GET",
			"/",
			["action" : "DescribeInstances", "version" : "2013-10-15"],
			["content-type" : "application/x-www-form-urlencoded; 
charset=utf-8",
			 "host" : service ~ ".amazonaws.com",
			 "x-amz-date" : xamztime],
			 cast(ubyte[])"");
	
	auto qParm = 
canonicalQueryString(r.canonicalRequest.queryParameters);

	auto sigHead = canonicalHeaders(r.canonicalRequest.headers);

	auto sigStr = signableString(r);

	auto sigKey = signingKey(secKey, curDateStr, zone, service);
	
	auto signature = sign(sigKey, 
cast(ubyte[])sigStr).toHexString().toLower();

	writeln();	
	writeln(qParm);
	writeln();
	writeln(sigHead);
	writeln();
	writeln(sigStr);
	writeln();
	writeln(signature);
	writeln();

	auto client = HTTP();
	client.clearRequestHeaders;
	client.addRequestHeader("content-type:", 
"application/x-www-form-urlencoded; charset=utf-8");
	client.addRequestHeader("host:", service ~ ".amazonaws.com");
	client.addRequestHeader("x-amz-date:", xamztime);
	client.addRequestHeader("Authoryzation:", "AWS4-HMAC-SHA256" ~ " 
" ~ "Credential=" ~ accKey ~ "/" ~ xamztime ~ "/" ~ zone ~ "/" ~ 
service ~ "/" ~ "aws4_request" ~ ", " ~ "SignedHeaders=" ~ 
"content-type;host;x-amz-date" ~ ", " ~ "Signature=" ~ signature);

	auto url = service ~ ".amazonaws.com?" ~ 
"Action=DescribeInstances&Version=2013-10-15";
	writeln(url);
	auto content = get(url, client);
	writeln(content);
}

Everything is compiling but im getting 400 (Bad Request):



action=DescribeRegions&version=2013-10-15

content-type: application/x-www-form-urlencoded; charset=utf-8
host: ec2.amazonaws.com
x-amz-date: 20151009T053800Z


AWS4-HMAC-SHA256
20151009T053800Z
20151009/us-east-1/ec2/aws4_request
888595748692147ceafafcae3941ec0d83ac42c97641e4d954d7447a00c56270

69b1e4c5212cc6b485569fdfb43f7dde94413b36c50393c55d4810ced47f167b

ec2.amazonaws.com?Action=DescribeRegions&Version=2013-10-15
std.net.curl.CurlException /usr/include/dlang/dmd/std/net/curl.d(824): HTTP
request returned status code 400 (Bad Request)
----------------
/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(pure  safe
bool std.exception.enforce!(std.net.curl.CurlException, bool).enforce(bool,
lazy const(char)[], immutable(char)[], ulong)+0x65) [0x5004bd]
/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(char[]
std.net.curl._basicHTTP!(char)._basicHTTP(const(char)[], const(void)[],
std.net.curl.HTTP)+0x1a2) [0x4fc0ba]
/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(char[]
std.net.curl.get!(std.net.curl.HTTP, char).get(const(char)[],
std.net.curl.HTTP)+0x6a) [0x4fbec2]
/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/
eader(_Dmain+0x840) [0x4f8f98]
/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(_D2rt6dmain211_d_run_mainUiPPaPUAAaZiZ6runAllMFZ
__lambda1MFZv+0x1f) [0x533eab]
/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(void
rt.dmain2._d_run_main(int, char**, extern (C) int
function(char[][])*).tryExec(scope void delegate())+0x2a) [0x533e06]
/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(void
rt.dmain2._d_run_main(int, char**, extern (C) int
function(char[][])*).runAll()+0x2b) [0x533e67]
/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(void
rt.dmain2._d_run_main(int, char**, extern (C) int
function(char[][])*).tryExec(scope void delegate())+0x2a) [0x533e06]
/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/heade
(_d_run_main+0x1d2) [0x533d86]
/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(main+0x12)
[0x52cb62]
/usr/lib/libc.so.6(__libc_start_main+0xf0) [0x7f9c39f71610]


What am i doing wrong? What is strange when when i do same with 
curl its responding normally of course is not authenticated).


ec2.amazonaws.com?Action=DescribeInstances&Version=2013-10-15
[1] 20390

<Response><Errors><Error><Code>MissingParameter</Code><Message>The request must
contain the parameter
AWSAccessKeyId</Message></Error></Errors><RequestID>e1352781-c2b4-4e74-ade3-80d655efd0ac</RequestID></Response>

Oct 08 2015

Laeeth Isharc <laeethnospam nospam.laeeth.com> writes:

On Friday, 9 October 2015 at 04:04:57 UTC, holo wrote:

 	r.dateString = 	client.addRequestHeader("Authoryzation:", 
 "AWS4-HMAC-SHA256" ~ " " ~ "Credential=" ~ accKey ~ "/" ~ 
 xamztime ~ "/" ~ zone ~ "/" ~ service ~ "/" ~ "aws4_request" ~ 
 ", " ~ "SignedHeaders=" ~ "content-type;host;x-amz-date" ~ ", " 
 ~ "Signature=" ~ signature);

authorisation ??? (Or maybe authorization) ie a typo (check other 
fields carefully too, just in case)



 	auto url = service ~ ".amazonaws.com?" ~ 
 "Action=DescribeInstances&Version=2013-10-15";
 	writeln(url);
 	auto content = get(url, client);
 	writeln(content);
 }

 Everything is compiling but im getting 400 (Bad Request):



 action=DescribeRegions&version=2013-10-15

 content-type: application/x-www-form-urlencoded; charset=utf-8
 host: ec2.amazonaws.com
 x-amz-date: 20151009T053800Z


 AWS4-HMAC-SHA256
 20151009T053800Z
 20151009/us-east-1/ec2/aws4_request
 888595748692147ceafafcae3941ec0d83ac42c97641e4d954d7447a00c56270

 69b1e4c5212cc6b485569fdfb43f7dde94413b36c50393c55d4810ced47f167b

 ec2.amazonaws.com?Action=DescribeRegions&Version=2013-10-15
 std.net.curl.CurlException /usr/include/dlang/dmd/std/net/curl.d(824): HTTP
request returned status code 400 (Bad Request)
 ----------------
 /tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(pure  safe
bool std.exception.enforce!(std.net.curl.CurlException, bool).enforce(bool,
lazy const(char)[], immutable(char)[], ulong)+0x65) [0x5004bd]
 /tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(char[]
std.net.curl._basicHTTP!(char)._basicHTTP(const(char)[], const(void)[],
std.net.curl.HTTP)+0x1a2) [0x4fc0ba]
 /tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(char[]
std.net.curl.get!(std.net.curl.HTTP, char).get(const(char)[],
std.net.curl.HTTP)+0x6a) [0x4fbec2]
 /tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/
eader(_Dmain+0x840) [0x4f8f98]
 /tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(_D2rt6dmain211_d_run_mainUiPPaPUAAaZiZ6runAllMFZ
__lambda1MFZv+0x1f) [0x533eab]
 /tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(void
rt.dmain2._d_run_main(int, char**, extern (C) int
function(char[][])*).tryExec(scope void delegate())+0x2a) [0x533e06]
 /tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(void
rt.dmain2._d_run_main(int, char**, extern (C) int
function(char[][])*).runAll()+0x2b) [0x533e67]
 /tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(void
rt.dmain2._d_run_main(int, char**, extern (C) int
function(char[][])*).tryExec(scope void delegate())+0x2a) [0x533e06]
 /tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/heade
(_d_run_main+0x1d2) [0x533d86]
 /tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(main+0x12)
[0x52cb62]
 /usr/lib/libc.so.6(__libc_start_main+0xf0) [0x7f9c39f71610]


 What am i doing wrong? What is strange when when i do same with 
 curl its responding normally of course is not authenticated).


 ec2.amazonaws.com?Action=DescribeInstances&Version=2013-10-15
 [1] 20390

 <Response><Errors><Error><Code>MissingParameter</Code><Message>The request
must contain the parameter
AWSAccessKeyId</Message></Error></Errors><RequestID>e1352781-c2b4-4e74-ade3-80d655efd0ac</RequestID></Response>

Oct 08 2015

holo <holosian gmail.com> writes:

I correct typo (thats whats happening when you trying to solve 
problem after night at 6:00 AM ).But still  have same error. When 
use curl with same generated creditentials im getting "400". What 
is strange when im using same generated headers and signature 
with command line curl im getting "500". My headers right now:

auto client = HTTP();
         client.clearRequestHeaders;
         client.addRequestHeader("Content-Type:", 
"application/x-www-form-urlencoded; charset=utf-8");
         client.addRequestHeader("Host:", service ~ 
".amazonaws.com");
         client.addRequestHeader("X-Amz-Date:", xamztime);
         client.addRequestHeader("Authorization:", 
"AWS4-HMAC-SHA256" ~ " " ~ "Credential=" ~ accKey ~ "/" ~ 
xamztime ~ "/" ~ zone ~ "/" ~ service ~ "/" ~ "aws4_request" ~ ", 
" ~ "SignedHeaders=" ~ "content-type;host;x-amz-date" ~ ", " ~ 
"Signature=" ~ signature);

         auto url = service ~ ".amazonaws.com?" ~ 
"Action=DescribeInstances&Version=2013-10-15";
         writeln(url);
         auto content = get(url, client);
         writeln(content);


Did i set up it correctly?

Oct 09 2015

holo <holosian gmail.com> writes:

I doped HTTP packets with wireshark, maybe that  will  be useful:

First recived:

  Hypertext Transfer Protocol
     HTTP/1.1 400 Bad Request\r\n
         [Expert Info (Chat/Sequence): HTTP/1.1 400 Bad 
Request\r\n]
         Request Version: HTTP/1.1
         Status Code: 400
         Response Phrase: Bad Request
     Transfer-Encoding: chunked\r\n
     Date: Fri, 09 Oct 2015 15:43:37 GMT\r\n
     Cneonction: close\r\n
     Server: AmazonEC2\r\n
     \r\n
     [HTTP response 1/2]
     [Next request in frame: 7739]
     HTTP chunked response

Second sent:

Hypertext Transfer Protocol
     GET /?Action=DescribeInstances&Version=2013-10-15 HTTP/1.1\r\n
         [Expert Info (Chat/Sequence): GET 
/?Action=DescribeInstances&Version=2013-10-15 HTTP/1.1\r\n]
         Request Method: GET
         Request URI: /?Action=DescribeInstances&Version=2013-10-15
         Request Version: HTTP/1.1
     Host:: ec2.amazonaws.com\r\n
     User-Agent: Phobos-std.net.curl/2.069 (libcurl/7.44.0)\r\n
     Accept: */*\r\n
     Content-Type:: application/x-www-form-urlencoded; 
charset=utf-8\r\n
     X-Amz-Date:: 20151009T174337Z\r\n
     Authorization:: AWS4-HMAC-SHA256 
Credential=AKIAIHGPAJQ2PJ47YKKA/20151009T174337Z/us-east-1/ec2/aws4_request,
SignedHeaders=content-type;host;x-amz-date,
Signature=3049eff7d6b5084c2fee71d10d5cffb17e0df1a9fe76963859c423f637d0496e\r\n
     \r\n
     [Full request URI: http://: 
ec2.amazonaws.com/?Action=DescribeInstances&Version=2013-10-15]
     [HTTP request 2/2]

What is strange why im getting first that 400 Bad Request packet 
and after that is sent GET one?

Oct 09 2015

Marc =?UTF-8?B?U2Now7x0eg==?= <schuetzm gmx.net> writes:

On Friday, 9 October 2015 at 16:01:33 UTC, holo wrote:
     Host:: ec2.amazonaws.com\r\n

This, ...

     Content-Type:: application/x-www-form-urlencoded; 
 charset=utf-8\r\n
     X-Amz-Date:: 20151009T174337Z\r\n
     Authorization:: AWS4-HMAC-SHA256 
 Credential=AKIAIHGPAJQ2PJ47YKKA/20151009T174337Z/us-east-1/ec2/aws4_request,
SignedHeaders=content-type;host;x-amz-date,
Signature=3049eff7d6b5084c2fee71d10d5cffb17e0df1a9fe76963859c423f637d0496e\r\n
     \r\n

... these ones, ...

     [Full request URI: http://: 
 ec2.amazonaws.com/?Action=DescribeInstances&Version=2013-10-15]
     [HTTP request 2/2]

... and this look wrong. Where does the additional ": " come from?

Oct 09 2015

holo <holosian gmail.com> writes:

OK i find out error, in addRequestHeader i was using ":" after 
header name what casing problem. I removed it and now im getting 
"unauthorized". Here is how it looks right now:

  HTTP/1.1 401 Unauthorized\r\n
         [Expert Info (Chat/Sequence): HTTP/1.1 401 
Unauthorized\r\n]
         Request Version: HTTP/1.1
         Status Code: 401
         Response Phrase: Unauthorized
     Transfer-Encoding: chunked\r\n
     Date: Fri, 09 Oct 2015 16:22:47 GMT\r\n
     Server: AmazonEC2\r\n
     \r\n
     [HTTP response 1/2]
     [Next request in frame: 8371]
     HTTP chunked response
         Data chunk (254 octets)
             Chunk size: 254 octets
             Data (254 bytes)

In data field i can read:

"AWS was not able to validate provided access credentials" Is my 
signing process incorrect?

Oct 09 2015

holo <holosian gmail.com> writes:

On Friday, 9 October 2015 at 16:30:26 UTC, holo wrote:
 OK i find out error, in addRequestHeader i was using ":" after 
 header name what casing problem. I removed it and now im 
 getting "unauthorized". Here is how it looks right now:

  HTTP/1.1 401 Unauthorized\r\n
         [Expert Info (Chat/Sequence): HTTP/1.1 401 
 Unauthorized\r\n]
         Request Version: HTTP/1.1
         Status Code: 401
         Response Phrase: Unauthorized
     Transfer-Encoding: chunked\r\n
     Date: Fri, 09 Oct 2015 16:22:47 GMT\r\n
     Server: AmazonEC2\r\n
     \r\n
     [HTTP response 1/2]
     [Next request in frame: 8371]
     HTTP chunked response
         Data chunk (254 octets)
             Chunk size: 254 octets
             Data (254 bytes)

 In data field i can read:

 "AWS was not able to validate provided access credentials" Is 
 my signing process incorrect?

Maybe i will put my present code again:



import std.stdio;
import std.string;
import std.file;
import std.datetime;
import std.process;
import std.digest.sha;
import std.net.curl;
import std.uri;
import sigv4;


auto zone = "us-east-1";
auto service = "ec2";


void main()
{
	auto accKey = environment["AWS_ACCESS_KEY"];
	auto secKey = environment["AWS_SECRET_KEY"];

	auto currentClock = Clock.currTime;

	auto currentDate = cast(Date)currentClock;
	auto curDateStr = currentDate.toISOString;

	auto currentTime = cast(TimeOfDay)currentClock;
	auto curTimeStr = currentTime.toISOString;

	auto xamztime = curDateStr ~ "T" ~ curTimeStr ~ "Z";

	string[string] empty;

	SignableRequest r;
	r.dateString = curDateStr;
	r.timeStringUTC = curTimeStr;
	r.region = zone;
	r.service = service;
	r.canonicalRequest = CanonicalRequest(
			"POST",
			"/",
			["action" : "DescribeInstances", "version" : "2013-10-15"],
//			["accept" : "*/*",
			["content-type" : "application/x-www-form-urlencoded; 
charset=utf-8",
			 "host" : service ~ ".amazonaws.com",
			 "x-amz-date" : xamztime],
			 cast(ubyte[])"");          
//cast(ubyte[])"Action=DescribeInstances&Version=2013-10-15");
	
	auto qParm = 
canonicalQueryString(r.canonicalRequest.queryParameters);

	auto sigHead = canonicalHeaders(r.canonicalRequest.headers);

	auto sigStr = signableString(r);

	auto sigKey = signingKey(secKey, curDateStr, zone, service);
	
	auto signature = sign(sigKey, 
cast(ubyte[])sigStr).toHexString().toLower();

	writeln();	
	writeln(qParm);
	writeln();
	writeln(sigHead);
	writeln();
	writeln(sigStr);
	writeln();
	writeln(signature);
	writeln();
	auto client = HTTP();
//	client.clearRequestHeaders;
	client.addRequestHeader("content-type", 
"application/x-www-form-urlencoded; charset=utf-8");
	client.addRequestHeader("host", service ~ ".amazonaws.com");
	client.addRequestHeader("x-amz-date", xamztime);
	client.addRequestHeader("authorization", "AWS4-HMAC-SHA256" ~ " 
" ~ "Credential=" ~ accKey ~ "/" ~ xamztime ~ "/" ~ zone ~ "/" ~ 
service ~ "/" ~ "aws4_request" ~ ", " ~ "SignedHeaders=" ~ 
"content-type;host;x-amz-date" ~ ", " ~ "Signature=" ~ signature);

	auto url = "ec2.amazonaws.com/?" ~ 
"Action=DescribeInstances&Version=2013-10-15";
	auto urlenc = encode(url);
	writeln(url);
	auto content = get(urlenc, client);
	writeln(content);
}

Is my signing process correct?

Oct 10 2015

holo <holosian gmail.com> writes:

After long fight with previous code i try to rewrite "one-to-one" 
python example from 
http://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html
(GET part) from begging to D with full success. Here is working code in clear
D. Im using hmac function which is available from 2.069 of phobos). It is not
too beautiful but like i said im beginner, here it is:




module sigawsv4;

import std.stdio, std.process;
import std.digest.sha, std.digest.hmac;
import std.string;
import std.conv;
import std.datetime;
import std.net.curl;

void main()
{
	auto accessKey = environment["AWS_ACCESS_KEY"];
	auto secretKey = environment["AWS_SECRET_KEY"];

	auto currentClock = Clock.currTime(UTC());

	auto currentDate = cast(Date)currentClock;
	auto curDateStr = currentDate.toISOString;

	auto currentTime = cast(TimeOfDay)currentClock;
	auto curTimeStr = currentTime.toISOString;

	auto xamztime = curDateStr ~ "T" ~ curTimeStr ~ "Z";

	
	string method = "GET";
	string service = "ec2";
	string host = "ec2.amazonaws.com";
	string region = "us-east-1";
	string endpoint = "https://ec2.amazonaws.com";
	string request_parameters = 
"Action=DescribeInstances&Version=2013-10-15";


	auto hmac_sha256(ubyte[] key, ubyte[] msg)
	{
		auto hmac = hmac!SHA256(key);
		hmac.put(msg);
		auto digest = hmac.finish;

		return digest;
	}

	alias sign = hmac_sha256;

	auto getSignatureKey(string key, string dateStamp, string 
regionName, string serviceName)
	{
		ubyte[] kString = cast(ubyte[])("AWS4" ~ key);
		auto kDate = sign(kString, cast(ubyte[])dateStamp);
		auto kRegion = sign(kDate, cast(ubyte[])regionName);
		auto kService = sign(kRegion,  cast(ubyte[])serviceName);
		auto kSigning = sign(kService, cast(ubyte[])"aws4_request");

		return kSigning;
	}


	string canonicalURI = "/";
	string canonicalQueryString = request_parameters;
	string canonicalHeadersString =  "host:" ~ host ~ "\n" ~ 
"x-amz-date:" ~ xamztime ~ "\n";
	string signedHeaders = "host;x-amz-date";
	string payloadHash = sha256Of("").toHexString.toLower;
	string canonicalRequest = method ~ "\n" ~ canonicalURI ~ "\n" ~ 
canonicalQueryString ~ "\n" ~ canonicalHeadersString ~ "\n" ~ 
signedHeaders ~ "\n" ~ payloadHash;


	string algorithm = "AWS4-HMAC-SHA256";
	string credentialScope = curDateStr ~ "/" ~ region ~ "/" ~ 
service ~ "/" ~ "aws4_request";
	string stringToSign = algorithm ~ "\n" ~ xamztime ~ "\n" ~ 
credentialScope ~ "\n" ~ 
sha256Of(canonicalRequest).toHexString.toLower;

	auto signingKey = getSignatureKey(secretKey, curDateStr, region, 
service);

	string signature = hmac_sha256(signingKey, 
cast(ubyte[])stringToSign).toHexString.toLower;

	string authorizationHeader = algorithm ~ " " ~ "Credential=" ~ 
accessKey ~ "/" ~ credentialScope ~ ", " ~ "SignedHeaders=" ~ 
signedHeaders ~ ", " ~ "Signature=" ~ signature;

	auto client = HTTP(endpoint ~ "?" ~ canonicalQueryString);
	client.method = HTTP.Method.get;
	client.addRequestHeader("x-amz-date", xamztime);
	client.addRequestHeader("Authorization", authorizationHeader);
	auto content = client.perform();

	writeln(content);
}

Now will try to "pack" it to some class and make it more usable 
and universal. I think that thread can be closed/solved (if 
something like this exist here)

Thank you all for help.

//holo

Oct 11 2015

Laeeth Isharc <laeethnospam nospamlaeeth.com> writes:

On Sunday, 11 October 2015 at 23:16:51 UTC, holo wrote:
 After long fight with previous code i try to rewrite 
 "one-to-one" python example from 
 http://docs.aws.amazon.com/general/latest/gr/sigv4-signed-r
quest-examples.html (GET part) from begging to D with full success. Here is
working code in clear D. Im using hmac function which is available from 2.069
of phobos). It is not too beautiful but like i said im beginner, here it is:

Thanks for sharing holo - you saved me a bit of time, as I needed 
just the same thing for another api (dnsmadeeasy).

Good luck with your own project.


Laeeth.

Oct 11 2015

Marc =?UTF-8?B?U2Now7x0eg==?= <schuetzm gmx.net> writes:

On Sunday, 11 October 2015 at 23:16:51 UTC, holo wrote:
 	auto hmac_sha256(ubyte[] key, ubyte[] msg)
 	{
 		auto hmac = hmac!SHA256(key);
 		hmac.put(msg);
 		auto digest = hmac.finish;

 		return digest;
 	}

 	alias sign = hmac_sha256;

std.digest.hmac already contains a helper function for this:

     hmac!SHA256(data, secret);

The order of arguments is this way to allow chaining with UFCS:

     data.hmac!SHA256(secret);

But it seems the documentation generator can't handle ddoc 
comments in templates, so unfortunately it doesn't appear in the 
documentation.

Oct 12 2015

holo <holosian gmail.com> writes:

Thank you for info, i changed my code to use that build in 
template and changed "cast(ubyte[]"  to "xxx.representation" and 
it is still working:



module sigawsv4;

import std.stdio, std.process;
import std.digest.sha, std.digest.hmac;
import std.string;
import std.conv;
import std.datetime;
import std.net.curl;

void main()
{
	auto accessKey = environment["AWS_ACCESS_KEY"];
	auto secretKey = environment["AWS_SECRET_KEY"];

	auto currentClock = Clock.currTime(UTC());

	auto currentDate = cast(Date)currentClock;
	auto curDateStr = currentDate.toISOString;

	auto currentTime = cast(TimeOfDay)currentClock;
	auto curTimeStr = currentTime.toISOString;

	auto xamztime = curDateStr ~ "T" ~ curTimeStr ~ "Z";

	
	string method = "GET";
	string service = "ec2";
	string host = "ec2.amazonaws.com";
	string region = "us-east-1";
	string endpoint = "https://ec2.amazonaws.com";
	string request_parameters = 
"Action=DescribeInstances&Version=2013-10-15";


	alias sign = hmac!SHA256;

	auto getSignatureKey(string key, string dateStamp, string 
regionName, string serviceName)
	{
		auto kString = ("AWS4" ~ key).representation;
		auto kDate = sign(dateStamp.representation, kString);
		auto kRegion = sign(regionName.representation, kDate);
		auto kService = sign(serviceName.representation, kRegion);
		auto kSigning = sign("aws4_request".representation, kService);

		return kSigning;
	}


	string canonicalURI = "/";
	string canonicalQueryString = request_parameters;
	string canonicalHeadersString =  "host:" ~ host ~ "\n" ~ 
"x-amz-date:" ~ xamztime ~ "\n";
	string signedHeaders = "host;x-amz-date";
	string payloadHash = sha256Of("").toHexString.toLower;
	string canonicalRequest = method ~ "\n" ~ canonicalURI ~ "\n" ~ 
canonicalQueryString ~ "\n" ~ canonicalHeadersString ~ "\n" ~ 
signedHeaders ~ "\n" ~ payloadHash;


	string algorithm = "AWS4-HMAC-SHA256";
	string credentialScope = curDateStr ~ "/" ~ region ~ "/" ~ 
service ~ "/" ~ "aws4_request";
	string stringToSign = algorithm ~ "\n" ~ xamztime ~ "\n" ~ 
credentialScope ~ "\n" ~ 
sha256Of(canonicalRequest).toHexString.toLower;

	auto signingKey = getSignatureKey(secretKey, curDateStr, region, 
service);

	string signature = hmac!SHA256(stringToSign.representation, 
signingKey).toHexString.toLower;

	string authorizationHeader = algorithm ~ " " ~ "Credential=" ~ 
accessKey ~ "/" ~ credentialScope ~ ", " ~ "SignedHeaders=" ~ 
signedHeaders ~ ", " ~ "Signature=" ~ signature;

	auto client = HTTP(endpoint ~ "?" ~ canonicalQueryString);
	client.method = HTTP.Method.get;
	client.addRequestHeader("x-amz-date", xamztime);
	client.addRequestHeader("Authorization", authorizationHeader);
	auto content = client.perform();

	writeln(content);
}

Oct 12 2015

yawniek <dlang srtnwz.com> writes:

On Saturday, 3 October 2015 at 03:11:06 UTC, holo wrote:
 Hello

 I'm trying to contact AWS API with D according to documentation:

 [...]

check https://github.com/yannick/vibe-aws
it has v4 implemented

Oct 03 2015

Vladimir Panteleev <thecybershadow.lists gmail.com> writes:

On Saturday, 3 October 2015 at 03:11:06 UTC, holo wrote:
 Last but not least, how to write such function in D:

 def sign(key, msg):
     return hmac.new(key, msg.encode("utf-8"), 
 hashlib.sha256).digest()

 ?

 I can't find in standard libraryt hmac function, is it existing?

The next version of D will have a std.digest.hmac module:

http://dlang.org/phobos-prerelease/std_digest_hmac.html

Oct 03 2015

D Programming

C/C++ Programming

Other

digitalmars.D.learn - AWS API Dlang, hmac sha256 function.