www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - AWS SDK

reply Jadbox <jadit2 gmail.com> writes:
Is there an AWS library in the works for D? It's seriously the 
main blocker for me to push adoption of the language internally.

If not, I can try to invest time into making one, but I could use 
help.

(fyi, there's one in the works for Rust: 
https://github.com/rusoto/rusoto)
Jun 26 2016
next sibling parent bachmeier <no spam.net> writes:
On Sunday, 26 June 2016 at 23:06:02 UTC, Jadbox wrote:
 Is there an AWS library in the works for D? It's seriously the 
 main blocker for me to push adoption of the language internally.

 If not, I can try to invest time into making one, but I could 
 use help.

 (fyi, there's one in the works for Rust: 
 https://github.com/rusoto/rusoto)
http://forum.dlang.org/post/jqotefrswxhaiqteefdq forum.dlang.org
Jun 26 2016
prev sibling next sibling parent yawniek <dlang srtnwz.com> writes:
On Sunday, 26 June 2016 at 23:06:02 UTC, Jadbox wrote:
 Is there an AWS library in the works for D? It's seriously the 
 main blocker for me to push adoption of the language internally.

 If not, I can try to invest time into making one, but I could 
 use help.

 (fyi, there's one in the works for Rust: 
 https://github.com/rusoto/rusoto)
what services do you need? the important bits, such as auth are around, but you likely need to implement some stuff for yourself. we have an incomplete s3 libary with https://github.com/tamediadigital/vibe-s3 there is also some code for kinesis and dynamo around but its likely outdated.
Jun 27 2016
prev sibling next sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Sunday, 26 June 2016 at 23:06:02 UTC, Jadbox wrote:
 Is there an AWS library in the works for D? It's seriously the 
 main blocker for me to push adoption of the language internally.

 If not, I can try to invest time into making one, but I could 
 use help.

 (fyi, there's one in the works for Rust: 
 https://github.com/rusoto/rusoto)
The C++ SDK appears to use a Java generator, maybe you can make it work for D since it is quite close to C++ (but probably a lot of work): https://github.com/aws/aws-sdk-cpp/tree/master/code-generation https://aws.amazon.com/blogs/aws/introducing-the-aws-sdk-for-c/
Jun 27 2016
prev sibling next sibling parent reply Brad Roberts via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 6/26/2016 4:06 PM, Jadbox via Digitalmars-d wrote:
 Is there an AWS library in the works for D? It's seriously the main
 blocker for me to push adoption of the language internally.

 If not, I can try to invest time into making one, but I could use help.

 (fyi, there's one in the works for Rust: https://github.com/rusoto/rusoto)
I have some old code here: https://github.com/braddr/downloads.dlang.org/tree/master/src It has sig v2 signing and some s3 object related code. Super minimal and sig v2 is out of date (though still very usable, in most aws regions). Ideally, someone would partner with aws -- there's a developer tools forum -- to add D to the suite of languages. I'm fairly sure they code generate at least a large part of the tool kits from the api definitions for each of the tons of services. Trying to manage them all by hand is a loosing battle. I've been tempted to do this a couple times, but the time investment would likely be more than I'm willing to spend.
Jun 27 2016
parent jadbox <jadit2 gmail.com> writes:
On Monday, 27 June 2016 at 17:53:32 UTC, Brad Roberts wrote:
 https://github.com/braddr/downloads.dlang.org/tree/master/src

 It has sig v2 signing and some s3 object related code.  Super 
 minimal and sig v2 is out of date (though still very usable, in 
 most aws regions).

 Ideally, someone would partner with aws -- there's a developer 
 tools forum -- to add D to the suite of languages.  I'm fairly 
 sure they code generate at least a large part of the tool kits 
 from the api definitions for each of the tons of services.  
 Trying to manage them all by hand is a loosing battle.

 I've been tempted to do this a couple times, but the time 
 investment would likely be more than I'm willing to spend.
Yep, the sdk would need to be autogenerated from their spec files. I use a great number of services from s3, Kinesis, SQS, SNS, Redshift, DynamoDB, EC2 cmds, Route 53 reconfiguring. This is a common story for most cloud developers. With the advent of microservices, it's a relatively easy sell to get developers to try a new language for a small service, but most of these services rely on cloud drivers.
Jun 27 2016
prev sibling parent reply Brad Roberts via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 6/27/16 10:53 AM, Brad Roberts via Digitalmars-d wrote:
 On 6/26/2016 4:06 PM, Jadbox via Digitalmars-d wrote:
 Is there an AWS library in the works for D? It's seriously the main
 blocker for me to push adoption of the language internally.

 If not, I can try to invest time into making one, but I could use help.

 (fyi, there's one in the works for Rust: https://github.com/rusoto/rusoto)
I have some old code here: https://github.com/braddr/downloads.dlang.org/tree/master/src It has sig v2 signing and some s3 object related code. Super minimal and sig v2 is out of date (though still very usable, in most aws regions). Ideally, someone would partner with aws -- there's a developer tools forum -- to add D to the suite of languages. I'm fairly sure they code generate at least a large part of the tool kits from the api definitions for each of the tons of services. Trying to manage them all by hand is a loosing battle. I've been tempted to do this a couple times, but the time investment would likely be more than I'm willing to spend.
I've started talking with a few of the aws sdk engineers (some of who I worked with when I was there) today about their sdk and the tooling involved. It might not be a horrible job to add another language to the mix. I'm going to meet with them and dig through some of the tooling to get a better feel for it. Later, Brad
Jun 27 2016
parent reply Ky-Anh Huynh <saigon example.net> writes:
On Tuesday, 28 June 2016 at 00:36:31 UTC, Brad Roberts wrote:
 On 6/27/16 10:53 AM, Brad Roberts via Digitalmars-d wrote:
 On 6/26/2016 4:06 PM, Jadbox via Digitalmars-d wrote:
 [...]
I have some old code here: https://github.com/braddr/downloads.dlang.org/tree/master/src It has sig v2 signing and some s3 object related code. Super minimal and sig v2 is out of date (though still very usable, in most aws regions). Ideally, someone would partner with aws -- there's a developer tools forum -- to add D to the suite of languages. I'm fairly sure they code generate at least a large part of the tool kits from the api definitions for each of the tons of services. Trying to manage them all by hand is a loosing battle. I've been tempted to do this a couple times, but the time investment would likely be more than I'm willing to spend.
I've started talking with a few of the aws sdk engineers (some of who I worked with when I was there) today about their sdk and the tooling involved. It might not be a horrible job to add another language to the mix. I'm going to meet with them and dig through some of the tooling to get a better feel for it. Later, Brad
Hi Brad, Do we have any good update on this? I'm writting my devops tools in Dlang now (Golang is too hot to use, isn't it:D). It's a great win if we can have AWS SDK support. I'm not a master however if there is any thing I can help, please let me know. Thanks for your reading.
Oct 17
next sibling parent Brad Roberts <braddr puremagic.com> writes:
On 10/17/2017 6:32 PM, Ky-Anh Huynh via Digitalmars-d wrote:
 On Tuesday, 28 June 2016 at 00:36:31 UTC, Brad Roberts wrote:
 On 6/27/16 10:53 AM, Brad Roberts via Digitalmars-d wrote:
 On 6/26/2016 4:06 PM, Jadbox via Digitalmars-d wrote:
 [...]
I have some old code here: https://github.com/braddr/downloads.dlang.org/tree/master/src It has sig v2 signing and some s3 object related code.  Super minimal and sig v2 is out of date (though still very usable, in most aws regions). Ideally, someone would partner with aws -- there's a developer tools forum -- to add D to the suite of languages.  I'm fairly sure they code generate at least a large part of the tool kits from the api definitions for each of the tons of services.  Trying to manage them all by hand is a loosing battle. I've been tempted to do this a couple times, but the time investment would likely be more than I'm willing to spend.
I've started talking with a few of the aws sdk engineers (some of who I worked with when I was there) today about their sdk and the tooling involved.  It might not be a horrible job to add another language to the mix.  I'm going to meet with them and dig through some of the tooling to get a better feel for it. Later, Brad
Hi Brad, Do we have any good update on this? I'm writting my devops tools in Dlang now (Golang is too hot to use, isn't it:D). It's a great win if we can have AWS SDK support. I'm not a master however if there is any thing I can help, please let me know. Thanks for your reading.
Unfortunately, I haven't made the time.  Too much contract work being thrown my way for the last year or so.
Oct 17
prev sibling parent reply Andre Pany <andre s-e-a-p.de> writes:
On Wednesday, 18 October 2017 at 01:32:51 UTC, Ky-Anh Huynh wrote:
 On Tuesday, 28 June 2016 at 00:36:31 UTC, Brad Roberts wrote:
 On 6/27/16 10:53 AM, Brad Roberts via Digitalmars-d wrote:
 On 6/26/2016 4:06 PM, Jadbox via Digitalmars-d wrote:
 [...]
I have some old code here: https://github.com/braddr/downloads.dlang.org/tree/master/src It has sig v2 signing and some s3 object related code. Super minimal and sig v2 is out of date (though still very usable, in most aws regions). Ideally, someone would partner with aws -- there's a developer tools forum -- to add D to the suite of languages. I'm fairly sure they code generate at least a large part of the tool kits from the api definitions for each of the tons of services. Trying to manage them all by hand is a loosing battle. I've been tempted to do this a couple times, but the time investment would likely be more than I'm willing to spend.
I've started talking with a few of the aws sdk engineers (some of who I worked with when I was there) today about their sdk and the tooling involved. It might not be a horrible job to add another language to the mix. I'm going to meet with them and dig through some of the tooling to get a better feel for it. Later, Brad
Hi Brad, Do we have any good update on this? I'm writting my devops tools in Dlang now (Golang is too hot to use, isn't it:D). It's a great win if we can have AWS SDK support. I'm not a master however if there is any thing I can help, please let me know. Thanks for your reading.
Hi, for an inhouse project I developed a generator which generates based on the API definitions from here https://github.com/aws/aws-sdk-js/tree/master/apis classes for the services and structures for the input/output requests. auto client = new AwsClient(); auto dynamoDBService = new DynamoDBService(client); CreateTableInput request = { tableName: "zapp-test1", provisionedThroughput: { readCapacityUnits: 1, writeCapacityUnits: 1 }, attributeDefinitions: [ {attributeName: "A", attributeType: ScalarAttributeType.S}, {attributeName: "B", attributeType: ScalarAttributeType.S} ], keySchema: [ {attributeName: "A", keyType: KeyType.HASH} ] }; dynamoDBService.createTable(request); Actually the generated classes do not call AWS directly but uses the AWS console client. Unfortunately the generator is proprietary. The good thing is, it was easily written within 4-5 hours. PS. Almost every service was working out of the box except EMR. There is a difference between the EMR commands / parameters in the API definitions and in the AWS console client. Kind regards André
Oct 18
parent reply aberba <karabutaworld gmail.com> writes:
On Wednesday, 18 October 2017 at 11:11:35 UTC, Andre Pany wrote:
 On Wednesday, 18 October 2017 at 01:32:51 UTC, Ky-Anh Huynh 
 wrote:
 On Tuesday, 28 June 2016 at 00:36:31 UTC, Brad Roberts wrote:
 On 6/27/16 10:53 AM, Brad Roberts via Digitalmars-d wrote:
 On 6/26/2016 4:06 PM, Jadbox via Digitalmars-d wrote:
 [...]
I have some old code here: https://github.com/braddr/downloads.dlang.org/tree/master/src It has sig v2 signing and some s3 object related code. Super minimal and sig v2 is out of date (though still very usable, in most aws regions). Ideally, someone would partner with aws -- there's a developer tools forum -- to add D to the suite of languages. I'm fairly sure they code generate at least a large part of the tool kits from the api definitions for each of the tons of services. Trying to manage them all by hand is a loosing battle. I've been tempted to do this a couple times, but the time investment would likely be more than I'm willing to spend.
I've started talking with a few of the aws sdk engineers (some of who I worked with when I was there) today about their sdk and the tooling involved. It might not be a horrible job to add another language to the mix. I'm going to meet with them and dig through some of the tooling to get a better feel for it. Later, Brad
Hi Brad, Do we have any good update on this? I'm writting my devops tools in Dlang now (Golang is too hot to use, isn't it:D). It's a great win if we can have AWS SDK support. I'm not a master however if there is any thing I can help, please let me know. Thanks for your reading.
Hi, for an inhouse project I developed a generator which generates based on the API definitions from here https://github.com/aws/aws-sdk-js/tree/master/apis classes for the services and structures for the input/output requests. auto client = new AwsClient(); auto dynamoDBService = new DynamoDBService(client); CreateTableInput request = { tableName: "zapp-test1", provisionedThroughput: { readCapacityUnits: 1, writeCapacityUnits: 1 }, attributeDefinitions: [ {attributeName: "A", attributeType: ScalarAttributeType.S}, {attributeName: "B", attributeType: ScalarAttributeType.S} ], keySchema: [ {attributeName: "A", keyType: KeyType.HASH} ] }; dynamoDBService.createTable(request); Actually the generated classes do not call AWS directly but uses the AWS console client. Unfortunately the generator is proprietary. The good thing is, it was easily written within 4-5 hours. PS. Almost every service was working out of the box except EMR. There is a difference between the EMR commands / parameters in the API definitions and in the AWS console client. Kind regards André
What's up with proprietary stuff in this? At least give us the setup for us to generate the apis on our own if you have your private stuff in there. I have been waiting for a D AWS SDK for "years"... At least any cloud SDK. The generated api approach seem like the way to go cus its a lot of work to build and maintain a D API by hand. I wanted to just do an S3 api plus some few services to use in some of our microservices. We opted to use Nodejs because it already has all the Cloud SDK for almost every provider. Technically D is much better... You know, cloud and microservices is the way to build any "scalable" server-side services in 2017 yet it seem those using D for server-side dev are still writing monoliths.
Oct 18
parent reply Andre Pany <andre s-e-a-p.de> writes:
On Wednesday, 18 October 2017 at 19:28:30 UTC, aberba wrote:
 What's up with proprietary stuff in this? At least give us the 
 setup for us to generate the apis on our own if you have your 
 private stuff in there. I have been waiting for a D AWS SDK for 
 "years"... At least any cloud SDK. The generated api approach 
 seem like the way to go cus its a lot of work to build and 
 maintain a D API by hand.

 I wanted to just do an S3 api plus some few services to use in 
 some of our microservices. We opted to use Nodejs because it 
 already has all the Cloud SDK for almost every provider. 
 Technically D is much better...

 You know, cloud and microservices is the way to build any 
 "scalable" server-side services in 2017 yet it seem those using 
 D for server-side dev are still writing monoliths.
Although I wrote the generator, it is property of my employer. I am unfortunately not allowed to make it public. The implementation is straight forward. For every api definition (e.g. https://github.com/aws/aws-sdk-js/blob/master/apis/iam-2010-05-08.normal.json) create a module with a class. Read the json file with std.json and for every operation in the json create a method with the same name in the class. For every shape in the json create a struct with the same name. Every operation defines either an input shape, output shape, or both. During the method call serialize the input struct to json, call the aws client with the json and deserialize the output json to the output struct. Kind regards André
Oct 18
parent reply ikod <geller.garry gmail.com> writes:
Hello,

On Wednesday, 18 October 2017 at 20:05:28 UTC, Andre Pany wrote:
 On Wednesday, 18 October 2017 at 19:28:30 UTC, aberba wrote:
 The implementation is straight forward. For every api 
 definition (e.g. 
 https://github.com/aws/aws-sdk-js/blob/master/apis/iam-2010-05-08.normal.json)
create a module with a class. Read the json file with std.json and for every
operation in the json create a method with the same name in the class.
 For every shape in the json create a struct with the same name.

 Every operation defines either an input shape, output shape, or 
 both. During the method call serialize the input struct to 
 json, call the aws client with the json and deserialize the 
 output json to the output struct.

 Kind regards
 André
Is there any docs from Amazon on this, beside json's itself? I will be very grateful for link.
Oct 18
next sibling parent Andre Pany <andre s-e-a-p.de> writes:
On Wednesday, 18 October 2017 at 20:19:20 UTC, ikod wrote:
 Hello,

 On Wednesday, 18 October 2017 at 20:05:28 UTC, Andre Pany wrote:
 On Wednesday, 18 October 2017 at 19:28:30 UTC, aberba wrote:
 The implementation is straight forward. For every api 
 definition (e.g. 
 https://github.com/aws/aws-sdk-js/blob/master/apis/iam-2010-05-08.normal.json)
create a module with a class. Read the json file with std.json and for every
operation in the json create a method with the same name in the class.
 For every shape in the json create a struct with the same name.

 Every operation defines either an input shape, output shape, 
 or both. During the method call serialize the input struct to 
 json, call the aws client with the json and deserialize the 
 output json to the output struct.

 Kind regards
 André
Is there any docs from Amazon on this, beside json's itself? I will be very grateful for link.
I think as a starting point the console client help is very interesting. http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-using.html Especially these two topics -Generate CLI Skeleton and CLI Input JSON Parameters -Controlling Command Output from the AWS Command Line Interface Kind regards André
Oct 18
prev sibling parent reply Stephan Dilly <stephan extrawurst.org> writes:
On 2017-10-18 20:19:20 +0000, ikod said:

 Hello,
 
 On Wednesday, 18 October 2017 at 20:05:28 UTC, Andre Pany wrote:
 On Wednesday, 18 October 2017 at 19:28:30 UTC, aberba wrote:
 The implementation is straight forward. For every api definition (e.g. 
 https://github.com/aws/aws-sdk-js/blob/master/apis/iam-2010-05-08.normal.json) 
 create a module with a class. Read the json file with std.json and for 
 every operation in the json create a method with the same name in the 
 class.
 For every shape in the json create a struct with the same name.
 
 Every operation defines either an input shape, output shape, or both. 
 During the method call serialize the input struct to json, call the aws 
 client with the json and deserialize the output json to the output 
 struct.
 
 Kind regards
 Andr
Is there any docs from Amazon on this, beside json's itself? I will be very grateful for link.
Are you going to take over the task? --Stephan
Oct 18
parent reply ikod <geller.garry gmail.com> writes:
On Wednesday, 18 October 2017 at 20:51:48 UTC, Stephan Dilly 
wrote:
 On 2017-10-18 20:19:20 +0000, ikod said:

 Hello,
 
 On Wednesday, 18 October 2017 at 20:05:28 UTC, Andre Pany 
 wrote:
 [...]
Is there any docs from Amazon on this, beside json's itself? I will be very grateful for link.
Are you going to take over the task? --Stephan
I'd like, but It depends on the required time investment. I need some time to look at the docs etc... Regards, Igor
Oct 18
parent reply Stephan Dilly <stephan extrawurst.org> writes:
On 2017-10-18 22:25:23 +0000, ikod said:

 On Wednesday, 18 October 2017 at 20:51:48 UTC, Stephan Dilly wrote:
 On 2017-10-18 20:19:20 +0000, ikod said:
 
 Hello,
 
 On Wednesday, 18 October 2017 at 20:05:28 UTC, Andre Pany wrote:
 [...]
Is there any docs from Amazon on this, beside json's itself? I will be very grateful for link.
Are you going to take over the task? --Stephan
I'd like, but It depends on the required time investment. I need some time to look at the docs etc... Regards, Igor
Hi Igor, I started a little tool (POC) that can parse the API spec and creates types and the API interface only for now: https://github.com/Extrawurst/aws-sdk-dlang-gen Its a start, not tested and does not come with any dependency to vibe or what. AWS specific request signing also needs to be done on top of it. but using that tool the types can be automatically generated. --Stephan
Oct 18
next sibling parent ikod <geller.garry gmail.com> writes:
On Wednesday, 18 October 2017 at 23:02:27 UTC, Stephan Dilly 
wrote:
 On 2017-10-18 22:25:23 +0000, ikod said:

 On Wednesday, 18 October 2017 at 20:51:48 UTC, Stephan Dilly 
 wrote:
 On 2017-10-18 20:19:20 +0000, ikod said:
 
 Hello,
 
 On Wednesday, 18 October 2017 at 20:05:28 UTC, Andre Pany 
 wrote:
 [...]
Is there any docs from Amazon on this, beside json's itself? I will be very grateful for link.
Are you going to take over the task? --Stephan
I'd like, but It depends on the required time investment. I need some time to look at the docs etc... Regards, Igor
Hi Igor, I started a little tool (POC) that can parse the API spec and creates types and the API interface only for now: https://github.com/Extrawurst/aws-sdk-dlang-gen Its a start, not tested and does not come with any dependency to vibe or what. AWS specific request signing also needs to be done on top of it. but using that tool the types can be automatically generated. --Stephan
Thanks, Stephan! I'm playing with it, but it will take some time. I'm also looking into some other bindings, both generated from descriptions and not. I have some experience with s3 and ec2 using boto library, but generate library code, tie it with http, auth, e.t.c is something different.
Oct 20
prev sibling parent aberba <karabutaworld gmail.com> writes:
On Wednesday, 18 October 2017 at 23:02:27 UTC, Stephan Dilly 
wrote:
 On 2017-10-18 22:25:23 +0000, ikod said:

 On Wednesday, 18 October 2017 at 20:51:48 UTC, Stephan Dilly 
 wrote:
 On 2017-10-18 20:19:20 +0000, ikod said:
 
  [...]
Are you going to take over the task? --Stephan
I'd like, but It depends on the required time investment. I need some time to look at the docs etc... Regards, Igor
Hi Igor, I started a little tool (POC) that can parse the API spec and creates types and the API interface only for now: https://github.com/Extrawurst/aws-sdk-dlang-gen Its a start, not tested and does not come with any dependency to vibe or what. AWS specific request signing also needs to be done on top of it. but using that tool the types can be automatically generated. --Stephan
I'm really happy about this.
Oct 28