www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Payload Details with std.net.curl:post

reply Kyle Ingraham <kyle kyleingraham.com> writes:
Happy Saturday everyone. I am using `std.net.curl:post` to 
request an OAuth access token from a Google API. Initially I was 
getting a 400 status code back but could not tell why. I used the 
`verbose` option on `HTTP()` to get more details:

```D
char[] tokenResponse;
auto connection = HTTP();
connection.verbose = true;

try
{
     tokenResponse = post
     (
     tokenRequestEndpoint,
     ["code": incomingCode, "redirect_uri": 
tokenRequestRedirectUrl,
      "grant_type": "authorization_code", "code_verifier": 
to!string(codeVerifier),
      "client_id": clientId, "client_secret": clientSecret],
     connection
     );
}
catch(HTTPStatusException e)
{
     writefln("Token request failed - Status code: %d - Message: 
%s\n", e.status, e.msg);
     return 1;
}
```

That `verbose` option gets me a detailed log but I cannot see the 
payload of my request nor that of the response:

```
*   Trying 172.217.165.10:443...
* TCP_NODELAY set


 POST /oauth2/v4/token HTTP/1.1


Host: www.googleapis.com
User-Agent: Phobos-std.net.curl/2.098 (libcurl/7.68.0)
Accept: 
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Content-Length: 16
Content-Type: application/x-www-form-urlencoded

* We are completely uploaded and fine
* Mark bundle as not supporting multiuse
< HTTP/1.1 400 Bad Request
< Content-Type: application/json; charset=utf-8
< Vary: X-Origin
< Vary: Referer
< Date: Sat, 27 Nov 2021 14:24:45 GMT
< Server: scaffolding on HTTPServer2
< Cache-Control: private
< X-XSS-Protection: 0
< X-Frame-Options: SAMEORIGIN
< X-Content-Type-Options: nosniff
< Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; 
ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; 
ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; 
v="46,43"
< Accept-Ranges: none
< Vary: Origin,Accept-Encoding
< Transfer-Encoding: chunked
<

```

Is there a way to see that information? Google's API does not 
seem to provide much more than a status code as to the reason for 
the failure.
Nov 27 2021
parent reply frame <frame86 live.com> writes:
On Saturday, 27 November 2021 at 17:41:55 UTC, Kyle Ingraham 
wrote:


 Is there a way to see that information? Google's API does not 
 seem to provide much more than a status code as to the reason 
 for the failure.


No, and it's a shame that we cannot use CurlOption.debugfunction 
because the Curl struct on HTTP is private and the only way to 
set this option is to use a raw Curl instance.

You are on your own working with onReceive(), setPostData() and 
std.uri.urlEncode() to encode your post data.
Nov 27 2021
parent reply Kyle Ingraham <kyle kyleingraham.com> writes:
On Saturday, 27 November 2021 at 19:21:28 UTC, frame wrote:

 On Saturday, 27 November 2021 at 17:41:55 UTC, Kyle Ingraham 
 wrote:


 Is there a way to see that information? Google's API does not 
 seem to provide much more than a status code as to the reason 
 for the failure.


 No, and it's a shame that we cannot use 
 CurlOption.debugfunction because the Curl struct on HTTP is 
 private and the only way to set this option is to use a raw 
 Curl instance.

 You are on your own working with onReceive(), setPostData() and 
 std.uri.urlEncode() to encode your post data.


I was afraid of that. Thank you for clarifying. I’ve tried other 
clients but they all seem to have trouble verifying HTTPS 
connections.

I’ll see about debugging with curl outside of D or go the manual 
route like you suggested.
Nov 27 2021
parent reply ikod <igor.khasilev gmail.com> writes:
On Saturday, 27 November 2021 at 20:31:16 UTC, Kyle Ingraham 
wrote:

 On Saturday, 27 November 2021 at 19:21:28 UTC, frame wrote:

 On Saturday, 27 November 2021 at 17:41:55 UTC, Kyle Ingraham 
 wrote:


 I was afraid of that. Thank you for clarifying. I’ve tried 
 other clients but they all seem to have trouble verifying HTTPS 
 connections.


Which clients did you try, and what the troubles they have?
Nov 27 2021
parent reply Kyle Ingraham <kyle kyleingraham.com> writes:
On Saturday, 27 November 2021 at 22:18:48 UTC, ikod wrote:

 On Saturday, 27 November 2021 at 20:31:16 UTC, Kyle Ingraham 
 wrote:

 On Saturday, 27 November 2021 at 19:21:28 UTC, frame wrote:

 On Saturday, 27 November 2021 at 17:41:55 UTC, Kyle Ingraham 
 wrote:


 I was afraid of that. Thank you for clarifying. I’ve tried 
 other clients but they all seem to have trouble verifying 
 HTTPS connections.


 Which clients did you try, and what the troubles they have?


Hi ikod. I tried `requests` on Windows 10 (Build 19042) and got 
the following exception:

```
object.Exception C:\Users\Kyle 
Ingraham\AppData\Local\dub\packages\requests-2.0.2\requests\source\requ
sts\streams.d(890): ssl connect failed: certificate verify failed
```

I checked the issues page for `hunt-http` and saw that there were 
similar reports there from Windows users (I have not verified 
them myself). I did not try `vibe-http` because the callback 
interface in the docs wasn't one that grabbed me.

I would have dug further into the solution for the `requests` 
exception as I like the interface it provides but I was eager to 
power through the OAuth bit of my prototyping so that I could 
experiment with the Google Photos API.

Do you have tips on why I might be getting that exception? I'm 
guessing I need to point `requests` at a trust store.
Nov 27 2021
parent reply ikod <igor.khasilev gmail.com> writes:
On Sunday, 28 November 2021 at 01:06:45 UTC, Kyle Ingraham wrote:

 On Saturday, 27 November 2021 at 22:18:48 UTC, ikod wrote:

 On Saturday, 27 November 2021 at 20:31:16 UTC, Kyle Ingraham





Hi Kyle,



 ```
 object.Exception C:\Users\Kyle 
 Ingraham\AppData\Local\dub\packages\requests-2.0.2\requests\source\requ
sts\streams.d(890): ssl connect failed: certificate verify failed
 ```

 I checked the issues page for `hunt-http` and saw that there 
 were similar reports there from Windows users (I have not 
 verified them myself). I did not try `vibe-http` because the 
 callback interface in the docs wasn't one that grabbed me.


 Do you have tips on why I might be getting that exception? I'm 
 guessing I need to point `requests` at a trust store.



Yes, trust store location can be the problem. I hope you checked 
this readme sections: 
[windows-ssl-notes](https://github.com/ikod/dlang-requests#windows-ssl-notes)
and [ssl-settings](https://github.com/ikod/dlang-requests#ssl-settings).

It is difficult to answer without having more details, but I 
recently fixed a bug that could lead to a similar problem on 
32-bit systems (regardless of OS). If this is your case, then you 
can try to use the latest commit from the github. Anyway, it 
would be nice if you create an issue on github and post more 
details there, so that I can try to reproduce the problem.
Nov 27 2021
parent reply Kyle Ingraham <kyle kyleingraham.com> writes:
On Sunday, 28 November 2021 at 07:27:35 UTC, ikod wrote:

 On Sunday, 28 November 2021 at 01:06:45 UTC, Kyle Ingraham 
 wrote:

 On Saturday, 27 November 2021 at 22:18:48 UTC, ikod wrote:

 On Saturday, 27 November 2021 at 20:31:16 UTC, Kyle Ingraham





 Hi Kyle,



 ```
 object.Exception C:\Users\Kyle 
 Ingraham\AppData\Local\dub\packages\requests-2.0.2\requests\source\requ
sts\streams.d(890): ssl connect failed: certificate verify failed
 ```

 I checked the issues page for `hunt-http` and saw that there 
 were similar reports there from Windows users (I have not 
 verified them myself). I did not try `vibe-http` because the 
 callback interface in the docs wasn't one that grabbed me.


 Do you have tips on why I might be getting that exception? I'm 
 guessing I need to point `requests` at a trust store.



 Yes, trust store location can be the problem. I hope you 
 checked this readme sections: 
 [windows-ssl-notes](https://github.com/ikod/dlang-requests#windows-ssl-notes)
and [ssl-settings](https://github.com/ikod/dlang-requests#ssl-settings).

 It is difficult to answer without having more details, but I 
 recently fixed a bug that could lead to a similar problem on 
 32-bit systems (regardless of OS). If this is your case, then 
 you can try to use the latest commit from the github. Anyway, 
 it would be nice if you create an issue on github and post more 
 details there, so that I can try to reproduce the problem.


I had read those sections but couldn't figure out which 
combination of certificates and function calls would make a 
difference for my URL.

I found https://pki.goog/repository/ which has links to all 
certificates relevant to Google URLs. I worked my way down the 
certificate chain and the call that worked was:

```D
auto rq = Request();
rq.sslSetCaCert(r"gtsr1.pem");
```

That certificate is for one of Google's root CAs but not the root 
of the certificate chain. With that combination my request worked 
and the payload was delivered successfully. I was also able to 
see the payload that had been sent in my request which was my 
motivation for starting this thread in the first place.

I think my trouble here was mostly due to my limited familiarity 
with SSL verification. I ended up using a CA certificate and the 
method made available was `sslSetCaCert`. A bit obvious when I 
look back. I had not used it at first because it was referenced 
as being needed for vibe.d and when perusing `requests` I thought 
that it wasn't being called for my setup.

Glad to have a working setup now. Thank you ikod for stopping by 
and asking about my experiences outside of `std.net.curl`.

P.S. `openssl s_client -connect <URL>:443` was useful for viewing 
the certificate chain.
Nov 28 2021
parent ikod <igor.khasilev gmail.com> writes:
On Monday, 29 November 2021 at 01:49:37 UTC, Kyle Ingraham wrote:

 On Sunday, 28 November 2021 at 07:27:35 UTC, ikod wrote:

 On Sunday, 28 November 2021 at 01:06:45 UTC, Kyle Ingraham 
 wrote:

 On Saturday, 27 November 2021 at 22:18:48 UTC, ikod wrote:






Hi Kyle!


 I found https://pki.goog/repository/ which has links to all 
 certificates relevant to Google URLs. I worked my way down the 
 certificate chain and the call that worked was:

 ```D
 auto rq = Request();
 rq.sslSetCaCert(r"gtsr1.pem");
 ```


Nice to hear that you solved this problem! Openssl under windows 
sometimes require
extra efforts. I tried to add support for native windows 
SecureChannel API in `requests`, but got stuck somewhere in the 
middle of the process.
Nov 29 2021