www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - help needed to bring Windows SecureChannel to dlang-requests

reply ikod <igor.khasilev gmail.com> writes:
Hello,

It's probably time to ask for help with long standing problem 
with SSL support for dlang-requests under Windows.

dlang requests is HTTP/FTP client library, inspired by 
python-requests. For long time it have support for ssl using 
dynamic load of openssl library version 1.0 or 1.1 (whatever user 
have in system). This works fine under Linux and OSX, but not 
under Windows.

The problem: in Windows to use OpenSSL you have to find, download 
and install openssl package from some thirtd party site. Library 
user is not always have enough skills to perform every step 
successfully, and what is worse - dlang requests have not enough 
skills to find installed library at runtime, so sometimes it 
succeed with SSL under Windows and sometimes - not.

I'd like to fix this problem using Windows native SecureChannel, 
but I know too little about windows programming and even less 
about secure channel. I need a person  skilled in both topics, 
who I can ask questions. I don't ask to write code for me, I just 
need some expertise.

If you feel you can help - please ping me on the project page - 
https://github.com/ikod/dlang-requests or by email igor.khasilev 
at gmail.com

Thanks!
Aug 23 2020
next sibling parent mw <mingwu gmail.com> writes:
On Monday, 24 August 2020 at 04:06:27 UTC, ikod wrote:

 Hello,

 It's probably time to ask for help with long standing problem 
 with SSL support for dlang-requests under Windows.


Since https://www.openssl.org/ is also open source, just 
wondering if you can link to static OpenSSL library (i.e directly 
provide the lib in 
https://github.com/ikod/dlang-requests/tree/master/lib, I saw 
win-i386 there already)?
Aug 23 2020
prev sibling next sibling parent reply novice3 <sorryno em.ail> writes:
On Monday, 24 August 2020 at 04:06:27 UTC, ikod wrote:

 The problem: in Windows to use OpenSSL you have to find, 
 download and install openssl package from some thirtd party


may be, link openssl statically?
Aug 23 2020
parent reply ikod <igor.khasilev gmail.com> writes:
On Monday, 24 August 2020 at 05:29:25 UTC, novice3 wrote:

 On Monday, 24 August 2020 at 04:06:27 UTC, ikod wrote:

 The problem: in Windows to use OpenSSL you have to find, 
 download and install openssl package from some thirtd party


 may be, link openssl statically?


Thanks for your reply!

I use dynamic linking because at OpenSSL have interface 
incompatibilities between version 1.0 and 1.1, and at the time of 
transition between these versions there were no clean and 
reliable way for dub to detect which version installed in system 
(so I was unable to select proper openssl calls at compile time). 
So I decided to load library dynamicaly and check which 
interfaces available during runtime. Here is details: 
https://github.com/ikod/dlang-requests/issues/45

Also, will static linking work reliable? I'm open to return back 
to static linking if this can be confirmed: 1) static linking 
will work ok for third-party packages installed on Windows, 2) 
with dub I can at compile time detect openssl version installed 
in system
Aug 23 2020
next sibling parent reply mw <mingwu gmail.com> writes:
On Monday, 24 August 2020 at 06:28:26 UTC, ikod wrote:

 Also, will static linking work reliable? I'm open to return 
 back to static linking if this can be confirmed: 1) static


static link means the generated exe will physically contain all 
the binary code which is pulled in from the static OpenSSL lib, 
and the exe will not depends on any other external files.

E.g. if the exe is linked with 1.1 OpenSSL static lib, even the 
user's machine only has 1.0 OpenSSL dll, it has nothing to do 
with the generated exe, won't affect its running.


 linking will work ok for third-party packages installed on 
 Windows, 2) with dub I can at compile time detect openssl 
 version installed in system
Aug 24 2020
parent reply ikod <igor.khasilev gmail.com> writes:
On Monday, 24 August 2020 at 07:11:14 UTC, mw wrote:

 On Monday, 24 August 2020 at 06:28:26 UTC, ikod wrote:

 Also, will static linking work reliable? I'm open to return 
 back to static linking if this can be confirmed: 1) static


 static link means the generated exe will physically contain all 
 the binary code which is pulled in from the static OpenSSL lib, 
 and the exe will not depends on any other external files.

 E.g. if the exe is linked with 1.1 OpenSSL static lib, even the 
 user's machine only has 1.0 OpenSSL dll, it has nothing to do 
 with the generated exe, won't affect its running.


 linking will work ok for third-party packages installed on 
 Windows, 2) with dub I can at compile time detect openssl 
 version installed in system




Yes, but I'm not distribute .exe, I'm distribute sources of the 
library which have to be compiled on the target machine.

Or I'm missed something?
Aug 24 2020
parent reply mw <mingwu gmail.com> writes:
On Monday, 24 August 2020 at 07:22:45 UTC, ikod wrote:

 On Monday, 24 August 2020 at 07:11:14 UTC, mw wrote:

 On Monday, 24 August 2020 at 06:28:26 UTC, ikod wrote:

 Also, will static linking work reliable? I'm open to return 
 back to static linking if this can be confirmed: 1) static


 static link means the generated exe will physically contain 
 all the binary code which is pulled in from the static OpenSSL 
 lib, and the exe will not depends on any other external files.

 E.g. if the exe is linked with 1.1 OpenSSL static lib, even 
 the user's machine only has 1.0 OpenSSL dll, it has nothing to 
 do with the generated exe, won't affect its running.


 linking will work ok for third-party packages installed on 
 Windows, 2) with dub I can at compile time detect openssl 
 version installed in system




 Yes, but I'm not distribute .exe, I'm distribute sources of the 
 library which have to be compiled on the target machine.


Just tell your library user also use static linking (and with the 
OpenSSL version you provided).

Actually I don't know what your worry is.
Aug 24 2020
parent ikod <igor.khasilev gmail.com> writes:
On Monday, 24 August 2020 at 07:27:14 UTC, mw wrote:

 On Monday, 24 August 2020 at 07:22:45 UTC, ikod wrote:

 On Monday, 24 August 2020 at 07:11:14 UTC, mw wrote:

 On Monday, 24 August 2020 at 06:28:26 UTC, ikod wrote:

 Also, will static linking work reliable? I'm open to return








...

 the library which have to be compiled on the target machine.


 Just tell your library user also use static linking (and with 
 the OpenSSL version you provided).

 Actually I don't know what your worry is.


Here is list of issues for windows + ssl
https://github.com/ikod/dlang-requests/issues/127
https://github.com/ikod/dlang-requests/issues/121 - altough 
solved by experienced user
https://github.com/ikod/dlang-requests/issues/52

I'd be really happy if this problem is not real and I'll forget 
about it.
Aug 24 2020
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On Monday, 24 August 2020 at 06:28:26 UTC, ikod wrote:


 I use dynamic linking because at OpenSSL have interface 
 incompatibilities between version 1.0 and 1.1, and at the time 
 of transition between these versions there were no clean and 
 reliable way for dub to detect which version installed in 
 system (so I was unable to select proper openssl calls at 
 compile time). So I decided to load library dynamicaly and 
 check which interfaces available during runtime. Here is 
 details: https://github.com/ikod/dlang-requests/issues/45

 Also, will static linking work reliable? I'm open to return 
 back to static linking if this can be confirmed: 1) static 
 linking will work ok for third-party packages installed on 
 Windows, 2) with dub I can at compile time detect openssl 
 version installed in system


If you provide an OpenSSL library in the Git repository you know 
exactly which version it is and which interface it provides.

--
/Jacob Carlborg
Aug 24 2020
prev sibling parent Jacob Carlborg <doob me.com> writes:
On Monday, 24 August 2020 at 06:28:26 UTC, ikod wrote:


 Also, will static linking work reliable? I'm open to return 
 back to static linking if this can be confirmed: 1) static 
 linking will work ok for third-party packages installed on 
 Windows,


I can add that static linking is required on Linux if you want to 
fully statically link the binary. `dlopen` is not supported on 
iOS and OpenSSL doesn't ship on iOS, IIRC.


 2) with dub I can at compile time detect openssl version 
 installed in system


Dub can execute arbitrary code at various stages during the 
build. Here's an example of DStep identifying libclang [1] [2].

[1] 
https://github.com/jacob-carlborg/dstep/blob/d355dec2d57e11459e938dddcb1f6dab34ef124b/dub.json#L34-L36

[2] 
https://github.com/jacob-carlborg/dstep/blob/master/configure.d

--
/Jacob Carlborg
Aug 24 2020
prev sibling next sibling parent Andre Pany <andre s-e-a-p.de> writes:
On Monday, 24 August 2020 at 04:06:27 UTC, ikod wrote:

 Hello,

 It's probably time to ask for help with long standing problem 
 with SSL support for dlang-requests under Windows.

 dlang requests is HTTP/FTP client library, inspired by 
 python-requests. For long time it have support for ssl using 
 dynamic load of openssl library version 1.0 or 1.1 (whatever 
 user have in system). This works fine under Linux and OSX, but 
 not under Windows.

 The problem: in Windows to use OpenSSL you have to find, 
 download and install openssl package from some thirtd party 
 site. Library user is not always have enough skills to perform 
 every step successfully, and what is worse - dlang requests 
 have not enough skills to find installed library at runtime, so 
 sometimes it succeed with SSL under Windows and sometimes - not.

 I'd like to fix this problem using Windows native 
 SecureChannel, but I know too little about windows programming 
 and even less about secure channel. I need a person  skilled in 
 both topics, who I can ask questions. I don't ask to write code 
 for me, I just need some expertise.

 If you feel you can help - please ping me on the project page - 
 https://github.com/ikod/dlang-requests or by email 
 igor.khasilev at gmail.com

 Thanks!


 From a security point of view this is fantastic. Because in case 
of a security issue, Microsoft will solve it automatically via 
the windows updates.

Security issues in OpenSSL causes the developers to distribute 
new software packages to their customers.

SecureChannel would be a huge improvement. Maybe a 
recommendation, maybe you can join forces with vibe-d developers 
on this topic?

Kind regards
Andre
Aug 24 2020
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On Monday, 24 August 2020 at 04:06:27 UTC, ikod wrote:


 dlang requests is HTTP/FTP client library, inspired by 
 python-requests. For long time it have support for ssl using 
 dynamic load of openssl library version 1.0 or 1.1 (whatever 
 user have in system). This works fine under Linux and OSX, but 
 not under Windows.


It's the same problem on macOS. Although, OpenSSL ships on macOS, 
it's been deprecated since at least nine major versions ago. The 
recommendation is if you want to use OpenSSL you should ship it 
with the application. One major reason being it's not ABI 
compatible between even minor versions. IIRC, OpenSSL doesn't 
ship on iOS at all.

I would love a version on macOS that uses platform provided 
TLS/SSL implementation.

--
/Jacob Carlborg
Aug 24 2020
parent reply ikod <igor.khasilev gmail.com> writes:
On Monday, 24 August 2020 at 09:57:16 UTC, Jacob Carlborg wrote:

 On Monday, 24 August 2020 at 04:06:27 UTC, ikod wrote:


 dlang requests is HTTP/FTP client library, inspired by 
 python-requests. For long time it have support for ssl using 
 dynamic load of openssl library version 1.0 or 1.1 (whatever 
 user have in system). This works fine under Linux and OSX, but 
 not under Windows.


 It's the same problem on macOS. Although, OpenSSL ships on 
 macOS, it's been deprecated since at least nine major versions 
 ago. The recommendation is if you want to use OpenSSL you 
 should ship it with the application. One major reason being 
 it's not ABI compatible between even minor versions. IIRC, 
 OpenSSL doesn't ship on iOS at all.

 I would love a version on macOS that uses platform provided 
 TLS/SSL implementation.


I think this is possible (thanks for reminding!), I have no 
problems with access and programming under OSX.

re your next message about linking with distributed library - I 
really dont want to distribute anything in binary form 
(especially security libs). I included these files as last resort 
for user.


 --
 /Jacob Carlborg
Aug 24 2020
parent Jacob Carlborg <doob me.com> writes:
On Monday, 24 August 2020 at 10:11:25 UTC, ikod wrote:


 I think this is possible (thanks for reminding!), I have no 
 problems with access and programming under OSX.


Cool. I think we have discussed this before, but unfortunately 
the Secure Transport library (which is the closest corresponding 
thing to OpenSSL) has been deprecated. The recommended 
alternative is the new Network framework. But that is only 
supported on macOS 10.15 and later (which is too recent). I think 
the best bet is the CFNetwork framework. It's higher level than 
Secure Transport or Network, it will handle both the encryption 
and the transport. It's supported since macOS 10.8 (really old) 
and iOS 2.0 (really old) and will take care of any underlying 
differences in the platform. I'm guessing this will require quite 
a lot of work to implement since you'll not work directly with 
sockets. Not sure if it's flexible enough to handle everything 
that your library can do.


 re your next message about linking with distributed library - I 
 really dont want to distribute anything in binary form 
 (especially security libs). I included these files as last 
 resort for user.


Fair enough.

--
/Jacob Carlborg
Aug 24 2020
prev sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Monday, 24 August 2020 at 04:06:27 UTC, ikod wrote:

 I'd like to fix this problem using Windows native 
 SecureChannel, but I know too little about windows programming 
 and even less about secure channel.


Aah, I want to do this for my http lib too. I know Windows fairly 
well, but not the SecureChannel api. So it has been sitting as a 
FIXME list of links for a long time now.

If you see a good example of it let me know and I'll post here if 
I can figure it out...
Aug 24 2020
parent Kagamin <spam here.lot> writes:
On Monday, 24 August 2020 at 12:13:07 UTC, Adam D. Ruppe wrote:

 If you see a good example of it let me know and I'll post here 
 if I can figure it out...


curl?
Aug 25 2020