www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - SecureD Futures (v2.0)

reply Adam Wilson <flyboynw gmail.com> writes:
Now that SecureD v1 is in the books I thought it would be worthwhile to 
explore what a second version could like. I specifically want to focus 
on expanding compatibility with other systems.

For example: AWS uses SHA2-256 for signing requests. As implemented 
today SecureD does not support this. However, this is one of the 
scenarios that falls under SecureD's mission of ease-to-use cryptography 
for non-transport layer security.

I am curious what people would like to see in SecureD moving forward and 
I would love to get your feedback on what works and what doesn't work in 
v1 of SecureD.

With this in mind here are my ideas for expanding SecureD's 
capabilities. I'll start with the idea that I think will be the least 
controversial.

1. Full support for SHA-2 and SHA-3 for Hashes and HMAC's.

This means the following modes would be supported:
SHA2: 224, 256, 384, 512, 512/224, 512/256
SHA3: 224, 256, 384, 512

The PBKDF2 implementation would also be updated to support the above 
hash functions.

Hash methods supporting salts will be added.

To maintain backwards compatibility with v1, methods that match the v1 
signatures and forward to the correct implementation would be provided 
as the v1 defaults are still secure.

This requires OpenSSL 1.1.1 at a minimum. Note that SHA3 support could 
be delayed until SecureD 2.1 if OpenSSL 1.1.1 is not available on a 
majority of systems prior to shipping 2.0. In such case OpenSSL 1.1.0 
would become the minimum supported version.

2. More Flexibility for Symmetric Encryption.

Currently SecureD only supports AES256-CTR-HMAC384 and the encrypted 
data is laid out as: HASH+IV+DATA. This is only useful in scenarios 
where the developer controls both the encryption and decryption of the 
data. Additionally, this does not support AEAD modes, only AE modes.

The first thing I would like to do is to support all AES, SHA2, and SHA3 

padding as that is the most common cipher mode in use.

One of the primary use cases for SecureD is long-term storage of data. 
To facilitate this requirement I would like to add a binary encoded 
header to each encrypted blob. This header would contain the necessary 
information to decode the blob (minus the key of course).

The header would cost 26 bytes per encrypted blob and would have the 
following layout:
struct cryptoHeader {
     ubyte hdrVersion;	// The version of the header
     ubyte encAlg;	// The encryption algorithm used
     ubyte hashAlg;	// The hash algorithm used
     uint  kdfIters;	// The number of PBKDF2 iterations
     ubyte authLen;	// The length of the authentication value
     ubyte saltLen;	// The length of the PBKDF2 salt
     ubyte ivLen;	// The length of the IV
     ulong encLen;	// The length of the encrypted data
     ulong adLen;	// The length of the additional data
}

The data layout would be: HEADER+AUTH+SALT+IV+ENC+AD
Any excluded elements would be marked as 0 in the header and completely 
excluded from the data layout.
The MAC would be computed as: HMAC(HEADER+SALT+IV+ENC+AD)

The full encryption process would be as follows:
- Create a PBKDF2 salt via RNG
- Create IV via RNG
- Create header from user inputs
- EncKey = PBKDF2 over RawKey unless hashAlg is None or kdfIters == 0
   - Use RawKey as-is if hashAlg is None or kdfIters == 0
- MacKey = PBKDF2 over EncKey once using same inputs as EncKey
- Encrypt data
- Compute HMAC(HEADER+SALT+IV+ENC+AD)
- Return RESULT(HEADER+AUTH+SALT+IV+ENC+AD)

Methods would be added to work with the following:
- Simple encrypted blobs (no header/auth/salt/iv/ad included)
- Creating and Verifying AE/AD MACs without headers
- Methods to read SecureD v1 encrypted data

Note that this plan is NOT compatible with the SecureD v1 encryption 
data layout and data would need to be re-encrypted.

3. Re-implement the OpenSSL Bindings

The Deimos/DUB bindings are ancient and do not support OpenSSL 1.1.x. We 
can re-implement the bindings taking care to only include the symbols we 
need.

4. Streams

I get asked about this, and, as much as I want to do this, it is not a 
simple topic. std.streams was removed and right now the only viable 
replacement is Vibe.D streams. But that means pulling in Vibe.D for a 
simple cryptography library. At this point that doesn't seem like the 
right idea. If someone is willing to step-up and do the work I'd be 
willing to look at it, but for now I want to wait on this, preferably 
for a standard/generic streams interface to be made available.

Please let know what you think! I am very interested to hear about what 
would make your life easier when working with SecureD an cryptography in 
general.

-- 
Adam Wilson
IRC: LightBender
import quiet.dlang.dev;
May 27 2018
next sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On Sunday, 27 May 2018 at 10:27:45 UTC, Adam Wilson wrote:

 Now that SecureD v1 is in the books I thought it would be 
 worthwhile to explore what a second version could like. I 
 specifically want to focus on expanding compatibility with 
 other systems.

 [...]



No, it’s not. Look at IOpipe and no further it provides the exact 
abstraction that works for any form buffered I/O doing stream 
processing at top speeds. I’m doing IOpipe regex, it looks to be 
just what the doctor ordered.


  But that means pulling in Vibe.D for a simple cryptography 
 library. At this point that doesn't seem like the right idea. 
 If someone is willing to step-up and do the work I'd be willing 
 to look at it, but for now I want to wait on this, preferably 
 for a standard/generic streams interface to be made available.

 [...]
May 27 2018
prev sibling next sibling parent reply Neia Neutuladh <neia ikeran.org> writes:
On Sunday, 27 May 2018 at 10:27:45 UTC, Adam Wilson wrote:

 Now that SecureD v1 is in the books


This would have been a great place to insert a brief description 
of what SecureD is or a link to the project.
May 27 2018
parent Adam Wilson <flyboynw gmail.com> writes:
On 05/27/2018 09:54 AM, Neia Neutuladh wrote:

 On Sunday, 27 May 2018 at 10:27:45 UTC, Adam Wilson wrote:

 Now that SecureD v1 is in the books


 
 This would have been a great place to insert a brief description of what 
 SecureD is or a link to the project.


Good point. SecureD is a cryptography library that is designed to make 
non transport-layer cryptography simple to use. I created it after 
working with OpenSSL and observing that the API design of OpenSSL 
actively encourages broken implementations. Think of it as a 
cryptography library for the rest of us.

You can read more about it here: http://code.dlang.org/packages/secured

-- 
Adam Wilson
IRC: LightBender
import quiet.dlang.dev;
May 27 2018
prev sibling parent reply sarn <sarn theartofmachinery.com> writes:
On Sunday, 27 May 2018 at 10:27:45 UTC, Adam Wilson wrote:

 struct cryptoHeader {
     ubyte hdrVersion;	// The version of the header
     ubyte encAlg;	// The encryption algorithm used
     ubyte hashAlg;	// The hash algorithm used
     uint  kdfIters;	// The number of PBKDF2 iterations
     ubyte authLen;	// The length of the authentication value
     ubyte saltLen;	// The length of the PBKDF2 salt
     ubyte ivLen;	// The length of the IV
     ulong encLen;	// The length of the encrypted data
     ulong adLen;	// The length of the additional data
 }


Should there be any kind of key identifier, or do you consider 
that a separate issue?


 - MacKey = PBKDF2 over EncKey once using same inputs as EncKey


How about this?

Use PBKDF2 to generate a key-derivation-key (KDK), then use 
HKDF-Expand (https://tools.ietf.org/html/rfc5869) to derive the 
encryption key and MAC key separately.

I think that's more standard.  I don't know how much it matters 
in practice, but a lot of cryptographers don't like generating 
MAC/encryption keys from each other directly.

Also, it's probably safer to use HKDF-Extract instead of using 
raw keys directly when not using PBKDF2.
May 27 2018
parent reply Adam Wilson <flyboynw gmail.com> writes:
On 05/27/2018 05:11 PM, sarn wrote:

 On Sunday, 27 May 2018 at 10:27:45 UTC, Adam Wilson wrote:

 struct cryptoHeader {
     ubyte hdrVersion;    // The version of the header
     ubyte encAlg;    // The encryption algorithm used
     ubyte hashAlg;    // The hash algorithm used
     uint  kdfIters;    // The number of PBKDF2 iterations
     ubyte authLen;    // The length of the authentication value
     ubyte saltLen;    // The length of the PBKDF2 salt
     ubyte ivLen;    // The length of the IV
     ulong encLen;    // The length of the encrypted data
     ulong adLen;    // The length of the additional data
 }


 
 Should there be any kind of key identifier, or do you consider that a 
 separate issue?
 


hashAlg is used for everything related to key derivation. Salts and IVs 
are random.


 - MacKey = PBKDF2 over EncKey once using same inputs as EncKey


 
 How about this?
 
 Use PBKDF2 to generate a key-derivation-key (KDK), then use HKDF-Expand 
 (https://tools.ietf.org/html/rfc5869) to derive the encryption key and 
 MAC key separately.
 
 I think that's more standard.  I don't know how much it matters in 
 practice, but a lot of cryptographers don't like generating 
 MAC/encryption keys from each other directly.
 


I like it. But it does require more space. We need three salts and three 
lengths in the header. One for the PBKDF2 KDK, one for the MAC key, and 
one for the encryption key.

Something like this:
struct cryptoHeader {
     ubyte hdrVersion;    // The version of the header
     ubyte encAlg;        // The encryption algorithm used
     ubyte hashAlg;       // The hash algorithm used
     uint kdfIters;       // The number of PBKDF2 iterations

     ubyte kdkSLen;       // The length of the KDK Salt
     ubyte macSLen;       // The length of the MAC Salt
     ubyte keySLen;       // The length of the KEY Salt
     ubyte ivLen;         // The length of the IV
     ulong encLen;        // The length of the encrypted data
     ulong adLen;         // The length of the additional data
     ubyte authLen;       // The length of the authentication value
}


 Also, it's probably safer to use HKDF-Extract instead of using raw keys 
 directly when not using PBKDF2.
 


It is. And we could dot that if you stick to the default encrypt/decrypt 
routines. I want to expand the symmetric capabilities of SecureD so I am 
going to rebuild the encrypt/decrypt routines as a composition of 
smaller methods. That will allow users to either use the default 
encryption (simple) or if they have to interoperate with other systems, 
use the component methods to perform their operations. I am not looking 
to cover all scenarios though, just the common ones. If you have 
something truly unusual, you'll still need to drop down the base crypto 
libraries.

-- 
Adam Wilson
IRC: LightBender
import quiet.dlang.dev;
May 27 2018
parent reply sarn <sarn theartofmachinery.com> writes:
On Monday, 28 May 2018 at 02:25:20 UTC, Adam Wilson wrote:

 I like it. But it does require more space. We need three salts 
 and three lengths in the header. One for the PBKDF2 KDK, one 
 for the MAC key, and one for the encryption key.


HKDF-Expand doesn't need a salt.  You just need one salt to make 
the KDK (whether you use PBKDF2 or HKDF-Extract for that) and no 
extra salts for deriving the encryption and MAC key.
May 27 2018
parent reply Adam Wilson <flyboynw gmail.com> writes:
On 05/27/2018 08:52 PM, sarn wrote:

 On Monday, 28 May 2018 at 02:25:20 UTC, Adam Wilson wrote:

 I like it. But it does require more space. We need three salts and 
 three lengths in the header. One for the PBKDF2 KDK, one for the MAC 
 key, and one for the encryption key.


 
 HKDF-Expand doesn't need a salt.  You just need one salt to make the KDK 
 (whether you use PBKDF2 or HKDF-Extract for that) and no extra salts for 
 deriving the encryption and MAC key.


Strictly speaking, it's is Optional but Strongly Recommended per RFC5869-3.1

The use case here is that this data is going into storage and that 
storage is cheap. We don't have to be strict on our byte budget. :)

https://tools.ietf.org/html/rfc5869

https://www.openssl.org/docs/man1.1.0/crypto/EVP_PKEY_CTX_set_hkdf_md.html

SecureD is supposed to be "Crypto done right." So I might as well do it 
right and follow the RFC.

-- 
Adam Wilson
IRC: LightBender
import quiet.dlang.dev;
May 27 2018
parent reply sarn <sarn theartofmachinery.com> writes:
On Monday, 28 May 2018 at 06:22:02 UTC, Adam Wilson wrote:

 On 05/27/2018 08:52 PM, sarn wrote:

 On Monday, 28 May 2018 at 02:25:20 UTC, Adam Wilson wrote:

 I like it. But it does require more space. We need three 
 salts and three lengths in the header. One for the PBKDF2 
 KDK, one for the MAC key, and one for the encryption key.


 
 HKDF-Expand doesn't need a salt.  You just need one salt to 
 make the KDK (whether you use PBKDF2 or HKDF-Extract for that) 
 and no extra salts for deriving the encryption and MAC key.


 Strictly speaking, it's is Optional but Strongly Recommended 
 per RFC5869-3.1


There's HKDF-Expand and there's HKDF-Extract.  HKDF-Extract takes 
an optional salt and HKDF-Expand doesn't use a salt at all (take 
a closer look at that RFC).  That OpenSSL routine is the 
combination of the two, so that's why it takes the salt.
May 28 2018
parent reply Adam Wilson <flyboynw gmail.com> writes:
On 05/28/2018 12:14 AM, sarn wrote:

 On Monday, 28 May 2018 at 06:22:02 UTC, Adam Wilson wrote:

 On 05/27/2018 08:52 PM, sarn wrote:

 On Monday, 28 May 2018 at 02:25:20 UTC, Adam Wilson wrote:

 I like it. But it does require more space. We need three salts and 
 three lengths in the header. One for the PBKDF2 KDK, one for the MAC 
 key, and one for the encryption key.


 HKDF-Expand doesn't need a salt.  You just need one salt to make the 
 KDK (whether you use PBKDF2 or HKDF-Extract for that) and no extra 
 salts for deriving the encryption and MAC key.


 Strictly speaking, it's is Optional but Strongly Recommended per 
 RFC5869-3.1


 
 There's HKDF-Expand and there's HKDF-Extract.  HKDF-Extract takes an 
 optional salt and HKDF-Expand doesn't use a salt at all (take a closer 
 look at that RFC).  That OpenSSL routine is the combination of the two, 
 so that's why it takes the salt.


I understand that. But the way I read the first paragraph of 3.1 is that 
yes, it can work without a Salt, but HKDF, in general, is strongly 
suggested to be used with a salt. If we are really concerned about bytes 
we could reuse the salt for all three steps (KDK/MAC/KEY), but TBH, I'm 
not worried about disk usage.

Let me ask the question a different way. What is the reason NOT to use 2 
different salts for the MAC and KEY generation steps?

-- 
Adam Wilson
IRC: LightBender
import quiet.dlang.dev;
May 28 2018
parent reply sarn <sarn theartofmachinery.com> writes:
On Monday, 28 May 2018 at 07:52:43 UTC, Adam Wilson wrote:

 I understand that.


Sorry, not for nothing, but you obviously don't.  For starters, 
if you were familiar with the key derivation tools available 
24hrs ago, you wouldn't have come up with PBKDF2 on PBKDF2.  I 
suggest slowing down a little, and asking people on a crypto 
forum if you're still not sure.  If you explain your problem from 
the start, they might even have some better ideas.

When that RFC (correctly) recommends using a salt, it's talking 
about HKDF-Extract, which is a tool for taking a large amount of 
mostly-random data and turning it into an appropriate KDK.  
That's not the problem you have here.  The problem you have is 
generating keys for different purposes from a KDK.  That's a 
problem HKDF-Expand addresses, and it doesn't use a salt.


 Let me ask the question a different way. What is the reason NOT 
 to use 2 different salts for the MAC and KEY generation steps?


You might choose to not use extra salts for the same reason 
you've already chosen to not encrypt three times, or add extra 
HMACs for each individual block of ciphertext: it's not solving 
any problems.
May 28 2018
parent reply Adam Wilson <flyboynw gmail.com> writes:
On 05/28/2018 04:02 PM, sarn wrote:

 On Monday, 28 May 2018 at 07:52:43 UTC, Adam Wilson wrote:

 I understand that.


 
 Sorry, not for nothing, but you obviously don't.  For starters, if you 
 were familiar with the key derivation tools available 24hrs ago, you 
 wouldn't have come up with PBKDF2 on PBKDF2.  I suggest slowing down a 
 little, and asking people on a crypto forum if you're still not sure.  
 If you explain your problem from the start, they might even have some 
 better ideas.
 
 When that RFC (correctly) recommends using a salt, it's talking about 
 HKDF-Extract, which is a tool for taking a large amount of mostly-random 
 data and turning it into an appropriate KDK. That's not the problem you 
 have here.  The problem you have is generating keys for different 
 purposes from a KDK.  That's a problem HKDF-Expand addresses, and it 
 doesn't use a salt.
 

 Let me ask the question a different way. What is the reason NOT to use 
 2 different salts for the MAC and KEY generation steps?


 
 You might choose to not use extra salts for the same reason you've 
 already chosen to not encrypt three times, or add extra HMACs for each 
 individual block of ciphertext: it's not solving any problems.


Ok, but I am not harming anything either. I asked specifically what 
would be reasons not to do something, you replied with "it's not solving 
any problems". In a technical sense you are correct. I am not solving 
any problems. However. I am not creating any either. Adding a salt to an 
HKDF does not hurt anything. Ever.

As to the PBKDF2 comment. I'll concede that I am not perfect. Which is 
why I asked for input and I obviously "slowed down" enough to bother to 
ask. And when my mistake was pointed out I corrected it without further 
comment. We make all make mistakes sometimes, and cryptography is much 
harder than most to get right. So when we aren't sure we ask. And my 
statement was posed as a question. I was inviting feedback.

One of the pillars of SecureD is that ONLY safe, well-known, algorithms 
are presented. If reasonable we will only present one algorithm for a 
specific purpose. If there is a good reason to add more than one 
algorithm, we will.

For example, I am well aware of the BCrypt/SCrypt family as well as 
Argon2. At this point Argon2 appears to be the heir-apparent. Sadly, 
Argon2 is so new that neither Botan or OpenSSL implement it. So that one 
is ruled out for practical purposes, at least for now. I would also like 
to point out the OpenSSL will never implement BCrypt, but does implement 
SCrypt, and an Argon2 implementation is an open question due to it's use 
of threads.

It turns out the key-stretching field is very active right now, and 
precisely what will become the de-facto standard is impossible to say. I 
don't feel comfortable trying to pick a winner to implement. For the 
moment, PBKDF2 is stable, broadly implemented, and most importantly, 
well-understood. SecureD v2 will default to 1,000,000 iterations which 
yields roughly 650ms on an Intel 8700K. As older hardware is unlikely to 
be as fast, times of over one second could be seen in production while 
still providing some buffer against the future. That said the default 
PBKDF2 method will be tune-able.

I have long hoped for a reasonable replacement for PBKDF2, and with all 
the recent activity it is likely that something will shake out. But the 
attacks against Argon2 coming out so soon after it's release do not bode 
well for it in the future. And SCrypt requiring 16MB of RAM to achieve 
BCrypt's security makes it challenging to recommend for use on servers, 
especially in today's cloud environments where memory costs money. I 
know PBKDF2 is the conservative choice, but SecureD is all about 
conservative. For the moment, there aren't any great options, so 
conservative wins by default.

SCrypt Memory Math:
Assuming a 4GB VM in the cloud and we are willing to dedicate 1GB to 
SCrypt per second at BCrypt equivalent security.
1024/16=64
That's 64 connections per second, that is ... not great. And dedicating 
1/4 of your systems RAM just for password hashing is being very 
generous. You're not going to see that in production because it is 
terribly expensive.

As PBKDF2 and SCrypt are the only key-stretching algorithms available in 
both OpenSSL and Botan, and I cannot recommend SCrypt for cloud/server 
scenarios. PBKDF2 it is.

To be perfectly honest, key-stretching is a mitigation. Weak passwords 
will be easily guessed no matter what algorithm is used. The real fix is 
to force better passwords.

-- 
Adam Wilson
IRC: LightBender
import quiet.dlang.dev;
May 29 2018
parent reply Brad Roberts <braddr puremagic.com> writes:
On 5/29/2018 1:57 AM, Adam Wilson via Digitalmars-d wrote:

 One of the pillars of SecureD is that ONLY safe, well-known, 
 algorithms are presented. If reasonable we will only present one 
 algorithm for a specific purpose. If there is a good reason to add 
 more than one algorithm, we will.


One of the pillars of crypto is that eventually a problem will be found 
with every algorithm, it's just a matter of time.  So, having just one 
available means that eventually the library will be horribly broken.  
The corollary here is that having a fallback is pretty essential.
May 29 2018
parent Adam Wilson <flyboynw gmail.com> writes:
On 05/29/2018 11:29 AM, Brad Roberts wrote:

 
 On 5/29/2018 1:57 AM, Adam Wilson via Digitalmars-d wrote:

 One of the pillars of SecureD is that ONLY safe, well-known, 
 algorithms are presented. If reasonable we will only present one 
 algorithm for a specific purpose. If there is a good reason to add 
 more than one algorithm, we will.


 
 One of the pillars of crypto is that eventually a problem will be found 
 with every algorithm, it's just a matter of time.  So, having just one 
 available means that eventually the library will be horribly broken. The 
 corollary here is that having a fallback is pretty essential.


Sure. For example, TLS 1.3 supports AES, Camellia, ARIA, and 
ChaCha20-Poly1305 for symmetric ciphers. However, when you look at the 
best-practice recommendations, AES is the clear winner. And when you 
look at what actually gets used, it's AES. AES is the Gold Standard and 
it would take a fundamental breakthrough in cryptanalysis for that to 
change.

But what happens to TLS1.3 if there is a fundamental breakthrough?
- ARIA uses an AES derived substitution-permutation network.
- Camellia uses the same S-box that AES does. (It was an AES contender)
- Poly1305 for ChaCha20 directly uses AES in it's PRF.
Which algorithm do you switch to? Anything that breaks AES can also be 
used against these other algorithms. In this case, ChaCha20 is probably 
the best answer, but only if you apply a different MAC to it.

It's a good thing that a fundamental breakthrough in cryptanalysis is 
unlikely in the foreseeable future. Of course, that's what we told 
ourselves before 3DES and Differential Cryptanalysis.

I don't disagree with you. But the overwhelming cacophony of choice 
presents it's own issues. (See: The Paradox of Choice) SecureD v2 could 
easily support these ciphers, and I don't really have a problem adding 
them for the purposes of compatibility. But the default is going to be 
AES for the forseeable future. The design premise of SecureD is to make 
the the safe defaults easy and everything else hard.

In cases where there are known weaknesses SecureD does intend to support 
multiple algorithms, but usually only two, for example SHA2 potentially 
suffers from the same weaknesses as SHA1, and as a result SecureD v2 
will be getting SHA3 support whenever OpenSSL 1.1.1 drops. But it's not 
going to add things like Whirlpool or Skein.

Or RSA versus ECC. I am considering adding Curve25519 support to v2, but 
with NSA/NIST deprecating ECC altogether (as it is less resistant to 
quantum computing than RSA, see: 
https://en.wikipedia.org/wiki/Elliptic-curve_cryptography#Quantu
_computing_attacks) 
it may be better to wait-and-see what comes of the work being done on 
quantum-resistant public-key cryptography. The new Supersingular Isogeny 
Diffie-Hellman (SIDH) looks like a strong contender, but nothing 
implements it, and it's way too new to put any trust in it yet.

-- 
Adam Wilson
IRC: LightBender
import quiet.dlang.dev;
May 29 2018