karolyi
/
py3-validate-email
3
3
Fork 4
Code Issues 1 Pull Requests Projects Releases Wiki Activity

SMTP validation raises SSLError with message DH_KEY_TOO_SMALL #79

New Issue
Closed
opened 2021-09-07 18:55:10 +02:00 by pomma89 · 13 comments
pomma89 commented 2021-09-07 18:55:10 +02:00 (Migrated from github.com)
Copy Link
  • I have read and understood the FAQ

Describe the bug

SMTP validation raises SSLError with following message:

ssl.SSLError: [SSL: DH_KEY_TOO_SMALL] dh key too small (_ssl.c:1129)

To Reproduce

I cannot publicly share the email address that triggers this error. I can send it privately, if it is needed.

My debug output

Output from the debug run described in the FAQ:

DEBUG:validate_email.smtp_check:Trying __SMTP_SERVER__ ...
16:40:42.117353 connect: to ('__SMTP_SERVER__', 25) None
16:40:42.477049 reply: b'220 Ready to receive mail -=- ESMTP\r\n'
16:40:42.480237 reply: retcode (220); Msg: b'Ready to receive mail -=- ESMTP'
16:40:42.480664 connect: b'Ready to receive mail -=- ESMTP'
16:40:42.482181 send: 'ehlo [__IP_ADDRESS__]\r\n'
16:40:42.504819 reply: b'250-Ready to receive mail -=-\r\n'
16:40:42.504894 reply: b'250-STARTTLS\r\n'
16:40:42.504916 reply: b'250-PIPELINING\r\n'
16:40:42.505653 reply: b'250 8BITMIME\r\n'
16:40:42.505695 reply: retcode (250); Msg: b'Ready to receive mail -=-\nSTARTTLS\nPIPELINING\n8BITMIME'
16:40:42.506105 send: 'STARTTLS\r\n'
16:40:42.530312 reply: b'220 ready for tls\r\n'
16:40:42.530540 reply: retcode (220); Msg: b'ready for tls'
16:40:42.633055 send: 'quit\r\n'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.9/site-packages/validate_email/validate_email.py", line 70, in validate_email
    return validate_email_or_fail(email_address, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/validate_email/validate_email.py", line 56, in validate_email_or_fail
    return smtp_check(
  File "/usr/local/lib/python3.9/site-packages/validate_email/smtp_check.py", line 202, in smtp_check
    return smtp_checker.check(hosts=mx_records)
  File "/usr/local/lib/python3.9/site-packages/validate_email/smtp_check.py", line 170, in check
    if self._check_one(host=host):
  File "/usr/local/lib/python3.9/site-packages/validate_email/smtp_check.py", line 141, in _check_one
    self.starttls()
  File "/usr/local/lib/python3.9/site-packages/validate_email/smtp_check.py", line 80, in starttls
    super().starttls(*args, **kwargs)
  File "/usr/local/lib/python3.9/smtplib.py", line 790, in starttls
    self.sock = context.wrap_socket(self.sock,
  File "/usr/local/lib/python3.9/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/local/lib/python3.9/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/usr/local/lib/python3.9/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: DH_KEY_TOO_SMALL] dh key too small (_ssl.c:1129)

I had to remove SMTP server name and IP, as I cannot publicly disclose them. However, I can send them privately.

Expected behavior

I think that the error should either be ignored, as it is done here:

https://github.com/karolyi/py3-validate-email/blob/master/validate_email/smtp_check.py#L84

Or handled, in order to produce an ambiguous result.

Please complete the following information:

  • OS: Linux on Docker
  • Flavor and Version: Debian 11 with Python 3.9
  • Your network environment: Azure App Service
  • Your exact py3-validate-email module version: 1.0.1

Additional context

The Docker container can correctly validate other email addresses, the error is raised by a specific SMTP server.

<!-- Please don't delete this template or we'll close your issue --> - [x] I have read and understood the [FAQ](https://github.com/karolyi/py3-validate-email/blob/master/FAQ.md) **Describe the bug** SMTP validation raises `SSLError` with following message: `ssl.SSLError: [SSL: DH_KEY_TOO_SMALL] dh key too small (_ssl.c:1129)` **To Reproduce** I cannot publicly share the email address that triggers this error. I can send it privately, if it is needed. **My debug output** Output from the debug run described in the FAQ: ```txt DEBUG:validate_email.smtp_check:Trying __SMTP_SERVER__ ... 16:40:42.117353 connect: to ('__SMTP_SERVER__', 25) None 16:40:42.477049 reply: b'220 Ready to receive mail -=- ESMTP\r\n' 16:40:42.480237 reply: retcode (220); Msg: b'Ready to receive mail -=- ESMTP' 16:40:42.480664 connect: b'Ready to receive mail -=- ESMTP' 16:40:42.482181 send: 'ehlo [__IP_ADDRESS__]\r\n' 16:40:42.504819 reply: b'250-Ready to receive mail -=-\r\n' 16:40:42.504894 reply: b'250-STARTTLS\r\n' 16:40:42.504916 reply: b'250-PIPELINING\r\n' 16:40:42.505653 reply: b'250 8BITMIME\r\n' 16:40:42.505695 reply: retcode (250); Msg: b'Ready to receive mail -=-\nSTARTTLS\nPIPELINING\n8BITMIME' 16:40:42.506105 send: 'STARTTLS\r\n' 16:40:42.530312 reply: b'220 ready for tls\r\n' 16:40:42.530540 reply: retcode (220); Msg: b'ready for tls' 16:40:42.633055 send: 'quit\r\n' Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/local/lib/python3.9/site-packages/validate_email/validate_email.py", line 70, in validate_email return validate_email_or_fail(email_address, **kwargs) File "/usr/local/lib/python3.9/site-packages/validate_email/validate_email.py", line 56, in validate_email_or_fail return smtp_check( File "/usr/local/lib/python3.9/site-packages/validate_email/smtp_check.py", line 202, in smtp_check return smtp_checker.check(hosts=mx_records) File "/usr/local/lib/python3.9/site-packages/validate_email/smtp_check.py", line 170, in check if self._check_one(host=host): File "/usr/local/lib/python3.9/site-packages/validate_email/smtp_check.py", line 141, in _check_one self.starttls() File "/usr/local/lib/python3.9/site-packages/validate_email/smtp_check.py", line 80, in starttls super().starttls(*args, **kwargs) File "/usr/local/lib/python3.9/smtplib.py", line 790, in starttls self.sock = context.wrap_socket(self.sock, File "/usr/local/lib/python3.9/ssl.py", line 500, in wrap_socket return self.sslsocket_class._create( File "/usr/local/lib/python3.9/ssl.py", line 1040, in _create self.do_handshake() File "/usr/local/lib/python3.9/ssl.py", line 1309, in do_handshake self._sslobj.do_handshake() ssl.SSLError: [SSL: DH_KEY_TOO_SMALL] dh key too small (_ssl.c:1129) ``` I had to remove SMTP server name and IP, as I cannot publicly disclose them. However, I can send them privately. **Expected behavior** I think that the error should either be ignored, as it is done here: https://github.com/karolyi/py3-validate-email/blob/master/validate_email/smtp_check.py#L84 Or handled, in order to produce an ambiguous result. **Please complete the following information:** - OS: Linux on Docker - Flavor and Version: Debian 11 with Python 3.9 - Your network environment: Azure App Service - Your exact `py3-validate-email` module version: 1.0.1 **Additional context** The Docker container can correctly validate other email addresses, the error is raised by a specific SMTP server.
karolyi commented 2021-09-07 19:17:45 +02:00 (Migrated from github.com)
Copy Link

Hey,

this seems odd.

Can you please share the email address privately so I can test with it? Just shoot an email to laszlo -at- karolyi -dot- hu with it.

Thanks.

Hey, this seems odd. Can you please share the email address privately so I can test with it? Just shoot an email to laszlo -at- karolyi -dot- hu with it. Thanks.
karolyi commented 2021-09-07 19:52:24 +02:00 (Migrated from github.com)
Copy Link

So, after having gotten your email, I went over to my server and issued a test. For me, starttls worked out right for the same host (smtp1p1) you specified.

I think the error must be on your end since I can't reproduce it. Can you tell me which openssl version is installed within that docker container? Just a hunch, but that might be the culprit.

So, after having gotten your email, I went over to my server and issued a test. For me, starttls worked out right for the same host (smtp1p1) you specified. I think the error must be on your end since I can't reproduce it. Can you tell me which openssl version is installed within that docker container? Just a hunch, but that might be the culprit.
pomma89 commented 2021-09-07 20:22:09 +02:00 (Migrated from github.com)
Copy Link

Hello @karolyi,

Thank you. openssl version command says:

OpenSSL 1.1.1k 25 Mar 2021

It seems that recent Debian releases have tightened SSL security:

https://askubuntu.com/a/1263098

I tried to apply these instructions to a test container and the validation for that email address started working:

https://askubuntu.com/a/1296578

What do you think about this? Should user code, not library code, handle SSLError, because it is something related to the environment configuration?

Hello @karolyi, Thank you. `openssl version` command says: `OpenSSL 1.1.1k 25 Mar 2021` It seems that recent Debian releases have tightened SSL security: https://askubuntu.com/a/1263098 I tried to apply these instructions to a test container and the validation for that email address started working: https://askubuntu.com/a/1296578 What do you think about this? Should user code, not library code, handle `SSLError`, because it is something related to the environment configuration?
karolyi commented 2021-09-08 00:08:32 +02:00 (Migrated from github.com)
Copy Link

RFC 3207 states that,

The SMTP client and server should note carefully the result of the
TLS negotiation. If the negotiation results in no privacy, or if it
results in privacy using algorithms or key lengths that are deemed
not strong enough, or if the authentication is not good enough for
either party, the client may choose to end the SMTP session with an
immediate QUIT command, or the server may choose to not accept any
more SMTP commands.

I'd like to follow this behavior which means we return None in validate_email() and raise SMTPTemporaryError in validate_email_or_fail().

Maybe adding an smtp_require_tls=False and an optionally passable SSLContext option is feasible here, to be able to not use STARTTLS when this exception is raised. smtp_require_tls would be True per default, resulting in the client deciding on an ambiguous response when the handshake fails.

@reinhard-mueller, do you have an opinion on this? Anything to add?

[RFC 3207](https://datatracker.ietf.org/doc/html/rfc3207) states that, > The SMTP client and server should note carefully the result of the > TLS negotiation. If the negotiation results in no privacy, or if it > results in privacy using algorithms or key lengths that are deemed > not strong enough, or if the authentication is not good enough for > either party, the client may choose to end the SMTP session with an > immediate QUIT command, or the server may choose to not accept any > more SMTP commands. I'd like to follow this behavior which means we return `None` in `validate_email()` and raise `SMTPTemporaryError` in `validate_email_or_fail()`. Maybe adding an `smtp_require_tls=False` and an optionally passable `SSLContext` option is feasible here, to be able to not use STARTTLS when this exception is raised. `smtp_require_tls` would be `True` per default, resulting in the client deciding on an ambiguous response when the handshake fails. @reinhard-mueller, do you have an opinion on this? Anything to add?
karolyi commented 2021-09-08 00:10:05 +02:00 (Migrated from github.com)
Copy Link

Even better, to signify that there was an actual failure of the TLS handshake, I'd introduce a new exception which can be catched, should the user decide that way.

Even better, to signify that there was an actual failure of the TLS handshake, I'd introduce a new exception which can be catched, should the user decide that way.
reinhard-mueller commented 2021-09-08 08:25:59 +02:00 (Migrated from github.com)
Copy Link

Maybe it would make sense to try to continue without TLS, like we already do for other kinds of SSL problems? Are there SMTP servers out there which refuse to communicate without TLS?

Maybe it would make sense to try to continue without TLS, like we already do for other kinds of SSL problems? Are there SMTP servers out there which refuse to communicate without TLS?
👍 1
karolyi commented 2021-09-13 18:21:46 +02:00 (Migrated from github.com)
Copy Link

Problem is, once you wrapped the connection in an SSL wrapper, you can't get it back. The same happens with other clients, they simply disconnect after the TLS negotiation has failed. Even if you can recover from SSLError at context.wrap_socket(), I think the server will either already assume you talk the TLS protocol, or just simply close the connection over the negotiation failure.

To be honest I'm against reconnecting immediately, none of other cliens do that either (other than ones that are written badly and want to spam). If they do, fail2ban is easily able to catch them, for example.

Normally a client should retry in about 5 minutes, but I'm not sure if they should remember that they have failed earlier so next time they would not use TLS negotiation.

Hence my opinion of failing silently and giving an ambiguous response.

Still, I'm open to suggestions.

Problem is, once you wrapped the connection in an SSL wrapper, you can't get it back. The same happens with other clients, they simply disconnect after the TLS negotiation has failed. Even if you can recover from `SSLError` at [`context.wrap_socket()`](https://github.com/python/cpython/blob/11749e2dc20ad6a76e9a39e948853e89b2b4bbed/Lib/smtplib.py#L785), I think the server will either already assume you talk the TLS protocol, or just simply close the connection over the negotiation failure. To be honest I'm against reconnecting immediately, none of other cliens do that either (other than ones that are written badly and want to spam). If they do, fail2ban is easily able to catch them, for example. Normally a client should retry in about 5 minutes, but I'm not sure if they should remember that they have failed earlier so next time they would not use TLS negotiation. Hence my opinion of failing silently and giving an ambiguous response. Still, I'm open to suggestions.
pomma89 commented 2021-09-13 21:34:39 +02:00 (Migrated from github.com)
Copy Link

Thank you, @karolyi, for the detailed explanation. I understand your point of view and what you suggested doing.

The dedicated error with the ambiguous response seems to be the best solution.

Let me know if a PR with this change would be appreciated.

Goodbye,
Alessio

Thank you, @karolyi, for the detailed explanation. I understand your point of view and what you suggested doing. The dedicated error with the ambiguous response seems to be the best solution. Let me know if a PR with this change would be appreciated. Goodbye, Alessio
reinhard-mueller commented 2021-09-14 09:46:34 +02:00 (Migrated from github.com)
Copy Link

@karolyi I see your point about not being able to go back into non-SSL. Given that, I agree with issuing an ambigous response, and I also agree that a new exception class (derived from SMTPError) is the optimal solution.

@karolyi I see your point about not being able to go back into non-SSL. Given that, I agree with issuing an ambigous response, and I also agree that a new exception class (derived from SMTPError) is the optimal solution.
karolyi commented 2021-09-14 10:37:17 +02:00 (Migrated from github.com)
Copy Link

PRs are welcome but I can prepare something in a couple days too.

PRs are welcome but I can prepare something in a couple days too.
pomma89 commented 2021-09-14 21:23:03 +02:00 (Migrated from github.com)
Copy Link

PRs are welcome but I can prepare something in a couple days too.

@karolyi, thank you. If, for whatever reason, you cannot work on this, I will be able to work on a PR during the weekend.

> > > PRs are welcome but I can prepare something in a couple days too. @karolyi, thank you. If, for whatever reason, you cannot work on this, I will be able to work on a PR during the weekend.
karolyi commented 2021-09-15 18:27:28 +02:00 (Migrated from github.com)
Copy Link

This should be fixed now in 1.0.2 although travis shits itself for some reason (probably there is some API glitch between them and github), and I can't get the green build logo displayed. Will work on that part, but the SSL error should be fixed now, with additional changes. For those, see changelog.txt.

This should be fixed now in `1.0.2` although travis shits itself for some reason (probably there is some API glitch between them and github), and I can't get the green build logo displayed. Will work on that part, but the SSL error should be fixed now, with additional changes. For those, see changelog.txt.
pomma89 commented 2021-09-15 22:47:21 +02:00 (Migrated from github.com)
Copy Link

Thank you, @karolyi. Tomorrow I will test the new version.

Goodbye,
Alessio

Thank you, @karolyi. Tomorrow I will test the new version. Goodbye, Alessio
👍 1
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
master
pypi-faggotry
v1.0.11
v1.0.10
v1.0.9
Labels
Clear labels   
bug

Something isn't working

  
duplicate

This issue or pull request already exists

  
enhancement

New feature or request

  
good first issue

Good for newcomers

  
help wanted

Extra attention is needed

  
invalid

This doesn't seem right

  
question

Further information is requested

  
stale

   
wontfix

This will not be worked on

No Label
bug
duplicate
enhancement
good first issue
help wanted
invalid
question
stale
wontfix
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: karolyi/py3-validate-email#79
No description provided.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?