PEP 493 – HTTPS verification migration tools for Python 2.7
- Author:
- Alyssa Coghlan <ncoghlan at gmail.com>, Robert Kuska <rkuska at redhat.com>, Marc-André Lemburg <mal at lemburg.com>
- BDFL-Delegate:
- Barry Warsaw
- Status:
- Final
- Type:
- Standards Track
- Created:
- 10-May-2015
- Python-Version:
- 2.7.12
- Post-History:
- 06-Jul-2015, 11-Nov-2015, 24-Nov-2015, 24-Feb-2016
- Resolution:
- Python-Dev message
Table of Contents
- Abstract
- Rationale
- Scope Limitations
- Requirements for capability detection
- Feature: Configuration API
- Feature: environment based configuration
- Reference Implementation
- Backporting this PEP to earlier Python versions
- Backporting PEP 476 to earlier Python versions
- Recommendation for combined feature backports
- Copyright
Abstract
PEP 476 updated Python’s default handling of HTTPS certificates in client modules to align with certificate handling in web browsers, by validating that the certificates received belonged to the server the client was attempting to contact. The Python 2.7 long term maintenance series was judged to be in scope for this change, with the new behaviour introduced in the Python 2.7.9 maintenance release.
This has created a non-trivial barrier to adoption for affected Python 2.7 maintenance releases, so this PEP proposes additional Python 2.7 specific features that allow system administrators and other users to more easily decouple the decision to verify server certificates in HTTPS client modules from the decision to update to newer Python 2.7 maintenance releases.
Rationale
PEP 476 changed Python’s default behaviour to align with expectations established by web browsers in regards to the semantics of HTTPS URLs: starting with Python 2.7.9 and 3.4.3, HTTPS clients in the standard library validate server certificates by default.
However, it is also the case that this change does cause problems for infrastructure administrators operating private intranets that rely on self-signed certificates, or otherwise encounter problems with the new default certificate verification settings.
To manage these kinds of situations, web browsers provide users with “click
through” warnings that allow the user to add the server’s certificate to the
browser’s certificate store. Network client tools like curl
and wget
offer options to switch off certificate checking entirely (by way of
curl --insecure
and wget --no-check-certificate
, respectively).
At a different layer of the technology stack, Linux security modules like
SELinux
and AppArmor
, while enabled by default by distribution vendors,
offer relatively straightforward mechanisms for turning them off.
At the moment, no such convenient mechanisms exist to disable Python’s default certificate checking for a whole process.
PEP 476 did attempt to address this question, by covering how to revert to the
old settings process wide by monkeypatching the ssl
module to restore the
old behaviour. Unfortunately, the sitecustomize.py
based technique proposed
to allow system administrators to disable the feature by default in their
Standard Operating Environment definition has been determined to be
insufficient in at least some cases. The specific case that led to the
initial creation of this PEP is the one where a Linux distributor aims to
provide their users with a
smoother migration path
than the standard one provided by consuming upstream CPython 2.7 releases
directly, but other potential challenges have also been pointed out with
updating embedded Python runtimes and other user level installations of Python.
Rather than allowing a plethora of mutually incompatible migration techniques to bloom, this PEP proposes an additional feature to be added to Python 2.7.12 to make it easier to revert a process to the past behaviour of skipping certificate validation in HTTPS client modules. It also provides additional recommendations to redistributors backporting these features to versions of Python prior to Python 2.7.9.
Alternatives
In the absence of clear upstream guidance and recommendations, commercial redistributors will still make their own design decisions in the interests of their customers. The main approaches available are:
- Continuing to rebase on new Python 2.7.x releases, while providing no additional assistance beyond the mechanisms defined in PEP 476 in migrating from unchecked to checked hostnames in standard library HTTPS clients
- Gating availability of the changes in default handling of HTTPS connections on upgrading from Python 2 to Python 3
- For Linux distribution vendors, gating availability of the changes in default handling of HTTPS connections on upgrading to a new operating system version
- Implementing one or both of the backport suggestions described in this PEP, regardless of the formal status of the PEP
Scope Limitations
These changes are being proposed purely as tools for helping to manage the transition to the new default certificate handling behaviour in the context of Python 2.7. They are not being proposed as new features for Python 3, as it is expected that the vast majority of client applications affected by this problem without the ability to update the application itself will be Python 2 applications.
It would likely be desirable for a future version of Python 3 to allow the default certificate handling for secure protocols to be configurable on a per-protocol basis, but that question is beyond the scope of this PEP.
Requirements for capability detection
As the proposals in this PEP aim to facilitate backports to earlier Python versions, the Python version number cannot be used as a reliable means for detecting them. Instead, they are designed to allow the presence or absence of the feature to be determined using the following technique:
python -c "import ssl; ssl.<_relevant_attribute>"
This will fail with AttributeError
(and hence a non-zero return code) if
the relevant capability is not available.
The feature detection attributes defined by this PEP are:
ssl._https_verify_certificates
: runtime configuration APIssl._https_verify_envvar
: environment based configurationssl._cert_verification_config
: file based configuration (PEP 476 opt-in)
The marker attributes are prefixed with an underscore to indicate the implementation dependent and security sensitive nature of these capabilities.
Feature: Configuration API
This change is proposed for inclusion in CPython 2.7.12 and later CPython 2.7.x
releases. It consists of a new ssl._https_verify_certificates()
to specify
the default handling of HTTPS certificates in standard library client libraries.
It is not proposed to forward port this change to Python 3, so Python 3 applications that need to support skipping certificate verification will still need to define their own suitable security context.
Feature detection
The marker attribute on the ssl
module related to this feature is the
ssl._https_verify_certificates
function itself.
Specification
The ssl._https_verify_certificates
function will work as follows:
def _https_verify_certificates(enable=True):
"""Verify server HTTPS certificates by default?"""
global _create_default_https_context
if enable:
_create_default_https_context = create_default_context
else:
_create_default_https_context = _create_unverified_context
If called without arguments, or with enable
set to a true value, then
standard library client modules will subsequently verify HTTPS certificates by default, otherwise they will skip verification.
If called with enable
set to a false value, then standard library client
modules will subsequently skip verifying HTTPS certificates by default.
Security Considerations
The inclusion of this feature will allow security sensitive applications to include the following forward-compatible snippet in their code:
if hasattr(ssl, "_https_verify_certificates"):
ssl._https_verify_certificates()
Some developers may also choose to opt out of certificate checking using
ssl._https_verify_certificates(enable=False)
. This doesn’t introduce any
major new security concerns, as monkeypatching the affected internal APIs was
already possible.
Feature: environment based configuration
This change is proposed for inclusion in CPython 2.7.12 and later CPython 2.7.x
releases. It consists of a new PYTHONHTTPSVERIFY
environment variable that
can be set to '0'
to disable the default verification without modifying the
application source code (which may not even be available in cases of
bytecode-only application distribution)
It is not proposed to forward port this change to Python 3, so Python 3 applications that need to support skipping certificate verification will still need to define their own suitable security context.
Feature detection
The marker attribute on the ssl
module related to this feature is:
- the
ssl._https_verify_envvar
attribute, giving the name of environment variable affecting the default behaviour
This not only makes it straightforward to detect the presence (or absence) of the capability, it also makes it possible to programmatically determine the relevant environment variable name.
Specification
Rather than always defaulting to the use of ssl.create_default_context
,
the ssl
module will be modified to:
- read the
PYTHONHTTPSVERIFY
environment variable when the module is first imported into a Python process - set the
ssl._create_default_https_context
function to be an alias forssl._create_unverified_context
if this environment variable is present and set to'0'
- otherwise, set the
ssl._create_default_https_context
function to be an alias forssl.create_default_context
as usual
Example implementation
_https_verify_envvar = 'PYTHONHTTPSVERIFY'
def _get_https_context_factory():
if not sys.flags.ignore_environment:
config_setting = os.environ.get(_https_verify_envvar)
if config_setting == '0':
return _create_unverified_context
return create_default_context
_create_default_https_context = _get_https_context_factory()
Security Considerations
Relative to the behaviour in Python 3.4.3+ and Python 2.7.9->2.7.11, this approach does introduce a new downgrade attack against the default security settings that potentially allows a sufficiently determined attacker to revert Python to the default behaviour used in CPython 2.7.8 and earlier releases.
This slight increase in the available attack surface is a key reason why:
- security sensitive applications should still define their own SSL context
- the migration features described in this PEP are not being added to Python 3
However, it’s also worth keeping in mind that carrying out such an attack
requires the ability to modify the execution environment of a Python process
prior to the import of the ssl
module. In combination with the ability
to write to any part of the filesystem (such as /tmp
), any attacker with
such access would already be able to modify the behaviour of the underlying
OpenSSL implementation, the dynamic library loader, and other potentially
security sensitive components.
Interaction with Python virtual environments
The default setting is read directly from the process environment, and hence works the same way regardless of whether or not the interpreter is being run inside an activated Python virtual environment.
Reference Implementation
A patch for Python 2.7 implementing the above two features is attached to the relevant tracker issue.
Backporting this PEP to earlier Python versions
If this PEP is accepted, then commercial Python redistributors may choose to backport the per-process configuration mechanisms defined in this PEP to base versions older than Python 2.7.9, without also backporting PEP 476’s change to the default behaviour of the overall Python installation.
Such a backport would differ from the mechanism proposed in this PEP solely in
the default behaviour when PYTHONHTTPSVERIFY
was not set at all: it would
continue to default to skipping certificate validation.
In this case, if the PYTHONHTTPSVERIFY
environment variable is defined, and
set to anything other than '0'
, then HTTPS certificate verification
should be enabled.
Feature detection
There’s no specific attribute indicating that this situation applies. Rather,
it is indicated by the ssl._https_verify_certificates
and
ssl._https_verify_envvar
attributes being present in a Python version that
is nominally older than Python 2.7.12.
Specification
Implementing this backport involves backporting the changes in PEP 466, 476 and
this PEP, with the following change to the handling of the
PYTHONHTTPSVERIFY
environment variable in the ssl
module:
- read the
PYTHONHTTPSVERIFY
environment variable when the module is first imported into a Python process - set the
ssl._create_default_https_context
function to be an alias forssl.create_default_context
if this environment variable is present and set to any value other than'0'
- otherwise, set the
ssl._create_default_https_context
function to be an alias forssl._create_unverified_context
Example implementation
_https_verify_envvar = 'PYTHONHTTPSVERIFY'
def _get_https_context_factory():
if not sys.flags.ignore_environment:
config_setting = os.environ.get(_https_verify_envvar)
if config_setting != '0':
return create_default_context
return _create_unverified_context
_create_default_https_context = _get_https_context_factory()
def _disable_https_default_verification():
"""Skip verification of HTTPS certificates by default"""
global _create_default_https_context
_create_default_https_context = _create_unverified_context
Security Considerations
This change would be a strict security upgrade for any Python version that currently defaults to skipping certificate validation in standard library HTTPS clients. The technical trade-offs to be taken into account relate largely to the magnitude of the PEP 466 backport also required rather than to anything security related.
Interaction with Python virtual environments
The default setting is read directly from the process environment, and hence works the same way regardless of whether or not the interpreter is being run inside an activated Python virtual environment.
Backporting PEP 476 to earlier Python versions
The backporting approach described above leaves the default HTTPS certificate verification behaviour of a Python 2.7 installation unmodified: verifying certificates still needs to be opted into on a per-connection or per-process basis.
To allow the default behaviour of the entire installation to be modified without breaking backwards compatibility, Red Hat designed a configuration mechanism for the system Python 2.7 installation in Red Hat Enterprise Linux 7.2+ that provides:
- an opt-in model that allows the decision to enable HTTPS certificate verification to be made independently of the decision to upgrade to the operating system version where the feature was first backported
- the ability for system administrators to set the default behaviour of Python applications and scripts run directly in the system Python installation
- the ability for the redistributor to consider changing the default behaviour of new installations at some point in the future without impacting existing installations that have been explicitly configured to skip verifying HTTPS certificates by default
As it only affects backports to earlier releases of Python 2.7, this change is not proposed for inclusion in upstream CPython, but rather is offered as a recommendation to other redistributors that choose to offer a similar feature to their users.
This PEP doesn’t take a position on whether or not this particular change is a good idea - rather, it suggests that if a redistributor chooses to go down the path of making the default behaviour configurable in a version of Python older than Python 2.7.9, then maintaining a consistent approach across redistributors would be beneficial for users.
However, this approach SHOULD NOT be used for any Python installation that advertises itself as providing Python 2.7.9 or later, as most Python users will have the reasonable expectation that all such environments will verify HTTPS certificates by default.
Feature detection
The marker attribute on the ssl
module related to this feature is:
_cert_verification_config = '<path to configuration file>'
This not only makes it straightforward to detect the presence (or absence) of the capability, it also makes it possible to programmatically determine the relevant configuration file name.
Recommended modifications to the Python standard library
The recommended approach to backporting the PEP 476 modifications to an earlier point release is to implement the following changes relative to the default PEP 476 behaviour implemented in Python 2.7.9+:
- modify the
ssl
module to read a system wide configuration file when the module is first imported into a Python process - define a platform default behaviour (either verifying or not verifying HTTPS certificates) to be used if this configuration file is not present
- support selection between the following three modes of operation:
- ensure HTTPS certificate verification is enabled
- ensure HTTPS certificate verification is disabled
- delegate the decision to the redistributor providing this Python version
- set the
ssl._create_default_https_context
function to be an alias for eitherssl.create_default_context
orssl._create_unverified_context
based on the given configuration setting.
Recommended file location
As the PEP authors are not aware of any vendors providing long-term support releases targeting Windows, Mac OS X or *BSD systems, this approach is currently only specifically defined for Linux system Python installations.
The recommended configuration file name on Linux systems is
/etc/python/cert-verification.cfg
.
The .cfg
filename extension is recommended for consistency with the
pyvenv.cfg
used by the venv
module in Python 3’s standard library.
Recommended file format
The configuration file should use a ConfigParser ini-style format with a
single section named [https]
containing one required setting verify
.
The suggested section name is taken from the “https” URL schema passed to affected client APIs.
Permitted values for verify
are:
enable
: ensure HTTPS certificate verification is enabled by defaultdisable
: ensure HTTPS certificate verification is disabled by defaultplatform_default
: delegate the decision to the redistributor providing this particular Python version
If the [https]
section or the verify
setting are missing, or if the
verify
setting is set to an unknown value, it should be treated as if the
configuration file is not present.
Example implementation
_cert_verification_config = '/etc/python/cert-verification.cfg'
def _get_https_context_factory():
# Check for a system-wide override of the default behaviour
context_factories = {
'enable': create_default_context,
'disable': _create_unverified_context,
'platform_default': _create_unverified_context, # For now :)
}
import ConfigParser
config = ConfigParser.RawConfigParser()
config.read(_cert_verification_config)
try:
verify_mode = config.get('https', 'verify')
except (ConfigParser.NoSectionError, ConfigParser.NoOptionError):
verify_mode = 'platform_default'
default_factory = context_factories.get('platform_default')
return context_factories.get(verify_mode, default_factory)
_create_default_https_context = _get_https_context_factory()
Security Considerations
The specific recommendations for this backporting case are designed to work for privileged, security sensitive processes, even those being run in the following locked down configuration:
- run from a locked down administrator controlled directory rather than a normal
user directory (preventing
sys.path[0]
based privilege escalation attacks) - run using the
-E
switch (preventingPYTHON*
environment variable based privilege escalation attacks) - run using the
-s
switch (preventing user site directory based privilege escalation attacks) - run using the
-S
switch (preventingsitecustomize
based privilege escalation attacks)
The intent is that the only reason HTTPS verification should be getting turned off installation wide when using this approach is because:
- an end user is running a redistributor provided version of CPython rather than running upstream CPython directly
- that redistributor has decided to provide a smoother migration path to verifying HTTPS certificates by default than that being provided by the upstream project
- either the redistributor or the local infrastructure administrator has determined that it is appropriate to retain the default pre-2.7.9 behaviour (at least for the time being)
Using an administrator controlled configuration file rather than an environment
variable has the essential feature of providing a smoother migration path, even
for applications being run with the -E
switch.
Interaction with Python virtual environments
This setting is scoped by the interpreter installation and affects all Python processes using that interpreter, regardless of whether or not the interpreter is being run inside an activated Python virtual environment.
Origins of this recommendation
This recommendation is based on the backporting approach adopted for Red Hat Enterprise Linux 7.2, as published in the original July 2015 draft of this PEP and described in detail in this KnowledgeBase article. Red Hat’s patches implementing this backport for Python 2.7.5 can be found in the CentOS git repository.
Recommendation for combined feature backports
If a redistributor chooses to backport the environment variable based configuration setting from this PEP to a modified Python version that also implements the configuration file based PEP 476 backport, then the environment variable should take precedence over the system-wide configuration setting. This allows the setting to be changed for a given user or application, regardless of the installation-wide default behaviour.
Example implementation
_https_verify_envvar = 'PYTHONHTTPSVERIFY'
_cert_verification_config = '/etc/python/cert-verification.cfg'
def _get_https_context_factory():
# Check for an environmental override of the default behaviour
if not sys.flags.ignore_environment:
config_setting = os.environ.get(_https_verify_envvar)
if config_setting is not None:
if config_setting == '0':
return _create_unverified_context
return create_default_context
# Check for a system-wide override of the default behaviour
context_factories = {
'enable': create_default_context,
'disable': _create_unverified_context,
'platform_default': _create_unverified_context, # For now :)
}
import ConfigParser
config = ConfigParser.RawConfigParser()
config.read(_cert_verification_config)
try:
verify_mode = config.get('https', 'verify')
except (ConfigParser.NoSectionError, ConfigParser.NoOptionError):
verify_mode = 'platform_default'
default_factory = context_factories.get('platform_default')
return context_factories.get(verify_mode, default_factory)
_create_default_https_context = _get_https_context_factory()
Copyright
This document has been placed into the public domain.
Source: https://github.com/python/peps/blob/main/peps/pep-0493.rst
Last modified: 2023-10-11 12:05:51 GMT