How I manage my Python environments on a Mac with two simple tools

Two things that allow me to manage my Python environments

There are many reasons to love Python. Simple and expressive syntax, a strong ecosystem of packages, interoperability with C, etc … Jupyter notebooks 😃!

No tool is perfect, however. Python is notoriously frustrating when it comes to managing environments. Because new features are continuously being added to newer Python versions, there is a chance that a package developed with Python 3.10.x will not be compatible with older interpreters, e.g. 3.7.x.

While it is hard to expect forward compatibility, it is also important to note that macOS 12.3.1 is shipped with Python 3.8.9. Meanwhile, Google Colab uses Python 3.7.11. Using a feature introduced in Python 3.8 in a package source will make the package incompatible with the default Python version in Google Colab!

Unfortunately, compatibility between Python versions is not the only concern when developing Python packages. Compatibility between dependencies is also something to be aware of. Only one version of a package can be installed for an interpreter at a time. Example, system interpreter has foo_package==0.8.1 installed, but foo_package==0.7.3 would be needed for development of testing.

Managing Interpreters versions

To host multiple versions of Python 3 and switch between them, we need to install pyenv. Pyenv allows you to install multiple versions of Python and switch between them on both global and per-project levels.

Not all versions are available to download through pyenv, though a majority of them are. Chances are, you will be able to find a version that is right for the specific consumer.

Managing dependencies

To better manage package dependencies, it is strongly advised to set up a virtual environment on a per-project basis. Virtual environments — local interpreters — allow you to install local dependencies. If project1 requires foo_package==0.7.3, you can install it locally, all the while your global interpreter can have foo_package==0.8.1 installed, or not installed at all!

Python’s venv module, shipped with Python interpreters starting with 3.3, allows you to copy an instance of your Python interpreter into the desired location (most often your project root) via CLI. Once installed, you can activate and deactivate that version by sourcing a shell script.

pyenv and venv together

Using both pyenv and venv allows for creating multiple local interpreters from multiple Python versions. No more Python incompatibility headaches 😎.