How to create and set up a virtual environment in Python
Make your Python project a success with the right tools and knowledge.
If you're first getting started with Python, these steps will take you through the steps to get up and running and hopefully provide you with some understanding of why you're executing various installation commands and what the tools are doing for you. Setting up a development environment for Python involves a couple of different tools that require some explanation to fully grasp their usefulness.
Perhaps you've been using Python for a little while. When you first set up your Python environment, you probably followed a guide like this one. But a few months have passed and now you've encountered a problem with your setup or you need to upgrade your version of Python, so you need to understand what's going on under the hood. In which case, this article is also for you! Bear in mind that this is just one way of setting up a Python environment. This works well with most tools, but it’s by no means the only way to get set up.
By the end of this article you will learn how to:
-
Install different versions of Python using pyenv
-
Manage the version of Python in your current context using pyenv's tools
-
Set up a virtual environment for Python dependencies using Pipenv
-
Understand the tools that Pipenv provides for collaborating with other engineers
A closer look at the Python setup process
Pyenv: Managing Python versions
The most important tool is the Python language itself. If you're running on Mac or Linux, you'll get a version of Python that comes with the operating system. The problem is you shouldn't use this version. It's important to the operating system and you can't upgrade it without causing problems on your computer. So, how can you get your own separate version of Python you can control? Or better yet, if you're working on multiple projects that use different versions of Python, how can you get as many versions of Python as you want and easily switch between them? This is the purpose of pyenv.
Installing and exploring pyenv
Pyenv is a tool that helps you install and manage multiple versions of Python. If you're already up and running with Python, you might have installed it already. Let's walk through some commands to install and understand how to use pyenv and what it's doing for us.
First, check if it's installed by running:
pyenv --version
brew install pyenv
Once it's installed, first check out what versions of Python are currently installed on your computer.
~ % pyenv versions
system
3.7.4
3.7.6
3.8.15
* 3.9.15 (set by PYENV_VERSION environment variable)
With the versions command, pyenv shows you all the versions of Python that it is aware of on your computer. If you're just getting started, you might only have the system version.
Next, take a look at what Python versions are available using the command:
~ % pyenv install --list
Available versions:
2.1.3
2.2.3
2.3.7
2.4.0
... # Many more versions listed here
Pyenv will show you a lot of available Python versions, from the standard CPython versions to Jython or Anaconda implementations. Let's not worry about all that and focus on the standard CPython versions.
Install a new version of Python. The following command will build Python from the source, so it's going to take a while to run.
pyenv install 3.11.3
While you wait, let's take a minute to understand what's going to happen. Each version pyenv installs is going to be located nicely in your pyenv root directory:
~ % ls ~/.pyenv/versions/
3.7.4 3.7.6 3.8.15 3.9.15 3.11.3
It's easy to uninstall old versions of Python using pyenv's uninstall command.
pyenv uninstall 3.7.4
With a new version installed, let's checkout the pyenv versions command again.
~ % pyenv versions
system
3.7.6
3.8.15
* 3.9.15 (set by PYENV_VERSION environment variable)
3.11.3
Your active version of Python will be denoted with the *. Helpfully, pyenv will tell you why this is the active version. To understand the path file of your current Python interpreter, try running:
~ % which python
/Users/drew/.pyenv/shims/python
You can see that pyenv has inserted itself into your PATH. Therefore, even if you choose to use the system Python, pyenv can still manage your versions. Pyenv has its own command to tell you where your executable lives.
~ % pyenv which python
/Users/drew/.pyenv/versions/3.9.15/bin/python
Installing and exploring pyenv
Pyenv gives you tools to select which version of Python it uses in a given context. In order from most specific to least, here is how pyenv chooses which Python version to use, how they are set, and the mechanism by which that version is stored. If multiple settings conflict, pyenv will choose the most specific setting. I'll use version 3.9.15 in my examples, but you can use any version you want.
1. Pyenv shell:
pyenv shell 3.9.15
Using this command will set your python version by setting the pyenv_version environment variable. This command is used in a shell-specific context and expires when your shell exits.
2. Pyenv local:
pyenv local 3.9.15
This command will create a .python-version file in your current directory. When pyenv is running, it will automatically switch to the version you specify with this command when you enter this directory and may revert to a different version when you leave it, depending on how your higher-level settings are configured.
3. Pyenv global:
pyenv global 3.9.15
This command is useful for ensuring you have a default version you want to use. You’ll want to use this command because otherwise, the default version will be the one that comes preinstalled on your system and you will want tighter control than that. This command sets this version in the ~/.pyenv/version file.
4. System Python:
Pyenv doesn't change your system Python so if you unset your shell, local, and global versions or remove pyenv from your computer, you'll be left with this version of Python. Again, you should avoid letting this be your default. You won’t want an operating system update to affect your Python projects.
Virtual environments: Creating and managing virtual environments using Python
By now you know how pyenv can help you manage multiple versions of Python, but applications are more than just their versions of Python. You probably have all kinds of dependencies installed on your projects. What happens if you want to use different versions of a dependency on different projects? Do you need to keep switching back and forth? This is where a Python virtual environment will help you.
Why do I need a virtual environment?
When Python installs a module using Pip, the module is installed into a folder called site-packages. This folder isn't unique to each project, but rather exists as a child folder of the directory where Python is installed. Thus, even if you're using pyenv to manage Python versions, each version of Python you have installed has one and only one site-packages folder. If multiple projects use the same version of Python, they will share this folder and the dependencies can clash.
Pipenv: Managing virtual environments
Several different tools can be used to create and manage Python virtual environments, but I'll be talking about Pipenv.
Pipenv has some advantages over other tools. It is widely used in the Python community and so it has a lot of support should you run into trouble. In addition to managing dependency versions, it also supports for separating packages used for developing an application from packages that must be bundled with an application when it is released into production. Let's take a closer look at how to get this set up:
Pipenv is a Python package, and therefore installed using Python's package manager: pip.
pip install pipenv
Once you've installed Pipenv, you can forget about using pip in your project context since Pipenv acts as a replacement. Pipenv uses pip under the covers but simplifies and enhances it with helpful tools.
Using Pipenv, let's create a virtual environment for your development:
~/src/learning/pythonProject % pipenv shell
Launching a subshell in a virtual environment...
. /Users/drew/.local/share/virtualenvs/pythonProject-ttZ yii2/bin/activate
~/src/learning/pythonProject % . /Users/drew/.local/share/virtualenvs/pythonProject-ttZ yii2/bin/activate
(pythonProject) ~/src/learning/pythonProject %
You will get two new files when you've created a new virtual environment. If you're using an existing repository they probably already exist. You will see the Pipfile which will track your dependencies, and the Pipfile.lock which will enable your builds to be deterministic. More on that later. This command spawns a subshell. You can tell the subshell is active because you can see the name of your virtual environment at the start of your command prompt. You can leave the subshell at any time by typing deactivate.
With your virtual environment active, let's install some sample dependencies to see how Pipenv manages them. Let's examine the following:
pipenv install flask==0.12.1
# Installs a specific version of flask
pipenv install numpy
# Installs the latest compatible version of numpy
pipenv install -e git+https://github.com/requests/requests.git#egg=requests
# Installs the module from a specific location on github
pipenv install -e /Users/drew/src/my-local-module
# Installs from a specific location on your local file system
pipenv install pytest --dev
# Installs pytest as a dev dependency
One other note about the -e flag. This makes the dependency editable, meaning that if changes are made to the dependency, Pipenv will get the latest changes and use them each time your code is run. This can be useful when developing a library and a module that depends upon it simultaneously.
The final command installs the dependency as a dev dependency. This means that it won't be included with code that goes to production servers. Running these commands will update your Pipfile, storing a list of dependencies you've installed on your project.
The Pipfile and Pipfile.lock files
If you're developing a great application and you want to start collaborating with another engineer, you will need a way to keep track of which dependencies are installed. Pipenv helps you handle this by creating a Pipfile which tracks your dependencies and the versions of them that you are using. The Pipfile is formatted using toml and you can update it manually to change dependency versions if you need. If you followed along, your Pipfile should look something like this:
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pip_conf_index_global"
[packages]
flask = "==0.12.1"
numpy = "*"
requests = {editable = true, git = "https://github.com/requests/requests.git"}
my-local-module = {editable = true, path = "/Users/drew/src/my-local-module"}
[dev-packages]
pytest = "*"
[requires]
python_version = "3.9"
python_full_version = "3.9.16"
Taking a look at the Pipfile after running the commands above, you can see a section, one called packages which contains the first four dependencies you installed. Flask will have a specific version attached to it, while others have a * instead of a version. This means that the version is said to be unpinned and can be updated without intervention by the developer. Try running the following command.
pipenv install
The install command will install dependencies from an existing Pipfile. You can add the --dev flag to install dependencies listed in the development dependencies section Pipfile. If any of these are unpinned, you will get the latest version of that module. But wait, if dependencies can be unpinned, how can we know that a build will be deterministic? That is, if a new version of dependency is released, will your build pipeline update to use it without you knowing and testing? This is where the Pipfile.lock comes to the rescue.
The Pipfile.lock, or lock file, is a very large file in json format, which you're not supposed to edit manually. Instead Pipenv provides a set of commands for working with it.
pipenv lock
The lock command creates or updates your lock file. In essence, this pins any unpinned dependencies to specific versions so that when you install from the Pipfile.lock you get exactly what versions you're expecting. This is helpful during a continuous delivery process so updated dependencies don't break your application. Once you have a lock file, you can install dependencies from the lock file to get exactly the versions you expect by using the sync command. Pipenv also helpfully provides the update command which runs pipenv lock followed by pipenv sync.
pipenv sync
# Or use the update command to lock, then sync
pipenv update
Under the hood, Pipenv maintains separate directories for all its different virtual environments. You can find out where your virtual environment lives by running this command:
pipenv --venv
[+Body]
You can remove your virtual environment by using the command below. You may need to delete and recreate the virtual environment from scratch such as when upgrading to a new version of Python.
~/src/learning/pythonProject % pipenv --rm
Removing virtualenv (/Users/drew/.local/share/virtualenvs/pythonProject-ttZ yii2)...
Conclusion: Continue learning about the Python development environment
The information in this article should help become more familiar with the structure of your Python development environment. Understanding how all the pieces fit together can help you troubleshoot problems that might arise as versions change and update. If you're interested in learning more, check out the informational resources below.
