Virtual environments and lovely Poetry

Since I started learning Python, the only package manager I've used is Pip. It's a package manager that handles the basics, but it can lead to some messy situations if you're not careful.

My biggest issue with using Pip wasn't really with Pip itself, but with not using a virtual environment. I couldn't figure out why, with each project I started, when I ran pip3 freeze > requirements.txt, my requirements.txt file would be filled with packages I wasn't even using. As it turns out, all those extra packages were coming from previous projects where I'd been installing packages without thinking, not realising that everything was being pulled from a global environment.

At the very start of my journey, I didn't even realise that was happening, so I was pushing "infected" requirements.txt files to GitHub. Later, I found that it can all be solved using a virtual environment so that any packages you install stay in that environment. They won't leak out into other projects that don't need them.

Setting Up Virtual Environments

To create a virtual environment using Pip on Mac, you can run this command:

python3 -m venv .venv

python3 -m venv is the command to create the environment, and .venv is the folder it will create with all the virtual environment contents inside of it. From what I've seen, .venv is the convention for naming the virtual environment folder. If it has not already been added to your .gitignore file, you should add it.

Once the venv has been created, you then need to activate it by running this command:

source .venv/bin/activate

Once the virtual environment has been activated, you can run pip install -r requirements.txt to install all the packages needed for your project. Note that once you're in a virtual environment, you no longer need to add the 3 to your commands. That means you can use python instead of python3 or pip instead of pip3.

Another benefit of Virtual Environments

Besides keeping your projects isolated and preventing package conflicts, virtual environments offer another significant benefit: they make it easier to reproduce your development environment. When you're working on a team or deploying your application, you can be confident that everyone is using the same package versions, reducing the "it works on my machine" problem.

While virtual environments solve many problems, there's an even more elegant solution that takes care of virtual environments and more: Poetry.

Poetry - A newer package manager

Poetry is a tool for dependency management and packaging in Python. It allows you to declare the libraries your project depends on and it will manage (install/update) them for you.

By 'dependency management', I mean Poetry handles not just the installation of packages (like pip does), but also resolves version conflicts, tracks both direct and sub-dependencies, and ensures consistent environments across different systems. Unlike pip, which requires separate tools for creating virtual environments and managing project metadata, Poetry integrates all these functions into a single, intuitive tool.

Installing Poetry

To install Poetry, you can use the following command:

pip install poetry

Yes - I know - a slap in the face for Pip to install its alternative. There are other ways to install Poetry (including pipx, which seems to be a preferred way).

Basic Poetry Usage

Once installed, you can create a new project with Poetry:

poetry new my-project

This creates a new directory with a basic project structure, including a pyproject.toml file, which is where Poetry keeps track of your project dependencies. A .toml ("Tom's Obvious, Minimal Language") file is a configuration file, like a .yml file.

To add a dependency to your project, you can use:

poetry add requests

This will add the requests library to your project and update the pyproject.toml file.

To install all dependencies for an existing project, simply run:

poetry install

Poetry creates a virtual environment for you automatically, so you don't need to manage it manually. To run commands in this environment, you can use:

poetry run python your_script.py

Poetry simplifies dependency management and makes it easier to build and share your Python projects. What I like about Poetry is how it treats virtual environments as a core feature, rather than an afterthought. Unlike pip, where you need to manually create and manage virtual environments, Poetry integrates this functionality seamlessly into its workflow. This "built-in" approach to virtual environments means you're less likely to make mistakes or forget this crucial step in your project setup. It's definitely worth exploring if you're working on Python projects and want a more streamlined, less error-prone way to manage your development environment.