Python Virtual Environments: venv, pip, and Dependencies

Why You Should Use Virtual Environments

Imagine you’re working on two projects: one is a data analysis script using an older version of NumPy (say, 1.18) because it relies on deprecated features, and another is a machine learning app that needs the latest NumPy (1.26) for performance tweaks. If you install these globally on your machine, boom—conflict! One project breaks the other.

This is where virtual environments shine. A virtual environment (often called a “venv”) is a self-contained directory that holds its own Python interpreter, libraries, and scripts. It’s isolated from your system’s Python, so you can have different versions of packages for each project without interference.

Key benefits for junior/mid devs:

• Isolation: Each project gets its own space. No more “it works on my machine” excuses when sharing code with teammates.
• Reproducibility: Share your environment setup easily, making it simple for others (or future you) to recreate it.
• Cleanliness: Avoid polluting your global Python with project-specific packages. This is crucial on shared servers or when experimenting.
• Version Control Integration: Pair it with Git to track code and dependencies together.

Without venvs, you risk “dependency hell”—a term devs use for when packages clash, leading to hours of debugging. For example, libraries like Django or Flask often pin specific versions of dependencies. Mixing them globally can cause cryptic errors like ImportError or AttributeError.

In my early days, I once broke a production server by installing a package globally that conflicted with system tools. Lesson learned: Always use venvs. It’s a best practice endorsed by the Python community, and tools like Poetry or Conda build on similar ideas, but venv is built-in and lightweight—perfect for starters.

Creating and Activating a Venv

Python 3.3+ comes with the venv module out of the box, so no extra installs needed. Let’s walk through creating one.

First, open your terminal (Command Prompt on Windows, or any shell on macOS/Linux). Navigate to your project directory. For this example, assume we’re in a folder called my_project.

Run this command to create a venv named env (you can name it anything, but env or .venv is common):

text

python -m venv env

What happens? This creates a new directory called env with:

• A copy of your Python interpreter.
• Folders for site-packages (where libraries go).
• Scripts like pip and activate.

On Windows, it might be py -m venv env if you have multiple Python versions.

Now, activate it to switch your shell to use this venv’s Python and pip.

On Unix-based systems (macOS/Linux):

text

source env/bin/activate

On Windows:

text

env\Scripts\activate

Your prompt changes—something like (env) user@machine:~/my_project$. This means you’re in the venv. Any python or pip commands now use the isolated versions.

To test, run:

text

python --version
pip --version

These should show the Python version you used to create the venv, and a fresh pip with no extra packages.

Deactivating is simple:

text

deactivate

Your prompt returns to normal.

Full example workflow in a script-like format (run these in your terminal):

1. Create project dir: mkdir my_project && cd my_project
2. Create venv: python -m venv env
3. Activate: source env/bin/activate (or Windows equivalent)
4. Check: which python (should point to env/bin/python on Unix)
5. Deactivate: deactivate

Pro tip for mid-level devs: Use .venv as the name and add it to .gitignore so it’s not committed to version control. Tools like VS Code auto-detect and activate venvs for you.

If you’re on Python <3.3 (rare these days), use virtualenv from PyPI, but stick with venv for modern setups.

Installing Packages with Pip

Pip is Python’s package installer—think of it as your project’s shopping cart for libraries. Inside a venv, pip installs packages locally, not globally.

Basic install: pip install package_name

For example, let’s install Requests for HTTP requests and Pandas for data handling.

Activate your venv first, then:

text

pip install requests pandas

This downloads and installs them into env/lib/pythonX.Y/site-packages.

To verify:

text

pip list

Output might look like:

text

Package            Version
------------------ ---------
certifi            2023.7.22
charset-normalizer 3.3.2
idna               3.4
numpy              1.26.4     # Installed as a Pandas dependency
pandas             2.2.0
python-dateutil    2.8.2
pytz               2024.1
requests           2.31.0
six                1.16.0
tzdata             2023.4
urllib3            2.2.0

Now, let’s use them in code. Create a file example.py in your project:

Python

import requests
import pandas as pd

# Fetch some data from an API
response = requests.get('https://jsonplaceholder.typicode.com/todos/1')
data = response.json()

# Create a simple DataFrame
df = pd.DataFrame([data])  # Wrap in list for single item

print(df)

Run it: python example.py

Output:

text

userId  id                                              title  completed
1   1  delectus aut autem                             False

See? Isolated and working.

For specific versions: pip install requests==2.28.0

Or from a file (we’ll cover this next): pip install -r requirements.txt

Uninstall: pip uninstall requests

Pip also handles upgrades: pip install –upgrade pip (do this in venvs to keep things fresh).

For mid devs: Use pip search (though deprecated, alternatives like PyPI web search exist) or pip show package for details:

text

pip show requests

Gives version, location, dependencies, etc.

Remember, always activate the venv before pip commands—otherwise, you might install globally by mistake.

Freezing Dependencies with requirements.txt

Once your project is set up, you need to “freeze” the dependencies—capture the exact packages and versions for reproducibility.

Enter requirements.txt: A simple text file listing packages.

To generate it:

Activate venv, then:

text

pip freeze > requirements.txt

This outputs something like:

text

certifi==2023.7.22
charset-normalizer==3.3.2
idna==3.4
numpy==1.26.4
pandas==2.2.0
python-dateutil==2.8.2
pytz==2024.1
requests==2.31.0
six==1.16.0
tzdata==2023.4
urllib3==2.2.0

Commit this to Git. Now, anyone cloning your repo can recreate the env:

1. Create venv: python -m venv env
2. Activate
3. Install: pip install -r requirements.txt

Boom—identical setup.

For dev vs prod, use separate files like requirements-dev.txt with extras like pytest.

Full example: Suppose you have a project with Flask.

Install: pip install flask

Code in app.py:

Python

from flask import Flask

app = Flask(__name__)

@app.route('/')
def hello():
    return "Hello, World!"

if __name__ == '__main__':
    app.run(debug=True)

Freeze: pip freeze > requirements.txt

Contents:

blinker==1.7.0
click==8.1.7
Flask==3.0.2
itsdangerous==2.1.2
Jinja2==3.1.3
MarkupSafe==2.1.5
Werkzeug==3.0.1

To install elsewhere: pip install -r requirements.txt

Advanced: Use pip-compile from pip-tools for locked dependencies, but freeze is fine for basics.

Common Mistakes and Best Practices

Even pros slip up, so let’s cover pitfalls.

Common Mistakes:

1. Forgetting to Activate: You run pip install outside the venv, polluting global space. Fix: Always check your prompt or use pip -V to see the path.
2. Ignoring Versions: Not freezing leads to “works for me” issues. Always generate requirements.txt.
3. Mixing Python Versions: Create venv with the wrong Python (e.g., system 3.8 but project needs 3.11). Solution: Use python3.11 -m venv env if you have multiple installs.
4. Large Venvs in Git: Never commit the env folder—it’s huge and machine-specific. Use .gitignore.
5. Outdated Packages: Forgetting to update can lead to security holes. Run pip list –outdated periodically.

Best Practices:

• One Venv per Project: Standard rule.
• Automate Activation: Use direnv or auto-venv in IDEs.
• Virtualenvwrapper: For mid devs, install this globally (pip install virtualenvwrapper) for commands like mkvirtualenv project—easier management.
• Pipenv or Poetry: Graduate to these for automatic venv + locking.
• Docker Integration: For deployment, combine venvs with containers.
• Testing: Always test installs in a fresh venv to catch issues.

In one project, a junior dev installed TensorFlow globally, conflicting with system CUDA. Switching to venv fixed it instantly.

Wrapping up: Mastering venvs, pip, and dependencies turns chaotic coding into smooth sailing. Start small—try it on your next side project.