This content originally appeared on Level Up Coding - Medium and was authored by Rukshan J. Senanayaka
Get Started with Poetry and pyproject.toml for the beginner: Modernizing Python Project Management for AI apps
Introduction
In recent years, Python has seen a surge in popularity, particularly in the realm of artificial intelligence and web development. With frameworks like FastAPI gaining traction in the AI-driven modern world, managing Python projects efficiently has become more crucial than ever. As projects grow in complexity, developers often face challenges with dependency management, packaging, and deployment.
Traditionally, Python developers have relied on pip and requirements.txt files to manage dependencies. However, this approach comes with its own set of problems, which we’ll explore in this article. We’ll then introduce Poetry and pyproject.toml as modern solutions to these issues, demonstrating how they can streamline your Python development workflow.
The Problems with existing pip and requirements.txt
Dependency Hell
One of the most significant issues with using pip and requirements.txt is the infamous “dependency hell.” This occurs when different packages in your project require conflicting versions of the same dependency. Resolving these conflicts manually can be time-consuming and error-prone.
Lack of Lock Files
requirements.txt files don’t provide a built-in mechanism for locking dependency versions. This can lead to inconsistencies between development and production environments, as well as among team members working on the same project.
Limited Metadata
requirements.txt files are simple text files that only list dependencies and their versions. They lack important metadata about the project itself, such as the project name, version, author, and other details that are crucial for packaging and distribution. If you are familiar with node.js development, think about the well-structured package.json file
No Built-in Virtual Environment Management
While virtual environments are essential for isolating project dependencies, pip and requirements.txt don’t provide built-in tools for creating and managing them. Developers often need to use separate tools like virtualenv or venv.
Inconsistent Dependency Resolution
pip’s dependency resolution algorithm has improved over time, but it can still struggle with complex dependency graphs, sometimes leading to suboptimal or inconsistent installations.
Enter Poetry and pyproject.toml
Poetry is a modern dependency management and packaging tool for Python that aims to solve many of the issues associated with pip and requirements.txt. It uses the pyproject.toml file, which is a standardized configuration file for Python projects.
Benefits of Using Poetry and pyproject.toml
- Dependency Resolution: Poetry uses a powerful dependency resolver that can handle complex dependency graphs, reducing the likelihood of conflicts.
- Lock Files: Poetry generates a poetry.lock file that ensures consistent installations across different environments and team members.
- Project Metadata: pyproject.toml allows you to specify detailed project metadata in a standardized format, much like the package.json file in node.js development
- Virtual Environment Management: Poetry can create and manage virtual environments for your projects automatically.
- Build and Packaging: Poetry simplifies the process of building and packaging Python projects for distribution.
- Standardization: pyproject.toml is a PEP 518 compliant file, promoting standardization across Python tools and projects.
Getting Started with Poetry and pyproject.toml
Let’s walk through a minimal example of setting up a FastAPI project using Poetry and PyCharm on macOS.
Assumptions
This article assumes you have already the following system on your mac machine. The steps should be similar for other OSs but not guaranteed.
- OS: MacOS
- IDE: PyCharm
Installation and Setup
- Install Poetry using pipx
pipx install poetry
2. Verify the installation
poetry --version
3. Open PyCharm and create a new project
- Select “FastAPI” as the project template
- Choose “Poetry” as the environment instead of venv
- Click “Create”
Project Structure
After creating the project, you’ll see a basic FastAPI project structure with a pyproject.toml file at the root. This file will contain your project configuration and dependencies.
The directory structure of the project will look something like below.
.
├── main.py
├── poetry.lock
├── pyproject.toml
└── test_main.http
The main.py file is the entry point of your application, containing the main code.
The poetry.lock file is automatically generated and maintained by Poetry, ensuring consistent dependency versions across different environments.
The pyproject.toml file is the configuration file for your project, specifying project metadata, dependencies, and build system requirements.
Finally, test_main.http is a file used for HTTP request testing, containing sample requests to test your API endpoints.
This structure reflects a modern Python project setup, emphasizing dependency management with Poetry and providing a clear separation between application code, project configuration, and testing resources.
Adding Dependencies (optional)
To add dependencies to your project, you can use the poetry add command or manually edit the pyproject.toml file.
For example, to add FastAPI and Uvicorn you can execute the following command. But since we’ve used the PyCharm template for Fast API, these are already installed and added to the pyproject.toml and you don’t need to do this manually here.
poetry add fastapi uvicorn
This will update your pyproject.toml file and create or update the poetry.lock file.
Running Your FastAPI Application
To run your FastAPI application, you can either,
- use Poetry’s run command
- use the PyCharm’s run button
Using the Poetry’s run command,
poetry run uvicorn main:app --reload
This ensures that your application runs within the Poetry-managed virtual environment.
Testing with Poetry
Poetry makes it easy to manage test dependencies and run tests for your project.
- Add test dependencies
poetry add --dev pytest httpx
This should add the following line to pyproject.toml (your version may differ)
[tool.poetry.group.dev.dependencies]
pytest = "^8.3.2"
2. Create a tests directory in your project root and add your test files
Create an emtpy file tests/__init__.py (although empty it must be present otherwise you’ll get module errors when run using the poetry command.
As a simple test case, you can test the default “hello world” endpoint by creating a file test_main.py with following content
from fastapi.testclient import TestClient
from main import app
client = TestClient(app)
def test_read_main():
response = client.get("/")
assert response.status_code == 200
assert response.json() == {"message": "Hello World"}
def test_read_hello():
response = client.get("/hello/TestUser")
assert response.status_code == 200
assert response.json() == {"message": "Hello TestUser"}
Now the directory should look something like below
.
├── main.py
├── poetry.lock
├── pyproject.toml
├── test_main.http
└── tests
├── __init__.py
└── test_main.py
3. Run tests using Poetry
poetry run pytest
This approach ensures that your tests run in the same environment as your application, maintaining consistency between development and testing.
Conclusion
Poetry and pyproject.toml offer a modern, robust solution to many of the problems faced by Python developers when managing dependencies and projects. By providing powerful dependency resolution, consistent environments, and standardized project metadata, they simplify the development workflow and reduce common pitfalls.
As Python continues to grow in popularity, especially in AI and web development domains, tools like Poetry become increasingly valuable. They allow developers to focus more on writing code and less on managing dependencies and project configurations.
While there may be a small learning curve when transitioning from pip and requirements.txt, the benefits of using Poetry and pyproject.toml far outweigh the initial investment. As you work on more complex Python projects, you’ll find that these tools save time, reduce errors, and promote better collaboration among team members.
By adopting Poetry and pyproject.toml in your Python development workflow, you’re not just solving immediate problems — you’re future-proofing your projects and aligning with modern Python best practices.
References
PEP 518 - Specifying Minimum Build System Requirements for Python Projects | peps.python.org
Get Started with Poetry and pyproject.toml was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.
This content originally appeared on Level Up Coding - Medium and was authored by Rukshan J. Senanayaka
Rukshan J. Senanayaka | Sciencx (2024-08-15T18:04:27+00:00) Get Started with Poetry and pyproject.toml. Retrieved from https://www.scien.cx/2024/08/15/get-started-with-poetry-and-pyproject-toml/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.