This content originally appeared on DEV Community and was authored by sc0v0ne
One of the most used tools within the development flow is the linter. With the linter we can organize and standardize so that developers of the same project always maintain the same standard.
Using as an example, you are on a project with different developers. There is a need to standardize the same code for everyone. From there you choose a linter and all the devs start using it. However, within the project there will not be just one type of file from a specific language, there will be several. You will need to install another linter for another language again. A big effort, right?
Imagine that there is a tool where you can unify all these linters. Could you imagine?
If you imagined pre-commit, congratulations, you are up to date with the technologies, but if you don't know it, don't worry. I will explain below.
Pre-commit
With pre-commit we can unify these tools as given in the example above, using Git hooks, we can be one step ahead of the commit by detecting any inconvenience in the code within development. Having this tool in hand, we define a file called .pre-commit-config.yaml. With it we add hooks for the tools that will pre-evaluate the code. But below it will be explored in more detail.
Below is the project. Don't forget to leave a star on this incredible project.
pre-commit
/
pre-commit
A framework for managing and maintaining multi-language pre-commit hooks.
pre-commit
A framework for managing and maintaining multi-language pre-commit hooks.
For more information see: https://pre-commit.com/
Tutorial
Starting the tutorial, I will be using 3 linters:
- Flake8: For Python files.
- Hadolint: For Docker files.
- Shellcheck: For Shell-Script files.
If you want to follow the explanation along with the project, you can find it at the link below:
sc0v0ne
/
learning_precommit
Learning precommit
Start Application
uvicorn src.app.main:app --reload
Run Tests
./src/tests/test_main.py
Run Pre-commit
./src/scripts/linter.sh
Creating a simple project
Guys, remembering this tutorial and the pre-commit, the codes used are simple with a focus on teaching.
Let's create an empty folder, then open it in the IDE (Integrated development environment). From the directory where you start the IDE, you will create a new folder called src. Inside it you will create two more, one being app and the other tests. Inside each one you will create a file called init.py.
Inside app we will create a new file called main.py. This file will be our initial start of the application, we will use FastAPI to help. Inside the file you will add the following code:
src/app/main.py
from fastapi import FastAPI
app = FastAPI()
@app.get("/")
def read_root():
return {"message": "Hey, Python!!!!!"}
@app.get("/items/{item_id}")
def read_item(item_id: int, q: str = None):
return {"item_id": item_id, "q": q}
In this code we have two paths to the API.
Now pay attention to the following code, let's create a new directory. This directory will be to simulate Python's Alembic. When we have a service using a database that needs a migration manager. This directory was created without any external connection or use. It is being created to apply the use of linter, simulating a real situation. Inside the app, we will create a directory called alembic, it has the following file structure.
src/
....app/
........__init__.py
........main.py
........alembic/
...............versions/
.......................1975ea83b712_create_account_table.py
................env.py
................README
................script.py.mako
src/app/alembic/versions
1975ea83b712_create_account_table.py
"""create account table
Revision ID: 1975ea83b712
Revises:
Create Date: 2011-11-08 11:40:27.089406
"""
# Example File
# revision identifiers, used by Alembic.
revision = '1975ea83b712'
down_revision = None
branch_labels = None
from alembic import op
import sqlalchemy as sa
def upgrade():
pass
def downgrade():
pass
src/app/alembic/
- env.py
- README
- script.py.mako
Now let's create a new file inside the tests directory, called test_main.py. This way we can check if our paths are passing the tests. Inside the test_main.py file, you will add the following code.
src/tests/test_main.py
from fastapi.testclient import TestClient
from app.main import app
client = TestClient(app)
def test_read_root():
response = client.get("/")
assert response.status_code == 200
assert response.json() == {"message": "Hey, Python!!!!!"}
def test_read_item():
response = client.get("/items/42?q=test")
assert response.status_code == 200
assert response.json() == {"item_id": 42, "q": "test"}
Outside the src directory, you will create a file called requirements.txt. Inside it you will add the following packages.
requirements.txt
fastapi==0.95.2
uvicorn==0.22.0
pytest==7.4.0
httpx==0.24.1
Time of truth, is it working?
First let's run our application.
uvicorn src.app.main:app --reload
# Output
INFO: Will watch for changes in these directories: ['/home/example/Workspace/learning_precommit']
INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO: Started reloader process [50974] using StatReload
INFO: Started server process [50976]
INFO: Waiting for application startup.
INFO: Application startup complete.
Let's access our URL.
Alright, let's go through the tests in our application.
pytest ./src/tests/test_main.py
#output
============================================ test session starts ============================================
platform linux -- Python 3.12.4, pytest-7.4.0, pluggy-1.5.0
rootdir: /home/example/Workspace/learning_precommit
plugins: anyio-4.4.0
collected 2 items
src/tests/test_main.py .. [100%]
============================================= 2 passed in 0.30s =============================================
Very good, we managed to create a simple application to apply the linter.
pre-commit
We have reached the long-awaited stage. Let's apply linters to our application using pre-commit.
Outside in the src directory, let's create a file called .pre-commit-config.yaml. This file will contain our hooks for each type of file we want to pass the linter to.
After creating the file, let's start adding the hooks:
Starting with Python, we will use flake8. You can choose other options, initially for this project I used this resource.
- repos: Our group of repositories containing configurations for hooks.
- repo: repository URL
- rev: revision or tag to clone
- hooks: Group of hooks
- id: Identification for output at the terminal
- additional_dependencies: Additional dependencies
- language_version: Version of Python being used
- exclude: A path or several to directories that will not be seen when executing the pre-commit, to use as an example, it was our alembic directory, which we do not want to be standardized, to remain the way I start or receive subsequent updates , for greater safety, I avoid putting the linter on it as a practice.
- files: Location containing the files to be viewed by pre-commit, if not declared, the entire project will be viewed.
Flake8
repos:
- repo: https://github.com/pycqa/flake8
rev: 6.0.0
hooks:
- id: flake8
additional_dependencies: [flake8==6.0.0]
language_version: python3.9
exclude: ^src/app/alembic
files: ^src/
PyCQA
/
flake8
flake8 is a python tool that glues together pycodestyle, pyflakes, mccabe, and third-party plugins to check the style and quality of some python code.
Flake8
Flake8 is a wrapper around these tools:
- PyFlakes
- pycodestyle
- Ned Batchelder's McCabe script
Flake8 runs all the tools by launching the single flake8 command
It displays the warnings in a per-file, merged output.
It also adds a few features:
-
files that contain this line are skipped:
# flake8: noqa
-
lines that contain a
# noqacomment at the end will not issue warnings. -
you can ignore specific errors on a line with
# noqa: <error>, e.g.,# noqa: E234. Multiple codes can be given, separated by comma. Thenoqatoken is case insensitive, the colon before the list of codes is required otherwise the part afternoqais ignored -
Git and Mercurial hooks
-
extendable through
flake8.extensionandflake8.formattingentry points
Quickstart
See our quickstart documentation for how to install and get started with Flake8.
Frequently Asked Questions
Flake8 maintains an FAQ in its documentation.
Questions or Feedback
…In the following hooks we have two differences, in the args, there are two ways that we can pass something that we want to ignore with the linter.
Hadolint
For Dockerfiles, we will use Hadolint. I've only been using it for a short time and I can't do without it anymore.
- repo: https://github.com/AleksaC/hadolint-py
rev: v2.12.1b3
hooks:
- id: hadolint
args: [--ignore, DL3008, --ignore , DL3007, --ignore, DL4006]
exclude: ^src/app/alembic
AleksaC
/
hadolint-py
Run hadolint in pre-commit without docker or system installation
hadolint-py
A python package that provides a pip-installable hadolint binary.
The mechanism by which the binary is downloaded is basically copied from shellcheck-py.
Getting started
Installation
The package hasn't been published to PyPI yet, and may never be, as its primary purpose doesn't require it. However you can install it through git:
pip install git+https://github.com/AleksaC/hadolint-py.git@v2.12.1-beta
To install another version simply replace the v2.12.0 with the version you want.
With pre-commit
This package was primarily built to provide a convenient way of running hadolint as a pre-commit hook, since haskell isn't supported by pre-commit. An alternative to this solution is to create a docker hook since hadolint provides a docker image, but I think that it has unnecessary amount of overhead.
Example .pre-commit-config.yaml with rules DL3025 and DL3018 excluded:
repos:
- repo: https://github.com/AleksaC/hadolint-py
rev: v2.12.1b3
hooks:
- id: hadolint
args: [--ignore, DL3025, --ignore, DL3018]
Shellcheck
For shell-script files, quite useful, as development throughout the day when you add incorrect patterns in the script, this tool points out the points for standardization.
- repo: https://github.com/shellcheck-py/shellcheck-py
rev: v0.10.0.1
hooks:
- id: shellcheck
args:
- --exclude=SC2046
- --exclude=SC2006
exclude: ^src/app/alembic
shellcheck-py
/
shellcheck-py
python3/pip3 wrapper for installing shellcheck
shellcheck-py
A python wrapper to provide a pip-installable shellcheck binary.
Internally this package provides a convenient way to download the pre-built shellcheck binary for your particular platform.
installation
pip install shellcheck-py
usage
After installation, the shellcheck binary should be available in your
environment (or shellcheck.exe on windows).
As a pre-commit hook
See pre-commit for instructions
Sample .pre-commit-config.yaml:
- repo: https://github.com/shellcheck-py/shellcheck-py
rev: v0.10.0.1
hooks:
- id: shellcheck
.pre-commit-config.yaml
repos:
- repo: https://github.com/pycqa/flake8
rev: 6.0.0
hooks:
- id: flake8
additional_dependencies: [flake8==6.0.0]
language_version: python3.9
exclude: ^src/app/alembic
files: ^src/
- repo: https://github.com/AleksaC/hadolint-py
rev: v2.12.1b3
hooks:
- id: hadolint
args: [--ignore, DL3008, --ignore , DL3007, --ignore, DL4006]
exclude: ^src/app/alembic
- repo: https://github.com/shellcheck-py/shellcheck-py
rev: v0.10.0.1
hooks:
- id: shellcheck
args:
- --exclude=SC2046
- --exclude=SC2006
exclude: ^src/app/alembic
With the file defined, let's put the pre-commit to run. I'm going to do it a little differently than the documentation, you can consult the documentation for more information. If you can run pip install pre-commit and start using its commands and run the .pre-commit-config.yaml file.
The way I will use pre-commit will be using a Dockerfile. Let's now create a directory called containers, inside it we will add a file called Dockerfile. Then add the code below:
Dockerfile
FROM python:3.9-slim
ARG DEBIAN_FRONTEND=noninteractive
RUN apt-get update && \
apt-get install -y --no-install-recommends git && \
rm -rf /var/lib/apt/lists/*
WORKDIR /linter
RUN pip install --no-cache-dir pre-commit
COPY containers/ containers/
COPY scripts/ scripts/
COPY src/ src/
COPY .pre-commit-config.yaml .
COPY .git .git
CMD ["pre-commit", "run", "--all-files"]
Notice in our Dockerfile that I add the command to install Git, this is necessary because pre-commit uses Git hooks. Next we will perform the pre-commit installation.
Now let's copy our project into the container, see that even the .pre-commit-config.yaml file is being copied, this will be necessary to execute our configurations inside the container.
Now let's create 2 shell-script files to run our container. Let's create a new directory called scripts. Inside it we will create two files build.sh and linter.sh.
scripts/build.sh
In this file we will add our build command for our Dockerfile.
#!/bin/bash
docker build . -f containers/Dockerfile -t project_linter:latest --rm
scripts/linter.sh
Then in the linter.sh file we will add the following code. Note that I pass the docker commands within conditions, this way I can run the linter for each type of file, if the linter detects any correction, the script will stop running until it is adjusted, to continue with the next step.
#!/bin/bash
./scripts/build.sh
docker run --rm project_linter:latest pre-commit run --all-files flake8
status=$?
if test $status -ne 0
then
exit $status
fi
docker run --rm project_linter:latest pre-commit run --all-files hadolint
status=$?
if test $status -ne 0
then
exit $status
fi
docker run --rm project_linter:latest pre-commit run --all-files shellcheck
status=$?
if test $status -ne 0
then
exit $status
fi
Testing the linter
Now is the moment of truth.
Let's run our script file, ./scripts/linter.sh.
[+] Building 6.3s (14/14) FINISHED docker:default
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 424B 0.0s
=> [internal] load metadata for docker.io/library/python:3.9-slim 1.1s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [1/9] FROM docker.io/library/python:3.9-slim@sha256:a6c12ec09f13df9d4b8b4e4d08678c1b212d89885be14b6c72b6 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 11.66kB 0.0s
=> CACHED [2/9] RUN apt-get update && apt-get install -y --no-install-recommends git && rm -rf /var 0.0s
=> CACHED [3/9] WORKDIR /linter 0.0s
=> [4/9] RUN pip install --no-cache-dir pre-commit 4.8s
=> [5/9] COPY containers/ containers/ 0.0s
=> [6/9] COPY scripts/ scripts/ 0.0s
=> [7/9] COPY src/ src/ 0.0s
=> [8/9] COPY .pre-commit-config.yaml . 0.0s
=> [9/9] COPY .git .git 0.0s
=> exporting to image 0.2s
=> => exporting layers 0.1s
=> => writing image sha256:2005b56faa9f3ad9304712b10515d759a5e84ec553924f6b24d5f0be30d7e664 0.0s
=> => naming to docker.io/library/project_linter:latest 0.0s
[INFO] Initializing environment for https://github.com/pycqa/flake8.
[INFO] Initializing environment for https://github.com/pycqa/flake8:flake8==6.0.0.
[INFO] Initializing environment for https://github.com/AleksaC/hadolint-py.
[INFO] Initializing environment for https://github.com/shellcheck-py/shellcheck-py.
[INFO] Installing environment for https://github.com/pycqa/flake8.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
flake8...................................................................Failed
- hook id: flake8
- exit code: 1
src/app/main.py:5:1: E302 expected 2 blank lines, found 1
src/app/main.py:9:1: E302 expected 2 blank lines, found 1
src/app/main.py:11:40: W292 no newline at end of file
src/tests/test_main.py:6:1: E302 expected 2 blank lines, found 1
src/tests/test_main.py:11:1: E302 expected 2 blank lines, found 1
src/tests/test_main.py:14:59: W292 no newline at end of file
Look what we have, we have several lines to be corrected. We will correct it in the next run again. Note below that our Flake8 linter has now passed. But our Dockerfile needs fixes, let's fix it then run it again.
[+] Building 0.7s (14/14) FINISHED docker:default
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 424B 0.0s
=> [internal] load metadata for docker.io/library/python:3.9-slim 0.5s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [1/9] FROM docker.io/library/python:3.9-slim@sha256:a6c12ec09f13df9d4b8b4e4d08678c1b212d89885be14b6c72b6 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 7.52kB 0.0s
=> CACHED [2/9] RUN apt-get update && apt-get install -y --no-install-recommends git && rm -rf /var 0.0s
=> CACHED [3/9] WORKDIR /linter 0.0s
=> CACHED [4/9] RUN pip install --no-cache-dir pre-commit 0.0s
=> CACHED [5/9] COPY containers/ containers/ 0.0s
=> CACHED [6/9] COPY scripts/ scripts/ 0.0s
=> [7/9] COPY src/ src/ 0.0s
=> [8/9] COPY .pre-commit-config.yaml . 0.0s
=> [9/9] COPY .git .git 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:605dd086f6a94f8d465092eaa8d027860358ce99e20bac4ec31a3e87199f5c7e 0.0s
=> => naming to docker.io/library/project_linter:latest 0.0s
[INFO] Initializing environment for https://github.com/pycqa/flake8.
[INFO] Initializing environment for https://github.com/pycqa/flake8:flake8==6.0.0.
[INFO] Initializing environment for https://github.com/AleksaC/hadolint-py.
[INFO] Initializing environment for https://github.com/shellcheck-py/shellcheck-py.
[INFO] Installing environment for https://github.com/pycqa/flake8.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
flake8...................................................................Passed
[INFO] Initializing environment for https://github.com/pycqa/flake8.
[INFO] Initializing environment for https://github.com/pycqa/flake8:flake8==6.0.0.
[INFO] Initializing environment for https://github.com/AleksaC/hadolint-py.
[INFO] Initializing environment for https://github.com/shellcheck-py/shellcheck-py.
[INFO] Installing environment for https://github.com/AleksaC/hadolint-py.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
Hadolint.................................................................Failed
- hook id: hadolint
- exit code: 1
containers/Dockerfile:11 DL3013 warning: Pin versions in pip. Instead of `pip install <package>` use `pip install <package>==<version>` or `pip install --requirement <requirements file>`
Corrections done successfully, all our linter passed.
[+] Building 0.7s (14/14) FINISHED docker:default
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 431B 0.0s
=> [internal] load metadata for docker.io/library/python:3.9-slim 0.5s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [1/9] FROM docker.io/library/python:3.9-slim@sha256:a6c12ec09f13df9d4b8b4e4d08678c1b212d89885be14b6c72b6 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 7.14kB 0.0s
=> CACHED [2/9] RUN apt-get update && apt-get install -y --no-install-recommends git && rm -rf /var 0.0s
=> CACHED [3/9] WORKDIR /linter 0.0s
=> CACHED [4/9] RUN pip install --no-cache-dir pre-commit==3.5.0 0.0s
=> CACHED [5/9] COPY containers/ containers/ 0.0s
=> CACHED [6/9] COPY scripts/ scripts/ 0.0s
=> [7/9] COPY src/ src/ 0.0s
=> [8/9] COPY .pre-commit-config.yaml . 0.0s
=> [9/9] COPY .git .git 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:1707d54754106d4eb8844d546a6b7a29b708ae2d1106e973d3d7d9b0e1c2b219 0.0s
=> => naming to docker.io/library/project_linter:latest 0.0s
[INFO] Initializing environment for https://github.com/pycqa/flake8.
[INFO] Initializing environment for https://github.com/pycqa/flake8:flake8==6.0.0.
[INFO] Initializing environment for https://github.com/AleksaC/hadolint-py.
[INFO] Initializing environment for https://github.com/shellcheck-py/shellcheck-py.
[INFO] Installing environment for https://github.com/pycqa/flake8.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
flake8...................................................................Passed
[INFO] Initializing environment for https://github.com/pycqa/flake8.
[INFO] Initializing environment for https://github.com/pycqa/flake8:flake8==6.0.0.
[INFO] Initializing environment for https://github.com/AleksaC/hadolint-py.
[INFO] Initializing environment for https://github.com/shellcheck-py/shellcheck-py.
[INFO] Installing environment for https://github.com/AleksaC/hadolint-py.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
Hadolint.................................................................Passed
[INFO] Initializing environment for https://github.com/pycqa/flake8.
[INFO] Initializing environment for https://github.com/pycqa/flake8:flake8==6.0.0.
[INFO] Initializing environment for https://github.com/AleksaC/hadolint-py.
[INFO] Initializing environment for https://github.com/shellcheck-py/shellcheck-py.
[INFO] Installing environment for https://github.com/shellcheck-py/shellcheck-py.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
shellcheck...............................................................Passed
My Latest Posts
About the author:
A little more about me...
Graduated in Bachelor of Information Systems, in college I had contact with different technologies. Along the way, I took the Artificial Intelligence course, where I had my first contact with machine learning and Python. From this it became my passion to learn about this area. Today I work with machine learning and deep learning developing communication software. Along the way, I created a blog where I create some posts about subjects that I am studying and share them to help other users.
I'm currently learning TensorFlow and Computer Vision
Curiosity: I love coffee
This content originally appeared on DEV Community and was authored by sc0v0ne
sc0v0ne | Sciencx (2024-07-20T01:49:21+00:00) Creating a Dockerfile to Perform pre-commit. Retrieved from https://www.scien.cx/2024/07/20/creating-a-dockerfile-to-perform-pre-commit/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.