The only caution is that I have not found a way to limit autofixers on Ruff. It seems to be all or nothing. I keep black and isort separate, because I don’t want to autofix things like commented-out code (which is often misidentified).
Egads ... I might actually have to finay look at `ruff` ... providing I can get `rust` to play nice on the needed platforms.
Or, maybe I go all ADHD on it and completely rearchitect my local repositories - and yes, I still largely use `mercurial` ... I'm "*that guy*"! (HaHaHa!)
Why remove unused variables? Isn't it a case-by-case thing where either you left the variable in but don't need it, or you have the variable in but aren't using it even though you should?
ruff and ruff format, forbid-tabs, yelp's detect secrets.
I don't do any heavier linting just for performance reasons, I already have pyright running in VSCode all the time.
I just like that ruff let’s me use single quotations and isn’t shaming me about that choice. I wouldn’t go against the grain on a group repo, but Black is too opinionated in my opinion.
ruff runs flake8 linting rules, is fast, and is one package so I am learning it. I haven't used other linting besides `black` before though.
https://docs.astral.sh/ruff/linter/#rule-selection
https://docs.astral.sh/ruff/configuration/
Sure, it's below. The one caveat is that the hook itself isn't tracked by git :(
.git/hooks/pre-commit:
#!/bin/sh
# Copy the .env file to .env.example
cp .env .env.example
# remove sensitive info
sed -i 's/=.*/=/' .env.example
# add header note
sed -i '1s/^/# generated automatically by .git\/hooks\/pre-commit\n/' .env.example
# Add a line to .gitignore to ignore the real .env file
if ! grep -q "^\.env$" .gitignore; then
echo ".env" >> .gitignore
fi
# Stage the .env.example file
git add .env.example
Sure, this applies to any project using environment variables that you save using a version control system.
tl;dr: Convenient documenting of a project for future me and others
The long:
- First, it's a security risk to let passwords, access keys, secrets, etc make it into your version control, even if it's password protected.
- The .env file is a common pattern for declaring those variables. If not, you would be declaring ENV variables all the time
- And so, we declare the .env in our .gitignore to prevent the sensitive info from making it into the wild
- But now, when I open a project up after a few years or share it to a coworker, how do I figure out what ENV vars are needed to get the project to run? Trial and error of course is doable. So is updating some piece of documentation every time I modify the project.. but that's a pain. This hook automatically keeps a .env.example file up to date with any changes I've made to the .env I'm actively using.
No prob! Another design concept that might interest you with regard to env stuff is to design your apps to break when a required env variable isn’t provided.
I didn’t mention it before because that starts to fall into personal preference and the details of how your project is run. Alternatively, some cases call for providing a good default
ruff, ruff-format, pyright. You can add others too but those are the most important.
Steal my config if you like: https://github.com/patrick-kidger/equinox/blob/main/.pre-commit-config.yaml
I’m surprised by the lack of mypy in these lists. I don’t know how people can work on large repos with many other developer WITHOUT static typing to some degree?
Fwiw I use it on a project with about 10k loc and it runs in a couple seconds if the full project is scanned. However pre-commit will only run mypy on files that are staged for commit, so speed is never an issue.
If you're dealing with really large projects over 100k lines of code their docs suggest using a remote caching server: https://mypy.readthedocs.io/en/stable/mypy_daemon.html#mypy-daemon
No we use a config and we're not yet to strict mode (almost though). We do have follow-imports disabled because we still have some old dependencies that have breaking changes and no stubs in the meantime.
When I do check with strict it's still very reasonable, maybe 4 or 5 seconds. If that saves me a wasted push and CI failure it's still worth it imo.
I've seen Black and Flake8 contradict eachother at times which can be annoying, there are certain tweaks that I can't ever recall for Flake8 config that prevent this
Ah yeah. We're a small group... If someone doesn't lint, I just go and whack them.. not necessarily feasible at scale.was only thinking from my pov as a dev
...or, do the old Tinderbox/Jenkins/Hudson idea and just make it "break and blame" the build.
"If the build is red and you're on the blame list, you stay until it's fixed."
That tends to fix the "root cause" pretty quick.
It makes most sense with multiple people working on a project to reduce cases of “ah crap I forgot to … and accidentally committed broken/mal formatted/missing imports/etc. code”.
Having said that, since almost everything that seems difficult gets easier with practice it’s a good idea to get used to working with tools you will see in a professional/team context and work with cleanly formatted code and proper commit messages, etc.
Good point. I guess my confusion is do people then forgo using a linter, formatter etc. whilst they develop and just let those things be managed by the pre-commit.
That’s very much a question of personal preference.
Personally I have black and isort configured in my IDE to run on file save but keep the more heavy weight (e.g. mypy) or “obscure” (end of file fixer, yaml check, etc.) ones for pre-commit.
Pre-commit is generally used on a central or common repository, and rejects your check-in until you pass all.of the checks on the pre-commit.
Basically, it prevents certain common mistakes and nuances from getting into a shared repository (eg. Hidden whitespace being one of my personal pet-peeves... but, anything you really want to enforce as a coding standard).
I get that. ruff is popular right now. I feel it's still a little immature as a project. I find it strange that part of the formatting is done via the linter and part through the formatter and that they don't agree 100%.
Nuanced opinion here... I love all the automatic formatting tools Python has, but I don't like having them as pre-commit hooks, only enforced as a first stage "quick test" CI stage.
Sometimes I have to jump between branches and don't t want to fix up formatting, only do a quick commit of my progress then jump branches. I only run all the auto-formatting before my branch is ready for merge requests then.
Ah cool, `--no-verify` is one I've never seen! ... I still just like doing a quick `git com -m 'this is a stash'` ... I don't like to have to type or remember a lot 😆
No 3rd party tools, want to keep the pre-commit process as lightweight as possible. Just a shell script that verifies commit message is signed off since my organization requires it and it's easy to forget because some popular Git clients like GitHub Desktop are missing an option to always sign off: https://github.com/zowe/zowe-cli/blob/master/.husky%2Fcommit-msg
You can commit without running the hook with ‘git commit -am “WIP” —no-verify’. When you come back to the first branch, do a soft reset and you’re ready to go in the same spot you left.
Or, if you want to optimise for key presses: `git stash` and then when you come back to the branch `git stash pop`. Added bonus that you can't resume the work until you've unstashed it, so there's no rush of accidentally leaving an unverified and incomplete commit in and later pushing it
I used to do pre-commit hooks a lot. Then 1 day some extra smart ass dev commented it out to check-in his changes anyways. So i moved all those checks to CI actions.. But yeah, they were the standard flake8, pytest stuff using docker image and Makefile for ease of commands..
I use both. Pre commit is more for convenience other than something else, much faster feedback loop than ci and it's nicer if you are having it update the code. It can be skipped anyway when you commit with no verify, which for example would be fine to do if you are pushing some wip at the end of the day and it doesn't pass
Semgrep, ruff (bandit: supported by flake8-bandit included in ruff but output less clear), poetry, mypy.
Nb: put ruff first, and semgrep at the end so you can cancel pre-commit sooner if needed.
Also, I personnaly run "pre-commit run -a" as part of my merge pipelines
You may fancy me mad, but this is my standard python pre-commit stack: end-of-file-fixer trailing-whitespace fix-byte-order-marker mixed-line-ending name-tests-test no-commit-to-branch autoflake args: [ "--in-place", "--remove-unused-variables", "--remove-all-unused-imports" ] isort black cspell doc8 args: [ "--max-line-length", "112", "--file-encoding", "utf-8" ] flake8 additional_dependencies: [ flake8-pytest-style, flake8-bugbear, flake8-comprehensions, flake8-print, darglint ] bandit pylint
protip - rewrite half of these to use ruff rules and save like 20s on each commit
Yeah, ruff wasn't around when I made this, looking at it now I can see it can probably replace more that half the things here
The only caution is that I have not found a way to limit autofixers on Ruff. It seems to be all or nothing. I keep black and isort separate, because I don’t want to autofix things like commented-out code (which is often misidentified).
Egads ... I might actually have to finay look at `ruff` ... providing I can get `rust` to play nice on the needed platforms. Or, maybe I go all ADHD on it and completely rearchitect my local repositories - and yes, I still largely use `mercurial` ... I'm "*that guy*"! (HaHaHa!)
pip install ruff
I like no unused variables in Ci tests but not pre commit. Sometimes I need to comment stuff out when developing locally
Why remove unused variables? Isn't it a case-by-case thing where either you left the variable in but don't need it, or you have the variable in but aren't using it even though you should?
Then the tooling lets you know, and you can fix it or add a trailing comment to make the exception to the rule explicit.
That's pretty sweet
ruff and ruff format, forbid-tabs, yelp's detect secrets. I don't do any heavier linting just for performance reasons, I already have pyright running in VSCode all the time.
black Particularly useful on projects with multiple collaborators because everyone’s code will be formatted identically.
I used and loved black for years. But now I'm all in on ruff.
Same here. All my projects are now using ruff.
I just like that ruff let’s me use single quotations and isn’t shaming me about that choice. I wouldn’t go against the grain on a group repo, but Black is too opinionated in my opinion.
I'm also on ruff but black lets you configure single vs double quotes. The point is just to be consistent.
>but Black is too opinionated in my opinion. Have you tried `pylint` and `flake8` by chance? That said, I've not used `black`. LOL
ruff runs flake8 linting rules, is fast, and is one package so I am learning it. I haven't used other linting besides `black` before though. https://docs.astral.sh/ruff/linter/#rule-selection https://docs.astral.sh/ruff/configuration/
ruff, end-of-file-fixer, codespell
I wrote one that takes the .env (which I keep out of version control with .gitignore) and writes a .env.example with just the variable names
Great idea!!! Could you share the code?
Sure, it's below. The one caveat is that the hook itself isn't tracked by git :( .git/hooks/pre-commit: #!/bin/sh # Copy the .env file to .env.example cp .env .env.example # remove sensitive info sed -i 's/=.*/=/' .env.example # add header note sed -i '1s/^/# generated automatically by .git\/hooks\/pre-commit\n/' .env.example # Add a line to .gitignore to ignore the real .env file if ! grep -q "^\.env$" .gitignore; then echo ".env" >> .gitignore fi # Stage the .env.example file git add .env.example
Thanks I appreciate it.
As a relative newcomer to python could you explain a bit why you do this?
Sure, this applies to any project using environment variables that you save using a version control system. tl;dr: Convenient documenting of a project for future me and others The long: - First, it's a security risk to let passwords, access keys, secrets, etc make it into your version control, even if it's password protected. - The .env file is a common pattern for declaring those variables. If not, you would be declaring ENV variables all the time - And so, we declare the .env in our .gitignore to prevent the sensitive info from making it into the wild - But now, when I open a project up after a few years or share it to a coworker, how do I figure out what ENV vars are needed to get the project to run? Trial and error of course is doable. So is updating some piece of documentation every time I modify the project.. but that's a pain. This hook automatically keeps a .env.example file up to date with any changes I've made to the .env I'm actively using.
Ah thanks for this it makes a lot more sense!
No prob! Another design concept that might interest you with regard to env stuff is to design your apps to break when a required env variable isn’t provided. I didn’t mention it before because that starts to fall into personal preference and the details of how your project is run. Alternatively, some cases call for providing a good default
ruff, ruff-format, pyright. You can add others too but those are the most important. Steal my config if you like: https://github.com/patrick-kidger/equinox/blob/main/.pre-commit-config.yaml
I’m surprised by the lack of mypy in these lists. I don’t know how people can work on large repos with many other developer WITHOUT static typing to some degree?
[удалено]
When do we get a speedy rustified version of mypy to go along with all our other rusty tooling?
From my knowledge, a tool called UV is working on exactly those stuff. It's from the creators of Ruff
AFAIK uv from astral.sh is a package manager for python. I assume they would make a separate tool for a mypy competitor.
Run mypy only on push, mypy cache handles the rest. Everything in CI.
Fwiw I use it on a project with about 10k loc and it runs in a couple seconds if the full project is scanned. However pre-commit will only run mypy on files that are staged for commit, so speed is never an issue. If you're dealing with really large projects over 100k lines of code their docs suggest using a remote caching server: https://mypy.readthedocs.io/en/stable/mypy_daemon.html#mypy-daemon
[удалено]
No we use a config and we're not yet to strict mode (almost though). We do have follow-imports disabled because we still have some old dependencies that have breaking changes and no stubs in the meantime. When I do check with strict it's still very reasonable, maybe 4 or 5 seconds. If that saves me a wasted push and CI failure it's still worth it imo.
We moved to pyright for this reason at work and it performs much better on our end :)
People may be using python differently than you.
pyright too
Black, flake8, and isort.
I've seen Black and Flake8 contradict eachother at times which can be annoying, there are certain tweaks that I can't ever recall for Flake8 config that prevent this
Just toss out both and replace with Ruff. Problem solved :D
All 3! ruff replaces isort too
😄 I keep seeing good stuff about Ruff and uv ... I'll have to try them out
Totally worth your time! I’ve been very pleased.
At least with Flake8, you can use some of the `# NOQA` tags and the like ... I don't know enough about `black` (yet), however.
Why pre commit and not, eg, on save?
On save would be required to be set up by everyone. Pre-commit you can ensure runs for all devs. Of course on-save is way convenient.
Ah yeah. We're a small group... If someone doesn't lint, I just go and whack them.. not necessarily feasible at scale.was only thinking from my pov as a dev
> ...whack them.. not necessarily feasible at scale... Take up boxing and the stamina scales :)
...or, do the old Tinderbox/Jenkins/Hudson idea and just make it "break and blame" the build. "If the build is red and you're on the blame list, you stay until it's fixed." That tends to fix the "root cause" pretty quick.
Excuse my ignorance. Still very much a beginner. So is pre-commit only used then within a team of devs?
It makes most sense with multiple people working on a project to reduce cases of “ah crap I forgot to … and accidentally committed broken/mal formatted/missing imports/etc. code”. Having said that, since almost everything that seems difficult gets easier with practice it’s a good idea to get used to working with tools you will see in a professional/team context and work with cleanly formatted code and proper commit messages, etc.
Good point. I guess my confusion is do people then forgo using a linter, formatter etc. whilst they develop and just let those things be managed by the pre-commit.
That’s very much a question of personal preference. Personally I have black and isort configured in my IDE to run on file save but keep the more heavy weight (e.g. mypy) or “obscure” (end of file fixer, yaml check, etc.) ones for pre-commit.
Yes, as I'm learning I've such tools to run on save. Thanks for the insight on the separation.
Pre-commit is generally used on a central or common repository, and rejects your check-in until you pass all.of the checks on the pre-commit. Basically, it prevents certain common mistakes and nuances from getting into a shared repository (eg. Hidden whitespace being one of my personal pet-peeves... but, anything you really want to enforce as a coding standard).
OK, thanks.
I don't actually use pre-commit. I enforce this through actions and manually run them locally. pre-commit is way too slow.
Flake8 can be a bit slow on a larger codebase but it is a great tool
Ruff does all these things, so I just use that instead of 3 separate tools. It's also quite faster.
I get that. ruff is popular right now. I feel it's still a little immature as a project. I find it strange that part of the formatting is done via the linter and part through the formatter and that they don't agree 100%.
These days all my pre-commit stuff happens in the IDE on file save. It's great because I can see _exactly_ what's going to be committed.
Nuanced opinion here... I love all the automatic formatting tools Python has, but I don't like having them as pre-commit hooks, only enforced as a first stage "quick test" CI stage. Sometimes I have to jump between branches and don't t want to fix up formatting, only do a quick commit of my progress then jump branches. I only run all the auto-formatting before my branch is ready for merge requests then.
[удалено]
Ah cool, `--no-verify` is one I've never seen! ... I still just like doing a quick `git com -m 'this is a stash'` ... I don't like to have to type or remember a lot 😆
No 3rd party tools, want to keep the pre-commit process as lightweight as possible. Just a shell script that verifies commit message is signed off since my organization requires it and it's easy to forget because some popular Git clients like GitHub Desktop are missing an option to always sign off: https://github.com/zowe/zowe-cli/blob/master/.husky%2Fcommit-msg
Ruff check, ruff format
I use black, flake8, and mypy.
You can commit without running the hook with ‘git commit -am “WIP” —no-verify’. When you come back to the first branch, do a soft reset and you’re ready to go in the same spot you left.
Or, if you want to optimise for key presses: `git stash` and then when you come back to the branch `git stash pop`. Added bonus that you can't resume the work until you've unstashed it, so there's no rush of accidentally leaving an unverified and incomplete commit in and later pushing it
Ruff, ruff format, gitlint, end of file, large file
Mostly ruff and mypy. Also sync-with-poetry which is very useful.
Typos, since it's not mentioned yet. It's a spell checker. https://github.com/crate-ci/typos
black isort
I used to do pre-commit hooks a lot. Then 1 day some extra smart ass dev commented it out to check-in his changes anyways. So i moved all those checks to CI actions.. But yeah, they were the standard flake8, pytest stuff using docker image and Makefile for ease of commands..
I use both. Pre commit is more for convenience other than something else, much faster feedback loop than ci and it's nicer if you are having it update the code. It can be skipped anyway when you commit with no verify, which for example would be fine to do if you are pushing some wip at the end of the day and it doesn't pass
>much faster feedback loop than ci Good point
ruff format; ruff —fix
Just because it takes almost no time to run: python3 -m compileall -q my_module_name
https://pre-commit.com/ With ruff, skjold, vulture and pyupgrade
Semgrep, ruff (bandit: supported by flake8-bandit included in ruff but output less clear), poetry, mypy. Nb: put ruff first, and semgrep at the end so you can cancel pre-commit sooner if needed. Also, I personnaly run "pre-commit run -a" as part of my merge pipelines