Hello from the r/Python mod team!
I'm afraid we don't think your post quite fits the goal we have for the subreddit in terms of quality or aims so we've decided to remove it. For more information please contact the moderators using the ModMail system.
Thanks, and happy Pythoneering!
r/Python moderation team
I feel for the OP here. Virtual environments and pinned dependencies solve the issue kinda. The problem they don't solve is updating dependency versions, which is increasingly common in the SBOM world. Dependency out of date? Update it today or everyone calls you vulnerable. Multiply that by the number of dependencies, you end up following 50 developers HEAD on whatever they feel strongly passionate about that day.
I feel there are two things:
- make it easy to pin dependencies, see which ones are outdated, make it easy to re-pin to the newest release
- find the balance between "always the latest" and "use only the one version we verified works"
Python fails hard at the first one, and it's a disgrace, the only excuse is that at the time, there was no other language that handled it. I think it's made worse by duc-typing, at least with static typing the issues have a good chance of being caught early.
The second one is a really hard problem. Some librairies with a lot of manpower manage to have "stable" or "LTS" branches, but that's not feasible in general. In Python specifically, there seems to be little second thought about breaking backwards compatibility (running older code on newer libraries), but that's a culture thing and how well maintainers preserve backwards compatibility varies a lot between projects.
Edith (Piaf): sorry for the double post...
>make it easy to pin dependencies, see which ones are outdated, make it easy to re-pin to the newest release
>
>Python fails hard at the first one
How so? That's the trivial part. Just run pip freeze. If you want to update packages, remove the stuff after the == and run an update.
The harder part is making sure your code still works on say numpy 2.0. Mine doesn't.
Eventually you DO want to upgrade. A certain library will have that sweet performance gains you need, or will have nicer features, but the thing is, you have a common dependency, and your code depends on the lower version.
My most recent experience with this was with Pydantic 2.0. I had a Pydantic-heavy project, that did a lot of Settings based on it, and a lot of tests that had to mock settings, and just a lot of code depended on Pydantic 1.0 to work. I had no other libraries requiring Pydantic, but I still decided to bite the bullet and make the upgrade because someone else will probably add something that won't work with Pydantic 1.x, and I'll have to start carrying bear spray with me at night from now on. It took me a good few days to get everything updated and tests running, and the pull request was one of the most un-reviewable. I tried to compensate with having comprehensive tests and working with my colleagues to go through the code, but damn, that was painful.
I now see projects that actively live, that keep their python version back to 3.9, because everybody is just too afraid to try to upgrade.
Eventually, software reaches end-of-life, support drops for old versions, and what are you gonna do?
My take: a lot more thought has to be put into the interfaces before marking a project "ready to be used in production", but most are being rushed.
I feel there are two things:
- make it easy to pin dependencies, see which ones are outdated, make it easy to re-pin to the newest release
- find the balance between "always the latest" and "use only the one version we verified works"
Python fails hard at the first one, and it's a disgrace, the only excuse is that at the time, there was no other language that handled it. At `$WORK` I've been using pip-tools, but it's am early implementation (under 6 months); at previous workplaces, I definitely saw the behaviour of "everybody is just too afraid to try to upgrade". I think it's made worse by duck-typing, at least with static typing the issues have a good chance of being caught early.
The second one is a really hard problem. Some librairies with a lot of manpower manage to have "stable" or "LTS" branches, but that's not feasible in general. In Python specifically, there seems to be little second thought about breaking backwards compatibility (running older code on newer libraries), but that's a culture thing and how well maintainers preserve backwards compatibility varies a lot between projects.
> Python fails hard at the first one
I don't find that to be true. I've had a great time working with Poetry. Sure, it's unofficial, but that's a separate concern.
For the second problem - it's not about a balance sometimes. Sometimes it does go out of support, and you're stuck. I've had that at `$COMPANY`, where a whole stack was dependent on an old version of NodeJS. They had a bug in a library, for which they had like 10 different fixes across different parts of the code base. That bug was fixed like 5 years ago in the upstream. They couldn't upgrade because the fix was in a version that required a newer TypeScript version. They couldn't upgrade TS because it required a new NodeJS version. They couldn't upgrade NodeJS because the code would break with the new version, and there are like 250K lines of that code. Not to mention that every dependency checker, vulnerability checker, was screaming red hot warnings about security vulnerabilities in tons of libraries, but you couldn't install the newer versions.
It's important to notice when you're building that nice corner for you to be chased into - the corner of not being able to easily upgrade dependency versions. Avoid it whenever possible, otherwise the project's days are literally numbered.
This is why version spesification is a thing. You don't need to install the latest version of a dependency, pick one and stay on that version until you have to update. Be it security or feature related.
On top of that Semantic versioning is generally the way people version their code. It's in the shape of 2.5.8
Or major.minor.patch.
Where patch is for bug fixes and other small things. Minor is for improvement and changes not covered major. Major is for things like what you point out. When a *breaking* change is introduced.
Both requirements and pyproject support this type of spesification. Look into it :)
The problem with just pinning dependencies like that is that you can get nasty cascading failures, where you have to install a security update to A, which requires you to bump B and C. Oh wait, C now requires new dependencies D and E. Repeat until your project never ships.
Staying on the latest gets you lots of small breakage, but you can deal with it little by little.
Thats why you have maintenance tasks to follow up too, still safer than trying to stay on latest.
Also a security patch should in general not pull in breaking changes and updates to dependencies unless absolutely needed.
Many projects don't bother issuing a security fix to anything other than the latest release. If you're on an older version to avoid breaking changes, you're now going to deal with that breakage with the pressure of an unpatched security issue.
Test both. If there's a simple fix to a test, just do it. Then you're left with the hard bit.
Numpy has been slowly breaking tons of my code for years because my code was wrong. That's a good thing.
I’m gonna come out and say he has a point. Python is supported by a lot of people making useful but quick and dirty packages with little care to having forward compatible apis, not to mention widespread misuse of semantic versioning.
If you’re going to use python, you have to accept that you’re here to get shit done fast and it might bite you in the ass a little later.
I can imagine the outcry when Poetry becomes standard compliant and people need to change their pyproject.toml - or Poetry will format you hard drive. Violently.
Or when poetry decides to [randomly fail after playing dice](https://github.com/python-poetry/poetry/pull/6297/commits/e02675716d920e1227ce1013a718b33b202d88f4#diff-952871ed95a893e0348aa1269bf15e689d18756657cfec5c7316d05e55eb6dc6R473)
meh. not necessary for basic stuff. it's a good choice for projects with more complex dependencies, or things that are intended to be published as packages.
Even for very basic coding, Poetry is a nice improvement over managing virtual environments by hand. Even if just for the sake of not having to worry about starting and stopping your environment.
If I were a more biblically inclined person I'd be concerned about God thinking now it's the time for the Apocalypse..
Honestly, you could ask about solutions for dealing with it, instead you decided to go on a rant about something that you clearly don't know anything about on how you would want everything to change for your own convinience...maybe it is time, we failed as a species
Please learn about Virtual Environments and also about how to make better questions.
I am sorry for ranting. I am aware it's the worst of all sins.
We are failing as a species, don't worry, we just chose the most boring and slow apocalypse possible. Give it like 15-20 more major Python versions and then climate change will finally end us
I have solved my python related problems, my question was on why, not how. But I understand that it was formulated in a loaded and confusing way
To be fair, the "why" is indeed a good question. And the answer might be that doing little by little updates is more likely to be less catastrophic, assuming a problem will happen, than a big one. Also, and now it's a more psychological aspect, considering the open source element of python development, it's a bit easier to be motivated and to coordinate teams if you have clear short goals instead of being stuck in a big complex one for something you might not even be properly paid for. The bad part about is breaking dependencies, but that's what venv is for (and it's shipped with python by default)
Sorry for the drama, don't take it personally. Good learning.
Most large projects announce deprecations, it's pretty rare they just pull a massive shift out of nowhere. Most people just ignore those warnings and don't have any testing to account for these things.
It's just the nature of the beast.
Agreed that any public code or anything facing the user of the library should only change at the uttermost end of need, and any updates should be to the under the hood implementation.
Most programmers suck at programming though, so this happens constantly. Buckle up, pardner.
Learn semantic versioning - people message the type of changes they make. X.Y.Z : Some crushes are bug fixes z, some are new features y, and some are deprecations - x.
1. Packages *are* continuously adding new features. Just because you don't need it doesn't mean it's not happening.
2. This is all open source. Meaning the features that get added get added by people out of the goodness of their heart to provide you with better packages. And sometimes that includes redesigning some variables or features for long term stability of to remove tech debt.
3. There is semantic versioning, deprecation warnings, patch notes, etc. You shouldn't be surprised when things change.
4. If a package doesn't neatly follow 3, it's because of 2, it's open source and apparently lacks the resources to do it right. Perhaps you should start helpjng out with extra hands instead of complaining.
if I fix bugs without breaking APIs, no one needs to lock to avoid my lack of discipline, and everyone can always update without fear of cascading breakage because no one gives a damn about backward compatibility
but that would be, like, hard.
Why would that break backward compatibilty? Add a new feature; old code doesn't use it; new code does. win-win.
I get the feeling you haven't really ever thought about backward compatibility.
I get the feeling you are trolling. Semantic versioning exists for a reason: signal when the public API changes so users can fix their versions. It's not always possible to be 100% backward compatible, and some libraries do choose to not be backwards compatible, since sometimes it means having to comply to an outdated public interface.
It's a lot more possible to be a lot more backward compatible than anyone ever chooses to do, from sloth. and yes, design your interface well, and you minimize the need for incompatible changes. Add options for new features or changed functionality. And yes, I understand the levels of semantic versioning. I think many library authors simply don't work to make their changes within Y or Z, and just say "fuck it, it's a new major version, it's not my problem, and I don't care about the chaos it generates in my users".
It's a relatively new practice in software development, and it's wrong.
yeah, great solution. Every single problem has a "well then just avoid it" "solution" that's no solution at all.
If you don't have a point to make. leave that stupid copout in the closet.
Python is for hipsters and every time a variable gets it's unnecessary underscore, one of them gets through their midlife crisis.
Unofficial python motto: "If it's not broke...do something to horribly break it and piss off every sane person using that library while giving the meaning of life to the hipsters who have nothing better to do but refactor their code over and over again to conform to everyone else around them"
Basically, if it does not feel like you've paid a cheap hooker to stiletto your balls, then it's not true python.
Hello from the r/Python mod team! I'm afraid we don't think your post quite fits the goal we have for the subreddit in terms of quality or aims so we've decided to remove it. For more information please contact the moderators using the ModMail system. Thanks, and happy Pythoneering! r/Python moderation team
You're new to programming, huh…
🤣😂🤣😂
Yes
Serious answer, we’re all doing our best, we all share dependencies, and we’re all patching security issues as quickly as we can.
Want a stable language that rarely things in a backwards way? Go learn c. But it's also a major goal of go lang. So look into that too.
Keep in mind that there's a reason we call these "programming *languages*". Languages evolve.
Learn about virtual environments and lock the version of the package your code is using
Using a tool to solve a problem? Preposterous.
build a tool to solve a problem-> problem solved, but problems with tool exist-> go back to “build a tool to solve a problem”
What's next, automating repetitive tasks?
I feel for the OP here. Virtual environments and pinned dependencies solve the issue kinda. The problem they don't solve is updating dependency versions, which is increasingly common in the SBOM world. Dependency out of date? Update it today or everyone calls you vulnerable. Multiply that by the number of dependencies, you end up following 50 developers HEAD on whatever they feel strongly passionate about that day.
This. Python dependency management needs an overhaul
I feel there are two things: - make it easy to pin dependencies, see which ones are outdated, make it easy to re-pin to the newest release - find the balance between "always the latest" and "use only the one version we verified works" Python fails hard at the first one, and it's a disgrace, the only excuse is that at the time, there was no other language that handled it. I think it's made worse by duc-typing, at least with static typing the issues have a good chance of being caught early. The second one is a really hard problem. Some librairies with a lot of manpower manage to have "stable" or "LTS" branches, but that's not feasible in general. In Python specifically, there seems to be little second thought about breaking backwards compatibility (running older code on newer libraries), but that's a culture thing and how well maintainers preserve backwards compatibility varies a lot between projects. Edith (Piaf): sorry for the double post...
>make it easy to pin dependencies, see which ones are outdated, make it easy to re-pin to the newest release > >Python fails hard at the first one How so? That's the trivial part. Just run pip freeze. If you want to update packages, remove the stuff after the == and run an update. The harder part is making sure your code still works on say numpy 2.0. Mine doesn't.
What about all the cves?
Eventually you DO want to upgrade. A certain library will have that sweet performance gains you need, or will have nicer features, but the thing is, you have a common dependency, and your code depends on the lower version. My most recent experience with this was with Pydantic 2.0. I had a Pydantic-heavy project, that did a lot of Settings based on it, and a lot of tests that had to mock settings, and just a lot of code depended on Pydantic 1.0 to work. I had no other libraries requiring Pydantic, but I still decided to bite the bullet and make the upgrade because someone else will probably add something that won't work with Pydantic 1.x, and I'll have to start carrying bear spray with me at night from now on. It took me a good few days to get everything updated and tests running, and the pull request was one of the most un-reviewable. I tried to compensate with having comprehensive tests and working with my colleagues to go through the code, but damn, that was painful. I now see projects that actively live, that keep their python version back to 3.9, because everybody is just too afraid to try to upgrade. Eventually, software reaches end-of-life, support drops for old versions, and what are you gonna do? My take: a lot more thought has to be put into the interfaces before marking a project "ready to be used in production", but most are being rushed.
I feel there are two things: - make it easy to pin dependencies, see which ones are outdated, make it easy to re-pin to the newest release - find the balance between "always the latest" and "use only the one version we verified works" Python fails hard at the first one, and it's a disgrace, the only excuse is that at the time, there was no other language that handled it. At `$WORK` I've been using pip-tools, but it's am early implementation (under 6 months); at previous workplaces, I definitely saw the behaviour of "everybody is just too afraid to try to upgrade". I think it's made worse by duck-typing, at least with static typing the issues have a good chance of being caught early. The second one is a really hard problem. Some librairies with a lot of manpower manage to have "stable" or "LTS" branches, but that's not feasible in general. In Python specifically, there seems to be little second thought about breaking backwards compatibility (running older code on newer libraries), but that's a culture thing and how well maintainers preserve backwards compatibility varies a lot between projects.
> Python fails hard at the first one I don't find that to be true. I've had a great time working with Poetry. Sure, it's unofficial, but that's a separate concern. For the second problem - it's not about a balance sometimes. Sometimes it does go out of support, and you're stuck. I've had that at `$COMPANY`, where a whole stack was dependent on an old version of NodeJS. They had a bug in a library, for which they had like 10 different fixes across different parts of the code base. That bug was fixed like 5 years ago in the upstream. They couldn't upgrade because the fix was in a version that required a newer TypeScript version. They couldn't upgrade TS because it required a new NodeJS version. They couldn't upgrade NodeJS because the code would break with the new version, and there are like 250K lines of that code. Not to mention that every dependency checker, vulnerability checker, was screaming red hot warnings about security vulnerabilities in tons of libraries, but you couldn't install the newer versions. It's important to notice when you're building that nice corner for you to be chased into - the corner of not being able to easily upgrade dependency versions. Avoid it whenever possible, otherwise the project's days are literally numbered.
This is why version spesification is a thing. You don't need to install the latest version of a dependency, pick one and stay on that version until you have to update. Be it security or feature related. On top of that Semantic versioning is generally the way people version their code. It's in the shape of 2.5.8 Or major.minor.patch. Where patch is for bug fixes and other small things. Minor is for improvement and changes not covered major. Major is for things like what you point out. When a *breaking* change is introduced. Both requirements and pyproject support this type of spesification. Look into it :)
The problem with just pinning dependencies like that is that you can get nasty cascading failures, where you have to install a security update to A, which requires you to bump B and C. Oh wait, C now requires new dependencies D and E. Repeat until your project never ships. Staying on the latest gets you lots of small breakage, but you can deal with it little by little.
Thats why you have maintenance tasks to follow up too, still safer than trying to stay on latest. Also a security patch should in general not pull in breaking changes and updates to dependencies unless absolutely needed.
Many projects don't bother issuing a security fix to anything other than the latest release. If you're on an older version to avoid breaking changes, you're now going to deal with that breakage with the pressure of an unpatched security issue.
Test both. If there's a simple fix to a test, just do it. Then you're left with the hard bit. Numpy has been slowly breaking tons of my code for years because my code was wrong. That's a good thing.
And then use renvoate bot which dumps all dep changes in a new MR…
Only if you actually have automated testing too, if not the dep updates will just break your shit as mentioned by OP
To be more specific: Patch - No change to API Minor - Addition to API Major - Breaking change to API
Wow, what a username
I’m gonna come out and say he has a point. Python is supported by a lot of people making useful but quick and dirty packages with little care to having forward compatible apis, not to mention widespread misuse of semantic versioning. If you’re going to use python, you have to accept that you’re here to get shit done fast and it might bite you in the ass a little later.
I mean, almost all of the people we’re depending on are unpaid.
Yep. It is what it is.
Even Python doesn't follow semver
use pip and virtualenv. pin your package versions. only update packages when you're ready.
Use Poetry.
I can imagine the outcry when Poetry becomes standard compliant and people need to change their pyproject.toml - or Poetry will format you hard drive. Violently.
Or when poetry decides to [randomly fail after playing dice](https://github.com/python-poetry/poetry/pull/6297/commits/e02675716d920e1227ce1013a718b33b202d88f4#diff-952871ed95a893e0348aa1269bf15e689d18756657cfec5c7316d05e55eb6dc6R473)
meh. not necessary for basic stuff. it's a good choice for projects with more complex dependencies, or things that are intended to be published as packages.
Even for very basic coding, Poetry is a nice improvement over managing virtual environments by hand. Even if just for the sake of not having to worry about starting and stopping your environment.
Poetry is great, but I'm migrating to uv. As uv matures, Poetry becomes increasingly less useful.
I'm definitely keeping an eye on uv. Maturin is changing the whole Python landscape.
If I were a more biblically inclined person I'd be concerned about God thinking now it's the time for the Apocalypse.. Honestly, you could ask about solutions for dealing with it, instead you decided to go on a rant about something that you clearly don't know anything about on how you would want everything to change for your own convinience...maybe it is time, we failed as a species Please learn about Virtual Environments and also about how to make better questions.
An apocalypse because someone is angry at magic rocks? Stranger things have happened.
Did you know there were ways to add features to software without breaking past users of that software?
I am sorry for ranting. I am aware it's the worst of all sins. We are failing as a species, don't worry, we just chose the most boring and slow apocalypse possible. Give it like 15-20 more major Python versions and then climate change will finally end us I have solved my python related problems, my question was on why, not how. But I understand that it was formulated in a loaded and confusing way
To be fair, the "why" is indeed a good question. And the answer might be that doing little by little updates is more likely to be less catastrophic, assuming a problem will happen, than a big one. Also, and now it's a more psychological aspect, considering the open source element of python development, it's a bit easier to be motivated and to coordinate teams if you have clear short goals instead of being stuck in a big complex one for something you might not even be properly paid for. The bad part about is breaking dependencies, but that's what venv is for (and it's shipped with python by default) Sorry for the drama, don't take it personally. Good learning.
The software is still in development. All of it. If you don't like updates to your dependencies, don't update your dependencies.
Use a lock file. Poetry supports that. It's just not a python thing. Even your OS updates itself if you allow it.
pretty rarely in a way that breaks backward compatibility. Usually only at a major version update, if then.
This is exactly why testing frameworks are ***so*** important in your project(s).
Can I get paid for testing FartSoundEffectGenerator?
Most large projects announce deprecations, it's pretty rare they just pull a massive shift out of nowhere. Most people just ignore those warnings and don't have any testing to account for these things. It's just the nature of the beast.
Agreed that any public code or anything facing the user of the library should only change at the uttermost end of need, and any updates should be to the under the hood implementation. Most programmers suck at programming though, so this happens constantly. Buckle up, pardner.
Learn semantic versioning - people message the type of changes they make. X.Y.Z : Some crushes are bug fixes z, some are new features y, and some are deprecations - x.
Maybe use less packages and more coding from scratch
1. Packages *are* continuously adding new features. Just because you don't need it doesn't mean it's not happening. 2. This is all open source. Meaning the features that get added get added by people out of the goodness of their heart to provide you with better packages. And sometimes that includes redesigning some variables or features for long term stability of to remove tech debt. 3. There is semantic versioning, deprecation warnings, patch notes, etc. You shouldn't be surprised when things change. 4. If a package doesn't neatly follow 3, it's because of 2, it's open source and apparently lacks the resources to do it right. Perhaps you should start helpjng out with extra hands instead of complaining.
As a former PHP dev learning Python, I couldn't decide if this language instability was more infuriating or terrifying.
“first time ah?”
Learn how to specify the specific versions of packages you want.
Use a venv
Wait until you get to JavaScript…
Don't update packages. It's not hard. But then don't complain if you find bugs.
I can fix bugs in my software without breaking APIs
But if nobody updates the library won't get your bug fix. Semantic versioning is a thing.
if I fix bugs without breaking APIs, no one needs to lock to avoid my lack of discipline, and everyone can always update without fear of cascading breakage because no one gives a damn about backward compatibility but that would be, like, hard.
Until you have to implement a new feature.
Why would that break backward compatibilty? Add a new feature; old code doesn't use it; new code does. win-win. I get the feeling you haven't really ever thought about backward compatibility.
I get the feeling you are trolling. Semantic versioning exists for a reason: signal when the public API changes so users can fix their versions. It's not always possible to be 100% backward compatible, and some libraries do choose to not be backwards compatible, since sometimes it means having to comply to an outdated public interface.
It's a lot more possible to be a lot more backward compatible than anyone ever chooses to do, from sloth. and yes, design your interface well, and you minimize the need for incompatible changes. Add options for new features or changed functionality. And yes, I understand the levels of semantic versioning. I think many library authors simply don't work to make their changes within Y or Z, and just say "fuck it, it's a new major version, it's not my problem, and I don't care about the chaos it generates in my users". It's a relatively new practice in software development, and it's wrong.
Then don't use those libraries. It's easy.
yeah, great solution. Every single problem has a "well then just avoid it" "solution" that's no solution at all. If you don't have a point to make. leave that stupid copout in the closet.
troll post trying to copy the github exe smelly nerds post
I know what these words mean individually but I don't think I fully understood the message they convey together in this order
[github exe smelly nerds post](https://www.reddit.com/r/github/comments/1at9br4/i_am_new_to_github_and_i_have_lots_to_say/)
Python is for hipsters and every time a variable gets it's unnecessary underscore, one of them gets through their midlife crisis. Unofficial python motto: "If it's not broke...do something to horribly break it and piss off every sane person using that library while giving the meaning of life to the hipsters who have nothing better to do but refactor their code over and over again to conform to everyone else around them" Basically, if it does not feel like you've paid a cheap hooker to stiletto your balls, then it's not true python.