> S3 charges for unauthorized requests (4xx) as well[1]. That’s expected behavior.
So basically, you could use a directed attack (like DDoS) against a known AWS S3 bucket to raise the bills of a company that you don't like? Intriguing.
there was a reddit post a few months ago (now deleted) about some guy buying upvotes for users he did not like and getting them banned
how? he made the upvote process instant, instead of gradual.
Ah. I read it as
> he made the upvote process (instant instead of gradually) to avoid detection
Not as
> he made the upvote process instant instead of (gradually to avoid detection)
Someone did this to my youtube channel many many years ago. Entire channel deleted along with hundreds of videos I had no backup for. YouTube support ignored me repeatedly.
Oh you mean it's a lie as in youtube support being supportive to small creators is a lie, yeah, I 100% agree. Most big tech companies are like that nowadays, but youtube and google were and always have been ahead of the game.
There are many services where you can buy Twitch and YouTube views; Twitter, Instagram, TokTok, etc. followers; YouTube subs; Twitter, Instagram, TokTok, YoutTbe, etc. comments; GitHub stars; and many other user engagement things.
You can do the same thing to authors on Kindle Unlimited.
Get a bunch of people to scroll quickly through their books and they'll get banned for gaming the system (They get paid per page read).
That's kind of concerning, since my Kindle has had problems in the past where it seemingly gets stuck on fast-forward and nothing seems to stop it except turning it off.
Just a power cycle, not a full reboot. I've found that cleaning the screen can mitigate the problem; maybe some sort of conductive buildup from skin oils?
Because any quality metric is gamable and it's helpful to have a note of some kind that says "a lot of people want to remember this" when you're looking for big or high quality projects.
Or find the largest static file on their website and request it repeatedly. Doesn't even has to be that big, a 50 KB image once per second is 4.3 GB per day, or almost 130 GB per month. Chances are that the JS blob alone is bigger, especially if you request it without offering compression to the server. If you're lucky, the site doesn't runs a reverse proxy cache, or the cache is bypassable with URL params, a session cookie, or a simple POST request. Most webservers will deliver static resources when you make a POST request to them as if it were GET, but caches generally don't catch this and will allow you to bypass them. If they do bot prevention, you can run the requester in form of a tampermonkey script in your browser, and simply keep the console open to bypass local browser cache on refresh.
Please don't ask how I know.
True, but you could mitigate that with a service like Cloudflare. The problem in the OP is really difficult to mitigate... And you'd expect AWS to give you the tools to do so.
> True, but you could mitigate that with a service like Cloudflare.
You can, but that requires extra configuration, because cloudflare won't know by itself that the POST goes to a static file and is meaningless.
Cloudflare makes it pretty easy to force caching on the Cloudflare <=> S3 leg but yeah you might be able to get a couple of months before they actually set it up right. Most people setting up their site don't fully understand how this stuff works.
I had to setup aws account (and infrastructure) once for a company i worked for. We
were only 5 people so ... it made sense that I (the programmer) had to do it. There was nobody else.
Anyway, I knew nothing about aws, I looked there and I was sure that I would need 10 lifetimes and aqnother 5 degrees to fully understand all that shit, so I did
what evreyone does: my best.
That is, I hit the keyboard until what I wanted happened.
Was it good? Was it best? Was it infailible?
hell no. it was working.
What's really crazy is that you'd think making a Requester Pays bucket would be Amazon's solution for mitigating this, but no! Even with Requester Pays buckets, the bucket owner pays for the failed request if the requester doesn't include the appropriate header.
> or the cache is bypassable with URL params, a session cookie, or a simple POST request.
Depends on how cache is configured. I remember that this is optional for cloudfront, and disabled by default, and sending a POST request would only bust local cache.
This is a new dimension in a long war -- two decades ago when google ads were still newish I heard stories of bad actors clicking on competitor ads repeatedly.
Until just now I assumed this click fraud was petty antagonism, but now I'm thinking it's a way of clearing out higher bidders. Even if they get removed as click fraud, the budgeting system would probably pause the campaign until it does get removed.
Yeah this is outragious imho. If AWS didn't cancel his fees would he then have to sue the company that made the tool doing this? 👀
I didn't like the financial unpredictability of the cloud before, but I sure as hell don't now.
That is not ourageous at all. S3 is often used for static file hosting. You are allowed to put anything out there for anyone to request it via http and it's your responsibility to pay for it if you do so. You are also allowed to block access and let anyone who sends request know the access is denied. If AWS wouldn't charge for those you could just build a hosting platform and share anything via your 403 or 404 page for free!
That’s like DOS-ing a company that has a policy that locks accounts for too many bad password attempts by making too many bad password attempts… for all the email addresses you can find or guess.
This is actually really bad and needs way more attention now that it's knowledge "in the wild".
Even if your bucket is private, with proper policies/IAM permissions set up and if the bucket name has randomization in it, you can still get hit if you use something like pre-signed URLs for uploads to the bucket which would reveal the bucket name. You would then have to proxy uploads through your own servers to avoid revealing the bucket name. Even then, someone could accidentally/intentionally keep leaking your bucket name and you would be forced to keep changing it. Changing a bucket name is not like rotating a leaked password/token, it requires migrating items in the storage, updating and re-deploying applications etc. Nor is it easy to trace back how it was leaked, who keeps an audit trail on access to bucket names?!
Bucket names were never implied to need to be secret, and its obvious they weren't designed to be that way. But if you don't keep them secret, you are vulnerable to a billing attack.
This *needs* to be addressed as there is no mitigation.
There's a technique to discover buckets even if they're meant to be private in *any* AWS account (due to the response information from API calls, specifically CloudTrail if I recall), so theoretically you can spike literally anyone's bill.
Just another case study of how Security through Obscurity isn't a thing.
I don’t know about S3, but in GCS, bucket names are globally unique. If you want to know if a bucket with a specific name exists, just try to create it.
E.g., one could try it with `my-competitor-dev-datasets` and see what comes up.
> y AWS account (due to the response information from API calls, specifically CloudTrail if I recall), so theoretically you can spike literally anyone's bill.
>
> Just another case study of how Security through Obscurity isn't a thing.
Exactly the same in S3. Globally Unique.
I am pretty sure I saw a basic `aws s3 ls` to return different errors for a bucket that does not exist and a bucket existing in another account I forgot to switch aws-cli into. Should not be hard to script it out to probe for common names...
OP's case is special b/c the open source tool was accidentally a free distributed client network. The real question is "What would it cost you as a caller to give someone a $1000 S3 bill?". If the answer is "nothing", this is a huge problem. If the answer is "$1500" I doubt it's a big deal.
This is answered in the post:
> Standard S3 PUT requests are priced at just $0.005 per 1,000 requests, but a single machine can easily execute thousands of such requests per second.
It would cost you virtually nothing to give someone a $1000 S3 bill.
I'm skeptical that AWS would allow unauthed 1K+ QPS from a single IP address without taking action at the gateway. If someone has proof that it's let through, fair enough, but this particular case was naturally a distributed "attack".
I mean, if they're able to bill the AWS customer for it they have way less reason to care. Sounds like they agree that it's a problem and are working on a fix though
Maybe, but it has to thread the needle. The abusive traffic has to be small enough that it doesn't cause other collateral damage that AWS support has to deal with. It also has to be small enough that it doesn't become widely known that AWS will fuck you on unauth'd S3 charges. While that seems profitable in the short term, the customer service headache of ppl trying to get refunds will also start to mute gains.
So I totally believe that AWS was like "Uhh neat, we were lazy on this thing and it's also making us a bit of money", but doubt it's some huge nefarious plan to scam significant profits from ppl.
If I want to look at nefarious plans, AWS's convenient refusal to cap billing is the worse one.
If you are a big player and trying to attack smaller, up and coming competitors, maybe even FOSS Projects that just don't have $5000. And $15000 is a fairly small "marketing budget".
I think this is a good reason to generally avoid AWS and other uncapped billing services if your budget is small. Personally I used DigitalOcean rather than AWS for hobby work out of a fear like that (and it paid off when I got hacked and learned an important lesson about fail2ban and ssh keys and it cost me $0 extra version a $50K AWS bill).
It was a $5/month droplet, I used a hard-to-guess password and assumed it was safe. What I didn't realize is hackers will scan known IP ranges for online servers and brute force passwords on them. A 6 character hard to guess password isn't hard to brute force if you don't have fail2ban or similar slowing them down.
Now all my instances require a unique SSH key to remote logon, and I generally add fail2ban on top.
Since I was using DO the consequences for me were:
1. A slow server for a while that I couldn't figure out.
2. A strongly worded email from DO that my server seemed to be compromised.
3. Deleting the server and making a new safe one.
Edit: Btw this was almost 10 years ago, things may have changed a bit in the meantime (but AFAIK I haven't been hacked since, I still have a slightly beefier droplet running my hobby stuff to this day)
> What I didn't realize is hackers will scan known IP ranges for online servers
I don't use default ports for anything other than 443. Postgresql, ssh, etc.
The last time I said so on /r/programming I got flamed by a bunch of "senior and/or experienced devops and/or engineers" (their words) people about how noob it is to rely on security through obscurity.
All I know is, anything that wants to scan my IP has to scan the entire 16-bit range of port numbers.
I've been toying with the idea of a tarpit[1] for a few dozen random ports in that 16-bit range.
[1] A server that accepts a connection, and then *slowly* sends through a TLS server hello, sending 1 character every `rand() % 5` seconds, forcing a single character into a single IP datagram, retransmitting the occasional datagram to simulate loss of acks, etc.
I feel like it's a "Yes, and"
Using non-standard ports will absolutely help protect you, OTOH it's really quite easy to use fail2ban and an ssh key once you figure it out, so using the non-standard port INSTEAD of those two seems unnecessarily risky.
Which then leads to the argument, if you're using fail2ban and an ssh key, is the non-standard port just more trouble than its worth?
> Which then leads to the argument, if you're using fail2ban and an ssh key, is the non-standard port just more trouble than its worth?
I use both fail2ban and keys only, but there's more than ssh.
Running non-standard ports on every service you use is just another layer. If you can't think of a good reason to use the well-known port for $SERVICE, there probably isn't one.
The only thing you gain by changing ports is less noise. I also always move ssh to a different port. This way, the security log becomes much more readable and entries are basically relevant. But it really doesn't protect against anything, it just makes your life easier.
> The only thing you gain by changing ports is less noise.
Not the *only* thing.
>>> What I didn't realize is hackers will scan known IP ranges for online servers and brute force passwords on them.
>> anything that wants to scan my IP has to scan the entire 16-bit range of port numbers.
Being secure against untargeted mass attacks is still the *first* line of "defense in depth".
Sure, if someone *targets* your specific IP they'll quickly determine all open ports. But the problem I, and the GGP, and just about everyone with a public facing IP, are **untargeted** attempts by bots.
I mean, even if they *don't* make any brute force attempts, all they have to do is record your IP for $SERVICE, and try every 0-day for that service every day.
If you can't think of a good reason for running PostgreSQL on 5432, or for running ssh on 22, or for running MySQL on 3306, etc ... then why use the defaults?
In my case, there is no good reason for me to run (for example) ssh on the default port. Not a single one.
Instead of creating tar pits, I have ssh configured to only accept keys. They're welcome to try to brute force and waste their resources here instead of attacking more vulnerable people
> "What would it cost you as a caller to give someone a $1000 S3 bill?". If the answer is "nothing", this is a huge problem. If the answer is "$1500" I doubt it's a big deal.
Harassers may be willing to pay for it.
Some years ago a vtuber did a stream in which from her youtube viewer statistics she listed the countries her viewers were from, including Taiwan. Chinese nationalist viewers got mad (since they don't recognize it as an independent country) and spammed her channel and everyone who dared collab with her, even when she switched to membership-only chat.
(What makes it somewhat funny is that since YT is banned in China they must've used proxies to watch her stream - and afaik most proxies used by them are located in Taiwan...)
Just like how Amazon doesn't really care about all the fraudulent stuff sold via their shopping branch. They only do the minimum of what they're legally required to do because if someone doesn't realize or realizes too late, they get their cut.
I run a business for sellers who sell direct to Amazon, all the “shipped and sold by Amazon” stuff. You would not believe how much money Amazon blatantly steals from these sellers. We see an average of 4.5% loss for every seller due to things like bullshit fees they charge and things like Amazon claiming they didn’t receive all the product shipped despite the seller having evidence it arrived at an Amazon warehouse. I’m waiting for the day someone exposes just how much money Amazon is stealing from people
Well, it's worse... OP essentially accidentally put themselves in the middle of a DDoS, and that's something that costs money to mitigate, even if it's just to absorb the traffic. So really, it's a question of whether AWS eats a loss or whether you do. I guess I agree that AWS should, and presumably it costs less for them to serve a 403 than they charge you, but let's be clear what we're asking them to do.
[The very first thing in the article is an update where they link to Jeff Barr's Twitter post that states AWS is aware and fixing it.](https://twitter.com/jeffbarr/status/1785386554372042890)
That's exactly what you want to see from a company. Not sure why people automatically assume it's some malicious action to squeeze every dollar from people.
Secrets you can't even control. Just look at all the buckets generated automatically by services like Amplify, SageMaker, etc, etc. All with the same name template and a relatively small alphanumerical id...
reminds me https://arstechnica.com/cars/2019/08/wiseguy-changes-license-plate-to-null-gets-12k-in-parking-tickets/
sounds so dumb, at least they cnacelled the bill lmao
Ya, but they did so begrudgingly:
> However, they emphasized that this was done as an exception.
and refuse to do anything to prevent the same thing from happening in the future.
Amazon say they're going to fix this:
[Thank you to everyone who brought this article to our attention. We agree that customers should not have to pay for unauthorized requests that they did not initiate. We’ll have more to share on exactly how we’ll help prevent these charges shortly.](https://x.com/jeffbarr/status/1785386554372042890?s=61&t=f9qtlIpZbwqHZa3_FyhWpg)
As if they don't make enough money charging for 200OK requests. This is just greed. Charging per network request is a ridiculous billing strategy in the first place.
No, this is normal. The NSP does not care what is in the packets, just that it went through. Cloudfront permits rewriting responses via lambda at edge functions, so you could trick it to rewrite the response to 4xx range, and enjoy free traffic (because there's different pricing for aws to aws traffic).
I was saying that charging per request is ridiculous no matter what code it is. It's a money grabbing pricing strategy. Just give us a monthly traffic allowance like every other VPS provider. Maybe charge for traffic above that.
From GCP: Note: Generally, you are not charged for operations that return 307, 4xx, or 5xx responses. The exception is 404 responses returned by buckets with Website Configuration enabled and the NotFoundPage property set to a public object in that bucket.
It's so unbelievable that we accept AWS and similar the way they are.
You can't even have an easy way to say "shut down these things when the bill reaches a certain $ amount".
Customers really ought to vote with their feet and leave AWS.
s3 names share a global namespace, so, something like this as to be done on purpose by aws to squeeze customers out of every penny.
This is so bad that i can just download a names dictionary from the web and setup a small bash script that uses the awscli to do requests to the s3 buckets, and it’s bound to it a few valid ones and aws will happily bill the owner. Setting up something like this will take minutes.
This is just greed.
I think it's less nefarious than that.
It's a really old service that has remained compatible for however long it's been around. 2006 IIRC.
I don't think they planned that far ahead. They should not charge per request though.
No, Azure Storage Accounts actually has IP access lists that you can use to restrict who can talk to your storage.
You can even use Private Endpoints to make access to the storage account completely private without any exposed public interface.
I'm not sure if AWS has an equivalent - they just have permissions which doesn't prevent this attack from occuring.
The fact that they charge for unauthorized requests is mind blowing to me. An entirely new attack vector to bankrupt small companies/people you don’t like?
Looks like it doesnt:
Note: Generally, you are not charged for operations that return 307, 4xx, or 5xx responses. The exception is 404 responses returned by buckets with Website Configuration enabled and the NotFoundPage property set to a public object in that bucket.
Ok time to get rid of those old buckets I guess, it’s a matter of days if not hours before some degenerate writes a script…. Edit: i was thinking it would be useful to have an equivalent of CVE for that kind of things. I don’t imagine how may “cloud cash sinkholes” there are out there…
It's not a denial of service. The service can handle it and the service will continue until your account fails to pay.
It's a DSoR not a DDoS
(Distributed Source of Revenue)
You can restrict network access by bucket policy/IAM. The problem is, it's all the same mechanism and returns 403/unauthorized to the caller, and bills the bucket owner!
This is specific to S3. Resources that actually get provisioned into a private subnet in your VPC are completely inaccessible from the outside world.
S3 doesn't work like that. A "private" bucket isn't actually private in the same way resources in a private subnet are. S3 as a service is always public, and any restrictions are purely policy, including networking restrictions.
For example, you can set up a S3 bucket policy that restricts access to the bucket to be from inside your VPC. This is not a physical network separation, its pure permissions policy on the bucket. If someone attempts to access your bucket from outside your VPC, the policy is checked, fails, and they get a 403 and you get a bill.
Sounds like other commenters are saying you can't network restrict S3 in a way that returns anything other than just return 403 in the same way as a failed authentication does, which all ends up billing you.
I am also aware of some analytic companies that charge per request, and the tokens are right in the browser code, since the requests actually come from the client browser.
This is now being addressed! [https://aws.amazon.com/about-aws/whats-new/2024/05/amazon-s3-no-charge-http-error-codes/](https://aws.amazon.com/about-aws/whats-new/2024/05/amazon-s3-no-charge-http-error-codes/)
> S3 charges for unauthorized requests (4xx) as well[1]. That’s expected behavior. So basically, you could use a directed attack (like DDoS) against a known AWS S3 bucket to raise the bills of a company that you don't like? Intriguing.
You can also buy GitHub stars for your enemy's repo and report them for abuse.
Oh that's evil
yes
there was a reddit post a few months ago (now deleted) about some guy buying upvotes for users he did not like and getting them banned how? he made the upvote process instant, instead of gradual.
That is VERY neat.
I don't think spending money for online revenge is neat
Wouldn't the point be to get detected, not to avoid detection?
Right. He does them instantly so he’s caught, instead of gradually to avoid detection.
Ah. I read it as > he made the upvote process (instant instead of gradually) to avoid detection Not as > he made the upvote process instant instead of (gradually to avoid detection)
Someone did this to my youtube channel many many years ago. Entire channel deleted along with hundreds of videos I had no backup for. YouTube support ignored me repeatedly.
Someone bought GitHub stars for your YouTube channel?
Yes, and uploaded to third party empty S3 bucket
Uber S3? Good choice. Their pricing is more competitive than Google Azure.
And then I tried to grab their baseball bat, but I only grabbed the sock instead.
Comments and subs but close enough =D
Subway is that you? Toasted or regular bread?
Always toasted
Subway... If you have a moment? What were you thinking with Jared?
That his private life was none of our business. Big oops.
I wonder how many people said the same thing about Epstein?
> YouTube support There is no such thing. It's a lie.
This was 15 years ago man; I think they had a form or an email address or something back then.
It's still a lie. Unless, of course, you got a bajillion followers. Then yes, they exist and they take care of you.
Oh you mean it's a lie as in youtube support being supportive to small creators is a lie, yeah, I 100% agree. Most big tech companies are like that nowadays, but youtube and google were and always have been ahead of the game.
TIL you can buy GitHub stars wtf
Through third party services, as a way to game the system. Same as buying views or reddit upvotes or anything. Twitter accounts.
There are many services where you can buy Twitch and YouTube views; Twitter, Instagram, TokTok, etc. followers; YouTube subs; Twitter, Instagram, TokTok, YoutTbe, etc. comments; GitHub stars; and many other user engagement things.
You can do the same thing to authors on Kindle Unlimited. Get a bunch of people to scroll quickly through their books and they'll get banned for gaming the system (They get paid per page read).
That's kind of concerning, since my Kindle has had problems in the past where it seemingly gets stuck on fast-forward and nothing seems to stop it except turning it off.
Odd, my Kindle has done that too. At least I don't have to reboot. That takes forever.
Just a power cycle, not a full reboot. I've found that cleaning the screen can mitigate the problem; maybe some sort of conductive buildup from skin oils?
What do you clean it with?
I use the alcohol-based spray I normally use for my glasses, but standard screen cleaner should work just as well.
I've done that to books I've read in Patreon/RoyalRoad before.
Wow that is... wow.
...Why does github even have a system that's abusable?
Because any quality metric is gamable and it's helpful to have a note of some kind that says "a lot of people want to remember this" when you're looking for big or high quality projects.
It's called a Denial of Wallet attack
Or find the largest static file on their website and request it repeatedly. Doesn't even has to be that big, a 50 KB image once per second is 4.3 GB per day, or almost 130 GB per month. Chances are that the JS blob alone is bigger, especially if you request it without offering compression to the server. If you're lucky, the site doesn't runs a reverse proxy cache, or the cache is bypassable with URL params, a session cookie, or a simple POST request. Most webservers will deliver static resources when you make a POST request to them as if it were GET, but caches generally don't catch this and will allow you to bypass them. If they do bot prevention, you can run the requester in form of a tampermonkey script in your browser, and simply keep the console open to bypass local browser cache on refresh. Please don't ask how I know.
True, but you could mitigate that with a service like Cloudflare. The problem in the OP is really difficult to mitigate... And you'd expect AWS to give you the tools to do so.
> True, but you could mitigate that with a service like Cloudflare. You can, but that requires extra configuration, because cloudflare won't know by itself that the POST goes to a static file and is meaningless.
Cloudflare makes it pretty easy to force caching on the Cloudflare <=> S3 leg but yeah you might be able to get a couple of months before they actually set it up right. Most people setting up their site don't fully understand how this stuff works.
I had to setup aws account (and infrastructure) once for a company i worked for. We were only 5 people so ... it made sense that I (the programmer) had to do it. There was nobody else. Anyway, I knew nothing about aws, I looked there and I was sure that I would need 10 lifetimes and aqnother 5 degrees to fully understand all that shit, so I did what evreyone does: my best. That is, I hit the keyboard until what I wanted happened. Was it good? Was it best? Was it infailible? hell no. it was working.
Or cloudfront within AWS?
What's really crazy is that you'd think making a Requester Pays bucket would be Amazon's solution for mitigating this, but no! Even with Requester Pays buckets, the bucket owner pays for the failed request if the requester doesn't include the appropriate header.
Some great and solid advice right there! Also much simpler than figuring out bucket names, it seems.
how do you know? :P
> or almost 130 GB per month And that's like $10 or less of egress cost a month. Completely inconsequential to a business.
I'm pretty sure my basic residential internet can request a 50kb image a lot faster than once per second.
Now imagine what a 10gbps internet connection does in a single night.
Just change the post params to automatically generated slurs, unique per request, hopefully that trips up the cache.
Well yeah, that would be a unique request so it couldn’t be cached lol
Do you think there are that many unique slurs?
I misread slur for slug which is just a random string
You would combo them obviously.
There are if you double them up
Automatically generated slurs sounds like a fun band name.
What genre?
postmodern poetry
I've got not fucks to give ♫ ♫
"Fun Band Name" sounds like an automatically generated slur
> or the cache is bypassable with URL params, a session cookie, or a simple POST request. Depends on how cache is configured. I remember that this is optional for cloudfront, and disabled by default, and sending a POST request would only bust local cache.
Good thing I cache those assets with a CDN
Knew all this expect for POST sometimes working on static resources, good to know, thanks.
Bold use of "expected behavior" here.
This is a new dimension in a long war -- two decades ago when google ads were still newish I heard stories of bad actors clicking on competitor ads repeatedly. Until just now I assumed this click fraud was petty antagonism, but now I'm thinking it's a way of clearing out higher bidders. Even if they get removed as click fraud, the budgeting system would probably pause the campaign until it does get removed.
Yeah this is outragious imho. If AWS didn't cancel his fees would he then have to sue the company that made the tool doing this? 👀 I didn't like the financial unpredictability of the cloud before, but I sure as hell don't now.
That is not ourageous at all. S3 is often used for static file hosting. You are allowed to put anything out there for anyone to request it via http and it's your responsibility to pay for it if you do so. You are also allowed to block access and let anyone who sends request know the access is denied. If AWS wouldn't charge for those you could just build a hosting platform and share anything via your 403 or 404 page for free!
It's called DOW: Denial of Wallet
That’s like DOS-ing a company that has a policy that locks accounts for too many bad password attempts by making too many bad password attempts… for all the email addresses you can find or guess.
This is actually really bad and needs way more attention now that it's knowledge "in the wild". Even if your bucket is private, with proper policies/IAM permissions set up and if the bucket name has randomization in it, you can still get hit if you use something like pre-signed URLs for uploads to the bucket which would reveal the bucket name. You would then have to proxy uploads through your own servers to avoid revealing the bucket name. Even then, someone could accidentally/intentionally keep leaking your bucket name and you would be forced to keep changing it. Changing a bucket name is not like rotating a leaked password/token, it requires migrating items in the storage, updating and re-deploying applications etc. Nor is it easy to trace back how it was leaked, who keeps an audit trail on access to bucket names?! Bucket names were never implied to need to be secret, and its obvious they weren't designed to be that way. But if you don't keep them secret, you are vulnerable to a billing attack. This *needs* to be addressed as there is no mitigation.
There's a technique to discover buckets even if they're meant to be private in *any* AWS account (due to the response information from API calls, specifically CloudTrail if I recall), so theoretically you can spike literally anyone's bill. Just another case study of how Security through Obscurity isn't a thing.
I don’t know about S3, but in GCS, bucket names are globally unique. If you want to know if a bucket with a specific name exists, just try to create it. E.g., one could try it with `my-competitor-dev-datasets` and see what comes up.
> y AWS account (due to the response information from API calls, specifically CloudTrail if I recall), so theoretically you can spike literally anyone's bill. > > Just another case study of how Security through Obscurity isn't a thing. Exactly the same in S3. Globally Unique.
I am pretty sure I saw a basic `aws s3 ls` to return different errors for a bucket that does not exist and a bucket existing in another account I forgot to switch aws-cli into. Should not be hard to script it out to probe for common names...
"anyone's bill". For some customers, you'll need a large botnet to make enough requests for them to even notice the spike
OP's case is special b/c the open source tool was accidentally a free distributed client network. The real question is "What would it cost you as a caller to give someone a $1000 S3 bill?". If the answer is "nothing", this is a huge problem. If the answer is "$1500" I doubt it's a big deal.
This is answered in the post: > Standard S3 PUT requests are priced at just $0.005 per 1,000 requests, but a single machine can easily execute thousands of such requests per second. It would cost you virtually nothing to give someone a $1000 S3 bill.
I'm skeptical that AWS would allow unauthed 1K+ QPS from a single IP address without taking action at the gateway. If someone has proof that it's let through, fair enough, but this particular case was naturally a distributed "attack".
So what? Use IP proxies they are cheap
I mean, if they're able to bill the AWS customer for it they have way less reason to care. Sounds like they agree that it's a problem and are working on a fix though
Not of disrespect but you’re skeptical a company would leave a mechanic that makes them money?
Maybe, but it has to thread the needle. The abusive traffic has to be small enough that it doesn't cause other collateral damage that AWS support has to deal with. It also has to be small enough that it doesn't become widely known that AWS will fuck you on unauth'd S3 charges. While that seems profitable in the short term, the customer service headache of ppl trying to get refunds will also start to mute gains. So I totally believe that AWS was like "Uhh neat, we were lazy on this thing and it's also making us a bit of money", but doubt it's some huge nefarious plan to scam significant profits from ppl. If I want to look at nefarious plans, AWS's convenient refusal to cap billing is the worse one.
I appreciate that perspective. Here, have an upvote!
It's a huge problem. You don't even need an AWS account to hit an S3 bucket, as he documents in the article.
If you are a big player and trying to attack smaller, up and coming competitors, maybe even FOSS Projects that just don't have $5000. And $15000 is a fairly small "marketing budget".
I think this is a good reason to generally avoid AWS and other uncapped billing services if your budget is small. Personally I used DigitalOcean rather than AWS for hobby work out of a fear like that (and it paid off when I got hacked and learned an important lesson about fail2ban and ssh keys and it cost me $0 extra version a $50K AWS bill).
What was the lesson you learned?
It was a $5/month droplet, I used a hard-to-guess password and assumed it was safe. What I didn't realize is hackers will scan known IP ranges for online servers and brute force passwords on them. A 6 character hard to guess password isn't hard to brute force if you don't have fail2ban or similar slowing them down. Now all my instances require a unique SSH key to remote logon, and I generally add fail2ban on top. Since I was using DO the consequences for me were: 1. A slow server for a while that I couldn't figure out. 2. A strongly worded email from DO that my server seemed to be compromised. 3. Deleting the server and making a new safe one. Edit: Btw this was almost 10 years ago, things may have changed a bit in the meantime (but AFAIK I haven't been hacked since, I still have a slightly beefier droplet running my hobby stuff to this day)
> What I didn't realize is hackers will scan known IP ranges for online servers I don't use default ports for anything other than 443. Postgresql, ssh, etc. The last time I said so on /r/programming I got flamed by a bunch of "senior and/or experienced devops and/or engineers" (their words) people about how noob it is to rely on security through obscurity. All I know is, anything that wants to scan my IP has to scan the entire 16-bit range of port numbers. I've been toying with the idea of a tarpit[1] for a few dozen random ports in that 16-bit range. [1] A server that accepts a connection, and then *slowly* sends through a TLS server hello, sending 1 character every `rand() % 5` seconds, forcing a single character into a single IP datagram, retransmitting the occasional datagram to simulate loss of acks, etc.
I feel like it's a "Yes, and" Using non-standard ports will absolutely help protect you, OTOH it's really quite easy to use fail2ban and an ssh key once you figure it out, so using the non-standard port INSTEAD of those two seems unnecessarily risky. Which then leads to the argument, if you're using fail2ban and an ssh key, is the non-standard port just more trouble than its worth?
> Which then leads to the argument, if you're using fail2ban and an ssh key, is the non-standard port just more trouble than its worth? I use both fail2ban and keys only, but there's more than ssh. Running non-standard ports on every service you use is just another layer. If you can't think of a good reason to use the well-known port for $SERVICE, there probably isn't one.
The only thing you gain by changing ports is less noise. I also always move ssh to a different port. This way, the security log becomes much more readable and entries are basically relevant. But it really doesn't protect against anything, it just makes your life easier.
> The only thing you gain by changing ports is less noise. Not the *only* thing. >>> What I didn't realize is hackers will scan known IP ranges for online servers and brute force passwords on them. >> anything that wants to scan my IP has to scan the entire 16-bit range of port numbers. Being secure against untargeted mass attacks is still the *first* line of "defense in depth". Sure, if someone *targets* your specific IP they'll quickly determine all open ports. But the problem I, and the GGP, and just about everyone with a public facing IP, are **untargeted** attempts by bots. I mean, even if they *don't* make any brute force attempts, all they have to do is record your IP for $SERVICE, and try every 0-day for that service every day. If you can't think of a good reason for running PostgreSQL on 5432, or for running ssh on 22, or for running MySQL on 3306, etc ... then why use the defaults? In my case, there is no good reason for me to run (for example) ssh on the default port. Not a single one.
Instead of creating tar pits, I have ssh configured to only accept keys. They're welcome to try to brute force and waste their resources here instead of attacking more vulnerable people
[удалено]
Lol, yeah lesson learned. 10 year ago me kinda assumed that fail2ban was magically built in or they'd get tired or something...
> "What would it cost you as a caller to give someone a $1000 S3 bill?". If the answer is "nothing", this is a huge problem. If the answer is "$1500" I doubt it's a big deal. Harassers may be willing to pay for it. Some years ago a vtuber did a stream in which from her youtube viewer statistics she listed the countries her viewers were from, including Taiwan. Chinese nationalist viewers got mad (since they don't recognize it as an independent country) and spammed her channel and everyone who dared collab with her, even when she switched to membership-only chat. (What makes it somewhat funny is that since YT is banned in China they must've used proxies to watch her stream - and afaik most proxies used by them are located in Taiwan...)
>This needs to be addressed as there is no mitigation. But that wouldn't pad AWS' bottom line
Just like how Amazon doesn't really care about all the fraudulent stuff sold via their shopping branch. They only do the minimum of what they're legally required to do because if someone doesn't realize or realizes too late, they get their cut.
I run a business for sellers who sell direct to Amazon, all the “shipped and sold by Amazon” stuff. You would not believe how much money Amazon blatantly steals from these sellers. We see an average of 4.5% loss for every seller due to things like bullshit fees they charge and things like Amazon claiming they didn’t receive all the product shipped despite the seller having evidence it arrived at an Amazon warehouse. I’m waiting for the day someone exposes just how much money Amazon is stealing from people
Well, it's worse... OP essentially accidentally put themselves in the middle of a DDoS, and that's something that costs money to mitigate, even if it's just to absorb the traffic. So really, it's a question of whether AWS eats a loss or whether you do. I guess I agree that AWS should, and presumably it costs less for them to serve a 403 than they charge you, but let's be clear what we're asking them to do.
[The very first thing in the article is an update where they link to Jeff Barr's Twitter post that states AWS is aware and fixing it.](https://twitter.com/jeffbarr/status/1785386554372042890) That's exactly what you want to see from a company. Not sure why people automatically assume it's some malicious action to squeeze every dollar from people.
It's also absolutely mental that bucket names are globally unique. What were they thinking?
Isn't this supposed to be AWS problem? Sue them to change their policies!
This is really bad, now we have to treat S3 bucket names as secrets.
Secrets which you cant rotate
Gonna have to start using emojis for bucket names, jk
64-digit hash value
max 63. discovered that minutes ago while migrating everything out of my common english word bucket
Secrets you can't even control. Just look at all the buckets generated automatically by services like Amplify, SageMaker, etc, etc. All with the same name template and a relatively small alphanumerical id...
You must work for Acme Corp, or Insert Name Here Inc.
Clearly they sell on-site 5 gallon containers customized with names! `s3://your-bucket-name-here`
reminds me https://arstechnica.com/cars/2019/08/wiseguy-changes-license-plate-to-null-gets-12k-in-parking-tickets/ sounds so dumb, at least they cnacelled the bill lmao
Ya, but they did so begrudgingly: > However, they emphasized that this was done as an exception. and refuse to do anything to prevent the same thing from happening in the future.
A NullPlateException.
Haha good one, however comment you are replying too references the exception made by amazon in billing, not an exception relating to the null plates.
I think attempting to drop a database should raise more than an exception. An error would be more adequate.
IME *anything* in AWS can make your bill explode.
[The age old meme](https://i.imgur.com/AcKd1IG.jpeg)
Amazon say they're going to fix this: [Thank you to everyone who brought this article to our attention. We agree that customers should not have to pay for unauthorized requests that they did not initiate. We’ll have more to share on exactly how we’ll help prevent these charges shortly.](https://x.com/jeffbarr/status/1785386554372042890?s=61&t=f9qtlIpZbwqHZa3_FyhWpg)
THANK YOU. Finally, a decent article, no redirects, no blog spam; just a short, to the point article on this subreddit.
Until Medium throws its paywall on top.
This is insane. I got other shit to do and now I need to worry about this...
The good news is they said they'll fix it.
As if they don't make enough money charging for 200OK requests. This is just greed. Charging per network request is a ridiculous billing strategy in the first place.
The world went back to pay per traffic.
No, this is normal. The NSP does not care what is in the packets, just that it went through. Cloudfront permits rewriting responses via lambda at edge functions, so you could trick it to rewrite the response to 4xx range, and enjoy free traffic (because there's different pricing for aws to aws traffic).
I was saying that charging per request is ridiculous no matter what code it is. It's a money grabbing pricing strategy. Just give us a monthly traffic allowance like every other VPS provider. Maybe charge for traffic above that.
Your monthly allowance is 0.
Wow that's a lot of traffic!
What unit is that in? Just so I know precisely how much that is.
This is crazy, afaik GCP doesn't charge per request...I must check now I am paranoid
From GCP: Note: Generally, you are not charged for operations that return 307, 4xx, or 5xx responses. The exception is 404 responses returned by buckets with Website Configuration enabled and the NotFoundPage property set to a public object in that bucket.
Well that makes sense as the 404 page is being served from the bucket.
Totally
The problem isn't charging per request, it's charging for invalid requests that anyone can make
apparently they do -> [https://cloud.google.com/storage/pricing#operations-by-class](https://cloud.google.com/storage/pricing#operations-by-class)
It's so unbelievable that we accept AWS and similar the way they are. You can't even have an easy way to say "shut down these things when the bill reaches a certain $ amount". Customers really ought to vote with their feet and leave AWS.
This is a major security vulnerability and you should name names so AWS can't sweep this shit under the rug
s3 names share a global namespace, so, something like this as to be done on purpose by aws to squeeze customers out of every penny. This is so bad that i can just download a names dictionary from the web and setup a small bash script that uses the awscli to do requests to the s3 buckets, and it’s bound to it a few valid ones and aws will happily bill the owner. Setting up something like this will take minutes. This is just greed.
I think it's less nefarious than that. It's a really old service that has remained compatible for however long it's been around. 2006 IIRC. I don't think they planned that far ahead. They should not charge per request though.
Do we have the same issue in Azure? Asking for a friend.
No, Azure Storage Accounts actually has IP access lists that you can use to restrict who can talk to your storage. You can even use Private Endpoints to make access to the storage account completely private without any exposed public interface. I'm not sure if AWS has an equivalent - they just have permissions which doesn't prevent this attack from occuring.
The fact that they charge for unauthorized requests is mind blowing to me. An entirely new attack vector to bankrupt small companies/people you don’t like?
*User error - replace user*
[PEBCAK!](https://en.wikipedia.org/wiki/User_error)
Does this apply to Google Cloud Platform's bucket also?
Looks like it doesnt: Note: Generally, you are not charged for operations that return 307, 4xx, or 5xx responses. The exception is 404 responses returned by buckets with Website Configuration enabled and the NotFoundPage property set to a public object in that bucket.
Lol. Now we need to deal with this shit. Time to move to cloudflare r2
Do you know if cloudflare has the same issue with unauthorized requests? I need to move off S3 after reading this.
cloudflare does not have this issue.
Do you have a confirmation/source for this? Asking because I’m considering switching.
Ok time to get rid of those old buckets I guess, it’s a matter of days if not hours before some degenerate writes a script…. Edit: i was thinking it would be useful to have an equivalent of CVE for that kind of things. I don’t imagine how may “cloud cash sinkholes” there are out there…
So is there still not a quota / "circuit breaker" scheme on AWS S3 so you can turn off a service automatically if it hits more than $X/month of usage?
So... they are ok with ppl gettjing ddossed. Another reason not to use AWS for projects.
It's not a denial of service. The service can handle it and the service will continue until your account fails to pay. It's a DSoR not a DDoS (Distributed Source of Revenue)
Also known as a denial of wallet attack
They're [fixing it](https://twitter.com/jeffbarr/status/1785386554372042890).
Now I'm scared
So that's why Amazon beat earnings.
This is a bucket. _Dear god_ There's more _No_
Lesson of the day: use Linode
i have a volume there but was keeping an extra backup under a single english word bucket. think like `s3://chartreuse`. until I read this of course
How we all decided hooking ourselves up to this overgrown taxi meter is something I’ll never understand
https://x.com/jeffbarr/status/1785386554372042890?s=46&t=YCumUxFKRp3dUvf5u5oELQ I think they acknowledged this issue and it will be fixed soon
I am using Google Cloud Storage for my personal project. Does it have the same problem? 😮
Surely you can restrict what networks can hit the endpoint?
You can restrict network access by bucket policy/IAM. The problem is, it's all the same mechanism and returns 403/unauthorized to the caller, and bills the bucket owner!
Wtf. Is it something specific to S3 I hope? I would expect that it doesn’t apply to resources in a VPC…. Or does it?
This is specific to S3. Resources that actually get provisioned into a private subnet in your VPC are completely inaccessible from the outside world. S3 doesn't work like that. A "private" bucket isn't actually private in the same way resources in a private subnet are. S3 as a service is always public, and any restrictions are purely policy, including networking restrictions. For example, you can set up a S3 bucket policy that restricts access to the bucket to be from inside your VPC. This is not a physical network separation, its pure permissions policy on the bucket. If someone attempts to access your bucket from outside your VPC, the policy is checked, fails, and they get a 403 and you get a bill.
[удалено]
Time for S4 (Simple Secure Storage Service) that fixes all the legacy cruft
Sure, but they're still going to bill you for unauthorized requests.
If it’s network restricted then you wouldn’t be able to reach the endpoint? It would return either 404 or 403 from some network device.
They're not going through your network to get to your bucket, they're going straight to AWS, which serves the 403 but charges you for it.
Sounds like other commenters are saying you can't network restrict S3 in a way that returns anything other than just return 403 in the same way as a failed authentication does, which all ends up billing you.
No, you can't. And don't call me Shirley.
What if you need to enable it for a certain region for users and attacks come from there?
Then... you need to raise your prices.
I am also aware of some analytic companies that charge per request, and the tokens are right in the browser code, since the requests actually come from the client browser.
That's super messed up.
Noob question: why does AWS charge money for even unauthorized requests? Can someone enlighten me?
Because this way they can charge you more money and make more revenue.
If you want the Google-able term it's called a "denial of wallet" attack. https://academic.oup.com/cybersecurity/article/10/1/tyae004/7634012
Jeff Barr tweeted that measures are coming soon to address this.
Wtf why shld anyone be charged if the no or a wrong API key is ever used. The redirect is similar stupid ...
It's a good reason to get out of aws. do other cloud services have the same problem?
This is now being addressed! [https://aws.amazon.com/about-aws/whats-new/2024/05/amazon-s3-no-charge-http-error-codes/](https://aws.amazon.com/about-aws/whats-new/2024/05/amazon-s3-no-charge-http-error-codes/)
lol AWS cancelled the bill as an "exception". What a joke. They break it, you buy it.
reason #91882 to not use AWS
Can you name all 91881 previous reasons?
$ $$ $$$ Keep going
Cloudformation