Avoid 4 Patterns of Cloud Cost Disaster — How Not to Accidentally Hit the Nuclear Launch Button
1. The Nuclear Chain Reaction — Anything Event Based & FaaS
The Event — Beware of event for example S3 file upload then trigger Lambda.
The event can occur in the way you are not expecting with unexpected number of data.
Never share or set match-all rule.
For example, you write Lambda to resize image that upload from your system but in the same bucket it also connect to web portal that later the customers upload 10,000 images.
You write rule that trigger your lambda upon file upload without specify path or separate bucket. Boom !
The Recursive — Calling Lambda upon some event and Lambda save back then you got the chain reaction. S3 lambda job that put file back in S3 then trigger again.
This sound unlikely, but normally the flow is long and at some part of the flow and only on some specific condition it trigger back to yourself.
2. Dam Failure — Anything with Big Data
Plugging anything into big data is always dangerous
High traffic —Open the web server then data transfer out without cache.
Replication — Accidentally large data replication on S3 etc.
Transform — Lambda trigger on large data set e.g., Network logs, All S3 bucket text analysis and extraction etc.
Even sneak and peek into the big data can be dangerous. This can trigger the table scan e.g., AWS QLDB can scan all table if misuse → huge cost
3. Land Mine Casualty — Anything Enterprise, Organization, Auditing, Support and License
Always Cautious with Anything Microsoft, Oracle and big Enterprise
— Window license (although you already turn it off), SQL Server License and plan, Oracle, SAP etc.
Marketplace License and Upfront Disaster — This is provided by third party 1-click provisioning. Be sure when clicking anything on the market place.
Organization & Auditing Enterprise solution is never free
— If the solution is not for normal people e.g., scan for treat, dividing organization into business unit, Data warehouse, multi-account configuration etc.
You must check pricing every time. Pricing can be easily misunderstood e.g., CloudTrail is only free for the first trail.
Support — e.g., Premium support, Enterprise support, Business support.
4. Sinking Ship — Anything Time-based Service You Forget to Close
Anything that charge you by hours. Normally, it is cluster / platform fee. e.g., Kubernetes, Kafka, Load balancer etc.
When you try service like this, close it as soon as possible. Otherwise you might forget and surprise with the bill at the end of the month.
It’s like poking the hole into your ship.
Extra Cautious on Azure
Every click can risk your career and wallet. Azure cost is never a joke.
It’s like walking in the war zone full of land mine.
Check pricing of each option* before clicking on it.
Every provision and config works like — Explosive Ordnance Disposal Operation (image below)
Something common and simple like load balancer (application gateway v2) can be so damn expensive.
Wow. You could feel better being fired than pay the Azure bill yourself !
Solution
Solution 1— Budget alert
Always setup incremental budget alert especially on the service like FaaS.
You will know and act in time.
Solution 2— Grant developer only their project privileges
Grant by tagging the project and grant to them such resources.
You will never have 100 people risk mistakenly open Kubernetes cluster. They do not have the right to.
Solution 3 — Grant Admin Right only on Common Service
Full admin is nothing different from leaving your kid with credit card. They can buy anything.
Anything at Walmart but not Louis Vuitton.
Walmart is like AWS EC2, S3, RDS, Lambda i.e., any common service. This will prevent the admin stepping into the landmine.
When they do something dangerous they are assign the privileges at that time. This work like Linux SUDO !
Hope this helps !