Keeping cloud costs under control

Cloud waste usually sits in things nobody looks at anymore: test environments running all night, disks left behind by servers that are long gone, old snapshots and logs without a retention period. A fixed routine is enough to deal with that: you know what is running and who owns it, and you look at the bill per owner every month.

I work on infrastructure and its cost in client projects myself. My examples come from AWS, but the approach works on any cloud.

Start with an inventory

List every account, subscription and project, and for each account the regions where something runs. A resource in a region nobody else uses will otherwise never be noticed.

For each resource, note what it is, which environment it belongs to (production, staging or test), who owns it and whether it is still needed. A spreadsheet is fine for the first pass. The goal is that every line on the bill maps to a person and a product.

Include the services around your cloud as well, such as monitoring, logging, CI minutes and a CDN. They arrive on a different invoice, but they are part of the same cost.

Tag every resource

Tags keep the inventory up to date. Three tags are enough to start: owner, environment and project. Define them in your infrastructure as code, so a new resource without tags does not pass review.

On AWS, user-defined cost allocation tags have to be activated in the Billing console before they appear in Cost Explorer and the cost allocation report, and it can take up to 24 hours before they show. Activate a new tag right away instead of waiting until you need a report.

Also keep a list of untagged resources and go through it at every review. Anything that stays on the list for more than a month gets an owner or gets removed.

Switch off what nobody uses

Test environments and development servers don't need to run at night or over the weekend. A schedule that stops them at 7 pm and starts them at 7 am is a few lines of code.

A server that is switched off still costs money. A stopped EC2 instance does not incur compute charges, but the storage of its EBS volumes is still billed. Elastic IP addresses also cost money regardless of the instance state.

Deleting things can leave pieces behind too. When you terminate an instance, AWS by default deletes the root volume but keeps other EBS volumes. Snapshots are separate from the instance and stay as well. So check every month for volumes that aren't attached to anything and for old snapshots without an owner.

My advice is to take a final backup and delete a resource if nobody claims it within two weeks.

Right-size based on measurements

Teams often size servers and databases generously, "to be safe", and never look at them again. Look at CPU and memory over a few weeks rather than one day, and make sure a busy period is included.

AWS Compute Optimizer does this for you once you opt in. By default it analyses CloudWatch metrics from the last 14 days, suggests other sizes and flags idle resources. Treat a suggestion as a starting point and make the decision yourself, because you know which peaks are coming.

A database deserves extra care, because downsizing needs a maintenance window. Also check whether it is large because it does work that an index or a cache could do more cheaply (see scalability: what you actually need). Whether a workload belongs on serverless or on your own servers depends on the shape of your traffic. That is the subject of serverless or your own servers.

Only once the sizes are right should you look at reservations or longer-term commitments. If you reserve first and downsize later, you pay for capacity you don't use.

Storage and logs grow quietly

Storage is cheap per gigabyte, but it grows every day and usually nobody cleans it up.

Put a lifecycle rule on every bucket. In S3 you can move objects to a cheaper storage class after a number of days, or let them expire so S3 deletes them for you. Transitions do carry request costs, so for very large numbers of small files, moving them can cost more than it saves.

Logs are the other quiet grower. Give every log group a retention period and only enable debug logging in production temporarily. A few weeks of logs in searchable storage followed by a cheap archive is enough for many teams.

Backups and snapshots grow quietly as well. Copies your retention policy does not require only cost money. How to set that policy is covered in backups and disaster recovery. Finally, keep an eye on data transfer, such as traffic to the internet and between regions. It grows with usage and often only gets noticed once it is a sizeable item.

Budgets and alerts

Set a budget with alerts per account or project, for example at 80 and 100 percent of what you expect. Send the alerts to a channel someone actually reads, not the inbox of an old account.

An alerts-only budget warns you, but it does not switch anything off. Google Cloud's documentation states that such a budget does not automatically cap usage or spending. Someone has to act on the alert.

A fixed one-hour review every month

Without a fixed review, this slips again. Book an hour every month with the team's tech lead and someone who knows the budget, such as the CTO or the founder. Go through the same list each time:

1. Compare cost per tag with last month and find the biggest increases.
2. Look at the ten largest cost items. Do you know why each one is there?
3. Go through the untagged resources.
4. Look for unattached volumes, old snapshots and servers that have been stopped for a long time.
5. Read the right-sizing suggestions, on AWS for example from Compute Optimizer.
6. Write down what you decide, who does it and by when.

The notes can be a few lines in your wiki. The next review starts by checking whether last month's actions were done.

Sources

• AWS: cost allocation tags
• AWS: EC2 instance state changes and billing
• AWS Compute Optimizer
• Amazon S3: managing the lifecycle of objects
• Google Cloud: create, edit or delete budgets and budget alerts