The recent AWS outage has been frustrating for many. If you’d like to know why this is happening and how to avoid it, read on!
Summary
December started with a bang thanks to the abundance of holidays. This included an increase in online retail sales, packages flooding delivery services, and high traffic on websites across America! But then something happened… OutAGES.
Amazon Web Services (AWS) US East 1 region went down in mid-December, which impacted many companies that rely heavily upon this service for their business all over America and Europe.
AWS Outage 2021 Details
There have been two outages so far in December. There was one on 7th December 2021 and one on 15th December 2021.
Both incidents involved the AWS network, where service reachability was being blocked by congestion. In the case of the December 7th outage, traffic loss owing to congestion was originating between AWS’ main and internal management networks. The situation on 15th December occurred inside their main network, where traffic from sources both on and off AWS was being discarded.
In the first case on 7th December, AWS devices for critical traffic routing were overwhelmed. In the second outage, some network functions in the data path were not working at full capacity. This prevented apps and services from reaching their intended audience.
Many problems have occurred in the past when a network’s control plane has not functioned correctly, and things have not gone as planned.
In December 2021, AWS experienced a significant outage that raised concerns among users and shed light on the history of AWS outages. This particular incident was primarily attributed to issues with AWS Fargate, a container orchestration service. An AWS outage occurs when there is a disruption in the availability of Amazon Web Services, impacting various businesses and services relying on AWS infrastructure. These outages can have widespread consequences, underscoring the critical importance of redundancy and disaster recovery planning for businesses relying on cloud services like AWS.
These recent events do not appear to be repeating this pattern. Even if the control plane is down, it is still possible to route traffic correctly.
What Services are affected?
AWS is the largest cloud provider globally, so when it goes down, it takes a lot of websites with it. AWS isn’t just the largest provider but also has the highest customer retention rate of its competitors. Over 11,000 websites were down – from AWS services and Amazon’s network of brands to seemingly unconnected sites including Netflix, Disney+, Delta airlines, and Instacart.
This outage affected several AWS services, including Amazon CloudWatch, Gateway API, Secure Token Service (STS), and container services like Fargate, ECS, and EKS. The popular EC2 service provides a virtual server capacity.
The API request did not affect running instances and containers, but they could not be used to modify or start new ones.
Amazon’s delivery people were unable to deliver items due to a problem with their custom-built app that specified which packages go whereas the servers could not communicate.
Alexa-powered speakers, lightbulbs, and smart gadgets suddenly stopped working.
Netflix had a massive drop in visitors at the same time; day traders on Robinhood also faced issues. Because, unbeknownst to many, AWS maintains so much of the internet, it’s difficult to isolate an area that wasn’t affected by the outage.
While Amazon has issued an apology and vowed to do better, this latest outrage is sure to raise doubts about AWS’s reliability.
It’s not just this week’s outage that has some people concerned about AWS. The service has had a history of outages, including a major outage in 2016 that affected companies like Netflix and Reddit.
Boost your earning potential with AWS expertise. Explore our certified AWS Courses for a high-paying career
What the AWS Outage Taught Us About Cloud Dependency?
An AWS outage doesn’t mean every application hosted on AWS stops working. In December 2021 we saw how dependencies between cloud services can affect the real-world impact of an outage. If one piece of infrastructure is down or slow, other services that rely on that piece may also be down, slow, or not fully functional.
This is especially relevant for organizations developing applications atop many managed AWS services. An application could be dependent on services like Amazon EC2, Amazon ECS, AWS Lambda, Amazon CloudWatch, IAM, networking, databases and APIs. Even if the application servers are up, a down supporting service can stop teams from deploying resources, viewing monitoring information, authenticating users, or doing administrative tasks.
So what happened in December 2021 is an important lesson for us, in terms of cloud architecture: Availability should be considered over the whole dependency chain, and not only at application-server level. AWS builds services with redundancy and fault isolation, but you still need to architect resilient applications in your own environments.
They need to watch for critical dependencies for each service and what to do with a failing dependency. They also need to differentiate between data-plane operations that service existing workloads and control-plane operations that create or modify resources. This distinction helps teams understand which application functionality may still be available during a disruption of an AWS service, and which functionality may not be available for some time.
What Caused The Outage in 2017, 2018, and 2019?
A little history. Last year’s outage had something to do with an error in one of Amazon’s pieces of code. That particular piece of code was designed to protect its AWS S3 storage system, which is how it protects and preserves data for clients that use it for their own cloud-based applications.
The 2017 outage lasted several hours, but because so many people rely on AWS today—including Netflix, Slack, and Trello—the potential impacts were great and immediate.
As such, there are certain expectations about availability (or lack thereof) as customers make decisions about where they want to host their services. Unfortunately for these companies and others like them, those expectations have been violated time after time by Amazon Web Services outages over the past few years.
So what should companies be doing? Should they be moving away from AWS?
That’s not really an option, as AWS is the dominant player in the cloud market. Instead, companies need to be prepared for outages and have backup plans in place. That means having alternative services ready to go if AWS goes down.
It’s also important to keep an eye on AWS’s reliability and make sure that you’re only using services that are reliable. If an outage does happen, you don’t want it to take your whole business down with it.
It’s not like only AWS had an outage; recently, on 1 Apr 2021, Microsoft Azure was also down for some time. AWS is an excellent service, but no service can be perfect; they become perfect with experience. Companies need to be aware of the risks and have backup plans in place in case of an outage.
How Should I Prepare for an AWS Cloud Instance Crash or Problem?
There’s no guarantee that you won’t be affected by an outage. Amazon Web Services (AWS) is known for its exemplary cloud service but can experience problems just like any other hosting provider. If you run a business that relies on a well-functioning server, you may have to take precautions in order to avoid downtime due to issues with your hosting company.
How to Design AWS Workloads for Greater Resilience?
The AWS outages in December 2021 are a reminder that when building cloud applications, you should build for failure, not assume that any one AWS service or region will always be available. AWS Regions and Availability Zones AWS has multiple Availability Zones . An application deployed to AWS is not necessarily resilient . The architecture should be designed to remove or reduce single points of failure.
Deploying across multiple Availability Zones is an important strategy. Applications can distribute work across Availability Zones, so that a failure at one site does not necessarily bring the entire application down. Organizations may also want to consider multi-Region architectures for workloads with higher availability requirements, depending on business needs, recovery objectives, cost, and complexity of the applications.
Teams should also build in redundancy for critical components including databases, compute, DNS, networking and storage. Load balancing distributes traffic among healthy resources and automated health checks and failover capabilities assist in redirecting users when an endpoint is down.
Another important practice is to have tested backup and disaster recovery practices in place. Backups aren’t enough. Periodic recovery exercises must be conducted by organizations to demonstrate that critical systems and data can be restored within the required Recovery Time Objective (RTO) and Recovery Point Objective (RPO).
“Monitoring is important. Tools such as Amazon CloudWatch can provide metrics, alarms and operational visibility to enable teams to quickly surface abnormal behavior. Monitoring must be linked with well-documented incident-response procedures so engineers know what to look for and what recovery actions to take.
The point is not to avoid all failure. Instead, a resilient AWS architecture tries to reduce the blast radius, maintain critical functionality, and quickly recover when individual components fail.
How Can I Get Help Immediately After an AWS Cloud Instance Crash?
In addition to preparing yourself for a possible outage ahead of time, it’s important to plan ahead should one occur. One step toward making sure you aren’t left without internet access after suffering from an AWS crash is having phone numbers handy.
AWS knowledge is helpful for a high-paying career. Explore more of our resources to get trained for AWS career.
- Explore AWS Route 53 interview questions.
- Practice for your interview with AWS live project training.
- Read top AWS IAM interview questions.
- Read concepts of data ingestion in AWS.
- Read the top EC2 interview questions and answers.
Don’t for to register for our free zoom webinars on cloud computing.
3. AWS Outage Checklist: What Businesses Should Do
A backup server > An AWS outage. You have to be ready. Organizations should have an outage response checklist that includes detection, communication, failover, recovery, and post-incident review. It provides a process framework to allow technical teams to react consistently and avoid making decisions when under pressure.
1. Health check the AWS service. When an application starts failing for no apparent reason, the first thing teams should do is determine whether the issue is isolated to their infrastructure or linked to an AWS service event. AWS service-status information can help you understand the impact of an incident.
2. Identify affected dependencies. See what AWS Services your app is using. Identify the problem area compute, storage, network, authentication, database, monitoring or some other supporting service.
3. Protect Critical Operations. Do not change configurations unless absolutely necessary during an incident. Teams need to be very careful about any additional risk to workloads that are still running from any administrative action.
4. Failover Procedures. If your architecture is built with redundant Availability Zones, Regions or other recovery environments, start the documented failover process. You have to test failover in the hope of a real event.
Conclusion
It’s important to know that AWS is also one of the leading providers of cloud computing services. This outage has caused many companies who rely heavily on this service for their business, all over America and Europe, to have significant downtime. To learn more about how you can best protect your company from outages like these, please reach out with any questions or concerns so we can help guide you through your options. You can also check our other blogs and training courses to learn and explore cloud technologies.
