AWS Morning Brief (Corey Quinn)

Links Mentioned

• Want to give your ears a break and read this as an article? You’re looking for this link: https://www.lastweekinaws.com/blog/reader-mailbag-accounts/

Sponsors

• StrongDM: https://strongdm.com
• Linode: https://www.linode.com

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

About Corey Quinn

Over the course of my career, I’ve worn many different hats in the tech world: systems administrator, systems engineer, director of technical operations, and director of DevOps, to name a few. Today, I’m a cloud economist at The Duckbill Group, the author of the weekly Last Week in AWS newsletter, and the host of two podcasts: Screaming in the Cloud and, you guessed it, AWS Morning Brief, which you’re about to listen to.

Links

• CHAOSSEARCH.io
• Amazon Route 53
  
  

Transcript

Corey: Corey: Welcome to AWS Morning Brief: Whiteboard Confessional. I’m Cloud Economist Corey Quinn. This weekly show exposes the semipolite lie that is whiteboard architecture diagrams. You see, a child can draw a whiteboard architecture, but the real world is a mess. We discuss the hilariously bad decisions that make it into shipping products, the unfortunate hacks the real world forces us to build, and that the best to call your staging environment is “theory”. Because invariably whatever you’ve built works in the theory, but not in production. Let’s get to it.

But first… On this show, I talk an awful lot about architectural patterns that are horrifying. Let’s instead talk for a moment about something that isn’t horrifying. CHAOSSEARCH. Architecturally, they do things right. They provide a log analytics solution that separates out your storage from your compute. The data lives inside of your S3 buckets, and you can access it using APIs you’ve come to know and tolerate, through a series of containers that live next to that S3 storage. Rather than replicating massive clusters that you have to care and feed for yourself, instead, you now get to focus on just storing data, treating it like you normally would other S3 data and not replicating it, storing it on expensive disks in triplicate, and fundamentally not having to deal with the pains of running other log analytics infrastructure. Check them out today at CHAOSSEARCH.io.

So I’m going to deviate slightly from the format that I’ve established so far on these Friday morning whiteboard confessional stories, and talk instead about a pattern that has tripped me and others up more times than I care to remember. So it’s my naive hope that by venting about this for the next 10 minutes or so, I will eventually be able to encounter an environment where someone hasn’t made this particular mistake. And what mistake am I talking about? Well, as with so many terrifying architectural patterns, it goes back to databases. You decide that you’re going to write a small toy application, You’re probably not going to turn this into anything massive. And in all honesty, baby seals will probably get more hits than whatever application you’re about to build will. So you don’t really think too hard about what your database structure is going to look like. You spin up a database, you define the database endpoint inside the application, and you go about your merry way. Now, that’s great. Everything’s relatively happy, and everything we just described will work. But let’s say that you hit that edge or corner case where this app doesn’t fade away into obscurity. In fact, this turns out to have some legs, the thing that you’re building now has attained business viability or is at least seeing enough user traffic that it now has to worry about load.

So you start taking a look at this application because you get the worst possible bug reports six to eight months later; it’s slow. Where do you start looking when something is slow? Well, personally, I start looking at the bar, because that is a terribly obnoxious problem to have to troubleshoot. There are so many different ways that latency can get injected into an application. You discover the person reporting the slowness is on the other side of the world with satellite internet connection that they’re apparently trying to set up to the satellite with a tin can and a piece of very long string. There’s a lot of failure states here that you get to start hunting down. The joys of latency hunting. But in many cases, the answer is going to come down to, oh, that database that you defined is now no longer up to the task. You’re starting to bottleneck on that database. Now, you can generally buy your way out of this problem by scaling up whatever database you’re using. Terrific, great, it turns out that you can just add more hardware, which in a time of cloud, of course, just means more money and a bit of downtime while you scale the thing up, but that gets you a little bit further down the road. Until the cycle begins to rinse and repeat, and it turns out, there are only instances that are so large that you’ll be able to get to power databases. Also, they’re not exactly inexpensive. Now, I would name exact sizes of what those databases might look like. But this is AWS, they’re probably going to release at least five different instance families and sizes, by the time I finish recording this. But it gets published later at the end of the week. So instead, there is an alternative here, and it doesn’t take much from an engineering or design perspective when you’re building out one of these silly toy apps that will never have to scale. What is that fix, you might wonder? Terrific question. Let me tell you in just a minute. 

In the late 19th and early 20th centuries, democracy flourished around the world. This was good for most folks, but terrible for the log analytics industry because there was now a severe shortage of princesses to kidnap for ransom to pay for their ridiculous implementations. It doesn’t have to be that way. Consider CHAOSSEARCH. The data lives in your S3 buckets in your AWS accounts, and we know what that costs. You don’t have to deal with running massive piles of infrastructure to be able to query that log data with APIs you’ve come to know and tolerate, and they’re just good people to work with. Reach out to CHAOSSEARCH.io. And my thanks to them for sponsoring this incredibly depressing podcast. 

So this is a pattern that increasingly, modern frameworks are recommending, but a number of them don’t. And I’m not going to name names, because I don’t want to wind up in a slap and tickle fight around which frameworks are good versus which frameworks are crappy. You can all make your own decisions around that. But the pattern that makes sense for this is even when you’re beginning with a toy app, go ahead and define two database endpoints, one for reads, And one for writes. Invariably, this is going to solve a whole host of problems with most database technologies. If you take a look at most applications, and yes, I know there are going to be exceptions to this, they tend to bottleneck on reads. If you have just a single database or database cluster, then all of the read traffic gets in the way of being able to write to that. That includes things that don’t actually need to be in line with the rest of what the application is doing. If you can have a read replica that’s used for business analytics, great. Your internal business teams can beat the living crap out of that database replica without damaging anything that’s in the critical path of serving users. And the writes can then go specifically to the primary node, w...

AWS Morning Brief for the week of Monday, June 24th, 2024, with Corey Quinn. 

Links:

• Amazon CloudWatch announces AI-Powered natural language query generation
• AWS Compute Optimizer supports rightsizing recommendations for Amazon RDS MySQL and RDS PostgreSQL
• AWS CloudFormation Linter (cfn-lint) v1

Last week in security news: AWS Firewall Manager supports referencing of Security Groups, Secure by Design: AWS to enhance MFA requirements in 2024, You Can't Control Your Data in the Cloud, and more!

Links:

• You Can't Control Your Data in the Cloud
• Chris Farris leaked 7 IAM keys in public (intentionally!
• Chris Farris also writes Security Hub gives me imposter syndrome
• Google is stuffing previously widely available security offerings into incredibly expensive paid-for tiers.
• AWS Firewall Manager supports referencing of Security Groups
• Secure by Design: AWS to enhance MFA requirements in 2024 
• Reported TorchServe Issue (CVE-2023-43654)
• Tip of the week: patch the embargoed libcurl high-severity issue that was made public after this recording but before publishing. 

Links Mentioned

• Want to give your ears a break and read this as an article? You’re looking for this link: https://www.lastweekinaws.com/blog/reader-mailbag-aws-services/

Sponsors

• StrongDM: https://strongdm.com
• Linode: https://www.linode.com

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

Links:

• Unconventional Guide to AWS Cost Management:https://www.duckbillgroup.com/resources/unconventional-guide-to-aws-cost-management/
• AWS Tagging Best Practices Whitepaper: https://docs.aws.amazon.com/whitepapers/latest/tagging-best-practices/welcome.html

Transcript

Corey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I’m going to just guess that it’s awful because it’s always awful. No one loves their deployment process. What if launching new features didn’t require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren’t what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.

Pete: Hello, and welcome to the AWS Morning Brief: Fridays From the Field. I'm Pete Cheslock.

Jesse: I'm Jesse DeRose.

Pete: And we're back again, Jesse. We are back. But really have we gone anywhere to begin with?

Jesse: We've been making our way slowly but surely through this Unconventional Guide. Lots of really interesting recommendations, lots of really interesting feedback from all of you, which we really, really appreciate. We can't wait to dive into some of those ideas deeper in future episodes.

Pete: Yeah. And don't forget, you can give us additional feedback and questions at lastweekinaws.com/QA, feel free to add your name. Or not. Doesn't matter. It can be totally anonymous. That's fine with us. So today, we're talking about a topic that is very near and dear to our hearts. 

Jesse: Yes.

Pete: It is tagging.

Jesse: Yes.

Pete: Tagging your resources in Amazon, or I mean really any cloud provider; any place you can tag something you probably should. And we're going to talk a little bit about strategies for that, how people use their tags, just all the fun things related to it. Tagging, it's easy to do, right, Jesse? You just tag your resources and all your problems go away.

Jesse: Yep. Thanks, everybody, have a good night.

Pete: So yeah, if you've enjoyed this podcast, please go to—no, I’m just kidding. 

Jesse: [laugh].

Pete: Tagging is probably the thing that most companies are doing poorly, simply because it's hard, and it's an afterthought, and if you didn't have a really solid forced strategy to ensure tags and force compliance, you're probably not going back to fix it.

Jesse: Yeah. It's not thought about as something that's a first-class citizen in the cloud world. When you think about the things that are important to your business model, you might think about getting your application out the door and running, maybe talking about business requirements for availability, failover, data retention, but tagging is nowhere on that list. That's not something that I think any organization thinks about as part of an MVP, let alone future iterations of their products.

Pete: Tagging feels much like the same feeling I get when my doctor says that I should eat more veggies. 

Jesse: Oof.

Pete: I know they're good for me; I know we need to do this. They have vitamins, and fiber, and all these wonderful things. But in order to make those veggies something I want to eat, we have to learn to make it more delicious. Personally, I find duck fat works to make them more delicious. I wish we could apply a duck fat strategy to the tagging problem.

Jesse: Yeah, it's not an easy problem to solve. Or rather, I should say it is an easy problem to solve, but it's not something that anybody is quickly incentivized to solve. Tagging, just for the sake of tagging, it doesn't work.

Pete: Yeah, it's that there really are no incentives for it. No good incentives. It's usually because someone came over to your desk and said, “Hey, what's this charge for? And who's using it? And what's the deal with this?” 

And you're going into Cost Explorer, and you're like, “Uh, I don't know. It's in this one account.” And that's as far as you can go to figure out who did what and why that thing is the way it is.

Jesse: Yeah. There are so many different tagging strategies that we've seen. We've seen some clients talk about tagging as a way to potentially penalize engineers who aren't tagging or who are spending too much money. We've seen organizations who are tagging to reward teams that are tagging all their spend or keeping their spend optimized. Across the board, there are just so many different ways to go about this.

Pete: So let's assume you are like most of the companies that we've seen. Definitely not all: there are some rare gems out there that are making tagging a long term and continual process, which we're actually going to talk about in a future episode, how to do that. But let's say you're just looking at your bill, you're looking at your usage, and you're saying to yourself, “Okay. I need to be better at this.” What do they say, “The journey of a thousand miles starts with a single step?” What is that first step?

Jesse: Yeah there's a lot of different ways to go about this. I think there's a couple great places to start. Now, I will say AWS has a thrilling 24-page best practices white paper that we’ll throw a link in the [show notes 00:05:18]. 

Pete: Have you read that, Jesse?

Jesse: I will say that I have read parts of it. I have not read all of it, and so I want to make it very, very clear to all of our listeners, this is not a document that needs to become the holy grail for your organization. I think in the same way that you could read the SRE book from Google and have some good takeaways, you can skim through this white paper, maybe read through a couple of the sections that seem most applicable to your organization, and then start with those ideas, start with those best practices, and then build them over time organically; develop them over time organically.

Pete: I like to read it some nights when I'm just having trouble sleeping, and maybe by page two or three I’m just out.

Jesse: Yeah. There's a lot of content in there talking about what to tag, why to tag. I think the best place for any organization to start is to think about what are the important things that we need to tag. And that's a conversation that's going to involve not just engineers, but also finance, potentially IT, maybe also security teams, depending on how your organization is built. Because ultimately, what you want to do is understand what are the things that my organization cares about when it comes to our cloud usage?&n...

Want to give your ears a break and read this as an article? You’re looking for this link. 

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

AWS Morning Brief for the week of October 30, 2023 with Corey Quinn. 

Links:

• Amazon Aurora MySQL includes optimizations that reduce the database restart time by up to 65%
• Amazon EKS adds support for customer managed IAM policies
• AMI Block Public Access now enabled for all new accounts and existing accounts with no public AMIs
• AWS Config now supports 19 new resource types
• AWS Marketplace announces enhanced private offer user experience for sellers
• AWS re:Post introduces Selections
• EC2 Hibernate now supports 20 additional instance families on EC2 Spot 
• Announcing general availability of Amazon EC2 M2 Mac instances for macOS 
• Comparing AWS Lambda Arm vs. x86 Performance, Cost, and Analysis
• How Infosys Built an Enterprise Knowledge Management Assistant Using Generative AI on AWS
• Rotate Your SSL/TLS Certificates Now – Amazon RDS and Amazon Aurora Expire in 2024
• Build ROSA Clusters with Terraform
• Build a web-based cryptocurrency wallet tracker with Amazon Managed Blockchain Access and Query 
• Why AWS is the Best Place to Run Rust
• What’s top of mind for Chief Data Officers going into 2024?
• EFA: how fixing one thing, lead to an improvement for … everyone | AWS HPC Blog
• Is Generative AI the Answer to All Questions?
• How to download your AWS Resilience Hub assessment results 
• Gain practical experience building with Amazon CodeWhisperer through AWS Jam
• Boost your AWS proficiency with Solution-Focused Immersion Days
• Summary of the AWS Service Event in the Northern Virginia (US-EAST-1) Region
• AWS Disallowing Resale of Reserved Instances that were purchased at a discount

Want to give your ears a break and read this as an article? You’re looking for this link.

https://www.lastweekinaws.com/blog/9-ways-aws-cdk-headdesk

Want to watch the full dramatic reenactment of this podcast? Watch the YouTube Video here: https://youtu.be/3Mf3_l6iEtA

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

AWS Morning Brief Extras edition for the week of March 29, 2023.

Want to give your ears a break and read this as an article? You’re looking for this link.https://www.lastweekinaws.com/blog/s3-as-an-eternal-service

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

Buy our merch 

• https://store.lastweekinaws.com

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

Want to give your ears a break and read this as an article? You’re looking for this link.

https://www.lastweekinaws.com/blog/status-paging-you

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

Want to give your ears a break and read this as an article? You’re looking for this link. 
https://www.lastweekinaws.com/blog/how-aws-measures-its-customers 

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

Links:

• Amazon CloudFront now supports the removal of response headers 
• Amazon SageMaker is now available in AWS Middle East (UAE) Region
• Amazon Neptune announces graph-explorer, an open-source visual exploration tool for low-code users
• An elastic deployment of Stable Diffusion with Discord on AWS 
• Measure the Business Impact of Personalize Recommendations 
• How Heineken’s Connected Brewery Ecosystem fuels automation 

Links:

• In this down market, it's good to know that jobs paying six (and rarely, seven!) figure salaries, giving bonuses, and of course including paid time off are still out there. Unfortunately they're working for cybercrime groups.
• Ian McKay is great--but given his history of creating awesome-yet-horrifying things in AWS I read this piece on Cedar (AWS's new policy language) 
• Popular drone manufacturer CrowdStrike reports on how Adversaries Can Persist with AWS User Federation,
• How to set up ongoing replication from your third-party secrets manager to AWS Secrets Manager 
• Want to chain roles in a way that works for more than an hour? Role Chain Juggling has you covered. 

Links:

• Google Cloud Build deep dive
• Andrea Brancaleoni found an ELB header security issue
• An article on You Can't Opt Out of Citizen Development
•  DOJ Announces It Won’t Prosecute White Hat Security Researchers
• Choosing the right certificate revocation method in ACM Private CA
• a somewhat... controversial AWS Security Maturity Model
•  AWS API calls that return credentials on GitHub

Want to give your ears a break and read this as an article? You’re looking for this link. https://www.lastweekinaws.com/blog/turn-that-sh—-off

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

AWS Morning Brief for the week of February 27, 2023 with Corey Quinn.

Links:

• Amazon OpenSearch Service now lets you schedule service software updates during off-peak hours 
• AWS App Runner now supports HTTP to HTTPS redirect
• Announcing the ability to enable AWS Systems Manager by default across all EC2 instances in an account 
• New: AWS Telco Network Builder – Deploy and Manage Telco Networks
• Developing portable AWS Lambda functions
• Using Porting Advisor for Graviton 
• Query data with DynamoDB Shell – a command line interface for Amazon DynamoDB
• AWS and Hugging Face collaborate to make generative AI more accessible and cost efficient
• Branch Insurance improves hiring diversity and accelerates app development using AWS AppSync
• Gain compliance insights using the open source community for AWS CloudTrail 
• The true costs of resiliency decisions 

Links:

• S3 Bucket Negligence Award
• Mandoogle on how AWS's instance metadata service can be abused by attackers
• Heroku apparently had its entire database breached last week
• Wiz Research discovered a new vulnerability in Azure’s PostgreSQL Flexible Server service.
• AWS deleted packages they'd pushed to public repositories
• A guide to Cloud Security Orienteering

Want to give your ears a break and read this as an article? You’re looking for this link.

https://www.lastweekinaws.com/blog/the-strange-too-familiar-tale-of-uncle-suitcase/

Want to watch the full dramatic reenactment of this podcast? Watch the YouTube Video here: https://youtu.be/x70EypnAH1Y

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

Links:

• Principals in AWS IAM: https://ben11kehoe.medium.com/principals-in-aws-iam-38c4a3dc322a
• You Don’t Need to Burn off Your Fingertips (and Other Biometric Authentication Myths): https://www.troyhunt.com/you-dont-need-to-burn-off-your-fingertips-and-other-biometric-myths/
• Amazon Detective offers Splunk integration: https://aws.amazon.com/about-aws/whats-new/2021/09/amazon-detective-splunk-integration/
• IAM Vulnerable - An AWS IAM Privilege Escalation Playground: https://labs.bishopfox.com/tech-blog/iam-vulnerable-an-aws-iam-privilege-escalation-playground

Transcript

Corey: This is the AWS Morning Brief: Security Edition. AWS is fond of saying security is job zero. That means it’s nobody in particular’s job, which means it falls to the rest of us. Just the news you need to know, none of the fluff.

Corey: This episode is sponsored in part by Thinkst Canary. This might take a little bit to explain, so bear with me. I linked against an early version of their tool, canarytokens.org, in the very early days of my newsletter, and what it does is relatively simple and straightforward. It winds up embedding credentials, files, or anything else like that that you can generate in various parts of your environment, wherever you want them to live; it gives you fake AWS API credentials, for example. And the only thing that these are empowered to do is alert you whenever someone attempts to use them. It’s an awesome approach to detecting breaches. I’ve used something similar for years myself before I found them. Check them out. But wait, there’s more because they also have an enterprise option that you should be very much aware of: canary.tools. Take a look at this: what it does is it provides an enterprise approach to drive these things throughout your entire environment and manage them centrally. You can even get a physical device that hangs out on your network and impersonates whatever you want to. When it gets Nmap scanned, or someone attempts to log into it, or access files that it presents on a fake file store, you get instant alerts. It’s awesome. If you don’t do something like this, instead you’re likely to find out that you’ve gotten breached the very hard way. So, check it out. It’s one of those few things that I look at and say, “Wow, that is an amazing idea. I am so glad I found them. I love it.” Again, those URLs are canarytokens.org and canary.tools. And the first one is free because of course it is. The second one is enterprise-y. You’ll know which one of those you fall into. Take a look. I’m a big fan. More to come from Thinkst Canary in the weeks ahead.

Corey: Ben Kiko, cloud robotics research scientist at iRobot—motto: “All IoT sucks, but ours is supposed to”—walks us through Principles in AWS IAM. It’s short, it’s concise, and it’s definitely worth taking the time to dig into what he has to say. If you only hunt down one thing from this podcast this week, this is the one.

[Version three of OpenSSL was released 00:03:19], so expect a few conversations around that. There’s also apparently a Rusttls, which is ostensibly OpenSSL rewritten in Rust for the modern era but is in practice just another talking point for the Rust evangelism strikeforce, who is actively encouraged not to find a way to leave a comment on this episode.

Sneak or Snack or Synack raised—however they’re pronounced—[raised a big funding round last week 00:03:19] and still stubbornly refuses to buy a vowel. More interestingly, they report that 50% of security jobs are unfilled. Further, any solution predicated on devs becoming security experts is doomed, which is exactly the point of this podcast. What you need to know about cloud security, minus the fluff and 
gatekeeping. Okay fine, yes, and some snark added to keep it engaging because my God, is it dull without that.

Another week, another [Azure Security failure 00:03:19]. This time a flaw existed that could leak data between users of Azure Container Services. Look, this whole thing is about AWS, so why do I talk about Azure issues like this? Simply put, people are going to bring it up in a cloud isn’t secure context, and you should be aware of what they’re talking about when they do. Azure, please get it together. Stuff like this hurts all cloud providers.

Corey: Troy Hunt has a post informing you that despite what your AWS bill may have you believe in the moment, self-immolation is unnecessary. Okay, that’s not actually his point, but specifically, You Don’t Need to Burn off Your Fingertips (and Other Biometric Authentication Myths) doesn’t hit quite the same way. It’s a super handy reminder that for most of you folks, adversaries are not going to steal your fingerprints to get into your systems. They’re either going to bribe you or hit you with a wrench until you tell them your password.

From the mouth of AWS horse—or from the horse’s AWS—Amazon Detective offers Splunk integration. Amazon Detective and the Case of the Missing Mountain of Money is apparently this month’s hot comic book.

And AWS—motto: “Opinions my own”—has a [security checklist 00:03:19], and it’s worth taking a look at because a few of these items that they issue from time to time are, like, “Use multiple AWS accounts,” directly contravenes older guidance. It’s always good to check on things like this around best practices that AWS is putting out there because even if you don’t make changes to your systems as a result, you should know where AWS’s head is at with respect to where the future of the industry is going.

And lastly, there was an interesting tool that came out called IAM Vulnerable. It’s an IAM privilege escalation playground that lets you muck around with exploiting improperly set IAM policies. It’s a good way to kill an hour on an afternoon when you’re not particularly motivated to do other things. Another good ‘I need a distraction’ task is rotating reused or weak passwords that you have in your password manager. And that’s what happened.

Announcer: Have you implemented industry best practices for securely accessing SSH servers, databases, or Kubernetes? It takes time and expertise to set up. Teleport makes it easy. It is an identity-aware access proxy that brings automatically expiring credentials for everything you need, including role-based access controls, access requests, and the audit log. It helps prevent data exfiltration and helps implement PCI and FedRAMP compliance. And best of all, Teleport is open-source and a pleasure to use. Download Teleport at

Want to give your ears a break and read this as an article? You’re looking for this link. https://www.lastweekinaws.com/blog/the-cloud-genie 

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

Links:

• “Cloud Security Breaches and Vulnerabilities”: https://blog.christophetd.fr/cloud-security-breaches-and-vulnerabilities-2021-in-review/
• S3 Bucket Negligence Award: https://mytechdecisions.com/audio/sennheiser-responds-after-customer-data-from-2018-was-exposed-online/
• Granted the role its support teams use to access customer accounts access to S3 objects: https://Twitter.com/0xdabbad00/status/1473448889948598275?s=12
• S3 Bucket Negligence Award: https://www.modernghana.com/news/1127205/report-ghana-government-agency-exposes-100000s.html
• “Simplify setup of Amazon Detective with AWS Organizations”: https://aws.amazon.com/blogs/security/simplify-setup-of-amazon-detective-with-aws-organizations/
• “AWSSupportServiceRolePolicy Informational Update”: https://aws.amazon.com/security/security-bulletins/AWS-2021-007/
• aws-sso-cli: https://github.com/synfinatic/aws-sso-cli

Transcript

Corey: This is the AWS Morning Brief: Security Edition. AWS is fond of saying security is job zero. That means it’s nobody in particular’s job, which means it falls to the rest of us. Just the news you need to know, none of the fluff.

Corey: Are you building cloud applications with a distributed team? Check out Teleport, an open-source identity-aware access proxy for cloud resources. Teleport provides secure access for anything running somewhere behind NAT: SSH servers, Kubernetes clusters, internal web apps, and databases. Teleport gives engineers superpowers. Get access to everything via single sign-on with multi-factor, list and see all of SSH servers, Kubernetes clusters, or databases available to you in one place, and get instant access to them using tools you already have. Teleport ensures best security practices like role-based access, preventing data exfiltration, providing visibility, and ensuring compliance. And best of all, Teleport is open-source and a pleasure to use. Download Teleport at goteleport.com. That’s goteleport.com.

Corey: Well, we’re certainly ending 2021 with a whirlwind in the security space. Log4J continues to haunt us, while AWS took not only an 
outage but also a bit of a security blunder that they managed to turn into a messaging win. Listen on.

But first, the Community. A depressing review of 2021’s “Cloud Security Breaches and Vulnerabilities.” Honestly, it seems like there are just so damned many ways for bad security to set the things we care about on fire. The takeaways are actionable though. Stop using static long-lived credentials and start with the basics before you get fancy.

Sennheiser scores itself an S3 Bucket Negligence Award, and of all the countries in which to suffer a data breach, I’ve got to say that Germany is at the bottom of the list. They do not mess around with data protection there.

And, Holy hell, AWS inadvertently granted the role its support teams use to access customer accounts access to S3 objects. It lasted for ten hours, and while there are mitigations out there, this is far from the first time that AWS has biffed it with regard to an unreviewed change making it into a managed IAM policy. This needs to be addressed. If you’ve got specific questions about how those things are handled, reach out to your account team; but it’s a terrible look. But there’s more to come in a second here.

Corey: This episode is sponsored in part by my friends at Cloud Academy. Something special for you folks: If you missed their offer on Black Friday or Cyber Monday or whatever day of the week doing sales it is, good news, they’ve opened up their Black Friday promotion for a very limited time. Same deal: $100 off a yearly plan, 249 bucks a year for the highest quality cloud and tech skills content. Nobody else is going to get this, and you have to act now because they have assured me this is not going to last for much longer. Go to cloudacademy.com, hit the ‘Start Free Trial’ button on the homepage and use the promo code, ‘CLOUD’ when checking out. That’s C-L-O-U-D. Like loud—what I am—with a C in front of it. They’ve got a free trial, too, so you’ll get seven days to try it out to make sure it really is a good fit. You’ve got nothing to lose except your ignorance about cloud. My thanks to Cloud Academy once again for sponsoring my ridiculous nonsense.

A bit off the beaten path, this week’s S3 Bucket Negligence Award goes to the government of Ghana. This one is pretty bad. I mean, you can’t exactly opt out of doing business with your government, you know?

Now, AWS has two things I want to talk about. The first is that they offer a way to “Simplify setup of Amazon Detective with AWS Organizations.” I’m actually enthusiastic about this one because there’s a significant lack of security tooling available to folks at the lower end of the market. A bunch of companies seem to start off targeting this segment, but soon realize that there’s a better future in selling things to bigger companies for $200,000 a month instead of $20.

Now, “AWSSupportServiceRolePolicy Informational Update.” Now, you heard a minute ago, I was initially extremely unhappy about this mistake. That said, I am such a fan of this notification that I can’t even articulate it without sounding like I’m fanboying. Because mistakes happen and talking about those mistakes and why defense in depth mitigates the harm of those mistakes goes a long way. This affirms my trust in AWS rather than harming it. Meanwhile Azure has absolutely nothing to say about why their tenant separation is aspirational at best.

And lastly a bit of tooling story here. To end up the year, I’ve been kicking the tires on aws-sso-cli over on GitHub, which is a tool for using AWS SSO for both the CLI and web console. I don’t know why the native SSO tooling is quite as trash as it is, but it’s a problem. There’s a lot of value to using SSO but AWS hides it as if the entire thing were under NDA. Thank you for listening. It’s been a heck of a year as we’ve launched the security portion of this weekly nonsense. I’ll talk to you more in 2022. Stay safe.

Corey: Thank you f...

Want to give your ears a break and read this as an article? You’re looking for this link.

https://www.lastweekinaws.com/blog/the-feudal-lords-of-amazon/

Want to watch the full dramatic reenactment of this podcast? Watch the YouTube Video here: https://youtu.be/g1guW6tiR50

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

Buy our merch 

• https://store.lastweekinaws.com

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

About Corey Quinn

Over the course of my career, I’ve worn many different hats in the tech world: systems administrator, systems engineer, director of technical operations, and director of DevOps, to name a few. Today, I’m a cloud economist at The Duckbill Group, the author of the weekly Last Week in AWS newsletter, and the host of two podcasts: Screaming in the Cloud and, you guessed it, AWS Morning Brief, which you’re about to listen to.

Links

• CHAOSSEARCH
• @QuinnyPig
• Chris Short’s LinkedIn: https://www.linkedin.com/in/thechrisshort/

Transcript

Corey: Welcome to AWS Morning Brief: Whiteboard Confessional. I’m Cloud Economist Corey Quinn. This weekly show exposes the semi-polite lie that is whiteboard architecture diagrams. You see, a child can draw a whiteboard architecture, but the real world is a mess. We discuss the hilariously bad decisions that make it into shipping products, the unfortunate hacks the real-world forces us to build, and that the best to call your staging environment is “theory”. Because invariably whatever you’ve built works in the theory, but not in production. Let’s get to it.

This episode is sponsored in part by ParkMyCloud, fellow worshipers at the altar of turned out [BLEEP] off. ParkMyCloud makes it easy for you to ensure you're using public cloud like the utility it's meant to be. just like water and electricity, You pay for most cloud resources when they're turned on, whether or not you're using them. Just like water and electricity, keep them away from the other computers. Use ParkMyCloud to automatically identify and eliminate wasted cloud spend from idle, oversized, and unnecessary resources. It's easy to use and start reducing your cloud bills. get started for free at parkmycloud.com/screaming.

When you're building on a given cloud provider, you're always going to have concerns. If you're building on top of Azure, for example, you're worried your licenses might lapse. If you're building on top of GCP, you're terrified that they're going to deprecate all of GCP before you get your application out the door. If you're building on Oracle Cloud, you're terrified, they'll figure out where you live and send a squadron of attorneys to sue you just on general principle. And if you build on AWS, you're constantly living in fear, at least in a personal account, that they're going to surprise you with a bill that's monstrous.

Today, I want to talk about a particular failure that a friend of this podcast named Chris Short experienced. Chris is not exactly a rank neophyte to the world of Cloud. He currently works at IBM Hat, which I'm told is the post-merger name. He was deep in the Ansible community. He's a Cloud Native Computing Foundation Ambassador, which means that every third word out of his mouth is now contractually obligated to be Kubernetes.

He was building out a static website hosting environment in his AWS account, and it was costing him between $10 and $30 a month. That is right aligned with what I tend to cost. And he wound up getting his bill at the end of the month: “Welcome to July, time to get your bill,” and it was a bit higher. Instead of $30, or even $40 a month, it was $2700. And now there was actual poop found in his pants.

This is a trivial amount of money to most companies, even a small company, and I say this from personal experience, runs on burning piles of money. However, a personal account is a very different thing. This is more than most people's mortgage payments if you don't make terrible decisions like I do, and live in San Francisco. This is an awful lot of money, and his immediate response was equivalent to mine. First, he opened a ticket with AWS support, which is an okay thing to do. Then he immediately turned to Twitter, which is the better thing to do because it means that suddenly these stories wind up in the public eye.

I found out roughly 10 seconds later, as my notifications blew up with everyone saying, “Hey, have you met Corey?” Yes, Chris and I know each other. We're friends. He wrote the DevOps’ish newsletter for a long time, and the secret cabal of DevOps-y type newsletters runs deep. We secretly run all kinds of things that aren't the billing system for cloud providers.

So, he hits the batphone. I log into his account once we get a credential exchange going, and I start poking around because, yeah, generally speaking, 100x bill increase isn't typical. And what I found was astonishing. He was effectively only running a static site with S3 in this account making the contents publicly available, which is normal. This is a stated use case for S3, despite the fact that the console is going to shriek it's damn fool head off at you at every opportunity, that you have exposed an S3 bucket to the world.

Well, yes, that is one of its purposes. It is designed to stand there, or sit there depending on what a bucket does—lay there, perhaps—and provide a static website to the world. Now, in a two-day span, someone or something downloaded data from this bucket, which is normal, but it was 30 terabytes of data, which is not. At approximately nine cents a gigabyte, this adds up to something rather substantial, specifically after free tier limits are exhausted, that's right: $2700.

Now, the typical responses to what people should do to avoid bill shocks like this don't actually work. “Well, he should have set up a billing alarm.” Yeah, aspirationally the AWS billing system runs on an eight-hour eventual consistency model, which means that at the time the bill starts spiking. He has at least 8 hours, and in some cases as many as 24 to 48, before those billing alarms would detect. The entire problem took less time than that.

So, at that point, it would be alerting after something had already happened. “Oh, he shouldn't have had the bucket available to the outside world.” Well, as it turns out, he was fronting this bucket with CloudFlare. But what he hadn't done is restrict bucket access to CloudFlare’s endpoints, and for good reason. There's no way to say, “Oh, CloudFlare’s, identity is going to be defined in an IAM managed policy.” He has to explicitly list out all of CloudFlare’s IP ranges, and hope and trust that those IP ranges will never change despite whatever networking enhancements CloudFlare makes, it's a game of guess and check and having to build an automated system around this. Again, all he wanted to do was share a static website. I've done this myself. I continue to do this myself and it costs me, on a busy month, pennies. In some rare cases, dozens of pennies.

Corey: This episode is sponsored in part by ChaosSearch. Now their name isn’t in all caps, so they’re definitely worth talking to. What is ChaosSearch? A scalable log analysis service that lets you add new workloads in minutes, not days or weeks. Click. Boom. Done. ChaosSearch is for y...

Links:

• Summit Route's AWS SCP Best Practices 
• Reported Apache Log4j Hotpatch Issues
• cloudtrail-partioner

AWS Morning Brief for the week of February 3, with Corey Quinn. 

Links:

• Amazon EFS is now available in the AWS Mexico (Central) Region
• Amazon EKS and Amazon EKS Distro now supports Kubernetes version 1.32
• Amazon S3 Metadata is now generally available
• Amazon Timestream for InfluxDb now supports Storage Scaling
• AWS Elastic Beanstalk now supports Python 3.13 on Amazon Linux 2023
• AWS Health now supports Internet Protocol Version 6 (IPv6)
• Announcing AWS Managed Notifications in the AWS Console Mobile App
• Announcing new AWS Wavelength Zone in Casablanca
• AWS now supports Zone Groups for Availability Zones
• Announcing the AWS CDK Glue L2 Construct
• Deploy DeepSeek-R1 Distilled Llama models in Amazon Bedrock
• Announcing upcoming changes to the AWS Security Token Service global endpoint
• Design patterns for multi-tenant access control on Amazon S3
• Amazon Nimble Studio Closed to New Customers

Want to give your ears a break and read this as an article? You’re looking for this link. https://www.lastweekinaws.com/blog/The-Actual-Next-1-Million-Cloud-Customers

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

About Corey Quinn

Over the course of my career, I’ve worn many different hats in the tech world: systems administrator, systems engineer, director of technical operations, and director of DevOps, to name a few. Today, I’m a cloud economist at The Duckbill Group, the author of the weekly Last Week in AWS newsletter, and the host of two podcasts: Screaming in the Cloud and, you guessed it, AWS Morning Brief, which you’re about to listen to.

Links

• CHAOSSEARCH
• Console Recorder on Chrome Web Store
• Console Recorder on Firefox Add-Ons
• Screaming in the Cloud
• Ian Mckay’s GitHub for AWSConsoleRecorder
• Twitter: @QuinnyPig

Transcript

Corey Quinn: Welcome to AWS Morning Brief: Whiteboard Confessional. I’m Cloud Economist Corey Quinn. This weekly show exposes the semi-polite lie that is whiteboard architecture diagrams. You see, a child can draw a whiteboard architecture, but the real world is a mess. We discuss the hilariously bad decisions that make it into shipping products, the unfortunate hacks the real-world forces us to build, and that the best to call your staging environment is “theory” because invariably, whatever you’ve built works in the theory, but not in production. Let’s get to it.

On this show, I talk an awful lot about architectural patterns that are horrifying. Let’s instead talk for a moment about something that isn’t horrifying. CHAOSSEARCH. Architecturally, they do things right. They provide a log analytics solution that separates out your storage from your compute. The data lives inside of your S3 buckets, and you can access it using APIs you’ve come to know and tolerate, through a series of containers that live next to that S3 storage. Rather than replicating massive clusters that you have to care and feed for yourself, instead, you now get to focus on just storing data, treating it like you normally would other S3 data and not replicating it, storing it on expensive disks in triplicate, and fundamentally not having to deal with the pains of running other log analytics infrastructure. Check them out today at CHAOSSEARCH.io.

You’ll notice that I periodically refer to various friends of the show as code terrorists. It’s never been explained, until now, exactly what that means and why I use that term. So, today, I thought I’d go ahead and help shine a little bit of light on that. One of the folks that I refer to most frequently is Ian Mckay, a fine gentleman based out of Australia. And he’s built something that is both amazing and terrible all at the same time, called Console Recorder. But let me back up before I dive into this monstrosity. Let’s talk about how we get things that we build in AWS into production. There are basically four tiers. Tier one is using the AWS web console itself, we click around and we build things. Great. Tier two is we use CloudFormation like sensible folks. Tier three is Terraform with all of its various attendant tooling, and then there’s the ultra tier four that I do, which is we use the AWS console and then we lie about it. Some folks are gonna play around here and say that oh, you should use the CDK, or something like that, or custom scripts that wind up spinning up production. 

And all of those are well and good, but only recently did CloudFormation release the ability to import existing resources. And even then, much like Terraform import, it’s pretty gnarly and not at all terrific. So, what do you wind up generally doing? Well, if you’re like me, you’ll stand up production resources inside of an AWS account. You will click around in the console—I always start with the console, not because I don’t know how these other tools work, that’s a side point, but rather because that helps me get a sense for how these services are imagined by the teams building them. They tend to assume that everyone who interacts with them is going to go through the console at some point, or at least they should. So, it gives me access and exposure to what their vision of this service is. Then once you’ve built something up, it often finds its way into production, if you at all like me, where I’ll spin something up just to test something and it works, and oh my stars, and suddenly you just want to get it out and not worry about it, so you don’t go back and rebuild it properly. 

So, now you’re left with this hand-built thing that’s just flapping around out there in production. What are you supposed to do? Well, according to the AWS experts, if we’re allowed to use that term to describe them, you’re supposed to smack yourself on the forehead, be convinced that you’re fundamentally missing the boat here, throw everything you’ve just built away and go back and do it properly. Which admittedly seems a little bit on the nose for those of us who’ve done exactly this far more times over the course of our career than we would care to count. So, today, however, I posit that there’s an alternate approach that doesn’t require support from AWS, which, to be honest, long ago seems to have given up on solving this particular problem in a way that human beings can understand. And I’d like to tell you about that, after this brief message from our sponsor.

In the late 19th and early 20th centuries, democracy flourished around the world. This was good for most folks, but terrible for the log analytics industry because there was now a severe shortage of princesses to kidnap for ransom to pay for their ridiculous implementations. It doesn’t have to be that way. Consider CHAOSSEARCH. The data lives in your S3 buckets in your AWS accounts, and we know what that costs. You don’t have to deal with running massive piles of infrastructure to be able to query that log data with APIs you’ve come to know and tolerate, and they’re just good people to work with. Reach out to CHAOSSEARCH.io. And my thanks to them for sponsoring this incredibly depressing podcast. 

This brings us back to where we started this conversation, with defining what a code terrorist was and pointing out Ian Mckay, out of Australia, who’s built something absolutely terrifying called Console Recorder for AWS. And this is a browser extension that works in Chrome, it works in Firefox, possibly others I stopped looking after those two. And what it does is you click a button in your browser when you start doing something inside the AWS console, and it does exactly what it says it would on the tin, where it starts recording what you’re doing. Their icon turns to a bright red record symbol, and you do whatever it is you’re going to do to spin something up. When you’re done, you hit the button again and say stop recording, and it opens a brand new tab. 

...

Links:

• Nick Jones' review of the AWS Security Model I linked to previously.
• Microsoft Azure has seen 6 'nightmare' cloud security flaws over the past year. 
• Unsecured Elasticsearch Data Replaced with Ransom Note
• AWS Systems Manager announces support for port forwarding to remote hosts using Session Manager 
• When and where to use IAM permissions boundaries 
• Security vulnerability in AWS's Managed Workflows for Apache Airflow

Want to give your ears a break and read this as an article? You’re looking for this link.

Sponsors

• nOps
• Linode

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

Links:

• Last Week in AWS Community Slack
• VPC Lattice
• AWS Supply Chain
• OpenSearch Serverless
• AWS Verified Access

Stay Up To Date with re:Quinnvent

• Sign up for the re:Quinnvent Newsletter
• Check out the re:Quinnvent playlist on YouTube
• If you’re on site:
  • Join Corey for a Nature Walk through the Expo Hall beginning at the Fortinet booth today (11/29/22) at 1pm PST or 
  • For drinks at Atomic Liquors tonight at 8:15 pm PST.
  • Tomorrow evening is re:Play, if you see Corey there, please say hello!

Help the show

• Share your feedback
• Subscribe wherever you get your podcasts
• Buy our merch

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

AWS Morning Brief for the week of February 24, with Corey Quinn. 

Links:

• Amazon ECS increases the CPU limit for ECS tasks to 192 vCPUs
• Amazon Q Developer now supports upgrade to Java 21
• AWS announces Backup Payment Methods for invoices
• AWS CodePipeline adds native Amazon EKS deployment support
• AWS Price List API supports AWS PrivateLink
• AWS CloudFormation: 2024 Year in Review
• Cost optimize your Minecraft Java EC2 Server
• Improving Security in Amazon WorkMail with MFA
• Update on Support for Amazon Chime
• Best practices to respond to security risks across your AWS Organizations
• Reduce IT costs by implementing automatic shutdown for Amazon EC2 instances
• How to restrict Amazon S3 bucket access to a specific IAM role
• Introducing the AWS Trust Center
• Is AWS Delivering on Its 3-Layer Approach to AI?

AWS Morning Brief Extras edition for the week of November 8, 2023.

Links

• AWS has been making a lot of noise about generative AI
• AWS's Responsible AI page
• AWS's Service Terms
• Amazon's updated management policy language
• Opt-Out of Amazon AI
• Opt-Out Terraform module
• Opt-Out Python script

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

Buy our merch 

• https://store.lastweekinaws.com

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

AWS Morning Brief for the week of October 16, 2023 with Corey Quinn.
 
Links:

• New Amazon CloudWatch metric monitors EC2 instance reachability to EBS volumes
• Announcing AWS Lambda’s support for Internet Protocol Version 6 (IPv6) for outbound connections in VPC 
• Announcing new AWS Network Load Balancer (NLB) availability and performance capabilities 
• Two billion downloads of Terraform AWS Provider shows value of IaC for infrastructure management
• Why purpose-built artificial intelligence chips may be key to your generative AI strategy
• How Zalando migrated their shopping carts to Amazon DynamoDB from Apache Cassandra 
• Unlocking cost-optimization: Open Raven’s journey with Amazon Aurora I/O-Optimized leads to 60% savings
• How does Cloud enable the transformation of Bank finance functions? 
• AWS Cloud Institute: Virtual training program for cloud developers

AWS Morning Brief for the week of Monday, June 10th, with Corey Quinn. 

Links:

• Introducing Amazon EC2 High Memory U7i Instances
• Amazon API Gateway integration timeout limit increase beyond 29 seconds
• Amazon CloudWatch GetMetricData API now supports AWS CloudTrail data event logging
• Amazon CloudWatch Logs announces Live Tail streaming CLI support
• Amazon CodeCatalyst now supports GitHub Cloud source code with blueprints
• Amazon EC2 instance type finder capability is generally available in AWS Console
• AWS Transfer Family increases message size and throughput limits for AS2
• Integrate AWS Cost Anomaly Detection Notifications with IT Service Management Workflow – Part1 Jira
• Introducing CloudFront Hosting Toolkit

Transcript

Corey: This episode is sponsored in part byLaunchDarkly. Take a look at what it takes to get your code into production. I’m going to just guess that it’s awful because it’s always awful. No one loves their deployment process. What if launching new features didn’t require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren’t what you expect? LaunchDarkly does exactly this. To learn more, visitlaunchdarkly.com and tell them Corey sent you, and watch for the wince.

Jesse: Hello, and welcome to AWS Morning Brief: Fridays From the Field. I’m Jesse DeRose.

Amy: I’m Amy Negrette.

Jesse: This is the podcast within a podcast where we talk about all the ways that we have seen AWS used and abused in the wild, with a healthy dose of complaining about AWS for good measure. I feel like it’s just kind of always necessary. There always has to be just that little bit of something extra; it’s the spice that really makes the dish. Today we’re going to be talking about the ‘Cloud Cost Management Team Starter Kit.’ Now, in a previous episode, we talked about the ‘Cloud Cost Management Starter Kit,’ which was a little bit more generalized, and one of the things that we talked about, ultimately, was building a team that is responsible for some of this work, some of this cloud cost management work.

So today, we’re going to take that one step further; we’re going to talk about all of the things that your cloud cost management team should ultimately be responsible for, what it should look like, how you might want to start building that team within your organization. So, I’m going to kick us off. I think one of the first things that is so, so critical for any team that is going to be doing any work is buy-in at the executive leadership level. You need to make sure that engineering leadership, the C-suite leadership has your back in everything that you’re doing. You need to make sure that the work that you’re doing has been signed off at the highest level so that that leadership can help empower you to do your work.

Amy: And we’ve referenced this before, and really, every time we talk about things like what makes a successful project is that as the one executing that project, you probably need the authority and actionable goals in order to do that, and the leadership is going to be the ones to lay that out for you.

Jesse: Absolutely. If you don’t have the backing of leadership, whether it is your boss, whether it is the C-suite, whether it’s a VP suite, you’re not going to get other people to listen to what you have to say; you’re not going to get other people to, broadly speaking, generally speaking, care about the work that you’re trying to do, the work that you’re trying to incentivize and empower other people in the organization to do.

Amy: And that kind of leads us into the next portion of it where you need to know what the responsibilities are and have that clear delineation so that you understand the things that is expected of you, what the engineering teams, what they’re expected to do, and product teams, and finance teams. Everyone has to have a pretty much fenced-in idea of what they’re allowed to do and what they are expected to deliver, just like in any project.

Jesse: Absolutely. It’s so critical for me to understand what I’m responsible for, you to understand what you’re responsible for. I can’t tell you how many times I’ve been in a meeting where somebody will say something generally like, “We should do X,” and then everyone nods and goes, “Oh, yeah, yeah, yeah. We should do X.” And then everybody leaves the meeting and thinks that somebody else is responsible for it, and nobody’s been clearly assigned that work, or nobody knows that work is ultimately their responsibility.

Amy: And if you don’t assign it, people are going to assume that this is going to be a thing that if they have time to, they’ll get to it. And we harp on it enough that whenever work is not prioritized, it is automatically deprioritized. That’s just the way task lists shake out, especially at the end of sprint meetings.

Jesse: Absolutely. And I think that’s one of the other things that’s so important, too, is that it’s not just about assigning the work, but it’s about making sure that everybody who is involved in the conversation, everybody who’s involved in the work agrees on what those boundaries are, agrees on who is responsible for what actions, more specifically speaking from a task responsibility perspective. Because at the end of the day, I want my team, whether that is my individual team or a cross-functional team, to all be bought into who’s responsible for what parts of the project. We all need to be on the same page in terms of, “Yes, this is my responsibility. This part of the work is my responsibility. I will take ownership over this,” so that we can all help each other.

Get that project goal together. One of the other big ideas that is so critical to starting a cloud cost management team is identifying and socializing your business KPI metrics. Now, this is something that some engineering teams already think about day-to-day. They might have ideas of service-level agreements, metrics, maybe service-level objective metrics, but there might be other business metrics that indirectly—or directly—relate to engineering work. It could be number of users using your SaaS platform, it could be number of API requests, it could be the amount of storage that customers are storing on your platform. You want to identify what these metrics are, and start measuring your cloud spend against these metrics.

Amy: And as far as cost optimization projects go, the KPIs may not line up directly against how many servers you’re standing up, or how many users are coming through. They’ll be very indicative because you are spending money per user and per resource, but perhaps your business goals are different. Maybe you’re not looking at trying to save money, but better understand where that money is going.

Jesse: Absolutely. It’s not just about how many instances are running per hour, it’s not just about how many servers are running per hour, or how many users per server. It’s really about understanding what are the core driving indicators of your business? What are the things that ultimately influence and impact how your workloads, and servers, and API functions, and everything, flow and grow and change over time?

Amy: These metrics also can be influenced by things that are not architecturally specific, like savings plans, or the saving you would get through reservations, or some other contractual deal you get from your provider.

Jesse: Yeah, that’s one of the hard things, too, that we always hear from our clients. There is this idea that they think that they are spending a certain amount of money because they’re getting discounts from savings plans, or from reserved instances or from an enterprise discount program, and maybe their usage is a lot higher than that, but because they get these discounts, they think that they’re actually using a lot less than they actually are. And while this is not something we’re talking about specifically or directly in this conversation, it is something to be mindful of because there definitely can be a difference between your usage and your overall spend if your company is investing in thin...

Links:

• Tiered storage for MSK
• Lambda telemetry API
• Resource Explorer Launched
• GP3 comes to RDS
• Amazon Time Sync is now available as a public NTP service
• Zurich region
• Spain Region
• Hyderabad Region
• Faster glacier restores
• multiple MFA devices
• Finch
• AWS Fault Isolation Boundaries whitepaper

Stay Up To Date with re:Quinnvent

• Sign up for the re:Quinnvent Newsletter
• Check out the re:Quinnvent playlist on YouTube

Help the show

• Share your feedback
• Subscribe wherever you get your podcasts
• Buy our merch

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

Transcript

Corey: This episode is sponsored in part byLaunchDarkly. Take a look at what it takes to get your code into production. I’m going to just guess that it’s awful because it’s always awful. No one loves their deployment process. What if launching new features didn’t require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren’t what you expect? LaunchDarkly does exactly this. To learn more, visitlaunchdarkly.com and tell them Corey sent you, and watch for the wince.

Jesse: Hello, and welcome to AWS Morning Brief: Fridays From the Field. I’m Jesse DeRose.

Amy: I’m Amy Negrette.

Tim: And I’m Tim Banks.

Jesse: This is the podcast within a podcast where we talk about all the ways we’ve seen AWS used and abused in the wild with a healthy dose of complaining about AWS for good measure. Today, we’re going to be talking about a recent addition to the AWS family: AWS Application Cost Profiler.

Tim: But hold on for a second, Jesse, because AWS Application Cost Profiler we can get to; that’s rather unremarkable. I really want to talk about how impressed I am with AWS InfiniDash. I’ve been benchmarking this thing, and it is fan… tastic. It’s so good. And we could probably talk about for a while, but suffice to say that I am far more impressed with AWS InfiniDash than I am with AWS Application Cost Profiler.

Jesse: You know, that’s fair. And I feel like InfiniDash should absolutely get credit where credit is due. I want to make sure that everybody can really understand the full breadth of everything that InfiniDash is able to accomplish. So, I want to make sure that we do get to that; maybe in a future episode, we can touch on that one. But for right now, I have lots of feelings about AWS Application Cost Profiler, and what better place to share those feelings than with two of my favorite people, Amy and Tim, and then all of you listeners who are listening in to this podcast. I can’t wait to dive into this. But I think we should probably start with, what is AWS Application Cost Profiler?

Amy: It is [unintelligible 00:01:54] in a trench coat.

Jesse: [laugh].

Amy: Which is the way AWS likes to solve problems sometimes. And in this case, it’s talking about separating billing costs by tenants by service, which is certainly a lot of things that people have problems with.

Jesse: That is a lot of buzzwords.

Amy: A lot of words there.

Jesse: Yeah. Looking at the documentation, the sales page, “AWS Application Cost Profiler is a managed service that helps us separate your AWS billing and costs by the tenants of your service.” That has a lot of buzzwords.

Tim: Well, to be fair, that’s also a majority of the documentation about service.

Jesse: Yeah, that is fair. That is a lot of what we saw, and I think we’ll dive into that with documentation in a minute. But I do want to call out before we dive into our thoughts on this service—because we did kick the tires on this service and we want to share what our experience was like, but I do want to call out that this problem that AWS Application Cost Profiler is trying to solve. This idea of cost allocation of shared resources, it is a real, valid problem and it is one that is difficult to solve.

Amy: And we’ve had clients that have had this very explicit problem and our findings have been that it’s very difficult to accurately splice usage and spend against what’s essentially consumption-based metrics—which is how much a user or request is using all the way along your pipeline—if they’re not using dedicated resources.

Jesse: Yeah, when we talk about cost allocation, generally speaking, we talk about cost allocation from the perspective of tagging resources, broadly speaking, and moving resources into linked accounts and separating spend by linked accounts, or allocating spend by linked accounts. But if you’ve got a shared compute cluster, a shared database, any kind of shared resources where multiple tenants are using that infrastructure, slapping one tag on it isn’t going to solve the issue. Even putting all of those shared resources in a single linked account isn’t going to solve that issue. So, the problem of cost allocation for shared resource is real; it is a valid problem. So, let’s talk specifically about AWS Application Cost Profiler as a solution for this problem. As I mentioned, we kicked the tires on this solution earlier this week and we have some thoughts to share.

Tim: I think one of the main things around this AWS Application Profiler like I said, there’s some problems that can be solved there, there’s some insights that people really want to gain here, but the problem is people don’t want to do a lot more work or rewrite their observability stack to do it. The problem is, that’s exactly what AWS Cost Profiler seems to be doing or seems to want you to do. It doesn’t get data from, I think it only gets data from certain EC2 services, and it’s just, it’s doing things that you can already do in other tools to do aggregation. And if I’m going to do all the work to rewrite that stack, to be able to use the Profiler, am I going to want to spend that time doing something else? I mean, that kind of comes to the bottom line about it.

Jesse: Yeah, the biggest thing that I ran into, or that I experienced when we were setting up the Cost Profiler, is that documentation basically said, “Okay, configure Cost Profiler and then submit your data.” And [unintelligible 00:05:54] stop, like wait, what? Wait, what do you mean, ‘submit data?’ And it said, “Okay, well now that you’ve got Cost Profiler as a service running, you need to upload all of the data that Cost Profiler is going to profile for you.” It boggles my mind.

Tim: And it has to be in this format, and it has to have these specific fields. And so if you’re not already emitting data in that format with those fields, now you have to go back and do that. And it’s not really solving any problems, but it offers to create more problems.

Amy: And also, if you’re going to have to go through the work of instrumenting and managing all that data anyway, you could send it anywhere you wanted to. You could send it to your own database to your own visualization. You don’t need Profiler after that.

Jesse: Yeah, I think that’s a really good point, Amy. AWS Cost Profiler assumes that you already have this data somewhere. And if not, it explicitly says—in its documentation it says, to generate reports you need to submit tenant usage data of your software applications that use shared AWS resources. So, it explicitly expects you to already have this data. And if you are going to be looking for a solution that is going to help you allocate the cost of shared resources and you already have this data somewhere else, there are better solutions out there than AWS Application Cost Profiler. As Amy said, you can send that data anywhere. AWS Application Cost Profiler probably isn’t going to be the first place that you think of because it probably doesn’t have as many features as other solutions.

Amy: If you were going to instrument things to that level, and let’s say you were using third-party services, you could normalize your...

AWS Morning Brief Extras edition for the week of February 22, 2023.

Want to give your ears a break and read this as an article? You’re looking for this link.https://www.lastweekinaws.com/blog/amazons-snowball-edge-frustrates-this-user

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

Buy our merch 

• https://store.lastweekinaws.com

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

Links:

• Infosys leaked FullAdminAccess AWS keys on PyPi for over a year.
• Rackspace has suffered a ransomware attack 
• AWS Security Hub now integrates with AWS Control Tower
• AWS Verified Access Preview — VPN-less Secure Network Access to Corporate Applications
• The Open Source Security Index 

Want to give your ears a break and read this as an article? You’re looking for this link.

Sponsors

• New Relic
• ExtraHop 

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

Last week in security news: Accelerating development with AWS CDK plugin – CfnGuardValidator, This week's S3 Bucket Negligence Award is brought to you by PwC Nigeria, The volkswagen open source tool, and more!

Links:

• Last week I talked about AWS Management Console Access incorrectly. My thanks to Timothy Ingalls on the Last Week in AWS community Slack for flagging this for me. Gold star for you!
• This week's S3 Bucket Negligence Award is brought to you by PwC Nigeria.
• FusionAuth has a great dive into their annual SOC 2 vendor selection process.
• My beloved Retool has a post talking about how an MFA failure mode led to a small number of customers being exposed. 
• Accelerating development with AWS CDK plugin – CfnGuardValidator 
• How to implement cryptographic modules to secure private keys used with IAM Roles Anywhere 
• Tool of the week: The volkswagen open source tool detects when your tests are being run in a CI server, and makes them pass.

Links:

• Amazon Chime announces new mobile apps with features to improve your meeting experience
• Amazon Detective improves search by supporting case insensitivity 
• AWS Activate is now open to all startups 
• AWS CloudFormation StackSets increases limits on three service quotas
• AWS IQ now supports partners and independent consultants in Australia, Europe, Japan, and other regions
• Announcing a new Cost Explorer console experience 
• Omdia study: how the media and entertainment industry uses cloud marketplace solutions
• Best Practices for Hosting Regulated Gaming Workloads in AWS Local Zones and on AWS Outposts
• Reducing AWS Fargate Startup Times with zstd Compressed Container Images
• Managing your Game Studio on AWS part 2
• Netflix innovates and entertains the world, powered by AWS 
• How to use AWS Config and CloudTrail to find who made changes to a resource 
• Introducing AWS Global Accelerator IPv6 
• Canary Testing with AWS App Mesh and Tekton
• The economic impact of AWS’s investment in Japan 
• Goldman Sachs and AWS examine efficient ways to load data into quantum computers 
• The importance of a mentor in your cloud learning journey 

Last week in security news: Delegating permission set management and account assignment in AWS IAM Identity Center, How AWS protects customers from DDoS events, The Tip of the Week, and more!

Links:

• Rhino Security has a two part post that talks about how they find Cognito misconfigurations
• Delegating permission set management and account assignment in AWS IAM Identity Center 
• How AWS protects customers from DDoS events 
• Now available: Building a scalable vulnerability management program on AWS
• Issue with Amazon WorkSpaces Windows Client Version 5.9 and 5.10
• The Tip of the Week is to delegate as much as humanly possible to accounts that aren't the management account for the org. As more services gain the ability to delegate management to designated accounts, it gets easier to restrict access to it

AWS Morning Brief for the week of March 6, 2023 with Corey Quinn.

Links:

• Amazon Aurora Serverless v1 now supports customer configurable maintenance windows
• Amazon CloudWatch Internet Monitor is now generally available
• AWS Lambda Powertools for .NET is now generally available
• Amazon Neptune Serverless now scales down to 1 NCU to save costs 
• AWS Control Tower announces a progress tracker for landing zone setup and upgrades
• In the Works – AWS Region in Malaysia 
• New – Amazon Lightsail for Research with All-in-One Research Environments 
• Announcing Amazon ECS Task Definition Deletion 
• Announcing the end of Windows Installer support for AWS Tools for Windows
• “Avatar: The Way of Water” and the future of filmmaking 
• A detailed overview of Trusted Advisor Organizational Dashboard 

AWS Morning Brief for the week of February 10, with Corey Quinn. 

Links:

• Amazon EBS now supports additional resource-level permissions for creating EBS volumes from snapshots
• Amazon Managed Service for Prometheus collector adds support for cross-account ingestion
• Amazon Redshift Concurrency Scaling is now available in 1 additional region
• Amazon Q Developer now troubleshoots AWS Console errors in all AWS Commercial regions
• Amazon Q Developer introduces a new, simplified setup experience for Pro tier subscriptions
• AWS IAM Identity Center now offers improved error messages and AWS CloudTrail logging for provisioning issues
• AWS Step Functions now supports 100,000 state machines and activities per AWS account
• Cost Optimization Hub supports more EC2 Auto Scaling group recommendations
• Announcing the general availability of AWS Database Migration Service Serverless support for files with an S3 source endpoint
• AI-Powered Football Match Analysis: SAP Sports One on AWS
• Introducing AWS CloudFormation Stack Refactoring
• AWS Tightens the Reins: New AWS SaaS Marketplace Rules Will Impact Your Commitments

Last week in security news: AWS IAM Identity Center session duration limit increases from 7 to 90 days, Access accounts with AWS Management Console PrivatAccess, A dive through using Amazon Athena in Incident Response, and more!

Links:

• This is an esoteric Firefox/Yubikey compatibility bug that I went blindly stumbling into and has been resolved.
• Chris Farris has a post up about deploying AWS Backup. 
• In preparation for re:Invent, the MGM had a massive cybersecurity issue
• Amazon EC2 now supports Block Public Access for Amazon Machine Images
• AWS IAM Identity Center session duration limit increases from 7 to 90 days
• AWS Identity and Access Management provides action last accessed information for more than 140 services 
• Access accounts with AWS Management Console Private Access 
• A dive through using Amazon Athena in Incident Response. This is important! 
• Corey will be hosting an AMA on 9/27 @ noon PDT over on Twitch. Bring questions!

AWS Morning Brief for the week of Monday, July 29th, with Mike Julian.

Links:

• Amazon VPC IPAM now supports BYOIP for IPs registered with any Internet Registry
• AWS Cost Categories now supports “Billing Entity” dimension 
• Enhance database performance with Amazon RDS dedicated log volumes
• QLDB deprecation--more on the way?

Want to give your ears a break and read this as an article? You’re looking for this link.

https://www.lastweekinaws.com/blog/an-aws-free-tier-bill-shock-your-next-steps

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

Want to give your ears a break and read this as an article? You’re looking for this link. https://www.lastweekinaws.com/blog/New-CEO-Onboarding-at-AWS

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

AWS Morning Brief for November 30th, 2023 with Corey Quinn.

Stay Up To Date with re:Quinnvent

• Sign up for the re:Quinnvent Newsletter
• Join the re:Quinnvent Scavenger Hunt
• Check out the re:Quinnvent playlist on YouTube

Help the show

• Share your feedback
• Follow wherever you get your podcasts
• Buy our merch

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

AWS Morning Brief for the week of October 2, 2023, with Corey Quinn.

Links:

• Amazon Chime adds Waiting Room capability to further secure your meetings
• Amazon CloudFront announces security recommendations 
• Amazon DocumentDB (with MongoDB compatibility) supports in-place major version upgrade
• Amazon EC2 Serial Console now available in additional AWS Regions 
• AWS Application Composer now supports all 1000+ AWS CloudFormation resources
• DynamoDB global tables is now available in all AWS Regions 
• Announcing incremental export to S3 for Amazon DynamoDB
• Amazon Bedrock Is Now Generally Available – Build and Scale Generative AI Applications with Foundation Models 
• How to import existing resources into AWS CDK Stacks
• Introducing dual-stack and IPv6-only support for Amazon Route 53 Resolver Endpoints

Want to give your ears a break and read this as an article? You’re looking for this link.

https://www.lastweekinaws.com/blog/shitposting-as-a-learning-style

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

Want to give your ears a break and read this as an article? You’re looking for this link.

https://www.lastweekinaws.com/blog/s3-encryption-at-rest-does-not-solve-for-bucket-negligence/

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

Buy our merch 

• https://store.lastweekinaws.com

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

Links:

• Join Corey in Phoenix next Sunday at 1PM at Zuzu for a community meet-up.
• Rackspace continues to trickle the truth out; it's now admitting that attackers accessed customer data 
• Tom Forbes scanned--wait, holy hell, he scanned every package on PyPi and found 57 live AWS keys. 
• In one year we're going to come back and see how accurate the heads of AWS security are with their predictions for cybersecurity in 2023
• Today's tip of the week is to go fire up your important AWS account(s) and validate that the root user doesn't have API credentials assigned.

AWS Morning Brief for the week of June 26, 2023 with Corey Quinn. 

Links:

• The FTC comment period about the business of cloud computing ended
• Amazon warehouse practices are now the focus of a senate probe
• The FTC is suing Amazon for its Prime enrollment dark patterns
• Amazon’s iRobot acquisition is now the subject of an EU investigation
• The launch of Amazon Clinic is being delayed after the senate asked some hard questions
• Announcing Amazon EC2 Hpc7g instances 
• AWS Lambda supports starting from timestamp for Kafka event sources
• AWS Step Functions launches Versions and Aliases 
• AWS Transfer Family announces structured JSON log format
• 5 Stages to Building a Successful Partner Practice with AWS
• Say Hello to 176 AWS Competency, Service Delivery, Service Ready, and MSP Partners Added or Renewed in May
• How GoDaddy Implemented a Multi-Region Event-Driven Platform at Scale
• New Amazon EC2 C7gn Instances: Graviton3E Processors and Up To 200 Gbps Network Bandwidth
• For actual technical depth, my thanks to David Cuthbert in the Last Week in AWS Slack Community for surfacing this AnandTech article.
• Stream VPC Flow Logs to Datadog via Amazon Kinesis Data Firehose
• Creating real-time flood alerts with the cloud
• Use AWS Private Certificate Authority to issue device attestation certificates for Matter
• Should I use the hosted UI or create a custom UI in Amazon Cognito? - Trick question, you should use recurring Last Week in AWS sponsor FusionAuth instead. 
• Coming soon: updates to AWS Certified Cloud Practitioner exam
• How I achieved all six specialty AWS Certifications on first attempt
• How to win a $5 Amazon Gift Card, just by signing up for the Amazon News newsletter 

About Corey Quinn

Over the course of my career, I’ve worn many different hats in the tech world: systems administrator, systems engineer, director of technical operations, and director of DevOps, to name a few. Today, I’m a cloud economist at The Duckbill Group, the author of the weekly Last Week in AWS newsletter, and the host of two podcasts: Screaming in the Cloud and, you guessed it, AWS Morning Brief, which you’re about to listen to.

Links

• CHAOSSEARCH
• Amazon Elastic File System
• Network File System
• AWS Fargate

Transcript

Corey: Welcome to AWS Morning Brief: Whiteboard Confessional. I’m Cloud Economist Corey Quinn. This weekly show exposes the semi-polite lie that is whiteboard architecture diagrams. You see, a child can draw a whiteboard architecture, but the real world is a mess. We discuss the hilariously bad decisions that make it into shipping products, the unfortunate hacks the real-world forces us to build, and that the best to call your staging environment is “theory”. Because invariably whatever you’ve built works in the theory, but not in production. Let’s get to it.

Corey: On this show, I talk an awful lot about architectural patterns that are horrifying. Let’s instead talk for a moment about something that isn’t horrifying. CHAOSSEARCH. Architecturally, they do things right. They provide a log analytics solution that separates out your storage from your compute. The data lives inside of your S3 buckets, and you can access it using APIs you’ve come to know and tolerate, through a series of containers that live next to that S3 storage. Rather than replicating massive clusters that you have to care and feed for yourself, instead, you now get to focus on just storing data, treating it like you normally would other S3 data and not replicating it, storing it on expensive disks in triplicate, and fundamentally not having to deal with the pains of running other log analytics infrastructure. Check them out today at CHAOSSEARCH.io.

I talked a lot about databases on this show. There are a bunch of reasons for that, but they mostly all distill down to that databases are, and please don't quote me on this as I'm not a DBA, where the data lives. If I blow up a web server, it can have hilarious consequences for a few minutes, but it's extremely unlikely to have the potential to do too much damage to the business. That's the nature of stateless things. They're easily replaced, and it's why the infrastructure world has focused so much on the recurring mantra of cattle, not pets.

But I digress. This episode is not about mantras. It's about databases. Today's episode of the AWS Morning Brief: Whiteboard Confessional returns to the database world with a story that's now safely far enough in the past that I can talk about it without risking a lawsuit. We were running a fairly standard three-tiered web app. For those who haven't had the pleasure because their brains are being eaten by the microservices worms, these three tiers are web servers, application servers, and database servers. It's a model that my father used to deploy, and his father before him.

But I digress. This story isn't about my family tree. It's about databases. We were trying to scale, which is itself a challenge, and scale is very much its own world. It's the cause of an awful lot of truly terrifying things. You can build an application that does a lot for you on your own laptop. But now try scaling that application to 200 million people. Every single point of your application architecture becomes a bottleneck long before you'll get anywhere near that scale, and you're gonna have oodles of fun re-architecting it as you go. Twitter very publicly went through something remarkably similar about a decade or so ago, the fail whale was their error page when Twitter had issues, and everyone was very well acquainted with it. It spawned early memes and whatnot. Today, they've solved those problems almost entirely.

But I digress. This episode isn't about scale, and it's not about Twitter. It's about databases. So my boss walks in and as we're trying to figure out how to scale a MySQL server for one reason or another, and then casually suggests that we run the database on top of NFS.

[Record Scratch]

Yes, I said NFS. That's Network File System. Or, if you've never had the pleasure, the protocol that underlies AWS’s EFS offerings, or Elastic File System. Fun trivia story there, I got myself into trouble, back when EFS first launched, with Wayne Duso, AWS’s GM of EFS, among other things, by saying that EFS was awful. At launch, EFS did have some rough edges, but in the intervening time, they've been fixed to the point where my only remaining significant gripe about EFS is that it's NFS. Because today, I mostly view NFS is something to be avoided for greenfield designs, but you've got to be able to support it for legacy things that are expecting it to be there. There is, by the way, a notable EFS exception for Fargate and using NFS with Fargate for persistent storage.

But I digress. This episode isn't about Fargate. It's about databases.

Corey: In the late 19th and early 20th centuries, democracy flourished around the world. This was good for most folks, but terrible for the log analytics industry because there was now a severe shortage of princesses to kidnap for ransom to pay for their ridiculous implementations. It doesn’t have to be that way. Consider CHAOSSEARCH. The data lives in your S3 buckets in your AWS accounts, and we know what that costs. You don’t have to deal with running massive piles of infrastructure to be able to query that log data with APIs you’ve come to know and tolerate, and they’re just good people to work with. Reach out to CHAOSSEARCH.io. And my thanks to them for sponsoring this incredibly depressing podcast. 

So I'm standing there, jaw agape at my boss. This wasn't one of those many mediocre managers I've had in the past that I've referenced here. He was and remains the best boss I've ever had. Empathy and great people management skills aside, he was also technically brilliant. He didn't suggest patently ridiculous things all that often, so it was sad to watch his cognitive abilities declining before our eyes. “Now, hang on,” he said, “before you think that I've completely lost it. We did something exactly like this before at my old job, it can be done safely, sanely and offer great performance benefits.” So, I'm going to skip what happens next in this story because I was very early in my career. I hadn't yet figured out that it's better to not actively insult your boss in a team meeting, based only upon a half baked understanding of what they've just proposed. To his credit, he took it in stride, and then explained how to pull off something that sounds on its face to be truly monstrous.

Now I've doubtless forgotten most of the technical nuance here, preferring ...

• Corey’s livetweet: https://twitter.com/quinnypig
• Eric Hammond’s old article: https://alestic.com/2014/09/aws-root-password/
• Lightspin found a vulnerability: https://blog.lightspin.io/aws-rds-critical-security-vulnerability
• Expel’s incident report: https://expel.com/blog/incident-report-from-cli-to-console-chasing-an-attacker-in-aws/
• Rhino Security Labs found a CVE in the AWS VPN Client: https://rhinosecuritylabs.com/aws/cve-2022-25165-aws-vpn-client/
• DarkReading’s profile of AJ Yawn: https://www.darkreading.com/edge-articles/bytechek-founder-aj-yawn-brings-discipline-to-everything-he-does
• NotGitBleed: https://www.notgitbleed.com/
• AWS Security Bulletins:
•  https://aws.amazon.com/security/security-bulletins/AWS-2022-005/
• https://aws.amazon.com/security/security-bulletins/AWS-2022-004/
• gimme-aws-creds: https://github.com/Nike-Inc/gimme-aws-creds
• Chamber: https://github.com/segmentio/chamber
• #lastweekinaws slack channel: https://og-aws-slack.lexikon.io/

Want to give your ears a break and read this as an article? You’re looking for this link. https://www.lastweekinaws.com/blog/a-brief-history-of-kubernetes-its-use-cases-and-its-problems

Want to watch the full dramatic reenactment of this podcast? Watch the YouTube Video here: https://youtu.be/StlZwvsq9tc

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

Buy our merch 

• https://store.lastweekinaws.com

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

Want to give your ears a break and read this as an article? You’re looking for this link.

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

Links

• Follow Last Week In AWS on Twitter
• AWS Outage Message
• "Kinesis Outage" by Ryan Frantz

Transcript
Corey: This episode is sponsored in part by our friends at Linode. You might be familiar with Linode; they’ve been around for almost 20 years. They offer Cloud in a way that makes sense rather than a way that is actively ridiculous by trying to throw everything at a wall and see what sticks. Their pricing winds up being a lot more transparent—not to mention lower—their performance kicks the crap out of most other things in this space, and—my personal favorite—whenever you call them for support, you’ll get a human who’s empowered to fix whatever it is that’s giving you trouble. Visit linode.com/screaminginthecloud to learn more, and get $100 in credit to kick the tires. That’s linode.com/screaminginthecloud.

Pete: Hello, everyone. Welcome to the AWS Morning Brief. It's Pete Cheslock again—

Jesse: And Jesse DeRose.

Pete: We are back to talk about ‘The Kinesis Outage.’

Jesse: [singing] bom bom bum.

Pete: So, at this point, as you're listening to this, it's been a couple of weeks since the Kinesis outage has happened, and I'm sure there are many, many armchair sysadmins out there speculating at all the reasons why Amazon should not have had this outage. And guess what? You have two more system administrators here to armchair quarterback this as well.

Jesse: We are happy to discuss what happened, why it happened. I will try to put on my best announcer voice, but I think I normally fall more into the golf announcer voice than the football announcer voice, so I'm not really sure if that's going to play as well into our story here.

Pete: It's going, it's going, it's gone.

Jesse: It’s—and it's just down. It's down—

Pete: It's just—

Jesse: —and it's gone.

Pete: No, but seriously, we're not critiquing it. That is not the purpose of this talk today. We're not critiquing the outage because you should never critique other people's outages; never throw shade at another person's outage. That's not only crazy to do because you have no context into their world. It's just, it's not nice either, so just try to be nice out there.

Jesse: Yeah, nobody wants to get critiqued when their company has an outage and when they're under pressure to fix something. So, we're not here to do that. We don't want to point any fingers. We're not blaming anyone. We just want to talk about what happened because honestly, it's a fascinating, complex conversation.

Pete: It is so fascinating and honestly, loved the detail, a far cry from the early years of Amazon outages that were just, “We had a small percentage of instances have some issues.” This was very detailed. This gave out a lot of information. And the other thing too is that, when it comes to critiquing outages, you have to imagine that there are unlikely to be more than a handful of people even inside Amazon Web Services that fully understand the scope of the size and the interactions of all these different services. There may not even be a single person who truly understands how these dozens of services interact with each other. 

I mean, it takes teams and teams of people working together to build these things and to have these understandings. So, that being said, let's dive in. So, the Wednesday before Thanksgiving, Kinesis decided to take off early. You know, long weekend coming up, right? But really, what happened was is that there was an addition of capacity to Kinesis, and it caused it to hit an operating system limit causing an outage. 

But interestingly enough—and what we'll talk about today—are the interesting and downstream effects that occurred via CloudWatch, Cognito, even the status page, and the Personal Health Dashboard. I mean, that's a really interesting contributing factor or a correlating outage. I don't know the words here, but it's interesting to hear that both CloudWatch goes down and the Personal Health Dashboard goes down.

Jesse: That's when somebody from the product side says, “Oh, that's a feature, definitely not a bug.”

Pete: But the outage to CloudWatch then even affected some of the downstream services to CloudWatch—such as Lambda—which also included auto-scaling events. It even included EventBridge, which was impacted, and that even caused some ECS and EKS delays with provisioning new clusters and scaling of existing clusters.

Jesse: So, right out of the bat, I just want to say huge kudos to AWS for dogfooding all of their services within AWS itself: not just providing the services to its customers, but actually using Kinesis internally for other things like CloudWatch and Cognito. They called that out in the write-up and said, “Kinesis is leveraged for CloudWatch, and Cognito, and for other things, for various different use cases.” That's fantastic. That's definitely what you want from your service provider.

Pete: Yeah, I mean, it's a little amazing to hear, and also a little terrifying, that all of these services are built based on all of these other services. So, again, the complexity of the dependencies is pretty dramatic. But at the end of the day, it's still software underneath it; it's still humans. And I don't want to say that I am happy that Amazon had this outage at all, but watching a company of this stature, of this operational expertise, have an outage, it's kind of like watching the Masters when Tiger Woods duffs one into the water or something like that. It's just—it's a good reminder that—listen, we're all human, we're all working under largely the same constraints, and this stuff happens to everyone; no one is immune.

Jesse: And I think it's also a really great opportunity—after the write-up is released—to see how the Masters go about doing what they do. Because everybody at some point is going to have to troubleshoot some kind of technology problem, and we get to see firsthand from this, how they go about troubleshooting these technology problems.

Pete: Exactly. So, of course, one of the first things that I saw everywhere is everyone is, on mass, moving off of Amazon, right? They had an outage, so we're just going to turn off all our servers and just move over to GCP, or Azure, right? 

Jesse: Because GCP is a hundred percent uptime. Azure is a hundred percent uptime. They're never going to have any kind of outages like this. Google would never do something to maybe turn off a service, or sunset something.

Pete: Yeah, exactly. So, with the whole talk about hybrid-cloud and multi-cloud strategies, you got to know that there's a whole slew of people out there, probably some executive at some business, who says, “Well, we need to engineer for this type of durability, this type of thing to happen again,” but could you even imagine the complexity...

AWS Morning Brief for the week of April 10, 2023 with Corey Quinn.

Links:

• Console Toolbar is now generally available for AWS CloudShell
• Announcing CSV Export for AWS Resource Explorer Search Results
• Announcing Utilization Notifications for EC2 On-Demand Capacity
• Everything you need to know about AWS Billing Conductor’s new pricing model
• How to use Amazon CloudWatch to monitor Amazon DynamoDB table size and item count metrics
• Implement resource counters with Amazon DynamoDB
• AWS Organizations, moving an organization member account to another organization: Part 3
• Build secure multi-account multi-VPC connectivity for your applications with Amazon VPC Lattice 
• Higher education cloud financial planning: A former CFO’s perspective
• How the Think Big for Small Business program helps small businesses win big contracts
• Amazon started passing out Small Business labels to giant companies.
• Perfect imperfections: how AWS is innovating on diamond materials for quantum communication with Element Six

Want to give your ears a break and read this as an article? You’re looking for this link.

https://www.lastweekinaws.com/blog/The-Compelling-Economics-of-Cloudflare-R2

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

Want to give your ears a break and read this as an article? You’re looking for this link. https://www.lastweekinaws.com/blog/The-Turbotax-of-AWS-Billing

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

Last week in security news: Faster AWS cloud connections with TLS 1.3, Belkin is crappy in many ways, the Tool of the Week, and more!

Links:

• Amazon bought Pillpack, since they wanted to get into being our pharmacy. Now Pillpack reports a data breach affecting more than 19,000 people.
• Belkin is crappy in many ways
• AWS partners bring choice of temporary elevated access capabilities to IAM Identity Center
• Exclude cipher suites at the API gateway using a Network Load Balancer security policy 
• Faster AWS cloud connections with TLS 1.3 
• Stronger together: Highlights from RSA Conference 2023 
• This is a fun tool: Is It AWS 

Want to give your ears a break and read this as an article? You’re looking for this link. https://www.lastweekinaws.com/blog/The-Unfulfilled-Promise-of-Serverless

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

Last week in security news: A deep dive into the DomainNetworks Snail Mail Scam by Krebs on Security, Tailscale is putting their terms and conditions on GitHub, The Tool/ Lesson of the Week, and more!

Links:

• A deep dive into who's behind the DomainNetworks Snail Mail Scam by Krebs on Security.
• Tailscale is putting its terms and conditions on GitHub and invites users to subscribe to see diffs instead of legalese.
• Three ways to accelerate incident response in the cloud: insights from re:Inforce 2023
• Tool/ Lesson of the week: git-landmine 

AWS Morning Brief for the week of January 21, with Corey Quinn. 

Links:

• AWS CodePipeline introduces new debugging experience in AWS Management Console
• The AWS Management Console now supports simultaneous sign-in for multiple AWS accounts
• EC2 Image Builder simplifies converting Windows ISO files to AMIs
• Now open — AWS Mexico (Central) Region
• AWS CDK is splitting Construct Library and CLI
• Amazon Bedrock launches with Claude 3.5 Sonnet in the AWS Top Secret cloud
• Preventing unintended encryption of Amazon S3 objects
• Secure root user access for member accounts in AWS Organizations | AWS Security Blog
• Cost-optimized log aggregation and archival in Amazon S3 using s3tar
• Issue with Amazon WorkSpaces, Amazon AppStream 2.0, and Amazon DCV (CVE-2025-0500 and CVE-2025-0501)

Sponsor
The Duckbill Group: https://www.duckbillgroup.com/

Want to give your ears a break and read this as an article? You’re looking for this link.

https://www.lastweekinaws.com/blog/orca-security-aws-and-the-killer-whale-of-a-problem

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

Links

• Follow Last Week In AWS on Twitter

Transcript

Corey: This episode is sponsored in part by Catchpoint. Look, 80 percent of performance and availability issues don’t occur within your application code in your data center itself. It occurs well outside those boundaries, so it’s difficult to understand what’s actually happening. What Catchpoint does is makes it easier for enterprises to detect, identify, and of course, validate how reachable their application is, and of course, how happy their users are. It helps you get visibility into reachability, availability, performance, reliability, and of course, absorbency, because we’ll throw that one in, too. And it’s used by a bunch of interesting companies you may have heard of, like, you know, Google, Verizon, Oracle—but don’t hold that against them—and many more. To learn more, visit www.catchpoint.com, and tell them Corey sent you; wait for the wince.

Pete: Hello, and welcome to AWS Morning Brief. I am Pete Cheslock. Corey, while being back from his paternity leave, is still not here. We are having too much fun. And by we, I mean I'm joined again with Jesse DeRose. Hey, Jesse. 

Jesse: Thanks as always for having me, Pete. 

Pete: It's so much fun to again chat with people outside of my little family unit, that we've just decided not to give this back to Corey. And luckily, Corey has many other podcasts that he does, he was pretty happy to give it away.

Jesse: I feel like you should never talk about your children that way, but he's got a plethora at this point. So, he's willing to kind of share the wealth.

Pete: Exactly. And if you notice, we have a new theme song that came out, I think it was last week was the first week that we brought in the new theme song, which is I think much in line with a previous episode where we talked about ’80s breakdancing movies that the new theme song kind of has that vibe to it.

Jesse: I hope you're wearing the Members Only jean jacket that I sent you, along with the shades to match the uniform.

Pete: Yeah. I mean, I was born in ’80, so the ’80s for me, I was very young. I'm kind of waiting for the ’90s movies to come around again because I want to rock out my JNCO jeans and my wallet chain. 

Jesse: [laugh], yes.

Pete: And all that good stuff.

Jesse: I am ready.

Pete: Exactly. Well, what are we talking about today? Well, earlier this week, AWS Storage Day 2020 happened on Tuesday. If you were a part of that, it was a free online event. As Amazon called it, a full day online event. Except it was only about four hours long, so kind of mailing it in on that one, huh?

Jesse: Can we start discussing that with our boss and say that a full day of work is technically just four hours? Can we just start working with that going forward?

Pete: Yeah, we'll just say it right now. So, hey, Corey, we're done for the day. Put in the old college four.

Jesse: [laugh]. That's what you say, “I put in the old college try. I just did my full day of four hours, according to AWS. So, this has been great. I'll talk to you tomorrow.”

Pete: Exactly. Well, Storage Day this year—it's the second year in a row if I'm remembering it correctly. 2019 was the last year they did that—and I feel like this kind of ties into the fact that there's just so many announcements that happened around re:Invent, that leading up into re:Invent, you have a lot of announcements to maybe soften the blow for a lot of folks. And Storage Day, really is just this whole day—well, four hours worth of a whole day—talking about everything related to storage. And we're talking about things like S3, EBS, EFS, FSx, for the five huge enterprises that probably use FSx. 

Although if you actually do use FSx, I'd be curious to hear about how you like it and what you think of it because we don't really hear a lot of people using it. But these are all the services, plus many more, that Amazon talked about as part of its Storage Day.

Jesse: Yeah, it was a really interesting discussion. I greatly appreciate that AWS broke out this discussion prior to AWS re:Invent, but they dropped a lot of knowledge on us all at once, and in, like, rapid-fire succession, I was really, kind of… not necessarily surprised, but there's a lot of information that they shared all at once. And I have to admit that after sitting through this presentation, I now have a greater appreciation for Apple's slow presentation style. As much as I hate it; as much as I hate sitting for an hour and a half for one announcement while they toot their own horn, I have to say that the buildup and getting me involved in the story and bringing me along with them. It works, it absolutely works. And it was kind of hard for me to pick up on all the things that went on during AWS Storage Day this year because there was a lot of things going on.

Pete: And honestly, the fact they give so much information is really amazing in, I guess, both their ability to tout, in many cases, minor feature changes that most SaaS businesses would just turn on and maybe blog about. But this is—obviously the engine of AWS is so good at discussing their wins. But you're right, it's just a huge amount. On Monday, Jeff Barr of course, wrote the blog post with a lot of these details, linking to countless other blog posts. And I think it really speaks to just how, probably every, or nearly every Amazon service ties into storage in some way. It's a huge, huge part of this ecosystem. 

Jesse: Absolutely. 

Pete: So, as you can imagine, there were so many new features that we're not even going to be able to cover them all throughout the course, but we did want to call out some of the big ones, or at least what we thought were the biggest ones, the most interesting new features, new product announcements that came out, and also just touch on some of the other things that we thought were pretty interesting as well. And yeah, there was a lot of fun stuff. I think the biggest one that was announced was the S3 Intelligent-Tiering, which is a class storage tier within S3, adds additional levels of archive access. So, if you imagine Intelligent-Tiering, you know, you have the automatic tiering of data from frequently accessed to infrequently accessed as things age out, they essentially automate that for you. So, as things are not accessed, you just start automatically paying less for them. And anything automatic in a cost savings world is going to help you save money. 

If you don't have to think about it and it just does it for you, it's fantastic. Well, Intelligent-Tiering added in these additional tiers—which they are Glacier—level tiers. They are additional places that your data can eventually move to as they start aging out based on a whole series of criteria. But there's caveats. There's more caveats now. 

Before, one of the interesting things that we actually learned as part of this—because it was buried in a pricing page footnote—is that when you store something into Intelligent-Tiering, there is a minimum storage time period that you will get charge...

AWS Morning Brief for the week of May 8, 2023 with Corey Quinn.

Links:

• Announcing Provisioned Capacity for Amazon Athena
• Amazon EFS Replication is now available in all AWS Regions
• Amazon Redshift launches ra3.xlplus instances in additional Middle East, Europe and Asia Pacific Regions 
• AWS Compute Optimizer now supports filtering by tags
• AWS Console Mobile Application launches push notifications
• Announcing AWS User Notifications general availability 
• Process price transparency data using AWS Glue 
• Patterns for building an API to upload files to Amazon S3
• Improve query performance and reduce cost using scheduled queries in Amazon Timestream
• Working with JSON data in Amazon DynamoDB 
• The history and future roadmap of the AWS CloudFormation Registry 
• Partnerships extend Just Walk Out technology to more colleges and universities
• Quickly build high-accuracy Generative AI applications on enterprise data using Amazon Kendra, LangChain, and large language models 
• Introducing AWS Verified Access – General Availability 

Want to give your ears a break and read this as an article? You’re looking for this link. https://www.lastweekinaws.com/blog/the-lessons-aws-infinidash 

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

AWS Morning Brief for the week of Monday, July 1st, with Corey Quinn. 

Links:

• Amazon DocumentDB announces IAM database authentication
• Amazon Redshift Query Editor V2 now supports 100MB file uploads
• Amazon Time Sync Service expands microsecond-accurate time to 27 EC2 instance types
• Announcing Amazon WorkSpaces Pools, a new feature of Amazon WorkSpaces
• AWS CodeBuild supports Arm-based workloads using AWS Graviton3
• Optimizing Amazon Simple Queue Service (SQS) for speed and scale
• Ten Ways to Improve Your AWS Operations

Want to give your ears a break and read this as an article? You’re looking for this link. https://www.lastweekinaws.com/blog/Developer-Portals-are-an-Anti-Pattern

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

AWS Morning Brief for the week of January 13, with Corey Quinn. 

Links:

• Amazon DynamoDB now supports configurable point-in-time-recovery periods
• AWS Compute Optimizer now expands idle and rightsizing recommendations for Amazon EC2 Auto Scaling groups
• Announcing the new AWS Asia Pacific (Thailand) Region | AWS News Blog
• Shaping the future of CDK together
• The benefits of an email-based AI virtual assistant on AWS
• Securing a city-sized event: How Amazon integrates physical and logical security at re:Invent

Sponsor
The Duckbill Group: https://www.duckbillgroup.com/

Want to give your ears a break and read this as an article? You’re looking for this link.

https://www.lastweekinaws.com/blog/my-mental-model-of-aws-regions

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

AWS Morning Brief for the week of June 20th, 2023 with Scott Piper filling in for Corey Quinn.

Links:

• AWS announces scripts to bulk updates policies per new AWS Billing and Cost Management permissions 
• Amazon QuickSight now supports APIs to automate and accelerate assets deployment
• Amazon Rekognition improves face search accuracy with user vectors
• AWS Config supports recording exclusions by resource type
• Falcon 40B foundation model from TII available on SageMaker JumpStart
• Amazon EMR supports price-capacity-optimized allocation strategy for EC2 Spot Instances
• Amazon Verified Permissions is now generally available
• Announcing Live Tail in Amazon CloudWatch Logs, providing real-time exploration of logs
• aidansteele/rdsconn

Links:

• Poisoned Python and PHP packages purloin passwords for AWS access
• No, your cloud environment doesn't need a sandbox
• Spring 2022 SOC reports are now available with 150 services in scope
• Canary Tokens

AWS Morning Brief for the week of August 7, 2023 with Corey Quinn. 

Links:

• In Amazon's earnings call, Andy Jassy said that every Amazon team is working on Generative AI
• Amazon Route 53 adds support for 14 additional Top-Level Domains 
• AWS NAT Gateway is now available in the AWS US West Phoenix Local Zone
• Amazon EBS announces up to 128 volume attachments per EC2 instance
• Introducing Amazon EC2 M7i-flex and M7i instances
• Amazon EventBridge Scheduler adds schedule deletion after completion
• AWS Application Composer updates: Undo and Redo, Export Canvas, and Local Sync Mode
• Now Open – AWS Israel (Tel Aviv ) Region
• Prime Day 2023 Powered by AWS – All the Numbers 
• Estimate cost savings for the Amazon Aurora I/O-Optimized feature using Amazon CloudWatch 
• Empowering your workforce with Amazon WorkSpaces services and Microsoft 365
• Exploring Fn::ForEach and Fn::FindInMap enhancements in AWS CloudFormation 
• Identify and optimize public IPv4 address usage on AWS 

AWS Morning Brief for the week of February 26, 2024 with Corey Quinn. 

Links:

• • AWS Incident Detection and Response now offers five minute response time for critical incidents
  • Multiple solution tagging on ACE opportunities 
  • Introducing the .NET 8 runtime for AWS Lambda 
  • Best practices for managing Terraform State files in AWS CI/CD Pipeline 
  • Understand Amazon S3 data transfer costs by classifying requests with Amazon Athena