TechOps Examples
Hey — It's Govardhana MK 👋
Welcome to another technical edition.
Every Tuesday – You’ll receive a free edition with a byte-size use case, remote job opportunities, top news, tools, and articles.
Every Thursday and Saturday – You’ll receive a special edition with a deep dive use case, remote job opportunities and articles.
Top engineers at Anthropic and OpenAI say AI now writes 100% of their code.
If you're not using AI, you're spending 40 hours doing what they do in 4.
These 100+ Claude Code hacks fix that and help you ship 10x faster.
Sign up for The Code and get:
100+ Claude Code hacks used by top engineers — free
The Code newsletter — learn the latest AI tools, tips, and skills to code faster with AI in 5 minutes a day
Architecture to Overcome API Gateway Pay Load Limit
Today's use case is about "Overcoming AWS API Gateway Payload Limits" with an ideal architecture example.
Imagine you're developing a document management mobile app, where users can upload large files such as PDFs, presentations, and spreadsheets. The app needs to store these documents securely and efficiently.
For this, utilizing AWS services like API Gateway and S3 is an ideal choice due to their scalability, security, and ease of integration.
A mobile client can directly upload files to an S3 bucket via API Gateway, as depicted below:
While this might seem straightforward, it has a critical limitation: API Gateway imposes a payload size limit of 10MB.
In this setup:
Mobile Client sends a request to API Gateway.
API Gateway attempts to upload the file to S3 Bucket.
Files under 10MB succeed, while those over 10MB fail.
This architecture is not only inefficient but also unsuitable for applications dealing with larger files.
Key considerations for API Gateway:
Feature | Value/Range |
|---|---|
Maximum routes per API | 300 |
Maximum integrations per API | 300 |
Maximum stages per API | 10 |
Maximum request payload size | 10 MB |
Maximum response payload size | 10 MB |
Request rate limit | 10,000 requests per second |
Burst limit | 5,000 requests per second |
Maximum Integration timeout | 30 seconds |
API key limit per account | 10,000 API keys |
Deployment limit per stage | 10 deployments |
Don't worry, we've got this covered 😃
Have you heard about Pre-Signed URLs?
A pre-signed URL is a URL that you can provide to your users to grant temporary access to a specific S3 bucket. With this URL, users can upload files directly to S3 without requiring AWS credentials or going through an intermediary service like API Gateway.
Pre-signed URLs are time-limited and can be configured to allow only specific actions (like uploads), ensuring secure and controlled access to your S3 buckets.
Let's transition to a better architecture as below:
In this setup:
Mobile Client sends a request to API Gateway.
API Gateway invokes a Lambda Function.
Lambda Function generates a pre-signed URL for the S3 bucket.
Lambda Function returns the pre-signed URL to API Gateway.
API Gateway forwards the pre-signed URL to the Mobile Client.
Mobile Client uses the pre-signed URL to upload the file directly to S3 Bucket.
Why is this the Better Choice?
This improved architecture offers several benefits:
No File Size Limitations: By allowing direct uploads to S3, the file size constraints of API Gateway are eliminated.
Enhanced Performance: Direct uploads reduce latency and improve the overall performance of the file upload process.
Scalability: This architecture can handle a larger number of uploads without additional complexity.
Security: Pre-signed URLs ensure that only authorized users can upload files, maintaining security without sacrificing convenience.
This way, you ensure a seamless, scalable, and efficient file upload process for your mobile app, enhancing user experience and operational efficiency.
In my experience, what I understand is 'architecture' design is a subjective choice and differs drastically with individuals and systems.
This solution is one of the choices - That's it!
