Blog
Engineering notes
Engineering notes from Azeem Subhani on SaaS, payments, booking, and AI application work: architecture, trade-offs, and what shipped.
26 posts
11 min read
Unexpected AWS Bill Investigation: Find the Usage Type and Resource
Trace a surprise AWS bill from the daily total to service, usage type, and resource, and learn which charges never carry a resource ID.
- Cloud
- AWS
- Cost
12 min read
Retry Storms: Exponential Backoff, Jitter, and One Retry Layer
Why capped exponential backoff still causes retry storms, how retries multiply across layers, and how to fix it with full jitter, budgets, and one retry layer.
- APIs
- Distributed systems
- Reliability
- AWS
11 min read
Transaction Open During an HTTP Call: Why the Pool Drains
A transaction held across a remote call ties up a connection and its locks. Find it, split the transaction, and recover safely with idempotency keys.
- Postgres
- Databases
- Reliability
- APIs
11 min read
P99 Latency: Why the Average Looks Fine and Users Still Retry
The mean looks healthy while one in a hundred requests is slow enough to trigger retries. Read the histogram, find the tail's shape, and fix it.
- Performance
- Reliability
- Observability
- Distributed systems
10 min read
Postgres Migration Locks the Table: lock_timeout and the Lock Queue
Why a waiting ALTER TABLE stalls every query on a Postgres table, and how lock_timeout, jittered retries, and CONCURRENTLY keep migrations safe.
- Postgres
- Databases
- Reliability
11 min read
Offset Pagination Is Slow and Skips Rows: Switch to Keyset Pagination
Why deep OFFSET pages get slow and why rows skip or repeat under writes, plus keyset pagination, cursors, indexes, and cheap totals in Postgres.
- Postgres
- Databases
- Performance
- APIs
9 min read
Android App Startup Time: Measure TTID and TTFD Before You Fix
Name the startup clock first: TTID or TTFD. Measure cold starts with Macrobenchmark, then decide whether profiles or deferred work will actually help.
- Mobile
- Android
- Performance
10 min read
RAG Retrieves Irrelevant Chunks: Evaluate Retrieval Before the Prompt
Your RAG answer is wrong because the right passage never reached the model. Label queries, measure recall, and fix chunking and filters.
- AI
- LLMs
- Data engineering
- Databases
13 min read
Duplicate Events and Idempotent Consumers: At-Least-Once Done Right
At-least-once delivery means duplicates. Dedupe with a stable key in the same transaction, ack after commit, and guard external side effects.
- Distributed systems
- Reliability
- Postgres
- APIs
9 min read
High Cardinality Metrics: When OpenTelemetry Overflow Silences Alerts
OpenTelemetry cardinality overflow keeps totals right but undercounts breakdowns, so error alerts can go quiet. Learn to detect, prevent, and bound it.
- Observability
- Reliability
- Distributed systems
10 min read
Kubernetes OOMKilled Debugging: Container Limit, Node, or Heap?
Tell a container limit kill, a node memory kill, and a runtime heap failure apart before you raise a Kubernetes memory limit.
- Kubernetes
- Reliability
- Cloud
10 min read
API Slow Under Load? Measure Connection Pool Wait Before Tuning
Pool wait, not query time, is often why APIs slow down under load. Measure wait and hold time, cap waiters, and size the pool for the database.
- Performance
- APIs
- Postgres
- Databases
4 min read
Typed API clients and token refresh: fixing auth-expiry bugs
Why booking flows fail when access tokens expire mid-checkout, and how typed API clients, token refresh, and route guards removed a class of those bugs.
- TypeScript
- Auth
- Next.js
9 min read
Debug Poor INP: Find the Slow Phase in Real User Data
Lab scores look fine but taps feel late. Split INP into input delay, processing, and presentation delay, reproduce it, and verify the fix in field data.
- Performance
- Web
- Observability
9 min read
Webhook Return 200 Before Processing: Store First, Then Do the Work
Return a webhook 200 before processing, but only after a durable insert. How to avoid lost events and double applies with Stripe-style retries.
- APIs
- Payments
- Reliability
- Distributed systems
11 min read
Next.js Stale Cache Serving the Wrong User: Find the Layer
Logged-in users see another account's data, or stay stale after revalidation. Map the symptom to the Next.js 16 cache layer and invalidate the right one.
- Next.js
- Web
- Security
- Performance
10 min read
NAT Gateway Cost and Data Transfer: Diagnose the Path, Then Fix It
Trace NAT gateway and cross-AZ charges back to the network path, then choose gateway endpoints, zone routing, or interface endpoints.
- Cloud
- AWS
- Cost
10 min read
Serverless Database Connection Exhaustion: Fix the Fleet Arithmetic
Serverless instances times pool size can exceed Postgres max_connections. Size pools per platform, add a pooler, and know what transaction mode breaks.
- Cloud
- AWS
- Postgres
- Databases
12 min read
LLM API Cost Unexpectedly High: Find Which Calls Drive the Bill
The token price did not change but your LLM bill did. Trace it by feature, call pattern, and token type, with retries and loops.
- LLMs
- AI
- Cost
- Observability
11 min read
Idempotency Key Race Conditions That Still Cause Duplicate Charges
Three ways an idempotency key still lets a charge run twice, and the claim, call, record design in Postgres that closes each race.
- APIs
- Payments
- Postgres
- Distributed systems
11 min read
N+1 Query Detection in Postgres: pg_stat_statements, Not EXPLAIN
Find N+1 queries in Postgres when EXPLAIN says every query is fast and the request is slow. Sort pg_stat_statements by calls, then join or batch.
- Postgres
- Databases
- Performance
- APIs
10 min read
Late Arriving Data Pipeline: Watermarks, Lateness, and Restatement
Yesterday's total changed because late events arrived. Choose a restate, reconcile, or ignore policy, then set watermarks and lookbacks.
- Data engineering
- Distributed systems
- Reliability
11 min read
IDOR and Broken Object Level Authorization: A Review Guide for APIs
Logged in is not allowed. Find every object id in a request, scope queries by tenant, test with two users, and stop trusting UUIDs as access control.
- Security
- APIs
- Postgres
- Web
5 min read
Five branded booking sites on one payments engine
How a motorsports booking platform serves five white-label venues from one shared reservation and Stripe payments engine, and what that design costs.
- Architecture
- Payments
- Multi-tenant
11 min read
Read Replica Stale Reads: Read-Your-Writes Routing for Postgres Lag
The user saved and saw old data. Prove replica lag with a WAL timeline, classify reads, and route read-your-writes without overloading the primary.
- Postgres
- Databases
- Reliability
- AWS
4 min read
Next.js product sites that search engines can actually read
A checklist for shipping Next.js marketing and product pages that crawlers can parse: server-rendered HTML, metadata, canonicals, sitemaps, and structured data.
- Next.js
- SEO
- Performance