<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[Databases]]></title><description><![CDATA[Databases]]></description><link>https://dbbyyaz.hashnode.dev</link><image><url>https://cdn.hashnode.com/uploads/logos/6af4e297ed0c9c85a1b00a48/899ffce0-012c-43d2-bc60-ff239ad10128.jpg</url><title>Databases</title><link>https://dbbyyaz.hashnode.dev</link></image><generator>RSS for Node</generator><lastBuildDate>Wed, 18 Nov 2026 13:31:01 GMT</lastBuildDate><atom:link href="https://dbbyyaz.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[Redis]]></title><description><![CDATA[Most people call Redis a key-value store, and honestly that undersells it. A normal key-value store gives you a key, a blob, and GET/SET. That’s it. Redis…]]></description><link>https://dbbyyaz.hashnode.dev/redis</link><guid isPermaLink="true">https://dbbyyaz.hashnode.dev/redis</guid><dc:creator><![CDATA[Yazdan Haider]]></dc:creator><pubDate>Fri, 13 Nov 2026 19:34:17 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/89d461e4-489f-415e-b348-7b86ec4d12ca.jpg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<h1>Redis's Five Fundamental Data Types: A Deep Dive</h1>
<p>Most people call Redis a key-value store, and honestly that undersells it. A normal key-value store gives you a key, a blob, and GET/SET. That’s it. Redis is more like a data-structure server. Every value has a type, and the type decides what the server can do for you, like atomic increments, set operations, ranking, or blocking pops. So picking a type isn’t about taste. You’re really asking “what operations do I need, and how fast do they have to be?”</p>
<p>In this article we’ll go through the five core data types, then persistence, config, replication, Sentinel, Cluster, and how to debug a misbehaving Redis.</p>
<hr />
<h2>1. Strings</h2>
<p>Strings are the simplest type. A key points to some bytes, which can be text, a number, or a serialized JSON blob, up to 512 MB. The part people miss is that INCR and DECR are atomic, so you can build rate limiters, view counters and ID generators without worrying about two clients stepping on each other. If you ever need to cache a serialized object, this is where it goes.</p>
<h2>2. Sets</h2>
<p>A set is a bunch of unique strings with no order. Adding the same thing twice does nothing, and checking if something exists (SISMEMBER) is fast. The fun part is that Redis can do the set math for you with SINTER, SUNION and SDIFF, so something like “which users saw both campaign A and B” is a single command. The downside is no ordering and no ranges. If you need those, look at sorted sets.</p>
<h2>3. Sorted Sets (ZSETs)</h2>
<p>Sorted sets are my favorite. Every member has a score, and Redis keeps them ordered by it. You can grab the top 10 (ZRANGE), ask “what rank is this player?” (ZRANK), or pull everything between two scores (ZRANGEBYSCORE), and all of it stays fast. Leaderboards are the obvious use case, but priority queues and sliding-window rate limits work well too. Members are unique, so if you need the same item twice with different values, this isn’t the type for you.</p>
<h2>4. Lists</h2>
<p>Lists keep items in insertion order, and pushing or popping at either end is fast. That’s why LPUSH with RPOP gives you a simple queue, and LPUSH with LTRIM gives you a capped feed of the latest N items. BLPOP and BRPOP let a worker sit and wait until a job shows up. Reaching into the middle of a big list is slow, though, so don’t use it like an array. For real event streams with consumer groups, use Streams (added in Redis 5) instead of a list.</p>
<h2>5. Hashes</h2>
<p>A hash is a small object under one key, made of field and value pairs. Say you store a user as a JSON string. Every small change means reading the whole thing, editing it, and writing it back. With a hash you just HSET the one field you care about. HINCRBY works as a per-field counter too. The limits are that values are plain strings, so there’s no nesting, and you can’t search by field value. If your data is deeply nested, a document database fits better.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/ea90ba04-8dd5-4a90-89de-25ceabb23ad9.jpg" alt="" style="display:block;margin:0 auto" />

<hr />
<h2>Key Benefits that Redis Provides</h2>
<p>So why use Redis at all? Mainly speed. Everything lives in RAM, so responses come back in under a millisecond. Command execution is single-threaded (newer versions use extra threads for network I/O), which means no lock fights between commands.</p>
<p>It also handles growth and failure. You can shard data across nodes with Cluster, keep copies with replication, and let Sentinel promote a replica if the primary dies. And because it’s so fast, it’s a natural cache. When memory fills up it evicts old keys using policies like LRU or LFU.</p>
<p>On top of that, the data structures give you features for free: sorted sets for rankings, HyperLogLog to count unique visitors with tiny memory, and Streams for event pipelines. It’s also easy to pick up. One config file, simple commands, and a client library for almost every language.</p>
<hr />
<h1>Data Persistence</h1>
<p>Lets move forward. If you were paying attention upto this point the first question coming to your mind would be if its stored in memory. Then that means data would be lost on restart. And yes my brother you are correct , partially that is since redis also saves data on disk:</p>
<ul>
<li><strong>RDB (Redis Database Snapshots):</strong> Redis takes point-in-time snapshots of your entire dataset at configured intervals (can be configured by you) and saves them to disk (.rdb). It is compact and fast for backups and restarts, with one trade-off: if Redis crashes between snapshots, you lose the writes made since the last save.</li>
<li><strong>AOF (Append-Only File):</strong> Every single write command is logged to an append-only file in real time. If the server crashes, Redis replays the log to reconstruct the exact state right up to the failure. You trade a slightly larger disk footprint and minimal write overhead for maximum durability.</li>
<li><strong>RDB + AOF Hybrid (Production Standard):</strong> In production, you rarely have to choose. Running both gives you fast cold starts from RDB snapshots while using AOF to replay only the most recent tail of writes—balancing raw performance with near-zero data loss.</li>
</ul>
<hr />
<h1>Configuration</h1>
<p>We will look now on some configurable values we can set in our configuration file. Based on your installation you will have configuration file either in <code>/etc/redis/redis.conf</code> or /etc/redis.conf.</p>
<p><em><strong>CLI commands take effect immediately but are not persisted</strong></em><em>;</em> <em><strong>config file settings survive restarts</strong></em></p>
<ul>
<li><p>We can bind redis to listen on a specific interface ( Bind it to private on production for Security)</p>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/65b5e74d-c908-45fb-8af3-66c8cd152ac8.png" alt="" style="display:block;margin:0 auto" /></li>
<li><p>Protected mode (<code>protected-mode yes</code>) acts as a safety net that automatically enforces loopback-only access (<code>127.0.0.1</code>, <code>::1</code>, and Unix domain sockets) if <strong>both</strong> of the following conditions are true: [<a href="https://gist.github.com/AysadKozanoglu/8d7cfdf32598d638e2b6f4e5bf02182a">1</a>, <a href="https://nhimg.org/redis-security-guide">2</a>]</p>
<ul>
<li><p><strong>No explicit bind address is set.</strong> The <code>bind</code> directive in <code>redis.conf</code> is not explicitly restricting network interfaces.</p>
</li>
<li><p><strong>No password is configured.</strong> Neither a legacy <code>requirepass</code> password nor an ACL password for the <code>default</code> user is set.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/33749aaf-e00c-4b64-ab06-1d3fb03aed67.png" alt="" style="display:block;margin:0 auto" /></li>
</ul>
</li>
<li><p>We can specify which port the redis will listen on crucial if multiple redis instances running on same server (Highly unlikely but is possible if resource constraint)</p>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/ceb0d513-5fc8-4731-a2ac-aedc262a7f1b.png" alt="" style="display:block;margin:0 auto" /></li>
<li><p>This feature is essentially used by redis to maintain max open tcp connections in case of sudden spikes you can increase it but it will also require some kernel tuning to accept more connection since the redis uses the accept() system call for accepting connections.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/4562bca6-9a1e-4929-8309-e8d27be32ac6.png" alt="" style="display:block;margin:0 auto" /></li>
<li><p><code>timeout and tcp-keepalive</code><strong>:</strong> <code>timeout 0</code> means Redis never closes idle connections, and <code>tcp-keepalive 300</code> makes it check dead connections every 5 minutes. If your app uses persistent connection pools (sessions, dashboards, auth APIs), this default is fine and I’d leave it. Only set a timeout if your clients open and abandon connections a lot.</p>
</li>
<li><p><code>unixsocket</code><strong>:</strong> disabled by default. Enable it only if your app runs on the same machine as Redis, since it skips the TCP stack entirely.</p>
</li>
</ul>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/f608aa3a-870d-4648-9cf2-5c979e595bce.png" alt="" style="display:block;margin:0 auto" />

<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/24dda027-c0ed-4d3f-be83-49eb5d154f0b.png" alt="" style="display:block;margin:0 auto" />

<ul>
<li><p>Location of pid file , <code>loglevel notice</code> sets the verbosity of Redis server logs to <code>notice</code>, which is the **default and recommended setting for production and location of redis log file.**Redis runs one command at a time, so each command on its own is atomic. But a sequence of commands isn’t. Your app sends GET, waits for the reply, thinks about it, then sends DECR. In between those two calls, Redis happily serves other clients. That gap is where the race lives. Both users read “1”, both decide they can buy, both decrement, and stock ends up at -1.</p>
<p>The fix is to move the decision into Redis. A Lua script runs as one block, so nothing else can sneak in between the check and the update.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/0b474fbf-edc6-45fc-a45e-aafbc99fda14.png" alt="" style="display:block;margin:0 auto" /></li>
<li><p><code>databases</code> Sets the number of isolated logical keyspaces. <code>set-proc-title</code> Dynamically updates the Redis process name in OS tools like <code>ps</code> and <code>top</code> to display runtime details such as its bound IP address, port, and cluster role.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/f11a1690-23aa-45cf-85f8-47008eb5e9a0.png" alt="" style="display:block;margin:0 auto" /></li>
</ul>
<p>lets keep the configuration file dissection upto here the following configurations focus on sharding and replication and clustering and read only mode.</p>
<hr />
<h1>The Race Condition Problem: Why Lua Scripts Exist in Redis</h1>
<p>Before we dive into replication, sharding, and clustering mechanisms, let's first address a common problem we face with this in-memory data store.</p>
<p>As you've heard, Redis is single-threaded and in-memory. Technically, the way it works is that it performs each request — whether it's a <code>SET</code> or a <code>GET</code> — one at a time. Only now consider you're using a shopping cart inside an e-commerce store, and we have <strong>one item left in stock</strong>. You add it to your cart. So yes, you took the last one.</p>
<p>But if it's a hot item and another person adds the same item to his cart at the same moment, now the <strong>same item is booked twice</strong> while the quantity of that item remains 1. This problem results in a <strong>race condition</strong>, which is solved using Lua scripts in Redis — they allow you to perform multiple Redis commands as a single unit, hence providing <strong>atomicity</strong>.</p>
<h2>Let's Walk Through What Actually Happens</h2>
<p>Redis runs one command at a time, so each command on its own is atomic. But a sequence of commands isn’t. Your app sends GET, waits for the reply, thinks about it, then sends DECR. In between those two calls, Redis happily serves other clients. That gap is where the race lives. Both users read “1”, both decide they can buy, both decrement, and stock ends up at -1.</p>
<p>The fix is to move the decision into Redis. A Lua script runs as one block, so nothing else can sneak in between the check and the update.</p>
<p>Both users see "1", both pass the check, both decrement. Stock ends at <code>-1</code>. <strong>One physical item, two confirmed purchases.</strong></p>
<h2>The Lua Fix: Make the Whole Check-and-Act One Unit</h2>
<p>The insight is simple: <strong>move the decision into Redis, so there is no gap between the check and the act.</strong> A Lua script runs as one indivisible unit — no other client's command can interleave while it executes.</p>
<pre><code class="language-lua">-- KEYS[1] = stock key,  ARGV[1] = quantity to buy
local stock = tonumber(redis.call('GET', KEYS[1]))
if stock and stock &gt;= tonumber(ARGV[1]) then
    redis.call('DECRBY', KEYS[1], ARGV[1])
    return 1          -- success
else
    return 0          -- sold out
end
</code></pre>
<p>User 2's script <strong>cannot start</strong> until User 1's script finishes (single threaded remember). There is no window between the read and the write. The check and the act are one step.</p>
<hr />
<h1>Redis Replication</h1>
<h2>The Core Distinction: Async by Default</h2>
<p>Redis replication is <strong>asynchronous by default</strong>. When you send <code>SET k v</code>, the primary applies it locally, appends it to the replication stream, and <strong>replies</strong> <code>OK</code> <strong>immediately</strong> — before the replica has applied anything. The primary does not wait for replica acknowledgment.</p>
<p><strong>Y</strong>our latency is bounded by the primary's local processing only. No network round-trip to replicas, no waiting on slow replicas, no coordination. This is why Redis does 100k+ writes/sec per node.</p>
<p><em>If the primary dies right after replying</em> <code>OK</code> <em>but before the replica got the write, that write is</em> <em><strong>gone</strong></em><em>. On failover, the replica is promoted — and it never had the write. This is replication lag manifesting as data loss.</em></p>
<p><strong>T</strong>he replica's offset trails (how much behind the replica) the primary's. Check <code>INFO replication</code> — <code>master_repl_offset</code> versus <code>slave_repl_offset</code>, and <code>master_last_io_seconds_ago</code>. Lag is a read-staleness risk and a failover-duration risk.</p>
<hr />
<h2>WAIT: Synchronous-Like, But Not Synchronous</h2>
<pre><code class="language-bash">SET k v
WAIT 2 1000    # block up to 1000ms until 2 replicas ack
</code></pre>
<p>Redis replication is asynchronous by default. When you run SET, the primary applies it, replies OK, and sends the change to replicas afterwards. It doesn’t wait for them. That’s why writes are so fast. The catch is that if the primary dies right after replying OK and before a replica got the write, that write is gone. This is replication lag turning into data loss.</p>
<p>If a write really matters, you can use WAIT. It blocks your client until N replicas acknowledge, or until the timeout hits. Always check the return value, since it can return fewer replicas than you asked for. And use it only on important writes, not every SET.</p>
<p><strong>What it does NOT do:</strong></p>
<ul>
<li>Doesn't guarantee primary-side durability — if the primary restarts from an old RDB snapshot, the write is lost even if replicas had it.</li>
<li>Doesn't cover other clients' writes — it's per-connection state, not a global barrier.</li>
<li>Can return short — <code>WAIT 2 1000</code> can return <code>1</code> after a second.</li>
<li>Adds latency — at minimum one round-trip to the slowest acking replica.</li>
</ul>
<p>Use it selectively on critical writes, not every <code>SET</code>.</p>
<p>We can also configure the redis to stream the write directly over the network to replicas even before writing but that assumes no network partition occurs and in that case the data will be lost.</p>
<hr />
<h2>How Replication Works: Full Sync and Streaming</h2>
<p>On first connect (or after falling too far behind), the replica does a <strong>full sync</strong> — the primary forks, writes an RDB snapshot, sends it, and buffers writes to replay after. This is expensive.</p>
<p>After the snapshot, every write is streamed continuously. If a replica drops out, Redis tries a <strong>partial resync</strong> using the <strong>replication backlog</strong> (<code>repl-backlog-size</code>, default <strong>1 MB</strong>). Do note the following important concept</p>
<ul>
<li>If the replica's offset is still inside the backlog, it gets only the missing bytes. If it fell out, it needs a full resync. Size the backlog for your longest tolerable disconnect multiplied by write rate — too small means frequent full resyncs, too large means memory cost. So keep its size large enough to handle temporary network outages and network latency drop.</li>
</ul>
<ul>
<li><code>replicaof &lt;primaryip&gt; &lt;primary port&gt;</code></li>
</ul>
<p>This is the persistent form. <code>redis-cli REPLICAOF host port</code> works at runtime but is forgotten on restart. <strong>Never use both</strong> — the CLI overrides at runtime, then a restart reverts to config, producing a topology that changes under you.</p>
<ul>
<li><code>replica-read-only yes</code></li>
</ul>
<p>This is <strong>read-only mode</strong> for replicas — on by default. Replicas reject writes. Why it matters: a write on a replica is not propagated back to the primary, so the replica silently diverges. When failover promotes it, it carries data the primary never had — corruption. Keep this on. Before setting up read scaling, run <code>CONFIG GET replica-read-only</code> and confirm it returns <code>yes</code>.</p>
<p>Note: this is not the same as "replica serves reads." It only means writes are refused. Whether your app reads from replicas is a client-side routing decision.</p>
<ul>
<li><code>repl-diskless-sync yes</code></li>
<li><code>repl-diskless-sync-delay 5</code></li>
</ul>
<p>Normally the primary forks, writes the RDB snapshot to disk, then ships the file. Diskless streams the RDB directly over the socket — no disk write.</p>
<p>The delay setting only matters with diskless on: after a sync request, the primary waits up to N seconds so other replicas arriving in that window can join the <strong>same</strong> snapshot transfer — one fork, many replicas.</p>
<h3><code>repl-timeout 60</code></h3>
<p>How long the replica waits for communication from the primary before dropping the link and retrying. Too short means spurious disconnects on busy networks (each may trigger a resync — expensive). Too long means slow failure detection and delayed failover. Raise it for high-latency links.</p>
<ul>
<li><code>min-replicas-to-write 1</code></li>
<li><code>min-replicas-max-lag 10</code>\</li>
</ul>
<p>Master auth is required for replica successfully connecting with master</p>
<ul>
<li><code>masterauth your-strong-password\ \ </code></li>
</ul>
<p><code>requirepass</code> is for clients connecting <em>to</em> this node.</p>
<p><code>masterauth</code> is what this replica uses to authenticate <em>to</em> the primary. A replica with <code>requirepass</code> but no <code>masterauth</code> will fail to sync against a password-protected primary — a common misconfiguration.</p>
<hr />
<h1>Redis Sentinel and Clustering Explained</h1>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/12a4fa80-7086-4f2e-9e3d-e425209913b4.png" alt="" style="display:block;margin:0 auto" />

<h2>Way 1: Standard Replication + Sentinel</h2>
<p>Use this when you want read scaling and automatic failover but your data still fits on one machine. Replication alone can’t fail over. You’d have to run <code>REPLICAOF NO ONE</code> yourself. Sentinel watches the primary, and when it goes down it picks a replica, promotes it, and reconfigures the old primary when it comes back. Run at least three Sentinels so they can agree the primary is really dead, and use a Sentinel-aware client (Lettuce, ioredis, redis-py).</p>
<h3>Configuration</h3>
<pre><code class="language-conf">replicaof &lt;master-ip&gt; &lt;master-port&gt;   # persistent form; CLI REPLICAOF is not
replica-read-only yes                 # rejects writes on replicas — prevents divergence
repl-diskless-sync yes                # stream RDB over network, no disk (more primary RAM)
repl-diskless-sync-delay 5            # batch multiple replicas into one sync
repl-backlog-size 10mb                # partial-resync window; too small = frequent full resyncs
repl-timeout 60                       # drop dead link after this long
</code></pre>
<h3>Durability &amp; Safety</h3>
<ul>
<li><code>min-replicas-to-write 1</code> + <code>min-replicas-max-lag 10</code>: primary <strong>refuses writes</strong> unless 1 replica is connected and within 10s lag.</li>
<li><strong>Availability trade</strong> — healthy primary goes read-only if replicas die.</li>
<li><code>requirepass</code>: clients auth to this node. <code>masterauth</code>: replica auths to primary. <strong>Need both.</strong></li>
</ul>
<h3>Sentinel (automatic failover)</h3>
<p>Standard replication alone can't fail over — you'd manually promote with <code>REPLICAOF NO ONE</code>.</p>
<p><strong>Sentinel</strong> automates: detect failure → elect replica → promote → reconfigure old primary when it returns.</p>
<ul>
<li><strong>Quorum:</strong> odd number, at least 3 Sentinels, so they agree the master is dead.</li>
<li><strong>Client requirement:</strong> app must use a Sentinel-aware client (Lettuce, ioredis, redis-py) to ask "who is master now?"</li>
</ul>
<hr />
<h2>Way 2: Redis Cluster (Horizontal Scaling)</h2>
<p>Use this when one machine isn’t enough for writes or memory. Keys are split across 16,384 hash slots using CRC16, and each master owns a range of them. You can talk to any node, and if you hit the wrong one it answers with <code>-MOVED</code> and your client retries on the right one. Failover is built in, so you don’t need Sentinel here.</p>
<h3>Core Concepts</h3>
<ul>
<li><strong>16,384 hash slots.</strong> Key → slot via <code>CRC16(key) mod 16384</code>. Not consistent hashing.</li>
<li><strong>Multiple masters</strong>, each owning a subset of slots (3 masters → ~5,461 slots each).</li>
<li><strong>Automatic routing:</strong> client can hit any node; wrong node returns <code>-MOVED &lt;slot&gt; &lt;ip:port&gt;</code> and the client re-issues to the correct one.</li>
<li><strong>Built-in failover:</strong> masters have replicas; on master death the cluster votes to promote the replica, which inherits the same slots.</li>
<li><strong>Hash tags</strong> <code>{...}</code><strong>:</strong> Redis hashes only the text inside braces. <code>{user100}:name</code> and <code>{user100}:age</code> land in the same slot — required for multi-key ops, <code>MULTI</code>, Lua.</li>
</ul>
<pre><code class="language-bash"># config template per port
port PORT
cluster-enabled yes
cluster-config-file nodes-PORT.conf
cluster-node-timeout 5000
appendonly yes
protected-mode no
dir .

# create 3 masters + 3 replicas
redis-cli --cluster create \
  127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 \
  127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005 \
  --cluster-replicas 1

# verify
redis-cli -c -p 7000 cluster info     # cluster_state:ok
redis-cli -c -p 7000 cluster nodes    # roles, IDs, slot ranges
redis-cli -c -p 7000 cluster keyslot user:1001
</code></pre>
<p><code>redis-cli</code> <strong>decides the master/replica split for you, then writes that role assignment into each node's metadata file (</strong><code>nodes.conf</code><strong>).</strong> There is no pre-existing "master flag" on any of those six servers — they all start as blank Redis instances.</p>
<h3>What <code>--cluster-replicas 1</code> actually means</h3>
<p>It says: for every master created, allocate <strong>one replica</strong>. With six nodes and this flag, the tool computes <code>6 / (1 + 1) = 3</code> masters, and the remaining 3 become replicas of those masters .</p>
<p>So the command produces:</p>
<ul>
<li><strong>3 masters</strong> (ports 7000, 7001, 7002 in the default layout)</li>
<li><strong>3 replicas</strong> (ports 7003, 7004, 7005), each following one master</li>
</ul>
<blockquote>
<h3>How the tool picks who is master vs replica</h3>
<p>The tool does <strong>not</strong> just take the first three as masters and the last three as replicas blindly. It runs an <strong>anti-affinity optimization</strong> first: it tries to avoid placing a master and its replica on the same physical host (based on the IP addresses you give). If you pass six <code>127.0.0.1</code> addresses, that constraint is satisfied because they're all "the same host," so the assignment becomes effectively positional — first half masters, second half replicas.</p>
<p>If you passed three different IPs with two ports each, the tool would try to ensure each master's replica sits on a <strong>different IP</strong> than the master, so a single machine failure doesn't take out both a master and its backup .</p>
</blockquote>
<hr />
<h1>Redis Diagnostics &amp; Monitoring</h1>
<p>Redis gives you four tools for debugging, and each one answers a different question. INFO tells you if something is wrong. MONITOR shows what commands are flowing. SLOWLOG shows which ones are slow. CLIENT LIST tells you who is sending them. Used in that order, they narrow the problem down.</p>
<table>
<thead>
<tr>
<th>Tool</th>
<th>Question</th>
<th>Granularity</th>
</tr>
</thead>
<tbody><tr>
<td><strong>INFO</strong></td>
<td>"What's the overall state? Is something off?"</td>
<td>Server-wide snapshot</td>
</tr>
<tr>
<td><strong>MONITOR</strong></td>
<td>"What commands are flowing <em>right now</em>?"</td>
<td>Every command, real time</td>
</tr>
<tr>
<td><strong>SLOWLOG</strong></td>
<td>"Which commands took too long?"</td>
<td>Filtered by execution time</td>
</tr>
<tr>
<td><strong>CLIENT LIST</strong></td>
<td>"<em>Who</em> is sending these commands?"</td>
<td>Per-connection</td>
</tr>
</tbody></table>
<hr />
<h3>1. INFO — Server-Wide Snapshot</h3>
<p>Built-in command returning a categorized dump of server stats and operational state.</p>
<pre><code class="language-bash">redis-cli INFO              # all sections
redis-cli INFO memory       # one section — cuts noise
redis-cli INFO replication  # replication status
</code></pre>
<p><strong>Sections:</strong> Server (version, uptime), Clients (connected, blocked), Memory (<code>used_memory</code>, <code>maxmemory</code>, fragmentation), Persistence (RDB/AOF status), Stats (hits/misses, commands processed), Replication (role, sync), CPU, Modules, Errorstats (flags misconfigs), Cluster, Keyspace (key count per DB).</p>
<p><strong>Why it matters:</strong> detect memory leaks, tune eviction policy, decide when to scale, quick health check.</p>
<p><strong>Key metrics to watch:</strong></p>
<ul>
<li><code>used_memory</code> vs <code>maxmemory</code> → headroom / eviction risk</li>
<li><code>mem_fragmentation_ratio</code> → memory efficiency</li>
<li><code>connected_clients</code> → connection pressure</li>
<li><code>keyspace_hits</code> / <code>keyspace_misses</code> → cache effectiveness</li>
<li><code>rejected_connections</code> → overload signal</li>
</ul>
<hr />
<h3>2. MONITOR — Live Command Stream</h3>
<p>Streams <strong>every command</strong> the server processes, in real time.</p>
<pre><code class="language-bash">redis-cli MONITOR    # Ctrl+C to stop
</code></pre>
<p>Each entry: <strong>timestamp</strong>, <strong>client IP + port</strong>, <strong>command + full arguments</strong>.</p>
<p><strong>Use cases:</strong> real-time debugging, diagnose unexpected behavior, verify commands reach Redis as expected.</p>
<p>A word of caution on MONITOR: it prints every command, including passwords sent with AUTH, and it slows down a busy server. Use it for short debugging sessions only, never leave it running on a production primary.</p>
<hr />
<h3>3. SLOWLOG — Capture Slow Commands</h3>
<p>Records commands exceeding a configurable threshold — for finding bottlenecks.</p>
<pre><code class="language-bash">SLOWLOG GET [count]   # retrieve entries
SLOWLOG LEN           # count
SLOWLOG RESET         # clear log
</code></pre>
<p>Each entry: <strong>ID</strong>, <strong>execution time in microseconds</strong>, <strong>command + args</strong>, timestamp, client ID/IP.</p>
<p><strong>Configuration:</strong></p>
<table>
<thead>
<tr>
<th>Parameter</th>
<th>Meaning</th>
<th>Example</th>
</tr>
</thead>
<tbody><tr>
<td><code>slowlog-log-slower-than</code></td>
<td>Threshold in <strong>microseconds</strong></td>
<td><code>10000</code> = 10 ms</td>
</tr>
<tr>
<td><code>slowlog-max-len</code></td>
<td>Ring-buffer size</td>
<td><code>128</code></td>
</tr>
</tbody></table>
<p><strong>Special values for threshold:</strong> <code>0</code> = log everything; <code>-1</code> = disable.</p>
<p><strong>Notes:</strong></p>
<ul>
<li>Storage is <strong>bounded</strong> — oldest entries silently dropped.</li>
<li>Only commands above threshold are recorded — routine traffic doesn't clutter it.</li>
<li><code>SLOWLOG RESET</code> clears logged data only; no effect on execution.</li>
</ul>
<hr />
<h3>4. The Drill-Down Workflow (the real lesson)</h3>
<p>Say memory keeps climbing. INFO confirms it (<code>used_memory</code> going up), but not why. MONITOR shows which commands are hitting Redis, but not whether any are slow. SLOWLOG tells you if it’s a slow-command problem or just heavy load. Finally CLIENT LIST shows which client is doing it. By then you know who the culprit is.</p>
<hr />
<blockquote>
<h3><strong>Redis looks simple when you start with SET and GET, but there’s a lot going on under the hood. Play around with replication and Cluster in a lab before you touch production, because breaking things on purpose is the best way to learn them.</strong></h3>
</blockquote>
]]></content:encoded></item><item><title><![CDATA[Database Security]]></title><description><![CDATA[In this article we will cover the security aspect in regards to databases as per CIS benchmarks (CIS Benchmarks are globally recognized, consensus-driven…]]></description><link>https://dbbyyaz.hashnode.dev/database-security</link><guid isPermaLink="true">https://dbbyyaz.hashnode.dev/database-security</guid><dc:creator><![CDATA[Yazdan Haider]]></dc:creator><pubDate>Fri, 13 Nov 2026 17:02:39 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/0e0f25e2-1b5d-493c-b45f-6e2caf4d338b.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>In this article we will cover the security aspect in regards to databases as per CIS benchmarks (<strong>CIS Benchmarks</strong> are <strong><mark>globally recognized, consensus-driven security configuration best practices developed by the</mark></strong> <a href="https://www.cisecurity.org/cis-benchmarks"><strong><u><mark class="bg-highlight text-highlight-foreground">Center for Internet Security (CIS)</mark></u></strong></a>).</p>
<p>So lets start this:</p>
<h2>Operating System Level Configuration</h2>
<p>Keep your database files off the system partition (<code>/</code>, <code>/var</code>, <code>/usr</code>). Why? Because if MySQL fills up the disk, it doesn't just take down MySQL — it takes down <code>systemd</code>, logging, SSH, everything that needs to write to <code>/</code>. That's a denial-of-service waiting to happen.</p>
<p>Put the data on its own partition and <strong>always a dedicated server for database which is in private or internal subnet and can only be accessed by applications or services that need to or authorized to communicate with it</strong>, and a runaway table or binlog only kills the thing it should kill. Blast radius contained.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/27aa6359-3e0a-40ac-b55f-3effeb13634d.png" alt="" style="display:block;margin:0 auto" />

<p>As you can see that it is default location so it is on the root / volume we will change it.</p>
<ul>
<li><p>First we take backup of our DB.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/8982c1d6-fde9-4b11-b155-4e956a54f2fa.png" alt="" style="display:block;margin:0 auto" /></li>
</ul>
<ul>
<li><p>As you can see the backup has been taken now</p>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/c33b3a25-7535-4864-ac99-d6b230e3f0c8.png" alt="" style="display:block;margin:0 auto" /></li>
</ul>
<p>Cant create a logical volume on my VM since It is part of my homelab and my disk almost free will cover the volumes part in separate linux series.</p>
<pre><code class="language-shell">sudo systemctl stop mysql                # Stop the Mysql Service
pgrep -a mysqld                          # should be empty
sudo cp -rp /var/lib/mysql /data/mysql   # Copy the files from the default dir
sudo chown -R mysql:mysql /data/mysql    # Change ownership of the dirs in new vol
sudo chmod 750 /data/mysql               # Assign permissions (user=rwx,group=rx,others=---)
</code></pre>
<p>After this step you have to make the mysql process read the new data dir by modifying its configuration file</p>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/5e1e0527-f2f7-4de4-9818-024043c5baac.png" alt="" style="display:block;margin:0 auto" />

<p>After this you may have to manage the Selinux contexts for the new volume for security if using a rhel based distro.</p>
<h2>1. Use a Dedicated Least Privileged Account</h2>
<p>Now we're tackling something that is absolutely foundational in Linux security: <strong>making sure the database service doesn't run as</strong> <code>root</code>.</p>
<p>MySQL has serveral privileges make sure you give them according to user's authorized access and not grant them unecessary privileges:</p>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/c128e34c-151d-4cf9-bcf9-8be749ca7830.png" alt="" style="display:block;margin:0 auto" />

<p>Imagine an attacker finds a SQL injection vulnerability in your app that allows them to write files. If MySQL is running as <code>root</code>, they just wrote a web shell to <code>/var/www/html/</code> and owned your entire server. If MySQL is running as the <code>mysql</code> user with no shell access, they are trapped in a sandbox. That's the blast-radius reduction we want:</p>
<p>Lets check on my server:</p>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/4c192aa7-9676-4351-8299-4bbe7c402b9f.png" alt="" style="display:block;margin:0 auto" />

<p>So the first check is passed as you can see the user of mysqld process is the mysql user.</p>
<p>we will run the following command to see if the user has access to interactive shell or not:</p>
<pre><code class="language-sql">sudo grep '^mysql:' /etc/passwd
</code></pre>
<p><strong>The shell is locked down</strong> (<code>/sbin/nologin</code>). If a hacker exploits a bug in the database and tries to drop into a command prompt, the OS will instantly kick them out.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/5d30b613-d04f-4bc3-b637-761d979307e7.png" alt="" style="display:block;margin:0 auto" />

<p>Or you can use mysql builtin plugin to make a user with nologin. You can use the following commands for that:</p>
<pre><code class="language-sql">CREATE USER 'proddbuser'@'localhost' IDENTIFIED WITH mysql_no_login;
</code></pre>
<pre><code class="language-sql">GRANT ALL PRIVILEGES ON my_database.* TO 'proddbhost'@'localhost';
</code></pre>
<hr />
<h2>2. Disable Command History</h2>
<p>Every time you type a command into the MySQL shell, the client secretly writes it down in a hidden file in your home directory (<code>~/.mysql_history</code>). Why is this a problem? Because DBAs are human, and humans type things like:</p>
<p><code>CREATE USER 'admin'@'localhost' IDENTIFIED BY 'SuperSecretPassword123!';</code></p>
<p>Congratulations, you just wrote your admin password in plaintext to a file on the disk. If an attacker gets local access to the OS, they are absolutely going to run <code>cat /root/.mysql_history</code> to see what you've been up to.</p>
<p>The fix is to simply redirect this history file to <code>/dev/null</code> (the Linux black hole) so it never gets saved.</p>
<p>But latest versions of mysql have this feature builtin.</p>
<hr />
<h2>3. Stop Using MYSQL_PWD</h2>
<p>You can pass a password to the MySQL client like this:<br /><code>export MYSQL_PWD="SuperSecretPassword"; mysql -u root</code></p>
<p>It’s incredibly convenient for scripts because you don't have to deal with interactive prompts. But as a security engineer, I hate it. Why? Because <strong>any user on the system who can read your process environment can steal the password.</strong></p>
<p>On Linux, environment variables are exposed in <code>/proc/&lt;PID&gt;/environ</code>. If a backup script runs with <code>MYSQL_PWD</code>, or if you leave it set in your SSH session, an attacker who gets a low-privileged shell can just read it. It's plaintext credential exposure.</p>
<p>Which is also the built in security feature already handled by mysql:</p>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/15a733e5-3302-453d-a7b6-e53c942121a7.png" alt="" style="display:block;margin:0 auto" />

<p>I ran the CIS audit command (<code>grep MYSQL_PWD /proc/*/environ</code>) and a secondary check using <code>ps eww -u mysql</code>. Both returned clean. The MySQL daemon is not leaking plaintext credentials in its environment, and no active scripts are using it.</p>
<hr />
<h1>BACKUP AND DISASTER RECOVERY:</h1>
<h2>1 . Regular Backups:</h2>
<p>We need to have database backups consistently for disaster recovery and incident response this can be done using crontab (I do this) or one of the following methods:</p>
<ul>
<li>For enterprise workloads, <strong>pgBackRest</strong> delivers reliable physical and Write-Ahead Log (WAL) backups with encryption, compression, and fast delta restores.</li>
</ul>
<ul>
<li>If you are pushing archives directly to cloud storage like AWS S3 or Google Cloud, <strong>WAL-G</strong> offers ultra-low latency.</li>
<li>For smaller datasets or granular table recovery, PostgreSQL's built-in <strong>pg_dump</strong> and <strong>pg_restore</strong> utilities remain simple and effective.</li>
<li><strong>Percona XtraBackup</strong> is the open-source industry standard for non-blocking "hot" backups on InnoDB, letting you capture incremental changes and recover quickly without taking your database offline.</li>
<li>For smaller transactional databases, standard logical tools like <strong>mysqldump</strong> and <strong>mysqlpump</strong> get the job done easily with bash scripting and cron.</li>
<li>For Windows-based SQL environments, third-party automation and snapshot platforms like <strong>Veeam</strong> can help in managing <code>.bak</code> files for automated schedules.</li>
</ul>
<h2>2. Secure Your Backups</h2>
<p>After we successfully configure and run our backups, the next critical step is securing them. Think about what a database backup actually is: it’s a complete, consolidated copy of your entire database—every table, every schema, every password hash, and every piece of sensitive customer data, all sitting in one place.</p>
<p>If a backup file gets compromised, the attacker essentially has the keys to the kingdom. They can exfiltrate the data, or worse, hold it for ransom. Even if they don't get your live server, they still have your data.</p>
<p>Now, here is a frustrating reality: you absolutely cannot give non-interactive access to a backup user if you are automating things via cron or using CLI-based tools like WAL-G, mysqldump, or pg_basebackup.</p>
<p>Why? Because automation requires the script to run without a human typing a password. To make this work, people usually hardcode credentials into a script or a plain-text .my.cnf file. If an attacker gets into your server, the first thing they will look for is your backup script—because they know it holds the master password to dump the entire database.</p>
<p>So, how do we solve this? We have to separate the creation of the backup from the storage of the backup, and we have to lock it down with both strict file permissions and strong encryption.</p>
<ol>
<li>Lock Down the Directory Never store backups in /tmp or /home. The backup directory itself should be owned by root (or a dedicated, locked-down backup account) and restricted with chmod 700. This ensures no other user on the OS can even see the files, let alone read them.</li>
<li>Encrypt at Rest (and in Transit) Even with 700 permissions, if an attacker gets root, they get the backup. You must encrypt the backup stream. Using OpenSSL or GPG, we can pipe the output of mysqldump directly into an encryption engine. This way, the backup is encrypted before it ever hits the disk.</li>
<li>We can and should also backup configuration files and SSL files.</li>
</ol>
<h2>3. Point-in-Time Recovery (PITR)</h2>
<p>Lets take an example that your write a cron script or schedule your backups at midnight 12:00 now what if a hardworking developer decides to work on his task late at night 3:00 AM and meanwhile in this process he out of an error copy pastes a mysql query which deletes the table users.</p>
<p>Now of course the first thing would be to restore the backup of 12:00 AM right.</p>
<p>WRONG</p>
<p>Because what about the users that were created after midnight thats where the binlog comes in:</p>
<blockquote>
<h3>Binlog: The "Archive" at the Server Level</h3>
<p><strong>Binlog</strong> belongs to the <strong>MySQL Server level</strong>, not the InnoDB engine level. Regardless of which storage engine you use, as long as binlog is enabled, MySQL records writes.</p>
<ul>
<li><strong>Logical log</strong>: It records specific SQL statements (like <code>UPDATE users SET ...</code>) or row-level changes, used for <strong>replication</strong> and <strong>point-in-time recovery (PITR)</strong> .</li>
<li>It and the redo log coordinate through <strong>two-phase commit</strong> to ensure that the binlog and the actual data remain consistent after a crash.</li>
</ul>
</blockquote>
<p>You will first restore the latest <strong>backup</strong> then:</p>
<h2>4. Recovery (PITR) Using Binlogs</h2>
<h3>Step 1: Find the "Disaster" Position in the Binlog</h3>
<p>Suppose you know the accidental deletion happened around <code>2026-xx-xx 03:00:00</code>. We need to find the exact byte position of that bad <code>DELETE</code> command in the binlog.</p>
<p>Use the <code>mysqlbinlog</code> tool to inspect the log content around that timeframe. We use <code>grep -C 15</code> to show 15 lines of context before and after the keyword:</p>
<pre><code class="language-bash">mysqlbinlog --start-datetime="2026-xx-xx 12:00:00" \
            --stop-datetime="2026-xx-xx 02:59:00" \
            --verbose /var/lib/mysql/binlog.000012 | grep -C 15 "DELETE"
</code></pre>
<p>In the output, you will see a line that looks like <code># at 232</code> right before the bad statement. <strong>This is the position where we must stop the replay.</strong></p>
<blockquote>
<p><strong>Note:</strong> We use <code>--start-datetime</code> and <code>--stop-datetime</code> <em>only</em> for locating the event. It is highly recommended <strong>not</strong> to use timestamps for the actual replay, because overlapping events or clock drifts can cause you to miss transactions. Always use exact positions.</p>
</blockquote>
<h3>Step 3: Execute the Replay</h3>
<p>Now we bring the database back to life. We will pipe the binlog into the MySQL client using <code>--start-position</code> (where the backup ended) and <code>--stop-position</code> (where the disaster began).</p>
<p>PITR is a superpower, but it requires <strong>binary logging to be enabled</strong> and <strong>binlogs to be backed up offsite</strong>. First set <code>log_bin = ON</code> and <code>binlog_expire_logs_seconds</code> to a safe number in the cnf file.</p>
<hr />
<h1>Identity and Access Management (IAM)</h1>
<h2>1. Password Expiration, Lockouts, and Socket Auth</h2>
<p><strong>Why it matters:</strong> Password expiration forces users to change their credentials periodically, limiting the window of opportunity if an account is compromised (e.g., through a data breach at a third-party site).</p>
<p>In MySQL, the global variable is <code>default_password_lifetime</code>. Run this query:</p>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/6d08e8b7-0cad-4bca-9cfc-77eba540affc.png" alt="" style="display:block;margin:0 auto" />

<ul>
<li>The value is <code>&gt; 0</code> but <code>&lt;= 365</code> (e.g., <code>365</code> means passwords expire in 1 year).</li>
<li>The value is <code>0</code> (never expires) or greater than <code>365</code> (Wrong move).</li>
</ul>
<p>You can set this globally at runtime, but it's safer to put it in the config file so it survives a reboot.</p>
<p>Add under <code>[mysqld]</code> section:</p>
<pre><code class="language-sql">default_password_lifetime=365
</code></pre>
<p>Restart MySQL, or run the SQL command: <code>SET GLOBAL default_password_lifetime=365;</code></p>
<p>To enforce it on a specific user (overriding the global default):</p>
<pre><code class="language-sql">ALTER USER 'app_user'@'localhost' PASSWORD EXPIRE INTERVAL 365 DAY;
</code></pre>
<hr />
<h2>2. Lock Out Accounts if Not Currently in Use</h2>
<p><strong>Why it matters:</strong> Dormant, unused accounts are a massive security risk. They often have outdated passwords and aren't monitored. If an attacker finds a dormant <code>admin</code> account, they can use it without raising suspicion.</p>
<p>To see which accounts are locked:</p>
<pre><code class="language-sql">SELECT User, Host, account_locked 
FROM mysql.user 
WHERE account_locked = 'Y';
</code></pre>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/d2faa3ee-d500-4c98-bb21-2ac03afb24ec.png" alt="" style="display:block;margin:0 auto" />

<p>If you find stale accounts or users that need temporary suspension, lock them so they cannot authenticate. Do not delete them until you are absolutely sure they aren't needed (to preserve their grant history).</p>
<pre><code class="language-sql">-- To lock an account:
ALTER USER 'demo'@'localhost' ACCOUNT LOCK;

-- To unlock an account:
ALTER USER 'demo'@'localhost' ACCOUNT UNLOCK;
</code></pre>
<p><em>Security Tip:</em> It is a best practice to immediately <code>LOCK</code> an account when you create it ahead of time, and only <code>UNLOCK</code> it once the user is ready to start.</p>
<hr />
<h1>Ensure Socket Peer-Credential Authentication is Used Appropriately (Manual)</h1>
<p><strong>Why it matters:</strong> The <code>unix_socket</code> authentication plugin allows local users to log in to MySQL using their <strong>Linux OS username</strong>—without typing a MySQL password. It checks the OS credentials and grants access.</p>
<p>This is fantastic for DBAs because it eliminates plaintext passwords (like <code>MYSQL_PWD</code>) and relies on the OS security model. However, if an attacker gets root OS access, they automatically get root MySQL access.</p>
<p>First, check if the plugin is active:</p>
<pre><code class="language-sql">SELECT PLUGIN_NAME, PLUGIN_STATUS
FROM INFORMATION_SCHEMA.PLUGINS
WHERE PLUGIN_NAME = 'auth_socket';
</code></pre>
<p>To see which MySQL users are using socket authentication:</p>
<pre><code class="language-sql">SELECT User, Host, plugin 
FROM mysql.user 
WHERE plugin = 'auth_socket';
</code></pre>
<ul>
<li><strong>The Pass/Fail:</strong> If your organization mandates passwords for <em>all</em> DB logins, and this plugin is active, it’s a <strong>FAIL</strong>. If you are using this for local DBA logins to harden the server, it is a <strong>PASS</strong> (provided the <code>Host</code> is restricted to <code>localhost</code>).</li>
</ul>
<h2>2. Bound to Specific IP Addresses</h2>
<p>By default, MySQL/MariaDB listens on <code>0.0.0.0</code> (all IPv4 interfaces) or <code>::</code> (all IPv6 interfaces). This means your database is accepting TCP connections on <em>every</em> network card plugged into the server—including public-facing ones.</p>
<p>As a security engineer, if a database is listening on a public IP, it's only a matter of time before bots find it and start brute-forcing your <code>root</code> password. By binding the database to a specific, private IP address (like your internal LAN or VPC IP), you limit the attack surface. The database can only be reached if a client is explicitly routing traffic to that specific interface.</p>
<p>The CIS benchmark says it perfectly: <em>"Limiting the IP addresses provides additional controls and restrictions on how client applications can connect."</em></p>
<h2>How to check</h2>
<p>Run this query:</p>
<pre><code class="language-sql">SELECT VARIABLE_NAME, VARIABLE_VALUE 
FROM performance_schema.global_variables 
WHERE VARIABLE_NAME = 'bind_address';
</code></pre>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/697e7a02-c4c3-4f65-9fd7-f35481b4cffa.png" alt="" style="display:block;margin:0 auto" />

<p><strong>The Pass/Fail:</strong></p>
<ul>
<li>The value is a specific IP (e.g., <code>127.0.0.1</code>, <code>192.168.1.50</code>, or a comma-separated list of private IPs).</li>
<li>The value is empty, <code>*</code>, <code>0.0.0.0</code>, or <code>::</code> (Wrong).</li>
</ul>
<p>To see exactly what your server is listening on right now at the OS level, run:</p>
<pre><code class="language-bash">ss -tulpn | grep mysqld
</code></pre>
<p>If you see <code>0.0.0.0:3306</code> or <code>[::]:3306</code>, you are listening to the world. That's a fail.</p>
<hr />
<h2>3. The Cloud / Private Interface Rule</h2>
<p>If your server has multiple network interfaces (for example, one private IP for application traffic, and one public IP because you're on a cloud provider), <strong>you must bind to the private interface.</strong></p>
<p>If you have an on-prem replication setup over a VPN and a public IP, you might bind to both the private IP and the public IP, but only if the firewall restricts access to that public IP. In 99% of cases, you only want to bind to the private IP or localhost.</p>
<h3>Applying the Configuration</h3>
<ol>
<li>Open your MySQL config file:</li>
</ol>
<pre><code class="language-bash">sudo vi /etc/my.cnf.d/mysql-server.cnf
</code></pre>
<p><em>(On Ubuntu, this might be</em> <code>/etc/mysql/mysql.conf.d/mysqld.cnf</code><em>)</em></p>
<ol start="2">
<li>Find the <code>[mysqld]</code> section and add/change the <code>bind_address</code> line.</li>
</ol>
<p><strong>Scenario A: Localhost only (Application and DB on the same server)</strong></p>
<pre><code class="language-ini">bind_address=127.0.0.1
</code></pre>
<p><strong>Scenario B: Specific private IP (Application on a different server)</strong> Find your private IP using <code>ip a</code> (e.g., <code>192.168.1.50</code> or AWS private IP):</p>
<pre><code class="language-ini">bind_address=192.168.1.50
</code></pre>
<p><strong>Scenario C: Multiple specific IPs (Comma-separated)</strong></p>
<pre><code class="language-ini">bind_address=192.168.1.50,10.0.0.5
</code></pre>
<p>After this you can restart mysql service so that it can load the configuration file again.</p>
<hr />
<h1>Hardening TLS and Encryption</h1>
<h2>1. Limit Accepted TLS Versions</h2>
<p>TLS 1.0 and 1.1 are ancient. They have known vulnerabilities (like BEAST and POODLE) and are officially deprecated by every security standard (PCI-DSS, NIST, etc.). If your server accepts them, an attacker might force a downgrade attack. We want to strictly enforce TLS 1.2 and 1.3.</p>
<p>By default mysql now uses tlsv1.2 and v1.3</p>
<pre><code class="language-sql">SHOW VARIABLES LIKE 'tls_version';
</code></pre>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/1ca32eeb-6ffb-4e3c-94e8-3b20a1de1cc7.png" alt="" style="display:block;margin:0 auto" />

<pre><code class="language-ini">tls_version=TLSv1.2,TLSv1.3
</code></pre>
<hr />
<h2>2. Require Client-Side Certificates (X.509)</h2>
<p>Normally, when you connect to MySQL over TLS, the server proves its identity to the client. But what if we want to prove the <em>client's</em> identity? X.509 client certificates act as a second factor. The server will refuse the connection unless the client presents a valid, signed certificate.</p>
<p>Run this to see who requires a certificate:</p>
<pre><code class="language-sql">SELECT user, host, ssl_type 
FROM mysql.user 
WHERE user NOT IN ('mysql.sys', 'mysql.session', 'mysql.infoschema', 'root');
</code></pre>
<p><strong>The Pass:</strong> The <code>ssl_type</code> column shows <code>X509</code> for your application users. <strong>The Fail:</strong> The <code>ssl_type</code> is empty (meaning passwords alone are enough).</p>
<p><strong>We will work on this implementation on a separate blog article.</strong></p>
<hr />
<h2>3. Ensure Only Approved Ciphers are Used</h2>
<p>TLS versions are the protocol; Ciphers are the actual math used to scramble the data. Some ciphers are weak (like RC4 or 3DES) and can be cracked. We want to restrict MySQL to only use modern, strong ciphers (specifically AES-GCM or ChaCha20).</p>
<pre><code class="language-sql">SHOW VARIABLES LIKE 'ssl_cipher';
</code></pre>
<p><strong>The Pass:</strong> The value is a specific, strong cipher suite (e.g., <code>ECDHE-ECDSA-AES128-GCM-SHA256</code>). <strong>The Fail:</strong> The value is empty (allows any cipher the OS supports, including weak ones).</p>
<p>Open your config file:</p>
<pre><code class="language-bash">sudo vi /etc/my.cnf.d/mysql-server.cnf
</code></pre>
<p>Add under <code>[mysqld]</code> (this is a common, highly secure cipher string) This works only if you have configured tls/ssl certificates:</p>
<pre><code class="language-ini">ssl_cipher=ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256
</code></pre>
<blockquote>
<p>The checklist is done, but the takeaway isn't "I hardened MySQL." It's that every item above was a blast-radius decision: the data dir on its own volume so a full disk doesn't take down sshd; the <code>mysql</code> user with <code>/sbin/nologin</code> so a SQLi file-write lands in a sandbox; <code>bind_address</code> on the private NIC so bots never see port 3306; encrypted backups in a <code>chmod 700</code> root-owned dir so a compromise doesn't hand over the whole dataset. None of these stop a determined attacker alone — together they turn "own the server" into "own one thing, slowly, and get logged.</p>
</blockquote>
]]></content:encoded></item><item><title><![CDATA[Database Optimization]]></title><description><![CDATA[In this section we will be discussing various strategies for optimizing databases and what happens if we ignore them. Database optimization is a two-way…]]></description><link>https://dbbyyaz.hashnode.dev/database-optimization</link><guid isPermaLink="true">https://dbbyyaz.hashnode.dev/database-optimization</guid><dc:creator><![CDATA[Yazdan Haider]]></dc:creator><pubDate>Fri, 13 Nov 2026 13:45:28 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/7c728822-0bf1-45ed-b8fd-221c9e952ec0.jpg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>In this section we will be discussing various strategies for optimizing databases and what happens if we ignore them.</p>
<p>Database optimization is a two-way street. You can write the cleanest application code in the world, or tune your database configuration to perfection, but if you ignore one side of the equation, your app will still crawl.</p>
<h2>Splitting Data: Horizontal vs. Vertical</h2>
<p>When a single database server starts groaning—queries take seconds instead of milliseconds, disks fill up, and connections max out—adding another index won't save you. Think of a giant spreadsheet: there are two ways to tear it apart.</p>
<pre><code class="language-plaintext">Horizontal Partitioning (Splitting by Rows): You keep the exact same columns, but divide the rows across smaller tables. You can split by Range (e.g., dates or sequential IDs), List (e.g., specific regions or states), or Hash (running an ID through a mathematical function to distribute rows evenly).

Vertical Partitioning (Splitting by Columns): You separate heavy columns from frequently accessed ones. Instead of loading a 5MB profile picture every time you check a user's email address, you store the lightweight metadata in one table and isolate the heavy blobs in another.
</code></pre>
<h2>1. Partitioning vs. Sharding: Where the Pieces Live</h2>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/a79405aa-d44a-4249-8090-932f5d0dd3e3.jpg" alt="" style="display:block;margin:0 auto" />

<p>Both techniques split large tables into smaller pieces, but they differ completely in infrastructure and complexity we will follow along with hands on so you better understand the concepts. The following is a table on my database with sample dataset which we will practice on:</p>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/5f221046-ff85-49c1-8e3e-5e2a3509d4a1.png" alt="" style="display:block;margin:0 auto" />

<pre><code class="language-plaintext">Partitioning (Same Server, Invisible to the App): Your table is split into smaller physical sub-tables on the exact same machine. Your application still queries one logical table name, and the database engine automatically routes the request to the right underlying partition.

Sharding (Different Servers, App-Aware): Your data is split across completely separate physical servers. Now your application has to become smart—it needs routing logic to know which server holds which user.
</code></pre>
<h3>Why Indexes Still Matter on Partitioned Tables</h3>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/bac66072-45ca-4b96-acbd-c8450a192c4f.png" alt="" style="display:block;margin:0 auto" />

<p>Databases do not automatically index regular columns—only Primary Keys get indexed out of the box. If you partition a table by date, and then query by a user's name without an index on that column, the database has to scan every single partition from top to bottom. Furthermore, in older database versions or certain engines, creating an index on a parent table doesn't always propagate to child partitions automatically—always verify that every child partition carries the indexes your queries rely on.</p>
<p>Lets look at what happens if we perform a query on the prepartitioned salary table:</p>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/f0ed6939-2fb2-4d9e-bc37-ac0209fff9c8.png" alt="" style="display:block;margin:0 auto" />

<p>As expected you can see from screenshot it loaded the whole table or in DBA words it scanned the whole table aka loaded it into memory for reading.</p>
<p>Lets create an index on this table:</p>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/99ef03ba-4f81-4ba5-afdb-bbe9ac60bc5d.png" alt="" style="display:block;margin:0 auto" />

<p>Look what happens if we run the same query again:</p>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/00cdd084-4e08-42df-8d16-33cb21920336.png" alt="" style="display:block;margin:0 auto" />

<p>This time it didnt load the whole table instead it just had to load a specific page containing our row into memory since data is stored in fixed sized pages which are of 16kb size and fetch our rows. Since a index is a separate file that tracks our rows for the column on which index is present.</p>
<h2>Pros and Cons of Partitioning</h2>
<p>Why it shines:</p>
<pre><code class="language-plaintext">Partition Pruning: Querying a specific month allows the database to skip every other partition completely.

Sequential Disk Reads: Because data from the same period is stored together physically, disk reads stay smooth and sequential instead of jumping randomly across years of records.

Instant Archiving &amp; Bulk Loads: You can load millions of rows into a standalone staging table, build its indexes offline, and snap it into the live table in milliseconds—or drop an entire year's partition instantly without locking rows.
</code></pre>
<p>Where it hurts:</p>
<pre><code class="language-plaintext">Moving Rows Across Partitions: Updating a record's partition key forces the database to delete it from one partition and insert it into another. Doing this in bulk can lock up your database.

Scatter Scans: Querying by a column other than the partition key forces the engine to search every partition, which is often slower than searching one unpartitioned table.
</code></pre>
<h2>2. Deadlocks and Two-Phase Locking (2PL)</h2>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/e11e5594-0903-475c-a623-b1fba13ffb23.jpg" alt="" style="display:block;margin:0 auto" />

<p>Deadlocks happen when two transactions lock resources in opposite orders and wait forever for each other to let go. For example, Transaction A inserts item 21 and tries to insert item 20, while Transaction B inserts item 20 and tries to insert item 21. Even though neither has committed yet, both hold internal transaction locks to verify unique constraints—creating a circular wait until the database kills one of them.</p>
<p><strong>How to prevent it:</strong> Always acquire locks and modify records in a consistent, deterministic order across your application.</p>
<p>This behavior stems from Strict Two-Phase Locking (2PL), the mechanism databases use to guarantee transaction isolation. Under Strict 2PL, a transaction enters a Growing Phase where it acquires all necessary locks, and it is forbidden from releasing any of them until the entire transaction commits or rolls back. Without this rule, another user could read half-finished changes before a crash triggers a rollback.</p>
<p>Lets see it in action:</p>
<ul>
<li><p>We will use two separate terminal windows connected to the database server to run overlapping transactions.</p>
<p>In <strong>Terminal 1</strong>, we start a transaction and update a row:</p>
</li>
</ul>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/1ee68701-8ea9-4a5e-8cb5-8b17fe65dbd3.png" alt="First transaction" style="display:block;margin:0 auto" />

<p>In <strong>Terminal 2</strong>, we start our own transaction and update a different row:</p>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/cf9068bf-e243-42c3-bb5b-304ff0dd7aab.png" alt="" style="display:block;margin:0 auto" />

<p>Now, let’s see what happens if <strong>Terminal 2</strong> tries to update the <em>same</em> row that Terminal 1 already touched (<code>dept_no = 'd002'</code>):</p>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/034fb72b-9c4b-424c-be44-d43eb29714e0.png" alt="" style="display:block;margin:0 auto" />

<p><strong>Uh-oh.</strong> The query hangs, and eventually fails with this error:</p>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/9be9549a-85dc-4f02-af49-aa49be6266de.png" alt="" style="display:block;margin:0 auto" />

<p>Why did this happen? Because Terminal 1 already grabbed an <strong>Exclusive Lock</strong> on that row and hasn't committed or rolled back yet. Terminal 2 is forced to wait in line. Once it waits longer than the <code>innodb_lock_wait_timeout</code> limit (50 seconds by default), MySQL gives up and bails out.</p>
<ul>
<li><p>Look at what happens when we complete the cross-over: Terminal 1 is waiting on Terminal 2's row, and now Terminal 2 tries to grab Terminal 1's row (<code>d001</code>).</p>
<p>MySQL instantly detects the circular wait and kills the query with:</p>
</li>
</ul>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/a9cb83a0-50df-4142-9a2a-0c253522a335.png" alt="" style="display:block;margin:0 auto" />

<p>The database doesn't let both transactions hang forever. It sacrifices one of them—rolling it back automatically—so the other one can finish its work.</p>
<h2>3. The OFFSET Pagination Trap</h2>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/88cf4876-01cb-4723-9483-38752a9a4cb9.jpg" alt="" style="display:block;margin:0 auto" />

<p>Traditional pagination asks the database to skip a set number of rows before returning the next page. Asking for page 10 is harmless. Asking for page 50,000 forces the database to read half a million rows from disk, load them into memory, throw all of them in the trash, and hand you the final ten. As your dataset grows, deep pagination spikes CPU usage to 100%.</p>
<p>The fix is <strong>Keyset (Cursor) Pagination</strong>: Instead of telling the database how many rows to skip, tell it the ID or timestamp of the last item the user saw. Using a B-Tree index, the database jumps straight to that exact spot and reads the next ten rows—zero wasted work.</p>
<p>Let’s see it in action, shall we?</p>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/0f88971a-2800-4685-8c23-cc5a738d1d09.png" alt="" style="display:block;margin:0 auto" />

<p>So our 100,001st row has this employee number (<code>20541</code>).</p>
<p>Let's check what this cost us:</p>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/cefec54e-6e87-43bd-b479-8379044f483a.png" alt="" style="display:block;margin:0 auto" />

<ul>
<li>It loaded 100,000 rows just to discard them and give us the next 10.</li>
</ul>
<h3>The Real-World Impact</h3>
<p>Let’s assume you have a blog or social media website with posts.</p>
<p>If we used offset pagination here, we would load all your post records every time a user navigated to the next page. Now consider you have 10 users visiting your blog, each browsing through 100,000 posts. Got the idea? It causes unnecessary strain on the database.</p>
<p>Now, what if your application instead said: <em>"Let's check what the last post I loaded on the previous page was (ID</em> <code>20541</code><em>). Let's load the next 10 straight from that row onto this page"</em>?</p>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/29fe9c5e-fb9f-47a9-91b3-f166511ee630.png" alt="" style="display:block;margin:0 auto" />

<h3>What Happened Here:</h3>
<ul>
<li><strong>Index range scan (The bottom line):</strong> Look at the text carefully: <code>over (20541 &lt; emp_no)</code>. MySQL used the B-Tree index like a phonebook. It jumped <em>directly</em> to <code>emp_no = 20541</code> and started reading from there. It only read <strong>10 rows</strong> (<code>rows=10</code>) and took <strong>0.834 milliseconds</strong>.</li>
<li><strong>Limit (The top line):</strong> It grabbed the 10 rows and stopped.</li>
<li><strong>Total Time:</strong> 0.838 milliseconds.</li>
</ul>
<h2>4. Connection Pooling: The VIP Club Gatekeeper</h2>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/cead017c-1424-4588-8a1b-c7203dcf02e4.png" alt="" style="display:block;margin:0 auto" />

<p>Opening a new database connection is expensive. A Connection Pool acts like a nightclub bouncer with a strict capacity limit (say, 20 active connections). When 10,000 requests hit at once, they queue up cleanly while those 20 connections cycle through fast queries in milliseconds.</p>
<p>Connection pooling is handled by your application using a client library to set up a pool of active connections, which are then assigned dynamically as incoming requests arrive. If the pool is maxed out—for instance, if you set your application's pool limit close to MySQL's <code>max_connections</code> ceiling—any incoming request beyond that limit has to wait in a queue until a connection frees up.</p>
<p>However, a connection pool is a shock absorber, not a magic wand. Simply cranking your pool size up to 500 when MySQL only allows 300 connections will crash your server.</p>
<p>True scale comes from reducing how often requests hit the primary database: putting Redis (covered in a separate article) in front for hot reads, and offloading read traffic to read replicas.</p>
<h2>5. Bloom Filters: Saving Disk Trips with Bits in RAM</h2>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/c90ef86f-bd93-4c72-ac29-691c1eccacdc.jpg" alt="" style="display:block;margin:0 auto" />

<p>Before checking a massive disk table to see if a username is already taken, you can check a Bloom Filter in RAM. Picture a wall of millions of tiny light switches, all starting at 0. When a user signs up, their username is passed through a few hash functions, flipping specific switches to 1.</p>
<pre><code class="language-plaintext">If you check a new username and any of its switches is 0, that username definitely does not exist—you never touch the database.

If all switches are 1, the username might exist (or other names happened to flip those same switches), so you verify against the database.
</code></pre>
<p>Tracking a million strings takes roughly 1.2 MB of RAM compared to 50–100 MB for a standard B-Tree index. Even with occasional false positives, a nanosecond check in RAM saves millions of slow disk reads.</p>
<h2>6. Locks, Isolation Levels, and MVCC</h2>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/4edf9bdd-8a0a-4755-a4cf-a813bf3d85fe.jpg" alt="" style="display:block;margin:0 auto" />

<p>Databases balance speed and data accuracy using Shared Locks (multiple readers can read simultaneously) and Exclusive Locks (one writer blocks everyone else), alongside four isolation levels (<code>READ UNCOMMITTED</code>, <code>READ COMMITTED</code>, <code>REPEATABLE READ</code>, and <code>SERIALIZABLE</code>).</p>
<p>MySQL's InnoDB engine defaults to <code>REPEATABLE READ</code>, yet readers don't block writers and writers don't block readers. It pulls this off using Multi-Version Concurrency Control (MVCC). Instead of locking rows during updates, InnoDB saves older versions of rows in an Undo Log. When you start a transaction, you get a point-in-time snapshot; if another user modifies a row while you're reading, InnoDB simply follows a pointer back through the undo log to show you the version that existed when your snapshot began.</p>
<p>Let's see it in example:</p>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/f68538bd-ca93-45d0-83c4-b8c72a3ab4c6.png" alt="" style="display:block;margin:0 auto" />

<p>Now let's go to the second transaction which will update this row:</p>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/c1f552b4-3427-48b0-9ce1-a95626ab5dbd.png" alt="" style="display:block;margin:0 auto" />

<p>It succeeded while previously on update we got deadlocks since both transactions were updating.</p>
<p>Now let's try reading the data again after committing the first transaction which read the title initially:</p>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/2776bdb6-c370-4323-9f70-4d31e506662e.png" alt="" style="display:block;margin:0 auto" />

<p>See, now you see fresh data. But what happens when you scale this across multiple servers in different data centers and a network cable gets cut?</p>
<p>That’s where the <strong>CAP Theorem</strong> comes in. In simple developer terms, when a network partition happens, you have to choose:</p>
<ul>
<li><strong>Consistency (CP):</strong> Every read receives the most recent write or an error. If servers can't talk to each other to sync up, they'd rather fail or block your request than serve you old data.</li>
<li><strong>Availability (AP):</strong> Every request receives a non-error response—meaning the system stays up and answers you immediately, even if the data it gives you is stale.</li>
</ul>
<p>You can't have both at the same time when the network drops. You have to decide whether your app prefers to crash/block or lie with old data.</p>
<h2>7. Replication Lag: Stale Reads, Not Locks</h2>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/6f168eb0-989f-42b0-ab3c-787232ce90fa.png" alt="" style="display:block;margin:0 auto" />

<p>In a Primary-Replique setup, asynchronous replication (where the primary database writes changes locally, fires off the event logs to the replicas, and immediately tells the client <em>"Success!"</em> without waiting for the replicas to finish applying them) means replicas can fall behind during heavy write spikes or network hiccups. Contrary to common belief, replication lag doesn't lock out your read queries—replication threads run independent background loops separate from user connections.</p>
<p>The real danger isn't database locks; it's <strong>stale reads</strong>. Imagine a user updates their profile on the primary server, immediately refreshes the page, and hits a lagging replica that still serves their old data.</p>
<p>Here is how asynchronous replication and lag actually behave in production:</p>
<ul>
<li><strong>How Asynchronous Replication Works (The Fire-and-Forget Model):</strong><ul>
<li>The Primary server executes a write query and records it in its binary log (binlog).</li>
<li>It ships those binlog events to the replicas over the network.</li>
<li>Crucially, <strong>the primary does not wait</strong> for the replicas to confirm they've successfully saved the data before returning a success response to the user. This keeps your writes blazing fast, but it means there is always a tiny window where the replica has stale data.</li>
</ul>
</li>
<li><strong>Independent Background Threads:</strong> Replication runs on its own dedicated threads on the replica. Normal user read queries won't block replication, but a massive wave of writes on the primary can easily saturate the network pipeline and cause replicas to fall behind.</li>
<li><strong>The User Experience Trap:</strong><ul>
<li>User Alice uploads a new profile picture, which hits the primary server.</li>
<li>She refreshes her browser the exact next second.</li>
<li>Her request lands on a read replica that is two seconds behind.</li>
<li>The replica happily serves up her <em>old</em> profile picture.</li>
<li>Alice thinks her upload failed and clicks "Save" again, unnecessarily spamming the server.</li>
</ul>
</li>
<li><strong>How We Mitigate It:</strong><ul>
<li><strong>Route Critical Reads to the Primary:</strong> For immediate read-after-write flows (like updating a profile, changing a password, or completing an e-commerce checkout), route those specific queries directly to the primary server instead of the replicas.</li>
</ul>
</li>
</ul>
<h1><em>Ending</em></h1>
<blockquote>
<p><em>Doing all this made me realize database optimization isn't about memorizing fancy tricks or chasing a mythical 0.001-millisecond query time. For a long time, terms like "MVCC," "sharding," and "replication lag" felt like magic words that senior engineers used just to sound smart.</em></p>
<p><em>Turns out, they are just practical solutions to very real bottlenecks:</em></p>
<ul>
<li><em>I learned the hard way why skipping 100,000 rows with</em> <code>OFFSET</code> <em>will eventually melt your CPU.</em></li>
<li><em>I watched MySQL gracefully kill my queries with deadlocks when two transactions tried to step on each other's toes.</em></li>
<li><em>I finally wrapped my head around how MVCC and undo logs let readers and writers coexist peacefully without blocking each other.</em></li>
<li><em>And I realized why replication lag means we can't always blindly trust a read replica right after updating our profile.</em></li>
</ul>
<p><em>You don't need to know everything overnight. But once you start looking at execution plans (</em><code>EXPLAIN ANALYZE</code><em>), treating disk reads as expensive, and respecting how databases handle concurrency under the hood, a lot of the mystery goes away.</em></p>
</blockquote>
]]></content:encoded></item><item><title><![CDATA[Database Internals]]></title><description><![CDATA[A database is a collection of data stored electronically so that it can be accessed, retrieved, managed, and updated efficiently. But wait! What about…]]></description><link>https://dbbyyaz.hashnode.dev/database-internals</link><guid isPermaLink="true">https://dbbyyaz.hashnode.dev/database-internals</guid><category><![CDATA[Databases]]></category><category><![CDATA[Devops]]></category><category><![CDATA[storage]]></category><dc:creator><![CDATA[Yazdan Haider]]></dc:creator><pubDate>Wed, 11 Nov 2026 22:07:59 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/78022d50-99fc-467f-9d79-209e398d1f53.jpg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<h2>What is a Database?</h2>
<p>A database is a collection of data stored electronically so that it can be accessed, retrieved, managed, and updated efficiently.</p>
<p>But wait! What about MongoDB, where data is stored as documents and different records can have different structures? Doesn’t that contradict our definition of a database as an organized collection of data?</p>
<p>Not really!</p>
<p>The word “organized” doesn’t mean the data must live in tables with predefined rows and columns. Different systems organize and store data in different ways.</p>
<p>The software that does this organizing is called a Database Management System (DBMS). MySQL, MongoDB, and Redis are all DBMSs. For example:</p>
<p>If your application has complex relationships between entities, such as users, orders, and products, you might choose a relational DBMS like MySQL. If your application needs a flexible schema where documents can have different fields, you might choose a document DBMS like MongoDB.</p>
<p>Consider the below picture for reference I added an another row with different field and it saved while the same one would give an error in a relational database</p>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/40a50f28-501e-41ca-bea7-d800bfb28cdc.png" alt="" style="display:block;margin:0 auto" />

<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/84fbfa6c-4f7f-4127-9fd2-7093edbcb5b1.png" alt="" style="display:block;margin:0 auto" />

<pre><code class="language-markdown">If you need to store frequently accessed, short-lived data such as sessions and cache entries, you might choose an in-memory DBMS like Redis this will be covered in a separate article.
</code></pre>
<p>In short, a database is the organized data itself, and a DBMS is the software that stores, retrieves, and manages it. Different DBMSs just do this in different ways. But that’s a story for another day, because today we’re focusing on how a DBMS works internally. We’ll explain only mysql and follow a query from the moment you enter it to the moment it reaches the storage engine.</p>
<p>Ready for this?</p>
<p>Do keep in mind that I’ll assume you already know the basics of the queries you enter in a database client.</p>
<p>And sorry for the little refresher above! I thought it would be unfair to throw technical jargon in your face right away.</p>
<p>Okay, so we know what a database is, and we know the DBMS is the software managing it. But what actually happens when you hit “Run” on a SQL query?</p>
<p>Let’s trace the life of a simple query: SELECT name FROM employees WHERE emp_no=10001.</p>
<p>Look at the diagram below. It’s a vertical pipeline. On the left, we have the “What Happens” (the internal machinery), and on the right, we have “What you enter” or what the system produces at that exact moment.</p>
<p>Here is the step-by-step breakdown of how your database processes that text.</p>
<h2>1. The Client &amp; The Text</h2>
<p>It all starts here. You type your SQL query into your client (like MySQL Workbench, DBeaver, or a terminal). At this stage, it is literally just plain text. The database has absolutely no idea what SELECT or users means yet. It’s just a string of characters being sent over a network.</p>
<h2>2. The Transport Layer (Connection Handler)</h2>
<p>Before the database even looks at your text, it needs to know who you are. The Connection Handler steps in. It checks your credentials (username/password), verifies which database you are trying to access, and assigns you a specific thread to handle your request. Think of this as the bouncer at a club checking your ID before you even get to the dance floor.</p>
<p>This is the connection handling done my the dbms itself:</p>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/02d8a307-4504-477b-aafb-289e63a642bb.png" alt="" style="display:block;margin:0 auto" />

<p>It verifies the user and hosts maintains the sessions on server side for the duration of client requests.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/9aacb6d5-7ff1-47e8-baa6-6e8727d4bda1.png" alt="" style="display:block;margin:0 auto" />

<h3>3. The Parser (The Grammar Checker)</h3>
<p>Now we enter the <strong>Query Processor</strong>. The first stop is the <strong>Parser</strong>.<br />Remember the purple box on the left? That’s the “Parsing of each word” breakdown. The parser takes your raw text and checks it against the SQL grammar.</p>
<ul>
<li>Is <code>SELECT</code> a valid keyword? Yes.</li>
<li>Is <code>FROM</code> in the right place? Yes.</li>
<li>Does the syntax make sense?</li>
</ul>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/ba875d9c-692e-4c17-9178-4a5fb6da0736.png" alt="" style="display:block;margin:0 auto" />

<p>If you made a typo, this is where it dies and throws an error. If it passes, the Parser spits out a <strong>Parse Tree</strong>. It breaks your query down into labeled pieces (words) so the system can understand the structure.</p>
<h3>4. The Optimizer (The Route Planner)</h3>
<p>This is arguably the smartest part of the database. The <strong>Optimizer</strong> takes the Parse Tree and asks: <em>“What is the absolute fastest way to get this data?”</em></p>
<p>Let’s say you have a million users. Should the database read every single row until it finds <code>id=5</code>? Absolutely not. The Optimizer looks at the available indexes and statistics. It realizes, <em>"Hey,</em> <code>id</code> <em>is a Primary Key! I can do a direct lookup instead of scanning the whole table."</em></p>
<p>It then creates an <strong>Execution Plan</strong>. In our diagram, the plan is simply: <em>“Primary key lookup, 1 row.”</em> It found the fastest route.</p>
<p>Lets look at an example so it is cleared:</p>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/7d481746-7b29-48ac-8832-96285fc81c9a.png" alt="" style="display:block;margin:0 auto" />

<p>If you look at both these<code>EXPLAIN</code> output, we asked the database to find an employee by their <code>first_name</code>.</p>
<p>Notice the <code>type</code> column. It says <strong>ALL</strong>. This is the optimizer’s way of saying, <em>"I have no shortcuts here, so I'm going to read the entire table from top to bottom."</em></p>
<p>Look at the <code>rows</code> column next: <strong>299,113</strong>. The database is estimating it will have to look at nearly 300,000 records just to find one person. This is exactly what happens when you don't have an index on a column. In our diagram, the optimizer would have looked at this and created an execution plan for a "Full Table Scan." It's slow, it's heavy on the disk, and it's exactly why adding indexes is so important.</p>
<p>But what if we refer the employee number which is the primary key</p>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/1b20b732-9af8-4c82-aa13-f7f75f379b8d.png" alt="" style="display:block;margin:0 auto" />

<p><em>P.S Since you may or may not know about indexes they are pointers to our data. An index maps to a page which has our data in it. In the case of not creating an index a primary key also by default in innodb creates a clustered index(will be covered later).</em></p>
<h3>5. The Executor (The Worker)</h3>
<p>The <strong>Executor</strong> is the muscle. It takes the Execution Plan from the Optimizer and actually does the work. It doesn’t care about grammar or optimization; it just follows the recipe.<br />It calls the <strong>Handler API</strong> with specific instructions: <em>“Go to the storage engine, read the row where key equals 5, and give me the data.”</em></p>
<p>This will be cleared with the following example:</p>
<img src="https://cdn.hashnode.com/uploads/covers/6af4e297ed0c9c85a1b00a48/367d0d2f-e405-471c-884a-915321e2b585.png" alt="" style="display:block;margin:0 auto" />

<p>As mentioned before see since a table with primary key already gets it index created when we used that index to find a row the executor didn't have to load all the table. But when we referenced the record using non-primary key (in short no index) then it had to load the table. This is where query optimization comes.</p>
<h3>6. The Storage Engine (InnoDB)</h3>
<p>Finally, we hit the bottom layer: the <strong>Storage Engine</strong>. In this example, we are using <strong>InnoDB</strong> (the default for MySQL).<br />The Storage Engine is the component that actually touches the hard drive. It handles the heavy lifting:</p>
<ul>
<li><strong>B-Tree:</strong> Navigating the index structure to find the key.</li>
<li><strong>Buffer Pool:</strong> Checking if the data is already in memory before going to the disk.</li>
<li><strong>Redo Log:</strong> Ensuring the data is safe if the power goes out.(Every write and change is added to this log(redo log) a tracker of sorts)</li>
</ul>
<p>Once InnoDB finds the row, it sends the data all the way back up the pipeline to the Client, and voila — you see your user’s name on the screen.</p>
<p><em>And that’s the journey of a single query. From a plain string of text to a parsed tree, through the brain of the optimizer, and finally out to the disk. It’s a lot of work for a few milliseconds of waiting, but understanding this pipeline is the first step to writing better, faster queries. Now go run an</em> <code>EXPLAIN</code> <em>and see what your database is really thinking.</em></p>
]]></content:encoded></item></channel></rss>