I was in the middle of writing a technical post, I knew I had read a great explanation of the topic somewhere. I could picture the page. I spent 40 minutes re-Googling variations of the same query, opening tabs, closing them, opening them again. Never found it.

That’s when I realized: I don’t have a knowledge problem. I have a retrieval problem. I read constantly and I retain almost none of it in a way that’s actually useful.

I had 147 bookmarks across three browsers. A Notion database with 100 articles I’d tagged “read later” and never opened. An Obsidian vault that felt productive to set up and then immediately became a burden to maintain.

The thing is, all of these tools have the same problem underneath: filing cabinets. You put something in, it stays exactly where you put it, and you need to remember where you put it to get it back. They don’t understand what you read. They don’t connect ideas across sources. They don’t know that the paper you read in January is related to the blog post you saved last week.

So I built something that does and I’m calling it Personal KnowOS.

The solution is simple: everything you actually read(not bookmark) becomes queryable knowledge without you doing anything. You browse normally. The system captures what you spend real time on, processes it, builds a graph of entities and relationships across your entire reading history, and gives you a conversational agent you can ask questions to. With citations. From your actual reading.

This post is about how I built it and why I made every decision I made. Including the ones I got wrong.

The whole system, fast

Chrome extension notices you’ve been reading something for real (not just landed on it) → POSTs cleaned content to a local FastAPI server → server immediately hands it off to a Celery worker and returns a job ID → worker chunks the text, embeds it into Qdrant, runs entity extraction into Neo4j, writes metadata to Postgres → Redis handles dedup and job tracking throughout.

When you ask a question, a LangGraph agent figures out what kind of question it is, decides whether to search your personal knowledge base or the live web (or both), runs hybrid retrieval across vectors + keyword + graph, re-ranks the results, pulls full context, and sends it to Gemini for synthesis with source citations.

That’s the shape of it.

Part 1: The Capture Layer

The first real question comes: how does content get in?

I went through the options a lot of options like browser history parser, chrome extension, desktop/mobile app.

A Chrome extension was the only path that made sense. It runs inside the browser. It sees the fully rendered DOM. It knows how long I’ve been on the page, how far I’ve scrolled, when the tab was in focus. It can run Readability.js right there in the browser to strip navigation, ads, and boilerplate before the content ever touches my server.

The extension has two parts: 1. a content script that lives in the page and handles DOM access and text extraction, and 2. a background service worker that manages tab events and makes the actual POST request to FastAPI. They talk to each other via chrome.runtime.sendMessage.

The hard part wasn’t the building. It was figuring out what not to capture.

On noise

The first version of this captured everything. Everything. And my retrieval results were immediately garbage: LinkedIn profile pages, my shopping cart, Google search results I’d glanced at for three seconds, cookie consent dialogs that had somehow scraped together 200 words.

The insight, which sounds obvious in retrospect: a personal knowledge system is only as good as the signal you feed it. Noise doesn’t just fail to help. It actively makes things worse. Every irrelevant document that makes it in is something that could surface in your results instead of the thing you actually want.

So I added three layers of filtering.

Behavioral gates: the extension only fires if you’ve spent at least 30 seconds on the page, scrolled at least 40% of the content, and the word count is over 250. These are proxies for “did I actually read this?” They’re not perfect but they’re pretty good.

Domain blocklists: social media feeds, email inboxes, shopping pages, banking sites. The extension doesn’t even look at these.

Content deduplication: SHA-256 hash of the cleaned text gets checked against Redis. If I’ve seen this content in the last 7 days, skip it. For near-duplicates (same article on the original site and three syndication farms), content similarity scoring at a 0.92 threshold.

Honestly, building the blocklist was more work than building the capture logic and more important.

Part 2: The Processing Pipeline

When content arrives at FastAPI, the server does exactly one thing: it hands the work to a Celery task and returns a job ID. The whole endpoint takes under 200ms.

I’m proud of this decision because the temptation is to just process everything synchronously in the route handler. It’s simpler to write. And it would be completely unusable: chunking, embedding, NER, and three database writes takes 10-30 seconds. The extension would time out. The browser would feel broken.

Celery running against a Redis broker separates the HTTP response from the actual work. The extension stays snappy. Failed tasks retry automatically. Workers scale horizontally. And I stream status back to the extension via Server-Sent Events, so there’s a little progress indicator rather than a black hole.

Chunking — the decision I got most wrong, most times

I went through three versions of chunking before landing on something that actually worked.

Version one: fixed-size chunks at 512 tokens. Dead simple to implement. Terrible in practice. Chunks that split mid-sentence, mid-argument, sometimes mid-concept. The embeddings for these chunks don’t represent coherent ideas, they represent arbitrary fragments. Retrieval was noisy in a way that felt random.

Version two: sentence-level chunks. Better, at least each chunk is grammatically complete. But individual sentences are often meaningless in isolation. “This has three important advantages” retrieved without the advantages is useless. And sentence-level chunks are often too short to embed a real idea.

Version three: which is what I run now: parent-child chunking. Small child chunks (256 tokens) get embedded and searched. Each child is linked to a large parent chunk (1024 tokens). When retrieval finds a relevant child, you fetch its parent for the LLM to actually read.

The insight: optimize search and read separately. Small chunks give precise retrieval because they capture narrow, specific semantics. Large chunks give the LLM enough context to actually understand and use what it found. Doing both in one chunk size means you’re compromising on both.

This single change improved answer quality.

For very long documents I also use semantic chunking, which splits on topic boundaries rather than token counts. The way you detect a boundary: measure cosine similarity between adjacent sentence embeddings. When it drops sharply, that’s a topic transition. Split there.

The knowledge graph

This is the piece that makes the system feel like memory rather than search.

After chunking, a spaCy NER pipeline runs over each document and pulls out named entities like people, organizations, places, concepts, products. These become nodes in Neo4j. Relationships extracted from sentence co-occurrence and dependency parsing become edges.

What this buys you: a retrieval path that pure vector search fundamentally cannot provide. Vector search is great at “what’s semantically related to this query.” Graph traversal is great at “what do I know about this specific entity.” They answer different questions.

If you search “Sam Altman” in a pure vector system, you’ll find documents where he’s prominently featured, where his name appears often enough to dominate the embedding. You might miss the document that mentions him once in passing, relevantly. Graph traversal finds every document with a node edge to him, regardless of prominence.

I added the graph last.

Part 3: Memory — Why You Need More Than One Database

The most common mistake I see in projects like this is treating storage as one thing. Pick a database, put everything in it. This doesn’t work because different data has wildly different access patterns and latency requirements.

I made three tiers.

Tier 1 is an in-process Python dictionary. Current session context and the agent’s conversation history live here. Latency is sub-millisecond. It dies on restart, which is fine, a session state doesn’t need to survive a restart. Running a Redis call for every agent state update during a conversation would be gratuitous overhead.

Tier 2 is Redis. Dedup fingerprints, job status, search result cache (24-hour TTL), the Celery task queue. Latency is around 1ms. Fast enough for the dedup check that runs on every ingested document, persistent enough to survive a server restart.

Tier 3 is the actual knowledge stores: Qdrant for vectors, Neo4j for the graph, Postgres for structured metadata. Latency is 5-50ms depending on query complexity. Durable, indexed, queryable. Each one handles the thing it’s genuinely designed for.

The rule I kept coming back to: match the storage to the access pattern. Don’t run a Qdrant vector search to check if you’ve seen a URL. Don’t store conversation history in Postgres. Don’t put embeddings in Redis.

Part 4: Retrieval — Why Hybrid Beats Simple

The question at the heart of the retrieval system: when someone asks “what do I know about transformer architectures?”, how do you find the right content across thousands of documents?

The obvious answer is vector search. Embed the query, find similar embeddings. It works. But it misses things.

Vector search fails on exact terms. If you’re looking for “Attention is All You Need”, the vector might surface conceptually related documents that never mention the paper by name. Acronyms, proper nouns, and technical jargon compress poorly into embedding space.

Pure keyword search (BM25) fails on semantics. “How do attention mechanisms work” might miss a document that explains the concept perfectly using different vocabulary, because none of the words match.

So I run three retrievers in parallel and merge the results.

First: vector search over Qdrant child chunks, but with HyDE(Hypothetical Document Embeddings). Instead of embedding the raw query, I prompt Gemini to generate three hypothetical answers to the question, then embed those. The hypothetical answers use domain vocabulary that matches how real documents are written, not how casual questions are phrased. For vague exploratory queries this dramatically improves recall.

Second: BM25 over full document text in Postgres. Catches exact terminology, model names, paper titles, acronyms.

Third: graph traversal over Neo4j for entity-centric queries. Start at the entity node, walk edges to related documents.

The three ranked result lists get merged with Reciprocal Rank Fusion. The formula: for each document, sum 1/(k + rank) across all lists where it appears, where k is typically 60. A document ranked highly in multiple lists scores much higher than one that appears in only one. No normalization needed.

After RRF, the top 20 candidates go to a cross-encoder re-ranker.

This is the step I resisted adding the longest, and it had the biggest quality impact.

Vector search uses bi-encoders: query and document are embedded separately, then compared. It’s fast but the comparison is shallow, you’re comparing two independent summaries rather than reading both together. A cross-encoder takes the (query, document) pair as one input and runs full attention over both simultaneously. Much more accurate. Also O(n) expensive, which is why you only use it on top-20 candidates, not the full corpus.

Top 5 after re-ranking get their parent chunks fetched. Those go to the LLM.

Part 5: The Agent

Without this layer, everything I’ve described is a sophisticated single-shot search system. One query, one retrieval pass, one answer. No iteration, no tool selection, no ability to say “I don’t have this in my personal KB, let me check the web.”

LangGraph lets you build a stateful graph of agent nodes where edges can be conditional. Each node is a function that reads from and writes to a shared state dictionary. The graph loops until a termination condition is met.

My agent’s state carries the original query, retrieved chunks, tool call history, intermediate reasoning, and conversation history. Nodes can inspect previous tool calls and decide differently based on what they found.

The routing logic works like this. The classification node reads the query and decides: is this temporal (”what did I read last week about X?”) → Postgres-filtered vector search. Entity-centric (”everything I’ve read about Anthropic”) → graph traversal first. Current events (”what happened with OpenAI yesterday?”) → DuckDuckGo directly. Exploratory (”what do I know about diffusion models?”) → HyDE + vector.

The synthesis node is the most constrained part of the system. Gemini gets the retrieved context and a prompt that says: cite every factual claim to a source ID. Do not introduce information that isn’t in the provided context. If context is insufficient, say so explicitly and ask if I want to search the web.

This matters more than it sounds. An unconstrained synthesis step will confidently blend retrieved knowledge with model priors, and you’ll get answers that feel right but contain information that wasn’t in anything you actually read. The citation constraint forces grounding.

Part 6: Making It All Run

Six Docker services in one compose file: FastAPI, Celery worker, Redis, Qdrant, Neo4j, Postgres. They share a Docker network and resolve each other by service name. Volumes for persistent data. Environment variables for secrets. Health checks so Celery doesn’t start before Redis is ready.

The embedding API call is the bottleneck at scale. It’s synchronous and has rate limits. The fix is batching: instead of embedding one chunk per API call, batch them. An async HTTP client (httpx) instead of requests helps too. At scale you’d also look at self-hosting an embedding model rather than paying per token.

What I Got Wrong

The chunking thing. I stayed with fixed-size chunks longer than I should have because they were simple and “good enough” felt like it was right around the corner. It wasn’t.

Underestimating noise. I genuinely did not expect cookie dialogs and Google search result pages to make it through. The behavioral triggers felt conservative when I wrote them. In practice they barely held back the flood. The blocklist was the most tedious thing I built and probably had the highest return.

Adding the graph last. I treated Neo4j as an enhancement rather than a core component. That means my entity extraction schema was designed around what was easy to bolt on, not what would be most useful for traversal. The Cypher queries feel improvised rather than planned. If I rebuilt this from scratch, the graph schema would be the first design artifact I produced.

Not adding the cross-encoder at the start. I assumed bi-encoder vector search would be close enough and I could always add re-ranking later. There are very few retrieval systems where you don’t want a cross-encoder re-ranker.

What I Want to Build Next

Running a cloud LLM over all my personal reading is a privacy tradeoff that bothers me more the longer I think about it. Local synthesis with Ollama and a quantized model is the most important thing on the roadmap. The quality hit is real but the privacy gain is worth it for something this personal.

Spaced repetition, the system knows what I’ve read and when. It should be able to surface “you read this 4 months ago, do you still remember this?” and give me a one-paragraph refresher. Forgetting curve modeling over ingested content.

Multi-modal capture. YouTube transcripts work partially. Podcasts, lectures, voice notes nothing. Whisper integration with timestamp-aligned chunking would bring audio into the knowledge base, which is where a huge chunk of my actual learning comes from.

The thing I want most and keep deprioritizing: graph visualization. I want to see my knowledge as a map. I want to see clusters forming around topics, connections appearing between ideas I didn’t know were related, blank spots in areas I think I understand. It would turn this from a search engine into something that actually feels like a second brain.

Subscribe now

If this article helped clarify things, consider subscribing so you don’t miss future deep-dives, and feel free to like or leave a comment—I’d love to hear your thoughts and what you’d like to see next.

Leave a comment

One of the confusing things when you first encounter MCP is — if AI agents can already call APIs and use tools, why do we need yet another protocol? Why didn't the community just stick with function calling and move on?

This confusion usually shows up right when you are starting to feel comfortable with how AI agents work. Then MCP appears, claiming to be a “universal standard for connecting AI to everything,” and suddenly the picture gets murky again.

The trick is to not start with the architecture. Start with the problem it actually solves.

The Problem: AI Agents Are Brilliant but Isolated

Today’s AI models can reason, write, plan, and summarize. But they are fundamentally disconnected from the systems where work actually happens.

Claude cannot see your Slack messages. It cannot push a commit to GitHub. It cannot query your company’s database. Not without someone writing custom glue code — every single time, for every single system.

Before MCP, connecting an AI to an external tool looked like this:

That second column is what the Model Context Protocol makes possible. And the USB analogy is not just poetic — it is precise.

Before USB, every device had its own proprietary connector. USB gave the world one standard plug. MCP does the same thing for AI agents and external systems.

What MCP Actually Is

MCP stands for Model Context Protocol. It is an open-source standard, originally developed by Anthropic, that defines how AI applications communicate with external systems — databases, file systems, search engines, Slack, Google Drive, GitHub, maps, and more.

One detail that is rarely explained clearly: MCP is a stateful protocol. Unlike a REST API call that fires and forgets, MCP maintains a persistent connection with a full lifecycle — it must be initialized, kept alive, and properly shut down. This makes it far better suited to the back-and-forth, multi-step nature of real agentic workflows.

💡 Why Stateful Matters

Imagine asking an agent to “review all open PRs and comment on anything touching the auth module.” That requires multiple tool calls, context carried across them, and real-time updates as results come back. A stateless REST call can’t do this well. A stateful MCP connection can.

The MCP ecosystem has three layers:

MCP Specification — the formal standard that defines exactly what clients and servers must implement. Language-agnostic and versioned.

SDKs — available in Python, TypeScript, and more, so developers can build MCP servers without implementing the protocol from scratch.

Dev Tools & Reference Implementations — ready-made MCP servers for common systems (filesystem, GitHub, web search, databases) that you can plug in immediately without writing any server code.

The Architecture: Three Actors, One Clear Job Each

MCP uses a client-server architecture — but with one extra piece that makes everything click. There are three actors, and each has a clean, non-overlapping role.

The Host is the AI application — Claude Desktop, VS Code with an AI extension, Cursor, or any custom agent framework. It is what the user interacts with. The host owns the conversation and orchestrates everything.

The Client lives inside the host. Each client maintains a dedicated, one-to-one connection with a single MCP server. If the host needs to talk to both GitHub and the filesystem, it spins up two separate clients — one for each. Clients are the translation layer between the host’s needs and the server’s capabilities.

The Server is the external system. A GitHub MCP server exposes repository operations. A filesystem MCP server exposes file reads and writes. Each runs as its own independent process.

🔑 Key Rule

The host never talks directly to a server. It always goes through a dedicated client. This separation keeps concerns clean, security manageable, and failures isolated — if one server crashes, only its client is affected.

The Two Layers Inside Every MCP Connection

Every MCP connection has two layers working together — an inner data layer and an outer transport layer. Think of it like a letter: the transport layer is the envelope and postal system, the data layer is the language the letter is written in.

The Data Layer (Inner)

All communication between a client and server uses JSON-RPC 2.0 — a lightweight remote procedure call protocol. Every message is a clean JSON object: a method name, a set of parameters, and an ID so responses can be matched back to their requests.

The most important job of the data layer is schema definition — formally describing what a server can do, what inputs it expects, and what shape the output takes. This is how an AI discovers available tools without any hardcoded knowledge. It just asks: “what can you do?” and the server tells it.

The Transport Layer (Outer)

This is how messages physically travel. MCP supports two mechanisms, chosen based on where the server lives.

The SSE (Server-Sent Events) part of the HTTP transport is worth pausing on. It means the server can push updates to the client as things happen — not just respond to requests. This is what enables real-time notifications in MCP: a long-running task can stream progress back to the agent as it works.

The Lifecycle: MCP Is Not Fire-and-Forget

Because MCP is stateful, connections have a proper lifecycle that needs to be managed. This is one of the most underexplained parts of the protocol.

MCP Connection Lifecycle

The initialization handshake is particularly important — it is where the client and server negotiate which features each supports, so they never assume capabilities the other side doesn’t have. This is what makes MCP robust across very different server implementations.

MCP Primitives: The Vocabulary of Agentic Action

This is where MCP gets genuinely interesting. The protocol defines a vocabulary of primitives — the building blocks that describe every possible action and piece of context that can flow between client and server. And it goes both ways.

The Sampling primitive deserves special attention. It means a server can ask the host’s AI to do reasoning mid-execution — without needing its own model access or API key. A GitHub MCP server could, during a PR review, ask Claude: “based on these diffs, does this look like a security issue?” and incorporate that judgment into its next action. Servers become genuinely intelligent, not just mechanical.

The Elicitation primitive is equally powerful. It means agentic workflows do not have to be fully autonomous or fully manual. A server can pause mid-task and surface a clarifying question to the user through the host’s interface — “which branch should I push to?” — before continuing. This is the missing piece for human-in-the-loop agentic systems.

Why MCP Feels Like HTTP for the Agentic Era

When the web was being built, HTTP solved a critical coordination problem: any browser could talk to any server without custom integration. You did not need to know whether a server ran Apache or Nginx. HTTP abstracted that away — and the web exploded.

MCP is doing the same thing for AI agents.

The web succeeded because it standardized communication. Any developer could build a website knowing any browser could render it. The agentic AI ecosystem will succeed for exactly the same reason — because any developer can build an MCP server knowing any AI host can use it.

The Real Takeaway

MCP is not just a developer tool. It is infrastructure — the quiet foundation being laid right now that will determine whether AI agents remain impressive demos or become genuinely useful participants in complex real-world workflows.

Once the mental model clicks, it becomes simple:

The host is the AI application. The client is the dedicated bridge to each external system. The server is the external system. The host never talks to servers directly — always through clients. The data layer defines what is said; the transport layer defines how it travels. Tools are what the AI does. Resources are what it reads. Prompts are how it is guided. And the whole thing is stateful — which is what makes long-running, multi-step agentic tasks possible.

The same way you do not think about TCP/IP every time you open a browser, you will eventually not think about MCP every time your AI agent reads a file, queries a database, or drafts a Slack message. It will just work.

And the reason it will just work is because someone built the standard properly.

Next up: Building your own MCP server from scratch — in under 50 lines of Python.

Till then, keep building — and stay curious.

Subscribe now

If this article helped clarify things, consider subscribing so you don’t miss future deep-dives, and feel free to like or leave a comment—I’d love to hear your thoughts and what you’d like to see next.

Leave a comment

In Part 1: The Quiet Foundation Behind Intelligent Systems, we built the ground floor — understanding what scalars, vectors, matrices, and linear equations actually are, and why machine learning quietly depends on them everywhere.

In Part 2, we shifted from what these objects are to what they do, looking at how matrices act as transformations, how data moves through space, and why these transformations matter more than the numbers themselves.

This article continues naturally from both.

If any idea here feels slightly unfamiliar, it’s not a dead end — it’s usually a sign that revisiting Part 1 or Part 2 for a few minutes will make everything click again.

So far, we’ve asked what vectors and matrices are, and why machine learning needs them.

Now we go one level deeper and ask a more practical question:

How do vectors interact, stretch, align, collapse, and quietly decide which directions matter — and why do those interactions show up everywhere from regularisation to PCA?

Vectors: Rows, Columns, and Direction

A vector is simply an ordered list of numbers, but the way we write it matters.

• A row vector is written horizontally

• A column vector is written vertically

In ML, column vectors are more common because they fit naturally into matrix multiplication.

Measuring Vector “Size”: Norms

When we talk about a vector’s length, we use something called a norm.

L1 Norm (Manhattan Distance)

The L1 norm adds up the absolute values of the components:

Intuition:

Imagine walking in a city laid out in blocks — you can only move left/right and up/down. L1 norm measures distance the way you’d walk, not fly.

L2 Norm (Euclidean Distance)

The L2 norm is the straight-line distance:

This is the distance your intuition already knows — shortest path through space.

Basic Vector Operations (Nothing Fancy)

Vector arithmetic is refreshingly simple:

• Addition / subtraction happens component-by-component

• Scalar multiplication stretches or shrinks every component equally

Orthogonal vectors — vectors at right angles — have a special property:

This means they share no direction in common.

Dot Product: More Than Just Multiplication

The dot product is defined as:

But the intuition matters more than the formula.

The dot product answers:

How much of one vector lies in the direction of the other?

If vectors point in the same direction → large dot product

If they’re perpendicular → zero

If they oppose each other → negative

Matrix Inverse: Undoing a Transformation

A matrix inverse is the idea of reversing a transformation.

If:

then applying A and then A-1 brings you back to where you started.

But not all matrices are reversible.

If:

then no inverse exists.

Why?

Because the transformation collapsed space — some information was lost forever.

Determinant: Area, Volume, and Orientation

The determinant tells us how a matrix scales space.

A negative determinant means the orientation flipped — like a mirror reflection.

Transformations compound exactly the way you’d expect.

Basis: Coordinates Without Coordinates

A basis is a set of vectors that lets you reach every point in a space.

Think of them as directions you agree to use.

• In 2D: usually two vectors

• In 3D: three vectors

They don’t have to be perpendicular — just independent.

Span: Everything You Can Reach

The span of a set of vectors is all points you can reach by combining them.

If you can walk forward, backward, and scale your steps — span is your reachable world.

A basis is simply a minimal spanning set.

No redundancy.

Dimension: How Many Directions Matter

The dimension of a space equals the number of vectors in its basis.

• Line → 1D

• Plane → 2D

• Volume → 3D

In ML, high-dimensional spaces are common — but often misleading.

Linear Dependence: Redundant Directions

If one vector can be written as a combination of others:

then the vectors are linearly dependent.

Intuition:

One direction wasn’t actually adding anything new.

A proper basis must be:

• spanning

• linearly independent

Eigenvalues and Eigenvectors: The Quiet Axes

Eigenvectors are special directions that don’t rotate under a transformation.

They only stretch or shrink.

• The eigenvector tells the direction

• The eigenvalue tells how much stretch happens

Eigenbasis

If you stack all eigenvectors as columns, you get an eigenbasis.

This basis describes the transformation in its most natural coordinate system.

To find eigenvalues:

This is the characteristic equation.

Dimensionality Reduction and PCA (High-Level)

Real datasets often live in high dimensions — but the information doesn’t.

Mean

The center of the data.

Variance

How spread out the data is.

Covariance

Whether two directions vary together.

Covariance checks:

Do these dimensions grow and shrink together?

Why PCA Works

PCA finds directions with maximum variance.

The eigenvector with the largest eigenvalue points along the direction where the data spreads the most.

Projecting onto that direction preserves maximum information.

That’s dimensionality reduction without throwing away meaning.

Closing Thoughts

Some topics — Gaussian elimination, PCA derivations, advanced matrix factorisations — deserve their own deep dives.

This series wasn’t about covering everything.

It was about building intuition strong enough that formulas stop feeling scary.

And once that intuition clicks, ML concepts start feeling… lighter

You can visit the notes for all the 3 parts with this link.

And for detailed video lectures - DeepLearning.AI(I myself studied from there) from their course Maths for ML and DS

Stopping here with Linear algebra. In the next articles, we’ll move into calculus, probability, and statistics, where the math becomes a little more explicit and a few new terms will show up. Until then, take it slow, revisit what felt fuzzy, and most importantly — keep learning and stay curious!

Subscribe now

If this article helped clarify things, consider subscribing so you don’t miss future deep-dives, and feel free to like or leave a comment—I’d love to hear your thoughts and what you’d like to see next.

Leave a comment

If Entropy and Gini almost always choose the same split in a decision tree, a very reasonable question arises — why do we even have two different metrics?

Why didn’t the community just settle on one and move on?

This confusion usually appears right when someone starts feeling comfortable with decision trees, and suddenly two unfamiliar formulas enter the picture, both claiming to measure something called impurity. The trick to understanding this comparison is to not start with formulas at all, but with what kind of question each metric is trying to answer.

Why Do We Measure Impurity in the First Place?

A decision tree grows by repeatedly asking simple questions like:

“Should I split the data based on feature X at value Y?”

To answer this, the tree needs a way to decide whether a split makes the data cleaner. Cleaner here simply means that the labels become more predictable after the split.

So impurity is not a complicated concept. It is just a measure of how mixed the classes are at a node.

• A node with only one class → perfectly pure

• A node with many classes mixed evenly → highly impure

Both Entropy and Gini exist to quantify this “mixed-ness”, but they look at it through slightly different mental lenses.

Entropy: Measuring Uncertainty

Entropy comes from information theory, but we do not need that background to understand its intuition.

Entropy is asking a very human question:

“How uncertain am I about the class label at this node?”

Imagine reaching a node in a tree and blindly picking a data point from it. If you have no idea what class it belongs to, entropy is high. If you can predict the class almost immediately, entropy is low.

Mathematically, entropy is written as:

But instead of focusing on the formula, focus on the behaviour:

• When all classes are equally likely → entropy is high

• When one class dominates → entropy drops

• When a node is perfectly pure → entropy becomes zero

Gini: Measuring Mistakes

Gini looks at the same situation, but asks a different question:

“If I randomly guess a class label according to the class distribution, how often would I be wrong?”

This framing is surprisingly powerful.

Instead of thinking about uncertainty, Gini thinks in terms of expected misclassification. It imagines a very lazy classifier that guesses labels based on proportions and asks how bad that strategy would be.

The Gini impurity is written as:

Again, intuition matters more than algebra:

• If one class dominates → squared probability is large → Gini is low

• If classes are evenly mixed → squared probabilities are small → Gini is high

• Pure node → Gini equals zero

A Single Running Example: The Bag of Balls

Let’s carry one example through the rest of the article.

Imagine a bag with balls of two colours.

Bag A

• 50 red

• 50 blue

Bag B

• 90 red

• 10 blue

From an entropy point of view:

• Bag A is maximally confusing

• Bag B is far less surprising

From a Gini point of view:

• In Bag A, you will be wrong very often if you guess

• In Bag B, guessing “red” almost always works

Same situation. Same conclusion.

Different thinking.

This is why most of the time both metrics rank splits in the same order.

Why Do Entropy and Gini Often Pick the Same Split?

This is one of the most important insights that is rarely stated clearly.

Decision trees do not care about the absolute value of impurity.

They only care about how much impurity is reduced by a split.

Both Entropy and Gini:

• Reward purer child nodes

• Penalize mixed distributions

• Prefer balanced and meaningful splits

Since both functions are monotonic with respect to class purity, the relative improvement after a split is often ranked the same.

In simple terms:

The tree does not care how you measure impurity, as long as impurity goes down.

When Do They Actually Differ?

Although rare, there are situations where Entropy and Gini disagree.

This usually happens when:

• Class distributions are highly skewed

• There are many classes

• The dataset is very small

Entropy is more sensitive to changes near probability extremes, meaning it reacts more strongly when rare classes appear or disappear. Gini, on the other hand, responds more smoothly.

So entropy may prefer splits that isolate rare classes slightly earlier, while Gini may stay conservative.

These differences are subtle, but they exist.

Why Most Libraries Default to Gini

If Entropy and Gini behave similarly, why does sklearn default to Gini?

The reason is refreshingly practical.

• Gini is computationally cheaper (no logarithms)

• Faster when building large trees

• Empirically shows similar performance

In large-scale systems, these small efficiency gains add up, and since accuracy differences are negligible, Gini becomes the sensible default.

Visual Intuition Without Graphs

You can imagine the two metrics like this:

• Entropy drops sharply as nodes become pure

• Gini decreases in a smoother curve

• Both reach zero at perfect purity

Entropy feels more “strict”, Gini feels more “forgiving”, but both guide the tree toward the same destination.

The Real Takeaway

Entropy and Gini are not rivals.

They are two languages describing the same idea.

• Entropy talks about uncertainty

• Gini talks about mistakes

• Decision trees only care about improvement

Entropy measures how confused the model feels; Gini measures how often it would be wrong.

As long as impurity decreases, the tree is happy.

And once this intuition clicks, the formulas stop feeling intimidating — they just become tools expressing a very human idea: make decisions that reduce confusion

Hope this article gave you some insights into Entropy and Gini impurity and now-onwards, you guys won’t feel scared of these terms. See you in next article.

Till then, keep learning—and stay curious! and let the math settle naturally.

Subscribe now

If this article helped clarify things, consider subscribing so you don’t miss future deep-dives, and feel free to like or leave a comment—I’d love to hear your thoughts and what you’d like to see next.

Leave a comment

In the previous article, we focused on what vectors and matrices are, and why machine learning needs them. Now we move one level deeper and ask a more practical question:

How do we actually solve systems of equations using matrices?

Systems of Equations as Matrices

Consider a very simple system of equations:

These are two equations with two unknowns. Instead of handling them separately, linear algebra allows us to bundle them together into a compact matrix form.

This system can be written as shown in the above figure.

This way of writing equations is extremely powerful. It allows computers—and us—to operate on entire systems at once instead of dealing with equations line by line.

Singular and Non-Singular Systems

Earlier, we introduced the idea that not all systems of equations behave the same way. Some give a unique solution, while others do not.

To formalise this idea, we introduce an important quantity called the determinant.

• If the determinant of a matrix is zero, the system is singular

  (the equations are linearly dependent or contradictory)

• If the determinant is non-zero, the system is non-singular

  (there is exactly one solution)

This single number tells us whether the system carries enough independent information to solve for the unknowns.

For a 2X2 matrix, the determinant is computed as:

|determinant| = (ad - bc)

You can remember this intuitively as:

multiply the diagonal elements, subtract the product of the off-diagonal elements.

For 3X3 matrix,

If this value becomes zero, the two rows are not giving independent information.

You can easily search on various methods of calculating the determinant on you-tube, the goal is not memorisation, but understanding why determinants matter: they tell us whether solving the system is even possible.

When Does a System Have a Solution?

Let us revisit a concrete example:

a + b = 10

a + 2b = 12

Here, we have:

• two variables

• two independent equations

This leads to a unique solution:

b = 2, and a = 8

This highlights a key idea:

To get a definite solution, the number of independent equations must match the number of unknowns.

If there are fewer equations, the solution space becomes larger.

If equations repeat or contradict each other, the system becomes singular.

Why Matrices Help Computers

Solving systems by substitution works for small examples, but machine learning deals with thousands of variables and equations.

Matrices allow us to:

• represent systems compactly

• apply systematic operations

• reduce equations step by step

This brings us to row reduction.

Matrix Row Reduction

The idea of row reduction is simple:

we perform operations that do not change the solution, but make the matrix easier to solve.

The goal is to reduce complex systems into simpler ones.

We aim to introduce zeros strategically so that back-substitution becomes easy.

Elementary Row Operations

Before learning different forms, here are the allowed operations:

1. Row swapping

   Swapping two rows changes the sign of the determinant.

2. Row scaling

   Multiplying a row by a scalar multiplies the determinant by that scalar.

3. Row addition

   Adding a multiple of one row to another does not change the determinant.

These operations preserve the solution while simplifying the system.

Rank: Measuring Information

The rank of a matrix tells us how much independent information the system contains.

Informally:

• Rank = number of independent equations

• Rank = number of pivots after row reduction

A useful relationship is:

Rank = Number of variables - Dimension of solution space

If the rank is less than the number of variables, the system is singular and does not have a unique solution.

Row Echelon Form (REF)

Row echelon form is a simplified version of a matrix where:

1. All zero rows are at the bottom

2. Each leading non-zero entry (pivot) is to the right of the one above it

3. All entries below each pivot are zero

In essence, the matrix becomes upper triangular.

The number of pivots in REF directly gives the rank of the matrix.

Reduced Row Echelon Form (RREF)

Reduced Row Echelon Form goes one step further:

• Each pivot is 1

• Each pivot column has zeros above and below

• Zero rows stay at the bottom

While REF is enough to solve systems, RREF makes the solution immediately visible.

REF vs RREF

• REF simplifies the matrix

• RREF completely solves it

• RREF is stricter and more structured

Both are achieved using the same row operations.

Gaussian Elimination

Gaussian elimination is the algorithmic process behind all of this.

Steps:

1. Convert equations into an augmented matrix

2. Apply row operations to reach REF

3. Use back-substitution to find variables

This procedure is the backbone of many numerical methods used in machine learning systems.

Don’t worry, if you didn’t get everything at first, reread the article.
This may feel heavy at first—and that is normal. These ideas form the mathematical backbone of optimisation, regression, and neural networks.

We will stop here for now. In the next post, we are gonna give a closure to this topic and get some more important concepts.

Till then, keep learning—and stay curious! and let the math settle naturally.

Subscribe now

If this article helped clarify things, consider subscribing so you don’t miss future deep-dives, and feel free to like or leave a comment—I’d love to hear your thoughts and what you’d like to see next.

Leave a comment

Our algorithms do something very similar when the dataset is small or noisy, because the optimisation process rewards any pattern that reduces training error, even if that pattern is a meaningless coincidence, and this is the exact moment when we need a concept called regularisation, which acts like a friendly mentor sitting beside the model and whispering that understanding matters more than memorising.

To keep things gentle and intuition driven, let us carry one simple example through the entire article. Imagine we want to predict house prices using only 120 records from a single neighbourhood, and the features include size, age, distance from metro, number of balconies, and a few quirky columns like “paint color style” that probably have little to do with actual value. A powerful model trained on this tiny table can easily find strange rules such as green-painted houses built in 2003 with two balconies are extremely expensive, and while that statement may be true for three rows in our data, it collapses brutally when a new family uploads their property next month.

A small picture of the problem

The gap between these two arrows is what regularisation tries to close, not by magic, but by reducing the model’s temptation to use extreme weights and complicated decision surfaces.

The Core Intuition

Complexity in a model is freedom, and freedom without constraints often becomes chaos. In linear models chaos shows up as huge coefficients jumping in opposite directions, and in deep networks it appears as neurons firing only for very specific training examples, like secret switches that recognise individual houses instead of general principles. Regularisation adds discipline so the model prefers simpler explanations.

In ordinary regression we minimise a loss that looks like

and the algorithm searches for any set of parameters that makes this expression small. Regularisation tells us to modify the goal and say with a long breath that error should be small and weights should not be unnecessarily large, leading to

where λ is the voice of the mentor and the term ∑wj^2 measures how wild the coefficients have become. For our house example this means that if the feature “paint color style” receives a giant weight just to fit a few points, the penalty will pull it back like an elastic band.

Bullets for the feeling

• Overfitting = memorising individual houses

• Regularisation = encouraging general taste

• Lambda = how strict we want to be

• Smaller dataset → stronger discipline

Bias and variance can also be narrated through the same streets. When we trained a very deep tree on those 120 houses, variance became high because predictions changed violently with minor data edits, and regularisation increases bias a little but reduces variance a lot, producing calmer prices that follow size and location rather than paint fantasies.

Regularisation in Classical ML

L2 – The Art of Shrinking (Ridge)

L2 regularisation assumes that many features contribute gently, and it prefers spreading responsibility across them instead of allowing one bully column to dominate. In coefficient space it literally creates a circular fence:

The maths square wj^2 grows fast, so extreme weights hurt badly. For houses this often highlights size, age, and metro distance while reducing balconies and paint color.

L1 – Learning to Ignore (Lasso)

L1 regularisation assumes that in a small and noisy dataset only a few features carry real meaning, and it prefers allowing the model to concentrate on those honest drivers instead of distributing attention across every quirky column. The loss becomes

and the absolute value has a firm habit of pushing some weights exactly to zero, which feels like feature selection happening naturally rather than being forced by us. In our neighbourhood example it might slowly realise during training that “paint color score’’ is helping only to fit three dramatic houses, and the penalty will keep reducing its coefficient until it finally becomes 0, leaving size, age, and metro distance as the main voices deciding the price. Beginners can watch this behaviour across epochs and notice with long curiosity that predictions turn simpler and more believable, as if the model has learnt to ignore gossip columns and listen only to sensible facts.

When to use which

• If we believe every house feature matters → L2

• If we suspect only a few real drivers → L1

• Unsure beginners → try both and watch curves

Training curves for the example tell long stories rather than short tweets, and beginners should read them like novels.

Moving to Deep Learning

Neural networks are more imaginative than linear models, because a tiny two-layer network for house prices may already contain 50,000 parameters, each capable of inventing secret codes for individual training homes.

The neural version of L2 is called weight decay, and the formula echoes the same philosophy:

but deep learning has additional human-friendly tricks.

Dropout – Forcing Team Work

During training we randomly switch off neurons:

where ○ means a sleeping neuron. This prevents the network from relying on one special path that recognises only three houses with green paint, and instead it learns distributed reasoning, like multiple students solving the assignment together.

Early Stopping – Leaving the Exam Hall on Time

If we keep training forever, the model eventually memorises the dataset completely, so we stop when validation error begins to rise, which in our example happened around epoch 18 even though training loss was still falling with temptation.

The Mathematics Without Fear

Regularisation modifies gradients. For L2, derivative of the penalty is

so each update becomes

which clearly shows an additional pull toward zero every iteration, like a constant reminder. Probabilistic people say this equals assuming a Gaussian prior on weights, but as beginners we can simply imagine that simpler explanations are more likely than bizarre ones.

Practical Tips for Beginners

• Choose lambda by watching validation curves

• Strong penalty for tiny datasets

• Do not regularise the bias term usually

• Remember that regularisation helps behaviour, not broken data

Common misunderstandings appear when beginners expect regularisation to rescue completely irrelevant features or believe that dropout is only for large networks, and our story demonstrates patiently that philosophy matters more than rigid rules.

Conclusion

Regularisation is less a formula and more a way of thinking that tells us with long compassionate sentences that models should learn like humans who generalise from experience instead of memorising accidental paint colors from a single lane. The house-price example we carried from start to end shows how gentle discipline turns a fragile predictor into a trustworthy appraiser, and I hope beginners feel encouraged to treat lambda as a friendly mentor rather than a scary Greek symbol.

Subscribe now

If this article helped clarify things, consider subscribing so you don’t miss future deep-dives, and feel free to like or leave a comment—I’d love to hear your thoughts and what you’d like to see next.

Leave a comment

Many learners begin their ML journey directly with code. At first, this feels productive. But over time, concepts like gradient descent, back propagation, feature spaces, and optimisation start to feel opaque. Almost always, the root cause is the same: shaky foundations in linear algebra.

So in this series, we slow things down. We start from the very basics and build upward—carefully, intuitively, and with machine learning in mind.

Machine learning systems do not “think” in words or concepts. They operate on numbers. More precisely, they operate on structured collections of numbers, and they learn by transforming those structures in consistent ways.

If this statement feels abstract right now, don’t worry. By the end of this series, it will feel obvious.

From numbers to structure

For this article, let’s consider a dataset of people’s characteristics - like height, weight, age, income.

Consider a dataset describing people:

• Height

• Weight

• Age

• Income

Each of these values is a single number. In mathematics, such a single number is called a scalar. Scalars are meaningful—temperature, age, price, probability—but on their own, they cannot represent complex data.

Machine learning rarely works with isolated numbers. Instead, it works with organised collections of numbers.

Vectors: Representing Data Points

Suppose we want to represent the heights of many individuals in a single, structured form. This naturally leads us to the concept of a vector.

A vector is simply an ordered list of numbers. In ML terms, you can think of it as one data point or one feature across many samples, depending on the context

Vectors allow us to perform mathematical operations such as addition, scaling, distance measurement, and projections—operations that are fundamental to learning algorithms.

From Vectors to Datasets

In practice, we don’t operate on one vector at a time. Algorithms like gradient descent (covered in a later post) work on entire datasets simultaneously.

Imagine a small dataset where each person is described by height, weight, and age.

But we want to operate on whole dataset for Gradient Descent like functions.(Soon post on GD).

Now the question becomes:

How do we represent this dataset so that a computer can efficiently work with it?

So, to represent this dataset we need something like this

Matrices: Structured Data at Scale

To represent an entire dataset compactly, we use a matrix.

A matrix is a rectangular array of numbers arranged in rows and columns:

• Each row typically represents one data sample

• Each column represents a feature

This structure is not arbitrary. Matrices allow us to express large collections of equations and transformations using concise mathematical notation—exactly what machine learning relies on.

Linear Equations: The Simplest Models

A linear equation describes a straight-line relationship. The most familiar form is:

y=mx+by = mx + by=mx+b

Here:

• x is the independent variable (input or feature)

• y is the dependent variable (output or target)

• m controls the slope

• b is the intercept

In machine learning, y is usually what we want to predict, while x represents known features such as age or height.

Even simple numbers can be expressed through equations. For example:

This may look trivial, but the idea is powerful: unknown values can be represented and solved using known quantities.

From One Equation to Many

Real-world datasets involve multiple features, leading to multiple equations that must be solved together.

Using matrix notation, this is commonly written as:

W*X+b=y^

This is known as a system of linear equations, where:

• W contains weights

• X is the input matrix

• b is the bias

• y^ is the predicted output

Systems of Equations: An Intuitive View

Think of a system of equations like a set of sentences describing reality.

example courtesy - DeepLearning.AI

• for the first system - this makes the definite colour to each one thus is considered complete and called Non-singular.

• for second system - it is giving same equations twice about the colour of same creature- thus redundant system of equations and are called Singular

• for the last system - the two sentences are contradictory and are called Singular

Systems of Equations in Numbers

1. Unique solution (non-singular)

   a+b=10

   a+2b=12

   → a=8,b=2

2. Infinite solutions (singular, redundant)

   2a+2b=20

   a+b=10

3. No solution (singular, contradictory)

   a+b=10

   2a+2b=24

In linear algebra terms:

• Singular systems are linearly dependent

• Non-singular systems are linearly independent

Not all equations are linear. and here are some example of non-linear equations:

Machine learning often starts with linear models because they are easier to analyze and optimize. Non-linearity is later introduced deliberately, usually through activation functions.

Geometry Behind the Math

• 2 variables → line

• 3 variables → plane

• More than 3 variables → hyperplane (cannot be visualized directly)

Despite the abstraction, these geometric interpretations guide how algorithms learn and generalise.

Stopping it here today, hopefully this much didn’t overwhelmed you. this was just the introduction, gear up for the next posts which will have a bit of mathematical representations and new terms introduced. Till then, keep learning—and stay curious!

Subscribe now

If this article helped clarify things, consider subscribing so you don’t miss future deep-dives, and feel free to like or leave a comment—I’d love to hear your thoughts and what you’d like to see next.

Leave a comment

This publication is my attempt to slow things down and focus on the fundamentals. Here, I write about AI and Machine Learning concepts the way I wish I had learned them—clearly, step by step, and with emphasis on understanding rather than memorization.

You’ll find concise explanations of Maths for ML, core ML concepts, PyTorch insights, and curated notes from courses and papers—written for clarity, not noise.

Over time, this will grow into a well-organized knowledge base I can return to—and that others can use as well.

The goal is simple: to make these ideas easier to learn, revisit, and truly understand.

If you’re building strong ML foundations or preparing for AI/ML roles, I hope this helps you learn with more confidence and less confusion.

Subscribe to get practical, high-signal ML content directly in your inbox.