Modern Search Engines in Depth: From Principles and Algorithms to Efficient Search Practice

This article aims to systematically lay out the architecture, core algorithms and information-retrieval methods of modern search engines. It draws on experience and research, so omissions are inevitable; corrections are welcome.

This piece is complementary to “Modern SEO in Depth: Principles, Algorithms and Practice”. If you’ve finished this article and are interested in SEO, you can read the follow-up, >click to jump

0. Basic Concepts#

0.1 The Definition of a Search Engine#

A search engine (Search Engine) is a class of system for efficiently retrieving information from massive datasets. It mainly consists of the following modules:

Crawling: responsible for discovering and downloading the enormous amount of web page content on the internet.
Indexing: processing the crawled content into data structures that machines can retrieve quickly.
Ranking: ranking the indexed candidate content by importance or relevance according to the query terms submitted by the user and their intent.
Query Interface: providing an interface for users to enter queries and presenting the processed search results to them.

0.2 How a Search Engine Works#

The typical workflow of a modern search engine can be summarised as:

graph TD
    A[源数据：网页、文档、多媒体等] --> B(爬取);
    B --> C{结构解析 + 数据清洗};
    C --> D[建立索引];
    D --> E[建立传统 + 语义/行为模型];
    E --> F[最终给用户展示的搜索结果UI];

0.3 Data Types#

Search engines have to handle all sorts of data types:

Structured data: data with fixed fields and a strict structure, such as databases, API responses, knowledge-base entries.
Unstructured data: free-text content with no predefined format, such as site body text, blog posts, news reports.
Semi-structured data: data organised with tags or markup, but with a looser structure, such as JSON, XML, HTML DOM structures.
Multimodal data: data containing multiple media forms, such as images, audio, video and so on. Such data must be recognised through OCR, ASR and similar technologies and converted into retrievable forms such as CLIP vectors.

1. Crawling `Crawling`#

Crawling is the first step by which a search engine obtains information: it traverses the internet through automated programs (that is, crawlers).

1.1 Link Discovery Mechanisms#

HTML <a> tag hyperlinks: the most important way pages reference one another.
Resource tags such as <link>, <script>, <img>: these tags may also hide URLs pointing to other resources.
The sitemap.xml submitted by a website: a “map” the webmaster proactively hands over, telling the crawler which pages the site has.
External link references (Backlinks): links from other sites that “recommend” this site — an important source of discovery.
Domain brute-force traversal / dictionary expansion: guessing possible URLs by exhaustively enumerating or combining common words.

1.2 Crawl Strategies `Crawl Strategies`#

BFS (breadth-first)
- Characteristics: breadth before depth, guaranteeing coverage.
- Application: usually used for general-purpose crawlers, ensuring important areas aren’t missed.
DFS (depth-first)
- Characteristics: fast penetration into a single domain.
- Application: suited to targeted collection, when you need to quickly obtain deep content on a topic or website.
Priority queue
- Strategy: score links by weight, content update frequency and other factors, prioritising “more important” pages.
- Effect: popular, authoritative, frequently updated pages are taken care of first.

1.3 Crawl Budget `Crawl Budget`#

A search engine will not crawl a website without limit; it has a “ceiling on crawl resources”, that is, a crawl budget.

Influencing factors: the website server’s response speed, the quality of page content, the quality of external links, site size, content update frequency.
Main effect: directly related to the speed at which a website’s pages are indexed by the search engine and their overall visibility in search results.

1.4 Crawl Control Rules#

Websites have several ways to guide the behaviour of search engine crawlers:

robots.txt
- Purpose: the file is usually placed in the website root directory and is used to forbid or allow crawlers to access specific paths.
- Example: Disallow: /private/ (tell the crawler not to enter this directory)
- Of course, it stops honest people, not scoundrels
<meta name="robots">
- Purpose: written directly in the <head> section of an HTML page, giving page-level control over the current page.
- Example: <meta name="robots" content="noindex, nofollow">
  - noindex: tells the search engine not to include this page in its index.
  - nofollow: tells the search engine not to follow links on this page.
HTTP Header X-Robots-Tag
- Purpose: control through the HTTP response header, allowing more granular cache and status control.
- Example: X-Robots-Tag: noarchive (tell the search engine not to archive this page)

Forbidding crawling ≠ forbidding indexing ≠ forbidding display These are three different concepts. For example, if robots.txt forbids a certain page from being crawled, but many other websites link to that page, the search engine may still know it exists and may even show its title in search results (though the content cannot be displayed). The three follow different logic, and this will be expanded on in detail in the SEO section.

2. Indexing `Indexing`#

The ultimate purpose of the index: to turn the enormous content collected by crawlers into a data structure the search engine can retrieve quickly.

2.1 Text Preprocessing#

Raw text goes through a series of processing steps before being indexed:

Tokenization
- English: relatively simple — usually just splitting words on spaces and punctuation.
- Chinese: because there are no natural separators between words, extra segmentation algorithms are needed:
  - Maximum matching (MM): try to match the longest word in the dictionary from left to right. Simple, but the error rate may be higher.
  - Reverse maximum matching (RMM): match from right to left; generally considered more effective than forward matching.
  - HMM (hidden Markov model): based on statistical probability models, better suited to large vocabularies and complex contexts.
  - BPE/SentencePiece: the mainstream approach in modern NLP — splitting words into subword units, which handles out-of-vocabulary words and works across languages.
Stopword filtering
- Removing words that are common but of little significance to search, such as 的、了、是 in Chinese and and、the、a in English.
Stemming / Lemmatization
- Unifying different forms of a word into their stem or base form — for example reducing → reduced → reduces all reduce to reduce. This effectively shrinks the index and improves the match rate.

2.2 Inverted Index `Inverted Index`#

The inverted index is one of the search engine’s most central data structures.

Core structure: it is like a dictionary whose keys are words and whose values are the list of document IDs containing that word ([document_id1, document_id2, ...]). In addition, information such as term frequency and positions is usually recorded.
What it enables:
- Fast lookup: it can quickly find all documents containing a particular keyword.
- Boolean operations: it easily supports logical operations such as AND, OR, NOT for complex queries.

2.3 Vector Index `Vector Index`#

Modern search engines rely increasingly on vector indexes to implement semantic search.

Embedding text into vector space: through learned models, text (whether a single word, a sentence or an entire document) is converted into a high-dimensional vector.
- Word2Vec / GloVe: mainly used to generate word vectors.
- BERT / RoBERTa / Embedding Models: generate more powerful semantic vectors capable of capturing the deeper meaning of text.
Using vector retrieval: these vectors are stored in specialised vector databases and support finding semantically similar content by computing distances between vectors.
- Common tools: FAISS, Milvus, HNSWlib and so on.
Effect: even if the keywords the user searches do not exactly match anything in a document, as long as it is semantically related it can still be retrieved effectively — achieving true semantic similarity search.

2.4 Content Deduplication `Content Deduplication`#

Search engines need to avoid indexing large amounts of duplicate content, which would degrade search quality. Content deduplication solves this problem.

Shingling (fixed-window slicing): split a document into many small fragments, compute a hash value for each fragment, and decide whether documents are duplicates by comparing the similarity of the hash sets.
SimHash (locality-sensitive hashing): generate a compact hash value for the whole document; similar documents receive similar SimHash values.
Purpose: prevent low-quality duplicate content — SEO scraper sites, mirror sites and the like — from polluting index quality, ensuring the diversity and authority of search results.

3. Ranking `Ranking`#

The core goal of a search engine’s ranking model is to measure the relevance between candidate documents and the user’s query, and present results in order of relevance. Modern ranking systems usually process in multiple stages to balance efficiency and accuracy:

3.1 First-stage Ranking / Recall#

Purpose: quickly and efficiently screen out the small amount of content that may be relevant to the query from the massive index. This stage prioritises recall and efficiency.
Methods:
- BM25 / TF-IDF: classic keyword-matching algorithms that compute relevance from term frequency, inverse document frequency and document length.
- Fast scoring based on the inverted index: use the inverted index to locate documents quickly and give them a preliminary score.
- Simple vector recall: perform nearest-neighbour search over the vector index to recall documents that are initially semantically relevant.

3.2 Re-ranking / Semantic Ranking#

Purpose: perform deeper semantic understanding and fine-grained ranking optimisation on the shortlisted results. This stage prioritises ranking precision.
Methods:
- LTR models: use machine learning methods to rank across a large number of features (keyword match, link quality, user behaviour, page content quality, etc.).
- Transformer representations (Embedding Models): use large pre-trained language models such as BERT and RoBERTa to encode the query and documents into semantic vectors, then match them finely by vector similarity.
- Reranker models: deep learning models designed specifically for interactive matching between a query and candidate document pairs, deeply assessing their semantic relevance.

The typical ranking pipeline of a modern search engine:

graph TD
    A[用户查询] --> B(倒排索引召回);
    B --> C(初排);
    C --> D(语义重排);
    D --> E(返回给用户);

3.3 A Citation-based Weight Model: `PageRank`#

PageRank is one of Google’s early core algorithms; it is used to assess the importance and authority of a web page. Its yardstick is not the page’s content quality itself, but:

Who cites this page, and how much it is cited

That is to say: The more it is cited by high-quality pages → the more important it is

The PageRank Formula#

The core idea of PageRank is to treat links on the internet as a kind of “vote”. The higher a page’s PageRank value ( $PR(p_i)$ ), the more important or authoritative it is generally considered to be.

PR(p_i) = \frac{1 - d}{N} + d \cdot \sum_{p_j \in M(p_i)} \frac{PR(p_j)}{L(p_j)}

Explanation:

Symbol	Meaning	How to read it
$PR(p_i)$	The PageRank value of page $p_i$	How important the page is on the Internet
$M(p_i)$	The set of all pages pointing to $p_i$	Who cites it
$L(p_j)$	The number of outbound links on page $p_j$	The citer’s contribution is split evenly
$d$	Damping factor (about 0.85)	Models the probability that the user keeps browsing
$\frac{1 - d}{N}$	Random-visit compensation	Prevents bias from dead links and closed-loop structures

note

A page’s importance depends in part on the importance of the pages citing it. The more important the citer, the greater the contribution; the more citations, the more the contribution is spread out.

PageRank measures the quality of the citation source, not the number of citations.

PageRank’s role in modern search engines#

PageRank’s status as the core of Google’s early ranking is long gone. Google stopped publicly updating the PageRank toolbar in 2016, and the direct weight of the original algorithm in today’s search engines is negligible.

The ranking algorithms of modern search engines are extremely complex, usually containing hundreds or even thousands of signals — far beyond a single PageRank model. However, the basic idea PageRank introduced, “links as votes” — its thinking and principles — is still folded into more complex and advanced link-analysis algorithms.

The PageRank-dominant era	Now
The main ranking basis	Merely one of many ranking signals
Links were highly trustworthy	Links are easily manipulated for SEO and must be judged alongside other factors
Little information, a relatively simple web structure	A complex web, where semantic understanding and user intent matter more

Today’s search ranking relies more on the following factors, among others:

Semantic matching: for example, using large models such as Embedding / Transformer to deeply understand the user’s query and the meaning of the content.
User behaviour feedback: such as CTR/Dwell Time; these metrics directly reflect user satisfaction with search results.
Content quality assessment: a comprehensive evaluation of content depth, authority, originality and expertise.
Site authority: assessing a website’s overall credibility through more complex link analysis, brand mentions, expert recommendations and the like.

Summary: the core logic of PageRank has evolved into an auxiliary factor within modern complex link analysis; its original form is no longer dominant.

note

PageRank profoundly influenced the development of search engines, but as a single ranking model it has become history. Still, its basic idea — “high-quality citations equal high value” — remains one of the foundations of link analysis in modern search engines. Citation quality > citation quantity

3.4 BM25 Text Relevance Scoring#

BM25 is a keyword-matching scoring algorithm widely used in information retrieval, used to assess the relevance between a document and a query. It is the cornerstone of the first-stage ranking in many search engines.

Formula:

\mathrm{BM25}(q, D) = \sum_{t \in q} \mathrm{IDF}(t)\cdot \frac{f(t, D)\,(k_1 + 1)} {f(t, D) + k_1\left(1 - b + b\cdot\frac{|D|}{\mathrm{avgDL}}\right)}

BM25 mainly solves several practical problems in text matching:

Problem	What BM25 does	Intuitive effect
Repeated words inflating weight	Term-frequency saturation curve	A word repeated 100 times in a document ≈ repeated 5 times, preventing excessive keyword stuffing.
Long documents shouldn’t get too much weight	Length normalization	Long articles aren’t naturally “more relevant” than short ones, removing the long-document advantage.
Rare words should be emphasised	IDF weighting	“Proper nouns” or “rare words” get higher weight than “ordinary words”.

It usually sits in the search engine’s first-stage ranking, giving preliminary scores to documents recalled through the inverted index, thereby shortlisting a top 100~1000+ candidates for the next stage.

3.5 Learning to Rank `LTR`#

As the quality gap between pages widened, user behaviour data accumulated and feature engineering matured, search engines needed a learnable ranking model to judge document relevance comprehensively. That is LTR.

Typical LTR models:

Model	Core idea	Training approach
RankNet	Based on pairwise comparison	“Document A should rank ahead of document B”
LambdaRank / LambdaMART	Directly optimises ranking metrics (e.g. NDCG)	The general choice of modern mainstream search engines; excellent results
ListNet / ListMLE	Directly optimises the ordering of the whole list	More theoretical, but training is usually more complex and difficult

Why use LTR?

Goes beyond traditional algorithms: algorithms like BM25 only look at textual features and cannot see information such as complex user behaviour and page quality.
Combines multi-dimensional features: LTR models can make comprehensive use of a huge number of features, including:
- CTR: whether the user clicked this result.
- Dwell Time: how long the user stayed on the page.
- Bounce Rate: whether the user quickly returned to the search results page.
- Historical search intent models: the user’s past search preferences and behaviour patterns.

If a user always clicks the 3rd result in the list and stays there for a long time, the ranking model will “learn” that this result may be higher quality and gradually move it to 1st place.

note

Search engine ranking is long past simply ranking “web pages written for machines”; it now truly ranks by “users voting with their feet”.

3.6 Neural Re-ranking#

Modern search — especially when handling complex queries and understanding deep user intent — has entered the stage of pure language understanding. Semantic re-ranking uses deep learning models to measure semantic relevance between query and document more precisely.

Models used:

Type	Example models	Characteristics
Sentence-embedding models	SBERT / E5 / bge	Produce dense vector representations of text, mainly used for coarse semantic recall or filtering.
Fine-ranking models	Cross-Encoder / ColBERT / RAG Reranker	Perform deep semantic comparison on a small set of candidates, computing fine-grained relevance.

Typical pipeline:

graph TD
    A[用户查询] --> B(倒排召回 / BM25);
    B --> C(粗语义召回 / Embedding);
    C --> D(精语义重排 / Cross-Encoder);
    D --> E(返回给用户);

Why not just let a large model rank every web page directly?

Because the computational cost differs exponentially — it is impossible to process all the data directly with the most complex model:

Model stage	Cost per ranking	Scale and feasibility
BM25	< 10 ms	Can quickly rank millions to hundreds of millions of web pages
Vector retrieval	1 ~ 100 ms	Can semantically filter thousands to tens of thousands of candidates
Large-model fine ranking	100 ~ 1000+ ms	Can only deeply process a few dozen shortlisted results

Ranking level	Method	Role	Scale
Recall	BM25 / TF-IDF	Quickly find possibly relevant documents	$10^6 \sim 10^8$
Semantic filtering	Vector retrieval	Find documents that are truly semantically relevant	$10^3 \sim 10^4$
Re-rank	Cross-Encoder / LTR	Find the documents that best meet the user’s needs	$10 \sim 10^2$

4. Using Search Engines Correctly#

4.1 Keywords#

The essence of information retrieval is extracting the core concepts. Avoid colloquial full sentences.

Bad example: 为什么我电脑上网速度很慢应该怎么办
Good examples:
- Breakdown: wifi slow fix driver
- Advanced: wifi latency diagnose "network adapter" windows

Core rule: start from Chinese semantics → extract key nouns → prefer English technical terms (if the problem is in a technical field).

4.2 Forcing Exact Matches#

Use double quotes " to force the search engine to match a word or phrase exactly, without splitting or rewriting it. This significantly reduces irrelevant results and improves precision.

"舞萌 DX" 评分
"start dash" maimai
"openwrt" "ipv6 relay"

4.3 Filtering by Index Scope#

Use the site: operator to limit the search to a specific website, going straight to professional sources.

site:xice.cx openwrt (find OpenWrt-related content on my blog)
site:zhihu.com maimai 难度 (search Zhihu for discussions of maimai difficulty)
site:wiki.archlinux.org f2fs (F2FS material in the Arch Linux Wiki)
site:github.com magisk (find Magisk-related projects on GitHub)

Master site:, and you can hit the most valuable, authoritative technical answers almost 100% of the time.

4.4 Filtering Out Useless Results#

Adding a minus sign - after the keywords excludes results containing specific words, effectively filtering out noise.

openwrt ipv6 优先级 -贴吧 -抖音 -小红书
maimai dx 曲目推荐 -bilibili

When Chinese search results are flooded with low-quality or marketing content, “subtraction” operators are often more important and more effective than “addition”.

4.5 Filtering by File Type#

Use the filetype: operator to search directly for a specific file type, such as PPT, PDF or Word documents.

maimai 音频 filetype:ogg (find maimai audio in OGG format)
互联网基础设施 filetype:pdf (find PDF documents about internet infrastructure)
openwrt 网络拓扑 filetype:docx (search for documents about OpenWrt networking)

With filetype:, in the vast majority of cases you can go straight to high-quality academic material.

4.6 Filtering by Time#

When information is time-sensitive (software versions, AI models, news events, for example), restricting the time range of search results is crucial.

openwrt wireguard 教程 2024 (restrict the search to 2024 WireGuard tutorials)
archlinux hyprland install 2025 (find 2025 Arch Linux Hyprland installation guides)

Or simply choose “Any time” → “Past year / Past month” in “Tools” on the Google search results page.

4.7 Reverse Lookup#

“Reverse search” can help you find the source of an image, the origin of a sentence, or similar content.

Search by image:
- Google Lens: comprehensive, good for identifying all kinds of images.
- Yandex Images: especially strong at recognising anime and ACG images.
Search for the source of a phrase (an interesting sentence/meme):
- Wrap the sentence exactly in double quotes: "原文句子" -微博 -抖音 -小红书

4.8 Turn Abstract Problems into Pattern Matching#

Don’t search directly for the specific “problem” you’ve encountered; search for the “structure” or “solution pattern” of that problem. This takes you straight from the symptom to a scientific diagnostic method.

For example, if you want to find a way to fix “unstable latency”, don’t search:

网速慢怎么办

Instead, convert it into technical concepts:

icmp latency waveform

Search is not “passively looking for an answer” but “actively locating information”. As long as you express the problem in a structured and precise enough way, the answer you seek always exists.

5. Common Misconceptions#

5.1 Using “Pouring Your Heart Out in Natural Language” Instead of Retrieval#

A search engine is not Chat GPT. It is better at handling structured keywords than long, rambling “confessions”.

Bad example 为什么我的 WiFi 总是特别卡而且有时候会掉线应该怎么办啊有没有办法解决

To a search engine, this kind of input is just noise; it is very hard to extract the core keywords accurately from it.

Good examples:
- wifi disconnect random fix / network unstable diagnose
- WiFi 掉线诊断 / 家宽网络抖动测试

Don’t pour your heart out to the search engine — state the “core words of the problem” directly.

5.2 Only Looking at Page One and Never Changing Keywords#

Most beginners, after a failed search, just complain:

“Baidu/Google can’t find anything useful at all”

A failed search is often not the search engine’s fault but a badly structured keyword. In that case, you should rewrite the keywords, not blindly page through results.

EXAMPLE

First input Rewrite
提高maimai分数 maimai 推分
wifi 卡 bufferbloat test
游戏延迟 udp relay latency diagnose

First input	Rewrite
`提高maimai分数`	`maimai 推分`
`wifi 卡`	`bufferbloat test`
`游戏延迟`	`udp relay latency diagnose`

5.3 Not Knowing How to Filter Out Invalid Information#

In the Chinese internet environment, where junk and low-quality content runs rampant, the minus-sign syntax - is especially important.

openwrt ipv6 配置教程
openwrt ipv6 教程 -知乎 -抖音 -小红书 -CSDN (directly stripping out content that is information-poor, repetitive or heavily marketing-driven)

5.4 Not Distinguishing “Answers” from “Explanations”#

Many people, when searching, habitually search for explanations, but what is truly valuable is often HOW TO DO.

Wrong input	Result	Solution search	Result
`什么是 bufferbloat`	Various theoretical definitions	`bufferbloat fix fq_codel`	Concrete configuration, optimisation commands and methods

5.5 Believing “Chinese Results Are Always Enough”#

In technical, academic or systems-related fields, search in English first. Because authoritative sources are usually originally in English.

Bad example: 游戏服务器 tick 解释
Good example: game server tick rate netcode

The information chain usually looks like this:

Paper/standard → English wiki / docs → foreign forums / blogs → Chinese translation/derivative works → 小红书 / 知乎 /CSDN

5.6 Treating “Searching” as “Finding Someone to Think for You”#

The essence of a search engine is not an answer machine; it is an efficient knowledge-mapping tool. You first need to be clear about what kind of information you’re looking for, and then search.

Classify first, then search:

Problem type	Search target	Example
Principles	wiki / RFC / MDN / Docs	`dns recursion rfc`
Configuration	Official docs / issues	`openwrt mwan3 configuration`
Troubleshooting	Log error codes	`systemd service failed exit code`

Search is not “handing the question over to the search engine unchanged”; rather, “the quality of the keywords you give determines the breadth and depth of the world you can see”.

6. How Search Engines Work Internally#

tips

This chapter focuses on readability and the core flow; it won’t grind through complex algorithmic details, but it will keep the overall structure and concepts accurate.

6.1 The Overall Search Engine Architecture#

A typical search engine’s internal architecture can be abstracted into the following five core modules:

graph TD
    A[User Query] --> B(Query Processing)
    B --> C(Index + Ranking System)
    C --> D(Document Storage + Cache)
    D --> E(SERP Rendering)

6.2 The Crawler System#

Crawlers proactively discover and download web page content across the internet.

Core mechanisms

Traversing links: continuously discovering new URLs through links on pages. Abiding by the robots.txt convention: respecting the crawl rules a website sets. Preventing infinite crawling (loop detection): avoiding endless loops or crawling redundant content. Adaptive update frequency: more popular, frequently updated pages are crawled at a higher frequency.

Basic flow: Initial URL list → crawl page → extract new links → add to the crawl queue → repeat this process.

Crawlers don’t mindlessly grab everything; they prioritise “more valuable pages” according to weight and importance.

6.3 Content Parsing and Normalization#

Crawled raw pages usually cannot be used for retrieval directly; they need a series of normalisation steps:

Step	Explanation
Remove HTML styling and scripts	Keep only the plain text content of the page.
Tokenization	Split text into meaningful word units; Chinese and English are handled with different strategies.
Lemmatization	Reduce different word forms to their base form, e.g. `studying → study`.
Stopwords	Remove common words with no retrieval value, e.g. `的 / a / the / 是 / 了`.
Inverted Index	The search engine’s core data structure, laying the foundation for fast retrieval.

An example of an inverted index:

"openwrt" → document IDs: [1, 3, 7, 9, ...]
"ipv6" → document IDs: [3, 9, 10, 14, ...]

After normalisation, each keyword corresponds to one or more document IDs. The smaller and more precise the keyword, the faster the search usually is.

6.4 Ranking#

A search engine’s ranking model is the key to the quality of search results. It usually consists of scores along two dimensions:

Type	Used for	Example
Static scoring	Judging a page’s overall intrinsic value	PageRank, domain weight, site reputation, etc.
Dynamic scoring	Reflecting the real-time interaction and satisfaction between users and pages	Behavioural statistics such as CTR, Dwell Time and bounce rate

A brief recap of how PageRank works: A page’s PageRank value is roughly equal to the sum of the PageRank values of all pages linking to it, distributed evenly according to the number of outbound links on those linking pages.

Dynamic ranking mechanisms allow search engines to continuously learn and evolve, adjusting rankings according to real user feedback.

6.5 Query Understanding#

When a user enters a query, the search engine performs complex “query processing” internally to better understand the user’s intent:

Spell correction: automatically fixing typos in the user’s input, e.g. maiami → maimai.
Synonym expansion: recognising and expanding synonyms, e.g. router ⇆ gateway ⇆ CPE.
Intent classification: determining whether the user’s search intent is to find “tutorials / principles / purchases / images / videos” and so on.
Entity recognition: identifying proper nouns or entities in the query — for example, recognising that 舞萌 is the name of a game rather than a video about “dance”.

The raw query a user enters is in fact rewritten by the search engine into a more accurate, richer internal query expression.

6.6 Presenting Results After Ranking#

The search engine results page (SERP, Search Engine Results Page) is not a pure list of links; it generates a layout dynamically according to user intent:

Technical questions: results from authoritative sites such as official documentation and Stack Overflow are usually shown first.
Encyclopedic queries: knowledge-aggregation platforms such as Wikipedia and Baidu Baike tend to be shown first.
Purchase intent: product cards and price comparisons are integrated, leading to e-commerce platforms (such as JD and Taobao).
Location-related queries: maps and business information are shown first.
Images/music/video: presented in a media grid view.

A search engine is not simply “guessing your intent”; it is performing precise intent classification and routing, directing the query to the most appropriate presentation template.

6.7 The User Feedback Loop `Feedback Loop`#

A modern search engine is a “living system” that continuously records and learns from user behaviour data, forming a feedback loop that keeps improving ranking results:

Metric	Meaning	Effect on ranking
CTR	Whether the user clicked this search result	High click-through rate → ranking weight ↑
Dwell Time	How long the user stayed on the clicked page	Long dwell time → indicates valuable content, ranking weight ↑
Pogo Sticking	The user quickly returns to the search results page after clicking	Indicates dissatisfaction with the content, ranking weight ↓

The more often you use a search engine, the more personalised its search results become.

At this point we’ve been through the broad logic of modern search engines. Here we’ll plant a flag: in the future we may do an episode of hand-holding you through building a search engine. And what would that be for? Don’t we already have Google/Bing? True — ~~but doing it really is a lot of fun~~.

Thanks For Watching.

You can reach me via Telegram or my personal email [email protected]. Corrections to this article are more than welcome — much appreciated.