This content originally appeared on DEV Community and was authored by Benny Guo
Introduction
Browser working principle is a piece of very important knowledge for frontend developers. We often use some knowledge of browser working principles to explain and understand the concept of repaint
, reflow
or CSS properties
.
Trying to figure out how the browser works by going through all the theory is rather ineffective and it's just too boring.
Here we will start from scratch and develop a simple browser using JavaScript. By creating a simple browser on our own, we will gain a deeper understanding of the browser working principles.
Less talk more code! Let's go!
Browser rendering process
General understanding of this process:
- First of all, the browser content is rendered in 5 different steps.
- When we access a web page from a URL, the page is parsed by the browser and rendered as a Bitmap.
- Last but not least, our graphics card renders the page so we can view it visually.
This is a browser's basic rendering process.
This part of the Frontend Advancement Series is only going to implement the basic functionality of the browser. For a real browser, it would include many more features, such as history, bookmarks management, user accounts, data syncing and many more.
Therefore the main goal for this part of the series is to have a good understanding of the browser's working principle.
In order to do that, we need to implement the entire process from URL request to Bitmap rendering.
Understanding the process of a browser
To understand the process a little deeper, we should go through each step of the process with more details:
- After a
URL
is entered into the browser, anHTTP
request is sent. The browser then parses the returned content and extracts the HTML. - After getting the HTML content, the browser will parse it and turn it into a
DOM
tree. - The
DOM
is basically nake at this time. The next step is to perform aCSS computation
to mount the CSS properties onto the DOM tree. At the end, we will get astyled DOM tree
. - The
styled DOM tree
we get after the computation is then useful to start forming out your page layout. - Each DOM will get a
calculated box
. (Of course, in the real browser, every CSS will generate a box, but for simplicity, we only need to calculate one box per DOM.) - Finally, we can start rendering the DOM tree, which should render CSS properties like the
background-image
or thebackground-color
onto an image. Next, the user will be able to see it through the API interface provided by the operating system and the hardware driver.
Use Finite-state Machine to parse character strings
It is important to understand one more thing before we dive into some coding.
A character string parser is required in many places throughout the browser's process. We will have a tough time implementing the code if we do not have a good "logic" management system to manage these different character string's parsing processes.
Therefore we need to use a state management system called "Finite-state Machine".
So what is Finite-state Machine (FSM)?
A Finite State Machine is a model of computation based on a hypothetical machine made of one or more states. Only one single state of this machine can be active at the same time. It means the machine has to transition from one state to another to perform different actions.
A Finite State Machine is any device storing the state of something at a given time. The state will change based on inputs, providing the resulting output for the implemented changes.
The important points here are the following:
-
Every state is a machine
- Every machine is decoupled from each other, it is a powerful abstract mechanism
- In each machine, we can do calculations, storage, output and etc.
- All these machines receive the same input
- Each state machine itself should have no state. If we express it as a pure function, it should have no side effects.
-
Every machine knows the next state
- Every machine has a definite next state (Moore state machine)
- Each machine determines the next state based on input (Mealy state machine)
For an in-depth explanation of Finite-state Machine, check out the article here.
How to implement FSM in JavaScript?
Mealy state machine:
// Every function is a state
// Function's parameter is an input
function state (input) {
// Inside the function, we can write our code
// for the current state
// Return the next State function
return state2;
}
/** =========
* To run the state matching
* ========= */
while (input) {
state = state(input);
}
- In the above code, we see that each function is a state
- Then the parameter of the function is
input
- The return value of this function is the next state, which implies that the next return value must be a state function.
- n ideal implementation of a state machine is: "A series of state functions that return a batch of state functions."
- When state functions are invoked, a loop is often used to obtain the input, then
state = state(input)
is used to let the state machine receive input to complete the state switch. -
Mealy
type state machine's return value must be based on theinput
to return the next state. -
Moore
type state machine's return value is not related to input, instead, it returns a fixed state.
What if we don't want to use FSM?
Let's take a look at what we can use if we don't want to use FSM to process the character strings in our simple browser.
In the same way that you would perform a science experience, you would attempt to defend your solution by using a different method or theory. It turns out that you already have the right solution.
What we do here is the same, let's look at how to implement the parse character strings without using a state machine.
We will learn this by going through a few challenges:
Challenge 1: Find the character "a" in a character string.
function match(string) {
for (let letter of string) {
if (letter == 'a') return true;
}
return false;
}
console.log(match('I am TriDiamond'));
Easy, isn't it?
Challenge 2: Find the character "ab" in a character string without using regular expression. Try implementing it with just pure JavaScript.
Tip: Look for
a
andb
directly, return when both are found
function matchAB(string) {
let hasA = false;
for (let letter of string) {
if (letter == 'a') {
hasA = true;
} else if (hasA && letter == 'b') {
return true;
} else {
hasA = false;
}
}
return false;
}
console.log( matchAB('hello abert'));
Challange 3: Find the character "abcdef" in a character string without using regular expression. Again try implementing it with just pure JavaScript.
There are 3 ways to approach this challenge.
Method1: By using storage space and move the key pointer to find our target.
/**
* @param {*} match String that you need to match
* @param {*} string String that you are matching against
*/
function matchString(match, string) {
// Break up matching string characters into an array
const resultLetters = match.split('');
// Break up the string characters that you are matching against into an array
const stringArray = string.split('');
let index = 0; // The matching index
for (let i = 0; i <= stringArray.length; i++) {
// Make sure the strings are absolutely matched
// eg. "abc" and "ab" should not be matched.
// Therefore we require the string characters have
// the correct orders
if (stringArray[i] == resultLetters[index]) {
// If one matching character is found
// index + 1 to move to the next character
index++;
} else {
// If the next character is not matched
// reset the index and match all over again
index = 0;
}
// If all characters of the string is matched
// return true immediately, which means
// `match` string is founded in our `string`
if (index > resultLetters.length - 1) return true;
}
return false;
}
console.log('Method 1', matchString('abcdef', 'hello abert abcdef'));
Method2: Using the substring
function to intercept the matching string characters to check whether they are equal to the answer.
function matchWithSubstring(match, string) {
for (let i = 0; i < string.length - 1; i++) {
if (string.substring(i, i + match.length) === match) {
return true;
}
}
return false;
}
console.log('Method 2', matchWithSubstring('abcdef', 'hello abert abcdef'));
Method 3: Search the characters one by one until you find the final result.
function match(string) {
let matchStatus = [false, false, false, false, false, false];
let matchLetters = ['a', 'b', 'c', 'd', 'e', 'f'];
let statusIndex = 0;
for (let letter of string) {
if (letter == matchLetters[0]) {
matchStatus[0] = true;
statusIndex++;
} else if (matchStatus[statusIndex - 1] && letter == matchLetters[statusIndex]) {
matchStatus[statusIndex] = true;
statusIndex++;
} else {
matchStatus = [false, false, false, false, false, false];
statusIndex = 0;
}
if (statusIndex > matchLetters.length - 1) return true;
}
return false;
}
console.log('Method 3', match('hello abert abcdef'));
Parsing characters using a state machine
Now let's look at how we process the characters by using a state machine.
To demonstrate how to process characters using a state machine, we going to solve the 3rd challenge using state machine:
Challange 3: Find the character "abcdef" in a character string without using regular expression. Again try implementing it with just pure JavaScript.
First, let's think about how are we going to do it with state machine:
- First of all, every state is a
state function
- We should have a
start
state and anend
state function, which we would call themstarting
andending
respectively - Every state function's name represents the previous matched state of a specific character
- Eg.
matchedA
means thea
character is being matched in the previous state function.
- Eg.
- The logic in each state matches the next character
- Therefore the current state function is processing the next state logic.
- Eg. If the current function name is
matchedA
, the logic inside it is to process when the character is equal tob
- If the match fails, return the
start
state - Because the last of the characters is an
f
, therefore aftermatchedE
succeeds, we can directly return to theend
state - The
End
state is also known as the 'Trap method' since the state transition is finished, we can let the state stay here until the loop is finished.
/**
* Character matching state machine
* @param {*} string
*/
function match(string) {
let state = start;
for (let letter of string) {
state = state(letter); // Switch state
}
// If the ending state is `end` return true
return state === end;
}
function start(letter) {
if (letter === 'a') return matchedA;
return start;
}
function end(letter) {
return end;
}
function matchedA(letter) {
if (letter === 'b') return matchedB;
return start(letter);
}
function matchedB(letter) {
if (letter === 'c') return matchedC;
return start(letter);
}
function matchedC(letter) {
if (letter === 'd') return matchedD;
return start(letter);
}
function matchedD(letter) {
if (letter === 'e') return matchedE;
return start(letter);
}
function matchedE(letter) {
if (letter === 'f') return end(letter);
return start(letter);
}
console.log(match('I am abcdef'));
Escalation of the challenge: Parsing of the character string "abcabx" with a state machine.
- The main difference in this challenge is that the letters "ab" appears twice.
- So the logic of our analysis should be:
- The first "b" is followed by a "c", while the second "b" should be followed by an "x"
- Go back to the previous state function if the character after the second "b" isn't an "x"
function match(string) {
let state = start;
for (let letter of string) {
state = state(letter);
}
return state === end;
}
function start(letter) {
if (letter === 'a') return matchedA;
return start;
}
function end(letter) {
return end;
}
function matchedA(letter) {
if (letter === 'b') return matchedB;
return start(letter);
}
function matchedB(letter) {
if (letter === 'c') return matchedC;
return start(letter);
}
function matchedC(letter) {
if (letter === 'a') return matchedA2;
return start(letter);
}
function matchedA2(letter) {
if (letter === 'b') return matchedB2;
return start(letter);
}
function matchedB2(letter) {
if (letter === 'x') return end;
return matchedB(letter);
}
console.log('result: ', match('abcabcabx'));
That's it!
After we had compared the parsing of a character string with and without a state machine. There is an obvious difference that we can observe.
When parsing with a state machine, the logic is much more manageable, while without a state machine it can be confusing and hard to understand.
The basics of HTTP protocol parsing
To understand the basic of the HTTP protocol, first we need to know what is the OSI Model.
The OSI Model (Open Systems Interconnection Model) is a conceptual framework used to describe the functions of a networking system.
ISO-OSI 7 layer model
HTTP
- Composition:
- Application
- Representation
- Conversation
In
node.js
, we have a very familiar package calledhttp
TCP
- Composition:
- Network
- There are two meanings for the term "internet"
- Protocol (
extranet
) of the application layer where the web page is located —— it is theinternet
that is responsible for data transmission - Company
intranet
—— it's thelocal
network build inside a company.
- Protocol (
4G/5G/Wi-Fi
- Composition:
- Data link
- Physical layer
- In order to complete an accurate transmission of data
- Transmissions are all done by point-to-point
- There must be a direct connection for transmissions
TCP and IP
-
Stream
- Stream is the main concept of transmitting data in the TCP layer
- A stream is a unit that has no apparent division
- It only guarantees that the order before and after is consistent
-
Port
- The TCP protocol is used by the software inside the computer
- Every piece of software get the data from the network card
- The port identifies which data is allocated to which software
- Just like the
net
package innode.js
-
Package
- Packages in TCP is transported one after another
- Each package can be large or small
- The size of each package is depended on the transmission capacity of your network intermediate equipment
-
IP Address
- An IP address is used to locate where the package should go.
- The connection relationship on the internet is very complicated, and there will be some large routing nodes in the middle.
- When we connected to an IP address, it first connects to the address of our house cable, then goes to the telecommunication company's cable.
- If you are visiting a foreign country's IP address, you will go to the main international address
- Every IP address is a unique identifier that connects to every device on the internet
- So the IP packet find out where it needs to be transmitted through the IP address
-
Libnet/libpcap
- The IP protocol needs to call these two libraries in
C++
-
Libnet
is responsible for constructing IP packets and sending them out -
Labpcap
is responsible for grabbing all IP packets flowing through the network card. - If we use switches instead of routers to build our network, we can use the
labpcap
package to catch many IP packages that do not belong to us
- The IP protocol needs to call these two libraries in
HTTP
- Composition
- Request
- Response
- HTTP works as a full-duplex channel, which means it can do both sending and receiving, and there is no priority relationship between them.
- In particular, HTTP must first be initiated by the client with a request
- Then the server comes back with a response
- So every request must have a response
Implement HTTP request
HTTP requests - server-side environment preparation
Before we write our own browser, we need to set up a node.js
server.
First by writing the following node.js
script:
const http = require('http');
http
.createServer((request, response) => {
let body = [];
request
.on('error', err => {
console.error(err);
})
.on('data', chunk => {
body.push(chunk.toString());
})
.on('end', () => {
body = Buffer.concat(body).toString();
console.log('body', body);
response.writeHead(200, { 'Content-Type': 'text/html' });
response.end(' Hello World\n');
});
})
.listen(8080);
console.log('server started');
Understanding HTTP Request Protocol
Before writing our client code, we need to understand the HTTP request protocol.
Let's first look at the request section of the HTTP protocol
POST/HTTP/1.1
Host: 127.0.0.1
Content-Type: application/x-www-form-urlencoded
field1=aaa&code=x%3D1
The HTTP protocol is a text type protocol, text type protocol is generally relative to the binary protocol. In another word, means that all the contents of this protocol are character strings and each byte is part of the character string.
- The first line:
request line
and contains three parts- Method: Eg.
POST
,GET
- Path: default is "
/
" - HTTP and HTTP version:
HTTP/1.1
- Method: Eg.
- Follow by
headers
- Each row is split with a colon in
key: value
format - Headers end with a blank line
- Each row is split with a colon in
- Last part is
body
- The content of this section is determined by
Content-Type
- The body's content format is based on
Content-Type
specify,
- The content of this section is determined by
Implement HTTP Requests
Goal:
- Design an HTTP request class
- Content-type is a required field with a default value
- Body is in key-value format
- Different Content-type affect body formatting
Request Class
class Request {
constructor(options) {
// Fill in the default values
this.method = options.method || 'GET';
this.host = options.host;
this.port = options.port || 80;
this.path = options.path || '/';
this.body = options.body || {};
this.headers = options.headers || {};
if (!this.headers['Content-Type']) {
this.headers['Content-Type'] = 'application/x-www-form-urlencoded';
}
// Convert the body format base on Content-Type
if (this.headers['Content-Type'] === 'application/json') {
this.bodyText = JSON.stringify(this.body);
} else if (this.headers['Content-Type'] === 'application/x-www-form-urlencoded') {
this.bodyText = Object.keys(this.body)
.map(key => `${key}=${encodeURIComponent(this.body[key])}`)
.join('&');
}
// Auto calculate body content length, if the length isn't valid, meaning it's an invalid request
this.headers['Content-Length'] = this.bodyText.length;
}
// Sending request, return Promise object
send() {
return new Promise((resolve, reject) => {
//......
});
}
}
Request Method
/**
* Request method using the Request Class
*/
void (async function () {
let request = new Request({
method: 'POST',
host: '127.0.0.1',
port: '8080',
path: '/',
headers: {
['X-Foo2']: 'custom',
},
body: {
name: 'tridiamond',
},
});
let response = await request.end();
console.log(response);
})();
Implement the send function
The logic of our send
function:
- Send function is in a form of Promise
- The response content will be gradually received during the sending process
- Construct the response and let the Promise resolve
- Because the process receives information one by one, we need to design a
ResponseParser
- In this way, the parser can construct different parts of the response object while gradually receiving the response information
send() {
return new Promise((resolve, reject) => {
const parser = new ResponseParser();
resolve('');
});
}
Implement HTTP response
Design the ResponseParser
The logic of our ResponseParser
:
- Need a
receive
function that collects the character string - Then use the state machine to process the string character by character
- So we need to loop each character string and then add the
recieveChar
function to process each of them
class ResponseParser {
constructor() {}
receive(string) {
for (let i = 0; i < string.length; i++) {
this.receiveChar(string.charAt(i));
}
}
receiveChar(char) {}
}
This is the basic structure of our ResponseParser
.
Understanding HTTP Response Protocol
In this section, we need to parse the contents in the HTTP response. So we will first analyse the HTTP response content.
HTTP / 1.1 200 OK
Content-Type: text/html
Date: Mon, 23 Dec 2019 06:46:19 GMT
Connection: keep-alive
26
<html><body> Hello World <body></html>
0
- The
status line
in the first line is opposite to therequest line
- The first part is the version of the HTTP protocol:
HTTP/1.1
- The second part is the HTTP status code:
200
(We can mark the state other than 200 as an error in our browser implementation to make it easier.) - The third part is HTTP status:
OK
- The first part is the version of the HTTP protocol:
- Follow by the
header
section- HTML requests and responses contain headers
- Its format is exactly the same as the request
- The final line of this section will be a blank line, used to divide the headers and the body content
- Body part:
- The format of the body here is also determined by Content-Type
- Here is a typical format called
chunked body
(A default format returned by Node) - The chunked body will start with a line with a hexadecimal number
- Follow by the content section
- Finally ended with a hexadecimal
0
, this is the end of the whole body
Implement the logic of send request
After we have a good understanding of the response protocol, we need a working send request to test and implement our Response Parser.
Design thoughts:
- Supports existing connections or adding new connections
- Passing the received data to the parser
- Resolve the Promise base on the parser's status
Let's see how we implement this.
send(connection) {
return new Promise((resolve, reject) => {
const parser = new ResponseParser();
// First check if connection is avaliable
// If not use Host and Port to create a TCP connection
// `toString` is used to build our HTTP Request
if (connection) {
connection.write(this.toString());
} else {
connection = net.createConnection(
{
host: this.host,
port: this.port,
},
() => {
connection.write(this.toString());
}
);
}
// Listen to connection's data
// Pass the data to the parser
// If parser had finished, we can start the resolve
// Then break off the connection
connection.on('data', data => {
console.log(data.toString());
parser.receive(data.toString());
if (parser.isFinished) {
resolve(parser.response);
connection.end();
}
});
// Listen to connection's error
// If the request had an error,
// first reject this Promise
// Then break off the connection
connection.on('error', err => {
reject(err);
connection.end();
});
});
}
/**
* Building HTTP Request text content
*/
toString() {
return `${this.method} ${this.path} HTTP/1.1\r
${Object.keys(this.headers)
.map(key => `${key}: ${this.headers[key]}`)
.join('\r\n')}\r\r
${this.bodyText}`;
}
Implement the RequestParser Class
Now let's implement the logic for our RequestParser
Class.
Logic:
- Response must be constructed by sections, so we are going to use Response Parser to assemble it.
- Use a state machine to analyze the text structure
Parsing the header
class ResponseParser {
constructor() {
this.state = this.waitingStatusLine;
this.statusLine = '';
this.headers = {};
this.headerName = '';
this.headerValue = '';
this.bodyParser = null;
}
receive(string) {
for (let i = 0; i < string.length; i++) {
this.state = this.state(string.charAt(i));
}
}
receiveEnd(char) {
return receiveEnd;
}
/**
* Waiting status line context
* @param {*} char
*/
waitingStatusLine(char) {
if (char === '\r') return this.waitingStatusLineEnd;
this.statusLine += char;
return this.waitingStatusLine;
}
/**
* Waiting for status line ends
* @param {*} char
*/
waitingStatusLineEnd(char) {
if (char === '\n') return this.waitingHeaderName;
return this.waitingStatusLineEnd;
}
/**
* Waiting for the Header name
* @param {*} char
*/
waitingHeaderName(char) {
if (char === ':') return this.waitingHeaderSpace;
if (char === '\r') return this.waitingHeaderBlockEnd;
this.headerName += char;
return this.waitingHeaderName;
}
/**
* Waiting for Header empty space
* @param {*} char
*/
waitingHeaderSpace(char) {
if (char === ' ') return this.waitingHeaderValue;
return this.waitingHeaderSpace;
}
/**
* Waiting for the Header value
* @param {*} char
*/
waitingHeaderValue(char) {
if (char === '\r') {
this.headers[this.headerName] = this.headerValue;
this.headerName = '';
this.headerValue = '';
return this.waitingHeaderLineEnd;
}
this.headerValue += char;
return this.waitingHeaderValue;
}
/**
* Waiting for the Header ending line
* @param {*} char
*/
waitingHeaderLineEnd(char) {
if (char === '\n') return this.waitingHeaderName;
return this.waitingHeaderLineEnd;
}
/**
* Waiting for Header content end
* @param {*} char
*/
waitingHeaderBlockEnd(char) {
if (char === '\n') return this.waitingBody;
return this.waitingHeaderBlockEnd;
}
}
Parsing the body content
Logic:
- Response body may have a different structure depending on the Content-Type, so we will use the structure of the sub-parser to solve this problem
- Take
ChunkedBodyParser
as an example, we also use a state machine to deal with the format of the body
Adding a state function for body parsing:
/**
* Response 解析器
*/
class ResponseParser {
constructor() {
this.state = this.waitingStatusLine;
this.statusLine = '';
this.headers = {};
this.headerName = '';
this.headerValue = '';
this.bodyParser = null;
}
/** ... Previous codes ... **/
/**
* Waiting for Header content end
* @param {*} char
*/
waitingHeaderBlockEnd(char) {
if (char === '\n') return this.waitingBody;
return this.waitingHeaderBlockEnd;
}
/** Adding a state function for body parsing **/
/**
* Waiting for body content
* @param {*} char
*/
waitingBody(char) {
this.bodyParser.receiveChar(char);
return this.waitingBody;
}
}
Adding ChunkedBodyParser
class:
class ChunkedBodyParser {
constructor() {
this.state = this.waitingLength;
this.length = 0;
this.content = [];
this.isFinished = false;
}
receiveChar(char) {
this.state = this.state(char);
}
/**
* Waiting for Body length
* @param {*} char
*/
waitingLength(char) {
if (char === '\r') {
if (this.length === 0) this.isFinished = true;
return this.waitingLengthLineEnd;
} else {
// Convert the hexdecimal number
this.length *= 16;
this.length += parseInt(char, 16);
}
return this.waitingLength;
}
/**
* Waiting for Body line end
* @param {*} char
*/
waitingLengthLineEnd(char) {
if (char === '\n') return this.readingTrunk;
return this.waitingLengthLineEnd;
}
/**
* Reading Trunk content
* @param {*} char
*/
readingTrunk(char) {
this.content.push(char);
this.length--;
if (this.length === 0) return this.waitingNewLine;
return this.readingTrunk;
}
/**
* Waiting for a new line
* @param {*} char
*/
waitingNewLine(char) {
if (char === '\r') return this.waitingNewLineEnd;
return this.waitingNewLine;
}
/**
* Waiting for line end
* @param {*} char
*/
waitingNewLineEnd(char) {
if (char === '\n') return this.waitingLength;
return this.waitingNewLineEnd;
}
}
Finally
In this section of the Frontend Advancement Series, we have implemented the browser HTTP Request, HTTP Response parser.
In the next section, we will talk about how to use the parsed HTTP to build a DOM tree.
Happy coding!~
Recommended Open Source Projects
Hexo Theme Aurora
Hexo Aurora Theme released today!
Benny Guo ・ Mar 30 ・ 4 min read
VSCode Aurora Future theme
A futuristic dark theme `Aurora Future`
Benny Guo ・ Apr 17 ・ 2 min read
Firefox Aurora Future
This content originally appeared on DEV Community and was authored by Benny Guo
Benny Guo | Sciencx (2021-05-13T13:17:44+00:00) How the browser works – HTTP request and parsing. Retrieved from https://www.scien.cx/2021/05/13/how-the-browser-works-http-request-and-parsing/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.