This content originally appeared on DEV Community and was authored by Sebastien Filion
Oh, hey there!
I'm glad you made it to this second post of the "Build the System: HTTP server" series.
This post is dedicated to decoding HTTP requests and encoding the response. I will also, offer a reliable way to test
our code for a more resilient project.
If you haven't read the first post of the series yet, I think you might want to. Just click here to read it.
I'll wait patiently for your return.
This article is a transcript of a Youtube video I made.
Alright, now that I know we're all on the same page, let's write some code.
For this project, I will use JavaScript and Deno, but the concepts don't change no matter what language or runtime you
are using.
Also one last disclaimer: this project first aim is to educate it will in no way be complete or the most performant!
I will discuss specifically the improvements we can bring to make it more performant and I will go through various
iteration with that in mind. At the end of the project, if there are part worth salvaging, I will replace the essential
parts.
All that to say, just enjoy the ride.
The first thing that I need to do is to announce listening on a port.
The incoming connection will be represented by a Readable/Writable resource.
First, I will need to read from the resource a specific amount of bytes. For this example, I will read around a KB.
The variable xs is a Uint8Array. I already wrote an article about this but long story short, a Typed Array is an array
that can only hold a specific amount of bit per item. In this case we need 8 bits (or one byte) array because you need 8 bits
to encode a single UTF-8 character.
? You will find the code for this post here: https://github.com/i-y-land/HTTP/tree/episode/02
As a convenience, I will decode the bytes to a string and log the result to the console.
Finally, I will encode a response and write it to the resource.
// scratch.js
for await (const connection of Deno.listen({ port: 8080 })) {
const xs = new Uint8Array(1024);
await Deno.read(connection.rid, xs);
console.log(new TextDecoder().decode(xs));
await Deno.write(
connection.rid,
new TextEncoder().encode(
`HTTP/1.1 200 OK\r\nContent-Length: 12\r\nContent-Type: text/plain\r\n\r\nHello, World`
)
);
}
Now, I will run the code:
deno run --allow-net="0.0.0.0:8080" scratch.js
On a different terminal session I can use curl to send an HTTP request.
curl localhost:8080
On the server's terminal, we can see the request, and on the client's terminal we can see the response's body:
"Hello, World"
Great!
To get this started on the right foot, I will refactor the code into a function named serve in a file called
server.js. This function will take a listener and a function that takes a Uint8Array and returns a Promise of a
Uint8Array!
// library/server.js
export const serve = async (listener, f) => {
for await (const connection of listener) {
const xs = new Uint8Array(1024);
const n = await Deno.read(connection.rid, xs);
const ys = await f(xs.subarray(0, n));
await Deno.write(connection.rid, ys);
}
};
Notice that the read function returns the number of byte that was read. So we can use the subarray method to pass
a lense on the appropriate sequence to the function.
// cli.js
import { serve } from "./server.js";
const $decoder = new TextDecoder();
const decode = $decoder.decode.bind($decoder);
const $encoder = new TextEncoder();
const encode = $encoder.encode.bind($encoder);
if (import.meta.main) {
const port = Number(Deno.args[0]) || 8080;
serve(
Deno.listen({ port }),
(xs) => {
const request = decode(xs);
const [requestLine, ...lines] = request.split("\r\n");
const [method, path] = requestLine.split(" ");
const separatorIndex = lines.findIndex((l) => l === "");
const headers = lines
.slice(0, separatorIndex)
.map((l) => l.split(": "))
.reduce(
(hs, [key, value]) =>
Object.defineProperty(
hs,
key.toLowerCase(),
{ enumerable: true, value, writable: false },
),
{},
);
if (method === "GET" && path === "/") {
if (
headers.accept.includes("*/*") ||
headers.accept.includes("plain/text")
) {
return encode(
`HTTP/1.1 200 OK\r\nContent-Length: 12\r\nContent-Type: text/plain\r\n\r\nHello, World`,
);
} else {
return encode(
`HTTP/1.1 204 No Content\r\nContent-Length: 0\r\n\r\n`,
);
}
}
return encode(
`HTTP/1.1 404 Not Found\r\nContent-Length: 0\r\n\r\n`,
);
},
)
.catch((e) => console.error(e));
}
Now that I have an way to parse the headers, I think it's a good opportunity to officialize all of this and write a new
utility function and the appropriate tests.
// library/utilities.js
export const parseRequest = (xs) => {
const request = decode(xs);
const [h, body] = request.split("\r\n\r\n");
const [requestLine, ...ls] = h.split("\r\n");
const [method, path] = requestLine.split(" ");
const headers = ls
.map((l) => l.split(": "))
.reduce(
(hs, [key, value]) =>
Object.defineProperty(
hs,
key.toLowerCase(),
{ enumerable: true, value, writable: false },
),
{},
);
return { method, path, headers, body };
};
// library/utilities_test.js
Deno.test(
"parseRequest",
() => {
const request = parseRequest(
encode(`GET / HTTP/1.1\r\nHost: localhost:8080\r\nAccept: */*\r\n\r\n`),
);
assertEquals(request.method, "GET");
assertEquals(request.path, "/");
assertEquals(request.headers.host, "localhost:8080");
assertEquals(request.headers.accept, "*/*");
},
);
Deno.test(
"parseRequest: with body",
() => {
const request = parseRequest(
encode(
`POST /users HTTP/1.1\r\nHost: localhost:8080\r\nAccept: */*\r\n\r\n{"fullName":"John Doe"}`,
),
);
assertEquals(request.method, "POST");
assertEquals(request.path, "/users");
assertEquals(request.headers.host, "localhost:8080");
assertEquals(request.headers.accept, "*/*");
assertEquals(request.body, `{"fullName":"John Doe"}`);
},
);
Now that I have a parseRequest function, logically I need a new function to stringify the response...
// library/utilities.js
import { statusCodes } from "./status-codes.js";
export const normalizeHeaderKey = (key) =>
key.replaceAll(/(?<=^|-)[a-z]/g, (x) => x.toUpperCase());
export const stringifyHeaders = (headers = {}) =>
Object.entries(headers)
.reduce(
(hs, [key, value]) => `${hs}\r\n${normalizeHeaderKey(key)}: ${value}`,
"",
);
export const stringifyResponse = (response) =>
`HTTP/1.1 ${statusCodes[response.statusCode]}${
stringifyHeaders(response.headers)
}\r\n\r\n${response.body || ""}`;
// library/utilities_test.js
Deno.test(
"normalizeHeaderKey",
() => {
assertEquals(normalizeHeaderKey("link"), "Link");
assertEquals(normalizeHeaderKey("Location"), "Location");
assertEquals(normalizeHeaderKey("content-type"), "Content-Type");
assertEquals(normalizeHeaderKey("cache-Control"), "Cache-Control");
},
);
Deno.test(
"stringifyResponse",
() => {
const body = JSON.stringify({ fullName: "John Doe" });
const response = {
body,
headers: {
["content-type"]: "application/json",
["content-length"]: body.length,
},
statusCode: 200,
};
const r = stringifyResponse(response);
assertEquals(
r,
`HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: 23\r\n\r\n{"fullName":"John Doe"}`,
);
},
);
So now, we have everything we need to refactor our handler function and make it more concise and declarative.
import { serve } from "./library/server.js";
import {
encode,
parseRequest,
stringifyResponse,
} from "./library/utilities.js";
if (import.meta.main) {
const port = Number(Deno.args[0]) || 8080;
serve(
Deno.listen({ port }),
(xs) => {
const request = parseRequest(xs);
if (request.method === "GET" && request.path === "/") {
if (
request.headers.accept.includes("*/*") ||
request.headers.accept.includes("plain/text")
) {
return Promise.resolve(
encode(
stringifyResponse({
body: "Hello, World",
headers: {
"content-length": 12,
"content-type": "text/plain",
},
statusCode: 200,
}),
),
);
} else {
return Promise.resolve(
encode(stringifyResponse({ statusCode: 204 })),
);
}
}
return Promise.resolve(
encode(
stringifyResponse({
headers: {
"content-length": 0,
},
statusCode: 404,
}),
),
);
},
)
.catch((e) => console.error(e));
}
So at this we can deal with any simple request effectively. To wrap this up and prepare the project for future iteration,
I will add a test for the serve function. Obviously, this function is impossible to keep pure and to test without
complex integration tests -- which I keep for later.
An actual connection is a bit figety so I thought I could mock it using a file as the resource since files are
readable/wriatable.
The first thing I did is to write a function to factorize an async iterator and purposely make it break after the first
iteration.
After that, I create a file with read/write permissions. With that, I can write the HTTP request, then move the cursor
back to the beginning of the file for the serve function to read back. Within the handler function, I make some
assertions on the request for sanity's sake, then flush the content and move the cursor back to the beginning before
writing a response.
Finally, I can move the cursor back to the beginning one last time, to read the response, make one last assertion then
cleanup to complete the test.
// library/server_test.js
import { assertEquals } from "https://deno.land/std@0.97.0/testing/asserts.ts";
import { decode, encode, parseRequest } from "./utilities.js";
import { serve } from "./server.js";
const factorizeConnectionMock = (p) => {
let i = 0;
return {
p,
rid: p.rid,
[Symbol.asyncIterator]() {
return {
next() {
if (i > 0) {
return Promise.resolve({ done: true });
}
i++;
return Promise.resolve({ value: p, done: false });
},
values: null,
};
},
};
};
Deno.test(
"serve",
async () => {
const r = await Deno.open(`${Deno.cwd()}/.buffer`, {
create: true,
read: true,
write: true,
});
const xs = encode(`GET /users/1 HTTP/1.1\r\nAccept: */*\r\n\r\n`);
await Deno.write(r.rid, xs);
await Deno.seek(r.rid, 0, Deno.SeekMode.Start);
const connectionMock = await factorizeConnectionMock(r);
await serve(
connectionMock,
async (ys) => {
const request = parseRequest(ys);
assertEquals(
request.method,
"GET",
`The request method was expected to be \`GET\`. Got \`${request.method}\``,
);
assertEquals(
request.path,
"/users/1",
`The request path was expected to be \`/users/1\`. Got \`${request.path}\``,
);
assertEquals(
request.headers.accept,
"*/*",
);
await Deno.ftruncate(r.rid, 0);
await Deno.seek(r.rid, 0, Deno.SeekMode.Start);
const body = JSON.stringify({ "fullName": "John Doe" });
return encode(
`HTTP/1.1 200 OK\r\nContent-Type: image/png\r\nContent-Length: ${body.length}\r\n\r\n${body}`,
);
},
);
await Deno.seek(r.rid, 0, Deno.SeekMode.Start);
const zs = new Uint8Array(1024);
const n = await Deno.read(r.rid, zs);
assertEquals(
decode(zs.subarray(0, n)),
`HTTP/1.1 200 OK\r\nContent-Type: image/png\r\nContent-Length: 23\r\n\r\n{"fullName":"John Doe"}`,
);
Deno.remove(`${Deno.cwd()}/.buffer`);
Deno.close(r.rid);
},
);
At this point we have a good base to work from. Unfortunately our server is a bit limitted, for example, if a request
is larger than a KB, we'd be missing part of the message, that means no upload or download of medium size files.
That's what I plan to cover on the next post. This will force us to be a little bit more familiar with
manipulation of binary bytes.
At any rate, if this article was useful to you, hit the like button, leave a comment to let me know or best of all,
follow if you haven't already!
Ok bye now...
This content originally appeared on DEV Community and was authored by Sebastien Filion
Sebastien Filion | Sciencx (2021-07-05T17:18:09+00:00) Building a HTTP Server from scratch: Understanding Request & Response. Retrieved from https://www.scien.cx/2021/07/05/building-a-http-server-from-scratch-understanding-request-response/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.