Create HTTP server from Scratch

I was testing some NGINX config files last night when I started to wonder how NGINX understands it all? Somehow this curiosity inspired me to go down the rabbit hole to understand the inner workings of HTTP. What I found was a relatively uncomplicated sets of rules that all web servers abide by. In this blog, I’ll try to explain what I understood from HTTP and build a simple HTTP server from scratch in Python.

Introduction

My goal is to create a web server which accepts a POST request and it will echo back whatever JSON body it receives. Let’s start by understanding what HTTP is.

HTTP or HyperText Transfer Protocol are a set of rules that regulate the transfer of hypertext files such as HTML, media file, text, json, XML, etc. ensuring the clients and servers understand each other.

An Overview of HTTP

HTTP is a protocol that was built on top of TCP or Transmission Control Protocol which is used to connect two computers and exchange data stream across the network. TCP’s role is to ensure data are reliably delivered from source to destination computer.

Source: https://developer.mozilla.org/en-US/docs/Web/HTTP/Overview

In most cases two computers or programs connected via HTTP works in a client-server configuration where there will be multiple clients such as Web Browsers, Terminals, etc. are connected to a server. The message sent by client are called Requests and the message sent by the server as an answer is called Responses.

HTTP message Format

Before we jump into building our HTTP server from scratch, we need to understand how we should format our messages so that it becomes a valid HTTP request or response.

HTTP Requests

A valid HTTP request contains three sections, a request line, headers and request body.

1. Request Line: This is the very first line of a HTTP request, this specifies the method we want to use such as GET, POST, PUT, etc. The actual URL or route we need to access in the server and the version of HTTP standard we are using. These days, most servers use either HTTP version 2 or HTTP version 3. But for our simple HTTP server we will be building for HTTP version 1.1 which is the simplest server we can build.
2. Headers: Headers are used to convey some additional information for the server such as who is the Host of the request, where did the request originate from what is the user agent, etc.
3. Body: A body contains the actual message we want to send the server. A body is only useful for some methods such as POST, PUT or PATCH and not required of methods such as GET, DELETE, etc.

HTTP Response

Similar to request, HTTP response also contains three sections. A Response line, Headers and Body.

1. Request Line: This is the very first line of a HTTP response, this the HTTP version used for the request, a status code such as 200 for success, 500 for server error, 404 for page not found, etc. And an optional phrase that accompanies the status code.
2. Headers: Headers are used to convey some additional information for the client such as who is the Content Type of the response, total length of the message, etc.
3. Body: The body contains the response that the server wants to send the client.

Python Implementation

Finally, we can start building a simple HTTP server in python using only the builtin libraries. We will be using the socket library which provides interfaces to open and manage a TCP connection. Since HTTP is a protocol built on top of TCP this should be an appropriate starting point for us.

import socket
server = socket.socket()

With these two lines we can instantiate a simple TCP socket, now we need to provide a host and a port so that we can start listening to it.

try:
    server.bind(('localhost',4000))
    server.listen()
    print("Server ready at localhost:4000")
except Exception as e:
    server.close()
finally:
    server.close()

This will create a simple TCP server that listens to every message that is sent to localhost port 4000. Now as long as our program is listening to localhost port 4000, no other program can listen to it. Because of this reason, we have enclosed everything inside a try-except-finally block to ensure that we release the port in case something goes wrong.

At this stage, even though our program is listening to the TCP connection we have nothing in place to actually accept a connection from a client and read the message it wants to send us. Let’s do that now.

try:
    server.bind(('localhost',4000))
    server.listen()
    print("Server ready at localhost:4000")
    
    while True:
        # Accept a connection
        (clientsocket, address) = server.accept()
        print(f"Connected to {address}")
except Exception as e:
    server.close()
finally:
    server.close()

Note that the main motivation behind this article is understanding and building a simple HTTP server, not a secure one. With that being said, we made an infinite while loop inside which we accept an incoming connection. server.accept wait for an incoming connection. Return a new socket representing the connection, and the address of the client.

On a side note, I have a huge OCD about nesting my code, so like to create a lot of helper functions that will help me avoid nested code as much as possible.

Let’s create a helper function that will read all the data that the client has sent us.

def get_all_request_data(clientsocket: socket.socket)->bytes:
    data = b''
  while True:
      try:
        temp_data = clientsocket.recv(4096, socket.MSG_DONTWAIT)
      data += temp_data
    except BlockingIOError as e:
      break
  return data

Let breakdown this function:

1. We passed in the clientsocket object which we received from server.accept() from previous step.
2. We created another infinite loop that reads the data that client sent the server in chunks of 4096 bytes using clientsocket.recv(4096, socket.MSG_DONTWAIT). By default socket.recv is a blocking function. Which means that it will wait for more data even if the client has sent everything. This might me useful for some use cases but we do not need this behaviour. So to override this, we used the flag socket.MSG_DONTWAIT. Adding this flag will raise a BlockingIOError when there is no more data to be read.
3. Throughout the while loop, we read the data in chunks and append it to a data variable and finally returned this variable.

Let’s use the helper function we created to fetch client TCP data in our server.

try:
    server.bind(('localhost',4000))
    server.listen()
    print("Server ready at localhost:4000")
    
    while True:
        # Accept a connection
        (clientsocket, address) = server.accept()
        print(f"Connected to {address}")
        
        try:
        data = get_all_request_data(clientsocket)
        print("Received:")
      print(data.decode())
      print("--------------------------")
    except Exception as e:
        clientsocket.close()
    finally:
      clientsocket.close()
except Exception as e:
    server.close()
finally:
    server.close()

At this point if you run the server and perform a curl request like so.

curl localhost:4000

You can see the raw HTTP request printed in your terminal.

Now let’s make a class that would help us define valid HTTP request and response.

class Request:
    def __init__(self, req_data: bytes) -> None:
        lines = req_data.split(b'\r\n')
        raw_request_line = lines[0].decode().split(" ")
        self.method = raw_request_line[0].strip()
        self.request_url = raw_request_line[1].strip()
        self.http_version = raw_request_line[2].strip()
        self.headers = {}
        for line in lines[1:-1]:
            if len(line) == 0:
                continue
            line = line.decode().split(":")
            self.headers[line[0].strip()] = line[1].strip()
            
        self.raw_message = lines[-1].decode()

    def get_response(self, 
                    data: bytes, 
                    status_code: int = 200, 
                    content_type: str = 'text/plain') -> bytes:
        response = b''
        response += f'{self.http_version} {status_code}\r\n'.encode()
        response += f"Content-Type: {content_type}\r\n".encode()
        response += f"Content-Length: {len(data)}\r\n".encode()
        response += b'\r\n'
        response += data
        return response

This class simply does some string manipulation to parse the request data from client based on the HTTP Request format that we discussed earlier. And similarly creates an HTTP response based on the HTTP Response format we discussed.

Let’s put this all together and build a simple HTTP server.

try:
    server.bind(('localhost',4000))
    server.listen()
    print("Server ready at localhost:4000")
    
    while True:
        # Accept a connection
        (clientsocket, address) = server.accept()
        print(f"Connected to {address}")
        
        try:
        data = get_all_request_data(clientsocket)
        print("Received:")
        print(data.decode())
        print("--------------------------")
      
      request = Request(data)
      
      response = request.get_response(
                            b'<h1>Welcome to my Server</h1>', 
                            content_type='text/html')
      print("Sending:")
      print(response.decode())
      print("--------------------------")
      
      clientsocket.send(response)
    except Exception as e:
        clientsocket.close()
    finally:
      clientsocket.close()
except Exception as e:
    server.close()
finally:
    server.close()

clientsocket.send(response) will send the HTTP formatted response back to the client. Now that we are able to send and receive valid HTTP messages, we can open http://localhost:4000 on our browser and see <h1>Welcome to my Server</h1> rendered.

Finally we can build our echo server, an echo server simply returns whatever message you send.

Final Code

import socket
import traceback

PORT = 4000

class Request:
    def __init__(self, req_data: bytes) -> None:
        lines = req_data.split(b'\r\n')
        raw_request_line = lines[0].decode().split(" ")
        self.method = raw_request_line[0].strip()
        self.request_url = raw_request_line[1].strip()
        self.http_version = raw_request_line[2].strip()
        self.headers = {}
        for line in lines[1:-1]:
            if len(line) == 0:
                continue
            line = line.decode().split(":")
            self.headers[line[0].strip()] = line[1].strip()
            
        self.raw_message = lines[-1].decode()

    def get_response(self, 
                    data: bytes, 
                    status_code: int = 200, 
                    content_type: str = 'text/plain'):
        response = b''
        response += f'{self.http_version} {status_code}\r\n'.encode()
        response += f"Content-Type: {content_type}\r\n".encode()
        response += f"Content-Length: {len(data)}\r\n".encode()
        response += b'\r\n'
        response += data
        return response

def get_all_request_data(clientsocket: socket.socket)->bytes:
    data = b''
    while True:
        try:
            temp_data = clientsocket.recv(4096, socket.MSG_DONTWAIT)
            data += temp_data
        except BlockingIOError as e:
            break
    return data

server = socket.socket()
try:
    server.bind(('localhost', PORT))
    server.listen()
    print(f"Server running at port {PORT}")

    while True:
        (clientsocket, address) = server.accept()
        print(f"Connected to {address}")
        
        try:
            data = get_all_request_data(clientsocket)
            print("Received:")
            print(data.decode())
            print("--------------------------")

            request = Request(data)
            allowed_methods = ["GET", "POST"]
            status_code = 200 if request.method in allowed_methods else 405
            
            if status_code != 200:
                response = request.get_response(
                    f"{request.method} NOT Allowed".encode(), 
                    status_code=status_code)
            else:
                if request.method == "POST":
                    content_type = request.headers.get('Content-Type', 'plain/text')
                    response = request.get_response(
                        request.raw_message.encode(), 
                        content_type=content_type)
                else:
                    response = request.get_response(
                        b'<h1>Welcome to my Server</h1>', 
                        content_type='text/html')
            print("Sending:")
            print(response.decode())
            print("--------------------------")
            clientsocket.send(response)
        except Exception as e:
            clientsocket.close()
        finally:
            clientsocket.close()
except Exception as e:
    server.close()
finally:
    server.close()

And here's the output

Conclusion

Even though what we build in this article works fine, this is no way production ready. There are obviously a lot of potential for future improvements such as can we set this up to search HTML files, manage routes, manage error code.

We can also explore on how to use TLS encryption and make our server compatible with HTTPS standard. Or maybe implement newer version of HTTP like version HTTP/2 or HTTP/3.

The possibilities are endless.

References

Majority of the information I got about how HTTP works from are from two sources.

1. The official HTTP document from W3C (https://www.w3.org/Protocols/rfc2616/rfc2616.html)
2. MDN web docs: An Overview of HTTP (https://developer.mozilla.org/en-US/docs/Web/HTTP/Overview)
3. Debian recv system call manual (https://manpages.debian.org/bookworm/manpages-dev/recv.2.en.html)