Today I wanna talk about something, without the invention of which you wouldn't be reading this article, yes I'm talking about the internet, the thing that we have started taking for granted now, and it's time we start paying attention to it, because soon, the way we use the internet is about to change. With the rapidly increasing number of internet users, Governments around the world are scrambling to gain more control over it and they are trying to regulate what you do over it.
Social Networking Websites are being asked to make their moderation policies more stringent and expressing your opinions freely on the internet without any repercussions will soon become next to impossible. Even now, anyone can take down a post or video. Talking from a developer's perspective, you wouldn't ever think about an open source project being taken down, but we all saw recently that youtube-dl got taken down because of a DMCA notice.
We get our internet connection at the merci of our Government even though we as a consumer pay a premium to use this luxury that is now a necessity, the government has the ultimate power to block and ban anything from the internet at least in their country.
You all know that China heavily censors their internet to protect their communist beliefs and to shut anyone who criticises the president and the government. Ok but that's China.
It's not just China. India for example, the largest democracy in the world, banned more than 150 apps and websites including PubG, Alibaba and Ali Express just because they were created by Chinese companies. You might say, that's politics.
It's not just politics, a few years ago India banned all Adult content from the Internet just because some politician didn't like that. A massive corporate entity persuaded the government to ban two music videos just because they didn't like the criticism in the lyrics.
How the government can block a website or content?
There are several ways a government entity can ban/block a website, app or any content on the internet.
-
Issuing a court order - The government can ask the judicial system to issue a court order to the company that owns the website or content and the company have to remove such content as long as they are operating in the country.
-
DNS Lookup Blocks - I will deep dive into this topic later in the post, but in simple terms, the government can ask the DNS providers to not resolve requests for a set of domains if the request is coming from their country. This has loopholes like the DNS Provider can decline the request if the company that owns the DNS Servers does not belongs to the country. These types of blocks can be bypassed by using a custom DNS like 1.1.1.1
-
ISP Blocks - The Internet Service Provider has the ability to not serve certain requests that are coming from or going to a banned IP address. I will talk about this later in the post.
But for all these bans can be bypassed by using a VPN.
There are some countries, where using VPN is banned and the ISP and the government can track and fine VPN users even though they are using an end to end encrypted network.
I happened to live in one of these countries, where using VPN is not illegal but using a VPN to bypass blocked content is illegal. (the only thing I need a VPM for)
This is when my curiosity peeked. How is my ISP able to detect that I am using a VPN and flag accordingly even though I'm using an end to end encrypted connection that too over HTTPS?
Bandwidth Throttling
You might have noticed in the title, I've written "throttles" in parenthesis instead of just blocks. That is because, in this country's laws, using VPN is not illegal. But using a VPN to bypass blocked websites and content is (basically only reason, I use VPN for)
Because of this reason, the ISP never completely blocks a VPN but throttles it. Now what does that mean?
Bandwidth throttling is the intentional slowing or speeding of an internet service by an Internet service provider (ISP)
This is exactly what I was experiencing. If you're not living in any of the European Union, your ISP most likely has two or even more that two network channels, a fast channel where you get very fast internet speed like 4G or 5G, and a slow channel where you get only a fraction of the internet speed you are paying for.
In my research, I speculate that my particular ISP has a third channel where there is next to nothing speed maybe like 20 or 30 bytes per second. And whenever they detect that I am using a VPN, they will flag that and put my connections on the third lane.
How a normal internet connection works?
Let's say that you want to connect to some website, idiomaticprogrammers.com, you will type idiomaticprogrammers.com onto your browser and it parses and detects that it is a website url.
Then the browser creates a packet that in simple terms looks like this.
This application, let's say Chrome, wants to get the webpage in idiomaticprogrammers.com and send this packet to the router. There are many layers to this process which is not relevant in this post I will talk about them in detail in a future post.
You router looks at this packet and adds a record to its entry and replaces the local IP address with the router's IP which is actually connected to the internet.
After that, the router searches for the IP Address to where you want to send this packet to. But only information we've provided is the domain name idiomaticprogrammers.com, so the router first prepares a different packet which looks something like this.
This Router want to do a DNS lookup for idiomaticprogrammers.com and send it to a Domain Name Server. Which usually is your ISP but your can manually set the IP Address of the DNS in your router like 1.1.1.1 is Cloudflare's DNS and 8.8.8.8 is Google's DNS (I personally use 1.1.1.1)
What is DNS Lookup? It is a process of asking what is the IP address associated by this domain name.
Once the router knows the IP address for the domain name it simply replaces the domain name with the IP address and sends the request to the ISP.
Then the ISP looks at the packet can see that you want to go to this IP so it sends the request to that IP and then returns whatever information the web server provides back to the client.
Note that, if a website is using HTTPS, that is you can see a secured lock in front of the URL, there is no way for the ISP to know what information is returned by the website. Everything is encrypted end to end. I will be making an entire post about how this works in the future. But for now, this is a basic idea of how things work on the internet.
How ISP blocks a particular domain?
Remember how a router sends a DNS Lookup request to the Domain Name Server? No matter what DNS you are using either you ISP or some custom one. This request has to pass via your ISP because they are the ones who controls your internet and responsible for movement of data to and from your computer.
Every ISP has a long list domain provided by the government that they decided to block. Whenever you send a DNS Lookup request, the ISP checks if this domain is in that list. If it is, the it will be flagged and you will see a message like this domain does not exist or a page by the ISP saying this domain is blocked in this country.
Now you may ask, what if I entered the IP address directly? That might be possible, provided two conditions, 1. This IP address is not in the blacklist of the ISP and 2. The web server is configured to receive requests from the bare IP address. Believe it or not, we backend engineers do a lot of heavy lifting for you frontend devs.
How VPN works?
Going forward with the same example, let's say (god forbid) idiomaticprogrammers.com is banned in your country but you want to read that latest article we've posted. You connect to a VPN paid or free doesn't matter, at a network level the following is happening.
Your local IP is let's say 192.168.1.27 and somewhere on the planet there exist a server that has it's own private IP let's say 10.0.0.2.
Note the word private, meaning you have no way to connect to the server with private IP 10.0.0.1. What VPN does is this.
As you can see in my diagram, the VPN treats your computer as if it's a part of their network hence the word "Virtual" Private Network.
Now if you connect to idiomaticprogrammers.com your computer will make a packet like this, compare this with the packet the computer creates when it's not connected to any VPN.
There are extra information and this time it's going straight to the public IP address of the VPN, The VPN software you're. using crypts this packet and makes a new packet which essentially says, 192.168.1.27 wants to send this encrypted message to this public address let's say 123.66.6.234. The router will make this entry and forward this to the ISP
ISP looks at the packet, and just forward it this IP and it does as instructed.
The VPN server looks at the packet, decrypts it and because there are protocols, forwards it to 10.0.0.2 which then does all other processes like DNS Lookup and GET request to the website, encrypts whatever data it gets and simply forwards them back to us.
Now for idiomaticprogrammers.com, that VPN server makes all the requests and no one but the VPN and ISP knows our identity and that is what these VPN companies say "Anonymous", I say not really.
How ISP throttles VPN?
In the last part you've seen that those encrypted VPN connections pass through ISP. So theoretically, an ISP and make a list of all the VPN servers and whenever some data comes from these Public IP addresses it just says, "You might be a VPN, we have this special channel for you sir" and simply place our connection in a low bandwidth channel where the Internet speed is barely 20 or 30 bytes per second.
The same happens when you connect via TOR Protocol.
If you find some inconsistencies or theoretical error in this post please leave a comment below and I will correct myself as soon as possible.