To give an overview of what the code was doing, it was making an API Post request which took more than 5-6 mins in some cases to return a response. So, as a sane thing to do, we added a request timeout of 10 mins. This is how the sample code looked.

from httpx import AsyncClient
async with AsyncClient(base_url="https://the-called-service", timeout=600) as client:
	_ = await client.post("/a_long_open_connection")

Assuming the connection returns in 4-5 mins, we should be seeing successes. But on the cluster, the requests were getting timed out after 600 seconds.

Now this was weird, as AWS has a pretty reliable network and considering that this was running quite nicely on my shitty home internet, something was wrong.

After a bit of searching i found this on AWS page.

Internet Connection Drops after 350 seconds

Problem
Your instances can access the internet, but the connection drops after 350 seconds.

Cause
If a connection that's using a NAT gateway is idle for 350 seconds or more, the connection times out.
When a connection times out, a NAT gateway returns an RST packet to any resources behind the NAT gateway that attempt to continue the connection (it does not send a FIN packet).

Solution
To prevent the connection from being dropped, you can initiate more traffic over the connection. Alternatively, you can enable TCP keepalive on the instance with a value less than 350 seconds.

By default linux has a keep alive timeout of 2 hours and this is well less than that. As to why AWS did it, i am not sure, and considering this, the code is running on EKS cluster which i have no control over. We needed a way to fix this.

So, how do we keep alive this connection. Well, lets just send periodic calls to some other endpoint which returns instantaneously.

So, we write a periodic async wrapper like so.  

from __future__ import annotations

import asyncio
import logging
from asyncio import ensure_future
from functools import wraps
from traceback import format_exception
from typing import Any, Callable, Coroutine, Union

from starlette.concurrency import run_in_threadpool

NoArgsNoReturnFuncT = Callable[[], None]
NoArgsNoReturnAsyncFuncT = Callable[[], Coroutine[Any, Any, None]]
NoArgsNoReturnDecorator = Callable[[Union[NoArgsNoReturnFuncT, NoArgsNoReturnAsyncFuncT]], NoArgsNoReturnAsyncFuncT]


def repeat_every(
        *,
        seconds: float,
        wait_first: bool = False,
        logger: logging.Logger | None = None,
        raise_exceptions: bool = False,
        max_repetitions: int | None = None,
) -> NoArgsNoReturnDecorator:
    """
    This function returns a decorator that modifies a function, so it is periodically re-executed after its first call.

    The function it decorates should accept no arguments and return nothing. If necessary, this can be accomplished
    by using `functools.partial` or otherwise wrapping the target function prior to decoration.

    Parameters
    ----------
    seconds: float
        The number of seconds to wait between repeated calls
    wait_first: bool (default False)
        If True, the function will wait for a single period before the first call
    logger: Optional[logging.Logger] (default None)
        The logger to use to log any exceptions raised by calls to the decorated function.
        If not provided, exceptions will not be logged by this function (though they may be handled by the event loop).
    raise_exceptions: bool (default False)
        If True, errors raised by the decorated function will be raised to the event loop's exception handler.
        Note that if an error is raised, the repeated execution will stop.
        Otherwise, exceptions are just logged and the execution continues to repeat.
        See https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.set_exception_handler for more info.
    max_repetitions: Optional[int] (default None)
        The maximum number of times to call the repeated function. If `None`, the function is repeated forever.
    """

    def decorator(func: NoArgsNoReturnAsyncFuncT | NoArgsNoReturnFuncT) -> NoArgsNoReturnAsyncFuncT:
        """
        Converts the decorated function into a repeated, periodically-called version of itself.
        """
        is_coroutine = asyncio.iscoroutinefunction(func)

        @wraps(func)
        async def wrapped() -> None:
            repetitions = 0

            async def loop() -> None:
                nonlocal repetitions
                if wait_first:
                    await asyncio.sleep(seconds)
                while max_repetitions is None or repetitions < max_repetitions:
                    try:
                        if is_coroutine:
                            await func()  # type: ignore
                        else:
                            await run_in_threadpool(func)
                        repetitions += 1
                    except Exception as exc:
                        if logger is not None:
                            formatted_exception = "".join(format_exception(type(exc), exc, exc.__traceback__))
                            logger.error(formatted_exception)
                        if raise_exceptions:
                            raise exc
                    await asyncio.sleep(seconds)

            ensure_future(loop())

        return wrapped

    return decorator

Thanks to the awesome package called fastapi-utils where this code is borrowed from. You can check the project out here.

And then we modify our initial code like so.

from httpx import AsyncClient
import asyncio

async with AsyncClient(base_url="https://the-called-service", timeout=600) as client:
	@repeat_every(seconds=60)
	async def continuous_poll():
    		_ = await client.get("/healthcheck")	
	
_ = asyncio.create_task(continuous_poll())
_ = await client.post("/a_long_open_connection")
    

And viola it now works.

Thats for all. Thanks for reading.

Potential Solutions

1. Shard count can be reduced by deleting or closing indices or by re-indexing into bigger indices, refer link and link
2. Finally we can also delete old/unnecessary indices to free up space and increase performance and then update the sharding strategy

There are a lot of chances that your WSL2 environment size is going to blow up. It is very easy for it to reach 100GB in size. In these cases it is better to move the vhdx file from the system drive to an external ssd or some other non system drive.

Although there are commands provided by WSL2 for these purposes with the flags --import and --export, these are not exactly stable.

The --import command has a bug wherein it needs ram, the size of vhdx to be imported correctly. Now obviously most of us dont have huge amounts of ram lying around.

Im going to show how you can do this an easy way without spending time doing import/export and waiting for those operations to blow up.

1. Shutdown wsl using wsl --shutdown command.

2. Find the path to your vhdx location. Refer Here

3. Now copy this vhdx file to the place where you want to migrate to.

4. Create a file with extension .reg and add these contents

   Windows Registry Editor Version 5.00
   
   [HKEY_USERS\SID\SOFTWARE\Microsoft\Windows\CurrentVersion\Lxss\{UUID}]
   
   "State"=dword:00000001
   
   "DistributionName"="distribution name"
   
   "Version"=dword:00000002
   
   "BasePath"="vhdx folder path"
   
   "Flags"=dword:0000000f
   
   "DefaultUid"=dword:000003e8

   • the vhdx folder path should be in the following format \\\\?\\E:\\WSL\\ubuntu-20
   • SID can be found by typing the following in command prompt whoami /USER
   • UUID can be generated randomly using anything. i used this website
   • distribution name can be any string identifier.

5. Double click the file and let it add content to your registry.

6. Run wsl --list and you should be able to see the new wsl entry created.

7. Clear the older entry by using wsl --unregister command.

NOTE:

1. If you get permission denied error, right click the .vhdx file, go the security and transfer the ownership of that file to the current user or the SID displayed above.
2. If the path in above format does not work, you can try the folowing format \\?\E:\WSL\ubuntu-20\

Thanks for reading and i hope this helped some of you guys facing this problem.

To use colab with vscode, first run the below code inside a colab notebook

import random, string
password = ''.join(random.choice(string.ascii_letters + string.digits) for i in range(20))

#Download ngrok
! wget -q -c -nc https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip
! unzip -qq -n ngrok-stable-linux-amd64.zip
#Setup sshd
! apt-get install -qq -o=Dpkg::Use-Pty=0 openssh-server pwgen > /dev/null
#Set root password
! echo root:$password | chpasswd
! mkdir -p /var/run/sshd
! echo "PermitRootLogin yes" >> /etc/ssh/sshd_config
! echo "PasswordAuthentication yes" >> /etc/ssh/sshd_config
! echo "LD_LIBRARY_PATH=/usr/lib64-nvidia" >> /root/.bashrc
! echo "export LD_LIBRARY_PATH" >> /root/.bashrc

#Run sshd
get_ipython().system_raw('/usr/sbin/sshd -D &')

#Ask token
print("Copy authtoken from https://dashboard.ngrok.com/auth")
import getpass
authtoken = getpass.getpass()

#Create tunnel
get_ipython().system_raw('./ngrok authtoken $authtoken && ./ngrok tcp 22 &')
#Print root password
print("Root password: {}".format(password))
#Get public address
! curl -s http://localhost:4040/api/tunnels | python3 -c \
    "import sys, json; print(json.load(sys.stdin)['tunnels'][0]['public_url'])"

What we are essentially doing here is, using ngrok to enable ssh connections to this instance.

Now just go to your ngrok page and get the ip and port. the username is root and the password is displayed when the above code is run.

Now, you can just use these and connect via VSCode

Disclaimer: The source code used above was from the following source

Most of the new generation routers ship with a restore mechanism via TFTP protocol and since TFTP is generally a part of the bootloader, it is almost always possible to restore a router using this method.

The idea behind TFTP restore is a simple one,

1. The router needs to be started in TFTP client mode
2. Routers asks a TFTP server running at <SOME_IP> to send it a file named  <FILENAME.bin>
3. If the server sends this file, router then uses this file to flash itself.

So now that we have this cleared, lets get started. And yes, you will need to connect to the router via an ethernet on the router LAN port because wifi and other stuffs won't work.

Start the router in TFTP client mode

Assuming that the router is connected to a computer via LAN cable, to start the router in this mode, we need to press a combination of hardware buttons on the router. In most of the router, this is holding the reset button and then pressing the power button.

You will see the networking adapter turn to unidentified network from unplugged cable mode.

Identifying the TFTP server IP and Filename

Now that we know how to start the router in TFTP mode, we need to know what IP and filename is the router looking at.

1. To do this, we first download a wonderful network monitoring tool called Wireshark.
   Download and install this tool, and then start monitoring the ethernet network adapter for your computer
2. Now restart the router in TFTP mode and you should start seeing some packets in Wireshark. What we are looking for are packets like this Who has 192.168.0.100? Tell 192.168.0.66
   Now what this packet says is that the router is looking for ip 192.168.0.100 as the TFTP server and that the ip of the router itself is 192.168.0.66
   Bingo, we got out TFTP server IP.
3. Now set the static ip of your computer to 192.168.0.100 and netmask as 255.255.255.0 and restart the router in TFTP mode again. Dont turn off wireshark yet, since we need the filename and keep on monitoring the network adapter.
4. This time on wireshark, you should see something like Read Request, File: recovery.bin
   What this means is that router is asking the server to send it this file over TFTP if it has it.
   Bingo, we got our filename too.

Flashing the firmware

1. Download and run TFTP server software. I personally use the portable edition of this TFTP server.
2. Download your router firmware file from the official website and rename it to the filename identified above and put it in a folder.
3. In the TFTP server software, point the directory to the directory where your renamed file is present, and for the server interface, select your LAN adapter with and make sure the IP is the one which you identified above.
4. Restart the router again in TFTP mode, and if everything goes right, the file transfer should start and once that is complete, give the router some time to flash and restore itself.

Voila, you have restored your router via TFTP and unbricked itself.

Thanks for reading. I hope this guide was helpful

WSL was good but it was not perfect for a linux dev running Windows 10. One reason was it was not fully compatible with linux. In the upcoming Windows 10 feature update, a new version of WSL is coming and it is called WSL2. It is optional and users can upgrade their exising WSL installations to WSL2.

Behind the scenes, WSL2 actually runs a lightweight VM. It boots up in seconds compared to what it would have taken for a normal VM to start.
Running graphical applications is also simple by using an XServer like MobaXTerm.

Due to it running inside a VM, the file access methods have changed, and the Distro installation/files is actually inside of a virtual hard drive and to access the full benefits and speeds of linux file systems, we need to copy our files inside the virtual hard drive.

Now, this might seem okay, but the problem is that the size of the virtual hard drive keeps on increasing automatically and if you delete a file from the linux file system, the space occupied by the VHD is still the same.

We need a way to manually free up this space from Windows 10.

Windows 10 Pro/Enterprise Method

if you have Pro or enterprise, you can do the following

• Find the path of the VHD: To do this, enter the following in powershell

wsl --shutdown
Get-AppxPackage -Name "*Distro Name*" | Select PackageFamilyName

---
PackageFamilyName
-----------------
CanonicalGroupLimited.Ubuntu18.04onWindows_79rhkp1fndgsc

• The VHD path would now be

%USERPROFILE%\AppData\Local\Packages\CanonicalGroupLimited.Ubuntu18.04onWindows_79rhkp1fndgsc\LocalState\ext4.vhdx

• Now run the powershell command

optimize-vhd -Path <PATH-TO-VHDX> -Mode full

Windows 10 Home

Home version of windows 10 doesn't expose Hyper-V tooling kit. Therefore we need to use the old method of diskpart to reudce the size.

• Find the .vhdx file path
• Run diskpart from command line
• Run the following commands

select vdisk file="<PATH-TO-VHDX>"
attach vdisk readonly
compact vdisk
detach vdisk

NOTE

• You can run the command sudo fstrim / inside the linux distro before shutting it down to improve the saving
• If reducing did not help you too much and you would like to move the data to a different drive, please refer the guide here

Thanks for reading

A lot of times we need to run some kind of automatic update scripts in our python api webserver.

One example of it might be automatically fetching and updating your machine learning models if they have been updated in a remote location with least disruption.

Also the automatic update task needs to be non blocking in nature.

Based on the above requirements, we can either use a threading or multiprocessing module to make it non blocking in nature

We use the threading module, because we require a access to shared variables of other object. Machine Learning model in this case.

Below is a scheduled task class which uses the power of Threading and Event api's to perform it.

import threading
import typing
from datetime import timedelta


class ScheduledTask(threading.Thread):
    def __init__(self, time_interval_seconds: int, scheduled_fn: typing.Callable, *args, **kwargs):
        threading.Thread.__init__(self)
        self.daemon = True
        self.stopped = threading.Event()
        self.time_interval_seconds = timedelta(seconds=time_interval_seconds)
        self.scheduled_fn = scheduled_fn
        self.args = args
        self.kwargs = kwargs

    def run(self):
        while not self.stopped.wait(self.time_interval_seconds.total_seconds()):
            self.scheduled_fn(*self.args, **self.kwargs)


class MachineLearningModel:
    def __init__(self):
        self.model = self._fetch_model()
        self.automatic_update_task = ScheduledTask(100, self._automatic_update_model)
        self.automatic_update_task.start()
    def _fetch_model(self):
        """Code to fetch model from remote repo"""
        pass
    def _automatic_update_model(self):
        self.model = self._fetch_model()
    def predict(self,input):
        """code to perform prediction"""
        pass


model_object = MachineLearningModel()
# model.predict()

Thats it.

Thanks for reading.

In this post, i try to show one of the ways of using authentication on standalone versions and replica set of MongoDb

STANDALONE INSTANCE

Connect to mongo shell on the mongodb instance and run.

use admin;
db.createUser(
    { 
        user: "adminUser", 
        pwd: "adminPassword", 
        roles: [
            { role: "userAdminAnyDatabase", db: "admin" },
            "readWriteAnyDatabase"
        ] 
    }
);

REPLICA SET

On primary node, connect to mongo shell and run the command.

use admin;
db.createUser(
    { 
        user: "adminUser", 
        pwd: "adminPassword", 
        roles: [{ 
            role: "userAdminAnyDatabase", db: "admin" }, 
            { role: "dbAdminAnyDatabase", db: "admin" }, 
            { role: "readWriteAnyDatabase", db: "admin" }, 
            { role: "clusterAdmin", db: "admin" }
        ] 
    }
);

Generate a keyfile for replicaset communication. and copy this to all instances of mongodb

openssl rand -base64 756 > mongodb.key
chmod 400 mongodb.key

NOTE:

• More roles are required for replica set compared to standalone deployments.

Restart the mongod instance

1. If using docker container, then run docker run -v source:/data/db -p 27017:27017 -d mongo:latest --auth after stopping the container.
2. If running standalone mongod instances, modify the config file and add these lines to it.

security:
    authorization: "enabled"
    keyFile: <path_to>/mongodb.key 

NOTE:

• If running on docker, exec into the container and run the mongo shell.
• For production, you should secure the deployments. You can refer for that here.

In this blog post i try to get around pricey User IP location services using a cloud function and with the help of tracked data by our friends at Google.

Below is a nodejs script to deploy on google cloud functions.

const cors = require('cors')
const cityTimezones = require('city-timezones');

// Set `useWhitelist` to `false` if you want to accept all requests.
const config = {
  useWhitelist: false
}

// Define from which origins requests are allowed.
const whitelist = [
  'https://YOURWEBSITE.com'
];

// Parse the whitelist and decide if the request is allowed.
const corsOptionsWhitelist = function (req, callback) {
  var corsOptions;

  if (whitelist.indexOf(req.header('Origin')) !== -1) {
    corsOptions = { origin: true }
  } else {
    corsOptions = { origin: false }
  }

  callback(null, corsOptions);
}

// Options when not using the whitelist.
const corsOptions = {
  origin: true
}

// Handle the response within this function. It can be extended to include more data.
function _geolocation(req, res) {
  // res.header('Cache-Control','no-cache');

  const data = {
    country: req.headers["x-appengine-country"],
    region: req.headers["x-appengine-region"],
    city: req.headers["x-appengine-city"],
    cityLatLong: req.headers["x-appengine-citylatlong"],
    userIP: req.headers["x-appengine-user-ip"],
    cityData: cityTimezones.lookupViaCity(req.headers["x-appengine-city"])
  }

  res.json(data)
};

// Export the cloud function.
exports.geolocation = (req, res) => {
  const corsHandler = config.useWhitelist ? cors(corsOptionsWhitelist) : cors(corsOptions);

  return corsHandler(req, res, function() {
    return _geolocation(req, res);
  });
};

This is the package.json contents

{
  "name": "gfc-geolocation",
  "version": "0.0.1",
  "dependencies": {
    "cors": "^2.8.4",
    "city-timezones": "^1.0.5"
    }
}

Google provides 2 million cloud functions invocation free per month. So effectively you are getting user location tracking at fraction of a price.

Disclaimer: This wonderful hack was contributed by Amit Kumar

CI/CD pipelines is a must for rapid prototyping and development of applications. Be it frontend or backend.

One of the open source tool for CI/CD is the one called Drone

Setup of this tool is extremely simple.

Requirements

• docker installation on your system

Just ssh into your server and run the following command

docker run \
  --volume=/var/run/docker.sock:/var/run/docker.sock \
  --volume=/var/lib/drone:/data \
  --env=DRONE_RUNNER_CAPACITY=10 \
  --env=DRONE_SERVER_HOST=drone.example.com \
  --env=DRONE_SERVER_PROTO=https \
  --env=DRONE_TLS_AUTOCERT=true \
  --publish=80:80 \
  --publish=443:443 \
  --restart=always \
  --detach=true \
  --name=drone \
  --env=DRONE_BITBUCKET_CLIENT_ID=${BITBUCKET_KEY} \
  --env=DRONE_BITBUCKET_CLIENT_SECRET=${DRONE_BITBUCKET_CLIENT_SECRET} \
  --env=DRONE_USER_FILTER=user1,user2,user3 \
  --env=DRONE_USER_CREATE=username:user1,admin:true \
  drone/drone:1

NOTES

• --volume=/var/lib/drone:/data : to persist to data to our local filesystem
• --env=DRONESERVERHOST=drone.example.com --env=DRONESERVERPROTO=https env=DRONETLSAUTOCERT=true --publish=80:80 --publish=443:443: required for exposing it to the internet and issuing https certificates. You can ignore all these options if you don't need the functionality.
• --env=DRONEBITBUCKETCLIENTID=${BITBUCKETKEY} --env=DRONEBITBUCKETCLIENTSECRET=${DRONEBITBUCKETCLIENTSECRET}: To link with bitbucket installation
• --env=DRONEUSERFILTER=user1,user2,user3 --env=DRONEUSERCREATE=username:user1,admin:true: Default user creation and filtering since drone installation is open to everyone by default

A lot of time, we need to run periodic jobs using Cron. Kubernetes is an amazing tool for it since it simplifies resource management by spinning up nodes in case of non availability of resources and releasing them in case of excess capacity.

Cron jobs can be deployed by referring to the below sample yaml and running kubectl apply -f

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: cron-job
  namespace: "${namespace}"
spec:
  schedule: "0/30 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: cron-jobs
            image: "${repo}:${DRONE_BUILD_NUMBER}"
            env:
              - name: run_environment
                value: ${run_environment}
          restartPolicy: OnFailure

This runs the cron job every 30 minutes

Notes

• Use envsubst to replace env variables in .yml file
• Install it in debian using command apt-get install gettext -y
• Run the deploy in CI/CD using the command cat deploy.yml|envsubst|kubectl apply -f -
• We set restartPolicy: OnFailure so that, it doesn't keep on restarting the job on completion of it.

Need to find the means of the multi-modal normal distribution

• In our day to day lives, we encounter many situations where data is generated with multiple peaks(modes).
• One such problem would be the identification of peak hour times in a public transport systems like metro or buses.
• We need to identify these peaks so that we can target increasing the frequencies of the buses/trains during the peak hours.

Generating random data with multiple peaks

Use the below code to generate multi-modal gaussian distributions

%matplotlib inline

import numpy as np
import pandas as pd

# Generating multiple gaussians 
distribution1 = np.random.normal(loc=0,scale=1.0,size=(300))
distribution2 = np.random.normal(loc=5,scale=1.0,size=(300))
distribution3 = np.random.normal(loc=10,scale=1.0,size=(300))
distribution4 = np.random.normal(loc=15,scale=1.0,size=(150))
distribution5 = np.random.normal(loc=-10,scale=1.0,size=(10))

combined_distribution = np.concatenate([distribution1,distribution2,distribution3,distribution4,distribution5])

combined_data_dataframe = pd.DataFrame(combined_distribution)
combined_data_dataframe.plot(kind='kde')

As you can see, we have created a random gaussian data with multiple means of 0,5,10,15 and -10

What happens if we,just calculate the mean?

print(combined_distribution.mean())
>>> 6.314271309260518

As you can see, the mean does not represent the peak due to multi-modality of the data.

Hence, to find multiple peaks programatically, we use one of the mixture models available in the scikit-learn package called GaussianMixtureModel

Why a mixture model?

• These models are based on the assumption that, there is a presence of another subpopulation within the main population.
• Can approximate the subpopulations while being not computationally heavy for mid sized data (100's of thousands).
• In case of a huge data, approximations about data can be made by sampling a small sample of data randomly and fitting the model on these samples. [See Central Limit Theorem]

Now, the code for it

from sklearn.mixture import GaussianMixture
mixture_model = GaussianMixture(n_components=5)
mixture_model.fit(combined_distribution.reshape(-1,1))
print(mixture_model.means_.astype(np.int32).reshape(-1))
print(mixture_model.weights_.reshape(-1))
>>> [14  5 10 -9  0]
>>> [0.1417197  0.28217055 0.2822368  0.00943396 0.28443898]

• We can combine use these weights as a representation of the density of the data at the peak.
• In order to automatically determine the peaks, we can use a variation of gaussian mixture called BayesianGaussianMixture along with the means_ and degrees_of_freedom_ attribute to select the proper peaks
• Alternative we can use the scipy.stats.gaussian_kde to find the density, but it smooths out the lower density values more than necessary.

Thank you for reading

Deploying services using Docker containers are all in the rage nowadays and Kubernetes provides a good way to manage them. Also, majority of the cloud providers have kubernetes as a service. So there's that too.

Before we begin

I assume that you have setup Kubernetes and have Kubectl configured on your local machine.
If not, i suggest you yo go and set those up before starting.
Now lets get started :D

via GIPHY

Installing Helm

What is helm and why do i need it?

• Helm is a package manager for kubernetes.
• Since we are going to install some services and their dependencies into the cluster, we need helm to make sure we can install them with simple set of commands.
• Alternatively, you can install these packages without helm. Just go their respective github repos and find the yaml files and install them using kubectl apply

Spin up your terminal and fire up the following commands to install helm.

kubectl --namespace kube-system create sa tiller
# create a cluster role binding for tiller
kubectl create clusterrolebinding tiller \
   --clusterrole cluster-admin \
   --serviceaccount=kube-system:tiller
echo "initialize helm"
# initialized helm within the tiller service account
helm init --service-account tiller
# updates the repos for Helm repo integration
helm repo update
echo "verify helm"
# verify that helm is installed in the cluster
kubectl get deploy,svc tiller-deploy -n kube-system

Basically what the above commands does is as follows

• Creates a service account called tiller in the namespace kube-system which is the system namespace that kubernetes creates after the cluster creation
• Assigns the cluster-admin role to the namespace so that we can install other charts via helm without any problems
• Install helm to the cluster.

Installing nginx ingress controller

Now that we've got helm setup and going, we need to install nginx ingress controller in the cluster using helm
To do this, we run

helm install stable/nginx-ingress \
    --namespace=kube-system \
    --name nginx-ingress \
    --set controller.kind=DaemonSet \
    --set controller.daemonset.useHostPort=true \
    --set controller.service.enabled=false

Basically what we are doing here is installing nginx with some specific settings so that it doesn't spin up loadbalancers during the actual creation of ingress.

• You can read more about DaemonSets here
• useHostPort basically tells the controller to use host network of the cluster nodes and allows us to access 80 and 443 ports from outside the cluster.

Installing certmanager

We need a way to issue certificates to the cluster and this is done by using certmanager.
Install it using helm by

helm install --name cert-manager --version v0.5.2 \
    --namespace kube-system stable/cert-manager

Now that we have installed certmanger, we need someone to issue the certificate for us, lets use LetsEncrypt for it.

Installing Letencrypt as certificate issuer

Open a text editor and enter the below

apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: "<YOUR_EMAIL>"
    privateKeySecretRef:
      name: letsencrypt-staging
    http01: {}
---
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: "<YOUR_EMAIL>"
    privateKeySecretRef:
      name: letsencrypt-prod
    http01: {}

save the above file as lets_encrpyt.yml and then run kubectl apply -f lets_encrypt.yml

Run an ingress

Now that we have setup letsencrypt as cluster issuer, we need to setup an ingress to handle the incoming connections.
Open another editor and enter the following

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/ssl-redirect: "true"
    certmanager.k8s.io/cluster-issuer: letsencrypt-prod
    certmanager.k8s.io/acme-http01-edit-in-place: "true"
    kubernetes.io/ingress.class: "nginx"
  name: custom-ingress-nginx
spec:
  rules:
    - host: <YOUR-HOSTNAME>
      http:
        paths:
          - path: /
            backend:
              serviceName: "<SERVICE-NAME>"
              servicePort: <SERVICE-PORT>
  tls:
    - secretName: test-eng-secret
      hosts:
        - <YOUR-HOSTNAME>

• we specify the cluster issuer name in the annotation to issue us certificates. Use letsencrypt-staging to see if your setup is working and then use the production one.
• And we specify the ingress class as nginx to use the nginx ingress controller.
• the <SERVICE-NAME> and <SERVICE-PORT> are the names and ports in the NodePort services exposed above.

Thanks for reading and look forward to my future struggles.