Build RESTful Microservices with AWS Lambda and API Gateway
Learn how to design, configure, secure and test HTTP endpoints, using AWS Lambda as backend.
Posts by Tag
- AWS 20
- Data Engineering 17
- Kubernetes 15
- NLP 14
- SQL 12
- postgres 10
- Kafka 10
- Time Series 9
- Docker 8
- AWS Compute 7
- Serverless 7
- DevOps 6
- Machine Learning 6
- AWS Networking 5
- Deep Learning 5
- SQLAlchemy 4
- Data Visualization 4
- Neo4j 3
- PCA 3
- R 3
- AWS Databases 3
- Unsupervised Learning 3
- Clustering 3
- Recommendation Systems 3
- Jenkins 2
- Classification 2
- Cloud 2
- IaC 2
- Pandas 2
- Neural Networks 2
- Pytorch 2
- CNN 2
- Airflow 2
- Data Cleaning 2
- Data Preprocessing 2
- Feature Engineering 2
- Faust 2
- AWS Sagemaker 2
- Spark 2
- git 1
- conda 1
- Linear Algebra 1
- LDA 1
- jupyter 1
- AWS BigData 1
- Logistic Regression 1
- Cross Validation 1
- regex 1
- AWS RedShift 1
- Supervised Learning 1
- pipelines 1
- EDA 1
- tsne 1
- Data Pipeline 1
- Dimensionality Reduction 1
- Avro 1
- KSQL 1
AWS
CICD on AWS using CodeBuild, CodeDeploy and CodePipeline
AWS Developer tools: CodeBuild, CodeDeploy and CodePipeline are discussed in the context of a CICD pipeline.
Develop, Test and Deploy a Serverless App using Cloud9
Cloud9 is a cloud-based IDE to build Cloud-Native applications. In this post, I will show how to develop, test and deploy a Serverless app using Cloud9.
AWS Step Function to create a state machine
In this post, we will see what AWS Step Functions are and what types of problems this service is the best suited to solve with an example.
Infrastructure as Code using AWS CloudFormation
IaaC is one of the pillars of Cloud DevOps and CloudFormation is the tool that lets us provision resources using declarative programming.
Create AWS Redshift cluster using AWS python SDK
The power of infrastructure-as-code is illustrated by launching a 4-node AWS Redshift cluster, performing some analysis, and destroying the resources, all us...
Introduction to AWS X-Ray Service
Why do we need AWS X-Ray service and what are the key components of X-ray service are covered in this post.
How to configure a static website on S3?
Configure a static website with S3 and distribute it using CloudFront.
AWS S3 Encryption Mechanisms
In this post, we will look at 5 ways to encrypt data in S3: SSE-S3, SSE-KMS, SSE-C, CSE-KMS and CSE-C.
AWS KMS Key Management
Rotation of CMKs, importing key material for CMKs and Deletion of CMKs are covered in this post.
AWS KMS Key Policies
Key policies are resource based policies which are tied to your CMK. And if you want a principal to be able to access your CMK, then a key policy must be in ...
AWS Key Management Service (KMS)
Many of the AWS services rely on KMS for their encryption needs. KMS allows encryption of data at rest. In this post, we will look at the components that mak...
Docker on AWS EC2
Install Docker on AWS, find and use images from public docker registry and finally build your docker images using Dockerfiles.
AWS SQS, SNS, SES overview
A high level overview of AWS SQS, SNS, SES
Storage fundamentals of AWS part 2: Glacier, EBS, EFS, EC2 Instance storage
Storage fundamentals of AWS are discussed here. Non-S3 is covered in-depth.
Storage fundamentals of AWS part 1
Storage fundamentals of AWS are discussed here. S3 is covered in-depth.
AWS Identity and Access Management (IAM)
The Identity and Access Management service, commonly referred to as IAM is a key security service within AWS and is likely the first service you will encount...
Build a Jenkins pipeline to upload a file to S3
Configure Jenkins plugins to talk to S3 and Github and build a simple pipeline which will upload a file checked into Github to S3. Each time you make a chang...
How to install Jenkins on AWS EC2 instance
This post covers the installation of Jenkins on EC2 and configures Blue Ocean plugin for building pipelines.
AWS Glossary
A glossary of terms pertaining to AWS
Data Engineering
Spark functional programming and pure functions
Spark’s use of functional programming is illustrated with an example. The intuition for using pure functions and DAGs is explained.
Spark modes, use cases, limitations and alternatives
When to use Spark? What are the modes in which Spark can run? When not to use Spark? and What are some alternatives to Spark? - these questions are answered ...
Build Stream Processing Applications using KSQL
KSQL is a SQL-like interface for building stream processing applications. In this post, I will show how to convert Kafka topics in Streams and Tables and the...
Build a production-grade data pipeline using Airflow
A data pipeline captures the movement and transformation of data from one place/format to another. It often runs on schedule and feeds data into multiple das...
Filter and enrich a stream of data using Faust
A common use-case in building stream processing applications is to filter and enrich an incoming stream of data and send it to a new topic or create an inter...
Process a stream of data using Faust
Faust is a stream processing python library which allows us to read a stream of data from a Kafka Topic, process it and store the processed data into another...
Storage options for stream processing apps
Faust provides a few options for state storage that we need to understand before we start building streaming applications in production. Here, we are going t...
Kafka: How Kafka stores data?
It is worth understanding how kafka stores data to better appreciate how the brokers achieve such high throughput. Kafka simply has a data directory on disk:...
Kafka: Data Schemas, Apache Avro and Schema Registry
The role of data schemas, Apache Avro and Schema Registry in Kafka is explained using an example.
Managing Kafka Topics
How to create and administer Kafka topics? In this post, I will show 3 methods to manage Kafka topics. This will be useful when troubleshooting arcane issues
A practical guide to using Kafka REST Proxy
The Confluent REST Proxy provides a RESTful interface to a Kafka cluster. Learn how to produce, consume, view and administer Kafka cluster using simple pytho...
Kafka Connect to stream data into Kafka
Use Kafka Connect to stream data from a log file and SQL table into Kafka using python and Kafka Connect HTTP REST API
Kafka commonly used commands
Covers commonly used commands when working in Kafka ecosystem. Namely, Kafka CLI; Kafka Connect; Kafka REST Proxy; KSQL; Faust
Build a data pipeline using Apache Airflow
I will show how to build a simple data pipeline using Apache Airflow to retrieve data from S3 and load it into Redshift cluster
The Rise of Cloud Data Lakes
Lets understand the rise and popularity of Data Lakes and the need for modern Cloud-based Data Lakes.
Create AWS Redshift cluster using AWS python SDK
The power of infrastructure-as-code is illustrated by launching a 4-node AWS Redshift cluster, performing some analysis, and destroying the resources, all us...
MapReduce Programming Example
The MapReduce programming technique was designed to analyze massive data sets across a cluster. In this post, we will get a sense of how Hadoop MapReduce works
Kubernetes
How to create an EKS cluster using eksctl?
Understand what EKS service provides, then create an eks cluster using eksctl and connect to the cluster using kubectl. Finally destroy the resources.
Kubernetes ConfigMaps and Secrets
Kubernetes provides ConfigMaps and Secrets resource kinds to allow you to separate configuration from pod’s specs. This separation makes it easier to manage ...
Kubernetes Volumes
Volumes, PersistentVolumes, PersistentVolumeClaims are explored in depth with the use of an example to demonstrate the need for these and how to use them.
Kubernetes Init Containers
Init Containers let you perform some tasks or check some preconditions before the main application container starts. In this post, I will use an example to s...
Kubernetes Liveness and Readiness Probes
Is your pod ready as soon as the container starts? This is the key question we will explore in this post through the use of Kubernetes liveness and readiness...
Kubernetes Rolling Updates and Rollbacks
Kubernetes by default uses RollingUpdates strategy. In this post, we will learn how to trigger, pause, resume and view a rollout and demonstrate a rollback.
Kubernetes Deployments in Depth
Instead of creating Pods, we will create deployments and use an example 3-tier application to illustrate scaling and load balancing.
Kubernetes Autoscaling Demonstration
Autoscaling uses metrics server to collect metrics about the cluster and uses this info to make scaling decisions. In this demo, I will show how to create me...
Kubernetes Service Discovery for a multi-pod design
In this post, I will show how to use Kubernetes service discovery mechanisms: env variables and DNS, to design a multi-pod n-tier application.
Kubernetes Multi-Container Pods
In this post, I will show how to run a multi-container pod that implements a three tier application in a Kubernetes namespace.
Demystifying Kubernetes Services
Learn why you can’t live without these Services in Kubernetes and what exactly they solve.
A practical guide to understanding Kubernetes pods
In this post, I will first cover what pods are, how to create, destroy and configure them. I will then run an Nginx web server on Kubernetes cluster.
Kubernetes Architecture
A basic level architecture overview of Kubernetes is the focus of this post. Clusters, Nodes, Control Plane, Pods, Services and Deployments are touched upon.
Deploying Kubernetes options
Once you’ve decided on using Kubernetes, you have a variety of methods for deploying Kubernetes. Single-node, Multi-node, Vendor managed, on-prem etc.
Kubernetes Overview
Overview of Kubernetes, what it does, how it does it and alternatives to Kubernetes.
NLP
Create features out of text
Before we can leverage text data in a machine learning model you must first transform it into a series of columns of numbers or vectors. There are many diffe...
Topic Modeling using NMF and LDA using sklearn
Are James Bond movies the best in Thriller movies category? - I will answer this using 1500 user movie reviews of 500 top thriller movies using NLP and Topic...
Group Cosmetics based on chemicals.
Using NLP and Dimensionality Reduction (t-SNE), group cosemetics based on their chemical ingredients.
Document Clustering
1000 user reviews are clustered using K-means and Agglomerative Hierarchical clustering algorithms. Silhouette analysis is conducted to evaluate the effectiv...
Clustering movies based on their plots
In this post, I will show how we can cluster movies based on IMDB and Wiki plot summaries. We will quantify the similarity of movies based on their plot summ...
Document Similarity
Two documents are similar if their vectors are similar. In this post, we will explore this idea through an example. A heatmap of Amazon books similarity is d...
Scrape IMDB movie reviews
Scrape IMDB movie reviews and construct a dataset. Perform shallow parsing on user reviews using spaCy and pattern.
Scrape, clean and normalize Gutenberg texts
Scrape, clean and normalize 100+ Gutenberg texts and apply basic text analysis
Lexical Diversity
Lexical diversity is a measure of how many different words that are used in a text. The goal of this notebook is to use NLTK to explore the lexical diversity...
NLP Glossary of terms
A glossary of NLP terms and notes
Understanding Vector Space Models
We will delve into the concept of Vector Spaces by developing an intuition behind this idea through some simple examples. There are many applications and alg...
Build a Machine Learning Pipeline to predict Multi-class-multi-label target variable
Predict School budgets using a machine learning pipeline. The target variable is multi-class-multi-label and we have a mix of numeric and text features. We w...
Text as numeric data
Text Analysis is a major application field for machine learning algorithms. However the raw data, a sequence of symbols cannot be fed directly to the algorit...
The World of Regex
When working with text data, you are often required to write regular expressions to pre-process the data, extract useful information from the data, create ne...
SQL
Data driven decision making part 1
Using simple SQL filtering, aggregations and joins answering business questions.
Exploratory Data Analysis of JSON data using PostgreSQL
Use PostgreSQL JSON operators and functions to explore and understand couple of datasets stored inside a database
Day 1: Leetcode SQL problems
A variety of SQL leetcode questions that I solved. This post captures Day 1 problems and their solutions.
How to work with ARRAYs in Postgres
When data is stored in arrays we can make use of some special Postgres operators like ANY and CONTAINS to query, filter and aggregate records
Retrieve data from API and store it in Postgres
Use requests library to send GET requests to an API endpoint, parse the response, handle pagination, and store the transformed data into a postgres table.
Setup sample database in Postgres
Setup dvdrental database locally for demonstrating features of PostgresSQL
Exploratory Data Analysis of JSON data using PostgreSQL
Use PostgreSQL JSON operators and functions to explore and understand couple of datasets stored inside a database
Convert SQLite database to Postgres database
Convert a SQLite database to a Postgres Database. Example when given database.sqlite and you want to do your analysis in Postgres, then read on
Working with Dates and Times in Postgresql
Work with dates and times in SQL
Query information_schema in Postgres
Since we don’t have DESCRIBE table in Postgres, we need to write this query again and again. So I am documenting it.
How to work with JSON data using PostgreSQL
We will look at a real-world scenarios of a health-care company data and learn how to query JSON data using normal PostgreSQL.
Postgres and Jupyter notebooks
Install postgres, administer it using cli, connect to it using jupyter notebook.
postgres
Data driven decision making part 1
Using simple SQL filtering, aggregations and joins answering business questions.
Exploratory Data Analysis of JSON data using PostgreSQL
Use PostgreSQL JSON operators and functions to explore and understand couple of datasets stored inside a database
How to work with ARRAYs in Postgres
When data is stored in arrays we can make use of some special Postgres operators like ANY and CONTAINS to query, filter and aggregate records
Retrieve data from API and store it in Postgres
Use requests library to send GET requests to an API endpoint, parse the response, handle pagination, and store the transformed data into a postgres table.
Setup sample database in Postgres
Setup dvdrental database locally for demonstrating features of PostgresSQL
Exploratory Data Analysis of JSON data using PostgreSQL
Use PostgreSQL JSON operators and functions to explore and understand couple of datasets stored inside a database
Convert SQLite database to Postgres database
Convert a SQLite database to a Postgres Database. Example when given database.sqlite and you want to do your analysis in Postgres, then read on
Working with Dates and Times in Postgresql
Work with dates and times in SQL
Query information_schema in Postgres
Since we don’t have DESCRIBE table in Postgres, we need to write this query again and again. So I am documenting it.
Postgres and Jupyter notebooks
Install postgres, administer it using cli, connect to it using jupyter notebook.
Kafka
Build Stream Processing Applications using KSQL
KSQL is a SQL-like interface for building stream processing applications. In this post, I will show how to convert Kafka topics in Streams and Tables and the...
Filter and enrich a stream of data using Faust
A common use-case in building stream processing applications is to filter and enrich an incoming stream of data and send it to a new topic or create an inter...
Process a stream of data using Faust
Faust is a stream processing python library which allows us to read a stream of data from a Kafka Topic, process it and store the processed data into another...
Storage options for stream processing apps
Faust provides a few options for state storage that we need to understand before we start building streaming applications in production. Here, we are going t...
Kafka: How Kafka stores data?
It is worth understanding how kafka stores data to better appreciate how the brokers achieve such high throughput. Kafka simply has a data directory on disk:...
Kafka: Data Schemas, Apache Avro and Schema Registry
The role of data schemas, Apache Avro and Schema Registry in Kafka is explained using an example.
Managing Kafka Topics
How to create and administer Kafka topics? In this post, I will show 3 methods to manage Kafka topics. This will be useful when troubleshooting arcane issues
A practical guide to using Kafka REST Proxy
The Confluent REST Proxy provides a RESTful interface to a Kafka cluster. Learn how to produce, consume, view and administer Kafka cluster using simple pytho...
Kafka Connect to stream data into Kafka
Use Kafka Connect to stream data from a log file and SQL table into Kafka using python and Kafka Connect HTTP REST API
Kafka commonly used commands
Covers commonly used commands when working in Kafka ecosystem. Namely, Kafka CLI; Kafka Connect; Kafka REST Proxy; KSQL; Faust
Time Series
Techniques for detecting stationarity in Time Series.
Visually and Statistically detect Stationarity in a time series. Methods to convert a non-stationary time series to stationary are discussed with examples.
Autoregressive Models: PACF and Information Criteria
Understand the basics of Autoregressive models and how we can use PACF and AIC/BIC to identify the order of AR model.
White Noise, Random Walk and Stationarity
How to know if your time series is White Noise (or) a Random Walk? How to detect a stationary process? What is the relation b/w these concepts?
Autocorrelation of a time series
Autocorrelation is the correlation of a single time series with a lagged copy of itself. In this post, we will explore this concept with couple of examples.
Correlation and Regression of two time series
How to calculate correlation and regression coefficients for two time series?
Time Series Window functions: Rolling and Expanding metrics
Using pandas we will explore two types of window functions: rolling and expanding metrics for time series data.
Resampling a time series: Downsampling
Downsampling and aggregation of time series to observe trends and comparing time series that have different frequencies.
Resampling a time series: Upsampling
Changing the time series frequency is a very common operation as there are many cases when you want to compare time series with different freq.
Basics of manipulating time series
Basics of manipulating time series like slicing, changing frequency, shifting, lags, diffs, percent_changes are explored in this post.
Docker
Build a data pipeline using Apache Airflow
I will show how to build a simple data pipeline using Apache Airflow to retrieve data from S3 and load it into Redshift cluster
Containerize a Flask application
The following topics are covered: Benefits of containerizing, why choose containers over VM, Docker basics, install postgres from Dockerhub and finally build...
Docker on AWS EC2
Install Docker on AWS, find and use images from public docker registry and finally build your docker images using Dockerfiles.
Docker Networking: bridge, host and none networks
Learn about networking fundamentals in Docker. I explore the three pre-configured networks with examples.
Docker Port Mapping by example
What is the difference between Expose, PublishAll and Publish when talking about port mapping in docker? Also learn how to run docker container in detached m...
Build a python image from a Container
Build a docker image from a Container using the docker commit command and modify the default command using the change flag
Build your first image using Dockerfile
Build your first docker image from a Dockerfile and run a simple go binary file inside the container.
How to explain Docker to your Grandma?
Suppose your gradma caught you twiddling with a Blue Whale Icon on your laptop and you have to explain what that is, how would you explain it?
AWS Compute
Introduction to AWS ECS
To understand EC2 Container Service AWS ECS, I will approach this from an Architectural Context, then I’ll cover the Computational Context, and then finally,...
Host a static website on AWS with CloudFront Distribution Enabled
Host a static gallery website on S3 and distribute the content the Edge locations using CloudFront Web Distribution.
AWS Elastic Beanstalk explained
We will explore how AWS Beanstalk can be used to help deploy and scale applications without having to worry about provisioning resources manually.
AWS Elastic Beanstalk, Lambda and Batch
Amazon Elastic Beanstalk, AWS Lambda and AWS Batch are covered in this post.
AWS ECS and ECR explained
Amazon EC2 Container Service and Elastic Container Registry are explained.
AWS EC2 Instance metadata
Create an Amazon EC2 Linux instance, connect to it and extract instance metadata.
AWS EC2 service overview
AMI, instance types, instance purchase options, tenancy, user data, storage options and security of EC2 instances are covered in this post.
Serverless
Build RESTful Microservices with AWS Lambda and API Gateway
Learn how to design, configure, secure and test HTTP endpoints, using AWS Lambda as backend.
React to AWS CodeCommit events with AWS Lambda
Process CodeCommit events with a Lambda Function to create custom SNS notifications, containing useful information about branch, author and message for each ...
Process S3 Events using AWS Lambda
Process AWS S3 events using AWS Lambda. A simple example of uploading an image to a bucket will create a zipped version of it in the same bucket with zip pre...
Overview of AWS Lambda
An overview of AWS Lambda service and its key features.
Develop, Test and Deploy a Serverless App using Cloud9
Cloud9 is a cloud-based IDE to build Cloud-Native applications. In this post, I will show how to develop, test and deploy a Serverless app using Cloud9.
AWS Step Function to create a state machine
In this post, we will see what AWS Step Functions are and what types of problems this service is the best suited to solve with an example.
Process SNS notifications using AWS Lambda
Learn to process SNS notifications with AWS Lambda function which will upload a file to S3 upon receipt of a message.
DevOps
Develop, Test and Deploy a Serverless App using Cloud9
Cloud9 is a cloud-based IDE to build Cloud-Native applications. In this post, I will show how to develop, test and deploy a Serverless app using Cloud9.
Infrastructure as Code using AWS CloudFormation
IaaC is one of the pillars of Cloud DevOps and CloudFormation is the tool that lets us provision resources using declarative programming.
Continuous Delivery requirements and tools
Continuous Delivery requirements and tools are discussed extensively in this post.
How to explain Docker to your Grandma?
Suppose your gradma caught you twiddling with a Blue Whale Icon on your laptop and you have to explain what that is, how would you explain it?
Build a Jenkins pipeline to upload a file to S3
Configure Jenkins plugins to talk to S3 and Github and build a simple pipeline which will upload a file checked into Github to S3. Each time you make a chang...
How to install Jenkins on AWS EC2 instance
This post covers the installation of Jenkins on EC2 and configures Blue Ocean plugin for building pipelines.
Machine Learning
Build a Machine Learning Pipeline to predict Multi-class-multi-label target variable
Predict School budgets using a machine learning pipeline. The target variable is multi-class-multi-label and we have a mix of numeric and text features. We w...
Binary Classification Problem
Approach to solving a binary classification problem
Clustering of NCSchools
Unsupervised Learning: Clustering using R and Python
Seismic Bump prediction using Logistic Regression
Predict Seismic bumps using Logistic Regression in R
PCA in R
PCA in R
Breast Cancer detection using PCA + LDA in R
Principal Components Analysis and Linear Discriminant Analysis applied to BreastCancer Wisconsin Diagnostic dataset in R
AWS Networking
Fundamentals of Networking in AWS using LucidCharts
Basic Networking resources within AWS is covered through the use of LucidChart diagrams.
AWS Elastic Load Balancing Service
Introduction to Elastic Load Balancing and types of ELBs is covered in this post.
Network Access Control Lists and Security Groups
Filter inbound and outbound traffic using NACLs and Security Groups within your VPC.
VPC subnets
Everything about VPC subnets and how to design for high availability and resiliency.
Create a VPC and associated network components: subnet, IGW, routing tables
AWS Virtual Private Cloud (VPC) allows you to provision a logically isolated section of the AWS Cloud where you can launch AWS resources in a virtual network...
Deep Learning
Neural Network Basic Introduction
What comes to mind when you think of a neural network? In this post, we will develop the intuition for understanding how a neural network works and introduce...
Faster R-CNN to detect objects on the road
Use Faster Region Based Convolutional Neural Network to detect objects under different driving conditions
Convolutional Neural Networks (CNN)
A glossary of terms used in CNN.
Multi Layer Perceptron (MLP)
How to construct a MLP in PyTorch. A glossary of terms.
Pytorch Neural Networks
This post will introduce how to use pytorch to build and train neural networks
SQLAlchemy
SQLAlchemy Object Lifecycle
The story of how an object goes from Pending to Persisted.
Flask and SQLAlchemy
Build a simple flask application which connects to postgres using SQLAlchemy.
SQLAlchemy Layers explained
SQLAlchemy offers several layers of abstraction and convenient tools for interacting with a database. In this post, we will understand the purpose of each la...
Should I use SQLAlchemy or psycopg2?
For a long time I have been a fan of psycopg2 and resisted to use SQLAlchemy, but may be not anymore.
Data Visualization
Deploy data dashboard to Heroku
Using Flask + Pandas + Plotly + Dash deploy an interactive data dashboard to Heroku
K-means clustering of NYC Airbnb data
Group Airbnb listings based on similarity. Used folium maps to display the clustering results for Chelsea neighborhood
New York City Airbnb t-SNE visualization
An interactive 3-D scatter plot of t-SNE features showing similar airbnb listings is shown here
New York City Airbnb price distribution
Visualize Airbnb price distribution across 5 boroughs of New York City. Using GeoPandas and Folium maps, created some interesting visualizations
Neo4j
Cypher Query Language to query Neo4j
In this post we will think about the graph patterns to apply to the graph database and then we will perform the queries using Cypher
Create a Neo4j property graph using Cypher
What is a property graph? How to create one in Neo4j using Cypher? are answered in this post using an example.
Neo4j and Graph database basics
In this post I cover how to install Neo4j Desktop on mac and explore what is available in the tool. Basics of graph database is discussed.
PCA
New York City Airbnb PCA
Used Dimensionality Reduction to reduce 2100 features to 50 principal features
PCA in R
PCA in R
Breast Cancer detection using PCA + LDA in R
Principal Components Analysis and Linear Discriminant Analysis applied to BreastCancer Wisconsin Diagnostic dataset in R
R
Seismic Bump prediction using Logistic Regression
Predict Seismic bumps using Logistic Regression in R
PCA in R
PCA in R
Breast Cancer detection using PCA + LDA in R
Principal Components Analysis and Linear Discriminant Analysis applied to BreastCancer Wisconsin Diagnostic dataset in R
AWS Databases
Create your first Amazon RDS Database
Create RDS subnet groups, RDS database clusters, manage access to RDS cluster using security groups and finally connect to your RDS cluster and create tables.
Databases on AWS
AWS RDS and AWS non-relation databases are two main families of Databases offered by AWS. In this post I will introduce the various databases available on AW...
AWS ElastiCache Service features and usage scenarios
Learn when to use ElastiCache Service within your applications to improve your overall performance.
Unsupervised Learning
Deploy data dashboard to Heroku
Using Flask + Pandas + Plotly + Dash deploy an interactive data dashboard to Heroku
K-means clustering of NYC Airbnb data
Group Airbnb listings based on similarity. Used folium maps to display the clustering results for Chelsea neighborhood
Clustering of NCSchools
Unsupervised Learning: Clustering using R and Python
Clustering
Document Clustering
1000 user reviews are clustered using K-means and Agglomerative Hierarchical clustering algorithms. Silhouette analysis is conducted to evaluate the effectiv...
Clustering movies based on their plots
In this post, I will show how we can cluster movies based on IMDB and Wiki plot summaries. We will quantify the similarity of movies based on their plot summ...
Clustering of NCSchools
Unsupervised Learning: Clustering using R and Python
Recommendation Systems
Content Based recommendations
Content based recommendation using Jaccard Similarity.
Making non-personalized recommendations
Make recommendations based on the knowledge of the crowd. Example, people who watch Gladiator have also watched the Matrix. We will take a dataset and genera...
Intro to Recommendation Systems
What are recommendation engines and what type of data is well suited for these problems?
Jenkins
Build a Jenkins pipeline to upload a file to S3
Configure Jenkins plugins to talk to S3 and Github and build a simple pipeline which will upload a file checked into Github to S3. Each time you make a chang...
How to install Jenkins on AWS EC2 instance
This post covers the installation of Jenkins on EC2 and configures Blue Ocean plugin for building pipelines.
Classification
Binary Classification Problem
Approach to solving a binary classification problem
Seismic Bump prediction using Logistic Regression
Predict Seismic bumps using Logistic Regression in R
Cloud
Deploy data dashboard to Heroku
Using Flask + Pandas + Plotly + Dash deploy an interactive data dashboard to Heroku
Create AWS Redshift cluster using AWS python SDK
The power of infrastructure-as-code is illustrated by launching a 4-node AWS Redshift cluster, performing some analysis, and destroying the resources, all us...
IaC
Infrastructure as Code using AWS CloudFormation
IaaC is one of the pillars of Cloud DevOps and CloudFormation is the tool that lets us provision resources using declarative programming.
Create AWS Redshift cluster using AWS python SDK
The power of infrastructure-as-code is illustrated by launching a 4-node AWS Redshift cluster, performing some analysis, and destroying the resources, all us...
Pandas
Manipulation DataFrames using Pandas
Everything about manipulation of dataframes using Pandas
Understanding Pandas melt and when to use it
Melting turns columns into rows. Whereas, pivot will take unique values from a column and creates a new columns.
Neural Networks
Convolutional Neural Networks (CNN)
A glossary of terms used in CNN.
Pytorch Neural Networks
This post will introduce how to use pytorch to build and train neural networks
Pytorch
Multi Layer Perceptron (MLP)
How to construct a MLP in PyTorch. A glossary of terms.
Pytorch Neural Networks
This post will introduce how to use pytorch to build and train neural networks
CNN
Faster R-CNN to detect objects on the road
Use Faster Region Based Convolutional Neural Network to detect objects under different driving conditions
Convolutional Neural Networks (CNN)
A glossary of terms used in CNN.
Airflow
Build a production-grade data pipeline using Airflow
A data pipeline captures the movement and transformation of data from one place/format to another. It often runs on schedule and feeds data into multiple das...
Build a data pipeline using Apache Airflow
I will show how to build a simple data pipeline using Apache Airflow to retrieve data from S3 and load it into Redshift cluster
Data Cleaning
New York City Airbnb Data Preprocessing
Dealing with outliers and choosing the type of scaling is covered in this post
New York City Airbnb Data Cleaning
Extract data from S3, clean the dataset, deal with missing values and load the cleaned dataset to S3
Data Preprocessing
New York City Airbnb PCA
Used Dimensionality Reduction to reduce 2100 features to 50 principal features
New York City Airbnb Feature Engineering
Create a binary bag-of-words representation for amenities and host verifications and Tfidf representation for description of each airbnb listing
Feature Engineering
Create features out of text
Before we can leverage text data in a machine learning model you must first transform it into a series of columns of numbers or vectors. There are many diffe...
New York City Airbnb Feature Engineering
Create a binary bag-of-words representation for amenities and host verifications and Tfidf representation for description of each airbnb listing
Faust
Filter and enrich a stream of data using Faust
A common use-case in building stream processing applications is to filter and enrich an incoming stream of data and send it to a new topic or create an inter...
Process a stream of data using Faust
Faust is a stream processing python library which allows us to read a stream of data from a Kafka Topic, process it and store the processed data into another...
AWS Sagemaker
Building a model using Sagemaker
General steps for using Sagemaker is discussed in this post. The Training process and inference process is explained in detail.
Using XGBoost in Sagemaker (Batch Transform) - High level API
As an introduction to using SageMaker’s High Level Python API we will look at a relatively simple problem. Namely, we will use the Boston Housing Dataset to ...
Spark
Spark functional programming and pure functions
Spark’s use of functional programming is illustrated with an example. The intuition for using pure functions and DAGs is explained.
Spark modes, use cases, limitations and alternatives
When to use Spark? What are the modes in which Spark can run? When not to use Spark? and What are some alternatives to Spark? - these questions are answered ...
git
How to force git to prompt for passwd on mac?
When you have multiple github profiles, you want to be prompted for your user and passwd.
conda
Conda Channels
Everything about conda channels
Linear Algebra
Linear Algebra basics using numpy
Basics of linear algebra using numpy. Things like calculating the norm, dot product using numpy.
LDA
Breast Cancer detection using PCA + LDA in R
Principal Components Analysis and Linear Discriminant Analysis applied to BreastCancer Wisconsin Diagnostic dataset in R
jupyter
Postgres and Jupyter notebooks
Install postgres, administer it using cli, connect to it using jupyter notebook.
AWS BigData
AWS Kinesis Overview
Get an overview of Amazon Kinesis to collect, process and analyze real-time streaming data.
Logistic Regression
Seismic Bump prediction using Logistic Regression
Predict Seismic bumps using Logistic Regression in R
Cross Validation
Seismic Bump prediction using Logistic Regression
Predict Seismic bumps using Logistic Regression in R
regex
The World of Regex
When working with text data, you are often required to write regular expressions to pre-process the data, extract useful information from the data, create ne...
AWS RedShift
Create AWS Redshift cluster using AWS python SDK
The power of infrastructure-as-code is illustrated by launching a 4-node AWS Redshift cluster, performing some analysis, and destroying the resources, all us...
Supervised Learning
Binary Classification Problem
Approach to solving a binary classification problem
pipelines
Build a Machine Learning Pipeline to predict Multi-class-multi-label target variable
Predict School budgets using a machine learning pipeline. The target variable is multi-class-multi-label and we have a mix of numeric and text features. We w...
EDA
Exploratory Data Analysis of JSON data using PostgreSQL
Use PostgreSQL JSON operators and functions to explore and understand couple of datasets stored inside a database
tsne
Group Cosmetics based on chemicals.
Using NLP and Dimensionality Reduction (t-SNE), group cosemetics based on their chemical ingredients.
Data Pipeline
Build a data pipeline using Apache Airflow
I will show how to build a simple data pipeline using Apache Airflow to retrieve data from S3 and load it into Redshift cluster
Dimensionality Reduction
New York City Airbnb PCA
Used Dimensionality Reduction to reduce 2100 features to 50 principal features
Avro
Kafka: Data Schemas, Apache Avro and Schema Registry
The role of data schemas, Apache Avro and Schema Registry in Kafka is explained using an example.
KSQL
Build Stream Processing Applications using KSQL
KSQL is a SQL-like interface for building stream processing applications. In this post, I will show how to convert Kafka topics in Streams and Tables and the...