Note: Opinions are of my own and do not represent my employer in any form Snowpipe Streaming API is a new feature that has been released by Snowflake (currently in preview). The purpose of the Snowpipe Streaming API is to write rowsets of data into Snowflake instead of using the…

Overview A year ago, I wrote an article on using dbt and Apache Airflow with Snowflake that received quite a bit of traction (screenshot below). That article was mainly focused on writing data pipelines with dbt and orchestrating them with Airflow.

With the rise of Blockchain, increasingly more decentralized applications are being deployed using Ethereum, Solidity and Web3.js. However, a point to note is that Blockchains by design are not ideal for storing a large amount of data. When a transaction is logged on a Blockchain (say executed via a Smart…

Note: Views expressed here are my own and do not represent my employer in any fashion. It is without a doubt the containers have become the default standard for running cloud native applications. The same has extended to data pipeline in which they are finally reproducible and they can be…

Recently, unstructured data was launched in Snowflake. With this, it Snowflake has really become the central Data Warehouse where you can store your structured, semi-structured and unstructured data. In this article, what we are going to do here is to explore storing unstructured data in Snowflake. This is done in…

Overview Numerous business are looking at modern data strategy built on platforms that could support agility, growth and operational efficiency. Snowflake is Data Cloud, a future proof solution that can simplify data pipelines for all your businesses so you can focus on your data and analytics instead of infrastructure management…

In this post, we will be looking at how we can use Apache Airflow to allow us to perform ETL and ELT jobs into any database/data warehouse. In this case, we will be using Snowflake as our data warehouse to store our data. A general architecture is shown as below. Set up of environment …

Note: Opinions expressed are solely my own and do not express the views or opinions of Snowflake, AWS or any other technology involved in this post — Introduction Extract, Transform and load are central tools to any data warehousing project. There have been many discussions into which is better? Leveraging code or using available ETL or ELT integration tools. With AWS Glue, software engineers are able to have a combination of customisable code alongside easy-to-use data transformation

Infrastructure as Code for the Snowflake data warehouse platform PS: Anything which has been written here is entirely in my own capacity and knowledge. More often than not we have heard of running Infrastructure-As-Code (IAC) to maintain the infrastructure state for cloud native applications. However, this got me thinking. With…