async VS dplyr

Before diving into advanced machine learning algorithms or statistical models, we need to start with the basics: collecting and organizing data. Fortunately, both Python and R offer a wealth of libraries that make it easy to collect data from a variety of sources, including web scraping, APIs, and reading from files. Key libraries in Python include requests, BeautifulSoup, and pandas, while R has httr, rvest, and dplyr.

It really can't be said enough how pandas is a mess. It has way too much surface area and no common thread pulling it all together. This gets obvious when you work with better dataframe libs like dplyr [1] or DataFramesMeta [2]. I've worked on production systems with all of these libs, this is not gratuitous bashing.

[1] https://dplyr.tidyverse.org/

There are two ways you can generally see the source code for packages. The simplest is to look for the github repository for the package (assuming it exists). For dplyr, it's here. Easiest way to find these is to google search "r github" plus the name of the package. Usually it'll be one of the first results. The github repo would also usually be linked on the package's CRAN page. However, be aware that this may be a development version of the package and not the same version that is currently released on CRAN (e.g. dplyr on CRAN is version 1.0.10, but on github it is listed as version 1.0.99.9000, which will probably become version 1.1.0 when it is released onto CRAN).

Tools used: various packages in R (tidycensus, dplyr, ggplot2, sf)

It came up with the idea of how to combine Data Engineering with Cloud and automation. I needed to find a data source as it would be an automated pipeline, so I needed a dynamic source. At the same time, I wanted to find a site where I thought retrieving data would not be a problem and do practice with both rvest and dplyr. After I had no problems with my experiments with Carvago, I added the necessary data cleaning steps. Another thing I aimed for in the project was to keep the data in different ways in different environments. While raw (daily CSV) and processed data were written to the Github repo, I wrote the processed data to PostgreSQL on AWS RDS. In addition, I sync the raw and processed data to S3 to be able to use it with Athena. However, I have separated some stages for GitHub Actions to be a good practice. For example, in the first stage, I added synchronization with AWS S3 as a separate action while scraping data, cleaning, and printing fundamental analysis to a simple log file. If there is no error after all this, I added a report with RMarkdown and the action that will be published on github.io. Thus, I created an end-to-end data pipeline where the data from the source is made to offer basic reporting with simple processing.

The API is heavily influenced by Tidy Data and borrows much of its design from dplyr. The philosophy is heavily influenced by this passage from dplyr's documentation:

Also - I might have been the one that put in the request for .by in dplyr 😅

For my day job I work on ibis. ibis lets users write queries using a familiar dataframe-like API, and then execute those queries on a number of SQL (and non-SQL) backends. Think of it like dplyr but for Python.

The special function mutate() and pivot_longer() come from the package dplyr which helps to make R-syntax more readable.