Scrapy VS seaborn

Scrapy is a robust and scalable open-source web crawling framework. It is highly efficient for large-scale projects and supports asynchronous scraping.

Language: Python | GitHub: 52.9k stars | link

One might ask, what about Scrapy? I'll be honest: I don't really keep up with their updates. But I haven't heard about Zyte doing anything to bypass TLS fingerprinting. So out of the box Scrapy will also be blocked, but nothing is stopping you from using curl_cffi in your Scrapy Spider.

Install scrapy (Offical website) either using pip or conda (Follow for detailed instructions):

Using Scrapy I fetched the data needed (activities and attendance). Scrapy handled authentication using a form request in a very simple way:

Scrapy is an open-source Python-based web scraping framework that extracts data from websites. With Scrapy, you create spiders, which are autonomous scripts to download and process web content. The limitation of Scrapy is that it does not work very well with JavaScript rendered websites, as it was designed for static HTML pages. We will do a comparison later in the article about this.

Below are the key insights. If you want to see the Python code I used to do this analysis and generate the charts using Seaborn, you can find my full analysis Jupyter notebook on my Github repo here: Tip Analysis.ipynb

seaborn

Additionally, Seaborn (https://seaborn.pydata.org/) is a great mention for people that want to use Matplotlib with better default aesthetics, amongst other conveniences:

"Seaborn is a Python data visualization library based on matplotlib. It provides a high-level interface for drawing attractive and informative statistical graphics."

Seaborn: built on top of matplotlib, adds a number of functions to make common statistical visualizations easier to generate.

pandas - The standard data analysis and manipulation tool numpy - scientific computing library seaborn - statistical data visualization sklearn - basic machine learning and predictive analysis CausalML - a suite of uplift modeling and causal inference methods PyTorch - professional deep learning framework PivotTablejs - Drag’n’drop Pivot Tables and Charts for Jupyter/IPython Notebook LazyPredict - build and work with and compare multiple models phidata - Build AI Assistants with memory, knowledge and tools. Lux - automates visualization and data analysis pycaret - low-code machine learning library. really nice Cleanlab - for when you are working with messy data drawdata - draw a dataset from inside Jupyter pyforest - lazy import popular data science libs streamlit - simple ui builder, useful for demonstrating ML results

How to Accomplish: Utilize visualization libraries like Matplotlib, Seaborn, or Plotly in Python to create histograms, scatter plots, and bar charts. For image data, use tools that visualize images alongside their labels to check for labeling accuracy. For structured data, correlation matrices and pair plots can be highly informative.

If you are doing data analysis I don't think any of the 3 pieces of software you mentioned are going to be that helpful.

I see these products as tools for data visualization and reporting i.e. presenting prepared datasets to users in a visually appealing way. They aren't as well suited for serious analytics.

I can't comment on Superset or Tableau but I am familiar with Power BI (it has been rolled out across my org), the type of statistics you can do with it are fairly rudimentary. If you need to do any thing beyond summarizing (counts, averages, min, max etc). It is not particularly easy.

For data analysis I use SAS or R. This software allows you do things like multivariate regression, timeseries forecasting, PCA, Cluster analysis etc. There is also plotting capability.

Both these products are kind of old school, I've been using them since early 2000's, the "new school" seems to be Python. Pretty much all the recent data science people in my organization use Python. Particularly Pandas and libraries like Seaborn (https://seaborn.pydata.org/).

The "power" users of Power BI in my organization tend to be finance/HR people for use cases like drill down into cost figures or Interactively presenting KPI's and other headline figures to management things like that.

It's referring to the seaborn library (https://seaborn.pydata.org/), a Python library for data visualization (built on top of matplotlib).

While it’s not perfect and it’s not ggplot2, Seaborn is definitely a big improvement over bare matplotlib. You can still use matplotlib to modify the plots it spits out if you want to but the defaults are pretty good most of the time.

https://seaborn.pydata.org/