Getting Started with a Web Scraping Project

This page summarizes the projects mentioned and recommended in the original post on dev.to

Our great sponsors
  • WorkOS - The modern identity platform for B2B SaaS
  • InfluxDB - Power Real-Time Data Analytics at Scale
  • SaaSHub - Software Alternatives and Reviews
  • simple-web-scraper

    Simple web scraper to get player data using beatiful-soup4 and PostgreSQL as a database. SQLAlchemy as an ORM

  • SQLAlchemy

    The Database Toolkit for Python

    We will now save the data to Postgres DB, this will give us persistence across sessions as well allow perform analysis on the data. To keep things consistent we'll use an Object Relational Mapper (ORM) SQLAlchemy to create the schema and psycopg2 to connect to the Postgres DB.

  • WorkOS

    The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  • pytest

    The pytest framework makes it easy to write small tests, yet scales to support complex functional testing

    Inorder to have this project be as well-rounded as possible we'll code a series of tests to test the code using pytest. Create a new directory called tests and create a new file called test_db.py in the tests directory. Inside create a test_players.py file.

  • Puts Debuggerer

    Ruby library for improved puts debugging, automatically displaying bonus useful information such as source line number and source code.

  • Visual Studio Code

    Visual Studio Code

    Visual Studio Code or any other IDE of your choice.

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts