Web Crawler in Go: Extracting Keyword-Relevant Text with Text Density

This page summarizes the projects mentioned and recommended in the original post on /r/golang

InfluxDB - Power Real-Time Data Analytics at Scale
Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
www.influxdata.com
featured
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com
featured
  • TD-Spider

    Via Text Density Simple Web Crawler With Go

    git link

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo
  • chromedp

    A faster, simpler way to drive browsers supporting the Chrome DevTools Protocol.

    There are a billion things that you need to consider when building a decent web crawler, especially interacting with pages in the modern web. For example, a lot of content is dynamically loaded by the browser nowadays, and won't show up if you make a simple HTTP request. Open your browser devtools and look at the network tab after you make a request, and you'll see it makes loads of auxiliary requests. Some content is also only loaded after you interact with it (e.g. hover, click). For that reason I'd recommend using something like chromedp and do browser based crawling, even if it's much slower.

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts

  • How can I do infinte scroll with go http

    1 project | /r/golang | 16 Nov 2022
  • Parsing Dynamic HTML content using goquery

    2 projects | /r/golang | 25 Feb 2022
  • Doubt in web scraping using go/colly

    1 project | /r/golang | 4 Dec 2021
  • Chromedp on real android device

    1 project | /r/golang | 29 Oct 2021
  • About chromedp

    1 project | /r/devopskhan | 25 Oct 2021

Did you konow that Go is
the 4th most popular programming language
based on number of metions?