Managing outdated pull requests is time-consuming. Mergify's Merge Queue automates your pull request management & merging. It's fully integrated to GitHub & coordinated with any CI. Start focusing on code. Try Mergify for free. Learn more →
Top 9 Go Crawling Projects
-
Project mention: Colly: Elegant Scraper and Crawler Framework for Golang | news.ycombinator.com | 2023-08-23
-
-
Mergify
Tired of breaking your main and manually rebasing outdated pull requests?. Managing outdated pull requests is time-consuming. Mergify's Merge Queue automates your pull request management & merging. It's fully integrated to GitHub & coordinated with any CI. Start focusing on code. Try Mergify for free.
-
I have tried the following. 1. Login to Okta via browser programatically using go-rod. Which I managed to do so successfully, but I'm failing to load up Slack as it's stuck in the browser loader screen for Slack. 2. I tried to authenticate via Okta RESTful API. So far, I have managed to authenticate using {{domain}}/api/v1/authn, and then subsequently using MFA via the verify endpoint {{domain}}/api/v1/authn/factors/{{factorID}}/verify which returns me a sessionToken. From here, I can successfully create a sessionCookie which have proven quite useless to me. Perhaps I am doing it wrongly.
-
hakrawler
Simple, fast web crawler designed for easy, quick discovery of endpoints and assets within a web application
This week i have used https://github.com/hakluke/hakrawler and I think it meets youre requirements.
-
cariddi
Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more
cariddi is an open source (https://github.com/edoardottt/cariddi) web security tool. It takes as input a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more.
-
-
webpalm
WebPalm is a powerful command-line tool for website mapping and web scraping. With its recursive approach, it can generate a complete tree of all webpages and their links on a website. It can also extract data from the body of each page using regular expressions, making it an ideal tool for web scraping and data extraction.
-
SonarQube
Static code analysis for 29 languages.. Your projects are multi-language. So is SonarQube analysis. Find Bugs, Vulnerabilities, Security Hotspots, and Code Smells so you can release quality code every time. Get started analyzing your projects today for free.
-
-
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
The latest post mention was on 2023-08-23.
Go Crawling related posts
- 🌐 "WebPalm: Unleash Websites" 🌐
- What is a good bug Bounty methodology?
- webpalm
- webpalm
- New Modern Crawling tool written with go
- New Modern Fast Crawler
- Webpalm - Modern fast web crawling tool in go
-
A note from our sponsor - Mergify
blog.mergify.com | 22 Sep 2023
Index
What are some of the best open-source Crawling projects in Go? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | colly | 20,763 |
| 2 | Ferret | 5,465 |
| 3 | rod | 4,182 |
| 4 | hakrawler | 3,941 |
| 5 | cariddi | 984 |
| 6 | Dataflow kit | 616 |
| 7 | webpalm | 285 |
| 8 | antch | 252 |
| 9 | bathyscaphe | 84 |
Collect and Analyze Billions of Data Points in Real Time
Manage all types of time series data in a single, purpose-built database. Run at any scale in any environment in the cloud, on-premises, or at the edge.
www.influxdata.com