katana
rod
Our great sponsors
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
katana
-
Originally a Covid project. Now a discount search engine.
Using a few different methods. Pulling the sites I'm using Puppeteer and Katana (https://github.com/projectdiscovery/katana). To process and extract the information is tricky, most websites selling things put time into their metadata; this does make it easier. Additionally, a lot of the larger stores have common patterns between them. Failing all of this, I trained a Tensor flow model to understand how to read product pages. However, it's far from perfect and a journey of continual improvement.
-
I’m messed up with Go libraries
Anyway, check out katana https://github.com/projectdiscovery/katana
- Katana
-
Go for web scraping
Checkout https://github.com/projectdiscovery/katana It is right up there with any advanced web scraper you will find in python world or beyond.
- A next-gen crawling and spidering framework
- New crawler released by projectdiscovery - Katana
- Katana: A next-generation crawling and spidering framework
rod
-
Need help authenticating to Okta programatically.
I have tried the following. 1. Login to Okta via browser programatically using go-rod. Which I managed to do so successfully, but I'm failing to load up Slack as it's stuck in the browser loader screen for Slack. 2. I tried to authenticate via Okta RESTful API. So far, I have managed to authenticate using {{domain}}/api/v1/authn, and then subsequently using MFA via the verify endpoint {{domain}}/api/v1/authn/factors/{{factorID}}/verify which returns me a sessionToken. From here, I can successfully create a sessionCookie which have proven quite useless to me. Perhaps I am doing it wrongly.
- Library to convert HTML to pdf in Golang
- Web scraping with Go
- Best option for browser automation
-
I’m messed up with Go libraries
I usually find libraries by googling them or searching awesome go on GitHub, for selenium/puppeteer I've always found go-rod useful and easy in every way, Sometimes I also Google "X in Nodejs for Golang"
-
Go for web scraping
I recently tried out https://github.com/go-rod/rod. I think it's based on chromedp (so, Chrome dev tools and headless browser) but it also has code to download and run a supported version of Chrome that doesn't interfere with your local browser.
- Reducir tiempo de Web Scraping con concurrencia - GO
-
VHS: CLI Home Video Recorder
One of the dependencies is `rod`[0], which is a web scraping/automation library, and I believe requires a browser to work. I don't know what they're using it for though as I haven't looked at the code (and I'm not familiar with Go anyways).
0: https://github.com/go-rod/rod
-
Thoughts on Go headless browser tools for testing and scraping?
I don't have personal experience, but https://github.com/go-rod/rod is far more active than chromedp
- Project with a Web scraper GO binary
What are some alternatives?
koin - Koin - a pragmatic lightweight dependency injection framework for Kotlin & Kotlin Multiplatform
playwright-go - Playwright for Go a browser automation library to control Chromium, Firefox and WebKit with a single API.
Kodein - Painless Kotlin Dependency Injection
chromedp - A faster, simpler way to drive browsers supporting the Chrome DevTools Protocol.
KODI - light-weight KOtlin Dependency Injection (KODI)
colly - Elegant Scraper and Crawler Framework for Golang
jellyfin-sdk-kotlin - Kotlin SDK for Jellyfin, supporting Android and JVM targets
WebDumper - A tool for scraping, dumping and unpacking (webpacked) javascript source files.
butterknife - Bind Android views and callbacks to fields and methods.
realize - Realize is the #1 Golang Task Runner which enhance your workflow by automating the most common tasks and using the best performing Golang live reloading.
Dagger2 - A fast dependency injector for Android and Java.
gotests - Automatically generate Go test boilerplate from your source code.