Creating a serverless function to scrape web pages metadata

This page summarizes the projects mentioned and recommended in the original post on dev.to

Our great sponsors
  • SurveyJS - Open-Source JSON Form Builder to Create Dynamic Forms Right in Your App
  • WorkOS - The modern identity platform for B2B SaaS
  • InfluxDB - Power Real-Time Data Analytics at Scale
  • metascraper

    Get unified metadata from websites using Open Graph, Microdata, RDFa, Twitter Cards, JSON-LD, HTML, and more.

  • First of all, we'll use the got npm package to fetch the website content (feel free to use any other fetching library), and the metascraper npm package to extract the metadata:

  • vercel

    Develop. Preview. Ship.

  • Next, install vercel as a devDependency:

  • SurveyJS

    Open-Source JSON Form Builder to Create Dynamic Forms Right in Your App. With SurveyJS form UI libraries, you can build and style forms in a fully-integrated drag & drop form builder, render them in your JS app, and store form submission data in any backend, inc. PHP, ASP.NET Core, and Node.js.

    SurveyJS logo
  • url-metadata-scraper

    Tiny Vercel serverless function to scrape metadata from a URL

  • You can find the entire source code of this project on GitHub — feel free to fork it or give it a try!

  • is-video

    Check if a filepath is a video file

  • The metascraper-video package depends on the is-video package to determine if a tag contains a valid video URL, and is-video depends on the video-extensions package that holds a list of valid video extensions. Unfortunately, the video-extensions package hasn't been updated in a while now, so it doesn't support the m3u8 video extension (which is a popular video extension on the web nowadays). Until this pull request is released and is-video is updated to use the latest version of video-extensions, you can use patch-package with the following diff to manually patch the m3u8 support into video-extensions (by putting it into patches/video-extensions+1.1.0.patch).

  • video-extensions

    List of video file extensions

  • The metascraper-video package depends on the is-video package to determine if a tag contains a valid video URL, and is-video depends on the video-extensions package that holds a list of valid video extensions. Unfortunately, the video-extensions package hasn't been updated in a while now, so it doesn't support the m3u8 video extension (which is a popular video extension on the web nowadays). Until this pull request is released and is-video is updated to use the latest version of video-extensions, you can use patch-package with the following diff to manually patch the m3u8 support into video-extensions (by putting it into patches/video-extensions+1.1.0.patch).

  • patch-package

    Fix broken node modules instantly 🏃🏽‍♀️💨

  • The metascraper-video package depends on the is-video package to determine if a tag contains a valid video URL, and is-video depends on the video-extensions package that holds a list of valid video extensions. Unfortunately, the video-extensions package hasn't been updated in a while now, so it doesn't support the m3u8 video extension (which is a popular video extension on the web nowadays). Until this pull request is released and is-video is updated to use the latest version of video-extensions, you can use patch-package with the following diff to manually patch the m3u8 support into video-extensions (by putting it into patches/video-extensions+1.1.0.patch).

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts