node-fetch VS Nock

To execute the algorithm, we will use Node.js (for the JavaScript runtime) and node-fetch (for network requests). This means we will run the code locally from the command line. For this project, we will have an output folder to store all the README data, as well as a list (queue) of repository URLs to visit. Before diving into the code, it is important to plan the input and output of the algorithm. For this web crawler, we will start at a valid GitHub repository page, which would be one URL string. After visiting each page with a README, we will export the data into a new file. Now lets cover the process of requesting a repository page from a URL. For this, we only care about saving the README file that is displayed, and we will ignore any other links that GitHub displays (such as the navbar). We will send a URL request with node-fetch, and retrieve the result of a HTML string. If we convert the HTML string to a DOM Tree, we can search for a specific element. GitHub stores the README file under a div with the class "markdown-body". We can use a library called 'jsdom' to use Browser API methods, and return a specific node.

Note: In case you run into install reference error: fetch isn’t defined, ensure you install node-fetch

Node Fetch is a JavaScript library tailored for Node.js that simplifies making HTTP requests. It offers a straightforward and Promise-based approach for fetching resources from the internet or server, such as GET, POST, PUT, and DELETE requests. Designed for server-side applications, it's compatible with the Fetch API, allowing easy code transition between client-side and server-side environments.

Would anyone be interested in an article about the crusade to move JS to ESM? I've been considering writing one, here's a preview:

Sindresorus wrote a gist "Pure ESM modules"[0] and converted all his modules to Pure ESM, breaking anyone `require`ing his code; he later locked the thread to prevent people from complaining. node-fetch released a pure ESM version a year ago that is 16x less popular than the CommonJS version[1]. The results of these changes broke a lot of code and resulted in many hours of developers figuring out how make their projects compatible with Pure ESM modules (or decide to ignore them and use old CommonJS versions)--not to mention the tons of pointless drama on GitHub issues.

Meanwhile, TC-39 member Matteo Collima advocated a moderate approach dependent on where your module will be run [2]. So the crusade is led not by the Church, but by a handful of zealots dedicated to establishing ESM supremacy for unclear reasons (note how Sindresorus' gist lacks any justifications). It's kind of like the Python 2 to 3 move except with even less rationale and not driven by the core devs.

0 - https://gist.github.com/sindresorhus/a39789f98801d908bbc7ff3...

1 - https://www.npmjs.com/package/node-fetch?activeTab=versions

2 - https://github.com/nodejs/node/issues/33954#issuecomment-924...

https://www.npmjs.com/package/node-fetch is pretty standard assuming you're referring to an HTTP client library

The API is CORS-enabled, meaning you’ll have to use the getArticles() functions from your backend. For making the actual request, you can use the fetch() function, available since Node.js v18. For older versions of Node.js, you can use a fetch()-compatible library like node-fetch.

I have the same setup. On node 18 fetch would not go through. I changed 127.0.0.1 to localhost in my config/env. More info here

I am making a bot that is supposed to take data from Upland's database from the account "dcrawtu15ye". I am using autocode to take it and I have found some ways to use it but some of my code still comes back as null. I have been using the eos docs to find info and all it can do right now is get account info if I use console.log(await rpc.get_account('dcrawtu1u5ye'));. I am using the dependency node-fetch. I wanted to know if there is something wrong with the code below. I also used greymass from this list and this article supposedly might help too.

The RSS feed's URL will then need to be requested over the network. The native fetch API of JavaScript will be used since it is the most efficient. It undoubtedly works in browsers, and it appears that Node has a pretty well-liked implementation of it.

docs: Documentation is an essential part of any package, as it provides users with the information they need to understand how to use it and how it works. The documentation usually includes usage instructions, API documentation, and more. It can also be included directly in the repository's README.md file, but it's often split into multiple files and stored in this folder for ease of navigation and maintenance. Although the documentation files can be in any format, the most common one is Markdown. Example from node-fetch.

So, why would you want a REAL server to mock request/reponses? You have a lot of intercepts today that sit on the network layer and you can define things like "If you send request to that endpoint, with that json, please return that Status" (for NodeJS example, Nock - https://github.com/nock/nock)

That said, folks seem to like "recording" features in these sorts of tools (Ruby's VCR, nock, etc), so maybe there's a future where I add something similar. I've always just found the ergonomics of those features awkward to deal with, especially having to flip back and forth between tests and fixtures files to figure out what's wired to what, but maybe there's a clean solution... perhaps a "live request" mode that just prints mock code snippets of request/response pairs passing through your app.

While not mocking per say I usually use nock for http calls. You can use nock.recorder.rec() to capture the http call to play back during test, That way you are always using "live" code but not making real calls to servers.

I looked at the service which is used to get the feed URLs from a blog URL and noticed it takes the html response of the blog URL and gets the links ( tags) by checking the type attribute value against a list of valid feed values. So, I decided to use a similar approach by getting the html response for a provided URL and checking the Content-Type header against a list of valid MIME types for a feed. I ended up updating the logic to test if a URL is a feed URL, returning it if true. If the URL is found to not be a feed URL, it would try to get the feed URLs assuming the URL is a blog URL. I tested and confirmed that the new logic worked for both blog and feed URLs. Then, I added some tests for the new function I added to test for a feed URL. Testing this ended up being simpler than I expected as all I had to do was mock the response of a test url (using nock), and then check if the function returned the correct boolean value for a url. I created a PR and noticed that some of the tests in another file were now failing. While I was investigating this, I got a review on my PR, requesting me to add another test to the file which had the failing tests. That file tested the API service as a whole. I found out that nock only mocks a URL's response for one request by default. And since I was now checking for a feed URL as well, the function which returned the feed URLs from a blog URL was throwing an error since the nock for that was used up. To fix this, I had to specify in the nock statement to mock the URL response for two requests:

There are libraries like https://github.com/nock/nock to prevent mocking the whole axios.

Use nock: https://github.com/nock/nock

Going based off the documentation I sent you in my last reply, there is an example that uses nock to emulate api responses. I haven't used nock myself, but the example seems pretty simple to use. You just need to take the example and change the response object to be the shape of what your getStuffFromDatabase function returns. That way your useCategory function runs as close to normally as possible, while providing a mock response value instead of hitting the database.

If you’re willing to venture into nodejs territory, then nock is a fantastic and simple to set up http mock server. https://github.com/nock/nock