rvest VS scikit-learn

You’re looking for the rvest package

Before diving into advanced machine learning algorithms or statistical models, we need to start with the basics: collecting and organizing data. Fortunately, both Python and R offer a wealth of libraries that make it easy to collect data from a variety of sources, including web scraping, APIs, and reading from files. Key libraries in Python include requests, BeautifulSoup, and pandas, while R has httr, rvest, and dplyr.

Tools: R + Rvest to scrape and clean the data. D3 to create the map. Svelte to put it all together.

so ik R has packages and native functions to help bypass this manual process. Eg scraping the wiki / gamepress unit list with Rvest may prove easier, furthermore you can specify web based sources when reading data. I'm not giga familiar with doing either myself, but maybe you can scrape data from the wikis or from repositories like the feh assets 1. But if youre able to set up a simple R script to read in new data and transform / clean it and save manual updates every 2 weeks

There are very similar tools in R that I cover in that tutorial. For example, rvest or xml2 should be able to do the job as both of them support XPath selectors (you can take the ones from the article - they should work in R too).

To get the data from the website, I need to use the package (a set of R code someone created and shared that's designed for a certain task) rvest, then I did a bunch of data munging in R to pull out the location/salary/age/etc. I saved that in a dataset and then used another package flexdashboard to make a webpage which I can essentially "one-click" publish using a free tool called RPubs.

It came up with the idea of how to combine Data Engineering with Cloud and automation. I needed to find a data source as it would be an automated pipeline, so I needed a dynamic source. At the same time, I wanted to find a site where I thought retrieving data would not be a problem and do practice with both rvest and dplyr. After I had no problems with my experiments with Carvago, I added the necessary data cleaning steps. Another thing I aimed for in the project was to keep the data in different ways in different environments. While raw (daily CSV) and processed data were written to the Github repo, I wrote the processed data to PostgreSQL on AWS RDS. In addition, I sync the raw and processed data to S3 to be able to use it with Athena. However, I have separated some stages for GitHub Actions to be a good practice. For example, in the first stage, I added synchronization with AWS S3 as a separate action while scraping data, cleaning, and printing fundamental analysis to a simple log file. If there is no error after all this, I added a report with RMarkdown and the action that will be published on github.io. Thus, I created an end-to-end data pipeline where the data from the source is made to offer basic reporting with simple processing.

I would try some more nuanced web scraping with a package like rvest

1) Use rvest to extract the contents of the XML file (i.e. loop over top-level nodes and pull any variable you're interested in into a column).

Thank you for your interest. There are some interesting examples in the SWE-bench-lite benchmark which are resolved by AutoCodeRover:

- From sympy: https://github.com/sympy/sympy/issues/13643. AutoCodeRover's patch for it: https://github.com/nus-apr/auto-code-rover/blob/main/results...

- Another one from scikit-learn: https://github.com/scikit-learn/scikit-learn/issues/13070. AutoCodeRover's patch (https://github.com/nus-apr/auto-code-rover/blob/main/results...) modified a few lines below (compared to the developer patch) and wrote a different comment.

There are more examples in the results directory (https://github.com/nus-apr/auto-code-rover/tree/main/results).

sklearn is adding support through the dataframe interchange protocol (https://github.com/scikit-learn/scikit-learn/issues/25896). scipy, as far as I know, doesn't explicitly support dataframes (it just happens to work when you wrap a Series in `np.array` or `np.asarray`). I don't know about PyTorch but in general you can convert to numpy.

Wow, from the upvotes on this comment, it really seems like a lot of people think that this is the correct behavior! I have to say I disagree, but if that's what you think, don't just sit there upvoting comments on Reddit; instead go to this PR and tell the Scikit-Learn maintainers not to "fix" this "bug", which they are currently planning to do!

I was in a similar boat to yours - Worked in data science and since then have made a move to data engineering and software engineering for ML services.

I would recommend you look into the Design Patterns book by the Gang of Four. I found it particularly helpful to make extensible code that doesn't break specially with abstract classes, builders and factories. I would also recommend looking into the book The Object Oriented Thought Process to understand why traditional OOP is build the way it is.

You can also look into the source code of popular data science libraries such as sklearn (https://github.com/scikit-learn/scikit-learn/tree/main/sklea...) and see how a lot of them have Base classes to define shared functionality between object of the same nature.

As others mentioned, I would also encourage you to try and implement design patterns in your everyday work - maybe you can make a Factory to load models or preprocessors that follow the same Abstract class?

It looks like you've been the victim of some misinformation. As Dr_Birdbrain said, an SVM is a convex problem with unique global optimum. sklearn.SVC relies on libsvm which initializes the weights to 0 [0]. The random state is only used to shuffle the data to make probability estimates with Platt scaling [1]. Of the random_state parameter, the sklearn documentation for SVC [2] says

Controls the pseudo random number generation for shuffling the data for probability estimates. Ignored when probability is False. Pass an int for reproducible output across multiple function calls. See Glossary.

[0] https://github.com/scikit-learn/scikit-learn/blob/2a2772a87b...

[1] https://en.wikipedia.org/wiki/Platt_scaling

[2] https://scikit-learn.org/stable/modules/generated/sklearn.sv...

Scikit-learn Documentation

Firstly, we need a connection to Memgraph so we can get edges, split them into two parts (train set and test set). For edge splitting, we will use scikit-learn. In order to make a connection towards Memgraph, we will use gqlalchemy.

The ML component is based on scikit-learn which differentiates it from purely list-based filters. It couples this with a full-featured wireless router (RaspAP) in a single device, so it fulfills the needs of a use case not entirely addressed by Pi-hole.