Python Big Data

Open-source Python projects categorized as Big Data
Edit details
Topics: #Python #Spark #data-engineering #Machine Learning #Data Science
Collect and Analyze Billions of Data Points in Real Time
Manage all types of time series data in a single, purpose-built database. Run at any scale in any environment in the cloud, on-premises, or at the edge.
www.influxdata.com
Sponsored

Top 23 Python Big Data Projects

  • data-science-ipython-notebooks

    Data science Python notebooks: Deep learning (TensorFlow, Theano, Caffe, Keras), scikit-learn, Kaggle, big data (Spark, Hadoop MapReduce, HDFS), matplotlib, pandas, NumPy, SciPy, Python essentials, AWS, and various command lines.

  • Cython

    The most widely used Python to C compiler

    Project mention: Ask HN: Is there a way to use Python statically typed or with any type-checking? | news.ycombinator.com | 2023-08-06

  • InfluxDB

    Collect and Analyze Billions of Data Points in Real Time. Manage all types of time series data in a single, purpose-built database. Run at any scale in any environment in the cloud, on-premises, or at the edge.

  • catboost

    A fast, scalable, high performance Gradient Boosting on Decision Trees library, used for ranking, classification, regression and other machine learning tasks for Python, R, Java, C++. Supports computation on CPU and GPU.

  • feast

    Feature Store for Machine Learning

    Project mention: What's Happening with Feast? | news.ycombinator.com | 2023-12-07

  • Stream-Framework

    Stream Framework is a Python library, which allows you to build news feed, activity streams and notification systems using Cassandra and/or Redis. The authors of Stream-Framework also provide a cloud service for feed technology:

    Project mention: Recommendations for an external messenger integration/API? | /r/rails | 2023-10-30

    I have looked into a getstream.io integration, however it seems that the Ruby SDK is really treated as a second class citizen. There's bugs with the documented API (I'm having issues even creating users and querying users), the usage of the gem is low and there is an open issue since May that no one has even looked at, which doesn't give me hope for long term support.

  • koalas

    Koalas: pandas API on Apache Spark

  • img2dataset

    Easily turn large sets of image urls to an image dataset. Can download, resize and package 100M urls in 20h on one machine.

    Project mention: OpenAI sued for web scraping from millions of internet users in order to train ChatGPT | /r/ArtistHate | 2023-06-30

    Lmao, no it doesn't. As we can see, their downloader uses very obscure "no ai" headers (which can be disabled, so its useless). They only claim it respects "robots.txt" because the google crawler respects it, if a site changes their robots.txt rules they don't remove it from their dataset, that is not "respecting". https://github.com/rom1504/img2dataset

  • Onboard AI

    Learn any GitHub repo in 59 seconds. Onboard AI learns any GitHub repo in minutes and lets you chat with it to locate functionality, understand different parts, and generate new code. Use it for free at www.getonboard.dev.

  • scikit-learn-intelex

    Intel(R) Extension for Scikit-learn is a seamless way to speed up your Scikit-learn application

  • DataEngineeringProject

    Example end to end data engineering project.

  • Hail

    Cloud-native genomic dataframes and batch computing

    Project mention: We're wasting money by only supporting gzip for raw DNA files | news.ycombinator.com | 2023-01-09

  • NIPY

    Workflows and interfaces for neuroimaging packages

  • opendata.cern.ch

    Source code for the CERN Open Data portal

    Project mention: NFS > FUSE: Why We Built Our Own NFS Server in Rust | news.ycombinator.com | 2023-09-19

    > XetHub has the world’s first natively cross-platform, user-mode filesystem implementation, allowing you to mount arbitrarily large datasets on your machine.

    Not really world's first. CERN has developed EOS (https://eos-web.web.cern.ch/) for many years, and even though it's not available natively on Windows, it is available on Linux and macOS. EOS uses FUSE, though, not NFS.

    > This enables you to, in just a few seconds, locally mount ~660 GB of Llama 2 models or write DuckDB queries to analyze large parquet files and scan just the data you need.

    If you mount all instances of EOS at CERN on your machine with the FUSE client, that in principle mounts hundreds of PB of data from LHC experiments, although much of this data requires special permissions to be accessed. However, there's also a lot of open data. See https://opendata.cern.ch/.

  • listenbrainz-server

    Server for the ListenBrainz project, including the front-end (javascript/react) code that it serves and all of the data processing components that LB uses.

    Project mention: EP Showing Twice | /r/lastfm | 2023-09-21

    If you really <3 clean data you can give listenbrainz.org a go (you can scrobble to that and last.fm concurrently so no need to ‘jump ship’)

  • eland

    Python Client and Toolkit for DataFrames, Big Data, Machine Learning and ETL in Elasticsearch

    Project mention: I'm getting elasticsearch.BadRequestError: BadRequestError(400, 'illegal_argument_exception', "specified fields can't be null or empty") using Eland library | /r/elasticsearch | 2023-05-02

    We have a fix for this issue reported here merged and pending a release. Hopefully that release will happen in the next few days, then you can upgrade and the default experience for everyone won't be as confusing :)

  • redislite

    Redis in a python module.

  • video2dataset

    Easily create large video dataset from video urls

    Project mention: FLaNK Stack Weekly for 17 July 2023 | dev.to | 2023-07-17

  • lithops

    A multi-cloud framework for big data analytics and embarrassingly parallel jobs, that provides an universal API for building parallel applications in the cloud ☁️🚀

    Project mention: Lithops: A multi-cloud framework for embarrassingly parallel jobs | news.ycombinator.com | 2023-01-14

  • amazon-s3-find-and-forget

    Amazon S3 Find and Forget is a solution to handle data erasure requests from data lakes stored on Amazon S3, for example, pursuant to the European General Data Protection Regulation (GDPR)

  • uproot5

    ROOT I/O in pure Python and NumPy.

    Project mention: Potential of the Julia programming language for high energy physics computing | news.ycombinator.com | 2023-12-04

    > I wasn't proposing ROOT to be reimplemented in JS. That was what the GP attributed to me.

    Sorry for assuming that. I really felt the pain of thinking of possibility of combining two things I hate so much together (JS+ROOT)

    > "Laypeople" may also think that code is optimized to the last cycle in something like HEP simulations. It's made fast enough and the optimization is nowhere near the level of e.g. graphics heavy games.

    I understand that in other areas there might be more sophisticated optimizations, but does not change things much inside HEP field community. And it is not optimized only for simulations but for other things too. It is not one problem optimization.

    > Real-time usage like high frequency large data collection will probably never happen on the "single language". But I'd guess ROOT is not used at that level either? Also at least last time I checked, ROOT is moving to Python (probably not for the hottest loops of the simulation though).

    I did not mean to indicate that ROOT is being used to handle the online processing (In HEP terms). It is usually handled via optimized C++ compiled code. My idea is that you will probably never use JS or any interpreted language (or anything other than C++ to be pessimistic) for that. ROOT at the end of the day is much closer to C++ than anything else. So learning curve wouldn't be that much if you come with some C++ knowledge initially.

    > Also at least last time I checked, ROOT is moving to Python (probably not for the hottest loops of the simulation though).

    I think you mean PyROOT [1]? This is the official python ROOT interface It provides a set of Python bindings to the ROOT C++ libraries, allowing Python scripts to interact directly with ROOT classes and methods as if they were native Python. But that does not represent and re-writing. It makes things easier for end users who are doing analysis though, while be efficient in terms of performance, especially for operations that are heavily optimized in ROOT.

    There is also uproot [2] which is a purely Python-based reader and writer of ROOT files. It is not a part of the official ROOT project and does not depend on the ROOT libraries. Instead, uproot re-implements the I/O functionalities of ROOT in Python. However, it does not provide an interface to the full range of ROOT functionalities. It is particularly useful for integrating ROOT data into a Python-based data analysis pipeline, where libraries like NumPy, SciPy, Matplotlib, and Pandas ..etc are used.

    > Off-topic: C++ interpretation like done in ROOT seems like a really bad idea.)

    I will agree with you. But to be fair the purpose of ROOT is interactive data analysis but over the decades a lot of things gets added, and many experiments had their own soft forks and things started to get very messy quickly. So that there is no much inertia to fix problems and introduce improvements.

    [1] https://root.cern/manual/python/

    [2] https://github.com/scikit-hep/uproot5

  • pmaw

    A multithread Pushshift.io API Wrapper for reddit.com comment and submission searches.

    Project mention: I am making a replacement service to keep the 3rd party apps running. Join me. | /r/Save3rdPartyApps | 2023-06-05

    Oh a whole lot of them. Pushshift was large enough to have its own libraries ( https://github.com/mattpodolak/pmaw ) so the first step will be to recreate the API 1:1 and get tools like reddit search and reveddit working again.

  • hazelcast-python-client

    Hazelcast Python Client

  • covid19-sir

    CovsirPhy: Python library for COVID-19 analysis with phase-dependent SIR-derived ODE models.

  • Lime-For-Time

    Application of the LIME algorithm by Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin to the domain of time series classification

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2023-12-07.

Python Big Data related posts

Index

What are some of the best open-source Big Data projects in Python? This list will help you:

Project Stars
1 data-science-ipython-notebooks 25,836
2 Cython 8,567
3 catboost 7,489
4 feast 4,964
5 Stream-Framework 4,706
6 koalas 3,312
7 img2dataset 2,841
8 scikit-learn-intelex 1,084
9 DataEngineeringProject 922
10 Hail 895
11 NIPY 706
12 opendata.cern.ch 608
13 listenbrainz-server 597
14 eland 574
15 redislite 555
16 video2dataset 352
17 lithops 286
18 amazon-s3-find-and-forget 221
19 uproot5 206
20 pmaw 205
21 hazelcast-python-client 113
22 covid19-sir 101
23 Lime-For-Time 86

Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com