Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality. Learn more ā
Top 23 Python GCP Projects
-
prowler
Prowler is an Open Source Security tool for AWS, Azure, GCP and Kubernetes to do security assessments, audits, incident response, compliance, continuous monitoring, hardening and forensics readiness. Includes CIS, NIST 800, NIST CSF, CISA, FedRAMP, PCI-DSS, GDPR, HIPAA, FFIEC, SOC2, GXP, Well-Architected Security, ENS and more
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
-
checkov
Prevent cloud misconfigurations and find vulnerabilities during build-time in infrastructure as code, container images and open source packages with Checkov by Bridgecrew.
-
-
cloud-custodian
Rules engine for cloud security, cost optimization, and governance, DSL in yaml for policies to query, filter, and take actions on resources
-
ethereum-etl
Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Google BigQuery https://goo.gl/oY5BCQ
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
OPAL
Policy and data administration, distribution, and real-time updates on top of Policy Agents (OPA, Cedar, ...) (by permitio)
-
fixinventory
Fix Inventory consolidates user, resource, and configuration data from your cloud environments into a unified, graph-based asset inventory.
-
dstack
dstack is an open-source orchestration engine for running AI workloads at scale in any cloud or data center. https://discord.gg/u8SmfwPpMd
-
optscale
FinOps and MLOps platform to run ML/AI and regular cloud workloads with optimal performance and cost.
-
-
opta
The next generation of Infrastructure-as-Code. Work with high-level constructs instead of getting lost in low-level cloud configuration.
-
energy-forecasting
š š§šµš² ššš¹š¹ š¦šš®š°šø š³-š¦šš²š½š š šš¢š½š ššæš®šŗš²šš¼šæšø | šš²š®šæš» š šš & š šš¢š½š for free by designing, building and deploying an end-to-end ML batch system ~ š“š°š¶š³š¤š¦ š¤š°š„š¦ + 2.5 š©š°š¶š³š“ š°š§ š³š¦š¢š„šŖšÆšØ & š·šŖš„š¦š° š®š¢šµš¦š³šŖš¢šš“
-
zentral
Zentral is a high-visibility platform for controlling Apple endpoints in enterprises. It brings great observability to IT and makes tracking & reporting compliance much less manual.
-
runhouse
Fast, Pythonic AI services and workflows on your own infra. Unobtrusive, debuggable, PyTorch-like APIs.
-
-
-
streamify
A data engineering project with Kafka, Spark Streaming, dbt, Docker, Airflow, Terraform, GCP and much more!
-
ethereum-etl-airflow
Airflow DAGs for exporting, loading, and parsing the Ethereum blockchain data. How to get any Ethereum smart contract into BigQuery https://towardsdatascience.com/how-to-get-any-ethereum-smart-contract-into-bigquery-in-8-mins-bab5db1fdeee
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
Project mention: Ask HN: Cloud security auditing for indie-grade projects? | news.ycombinator.com | 2023-12-04Which cloud provider?
https://github.com/prowler-cloud/prowler is easy to get going with, and gives decent results. It's much stronger at AWS than GCP or Azure.
Steampipe can be a little harder to wrap your head around, but scales really well and has broader support: https://hub.steampipe.io/mods?objectives=security
Project mention: A Deep Dive Into Terraform Static Code Analysis Tools: Features and Comparisons | dev.to | 2024-04-16Checkov Owner/Maintainer: Prisma Cloud by Palo Alto Networks (acquired in 2021) Age: First released on GitHub on March 31st, 2021 License: Apache License 2.0
Prowler and ScoutSuite are a good start for cloud stuff.
Project mention: Cutting down AWS cost by $150k per year simply by shutting things off | news.ycombinator.com | 2024-01-22> The best optimization is simply shutting things off
This is the way.
A similar idea has been bouncing around in my mind for a while now. An ideal, turnkey system would do the following:
- Execute via Lambda (serverless).
- Support automated startup and shutdown of various AWS resources on a schedule influenced by specially formatted tags.
- Enable resources to be brought back up out of schedule when demand dictates.
- Operate as a TCP/HTTP proxy that can delay clients so that a given service can be started when it is dormant or, even better, the service isn't serverless but you want it to be. This can't work for everything, but perhaps enough things such that the need to run always on services is reduced.
Cloud Custodian [1] can purportedly do some of this, but I've been reluctant to learn yet another YAML-based DSL to use it.
So this is my "make things designed to be always-on serverless instead" project and the work AWS has done to make Java apps function on Lambda keeps me thinking about the potential to take things that 1) have a relatively long startup time and 2) are designed to be long running service loops, and find a way to force them into the serverless execution model.
[1] https://cloudcustodian.io/
Project mention: Blockchain transactions decoding: making wallet activity understandable | dev.to | 2023-10-27Event is a log entity which EVM smart contracts can emit during transaction execution. Events are very good at signalling that an some action has taken place on-chain. Applications can subscribe and listen to events to trigger some off-chain logic or they can index, transform and store events in some off-chain storage (look at The Graph protocol or Ethereum ETL).
Another tool that can help you deploy a Policy as Code-based solution in 2024 is OPAL, the Open Policy Administration Layer. OPAL is an open-source project that provides a comprehensive policy-based service for applications. With one click, you can deploy a full architecture of a Git-based centralized policy store with decentralized policy engines running as a sidecar with your applications. OPAL also provides a unified architecture to sync all the data you need with the policy engines.
Project mention: Show HN: Fix ā An open source cloud asset inventory for cloud security engineers | news.ycombinator.com | 2024-03-27The reasoning is explained in the very section of our Github org README you quoted this sentence from. Our main open source project is Fix Inventory (https://github.com/someengineering/fixinventory) and that is very well documented (https://inventory.fix.security) and uses no commercial 3rd party libraries.
The Fix SaaS frontend that you're referring to and that you find at https://fix.security builds upon Fix Inventory. We could have just made it closed-source like every other SaaS (think Grafana Cloud). But because I'm a big proponent of OSS we decided to open source our entire SaaS stack, frontend, backend as well as all internal tooling. The main intend here is transparency, not so you spin up your own SaaS environment.
Essentially we develop the SaaS for ourselves first and foremost, but saw no reason to make it closed source. So that is why it might be using any number of commercial 3rd party add-ons.
> I'm curious to know what Material UI provided that any other open-source UI library did not.
I believe it was some MUI X table features like multi row sorting that we didn't feel like re-implementing. I'm sure there's other open source libs that would do that, but we've settled on MUI and are not going to start mixing different UI libraries for different visual elements if we don't absolutely have to.
Project mention: Ask HN: How does deploying a fine-tuned model work | news.ycombinator.com | 2024-04-23You can use https://github.com/dstackai/dstack to deploy your model to the most affordable GPU clouds. It supports auto-scaling and other features.
Disclaimer: Iām the creator of dstack.
Project mention: Profile and instrument ML experiments and optimize their performance expenses | news.ycombinator.com | 2023-09-27
Project mention: Full Stack 7-Steps MLOps Framework ā Learn ML and MLOps by Building an ML System | news.ycombinator.com | 2023-09-26
Project mention: A user-built manual on using TPUs for AI and LLMs on Google Cloud | news.ycombinator.com | 2023-08-04
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
Python GCP related posts
- Let's build your first ML app in Google Cloud Run
- Any better alternatives to fine-tuning GPT-3 yet to create a custom chatbot persona based on provided knowledge for others to use?
- Azure and M365 Secure Config Review
- Can I use AWS to build an adult website?
- Apim + function app & event grid
- BigQuery Remote Functions using Goblet
- Invoice granularity: Show different accounts/cost allocation tags on invoice
-
A note from our sponsor - InfluxDB
www.influxdata.com | 26 Apr 2024
Index
What are some of the best open-source GCP projects in Python? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | prowler | 9,547 |
| 2 | docker-android | 8,086 |
| 3 | metaflow | 7,586 |
| 4 | checkov | 6,512 |
| 5 | ScoutSuite | 6,159 |
| 6 | cloud-custodian | 5,201 |
| 7 | ethereum-etl | 2,819 |
| 8 | OPAL | 2,281 |
| 9 | fixinventory | 1,533 |
| 10 | dstack | 1,087 |
| 11 | optscale | 969 |
| 12 | skyplane | 968 |
| 13 | opta | 907 |
| 14 | energy-forecasting | 786 |
| 15 | zentral | 720 |
| 16 | runhouse | 709 |
| 17 | PurplePanda | 629 |
| 18 | spotty | 491 |
| 19 | redun | 484 |
| 20 | streamify | 474 |
| 21 | kcli | 460 |
| 22 | tpu-starter | 452 |
| 23 | ethereum-etl-airflow | 387 |
Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com