google-cloud-storage

Open-source projects categorized as google-cloud-storage
Edit details
Language: + Go + Python + Java + JavaScript + TypeScript + C++ + PHP + C + Emacs Lisp + R
Topics: S3 aws-s3 GCP Golang Storage
Power Real-Time Data Analytics at Scale
Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
www.influxdata.com
Sponsored

Top 23 google-cloud-storage Open-Source Projects

  • rclone

    "rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Yandex Files

    • Project mention: Supabase Storage: now supports the S3 protocol | dev.to | 2024-04-18

    rclone: a command-line program to manage files on cloud storage.

  • migrate

    Database migrations. CLI and Golang library.

    • Project mention: Using migrations with Golang | dev.to | 2024-04-17

    Go does not natively support the use of migrations, but we could use the ORM that has this functionality, such as GORM which is the most used by the community, but We can use migrations without using an ORM, for this we will use the golang-migrate package.

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo

  • thanos

    Highly available Prometheus setup with long term storage capabilities. A CNCF Incubating project.

    • Project mention: Looking for a way to remote in to K's of raspberry pi's... | /r/sysadmin | 2023-12-10

    Monitoring = netdata on each RPi https://www.netdata.cloud/ binded to the vpn interface being scraped into a prometeus thaons https://thanos.io/ setup with grafana to give management the Green all is good screens (very important).

  • functions-samples

    Collection of sample apps showcasing popular use cases using Cloud Functions for Firebase

    • Project mention: I just realized how expensive Firebase is for Social Media Apps | /r/Firebase | 2023-06-20

    I tried to make a reddit like app. I used both realtime-database and firestore as database. The billing of the two is different from each other. I used realtime-database for frequently updated data (like or upvote, downvote count for ex.) and firestore for more stable and large data (post, comment, community and user data..). While doing this, I only used database rules, I did not use Cloud functions. So, I suggest you use a hybrid database to minimize your bill as much as possible and you can do amazing things with database rules. Have a good works.

  • SFTPGo

    Fully featured and highly configurable SFTP server with optional HTTP/S, FTP/S and WebDAV support - S3, Google Cloud Storage, Azure Blob

    • Project mention: What you guys are hosting instead of Nextcloud? I'm sick of it. | /r/selfhosted | 2023-11-29

    EDIT: Thanks for the recommendations from all of you!! I've chose to use the below: - Files: sftpgo - Calendar: baikal - Notes: memos (But beware, it sends opt-out telemetry) - Network folder: webdav on sftpgo

  • kopia

    Cross-platform backup tool for Windows, macOS & Linux with fast, incremental backups, client-side end-to-end encryption, compression and data deduplication. CLI and GUI included.

    • Project mention: DwarFS – The Deduplicating Warp-Speed Advanced Read-Only File System | news.ycombinator.com | 2024-04-11

    I think Kopia would be great for your use case

    https://kopia.io/

    It has a great system to snapshot files but only store data if it's changed. I use it in an environment where I can't use something like zfs to snapshot data because I don't have the ability to make decisions about what filesystem we're using. It's been amazing, love it so much!

  • goofys

    a high-performance, POSIX-ish Amazon S3 file system written in Go

    • Project mention: Is Posix Outdated? | news.ycombinator.com | 2023-10-19

    The author needs to ask themselves: in this cloud technology stack, is there POSIX involved somewhere lower down, where I can't access it? The answer is, of course, "yes". The sort of cloud storage systems described all run on top of POSIX APIs. They provide convenience (cost efficiency is more debatable) compared to the POSIX alternative, but that's because they exist at an entirely different conceptual layer (hence the presence of POSIX anyway, just buried).

    Your point about surfacing a POSIX that's actually there but hidden and thus visible to low-level Amazon employees building the S3 service which makes it invisible to S3 end customers is true but isn't the the point of the article. The author is saying there are motivations for a POSIX-like api visible also the end user.

    So your explanation of stack looks like 2 layers: POSIX api <-- AWS S3 built on top of that

    Author's essay is actually talking about 3 layers: POSIX <-- AWS S3 <-- POSIX

    That's why the blog post has the following links to POSIX-on-top-of-S3-objects :

    https://github.com/s3fs-fuse/s3fs-fuse

    https://github.com/kahing/goofys

    https://www.cuno.io/

  • WorkOS

    The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

    WorkOS logo

  • lakeFS

    lakeFS - Data version control for your data lake | Git for data

    • Project mention: A Step-by-Step Guide to Implementing Data Version Control | dev.to | 2023-09-04

    # Download the LakeFS binary wget https://github.com/treeverse/lakeFS/releases/latest/download/lakefs # Make the binary executable chmod +x lakefs # Initialize LakeFS with S3 as the storage backend ./lakefs init --backend s3 --s3-gateway-endpoint --s3-region --s3-force-path-style --s3-access-key --s3-secret-key

  • wal-e

    Continuous Archiving for Postgres

    • Project mention: Run PostgreSQL. The Kubernetes Way | news.ycombinator.com | 2023-09-22

    See the GitHub: https://github.com/wal-e/wal-e

    Unmaintained would’ve made more sense to say, but the maintainer choose the words “obsolete” so I took those. :)

    Seems to be obsolete due to a lack of interest and contributions.

  • s3proxy

    Access other storage backends via the S3 API

    • Project mention: Ask HN: AWS S3, Cloudflare R2, GCS, Wasabi, or B2? | news.ycombinator.com | 2024-04-01

  • berglas

    A tool for managing secrets on Google Cloud

    • Project mention: How to deploy a Django app to Google Cloud Run using Terraform | dev.to | 2024-01-01

    Secret Manager: secure storage for sensitive data e.g passwords.

  • DataflowTemplates

    Cloud Dataflow Google-provided templates for solving in-Cloud data tasks

  • fake-gcs-server

    Google Cloud Storage emulator & testing library.

    • Project mention: Show HN: Winglang – a new Cloud-Oriented programming language | news.ycombinator.com | 2023-12-06

    > not really.. as soon as you need cloud resources that are not in your k8s cluster, you end up with the cloud in your development loop.

    https://github.com/localstack/awscli-local

    https://github.com/fsouza/fake-gcs-server

    https://cloud.google.com/sdk/docs/downloads-docker

    I’ve used all of these locally to great success.

    Glad to hear you guys want to interop with npm. Is wing going to be a superset of js then?

  • tus-node-server

    Node.js tus server, standalone or integrable in any framework, with disk, S3, and GGC stores.

    • Project mention: Supabase Storage: now supports the S3 protocol | dev.to | 2024-04-18

    Supabase contributed some advanced features from the Node implementation of TUS Spec including distributed locks, max file size, expiration extension and numerous bug fixes:

  • google-cloud-cpp

    C++ Client Libraries for Google Cloud Services

    • Project mention: Experience Continuous Integration with Jenkins | Ansible | Artifactory | SonarQube | PHP | dev.to | 2024-02-24

    Try to utilize your AWS free tier as much as you can, you can also register a new account if you have exhausted the current one. Alternatively, you can use Google Cloud (GCP) to rent virtual machines from this cloud service provider - you can get $300 credit here or here (NOTE: Please read instructions carefully to get your credits)

  • PHP-FFmpeg-video-streaming

    📼 Package media content for online streaming(DASH and HLS) using FFmpeg

  • Firebase-ESP-Client

    🔥Firebase Arduino Client Library for ESP8266, ESP32 and RP2040 Pico. The complete, fast, secured and reliable Firebase Arduino client library that supports RTDB, Cloud Firestore, Firebase and Google Cloud Storage, Cloud Messaging and Cloud Functions for Firebase.

    • Project mention: Can someone recommend a library for sending data to Google Firebase's Realtime Database using Adafruit Feather M0 WiFi? | /r/adafruit | 2023-06-29

    I'm using Firebase-ESP-Client but there is a problem like in the attached video.

  • emacs-easy-hugo

    Emacs major mode for managing hugo

    • Project mention: emacs-easy-hugo: Emacs major mode for managing hugo | /r/planetemacs | 2023-10-04

  • helm-gcs

    Manage Helm 3 repositories on Google Cloud Storage 🔐 **privately**

  • csi-gcs

    Kubernetes CSI driver for Google Cloud Storage

    • Project mention: Google Cloud Storage FUSE | news.ycombinator.com | 2023-05-02

    You're right their Apache licenses are different:

    https://github.com/ofek/csi-gcs/blob/master/LICENSE-APACHE

    https://github.com/GoogleCloudPlatform/gcs-fuse-csi-driver/b...

    OP should submit a PR to correct this. IANAL but pretty sure they're supposed to use the original copy including copyright notice "Copyright 2020 Ofek Lev"

  • gcp-storage-emulator

    Local emulator for Google Cloud Storage

  • Google-Cloud-Guide

    Google Cloud Platform (GCP) Guide. Learn all about Google Cloud Tools, Services, and Certifications.

  • googleCloudStorageR

    Google Cloud Storage API to R

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    SaaSHub logo
NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).

google-cloud-storage related posts

Index

What are some of the best open-source google-cloud-storage projects? This list will help you:

Project Stars
1 rclone 43,720
2 migrate 13,946
3 thanos 12,577
4 functions-samples 11,972
5 SFTPGo 8,117
6 kopia 6,270
7 goofys 5,031
8 lakeFS 4,058
9 wal-e 3,423
10 s3proxy 1,595
11 berglas 1,224
12 DataflowTemplates 1,089
13 fake-gcs-server 959
14 tus-node-server 723
15 google-cloud-cpp 509
16 PHP-FFmpeg-video-streaming 504
17 Firebase-ESP-Client 473
18 emacs-easy-hugo 341
19 helm-gcs 277
20 csi-gcs 152
21 gcp-storage-emulator 131
22 Google-Cloud-Guide 124
23 googleCloudStorageR 103

Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com