elasticsearch-py
vector
elasticsearch-py | vector | |
---|---|---|
21 | 97 | |
4,143 | 16,561 | |
0.4% | 1.8% | |
8.9 | 9.9 | |
6 days ago | 7 days ago | |
Python | Rust | |
Apache License 2.0 | Mozilla Public License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
elasticsearch-py
- Verify Connection to Elasticsearch (2021)
- An alternative to Elasticsearch that runs on a few MBs of RAM
- Help With Psort.py -> ELK
- Elastic Open Sources Their Endpoint Security Protection YARA Ruleset
-
OpenSearch – open-source search and analytics based on Apache 2.0 Elasticsearch
FD: I have a friend who works at Elastic, though he doesn't really colour my opinions of things.
> Firstly, dick moves like this: https://github.com/elastic/elasticsearch-py/pull/1623
I understand that this is unpopular, but you can make a very strong argument that it's to prevent weird errors in the future. I'm also guilty of littering my code with Asserts to ensure the universe is working fine.
The alternative is to allow it to work and then you end up with weird issues like when you connect mysql client to mariadb server (and vice-versa): https://stackoverflow.com/questions/50169576/mysql-8-0-11-er...
> Secondly, I don't buy the argument from Elastic any more. Yes, the ethical thing to do when you're making money from someone's work is at least contribute back. At the same time though, they're making money from packaging it up and selling it _as a service_. That "as a service" part is where they're making the bucks.
That's just an opinion, yes they have a service, and yes it competes with Amazon. Is it cool for Amazon to take a body of work and sell it without supporting it? Are amazon actually supporting it? Is it the same as Elastic using Lucene? (not really because Elastic submits a the majority of fixes to Lucene, but, you get it).
it's kinda gray, I'm sure Amazon thinks they're the good guy, but it's hard for me to look at Elastic as the bad guy in all this.
- Struggling reading code with type hints
-
I Don't Think Elasticsearch Is a Good Logging System
Oh man, https://github.com/elastic/elasticsearch-py/issues/1734 is a disappointing read. I know ES wants to save their business, but alienating users isn't exactly the path to success.
- Elasticsearch adding code to reject connections to OpenSearch clusters or to clusters running open source distributions of ES7
- Official Elasticsearch Python library no longer works with open-source forks
vector
-
What is a low/reasonable cost solution for service log storage and querying?
I am thinking about using https://vector.dev/ but would also love opinions on the best deal for lower or reasonable cost storage/querying of logs. Thanks!
-
Docker Log Observability: Analyzing Container Logs in HashiCorp Nomad with Vector, Loki, and Grafana
job "vector" { datacenters = ["dc1"] # system job, runs on all nodes type = "system" group "vector" { count = 1 network { port "api" { to = 8686 } } ephemeral_disk { size = 500 sticky = true } task "vector" { driver = "docker" config { image = "timberio/vector:0.30.0-debian" ports = ["api"] volumes = ["/var/run/docker.sock:/var/run/docker.sock"] } env { VECTOR_CONFIG = "local/vector.toml" VECTOR_REQUIRE_HEALTHY = "false" } resources { cpu = 100 # 100 MHz memory = 100 # 100MB } # template with Vector's configuration template { destination = "local/vector.toml" change_mode = "signal" change_signal = "SIGHUP" # overriding the delimiters to [[ ]] to avoid conflicts with Vector's native templating, which also uses {{ }} left_delimiter = "[[" right_delimiter = "]]" data=<
- FLaNK AI Weekly 18 March 2024
- Vector: A high-performance observability data pipeline
-
Hacks to reduce cloud spend
we are doing something similar with OTEL but we are looking at using https://vector.dev/
-
About reading logs
We don't pull logs, we forward logs to a centralized logging service.
-
Self hosted log paraer
opensearch - amazon fork of Elasticsearch https://opensearch.org/docs/latestif you do this an have distributed log sources you'd use logstash for, bin off logstash and use vector (https://vector.dev/) its better out of the box for SaaS stuff.
-
creating a centralize syslog server with elastic search
I have done something similar in the past: you can send the logs through a centralized syslog servers (I suggest syslog-ng) and from there ingest into ELK. For parsing I am advice to use something like Vector, is a lot more faster than logstash. When you have your logs ingested correctly, you can create your own dashboard in Kibana. If this fit your requirements, no need to install nginx (unless you want to use as reverse proxy for Kibana), php and mysql.
-
Show HN: Homelab Monitoring Setup with Grafana
I think there's nothing currently that combines both logging and metrics into one easy package and visualizes it, but it's also something I would love to have.
Vector[1] would work as the agent, being able to collect both logs and metrics. But the issue would then be storing it. I'm assuming the Elastic Stack might now be able to do both, but it's just to heavy to deal with in a small setup.
A couple of months ago I took a brief look at that when setting up logging for my own homelab (https://pv.wtf/posts/logging-and-the-homelab). Mostly looking at the memory usage to fit it on my synology. Quickwit[2] and Log-Store[3] both come with built in web interfaces that reduce the need for grafana, but neither of them do metrics.
- [1] https://vector.dev
-
Retaining Logs generated by service running in pod.
Log to stdout/stderr and collect your logs with a tool like vector (vector.dev) and send it to something like Grafana Loki.
What are some alternatives?
searxng - SearXNG is a free internet metasearch engine which aggregates results from various search services and databases. Users are neither tracked nor profiled.
graylog - Free and open log management
quickwit - Cloud-native search engine for observability. An open-source alternative to Datadog, Elasticsearch, Loki, and Tempo.
Fluentd - Fluentd: Unified Logging Layer (project under CNCF)
helm-charts
agent - Vendor-neutral programmable observability pipelines.
orama - 🌌 Fast, dependency-free, full-text and vector search engine with typo tolerance, filters, facets, stemming, and more. Works with any JavaScript runtime, browser, server, service!
syslog-ng - syslog-ng is an enhanced log daemon, supporting a wide range of input and output methods: syslog, unstructured text, queueing, SQL & NoSQL.
qryn - qryn is a polyglot, high-performance observability framework for ClickHouse. Ingest, store and analyze logs, metrics and telemetry traces from any agent supporting Loki, Prometheus, OTLP, Tempo, Elastic, InfluxDB and many more formats and query transparently using Grafana or any other compatible client.
OpenSearch - 🔎 Open source distributed and RESTful search engine.
evtx2es - A library for fast parse & import of Windows Eventlogs into Elasticsearch.
tracing - Application level tracing for Rust.