vector
opensearch
Our great sponsors
vector | opensearch | |
---|---|---|
96 | 9 | |
16,427 | 715 | |
4.8% | - | |
9.9 | 0.0 | |
about 3 hours ago | 8 months ago | |
Rust | Python | |
Mozilla Public License 2.0 | Creative Commons Attribution Share Alike 4.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
vector
-
Docker Log Observability: Analyzing Container Logs in HashiCorp Nomad with Vector, Loki, and Grafana
job "vector" { datacenters = ["dc1"] # system job, runs on all nodes type = "system" group "vector" { count = 1 network { port "api" { to = 8686 } } ephemeral_disk { size = 500 sticky = true } task "vector" { driver = "docker" config { image = "timberio/vector:0.30.0-debian" ports = ["api"] volumes = ["/var/run/docker.sock:/var/run/docker.sock"] } env { VECTOR_CONFIG = "local/vector.toml" VECTOR_REQUIRE_HEALTHY = "false" } resources { cpu = 100 # 100 MHz memory = 100 # 100MB } # template with Vector's configuration template { destination = "local/vector.toml" change_mode = "signal" change_signal = "SIGHUP" # overriding the delimiters to [[ ]] to avoid conflicts with Vector's native templating, which also uses {{ }} left_delimiter = "[[" right_delimiter = "]]" data=<
- FLaNK AI Weekly 18 March 2024
- Vector: A high-performance observability data pipeline
-
Hacks to reduce cloud spend
we are doing something similar with OTEL but we are looking at using https://vector.dev/
-
About reading logs
We don't pull logs, we forward logs to a centralized logging service.
-
Self hosted log paraer
opensearch - amazon fork of Elasticsearch https://opensearch.org/docs/latestif you do this an have distributed log sources you'd use logstash for, bin off logstash and use vector (https://vector.dev/) its better out of the box for SaaS stuff.
-
creating a centralize syslog server with elastic search
I have done something similar in the past: you can send the logs through a centralized syslog servers (I suggest syslog-ng) and from there ingest into ELK. For parsing I am advice to use something like Vector, is a lot more faster than logstash. When you have your logs ingested correctly, you can create your own dashboard in Kibana. If this fit your requirements, no need to install nginx (unless you want to use as reverse proxy for Kibana), php and mysql.
-
Show HN: Homelab Monitoring Setup with Grafana
I think there's nothing currently that combines both logging and metrics into one easy package and visualizes it, but it's also something I would love to have.
Vector[1] would work as the agent, being able to collect both logs and metrics. But the issue would then be storing it. I'm assuming the Elastic Stack might now be able to do both, but it's just to heavy to deal with in a small setup.
A couple of months ago I took a brief look at that when setting up logging for my own homelab (https://pv.wtf/posts/logging-and-the-homelab). Mostly looking at the memory usage to fit it on my synology. Quickwit[2] and Log-Store[3] both come with built in web interfaces that reduce the need for grafana, but neither of them do metrics.
- [1] https://vector.dev
-
Retaining Logs generated by service running in pod.
Log to stdout/stderr and collect your logs with a tool like vector (vector.dev) and send it to something like Grafana Loki.
-
Lightweight logging on RPi?
I would recommend that you run vector as a systems service so you don't have to worry about managing it. Here is a basic config to do that - https://github.com/vectordotdev/vector/blob/master/distribution/systemd/vector.service .
opensearch
-
Firefox Address Bar Tips
Or just right-click the input field, and if the browser recognizes it as a search field (they're good at it by default, but you can implement https://github.com/dewitt/opensearch to make extra sure), you'll get an option to create a search from it, with a keyword of your choosing (haven't tried Safari).
- Looking for Search API/DSL Design Resources
-
Is there a place with a bunch of free API's I can use to practice making CRUD applications?
https://github.com/public-apis/public-apis https://data.gov.sg/developer http://opendatacommunities.org/ https://www.data.gov/developers/apis https://data.sfgov.org/browse?limitTo=datasets&utf8=? https://mashupaustralia.org/open-access-to-psi/ https://catalogue.data.govt.nz/dataset https://rapidapi.com/ https://www.programmableweb.com/apis https://rapidapi.com/collection/list-of-free-apis https://any-api.com/ https://compassioninpolitics.wordpress.com/2009/06/03/best-free-apis-for-web-developers/ https://rapidapi.com/pozzad/api/email-validator-1 https://public-apis.xyz/ https://developers.facebook.com/ https://dev.twitter.com https://www.programmableweb.com/news/5-weather-apis-weatherbug-to-weather-channel/2009/04/15 https://duckduckgo.com/api https://openweathermap.org/api https://github.com/dewitt/opensearch http://www.atomenabled.org/ https://openid.net/developers/ http://developer.foodessentials.com/ https://postcoder.com/docs/address-lookup https://github.com/public-apis/public-apis https://www.data.gov/developers/apis https://rapidapi.com/ https://www.programmableweb.com/apis https://rapidapi.com/collection/list-of-free-apis https://any-api.com/ https://compassioninpolitics.wordpress.com/2009/06/03/best-free-apis-for-web-developers/ https://rapidapi.com/pozzad/api/email-validator-1 https://public-apis.xyz/ https://www.programmableweb.com/news/5-weather-apis-weatherbug-to-weather-channel/2009/04/15 https://duckduckgo.com/api https://openweathermap.org/api https://postcoder.com/docs/address-lookup https://postcoder.com/docs/address-lookup http://vendapin.com/cecb46.html https://my-json-server.typicode.com/ https://jsonplaceholder.typicode.com/ https://reqres.in/ https://my-json-server.typicode.com/ https://randomapi.com/pricing https://randomuser.me/ https://reqres.in/ https://randomapi.com/pricing http://www.mindit-bookmarking.com/ https://theroadtodelphi.com/2010/08/07/using-the-google-maps-api-v3-from-delphi-part-i-basic-functionality/ https://theroadtodelphi.com/2010/08/07/using-the-google-maps-api-v3-from-delphi-part-i-basic-functionality/ https://www.geoapify.com/map-marker-icons-generator-create-beautiful-icons-for-your-map https://elevation-api.io/ https://open-elevation.com/ https://www.thoughtco.com/save-a-web-page-as-html-or-mht-1058361 https://theroadtodelphi.com/2010/08/07/using-the-google-maps-api-v3-from-delphi-part-i-basic-functionality/ https://www.programmableweb.com/apis/directory https://www.sitepoint.com/10-html5-apis-worth-looking/ http://www.99lime.com/elements/ https://docs.angularjs.org/api https://docs.angularjs.org/api/ng/service/$http https://docs.angularjs.org/api/ngMock/service/$httpBackend https://developers.google.com/earth/ http://www.worldwindcentral.com/wiki/Google_Earth_comparison https://theroadtodelphi.com/2010/08/07/using-the-google-maps-api-v3-from-delphi-part-i-basic-functionality/ https://jasontpenny.com/blog/2009/01/11/google-maps-in-a-twebbrowser-from-delphi-directions/ https://geochalkboard.wordpress.com/2009/03/11/density-mapping-in-google-maps-with-heatmapapi/ https://www.ixxat.com/products/products-industrial/protocol-sw-and-apis https://docs.microsoft.com/en-us/dotnet/api/system.componentmodel.backgroundworker?redirectedfrom=MSDN&view=netframework-4.7.2 http://custforum.axis.com/viewtopic.php?t=1963&sid=7990ea009d57544606efd627f3a40541 https://stackoverflow.com/questions/5137717/seeking-simple-coding-example-for-axis-webcam-using-vapix
-
Show HN: Search Engine for Blogs
I still quite like the idea of having a number of independent search engines each indexing their own specialist subjects, and one or more federated search front-ends which can pull these together.
Doing it with APIs is a little tricky to make work in a usable way though. There have been various attempts at standardised APIs, e.g. OpenSearch[0], and metasearch engines like searX[1] have what are essentially pluggable scrapers, but there are still fundamental issues like getting different results at different times and having different ranking mechanisms.
Integrating at the index level could make a more usable search, but there are lots of other issues with this approach, e.g. those with Apache Solr's Cross Data Centre Replication[2]. And yes, the volumes of data may also be an issue, given a search index will typically be slightly larger than the compressed data size, e.g. the 16M wikipedia docs are approx 32Gb compressed and approx 40.75Gb in a search index.
[0] https://github.com/dewitt/opensearch , unrelated to Amazon's Elasticsearch fork
[1] https://github.com/searx/searx
[2] https://solr.apache.org/guide/8_11/cross-data-center-replica...
-
How to implement a search shortcut (OpenSearch) on any website
GitHub
-
Opensearch 1.0.0 Announced
There is already an open source project called OpenSearch which used to be located on the same domain. The original project is a specification for how to share search results with a common API.
- AWS released OpenSearch, a community-driven, open source fork of Elasticsearch and Kibana
- AWS releases forked Elasticsearch code. Announces new name: OpenSearc
What are some alternatives?
graylog - Free and open log management
sonic - 🦔 Fast, lightweight & schema-less search backend. An alternative to Elasticsearch that runs on a few MBs of RAM.
Fluentd - Fluentd: Unified Logging Layer (project under CNCF)
agent - Vendor-neutral programmable observability pipelines.
OpenSearch - 🔎 Open source distributed and RESTful search engine.
syslog-ng - syslog-ng is an enhanced log daemon, supporting a wide range of input and output methods: syslog, unstructured text, queueing, SQL & NoSQL.
OpenSearch-Dashboards - 📊 Open source visualization dashboards for OpenSearch.
fake-store-api - FakeStoreAPI is a free online REST API that provides you fake e-commerce JSON data
tracing - Application level tracing for Rust.
Typesense - Open Source alternative to Algolia + Pinecone and an Easier-to-Use alternative to ElasticSearch ⚡ 🔍 ✨ Fast, typo tolerant, in-memory fuzzy Search Engine for building delightful search experiences