Our great sponsors
-
-
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
If instead of removing the data you might want to do some in depth analysis on the message structure you could also write a series of sanitizer rules as well (https://github.com/fluent/fluent-plugin-sanitizer ).
Lastly, I know the question was around Fluentd - however if you are looking for super high performance I also recommend taking a look at Fluent Bit (https://fluentbit.io) it's more performant sub-project. Simliar functionality with grep (https://docs.fluentbit.io/manual/pipeline/filters/grep) exists, and there are examples of using the in-built LUA scripting for more processing type filtering (https://github.com/couchbase/couchbase-fluent-bit#redaction)
Related posts
- Fluentbit Kubernetes - How to extract fields from existing logs
- Parsing multiline logs using a custom Fluent Bit configuration
- Preferred Linux Distribution for production and why?
- How can I conditionally parse logs with regeX to assign “codes” to events?
- Is it possible to change the PAGESIZE/Kernel to RHCK of OKE image?