thinkdeep VS ApacheKafka

A complex codebase needs to be kept as simple as possible. In Predecos, one way this is done is through use of full-stack JavaScript, meaning developers only need experience with one language to maintain the codebase. Some differences exist in NodeJS when compared with browser-based JavaScript but those are minor compared with having to work in multiple languages. Front-end and back-end dependencies are also kept as uniform as possible. For example, Mocha and Chai are used in both the front-end and back-end meaning experience with said tools enables development in either. Developer setup and deployment should also be straightforward. Predecos uses Helm v3.8.2 to trigger one-click deployment of all the microservices. After doing this and setting a couple environment variables, one can easily start the front-end by using yarn run start then they are ready for development. This is also valuable when doing a deployment. When CircleCI went down, the impact was slight because automated deployment and manual deployment were very similar. All that was required was modifying the CircleCI config for the terminal, building the necessary docker images manually, pushing those images and running helm upgrade. A simple and fast mitigation that saved a lot of time. Diverging from simplicity was sometimes necessary though. Some may argue that use of Kubernetes makes the codebase difficult to understand by virtue of its high learning curve. However, the stability, reliability, documentation and wide-spread adoption of Kubernetes made it hard to resist. Google Cloud Platform (GCP), Amazon Web Services (AWS), Microsoft Azure and DigitalOcean all have Kubernetes offerings. This means applications developed on Kubernetes are largely portable from one cloud provider to another. Predecos uses DigitalOcean’s managed Kubernetes product because it is low-cost when compared to the other options. Should the desire for features such as, for example, encryption at rest arise, one option would be to migrate to Google Kubernetes Engine (GKE). Some modifications would be necessary but because Predecos is in a proof-of-concept (PoC) state and minimal data has been stored that migration would likely be simple.

Senden an Kafka (d. h. Senden der Daten an Apache Kafka)

Stream-processing platforms such as Apache Kafka, Apache Pulsar, or Redpanda are specifically engineered to foster event-driven communication in a distributed system and they can be a great choice for developing loosely coupled applications. Stream processing platforms analyze data in motion, offering near-zero latency advantages. For example, consider an alert system for monitoring factory equipment. If a machine's temperature exceeds a certain threshold, a streaming platform can instantly trigger an alert and engineers do timely maintenance.

Apache Kafka is a distributed streaming platform capable of handling high throughput of data, while ReductStore is a databases for unstructured data optimized for storing and querying along time.

*Push data *(original source image, GPS, timestamp) in a common place (Apache Kafka,...)

RabbitMQ comes with administrative tools to manage user permissions and broker security and is perfect for low latency message delivery and complex routing. In comparison, Apache Kafka architecture provides secure event streams with Transport Layer Security(TLS) and is best suited for big data use cases requiring the best throughput.

Apache Kafka is a distributed streaming platform to share data between applications and services in real-time.

also, this knowledge applies to learning more about data engineering, as this field of software engineering relies heavily on the event-driven approach via tools like Spark, Flink, Kafka, etc.