Light Service VS pathway

check out the `light-service` gem - it lets you create services that use this pattern and compose them into complex flows: https://github.com/adomokos/light-service

Similar to the above recommendations of Railway Orientated Programming and the Interactor gem, I use the Light Service gem. While it can sometimes be a bit verbose it excels in code reuse, testability, and composition.

View on GitHub

LightService

You can implement the service pattern as a PORO, but it helps to use one of the gems available to aid in code structure, orchestration, and error handling. The LightService gem provides a nice implementation without adding hardly any overhead. It’s design is very simple, as shown in the diagram below. An organizer is used to define the sequential workflow, which is made up of one or more actions. An error in any of the actions will short-circuit the entire workflow.

We understood how streaming databases differ from traditional databases, stream processing engines, conventional analytics databases, or OLAP databases. Now let’s focus on when and why we can use stream data processing frameworks for Python as an alternative to streaming databases. Python is the go-to language for data science and machine learning. There are some stream-processing libraries and frameworks in Python such as Bytewax, Quix, GlassFlow, Pathway. They have been developed to cope with the challenges Python Engineers face with Apache Kafka or Flink since they do not natively support Python.

Maintain a data snapshot to observe variations in sales prices over time, as Pathway provides a built-in feature to compute differences between two alterations.

Pathway (https://github.com/pathwaycom/pathway) is a data processing framework we are developing that unifies stream and batch processing of large datasets. It lets developers concentrate on writing the data processing logic, without worrying about tracking changes to data and updating the results. The same code can then be run on batch data (e.g. during testing) or on real-time data streams (i.e. online query processing)

In the LLM app, Pathway allows concentrating on prompt building and querying the LLM APIs as if the corpus of documents were static, while all updates to it are handled by the framework itself.