Parsing logs from multiple data sources with Ahana and Cube

This page summarizes the projects mentioned and recommended in the original post on dev.to

Our great sponsors
  • InfluxDB - Power Real-Time Data Analytics at Scale
  • WorkOS - The modern identity platform for B2B SaaS
  • SaaSHub - Software Alternatives and Reviews
  • cube.js

    📊 Cube — The Semantic Layer for Building Data Applications

  • const measureNames = [ 'perc_10', 'perc_10_value', 'perc_20', 'perc_20_value', 'perc_30', 'perc_30_value', 'perc_40', 'perc_40_value', 'perc_50', 'perc_50_value', 'perc_60', 'perc_60_value', 'perc_70', 'perc_70_value', 'perc_80', 'perc_80_value', 'perc_90', 'perc_90_value', 'perc_100', 'perc_100_value', ]; const measures = Object.keys(measureNames).reduce((result, name) => { const sqlName = measureNames[name]; return { ...result, [sqlName]: { sql: () => sqlName, type: `max` } }; }, {}); cube('errorpercentiles', { sql: `with sagemaker as ( select model_name, variant_name, cast(json_extract(FROM_UTF8( from_base64(capturedata.endpointinput.data)), '$.correlation_id') as varchar) as correlation_id, cast(json_extract(FROM_UTF8( from_base64(capturedata.endpointoutput.data)), '$.prediction') as double) as prediction from s3.sagemaker_logs.logs ) , actual as ( select correlation_id, actual_value from postgresql.public.actual_values ) , logs as ( select model_name, variant_name as model_variant, sagemaker.correlation_id, prediction, actual_value as actual from sagemaker left outer join actual on sagemaker.correlation_id = actual.correlation_id ) , errors as ( select abs(prediction - actual) as abs_err, model_name, model_variant from logs ), percentiles as ( select approx_percentile(abs_err, 0.1) as perc_10, approx_percentile(abs_err, 0.2) as perc_20, approx_percentile(abs_err, 0.3) as perc_30, approx_percentile(abs_err, 0.4) as perc_40, approx_percentile(abs_err, 0.5) as perc_50, approx_percentile(abs_err, 0.6) as perc_60, approx_percentile(abs_err, 0.7) as perc_70, approx_percentile(abs_err, 0.8) as perc_80, approx_percentile(abs_err, 0.9) as perc_90, approx_percentile(abs_err, 1.0) as perc_100, model_name, model_variant from errors group by model_name, model_variant ) SELECT count(*) FILTER (WHERE e.abs_err <= perc_10) AS perc_10 , max(perc_10) as perc_10_value , count(*) FILTER (WHERE e.abs_err > perc_10 and e.abs_err <= perc_20) AS perc_20 , max(perc_20) as perc_20_value , count(*) FILTER (WHERE e.abs_err > perc_20 and e.abs_err <= perc_30) AS perc_30 , max(perc_30) as perc_30_value , count(*) FILTER (WHERE e.abs_err > perc_30 and e.abs_err <= perc_40) AS perc_40 , max(perc_40) as perc_40_value , count(*) FILTER (WHERE e.abs_err > perc_40 and e.abs_err <= perc_50) AS perc_50 , max(perc_50) as perc_50_value , count(*) FILTER (WHERE e.abs_err > perc_50 and e.abs_err <= perc_60) AS perc_60 , max(perc_60) as perc_60_value , count(*) FILTER (WHERE e.abs_err > perc_60 and e.abs_err <= perc_70) AS perc_70 , max(perc_70) as perc_70_value , count(*) FILTER (WHERE e.abs_err > perc_70 and e.abs_err <= perc_80) AS perc_80 , max(perc_80) as perc_80_value , count(*) FILTER (WHERE e.abs_err > perc_80 and e.abs_err <= perc_90) AS perc_90 , max(perc_90) as perc_90_value , count(*) FILTER (WHERE e.abs_err > perc_90 and e.abs_err <= perc_100) AS perc_100 , max(perc_100) as perc_100_value , p.model_name, p.model_variant FROM percentiles p, errors e group by p.model_name, p.model_variant`, preAggregations: { // Pre-Aggregations definitions go here // Learn more here: https://cube.dev/docs/caching/pre-aggregations/getting-started }, joins: { }, measures: measures, dimensions: { modelVariant: { sql: `model_variant`, type: 'string' }, modelName: { sql: `model_name`, type: 'string' }, } });

  • vue-chartjs

    📊 Vue.js wrapper for Chart.js

  • In the ChartView component, we use the vue-chartjs library to display the charts. Our setup script contains the required imports and registers the Chart.js components:

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo
NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts