Our great sponsors
-
I think it is more about efficiency and not about limits. If you use JDBC/ODBC, it means your process is running outside of Snowflake, you need transfer data over network into your engine. If you use SparkSQL to read a JDBC source, it doesn't support streaming data - you need to load whole data into DataFrame before you process it. To stream you will need some custom workaround, for example that. If you work with Snowpark you don't need it (not sure how it works internally).
-
dbt-fal
Discontinued do more with dbt. dbt-fal helps you run Python alongside dbt, so you can send Slack alerts, detect anomalies and build machine learning models.
Happy data scientists use fal and dbt
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
Related posts
- dbt for ML Engineering
- Dbt-fal: a dbt Python adapter with local code execution
- I'm getting elasticsearch.BadRequestError: BadRequestError(400, 'illegal_argument_exception', "specified fields can't be null or empty") using Eland library
- Mastering Data Science: Top 10 GitHub Repos You Need to Know
- is it possible to use log data from elastic search and visualise it to a custom made dashboard in python?