What steps do you apply in the "L" when doing ELT?

• Scrapy

  180 50,824 9.6 Python

  Scrapy, a fast high-level web crawling & scraping framework for Python.

  

The sha256 is there establish the uniqueness of the file. It isn’t great for capturing whether or not you have already seen the file before, tho, because it is rather expensive to calculate (imagine your csv file were gigabytes on size — you would have to stream in whole file down in order to see if it had changed!). In the past I have used a sha256 of information that the server hosting the file gives me about the file — it’s last modification time, for example, and it’s file size. This is a surprisingly subtle question, though, so I would encourage you to ponder it deeply, as it is the hardest part of the ‘E’ step! You might consider seeing how a framework like scrapy (https://scrapy.org/) solves this particular thorny issue :)

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

Suggest a related project