configu VS parquet-format

As we celebrate Hacktoberfest, we at Configu invite you to be part of our mission to redefine software configuration management. We've set out to tackle the persistent challenge of configuration chaos, and we're making strides every day. If you're searching for a place to make a significant impact this Hacktoberfest, consider Configu. Delve into our open-source repository, understand our vision, and contribute to shaping our journey. If you're unsure where to begin or need some help along the way, our Configu Discord community is always here to guide you. For newcomers, we recommend starting with issues labeled 'good-first-issues'.

Enter Configu, an open source implementation of the concept of Configuration-as-Code, ensuring that the code remains the source of truth. But we didn't stop there. We've just launched a new feature called configu find that takes configuration management to the next level.

Thanks Geva for sharing my article.

In this piece, I delve into the world of application configuration, discussing the drawbacks of using environment variables stored in env files and introducing the powerful alternative - Configuration as Code (CaC).

Looking forward to your thoughts! and If you're intrigued, I invite you to explore our open-source project on GitHub: https://github.com/configu/configu

View on GitHub

Right, there's all sorts of metadata and often stats included in any parquet file: https://github.com/apache/parquet-format#file-format

The offsets of said metadata are well-defined (i.e. in the footer) so for S3 / blob storage so long as you can efficiently request a range of bytes you can pull the metadata without having to read all the data.

As i read here https://github.com/apache/parquet-format/blob/master/LogicalTypes.md , they are store in Integer formats and these integers represent the number of days (for Date) or number of milliseconds, microseconds or nanoseconds (for DateTime) since 1970-01-01. This works as expected with the parquet file that written by our ETL tool from internal database --> S3, all Data/DateTime columns are Integers, means that in Glue Job, i have to convert these Integers back to Date/Datetime value to do some transformation on them. But when parquet files are written by Spark, they are Date/DateTime (or TimeStamp to be more concise) format not Integers (i checked by read these files again in other Glue Job) and that make me confused.

Date is confusing with a timezone (UTC or otherwise) and the doco makes no such suggestion.

The Parquet datatypes documentation is pretty clear that there is a flag isAdjustedToUTC to define if the timestamp should be interpreted as having Instant semantics or Local semantics.

https://github.com/apache/parquet-format/blob/master/Logical...

Still no option to include a TZ offset in the data (so the same datum can be interpreted with both Local and Instant semantics) but not bad really.