Querying Parquet with Millisecond Latency

Scribd and Slideshare have been using AWS S3 for almost twenty years and store hundreds of billions of objects making storage management quite a challenge. My focus at Scribd has generally been around data and storage but only in the past twelve months have I started to really focus on one of our hardest technology problems: cost-effective storage and availability for the hundreds of billions of objects that represent our content library.

The FDAP stack, which consists of Apache Flight, DataFusion, Arrow, and Parquet, finally permits developers to build new systems without reinventing the wheel, resulting in more features and better performance than legacy designs.

Over the past week I have made a more conscious effort to keep track of some really interesting articles that came through my feed reader. I am a big fan of the open web and the power of RSS for disseminating interesting information from actual people. Below are some really interesting posts I have read recently!