On June 24, 2024, the first San Francisco Bay Area DataFusion meetup happened. I had the opportunity to help with the organization of the event and also attend. The event had a lot of content from six different companies. These companies ranged from startups to scale-ups and big Fortune 500 companies. Leaving the event, I felt I had experienced something significant, and I want to share it with you.
Three weeks ago, VeloxCon took place in San Jose. The event was a great opportunity for people who are interested in execution engines and data processing at scale to learn about the current state of the project. Most importantly, though, it was an amazing opportunity to get a glimpse of what the future of data processing will be like. From what we saw at the event, this future is very exciting!
After a few years and with the hype gone, it has become apparent that MLOps overlap more with Data Engineering than most people believed. Let’s see why and what that means for the MLOps ecosystem. MLOps is a relatively recent term. A quick search on Google Trends reveals that the term started being searched for, around the end of 2019.
DuckDB provides 14 SQL window-related functions in addition to all the aggregation functions that can be combined with windows. Snowflake, on the other hand, offers more than 70 functions that can be used with SQL windows. PostgreSQL also supports 11 SQL window-related functions, as well as all the aggregation functions that are packaged by default, in addition to any user-provided aggregation function. Hopefully, the above information has captured your attention and helped you realize how important SQL windows are, based on the effort database vendors are making to add support for them.
> TL;DR: We have an API Economy today, but it's not exactly what people were expecting 10 years ago when it started. The good news is we're starting a new phase of innovation that will change things up. Let's talk about why and how. At the start of the 2010s, the Cloud was transforming everything. SaaS was everywhere. CRMs like Salesforce, Ticketing Systems like Zendesk, and e-commerce solutions like Shopify were all transforming every aspect of the economy.
Ok so we talked on how SQL deals with Dates and there's a rich arsenal of tools to help us do whatever we want with time information in SQL. But, working with dates is still a pain.
> TL;DR: Working with Time in SQL is one of the most common tasks that people are seeking help for. In this article I will cover the most common ways to work with time in SQL. !mumf in time # SQL Timestamp, Date and Time Data Types The SQL standard defines the datetime datatype which can be further specified using the following descriptors:
!bunny # Introduction In this post I want to take you through a journey that starts with Github Issue #1284 requesting support for nanosecond/microsecond precision in TIMESTAMP for Trino and ends with the merge of Github PR #3783 that added support for the parametric TIMESTAMP type to the Trino query engine. This journey includes a number of surprises too!
Snowflake is investing heavily in improving the experience of building streaming pipelines. Let’s see how you can work with streaming data today on snowflake and what are some of the upcoming new features that are either rumored or announced already.
Snowflake pricing is a bit complicated, this is a result of *usage based pricing* together with their evolution into a *serverless database*. Let’s start with the fundamentals. The pricing landing page is an excellent starting point to the Snowflake pricing model and to its complexity. Pricing is affected by the following factors: