Jack Vanlightly

Missed one aspect of writing the TLA+ spec yesterday. How to gain confidence it is correct. https://lnkd.in/dQn2VUUq

8

I just updated my first table format comparison blog post to fix some missing details of Apache Hudi. https://lnkd.in/dwSUS6qS

5

Confluent's Tableflow is getting closer and closer to GA. Materializing Kafka topics as Iceberg tables, but with a great deal of thought around schema conversion, schema evolution and such. It's amazing how much work goes into making a product robust to schema changes. After the initial announcement I wrote two articles explaining Tableflow and its role in the future of bringing operational and analytics planes together. * Tableflow: the stream/table, Kafka/Iceberg duality: https://lnkd.in/gZbQ2cZ8 * Hybrid Transactional/Analytical Storage: https://lnkd.in/dTg6DKVy

102

4 comentarios

Good energy here, really appreciated this meetup! 🤩 Look at the recording and recognize the common mistakes with Kafka (the audience definitely did!). How to handle invalid messages, manage schema evolution, and ensure data quality. It’s a tough task, whether you’re using CI/CD, a custom SDK, or a Kafka proxy. #meetup #apachekafka

5

In this demonstration, Tom Scott tackled a real-world scenario where a retailer's transaction data was producing erroneous results due to an unusually large exchange rate for the Zimbabwean Dollar. This anomaly caused transactions with this currency to be recorded with a zero value, effectively making them invalid and impacting financial reports and operations. This anomaly caused the transaction amounts to reduce to zero, disrupting operations and creating confusion. By leveraging #indexing, we swiftly identified the root cause of the problem—the incorrect exchange rate—allowing us to rectify the issue promptly. This example highlights how indexing significantly accelerates problem-solving, enabling teams to quickly diagnose and address data anomalies without the need for unnecessary data movement and the complexity of #ETL pipelines. #Kafka #DataStreaming #Indexing #DataAnalytics #BigData #RealTimeData #ETL #DataManagement #StreamProcessing

9

The final deep dive on change query support in the table formats is ready. This one covers Apache Paimon, focusing on understanding the change query support through the lens of its internals. https://lnkd.in/dFfM_8nA

28

The new version of #apachepinot 1.2 is out now, packed with exciting enhancements! Check out the release notes here: https://lnkd.in/eNNfaAqc A recent blog post from StarTree delves into some of the standout features. Read more about the innovation in Apache Pinot 1.2 here: https://lnkd.in/e5wHtKgy

27

Interesting post by João Salgado about partitioning vs. sharding in #Postgres which lists trade-offs for each. It's great looking at the list of cons for partitioning and seeing that #TimescaleDB actually helps with most of them - Complexity ⬇️ because we transparently manage partition creation and lifecycle for you as data is ingested - Optimization Challenges ⬇️ because we have planner hooks that help with handling data exclusion (you still do need to pay attention to your partitioning key though). - Storage Overhead ⬇️ (although Postgres doesn't really have much overhead for partitions) because you can just choose to compress your data (we often see a 95x reduction in size). Timescale 🚀

18

1 comentario

New Blog Alert: Maximize your fine-tuned model performance with the new Azure AI Evaluation SDK! In the journey of distilling large models like Meta Llama 3.1 405B into efficient smaller versions, evaluating the performance is crucial to ensure efficacy, accuracy, and readiness for deployment. 🔍 Learn how the Azure AI Evaluation SDK can: - Enhance your model assessment with advanced metrics - Provide insights with GPT-model based and math based evaluations - Enable bulk metric evaluation for comprehensive analysis Join me as we compare distilled models against baseline performance and explore valuable metrics that can drive continuous improvement. Whether you're in AI development or in search of efficient deployment strategies, this blog post is tailored for you. 👉 Read more: https://lnkd.in/gfQbNy3p #AzureAI #MachineLearning #AIEvaluation #AICommunity #ArtificialIntelligence #TechInnovation

30

I came across this article about "SQL as fast as NoSQL" today (as always great content Franck Pachot and Yugabyte). I know it's not the point of the article, but what took my eye was the table showing independent ingest benchmarking, with TimescaleDB topping the list 👀. I got old mate ChatGPT to work out the relative ingest rates as percentages 🔥: --- Here are the relative insert speeds as percentages compared to TimescaleDB #TimescaleDB: 100.00% (600,000 inserts/sec) #InfluxDB: 76.67% (460,000 inserts/sec) #PostgreSQL: 71.33% (428,000 inserts/sec) #Cassandra async inserts: 68.33% (410,000 inserts/sec) #Cassandra sync inserts: 64.83% (389,000 inserts/sec) #YugabyteDB YSQL: 49.17% (295,000 inserts/sec) #YugabyteDB YCQL: 48.00% (288,000 inserts/sec) #Elasticsearch: 28.33% (170,000 inserts/sec) #ArrangoDB: 22.83% (137,000 inserts/sec) #CockroachDB: 15.17% (91,000 inserts/sec)

57

7 comentarios

Your Datalake - in 2024 don't just consider the job done with Iceberg-ing and Object Storage-ing . Take a look at Apache Doris. We are coming full circle where 'integrated solutions' such as Clickhouse and Apache Doris are capable of doing the heavy lifting. They even include features such as federated querying. Apache Doris is my current favourite. This article by Dien, Tran Thanh gives you a great inside look into a migration from BigQuery to Apache Doris. https://lnkd.in/eGYbvCKx

12

BigQuery now has support for Delta Lake tables through Big Lake. #bigquery #googlecloud #databricks

32

It always puzzled my why Automatic DDL replication, while Standard in MySQL is so hard in PostgreSQL. Good to see some progress being made https://lnkd.in/ddi83nkN #postgres #mysql

25

9 comentarios

I still remember that in 2017, when I was at the Reltio conference, it felt like the entire internet had ceased. It turned out to be an AWS-S3 outage, and that was when we all realized S3 is the backbone of the internet. Even today, S3 continues to hold the highest market share(31%), followed by Azure (25%) and GCP (11%). (Posted news in comment for those who forgot) As always 🎯 Mark Freeman II and Chad Sanderson have done a fantastic job explaining why DQ is so difficult for raw storage like S3. Excerpt from blog : "The strength of S3 is also its Achilles' heel: you can store virtually anything in it. Structured data, unstructured data, incomplete data, image data, audio data, Avro, Parquet, JSON—you name it, and you can likely store it in S3. This flexibility is incredibly powerful for data workflows involving machine learning or downstream workflows requiring diverse use cases. However, it also creates a long tail of potential data quality problems, with silent edge cases often causing the most significant issues." @Telmai, we have always believed that data quality monitoring should start at the data sources(shift-left), including landing zones (bronze) or data lakes. Hence, we natively support and automate DQ and monitoring of data lakes and all the raw formats. When you design your data reliability strategy, you should treat this as table stakes. I highly recommend this article and 🎯 Mark Freeman II's post on data quality and contracts to get such valuable insights. https://lnkd.in/gd27Zbik #dataquality #datalake #lakehousearchitecture #dataobservabily #dataobservability

48

5 comentarios

I'm very happy to share a technical article I co-created with Deeksha Gunde, that walks through enabling a federated Airflow environment using AWS SNS/SQS. https://lnkd.in/e4RwPirp #aws #dataengineering #analytics #bigdata

33

Tune in on December 17, as we're diving into the future of table optimization with Beyond Partitioning: Using #LiquidClustering to Escape from Hive-Style! I Say goodbye to the costly guessing game of partition optimization. With liquid clustering, your tables can adapt effortlessly to shifting data volumes—no more stress about getting your partition strategy right from day one. For years, we’ve needed to play an expensive guessing game — will my partition strategy result in too small, too big, or just right file sizes? Do the decisions today still make sense 6 months down the road? What about 1 year? What if you didn’t need to make so many decisions on day one? What if your table could instead adapt to the volume of data with minor effort on your behalf? Welcome to the world of liquid-clustering! In this session you’ll learn to harness the power of adaptive clustering and come to an understanding that today is the day you can finally escape the shackles of hive-style partitioning https://lnkd.in/g_u44AZD

2

Bloom indexes in #PostgreSQL efficiently test for the presence of a set of values, especially useful for multi-column queries where traditional indexes may be less efficient. They are ideal for checking combinations across columns without needing explicit indexes on each. #PostgreSQLPerformance @MinervaDB #dba https://zurl.co/OKDl

5