Marc Lamberti

Remember back in 2012, when PostgreSQL introduced JSON support ? https://lnkd.in/e_Qb8rpc Now in PostgreSQL 17 we have JSON_TABLE and new JSON functions !

3

Data Engineers ! This one's for you: 🧠 The Economics of Generative AI Stephanie Kirmer 🧠 Connecting ML Models in Sagemaker to Snowflake and dbt Hugo Lu 🧠 Introduction to Interpretable Clustering NAKUL UPADHYAY 🧠 Snowflake automatic clustering cost estimation Tim Sander 🧠 Death to Tech Debt? Ari Joury, PhD Check it here; https://lnkd.in/ebdFKNZ5 #dataengineering #dataroundup #startup

5

SingleStore Announces Bi-Directional Integration with Apache Iceberg Learn more & get our take 👇 https://lnkd.in/gNTBCWAE "Our vision has always been to provide one single data store for all companies to be able to take advantage of speed, scale and simplicity. Our data platform is designed to unlock all types of enterprise data — including data that is frozen in data lakes." — Raj Verma, CEO at SingleStore "SingleStore’s data platform is indispensable for analyzing our data and delivering real-time insights to our customers. This latest update continues to push the boundaries of what we can accomplish by harnessing all of our enterprise data in real time." — Mauricio Aristizabal, Principal Data Architect and Founder at Giist Social "The integration with Iceberg, plus enhancing our platform’s scaling and processing capabilities will allow users to access and harness their data in real time, and build intelligent apps." — Nadeem Asghar, Head of Engineering at SingleStore #apache #product #SoTNews

1

Apache XTable provides users with the ability to translate metadata from one #lakehouse table format to another omni-directionally. What exactly happens after the XTable "Sync" Process is run? The sync process provides users with the following: ✅ Syncs the data files along with their column-level statistics and partition metadata information ✅ All the schema-level updates in the source table are reflected on to the target format metadata ✅ Metadata maintenance for the target table format. - If the target format is Apache Hudi, unreferenced files will be marked as ‘cleaned’ to control metadata table size - If the target format is Apache Iceberg, snapshots will be expired after a configured amount of time - If the target format is Delta Lake, the transaction log will be retained for a configured amount of time ⭐️ Want to try out XTable? - Here is a link to the getting started page: https://lnkd.in/gHMBQeqV #dataengineering #softwareengineering

49

1 commentaire

🎩 Why fuss with Airflow and Prefect when you can Beat it with Celery? Sure, Airflow and Prefect get all the fancy workflow orchestration headlines, but sometimes you just want something that gets the job done without needing a PhD in DAGs. That’s where Celery Beat + Redis come in, offering a no-nonsense, efficient solution for scheduling tasks and orchestrating workflows. What makes this combo my go-to for simpler workflows: 🕒 Simple yet Powerful Scheduling: Celery Beat handles periodic tasks effortlessly, no complex DAGs required. ⚡ Redis Backend: Lightning-fast task queuing with Redis keeps things moving smoothly in high-throughput systems. 🔗 Workflows, But Streamlined: Task chaining, grouping, and conditional logic are all a breeze, perfect for straightforward automation needs. 📈 Easy Monitoring & Scaling: Keep track of task execution and scale when needed, without the overhead. Now, don’t get me wrong; Airflow and Prefect absolutely shine when it comes to complex, multi-layered workflows and data pipelines. If you’re building intricate, long-running data jobs or need advanced DAG management, they’re fantastic tools for the job. But for many of my use cases, Celery’s simplicity and flexibility strike the perfect balance. How are you orchestrating your workflows? Do you prefer the simplicity of Celery or the power of Airflow/Prefect for your more complex projects? Celery Documentation: https://lnkd.in/g3DCmxdH Celery Beat: https://lnkd.in/gauGBYWe #CeleryBeat #Redis #WorkflowAutomation #Airflow #Prefect #Python #TaskScheduling #DistributedSystems

16

Great things do happen these days in the big data domain. Table formats seems to consolidate, query engines do get more traction, AI is on the go. The Apache Software Foundation thanks for making this a TLP project. #apache #datafusion #bigdata

6

Attention Snowflake users ! Wouldn't it be great if every time anything anomalous happened in your infrastructure it got flagged? Well in Orchestra that's pretty easy to set-up. Check it out below! https://lnkd.in/eP8bj6MU #dataengineering #anomalydetection #snowflake

7

🎯 You shouldn't miss this last chat about #PostgreSQL and #Compression with Nikolay Samokhvalov and Michael Christofides from Postgres.FM brought. 🔥 My #suggestion? While listening to the #podcast #chat, access the video's #information for a #collection of #blogs and #publications about #compression on the #Postgres #World. 🐘 #databases #postgres #data #compression #cpucycles #toast

6

Question: Considering migrating to Unity Catalog? Curious about potential errors or the rollback process? Watch this quick clip of Blueprint's Databricks Champion, Shannon Lowder, explaining just how easy it is to manage and reverse a migration if needed. #unitycatalogmigration #ucmigration #unitycatalog #databricks

8

#DataEngineering 𝗣𝗿𝗼 𝗧𝗶𝗽𝘀 𝗳𝗿𝗼𝗺 𝗙𝗮𝗯𝗿𝗶𝗰𝗲 𝗗𝗲𝘀𝗲𝘆𝗻, 𝗗𝗮𝘁𝗮𝗯𝗿𝗶𝗰𝗸𝘀 𝗖𝗵𝗮𝗺𝗽𝗶𝗼𝗻 Join our colleague, Fabrice Deseyn, for a deep dive into the world of Databricks. In a recent interview with Youssef Mrini on the #NextGenLakehouse YouTube channel, Fabrice shares his insights and journey of becoming a Databricks champion. Beyond his champion status, Fabrice provides expert guidance on navigating Unity Catalog migrations. Discover practical tips and strategies to streamline the process and ensure a successful transition. Watch the video now 👉 https://lnkd.in/e2zFw53p #Databricks

11

1 commentaire

Join PyTorch at Snowflake's #DataCloudDevDay on June 6 in San Francisco to learn how to accelerate GenAI performance with Native PyTorch. Register: https://hubs.la/Q02xQStt0

18

JSON Handling and Type Management with Large Datasets in Go -- Ep. 4 is out now! In this episode, Miki Tebeka tackles the complexities of working with JSON data, especially when dealing with extensive datasets and optimizing type management. Here, Miki introduces key strategies for managing large JSON documents, avoiding unnecessary type definitions, and using Go’s encoding/json package to handle custom serialization needs effectively. Things you will learn in this video: ✅ Streamlining JSON Handling: How to avoid excessive type declarations by using anonymous structures and selective decoding for large JSON datasets. ✅Implementing Custom Serialization: Techniques for defining custom JSON serialization in Go with the Marshaler and Unmarshaler interfaces. ✅Managing Mixed-Type Arrays: Strategies to work effectively with JSON arrays containing various data types, ensuring smooth integration with Go’s type system. 📰Watch Now: https://lnkd.in/ghjz58pC Stay tuned for more episodes from Miki Tebeka in this mini series, highlighting Intro to #JSON for Engineers & Managers.🤓 #golang

23

Sharing one of my recent projects : A Multi-threaded Proxy Server in C with Caching! Key Features: ->Multi-threaded Handling: Efficiently manages multiple client requests concurrently using POSIX threads. ->Optimized Caching: Integrated the uthash library for O(1) average time complexity in cache lookups in LRU cache , significantly boosting performance. ->Thread Safety: Ensured robust synchronization and error handling with mutex locks and semaphores. I learnt systems programming, performance optimization, and thread safety. Looking forward to applying these skills in future challenges! #CProgramming #SystemsProgramming #Multithreading #Caching #PerformanceOptimization #SoftwareDevelopment

80

2 commentaires

Umair Shahid: Understanding Factors Impacting Data Replication Latency in PostgreSQL Across Geographically Distributed Nodes 🛠️ PostgreSQL Geeks, Ever Wonder Why Data Replication is as Fast as Your Grandma’s Internet? 🛠️ Check out Umair Shahid’s dive into the treacherous waters of data replication latency in PostgreSQL across geographically distributed nodes! 🌍 In today’s interconnected world, where your cat video server needs to deliver meows at lightning speed to enthusiasts worldwide, understanding data replication is crucial. Ensuring business continuity, disaster recovery, or just keeping your global users happy means mastering data replication between nodes scattered around the globe. 🌐 Umair Shahid’s latest post, *"Understanding Factors Impacting Data Replication Latency in PostgreSQL Across Geographically Distributed Nodes,"* is a must-read if you’re wrestling with the same challenges we all are—managing colossal data heaps between continents without pulling your hair out. Read the full piece that appeared on Stormatics: [https://lnkd.in/edg4ZZx3] Thanks to Umair for tackling this beast and helping us tame these replication monsters. Let’s train our nodes to replicate faster than a hyperactive rabbit! 🐇💻 #PostgreSQL #DataReplication #GloballyDistributedNodes #Latency #TechReads --- Check out more at Stormatics: [https://stormatics.tech] You can find the full article here: https://postgr.es/p/6CV

4

Excellent talk by our Daniel Balosh on give a well thought recepie to thoroughly go from a tech debt situation with an overwhelmed MySQL DB and reuduce it’s CPU usage by 70% in just three weeks!! Also we at Eleos Health are #hiring - #dataplatform and #fullstack #teamlead #qa #dsci #dataengineering - DM me Alon Alon Dror Merav Liat Tal Elad Sapir Samuel Amit Lidor Yacov Andrew Almog Michelle Raz Naomi Lior Or Lior Yair

17

1 commentaire

Reasons to monitor dbt in Orchestra 🕊️ Ease of use Orchestra is the easiest way to set-up dbt core orchestration and monitoring. The Data Catalog even gets updated without any additional integration 💸 Cost Orchestra provides an execution environment for dbt core with lightweight pricing, which can save on existing dbt compute costs. We also support monitoring dbt core projects in AWS EC2, Virtual Machines, etc. so you can run dbt core your way without compromising on visibility 🔀 End to End Orchestration There is sometimes more to life than synchronously pushing down SQL Queries to your data warehouse. Orchestra provides a feature-complete, end-to-end orchestration framework so you can trigger and monitor any job from a single place What other problems do you want us to solve? See it in action below. https://lnkd.in/eTzuG7Ty #dataengineering #analytics #dbt

2

Apache Arrow Flight for Columnar OLAP workloads. Why don’t ODBC & JDBC fit in today’s analytical world? Well, the reason is that these protocols were designed particularly for row-based workloads. Back in the days, most of the systems were row-based & there was a need for a standard protocol to transfer data between systems. ODBC/JDBC solved this problem & eventually became a standard! ✅ But if we look today, with OLAP analytics, specifically with huge volumes of data, more & more databases/stores are becoming columnar. And most client tools involved in OLAP workloads are also columnar to take advantage of those same performance benefits. But there is no 'columnar' transfer standard! ❌ This means if you have a columnar database & you are using ODBC/JDBC, you have to first convert it into row & if the client is columnar as well, you will need to convert it back to column. This process is called serialization/deserialization (serde) and can take 60-90% of the time of a transfer, which impacts performance a lot. Enter "Arrow Flight" - a standard protocol for efficiently transferring Arrow data. Check out the recording from a past talk in comments! #dataengineering #softwareengineering

162

11 commentaires