Tim Yim

➰ APIContext's latest insights on cloud-based API performance have been featured on Nordic APIs! In the 2024 report, we delve into crucial aspects affecting API efficiency: 🔐 Authentication is key. Ensuring robust authentication mechanisms in APIs is crucial, yet offloading this function can introduce complex challenges. 📖 DNS configurations matter. How you handle DNS settings can significantly impact API performance, particularly when tied to authentication processes. 🌐 The role of third parties. Critical third-party services play a significant role in 🔗 API transactions, underscoring the need for careful integration and monitoring. ⛅ Not all clouds are created equal. Differences in cloud services can drastically affect API performance, an essential consideration as APIs account for over 80% of all internet traffic. #APIs #CloudComputing #TechInsights #APIContext

15

1 Comment

In case you missed it, we have now released the second issue of our Confluent Chronicles – a fun and entertaining take on the fundamentals of stream processing, Apache Flink, and Confluent! We had such great feedback on the first issue that we couldn’t just stop there – you can check out and download the latest edition here: https://lnkd.in/gEFxA7af Would love to hear what you think of it!   #ApacheFlink #ApacheKafka #Confluent

39

1 Comment

Having trouble scaling PostgreSQL with AWS RDS? Considering migrating to AWS Aurora to solve this problem? I wrote an article about the Pro's and Con's of choosing AWS Aurora. Its the first in a series where I will also be looking at newer Postgres based options such as Timescale and Neon as well as talking about things you should be considering when benchmarking or designing your application for scale. If you are having problems like this and need help, StepChange.work would be happy to hop on a call and discuss your problems and how we can be of assistance. We have decades of experience building products at scale at top companies such as Meta and Coinbase. https://lnkd.in/gMgAbA4D

19

2 Comments

Postgres is the top database among developers two years in a row, according to the new Stack Overflow 2024 Developer Survey just released today. 51.9% of developers say they love and use PostgreSQL more than any other database. EDB is proud to be part of the vibrant open source community building Postgres, and these survey results highlight the continued growth of Postgres. Read the full Stack Overflow survey here: https://lnkd.in/dAE64Zba #Postgres #PostgreSQL #justsolveitwithPostgres #EDBPostgresAI #data #AI #database #DBA #opensource #stackoverflow #developers

19

1 Comment

Just a few weeks after the GA release of Arm64 runners, we’re already seeing our clients achieve noticeable reductions in build times and lower runtime costs. If you're looking to scale your CI/CD operations and accelerate feedback loops, now is the time to explore the benefits of Arm64! Key takeaways: - ⚡ Faster build times to streamline development. - 💰 Cost savings through optimized runtimes. Ready to take your CI/CD to the next level? #CI/CD #Arm64 #DevOps #BuildOptimization #CostEfficiency #ContinuousIntegration target: https://lnkd.in/gY-_n6JJ

77

Well done #AWS for promptly fixing an S3 billing edge-case! Interesting S3 cost issue in this blog: "How an empty S3 bucket can make your AWS bill explode". In brief: external unauthorised (denied) requests to a new empty S3 bucket were being charged - $1,300 within two days. The author had inadvertently chosen the same bucket name that was the placeholder default configuration in a popular open-source tool!    https://lnkd.in/eNEqedR8 That blog was published on 29th April 2024. On 13th May, AWS fixed this problem: "Amazon S3 will no longer charge for several HTTP error codes". https://lnkd.in/eMXuhhu8 To quote that announcement: "bucket owners will never incur request or bandwidth charges for requests that return an HTTP 403 (Access Denied) error response if initiated from outside their individual AWS account or AWS Organization."

6

Worth reading: How the Apache Arrow Format Accelerates Query Result Transfer by Ian Cook One of the hidden problems of intersystem communication is transcription of data formats. Ian does a wonderful job of explaining the "why" of Apache Arrow as well as the "how" of its advantages. Read on...

8

The speed at which these models improve is staggering, so is the cost to build them. Only way to keep up is to leverage principles such as AWS Bedrock so you have the flexibility you need to quickly select the model that works best for your task. https://lnkd.in/gNAR63YC https://lnkd.in/g_FtP6e9

36

Is your observability data getting out of control? Three ways you can start to tame it using the standard OpenTelemetry (OTel) collector. https://lnkd.in/gbpafUKw #KubeCon #observability #OTel #ControlTheory #ControlItAll

3

1 Comment

- Incremental Backup - Updates to Logical Replication - Improved JSON support With the release of PostgreSQL 17 this week, Postgres continues to evolve into a platform that can handle more workloads than ever before. EDB is proud to have contributed to key features to Postgres 17, showing our commitment to the community and pushing the limits of what open source can do. Read the new InfoWorld article to discover how these advancements impact complex transactions and AI workloads, with fresh insights from EDB SVP of Database Servers and Tools Tom Kincaid. ➡️ https://lnkd.in/eAKbfxiR #Postgres #PostgreSQL #justsolveitwithPostgres #EDBPostgresAI #data #AI #database #DBA #opensource #PG17 #Postgres17 #PostgreSQL17 #JSON #OpenSourceInnovation

16

1 Comment

The past two years at StarTree, working with the Apache Pinot community, have been fantastic. I’ve been able to get to know some incredibly talented people, and it’s validated my perspective that the advent of event-driven architecture is real and pressing, and even more interesting than I realized. Looking back, I’d never spent that much time focusing on analytics and the huge set of tools and concerns that attend it. It’s a critical ecosystem with a history deeper than digital computing itself, and I anticipate a lot of fruitful work with the Pinot community in the months and years to come. That being said, it’s ironic that my time working with this community has crystallized an important idea I’d been developing since before I joined it. The disruption introduced into our world by the perfect storm of the cloud, microservices, and ubiquitous Kafka has created an entirely new approach to building business software. It’s new, uncharted territory to be explored—and I want to help map out as much of it as possible, together with the hundreds and thousands developers who are doing the work of building applications in this new world. That’s why I’ve decided to transition from StarTree, and return to Confluent as VP of Developer Relations. The most valuable thing we can do as developers is write differentiated code that provides meaningful value to our customers or users. That code sits atop a huge stack of what is, with respect to their businesses, undifferentiated software—various kinds of frameworks and infrastructure components—that solves hard problems that application developers shouldn’t have to spend time on. They shouldn’t build that software, but they need to know how to navigate the terrain and make choices in the space they can trust. The problem is that the event-driven revolution has thrown a somewhat stable stack into relative disarray, so some of us are lured into creating little bits of data infrastructure in that unknown area between our code and Kafka. Those of us who are able to avoid that temptation still aren’t totally clear on what all that terrain looks like, and what infrastructure should even live there. History suggests this could take another five to ten years to stabilize. It will require the input of many communities, vendors, and a world full of developers. In my opinion, Confluent is the company that is best positioned to significantly move the needle on this problem, and that’s why I’m excited to (re)join them. One company can’t do it all, but their vision of the Data Streaming Platform is exactly what I’m talking about here. The opportunity to lead a world-class developer relations team at the company which has pioneered the journey so far, and is building the most comprehensive vision for its next stage, is a tremendous one. I’m delighted to be back, and eager to do my part to help chart our course to the future. It is, as I am wont to say, a wonderful time to be alive.

917

291 Comments

Folks I've spoken to about Amazon have likely heard me talk about the importance of the PR/FAQ process, a.k.a. "the working backwards process". This blog posts gives a very nice example of a PR/FAQ. https://lnkd.in/edTeuVRQ. Note that the blog doesn't capture the presumably 100s of reviews/discussions/rewrites that this doc represents, which is a part of the real value. When the process is done it all looks so simple, but getting a large organization to make a decision to something simple and do it in a BIG way is the challenge the fun of it all.

63

3 Comments

As you probably heard ScyllaDB is no more #opensource after consolidating on Source Available license with free tier for small scale installations. This is unfortunate news as I think ScyllaDB served important niche, for use cases where high performance and scalability are important. I do not expect it to be forked in meaningful way due to relatively small development community and restrictive Open Source license (AGPL) the Open Source edition used. I think though we see some people giving Apache Cassandra a second look, especially considering performance improvements in recent release as well as looking at Yugabyte as another solution offering some level of Cassandra Compatibility. Neither are complete ScyllaDB replacements though Despite those changes I appreciate what Dor Laor and his team brought to Open Source Database community and I hope they continue to contribute to Open Source, despite their main product now using Proprietary, Source Available License For more thoughts read complete article https://lnkd.in/eyUAYWbx

156

35 Comments

Truth. This blog articulates the shortcomings of most tiered storage solutions in delivering predictable performance at scale. Without some rebuild that harnesses cloud primitives like AWS S3 and GCP GCS, mere incremental adjustments to existing products won't adequately serve users. From a product manager's perspective, sometimes incremental adjustments isn't a path to sustainable product adoption and success.

11

Really interesting article on how to build #serverless data pipelines with Kestra, Modal, dbt, and BigQuery. It got me digging deeper into Kestra and I’m seriously impressed by their open-source orchestration platform. The workflow UI is super slick and you can easily integrate and build automations with just about any data stack you can think of. They’re also working on a cloud version that looks like it’ll be completely serverless! 🙌 https://lnkd.in/epZfuPrg

30

2 Comments

It's official! 🎉🌟 #ApachePaimon has graduated as a top-level project within The Apache Software Foundation 🚀 This momentous occasion marks the project’s maturity and widespread adoption by the community. But what exactly is Apache Paimon? It’s a cutting-edge data lake format that enables real-time lakehouse architectures built with #apacheflink and #apachespark for streaming and batch operations. By integrating Paimon into your data architecture, you unlock a world of possibilities for cost-effective, efficient, and scalable data pipelines. 🌐 So, what does this mean for customers who have adopted or are considering data lakes? Let’s break it down: 1️⃣ Cost-Effective Data Pipelines: Apache Paimon offers a more affordable solution compared to traditional methods. By replacing Apache Hudi in many deployments, Paimon has demonstrated its ability to reduce costs while enhancing performance. Say goodbye to expensive data processing and hello to cost-efficient pipelines! 💰 2️⃣ Improved Data Freshness: With Paimon, you can achieve near-real-time data updates, ensuring that your data is always fresh and up-to-date. This is crucial for businesses that rely on timely insights to drive decision-making and stay ahead of the competition. 🕒 3️⃣ Unified Batch and Streaming Architecture: Paimon supports both batch and streaming operations, providing a seamless and integrated approach to data processing. This means you can handle batch, streaming, and OLAP data with the same query and table abstraction, simplifying your data architecture and reducing complexity. 🔄 4️⃣ Strong Community and Ecosystem Support: As part of the Apache Software Foundation, Paimon benefits from a vast ecosystem of contributors and users. This active community ensures ongoing development, support, and a rich set of tools and integrations to help you build powerful data solutions. 🌐 https://lnkd.in/gTPz-pV6 #ApachePaimon #DataLakes #DataEngineering #Lakehouse #StreamingProcessing #BatchProcessing #CostEfficiency #DataFreshness #UnifiedArchitecture #CommunitySupport

33

With the release of PostgreSQL 17, Postgres continues to evolve into a platform that can handle more workloads than ever before. EDB is proud to have contributed to key features like incremental backups and enhanced SQL/JSON support, showing our commitment to the community and pushing the limits of what Postgres can do. Tom Kincaid, our SVP of Database Servers and Tools, breaks down these advancements and how they impact complex transactions and AI workloads in this InfoWorld article here ➡️ https://lnkd.in/ega5ggVS #Postgres #PostgreSQL #justsolveitwithPostgres #EDBPostgresAI #data #AI #database #DBA #opensource #PG17 #Postgres17 #PostgreSQL17 #JSON #OpenSourceInnovation

5

How are CloudNativePG and Kubernetes reshaping Postgres deployments across cloud, hybrid, and on-prem environments? At PGConf.EU 2024, Gabriele Bartolini and Marc Linster will present "Postgres: From Cloud to Hybrid and On-Prem Again," discussing: ✅ The flexibility of running Postgres databases with CloudNativePG across VMs or bare metal to meet high-performance needs ✅ Ensuring data portability and compliance with the EU Data Act by enabling easier switching between service providers ✅ How DBAs can retain control of their data with hybrid architectures, avoiding vendor lock-in and improving flexibility Learn more about their talk and what EDB is building to support Postgres workloads of the future here ➡️ https://lnkd.in/gZ2xxi-s #EDBPostgresAI #JustSolveItWithPostgres #PGConfEU #CloudNative #Kubernetes #HybridArchitecture #CloudComputing #PostgreSQL #OpenSource

11