Mayank Kapoor on Hidden Costs of Complexity in Enterprise AI

Wow!!!! Loved hosting Erica Volini, Chief Customer Officer at MongoDB, here at MongoDB.local Bangalore on The Ravit Show. Erica has had a front-row seat to some of the biggest enterprise shifts of the last two decades. Deloitte. ServiceNow growing from 1.5 billion to over 10 billion in revenue. And now MongoDB at the center of the AI moment. So when she talks about how companies actually navigate change, you listen. Here is what we got into. I asked her how this AI moment compares to the transformations she has seen before. Her answer was honest. Faster, messier, and the gap between leaders who are experimenting and leaders who are deploying is wider than people realize. We talked about what she is actually hearing from enterprise leaders right now. Where they are excited, and where they are stuck. The stuck part was the more interesting half. We spent real time on India. MongoDB is behind half of India's top 100 companies and more than 50 unicorns. I asked her what that signals about where India is headed as an AI market. Her read on the speed of adoption here was sharper than I expected. Her background in human capital is rare for someone in her role, and that came through. She thinks about AI as much through the lens of people and skills as she does through the lens of platforms. That framing showed up strongly when we got to MongoDB's commitment to upskilling two million Indian builders by 2030. She made the case for why the developer pipeline matters as much as the product itself, and I agreed with most of it. A few things stayed with me from this conversation. The companies winning with AI right now are not the ones with the biggest budgets. They are the ones whose people are ready to use it. India is not just adopting AI. India is shaping how AI gets built for the rest of the world. And the next two years will separate the enterprises that treated AI as a project from the ones that treated it as a rewiring. More conversations coming soon, stay tuned!!!! #data #ai #mongodb #lmongodbocal #theravitshow

400 terabytes of logs per day. 100 million logs per day on Kubernetes. Those are not benchmark numbers. They are production workloads from teams that shipped their talks at OpenSearchCon India 2026. Five observability sessions are now live. Rashmi Ramanathan and Buden Saheb Shaik from Freshworks explain how ODD replaced ELK at 400TB scale. Sravanthi Naga from Pegasystems covers performance engineering for a Kubernetes observability platform at 100M logs daily. Jeevitha Gajendran walks through turning noisy logs into actionable insights with Fluent Bit, OpenSearch, and RAG. Bharav Patel from Amazon Web Services (AWS) covers advanced data transformation pipelines with PPL. And the Freshworks team shows how they built a full observability suite inside OpenSearch Dashboards. If you are running any kind of log infrastructure, at least one of these will be directly useful. https://bit.ly/4eSuFdV

My first Elastic community event and what a Saturday morning it turned out to be! Excited to have attended the Elastic GenAI Meetup at SmartSense Consulting Solutions, GIFT City, Gandhinagar! Until now most of my community time has gone into AWS meetups, so stepping into an Elastic-focused event was a refreshing shift As someone building my career in Cloud & DevOps, understanding Elasticsearch, Vector Search, and RAG (Retrieval-Augmented Generation) isn't optional anymore it's foundational. Before we bring Elastic into production for log management, observability, and real-time search, it's important to first understand how the underlying search and indexing actually work. Today's sessions helped me build exactly that base. Some of my key takeaways: 🔍 How Elasticsearch powers fast, scalable search under the hood 🧠 Where Vector Search fits into modern GenAI applications 📊 Why RAG is becoming essential for AI-powered systems 📈 How all of this ties back to real-world DevOps use cases like logging and monitoring Grateful to connect with so many passionate developers, cloud enthusiasts, and AI professionals today, and to learn directly from industry experts. Every conversation added something new to my Cloud, DevOps, and AI journey. A big thank you to Elastic User Group and SmartSense Consulting Solutions for putting together such a well-organized community event Looking forward to applying these learnings in real projects soon! If you're into AI, Cloud, or Search technologies, let's connect and grow together Dhairya Panchal | Achyut Hadavani | Shiv Jani | Bhargav Parmar | Gautam Modi| Sumit Yadav | Rohit Rathod | KUNJ SHAH | Savya Thakkar | Elastic #Elastic #Elasticsearch #GenAI #RAG #VectorSearch #DevOps #Cloud #GIFTCity #Ahmedabad #CommunityLearning #ElasticUserGroupGujarat

𝗞𝘂𝗯𝗲𝗿𝗻𝗲𝘁𝗲𝘀 𝗛𝗣𝗔 𝗶𝘀𝗻'𝘁 𝗮 𝘀𝗶𝗹𝘃𝗲𝗿 𝗯𝘂𝗹𝗹𝗲𝘁. Many engineers use **Horizontal Pod Autoscaler (HPA)** for every workload. That's a mistake. HPA scales based on **resource metrics** like: * CPU * Memory * Custom metrics (Prometheus) But what if your application processes messages from **Amazon SQS**, **Kafka**, or **RabbitMQ**? Imagine this: * 📩 100,000 messages are waiting in SQS. * 🖥️ Your pods are using only **10% CPU**. HPA says: > "No need to scale." Your queue says otherwise. This is where **KEDA (Kubernetes Event-Driven Autoscaling)** shines. ✅ Scales based on **external events**, not just CPU or memory. Supports event sources like: * Amazon SQS * Kafka * RabbitMQ * Redis * Azure Service Bus * Google Pub/Sub * Prometheus * AWS CloudWatch (70+ scalers) ### HPA vs KEDA 🔹 **HPA** * CPU/Memory-based scaling * Great for REST APIs and stateless services * Cannot scale from **0 → N** 🔹 **KEDA** * Event-driven scaling * Perfect for queue workers, ETL jobs, AI inference, and background processing * Supports **scale-to-zero**, reducing idle infrastructure costs 💡 **Rule of thumb:** * **Use HPA** when requests directly hit your application. * **Use KEDA** when work arrives through queues or events. **Don't scale based on how busy your containers are. Scale based on how much work is waiting.** That's the difference between reactive infrastructure and efficient cloud-native architecture. #Kubernetes #KEDA #HPA #AmazonEKS #AWS #CloudNative #DevOps #PlatformEngineering #SystemDesign #Autoscaling

Great products don't win because they have more features - they win because they fit naturally into a customer's ecosystem and deliver measurable business outcomes. At a recent invite-only #MongoDB.local event, one architecture discussion stood out. A typical streaming pipeline looked like this: MongoDB → Kafka (ECS) → Amazon S3 → Snowflake The more strategic question wasn't about the pipeline - it was: What if Apache Iceberg became the foundation? That single decision enables: ✅ Cloud-agnostic architecture (AWS, Azure & GCP) ✅ Lower storage and compute costs ✅ Reduced operational complexity through managed services This is where product strategy becomes powerful. The conversation shifts from: "We manage your database." to "We manage your entire data ecosystem around your transactional core." That's not feature expansion - it's product expansion. Key Product Strategy Takeaways: • Ecosystem Fit > Feature Push - Integrate into existing architectures and solve business outcomes. • Bridge OLTP & OLAP - Own the journey from transactional to analytical workloads. • Open Standards Matter - Technologies like Apache Iceberg reduce vendor lock-in and enable cloud portability. • Manage Complexity - Customers value platforms that simplify streaming, governance and multi-cloud operations. • Expand the Platform - Land with the database, grow into data mobility, streaming and analytics. The biggest takeaway? Enterprise customers don't buy technology because it's technically superior. They buy solutions that reduce complexity, lower costs, scale with the business and fit seamlessly into their ecosystem. That's where product strategy creates lasting competitive advantage. #ProductManagement #ProductStrategy #EnterpriseArchitecture #DataEngineering #CloudComputing #DataPlatform #DigitalTransformation #DataArchitecture #Analytics #StreamingData #OpenSource #CloudNative #TechnologyLeadership #Innovation #SoftwareEngineering #softwareEngineering #director #EngineeringManager #CorporateStrategy #MongoDB #MongoDBAtlas #ApacheIceberg #Kafka #Snowflake #fbsindia #ford #leadership #innovation

The Call for Speakers for AWS Community Day Mumbai 2026 is officially open. Stepping onto a stage like this is about more than just delivering a slide deck. Standing in front of an audience of over 500+ cloud enthusiasts, platform engineers, and tech leaders is an incredible opportunity to spark meaningful technical conversations. It is a chance to share the hard-learned lessons from your own production environments, establish your voice in the ecosystem, and directly shape how the local community approaches modern infrastructure problems. If you have navigated a complex migration, built something impactful with GenAI, or solved a tricky architecture bottleneck, your insights are exactly what the community needs to hear. We are looking for technical sessions, real-world case studies, and deep dives across a wide range of areas, including Cloud Architecture, AI/ML, DevOps, Containers, Serverless, and Security. To help tailor your proposal, here are the key details for this year's submissions: * Session Formats: You can submit for either a quick, high-impact 15-minute session or a comprehensive 30-minute deep dive. * Talk Levels: We welcome content across the entire spectrum, whether it is an introductory Level 100/200 talk or an advanced Level 300/400 technical breakdown. * Deadline: The CFP will remain open until July 31, 2026. If you want to share your engineering journey and give back to the local tech community, we would love to see your proposal. You can submit your session details directly through the portal. The event will take place in Mumbai on October 17, 2026. Link to submit: https://lnkd.in/gDWCetHm #aws #mumbai #speaker #cloud #awsbuilder #communityday

Excited to be speaking about MongoDB at the Core of a No-Code Platform at the upcoming mongodb.local event in Bangalore on June 30th. Over the years, we have built our Workhall platform that enables organizations to create applications(includes Business Workflows, Data Orchestration Layer, Reports, AI-powered experiences, UI and more). Platform comes with unique architectural challenges, especially when customers are building thousands of custom applications on a single platform. In this session, I will share how Workhall leverages MongoDB to address many of these architectural challenges. Looking forward to connecting with fellow builders, architects, and the MongoDB community in Bangalore. #MongoDB #NoCode #LowCode #SaaS #ProductEngineering #SoftwareArchitecture #CloudNative #EnterpriseSoftware #DigitalTransformation #Workhall #ApplicationDevelopment

𝐀𝐈 𝐦𝐨𝐝𝐞𝐥𝐬 𝐠𝐞𝐭 𝐭𝐡𝐞 𝐡𝐞𝐚𝐝𝐥𝐢𝐧𝐞𝐬, 𝐛𝐮𝐭 𝐀𝐖𝐒 𝐢𝐧𝐟𝐫𝐚𝐬𝐭𝐫𝐮𝐜𝐭𝐮𝐫𝐞 𝐤𝐞𝐞𝐩𝐬 𝐭𝐡𝐞𝐦 𝐚𝐥𝐢𝐯𝐞. As a DevOps engineer, I see a massive gap between a data scientist training an ML model and actually scaling it securely in production. Passing a Jupyter Notebook to an operations team is no longer a viable workflow. Moving into 2026, LLMOps and MLOps are completely redefining our cloud architectures. It requires moving away from single-account setups and building structured, automated pipelines across isolated environments. Here is exactly how we look at modern AI infrastructure scaling, inspired by the reference architecture below: 1) Separation of Concerns: Driving the lifecycle across isolated AWS accounts 2) Development, Automation, Staging, and Production: to guarantee compliance and security. 3) Hybrid Orchestration: Leveraging Azure DevOps for git-driven Model Build and Model Deploy pipelines, seamlessly executing infrastructure updates via AWS CloudFormation. 4) Automated Lineage: Relying on Amazon SageMaker Pipelines for preprocessing, training, and registering models directly into a secure Model Registry with EventBridge-triggered approvals. 5) Production-Ready Inference: Abstracting model delivery into flexible endpoints: from low-latency Real-time inference to cost-effective Serverless inference, fronted by Amazon API Gateway and AWS Lambda. 6) Continuous Governance: Closing the loop with SageMaker Model Monitor and CloudWatch to catch data drift and latency spikes before they impact users. The real engineering challenge isn't just writing the model code. It is building the automated, scalable, and cost-effective pipelines that keep those models reliable at scale. #AWS #DevOps #MLOps #CloudArchitecture #PlatformEngineering

How I Cut API Latency by 35% We had a problem: AI workloads were causing API timeouts during peak hours. Customers were frustrated. The team was frustrated. The root cause? We were processing everything synchronously. One request = one process. If it took 10 seconds, the API hung for 10 seconds. The fix was embarrassingly simple: move heavy workloads to async queues. Accept request Enqueue task to AWS SQS Return job ID instantly User polls for results Result: API latency dropped by 35%. No more timeouts. Happier users. Lesson: Sometimes the simplest solution is the right one. I'm exploring AI engineering roles, if you're hiring, DM me. #PerformanceOptimization #AWS #SoftwareEngineering #Backend #Async

🚀 New Medium Article Published! Manual infrastructure provisioning can quickly become a bottleneck for engineering teams, especially when working with temporary Amazon EMR environments. In my latest article, I share how I designed an API-driven automation framework using AWS Lambda to simplify the complete lifecycle of EMR Sandbox provisioning—from creation and validation to termination and rehydration. The article covers: • Designing an API-first provisioning workflow • Automating infrastructure with AWS Lambda • High-level architecture and orchestration • Lessons learned while building the solution If you're interested in AWS, Data Engineering, Platform Engineering, or Cloud Automation, I'd love to hear your feedback. 📖 Read it here: https://lnkd.in/gMvabyxN #AWS #DataEngineering #AWSLambda #AmazonEMR #CloudComputing #Automation #DevOps #PlatformEngineering