How to Reduce Friction in Data Workflows

I’ve noticed a recurring theme in my recent discussions with large organisations.   API friction is a hidden cost centre. And it compounds quietly, every single day.   In most enterprises, developers spend around 3 hours each week dealing with: inconsistent API contracts unclear or custom authentication flows documentation that no longer matches the implementation duplicated services that nobody realised already existed   That’s 20 workdays per developer, per year — before even considering partners, integrators or external ecosystems.   At that point, it’s no longer simply a technical inefficiency. It’s a business and ROI issue. It impacts delivery timelines, onboarding speed, incident recovery, compliance, and customer experience.   During these conversations, leaders often ask: “Okay, but how does standardisation actually help?”   My answer is usually along the following lines: Start with contract-first API design (OpenAPI / AsyncAPI), so design, tests, SDKs and docs all come from the same source of truth. Move to one authentication model (OAuth2 + OIDC) instead of several slightly different ones — it reduces support and integration friction. Generate documentation automatically as part of the build pipeline (if docs can drift, they will drift). Define a few clear conventions for naming, pagination, error structures and versioning — predictability is a performance multiplier. Maintain a shared API catalogue so teams can discover what already exists (otherwise they rebuild it again). And when possible, align with recognised open standards like the work carried out in ETSI TC DATA, which focuses on interoperable data architectures and API patterns for distributed data ecosystems.   This isn’t about adding control or bureaucracy. It’s about removing friction — the kind that slows everything down without anyone noticing it directly.   The outcomes are very tangible: ✅ Faster onboarding of internal teams and partners ✅ Lower long-term integration & maintenance costs ✅ Fewer incidents + smoother change management ✅ Stronger compliance posture ✅ Predictability at scale   If this resonates, comment ROI — I’ll share a simple API Friction Cost Calculator that makes this visible in under 2 minutes.

At Amazon, I’ve built pipelines that move thousands of gigabytes of data. At Amazon, I’ve also built platforms used by hundreds of teams across the organization. But do you know how I got the opportunity to do these things? → It was because of one simple mindset shift: I stopped thinking like a pipeline builder. And started thinking like a product builder. Here’s what that shift looks like in real life 👇 1. Optimize for adoption, not just execution A fast Spark job is nice. But a pipeline that any team can deploy, monitor, and debug without you? That’s a game-changer. If your internal users are struggling, that’s a UX bug. 2. Design APIs, not one-off scripts Your Airflow DAGs and Glue jobs should feel like APIs. Versioned, observable, with clear inputs/outputs. That’s how you build trust at scale. 3. Surface friction like a PM If people keep pinging you for creds, schemas, or weird Athena errors, that’s a signal. Treat those moments like product bugs. Fix them once, and fix them for everyone. 4. Metrics = feedback loops In product, you track conversion. In data platforms, track usage: → How many teams use your tools? → How often do they fail? → Who’s stuck? These are your feature requests. 5. Think enablement > control Great platforms don’t block, they enable. Guardrails should guide, not restrict. Make it easy to do the right thing. I’ve learned this the hard way. When you think like a product builder, your work scales. It doesn’t stop at you. It becomes a system that helps others move faster. So next time you're building a data pipeline, ask yourself: What would this look like if it were a product? Let’s build platforms that people actually want to use.

𝐌𝐨𝐬𝐭 𝐑𝐀𝐆 𝐬𝐲𝐬𝐭𝐞𝐦𝐬 𝐝𝐨𝐧’𝐭 𝐟𝐚𝐢𝐥 𝐛𝐞𝐜𝐚𝐮𝐬𝐞 𝐨𝐟 𝐭𝐡𝐞 𝐋𝐋𝐌. 𝐓𝐡𝐞𝐲 𝐟𝐚𝐢𝐥 𝐛𝐞𝐜𝐚𝐮𝐬𝐞 𝐭𝐡𝐞 𝐫𝐞𝐭𝐫𝐢𝐞𝐯𝐚𝐥 𝐩𝐢𝐩𝐞𝐥𝐢𝐧𝐞 𝐢𝐬 𝐭𝐨𝐨 𝐬𝐥𝐨𝐰. After spending months building real RAG systems, one thing became clear: Speed is not an accident. It’s engineered. If I had to rebuild a fast, production-grade RAG pipeline today, these are the 7 techniques I would start with 👇 1.Vector Database Optimization → Switch to ANN search (HNSW, IVF) → Optimize indexes for your dataset size → Reduce embedding dimensions where possible → Use quantization to speed up similarity search 2.Caching Strategies → Query caching for repeated questions → Embedding + context caching → Multi-level caching with in-memory + Redis 3.Reranking Optimization → Two-stage retrieval (fast fetch, small rerank) → Lightweight cross-encoders → Confidence-based filtering → Hybrid lexical + vector search 4.Context Window + Prompt Optimization → Dynamic chunk selection → Smaller chunks (256–512 tokens) → Summaries instead of raw text → Tight, token-efficient prompts 5.Model Selection and Optimization → Smaller embedding models → Faster LLMs for simple queries → Quantized local models → Smart routing based on complexity 6.Parallel Processing → Parallel retrieval across vector stores → Async chunk embedding → Batch I/O operations 7.Smart Routing and Query Classification → Intent classification → Complexity scoring → Domain-specific routing → Cache-first flow 👉 Fast RAG isn’t just about picking a good vector DB or chunk size. 👉 It’s about engineering every stage of the pipeline so retrieval, reranking, context prep, and generation work together with minimal friction. 👉If you want smoother UX, lower latency, and happier users, start here. ♻️ Repost to help someone fix their slow RAG pipeline. ➕ Follow Naresh Edagotti for more practical AI systems breakdowns.

What They Don’t Teach You About Data Engineering #2: Use the Boring Tools That Work   A solid data stack is often boring on purpose. You don’t need the latest real-time feature store with vector embeddings if your pipeline could just run once every night. In many real-world projects, progress comes from simple tools that do their job well. Docker to make environments reproducible. Postman to test APIs quickly. DuckDB to explore data locally without spinning up infrastructure. Cursor to move faster while writing and debugging code. None of these tools are particularly flashy. But they remove friction. And removing friction is what keeps projects moving. A lot of early-career engineers feel pressure to build something impressive with the newest technologies. But most companies don’t hire data engineers to experiment with hype. They hire them to make data reliable, accessible, and usable. That often means choosing tools that are: ✅ easy to understand ✅ quick to set up ✅ stable in production ✅ well-documented “Boring” tools usually check all those boxes. The goal isn’t to build the most complex architecture. The goal is to build something that works consistently, is easy to maintain, and that your teammates can actually understand six months later. Don’t chase fancy tech for LinkedIn likes. Chase what gets the job done well.

On the surface, Change Data Capture (CDC) sounds straightforward: track changes in a database and move them somewhere else—easy, right? Not quite. CDC is inherently intensive. You're dealing with high-frequency, high-volume transactional data, often across multiple heterogeneous systems. These aren't batch jobs you run once a day. This is real-time, always-on infrastructure. And building a production-grade CDC pipeline isn't just about tapping into a database log—it’s about handling scale, latency, schema drift, fault tolerance, and operational complexity. What’s often underestimated is what it really takes to make CDC pipelines low-maintenance and reliable: Log parsing and recovery logic for dozens of different database engines and versions. State management at scale, across distributed systems, with the ability to checkpoint and recover precisely. Schema evolution support, where downstream systems need to stay in sync even as upstream schemas change. Backpressure management, because targets are rarely as fast as sources. Security, governance, and observability baked in, because no enterprise trusts black boxes. Exactly-once semantics, or at least the illusion of it, depending on how strict your system needs to be. And above all—resource efficiency, because CDC is always running and needs to be lightweight enough to coexist with mission-critical OLTP workloads. So when you see a CDC product that "just works"—that lets you point to a source database and stream data reliably into a lake or warehouse with minimal config—it’s worth recognizing the layers of engineering behind that simplicity. Making CDC look easy is hard. And doing it in a way that scales without becoming a constant ops burden? That takes serious design around performance, durability, and observability. At Striim, we’ve been building CDC pipelines for years, and we’ve learned that the real product isn’t just data movement—it’s trustworthy, low-friction data movement. That means automating the hard parts, surfacing the right signals, and giving teams confidence that their real-time pipelines won’t become their next incident. Because in the end, CDC isn’t a feature. It’s infrastructure. And infrastructure only works if it’s invisible when you need it to be and transparent when things go wrong.

Struggling with slow Databricks queries? Here's a quick overview of the 6 key Delta Lake optimization techniques that can cut your query times from hours to minutes. 📋 𝗭-𝗢𝗿𝗱𝗲𝗿𝗶𝗻𝗴 It organizes your data like a well-sorted library. It clusters related data together on disk, reducing the amount of data Spark needs to scan. When to use: → Multi-column filters in your queries → Tables with 2-4 frequently queried columns → Data that doesn't change often When not to use: → Single column queries (regular sorting works better) → Tables with frequent writes (overhead too high) → More than 4 included columns (diminishing returns) 📋 𝗖𝗼𝗺𝗽𝗮𝗰𝘁𝗶𝗼𝗻 Combines small files into larger ones to reduce metadata overhead and improve I/O. When to use: → Many small files (< 100MB each) → Streaming workloads creating lots of tiny files → Before running large analytical queries When not to use: → Files already optimal size (100MB-1GB) → Real-time streaming requirements → Limited compute resources 📋 𝗣𝗮𝗿𝘁𝗶𝘁𝗶𝗼𝗻𝗶𝗻𝗴 Physically organizes data into separate folders based on column values. When to use: → Clear filtering patterns (date, region, category) → Low cardinality columns (< 1000 partitions) → Queries consistently filter on partition column When not to use: → High cardinality columns (creates too many small files) → No consistent query patterns → Columns that change frequently 📋 𝗔𝘂𝘁𝗼 𝗖𝗼𝗺𝗽𝗮𝗰𝘁𝗶𝗼𝗻 Automatically triggers compaction during writes without manual intervention. When to use: → Streaming pipelines → Frequent incremental loads → Teams without dedicated optimization workflows When not to use: → Batch processing with controlled file sizes → Cost-sensitive environments (adds compute overhead) → Custom compaction logic needed 📋 𝗟𝗶𝗾𝘂𝗶𝗱 𝗖𝗹𝘂𝘀𝘁𝗲𝗿𝗶𝗻𝗴 Dynamically reorganizes data based on query patterns, adapting to your workload without manual tuning. When to use: → Evolving query patterns → Multiple clustering columns needed → Want to replace both partitioning and z-ordering When not to use: → Simple, stable query patterns → Tables smaller than 1GB 📋 𝗩𝗮𝗰𝘂𝘂𝗺 Removes old file versions to reclaim storage space. When to use: → After major data changes → Storage costs are high → Old versions no longer needed for time travel When not to use: → Need historical versions for auditing → Within retention period of critical data → During active read operations 🚩 𝗖𝗼𝗺𝗺𝗼𝗻 𝗠𝗶𝘀𝘁𝗮𝗸𝗲𝘀  1. Over-partitioning small datasets  2. Z-ordering on too many columns  3. Skipping vacuum in cost-sensitive environments  4. Using liquid clustering on tiny tables  5. Manual compaction on auto-compaction enabled tables  6. Z-ordering on high cardinality columns like IDs or timestamps  7. Not monitoring file sizes after optimization (missing the target 100MB-1GB range)  8. Applying same optimization strategy across all tables regardless of usage patterns Do you know other optimization strategies?

Most teams don't have a data problem. They have a governance problem. Data governance isn't about creating more policies, more approvals, or more documentation. It's about creating trust in the data people use every day. 𝐇𝐞𝐫𝐞'𝐬 𝐡𝐨𝐰 𝐦𝐨𝐝𝐞𝐫𝐧 𝐝𝐚𝐭𝐚 𝐠𝐨𝐯𝐞𝐫𝐧𝐚𝐧𝐜𝐞 𝐚𝐜𝐭𝐮𝐚𝐥𝐥𝐲 𝐬𝐭𝐚𝐫𝐭𝐬: → 𝐒𝐭𝐚𝐫𝐭 𝐖𝐢𝐭𝐡 𝐎𝐧𝐞 𝐑𝐞𝐚𝐥 𝐁𝐮𝐬𝐢𝐧𝐞𝐬𝐬 𝐏𝐫𝐨𝐛𝐥𝐞𝐦 Focus on broken dashboards, KPI mismatches, poor data quality, or access confusion. Solve a business pain point before building governance programs. → 𝐊𝐞𝐞𝐩 𝐆𝐨𝐯𝐞𝐫𝐧𝐚𝐧𝐜𝐞 𝐒𝐦𝐚𝐥𝐥 𝐚𝐭 𝐅𝐢𝐫𝐬𝐭 Start with one domain, a few critical datasets, and a clear source of truth. Successful governance scales incrementally. → 𝐃𝐞𝐟𝐢𝐧𝐞 𝐂𝐥𝐞𝐚𝐫 𝐎𝐰𝐧𝐞𝐫𝐬𝐡𝐢𝐩 Every dataset should have an owner, a steward, and a platform team responsible for reliability and access. Accountability eliminates ambiguity. → 𝐒𝐢𝐦𝐩𝐥𝐢𝐟𝐲 𝐀𝐜𝐜𝐞𝐬𝐬 𝐌𝐚𝐧𝐚𝐠𝐞𝐦𝐞𝐧𝐭 Adopt role-based access, least-privilege principles, data masking, and audit trails. Security should enable productivity, not block it. → 𝐌𝐨𝐧𝐢𝐭𝐨𝐫 𝐃𝐚𝐭𝐚 𝐐𝐮𝐚𝐥𝐢𝐭𝐲 𝐂𝐨𝐧𝐭𝐢𝐧𝐮𝐨𝐮𝐬𝐥𝐲 Track freshness, completeness, duplicates, schema changes, and pipeline health. Trust is built through consistency. → 𝐒𝐭𝐚𝐧𝐝𝐚𝐫𝐝𝐢𝐳𝐞 𝐁𝐮𝐬𝐢𝐧𝐞𝐬𝐬 𝐌𝐞𝐭𝐫𝐢𝐜𝐬 Align definitions for revenue, retention, conversions, and operational KPIs. Most reporting conflicts come from inconsistent definitions. → 𝐊𝐞𝐞𝐩 𝐃𝐨𝐜𝐮𝐦𝐞𝐧𝐭𝐚𝐭𝐢𝐨𝐧 𝐏𝐫𝐚𝐜𝐭𝐢𝐜𝐚𝐥 Document purpose, ownership, source systems, update schedules, and sensitivity levels. The best documentation is the documentation people actually use. → 𝐂𝐫𝐞𝐚𝐭𝐞 𝐚 𝐆𝐨𝐯𝐞𝐫𝐧𝐚𝐧𝐜𝐞 𝐎𝐩𝐞𝐫𝐚𝐭𝐢𝐧𝐠 𝐑𝐡𝐲𝐭𝐡𝐦 Review incidents weekly. Audit access monthly. Improve standards quarterly. Modern governance isn't a technology project. It's an operating model for trusted, scalable, and AI-ready data. The organizations that move fastest aren't the ones with the most governance. They're the ones with the clearest governance. PS: Governance should reduce friction, not create it. If your governance process slows delivery more than it improves trust, it's time to simplify. Follow Ashish Joshi for more insights

Critique this (real) team's experiment. Good? Bad? Caveats? Gotchas? Contexts where it will not work? Read on: Overview The team has observed that devs often encounter friction during their work—tooling, debt, environment, etc. These issues (while manageable) tend to slow down progress and are often recurring. Historically, recording, prioritizing, and getting approval to address these areas of friction involves too much overhead, which 1) makes the team less productive, and 2) results in the issues remaining unresolved. For various reasons, team members don't currently feel empowered to address these issues as part of their normal work. Purpose Empower devs to address friction points as they encounter them, w/o needing to get permission, provided the issue can be resolved in 3d or less. Hypothesis: by immediately tackling these problems, the team will improve overall productivity and make work more enjoyable. Reinforce the practice of addressing friction as part of the developers' workflow, helping to build muscle memory and normalize "fix as you go." Key Guidelines 1. When a dev encounters friction, assess whether the issue is likely to recur and affect others. If they believe it can be resolved in 3d or less, they create a "friction workdown" ticket in Jira (use the right tags). No permission needed. 2. Put current work in "paused" status, mark new ticket as "in progress," and notify the team via #friction Slack channel with a link to the ticket. 3. If the dev finds that the issue will take longer than 3d to resolve, they stop, document what they’ve learned, and pause the ticket. This allows the team to revisit the issue later and consider more comprehensive solutions. This is OK! 4. After every 10 friction workdown tickets are completed, the team holds a review session to discuss the decisions made and the impact of the work. Promote transparency and alignment on the value of the issues addressed. 5. Expires after 3mos. If the team sees evidence of improved efficiency and productivity, they may choose to continue; otherwise, it will be discontinued (default to discontinue, to avoid Zombie Process). 6. IMPORTANT: The team will not be asked to cut corners elsewhere (or work harder) to make arbitrary deadlines due to this work. This is considered real work. Expected Outcomes Reduce overhead associated with addressing recurring friction points, empowering developers to act when issues are most salient (and they are motivated). Impact will be measured through existing DX survey, lead time, and cycle time metrics, etc. Signs of Concern (Monitor for these and dampen) 1. Consistently underestimating the time required to address friction issues, leading to frequent pauses and unfinished work. 2. Feedback indicating that the friction points being addressed are not significantly benefiting the team as a whole. Limitations Not intended to impact more complex, systemic issues or challenges that extend beyond the team's scope of influence.

A machine operator taught me more about ERP in one hour than I had learned from 25 years of tech consulting. I thought ERP was about technology. Then a factory worker proved me wrong. We were rolling out a new ERP module at a manufacturing plant. Flawless on paper. ✓ Clean data. ✓ Tight integrations. ✓ Seamless training. But something was OFF. The line supervisor pulled me aside and said... “This system might be perfect for your screen… but it’s slowing us down.” So I shadowed one of the operators for an hour. No slides. No dashboards. Just observation. What I saw changed how I approach ERP forever. 𝐇𝐞𝐫𝐞’𝐬 𝐰𝐡𝐚𝐭 𝐭𝐡𝐞 𝐨𝐩𝐞𝐫𝐚𝐭𝐨𝐫 𝐬𝐡𝐨𝐰𝐞𝐝 𝐦𝐞: 🚩 He had to memorize 6 extra clicks to complete a job entry. 🚩 The screen timeout was cutting him off mid-task. 🚩 The dropdowns weren’t sorted by frequency of use. 🚩 The barcode scanner worked technically, but needed two hands—slowing the line. And the kicker? → They stopped trusting the system because of one bug that went unresolved for weeks. → Not once did he mention “data models,” “integrations,” or “APIs.” But what he showed me was everything ERP is meant to be about: ☑️ Process. People. Flow. Friction. Trust. That day, I stopped thinking like a consultant. I started thinking like a user. 🧠 And here’s the uncomfortable truth we don’t talk about in ERP consulting ⤵️ ↳ A factory worker’s 3-second delay is more expensive than a 3-month delay in your data migration. ↳ If you want ERP success, don’t just talk to the CIO or the CFO. ↳ Go to the floor. Watch the workflow. Listen to the friction. 𝐁𝐞𝐜𝐚𝐮𝐬𝐞 𝐫𝐞𝐚𝐥 𝐄𝐑𝐏 𝐢𝐧𝐬𝐢𝐠𝐡𝐭𝐬 𝐝𝐨𝐧’𝐭 𝐜𝐨𝐦𝐞 𝐟𝐫𝐨𝐦 𝐝𝐢𝐬𝐜𝐨𝐯𝐞𝐫𝐲 𝐜𝐚𝐥𝐥𝐬. 𝐓𝐡𝐞𝐲 𝐜𝐨𝐦𝐞 𝐟𝐫𝐨𝐦 𝐛𝐨𝐨𝐭𝐬, 𝐠𝐥𝐨𝐯𝐞𝐬, 𝐚𝐧𝐝 𝐠𝐫𝐞𝐚𝐬𝐞 𝐬𝐭𝐚𝐢𝐧𝐬. Are we spending too much time perfecting the tech and not enough time watching how humans actually use it? Would love to hear your thoughts. P.S. The image is created from ChatGPT and is for representation purpose.

Recently helped a client cut their AI development time by 40%. Here’s the exact process we followed to streamline their workflows. Step 1: Optimized model selection using a Pareto Frontier. We built a custom Pareto Frontier to balance accuracy and compute costs across multiple models. This allowed us to select models that were not only accurate but also computationally efficient, reducing training times by 25%. Step 2: Implemented data versioning with DVC. By introducing Data Version Control (DVC), we ensured consistent data pipelines and reproducibility. This eliminated data drift issues, enabling faster iteration and minimizing rollback times during model tuning. Step 3: Deployed a microservices architecture with Kubernetes. We containerized AI services and deployed them using Kubernetes, enabling auto-scaling and fault tolerance. This architecture allowed for parallel processing of tasks, significantly reducing the time spent on inference workloads. The result? A 40% reduction in development time, along with a 30% increase in overall model performance. Why does this matter? Because in AI, every second counts. Streamlining workflows isn’t just about speed—it’s about delivering superior results faster. If your AI projects are hitting bottlenecks, ask yourself: Are you leveraging the right tools and architectures to optimize both speed and performance?