Building Enterprise-Grade GenAI: Security, Scale & Trust

As organizations accelerate their adoption of generative AI (GenAI), one thing has become clear: enterprise-grade AI isn’t just about plugging in a foundation model and hoping for magic. It’s about building intelligent systems that are secure, scalable, explainable, and tightly aligned with business outcomes.

Recently, I had the opportunity to join an expert panel titled “Building Enterprise-Grade GenAI Applications: Balancing Innovation, Security & Scalability”—hosted by Data Science Connect and sponsored by AtScale, Vespa AI, Nexla, Informatica, and Skyflow.

Joining me were:

Saket Saurabh, Co-founder & CEO, Nexla
Jon Bratseth, CEO, Vespa AI
Manish Ahluwalia, Field CTO, Skyflow
Bheem Manta, Technical Architect, Informatica

What followed was a rich, multi-layered conversation that surfaced practical strategies for enterprise AI success—and the pitfalls to avoid.

Why GenAI Fails in the Enterprise Without Governance

Enterprise AI doesn’t operate in a vacuum. Unlike public tools that can afford to be flexible and experimental, enterprise environments are bound by strict regulatory, privacy, and security standards. The more data an enterprise has, the more constraints it faces. And when data is locked down due to lack of confidence in access controls, GenAI simply can’t deliver its full value.

What emerged in our discussion is that enterprises won’t win with GenAI unless they can safely unlock their data. That starts with strong governance—an architecture designed to ensure data is only seen and used by the right people for the right reasons. Without that foundation, most organizations will either stall progress or expose themselves to risk.

Accuracy and Trust: The Core Enterprise Challenge

A major difference between GenAI for consumers and GenAI for businesses is that enterprise decision-making has real consequences. In the world of BI, if a data point is wrong—if a number is off—there’s a significant impact. It could trigger poor decisions, break processes, or even cost someone their job.

We’ve put LLMs to the test at AtScale. When used in isolation, even with a best-in-class cloud data platform, they often produce inaccurate results. That’s because they lack the business context needed to understand how data should be interpreted and connected. Injecting that context through a semantic layer not only improves accuracy, but also provides the consistency and transparency enterprise teams demand.

The Semantic Layer as a GenAI Accelerator

Throughout the panel, it became clear that semantic context is the missing link in most enterprise GenAI architectures. When you build applications that understand your business’s specific metrics, hierarchies, and terminology, you not only improve the quality of results, but also how you manage access, apply governance, and maintain control.

This is a core principle in our work at AtScale. We map raw data to business concepts so teams can query and analyze with confidence—without ever compromising sensitive tables or losing control of how data is used. Semantic understanding becomes a mechanism for both trust and security, especially when enabling natural language query.

Scaling GenAI in Production: Why Architecture Matters

Moving from proof of concept to production requires more than fine-tuning prompts or experimenting with APIs. It demands architectural rigor. For GenAI to scale in the enterprise, it must be built on secure data foundations with clear access control and governance policies in place.

One point I emphasized during the discussion is that enterprises need to start with the data itself—ensuring it’s structured, secured, and accessible only by those with the right permissions. Without that baseline, no amount of GenAI innovation will hold up in production.

And as we scale, observability becomes critical. If users get inaccurate results, they’ll stop using the system altogether. Adoption is a direct signal of trust. If no one is querying your agent, it’s probably not delivering value. That’s why it’s essential to track usage patterns, evaluate feedback, and build mechanisms to continuously improve quality over time.

Future-Proofing Your GenAI Strategy

Looking ahead, we’ll see enterprise GenAI shift from knowledge tasks to process automation. As Bhim noted,

“We’re already seeing customers move from using iPaaS for deterministic processes to agentic applications where the LLM drives orchestrations.”

To make that leap, enterprises need semantic interoperability, model flexibility, and a platform approach that balances modularity with control. The semantic layer plays a pivotal role—not just for analytics but also for powering intelligent systems that interact with enterprise data safely and effectively.

Organizations need to embed natural language queries into their enterprise architecture with trust and accountability rather than treating natural language queries as a novelty. That starts with encoding business context into the AI stack, not expecting a general-purpose LLM to understand a company’s unique definitions, hierarchies, and KPIs.

The Road Ahead: GenAI Agents, Enterprise Integration & Business Value

Building enterprise-grade GenAI applications isn’t about chasing hype. It’s about delivering consistent, accurate, governed outcomes that the business can depend on. That requires collaboration across data, security, and engineering teams, and recognizing that context and control unlock true intelligence at scale.

To catch the full session or revisit the insights from our fellow panelists, you can watch the webinar recording here.