Building the Foundation: Technical Components for Enterprise GenAI Success

As enterprises begin exploring generative AI, understanding the core technical components becomes essential—even for non-technical leaders. This article serves as a primer for anyone new to GenAI, breaking down the key building blocks in accessible terms. Whether you’re a business leader making investment decisions, a developer implementing your first GenAI project, or a product manager planning AI features, understanding these fundamental components will help you navigate the rapidly evolving landscape with confidence.

The Basic Building Blocks of GenAI Systems

At its core, any enterprise GenAI application consists of four main components:

1. Models: The AI Engine

Large language models (LLMs) like GPT-4, Claude, or Gemini form the foundation of any GenAI application. These models have been trained on vast amounts of text data and can generate human-like text based on the prompts they receive.

Patrick Debois, in his 2023 DevOps Enterprise Summit presentation “Bringing GenAI from Promise to Reality,” highlights several important considerations when selecting models:

• Context Window Size: This determines how much text the model can process at once. Smaller models might handle only 2,000 tokens (roughly 1,500 words), while larger ones can process 100,000+ tokens. This affects how much information you can feed into the model at once. 
• Performance and Speed: Models vary significantly in how quickly they respond. For real-time applications like chatbots, speed matters tremendously. For batch processing done overnight, it may be less important.
• Cost Structure: Models typically charge based on the number of tokens processed—both input and output. As Debois notes, costs can add up quickly, especially for large-scale deployments.
• Capabilities: Different models excel at different tasks. Some are better at creative writing, others at code generation, and others at precise factual responses.

When selecting models, organizations need to weigh these factors against their specific use cases. A customer service application might prioritize speed and accuracy, while a content generation tool might value creativity and fluency.

2. Data Integration: Grounding AI in Your Business Reality

One limitation of foundation models is that they only know what they were trained on—which typically doesn’t include your organization’s proprietary information. Connecting these models to your business data is essential for making them useful.

The most common approach is called Retrieval Augmented Generation (RAG). Damon Edwards and colleagues explain in the paper Enterprise GenAI Delivery Patterns that RAG works by:

1. Indexing your enterprise content (documents, knowledge bases, databases).
2. Finding relevant information when a user asks a question.
3. Including that information in the prompt sent to the model.

This approach has several advantages:

• It keeps your data secure, as it’s only retrieved when needed.
• It ensures responses reflect the most current information.
• It reduces hallucinations (made-up answers) by grounding responses in real data.

Organizations are increasingly treating RAG capabilities as a platform service. As Edwards and colleagues note, “A centralized RAG system can be provided to the team to vectorize and store data through an API.” This allows different teams to leverage organizational knowledge without building their own infrastructure.

3. Orchestration and Integration: Connecting the Pieces

To build useful applications, GenAI needs to be integrated with existing systems and workflows. Patrick Debois explains that this often involves:

• APIs and Function Calling: Enabling models to trigger actions in other systems, like searching a database or creating a ticket.
• Middleware: Tools like LangChain and LlamaIndex that simplify connecting models with data sources and other systems.
• Orchestration: Managing the flow of information between different components.

Anand Raghavan from Cisco emphasizes the importance of orchestration in his 2024 Enterprise Technology Leadership Summit presentation: “When the user asks a question, how do you understand the intent behind that? And based on that intent, you might want to route it to a different model service that handles the question differently.”

For example, a customer support application might route product questions to a knowledge base, billing questions to a payments system, and complex problems to human agents—all using the same conversational interface.

4. Safety and Governance: Ensuring Responsible Outputs

The final critical component is ensuring AI systems behave appropriately. Debois highlights two key elements:

• Input Guardrails: Preventing users from asking harmful questions or trying to manipulate the system.
• Output Filtering: Ensuring responses are appropriate, accurate, and aligned with organizational values.

Edwards and colleagues note that many organizations are creating “Guardrails-as-a-Service” that can be used across multiple AI applications. These provide consistent safety checks without each team having to reinvent the wheel.

Building an Enterprise Platform for GenAI

As organizations move beyond initial experiments, many are taking a platform approach to streamline development and ensure consistency. John Rauser from Cisco explains that this helps avoid “an explosion of tools and instrumentation” that can quickly become unmanageable.

Other key platform capabilities highlighted in the paper “Enterprise GenAI Delivery Patterns” include:

1. LLM Proxy Services

A centralized service that manages connections to LLM providers offers numerous benefits:

• Authentication and access control.
• Usage tracking and cost monitoring.
• Consistent logging for audit trails.
• Caching to reduce redundant requests and costs.
• Standardized error handling.

This approach also makes it easier to switch between different model providers as the technology evolves.

2. Data and Knowledge Services

Centralizing how organizational knowledge is accessed and managed helps ensure:

• Consistent data security and access controls.
• Optimized retrieval strategies.
• Reuse of common knowledge bases.
• Higher-quality results through shared improvements.

As the authors note, this is particularly valuable when dealing with sensitive data that requires careful handling.

3. Evaluation and Feedback Systems

GenAI systems improve through feedback, both automated and human. Platform services can help by:

• Tracking user satisfaction with responses.
• Identifying potential issues through automated checks.
• Collecting examples where the system performed well or poorly.
• Supporting continuous improvement through structured testing.

Real-World Architecture Examples

Anand Raghavan from Cisco provides a comprehensive view of what production architecture looks like in practice, in his recent ETLS presentation. Their approach includes:

• Data Layer: Specialized stores for structured, unstructured, vector, and time-series data.
• Model Infrastructure: Runtime environments, hosting, repositories, and training pipelines.
• Feedback Management: Systems to capture user feedback and improve models over time.
• Model Evaluation: Measuring performance against quality metrics.
• Usage Analytics: Understanding how users interact with AI capabilities.

This layered approach allows for flexibility while maintaining governance and control.

Making Strategic Technology Choices

Organizations face critical decisions about their technology strategy. John Willis warns that making the wrong choices can lead to significant technical debt: “Some things we build will be an instant legacy, as technology evolves so fast.”

Key decision points include:

Proprietary vs. Open Source Models

Organizations must choose between:

• Commercial APIs (OpenAI, Anthropic, cloud providers): Typically higher quality but with per-token costs and potential data privacy concerns.
• Open-source models (Llama, Gemma, Mistral): More control and fixed costs, but potentially lower quality and higher operational complexity.

Patrick Debois notes that many organizations are taking a hybrid approach, using different models for different use cases based on their requirements.

Build vs. Buy for Infrastructure

For components like vector databases, orchestration tools, and evaluation frameworks, organizations must decide whether to:

• Build custom solutions: More control and customization, but higher development and maintenance costs.
• Adopt existing tools: Faster implementation but potentially less flexibility.

Some organizations often benefit from using existing tools for common needs while investing in custom solutions for their unique requirements.

Conclusion

Building a solid technical foundation for enterprise GenAI requires careful consideration of models, data integration, orchestration, and safety mechanisms. By understanding these core components, organizations can make informed decisions about their technology strategy and build systems that deliver real business value.

As Patrick Debois concludes in his presentation, the GenAI landscape requires ongoing attention to “the security of the data, what goes in and the model, the quality.” Organizations that invest in robust, flexible infrastructure while maintaining strong governance will be best positioned to derive sustainable value from these powerful technologies.

In our next article, we’ll explore how enterprises are structuring their teams and governance processes to effectively manage AI initiatives at scale.

- About The Authors

Leah Brown

Managing Editor at IT Revolution working on publishing books and guidance papers for the modern business leader. I also oversee the production of the IT Revolution blog, combining the best of responsible, human-centered content with the assistance of AI tools.

Follow Leah on Social Media

•