How to select the best LLM and AI strategy for your organization?

Organizations are increasingly recognizing the potential of generative AI technologies like ChatGPT, shaping their policies around these advancements. This strategy often leads to policies that are a response to the AI-related disruptions experienced in late 2022. Now, as the initial adjustment phase concludes, there is a golden opportunity to explore how AI can be smoothly implemented into organizational frameworks.

This exploration leads to crucial questions, such as identifying the most effective AI strategies to bolster an organization and choosing the best AI solutions for specific needs. Moreover, crafting a bespoke AI strategy could be key to securing a competitive advantage or meeting unique requirements, beginning with the careful selection of suitable Large Language Models (LLMs).

Selecting the ideal LLM for an organization is a considerable challenge. The AI landscape is broad, with countless options designed to cater to the unique needs and use cases of any organization. Following consultations with CEOs and CIOs, we recommend six critical steps to initiate this process.

six steps to select the best LLM for your organization

Step: 1: Define your AI strategy

Interacting with generative AI presents both opportunities and challenges. The potential for innovation is immense, yet there exists the risk of overlooking market opportunities or falling into indecision. Thus, the primary challenge for organizations is to identify where to begin. Considering AI's disruptive potential, creating a specific AI strategy that complements the broader corporate strategy is advisable. An AI strategy defines how AI will benefit the organization, its customers, and stakeholders, emphasizing realistic assessment of the organization's capabilities.

McKinsey's three AI archetypes—Taker, Shaper, Builder—offer valuable perspectives on organizational approaches to AI adoption. Does your organization rely on readily available AI solutions, thus being a 'Taker'? Or does it act as a 'Shaper', customizing AI Platforms as a Service (PaaS) for unique AI applications? Perhaps it's a 'Builder', venturing into developing proprietary Large Language Models (LLM)?

It's important that organizations grasp their current standing, available options, and the steps required to effectively utilize AI technology. This must be done with a keen awareness of their capabilities and strategic goals.

Step 2: Form an Artificial Intelligence advisory board

Before advancing with the AI strategy, it's crucial to ensure that all key stakeholders are thoroughly briefed and aligned with its direction. To ground this approach, forming an Artificial Intelligence Advisory Board with key stakeholders is essential to foster a unified vision. Transparency and realism about the strategy's benefits and potential limitations are imperative. Emphasize the importance of open dialogue through essential presentations and workshops for the advisory board. At this juncture, clear and effective communication is paramount. Furthermore, considering the ethical implications of artificial intelligence is paramount. Notably, there are exemplary guidelines on this matter, such as Microsoft's responsible AI practices, which provide an outstanding reference point.

Step 3: Establish the criteria for selecting your LLM

It's essential to grasp that Large Language Models (LLMs) vary greatly in their architecture and functionalities. The true intricacies and nuances pose the biggest hurdles. AWS provides a set of criteria dimensions as an initial guide.

• Accuracy: Measure of accuracy in response, completeness and coverage of facts

• Speed: Measure of time to first byte and complete result

• Economics: Measure of the cost to host and invoke LLM

• Transparency: Measure of hallucination in responses and accuracy of citations and sources.

• Responsibility: Measure and management of security, privacy and governance.

Large Language Models (LLMs) are being incorporated into a wide variety of technological platforms, raising an important question about their compatibility with your current technical infrastructure and principles. At this stage, the architectural team is crucial in establishing the criteria and ensuring alignment with contemporary architectural principles.

Step 4: Evaluate your Large Language Models

By now, you should have narrowed down your choice of Large Language Models (LLMs) to a select few that have the capability to enhance your AI strategy and meet the established criteria dimensions effectively. Several strategies can be employed to rank the chosen LLM. The majority of these ranking methodologies adopt a version of the ELO system, which is similarly utilized for ranking chess players.

The LMSYS chatbot arena offers an impressive initiative to rank Large Language Models (LLMs) by leveraging a public platform focused on assessing LLMs' performance. This platform has amassed more than 500,000 human preference votes, employing the Elo ranking system to accurately evaluate LLMs according to user feedback.

You can also perform your own LLM ranking by leveraging Arthur Bench, an open-source Python library that integrates seamlessly with LLM PaaS solutions such as AWS Bedrock, or with LLM integration frameworks like LangChain. The ranking process effectively engages stakeholders and garners widespread support for the subsequent steps to be taken.

Step 5: Outline your strategy for implementation

Every decision taken in the previous steps should be meticulously recorded and serve as the cornerstone of your strategy for implementation. Building on the previous steps, here are some potential scenarios that could shape your implementation strategy.

• Scenario A: Opt for the standard Large Language Model (LLM) solutions provided by your current cloud supplier. Although this may appear as a less-than-ideal outcome, there are numerous compelling reasons to remain with your existing provider, including considerations related to architecture, security, knowledge, SLAs, and administrative processes.

• Scenario B: Choosing a Private LLM. Numerous organizations encounter challenges when applying proprietary data to public LLMs. Nevertheless, there are options available for operating LLMs within your company's private cloud network, ensuring it is safeguarded from external access. For example, with LocalAI, you have the opportunity to operate your preferred Large Language Model (LLM) within a Docker container.

• Scenario C: Selecting one or more Large Language Models (LLMs) is crucial, as different LLMs deliver varied outcomes. Therefore, it's likely that integrating more than one LLM will best meet your company's needs. In these cases, a Platform as a Service (PaaS) solution, like AWS BedRock, provides a solid base to implement your multi LLM strategy effectively. Furthermore, LLM integration frameworks, such as LangChain, enable the seamless 'chaining' of these LLMs within a business application, enhancing functionality and efficiency.

• Scenario D: Not choosing a LLM, opt for one of the shelve solutions. One possible result of the preceding steps is the realization that your use case may not be as unique as initially thought. Upon careful consideration, a Software as a Service (SaaS) solution with integrated AI might emerge as the optimal choice. Another approach could be to choose ready-made AI solutions categorized by functionality, such as Text-to-Speech or Intelligent Search. For example, Azure AI offers a comprehensive suite of AI solutions that can scale with a customer's evolving AI proficiency, ranging from machine learning to low-code virtual assistant solutions.

Step 6: Assess the results and iterate as necessary

Regardless of the method chosen for AI deployment, it's critical to quickly identify and address any associated risks. Microsoft outlines four essential phases to be integrated into the AI application lifecycle:

1. Identify: Pinpoint and prioritize potential harms your AI system might cause by engaging in iterative red-teaming, stress-testing, and thorough analysis.

2. Measure: Quantify the frequency and severity of these harms by setting clear metrics, developing measurement test sets, and conducting iterative, systematic testing—both manually and automatically.

3. Mitigate: Reduce harms using strategies and tools like prompt engineering and content filters. After implementing mitigations, repeat measurements to gauge their effectiveness.

4. Operate: Establish and carry out a plan for deployment and operational readiness.
   

In conclusion, the journey of incorporating Large Language Models (LLMs) into your organizational framework is both intricate and rewarding. By meticulously following the outlined steps, from evaluation to the implementation strategy, organizations can not only leverage the cutting-edge capabilities of LLMs but also ensure that their deployment aligns with the company's values and technical requirements. The final assessment phase is crucial for maintaining the integrity and effectiveness of your AI systems, allowing for continuous improvement and adaptation to emerging challenges. With careful consideration, strategic planning, and ongoing assessment, your organization can harness the power of LLMs to drive innovation, enhance productivity, and create more meaningful interactions. Adopting such advanced technology is not just a leap towards future-proofing your business but also a commitment to responsible and impactful use of AI.