industry: How Shopify built an AI stack that doesn't care which

Shopify has engineered an advanced AI stack designed for unparalleled resilience and flexibility, ensuring continuous operation regardless of changes or outages among large language model (LLM) providers. Central to this strategy is an internal LLM proxy, which automatically reroutes engineers to alternative AI services if a primary model becomes unavailable or undergoes updates. This innovative approach, detailed by Farhan Thawar, Shopify’s head of engineering, on a recent VentureBeat Beyond the Pilot podcast, prevented workflow interruptions when Claude Fable 5 unexpectedly shut down, seamlessly shifting users to Claude Opus or GPT 5.5.

This sophisticated proxy allows Shopify to purchase AI tokens in bulk, providing comprehensive usage reporting and enabling instant failover. Thawar emphasizes that this system is critical for enterprises to avoid being "super tied" to a single provider, offering a vital backup plan against disruptions. The ability to "spray across different providers" ensures that engineering workflows remain uninterrupted, protecting against the inherent volatility of the rapidly evolving AI model landscape.

Beyond vendor independence, Shopify heavily leverages AI model distillation to enhance efficiency and accuracy. This process involves training smaller "student models" on specific tasks, learning from larger "teacher models." These specialized small language models (SLMs) often outperform generalized, off-the-shelf LLMs for particular functions, such as those performed by Shopify's flagship AI assistant, Sidekick, which automates various merchant subtasks.

Thawar highlights significant benefits from using distilled SLMs, noting they can be two times faster and cheaper than larger models, with extreme cases showing up to a 30-fold improvement in cost and speed. Crucially, this strategy also prioritizes accuracy, ensuring that specialized models deliver precise results for their targeted applications. This balance of cost, latency, and precision is central to Shopify's AI philosophy.

Shopify's Universal Distillation Pipeline (UDP) streamlines the creation and deployment of these specialized models. Engineers input a teacher model, training data, evaluations, and a target model (e.g., distilling Opus 4.8 down to Qwen 3.5). The pipeline then runs for approximately a day, generating an evaluation report on the fine-tuned model's speed, cost, and accuracy for the specific subtask. If the trade-off is favorable, engineers can deploy the model without needing further approval, demonstrating an agile and efficient development cycle. The internal platform, Tangle, provides real-time visualization of this process.

Thawar envisions a future where the distillation pipeline becomes even more autonomous. His "dream" is for users to eventually provide only the teacher model, data, and evaluations, and let the system intelligently recommend the optimal distillation target model, potentially uncovering surprisingly small yet effective models that could even run on mobile devices. This forward-looking perspective underscores Shopify's commitment to pushing the boundaries of AI optimization.

Shopify encourages its engineers to move from "AI reflexivity" – using AI without deep thought – to "AI leverage," where they strategically integrate AI to maximize workflow benefits. To support this, the company provides access to various AI harnesses like Claude Code, Codex, and GitHub Copilot, allowing engineers to experiment and find tools best suited for their individual workflows.

To manage resource consumption and promote thoughtful AI use, Shopify implemented a detailed usage dashboard. This dashboard tracks not only token spend but also identifies users with high-cost token usage, time spent on reasoning tasks, and the types of models utilized across different disciplines. "Circuit breakers" are in place to flag unusually long-running models or high token consumption, prompting users with inquiries like, "Did you mean to spend this?" This proactive monitoring helps prevent accidental overspending and encourages mindful AI adoption.

This robust AI ecosystem is built upon Shopify's foundational philosophy of prioritizing infrastructure development. Thawar explicitly states, "We've always built more infra. We will continue to always build more infra." This commitment ensures that scalable, resilient systems are in place before features are deployed, forming a solid base for advanced AI capabilities.

Further illustrating its comprehensive AI strategy, Shopify also deploys internal AI agents such as River, which creates a "substrate of information" across the company, and OpenClaw, an agent that demonstrated impressive contextual awareness by deducing Thawar's travel plans from his calendar. These examples highlight the practical application and ongoing evolution of AI agents within Shopify's operational framework.

FAQ

Q: What is Shopify's LLM proxy and why is it important?

A: Shopify's LLM proxy provides engineers with access to multiple AI model providers, automatically switching between them if one experiences an outage or update. This is crucial for ensuring uninterrupted workflows, preventing vendor lock-in, and establishing a robust backup plan in the volatile AI landscape.

Q: How does Shopify leverage AI model distillation, and what are its benefits?

A: Shopify uses distillation to create specialized small language models (SLMs) from larger "teacher" models. These SLMs are tailored for specific tasks, offering significant improvements in speed (up to 30x faster), cost (up to 30x cheaper), and accuracy compared to generalized models. This strategy supports applications like Shopify's Sidekick AI assistant.

Q: What is the significance of Shopify's shift from "AI reflexivity" to "AI leverage"?

A: The shift from "AI reflexivity" to "AI leverage" encourages engineers to think deeply and strategically about how AI can best be integrated into their workflows, rather than using it without critical consideration. Shopify supports this by providing diverse AI tools, monitoring usage, and implementing "circuit breakers" to promote mindful and efficient AI adoption.