A claim worth examining: the majority of generative AI pilots launched in 2023 were logged as successful in corporate reports. Yet when I look at what is actually running in production at the start of 2024, the number is significantly smaller. The rest live in presentation decks or in a recorded demo prepared for the board. The problem is not that companies failed to experiment — they experimented extensively. The problem is that most companies measured AI success the same way they measured ERP go-live: is the system up? Are users logging in? When both answers were yes, the pilot was declared a win. But the value of generative AI does not live in uptime. It lives in business output. Closing that gap is the central management task of 2024 — and the organizations that treat it as a communication problem rather than a measurement problem will still be running pilots in 2025.Consider a representative case type I encounter frequently in Turkish manufacturing: a mid-sized metal components producer in Kayseri, 312 employees, exporting to five countries, that launched a RAG-based technical documentation assistant in mid-2023. The pilot ran for eight weeks, the demo worked without fault, and the IT director presented the system to the board. Three months later, the majority of engineers were not using it regularly. The reason was straightforward but entirely preventable: over forty percent of the firm’s technical documents had never been digitized — they sat in physical archives. The RAG architecture had correctly indexed the data it received. The data itself was incomplete. The demo succeeded because the demo dataset had been carefully curated. Production was a different environment, and no one had formally asked whether the production data was ready before go-live was declared. This is the pilot-value gap in its most common form.This case surfaces two foundational questions for generative AI governance in 2024. First: have you actually ingested the right data? Second: how are you measuring success? RAG is a genuinely useful architecture — it connects an organization’s own knowledge base to an LLM, reduces hallucination risk, and keeps sensitive data from leaving the company’s environment, which matters considerably under Turkey’s KVKK data protection law. But RAG produces results proportional to the quality of the underlying data. Deploying RAG on incomplete or unstructured archives is not an AI problem — it is a data readiness problem that AI cannot solve for you. The diagnostic question before any RAG deployment should be: what percentage of the relevant knowledge base is clean, current, and accessible in a format the system can index? If you cannot answer that number, the pilot is not ready to start.On measurement, the Kayseri case illustrates what happens when the wrong proxy metric is used. Had the right questions been set in advance, the team would have tracked: how many hours per engineer per week are spent searching technical documents? Did that figure change after six weeks? Did the rate of production errors caused by incorrect part specifications decline? None of these were tracked because the agreed success metric was system availability. The distance between ‘the system is running’ and ‘the system is creating value’ is precisely where the pilot-value gap widens. To close it in 2024, every AI initiative needs three things documented before a single model is deployed: which business metric must move, by how much, and what happens if it does not move within eight weeks. If you cannot write those three lines, the initiative is not ready.Turkey’s specific dynamics add complexity to this picture. The EU AI Act, formally adopted in April 2024, enters phased application from 2025 onward. Turkish companies that export to EU markets — particularly in textiles, automotive components, and processed food — will face compliance questions they have not encountered before. An Ankara-based enterprise software company serving European clients recognized this early and began classifying AI features by risk category during development; decision-support modules designated high-risk under the Act pass through separate review workflows before being released to EU customers. That decision did not come from legal obligation alone — the Act was not yet in force — but from the recognition that customer trust and regulatory exposure are not the same thing and that building for the stricter standard costs less than retrofitting later. KVKK’s overlap with AI training data remains unresolved in Turkish regulatory guidance; sector-specific clarifications from the Personal Data Protection Authority are expected during 2024, and this uncertainty is keeping several companies’ fine-tuning plans in a holding pattern.Small language models, or SLMs, are gaining practical attention in this context — but they deserve a calibrated assessment rather than enthusiasm. Models in the 7B-to-13B parameter range can run on on-premises servers, send no data to external APIs, and present a more defensible posture under KVKK. For well-scoped, domain-specific tasks — classifying inbound customer requests, extracting structured data from a fixed document format, answering questions from a curated internal knowledge base — they perform adequately and at a fraction of the API cost of larger models. The cost argument is not trivial: for Turkish companies, large-model API pricing in US dollars carries a currency risk that SLM deployment eliminates. But SLMs are not substitutes for larger models across the board. Multi-step reasoning, long-context legal or technical analysis, and high-quality multilingual output still require larger architectures. The right question is not ‘which model is best’ but ‘which model is sufficient for this specific task’ — and that question should be answered before procurement, not after.AI agents deserve particular caution. Vendor messaging in 2024 frequently overstates what agentic systems can do autonomously. In practice, what an agent can reliably perform depends entirely on which systems it is connected to, at what permission level, and whether there is a documented human escalation path for edge cases and failures. The right set of questions before deploying any agentic workflow: what can this agent execute without human review, what requires explicit human approval, and how are errors logged and attributed? Without clear answers to all three, an agent system creates accountability gaps — gaps that, for companies operating in the EU market, can trigger high-risk classification under the AI Act. The experimental era is over. Every system now needs a legible accountability chain: who approved this output, what data was it based on, and who is responsible if it is wrong? Companies that can answer those questions for their AI systems in 2024 are building something durable. The ones still measuring success by demo quality are preparing for a second year of pilots.The Kayseri engineers who went back to their paper archives were not resisting technology. They were working around a system that had been declared production-ready before the production environment was actually understood. That is the honest diagnosis of where most generative AI programs stand at the start of 2024. The path from pilot to measurable value is not paved with better models or larger budgets — it is paved with the discipline to define what value means before the first prompt is written.
This article was originally published in Turkish by Gökhan MERCANOĞLU on January 1, 2024. The English edition has been reviewed and edited by the author.