In 2023, two major themes shaped much of the coverage of generative AI: Its unique capabilities in customer experience (CX) and unified communications (UC) and how enterprises should protect their data. No Jitter’s (NJ) first Conversations series dove into these topics, as did several articles in the Number of the Month series.
During the first quarter of 2024, No Jitter’s second Conversations series asked industry leaders to talk about how generative AI can boost productivity – and how we define what productivity even is – with the goal of helping those charged with evaluating and/or implementing Gen AI to have a better sense of which technologies will best meet the needs of their organizations and customers.
Generative AI (Gen AI) refers to the category of large language model (LLM)-powered solutions that can be used to automate tasks, generate content, and potentially improve decision-making. Gen AI-powered solutions have been integrated into contact center as a service (CCaaS), unified communications as a service (UCaaS), collaboration tools and document creation products.
Gartner’s January 9, 2024 report, AI’s Impact on the Employee Experience, made several points relevant to how AI may impact employee productivity. In this coming world where (generative) AI is used every day – hence “everyday AI” – Gartner believes that Everyday AI will help:
Garner goes on to list some other specific value associated with the introduction of Gen AI, but also states that Gen AI “spawns nontrivial compute costs, with vendors seeking to pass on costs, which sometimes represent significant uplifts from the base application price.”
Simply put, Gen AI productivity equates to ‘time saved.’ This is not how economists define productivity, but it is how most UCaaS, CCaaS and collaboration vendors describe the benefits associated with using a Gen AI assistant. Usually, this ‘time saved’ refers to the rote aspects of work – sending a simple email, drafting documents and presentations, summarizing a meeting or chat channel. Gen AI assistants can also summarize the results of an internal knowledge base search or the results of Internet search.
So, the vendor position is that by using a Gen AI assistant, a knowledge worker can save X minutes per day. And because of that time saved, the knowledge worker can do other things that boost revenue for their company, help more customers, or add value in some other way.
Most of these capabilities benefit knowledge workers, which is a term coined by Peter Drucker. And it’s because of this that defining productivity is so difficult – knowledge work isn’t as easy to quantify and measure as other occupations. It’s comparatively easy to define productivity for a mason, for example, but more difficult for a novelist.
These and other related topics will be tackled in the collection of articles below. We have provided short summaries of many published articles, whitepapers and academic articles (as well as links to the original content) that all provide varying definitions and datapoints that help answer the question: How does generative AI increase productivity for businesses?
According to this IBM article, “a knowledge worker is a professional who generates value for the organization with their expertise, critical thinking and interpersonal skills.” The IBM article goes on to state that knowledge workers are different from information workers. According to IBM, an information worker applies information to perform a task while “in the hierarchy of today’s workplace, knowledge workers oversee[s] the daily work of the information worker.”
So the opportunity for Gen AI-powered solutions, according to IBM, is to enable knowledge workers to “quickly gather information about a topic, search for solutions to business problems and flesh out innovative ideas.”
According to J. P. Gownder, vice president, principal analyst with Forrester, “At this point Gen AI is more about removing the drudgery associated with knowledge work.” And by knowledge work, Gownder is referring to the work product of the approximately 90 million U.S. workers who “sit behind desks” for the majority of their workday – those in business and financial operations occupations, management, computer and mathematical jobs, legal occupations, office and administrative support roles, etc. Forrester uses the occupation data and categories provided by the U.S. Bureau of Labor Statistics for its knowledge worker estimates.
The Ramírez and Nembhard paper states that productivity measurements for manual workers are mostly based on output or throughput – total, per hour, etc. By contrast, the authors state that “knowledge work is not easily observable or measurable.”
The paper goes on to present a summary of various frameworks that can be used to measure KW productivity, some of which include:
Gen AI can play a role across several of these points, perhaps by reducing the amount of resources and time it takes to produce the required outputs – and then by replicating, and systematizing, the "output creation process."
In their research paper, authors Erik Brynjolfsson, Danielle Li and Lindsey Raymond, studied the staggered introduction of a generative AI-based conversational assistant using data from 5,000 customer support agents. Access to the tool increases productivity, as quantitatively measured by issues resolved per hour, by 14 percent on average, with the greatest impact on novice and low-skilled workers, and minimal impact on experienced and highly skilled workers.
Erik Brynjolfsson is, among other things, a professor and Senior Fellow at the Stanford Institute for Human-Centered AI (HAI). He has written extensively on AI, particularly the use of generative AI in the contact center (one of his studies will be referenced later).
In this Microsoft WorkLab Podcast, Brynjolfsson made several interesting points the first being that technologies that imitate humans tend to drive down wages; technologies that complement humans tend to drive up wages.
“Don’t just look at your existing processes and think, oh, how can I replace this worker with a piece of software or an AI? It’s okay to drive down labor costs. I mean, it’s great for us to be able to get things cheaper. But there’s way more upside in doing new things, or delivering things in an entirely different way,” Brynjolfsson said. “That takes a little more creativity on the part of managers but ultimately leads not just to more total output and more value created, but also leads to more broadly shared prosperity because you’re keeping humans as part of the production process and not replacing them.”
According to a global survey of 700 CIOs and other senior IT leaders in enterprises with more than 2,000 employees commissioned by Ardoq in partnership with Slalom and conducted by Coleman Parkes between March and April 2024, the average enterprise spends $43.4 million annually on emerging technology projects.
Despite these expenditures, 69% of CIOs say predicting the ROI on such investments is little more than a ‘finger in the air’ exercise. Other findings include:
A November 2023 report from Gartner, How Generative AI Impacts Knowledge Management, argues that generative AI can turn normal interactions (unstructured data) into structured knowledge assets which allows a company to “capture institutional and subject matter knowledge in external, persistent forms.”
This could involve a subject matter expert talk about what they know into a webcam. AI can then transcribe that talk and an LLM can summarize it, categorize it and format it for the knowledge base. As the report states, this process “allows knowledge resources from across the enterprise to be used as raw material for Gen AI” which can then be repurposed for a wide range of use cases – customer service, internal enterprise search, etc.
The report also acknowledges that LLMs can improve “knowledge workers’ productivity by reducing the significant amount of their time that they spend searching for and making sense of information.”
In July 2023, McKinsey published a report (Generative AI and the future of work in America) that forecasted that “by 2030, activities that account for up to 30 percent of hours currently worked across the US economy could be automated—a trend accelerated by generative AI.” McKinsey wrote that Gen AI is likely to enhance the way the way STEM, creative, and business and legal professionals work rather than eliminate many of those jobs. However, automation (broadly defined) may cause employment to decline in other job categories such as, office support, customer service, and food service employment.
McKinsey wrote that, “When machines take over dull or unpleasant tasks, people can be left with more interesting work that requires creativity, problem-solving, and collaborating with others. Workers will need to gain proficiency with these tools and, importantly, use the time that is freed up to focus on higher-value activities.”
In the 2023 report A New Era of Generative AI for Everyone, Accenture found that across all industries 40% of all working hours can be impacted by LLMs like GPT-4. “This is because language tasks account for 62% of the total time employees work, and 65% of that time can be transformed into more productive activity through augmentation and automation.”
Released in late 2023, Cisco’s inaugural AI Readiness Report developed the Cisco AI Readiness Index that is based on a double-blind survey of “8,161 senior business leaders at organizations with 500 or more employees with responsibility for AI integration and deployment within their organizations.”
Cisco’s report includes both generative and predictive AI. This summary of Cisco’s report highlights survey results that relevant to measurements of ROI and productivity associated with AI-based solutions.
In late 2023, AWS partnered with Access Partnership to survey 3,297 employees and 1,340 organizations in the United States (US) across industries. The following are some of the key findings from that survey:
In September 2023, a group of academics and consultants from Harvard Business School, The Wharton School, Warwick Business School, MIT Sloan School of Management and Boston Consulting Group (BCG) released a working paper titled Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge.
This report shares the methodology and findings of a study which the authors conducted. It involved 758 consultants at BCG. The experimenters created three test groups: No AI access, GPT-4 AI Access or GPT-4 AI Access with a prompt engineering overview. The study participants were randomly assigned to each group and then the experimenters created a set of “18 realistic consulting tasks within the frontier of AI capabilities.”
The authors said that these were “complex tasks, selected by industry experts to replicate real-world workflows as experienced by knowledge workers. Most knowledge work includes this sort of flow, a set of interdependent tasks, some of which may be good fit for current AI, while some are not. We examine both kinds of tasks.”
The study authors were testing the idea that some “tasks are easily done by AI, while others, though seemingly similar in difficulty level, are outside the current capability of AI.” This is their “jagged technological frontier.”
The study found that:
In March 2023, two MIT Ph.D. students, Shakked Noy and Whitney Zhang, released a working paper, "Experimental Evidence on the Productivity Effects of Generative Artificial Intelligence” which presented the findings of their experiment incorporating ChatGPT in mid-level professionals' occupation-specific writing tasks. Their experiment involved 444 college-educated professionals who were assigned two occupation-specific and incentivized writing tasks. The tasks involved writing 20–30-minute assignments designed to resemble real tasks performed in each occupation.
The experiment found that ChatGPT increased the output quality of low-ability workers while reducing their time spent, and it allows high-ability workers to maintain their quality standards while becoming significantly faster. But the experimenters noted that ChatGPT increased “productivity primarily by substituting for worker effort.” The authors acknowledged that the experimental tasks were short, self-contained and lacked a dimension of context-specific knowledge which may have inflated their estimates of ChatGPT’s usefulness.
In 2012, McKinsey released a report entitled “The social economy: Unlocking value and productivity through social technologies.” This report focused on the implementation of social platforms – YouTube, Facebook – which were new at the time. The report states that social technologies have the potential to raise knowledge worker productivity, streamline communications and collaboration and lower the barriers between functional silos.
Although the report’s particulars are obviously dated, there was one data point that was interesting because it shed some light on the amount of time knowledge workers spend on what could be termed ‘rote’ tasks: “Knowledge workers spend about 28 hours each week writing emails, searching for information and collaborating internally.”
This data point was cited in some of Glean’s online material (see No Jitter’s Conversation with Glean) and it also relates, indirectly, to Slack’s claim that use of Gen AI in its product saved users 97 minutes per week and that it’s latest Recap feature is a further time-saver.
In October 2023, Forrester’s J.P. Gownder (and his co-authors) published a report that presented a similar framework to the one presented by IDC but was focused on the business case for Microsoft Copilot. That report states that the business case “rests on the assumption that saving time is its core benefit, and that this frees up time to be productive on other tasks.”
Forrester’s model uses employee salary, assumed time savings from using Copilot, the variable impact of using Copilot by job tasks, to calculate the break-even point for different salary levels. Forrester’s modeling suggested that when deploying to “knowledge workers, the financial risks appear to be low even with conservative assumptions in mind.”
In part three of their three part series in which authors Brent Kelly and Kevin Kieller review the capabilities of the AI assistants offered by Zoom, Cisco, Microsoft, and Google, they observe three risks associated with using Gen AI in UCaaS: inaccurate information, productivity loss and data loss.
The authors state that while been an understandable focus on how Gen AI can speed up specific business processes, quickly creating detailed explanations, alleviating the need to take meeting notes, leveraging summarization to eliminate reading lengthy documents, chat, and email threads.
The authors note that there are cases where using Gen AI yields unacceptable results and then requires completing the process manually, taking a greater amount of time and effort than if one had just proceeded manually at the outset.
They suggest, then, that the “challenge is to track overall productivity increases or decreases to better understand those use cases when it does make sense to use Gen AI and how best to use it.”
As the preceding discussion shows, a great deal of work has gone into defining what productivity means for generative AI-powered applications. See this article for more on particular Gen AI applications, uses cases and how the technology has been implemented to date.
