Insight Tech

Christina Cardoza: Hello and welcome to the IoT Chat, where we explore the latest developments in the Internet of Things. I’m your host, Christina Cardoza, Editorial Director of insight.tech. And it’s that time of year again where we’re going to look at the top IoT and AI predictions for 2024. And joining us again on this topic we have Martin Garner and Bola Rotibi from CCS Insight.

So for those of you who aren’t familiar or didn’t get to hear our last podcast with these two experts, let’s get to know them a little bit before we jump into the conversation. Bola, I’ll start with you. What can you tell us about what you do at CCS Insight?

Bola Rotibi: I’m the Chief of Enterprise Research at CCS Insight. I have a team that covers software development, workplace transformation, cloud, AI of course, security—quite a whole load, but everything to do with the enterprise.

Christina Cardoza: Great. And Martin, welcome back to the show. What can you tell us about CCS Insight and yourself?

Martin Garner: Thank you, Christina. Well, yes, I’ve been at CCS Insight for 14 years. I lead the work we do in IoT, mostly focusing on the industrial side, and so work very closely with Bola on those areas.

Christina Cardoza: Great. And, Martin, since you’ve been at the company for, like you said, 14 years now, we’ve worked a bit over the last couple of years. You guys always have an IoT prediction you put out at the end of the year, and we’ve always worked to talk about that and to see what’s coming next.

So I’m wondering, in the last year or even the last couple of years there’s been a big push to edge and AI. So I’m wondering how that impacted this year’s predictions and what we see coming for 2024.

Martin Garner: Sure. Well, obviously 2023 was a huge year for both of those areas. Edge was already growing strongly coming into this year, and I think 13 months ago ChatGPT had just launched and has clearly had a massive impact during this year and has really opened the world’s eyes to what AI can do—and quite a few things that it can’t do yet. So I think it’s fair to say that the world is now thinking rather differently about the two areas, and we are too.

So in our predictions last year we had lots about edge and some AI. I think for anyone who downloads the booklets at the end of the podcast you’ll see that we really dialed up the AI predictions this year. And it’s important: it’s not just generative AI like ChatGPT, and it’s not just what individuals can do with AI.

And there’s one prediction I’d like just to give as a little example of that, which is that by 2028 a major healthcare provider offers its customers a digital twin service that proactively monitors their health. Now today there is a lot of data about you scattered around—so we have online health records in many countries; we have fitness bands; we have smart watches and they show activities, sleep, and so on. There’s lots of data on your phone, but it’s all a bit messy and it’s not joined up at all.

So we think that healthcare providers over three to four years will start to combine these sources. We’ll use AI and we’ll basically do what the industrial guys are already doing, which is predictive maintenance on people. And of course the aim is early intervention gives smaller intervention, and it’s usually cheaper intervention too. The outcomes are just better if you get there earlier. So that’s an example of how we think AI will start to develop from here.

Christina Cardoza: Yeah. And I love what you said—we’ve learned a lot about what we can actually do with AI, or what’s possible. There’s always that hype or that trend that everybody wants to jump on right away. So I want to dig a little bit into that.

But before we get there, just this movement to—with AI and this movement to edge and having, wanting to get information at real time and faster—there’s been a lot of movement also to the cloud with all of these solutions and applications. And I know there’s always that cloud-edge versus on-premise debate. Will that continue? Or will people continue to move to the cloud over the next couple of years, or is on-premise still going to remain steady?

Martin Garner: Well, it’s a good question, and I think that debate is going to remain live for a good few years. And I want to draw out just a couple of ways of that. So one is quite short term. I think in many countries there are fears about where the economy’s going. We’re not quite out of all the pandemic effects yet. So one prediction is that recession fears push workloads from the cloud to on-premises through 2024.

Now, the best candidates for that are companies which are using hybrid cloud. It’s not so much those who’ve gone all-in on the cloud. And there are a few new initiatives, like HPE GreenLake and Dell APEX, which bring the flexible consumption model that we have in the cloud—they bring it to hardware and allow you to do that at the edge as well. So it may be that a hardware refresh in some of those companies is a good opportunity to rationalize some aspects, sort of repatriate some of their workloads to on-premises and take some cost savings while they’re doing it. So that’s the short term.

Longer term, we think there are several areas where there are sort of pendulum swings, like insourcing versus outsourcing, and edge to cloud could be one of those. And it’s partly also that there are new tools and different economics coming up in both areas. So it’s all changing quite fast.

But one area where we have another prediction is that a repricing of cloud providers’ edge computing services takes place by the end of 2028. Now, what does that mean really? Well, the big cloud providers, they all have edge computing services, but actually the public cloud is the big part of the business. And the edge services, they’re sort of on-ramp for the public cloud, and they’re priced to do that. But the forecasts for edge computing, they show it growing bigger than the public cloud over five years, possibly a lot bigger than the public cloud.

Now, if that happens, then we’d be in a position where the big part is subsidized by the smaller part, and that makes no sense at all. So we really expect the edge-cloud services, their prices to be moved upwards over three to five years, and that could be a significant price rise. So we think many industrial companies might want to consider their own edge computing, just in case—put themselves in a better position so they have more options. It’s edge as a hedge, if you like, and kind of gives them more options about how they go forwards.

Christina Cardoza: Yeah, absolutely. And that’s a great point. Companies, they don’t want to be locked in with a vendor. Like you said, more opportunities. They want to make sure any investments they make—because they do make a lot of investments to move to these solutions to build these—they want to make sure any investments they do make they can future-proof them. I think a lot of it driving it too is the type of solution that they’re building. If it’s more critical workload or they need more security, they tend to move on premise. If they need real-time analytics or they need faster results, that’s sometimes the cloud and the—where the edge computing comes in.

So I want to talk a little bit about AI development with you, Bola. What can we expect in terms of the AI solutions or the developments happening over the next year within enterprises?

Bola Rotibi: Well, I think—I mean it’s a good question actually, because I think we’re going to expect quite a lot, and I think, whilst 2023 has been kind of a year of—we’ve seen a year of launches, especially from many of the IT solution providers, there’s been a wealth of new tools. I mean, obviously when ChatGPT launched last in the end of November 2022, it really spawned massive interest, and we’ve seen it exponentially grow, especially with generative AI.

But I would say that the development of AI has actually been happening for quite some time, because it’s been machine learning and various other sort of models and algorithms that people have used behind the scenes. So anything that you’ve been using on your mobile phone to search through pictures and all those, those are kind of like AI solutions, AI models. So this AI itself is not new, right?

But what we are seeing is the power of generative AI, especially as we think about the productivity. You know, everyone has sort of latched on to the conversational aspect of generative AI and its ability to simplify even sort of complex queries and bring back concise information that is relevant information. So everyone’s jumping on that because we see that as a productivity solution.

So what we’re seeing now is a launch of lots and lots of solutions. Over the last year we’ve seen pretty much every provider—Intel amongst those as well—bringing out generative AI capabilities, as well as beefing up their own AI solutions. But one of the things that we’ve seen here, we’ve seen Microsoft launch with the Copilot AI-powered assistance. We’ve seen others from other providers like AWS with Amazon Q.

But I think what we’re—what one of the things that we do think is that despite all this frothiness around the interest, we did have a prediction which actually said AI investment and development will accelerate in 2024, it does accelerate in 2024—despite calls for caution. And we’re already seeing the throughput of this already. And the reason we see this is that actually quite a few of the main protagonists, I mean over the recent months have said, “Well, hold on just a minute. We need to kind of slow this down. Things are moving rather fast.” And people are a bit worried about security; they’re worried about whether the regulations are out there, whether they are effective enough. And all of this caution is still kind of wrapped around.

But at the same time I think there’s a real thirst to get AI, to develop it, to really get their hands on it. Because people have been blown away by the new experiences and the engagement levels that they can have, especially with generative AI. So I think there’s going to be even more acceleration next year.

Christina Cardoza: Yeah, absolutely. And a lot to dig in there. I want to start off with the generative AI piece. And, like you mentioned, the possibilities—it brings so many benefits that a lot of companies are trying to jump on this, and I always hear in the back of my head, one of our guests on the podcast was saying, “You want to create solutions that solve a problem, not create solutions just to create a solution, just to jump on the latest trend.”

So I’m curious with generative AI, because obviously, we’re seeing all of these companies talk about it—all of these releases and with AI toolkits adding generative AI capabilities, there’s been a lot of hype. What’s the reality? What do you think will actually come out of this space in the next year, and what can we be looking for? What are the applications or use cases that are possible?

Bola Rotibi: Oh gosh, that’s lots and lots of questions. Well, first of all I think there is a lot of froth around with generative AI, but that’s because everyone’s been, as I said earlier, been blown away by the productivity that they’ve seen from it and actually the intuitiveness of it all, and just really being able to talk or put together natural language queries and then get back concise information.

That said, I mean, one of our predictions that we outlined is that despite all of the excitement around generative AI and the fact that we’ll see lots and lots of new tools, we do think that, 2024, we’ll see some level of slowdown—partly because as people kind of get to the grips with the reality of the costs, some of the risks and the complexities that really, that become, start to be exposed and have been starting to be exposed this year, I think that we’ll see a slow, a slight slowdown.

But, despite that, I mean I’ve—it’s a bit of a thing where you’re sort of saying on one hand it’s going to be really, really great and really fast. On the other hand we’re going to see some slowdown. What I think, as with any technology, there will be people just trying all sorts of capabilities.

Nowadays you can go in and you can start typing in queries, and then immediately get very, very concise answers back. And what we’re also seeing is that we’re seeing generative AI in multiple perspectives. So we’re seeing it as an assistant to providing information to relevant information across lots and lots of different data types. We’ve also seen it being able to create, generate code that is very good quality—that at least is a starting point, especially the experts to finalize it and sort of do a little bit more quality control.

So I think we’re going to see—so on one side I think 2024 is really where people get to start to play with it properly. And the tools, they’ll get to start to understand some of the limitations of the tools, as well as what they’re going to be able to achieve. So I think it’s going to be, on one hand the hype of 2023 will start tempering down into much more of a more level-headed sort of approach to it and more nuanced, as well as the excitement of actually delving into some of the capabilities, like generated code. We’ll start seeing it across different types of workplace solutions, helping knowledge workers, but also helping expert professionals—whether those be expert developers or other professions as well.

Christina Cardoza: Yeah, it’s interesting because we may see a slowdown in some of the hype or everybody jumping onto this trend. But, like you mentioned, we’re going to see an increase in acceleration in AI investments maybe in other areas. And, like you mentioned, that we’re going to see an acceleration despite the need to maybe take a slowdown to take a step back and look how we’re developing things.

And, Martin, I know we’ve spoken in conversations outside of the podcast that AI oversight committees, this might be something that is coming. So can you talk a little bit more about what that means? How can we develop, or how do you anticipate ethical AI initiatives to spring up as this investment and this acceleration continues?

Martin Garner: Well, yes, of course. And you’re right, we touched on it before. It’s a big subject and we could talk all day about that. We’re not going to. So the short answer is, yes, there’s going to be a lot more of that. And I think you already said it, Christina, that AI does have the potential for many, many good uses in society. But also used wrongly it has the potential to do a huge amount of damage, and it’s a bit like medicine, where regulated drugs are generally good for society and unregulated drugs like opioids and fake Wegovy and things, generally not so good.

And the big difference with medicine is that there’s no professional body, there’s no Hippocratic oath. You can’t be struck off as an AI practitioner, at least not yet. And so at the moment instead we have the opposite, where the AI leading companies, they seem to be on a big push. As soon as something new is developed they open source it and push it out into the world as fast as possible. And that obviously puts a huge imperative on suppliers and developers to take their own ethical stance in how they use it—which customers are they going to deal with or not deal with, how to train their staff, internal governments. There’s lots to get right there. But it also puts an onus on companies who are using AI as customers, and they need to step up too.

And so we do have a prediction, which is that AI oversight committees become commonplace in large organizations in this coming year, 2024. And those are likely to be committees of ethics experts, AI experts, legal advisors, data scientists, HR, representative from the different business units, and so on. And they’re going to have to review the use of AI across the company and in the company’s products. And their job really is to bridge the gap between the tech teams who are all engineers—they’re not typically ethicists—and the organization and its goals and what it wants to do with it.

And we think that’s going to be quite a significant overhead for a lot of companies. Quite difficult to get right. Lots of training to come up to speed and to stay on top of it, because it’s all moving so fast that the committee will have to move fast too. And all that because the AI industry is largely not doing a good job of self-regulation.

Christina Cardoza: One thing I’m curious about is the EU is putting together the AI Act, and I’m wondering what will this mean for developing solutions with the GDPR? You know that that was in the EU, but that was a global impact. So do you anticipate the same thing happening when the AI Act has passed?

Martin Garner: I think probably Bola is the better one to talk on that, if I may. Can I hand that one to Bola? I know you’ve been looking—

Bola Rotibi: Yeah, no. Yes, I think definitely actually, because at the end of the day I do think that the regulators are coming together. The EU has been first out the door for starters with the EU AI Act. But when the act does come in, it will be like the GDPR Act. And I think what will happen, and we’ve already seen the ratification of the Digital Markets Act, and we’ve already seen the effects of that with some of the companies who are being highlighted are still trying to get bedded in into whether they are the main gateways.

But I think when it finally does come through, I think there will be sort of a bedding-in process as people try to get used to it, try to understand what it means, what all the constructs, see all of the rules and regulations. So there will be kind of like teething problems, but I think it will actually, it will become a thing, a regulation for people to rally around.

But the EU is not the only one. We’ve got the acts in the US, and we’ve also got the UK pushing to having a really strong play with AI regulations. And then we’ve got China and in other regions as well. So I think we’re going to start seeing some level of improvement in terms of—towards the end of 2024, certainly, as people rush out to see the frameworks regulation. And I think that will be really important.

The other thing that’s actually happening, and it’s not just about regulation at the international level and the national level, but it’s also what the industry is doing. And I think there’s some really exciting things that have been happening. Recently IBM and Meta and Intel amongst—as part of 50 organizations that have launched the AI Alliance, which is aimed to bring the industry together to work on collectively—like any other body—to standardize, like working groups, to bring working groups together, to come up with ideas for strategies and approaches to handling certain AI challenges and opportunities, to be the home or the hub for interactions between end users as well.

Christina Cardoza: I love seeing all of these AI efforts, especially from big tech companies, because AI has a lot of benefits, it has a lot of opportunities and ability to transform industries and transform our lives for the better. But there is that level of uncomfortability from end users—how their privacy is going to be used, how secure the data is, what’s it going to do, how it really is going to transform their lives. So I think this helps put a safeguard and minimize some of the risks that people think that are out there.

And I’m curious—because obviously we can’t wait for some of these acts or some of these efforts to be passed or to take off before we start developing in a more ethical way—so I’m just curious, Bola, you work with developers. How can they continue to develop these AI solutions with privacy and ethics in mind and ensure the safety and security of their solutions and of the end users?

Bola Rotibi: Well, I mean it’s a good one actually, because I think invariably what’s going to happen is that I think lots and lots of—how to say—there’s likely to be oversight committees within organizations. That’s something that we’ve actually put down in the prediction of ours. But I think one of the things is that there will be communities that will work within organizations to kind of understand what it is that they can do. Many of the tools that they are actually using now and many of the providers are building in that responsible AI, ethical AI, from the ground up.

What developers have to then start thinking about—and this isn’t just on the developers, because actually at the end of the day a developer might think, “Well, actually, I’m building code, I’m generating—I’m building a code, I’m writing an application.” But at the same time, in the same way, that security is actually down to everyone. Everyone in that workflow. So is responsibility—responsible AI and an ethical approach. It’s down to everyone.

So I don’t see it’s just a developer requirement, but at the same time you have to say there needs to be frameworks in place, and that needs to be driven around the whole organization. It needs to come from the ground up. It needs to go from the top down. So there needs to be some principles that are used and distributed and circulated across the organization. And there needs to be some sort of guidelines.

Now, the tools are coming with guidelines and guardrails, right? In terms of, “Okay, how do I prevent certain things happening?” Because you can’t expect anyone who’s always developing to have everything in their heads about, “Oh, okay, is this ethical?” Of course, you could ask yourself, “Well, just because I can do it, should I do it?” You know? And that’s always the case.

But, at the same time, if you want a level of consistency, we need to have, provide, those guidelines and to the organization, to the development organization, right across the board. But there needs to be guardrails. And I think many of the tools are recognizing this so that they can actually allow organizations of any size to put their own policies in.

So I see going forward that there will be a layered approach. So that may be an oversight committee within the organization that thinks about what it is that the organization is from an ethical standpoint, from a responsibility standpoint, and start building policies. And those policies will be driven and put into the tools so that they act as guardrails. But there’s also going to be guidance and training of developers in terms of taking an ethical approach, taking a responsible-AI approach.

So I think, going forward, it’s a bit like how do you do right, knowing right from wrong? But this is something that the development community already is aware of. There’s a lot of things out there, a lot of programs, a lot of initiatives out there, things like aiming for doing code for good. Lots of organizations have been thinking about impact, about sustainability, and all those kind of things. So there is a body, sort of a body, a wealth of already ideas and initiatives to make the people think at multiple levels, not just about responsible AI, but doing the right thing, thinking about sustainability. So I think the approach is actually is already there for many developers, but they do need help.

Christina Cardoza: Yeah, and I love what you said: it’s not—one camp is not responsible for this. It’s really a collaborative effort from everybody building it, everybody using these types of solutions, everybody touching AI. So I love that.

We are running a little bit out of time, but before we go I just want to change the conversation to a different topic real quickly while I have Martin on the podcast, because Martin, we’ve talked over the last couple of years: We’ve been talking about 5G. Are we looking to 6G? Where are we in the 5G efforts? Is it still beginning? So I’m just curious, as we’re talking about AI and Edge, where 5G fits into all of this, what the impact of AI is going to be on f 5G networks, and when is it time to start looking at 6G?

Martin Garner: Yeah, 6G, I love that. We’re not, we don’t quite have all the good bits of 5G yet, do we? They’re coming, but they’re not quite here yet. So—but there is work going on 6G.

So in terms of AI and 5G I think the first thing is that organizations who are starting to use 5G in their factory, in their warehouse, in private networks up and down the country and so on—one of the things 5G will do is enable a lot more use of AI thanks to the very high capacity, the time-sensitive networking location services it will bring in. And we’ll see a lot more AI in use around those domains. A lot more autonomous vehicles and things. We can already see good examples of autonomous trucks used in mines in Latin America and in ports in many different continents and so on. Lots and lots more of that to come with 5G and the newer bits of 5G, which are nearly here, as one of the key sort of enablers of that.

But I think the other interesting bit is the impact of AI on the network itself. Now, there are several tricky aspects if you try to buy and use 5G. So things like coverage planning on public and private networks. Very good candidate for using AI to make that simpler so that more people can do it. Also, 5G networks, they’re complicated things. They really are a very high number of settings and things. And so the whole optimization and management is a big deal in a 5G network.

And we have a prediction around that, which is that AI enables 5G networks to move beyond five-nines availability, which anyone who’s used a cellular network will appreciate the importance of that. And that would come through by analyzing traffic patterns and ensuring that the network is set up best to handle that type of traffic, identify problems, predictive maintenance, and configure the network—if things are going to go wrong—so it has graceful degradation or even become self-healing.

And we think there’s a lot more that networks themselves can do with AI to become much better quality and really support not only people up and down the country on the public networks, but also really the OT world—which absolutely, if they use it a lot, they will depend on it, and it’ll be very expensive when things go wrong. So supporting that requires that degree of quality.

Anyway, then 6G. It is a tiny bit early for 6G, but work is going on of course. And over the next five years or so we’re going to be building 6G networks. We think 2030 is going to be a bit of a headline year for 6G. And we have a few 6G predictions. Here’s one: which is that by 2030 the first 6G-powered massive twin city is announced. And we think that cities will be a great showcase and massive twinning is one of the best use cases, because all the layers of a city could be potentially included in the model there. And the 6G network could enable a sort of full representation of the whole city environment.

We don’t think that’s likely to start with older cities—much more with a new project such as they’re doing in Saudi Arabia at the moment. And if we do that, they’re going to need 6G just for the sheer volume and speed of the data in real time that runs through a city. So that was a really good example. We think 2030, big headline year for that.

Christina Cardoza: Yeah, absolutely. And I agree, we still have to fix or finish the 5G aspects that we have there, but of course you have businesses or companies out there already trying to talk about 6G. And I agree that for all of these AI solutions and benefits and opportunities we want to take advantage of we need to make sure the infrastructure is there to make it possible first.

Martin Garner: Yeah, and working as well as it can.

Christina Cardoza: Yeah, absolutely. So this has been a very big conversation. Obviously there’s lots to look forward to. And we have only touched a little bit of the research paper, the CCS Insight IOT predictions that come out every year. We’ve only touched a small subset and this has already been such a big conversation.

So before we go I just want to throw it back to you guys. Are there any final thoughts or key takeaways that you want to leave our listeners with today? Bola, I’ll start with you.

Bola Rotibi: Well, yes, I think the one thing that I do feel that is going to become even more prominent, if people haven’t already identified it yet, which is—and we have a wonderful prediction about this—which is that proficiency in generative AI is a common feature of job adverts for knowledge workers by 2025. And I think it’s actually going to be probably that you could stretch that out to more than knowledge workers—probably for everyone.

And the reason we say that is because what we’ve seen, this swathe of recent announcements for generative AI, we’ve seen them being embedded into workplace tools such as Microsoft Office, Google Workspace—all of the different types of solutions that are out there. I mean, Martin actually mentioned earlier that pretty much you now get generative AI capability right across the board—almost any tool you open up and there’s a generative AI chat box there.

And I think the thing is we have data already showing that the actual—the more proficient you are, i.e., the better that you are in creating those prompts, which means if you’re trained as well, the better the productivity and the better the quality. And I think people are going to recognize that that is going to be a real factor to having an efficiency factor and an effectiveness factor for the workplace, workflows, and especially if you’re trying to streamline your workflows, make them more effective.

So I think that proficiency side is going to be a real big thing. So my advice is to start looking at how best to build out those prompts, and taking some sort of training support in that. It’ll follow the way of many technologies—you know, the better you are with them, the more effective that you can get them to be.

Martin Garner: And I think there is already a job title called Prompt Engineer, isn’t there? I’ve heard about this.

Bola Rotibi: Exactly, yes.

Martin Garner: And what I think you’re saying, Bola, is that that’s going to evaporate and everybody’s going to be good at it.

Bola Rotibi: Yeah, exactly. I mean, prompt engineering is going to be like, well, that’s it. Just another engineering. But the thing is is that we can joke about it, but I think really in the same way that—and it’s going to be a little bit more than just being good at search terms. I think everybody’s going to have to really sort of learn how best to construct your prompts to get the best insights and the best information out of it.

Christina Cardoza: Yeah, absolutely. It’s almost like the AI developers, you don’t really call them AI developers anymore because everyone’s developing for AI. Everyone needs to know how to develop for AI, so they’re just developers. But, Martin, anything you want to leave us with today?

Martin Garner: No, no, I think that sort of wrapped it up really quite nicely. Thank you, Christina.

Christina Cardoza: Yeah, absolutely. Well, thank you guys again. It’s always a pleasure to have both of you on the podcast, and it has been a great, insightful conversation. I invite all of our listeners to take a look at the CCS Insight 2024 prediction for IoT. We will host it and make sure that we make a link available to our listeners to check that out. And I want to thank you guys for listening to the podcast. Until next time, this has been the IOT chat.

The preceding transcript is provided to ensure accessibility and is intended to accurately capture an informal conversation. The transcript may contain improper uses of trademarked terms and as such should not be used for any other purposes. For more information, please see the Intel^® trademark information.

This transcript was edited by Erin Noble, copy editor.

Christina Cardoza: Hello and welcome to the IoT Chat, where we explore the latest developments in the Internet of Things. I’m your host, Christina Cardoza, Editorial Director of insight.tech, and today we’re going to be talking about generative AI trends and opportunities with Waleed Kadous from Anyscale, Teresa Tung from Accenture, and Ramtin Davanlou, also from Accenture. But before we get started, let’s get to know our guests. Teresa, I’ll start with you. What can you tell us about yourself and Accenture?

Teresa Tung: I’m our Cloud First Chief Technologist, where I have the best job because I get to predict the next generation of impactful technologies. In this role I’ve gotten to work on big data, edge computing, AI, and lately you can bet it’s generative AI almost all the time.

Christina Cardoza: Great. That’s not an easy job you have there—having to predict what’s coming next or stay on top of the trends. But excited to get a little bit in to this conversation with you.

Before we go there, Ramtin, you’re also from Accenture. But what do you do there, and what can you tell us about your role at the company?

Ramtin Davanlou: Sure. Thanks a lot for having me, Christina. I’m a Director from Accenture Data and AI Group, based out of San Francisco, and I also serve as the CTO in the Accenture and Intel partnership, where we have a big focus on scaling GenAI applications and solutions with Intel hardware and software.

Christina Cardoza: Yeah, absolutely. I know Intel, the hardware and the software—they’re powering a lot of the use cases and the opportunities that we’re going to get into in just a moment.

But before we get there, Waleed, last but not least, what can you tell us about yourself and Anyscale?

Waleed Kadous: Yeah, so my name’s Waleed Kadous. I’m the Chief Scientist at Anyscale. Anyscale is a company that makes it easy for companies to scale up their machine learning infrastructure, with a special focus on scaling up generative AI—both the training, fine tuning, and inference stages. And we help our customers to deploy applications that run at huge scale.

We’re the company behind an open source project called Ray that’s very popular for doing distributed computing. It’s used by OpenAI and Cohere to train their models. So, yeah, it’s a really exciting job I have helping customers to work out how to deploy really cutting edge technologies, including IoT.

Christina Cardoza: Yeah, absolutely. And I can imagine open source—that behind a lot of the use cases it’s helping developers and businesses get started, and just the community out there to work together and build upon some of the successes that we’re seeing in that community.

So, like Teresa mentioned, generative AI—it seems to be the next big thing. We keep hearing about it; at all the latest conferences we’re talking about generative AI. A lot of it is around ChatGPT, so we’re not exactly sure—it’s a new thing, not exactly sure what this is. So, Ramtin, I’d love to start there, if you can give us a little explanation about, when we’re talking about generative AI and especially the opportunities out there for businesses, what do we—what is this idea of generative AI, and what are really the opportunities businesses are looking for?

Ramtin Davanlou: Yeah, yeah, sure. I think everybody has used generative AI or knows about it by now—unless they’ve been living off grid in the past year and this is the first content that they watch after they come back. But I think I can just put it in terms of its potentials and a summary of what is this basic entity that we have created. And it’s like—this is especially true when, over Thanksgiving dinner, all the conversation is centered around generative AI. So that’s really, really popular, and everybody is trying to make sense of it.

And so what a lot of companies like OpenAI, Google, AWS, and many more were able to do was they use their massive compute resources and massive data sets—a majority of the texts that you can find on the internet or on specific subjects—and they train these AI models—aka large language models—capable of generating new content. And this content is in many different forms. It is text, images, video, voice, or even code, computer code. And text is especially important because that’s what all humans basically do in communication, right?

So it boils down to text, and many of these AI models are able to generate responses that are really good on any given topic—better than an average person or even an average expert on that topic, right? And that creates a lot of new opportunities. Companies can now take these models, fine tune them a little bit so that the model behaves in certain ways and gains knowledge more about a specific context. And if you think about most of what white collar labor is doing every day at work—the conversation that we are having right now, it is basically text that we are generating, right? And that is the means of communication and building net new knowledge.

So what these LLMs cannot do now, but may soon be able to do, is creating that net new knowledge. But they can do everything that has been created, and that poses an opportunity for us to just focus on that net new value that we can generate. So the current workforce will focus on creating net new things.

And companies are thinking about how to use this entity to kind of enhance or substitute a lot of things that we are doing these days—sending emails, creating slides, right? All of the content that you’re creating to be able to communicate with each other. So this has huge implications for service industries and for manufacturing when you combine it with robotics. And at Accenture we are helping a lot of our clients to reinvent themselves using this transformative technology.

Christina Cardoza: Yeah. You mentioned the manufacturing spaces and other industries: it’s exciting when you have a capability like this that can really help transform all industries, and everybody’s using it in different ways. And I love how at the beginning of your response you talked about how it was dominating the Thanksgiving conversations much better than politics. But I’m sure—you know what? You can find, having these everyday conversations about this, is there comes a lot of misconceptions. A lot of interest comes with a lot of misconceptions.

So, Waleed, since you’re working in the open source world, you’re working with a lot of developers and people who are looking to develop, deploy, and manage these types of applications. Curious what you think. What are some of the considerations out there that businesses should be thinking about when developing these solutions? What are some of the misconceptions you’re seeing out there?

Waleed Kadous: Yeah. So in terms of the things that businesses should consider, I think one of them is to consider the quality of the output from these models. And in particular there’s a problem with LLMs called hallucination, and that’s where they may—they confidently assert things that are completely untrue. Now, over the last six months we’ve seen developments in an area called retrieval augmented generation which helps to minimize that. And we can talk a little bit about that, but making sure that the quality of the responses is accurate is one of the key considerations.

A second consideration is data cleanliness and making sure what is the information that these LLMs have access to? What are they disclosing, and what do they have the power to inform people of? Is there leakage between different users? Can someone reverse engineer the data that was used to train these models from the data themselves? It’s still a new frontier, and we’re still seeing issues that crop up in that front.

And then the final one is LLMs are expensive. I mean, really expensive. If you naively go and use GPT-4, which is probably the leading LLM out there today, you can easily blow a hundred thousand dollars in a month. So you really have to think about the cost and how you keep the cost under control.

And finally, there’s the final thing I would say is evaluation. How do you make sure that the system is actually producing high-quality results? What base data are you using to ensure that it’s doing the right thing? And that’s an area—I mean, the other misconception that we sometimes see people have is there’s this mechanism called fine tuning: let’s just fine tune and solve the problem. There’s a lot more nuance to solving problems with LLMs than we see, but there’s also a lot of potential applications.

And if I try to think about the applications, often when we talk to our users, they’re like, “Well, I can’t conceptualize what I need in terms of using an LLM.” And so we’ve worked out: these are the top things to look for in terms of use cases. The first one is summarization. Are there areas where you have a lot of information that you can condense, and where condensing it is useful? So we have a company called Merlin that provides a Chrome plugin that summarizes data.

The second is the retrieval-augmented-generation family. And that’s situations where you don’t just naively ask the LLM for questions; you actually provide it with context that helps answer the questions. So, for example, if it’s a customer support application, you wouldn’t just ask it about the customer support. You have an existing knowledge base of answers to questions that you pull the data from, and then say, “By the way, Mr. LLM or Mrs. LLM, here are the data points that you can use to help answer this question.”

I think one of the most interesting applications was what you might call “talk to the system” applications—especially interesting in IoT. So, imagine this as kind of a dashboard on steroids, a dashboard you can talk to. And I’ve seen a company that does expertly, that does Wi-Fi installations across companies for retailers, and what you can do is you can ask it questions, like, “Hey, which areas are we seeing the routers working too hard in?” And it will go, and it will query that information in real time and give you an update. And I really think that model is kind of the most interesting one for IoT.

And then the final one is really this in-context development and in-context application development. Perhaps the best known one of those is Copilot, right? When you’re writing your code, as Ramtin was talking about, it would give you suggestions about how to write even better, higher-quality code. And we’ve seen some of our companies deploying that. And, roughly, that order is the order of difficulty. In-context applications are the most difficult, but they’re also the highest potential.

And one thing I’d especially like to highlight is the idea that I know a lot of people are not quite sure: is this hype, or, is this not real? The really interesting thing to me is already we’re seeing massive proof points of the effectiveness of generative AI. We’ve seen GitHub do studies that show that it boosts the developer productivity by 55%. We’ve seen research about a Fortune 500 company that has shown that it can boost customer support quality and responsiveness by 14%. It’s just like, this is not kind of a hype thing. There’s already, even at this early stage, some incredible proof points that this works. But we’re still at the genesis of this technology, and we’re still collectively all learning how to use it.

Christina Cardoza: Yeah, absolutely agree. There’s always that concern: is the hype reality? Am I going to invest X amount of dollars in this and then tomorrow there’s going to be a new thing? So businesses are always making sure—are looking to future proof the investments that they are making. But, like you said, we have seen some proofs of concept, some success in the early days of experimentation. But, like you mentioned, cost is a big factor, and how can we go from those proofs of concept, experimentation solutions, and actually bring these solutions to life at production and at scale?

So, Teresa, I know Accenture does a lot of work with your end users trying to help them get to the next level of some of their efforts. So what can you tell us about where your customers are and what the next level is?

Teresa Tung: I would say, as an industry, most companies have started their proofs of concept, and many are starting with managed models like OpenAI. And these amazing general-purpose models address many use cases and offer a really great way to get started. But, as Waleed mentioned, cost in the long term is a factor, and this could be an order of magnitude bigger than many companies might be willing to pay. So, as generative AI pilots mature into production, companies now need to look at rightsizing that cost, rightsizing it for the performance, and even factoring in their sustainability goals.

When these models become more critical to the business, we’re also seeing companies want to take ownership and to control their own destiny, right? Rather than using a managed model, they might want to be able to take and create their own task-specific, enterprise-specific model. So these sub–10 billion parameter models are just more customized for them. And so what we’re seeing is companies beginning to adopt a number of models for different needs. So, yes, there will still be the general-purpose model available, but we’ll also have these fit-for-purpose models as well.

Waleed Kadous: So, to give a concrete example of what Teresa’s talking about: one of the experiments we did at Anyscale is we looked at translating natural language to SQL queries. And the general-purpose model, GPT-4, was able to produce an accuracy of about 86%, 80%. But by training a small, specific model that was only 7 billion parameters, that cost about one 100th the cost, we were able to achieve 86% accuracy in conversion. So this challenging mode of what are now being called SMSs versus LLMs—small specific models versus large language models—is kind of the evolving discussion that’s happening in the industry right now.

Christina Cardoza: Great. And we’ve been talking about these industries and these use cases at a high level, but I would love to dig in a little bit more and learn exactly how customers are using this—in what industries, where really are they finding that the biggest opportunities are, or the biggest areas that they want to start applying these generative AI solutions. So, Teresa, I’m wondering if you have any examples or use cases that you can share with us?

Teresa Tung: Yeah, I have a few. And I think Waleed had already done a great overview of some. I’m going to give a different perspective; I’m going to think about it in terms of things you can buy, things you’re going to boost, and things you’re going to build, right? So buying, being able to buy these generative-AI-powered applications for things like software development, marketing, some of these enterprise applications—that’s quickly becoming the new normal. So these applications use a model trained on these third-party data, and it gives everyone a jumpstart. And that’s the point—everyone is going to be able to capture these efficiencies, so don’t get left behind.

Boost is a second category, and this is where things like knowledge management or being able to apply a company’s first-party data—so, data about your products, your customers, your processes. And to do that you’re going to need to get your data foundation in order. And so using something like retrieval augmented generation is a great way to start, right? You don’t have to get a whole lot of data, and as you go along you can be able to create that data foundation.

And then, finally, in terms of build, we’re talking about companies being able to even maintain their own custom models. So, likely starting with the pre-trained open model and adding their own data to it. And this, again, gives them a lot more control and a lot more customization within the model.

Christina Cardoza: Great. And as we’re talking about building and boosting, deploying, managing these applications, and being able to work with large-scale models, it just comes to me that it’s—there’s a lot that goes into building these applications, and at insight.tech we’re always talking about “better together.” It’s usually not one company that’s doing this alone; it’s really an ecosystem or partnership that’s making this happen.

And, Ramtin, you talked about Intel hardware and software at the beginning of the conversation. I should mention, the “IoT Chat” and insight.tech, we are owned and sponsored by Intel, but of course it creates some of these great partnerships with companies like Anyscale and Accenture. So, I’m just curious, in terms of the hardware and the software, and the build and boost, how important it is to work with companies like Intel or any other partners that you have out there, Ramtin?

Ramtin Davanlou: Yeah, yeah, of course. I think partnerships are essentially very important in this area. Because if you are trying to build an end-to-end GenAI application and a developer ecosystem, you’re going to need a few of these suppliers, technology suppliers, to come together, right? And companies typically have to solve for a few things. This includes infrastructure and compute resources. You need a very efficient ML Ops, basically, tool to help you kind of manage this—everything you do, from development to managing and monitoring and deploying the models in production, right? And you also need a third-party software in a lot of cases, or open source software. A lot of clients are thinking about building larger platforms that could support several different use cases.

And this is an effort, like what Teresa mentioned and Waleed mentioned as well, to reduce the cost of this when you do this at scale. So instead of just using or building new platforms for every new use case, companies realize that they’re going to need this for many, many different use cases. So why not build a platform that you can reuse for all of these different cases? And this helps basically with total cost of ownership at the end of the day. And that means bringing several different technology pieces together, right?

For example, what we have built with Intel is a generative-AI playground, where we have used Intel Cloud, Intel Developer Cloud, and Gaudi tools—which is an AI accelerator specifically built for deep-learning applications, both training and—. So you can basically use GaudiTools in Intel Developer Cloud to fine-tune your models. But once you want to deploy that in scale you can go and use AWS, right? And that’s what we have done. In this playground you can basically bring in—do the development and fine-tuning of your models in IDC—Intel Developer Cloud—and then deploy at scale on AWS.

And we’ve used some of the Intel software like Converge, which is the ML Ops tool that you can use to make your data scientists and engineers collaborate in the same environment. And one of the big advantages of Converge is that it also allows you to use different compute resources across different cloud. So you can use compute resources in your on-prem environment, on Intel Developer Cloud, and on AWS all in the same workflow, right? Which is a huge advantage.

So at the time of deployment we realized that we need a tool that—or library that—helps us distribute the workloads. So if you’re getting 1,000 inferences per second, and then this has fluctuating demand—it goes up and down based on what’s going on, the time of day, and stuff like that—so you need to have a very efficient way to distribute the workloads. And we learned that this library called TGI from Hugging Face is very helpful. And that’s when you see there’s a lot of these different components and pieces that need to come together so that you can have an end-to-end GenAI application.

Christina Cardoza: Absolutely. It all goes back to the future proofing and protecting your investments, making sure that the hype is reality. I love, like you mentioned, being able to reuse some of the models or the applications and solutions you have out there—that’s always important.

So, Waleed, given that you are in the open source community and you are working with a lot of customers to make sort of these partnerships happen, I’m curious what you’re seeing. The use cases that you’re seeing, and how partners like Intel or any other partners you’re working with make those use cases a reality.

Waleed Kadous: Yeah, we’re definitely seeing a lot of interest in the different stages—the training, the functioning, and inference stages. So I think the points that Ramtin is making are valid, but one particular thing that has come up is this idea of open source. So there’s both open source models—so, for example, Meta has released a model called Llama 2 that we’ve seen very, very good results with. It’s maybe not quite on par with GPT-4, but it’s definitely close to GPT-3.5, the model one notch down.

And so there’s both open models and of course open source software. So TGI, for example, is not quite open source. It’s got a bit of a weird license, but there are systems like VLLM out of Berkeley, which is a really high-performance deployment system, as well as Ray LLMs. VLLM manages a single machine; Ray LLM gives you that kind of scalability across multiple machines, to deal with spikes and auto-scaling and so on.

We’re seeing a flourishing of the open source world in particular, because there are certain things that people like. Not everybody likes entrusting their entire data to one or two large companies, and vendor lock-in is a real concern. So we’re seeing people flock to open source solutions for cost reasons—they are cheaper—but mainly for flexibility reasons in terms of: I can deploy this in my data center, or I can deploy this in my own AWS Cloud and nobody has access to it except me.

And that flexibility of deployment and that availability of models at every size—from 180 billion down to 7 billion and below—these are the reasons that people are tending towards open source. And we’ve seen many of our customers—we did a profile of what it would take to build an email summarization engine, where if you used something like GPT-4 it would cost $36,000, and if you used open source technologies it would be closer to $1,000, for example.

And what that shows is not just that—it’s the other question, is are you really comfortable sending all of the emails that are in your company to any third party, whether it’s OpenAI or someone else. And so we’ve seen a lot of interest from all levels, from startups that tend to be more cost focused, to enterprises that tend to be more privacy- and data-control focused in open source models.

It’s not that open source models are perfect and open source technologies are perfect, it’s just that they’re flexible and you become part of a community. I think when Teresa was talking earlier about the—or you were talking, Christina, earlier—about this idea of building things together, that’s really the ethos behind the open source movement that we’ve seen, and we’ve really seen a lot of dynamism in that area, and every week there’s new models. It’s just a really, really dynamic space right now. And maybe open source models lag a little bit, but they’re continually improving at a very, very fast rate.

And if you look at the history of things like Linux and so on, you see that same pattern, that sometimes they lag a little bit, but just the breadth of applications that they end up being part of becomes the reason that people flock to these open source models. Just the fact that they’re part of a community that exists, that’s also one of the reasons that there are places like Hugging Face that are incredibly popular in the community as locuses of this open source movement.

Christina Cardoza: Absolutely. And I think, being in this open source space, it’s also when we talk about hype versus reality. There’s always the hype of what businesses want or what they think these solutions can do, and the reality of what they actually can do—not only if generative AI is hype. So I think being in the open source space you’re in an interesting area, where you can see the limitations. We’re still need to figure out exactly how to work with LLMs in all different types of use cases and scenarios and industries.

So I’m curious, Waleed, how do you think this space will mature? What do you think really needs to happen within the community—open source or outside of open source—to really make generative AI more mainstream?

Waleed Kadous: What we also see is open source models becoming easier to use. So, for example, Anyscale now offers a hosted version of many of these models that’s both price effective but also is identical to the OpenAI API. So you can literally just change a few variables and use it.

And that effort to make LLMs easier to use is one of the increasing trends. I think what will also happen is the continual looking of the life cycle. So really what we haven’t worked out so far is how to make LLMs better over time. If an LLM makes a mistake, how do you correct it? That sounds like such a simple question, but the answer is actually nuanced. And so what we’re seeing is a massive improvement in the evaluation and monitoring stages that companies like LangChain and LangFusion are really taking the lead on.

And then, finally, so far the focus has been on Large Language Models—text in, text out—but as Ramtin pointed out, we’re starting to see the evolution of both open source and closed, multimodal models—things that can process images or output images as well as text. And just as there is Llama for text, there’s now LLaVA for video and vision-based processing. And so we’re going to see some multimodal applications start to come up in the coming years.

I would say though, that I still think much of the focus would be on Large Language Models. Every business in the world uses language, uses words, as Ramtin pointed out. Not everybody, not every business in the world really needs to process images. Lawyers probably don’t need to process images all that often, right? So I think much of the growth will be in the single mode, text-based mode as things move on.

Christina Cardoza: Well, I’m excited to see where else this space is going to go. Like we’ve talked about in this conversation, there are a lot of opportunities, a lot of places to get started, but there’s obviously a lot of work still to be done and a lot we can still look forward to.

So, we are running a bit out of time, but before we go I would love to throw it back to each one of you—any final thoughts or takeaways? So, Teresa, I’ll start with you. Is there anything else you wanted to add, or anything looking towards the future you’re excited that we’re going to get to?

Teresa Tung: I think we still just need to remember that AI is about the data, and so being able to use this new class of generative AI for your business and to differentiate. One, hopefully the takeaway is that you could realize how easy it is to begin owning your own model. But it starts with that investment—getting your data foundation ready. And the good news is you could also use generative AI to help get that data supply chain going. So it is a win-win.

Christina Cardoza: Yeah. I love that final thought, that AI is about the data. There’s always such a—when something like this happens and there’s a big trend in a space, there’s always that instinct to jump on it.  But you have to actually be solving a problem or resolving something that the businesses need. Don’t just jump on it just to jump on it. What is the data telling you, and how can you use these solutions to help tell that story through your data? So that’s great to hear.

And, Ramtin, anything you would like to add?

Ramtin Davanlou: Yeah. I think regulatory and ethical compliance and dealing with hallucinations, like Waleed mentioned, and other topics under what we call responsible AI are the biggest challenges for companies to overcome. And also navigating the cultural change that’s needed to use GenAI at scale is really key to success.

Christina Cardoza: Yeah. And I think it’s ethical AI and responsible AI—it’s not only an issue for businesses, but also consumers or customers. They can be skeptical of these solutions sometimes, so building these with responsible AI and ethical AI in mind, that’s definitely important.

Waleed, last but not least—again, is there anything else you wanted to add to round out this conversation?

Waleed Kadous: I think it’s important to get started now, and it doesn’t have to be complicated. The tools for prototyping are becoming easier and easier. OpenAI, for example, recently released GPTs, that make it very easy to build a retrieval augmented generation system with a clean UI.

There will be these stages, but think about it as a staged process. Let’s build a prototype, make sure that users like it, even if it costs a lot of money. We’ll build that on top of GPT-4 turbo just to prove that there’s value there. And then come at the cost—and to some extent the quality issues—as a secondary issue, as the usage of those particular tools come up. So it’s now becoming much easier to prototype.

And one other thing is to not just think about this in terms of central enterprise organizations, but how can you empower employees to build their own LLMs? And for that initiative around LLMs to come not from some central machine learning group or something like that, but to give people tools to optimize their own workflows and to improve themselves.

And I think that’s really one of the most exciting trends. Rather than seeing this as a substitute technology, to see it as an augmentative technology that helps people do their jobs better. And I really like that mode of us focusing in AI on that, and empowering people to use LLMs in a way that makes them feel empowered rather than eliminated.

Christina Cardoza: Yeah, absolutely. I think my biggest takeaway from this conversation is generative AI is here to stay, and it’s only going to get bigger and help us do better things. So, like you said, Waleed, get started now, and if you’re skeptical or you don’t know where to start, or you don’t know how to take your small wins to the next level, we have great partners like the ones on this podcast: Anyscale or Accenture.

I encourage all of our listeners to visit their websites to stay up to date in this space, and to see how they can help you take your solutions and efforts to the next level, as well as visit insight.tech as we continue to cover these partners in this space. So I want to thank you guys all again for joining the conversation and for the insightful, informative conversation. And thanks to our listeners for tuning in. Until next time, this has been the IoT Chat.

The preceding transcript is provided to ensure accessibility and is intended to accurately capture an informal conversation. The transcript may contain improper uses of trademarked terms and as such should not be used for any other purposes. For more information, please see the Intel^® trademark information.

This transcript was edited by Erin Noble, copy editor.