The Unmatched Feeling of a Personalized, In-Store Experience

Philosophers have long known the importance of feeling recognized and being seen. But what the greatest thinkers of the ages could not have anticipated is that a technological solution would emerge that would make it possible to be recognized and feel part of a community by merely setting foot in a store or restaurant.

Many of us have experienced the simple joys of being a “regular” at a restaurant or a shop—feeling at ease, cared for, and perhaps even pampered. Being acknowledged by name, receiving a phone call when a much-anticipated product arrives, or having a dish prepared that accounts for your tastes and food allergies can have a significant impact on your mood, overall experience, and make you want to return just to feel that way again.

Personalization for Every Customer

Naturally, business owners have wondered whether it might be possible to make all customers feel this way, even if they have visited the store only once before.

For a long time, they could rely only on their staff’s memories. But now, thanks to the synergy between human ingenuity and cutting-edge technology, the mnemonic work can be relegated to machines, while clerks and servers can appreciate the look of satisfaction on their customers’ faces as they make them feel at home. No online transaction could ever replace the warmth and humanity of another person being attentive to your tastes and interests, and helping you to find something perfect for you.

Personalized greetings are the next frontier for #retail, thanks to groundbreaking #technologies and the incredible R&D efforts of @Intel and its partners. via @insightdottech

As a leader in the design and implementation of cutting-edge technology, Intel continually “pampers” us by creating intelligent devices that enrich the customer experience. Anything but a solo enterprise, Intel is constantly seeking partners to help it come up with creative solutions. One such partner is ThunderSoft, a world-leading OS  technology provider.

Helping Staff Provide an Excellent Customer Experience

By leveraging Intel technology—including real-time sensor capabilities based on the EdgeX framework—ThunderSoft has created an intelligent proximity device (commonly referred to as “edge computing”) that controls cameras and is embedded with facial recognition software. Essentially, the ThunderSoft Smart Retail Solution connects to the store’s ERP system to extract information about each customer, and then makes that information readily available to every staff member in the store.

Smart retail, AI in retail
Figure 1. From identifying VIP customers to assessing customer behavior, the ThunderSoft Smart Retail Solution gives staff the data they need to deliver an amazing customer experience. (Source: Intel)

In addition to recognizing the customer’s gender and age range, staff members can access relevant information about them in real time—such as their past purchases, color preferences, sizes, specific models, or favorite food items. Beyond the customer’s personal details, the information can be further customized based on the store’s management model or type of inventory—a comprehensive set of information helping staff give every customer an excellent experience.

Secure and Cost-Effective Personalization

Although the platform connects with the most popular ERP or CRM software, sensitive information is protected and retained locally, reducing the surface area for a potential attack. Customer face recognition data are stored in a way that potential hackers cannot reconstruct the original image, and data is transferred in an encrypted form, according to the most rigorous security standards. In the case of proprietary ERP software, an application interface can be established through APIs.

Since the total cost of ownership is minimal, the system can be adopted by large chains and family businesses alike. What’s more, ThunderSoft engineers have made tremendous efforts to simplify the app interface so that staff can use the solution with ease and efficiency—and the company offers training so staff can start using the system straight away. This combination of simplicity and low cost make the Smart Retail Solution almost immediately accessible to most shops, restaurants, centers, and distribution chains.

Without question, personalized greetings are the next frontier for retail, thanks to groundbreaking technologies and the incredible research and development efforts of Intel and its partners. In addition to creating value for businesses, it allows us to take a giant step forward in customer service and cater to the attention-emotion economy. Rather than feeling skeptical and as though a shop is just looking to turn a profit, technology-fueled relationships make us feel accepted and cared for.

Ultimately, we want to be seen, recognized, and reflected in the people around us. Knowing this, can you imagine how ThunderSoft’s solution will improve your customers’ experiences?

Build PoCs Faster with 11th Gen Intel® Core™ Processor Testbed

It’s hard to believe, but the concept of Industry 4.0 is already 10 years old. It was first introduced by the German government at the 2011 Hannover Messe trade show as a long-term strategy to computerize manufacturing.

Where are we with the Industry 4.0 concept a decade later?

“There’s always hype about the technology cost improvements and process improvements,” says Reiner Grübmeyer, Director of Product Management at Kontron, a leader in embedded computing technology. “People want to improve and ease their installations, especially for cost reasons, but also to change production time.”

“It is all becoming much faster,” he continues. “But on the other hand, it always has to be controlled, and there are some well-established standards that cannot really be broken up quickly.”­

In short, we still have a way to go before realizing Industry 4.0’s full cost, efficiency, and reliability promises of a decade ago. And that’s because, according to Grübmeyer, “the technology is not completely live yet.”

The availability of PoC testbeds like the NGSF will allow #industrial operators to refine their implementations, measure ROI, and begin their #DigitalTransformation at the optimal time. @kontron via @insightdottech

Shaping the Future of the Smart Factory

To Grübmeyer, “live” means persuading internal and external customers of Industry 4.0 technology by demonstrating something with easy installation and easy tools, which is also simple to understand.

Historically, one of the best ways to accomplish this is working with prototypes or proofs of concept (PoCs). But, as he points out, there’s no harmonized configuration tool that simplifies the integration of Industry 4.0’s many diverse components.

In response, Intel® and Kontron are collaborating on the Next-Generation Smart Factory (NGSF) testbed, an industrial technology demonstrator that provides all the ingredients necessary to build your own PoC on-prem. The full NGSF testbed, first showcased at the 2020 Intel® Industrial Summit, incorporates different wireless standards, Ethernet Time-Sensitive Networking (TSN) switches, industrial PCs as well as servo motors, all operating on an OPC UA Publisher Subscriber protocol.

Users can mix and match portions of the pre-built NGSF testbed with their own solutions to validate a specific use case in the context of a full-fledged, working Industry 4.0 installation. The testbed was designed around off-the-shelf, Intel-powered hardware platforms like Kontron embedded box PCs, but also integrates optimization solutions like the Intel® OpenVINO Toolkit and operational network orchestration software such as Intel® Edge Controls for Industrial.

“The NGSF was developed to bring all the Intel technology inside a single testbed,” says Walfred Tedeschi, software application engineer in the IoT group at Intel. “For everything in industrial, there is a counterpart inside of the testbed: the orchestration, the hypervisors, the artificial intelligence on the edge, the stacks, and so on.”

For example, two critical capabilities Grübmeyer says are needed for industrial IoT architectures—real-time converged networks and workload consolidation—are supported by commercial products like 11th Gen Intel® Core embedded processors. The new IoT-centric processors include features such as:

  • Intel® Time Coordinated Computing (Intel® TCC) that minimizes jitter and latency across enterprise and operational networks
  • Hardware-assisted virtualization and acceleration that enable multiple, mixed-criticality workloads to run simultaneously on the same device
  • Functional safety (FuSa) design elements that combine with an Essential Design Package (Intel® FSEDP) to simplify compliance with industry standards

Industry 4.0: Coming to an IPC Near You

Now that industrial operators have a ready-made PoC build environment at their fingertips, is the critical mass horizon for Industry 4.0 deployment any closer? That depends on what critical mass looks like to you.

For Grübmeyer, testbeds like the NGSF offer a glimpse of the future of industrial systems that will operate more like modern consumer electronics than traditional industrial devices. And thanks to embedded controllers based on high-performance processors that support workload consolidation, that future will be increasingly software-defined.

“It’s a step in this vision of software-defined industrial systems, where you have a platform that is capable of doing almost everything and then you just put applications on top of it,” the Kontron technologist says. “We have predictive maintenance. We have object recognition. But I think to get to the flexibility and ease of application exchange with the simplicity of a PC is the longer-term goal.”

Much how desktop workstations revolutionized the enterprise value chain, running multiple high-value apps on a single industrial PC, could extend smart factory benefits from point deployments to entire supply chains. But if nothing else, the availability of PoC testbeds like the NGSF will allow industrial operators to refine their implementations, measure return on investment, and begin their digital transformation at the optimal time.

“They now have a recipe to achieve this,” Grübmeyer says. “They can see Industry 4.0 is working and there is no big complexity behind it. That is one step further to getting it rolled out in the field.”

Smart Shopping Carts: Self-Checkout on the Go

Grocery shopping is one of life’s necessary evils. Who hasn’t circled a store several times trying to locate that one ingredient—or find an employee who can direct you to it? And then there are the long checkout lines. Self-serve kiosks speed things up a bit, but they can be glitchy, not recognizing the item you’ve already put into the bag.

And overall traditional systems are not hugely efficient. Shoppers need to put goods from the cart onto the self-checkout counter, from there on a scale, and then back into the cart again before leaving the store.

Fortunately, all those frustrations may soon be a thing of the past. Innovative technologies can provide the ultimate in self-checkout with smart carts that let customers shop, scan, and go.

“There’s high demand for non-human interaction compared to the traditional way of shopping, putting everything on the belt, and having a chat with the cashier,” says Bernard van Strien, marketing and communications manager for faytech AG, a leader in touchscreen solutions. “While some people will still prefer this method, the grocery store is slowly becoming more modern, and technology can make the experience better for customers and stores.”

The Smart Shopping Cart Experience

Smart retail carts like the EASY Shopper, created through a partnership between faytech AG and Pentland Firth Software GmbH, deliver a unique, streamlined experience. Before entering the store, customers have the option to make a shopping list with the solution app. Once inside, they activate their cart by scanning their smartphone QR code.

If the customer made a list, it would appear on a touchscreen on the cart. The screen will also display a map of the store to help customers find the items they want and give shoppers the shortest route for gathering their items. It also shares personalized promotions or special offers.

As shoppers move through the store, they scan their groceries using the barcode scanner and put them into the cart, starting the checkout process cashier-free. When they’re ready to leave, the shopper scans their QR customer code again at a cashier station, where the transaction is verified and completed. The shopper can pay directly within the app using preloaded banking information, or they can provide a credit or debit card or cash.

#SmartCarts not only improve the #CustomerExperience; they enable stores to gather insights. @FayTechUSA via @insightdottech

This easy process not only speeds things up for the customer, but also gives more time for the cashiers, who can perform other tasks within the store (Video 1).

Video 1. Smart shopping carts provide navigation, cost-saving promotions, and checkout rolled into one. (Source: faytech AG)

New Opportunities with Retail Analytics

Smart carts not only improve the customer experience; they enable stores to gather insights. When the shopper scans their QR code to login, the system’s analytics starts collecting information about the purchasing behavior of the customer. For example, it tracks the length of the shopping session, what is put in the cart, and what is put back.

Insights drive marketing opportunities. Targeted ads are displayed on the cart’s screen as the customer navigates the store, suggesting items that complement what they have in their carts and offering targeted offers. This capability opens the door for new forms of revenue.

“Stores can offer contracts to brands,” says Bernard. “For example, they can create popup messages for customers who walk down certain aisles. This creates new revenue opportunities for the retailer.”

­­The technology also helps reduce theft. The cart can include a video camera and a scale. Every product has its own weight in the system, and the cart can tell if there is a discrepancy. At the checkout, the cashier will be notified if something is off, and they can check to see if something wasn’t scanned.

Retailer Rolls Out Innovations

Edeka, one of the largest supermarket chains in Germany, uses the solution in about 60 of its stores, with plans to expand to 40 more by the end of 2021, says Bernard.

“Their goal in rolling out EASY Shopper was to step away from the traditional way of shopping and provide a more modern, innovative solution,” he says. The technology solves many of their challenges, such as finding creative ways to upsell or cross-sell products and speed up the checkout process, which had been slow.

Cart-to-Cloud Retail Tech

EASY Shopper runs on an Intel® processor-based PC and connects to a private server through Microsoft Azure cloud. The cart integrates directly to a store’s enterprise resource planning (ERP) platform, which provides real-time updates for pricing and inventory.

If the cart loses its wireless connection, the shopping and payment process can continue, says Bernard. The components are ruggedized and can be taken outside if it’s raining or withstand bumps or shocks in a parking lot. Additionally, the GPS function allows the retailer to constantly track the location of the cart, which in turn prevents theft, also in the parking lot.

And the technology itself is completely customizable. In the end, it’s the flexibility to meet the shopper where they are in terms of the experience that they’re looking for. By removing the frustrations and challenges of the store experience, grocers and shoppers can both benefit.

“It’s really what your imagination might lead you to,” says Bernard. “The platform is designed so that if you’ve got a great idea, you’ve got the right software, the right skills, you can create an integrated solution, for your particular use case in the retail space.”

Improve Warehouse Operations with Machine Vision and Edge AI

As more industries realize the value of connected machines and devices, IoT adoption is growing by leaps and bounds. By 2025, there will be more than 25 billion IoT devices in industries ranging from electricity and gas to retailing, wholesaling, and transportation.

Machine vision enables machines to identify objects and analyze scenes and activities in real-life environments. When paired with AI, machine vision can simulate the human capability of solving complex problems. This Industry 4.0 approach drives industries from automation (where humans program machine decisions) to autonomy (where the machine makes decisions based on real-time data).

One company that excels in providing its customers this advanced capability is ADLINK, a global company that uses embedded computing technology to deliver the Analog-Digital LINK (hence the name) required to accelerate deployment of artificial intelligence at the network edge.

A More Powerful Edge AI Solution

The IoT gives businesses real-time information they can use to solve problems quickly and operate more efficiently. But traditional IoT technology has some significant limitations. Because it uses wide-area networking to transmit data to the cloud for processing before returning results to the decision-maker, it does not deliver the real-time decision support needed for warehouse robotics. It also requires huge bandwidth resources.

These shortcomings have given rise to new approaches that combine machine vision technologies with artificial intelligence.

“Edge computing is about applying the right data, at the right time, in the right place, to drive the right decision and take the right action. Accomplishing that outcome demands the employment of artificial intelligence at the edge,” says Daniel Collins, ADLINK Senior Director of Edge Solutions.

The Smart Pallet solution couples #MachineVision and #ArtificialIntelligence to provide fully automated visibility of all #warehouse packages and pallets. @Adlink_IoT via @insightdottech

Smarter Warehouse Solutions with Machine Vision and Edge AI

An excellent example of effectively deploying computer vision at the edge is the ADLINK Edge Smart Pallet solution. This product is an ecosystem of warehouse technology, including applications, sensors, analytics engines, and operational systems.

Connected through the ADLINK Data River, these components combine to deliver autonomous Industry 4.0 solutions to address multiple customer pain points at the edge, such as:

  • Packages that contain the wrong merchandise or are lost, stolen, or delivered to the wrong place.
  • Lack of inventory visibility during packaging, palletization, and distribution.
  • Siloed and unconnected systems that make automation and process monitoring impossibly complex, intrusive, and expensive.
  • Bandwidth and latency challenges posed by transmitting data to and from the cloud.
  • Privacy concerns associated with some 5G WAN solutions.

The Smart Pallet solution couples machine vision and edge AI technology to provide fully automated visibility of all warehouse packages and pallets. This methodology replaces traditional barcode hand-scanning with machine learning, helping to automate all aspects of the manufacturing process from pallet stacking and box finding across warehouse facilities. It also eliminates inventory errors caused by misplaced and non-scanned items (Video 1).

Video 1. The ADLINK Smart Pallet solution uses machine vision and AI to determine whether warehouse packages have been loaded onto the right pallet. (Source: ADLINK)

By capturing multiple image data streams and applying high-performance computational power, machine learning at the edge can increase warehouse automation while simultaneously improving efficiency and accuracy.

Edge computing systems also eliminate the need to transmit data to the cloud, using a high-bandwidth local area network to collect and process data locally. Smart Pallet connects to new and existing equipment, using a vendor-agnostic approach built on open standards. The entire platform is architected with modular components to ease integration with existing IT and OT systems.

Creating the Machine Vision Solution

One of the biggest technical challenges ADLINK had in developing the solution was collecting enough data to train the artificial intelligence models. It solved that problem by using Intel® technology in the computer vision cameras.

The solution’s standard package consists of an ADLINK NEON-1000-MDX, industrial-grade smart camera with onboard storage and compute. That component uses an Intel Atom® processor, an Intel® Movidius Myriad video accelerator, and the Intel® OpenVINO Toolkit machine vision framework—which comes with pre-trained AI models. Using this framework as a starting point, ADLINK created its own AI models to locate individual packages and provide a visual display of the pallet on a computer monitor.

As a longtime Intel partner, ADLINK also relies on Intel’s supply chain security. This aspect is of critical importance to customers who are required to meet legal and regulatory certifications.

Machine Vision Systems Help Boost Efficiency and Stop Theft

Automatically reading barcodes saves enormous amounts of time for warehousing and manufacturing operations. After adopting ADLINK, a global meat processing factory reduced scanning time by 90% and improved overall processing speed by 41%. The plant also significantly reduced labor costs and scaled its business to create $340,000 of additional revenue.

Another company discovered the value of the smart pallet solution after shipping customers 500 empty packages—which were supposed to contain cell phones. It turned out they had all been stolen.

To prevent this type of loss from ever happening again, the company installed the ADLINK system and began using computer vision to monitor boxes and pallets. The deployment enabled them to detect and prevent theft during the packaging process, leading to a dramatic reduction in product shrinkage.

By blending machine vision and artificial intelligence at the network’s edge, ADLINK has established itself as an industry leader in bringing analog data into the digital world. Once companies have its solution in place, they can easily customize it for new uses.

“Everything is modular,” Collins says. “You just plug in the building blocks you need and extend them to something else. The possibilities are endless.”

 

This article was edited by Christina Cardoza, Associate Editorial Director for insight.tech.

This article was originally published on August 24, 2021.

In-Store Shopping Matters More Than You Think

A conversation with Trevor Sumner @perchexperience

[player]

Would you believe that in-store shopping is more important than ever? While shoppers flocked to e-commerce sites at the beginning of the pandemic, traffic to physical stores is already back above 2019 levels.

The benefit of online shopping over in-store shopping has been its ability to provide rich information and details about products and consumers, but that’s all changing. Leveraging new technologies like computer vision and IoT, physical retailers can provide the same level of product detail as ecommerce while shining a light on how consumers are interacting and engaging with brands.

Listen to this podcast to learn why in-store shopping is on the rise, what the future of physical retail looks like, and how digital displays benefit both consumers and retailers.

Our Guest: Perch

Our guest this episode is Trevor Sumner, CEO at Perch, a leader in interactive physical digital and retail displays. Trevor is a technologist, adventurer, and a native New Yorker. He was recently named a Top 100 Retail Influencer for 2021 by Rethink Retail and Top 50 Retail Technology Influencer by the Retail Technology Innovation Hub. Follow him on Twitter at @trevorsumner.

Podcast Topics

Trevor answers our questions about:

  • (1:37) How retail has evolved over the past 18 months
  • (3:14) The trajectory of e-commerce vs. brick and mortar
  • (5:38) The role of physical stores in the future
  • (7:45) How retailers can personalize their shopping experiences
  • (9:32) How computer vision provides more insight into the in-store experience
  • (23:22) How to create an omnichannel experience
  • (31:34) The future of physical retail

Related Content

To learn more about the future of in-store retail, read Interactive Digital Displays Let Every Product Tell a Story or listen to our podcast on Data and the Retail Experience. For the latest innovations from Perch, follow them on Twitter at @perchexperience.

“Apple  “Spotify”  “Google  “”

Transcript

Kenton Williston: Welcome to IoT Chat, where we explore the trends that matter for consultants, systems integrators, and end users. I’m Kenton Williston, the editor-in-chief of insight dot tech.

Every episode I talk to a leading expert about the latest developments in the Internet of Things.

Today I’m talking about the revitalization of retail of with Trevor Sumner, the CEO of Perch.

Last year was a tough time for stores, but now in-person shopping has coming roaring back. What’s behind this trend? What can retailers do to keep the momentum going? And how can they bring together the best of online and physical retail to boost sales in both domains?

Trevor, I’m glad you’re here to answer these questions!

Trevor Sumner: Thank you. It’s great to be here.

Kenton Williston: And can you give me a quick summary of your role at Perch.

Trevor Sumner: Yeah, absolutely. My name is Trevor Sumner. I’m the CEO here at Perch. We do interactive retail displays that use computer vision to detect which products you touch and then wake up, like Minority Report, and automatically start telling you about it. I’m a longtime startup entrepreneur. I’ve been through a bunch of different high-growth companies that have IPO’d and mergered. And I started a company that grew to over a hundred people. It sold to the Blackstone Group. I’m an avid mentor at various different accelerators; occasional angel investor. I’m an adventure scuba diver and have scuba dived in every continent, including Antarctica. And I’m a native New Yorker.

Kenton Williston: Very exciting list there.

Trevor Sumner: Yeah. I like to get in there, man. Like, let’s start it. Let’s get them excited.

Kenton Williston: Excellent. So, over the course of the last three or four years, of course, things have changed a lot, particularly in the last 18 months or so. And I’m wondering how that has affected your business, and how the company has evolved?

Trevor Sumner: Yeah, absolutely. And COVID has been this technology accelerant in so many different ways. And while we are focused on in-store retail, and certainly some of our biggest clients were at the time—Macy’s and they closed their doors—they very much updated, and took a lot of the products off the shelves. Interestingly, now they just recently reported that foot traffic is back above 2019 levels, which is amazing. We focused a lot of our technology on grocery and mass retail, so that those investments really paid off, as these were effectively essential businesses and investing even more in the in-store experience. And so, COVID has been this really interesting technology accelerant, where people need more insight into what’s happening in-store in real time.

They need ways to connect to the shoppers without a sales associate, because there are a lot less sales associates. And they’re focused on other things, like inventory management and making sure that their products are on the shelves. But, because of that, we are seeing a lot of our customers thinking about how to use our technology to unify commerce and convert in-store shoppers into omnichannel shoppers. It’s been really exciting, and also really hard, because a lot of our partners were struggling with what these changes meant. And, of course, that vision has been changing almost month to month. So it’s been a very uncertain environment in which to operate, which is difficult as a leader of a company. But, at the same time, the technology acceleration has made our product set even more inevitable.

Kenton Williston: Yes. A couple of things I want to touch on there. One you talked about was, of course, the shift in balance towards e-commerce during the pandemic—for very obvious reasons people were not going into stores. And I’d like to know where you see that trajectory going now.

Trevor Sumner: It’s fascinating, right? I mean, I think there’s this narrative that brick and mortar is dead, which is absolute lunacy. Brick and mortar has been increasing 1.5% to 2% year over year. Last year we had a deadly pandemic, which meant that people were literally risking their lives to go into stores. And yet, it was a flat year for physical retail. Last year was a boon to e-commerce. I think this year it will be harder to maintain and extend those gains. In fact, CoreSite just reported that e-commerce penetration at department stores has declined year over year. So I think it’s uneven, too. I think it’s hard to talk about retail in generalities. So, like last year, grocery stores did very well. Target did very well, essential businesses, but also pets did extraordinarily well.

Sporting goods did well. Home goods. Home Depot showed a 30% year-over-year growth. To put that in perspective, just the growth at Home Depot is a Fortune 50 company, right? And you look at apparel, and that’s a whole different story in terms of the declines and the tough year that they had. So Amazon lost e-commerce share, even though they grew. They didn’t grow as fast as the brick-and-mortar stores that had e-commerce presence. And part of that is because the physical stores themselves delivered about 40% of e-commerce orders for the first time. And so, there’s certain industries where the store has to be a key part of the fulfillment and thinking through customer acquisition, ordering, and fulfillment as three different separate pieces of the puzzle that are blending in all these different ways. It becomes less and less helpful to think about e-commerce separate from brick and mortar.

Kenton Williston: Bunch of really interesting points there. As a first, I have to say that I feel like I contributed more than my fair share at a Home Depot over the last year. So, I bought a house in the middle of all this nonsense, and I’ve got to enjoy just the skyrocketing lumber prices and everything else. Similarly, I’m like, yeah, okay—I basically didn’t buy clothes last year. Why? I’m not going into the office. I could be in my pajamas all day if I want to. And so, having said that, I’m going to ask you a very general question about this idea of fulfillment, and where you see the role of the physical stores going forward. Is it mostly a fulfillment role? Is it mostly a discovery role? Is it different by sector? How’s that going to work out?

Trevor Sumner: So, let’s be very clear. When we talk about this future where—oh, wouldn’t it be amazing if I could just order anything? Any time—it just shows up at my door. We just got that last year. How terrible was it? How soulless was that experience? I bought a Christmas gift for my nephew that I never saw, never wrapped, wrote the card on Amazon. I’d don’t even know what it looked like. Probably some fake scribbled thing on a nondescript card. And didn’t really get to see him opening it, except kind of on Zoom. It was fine, but was it like, me shopping for a gift for him? No. We crave shopping to connect with products. This connection between people and products is fundamental to shopping. And so, I think increasingly we’re seeing people think about the store in the back of the house for fulfillment, but the front of house isn’t going away; it’s just going to be optimized in all these new and exciting ways, in part powered by IoT.

Having local Walmarts everywhere near you so that they can more optimally ship to you and think about last-mile fulfillment is really incredibly interesting on a business level for margins and for costs and optimizing profits. But it’s definitely not about optimizing the shopping experience. And so, I think brick and mortars can be extraordinarily strong on a continuous basis.

Kenton Williston: Your story about the Christmas present brings to mind the movie Brazil, and there’s this scene where the protagonist goes into an office, and there’s just this stack of prewrapped presents, and his boss is like, “Oh, here you go. Merry Christmas.” And it’s just like, so generic and anonymized. It definitely feels like that, I think, a lot of times when you’re shopping online—that it’s just a box, very impersonal. So, what can stores do to really enhance that connectivity with their products, and the very personalized experience?

Trevor Sumner: Yeah. So what we do at Perch, which I think is pretty fundamental, is the way to think about it is—online, the way you connect with products or learn about products is you click on it. And when you click on them, you go to what’s called a product detail page, or PDP in the industry. And it’s got videos, ratings, and reviews, and all this stuff that you’re looking for for research. And, fundamentally, we keep telling ourselves that the reason that we go in stores is because it’s better for product discovery, but, ironically, it’s also the only place where you don’t get that PDP product-level, detail information—which is why 87% of people start their product research online, because that’s where the content is.

So I see this future of in-store experience about bringing the same tools and digital into the store to combine the best of physical and digital shopping, right? So I can touch the product. I can look at the product. I can get the joy of holding it in my hands and looking at multiple different products at once in a physical and real way—in a way that a list of SKUs on Amazon just never gives me—but also deliver the product storytelling.

If you look at the brands that you really feel that emotional connection to, fundamentally those are stories that are being told. In fact, there was a recent study that went out, that you are 50% more likely to have an emotional connection with a brand in-store than online, if you just do e-commerce, right? Because you could do that storytelling and you have that physical kind of connection. And that’s a little bit why so many people are spending so much money to do all this brand advertising outside of the stores, is because they don’t have those capabilities in the store. So, fundamentally what we’re doing at Perch—we use computer vision to detect which products you pick up at the shelf. And the moment you pick it up, it wakes up and starts telling you about the product. So, it could be videos, ratings reviews, other complementary products—maybe comparing products in a product family, and providing all the tools you need to understand whether you really want that product and that product’s right for you. And so, to me, that product pickup is the same as clicking online, right? So you can click on a product online to get more information. Now those clicks at retail at the shelf that you do when you pick up a product and look at it, that’s an expression of interest. And now we can provide the right message at the right time, and help brands connect on a meaningful basis with the shoppers that are considering that.

Kenton Williston: Nice. So all this discussion about brand and the personal connection totally resonates with me. So, what it brings to mind for me immediately is the Fenty brand by Rihanna. And I really like that brand for a lot of different reasons. Most notably, it is products designed for a diverse target audience. So, it’s different skin tones, different body types, in a way that many other brands have been pretty terrible about addressing. So, just what the product does is pretty great and pretty exciting, but the fact that it’s coming from Rihanna and it’s got all that exciting creativity behind it really just adds to that brand story. And so I totally get how this concept applies to things like cosmetics or handbags, or if we’re talking about jewelry—these sorts of things that are, let’s say, more fashionable than functional in certain ways. But how would this work if you’re talking about, not something like clothing, but something that’s like a refrigerator? What’s the story element that drives those sorts of products?

Trevor Sumner: I think of what we do as product-level marketing. So I would ask you: which categories or products do you want to look at—ratings, reviews, videos, and complementary products—what would you say? All of them, right?

Kenton Williston: Yeah, pretty much right. And reflecting back on this last 18 months, that’s been a big part of the struggles I’ve had across the board—is, yes, I don’t want to go into the grocery store and get my groceries, but also there’s been so many different options for getting things delivered. Which one is the best one for me? And that’s been a little hard to determine.

Trevor Sumner: Yeah. So it sounds like you just bought a home, and you went to Home Depot to buy a fridge, and there are over 300 different fridges. They can’t fit 300 fridges on the showroom floor. So they tell you there are all these different configurations. And you’re like, “Ah, I kind of like, maybe LG—I don’t know.” And then within LG there are probably 20 different SKUs, and they’re different configurations. How do you pick a fridge? It seems like a really hard problem, right? But if you go online, you can actually visualize the different configurations of the fridge. Are you doing a double door? Single door? Where’s the ice maker? Where’s the water filter? What type of shelving configuration can you have? Is this efficient? What’s the freezer look like? What’s the balance between the freezer and the fridge? Can you multi-sub? Does it have different finishes that can match your kitchen? There’s so many different questions that you can answer that would be very hard to do with physical retail on its own.

Kenton Williston: Yeah, I think I accidentally picked a really good example, because, as you’re saying this I realize, “Oh, you know, I’ve actually been seeing online ads for the latest generation Samsung refrigerators.” And one of the things that they’re pushing is it’s very customizable in its appearance. You can mix and match all kinds of different color schemes. And somebody who is trying to figure out what to do with their kitchen—that’s a real big bonus that not everybody offers me. I can get the stainless steel or the gloss black or whatever, but this is a totally different idea, speaking of something that’s perfect for your space. And I’m like—yes, that’s great, love it.

Trevor Sumner: So, that’s a simple example of, “Hey! I click on a product—tell me about what options I have available to me.” Now, we deliver a lot of media and product-level media out there in the market. But also what we’re seeing is, there are all these tools online that help you decide what your needs are. So, let’s stick with the fridge for a second. So, in an ideal experience you go into Home Depot and you fill out a quiz. Because if you talk to a sales associate, they’re going to ask you a bunch of questions. Maybe they’re going to push something on you because it’s the one that they’re familiar with, or it’s the one in stock, it’s one on sale, it’s the one they get the biggest commission on. You never know. But what if there was a tool that said, “Okay, tell me, how many people are you feeding? How many kids do you have? How important is the freezer? Where do you live?”

And then it said, “Based on your answers to the questions we just asked, here are 10 fridges that we recommend. And here’s how they compare, so you can do it.” Now, that sounds like a wonderful experience—to try and figure out which fridge you are, and to give you the confidence that you need. We’re picking fridges—which is frankly a little bulkier than generally what we do, because you’re not touching fridges, but we actually have done a fridge concept.

But think about that for every product set. You mentioned beauty. We’re working with Johnson & Johnson to bring their Skin360 tool. So, it’s a front-facing camera that looks at your face and says, “Okay, based on your skin type, based upon your wrinkles or your sun spots, dry spots. . .” It does an analysis of many different points of your face and asks you a couple questions about what you care about most, and says, “Here are the products that we would recommend.” Whether it’s finding the right refrigerator, the right electronics, the right computers, the right TV, the right deodorant—all of these things require some digital content. And we’re trying to bring that to the shelf where 85% of transactions occur.

Kenton Williston: You threw in a kind of an oddball example there in some ways—of deodorant. So far in our conversation I’ve been envisioning what purchase is doing is being centered around high-consideration, high-cost sorts of products. Obviously deodorants don’t really fit in there. So, presumably you’re doing business with brands that have these sorts of consumer packaged goods. And what does the model look like there, and how is it different from something like cosmetics?

Trevor Sumner: Yeah. So, our business model is basically per screen. And we generate—what’s amazing about our approach is it typically generates somewhere between 30% and up to 180% sales lift. So about 80% sales lift on average; 80% sales lift has to equal the cost of the program, which happens to be somewhere between $150 to $300 per month per screen, depending on volumes and some other things. And so you’re right that there are certain categories that lend themselves very clearly to this type of technology—anything high margin, high education, high brand building in mass retail and CPG. And interestingly, grocery is our fastest growing market segment. It’s really about beauty. It’s about baby. It’s about pets, and it’s about health. And going full circle to, “Hey, what about deodorant? What does that look like?”

Do you know what’s in your deodorant? There’s a lot of talk about aluminum, and is aluminum good or bad for you? And how important are natural ingredients? And how often do you sweat? Do you need something that’s a deodorant, or also an antiperspirant? Do you care about natural ingredients? We could see this going lots of different ways. Now, to be fair, we haven’t done anything in the deodorant side of the house, but, again, this is just the role of education. And right now it’s fascinating that the physical store, where 85% of transactions occur, is this digital desert of product content where you can research these things, and that’s changing.

And so one of the things that’s really remarkable about what we’re seeing right now is about 1% of digital media spent is happening in-store, where these 85% of transactions occur. So, there’s a multibillion-dollar shift to driving digital in-store, and it’s going to be done in a couple different, interesting ways. I think there are going to be digital signage networks that are on the walls, that are basically banner ads for promotional space for people. There’s going to be digital at the shelf that’s contextual, reacting to what products you touch, or it’ll do front-facing cameras that do demographic segmentation. So, me as a 45-year-old male will get a different message than a Gen Z woman.

So it’ll be exciting, it’ll be personalized, it will be contextual. And right now there are billions of dollars being spent going into this, because the store is a giant black box—not just for media, but also for data and what people are doing. And so I think that’s the other area that’s really driving this expansion—is to start understanding how shoppers shop in-store. And now that we’re shining a light on that with sensors and IoT data, it turns out some of the things that we’ve always thought to be true aren’t really true. And it’s causing us to rethink the way we merchandise and market. And it’s going to lead to a revolution in the way we think about the in-store experience.

Kenton Williston: Do you have an example of bad assumptions that people have been using for years that are being totally debunked now?

Trevor Sumner: Yeah, so I think that the greatest one—we just did a large launch with Purina across 200 grocery stores, and they did an endcap. So, that’s the area at the end of an aisle. It’s really valuable space. And there’s something called a planogram, and basically it’s a diagram of where you put your products on the shelf. Is it on the top shelf, on the mid-shelf; is it on the left, is it on the right, et cetera.

And if you ask anybody in retail what is the most valuable area of the planogram on an endcap, they will say: eye level. In fact, they’ll either say “eye level,” or they’ll say the same thing—which is, “eye level is buy level.”

It’s such an ingrained axiom that there’s an idiom for it: “eye level is buy level.” And so I asked them, “Is it true?” All of a sudden they’re like, “I don’t know. That’s what I’ve always been told.” And the answer is, while it is true that being at eye level is beneficial—it shows about a 25% engagement and sales lift to be at eye level versus at middle or lower, on the lower shelf—it turns out that the edges of the endcap are more valuable. They show about 35% to 50% sales and engagement lift. And nobody knew that, in part because earlier studies just looked at the aisle, and because the aisle just draws along—period—there aren’t really edges. And we showed this to Purina. We showed this to our customers, and they’re like, “Oh my God—this changes everything about how we think about planograms, and which products to put in which positions and how we merchandise.”

And that’s just one example of just changing norms. There are a lot of areas where we just agree we have no visibility into. So, I’ll give another example, is Johnson & Johnson. When we launched with them, they had Kerry Washington for Neutrogena and Jennifer Aniston for Aveeno. And then, on the bottom shelf, they had Clean & Clear, which was a new launch for them. And they said, “We don’t have a spokesperson, so we’re just going to use Instagram influencers. Can Perch tell us how much that hurts us, not to have a spokesperson?”

And so, we can look at the conversion ratio from click to screen to look at digital engagement and from click to sales, and take a look at that. And what we found was that the Instagram influencers were actually 22% better at driving them to engage with the screen, and drove 10% additional sales lift. I think the real answer to the influencer versus spokesperson game is it’s different for different people. If you’re a boomer or an old Gen X, like me, maybe Jennifer Aniston and Kerry Washington are better at converting me. But for Gen Z and millennials, maybe Instagram influencers are.

And with front-facing cameras, we can start testing these things, and actually just provide you a report and say, “Here’s the content that influences women and men by age, demographic.” You can look at sentiment analysis, eye tracking. All this stuff is coming, and it’s doing it in a way that’s seamless, that’s privacy protected. It’s all anonymous. It doesn’t record your identity. None of the video makes it into the cloud. So it can do this in a way that doesn’t sacrifice privacy, but helps the brands get all this data to a black box where most of their sales occur.

And that’s what’s really fascinating to me, is how all these sensors and data can shine the light into this black box and really change the way that we think and work.

Kenton Williston: So, Trevor, I’ve got to say, you’re doing good work to make me feel like an old man. You’re hearing about these influencers. So, I’ve got a 10-year old, and let me tell you, she cares a lot more about what Mr. Beast or some Minecraft speedrunner thinks about things, than a George Clooney or a Brad Pitt. That means nothing to her.

Trevor Sumner: Yeah. It’s changing generationally, and people bring—going back to e-commerce versus stores, 82% of millennials and Gen Z prefer to shop in stores. So stores continue to be the center of where it’s at; but we’ve got to merge some of these new behaviors, new desires, new demands for information. There’s a lot of social shopping that happens in-store, where people text their friends or take pictures and other things of products. How do we enable social shopping as fundamental to the physical shopping experience? There’s fascinating ways that you can do this using digital and screens and mobile, and integrating them all together.

Kenton Williston: Speaking of that integration, I think a key question that’s been lurking in the back of my mind is that, as we started talking about at the beginning of our conversation, in some ways this is nothing new. People have been talking about omnichannel and unified commerce and so forth for quite some time. In fact, I think it’s pretty well known at this point that a lot of people are doing comparison shopping. If they’re looking in, say, an electronics store, they’ll be checking what the alternatives are on Amazon, or even at the competitor’s store—they’ve got something in stock down the road.

But I think there’s still a ways to go to even take the existing systems and really make them a cohesive, coherent experience. Then all these additional sensors you’re talking about add another layer of complexity. So, what do you see as the path forward in bringing all of this rich data together and giving the customer a really pleasing, coherent experience that’s informed by all this data?

Trevor Sumner: Great question. I think it’s the reliance on the mobile phone—the over-reliance on the mobile phone has actually been a challenge for retailers. So, if I’m walking into Best Buy, and in order to get more information about a product I’ve got to pull up my mobile phone, where am I going to go? Is it the Best Buy app? Is it Amazon? Is Best Buy going to lose the sale because you forced me on my mobile, and now I’m going to do comparison shopping and I’m going to just pick on the best price?

And, of course, Best Buy has a price guarantee to help ensure that you get it while you’re there. But once you send them onto the mobile phone, you are sending them out of the shopping experience that you own and into to the World Wide Web, and it can be anything. Do you really want to keep them there? You want to have it heavily integrated.

And this is why all the major retailers are launching loyalty programs, shopper rewards. Target Red Circle, you get 2% cash back; or CVS CarePass, you get 20% off their generic products, because they want to keep you in the store, in their world, and financially pay you to do that. The question’s really more about how the data gets put together in a way that can make these experiences more cohesive. And I think that’s actually one of the things that COVID really accelerated—which is, to make this happen you have to have all the product content accessible via API, know the inventory in the store, online—all these different places. And that’s just a simple foundational layer that you need to deliver these experiences, and pre-COVID it was not there. And, because of COVID, so many stores had to launch “buy online, pick up in store.”

All of a sudden they’re like, “Oh, we absolutely need product information to be consistent. I need to know where it is in the store, what’s my inventory.” And be able to send orders per store, to online, delivered. And so they had to create these foundational layers, and that is going to accelerate our ability to provide these types of experiences to connect people between products and shoppers in more meaningful ways, because those basic building blocks have been built. And so that’s been a great acceleration as part of this.

Now the next part is, “Great—how do we determine context so that we give you relevant things?” So, we got all excited about Beacons about four or five years ago. And then people are like, “Actually, I really hate Beacons, because as I walk through your shop, you keep on just sending me, ‘Here’s a coupon. Here’s a coupon. Buy this. Buy this.’“ And your phone keeps going off, and you’re like, “No, no, no, no! I need to find a way to turn this off.” Because great experiences need to be contextual, and they need to be relevant.

Tell me about the thing that I’m interested in. One of the reasons that people are launching these loyalty programs is so that I can connect your online shopping to your in-store shopping, and know on a product- and SKU-level basis. But ultimately, for us, we think that the most important signals—which products are you interested in right now? And those are the ones that you’re touching. And those are all the different types of signals that you can meld together, and they’re all sensor-based, IoT, to be able to bring that into store.

And so some of that will be connectivity to the mobile phone to get identity, to get integration to loyalty rewards and mobile apps. And that’s why there’s such a big push for mobile. At first for mobiles use, but long term to be this connectivity to the store—where you can scan products, take pictures of products. Where it dynamically sends you information on products that you’re interacting with, for cashierless checkout—all these types of things, the mobile phone is a really key piece there. But, to me, there’s a balance between not taking people out of the physical shopping experience and throwing them just into the mobile phone. It has to be blended together.

Again, all these building blocks are finally coming together with 4G and 5G in-store. Walgreens is sliding up 9,000 stores with 5G. It’s all for the ways these devices’ sensors can all start talking together in-store, and it’s going to lead to a Cambrian explosion of new ways to think about the merging of digital and physical commerce.

Kenton Williston: Yeah, I agree. It does lead me, though, to a question, which is, at the end of the day, what we’re talking about here is delivering an experience and a set of messages—the right time, the right place, to the right person. I’m curious how that actually works at Perch. You’ve got the technology platform. How does the messaging work? Do you have partners that you work with there? Is it something that’s done by your customers? How does that get constructed, and how is the success evaluated?

Trevor Sumner: Yes. So what’s amazing is all the real content that you need to help sell a product is already there. We don’t shoot Jennifer Aniston videos or Kerry Washington videos—they’re already there. Or the Instagram influencer videos, depending on which is most effective. Ratings and reviews—already there. Product comparison sheets, already there. If you just provide the basic levels of information that we can find online, the in-store experience gets enhanced five-fold. In fact, typically, compared to digital signage, which typically shows about 1% to 4% sales lift, we’re showing 30% to 80% sales lift.

Kenton Williston: Wow, yeah.

Trevor Sumner: It’s literally an order of magnitude better to provide that level of context. That context is key. So, the content is there. We work with agencies, we work with integrators, we work with people like Johnson & Johnson. We had talked about the Neutrogena Skin360 app, which scans your face and recommends skin products. They’ve already invested in that. The only question is, how do we bring it in-store? How do we meld it in a way that engages people in-store? A lot of people just try and put their website in-store, and it’s just super frustrating to a shopper because, if I wanted to go to your website, I would have gone to your website. The shopping behaviors and interaction modes are much different.

You’re not going to click six, seven levels deep into a website in-store. It’s got to be cleaner—the most important information has to be bubbled up immediately. You get two or three clicks on just product exploration. You’ll get more on things, like virtual try-on or skin analysis or needs-analysis quizzes, but you don’t have a lot. It’s got to be very quick. It’s got to deliver value immediately. One of the things that we find is, if you pick up a product and all of a sudden the product shows up on the screen, people are like, “Whoa, that’s really cool. What else can you tell me?” Again, we see about 10x the engagement on the screen than typical digital signage.

Kenton Williston: Yeah. That makes sense. Again, it all goes back to that question of context.

Trevor Sumner: Yeah. So I joke around with one of my investors that I’m just an arms dealer. I’m helping all brands to deliver their digital messages and connect with their customers. So, brands are bringing it in and competing, just like they do right now for the existing offerings. They’re competing for endcap space. They’re competing for inclusion and priority on websites’ search results pages. So it is intense competition between brands—always is, within the retailer, because the retailer owns the traffic, the retailer owns the shopper. So I think, short term, this is going to be driven a lot by brands; but long term, it’s going to be driven by retailers. If you look at it, many of the major retailers are investing very deeply in these types of digital networks. CVS Health has their HealthHUBs. Kroger is working with a company that they bought called 84.51°. There’s a company called Quotient Technology, working at Ahold, Target. Walmart changed their Walmart Media Network, which was buying advertising online; now it includes the ability to bundle in in-store screen impressions.

So it’s all blending together, and retailers are going to offer this as a service. Ultimately I think that’s the future. Right now, it’s Purina getting their own dedicated endcap. But I think the future is more like what Macy’s envisioned with their fragrance bars—is that every endcap, if you think about it like an Amazon search—you search for, let’s say, pet toys or dog toys, because I’ve got a pandemic puppy named Roxy. I search for pet toys and what comes up? It’s the most popular toys. It’s the Amazon-curated best toy, best-for-price—whatever categories to put in that’s curated. Then it’s advertising for brands to compete, to be part of that message. I think that’s what the future of endcaps at retail will be, which is: retailers will offer this up across multiple different brands.

Some of them they’ll curate, because they got to have their stars there. Some of them people will pay to tell their stories—to use it as a platform for experimenting within messaging. I think that, ultimately, is going to drive a lot of category lift. So, right now, I’m an arms dealer to individual brands and some retailers. Long term, I think retailers are going to be the arms dealers and provide this platform for interaction, meaningful engagement, and data to each of the brands.

Kenton Williston: Yeah, that’s really interesting. I say—just everything I’m hearing here, really poised again and again to—there’s a real need for rethink of the basic infrastructure of how physical retail works that will end up looking a lot more like e-commerce.

Trevor Sumner: Yeah, absolutely. I mean, this is my e-commerce analogy. Let’s look at how these businesses are managing themselves right now. Right now they have data about the people coming in the stores, the number of people—think of that in the e-commerce world as your unique visitors to your website. Then they’ve got data on the sales that happen. For a brand, you may just get sales in Texas. You don’t even get store-level sales. So, think about if you’re an e-commerce site that you just look at unique visitors in sales, and you have no idea where people go on your site—where they click, if they add stuff to cart, which content changes the way that they shop, how your search results or your category pages change buying patterns. None of that. That’s where we are with stores right now. That’s why I call it a black box. Now we can start instrumenting all of these things. If you think about what we are going to learn through that process, I mean, it’s phenomenal.

One of the things is, if you get a shopper to scan a QR code or something, now you can retarget them when they leave—because I scanned that puppy food’s QR code, you know I have a puppy. So when I leave, now you can market me as a puppy owner. We’re collecting all this data that is going to be extraordinarily valuable. I think that’s why, in part, I’m so excited about stores, because stores now are the dominant form and channel—what are they going to be when we make them more profitable, more efficient, more engaging, more educational, more integrated into personalization, “know what I want.” All those things—how much better are stores going to be? When you paint that picture, I couldn’t be more bullish on the bright future of brick-and-mortar retail.

Kenton Williston: Awesome. That seems like a great place to wrap our conversation. So, Trevor, I just want to thank you again for joining us today.

Trevor Sumner: Absolutely. Thanks for the great conversation.

Kenton Williston: And thanks to our listeners for joining us. To keep up with the latest from Perch, follow them on Twitter at perch experience and on LinkedIn at perch-interactive-inc.

If you enjoyed listening, please support us by subscribing and rating us on your favorite podcast app.

This has been the IoT Chat. We’ll be back next time with more ideas from industry leaders at the forefront of IoT design.

Retail Self-Service Kiosks Get Personal

After more than a year of interacting with the world through technology, people no longer assume a face-to-face interaction when accessing retail, banking, or many other services. Consumers no longer have to wait in line to get an engaging and personalized experience. And with new technological innovations, businesses now have an opportunity to learn more about what their consumers want, and to implement new multichannel strategies.

We sat down with David Frei, Vice President of Strategic Partnerships for worldwide kiosk manufacturer Pyramid Computer, to discuss how the role of self-service kiosks is changing in the current environment, how businesses can get the most out of the kiosk experience, and where we go from here.

What kinds of use cases are you seeing for kiosks, and how have those been changing lately?

In the past few years, kiosks have really been established as one preferable digital channel when it comes to upselling and queue busting in pioneering industries like hospitality and retail. As far as KPIs, there are great reports available that prove upside potential such as waiting-time reductions of a minimum 20 seconds per session. That’s really pre-COVID.

Not to mention that health has become, and hopefully always had been, the most important goal and KPI in that sense. In the previous year, not least due to COVID, there have been new use cases, and a growing focus on new values.

For example, one of our biggest successes has been in temperature-measurement and guest-screening environments. Kiosks can provide a new layer of protecting visitor and staff health when entering in any kind of building.

#Kiosks have really been established as one preferable digital channel when it comes to upselling and queue busting in pioneering industries like #hospitality and #retail. @polytouch_de via @insightdottech

 What are some of the big trends you’re seeing in tokenized or pay-by-face and loyalty programs?

Loyalty programs are obviously a huge topic for all of our customers across multiple segments. There’s that golden rule: nothing new; no higher cost for new-customer acquisition.

So when someone asks me, “How do we get our existing customers back to our store?” I can only answer that the best way is by having a fundamental multichannel strategy and knowledge about your customers’ preferences. And then answering the question: What would be the easiest way to remember your customer on-site?

For some of our clients, we would answer: face identity. The whole tokenization and GDPR topic by sight, by facial recognition, is an incredible, interesting application field. First, identify the process, and then you’ll always be remembered, not only in front of the kiosk, but also, for example, with digital signage.

Then there’s also personalized menu adjustments. If you don’t want to give your full set of data, it’s enough that the system knows your demographics to be able to adjust the menu board, make it more relevant, and create an easier checkout process.

And what we have also tried in the past, which is very efficient, is mood detection—depending on the mood of the user, we can offer the most relevant products.

How would that work in practice? Can you give an example?

We tested software that basically detects very significant moods, like if the customer is smiling, not smiling, with a group of people, or a single person. And then, depending on other information—such as the weather outside, et cetera—there’s an algorithm to give the best offers to those special conditions. This kind of personalization is what drives consumers.

How are your customers using these new technologies to create more inviting interfaces and better personalization?

In terms of innovative POS interfaces, during the pandemic last year we field-tested new technologies, including gesture control, eye tracking, and even voice interaction.

This was in the context of drive-through, click-and-collect, and self-ordering—mostly in the restaurant environment. At the time, none of these methods were really established as an alternative for the touch interface, which is still the most intuitive. But they did provide interesting user information, which can be leveraged to improve upselling, speed at the point of sale, and especially customer loyalty.

It is really all about personalization. For example, there are interesting conclusions our customers can draw if you analyze the items that guests are looking at while standing in front of the kiosk, and whether they purchase those items or not. So you can then use that data and present those items—we call them “items of best chance”—to all the following customers with a similar demographic structure, including gender and age.

Talk about the trend of kiosks integrating more with other on-site devices. What needs to happen for that to be done effectively?

This is definitely key. A kiosk is necessarily only a piece of the digital puzzle; you have the greatest effect when integrating seamlessly into existing infrastructures. And that could be like an ERP system, which contains all article information, customer information, et cetera. Or in the restaurant environment, the existing POS, which still transfers all the article data, but also all the payment processing.

There are also other components, such as the web, mobile, or the delivery piece of the digital puzzle. Nowadays you really need to have the whole multichannel or omnichannel approach. Not to mention the on-premises data processing, which simply requires a specific server infrastructure. All that is, of course, a pretty comprehensive journey.

What final thoughts would you like to leave with our audience?

One lesson that I had during the past year while testing all these very innovative approaches is how valuable it can be to approach digitalization by starting from the beginning. It’s clear that we are operating in a really incredible, interesting, and fast-changing world, and exciting innovations are appearing every day.

There’s a great temptation for customers to start digitalization and have the desire to do everything at once. I’m a big fan of a more conservative digitalization approach, where I see real value in simplicity by going step by step. This means starting a very solid, fundamental digital journey with components that already have proven their return of investment, and where you have comparable low risk.

New Healthcare Demands Present SIs with New Opportunities

The pandemic forced healthcare organizations to quickly come up with new digital strategies and address gaps it had been putting off. Virtual care or telehealth, health-monitoring devices, artificial intelligence, and the cloud were used to address the immediate impact from the pandemic, and as a result have forever changed the expectations of the healthcare industry.

While these technologies were implemented out of necessity, the added benefits are giving systems integrators (SIs) new possibilities to expand healthcare’s digital strategies. With healthcare organizations and hospitals seeing less of a strain on resources and more areas for revenue, and patients receiving better access to care and an overall easier experience, there are new opportunities to advance and improve upon these digital solutions in the future as more organizations look to invest in the digital health space (Figure 1).

on-demand healthcare, virtual healthcare services, digital transformation in healthcare
Figure 1. Digital health investments and telehealth annual revenue has almost doubled in the last year. (Source: McKinsey & Company)

“These solutions are definitely key to success…with a greater opportunity within the healthcare space,” says Matt Jordan, Healthcare Business Development Manager at BlueStar, a global distributor supporting SIs, VARs, and MSPs. “Telehealth is obviously not going to go away. I think it’s one of the things where it probably would have been implemented sooner, but you just may not have had the strength of some of the software partners that had their solution baked and ready to go.”

Healthcare’s Accelerated Digital Transformation

Telehealth was the only immediate solution that could deal with the demand of COVID, Jordan explains. Hospitals were at overcapacity and having to open temporary triage stations to treat everyone. On top of that, patients still had appointments, medications, and treatments they could not wait until the unforeseeable future to receive.

“Telehealth became the only stopgap measure that health systems quickly had to adopt as a technology because you simply couldn’t get in the door,” says Jordan.

So, as a reactive measure, healthcare organizations started to make use of consumer videoconference solutions like Google Meet and Zoom. Once the initial shock of COVID-19 wore off, the industry started to realize these solutions were not built or ideal for healthcare services. It started to look for more proactive approaches to addressing the new demands.

With almost every aspect of healthcare now impacted – communication systems, supply chain, standard care services – the industry turned to partners and SIs to tackle healthcare’s specific needs.

With #healthcare organizations and #hospitals seeing less of a strain on resources and more areas for revenue due to virtual care, there are new opportunities to advance and improve upon #digital solutions. @think_bluestar via @insightdottech

For instance, telehealth had to be reimagined to not only provide home access to healthcare providers but limit physician interaction with infected patients through remote patient monitoring at home and in the hospital.

At the beginning of the pandemic, BlueStar ISV partner VeeMed worked with healthcare system provider Banner Health to transform existing hospital TVs and displays into telehealth endpoints leveraging its virtual care platform. This allowed Banner physicians to gain instant feedback about a patient or conduct immediate telehealth sessions without having to expose themselves to the virus.

There was also a huge demand for temperature monitoring or screening to make sure anyone granted access into a hospital or building was not running a fever and putting everyone at risk. Video and photo editing software provider CyberLink redesigned its AI facial recognition technology solution FaceMe to take temperatures, check for proper-fitting face masks, and ensure social distancing, explains Michael Freeman, Networking Practice Lead at BlueStar.

“You could do all this through walking through a gateway, so to speak, that had cameras on it and thermal sensors on it, and had AI capabilities built into it from a platform perspective, and be able to very quickly recognize whether this person was conforming with the requirements that were in place,” he says.

CyberLink was able to repurpose this technology for the healthcare industry by partnering with BlueStar, which brought together servers, cameras, and sensors from IoT supplier Advantech to FaceMe. The solution also leveraged Intel® OpenVINO for its AI capabilities and flexibility.

“We bring the piece parts together,” says Freeman. “We’re really trying to stay ahead of that curve, look at all the software entities out there and what they do, and how they can interplay with some of the other technologies that we offer.”

 The Future of Healthcare Solutions

Freeman sees facial recognition technology like FaceMe being used at events to screen large numbers of people at once or provide people access to parts of a building without having to use a keycard.

But for these technologies to continue to transform and add value to ongoing wants and expectations, it will take collaboration and partnership with hardware and software providers. Just as VeeMed did with Banner Health, BlueStar works with partners to analyze healthcare demands and understand net new growth opportunities. Jordan explains that just remote patient monitoring alone is going to be worth $117 billion by 2025.

“It’s really just more of engagement and networking with some of the key hardware providers, with some of the key software providers like VeeMed, for example, with their telehealth platform and understanding just some of the educational pieces of the technology,” he says.

Through BlueStar’s healthcare initiative VARMED, it identifies key players in the space, takes time to vet solutions, creates unique healthcare services, and educates and trains its partner community so they can feel confident these applications will be successful.

Freeman adds that BlueStar’s partnership with Intel has also been key in introducing new opportunities and onboarding different partners to build solutions.

“Our relationship with Intel continues to grow, coming out of a pilot to a tier-two aggregator, and part of that benefit has been the bilateral introduction of different software and hardware platforms that can be utilized to put together a solution. I think that’s been an invaluable part of our relationship thus far,” says Freeman.

Jordan expects to see more tools and technology that help healthcare providers retain staff, maintain operational efficiency, minimize potential patient errors, provide more convenience to care, and better access to information in the future.

The Future of Retail? Self-Service Kiosks

What exactly is driving the emergence of self-service kiosks, and why are they so popular? They have been critical in the retail sector during the pandemic, but that’s not the only use case. And kiosks can provide more than an alternative to face-to-face interactions.

But there are important questions to consider: How do you incorporate them into your establishment without compromising data security? And where does the kiosk reside within the larger IT infrastructure? We explore the answers to these questions, and more, with Dylan Waddle, Chief Operating Officer for global provider of kiosk solutions M3 Technology Solutions (M3t).

How do you see the self-service kiosk trend continuing as we move now into something that resembles normalcy?

We’ve been talking with senior executives from retailers around the country, and one of the things they’re heavily focused on is what the in-store experience feels like post-COVID. They’re thinking about inventory levels. What the consumer experience looks like? What happens from the time the consumer walks in the door to the time they leave? How many employees can they have in-store? What’s going to be safe for the consumer?

Pre-COVID, kiosks were seen as more of a convenience, but not necessarily something that every store had. And now, post-COVID, we feel like the retailers, banks, et cetera, are heavily focused on kiosks and the role that they will play for the consumer.

They’re thinking a lot about what a consumer-service representative does when you’re in-store. Do they help you with your transaction initially on a kiosk? And then there’s other functions like wayfinding, purchase, and paying for things right there on the terminal.

It’s really a complete refresh around the consumer experience in a retail establishment.

And banks, retailers—pretty much any kind of experience—is going to begin with a kiosk. People are a lot more comfortable with interacting with a kiosk—as long as the flow is simplified and they feel like it’s an easy way to conduct business.

“Post-COVID, we feel like the retailers, #banks, et cetera, are heavily focused on #kiosks and the role that they will play for the consumer.” —Dylan Waddle, COO of M3t via @insightdottech

How has the financial services sector coped with the changing environment?

When I think about financial services, I always think two steps behind. I definitely believe that touchless is the future for these types of services; however, banks and financial credit unions are just much slower to move to the latest and greatest technology.

What we see from our perspective is more of a limited-touch version. So we reduce the number of touches per use. We are also focused heavily on identity authentication through the use of face- and voice-recognition technology. That’s including a one-time use code that’s sent to your cell phone. We truly believe that, once we authenticate your identity as part of the initial financial services transaction, we can reduce the number of touches.

The other hurdle that comes into play is data storage. Certain governments restrict the amount of data that can be stored, and then consumers have to be given the ability to decide if they want their personal data stored for use of more advanced technology. So we’ve got to give the consumer the flexibility to decide that.

We also have to deal with things like ADA compliance. So, Braille and things of that sort must be included, either on the physical PIN pad or on-screen. We’ve seen some new technologies around that—where they actually print the Braille on the screen so you can feel it when you’re entering your PIN code.

So there are a lot of different pieces that come into play when you think about how financial services takes the next step. How do they move down the road? They are anxious to advance, but at the same time they’re also extremely conservative.

One interesting financial application is converting cashless payments into cash payments via a kiosk. Can you give us a little more detail about that?

That’s a unique feature that M3t offers, and it is really allowing consumers to insert cash. We live in a world where we’re moving from a cash society to a more digital payment–type society. So M3t lives right in that space.

And one of the things we really specialize in is accepting cash and loading those cash funds to a digital wallet—whether it be on your phone or a physical card. We actually have the capability to issue the physical card right at the kiosk so you can load that card. You can also issue change, if you want to give someone change from a transaction, by loading the physical card.

We see that in open-loop use cases, as well as closed loop—such as public transit, where the card is only used for that one specific purpose. Open loop being more like a branded card—a branded MasterCard that can be taken anywhere and used.

Our kiosks have the flexibility to allow you to return to the terminal, stick the card in, and remove your cash funds from that card as well. It’s all about flexibility and driving that consumer experience for the future.

When you think cash, once it’s inserted in the kiosk it’s almost like the sky’s the limit to the functionality. But we’re also doing things like bill payment for cities. In the city of Austin you can insert cash in the kiosk and you can pay for a permit. If you’re going to build a deck on your house, or something: insert cash, pay for that permit. So that’s really where those kiosks live, and that market for us is growing significantly.

There’s a huge trend with kiosks being able to recognize repeat visitors in a way that is anonymous and without storing personal data. How do you see this playing out in financial services?

Absolutely. I would say this is one of the very first industries to move toward tokenization. And the level of compliance they’re required to maintain on an ongoing basis—tokenizing the data is really the only ultra-secure way to do it. Encrypting the data was sort of like step one, storing a token, or even using a third-party tokenization service is the next.

A good example of that would be Trustly, where they provide a token so that you’re not actually storing that data. We heavily believe in that, because we’re trying to provide, in essence, another level of security for personal data.

I think the consumer is still a bit behind. They’re still trying to understand whether their data being stored securely. And then they see on television that there’s been a data breach and people’s information has been stolen. So they’re are nervous, and rightfully so. It’s always a grind between: Do you want the technology to provide you ultra-convenience and let it store your data? Or do you want to go through a more manual process?

From a financial services perspective, we’re pushing heavily in that direction. But we’re also mindful that some consumers are ultra-conservative about letting technology store data. And we certainly understand that. I think that from a financial services perspective, it touches every single person’s life as we go forward.

What do you think are some of the most critical considerations in how kiosks reside within the larger IT infrastructure?

When you really think about how a kiosk lives in a corporate IT infrastructure, most of that discussion is around monitoring, maintaining, applying patches correctly. Intel® vPro® is a pretty amazing tool for supplementing a remote-connection tool—allowing access to the BIOS and giving you the complete flexibility to even restart the computer when it’s down.

vPro is amazing for that purpose. We do that in concert with what we call our kiosk-management system, or route-management system—providing a real-time view of all the terminals deployed in your network.

It really comes down to the merging of IT initiatives with these terminals, and how that gets maintained and handled correctly. The other piece that goes in concert with that is the encryption of the data and then the tokenization of that data. So that you’re completely securing the data all the way from the terminal through the network to the processor and back.

Is there anything you would like to leave with our audience as a key takeaway?

I would just like to say that, as people consider the future for their establishments—whether it be retail, banking, gaming, hospitality—to please consider the crucial role that kiosks play in supplementing an organization.

And also, think about melding with the online digital experience that consumers are having on their phones—how these two things come together as they enter your store or your establishment of almost any type. And these things play a big part in reducing inventory and providing overall greater satisfaction when customers are in the store.

We believe the kiosk plays a major role as we go forward. So we would encourage people to take a look at them, consider them, think about how it could work in their environment. We’re happy to help in any way we can.

Digitize Healthcare (Faster) with Siemens Healthineers

Peter Shen, digitizing healthcare

[Player]

Have you noticed a change in healthcare? You are not alone! The pandemic pushed healthcare organizations to rapidly digitize with telehealth and other high-tech services. Now that patients and providers are getting more comfortable with these services, we can expect new opportunities and innovations to continue.

In this podcast episode, we explore the lessons learned early in the pandemic, new expectations for the healthcare industry, how digital innovations are benefitting not only patients but caregivers, and how organizations can manage and understand the wealth of health data now available at their fingertips.

Our Guest: Siemens Healthineers

Our guest this episode is Peter Shen, Vice President of Innovation and Digital Business at Siemens Healthineers, a leading medical technology company. Peter has worked at Siemens for over 22 years in various roles, including Vice President of Business Development, Director of Sales & Consulting, and National Sales Director. His favorite part about working for the company is seeing how new and emerging technologies and platforms are influencing and changing the healthcare industry.

Podcast Topics

Peter answers our questions about:

  • (2:36) What key advancements he saw being applied during the pandemic
  • (7:57) The role of emerging technologies like AI and edge computing
  • (10:02) The ongoing evolution of the healthcare industry
  • (13:08) What Siemens Healthineers is doing to create a more standardized ecosystem
  • (25:12) How wearables can help provide a more personalized healthcare experience
  • (28:18) What the future of healthcare digitization looks like, and how to prepare

Related Content

To learn more about the digitization of healthcare, read Teamplay Cloud Platform Brings Health Data Out of the Dark. For the latest innovations from Siemens Healthineers, follow them on Twitter at @SiemensHealth.

Apple Podcasts  Spotify  Google Podcasts  

Transcript

Kenton Williston: Welcome to the IoT Chat, where we explore the trends that matter for consultants, systems integrators, and end users. I’m Kenton Williston, Editor-in-Chief of insight.tech. Every episode I talk to a leading expert about the latest developments in the Internet of Things. Today I’m talking about the digitalization of health care with Peter Shen, Vice President of Innovation and Digital Business at Siemens Healthineers. How did the pandemic change health care delivery, and what lessons can we take forward? How can we better use data to benefit both patients and providers? And what can we do about the fragmentation of health care technology? I can’t wait to find out. So, Peter, welcome to the show.

Peter Shen: Yeah, thanks so much, Ken. Really happy to be here.

Kenton Williston: Peter, tell me a little bit about Siemens Healthineers. What is the company? What does it do?

Peter Shen: Yeah, absolutely. We’re a leading medical technology company with over 170 years of experience, actually, within health care, and over 18,000 patents globally. We’ve got about 65,000 different dedicated colleagues in 70 countries focused on health care, and we continue to innovate and shape the future of health care. Over 5 million different patients globally, everyday kind of benefit from our innovative technologies and services in the areas of diagnostic and therapeutic imaging, laboratory and diagnostic and molecular medicine, as well as digital health and enterprise services.

Kenton Williston: Can you tell me a little bit about your background and what you did before your current role?

Peter Shen: Yeah, sure. I’m the head of Innovation and Digital Business here at the Siemens Healthineers, specifically focused on new and innovative technologies that are coming to the forefront within health care. I’ve been at Siemens Healthineers here for now 22 years, in all sorts of different roles and capacities. Everything from sales to business management to product development, and have been living here in Silicon Valley for that entire time as well, so have seen a lot of changes in technology and everything going on right in the heart of things here in California.

Kenton Williston: Yeah, for sure. Same here. I’ve been here in the Valley for 20 years, and it’s been quite an interesting ride, to say the least. What’s your favorite thing about your current role?

Peter Shen: The thing I really like the most, Kenton, is really just being able to see a lot of these new and emerging technologies and platforms, and figuring out a way how they can influence and change health care overall.

Kenton Williston: Yeah, for sure. And obviously the last 18 months or so has been a perfect case study—in that, obviously, not the way you would want to innovate quickly, but kind of forced us all to do so. What were some of the key advancements you saw during the not-quite-finished pandemic that we’re still working through?

Peter Shen: Yeah. Certainly I think the pandemic, if anything, really just emphasized digitalization, and new technologies are playing a significant role within health care. And, more importantly, it’s helping in terms of trying to ease the burden that was on our health care workforce and a lot of the folks that were taking care of those that were sick or are sick during the pandemic here. I think some of the initial lessons that we’ve learned here are that we need to be flexible, I think, especially from a health care provider standpoint, and staff flexibility—trying to be able to manage changing demands on a temporary basis, and trying to work on demand to help scale based on the different needs of the patient population.

From a digital perspective, I think the pandemic’s really taught us to redefine the way that we deliver health care, and the way that we work with health care here—forcing us to create standards, standardize, and try to efficiently manage operations—obviously trying to maximize the safety of our patients and our interactions with them. And then trying to overall reduce that workload that’s facing us. Certainly technologies, like a lot of remote-type of solutions, I think, came to the forefront here. A lot of digital technologies around telehealth and remote capabilities to take care of the patients and to protect the patient in terms of things like real-time location services—to identify where the patient is and how do we take care of that patient. A lot of people also tend to forget that these remote and telehealth services not only benefit the patient, but they also benefit our caregivers as well.

Our caregivers could use technologies to be able to, let’s say, remotely monitor and operate a scanning device, an imaging device, let’s say, so if they needed to take an X-ray or a CT scan or an MRI scan of the patient, they could actually use some of these things like remote scanning and be able to sit physically in a different geographical location or a different area of the hospital and remotely scan or take care of a particular patient. Some of these remote solutions actually were a big benefit for, again, not just the patient, but also for our caregivers as well here.

Kenton Williston: Yeah. I’m really glad you brought that up, because I totally agree. It’s easy to lose sight of how important all these innovations are for the caregivers. Again, casting my memory back to the start of the pandemic, people were literally out in the streets cheering for our caregivers. And I think that newfound appreciation for our caregivers is still there, but it’s not as present. It’s not so much at the forefront of everyone’s minds. But you know they still have really tough jobs to do. And I think anything we can do with technology to help—of course it can help the patients, but, you know, like you said, it’s just going to help them and have their quality of life be so much better, and not get burned-out nurses and things like that.

Peter Shen: Yeah, absolutely. We’ve seen with some of our partners and our customers that they’re actually seeing kind of the pandemic as really an opportunity here to leverage digital innovations to be able to strengthen their workforce—trying to allow them to be more flexible, to be able to broaden their education and certification and training—taking online classes versus having to physically go to some sort of a class or education or whatnot. The ability, again, as we talked about, to deliver remote services so they don’t have to physically drive or go to a certain location—that’s a very helpful as well.

It’s more than just the efficiency and connectivity, but really the ability to allow flexibility, and allowing both the patient and the caregiver some ability to adapt to the changing environment.

Kenton Williston: Yeah. And I would hazard to guess that cost is an important factor here too, right? You talked about imaging, and I think that’s been pointed to as, in the US for sure, a significant factor in high health care costs—in that imaging has been a profit center for a lot of medical organizations, which, understandable, but you know, better for us as a population if we can centralize that a little bit and make it more cost effective.

Peter Shen: Yeah. I think it’s cost effective, being more efficient. So, knowing the fact that we might have to do a diagnostic test or an imaging test to try to figure out what’s going on—how can we leverage technology? How can we leverage concepts like artificial intelligence to maybe allow us to be more efficient in terms of when we’re administering that test, and more accurate or more precise in the way that we’re diagnosing those types of exams as well. I think those are real big drivers that, again, have been accelerated by the pandemic here. I think digitalization has always been a desire, I think, for health care institutions to want to move in that direction, but certainly the pandemic has really accelerated the time frame around that.

Kenton Williston: I’d like to dig a little bit deeper into that point you made about AI, because I do think it’s a pretty important part of the overall digital transformation of the industry. Where do you see AI? And, for that matter, the concept of Edge computing. That is, you putting this really powerful computational engine very close to the patient, as opposed to in the data center, or somewhere else like the cloud. Where do you see AI and Edge computing playing a role as we move forward?

Peter Shen: Yeah, it’s a great question, Kenton. I think there’s been this exponential growth of health care data that actually contains a wealth of critical clinical and operational information to help us treat the patient. The solutions that we create here at Siemens Healthineers, like our CTs and MRI imaging devices or laboratory processing units, they contribute to this challenge through this growing amount of data about the patient here. The goal now becomes—how do we process all this information, all this data, in a timely fashion so that we can deliver those important clinical results back to the physician, so that they can make that diagnosis or that treatment for the patient here.

This is where I think it’s important to figure out what is the right technology to try to address these challenges here and try to be able to find, again, those important clinical results here. So we’ve got to focus on developing technologies and solutions that process these critical clinical findings as quickly as possible. And that’s where the technologies like artificial intelligence become so important. Great example of that is a new AI platform that we’ve created here at Siemens Healthineers that we call the AI-Rad Companion, that we brought to market just a few years ago. The AI-Rad Companion actually leverages artificial intelligence here to process large amounts of imaging data, to help really identify, characterize, and quantify clinical results automatically for that physician, so they can immediately review them and create that diagnosis.

Kenton Williston: I have to say I’ve been the direct beneficiary of some of these imaging technologies myself. So, a couple of years ago I had to have a surgery and pre-surgery, and went in for an MRI because they were going to do an imagery-guided surgery, right? So they weren’t just depending on the real-time eyeballs of the surgeon to go and do what they needed to do, but were actually doing it in conjunction with a 3D image. Love that we’ve got these kinds of technologies, but that same experience was also shocking in the sense that, when I got my MRI, the way I conveyed it to my surgeon was I had to physically carry a CD to them. Really old school. Kind of made me feel like back in my college days—the sneaker net. I think that really speaks to how disjointed the industry is. One of the things you said earlier was, during the pandemic how there was a real, I think, awakening about how important standards were, and doing things in a coordinated, collaborative way.

How do you think the industry needs to continue evolving on this point—whether that’s from the perspective of digital transformation or just how technologies interoperate?

Peter Shen: Yeah. Your example is a great example of where there are still challenges in health care in terms of this digital adoption that’s going on here. Like we talked about, especially in the world of health care, we want to make these informed decisions about what to do with the patient—whether we’re diagnosing something, or we’re trying to treat the patient with something here. And I think it’s not just the sheer quantity of information and data that we talked about earlier, but it’s also ensuring that we have a high quality in this information. And that means the accuracy, the completeness, the timeliness of getting information—such as like critical results of a patient based on their exam, where if somebody is on the surgical table, trying to get that information. It’s so critical in terms of making our informed decision with this high quality.

That means things like accessibility is real critical. And other examples are active participation of patients in their health care—whether it’s the wearables that we’re all familiar with, or your active engagement in terms of monitoring your vital signs and whatnot. Getting that information to the clinician becomes very important as well. And trying to figure out a very easy, technology-enabled way of getting that information directly to your clinicians is just as important as well. We have to have almost like a digital health platform, if you will, that could gather all these different, disparate, growing, large amounts of health care data that we’re talking about, and trying to consume all this information in a timely manner—easy effort, so that the provider, the clinician here, your doctor, can effectively see all this information, gather all this information, digest all this information in a real simple way.

Kenton Williston: Yeah. What’s Siemens specifically doing towards that end? One of the things that comes to mind for me is how there’ve been all these different EHR systems, right? You kind of have a platform, but the problem is there’s multiple flavors of these things. Just like the personal example I experienced, I was going between different health care organizations, and that linkage totally broke down between them. What’s Siemens doing to create more of a standardized ecosystem that folks can plan together?

Peter Shen: Like you talked about, there are several characteristics that are necessary in order for this digital health platform to be successful. In the digital transformation of health care, I think this platform, like we talked about, needs to be accessible to broaden the digital portfolio of clinical and operational tools that might be available to the clinician or the end user here. It needs to be flexible in order to leverage technologies, to allow for the ease of deployment without the dependency on technical limitations or infrastructure. And then it’s got to be scalable to be able to facilitate organizational growth. And, finally, it’s got to also provide this interoperability that you mentioned, as well. So, being able to drive the connectivity amongst different systems and to simplify the whole concept of information sharing.

Here at Siemens, we actually created a digital health platform that we call the teamplay digital health platform. It’s a bit of our ecosystem, if you will, that brings together data and evolving applications, providing this unifying platform for accessibility, flexibility, scalability, and interoperability that we talked about here. And that platform leverages the latest different computing technologies that are out there, provides the flexibility to not only be both a cloud-based solution that we’re used to, but also maybe an on-premise solution, or within the walls of the institution for security purposes, or whatnot.

And the other important thing is that there’s some independence there that allows a consistent performance to deliver those critical results, regardless of the type of platform, or the solutions that are involved in that particular platform, to be able to deliver the right results to the physician without worrying about technology challenges.

Kenton Williston: I want to come back and talk more about that performance angle. You’re definitely triggering some questions in my mind. But before it gets to that, how do people actually partner up with you? Is this something that Siemens itself goes out and asks people to join, or are partners coming to you? How does that work?

Peter Shen: Yeah. It’s a combination of both. Certainly, I think, we at Siemens Healthineers, we pride ourselves in innovation, and certainly we’re developing a lot of clinical and operational solutions organically within our organization. But there’s also an opportunity for inorganic growth as well, where we recognize that some of our customers, some of those different health care providers, they want specific solutions that either we haven’t fully developed yet, or may not be in an area of expertise for us. And that’s where it’s important to kind of have, again, third-party clinical and operational partners. Certainly we love reaching out to certain partners to be able to explore what we might be able to do together.

Conversely, many partners also reach out to us, knowing the fact that we’ve got a digital health platform here that is quite ubiquitous in a lot of different health care settings already throughout the globe.

Kenton Williston: As promised, I do have questions about this idea of performance. What does it mean that you’re resolving performance concerns and making that a non-issue?

Peter Shen: From a practical standpoint here, we talked about this example earlier of trying to identify a nodule in a static chest CT image, and then maybe looking at the heart ventricle while it’s beating, or whatnot, in a functional MRI exam. The technology requirements to analyze and characterize, let’s say that nodule within an image—a picture, if you will—those might be vastly simpler, versus trying to take now a beating heart and trying to analyze, let’s say, the volume of that heart as it’s in motion and as it’s beating to try to figure out whether it’s getting enough blood to survive, let’s say.

And I think those different processing requirements, those needs to process all that information, whether it’s a static image or it’s this beating heart now, those are vastly different. And in the big picture, though, for our customers, for the clinician, for that doctor, he or she actually doesn’t care, especially in a critical situation, doesn’t care about how complex it is to interpret that image or that moving image or whatnot. He or she really just needs to get that critical clinical result. Is it malignant or is it benign? Do they need to go to surgery or not? And so this is where what we want to do with our digital health platform is we want to really move away from having our end users, those clinicians, having to worry about the technical limitations and the technology challenges that are there in order to get that critical clinical result.

Kenton Williston: Is this something you’re doing by making sure, for example, they’ve got on-prem sufficient compute power? Is it extending things into the cloud so you can flexibly handle whatever they need to do? What are you actually doing to make sure they’ve gotten the performance they need?

Peter Shen: Yeah, absolutely. It’s a combination of both of those things. I think it’s creating on-premise solutions to make sure that we’re delivering those results in a timely fashion. It’s leveraging cloud computing to be able to always make sure that our customers and those clinicians have the latest and greatest AI algorithm to be able to process those studies that are there. It’s Edge computing, where we’re able to take a hybrid of those scenarios to be able to, again, deliver the result in a timely fashion, leveraging whatever technologies that are out there. It’s certainly doing those aspects. I think the other piece—also leveraging a lot of the technology partners that we have as well to make sure that those capabilities are there too.

We have a wonderful partnership with Intel, for example, that is so critical to how our digital health platform works here. The processing performance that we get from Intel to help us design our AI algorithms and deliver those critical findings that are assimilated by solutions like the AI-Rad Companion are so important. What’s great about our partnership with Intel is that their OpenVINO toolkit allows us to configure and optimize those different AI algorithms for our platform here. And, quite frankly, it allows us on the Siemens Healthineers side to actually focus on developing our algorithms and clinical solutions to process those clinical findings, without having to worry about technical or infrastructure limitations here.

Kenton Williston: Peter, I feel like you just totally read my mind, because as you were talking I was thinking about—this is kind of a conversation about scalability, right? You can put computing at the Edge, in the cloud, in the data center. And I was already thinking about OpenVINO, and was going to ask you about that very thing. Because one of the things that’s so cool about that platform is that it’s designed to scale all different kinds of compute hardware, so that you can put things on the Edge, if that’s where they need to be, put them in the on-prem data center, put them in the cloud. It can do all of the above, right?

Peter Shen: Yeah, absolutely. I’ll give you another practical example of where Intel’s OpenVINO toolkit really helps us out. If we go back to that AI-Rad Companion solution that we mentioned earlier, one of the hallmarks about the AI-Rad Companion is that, with that solution, we actually run multiple AI algorithms at the same time on a particular image that’s being processed by that particular solution.

This is important from a practical sense, because in the real world the clinician actually might not know what’s the disease or ailment that’s affecting the patient that he or she sees. This is where, now as I create these AI algorithms, I have to have multiple computing powers, multiple capabilities here to—if I want to run these algorithms all at the same time. And, again, this is where OpenVINO makes things so much easier for our team. Because it really is now—with OpenVINO I can actually create these AI algorithms that really address these different clinical areas, and then have them all run or operate at the same time—all to meet this physician’s need at the end, which is that they need to get those critical results back so they can figure out how to best diagnose this patient.

That’s kind of a real-world example of how something like OpenVINO is really helping us in terms of the development of a technology like AI, and then the practical application of that technology in the real world.

Kenton Williston: That’s really cool. I should mention, too, for our listeners, that this podcast is an Intel production. One thing that really, I think, is interesting about this example you’re giving of somebody coming in and there’s a lot of different directions you could explore. A lot of the conversations about AI to date have been around sort of the point solutions—like, I need to see if there’s a nodule in the lung or not, right? That’s a single question. But really, caregivers can benefit from having a broader perspective on what the patient’s issues could actually indicate.

And I’ll never forget when my wife was pregnant, we went in—we’re talking to her OB-GYN, and we’re talking about this and that and the other. And she got out her little pocket book and flipped through to do a little quick analysis of what these symptoms could add up to. And this is somebody who is very experienced, really knew what she was doing, but it’s like you just can’t keep it all in your brain. And I think there’s some really interesting opportunities for AI to go beyond individual diagnoses to do things like better understand the patient as a whole, and even know what questions to ask to begin with.

Peter Shen: Yeah. You’re absolutely right, Kenton. I think that’s where we at Siemens Healthineers actually see the greatest potential for technology like artificial intelligence. And I think here at Siemens Healthineers we really strive to be that leader in clinical decision support, not only in the point of diagnosis, but through the entire care continuum for the patient. And that also means driving concepts like personalized medicine. It’s not just trying to figure out what’s the right diagnosis, but also maybe trying to figure out what’s the optimal therapy plan or treatment plan for that individual patient.

We’re in the process right now of developing another solution that we call the AI-Pathway Companion, which is really looking to leverage, as you mentioned, patient data from all these multiple sources. Not just imaging data, which we’ve talked a lot about here, but also: let’s look at the patient’s laboratory results, let’s look at the patient’s pathology report, or even their genomic data and history to maybe take in all these different disparate pieces of data. Leverage AI to ingest all this information. And then, more importantly, find correlations between all those different, disparate pieces of data. And then analyze that data to actually then try to either create kind of a guide, or predicted or optimized personalized treatment plan for that individual patient.

Kenton Williston: Yeah, totally. And this reminds me of something you said earlier in our conversation about even integrating data from wearables—which now Apple watches and the like have a lot of really sophisticated sensors in there, and, you know, can really help understand on an ongoing basis how an individual’s health is trending and how a larger population is doing. One of my aunts who’s in a retirement home now—they had a cohort wearing Apple watches for a while there to do various kinds of studies and understand how they could help them stay healthier longer. And I think that’s really cool.

Peter Shen: Yeah, absolutely. And, again, if we can leverage things like AI to help us figure out, “Hey, if this worked for your aunt, can we take the same regimen, same treatment, or whatever it is, and maybe help all of her colleagues also who are maybe either in the same environment or suffering from the same challenge or whatever it is.”

Kenton Williston: Yeah, totally. And this does lead me to a question. I mean, it’s really great to think about bringing all these different data points together. And I love what you guys are doing with this teamplay idea. But I do have a question about how you ensure all these different solutions coming from different partners are able to be integrated, and to make sure they’re secure, to make sure they’re compliant. Because, of course, as you well know, in the health care space there’s just a lot of regulations, and you have to take really, really good care of your patient data. How is Siemens handling these issues?

Peter Shen: Yeah. It’s a great question, Kenton. And obviously one that’s sensitive, to not only the patient and their information, but then also to the provider themselves—the health care giver as well, in terms of making sure that their patient’s information is in good standing and safe.

The great part within Siemens Healthineers is that that data security and patient information security is actually built in and inherent to even the design of our solutions. As we develop things like the teamplay digital health platform, as we design applications like the AI-Rad Companion or AI-Pathway Companion, security is kind of inherent in it. The ability to compartmentalize the data—if it’s not necessary, then let’s remove kind of all the patient information that’s there—so, all their PHI information or protected health information associated with the patient. If there isn’t a need for that information, then there’s no need within our applications to actually keep that information. That design thinking is inherent in all the products that we’ve created. And then, of course, as we implement these solutions and roll these solutions out, we’ve created different—either physical barriers or even technology barriers—where we’re isolating individual patient data or institutional data so that they’re separated from other organizations or whatnot as well.

It’s really principles that we adhere by, even from the initial development of our solutions, to make sure that we address data security and data privacy from the get-go.

Kenton Williston: Got it. Well, it’s certainly comforting to hear. Looking forward, what else can we expect in this area of health care digitization? And what should health care organizations be doing to prepare themselves for what’s coming?

Peter Shen: The future of health care continues to be dynamic, and digitalization plays a significant part of it. We talked about a couple of different concepts around leveraging technologies like AI in terms of simulating all the data that’s there. I think what we see in the future here is also the ability to—if we can assimilate all those different, disparate pieces of data—we can not only use that to do some predictive analytics, to figure out what works and what doesn’t work. But if you think about it, if we have information about your imaging exams, if we have information about your lab results and your pathology reports, and let’s say genomic makeup or whatnot—what we’re doing here at Siemens Healthineers is we can actually start to create, if you will, a digital twin of the patient. Kind of a digital replica of the patient.

And by creating that replica here, that digital twin could be used to simulate different diagnostic or therapeutic decisions to test them virtually, and see what the response is of the virtual patient before we do that exam or do that procedure on the patient for real. The excitement here, again, is all based on the data, but if we can leverage these technologies to gather all this information, not only can we, again, make diagnostic and therapeutic decisions, but we can actually then create this digital twin of the patient and use that digital twin to help us actually drive concepts of wellness going forward.

Kenton Williston: All right, well, listen, Peter, it’s been really great getting your perspective. Really appreciate your time today.

Peter Shen: This was fantastic, Kenton. Really appreciated the time as well. Enjoyed the conversation, and all is very exciting here in the world of health care.

Kenton Williston: And with that, I’d just like to thank our listeners for joining us. And to keep up with the latest from Siemens Healthineers, you can follow them on Twitter at Siemens Health and on LinkedIn at linkedin.com/company/siemens-healthineers. And, of course, if you enjoyed listing, please support us by subscribing and rating us on your favorite podcast app. This has been the IOT Chat. We’ll be back next time with more ideas from industry leaders at the forefront of IOT design.

Machine Vision—Coming to a Kiosk Near You

Self-service kiosks have changed the way consumers interact with retailers, banks, and the hospitality industry. Now, the way consumers interact with kiosks is undergoing a revolution. But it’s going to demand a lot in terms of new hardware, new software, and new connectivity to back-end systems.

We look at the changing role of self-service kiosks across multiple industries with Stephen Borg, Group Chief Executive Officer for AI technology company meldCX. We explore the role of machine vision in creating a seamless and connected experience, and how businesses can get the most out of their kiosks.

How do you see the ecosystem behind kiosks changing?

It’s quite interesting, in that we started from a software platform helping customers or partners execute more quickly. What we found, especially around machine vision, was that the integration between software and physical-device design needed to be more tightly integrated. What we do is help design that service, or work with partners to do that.

But most of all we’re finding that our customers are trying to replicate or create a better customer experience. It’s not just about pulling people through quicker, or line busting, or those traditional use cases. Usually the checklist we get is: “I want to create a richer, more engaging experience while minimizing the number of touches. But I want it to be more personal.”

Our mission is a seamless experience—not only for the customers using the #kiosk, but for the businesses deploying it. via @insightdottech

How is the industry responding to demands for new technologies like machine vision?

We’re seeing machine vision playing a really big part in those applications that make the whole process seamless. A great example is in postal services, where it can be quite complex—sending a parcel and making sure you fill out your forms right.

Machine vision does handwriting recognition and automatically detects the destination and what else we need to know, or verifies the address. It cleans the data on the way through to ensure your parcel gets there.

And it’s being used to connect an experience. We’re working with a retail bank right now, where through tokenization it distinguishes your skill level in using that kiosk. So it can go straight past any instructional content and get you right to the point. Because that’s your expectation—when you’ve used the kiosk once or twice, you want that interaction to be quite seamless.

Hotel check-in is another example of machine vision and AI applications being brought into new environments, such as the work we’re doing with the Marriott Group. They want to create a universal premium experience down to a kiosk device. For check-in, valet, any service that you typically require.

These are some of the use cases driving the need to tightly integrate AI or machine-vision solutions back into kiosk applications.

What are people doing to make sure high-traffic kiosks stay clean and safe to use?

The first thing we did was to open our platform to a lot of different options, such as eye and finger tracking—all these things where you don’t have to physically touch the device. What we found was that the end user wasn’t quite adapted to that. And so it didn’t create the best experience.

We found two main trends. Antimicrobial is one of them, but there’s another area, which is quite interesting. We created a piece of AI that allows you to heatmap touched areas on a kiosk. It uses a combination of the pressure sensor and touchscreen—and if there are any physical cameras in the screen—it allows you to create a complete digital manifest of areas that were touched.

Research we did with customers found that they were concerned about cleaning—that the kiosks were cleaned, and for long enough in the correct areas.

We created another AI tool that sits in the background and keeps a manifest of everything that’s touched or interacted with. You can set thresholds at a corporate level, and it messages a local attendant, or it can even stop the kiosk being used if it hits a threshold. And then it goes ahead and creates a complete digital manifest of who cleaned it and when.

Once you put it in that mode, it shows all the heavy-usage areas, and if there’s one area in a touchscreen that’s heavily used, it would literally make you rub that out. It’s like you’re rubbing an eraser. And we found that to be hugely popular, because it gave our customers confidence that their staff on-site were cleaning appropriately. And it gave them a full audit of their activity for cleaning.

This idea was started by a customer that had an outbreak in Australia even though it was regularly cleaning its kiosk. But they weren’t paying attention to the other devices that are on the kiosk. They were cleaning the screen, but they weren’t cleaning, say, the PIN pad. So this system would create a process flow and say, “Clean PIN pad now” on the screen.

What are some of the broader ways your customers are using machine vision for kiosk applications?

We have a customer that wants to detect the type of handbags females are holding when they’re interacting with their devices. So they know what their spending capacity is—which is really interesting. So if that’s in a mall, they’ll know: “Do I arrange to have a Gucci in this shopping center? Or is it a Coach?”

And, more interestingly, there’s product recognition. We’re seeing more and more customers, especially in grocery, that want to reduce waste. They want to be more conscious, not only about wasted packaging, but people taking only the portions they need.

So we use deep learning to detect the device or object—even through bags—and let the kiosk know what that is. The customer has a very seamless experience; they don’t need to enter a PLU code or a barcode or scan anything; they just put the object on there, and it’s remarkably quick.

Recognizing labels as well, to make sure there’s certain compliance—we’re finding that on deli and meat products. We’ve got a customer that will make sure a client doesn’t leave with something that’s out of date, or very close to out of date. All those type of things you’re starting to see to make that shopping experience more convenient. But they also have other goals—being environmentally conscious, reducing waste, and food safety.

What do you see as being the critical considerations of integration with the larger software and services universe?

We’ve been heavily focused on that because our mission is a seamless experience—not only for the customers using the kiosk but for the businesses deploying it.

And Intel® has been fantastic, giving us access to tools and getting our kits ready-to-market so others can use them. And further, we’ve created some universal APIs—not only to common peripherals but over 3,000 integrations to things like Salesforce and ServiceNow. So customers can easily take their API token, apply it, and they’re ready to go.

We have worked with Intel on multiple solutions for other kiosk manufacturers.

I’ll give you an example. There are a few legacy-payment terminal types that might have a situation where the PIN pad gets out of sync with the kiosk software itself. That situation might require a hard reboot or PIN pad timeout—which ultimately creates a bad customer experience. They’re in a state where they don’t know if their payment’s being processed; they don’t know how to move on.

In this case, we have different layers of what we look at. There are AI timers, and those look at various operations at a kiosk and automatically intervene. They might cut power to a PIN pad and re-engage that power while you’re in that transaction so you can continue. Or they might cycle a card reader. Or any of those things that would typically be done when you’re calling a support desk—it does that automatically, and sometimes seamlessly.

Is there anything that we didn’t get to that you’d like to add?

The one thought I’d leave you with is that when we talk to customers now, especially about kiosks, it’s so advanced, and there’s this perception that they’re very transactional. We really start the journey by asking our them: “What would you like the kiosk to hear, to see, and to do?” Because that’s what it’s really about.

And when customers think more broadly, you see some really interesting use cases, and those manifest into great experiences.