What happens when you harness digital convergence

Over the last several decades, technology has changed the way we do our jobs, from desktop computers and spreadsheets to analytics-based decision-making and cloud-powered everything. But as the saying goes: You ain’t seen nothing yet. That’s because the next wave of innovation centers around how technology can truly augment our limited capabilities. We believe the greatest gains come when you smartly combine people and technology.

We’re already starting to see its power in large and small ways. Upskilled finance pros who can use automation to take on time-consuming data tasks so they can focus on analysis; virtual reality training that immerses learners and makes them more empathetic. A new breed of robot designed to work side-by-side with nurses so they spend more time with patients. Where the old narrative used to be about robots taking jobs, it’s now evolved to robots helping us do a better job.

It’s a courageous outlook, especially in the face of hard problems related to trust, including equity, ethics, data privacy and security. But it’s not too late: We must confront these challenges now as we build the future. It’s why we recently gathered — in collaborative virtual reality — for our annual Emerging Technology Exchange, which brings together senior executives to explore and discuss what’s next. They get hands-on with technology, engage with academics and startups and think about use cases that can change their business or industry.

Emerging technologies + people = greater value

The right combination of people and technology is about the sum being greater than the whole of its parts. The idea also applies to the tech itself. In our labs, we’ve looked at how tech building blocks like artificial intelligence (AI), Internet of Things (IoT), and virtual reality (VR) are being used together in transformative ways. We’ve defined six power combinations that can shape the future: extended reality, immersive interfaces, hyperconnected networks, and working autonomy, along with automating trust and digital reflection.

What’s ahead for futuristic technology and society

Scott Likens, PwC’s emerging tech leader, kicked off the discussion with a question: What’s the biggest benefit of the technologies you are working on? Here’s what our panelists had to say.

Christoph Fleischmann

Founder and CEO of Arthur Technologies, maker of AR and VR dynamic collaboration solutions

“We have this technology that could completely remove this concept of distance in the world. Next to AI and automation, AR and VR will have the biggest impact on the way we work together.”

In the developed world, that means businesses can hire the most qualified talent regardless of where they’re located. And for employees, it means the freedom to go live and travel wherever you want without compromising your career. But Fleischmann points out that the technology can have an ever bigger impact in the developing world. Opportunities can be equal for everyone, regardless of location or background.

For a deeper dive into extended reality and what Fleischmann is working on at Arthur Technologies, see the discussion below.

Pattie Maes

Leader of MIT Media Labs’ Fluid Interface Research Group

“The line between people and their tools gets thinner every day. We have the unprecedented opportunity to enhance human decision-making for the betterment of society.”

Leading a variety of research projects that explore human computer interaction, Maes sees a future where interfaces may become smarter, more perceptive and more aware of what a person is doing. One example: combining deepfake technology with conversational interfaces to create realistic-looking agents that can engage with us in real-time. Other research in her lab aims to augment capabilities for those who are differently abled, such as memory augmentation for older adults or communication enhancement for individuals with autism or ALS.

For more on Maes’s work at MIT in immersive interfaces, see the discussion below.

Ramona Pierson

PwC’s Head of Product Innovation for Societal Change

“We can provide immersive learning to help people gain new skills, upskill and even create new types of content that evolves as technology does, so we are not waiting for universities to catch up with diploma programs.”

Using technologies like deepfakes for good and generative adversarial networks, Pierson and her team at PwC Labs can enable businesses to automatically create new learning content. But there’s much more to building a learning organization. Pierson reminds executives they’ll need to think about how to help people evolve their skills faster and figure out how to give teams the freedom to take risks — a key part of learning.

For a closer look at Pierson’s work at PwC on the ProEdge platform, see the discussion below.

Christian Sanz

CEO of Skycatch, a drone-based intelligence platform

“The most interesting thing about drones is the data you collect. As we get really good at collecting high-quality data, high-precision information, humans can only consume so much of it. The big breakthrough will be in using machines to interpret that information.”

Sanz goes on to describe the example of a geologist who can be helped by deep learning. While the geologist can’t easily review thousands of digital files, an AI model that’s been trained to mirror what the geologist does would take on the initial analysis.

Learn more about hyperconnected networks and what Sanz is doing at Skycatch below.

Andrea Thomaz

CEO and cofounder of Diligent Robotics

“We’re building robots that can work side by side with people. We’re very excited about the benefit this brings, allowing us to reinvent entire workflows and workforces and think about what we want people doing in this environment.”

Thomaz is also quick to point out the importance of reframing the conversation. Business leaders should invest in the change management that’s necessary to rethink their entire operations. It’s about asking what we want people to spend their time doing and where they are applying their skills.

Meet Moxi the robot and learn more about working autonomy in the discussion below.

Trust in our tech is critical

The tech leaders at the Exchange see a promising future in emerging tech, but they also acknowledge that we should confront trust-related challenges head-on. Many of these issues are not about the technology itself but rather who has access to it, who controls it and who benefits from it. They also talked about how responsible technology can’t be left solely to technologists. We need to make sure everyone is a part of the conversation, including business leaders, citizens and regulators.

Mary Shelton Rose, PwC's east region vice chair, also reminded us that, at the same time, tech can help us enhance trust: “Our mission is to build trust and solve complex problems. And you cannot do that without technology at the heart of it.”

Five ways emerging tech convergence is changing the business world

Combining people and technology in powerful new ways isn’t a futuristic vision. It’s here today. See how they’re already changing business now and what you can expect tomorrow.

Customizing the remote work experience with virtual reality (VR) software

What it is

When it comes to virtual reality (VR), the hardware headsets that make the experience possible are usually the focus. But where the real magic happens is in the software. This is especially true in the business world where VR’s value has been slowly catching on. That’s why the Arthur platform is so central to our efforts here at PwC. It lets you gather with colleagues and clients in an immersive setting that feels like the real world. Share ideas, solve problems and even just socialize — without video-call fatigue or the temptation to multi-task. Describing its promise, Arthur Technology founder and CEO Christoph Fleischmann says, “When I put on a headset, the feeling of presence that I have enables me to connect on a deeper level with my colleagues. Not only that, but with the space where I meet them. We also have better cognitive bandwidth in VR. Instead of compressing all the information I want to share on a small screen, we can spread it out in a vast 3-D space.”

How it works

Arthur is a cloud-based service that companies access through enterprise-grade VR headsets like Oculus for Business. Once inside Arthur’s virtual office environment, you’ll find a number of pre-built collaboration tools for presentations, white boarding, file sharing and training. Your avatar and those of your colleagues or clients can seamlessly talk, share objects (thanks to the VR handheld controllers) and even just catch up by the virtual water cooler — or in our case at the Exchange, a virtual happy hour. The experience is surprisingly effective: Attendees at the Exchange interacted in the same focused way that we would have in person, thanks to the immersive environment that does away with distractions like email or mobile phones that compete with our attention when on video calls. Beyond Arthur’s prebuilt office environment, you can also customize your setting and activities without needing to be a programmer. In our case, we created the Mars space station. And the team from MIT had fun building a replica of the Media Lab, as well as a giant dinosaur that other guests could ride.

Use cases

Virtual events like the Emerging Tech Exchange can bring together remote attendees and speakers who can share presentations and videos on multiple screens within the environment and people can interact much the same way they would at a conference.
Collaboration and ideation sessions, which PwC has done with clients in various industries, where small teams can set up virtual workshops to brainstorm and problem-solve, such as identifying new products and improving service areas.
Training sessions that bring together colleagues from around the world like what the United Nations is piloting at its International Training Centre.
Team meeting and building for geographically dispersed teams, where they can gather and work together more productively.

Why it matters

Arthur demonstrates the promise of extended reality, one of PwC’s emerging tech convergence themes. Using VR or AR along with other tech like AI, people can upskill, collaborate and perform work in new ways. These provide greater access and are proving to be more effective, safer and less expensive than many current methods.

Enhancing learning and empowerment through AI-generated deepfakes for good

What it is

When you hear the term “deepfake,” you probably jump to thoughts of fake news or humorous videos of celebrities and politicians that make us laugh and worry at the same time. But when Pattie Maes and her PhD students from the MIT Media Labs’ Fluid Interfaces Research Group show you their work that bridges human cognition and AI, you may begin to see untapped potential. Says Maes, “I am most excited about the use of this tech for self development and learning. What we have shown is that when people embody others — it might be a younger or older version of themselves or a different person — they can learn new skills, develop new types of behaviors and, ultimately, change their beliefs and values through this virtual embodiment technique. We can help people become better, more capable versions of themselves.”

How it works

Maes and her team have developed what they describe as an “AI-generated characters-for-good pipeline” — a pipeline is a standardized process for working with data and machine learning models so you can scale your efforts. This foundation makes it possible to quickly take live or recorded video, audio or text and combine it with a target persona, like Albert Einstein, to create an output that features the persona delivering the original content. Leveraging different algorithms in the pipeline, the facial expressions are copied and generated, the lip-syncing is accurate and the content can even be translated. And the last crucial element of the pipeline process is to embed a watermark in every frame of the video so viewers will know it’s synthetically generated.

Use cases

The film industry has used the technology to take footage of an actor and depict the character as younger and older versions of themselves.
In education or enterprise learning and development, it enables customized remote learning taught by a historical figure, celebrity or other notable person of the student’s choosing. For example, you can learn physics from Einstein or get mentored by your favorite CEO.
Multinationals in any industry can more widely share customer training or corporate messaging through instant transcription, translation and lip-syncing of an original video to multiple language versions.
Museums can use augmented reality to let visitors hold up a smart phone and get more information from an interactive version of a painting — for example, the Mona Lisa narrating her origin story.
Personal development, including building confidence and empathy, can also benefit from this technology. Research studies have shown that people gain confidence when borrowing the voice and mannerisms of leaders or become less biased after embodying others.
Online privacy is another area of promise. Patients can interact with mental health providers using a digital persona that both masks their identity and, at the same time, retain all of the mannerisms and expressions of the patient for the health provider to evaluate.

Why it matters

Synthetic media is one example of immersive interfaces — new ways of interacting with computers that enable us to work and learn in new ways. Like our other emerging tech convergence themes, it points to a future where technology is even more integrated with our work and lives and helps us bolster our human capabilities.

Targeting and measuring upskilling through knowledge graphs and a dynamic platform for learning, doing and sharing

What it is

ProEdge is a holistic upskilling platform, which was developed out of our own experience upskilling PwC’s entire US workforce — some 55,000 employees and counting. ProEdge is a single platform where executives can design and track upskilling plans for their company or teams, and it’s also the place where employees can learn and practice skills. “One of the hardest pieces of building personalized learning is really understanding the velocity of upskilling — what you’re upskilling to and how you’re getting the data to compare if upskilling is happening or not,” explains PwC’s Pierson.

How it works

At the core of ProEdge is data — a lot of it. We’ve compiled more than 250 million data points about corporate skills from job boards, professional networking sites, labor data and other sources. This collection of data is known as a knowledge graph, which is a model of data and relationships in a particular domain (in this case, skills). Powered by this intelligence, you can view the skills trending in your industry and lagging in your organization. Then you can set your plan for specific skills and teams and track progress from an executive dashboard. Your employees can follow customized learning pathways generated by ProEdge’s AI engine that include content from leading providers. Even better, they can put those skills to work within ProEdge’s Create module, earning valuable functionally relevant credentials. They can also share what they’ve built with their new skills, letting other employees take advantage of digital assets like automations or visualizations.

Use cases

Company leaders can see how their industry is changing and get a visual snapshot of the skills they currently have and where they need to focus. They can also set concrete upskilling targets, such as increase the number of finance team members trained in process automation by 40%.
Department heads or team leaders can create learning pathways for their people, assigning them to employees and then monitoring progress.
Employees can personalize when and how they learn, consuming content from leading providers and getting a chance to apply the skills via project-based learning.
Organizations can create a culture of learning by encouraging people to share the projects and digital assets that they’ve created so that others may benefit.

Why it matters

Keeping pace with technology and its potential for redefining jobs and the future of work is critical for success going forward. And it’s an ongoing process to build this type of learning organization where company leaders and employees feel informed and empowered to chart their course. Central to delivering on that promise is real-time data, processes and systems to drive continual upskilling.

Modeling our world through drones, AI and edge computing to see things more precisely, efficiently and safely

What it is

At its core, Skycatch is a data platform. But that simple label belies the innovative way it gathers, processes and visualizes data for things like commercial construction sites, remote mines and critical infrastructure. CEO Christian Sanz likes to call it an awareness platform. His goal? “Create a real-time 3-D gateway so that industrials can more quickly, reliably and cost-effectively retrieve data from the physical world, then go back in time, fast forward with predictive analytics and, ultimately, get a real-time snapshot of what’s going on. That’s the future.”

How it works

An engineer, geologist or other specialist uses an app to plan where a drone will fly and the data it will collect. After the flight, data is processed and analyzed in the field via an edge computing base station. Its built-in processing power, coupled with machine learning, means large data files don't need to be uploaded to the cloud before they can be viewed and analyzed. With the data visualized as a precise 3-D model (within five centimeters of accuracy), industrial users can then integrate it with specialty CAD, GIS or other specialized software. The time savings is considerable: Scanning terrain to design optimal and safe pits in what’s called high-wall mining, for example, takes just a few hours to create a highly accurate model, compared with the old way, which took from one to several weeks. Other mining uses include mounting a laser scanner in a truck bed, driving the truck around the mine perimeter, transferring the data to the cloud for processing, then augmenting it with manual mapping.

Use cases

Industrial equipment makers have used the precise models to automate routes for heavy equipment; some even control their vehicles autonomously.
Construction companies or corporate builders have quickly surveyed sites for accuracy and compared current state progress to plans, saving hundreds of hours and correcting issues as they arise.
Mining companies have created digital twins of mines to predict how changes to the environment and other factors affect productivity, safety and planning.
International relief organizations have assessed damage to historic sites after a large earthquake in order to prioritize and plan relief efforts.

Why it matters

Even if you’re not a construction or industrial company that will make use of aerial data collection, Skycatch illustrates the potential of hyperconnected networks and digital reflection. Through a combination of AI-based edge computing, other networking like 5G and cloud, data intensive-processing and analysis can happen anywhere you need it. Precise digital twins let you see what is and explore what if.

Improving patient care with robots that safely work alongside nurses and take on noncritical tasks

What it is

Meet Moxi, a robot that’s on the frontlines of the COVID-19 pandemic. Unlike many robots in use today that perform tasks separately from human workers — often in protected areas — Moxi is designed to interact with nurses and other hospital staff. Explains Andrea Thomaz, CEO of Diligent Robotics, “We’re taking on fetching and gathering tasks that fall to frontline workers in a hospital. This same kind of trend applies to other industries.”

How it works

A nurse can use a mobile device or kiosk to request Moxi’s assistance with time-consuming, noncritical tasks — such as picking up a patient’s prescription from the pharmacy or delivering a blood sample to the lab — that Diligent estimates can eat up 30% of a nurse’s time and take them away from direct patient care. Moxi’s onboard lidar, sensors and cameras enable it to map and autonomously navigate the environment. And its precise, lightweight arm allows Moxi to manipulate objects, open doors, and perform other tasks without human assistance. Other important features are its moveable head and LED eyes, which enable Moxi to give off social cues so that human coworkers know its intent and are more comfortable interacting with the robot.

Use cases

Hospitals are deploying Moxi to take on tasks that would normally require a nurse to step away from direct patient care. This is helping hospitals address nurse shortages and burnout, which have been even more acute during the pandemic.
In the future, Moxi’s instructions could come from another computer, instead of a nurse. For example, a patient’s electronic healthcare record could automatically trigger a request to retrieve necessary medication and have it waiting in the patient’s room when admitted.
Industries such as hospitality, retail and manufacturing, among others, can benefit from robots that can interact with human workers to relieve them of time-consuming, dangerous or dirty tasks.

Why it matters

The future of work isn’t simply about automating current processes or replacing human workers with digital ones. Working autonomy, one of PwC’s emerging tech convergence themes, is where the real promise is: reimagining how things are done so that tech can free people up to spend their time on what brings the most value and fulfillment.