Physical Cyber Social Computing for Human Experience
Amit Sheth Pramod Anantharam
Ohio Center of Excellence in
Knowledge-enabled Computing (Kno.e.sis)
Wright State University
Dayton, OH, USA
amit@knoesis.org
ABSTRACT
Computing has a critical role in solving some of the grand
challenges spanning health and wellbeing, sustainability, and
prevention of crime. Traditionally, computing has focused
on a single narrow view of the problem to provide solutions
ignoring the essential human experience component. The
availability of low-cost sensors and mobile devices leading to
simultaneous and continuous access to events spanning phys-
ical, cyber, and social worlds demands rethinking of the tra-
ditional computational approaches. We propose Physical-
Cyber-Social (PCS) computing, that takes a human centric
and holistic view of computing by analyzing observations,
knowledge, and experiences from physical, cyber, and social
worlds. We exemplify real-world problems that demands
the approach of PCS computing and outline the research
challenges in building algorithms and techniques for PCS
computing.
Categories and Subject Descriptors
H.4 [Information Systems Applications]: Miscellaneous
General Terms
Theory, Algorithm
Keywords
Physical-Cyber-Social Systems, Machine Perception, Social
Computing, Semantic Computing, Computing for Human
Experience, Healthcare, Traffic Analytics, Intelligence
1. INTRODUCTION
The societal challenges such as healthcare and wellbeing
(e.g., hospital readmission, assisted living), sustainability
(e.g., traffic, water, energy), crime prevention and mitiga-
tion (e.g., intelligence applications) needs a computational
core for observation, comprehension, and action. Most of
the existing computational techniques have focused on the
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise, to
republish, to post on servers or to redistribute to lists, requires prior specific
permission and/or a fee.
WIMS13, June 12-14, 2013 Madrid, Spain
Copyright
c 2013 ACM 978-1-4503-1850-1/13/06... $10.00. and knowledge.
Ohio Center of Excellence in
Knowledge-enabled Computing (Kno.e.sis)
Wright State University
Dayton, OH, USA
pramod@knoesis.org
development of algorithms and techniques that can act on
a specific representation of real-world problems [6]. A vast
majority of current approaches address a sliver of the real
problem, primarily focusing on limited forms of data and
its analysis. An often used approach is to use a limited
form of data as input to the algorithms in a reactive manner
(e.g., search query). However, they miss out on the richness
of human-level information processing which can be repre-
sented as abstractions over multimodal and multi-sensory
data. With over 40 billion devices and sensors expected
to be connected to the Internet by 2015, and an extensive
amount of collectively created and curated information on
the Web (e.g., Wikipedia and Linked Open Data), we now
have a deeper and continuous view of the events that sur-
round us. This trend is accelerated by the use of smart-
phones which record fine grained observations such as phys-
ical movements/activity along with low-cost devices report-
ing observations to the phone or directly via the Internet
(aka, Internet of Things - IoT). We focus on the rapidly
evolving ecosystem of computing encompassing physical (sen-
sors/devices/IoT captured observations), cyber (Web-based
information/knowledge and tools, etc.), and social (social or
human-in-the-loop sensing, mobile interactions, etc.) com-
ponents, what we call Physical-Cyber-Social (PCS) comput-
ing, to develop advanced algorithms and applications that
act to enhance human experience. PCS computing incor-
porates physical, cyber, and social components to assist or
interact with humans in a palatable way using higher level
abstractions. The technology itself is expected to disap-
pear into the background [17]. We have defined this human
centric vision of computing as Computing for Human Expe-
rience (CHE) [13]. CHE emphasizes the unobtrusive, sup-
portive, and assistive role of technology in improving human
experience, so that technology takes into account the hu-
man world and allows computers themselves to disappear in
the background [17].
2. FROM SEARCH TO SOLUTION, FROM
DATA TO MEANING
Using a massive amount of multimodal and multi-sensory
observations collected in a PCS system1 , the computational
approaches should move toward proactive creation of ques-
tion representations, for example, early detection and man-
1
In conjunction with PCS, we use system to refer to em-
bodiment in software and its components, and use com-
puting to refer to algorithms and techniques including the
operators discussed later, that act upon observations, data,
Figure 1: Comparison of a search engine such as Google vs. an answer engine such as WolframAlpha used
to find information about a symptom of heartburn

agement of an impending heart problem, detection of on-

set/progression of Parkinsons Disease, early detection of Table 1: Comparison of traditional computational
traffic delays, and early detection of crime. All these chal- paradigm with PCS for Human Experience
Application, Traditional Computational PCS Computing for Human
lenges require us to ask contextually relevant questions over Activity Paradigm Experience
of Human
multimodal and multi-sensory observations. One form of Interest
Healthcare threshold based monitoring actionable information in
deriving question representation is the use of background the form of risk score with
knowledge of the domain in asking contextually relevant Intelligence monitoring raw data from
explanation
actionable information in
questions [11, 10]. For example, observing elevated blood surveillance cameras, mobile the form of threats obtained
phone logs, open source in- from threat modeling and
pressure may result in a question such as as what are the telligence assessment
Traffic Man- monitoring speed of vehicles, actionable information to
reasons for elevated blood pressure?. These representations agement prediction of delays mobilize units based on
are actively generated by observing the state of the person event types causing delays
and worst affected locations
rather than waiting till this question is asked in a reactive
manner. The actionable information that a human needs
will be such as reduce the salt intake or try reducing your
stress level at work depending on the result of processing such as demographics, prescribed medication, co-occurring
observations from PCS system. symptoms, and connection to smoking and obesity. While
Consider the following example, when Mark experiences Mark can quickly glean the knowledge of heartburn symp-
heartburn, he translates the information need from the phys- tom and its causes, he still needs to make a connection be-
ical world (the physical experience of discomfort due to heart- tween his personal activities and observations to population
burn) into a search query heartburn. Figure 1 shows an level statistics. Mark is still not sure whether to schedule an
attempt by Mark to find information about the symptom appointment with his physician or visit the ER.
heartburn using a search engine such as Google vs. an an- We need a computational framework that can proactively
swer engine WolframAlpha. When using Google, the search sense the discomfort of Mark, ask contextually relevant ques-
for heartburn results in a ranked list of relevant documents tions, and personalize the analytics leveraging background
from authoritative sources such as WebMD and community knowledge. Mark needs some actionable information rather
knowledgebase such as Wikipedia. While the information than relevant documents describing his symptoms or a popu-
from authoritative sources are useful, it does not explain lation level statistics of his symptom. Beyond understanding
why Mark has a heartburn. Mark has to combine informa- the reason for his heartburn, Mark needs to act on it by tak-
tion about possible causes of heartburn (gleaned by read- ing appropriate measures such as decrease fatty food intake,
ing and abstracting from authoritative sources) and his past have timely food intake, schedule doctors appointment, or
activities (in the physical world) to explain his heartburn make arrangements for an ER visit (actionable information).
making a personalized interpretation. Comparison of the actionable information provided by
An answer engine such as WolframAlpha provides Mark PCS computing with the traditional computing paradigm
with lot more relevant information such as population level is presented in Table 1. Providing actionable information
statistics of heartburn cases in the US based on parameters requires the use of background knowledge, semantic inte-
gration, interpretation, and abstraction.
Figure 2: PCS operators for semantic integration and semantic abstraction to transcend from massive data
to actionable information an asthma scenario
3. PCS COMPUTING
The central idea of PCS computing [14] is to provide ac-
tionable information in the form of abstractions that puts
human at the center of computing by utilizing knowledge,
observations, and experiences from physical-cyber-social sys-
tems.
There have been different computing paradigms ranging
from the focus on making machines intelligent to the fo-
cus on human experience. Artificial Intelligence (AI) fo-
cuses on building machines that can simulate human intelli-
gence. Ambient Intelligence makes the surroundings of hu-
mans an intelligent space. Human Computer Interaction
(HCI) builds seamless interfaces between humans and ma-
chines. Computing for Human Experience [13] is a human
centric vision of computing where the focus is on using com-
putation for enhancing human experience. Ambient Intel-
ligence is more machine centric compared to HCI as HCI
puts human experience first and machine intelligence next.
Table 1 provides a comparison of solutions offered by tradi-
tional computational paradigm and PCS computing.
PCS Operators: The computational algorithms for PCS
computing can be thought of as operators on data for seman-
tic integration, annotation, interpretation, and abstraction.
Two such operators are described here (Figure 3).
Horizontal Operators: Horizontal operators provide opera-
tions for semantic integration of data from multisensory and
multimodal observations. These operators play a key role in
PCS systems as the observations may span from machine
sensors (quantitative observations) to citizen sensors (qual-
itative observations). One modality may complement the
other or provide corroborative evidence for the other.
Vertical Operators: Vertical operators provide operations
for transforming massive amounts of data to abstractions
that are palatable to people. These operators are unique in
the following two ways: (1) the operator is agnostic of the
source of the observations, and (2) the knowledge base is
not limited to a formal ontology - it spans from statistical
knowledge to social experiences.
Figure 2 exemplifies transcending from multimodal and
multi-sensory data to actionable information for asthma at-
tack prevention and mitigation. Mark has heartburn which
is not an obvious indication of an asthma problem unless we
have the contextual information of Marks history of asthma
and the domain knowledge that states the relationship be-
tween heartburn and asthma [8]. Given that heartburn is a
generic symptom (heartburn is a common symptom of many
ailments), any search or answer engine will provide informa-
tion that may not be immediately relevant to Mark. The
association between Marks house windows and the level of
CO2 in the house may not be immediately observable by
humans. While the action of closing windows before leav-
ing for work would be the actionable information that a
human expects to know. This actionable information is
a contextual (e.g., open windows) and personalized (e.g.,
Marks asthma history) interpretation of observations span-
ning physical (e.g., open window), cyber (e.g., knowledge of
CO2 causing asthma aggravation), and social (e.g., report
of air pollutants/asthma attacks near Marks house) worlds.
4. PCS COMPUTING EXAMPLES
4.1 Intelligence Scenario
Attacks on physical, cyber, and social infrastructures have
been on a constant rise [1, 2, 5, 4]. Intelligence agencies
should perform simultaneous (multi-sensory) and continu-
ous (real-time) monitoring for identifying malicious activi-
ties. Forensic analysis of these observations is a reactive way
of solving the problem. Proactive assessment of risks based
on observations from multisensory and multimodal sources
Figure 3: Example intelligence scenario with questions (from decision makers) and utilization of multimodal
and multi-sensory observations in answering these questions
is crucial for enhanced situational awareness for protection
of integrity and lives of humans. Conventional threat de-
tection techniques may rely on one modality, resulting in
an incomplete picture of the threats. Often decision makers
are inundated with lot of information and it is challenging
to focus on the most important piece of information. An-
swering some of the questions (by a decisionmaker) posed
in Figure 3, requires gathering evidence from heterogeneous
sources (machine sensor to citizen sensor observations). The
knowledge that may be required in answering each question
is listed along with the question in Figure 3.
4.2 Healthcare Scenario
Healthcare has great inertia in being reactive in treating
ailments/diseases leading to an expensive healthcare system
which constitutes 17.6% of GDP for 2010 in the US. This
type of healthcare system is not affordable to the increas-
ing and aging world population [16]. We envision a proac-
tive and personalized healthcare by analyzing contextually
relevant observations. With the increasing adoption and
decreasing cost of sensors and mobile devices for monitor-
ing health and wellbeing [7], we envision the transformation
of healthcare from reactive to proactive and personalized
healthcare. PCS computing will take massive amounts of
multisensory (e.g., heart rate, blood pressure, weight) and
multimodal (e.g., machine sensors, textual observations from
EMR, social media interactions) observations, and perform
contextual and personalized analytics. With proactive care,
Mark (from the example introduced earlier in search to solu-
tion section) need not search for heartburn on a computer.
PCS computing will identify the physical discomfort of Mark
through a galvanic skin response sensor and proactively ask a
contextually relevant question to Mark (Figure 4). Further,
based on his answer, the PCS computing will provide action-
able information to Mark (e.g., risk score on a scale of 5).
Since Mark has a history of asthma, exposed to CO2, and
experiencing heartburn symptom after having his meal,
the most likely reason for his heartburn can be asthma [8].
The actionable information in Marks case is to undersand
that the CO2 level at home is high and the windows are left
open during the day. The action to be taken by Mark is to
close the windows (action in the physical world) of his house
to reduce the ingush of CO2 along with setting up an ap-
pointment with Marks doctor. Action of Mark will reduce
his asthma attacks at home and addresses the root cause
of the heartburn symptom leading to an enhanced human
experience.
4.3 Traffic Scenario
Traffic jumped 236% while the population increased by
only 20% in the US from 1982 to 2001 [3]. Indian cities
had over 285 million people by 2001, which is almost the
population of the entire United States [12]. This trend of
concentrated population in cities is only going to get worse
with depleting resources. We need a deeper understanding
of the sustainability problems such as traffic management,
water, energy, and other resources. Solutions for these prob-
lems should not be confined to analysing one modality re-
sulting only in a single perspective of the problem. We need
a holistic approach to this problem which can semantically
integrate observations and derive insights from the data del-
uge. Figure 5 shows traffic related observations from 511.org
which has both machine sensor and textual observations.
The complementary nature of observations are exemplified
in the figure. Slow moving traffic is observed by the machine
sensors on a particular link (link is a unit that constitutes
a road). Examining the scheduled events list (provided by
511.org) shows that there is a baseball game near the link.
The schedule information of the venue indicates a scheduled
event of Mariners playing a baseball game at the coliseum
Figure 4: PCS computing vision of providing personalized, actionable information proactively by considering
contextually relevant observations from physical, cyber, and social modalities

Figure 5: Traffic observations spanning machine sensors, authoritative reports of traffic incidents, and event
schedule information

Figure 6: Hierarchy of PCS computing challenges
depicting a broader view beyond computational
challenges
complementing all the other modalities.
Understanding traffic events would allow decision makers
to ask questions such as: What are the common event-types
causing delays in San Francisco (SFO)? What are locations
worst affected by delays in SFO? Which location in SFO
needs attention when there is poor visibility?
5. KEY RESEARCH CHALLENGES
Figure 6 organizes the research challenges for PCS com-
puting into computational challenges and interdisciplinary
challenges.
Computational challenges include the algorithmic chal-
lenges in dealing with observations from the PCS system.
Horizontal operations: Horizontal operations should be ca-
pable of dealing with semantic heterogeneity for semantic in-
tegration, interpretation, and understanding of multimodal
and multi-sensory observations: (a) multimodal: data, knowl-
edge, and experiences span physical, cyber, and social modal-
ities, (b) multi-sensory: data from a variety of sensors, in-
cluding those that capture human senses (vision, sound,
touch, and smell).
Vertical operations: Vertical operators should be capable of
dealing with a massive amount of multimodal and multi-
sensory observations and their semantic interpretation to
generating human intelligible abstractions. Change is in-
evitable in the real-world (Physical) and these abstractions
change continuously, reflecting changes in the real-world.
Research on Continuous Semantics [15] deals with the ability
to understand and incorporate new entities/concepts with
respect to a situation/event.
PCS computing is a broader vision and needs the inter-
disciplinary challenges to be addressed. Human interactions
are an integral part of computing, where humans can guide
the computation, provide missing information that the sys-
tem seeks, geared toward understanding of the situation.
The AI, Intelligent Systems, Semantic Web, and Sensor Web
communities should embrace research in other domains such
as Social, Cognitive, and Brain & Neuroscience for develop-
ing PCS computing algorithms.
6. CONCLUSIONS
Some of the grand challenges in healthcare, sustainability,
crime prevention and mitigation require a holistic approach
to computing for providing actionable information. With
the increased digitization of the physical world culminating
in a massive data generation from sensors, mobile devices,
and personal/social observations has led to a deeper view
into our physical, cyber, and social worlds. The data gen-
eration rate has surpassed our ability to store all the obser-
vations [9]. We envision PCS computing to derive insights
from these observations to provide actionable information
to humans. Providing actionable information by taking a
human centric approach is the vision of PCS computing.
We exemplified the need for research toward a new com-
putational paradigm while we embark on solving some the
greatest challenges faced by humanity.
7. ACKNOWLEDGMENTS
Much of the work discussed here results from the work of
our Semantic Sensor Web team, of which Cory Henson and
Prof. T.K. Prasad are an integral part. We also acknowledge
partial support from NSF awards IIS-1111182: SoCS: Social
Media Enhanced Organizational Sensemaking in Emergency
Response and IIS-1143717: EAGER - Expressive Scalable
Querying over Integrated Linked Open Data. Any opinions,
findings, and conclusions or recommendations expressed in
this material are those of the author(s) and do not necessar-
ily reflect the views of the National Science Foundation.
8. REFERENCES
[1] Boston bombing on April 15th. http://en.
wikipedia.org/wiki/Boston_Marathon_bombings,
April, 2013.
[2] Dramatic increase in the attacks on cyber
infrastructure.
http://www.homelandsecuritynewswire.com/
dramatic-increase-critical-infrastructure-
cyber-attacks-sabotage, April, 2013.
[3] IBM Smarter Traffic initiative:. http://www.ibm.com/
smarterplanet/us/en/traffic_congestion/ideas/,
April, 2013.
[4] McAfee global report on the threats facing key
industries.
http://www.mcafee.com/us/resources/reports/rp-
in-crossfire-critical-infrastructure-cyber-
war.pdf, April, 2013.
[5] McAfee second annual infrastructure protection
report.
http://www.mcafee.com/us/resources/reports/rp-
critical-infrastructure-protection.pdf, April,
2013.
[6] Physical-Cyber-Social Computing Systems.
http://ngs.ics.uci.edu/blog/?p=1501, April, 2013.
[7] Quantified Self, self knowledge through numbers.
http://quantifiedself.com/, April, 2013.
[8] Relationship between heartburn and asthma. http:
//www.webmd.com/asthma/guide/heartburn-asthma,
April, 2013.
[9] Sensor Networks top Social Networks for Big Data.
http://gigaom.com/2010/09/13/sensor-networks-
top-social-networks-for-big-data-2/, April, 2013.
[10] C. Henson, A. Sheth, and K. Thirunarayan. Semantic
perception: Converting sensory observations to
abstractions. Internet Computing, IEEE, 16(2):2634,
2012.
[11] C. Henson, K. Thirunarayan, and A. Sheth. An
efficient bit vector approach to semantics-based
 machine perception in resource-constrained devices. In
The Semantic WebISWC 2012, pages 149164.
Springer, 2012.
[12] J. Pucher, N. Korattyswaroopam, and N. Ittyerah.
The crisis of public transport in india: overwhelming
needs but limited resources. Journal of Public
Transportation, 7:95113, 2004.
[13] A. Sheth. Computing for human experience:
Semantics-empowered sensors, services, and social
computing on the ubiquitous web. Internet
Computing, IEEE, 14(1):8891, 2010.
[14] A. Sheth, P. Anantharam, and C. Henson.
Physical-cyber-social computing: An early 21st
century approach. Intelligent Systems, IEEE,
28(1):7882, 2013.
[15] A. Sheth, C. Thomas, and P. Mehra. Continuous
semantics to analyze real-time data. Internet
Computing, IEEE, 14(6):8489, 2010.
[16] E. J. Topol. The creative destruction of medicine:
How the digital revolution will create better health
care. Basic Books (AZ), 2012.
[17] M. Weiser. The computer for the 21st century.
Scientific american, 265(3):94104, 1991.