Author(s):

  • Shaukat Ali,
  • Shah Khusro,
  • Akif Khan,
  • Inayat Khan,
  • Iftikhar Alam
  • Salman Faiz Solehria 

Abstract:

The advancements in sensors, processing, storage and networking technologies have turned smartphone into de facto lifelogging device. Realizing the lifelogging potential of smartphone, researchers have postulated with smartphone-based lifelogging systems with varying capabilities of capturing and storing information about peoples’ daily life activities, behaviors, interactions, locations, environments, etc., into their lifelog archives. However, the systems are lacking with several capturing, storing, retrieving and design issues, importantly not organizing the lifelog information with the same semantics as they exist in the real world and encoded in human episodic memory. The semantic organization of lifelog information will develop a true digital prosthetic memory to solve problems in different real-world domains including memory augmentation. In this paper, we have presented the architecture and implementation of Semantic LifelOGging (SLOG) framework for smartphone-based semantic lifelogging using Semantic Web technologies. The SLOG framework incorporates smartphone sensors and processing capabilities, and Semantic Web technologies to develop a semantically enriched digital prosthetic memory on smartphone. The semantic lifelog data model is developed by extending SmartOntoSensor ontology with relevant concepts and properties to represent, store and relate lifelog and contextual information. The semantic lifelog data model will provide contextual semantics to organize, annotate and relate lifelog information like human episodic memory and provide enough contextual cues to recall lifelog information like associative recall in human memory. The proof-of-concept applications are developed to show practicality of the SLOG framework. The quantitative and empirical evaluations have shown the SLOG framework and system has improved features and is acceptable in usability and effectiveness.

Documentation:

https://doi.org/10.1007/s00779-022-01701-0

References:

Gurrin C, Smeaton AF, Doherty AR (2014) LifeLogging: personal big data. Found Trends Inf Retr 8(1):1–125

Article  Google Scholar 

Ali S (2018) Exploiting sensor data semantics for smartphone-based lifelogging: Towards the development of digital prosthetic memory on smartphone. PhD Dissertation, University of Peshawar, Pakistan

Chen Y, Jones GJF (2010) Augmenting human memory using personal lifelogs. In Proceedings of the 1st Augmented Human International Conference. ACM, pp. 1–9

Hodges S, Williams L, Berr E, Izadi S, Srinivasan J, Butler A, Smyth G, Kapur N, Wood K (2006) SenseCam: A retrospective memory aid. In Proceedings of the 8th international conference on Ubiquitous Computing. Springer, pp. 177–193

Aizenbud-Reshef N, Belinsky E, Jacovi M, Laufer D, Soroka V (2008) Pensieve: augmenting human memory. In Proceedings of the CHI ’08 Extended Abstracts on Human Factors in Computing Systems. ACM, pp 3231–3236. https://doi.org/10.1145/1358628.1358836

Chennuru SK, Chen P-W, Zhu J, Zhang JY (2010) Mobile Lifelogger – recording, indexing, and understanding a mobile user’s life. In Proceedings of the International Conference on Mobile Computing, Applications, and Services. Springer, pp 263–281

Klapperich H, Uhde A, Hassenzahl M (2020) Designing everyday automation with well-being in mind. Pers Ubiquit Comput 24(6):763–779

Article  Google Scholar 

Belimpasakis P, Roimela K, Yu Y (2009) Experience Explorer: a life-logging platform based on mobile context collection. In Proceedings of the Third International Conference on Next Generation Mobile Applications, Services and Technologies – NGMAST ’09. IEEE, pp 77–82. https://doi.org/10.1109/ngmast.2009.49

Albatal R, Gurrin C, Zhou J, Yang Y, Carthy D, Li N (2013) SenseSeer: mobile-cloud-based lifelogging framework. In Proceedings of the IEEE International Symposium on Technology and Society (ISTAS). IEEE, pp 27–29

Mohamed EST (2012) Designing and evaluating a user interface for continues embedded lifelogging based on physical context. PhD Dissertation, Newcastle University, England

Karkkainen T, Vaittinen T, Vaananen-Vainio-Mattila K (2010) I don’t mind being logged, but want to remain in control: a field study of mobile activity and context logging. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, pp. 163–172

Bush V (1996) As we may think interactions 3(2):35–46. https://doi.org/10.1145/227181.227186

Article  Google Scholar 

Oren E (2006) An overview of information management and knowledge work studies: lessons for the semantic desktop. In Proceedings of the 5th International Conference on Semantic Desktop and Social Semantic Collaboration- SemDesk’06. ACM, pp. 14–24

Dodge M, Kitchin R (2007) ‘Outlines of a world coming into existence’: pervasive computing and the ethics of forgetting. Environ Plann B Plann Des 34(3):431–445

Article  Google Scholar 

Kalnikaite V, Sellen A, Whittaker S, Kirk D (2010) Now let me see where i was: understanding how lifelogs mediate memory. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, pp. 2045–2054

Ali S, Khusro S, Khan A, Khan I, Solheria SF (2019) An insight of smartphone-based lifelogging research: Issues, challenges, and research opportunities. Proceedings of the Pakistan Academy of Sci: A Physical and Computational Sci 56(3):1–16

MathSciNet  Google Scholar 

Sellen A, Whittaker S (2010) Beyond total capture: A constructive critique of lifelogging. Commun ACM 53(5):70–77. https://doi.org/10.1145/1735223.1735243

Article  Google Scholar 

Rawassizadeh R (2012) A holistic multi-purpose life logging framework. PhD Dissertation, University of Vienna, Austria

Siewiorek D, Smailagic A, Furukawa J, Krause A, Moraveji N, Reiger K, Shaffer J, Wong FL (2003) SenSay: A context-aware mobile phone. In Proceedings of the 7th IEEE International Symposium on Wearable Computers. IEEE, pp. 248–253

Shah M, Mears B, Chakrabarti C, Spanias A (2012) Lifelogging: Archival and retrieval of continuously recorded audio using wearable devices. In Proceedings of the IEEE International Conference on Emerging Signal Processing Applications (ESPA). IEEE, pp 99–102

Wolf K, Schmidt A, Bexheti A, Langheinrich M (2014) Lifelogging: you’re wearing a camera? IEEE Pervasive Comput 13(3):8–12

Article  Google Scholar 

Gurrin C, Johansen H, Sødring T, Johansen D (2014) Digital chronofiles of life experience. In Proceedings of the International Conference on Social Informatics. Springer, pp 264–273

Rawassizadeh R, Tomitsch M, Wac K, Tjoa AM (2013) UbiqLog: a generic mobile phone-based life-log framework. Pers Ubiquit Comput 17(4):621–637. https://doi.org/10.1007/s00779-012-0511-8

Article  Google Scholar 

Vemuri S, Schmandt C, Bender W (2006) iRemember: a personal, long-term memory prosthesis. In Proceedings of the 3rd ACM Workshop on Continous Archival and Retrieval of Persoanl Experiences CARPE’ 06. ACM, pp 65–74

Ali S, Khusro S, Rauf A, Mahfooz S (2014) Sensors and mobile phones: Evolution and state-of-the-art. Pak J Sci 66(4):386–400

Google Scholar 

Ali S, Khusro S, Khan AA, Hassan L (2014) A survey of mobile phones context-awareness using sensing computing research. J Engineering and Applied Sci 33(4):75–93

Google Scholar 

Myka A (2005) Nokia lifeblog — towards a truly personal multimedia information system. In Proceedings of the Workshop on Mobile Data Banken and Information Systemes – MDBIS’05. pp 21–30

Ahmed M, Hoang HH, Karim MS, Khusro S, Lanzenberger M, Latif K, Michlmayr E, Mustofa K, Nguyen H, Rauber A (2004) ‘SemanticLIFE’-a framework for managing information of a human lifetime. In Proceedings of the Sixth International Conference on Information Integration and Web Based Applications & Services. iiWAS2004, pp. 725–734

Froehlich J, Chen MY, Consolvo S, Harrison B, Landay JA (2007) MyExperience: A system for in situ tracing and capturing of user feedback on mobile phones. In Proceedings of the 5th International Conference on Mobile Systems, Applications and Services. ACM, pp. 57–70

Packer HS, Smith A, Lewis P (2012) MemoryBook: generating narratives from lifelogs. In Proceedings of the 2nd Workshop on Narrative and Hypertext. ACM, pp 7–12

Memon MA, Bhatti S, Mahoto NA (2016) A Digital Diary: Remembering the past using the present context. Mehran University Research J Engineering Technol 35(2):275–286

Google Scholar 

Sangkeun L, Gihyun G, Inbeom H, Sang-goo L (2010) LifeLogOn: A practical lifelog system for building and exploiting lifelog ontology. In Proceeding of the IEEE International Conference on Sensor Networks, Ubiquitous, and Trustworthy Computing (SUTC). IEEE, pp 367–373. https://doi.org/10.1109/sutc.2010.33

Zafeiropoulos A, Konstantinou N, Arkoulis S, Spanos DE, Mitrou N A (2008) Semantic-based architecture for sensor data fusion. In Proceedings of the Second Internation Conference on Mobile Ubiquitous Computing, Systems, Services and Technologies. IEEE, pp 116–121. https://doi.org/10.1109/ubicomm.2008.67

Tuffield MM, Loizou A, Dupplaw D (2006) The Semantic Logger: Supporting service building from personal context. In Proceedings of the 3rd ACM Workshop on Continuous Archival and Retrival of Personal Experences – CARPE ’06. ACM, pp. 55–64

Song I, Haam D, Kim H, Kim MH (2010) OntoLMS: An ontology-based lifelog management system. In Proceedings of the 12th International Asia-Pacific Web Conference. IEEE, pp 341–343

Ali S, Khusro S (2016) POEM: practical ontology engineering model for semantic web ontologies. Cogent Engineering 3(1):1193959. https://doi.org/10.1080/23311916.2016.1193959

Article  Google Scholar 

Cohen G, Conway MA (2008) Memory in the real world. Psychology Press, Hove East Sussex, United Kingdom

Google Scholar 

Tulving E, Murray D (1985) Elements of episodic memory. Can Psychol 26(3):235–238

Article  Google Scholar 

Tulving E, Thomson DM (1973) Encoding specificity and retrieval processes in episodic memory. Psychol Rev 80(5):359–380

Article  Google Scholar 

Ali S, Khusro S, Ullah I, Khan A, Khan I (2017) SmartOntoSensor: ontology for semantic interpretation of smartphone sensors data for context-aware applications. Journal of Sensors 17(2017):1–26

Article  Google Scholar 

Preuveneers D, den Bergh JV, Wagelaar D, Georges A, Rigole P, Clerckx T, Berbers Y, Coninx K, Jonckers V, Bosschere KD (2004) Towards an extensible context ontology for ambient intelligence. In Proceedings of the Second European Symposium on Ambient Intelligence. Springer, pp. 148–159

Kim I-J, Ahn SC, Kim H-G (2007) Personalized life log media system in ubiquitous environment. In: Ubiquitous Convergence Technology. Springer, pp 20–29

Weiser M (1994) The world is not a desktop. interactions 1 (1):7–8

Indulska J, Sutton P (2003) Location management in pervasive systems. In Proceedings of the Australasian Information Security Workshop Conference on ACSW Frontiers. Australian Computer Society Inc, pp 143–151

Ali S, Khusro S (2016) Mobile phone sensing: a new application paradigm. Indian J Sci Technol 9(19):1–42

Article  Google Scholar 

Wang P (2011) Semantic interpretation of events in lifelogging. PhD Dissertation, Dublin City University, Ireland

Smeaton AF, Over P, Kraaij W (2009) High-level feature detection from video in TRECVid: A 5-year retrospective of achievements. Multimedia Content Analysis, Signals and Communication:151–174

Arndt N (2011) TriplePlace: a flexible triple store for Android with six indices. https://core.ac.uk/reader/226099420#page=15. Accessed 20/03/2020 2020

Möller K, Handschuh S (2007) Towards a light-weight semantic desktop. In Proceedings of the Semantic Desktop Design Workshop (SemDeskDesign 2007) at ESWC2007.

Bell G (2001) A personal digital store. Commun ACM 44(1):86–91

Article  Google Scholar 

Allen AL (2008) Dredging up the past: Lifelogging, memory, and surveillance. The University of Chicago Law Review 75(1):47–74

Google Scholar 

Strahilevitz LJ (2005) A social networks theory of privacy. The University of Chicago Law Review:919–988

Langheinrich M (2001) Privacy by design—principles of privacy-aware ubiquitous systems. In Proceedings of the International conference on Ubiquitous Computing. Springer, pp 273–291

Khan I, Khusro S, Ali S, Din AU (2016) Daily life activities on smartphones and their effect on battery life for better personal information management. Proceedings of the Pakistan Academy of Sci: A Physical and Computational Sci 53(1):61–74

Google Scholar 

Khan I, Ali S, Khusro S (2019) Smartphone-based lifelogging: an investigation of data volume generation strength of smartphone sensors. In, Cham, 2019. Simulation tools and techniques. Springer International Publishing, pp 63–73

Brooke J (1996) SUS: a “quick and dirty” usability. Usability evaluation in indmstry:189

Mazumdar S, Petrelli D, Ciravegna F (2014) Exploring user and system requirements of linked data visualization through a visual dashboard approach. Semantic Web 5(3):203–220

Article  Google Scholar 

Bangor A, Kortum P, Miller J (2009) Determining what individual SUS scores mean: adding an adjective rating scale. J Usability Stud 4(3):114–123

Google Scholar 

Zeis C, Regassa H, Shah A, Ahmadain A (2001) Goodness-of-fit test for rating scale data: applying the minimum chi-sqaure method. J Econ Soc Meas 27(1–2):25–39

The SELF Institute