Skip to main content

Research Theme: Intelligent Sensing Laboratory

Research Theme: Intelligent Sensing Laboratory

The Intelligent Sensing Laboratory was built and equipped with £0.75M funding from Newcastle University Vice-Chancellor’s office as part of the University’s £30M Research Investment Fund. This laboratory was officially opened on October 13, 2016.

The lab conducts cross-disciplinary work. The research is focused on wearable and non-wearable multi-sensor data acquisition, and interpretation for healthcare and security applications.

Multimodal Information Processing

Signal and Information Processing is particularly well suited to deal with multimodal data. Multimodal data includes audio, video, and infrared. Current and future sensor systems, such as the sensors in smart devices, will provide ever more data for analysis.

The next generation of artificially intelligent systems include:

  • automated security and surveillance
  • human anomaly detection
  • human identification and tracking in cluttered and congested environments
  • speech enhancement and separation in challenging environments

Systems such as these will need to process multimodal data.

We develop algorithms and technologies to enable significant advances towards this vision.

Multimodal Data Processing - human detection

Multimodal human anomaly detection

Human anomaly detection is a challenging task for reliable artificial intelligence.

Human behaviour, both as individuals and in groups, is complex. Such behaviour needs robust features for modelling, indexing, and classification.

Our research into human behaviour analysis includes:

  • multimodal action recognition
  • contextual information retrieval
  • information fusion

We are investigating advanced deep learning techniques. Our main focus is to provide network explainability and human-informed systems.

Multimodal data processing - human behaviour anomaly
Multimodal data processing - human behaviour anomaly

Multimodal human identification and tracking

Our research involves:

  • core signal processing for multiple human localisation
  • tracking in cluttered and congested environments

Core signal processing techniques provide us with the solution for Multimodal Human Tracking (MHT). These techniques include:

  • social interaction modelling
  • probabilistic clustering
  • non-linear filtering
  • efficient data association
  • particle flow based on stochastic differential equations
  • multi-level cooperative fusion
People being tracked walking down a busy high street
People being tracked walking down a busy highstreet

Multimodal speech processing

We have established complete mathematical proofs for nonlinear signal processing theory. These are major breakthroughs. From this, we have established non-linear models for optimal signal separation and information retrieval.

A major success is our development of a complete framework using statistical methodologies for:

  • signal fusion
  • separation of complex nonlinearly mixed signals
  • informational retrieval

This work has radically challenged conventional approaches. Research for speech and audio show outstanding performance for various cases:

  • over-determined, where the number of channels is larger than the number of sources
  • determined, where both are equal
  • under-determined, where the number of channels is smaller than the number of sources

Under-determined cases include binaural and single channels.

The work has contributed to the fundamentals of nonlinear signal processing theory. It has exploited advanced machine learning techniques. It paves the way for the research community to adopt and develop interdisciplinary solutions.


We have received funding from research councils and industry, including:

  • Engineering and Physical Sciences Research Council (EPSRC)
  • Biotechnology and Biological Sciences Research Council (BBSRC)
  • Newton Fund
  • Ministry of Defence (MoD)
  • DSTL
  • Thales