A Hybrid Indoor Positioning System for the Blind and Visually Impaired Using Bluetooth and Google Tango

Abstract

Blind & visually impaired individuals often face challenges in wayfinding in unfamiliar environments. Thus, an accessible indoor positioning and navigation system that safely and accurately positions and guides such individuals would be welcome. In indoor positioning, both Bluetooth Low Energy (BLE) beacons and Google Tango have their individual strengths but also have weaknesses that can affect the overall usability of a system that solely relies on either component. We propose a hybrid positioning and navigation system that combines both BLE beacons and Google Tango in order to tap into their strengths while minimizing their individual weaknesses. In this paper, we will discuss the approach and implementation of a BLE- and Tango-based hybrid system. The results of pilot tests on the individual components and a human subject test on the full BLE and hybrid systems are also presented. In addition, we have explored the use of vibrotactile devices to provide additional information to a user about their surroundings.

Introduction

Blind & visually impaired (BVI) individuals often face challenges in wayfinding and navigation, especially in unfamiliar indoor environments. There is, therefore, a great need for reliable, indoor location-based services to assist BVI users in unfamiliar indoor environments. To address this need, we assessed the utility of combining Bluetooth Low Energy (BLE) beacons and a Google Tango device as a robust system for indoor positioning and navigation. Both BLE beacons and Tango have their respective strengths; however, their individual weaknesses prevent them from being fully used in indoor assistive navigation. We hypothesize that a hybrid of the two systems could form an extremely accurate and useful indoor localization system for the visually impaired. We also hypothesize that users can benefit from a supplementary system that can alert them of obstacles in their vicinity (such as walls and people). We tested this using vibrotactile sensors placed on the user’s wrists. The paper follows the following outline:

1. We describe the approaches and implementations of the individual BLE and Tango system, as well as those of the hybrid and supplementary vibrotactile systems.
2. We discuss the results of pilot tests and identify the technical limitations and advantages of the individual BLE and Tango systems.
3. We report the results of and discuss the conclusions drawn from a human subject test on the standalone BLE and the hybrid systems (and the vibrotactile supplement).

Related Work

Recent advances in computer vision have provided platforms for developing assistive technologies for the visually-impaired, particularly using Google Tango. Li, et al (2016) have proposed ISANA, context-aware indoor navigation implemented using Tango. Work by Winterhalter, et al (2015) utilized a Tango device and particle filter localization, instead of the Tango onboard SLAM, to estimate the 6DOF pose of the Tango device within a 2D floor plan, which eliminates an initial mapping phase with the Tango. External RGB-D cameras have also been used to build a vicinity map based on 3D data and perform path planning to provide 3D traversability (Schwarze, et al, 2016).

There have also been numerous projects which have used BLE beacons in assistive navigation. Perhaps the most relevant is NavCog, a “smartphone mobility aid” which solely used BLE beacons to provide turn-by-turn navigation and information about nearby points-of-interest (Ahmetovic, et al, 2016). Another system (Bohonos, et al, 2007) proposed the use of BLE beacons as part of a system to provide the visually impaired with information about the topology of an approaching urban intersection. Further work proposed beacons as part of an indoor “traffic sign system” for the cognitively-impaired that downloads images with directional instructions onto a device when the user reaches a hallway intersection (Chang, et al, 2008).

Discussion

In the following section, we discuss the system components used in our proposed hybrid system and present the results of human subject experiments.

System Components

For our preliminary evaluation of the system components, we performed pilot tests at Lighthouse Guild, a center providing vision and healthcare services for BVI in NYC. The pilot tests with the BLE system were performed using a Samsung Galaxy S4, and the tests with the Tango system were performed using a Lenovo Phab 2 Pro.

BLE Beacons

BLE beacons broadcast an identifier and other relevant information to nearby receiver devices. Specifically, we used Proximity and Location Beacons from Estimote (shown in the inset in Figure 1). These beacons have a small form-factor, long battery life, and are relatively low-cost.

With these beacons, we could use one of many different methods to utilize the Received Signal Strength Indicator (RSSI) value of each beacon and, thus, localize a device. We found that BLE signals are extremely noisy, because they are easily attenuated by materials commonly found in a building (Kara and Bertoni, 2001). Thus, we opted to use the “fingerprinting” method, which compares the current RSSIs with pre-built snapshots (or “fingerprints”) of the area’s radio landscape (Subhan, et al, 2011). Fingerprinting inherently looks for similarity and not exactness in RSSIs and is, thus theoretically, very suitable for rapidly fluctuating signals.

Figure 1 shows a detailed map of our beacon testbed. Our test system used fingerprints which were captured in an offline run before our tests. During localization, the current signal conditions were compared to the pre-recorded fingerprints using a custom variation of the k-nearest neighbor algorithm, which assigned different weights to fingerprints based on reliability and calculated the uncertainty of the final position.

Due to the noisy nature of the RSSIs, we observed that the predicted location tended to jump around. We also saw that even slight variations to our algorithm produced different results with varying degrees of accuracy (Figure 2). However, we also saw that the system seemed to be trying to follow the general trend found in the true path. Thus, the beacons may prove useful in calculating a coarse location for the user; however, greater accuracy would be required for a fine location application.

Google Tango

A Google Tango-enabled device contains an RGB-D camera that has been integrated into an Android device (Li, et al, 2016) with capabilities of 6-degrees-of-freedom VIO (visual-inertial odometry) and feature-based indoor localization, which allows the device’s pose (orientation and position) to be estimated as it moves through a 3D environment, without the use of GPS or other external signals. Tango also includes functionality for recording an Area Description File (ADF), a feature map of an indoor environment in a compressed format, which allows for re-localization within that environment. By utilizing these built-in capabilities of the Tango device, we could create detailed 3D models of indoor environments (Figure 3) and achieve extremely accurate real-time indoor localization.

During our pilot tests, we found that the Tango device was prone to small errors during the estimation of each camera position. (Sensors are sensitive to some noise and are not 100% accurate.) This made the estimated position of the device drift over time, causing drift error (maximum = 0.38 m, average = 2.9 cm). The Tango API itself attempts to correct some of this drift error by utilizing a loop closure approach. While an ADF is being recorded and the device returns to the origin of the ADF (thus closing the loop), Tango will attempt to correct the accumulated drift errors by adjusting the estimated trajectory to match the real trajectory. Furthermore, we encountered another limitation with the Tango device in that there is a limit on the size of an ADF (approximately one floor of a building).

BLE-Tango Hybrid System

In order to overcome the technical limitations of the individual systems, a hybrid localization system that utilizes BLE beacons and a Google Tango device was implemented:

A Tango device was used to record an ADF (feature map) for each floor in a building. Given that a 2D floor plan of each floor was available, an affine transformation was utilized to translate, scale and orient the 3D ADF to align with the floor plan (Figure 4). The affine transformation function in the OpenCV library was used, which given 3 pairs of corresponding 2D points, returns a 2 x 3 transformation matrix M. We assumed that all 3D pose information returned by the Tango API was on a plane (the floor), and therefore the z-component of the 3D coordinates could be discarded. The 2D affine transform could now be applied. The position of the user was then tracked on an adaptive app interface consisting of the 2D floor plan of the environment that the user was currently in.

As mentioned previously, the ADF size limitation is approximately one floor. However, since beacons have proven to be reliable in determining a rough location, they could be used to determine the floor a user is on and automatically load the corresponding ADF. Alternatively, if the floor is too large (and thus consists of multiple ADFs), we could use the location returned from the beacons to select the ADF that is closest to this coarse location.

Map Annotation

A map annotation module was developed in order to place waypoints on the 2D floor plan associated with the ADF. Once Tango has mapped a floor and is localized, waypoints can be marked on the 2D floor plan using the app’s adaptive interface. The waypoint on the map will correspond to the real-world location in the 3D ADF (Figure 5, left).

Navigation

An application was developed for blind and visually impaired individuals, consisting of a fully-developed “Navigator” component, which provided turn-by-turn navigation using either BLE beacons, Google Tango, or BLE-Tango hybrid localization (Figure 5, right). Using the waypoints set by the map annotation module on the 2D floor plan, a visually impaired user can receive turn-by-turn audio directions to navigate to any destination. When the user selects their destination, the app executes Dijkstra’s algorithm (Dijkstra, 1959) to determine the shortest path between the origin and destination nodes; it is this path that the Navigator will guide the user along.

Vibrotactile Assistance

We also examined the use of vibrotactile devices as a supplement to our application so that a user could gain an awareness of their immediate surroundings. We used simple, small 3D-printed devices that have infrared sensors that point straight out of the top portion of the device. The devices are attached to the user’s wrists using straps. When the device’s infrared sensor detects a close object, it vibrates (with the vibration speed dependent on the proximity of the object to the sensor). By moving their wrists, a user could gain an idea of how close they are to various objects/obstacles (such as walls) in addition to their relative orientation to these obstacles.

Human Subject Experiments

In order to evaluate the usability of the BLE and hybrid systems and the vibrotactile supplement by BVI individuals, we performed human subject experiments at Lighthouse Guild’s new building. For these experiments, we exclusively used the Lenovo Phab 2 Pro for both pure BLE and hybrid navigation. This study was approved by the Institutional Review Board of the City University of New York.

Participants & Materials

A convenience sample of 11 adults who were diagnosed as totally blind, legally blind, partially sighted, or low vision were offered participation in the study. There were 9 (81.8%) participants 55 years old or older, 1 (9.1%) participant 45-54 years old, and 1 (9.1%) participant 18-24 years old. In our study, we had 4 (36.4%) females, and 7 (63.6%) males. In total, 6 (54.5%) participants were diagnosed with total blindness, 4 (36.4%) participants were diagnosed with low vision, and 1 (9.1%) participant was partially sighted. We administered two types of surveys, a pre-experiment survey and a post-experiment survey. The pre-experiment survey included a demographics section, which asked the participants to disclose their gender, age, level of visual impairment, and a section to assess the participants’ familiarity with smartphones and their difficulty in indoor navigation. The post-experiment survey was divided into two sections assessing the perceived helpfulness, safety, ease of use, and overall experience of (1) the BLE and hybrid navigation apps and (2) supplementing hybrid navigation with vibrotactiles.

Procedure

The participants were divided into two groups, and both groups completed 6 navigation trials (paths) as shown in Table 1 below. The participants used their white cane or guide dog in all but one trial. The purpose of the navigation experiments was to compare different methods: (1) hybrid navigation vs. BLE navigation; (2) hybrid navigation vs. hybrid navigation supplemented with vibrotactiles; and (3) hybrid navigation vs. hybrid navigation supplemented with vibrotactiles only (without a white cane or guide dog). Each pair of paths (1 + 2, 3 + 4, 5 + 6)were virtually identical (equal distance, amount of turns, and narrowness) which allowed us torecord 11 data points for each method.

Table 1. Navigation experiments

Path	Group 1	Group 2
1	Hybrid	BLE
2	BLE	Hybrid
3	Hybrid	Hybrid (+ vibrotactiles)
4	Hybrid (+ vibrotactiles)	Hybrid
5	Hybrid	Hybrid (+ vibrotactiles ONLY)
6	Hybrid (+ vibrotactiles ONLY)	Hybrid

Results

The pre-experiment survey found that most of the participants relied on others for assistance in navigating indoors, with over 50% stating they relied extremely on others. As can be seen in Figure 6, although most of the participants claimed to find it easy to navigate indoors in familiar environments, almost all of them found it difficult to navigate in unfamiliar indoor environments.

Figure 7 shows the trajectories taken by one subject using BLE and Hybrid navigation. Paired-samples t-tests were conducted to compare duration, total bumps, and total researcher interventions (reminders) between methods. Comparisons were made between (1) BLE-based navigation and hybrid-based navigation, (2) hybrid-based navigation and hybrid-based navigation supplemented with vibrotactiles, and (3) hybrid-based navigation and hybrid-based navigation supplemented with vibrotactiles only (i.e., without a white cane or guide dog).

We made the following findings: (1) Our primary finding was that hybrid-based navigation required significantly less interventions and assistance from the researchers than BLE-based navigation (p = 0.0096); there was no significant difference in trip duration and total number of bumps between BLE- and hybrid-based navigation (Table 2). (2) We found no significant differences in trip duration, total number of bumps, and total number of researcher interventions and assistance between hybrid-based navigation supplemented with and without vibrotactiles (Table 3). (3) We found no significant differences in trip duration, total number of bumps, and total number of researcher interventions and assistance when comparing the hybrid navigation app and replacing the user’s usual aid (cane or guide dog) with vibrotactiles against not replacing the user’s usual aid with vibrotactiles (Table 4).

The post-experiment survey found that six participants (54.5%) stated that they preferred the Hybrid app, three (27.3%) preferred the BLE app, and two (18.2%) had no preference. About two-thirds (54.5% – somewhat worse and 9.1% – much worse) of the participants agreed that the BLE app was worse than the Hybrid app, but approximately 80% (45.5% – somewhat better and 27.3% – much better) of the participants agreed that the BLE app was better than no navigation app for indoors at all.

Table 2. Results of t-test and Descriptive Statistics for BLE and Hybrid navigation (M = mean, SD = Standard Deviation, n = sample size, t = test statistic, p = probability value, df = degrees of freedom; 95% confidence interval).

	BLE M	BLE SD	BLE n	Hybrid M	Hybrid SD	Hybrid n	t	p	df
total interventions	1.27	0.64	11	0.36	0.50	11	3.19	0.0096	10
trip duration(s)	75.18	17.12	11	64.72	15.13	11	1.65	0.12	10
total bumps	0.09	0.30	11	0.18	0.40	11	-1.00	0.34	10

Table 3. Results of t-test and Descriptive Statistics for Hybrid Navigation supplemented with Vibrotactiles (in table: “Vibros”) (M = mean, SD = Standard Deviation, n = sample size, t = test statistic, p = probability value, df = degrees of freedom; 95% confidence interval)

	Hybrid M	Hybrid SD	Hybrid n	Hybrid  + Vibros M	Hybrid  + Vibros SD	Hybrid   + Vibros n	t	p	df
total interventions	0.36	0.50	11	0.54	0.52	11	0.80	0.44	10
trip duration(s)	37.18	11.36	11	46.45	31.13	11	0.97	0.35	10
total bumps	0.36	0.50	11	0.18	0.40	11	-1.00	0.34	10

Table 4. Results of t-test and Descriptive Statistics for Hybrid Navigation supplemented with Vibrotactiles (in table: “Vibros”) ONLY. (M = mean, SD = Standard Deviation, n = sample size, t = test statistic, p = probability value, df = degrees of freedom; 95% confidence interval)

	Hybrid M	Hybrid SD	Hybrid n	Hybrid + Vibros only M	Hybrid + Vibros only SD	Hybrid + Vibros only n	t	p	df
total interventions	0.36	0.50	11	0.54	0.68	11	0.69	0.50	10
trip duration(s)	45.54	12.54	11	50.09	16.50	11	0.63	0.53	10
total bumps	0.09	0.30	11	0.27	0.46	11	1.00	0.34	10

Approximately 90% of the participants agreed that the Hybrid app was helpful, 90% agreed that using the Hybrid app felt safe, and 100% agreed that they could easily reach their destination when using it (see Figure 8). Supplementing the Hybrid navigation system with vibrotactiles for obstacle avoidance was well-received by the participants. Approximately two-thirds of the participants (see Figure 9) felt safe when their canes or guide dogs were replaced with the vibrotactiles.

Conclusions

Through our work, we have determined that a hybrid system composed of both BLE beacons and Google Tango was both accurate and robust. Tango is capable of providing a highly accurate, fine location; however, it can only create feature maps of approximately one floor at a time. Since the BLE beacons are excellent at coarse location detection, we used them to determine which ADF Tango must load in order to report the correct position to the user. According to subject evaluations, users felt that the BLE-Tango hybrid system helped them reach their destinations easily and safely. Although users perceived the hybrid system as better than the standalone BLE system, the BLE system was equally successful in guiding users to a destination quickly, safely, and with little assistance from the researchers. This finding is noteworthy since with a BLE system alone, a Tango sensor is not needed, the smartphone’s power consumption would be less, and any device (not just a Google Tango device) could be used for navigation. The vibrotactiles supplement proved to be a promising avenue for future work – as no decreases in speed and safety were observed when the standard cane or guide dog was replaced with vibrotactile devices.

Acknowledgments

This research was performed under appointment to the U.S. Department of Homeland Security (DHS) Science & Technology (S&T) Directorate, Office of University Programs, Summer Research Team Program for Minority Serving Institutions, administered by the Oak Ridge Institute for Science and Education (ORISE) through an interagency agreement between the U.S. Department of Energy (DOE) and DHS. ORISE is managed by ORAU under DOE contract #DE-AC05- 06OR23100 and #DE-SC0014664. All opinions expressed in this paper are the author’s and do not necessarily reflect the policies and views of DHS, DOE, or ORAU/ORISE. This work is also supported by the U.S. National Science Foundation (NSF) through Award #EFRI-1137172, Award #CBET-1160046 and Award #CNS-1737533, the VentureWell (formerly NCIIA) Course and Development Program (Award #10087-12), a Bentley-CUNY Collaborative Research Agreement 2017-2020, as well as NYSID via the CREATE (Cultivating Resources for Employment with Assistive Technology) Program. We would like to thank Manjekar Budhai and Dr. Hao Tang for their assistance in preparing for our human subject test. We would also like to thank our subjects for their participation and cooperation as well as for providing extremely helpful feedback in improving our system.

Works Cited

Ahmetovic, Dragan, et al. “NavCog.” Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services – MobileHCI ’16, 2016.

Bohonos, S., et al. “Universal real-time navigational assistance (URNA).” Proceedings of the 1st ACM SIGMOBILE international workshop on Systems and networking support for healthcare and assisted living environments – HealthNet ’07, 2007.

Chang, Yao-Jen, et al. “Mobile computing for indoor wayfinding based on bluetooth sensors for individuals with cognitive impairments.” 2008 3rd International Symposium on Wireless Pervasive Computing, 2008.

Dijkstra, Edsger W. “A note on two problems in connexion with graphs.” Numerische Mathematik, vol. 1, no. 1, 1959, pp. 269-271.

Kara, Ali, and Henry L. Bertoni. “Blockage/shadowing and polarization measurements at 2.45 GHz for interference evaluation between Bluetooth and IEEE 802.11 WLAN.” IEEE Antennas and Propagation Society International Symposium. 2001, 2001.

Li, Bing, et al. “ISANA: Wearable Context-Aware Indoor Assistive Navigation with Obstacle Avoidance for the Blind.” Lecture Notes in Computer Science, 2016, pp. 448-462.

Schwarze, Tobias, et al. “A Camera-Based Mobility Aid for Visually Impaired People.” Künstliche Intelligenz, vol. 30, no. 1, Feb. 2016, pp. 29-36.

Subhan, Fazli, et al. “Indoor positioning in Bluetooth networks using fingerprinting and lateration approach.” 2011 International Conference on Information Science and Applications, 2011.

Winterhalter, Wera, et al. “Accurate indoor localization for RGB-D smartphones and tablets given 2D floor plans.” 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2015.

---

Vishnu Nair, Christina Tsangouri
Dept. of Computer Science, The City College of New York (CUNY)

Bowen Xiao
Dept. of Computer Science, The City College of New York (CUNY), Academy of Eng. and Design Tech., Bergen County Academies

Greg Olmschenk, Zhigang Zhu
Dept. of Computer Science and the Graduate Center, The City College of New York (CUNY)

William H. Seiple
Lighthouse Guild, New York

---

Keywords

Indoor navigation; indoor positioning; Bluetooth Low Energy (BLE); beacons; Google Tango, assistive technology