Integrate Bluetooth Joystick to mobile VR set

[AndriyPt]

Joystick will be used for teleoperation.
Mobile browser should read commands from joystick and move robot accordingly.
There should be ability to calibrate joystick on the web page.

[MaxZhaloba]

@AndriyPt, Do I understand you correctly that you suggest to use the Web Bluetooth API? It's still being developed and not a standard yet. https://developer.mozilla.org/en-US/docs/Web/API/Web_Bluetooth_API
Current support in browsers.

[AndriyPt]

@MaxZhaloba I do not have any preference in solutions. Feel free to choose any :)

[MaxZhaloba]

At first it appeared to me, that the most straightforward way will be just to initialise the Blutooth connection between the joystick and the robot. This will eliminate the necessity to implement the Web BT API, which is currently a very new standard. It's not supported by Safari so far, and the support in Chrome might vary between platform and OS version (if supported in iOS or requires Android 6+ only), so this question needs investigation.

But, on second thought, I realised that we cannot connect the joystick to robot, because the Blutooth radius is too short (typically up to 10 meters) and the robot can be located on a great distance from operator. So, joystick needs to be connected to the VR smartphone in any case. And the phone will send the joystick data to the robot via WiFi. This means that Web BT is again an option for us (option 1).

Otherwise we will need to develop (or find and reuse) a dedicated mobile application (option 2) that will perform one of the following:
a) Connect to ROS and send the joystick data directly as ROS packet
b) Connect to some other server running on robot or a dedicated PC. This server will convert the joystick data (received in a custom or standard form/protocol) to ROS packets. Ideally, such mobile application should work on both Android and iOS.

So, for the beginning, I'm going to investigate:

1. on which platforms the Web BT actually works
2. if there exist any ready open-source solutions for iOS/Adroid BT server which allow to connect a joystick and send data over network.

@AndriyPt, how does this all sound to you?

[AndriyPt]

@MaxZhaloba sounds good to me. Also I would check how web browser is reacting on joystick commands. Maybe it could read some key pressed events.
Also you could have a look on this application https://play.google.com/store/apps/details?id=com.robotca.ControlApp&hl=en

[MaxZhaloba]

I've just tested the BT joystick on iOS 11.2.6. It reacts only to pressing the front bottom button (button 2) and button B. Other buttons adjust the volume or give no response.

So, all browsers (Safari, Chrome and Firefox) return such object on keydown event:

{"isTrusted":true}

The kypress data is practically identical for both buttons and on all browsers:

{"originalEvent":{"isTrusted":true},"type":"keypress","target":{},"currentTarget":{"location":{"href":"http://192.168.1.134:8080/","protocol":"http:","host":"192.168.1.134:8080","hostname":"192.168.1.134","port":"8080","pathname":"/","search":"","hash":"","origin":"http://192.168.1.134:8080","ancestorOrigins":{}},"jQuery3310245588629896824621":{"events":{"keypress":[{"type":"keypress","origType":"keypress","data":null,"guid":5,"namespace":""}]}}},"timeStamp":21263337817791000,"jQuery331024558862989682462":true,"delegateTarget":{"location":{"href":"http://192.168.1.134:8080/","protocol":"http:","host":"192.168.1.134:8080","hostname":"192.168.1.134","port":"8080","pathname":"/","search":"","hash":"","origin":"http://192.168.1.134:8080","ancestorOrigins":{}},"jQuery3310245588629896824621":{"events":{"keypress":[{"type":"keypress","origType":"keypress","data":null,"guid":5,"namespace":""}]}}},"handleObj":{"type":"keypress","origType":"keypress","data":null,"guid":5,"namespace":""},"data":null}

event.which returns code 85 for both buttons.

I've also noticed a difference in handling of button 2 and button B. When joystick is paired with smartphone, and I disable the bluetooth connection, then enable it and the phone connects to joystick, those two buttons act for adjusting the volume just as other buttons. When I leave the blutooth connection enabled, switch off the joystick and then switch it on, I get a popup message on my phone, stating that joystick wants to initiate the connection. After I allow it, button B and button 2 start to trigger the events in browsers.

To summarise, this approach doesn't work for us, because most buttons do not trigger browser events, neither the X/Y manipulator does.

[AndriyPt]

@MaxZhaloba please check specification of the joystick here https://www.kotulas.com/wcsstore/KotulaStorefrontAssetStore/images/pdf/54632.pdf
There are different modes which could be switched. Please give a try to VR and Mouse modes and let us know how browser reacts to these buttons ;)

[MaxZhaloba]

Hooray! This joystick recognizes all buttons and the X/Y thumbstick in additional modes.

It triggers 3 events (keydown, keypress, keyup) when I press any key, and then 3 events again when I release it. The event.which property returns unique scancodes which are different when I press and release any button.

Here's the detailed info. Each key pressing typically triggers 2 keypress events which are sent when the corresponding button is pressed and released. This data is collected in iOS Safari.

Mode	Button	Code when pressed	Code when released	Remark
A	Up	Page scrolling
A	Down	Page scrolling
A	Left	Page scrolling
A	Right	Page scrolling
A	1	Volume on/off
A	2	Volume down
A	@	Play/pause
A	Power	-
A	A	Volume on/off
A	B	Volume on/off
A	C	Ring volume down
A	D	Minimize browser
B	Up	1092	1081	m s1089 s1074 (explained below)
B	Down	-	-
B	Left	1095	1103	m
B	Right	o1091	1094	s1089 s1074
B	1	1075	1072
B	2	1088	1082
B	@	-	-
B	Power	-	1096 & 1100
B	A	1075	1072
B	B	1088	1082
B	C	1085	1077
B	D	1086	1090
C	Up	1094	1091	m s1095 s1103 (on slight pitch)
C	Down	o1095	-	m+s
C	Left	1092	1081	m s1089
C	Right	1074	1089	m s1095 s1103
C	1	1075	1072
C	2	1088	1082
C	@	-	1086 & 1090
C	Power	-	1077 & 1085
C	A	1075	1072
C	B	1088	1082
C	C	-	-
C	D	-	-
D	Up	1094	1091	m s1095 s1103
D	Down	1095 (only after Up)	-	m
D	Left	1092	1081	m s1074 s1089
D	Right	-	-	m s1089 s1084
D	1	1075	1072
D	2	1088	1082
D	@	-	1086 & 1090
D	Power	-	1077 & 1085
D	A	1075	1072
D	B	1088	1082
D	C	-	-
D	D	-	-

Legend:
*m - Moves the page
*s - Generates this code (event) in some (odd) cases. Sometimes this depends on thumbstick manipulation speed.
*o - Sometimes this code and browser event is omitted and the thumbstick gives no reaction

It seems that the thumbstick supports a kind of differentiation in movement. Also, there's memory which is seemingly designed to initiate continuous movement, e.g. in some cases Down triggers the event only if Up was pressed right before it. I'll investigate in more detail and get back with results.

[MaxZhaloba]

ROS Control looks interesting, but unfortunately I don't have any Android device to test it. I'd like to come up with solution that would support the BT joystick on iOS too.

[MaxZhaloba]

It turns out that Web Bluetooth is not supported under iOS by any browser so far (status). Actually, there's one program which implements support in iOS. Interestingly, it's open-source but costs 2$ in AppStore.

Also, Google Chrome supports Web BT only on sites downloaded via HTTPS (source).

Web BT can be tested using these pages.

It seems that we won't be able to utilise this technology at the moment.

[MaxZhaloba]

I've googled the word combination "bluetooth over ip" (exact match). It turns out to be rather a theoretical thing, and nobody has implemented this so far. Here's a stackoverflow thread with the reasoning. So, at the bare minimum, the app on the phone will need to recognise the joystick and send its data in some other format than encapsulated bluetooth.

[MaxZhaloba]

@AndriyPt, what if we use another phone with gyro and magnetometer to control the robot?

I think it might produce even better user experience than a real joystick with thumbstick. It's more intuitive to rotate and tilt the phone in your hands rather than to use a thumbstick, especially when you're in VR and you don't see it.

Nowadays not everyone has a joystick, but it's easy to find another phone or lend it from your relatives :)

And this will be easier in terms of implementation, since we already know that reading from IMU works well in web pages. And it's cross-platform. I'd prefer to focus on this approach as primary instead of the original request. This will work better for the demo and can make an impression.

[MaxZhaloba]

I can see 3 options:

1. Read the IMU data from the same phone that operator uses as VR set. In this case, his head tilt will cause the robot to move and rotate, whereas head rotation will be translated to robot head.
   I'm doubtful if this is a fairly good option in terms of user experience. It might cause vertigo and dizziness. But we cannot know unless we actually try it. I, personally, haven't payed any real VR video games so far, so for me it's hard to tell in advance.
2. The operator will use a second phone to control the robot. Screen tap, sliding or gestures can invoke additional actions.
3. Combination of option 1 and 2. Read some data from the VR phone, and other data from the hand-manipulation phone. From what I see, it would be natural to transfer operator's head and body rotation to robot head rotation. And we can still read the tilts/rotation from the hand-phone. Sounds like a more suitable option.

[AndriyPt]

@SystemDiagnosticss Do you have Android phone? This might be good demo for presentation. I guess for now it could be limited to Andoid phones.

[SystemDiagnosticss]

@AndriyPt Yes :)

[MaxZhaloba]

@AndriyPt, Why don't you like an approach with second phone to control the robot?

If you'd still like to use the approach with hardware joystick I can implement it for the one that you gave me using the keypress event. But then the implementation will be hardwired to this model unless we add some configuration page. The only thing is that it would be nice to check if this joystick sends the same codes on Android as in table above.

[AndriyPt]

@MaxZhaloba for demo purposes I would like to have bluetooth joystick integrated.
Using keypressed codes. Please talk with @SystemDiagnosticss to get Android phone for testing.
Regarding other approaches for teleoperation (two phones, keyboard, gesture tracking device, cyberglove etc) it is fine to have them but after demo :)
We could discuss it after next project meeting.

[MaxZhaloba]

@SystemDiagnosticss, when you'll be adding joystick key mappings please implement them as JS arrays (e.g.: var keyUp = [1092, 1094];) or in other flexible way.

[MaxZhaloba]

This issue has 2 open branches in repository: F#86_bt_joystick and F86_bluetooth_joy.

@SystemDiagnosticss, is it ok to continue the development in F86_bluetooth_joy?

[SystemDiagnosticss]

@MaxZhaloba in this brunch (F86_bluetooth_joy) old version vr.js and main.js . I think better solution for you is create new branch for this issue

[MaxZhaloba]

@AndriyPt, We've investigated the availability of Gamepad API using our BT joystick on different platforms with @SystemDiagnosticss. We used such tools as https://html5gamepad.com/ and http://luser.github.io/gamepadtest/. Here are our results.

OS	Browser	Performed by	Result
iOS	Safari, Chrome and Firefox	@MaxZhaloba	Not recognized
Android	Chrome	@SystemDiagnosticss	Supported
Ubuntu	Chrome and Firefox	@MaxZhaloba	Not recognized
Windows 8 and 10	Chrome	@MaxZhaloba	Supported in modes B and C

I'm doubtful whether this issue is something to do with this particular joystick or all BT joysticks in iOS. I think we need another BT joystick to compare the results since Can I Use clearly states that iOS is supported.

If testing with another joystick wouldn't make any difference then such coverage won't make it possible to support the iOS at the moment and – for me – to develop the joystick configuration tool. So, I'm not sure if we could stick to Gamepad API in this case. We might limit the support to Android devices and extend the support when Gamepad API becomes available on iOS.

[AndriyPt]

@MaxZhaloba thank you for good analysis. I guess sticking with Android for now is good idea.

[MaxZhaloba]

@SystemDiagnosticss, are the more recent version of these libraries already in kinetic-devel? If so, then they should have been updated when I've merged it into F86_bluetooth_joy.

[MaxZhaloba]

@AndriyPt This feature is implemented on the whole. I saved the configuration data to local storage (several lines of code) to make it running. I'd prefer a separate task for backend storage functionality or combination of both storage options to keep this issue from bloating.

There's one issue with the joystick I ended up applying a workaround so far. Axis #⁠9 gives some unuseful data and misleads the configuration routine. Ideally, I'd like to ignore the data from this axis somehow. Currently this scenario is resolved with magic number at config.js line 265. Please let me know if you have any ideas how to approach this issue.

leobot/leobot_web_server/web_content/js/config.js

Line 265 in cbed16f

if (absPitch > pitchThreshold && i!==9) {

To facilitate the testing of configuration wizard on PC I added the keypress handlers for such keys: Enter, Space, Backspace and letter R.

At this stage the wizard allows to assign only one button or thumbstick movement for each robot action. We can extend this functionality in future.

Remaining subtasks required to finish this task:

1. Download and substitute links for local copies of JS libraries.
2. Test the functionality on Android with @SystemDiagnosticss.
3. Some bugs occuring during configuration in corner cases.
4. Other issues which will be found during testing and later.

Other related functionality (good to have at some point):

1. Validate the configuration before saving. Currently the configuration is saved "as is".
2. Restyle the UI for mobile layouts.

Please let me know what you think on this matter.

[AndriyPt]

@MaxZhaloba thank you for detailed report.
Whenever code is ready for review feel free to create Pull Request.
Regarding your question: What is Axis #9?

[MaxZhaloba]

@AndriyPt, Gamepad API represents joystick state as number of buttons and axes. Axes have a number and a value in range [-1; 1]. It could have been sufficient to use 2 axes for this gamepad but in fact it's represented with 10.

Axes ## 0 and 1 return continuous readings for horizontal/vertical thumbstick tilt (depends on mode B/C).
Axes ## 2-8 return some values which never change.
Axis #⁠9 returns constant values.

I had't understood it before but now I've found out that axis #⁠9 represents thumbstick position in polar coordinates, where 3.28571 corresponds to center, despite the spec says that axis data should change in range [-1; 1].

Here's the data for mode B:

	Left			Right
Top	0.42857	0.71429	0.71429	0.71429 / 1
	0.14286	3.28571	3.28571	1
	-0.14286	3.28571	3.28571	-1
Bottom	-0.42857	-0.42857	-0.71429	-1 / -0.71429

The axis data changes discontinuosly so that these are all possible values.

The difference in mode C is that notional zero is found at bottom position and all values turn 90 deg counterclockwise.

Calibration routine detects the axis which gives the max delta from centered position. For this reason the axis #⁠9 always wins.

Off the top of my head I have an idea that we could detect axes which return non-standard readings out of the range [-1; 1] in centered position and ignore them during joystick calibration.

[AndriyPt]

@MaxZhaloba thank you for explanations.
A few questions:

1. Why not just use axis 1 and 2 values?
2. Polar coordinates of the centre should be represented by two values. What does 3.28 means? In which units is it measured?

[MaxZhaloba]

@AndriyPt

1. The configuration routine is flexible and unrelated to certain joystick model. Some joysticks have more than one thumbstick (i.e. DualShock) so I want to make it possible to configure and use any of available axes.
2. We don't know. This is actual data I captured. I interpret it as angle in polar system with special value for centered position, which is read as 3.28. If you know the joystick model I can try to find some official specifications on this issue.

[MaxZhaloba]

@AndriyPt, as for the units – as i see it – the value for axis #⁠9 is measured in their own normalized units which equal to rad/pi or deg/180. They cannot use radians or degrees becase according to Gamepad API the axis values sould be found in range of [-1; 1].

[MaxZhaloba]

@AndriyPt, I have some news related to testing of configuration page. We've tried to connect the Joystick to @SystemDiagnosticss Android phone and it remained unvisible despite the site https://html5gamepad.com showed the joystick. Now I've returned home and launched the Lineage OS Android 7.x x64 build from flash drive on my laptop. It supports my blutooth adapter and it connected to jostick successfully. I've tested the configuration page on the latest Chrome and it works fine despite joystick is represented with significantly different software interface. More specifically, it doesn't expose any axis data and all thumbstick movements are represented as button pressings. Such use case is supported by the configuration routine.

Interestingly, my 2.4 GHz mouse is also available via Gamepad API and added to list of devices for configuration under this Android x64 OS.

Long story short, I don't have any idea how to debug this issue which occurs on Igor's Android phone since I don't know where to lend a physical Android device for longer term and under Android for PC everything works fine.

[MaxZhaloba]

Actually, we can meet somewhere in the city with @SystemDiagnosticss to investigate this issue more thoroughly.

[MaxZhaloba]

Hi @AndriyPt,

I've tested that vr.html receives button and axis pressing from joystick on Android. But there's some issue with sending of commands to ROS so that robot doesn't move in simulation. All movement actions are implemented as functions so it will be easy to adjust them for all input methods.

Joystick configuration is performed on config.html page. The implementation is available in branch F86_bluetooth_joy.

I'm finishing my work on this task. Please reassign according to your priorities.