WebRTC Application with Virtual Board integration using Ionic
Building a video call app integrated with a virtual board to interact and collaborate using Web Sockets and WebRTC.
Having spent most of my time, during the pandemic, working with teams over Zoom, Google Meet and countless other video conference apps, I noticed most video chat applications don’t provide a virtual board for peers to collaborate and share their thoughts on. I was then inclined to explore development on this horizon and technologies that power them.
After some quick research on WebRTC, I came across important terms like Signaling, STUN & TURN Servers, ICE candidates, Offers and Answers which helped me gain further insight on how various operations work.
For rapid, cross platform compatible development I chose to proceed with latest Ionic 5 along with Apache Cordova to handle device specific hardware interactions. Ionic, with its strong developer community, helps build cross-platform mobile first applications with a single codebase. This helps you to charge forward with ability to deploy it as either a web app, iOS/android builds or even a PWA.
Well that being said, WebRTC provides real-time peer to peer direct communication capabilities on all modern browsers as well as on native clients for all major platforms. It supports video, voice, and generic data to be sent between peers, allowing developers to build powerful voice and video communication solutions.
Working
A WebRTC application will usually go through a common application flow. Accessing the media devices, opening peer connections, discovering peers, exchanging ICE candidates and start streaming. This process of exchanging device and network details prior establishing stable connection is facilitated using Web Sockets.
The following code shows a way to create “offers”, a Session Description Protocol message, to be sent to another peer:
With the above setup, peers can create and send offer via Web Sockets. A sample code that listens for offers and send back “answer” is as follows :
Peers must share information about the media along with network details. This is known as ICE candidate and exchanged between peers. The below function is called when icecandidate event occurs on an RTCPeerConnection instance. For further details refer documentation . A sample code that sends ICE Candidates between peers is as follows :
That sums up the video calling feature of the app.
The virtual board uses event handlers to detect touch, stylus pen and mouse movements. Every stoke over the board is registered as a list of data points and broadcast to all room users using web servers, deployed on Heroku. Over the receiver’s end, the data points is appended to a list and entire board is redrawn. This helps reduce the size of data being transferred significantly and thereby reduces load on the server. Features included are ability to undo and reset the board content, grant and revoke edit access to the virtual board, adjust stylus color and size, timeline and scaling content based on device screen size.
Note: WebRTC applications require a server for relaying the traffic between peers, since a direct socket is often not possible between the clients, unless they reside on the same local network. Running a TURN Provider is resource intensive and I found none of those “free” TURN providers like this actually work. Thus this application is currently limited to clients residing on same local network.
To my understanding, most video chat applications don’t provide a virtual board for peers to collaborate and share their thoughts on. My current application, CanvaShare, does the exact same by allowing users to not just connect over video call but also interact using a virtual board with access controls included. The data is transferred to individual room members using web sockets and not stored on any database for better security and storage cost.
Conclusion
I have built ‘CanvaShare’ — an video call app with virtual board integration using Ionic 5 and WebRTC that helps collaborate from anywhere by allowing users to create and join rooms. The user login and authentication are performed using Firebase authentication for the web & native apps, and Cloud Firestore. The source code for both client and server side are available on Gitlab. I hope this app and its development can help creators, schools and colleges for starters.
