BabylonJS inside React Native inception style

Okay, this article is going to be really funky. Because the solution is stupid in itself, but terribly effective in hindsight. Please be sure to check your architecture sanity altogether, because long story short : it’s going to get ugly really fast.

So the idea came as a solution to a problem I’m having at work (oh yeah, I’ve got a new job in Bordeaux, at AirInt Services) : we’re doing a React Native app, primarily for iOS, but also Windows and we’re not excluding an Android port. I wont go over the debate as to if this was the best technology for this, because I really don’t care. ES6 is a fine language, React Native dev environments are intuitive and they work, and I like it. Our app requires a technical 3D view which in itself is pretty simple, and needs to be for readability.

Our first approach was when I worked with them three years ago : the app was exclusively on iOS, was using classic iOS SDKs and it worked just fine. So we implemented a very small in house engine to render our scenes, with OpenGL ES2 at the time, and a fair amount of elbow grease. It worked just great, it was a magnificent piece of software, and I’m still kinda proud of it. But earlier this year, there was a demand for a windows version, which led to the react native version which is all new and improved. The way react native works for the 3D view is that my (not then but now) colleagues implemented the iOS engine as a native component, and I made a port of the engine on windows using DirectX. That’s when I got to thinking about this : what a nightmare to maintain.

And for a good reason : I gained a lot of skills in the past three years (plus iOS got ES3 in between). So I updated the iOS engine to be on par with the windows engine’s new shiny features, and I came out with this face :

So yeah, this was going to be a tough job keeping both versions updated, knowing that :

• different platforms with different technologies and languages
• some technical choices I made three years ago are really debatable and are deeply rooted inside the engine lifecycle
• just going to React Native needed another technical point of view than the one we adopted. Our architecture wasn’t fit for the situation anymore

So I started experimenting. In React Native, a lot of base stuff is effectively native behind the scene. The fact is that it’s sufficiently well done for your to be able to build a classic mobile app without ever touching much native code (which is dependent on the platform). But there’s a bunch of hardcore javascript people that keep pushing the crossplatform side of things to a whole new level. And for good reasons too : the windows version doesn’t support all React Native features because they don’t have their native implementations yet. It’s coming along quite nicely though; even though features might be missing, they’re always up to date on version numbers with the React Native project. There doesn’t seem to be a lot of core contributors to this port, so hats off for the work already done (and I feel the team deserves more appreciation from both Microsoft and React communities, there’s a lot of things unsaid and it’s not cool for them).

So that triggered me to think about this : how about a pure javascript 3D engine, lol? No native code, just install it and run it in your app without having to maintain different versions. So I started doing research on the subject, and there are a few webgl experiments online. Problem is that the same way we work with our native engines, there’s a native code implementation marshalled into RN. And guess what : there’s never a windows version. Mostly because devs are on totally different work platforms, and also because they don’t care about it. Not going to sugarcoat it, but the RN windows port needs extra attention when half (more like 99% actually) the packages you use are not compatible.

So I kinda let go of this idea because I got really lazy about implementing a webgl context for RN. Instead, I chose a pretty stupid thing : webviews. You’re probably thinking “mate, why?”. And that’s also what I thought when I made my POC. And not only did it work perfectly on both iOS and Windows (haven’t tested Android yet, working on it!), but it proved surprisingly flexible to use. There’s of course a few concessions to make :

• debugging is way more difficult (not really possible). Logging is doable.
• performance inside the webview is nice, but it can get way better on iOS with a small twist. More on that later.
• it’s really funky tbh. I kinda feel dirty.

Since I didn’t feel like developping a full webgl engine, I decided to use my favorite javascript engine, BabylonJS. It’s easy to use and contains all the features we need at work. So this is how this works (you might feel uncomfortable) :

1. if you intend to load models (which you probably do), you’re going to need to serve your files through a web server
2. your babylon RN “engine” is basically a website
3. the RN webview has a injectJavaScript method and a callback, and these will now be your way of communicating between the RN context and the engine context
4. The above doesn’t really apply if you want to do things correctly on iOS
5. Some packages are custom and not yet merged inside their respective repos, so there’s going to be some acrobatics with package.json
6. Getting this to work on React Native Windows is an art form at times
7. you might go crazy trying to debug this

So, if you’re still up for this, let’s go! I’ll divide the article in multiple chapters because there’s a lot of details to work on.

Setting up your project

I’m going to assume you know how to create a React Native project. Check out the official documentation if not, and if that’s the case, you should probably look at less crazier react native stuff first before trying this out.

I’m using yarn, because yarn is cool when you’re installing a lot. NPM is fine too, juste hella slow on Windows for RN projects. Version numbers I use are contextual and more like a “minimum version” indicator, feel free to use updated packages if they work. These are the packages we are going to use :

• React Native (Windows on parr) v 0.49.0 : not much to say, I’m going to assume this is the minimum version for Windows, but probably not for iOS
• react-native-wkwebview-reborn@^1.10.0 : let's get this out of the way right now. iOS RN ships with the default as the WebView component, but to be honest, it's kinda shitty. iOS has the new and improved , which by the way is available starting iOS version 8, which is also the minimum iOS version needed to run a RN app. Beats me as to why they chose to implement the old one, moreover when even Apple kindly asks not to use it anymore. So this package comes with a WkWebView component that has the advantage of being non blocking on the render size and wayyyyyyyy more perfs. This comes with a few edits : injectJavaScript becomes evaluateJavascript on the app side and becomes on the webview side. I'll agree this is a really ugly function prototype, but the major difference is that you can transmit real js objects back to your app from the webview with no lousy string cast in between, which is neat. Check out the linking part on their repo for your Xcode project.
• react-native-static-server : this one is the only one where I had to go and code something native. Since this hasn't yet been merged into the official package, you'll need to use my personal repo for now. You can do this by manually adding "react-native-static-server":"git://github.com/JulienNoble/react-native-static-server.git" in your dependencies to your packages.json. This package's name is pretty self explanatory, it's a static file webserver that runs inside your app, allowing you to serve files through http://localhost . Check out the linking part on my repo for the different platforms (auto link doesn't work on Windows).
• react-native-fs@^2.8.5 : the classic react native filesystem library, works on iOS, Android and Windows, gets the job done.

Once you’re done with that, you’ll need some place in your project to store your engine files. Be smart, don’t put the code next to your app’s code, for the simple reason that it’s not the same context. I like to create an folder alongside my folder, which I can then link in xcode and visual studio for deployment. Deployment is also the only part which is going to require some native code to setup. What we’re doing is copying our engine folder to the app Documents, which is a lawless zone for file permissions on all platforms. So let’s do this :

iOS deployment setup

Add your engine folder as an alias in your project. Then, inside your entry point class (or anywhere that gets executed once at the beginning of your app), add a method to call the engine copy to the app documents.

#import <WebKit/WKWebsiteDataStore.h>
- (void)createCopyOfEngineIfNeeded {
// clear webview cache, important for development. Can be removed when deploying for prod
NSSet *websiteDataTypes = [WKWebsiteDataStore allWebsiteDataTypes];
NSDate *dateFrom = [NSDate dateWithTimeIntervalSince1970:0];
[[WKWebsiteDataStore defaultDataStore] removeDataOfTypes:websiteDataTypes modifiedSince:dateFrom completionHandler:^{
}];
// copy folder
BOOL success;
NSFileManager *fileManager = [NSFileManager defaultManager];
NSError *error;
NSArray *paths = NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES);
NSString *documentsDirectory = [paths objectAtIndex:0];
NSString *enginePath = [documentsDirectory stringByAppendingPathComponent:@"/engine"]; // I copy it to an "engine" subfolder inside the app documents
if ([fileManager fileExistsAtPath:enginePath])
[fileManager removeItemAtPath:enginePath error:&error];
NSString *defaultEnginePath = [[[NSBundle mainBundle] resourcePath] stringByAppendingPathComponent:@"/engine"]; // and it's in an "engine" folder in my xcode project
success = [fileManager copyItemAtPath:defaultEnginePath toPath:enginePath error:&error];
NSAssert(success, @"Failed to create 3d engine files with message '%@'.", [error localizedDescription]);
}

Windows deployment setup

Create your engine folder in your VS project, and add all files as links inside this folder. Then, inside your entry point class (or anywhere that gets executed once at the beginning of your app), add a method to call the engine copy to the app documents.

private async void CopyEngine()
{
// clear webview cache, important for development. Can be removed when deploying for prod
await Windows.UI.Xaml.Controls.WebView.ClearTemporaryWebDataAsync();
// copy folder
var appInstalledFolder = Windows.ApplicationModel.Package.Current.InstalledLocation;
var engine = await appInstalledFolder.GetFolderAsync("engine");
var engineDest = await ApplicationData.Current.LocalFolder.CreateFolderAsync("engine", CreationCollisionOption.OpenIfExists);
foreach (var file in await engine.GetFilesAsync())
{
await file.CopyAsync(engineDest, file.Name, NameCollisionOption.ReplaceExisting);
}
}

One last thing : update your Package.appxmanifest to allow Internet (Client & Server) and Private Networks (Client & Server) for your app. If you don't do this, you're going to have a bad time.

Having a sane debug workflow (on iOS at least)

As you’ll soon discover, developping in such an environment is far from practical. Also I would encourage you to primarly use this as an integration means instead of a development setup. If you want to have a simpler life, mock it in your browser, mate. That said, we still want to have a bit of info coming back up from the engine, stuff like “every thing’s okay boss, we’re rocking this render” to “3D is kill”. To do this, a bit of javascript great replacement kung fu techniques :

// check if we're on iOS
var userAgent = window.navigator.userAgent.toLowerCase(),
ios = /iphone|ipod|ipad/.test(userAgent);
// "helper" function to help you get stuff back to the app
const sendMessage = message => {
if (window.webkit.messageHandlers.reactNative.postMessage) {
// iOS
window.webkit.messageHandlers.reactNative.postMessage(message);
} else {
// Windows (and probably Android too)
window.postMessage(JSON.stringify(message));
}
};
// helper log function
const log = (message, type) => {
if (ios)
sendMessage({
message: `[BabylonView] ${message}`,
type,
});
};
// replacing console.log and error event to redirect their outputs to the app
console.log = (...args) => {
for (let i = 0; i < args.length; i++) {
log(`[window] ${args[i]}`, 'info');
}
};
window.onerror = (message, url, linenumber) => {
log(`JavaScript error: ${message} on line ${linenumber} for ${url}`, 'error');
};

This whole setup is going to redirect your entire log flow towards the app, inside which you can then output to your debugger. One thing you’ll notice, is the strange line with the . This one’s extra funky, but basically we detect if you’re on a Windows platform. The reason I do this, is because the React Native CoreApplication instance on Windows tends to gracefully crash if you abuse this call, which kills your whole app, which is not a good thing. So, Windows gets no logs, sorry kids. This goes without saying that this javascript has to be called in an html file, like, for instance, an , but feel free (and also encouraged) to put the js part in a separate file that you import in the html.

On the app side, it’s time we create our component. Let’s start with the webview quirk. We want to have a unique component platform independant to use as a view, but we are going to need WebView for Windows and WkWebView for iOS. Let’s create two files that we’ll call CustomWebView.ios.js and CustomWebView.windows.js . The platform indicator after the first dot is a magic marker for react native to know which version to take on build. We'll thus have a unique importable for our jsx. Here goes :

// CustomWebView.ios.js
import WKWebView from 'react-native-wkwebview-reborn';
export const CustomWebView = WKWebView; // marvelous
// CustomWebView.windows.js
import { WebView } from 'react-native';
import PropTypes from 'prop-types';
import React, { Component } from 'react';
export class CustomWebView extends Component {
constructor(props) {
super(props);
this.registerWebView = this.registerWebView.bind(this);
this.evaluateJavaScript = this.evaluateJavaScript.bind(this);
this.onMessage = this.onMessage.bind(this);
}
onMessage(event) {
try {
const data = JSON.parse(event.nativeEvent.data);
this.props.onMessage({
body: data,
});
} catch (e) {
console.log(e);
}
}
registerWebView(ref) {
this.webview = ref;
}
evaluateJavaScript(script) {
if (this.webview) {
this.webview.injectJavaScript(`window.${script}`);
}
}
render() {
return (
<WebView
ref={this.registerWebView}
source={this.props.source}
onLoadEnd={this.props.onLoadEnd}
onMessage={this.onMessage}
/>
);
}
}
CustomWebView.propTypes = {
source: PropTypes.object,
onLoadEnd: PropTypes.func,
onMessage: PropTypes.func,
};

We’ll wrap it around a neat little component to which we can feed actions that the app can use to control the webview, ensuring some far fetched form of context separation.

// BabylonView.js
import PropTypes from 'prop-types';
import React, { Component } from 'react';
import StaticServer from 'react-native-static-server';
import { CustomWebView } from './CustomWebView';
const RNFS = require('react-native-fs');
export class BabylonView extends Component {
constructor(props) {
super(props);
this.registerWebView = this.registerWebView.bind(this);
this.onLoadEnd = this.onLoadEnd.bind(this);
this.onMessage = this.onMessage.bind(this);
}
componentDidMount() {
// your app document folder
const documentDir = RNFS.DocumentDirectoryPath;
this.server = new StaticServer(8800, `${documentDir}/`);
this.server.start();
}
onLoadEnd() {
if (this.webview) {
// init code goes here
}
}
// keep a ref of our webview
registerWebView(ref) {
this.webview = ref;
}
onMessage(event) {
if (event.body.message) {
if (event.body.type) { // send log to app debugger
switch (event.body.type) {
case 'error':
console.error(event.body.message);
break;
case 'info':
console.debug(event.body.message);
break;
default:
console.log(event.body.message);
break;
}
} else console.log(event.body.message);
}
}
render() {
return (
<CustomWebView
ref={this.registerWebView}
source={{ uri: 'http://localhost:8800/engine/' }}
onLoadEnd={this.onLoadEnd}
onMessage={this.onMessage}
/>
);
}
}

Strap it in your app, launch it, add some flavor text in your index.html page to know you’re in the webview, and some forced test logs and you should have something like that :

If you use log('bla', 'info'); and see nothing, check the dropdown next to the filter textbox in your debugger and set it to verbose. They should appear as blue. In this setup, React Native will also scream at you if there's an internal error. We're good to finally start doing some 3D!

Making a BabylonJS view

From this point, it’s important to know that you can do everything a browser can do. Want to code your entire app inside the webview? Knock yourself out, even though that’s actually pretty stupid. So what we’re going to do is create a classic website babylonjs context. First you’re going to need a few lib files :

• the babylonjs engine itself : https://cdn.babylonjs.com/babylon.js
• pep.js, which is going to handle pointer events : https://code.jquery.com/pep/0.4.1/pep.js

Save these files next to your . I have to stress out this is if you’re going from scratch, you would be wise to develop your 3D view outside of React Native first, and then integrate it inside the app. Your should look like this :

Next, let’s fill up our . It’s not going to change much over development, it’s basically just an entry point for the webview :

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<title>BabylonView</title>
<style>
html,
body {
overflow: hidden;
width: 100%;
height: 100%;
margin: 0;
padding: 0;
}
#renderCanvas {
width: 100%;
height: 100%;
touch-action: none;
}
</style>
<script src="./babylon.js"></script>
<script src="./pep.js"></script>
</head>
<body>
<canvas id="renderCanvas" touch-action="none"></canvas>
<script src="./engine.js"></script>
</body>
</html>

engine.jsis the file that currently contains our log utils, but is going to include babylonjs init and stuff. Let’s make a simple scene with an interactable camera :

// ... log utils
// Babylon
const canvas = document.getElementById('renderCanvas');
const engine = new BABYLON.Engine(canvas);
const setup = () => {
const scene = new BABYLON.Scene(engine);
const camera = new BABYLON.ArcRotateCamera(
'Camera',
Math.PI,
Math.PI / 4,
10,
BABYLON.Vector3.Zero(),
scene,
);
camera.attachControl(canvas, true);
const light = new BABYLON.HemisphericLight(
'light',
new BABYLON.Vector3(0, 1, 1),
scene,
);
const box = BABYLON.MeshBuilder.CreateBox('box', {}, scene);
return scene;
};
const scene = setup();
engine.runRenderLoop(() => {
scene.render();
});
window.addEventListener('resize', () => {
engine.resize();
});

This is a basic babylon scene setup. It’s pretty straightforward, and I wont dive deep into the lib’s usage, since it’s basically the same as any website with babylon.

I’m running this on the xcode 9.1 iOS simulator on a 2013 macbook pro (I don’t even have a real graphics card), getting around 20 fps and crisp response to gestures. On device, I get a good 40 fps on an iPad 2 (yes, the second generation of ipads, from 2011). Keep in mind this is a plain stupid scene. Don’t expect it to run well on complex scene. As a rule of thumbs, I’d say the first iPad Air is a good minimum for a fluid 60 fps on complex scenes. With our internal tests, we’re able to load on device a view with about half a million polygons (there’s a lot of instancing, calm your tits) and custom gestures at 60. All in all, this is extremely surprising performance wise : the respective webviews of each platform have become pretty darn powerful.

Getting stuff to and from the app

Spoilers : we already have what we need to get data back to the app. The sendMessage() helper function takes anything and brings it back to the app. The data gets projected into the "body" property of a javascript event object in onMessage(event) in BabylonView.js . Currently it's only handling logs coming back up from the webview, but go wild if you have to. For instance, we use this helper to build cache meshes on device and bring back the meshes data to save on our device's document folder. We also use it to bring back custom pointer events to the app, like tap and swipe. It works okay, but beware of Windows, the platform really doesn't like too much communication coming from the webview.

The other way around is also pretty simple, albeit not aesthetically pleasing. uses evaluateJavascript() instead of injectJavascript , and I decided to keep this name for our CustomWebView . So what we can do to pass data to the webview is send a string script as argument to the method, and it will execute in the webview context. I'll agree this is far from practical, but heh, it works. We use it for scene description for instance. The way this works is simple : you declare a method in the object and call it without the window reference :

// on the engine side
window.doSomething = (someObjectString) => {
const someObject = JSON.parse(someObjectString);
console.log(someObject.stuff);
}
// on the app side (in BabylonView.js for instance)
const aThing = {
stuff: 'hello',
};
this.webview.evaluateJavascript(`doSomething('${JSON.stringify(aThing)}')`);

It’s kinda ugly, but it does the job. Go wild. From this point, you can import the babylon loaders and copy your model and texture files along with your engine. Our internal architecture gets a list of cached meshes with a position map of these meshes, and the engine basically analyzes the description and builds the scene.

Stuff was done, moments were had

So it all boils down to this question : yes it’s stupid, but it works very nicely, so are you in? We made the decision to try this on our software, we don’t have any data yet to support this as being a good idea. But the results are already visible : non blocking loading and manipulation of the scene, great performances on complex datasets, and very stable. This is still all very experimental.

Check out a sample POC on my Github.

Have fun, code safe!
Tuxic

Originally published at http://cubeslam.net on November 16, 2017.