{"_id":"5ad78c382cfba50003554c46","category":{"_id":"5ad78c362cfba50003554bfe","version":"5ad78c362cfba50003554bfc","project":"5a065a6134873d0010b396ab","__v":0,"sync":{"url":"","isSync":false},"reference":false,"createdAt":"2016-07-18T21:08:45.730Z","from_sync":false,"order":1,"slug":"develop","title":"Develop"},"project":"5a065a6134873d0010b396ab","user":"57bb7e47afc18c0e00529cf3","parentDoc":null,"version":{"_id":"5ad78c362cfba50003554bfc","project":"5a065a6134873d0010b396ab","__v":1,"createdAt":"2018-04-18T18:19:34.288Z","releaseDate":"2018-04-18T18:19:34.288Z","categories":["5ad78c362cfba50003554bfd","5ad78c362cfba50003554bfe","5ad78c362cfba50003554bff","5ad78c362cfba50003554c00"],"is_deprecated":false,"is_hidden":false,"is_beta":false,"is_stable":true,"codename":"","version_clean":"1.6.0","version":"1.6.0"},"__v":0,"updates":[],"next":{"pages":[],"description":""},"createdAt":"2017-09-18T00:22:17.631Z","link_external":false,"link_url":"","githubsync":"","sync_unique":"","hidden":false,"api":{"results":{"codes":[]},"settings":"","auth":"required","params":[],"url":""},"isReference":false,"order":14,"body":"[block:callout]\n{\n  \"type\": \"info\",\n  \"title\": \"ViroCore uses ARCore for AR Tracking\",\n  \"body\": \"ARCore supports the following devices: Google Pixel, Pixel XL, Pixel 2, Pixel 2 XL, Samsung Galaxy S8 running N and later.\"\n}\n[/block]\nThe Viro platform supports Augmented Reality (AR) development through integration with ARCore. This guide will first give an overview of AR and provide a high-level overview of the components and features that enable developers to build AR experiences.\n[block:api-header]\n{\n  \"title\": \"Device Support\"\n}\n[/block]\nAR is not supported on all platforms. The full list of devices supporting AR can be found on Google's [ARCore page](https://developers.google.com/ar/discover/). To programmatically detect if a device supports ARCore, use the [ViroViewARCore.isDeviceCompatible(Context)][1] method. If a given device isn't compatible, an UnavailableDeviceNotCompatibleException will be thrown if you attempt to construct a ViroViewARCore.\n\nIf a device *is* compatible, then when the ViroViewARCore is constructed users will be prompted to download ARCore, if they haven't already. This is an asynchronous flow that you can listen to by passing a [StartupListener](https://developer.viromedia.com/virocore/reference/com/viro/core/ViroViewScene.StartupListener.html) into the ViroViewARCore constructor. If the download and install of ARCore was successful, the StartupListener's onSuccess() callback will be invoked. Otherwise, onFailure() will be invoked with the corresponding error code.\n\nTo summarize, the workflow is as follows:\n\n[1]:https://developer.viromedia.com/virocore/reference/com/viro/core/ViroViewARCore.html#isDeviceCompatible(android.content.Context)\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/fbce526-Install_Flow.001.jpeg\",\n        \"Install Flow.001.jpeg\",\n        1024,\n        768,\n        \"#fbfbfb\"\n      ]\n    }\n  ]\n}\n[/block]\n\n[block:api-header]\n{\n  \"title\": \"Required Camera Permissions for AR\"\n}\n[/block]\nViroCore requires access to the device's camera for AR enabled features like tracking / SLAM to function properly. This permission check automatically occurs during the onActivityResumed phase within the ViroViewARCore view. If the user had already previously given Camera permissions for your application, everything should work as per normal. \n\nHowever, if no Camera permissions had yet been given, Viro will automatically ask for Camera permissions via Android's requestPermissions() API. This should pop up a friendly permission request dialog as shown below:\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/304c212-device-2018-02-26-160020_copy_2.png\",\n        \"device-2018-02-26-160020 copy 2.png\",\n        248,\n        172,\n        \"#0c0c0c\"\n      ]\n    }\n  ]\n}\n[/block]\nIf user then denies camera permissions, the StartupListener's onFailure() callback will be invoked with the corresponding CAMERA_PERMISSIONS_NOT_GRANTED error code. Note that like any other Android permissions, you can also preemptively call and check for Camera permissions within your Android application before constructing the ViroViewARCore control.\n[block:api-header]\n{\n  \"title\": \"Developing for AR\"\n}\n[/block]\n## Camera Tracking\nThe Viro platform supports development of 6 degrees-of-freedom (6DOF) AR experiences. The platform tracks the user's real-world rotation and position as he or she moves, and keeps the virtual camera in sync with that movement. The platform maintains a right-handed coordinate system, where the origin of the system is the user's location at the time AR tracking was initialized. The camera's forward vector of `[0, 0, -1]` and up vector is `[0,1,0]`.\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/3a0e825-viro_camera_diagram.png\",\n        \"viro_camera_diagram.png\",\n        607,\n        676,\n        \"#3cbbfb\"\n      ]\n    }\n  ]\n}\n[/block]\n## AR Scenes and Anchors\n\nAR experiences are created in Viro by creating a [ViroViewARCore](https://developer.viromedia.com/virocore/reference/com/viro/core/ViroViewARCore.html), and configuring and setting an [ARScene](https://developer.viromedia.com/virocore/reference/com/viro/core/ARScene.html). ARScene blends virtual 3D content with the device camera's view of the real world. Similar to a normal Scene, an ARScene contains a hierarchy of Node objects representing the virtual 3D world. Behind these objects the camera's live video feed is rendered. ARScene ensures the two worlds -- real and virtual -- stay in sync with a unified coordinate system.\n\nThere are two ways to add virtual content to the ARScene. The first is to simply place content manually. The origin of the coordinate system is the initial position of the user, and the scale is meters. To add content you only need to set the position of the Node within this coordinate system, and add it to the scene graph. You can find more information on Scenes in our [Scenes guide](doc:scenes).\n\nThe second way to add virtual content is to use [ARAnchor](https://developer.viromedia.com/virocore/reference/com/viro/core/ARAnchor.html). ARAnchors represent features detected in the real-world. You can associate (or \"anchor\") your content to these features by adding your content to the [ARNode](https://developer.viromedia.com/virocore/reference/com/viro/core/ARNode.html) that corresponds to each detected ARAnchor. To do this, implement the [ARScene.Listener](https://developer.viromedia.com/virocore/reference/com/viro/core/ARScene.Listener.html). The ARScene.Listener callbacks are invoked whenever ARAnchors are found (or updated, or removed) in the real-world.\n\nIn the example below, we create an ARScene and set it up to add a virtual Box to sit on every detected real-world plane.\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"final ARScene scene = new ARScene();\\n\\nscene.setListener(new ARScene.Listener() {\\n    public void onAmbientLightUpdate(float lightIntensity, float colorTemperature) {\\n    }\\n  \\n    public void onAnchorFound(ARAnchor anchor, ARNode arNode) {\\n        // Create a Box to sit on every plane we detect\\n        if (anchor.getType() == ARAnchor.Type.PLANE) {\\n            Box box = new Box(1, 1, 1);\\n            Node boxNode = new Node();\\n            boxNode.setGeometry(box);\\n            arNode.addChildNode(boxNode);\\n        }\\n    }\\n  \\n    public void onAnchorRemoved(ARAnchor anchor, ARNode arNode) {\\n      \\n    }\\n  \\n    public void onAnchorUpdated(ARAnchor anchor, ARNode arNode) {\\n      \\n    }\\n  \\n    public void onTrackingInitialized() {\\n      \\n    }\\n});\\n\\nViroView view = new ViroViewARCore(context, null);\\nview.setScene(scene);\",\n      \"language\": \"java\"\n    }\n  ]\n}\n[/block]\n## AR Features\n\nViro supports numerous AR features, some of which are detailed below.\n\n### AR Hit Testing\nViroViewARCore has a variety of methods used to \"hit-test\" against the real-world. These hit tests can be used to determine, to the best of ARCore's ability, what real-world features exist at a given point on the 2D screen. Note that since a single 2D point on the view corresponds to a 3D ray in the scene, multiple results may be returned (each at a different depth). The results may be anchors, or they may be feature points that have not yet been fully identified.\n\n### Fixed to World Dragging\nNormally when dragging a Node through a DragListener, the Node when dragged stays at a fixed distance from from the user, as though the user is dragging the Node across the inner surface of a sphere. This is called `FixedToWorld` dragging. \n\nViro also supports `FixedToWorld` dragging in AR, where instead of keeping a dragged Node's distance from the user fixed, the dragged Node's distance is instead determined by its intersection with the nearest real-world object. This is useful for dragging a virtual object across a real-world surface, for example.\n\n### Portals\n\nPortals are an AR effect where a 'window' or 'door' is displayed that users can use to peer into a virtual world, as shown below.\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/899483a-AR_Portal_360_Video.gif\",\n        \"AR Portal 360 Video.gif\",\n        300,\n        534,\n        \"#534643\"\n      ]\n    }\n  ]\n}\n[/block]\n[PortalScene](https://developer.viromedia.com/virocore/reference/com/viro/core/PortalScene.html) is the root of the subgraph of Nodes that is displayed through a Portal. Each PortalScene can contain any number of child nodes and content, and each PortalScene can have its own background texture. If a PortalScene is set to passable, users are able to walk through the Portal into the PortalScene.\n\nThe following example shows how to create a simple Portal that transports you from your current location to a beach-like resort.\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"// Add a Light so the ship door portal entrance will be visible\\nOmniLight light = new OmniLight();\\nlight.setColor(Color.WHITE);\\nlight.setPosition(new Vector(0, 1, -4 ));\\nscene.getRootNode().addLight(light);\\n\\n// Load a model representing the ship door\\nObject3D shipDoorModel = new Object3D();\\nshipDoorModel.loadModel(Uri.parse(\\\"file:///android_asset/portal_ship.vrx\\\"), Object3D.Type.FBX, null);\\n\\n// Create a Portal out of the ship door\\nPortal portal = new Portal();\\nportal.addChildNode(shipDoorModel);\\nportal.setScale(new Vector(0.5, 0.5, 0.5));\\n\\n// Create a PortalScene that uses the Portal as an entrance.\\nPortalScene portalScene = new PortalScene();\\nportalScene.setPosition(new Vector(0, 0, -5));\\nportalScene.setPortalEntrance(portal);\\n\\n// Add a 'beach' background for the Portal scene\\nfinal Bitmap beachBackground = getBitmapFromAssets(\\\"beach.jpg\\\");\\nfinal Texture beachTexture = new Texture(beachBackground, Texture.Format.RGBA8, true, false);\\nportalScene.setBackgroundTexture(beachTexture);\\n\\nscene.getRootNode().addChildNode(portalScene);\",\n      \"language\": \"java\"\n    }\n  ]\n}\n[/block]\nTo try this example, you can download the beach 360 photo and portal ship door assets [here](http://dq0qxqxzgyp27.cloudfront.net/virocore/readme_example_assets/portal_example_assets.zip). Unzip these and store in your applications assets folder. The resulting scene is shown below.\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/ecac665-screen.png\",\n        \"screen.png\",\n        1440,\n        2560,\n        \"#685640\"\n      ]\n    }\n  ]\n}\n[/block]\n### Video and Still Capture\nViro supports video and still image capture, to make sharing AR experiences easy. To do this, simply grab the [ViroMediaRecorder](https://developer.viromedia.com/virocore/reference/com/viro/core/ViroMediaRecorder.html) from the ViroViewARCore.\n\n### Point Cloud Rendering\nEnable point cloud rendering in ARScene to display on-screen the rough feature points that ARCore is detecting. This is primarily useful for debugging and analysis.","excerpt":"","slug":"augmented-reality-ar","type":"basic","title":"Augmented Reality (AR)"}

Augmented Reality (AR)


[block:callout] { "type": "info", "title": "ViroCore uses ARCore for AR Tracking", "body": "ARCore supports the following devices: Google Pixel, Pixel XL, Pixel 2, Pixel 2 XL, Samsung Galaxy S8 running N and later." } [/block] The Viro platform supports Augmented Reality (AR) development through integration with ARCore. This guide will first give an overview of AR and provide a high-level overview of the components and features that enable developers to build AR experiences. [block:api-header] { "title": "Device Support" } [/block] AR is not supported on all platforms. The full list of devices supporting AR can be found on Google's [ARCore page](https://developers.google.com/ar/discover/). To programmatically detect if a device supports ARCore, use the [ViroViewARCore.isDeviceCompatible(Context)][1] method. If a given device isn't compatible, an UnavailableDeviceNotCompatibleException will be thrown if you attempt to construct a ViroViewARCore. If a device *is* compatible, then when the ViroViewARCore is constructed users will be prompted to download ARCore, if they haven't already. This is an asynchronous flow that you can listen to by passing a [StartupListener](https://developer.viromedia.com/virocore/reference/com/viro/core/ViroViewScene.StartupListener.html) into the ViroViewARCore constructor. If the download and install of ARCore was successful, the StartupListener's onSuccess() callback will be invoked. Otherwise, onFailure() will be invoked with the corresponding error code. To summarize, the workflow is as follows: [1]:https://developer.viromedia.com/virocore/reference/com/viro/core/ViroViewARCore.html#isDeviceCompatible(android.content.Context) [block:image] { "images": [ { "image": [ "https://files.readme.io/fbce526-Install_Flow.001.jpeg", "Install Flow.001.jpeg", 1024, 768, "#fbfbfb" ] } ] } [/block] [block:api-header] { "title": "Required Camera Permissions for AR" } [/block] ViroCore requires access to the device's camera for AR enabled features like tracking / SLAM to function properly. This permission check automatically occurs during the onActivityResumed phase within the ViroViewARCore view. If the user had already previously given Camera permissions for your application, everything should work as per normal. However, if no Camera permissions had yet been given, Viro will automatically ask for Camera permissions via Android's requestPermissions() API. This should pop up a friendly permission request dialog as shown below: [block:image] { "images": [ { "image": [ "https://files.readme.io/304c212-device-2018-02-26-160020_copy_2.png", "device-2018-02-26-160020 copy 2.png", 248, 172, "#0c0c0c" ] } ] } [/block] If user then denies camera permissions, the StartupListener's onFailure() callback will be invoked with the corresponding CAMERA_PERMISSIONS_NOT_GRANTED error code. Note that like any other Android permissions, you can also preemptively call and check for Camera permissions within your Android application before constructing the ViroViewARCore control. [block:api-header] { "title": "Developing for AR" } [/block] ## Camera Tracking The Viro platform supports development of 6 degrees-of-freedom (6DOF) AR experiences. The platform tracks the user's real-world rotation and position as he or she moves, and keeps the virtual camera in sync with that movement. The platform maintains a right-handed coordinate system, where the origin of the system is the user's location at the time AR tracking was initialized. The camera's forward vector of `[0, 0, -1]` and up vector is `[0,1,0]`. [block:image] { "images": [ { "image": [ "https://files.readme.io/3a0e825-viro_camera_diagram.png", "viro_camera_diagram.png", 607, 676, "#3cbbfb" ] } ] } [/block] ## AR Scenes and Anchors AR experiences are created in Viro by creating a [ViroViewARCore](https://developer.viromedia.com/virocore/reference/com/viro/core/ViroViewARCore.html), and configuring and setting an [ARScene](https://developer.viromedia.com/virocore/reference/com/viro/core/ARScene.html). ARScene blends virtual 3D content with the device camera's view of the real world. Similar to a normal Scene, an ARScene contains a hierarchy of Node objects representing the virtual 3D world. Behind these objects the camera's live video feed is rendered. ARScene ensures the two worlds -- real and virtual -- stay in sync with a unified coordinate system. There are two ways to add virtual content to the ARScene. The first is to simply place content manually. The origin of the coordinate system is the initial position of the user, and the scale is meters. To add content you only need to set the position of the Node within this coordinate system, and add it to the scene graph. You can find more information on Scenes in our [Scenes guide](doc:scenes). The second way to add virtual content is to use [ARAnchor](https://developer.viromedia.com/virocore/reference/com/viro/core/ARAnchor.html). ARAnchors represent features detected in the real-world. You can associate (or "anchor") your content to these features by adding your content to the [ARNode](https://developer.viromedia.com/virocore/reference/com/viro/core/ARNode.html) that corresponds to each detected ARAnchor. To do this, implement the [ARScene.Listener](https://developer.viromedia.com/virocore/reference/com/viro/core/ARScene.Listener.html). The ARScene.Listener callbacks are invoked whenever ARAnchors are found (or updated, or removed) in the real-world. In the example below, we create an ARScene and set it up to add a virtual Box to sit on every detected real-world plane. [block:code] { "codes": [ { "code": "final ARScene scene = new ARScene();\n\nscene.setListener(new ARScene.Listener() {\n public void onAmbientLightUpdate(float lightIntensity, float colorTemperature) {\n }\n \n public void onAnchorFound(ARAnchor anchor, ARNode arNode) {\n // Create a Box to sit on every plane we detect\n if (anchor.getType() == ARAnchor.Type.PLANE) {\n Box box = new Box(1, 1, 1);\n Node boxNode = new Node();\n boxNode.setGeometry(box);\n arNode.addChildNode(boxNode);\n }\n }\n \n public void onAnchorRemoved(ARAnchor anchor, ARNode arNode) {\n \n }\n \n public void onAnchorUpdated(ARAnchor anchor, ARNode arNode) {\n \n }\n \n public void onTrackingInitialized() {\n \n }\n});\n\nViroView view = new ViroViewARCore(context, null);\nview.setScene(scene);", "language": "java" } ] } [/block] ## AR Features Viro supports numerous AR features, some of which are detailed below. ### AR Hit Testing ViroViewARCore has a variety of methods used to "hit-test" against the real-world. These hit tests can be used to determine, to the best of ARCore's ability, what real-world features exist at a given point on the 2D screen. Note that since a single 2D point on the view corresponds to a 3D ray in the scene, multiple results may be returned (each at a different depth). The results may be anchors, or they may be feature points that have not yet been fully identified. ### Fixed to World Dragging Normally when dragging a Node through a DragListener, the Node when dragged stays at a fixed distance from from the user, as though the user is dragging the Node across the inner surface of a sphere. This is called `FixedToWorld` dragging. Viro also supports `FixedToWorld` dragging in AR, where instead of keeping a dragged Node's distance from the user fixed, the dragged Node's distance is instead determined by its intersection with the nearest real-world object. This is useful for dragging a virtual object across a real-world surface, for example. ### Portals Portals are an AR effect where a 'window' or 'door' is displayed that users can use to peer into a virtual world, as shown below. [block:image] { "images": [ { "image": [ "https://files.readme.io/899483a-AR_Portal_360_Video.gif", "AR Portal 360 Video.gif", 300, 534, "#534643" ] } ] } [/block] [PortalScene](https://developer.viromedia.com/virocore/reference/com/viro/core/PortalScene.html) is the root of the subgraph of Nodes that is displayed through a Portal. Each PortalScene can contain any number of child nodes and content, and each PortalScene can have its own background texture. If a PortalScene is set to passable, users are able to walk through the Portal into the PortalScene. The following example shows how to create a simple Portal that transports you from your current location to a beach-like resort. [block:code] { "codes": [ { "code": "// Add a Light so the ship door portal entrance will be visible\nOmniLight light = new OmniLight();\nlight.setColor(Color.WHITE);\nlight.setPosition(new Vector(0, 1, -4 ));\nscene.getRootNode().addLight(light);\n\n// Load a model representing the ship door\nObject3D shipDoorModel = new Object3D();\nshipDoorModel.loadModel(Uri.parse(\"file:///android_asset/portal_ship.vrx\"), Object3D.Type.FBX, null);\n\n// Create a Portal out of the ship door\nPortal portal = new Portal();\nportal.addChildNode(shipDoorModel);\nportal.setScale(new Vector(0.5, 0.5, 0.5));\n\n// Create a PortalScene that uses the Portal as an entrance.\nPortalScene portalScene = new PortalScene();\nportalScene.setPosition(new Vector(0, 0, -5));\nportalScene.setPortalEntrance(portal);\n\n// Add a 'beach' background for the Portal scene\nfinal Bitmap beachBackground = getBitmapFromAssets(\"beach.jpg\");\nfinal Texture beachTexture = new Texture(beachBackground, Texture.Format.RGBA8, true, false);\nportalScene.setBackgroundTexture(beachTexture);\n\nscene.getRootNode().addChildNode(portalScene);", "language": "java" } ] } [/block] To try this example, you can download the beach 360 photo and portal ship door assets [here](http://dq0qxqxzgyp27.cloudfront.net/virocore/readme_example_assets/portal_example_assets.zip). Unzip these and store in your applications assets folder. The resulting scene is shown below. [block:image] { "images": [ { "image": [ "https://files.readme.io/ecac665-screen.png", "screen.png", 1440, 2560, "#685640" ] } ] } [/block] ### Video and Still Capture Viro supports video and still image capture, to make sharing AR experiences easy. To do this, simply grab the [ViroMediaRecorder](https://developer.viromedia.com/virocore/reference/com/viro/core/ViroMediaRecorder.html) from the ViroViewARCore. ### Point Cloud Rendering Enable point cloud rendering in ARScene to display on-screen the rough feature points that ARCore is detecting. This is primarily useful for debugging and analysis.