Today I want to present a small project with a lot of potentialities, an animated newspaper.

Using the ARKit framework from Apple, I’ll show you how to play a video while passing on the newspaper pages with your camera!

The used newspapers

The scope of this project is to overlap a defined image on the newspaper with a video, for a better experience while reading the news.

Prepare your data

1. Start collecting some real newspaper, or to make the demo fast, use an online version, save it to a folder.
2. Crop the images you want to overlap and save locally (the image you crop are replaced with a video during this test, so choose something regular to simplify the test).
3. Download some video from youtube*

*To download a video on youtube, use this simple trick:

• Go to Youtube, search for a video, for example https://www.youtube.com/watch?v=Z75n-MSlXiQ
• After the word youtube and before .com, add the string “pp“, like youtubepp.com and you can download the video 🙂

Create a new XCode project

Open Xcode and create a new ARKit project.

1. Clean the demo data, remove all lines in viewDidLoad except the sceneView.delegate = self
2. Remove all the demo ARSCNViewDelegate.

Add resources

Now you need to add the saved resources in your project. Pay attention because is not the classic drag ‘n drop on the assets folder. There is something new…

• Create a new group called “Video“, and drag-n-drop all videos. The best experience is with videos of the same size, else you need to choose the right size for each video.
• In the Assets,xctassets drag-n-drop all the newspaper images, not the cropped images!!!
• Now, always in the Assets, right-click and select “New AR resource group“. In this group, you should add the cropped newspaper images.

Errors? Yes, the newly added images are without size because are AR resources.

To fix this, open the right settings panel and set the size you prefer. (Make some tests to decide the right size… because the size are in meters).

Note: use the same name for each resource category to simplify your test.

We use these images to show a video overlay on it, we replace the photo with a video.

Load all images

In the viewWillAppearfor instance, you can load the cropped AR images:

override func viewWillAppear(_ animated: Bool) {
     super.viewWillAppear(animated)
     
     let configuration = ARWorldTrackingConfiguration()

     if let imageToTrack = ARReferenceImage.referenceImages(inGroupNamed: "AR-Newspapers", bundle: Bundle.main) {
         configuration.detectionImages = imageToTrack
         configuration.maximumNumberOfTrackedImages = 1
     }

     sceneView.session.run(configuration)
 } 

Pay attention to the inGroupNamed that is the name of the “group” of AR images you have created the step before in the Assets.xctassets.

Add some logs, run and make sure that the code works well and the images are loaded.

Recognize AR images

You should implement a delegate now, the renderer(_ renderer: SCNSceneRenderer, nodeFor anchor: ARAnchor) -> SCNNode? {}

This method is called when an image of the AR images are recognized.

Here you can check if the recognized image is named as your image… and add a video view.

func renderer(_ renderer: SCNSceneRenderer, nodeFor anchor: ARAnchor) -> SCNNode? {

let node = SCNNode()

// filter for images
if let imageAnchor = anchor as? ARImageAnchor { 
  // check the correct image
  if imageAnchor.name! == "your-ar-image-name {
    // create and play a video node
    let currentVideoNode: SKVideoNode = SKVideoNode(fileNamed: "\(imageAnchor.name!).mp4")
    currentVideoNode.play()

    // create a scene with the video size
    let videoScene = SKScene(size: CGSize(width: 640, height: 480))
 
    // position the video on the scene
    currentVideoNode.position = CGPoint(x: videoScene.size.width / 2, y: videoScene.size.height / 2)
    // flip
    currentVideoNode.yScale = -1.0
    // add the the scene
    videoScene.addChild(currentVideoNode)

    // create a plane
    let plane = SCNPlane(
        width: imageAnchor.referenceImage.physicalSize.width,
        height: imageAnchor.referenceImage.physicalSize.height)

    // with the video
    plane.firstMaterial?.diffuse.contents = videoScene

    let planeNode = SCNNode(geometry: plane)
    // rotate -90°
    planeNode.eulerAngles.x = -.pi / 2
    node.addChildNode(planeNode)

  }

  // if return nil, nothing is added on view
  return node
}

Next step is… nothing, you’ve done!

Embarrassingly simple. Apple did the complex work behind.

Demo

A video that show you what you can do with this snippet of code. Due to the simplicity, no source code is available.

Enjoy reading!