Reading Time: 7 minutes

Hello again! I do hope that you are following our streak of blogs on Augmented reality and augmented image. What? Are you new here? Okay, have a look into these for better understanding and also to keep up with the flow of the blogs.

• What is Augmented Reality and why is it in-demand?
• Why use Augmented Reality for your Business?
• Augmented Reality Project- A TCW Case Study

You can also check out our video on the same, on our Youtube channel at TheCodeWork.

Okay, now back to the context. So as you already know, we have been working on several segments of Augmented reality and these case studies are a detailed approach to sharing our ideas and work process with you. 

It is welcome, right?

So here, we are going to have a detailed look into the project of Augmented Image, developed by our very own Android Development team at TheCodeWork.

So here we go into the world of illusion blend in reality once again! Follow up.

Overview of the Augmented Image Project

The APIs of Augmented Images in ARCore allows you to develop AR apps that have the ability to augment and detect 2D images in the respective user’s environment. These images can be posters or simply any product packaging. 

This project deals with the detection of such 2D images and producing their augmented image to exhibit information with the help of ViewRenderable or produce a 3-D model. Sounds cool right?

So it works like this;

When ARCore starts tracking an image, it provides estimations for the position of the image and the orientation of every frame. This entire tracking activity takes place on the device itself. 

Bonus Point: You don’t require any active internet connection to detect or track these images. 

ARcore basically scans the image that you store in the database when the image is detected. It also creates a 3D image of that specific image.

Magic!

Augmented Images for Android and how it works?

To begin with, you have to provide some reference images. After that, ARCore makes use of some machine vision algorithm to obtain characteristics from the grayscale data in every image. Consequently, it saves a representation of these characteristics in several Augmented Image databases. See the flow already?

During runtime, ARCore explores these saved characteristics on flat planes in the respective user’s surroundings. This further allows ARCore to detect the images in the real world and assess their orientation, position, and size even if they are not provided.

What all can Augmented Images do?

ARCore has the ability to track around 15-20 images together. However, ARCore does not track or detect recurring data of the same image. 

Also, guess what? The database of every Augmented Image has the ability to save information of around 1,000 reference images. How cool is that? It can store an endless number of databases. However, there is a catch. Out of these, only one particular database remains active at any particular given time.

In addition to all this, you can also add images to the database of Augmented Image during runtime. But do remember that, there is a limit of every database and that is around 1,000-image per database.  

Bonus: You can also download databases that are created in the past, by using a decent network connection. 

Okay, so now comes a bit of a tutorial for you. Are you ready? Here, we go;

When you add an image to the database, try and provide the physical size of that image. This will result in an enhanced image detection performance. Also,

• If you skip the part and do not provide any physical size of the image, then ARCore simply estimates the physical size and keeps iterating this estimation over time.
• And with the physical size (if provided), ARCore utilizes the available physical size and estimates the orientation and position of the image. So what it does is, ignores the discrepancy between the actual size and the one that is already provided.

Makes sense, right?

Check out the type of images that ARCore tracks or responds to;

• Fixed images on flat surfaces. For example, a picture that is hanging on the wall or a simple magazine on a flat table.

• Images that are set in motion. For example, an advertisement that is displayed on a moving bus. I am sure you get the type of examples that I am suggesting here. 

When ARCore starts to track an image, it produces estimates for the orientation and position of the image in every frame. And through this process, ARCore consistently refines the estimates as it keeps collecting more data on the same.

Now here’s the flow of it;

On detecting an image, ARCore keeps “tracking” the orientation and position of the image even if the image is no more in the view of the camera angle. I call it “efficiency” though!

However, while doing so, ARCore takes the orientation and position of the image to be static in the given surroundings. 

With the help of TrackingMethod, you can check if the ARCore is tracking the image based on the camera view detection (FULL_TRACKING). Similarly, you can also check if the ARCore is tracking the image purely based on the last detected position of the same (LAST_KNOWN_POSE).

Note: All these tracking activities take place on the device itself without the need for an active internet connection.

Requirements to build Augmented Image

Now let’s take a look at what all you need to build an augmented image; 

• The image has to take up a minimum of 25% of the camera view/frame for it to be easily detected.
• The object has to be smooth and flat. For example, it should not be wrinkled or wrapped around any curved surface.
• Make sure that the image is placed in a clear line of view of the camera angle. The image should not remain hidden in any way. The view needs to be very clear. That’s the whole point.

Best practices

Check out the ways to select reference images and what we incorporated for our project;

• The file format of these reference images can be either PNG or JPEG.
• The image resolution must be a minimum of 300 x 300 pixels. Trust me, high resolution has nothing to do with the performance
• As Color information is not really important here. You can use both grayscale and color images. No need to be specific about this.
• Do not use highly compressed images as that may hinder the process of feature extraction.
• Also, images with very few details or too many geometric features  ( QR codes, barcodes, logos, etc) results in bad detection as well as awful tracking performance.
• For the same purpose, do not use images that have repetitive patterns.
• Use the arcoreimg mechanism that is incorporated in the ARCore SDK. this helps with the image quality. Keep it between 0 to 100.

 We suggest a score of 75 for reference images though. 

Here are two examples:

Installation steps

There are some following steps

• In build.gradle of App, add a Sceneform UX library, Firebase ML Kit, and Sceneform Assets library.
  
• Select the Compile Option to assist Java for the Sceneform library.
  
• In Android Manifets.xml file add permission for AR features in-app, Camera permission, Internet permission add metadata to make it available for “Google Play Services for AR”.
  
• Add ArSceneView, provided by Sceneform ux library.
  
• Install Google Sceneform tools (beta) plugin from the android studio only.
  
• Create an augmented image.
  
• Add image in the drawable folder in-app.
  
• Add glb file in assets folder in-app.
  
• Next, check the image inside your camera frame with that of the augmented image.
  
• If it matches, just place an AR Object.
  
• Run the App.

Libraries/Dependencies with their versions :

Sceneform– 

Sceneform is basically a 3D framework making it easier to build ARCore applications without using OpenGL. In our project,we used the 1.15.0 version of google.ar.sceneform.ux:sceneform-ux.

Permissions :

We are using Camera permission and hardware.camera.ar permission.

Output will be

After scanning the image/poster you will see the 3D model of the respective image.

Challenges

In the beginning scanning, the image was quite difficult for us because we are scanning random images.

Overcoming challenges

After the research, we find that we can only scan images that we stored in our projects assets folder.

What we tried to do but did not succeed

In our project, we had to identify the pressure sensor. Unfortunately, the android ARCore SDK fails to support 3-D object recognition. So we failed to identify the sensor. 

So what we did instead is, we just attached an image on top of the sensors using the ‘Sensor Id’. This is exactly what we used for the augmented image too.

What are our future plans regarding the development of the project?

Our future plan for this project is to make the 3d model move, whenever we touch the screen. How cool would that be?

Conclusion

So that was all about this segment of our project on Augmented Image. I do hope that this was helpful in giving you a peek into our workflow and thought process too. 

The target industries for this project include gaming, e-commerce, health tech, and education. So you already have an idea of the vast impact that this field of research and work has on today’s world and business combined. 

Here’s to the aspiration of being a valuable cog in the tool.

Also, if you are an entrepreneur who wants to build something from scratch, then do reach out to us at TheCodeWork. We are a bunch of startup enthusiasts and it would be a pleasure to be of any assistance.

Till then, happy brainstorming.