Beray Bentesen in Xamarin Android

Get Started with the Google Mobile Vision API - Xamarin Android

 

This article will be covering :

  • Installing Google Mobile Vision API
  • Detecting landmarks (e.g. mouth, eyes)
  • Using Face api with Xamarin Android project

The Mobile Vision Api contains three Apis; Face Detection, Barcode Detection and the Text. This article cover Face Detection.

Face classification is used to check the landmarks(e.g. smiling faces,closed eyes).

Face Detection Api also detects faces at different angles and reports the Euler Y and Euler Z angles

Getting Started

  • In order to install Vision Api you have to include the Xamarin.GooglePlayServices.Vision library (32.940.0-beta 3 is the latest version)
  • You can use either the Nuget Package Manager,or install it through Package Manager Console

Working with Android Project

  • To enable available libraries that present for face detection, add this meta-data to your manifest file
<meta-data  
android:name="com.google.android.gms.vision.DEPENDENCIES"  
android:value="face" />  
  • Replace your layout code with the following layout. The button starts processing of the image (from Drawable folder), process it and then displays in the ImageView.
  • The image can be loaded from the device either via the camera or the gallery. Camera and gallery samples will be posted soon.
<RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android"  
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:orientation="vertical">
    <Button
        android:layout_alignParentBottom="true"
        android:layout_centerHorizontal="true"
        android:id="@+id/processButton"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:onClick="processImage"
        android:text="Process Image" />
    <ImageView
        android:id="@+id/image_view"
        android:layout_width="match_parent"
        android:layout_height="match_parent"
        android:contentDescription="imageForDetection" />
</RelativeLayout>  

Updating Activity

  • Add following variables into your Activity
public Paint rectPaint;  
public Bitmap imageBitmap;  
public Bitmap BitmapForMemory;  
public Canvas canvas;  
public Button processButton;  
public ImageView processImageView;  
  • Defining Button and ImageView with Click event handler.
processButton = FindViewById<Button>(Resource.Id.processButton);  
processImageView = FindViewById<ImageView>(Resource.Id.image_view);

processButton.Click += (sender, e) =>  
            {
                new Thread(() => RunOnUiThread(processImage)).Start();
            };
  • Create processImage method which works asynchronous to be able call await operator.
public async void processImage()  
            {
              ... 
            }
  • Creating a new BitmapFactory.Options object and set inmutable to true. Setting immutable true means you are able to apply effects programmatically to bitmap.
var bitmapOptions = new BitmapFactory.Options();  
bitmapOptions.InMutable = true;  
  • Then call initializeBitmap method. This method creates a Bitmap with the DecodeResource method from BitmapFactory class.
await Task.Run(() => initializeBitmap(bitmapOptions));  
  • initializeBitmap method
public void initializeBitmap(BitmapFactory.Options bitmapOptions)  
        {
            imageBitmap = BitmapFactory.DecodeResource(Resources, Resource.Drawable.phy,bitmapOptions);
            BitmapForMemory = Bitmap.CreateBitmap(imageBitmap.Width, imageBitmap.Height, Bitmap.Config.Rgb565);
        }
  • Continue with FaceDetector Api. This project uses image from Drawable folder that is why TrackingEnabled set false
FaceDetector faceDetector = new FaceDetector.Builder(this)                                                .SetTrackingEnabled(true)                                                   .SetLandmarkType(LandmarkDetectionType.All).Build();
  • Checking device support
if (!faceDetector.IsOperational)  
            {
            new AlertDialog.Builder(this).SetMessage("Device does not support face detection").Show();
            }
  • Creating a frame using the default bitmap and calling on the face detector to get the face objects.
else {  
        processButton.Visibility = ViewStates.Gone;
        var frame = new Frame.Builder().SetBitmap(imageBitmap).Build();
        var sparseArray = await Task.Run(() => faceDetector.Detect(frame));
     }
  • detectFaces method draws landmark for each face that found in the image.
  • After drawing landmarks, BitmapForMemory will be available for using as ImageDrawable.
  • The last step is releasing faceDetector
        await Task.Run(() => detectFaces(sparseArray));
        processImageView.SetImageDrawable(new BitmapDrawable(Resources,BitmapForMemory));
        faceDetector.Release();
  • detectFaces method (float variables can be used for additional processes)
public void detectFaces(SparseArray sparseArray)  
        {
            for (int i = 0; i < sparseArray.Size(); i++)
            {
                var face = (Face)sparseArray.ValueAt(i);

                //float left = face.Position.X;
                //float top = face.Position.Y;
                //float right = left + face.Width;
                //float bottom = right + face.Height;

                drawLandmarks(face, i + 1);
            }
        }
  • drawLandmarks method draws circles for each landmark(left eye, right eye etc.)
public void drawLandmarks(Face face,int i)  
        {
            foreach (Landmark landmark in face.Landmarks)
            {
                createPaint(Paint.Style.Stroke,Color.Red);
                var cx = (int)(landmark.Position.X);
                var cy = (int)(landmark.Position.Y);
                canvas.DrawCircle(cx, cy, 30, rectPaint);
                drawFaceCount((int)landmark.Type, cx, cy,i);
            }
        }
  • drawFaceFount method draws text with number of detected face
public void drawFaceCount(int landmarkType, float cx, float cy, int face)  
        {
            if (landmarkType == 0)
            {
                createPaint(Paint.Style.FillAndStroke,Color.Yellow);
                string type = "Face : " + face;
                rectPaint.TextSize = 90;
                canvas.DrawText(type, cx - 70, cy + 190, rectPaint);
            }
        }
  • Helper method for painting
public void createPaint(Paint.Style style,Color color)  
        {
            rectPaint = new Paint();
            rectPaint.StrokeWidth = 5;
            rectPaint.Color = color;
            rectPaint.SetStyle(style);
        }

Result