使用TensorFlow Lite和Azure自定义视觉对Android上的图像进行分类

图像分类库

1. Azure自定义视觉服务

1. 下载培训图片
2. 在图像中标记标签/对象标签
3. 重复（通过更多训练数据使模型变得更好）
4. 就是这样-Custom Vision负责其余工作！

2. TensorFlow Lite

使用Azure + Xamarin.Android实现图像分类

1.模型训练

• 名称： XamarinImageClassification
• 说明：识别图像中的对象
• 资源： [创建新资源]
• 项目类型：分类
• 分类类型：多标签（每个图像有多个标签）
• 域：常规（紧凑型）
• 出口能力：基本平台

2.从Azure自定义视觉服务中导出经过训练的模型

3.将TensorFlow Lite导入我们的Xamarin.Android应用

• labels.txt包含准备在Custom Vision网站上进行培训期间创建的图像标签
• models.tflite是我们用于预测的机器学习模型。

• 模型
• labels.txt

4. Xamarin.Android的图像分类代码的实现

ImageClassificationModel.cs

public class ImageClassificationModel 
{ 
    public ImageClassificationModel(string tagName, float probability) 
    { 
        TagName = tagName; 
        Probability = probability; 
    } 
   
    public float Probability { get; } 
    public string TagName { get; } 
}

TensorflowClassifier.cs

using System.Collections.Generic;
using System.IO;
using System.Linq;
using Android.App;
using Android.Graphics;
using Java.IO;
using Java.Nio;
using Java.Nio.Channels;

public class TensorflowClassifier
{
    //FloatSize     4,        4 
    const int FloatSize = 4;
    //PixelSize -     3,        : ,   .
    const int PixelSize = 3;

    public List<ImageClassificationModel> Classify(byte[] image)
    {
        var mappedByteBuffer = GetModelAsMappedByteBuffer();
        var interpreter = new Xamarin.TensorFlow.Lite.Interpreter(mappedByteBuffer);

        //   ,       
        var tensor = interpreter.GetInputTensor(0);
        var shape = tensor.Shape();

        var width = shape[1];
        var height = shape[2];

        var byteBuffer = GetPhotoAsByteBuffer(image, width, height);

        // StreamReader    labels.txt
        var streamReader = new StreamReader(Application.Context.Assets.Open("labels.txt"));

        // labels.txt  List<string>
        var labels = streamReader.ReadToEnd().Split('\n').Select(s => s.Trim()).Where(s => !string.IsNullOrEmpty(s)).ToList();

        //     Java.Lang.Object,    Xamarin.TensorFlow.List.Interpreter.
        var outputLocations = new float[1][] { new float[labels.Count] };
        var outputs = Java.Lang.Object.FromArray(outputLocations);

        interpreter.Run(byteBuffer, outputs);
        var classificationResult = outputs.ToArray<float[]>();

        // classificationResult     ,  ,     
        var classificationModelList = new List<ImageClassificationModel>();

        for (var i = 0; i < labels.Count; i++)
        {
            var label = labels[i]; classificationModelList.Add(new ImageClassificationModel(label, classificationResult[0][i]));
        }

        return classificationModelList;
    }

    // model.tflite  Java.Nio.MappedByteBuffer,    Xamarin.TensorFlow.Lite.Interpreter.
    private MappedByteBuffer GetModelAsMappedByteBuffer()
    {
        var assetDescriptor = Application.Context.Assets.OpenFd("model.tflite");
        var inputStream = new FileInputStream(assetDescriptor.FileDescriptor);

        var mappedByteBuffer = inputStream.Channel.Map(FileChannel.MapMode.ReadOnly, assetDescriptor.StartOffset, assetDescriptor.DeclaredLength);

        return mappedByteBuffer;
    }

    //     TensorFlow
    private ByteBuffer GetPhotoAsByteBuffer(byte[] image, int width, int height)
    {
        var bitmap = BitmapFactory.DecodeByteArray(image, 0, image.Length);
        var resizedBitmap = Bitmap.CreateScaledBitmap(bitmap, width, height, true);

        var modelInputSize = FloatSize * height * width * PixelSize;
        var byteBuffer = ByteBuffer.AllocateDirect(modelInputSize);
        byteBuffer.Order(ByteOrder.NativeOrder());

        var pixels = new int[width * height];
        resizedBitmap.GetPixels(pixels, 0, resizedBitmap.Width, 0, 0, resizedBitmap.Width, resizedBitmap.Height);

        var pixel = 0;

        //     Java.Nio.ByteBuffer
        for (var i = 0; i < width; i++)
        {
            for (var j = 0; j < height; j++)
            {
                var pixelVal = pixels[pixel++];

                byteBuffer.PutFloat(pixelVal >> 16 & 0xFF);
                byteBuffer.PutFloat(pixelVal >> 8 & 0xFF);
                byteBuffer.PutFloat(pixelVal & 0xFF);
            }
        }

        bitmap.Recycle();

        return byteBuffer;
    }
}

进一步研究的材料

• 完整的github代码
• Azure自定义视觉服务
• Azure认知服务
• 社区AI展览：Custom Vision，Xamarin和Simpsons