Crie uma exposição interativa com o .NET, Azure Functions e a magia dos serviços cognitivos

No início de janeiro, o Moscow Electromuseum anunciou uma seleção aberta de exposições para participação na exposição Open Museum 2020 . Neste artigo, mostrarei como transformei a ideia de um retrato cognitivo, sobre o qual já escrevi , em uma exposição interativa e como, depois que o museu foi colocado em quarentena, essa exibição se tornou virtual. Sob o corte - uma excursão ao Azure Functions , Bot Framework , serviços cognitivos e detecção de rosto em aplicativos UWP .

AI April. , Microsoft AI . — . .

, :

, 2019, People Blending	, 2020, Cognitive Portrait

OpenMuseum, , - . , "" .

, . , "" , , :

• , .
• , , .
• , , .
• , , , "" .

, :

• UWP-, , - . UWP- Raspberry Pi Windows IoT Core. :
  
  • , (face detection).
  • — .
  • , .
• , :
  
  • .
  • , , .
  • Azure Storage.
  • , , Azure Storage .

, , - . , - , , -.

@PeopleBlenderBot. REST API, . , , " ", . , , , " ".

@PeopleBlenderBot , ( ) , , . , . , .

.

UWP

, Windows — " ". , Raspberry Pi Windows 10 IoT Core. , Raspberry Windows , Intel NUC.

:

XAML ( ):

<Grid Background="{ThemeResource ApplicationPageBackgroundThemeBrush}">
    <Grid.ColumnDefinitions>
        <ColumnDefinition Width="*"/>
        <ColumnDefinition Width="*"/>
    </Grid.ColumnDefinitions>
    <Grid.RowDefinitions>
        <RowDefinition Height="*"/>
        <RowDefinition Height="170"/>
    </Grid.RowDefinitions>
    <Grid x:Name="FacesCanvas" Grid.Row="0" Grid.Column="0">
        <CaptureElement x:Name="ViewFinder" />
        <Rectangle x:Name="FaceRect"/>
        <TextBlock x:Name="Counter" FontSize="60"/>
    </Grid>
    <Grid x:Name="ResultCanvas" Grid.Row="0" Grid.Column="1">
        <Image x:Name="ResultImage" Source="Assets/bgates.jpg"/>
    </Grid>
    <ItemsControl x:Name="FacesLine" Grid.Row="1" Grid.ColumnSpan="2" 
              ItemsSource="{x:Bind Faces,Mode=OneWay}"/>
</Grid>

:

• ViewFinder .
• FaceRect Counter ViewFinder, 3-2-1, .
• ResultImage — , .
• FacesLine , . Faces, Observable. , , Faces.
• XAML .

UWP , . , ( ).

, , , ViewFinder:

MC = new MediaCapture();
var cameras = await DeviceInformation.FindAllAsync(
                              DeviceClass.VideoCapture);
var camera = cameras.First();
var settings = new MediaCaptureInitializationSettings() 
                          { VideoDeviceId = camera.Id };
await MC.InitializeAsync(settings);
ViewFinder.Source = MC;

. FaceDetectionEffect, FaceDetectedEvent :

var def = new FaceDetectionEffectDefinition();
def.SynchronousDetectionEnabled = false;
def.DetectionMode = FaceDetectionMode.HighPerformance;
FaceDetector = (FaceDetectionEffect)
     (await MC.AddVideoEffectAsync(def, MediaStreamType.VideoPreview));
FaceDetector.FaceDetected += FaceDetectedEvent;
FaceDetector.DesiredDetectionInterval = TimeSpan.FromMilliseconds(100);
FaceDetector.Enabled = true;
await MC.StartPreviewAsync();

, FaceDetectedEvent, , . Counter, 3, 2 1. 0, MemoryStream, CallCognitiveFunction, :

var ms = new MemoryStream();
await MC.CapturePhotoToStreamAsync(
      ImageEncodingProperties.CreateJpeg(), ms.AsRandomAccessStream());
var cb = await GetCroppedBitmapAsync(ms,DFace.FaceBox);
Faces.Add(cb);
var url = await CallCognitiveFunction(ms);
ResultImage.Source = new BitmapImage(new Uri(url));

, - REST POST, , - , URL . , URL Source ResultImage, ( UWP ).

. GetCroppedBitmapAsync, , Faces, ( DataBinding).

CallCognitiveFunction , , REST HttpClient:

private async Task<string> CallCognitiveFunction(MemoryStream ms)
{
    ms.Position = 0;
    var resp = await http.PostAsync(function_url,new StreamContent(ms));
    return await resp.Content.ReadAsStringAsync();
}

Azure Function

Azure Functions Python. Azure Functions :

• , . Azure Functions (serverless).
• functions consumption plan, , . (15 ), . , .
• , .
• Azure Function . REST-, , , .
• Azure Function , . , , .

, , . Azure Function, -, . , , — . boilerplate code!

Azure Functions , , - , -, IoT- ..

Python OpenCV — . , Azure Functions Python, — (V2) .

. , . VS Code, — .

CLI:

func init coportrait –python
cd coportrait
func new --name pdraw --template "HTTP trigger"

, Azure Functions Core Tools.

:

• Python ( , __init__.py).
• function.json, , .. , .

HTTP-, function.json :

{
  "scriptFile": "__init__.py",
  "bindings": [
    {
      "authLevel": "function",
      "type": "httpTrigger",
      "direction": "in",
      "name": "req",
      "methods": [ "post" ]},
    {
      "type": "http",
      "direction": "out",
      "name": "$return" 
}]}

, req HTTP-, — HTTP response, ($return). , POST-. "get", .

__init__.py, ( ):

def main(req:func.HttpRequest) -> func.HttpResponse:
    logging.info('Execution begins…')
    return func.HttpResponse(f"Hello {name}!")

, req — . , JPEG, get_body, numpy- OpenCV:

body = req.get_body()
nparr = np.fromstring(body, np.uint8)
img = cv2.imdecode(nparr, cv2.IMREAD_COLOR)

azure.storage.blob.BlockBlobService. , , , , . Azure Python .

cin:

blob = BlockBlobService(account_name=..., account_key=...) 
sec_p = int((end_date-datetime.datetime.now()).total_seconds())
name =  f"{sec_p:09d}-{time.strftime('%Y%m%d-%H%M%S')}"
blob.create_blob_from_bytes("cin",name,body)

. 10 , , . , , . 1 2021 , ( ) , YYYYMMDD-HHMMSS. 2021 ...

Face API , :

cogface = cf.FaceClient(cognitive_endpoint,
                        CognitiveServicesCredentials(cognitive_key))
res = cogface.face.detect_with_stream(io.BytesIO(body),
                        return_face_landmarks=True)
if res is not None and len(res)>0:
    tr = affine_transform(img,res[0].face_landmarks.as_dict())
    body = cv2.imencode('.jpg',tr)[1]
    blob.create_blob_from_bytes("cmapped",name,body.tobytes())

affine_transform . , , JPEG- cmapped.

, , 10 cmapped . list_blobs , 10 islice, imdecode numpy- :

imgs = [ imdecode(blob.get_blob_to_bytes("cmapped",x.name).content) 
         for x in itertools.islice(blob.list_blobs("cmapped"),10) ]
imgs = np.array(imgs).astype(np.float)

, . , float, — np.uint8:

res = (np.average(imgs,axis=0)).astype(np.uint8)

, out, cv2.imencode, create_blob_from_bytes:

b = cv2.imencode('.jpg',res)[1]
r = blob.create_blob_from_bytes("out",f"{name}.jpg",b.tobytes())
result_url = f"https://{act}.blob.core.windows.net/out/{name}.jpg"
return func.HttpResponse(result_url)

— out, , , .

__init__.py, requirements.txt:

azure-functions
opencv-python
azure-cognitiveservices-vision-face
azure-storage-blob==1.5.0

, :

func start

-, URL , Postman. POST, - .
URL function_url UWP-, .

Azure Function , Python Azure Function Azure Portal, Azure CLI:

az functionapp create --resource-group PeopleBlenderBot
    --os-type Linux   --consumption-plan-location westeurope
    --runtime python  --runtime-version 3.7
    --functions-version 2
    --name coportrait --storage-account coportraitstore

Azure Portal , ( ) . Azure CLI , .

, , :

func azure functionapp publish coportrait

Azure Portal, URL:

URL : https://coportrait.azurewebsites.net/api/pdraw?code=geE..e3P==, .. . ( ) function_url UWP-, , !

-

, . — Microsoft Bot Framework -.

, , . , -.

- C# , . - Python, .NET , .

, VS Bot Template Visual Studio, Echo:

PeopleBlenderBot.

Bots\EchoBot.cs, OnMessageActivityAsync. .. turnContext. — Activity, Text, Attachments. , :

if (turnContext.Activity.Attachments?.Count>0)
{
   // do the magic
}
else
{
    await turnContext.SendActivityAsync("Please send picture");
}

, ContentUrl Attachment. if-, http-:

  var http = new HttpClient();
  var resp = await http.GetAsync(Attachments[0].ContentUrl);
  var str = await resp.Content.ReadAsStreamAsync();

Azure Function, POST-. UWP-:

  resp = await http.PostAsync(function_url, new StreamContent(str));
  var url = await resp.Content.ReadAsStringAsync();

URL , . , Hero-card, attachment-:

  var msg = MessageFactory.Attachment(
      (new HeroCard()
      { Images = new CardImage[] { new CardImage(url) } }
      ).ToAttachment());
  await turnContext.SendActivityAsync(msg);

, Bot Framework Emulator:

, , , Azure CLI ARM-. , Azure Portal:

1. Web App Bot , Echo Bot . :
   
   • Bot Connector App, , , Telegram
   • Bot Web App, -,
2. appsettings.json PeopleBlenderBot. App Id App Password, bot connector.
3. Visual Studio PeopleBlenderBot Publish. Existing Web App, , 1. .

, Bot Connector App Azure Portal, , Web Chat:

— ! , Telegram Channels :

, . , Mechanium, Moscow Maker Faire 2019. , - . — Science Art, — — . , , ! - — pull request ! , , . — , - !

-, , Telegram @peopleblenderbot, . ! , , , , . , , .

! , , !