In early January, the Moscow Electromuseum announced an open selection of exhibits for participation in the Open Museum 2020 exhibition . In this article I will tell you how I turned the idea of a cognitive portrait that I already wrote about into an interactive exhibit, and how after the museum was quarantined, this exhibit became virtual. Under the cut - an excursion to the Azure Functions , Bot Framework , cognitive services and face detection in UWP applications .
AI April. , Microsoft AI . — . .
, :
OpenMuseum, , - . , "" .
, . , "" , , :
, :
, , - . , - , , -.
@PeopleBlenderBot. REST API, . , , " ", . , , , " ".
@PeopleBlenderBot , ( ) , , . , . , .
.
UWP
, Windows — " ". , Raspberry Pi Windows 10 IoT Core. , Raspberry Windows , Intel NUC.
:
XAML ( ):
<Grid Background="{ThemeResource ApplicationPageBackgroundThemeBrush}">
<Grid.ColumnDefinitions>
<ColumnDefinition Width="*"/>
<ColumnDefinition Width="*"/>
</Grid.ColumnDefinitions>
<Grid.RowDefinitions>
<RowDefinition Height="*"/>
<RowDefinition Height="170"/>
</Grid.RowDefinitions>
<Grid x:Name="FacesCanvas" Grid.Row="0" Grid.Column="0">
<CaptureElement x:Name="ViewFinder" />
<Rectangle x:Name="FaceRect"/>
<TextBlock x:Name="Counter" FontSize="60"/>
</Grid>
<Grid x:Name="ResultCanvas" Grid.Row="0" Grid.Column="1">
<Image x:Name="ResultImage" Source="Assets/bgates.jpg"/>
</Grid>
<ItemsControl x:Name="FacesLine" Grid.Row="1" Grid.ColumnSpan="2"
ItemsSource="{x:Bind Faces,Mode=OneWay}"/>
</Grid>
:
ViewFinder .FaceRect Counter ViewFinder, 3-2-1, .ResultImage — , .FacesLine , . Faces, Observable. , , Faces.- XAML .
UWP , . , ( ).
, , , ViewFinder:
MC = new MediaCapture();
var cameras = await DeviceInformation.FindAllAsync(
DeviceClass.VideoCapture);
var camera = cameras.First();
var settings = new MediaCaptureInitializationSettings()
{ VideoDeviceId = camera.Id };
await MC.InitializeAsync(settings);
ViewFinder.Source = MC;
. FaceDetectionEffect, FaceDetectedEvent :
var def = new FaceDetectionEffectDefinition();
def.SynchronousDetectionEnabled = false;
def.DetectionMode = FaceDetectionMode.HighPerformance;
FaceDetector = (FaceDetectionEffect)
(await MC.AddVideoEffectAsync(def, MediaStreamType.VideoPreview));
FaceDetector.FaceDetected += FaceDetectedEvent;
FaceDetector.DesiredDetectionInterval = TimeSpan.FromMilliseconds(100);
FaceDetector.Enabled = true;
await MC.StartPreviewAsync();
, FaceDetectedEvent, , . Counter, 3, 2 1. 0, MemoryStream, CallCognitiveFunction, :
var ms = new MemoryStream();
await MC.CapturePhotoToStreamAsync(
ImageEncodingProperties.CreateJpeg(), ms.AsRandomAccessStream());
var cb = await GetCroppedBitmapAsync(ms,DFace.FaceBox);
Faces.Add(cb);
var url = await CallCognitiveFunction(ms);
ResultImage.Source = new BitmapImage(new Uri(url));
, - REST POST, , - , URL . , URL Source ResultImage, ( UWP ).
. GetCroppedBitmapAsync, , Faces, ( DataBinding).
CallCognitiveFunction , , REST HttpClient:
private async Task<string> CallCognitiveFunction(MemoryStream ms)
{
ms.Position = 0;
var resp = await http.PostAsync(function_url,new StreamContent(ms));
return await resp.Content.ReadAsStringAsync();
}
Azure Function
Azure Functions Python. Azure Functions :
- , . Azure Functions (serverless).
- functions consumption plan, , . (15 ), . , .
- , .
- Azure Function . REST-, , , .
- Azure Function , . , , .
, , . Azure Function, -, . , , — . boilerplate code!
Azure Functions , , - , -, IoT- ..
Python OpenCV — . , Azure Functions Python, — (V2) .
. , . VS Code, — .
CLI:
func init coportrait –python
cd coportrait
func new --name pdraw --template "HTTP trigger"
, Azure Functions Core Tools.
:
- Python ( ,
__init__.py). function.json, , .. , .
HTTP-, function.json :
{
"scriptFile": "__init__.py",
"bindings": [
{
"authLevel": "function",
"type": "httpTrigger",
"direction": "in",
"name": "req",
"methods": [ "post" ]},
{
"type": "http",
"direction": "out",
"name": "$return"
}]}
, req HTTP-, — HTTP response, ($return). , POST-. "get", .
__init__.py, ( ):
def main(req:func.HttpRequest) -> func.HttpResponse:
logging.info('Execution begins…')
return func.HttpResponse(f"Hello {name}!")
, req — . , JPEG, get_body, numpy- OpenCV:
body = req.get_body()
nparr = np.fromstring(body, np.uint8)
img = cv2.imdecode(nparr, cv2.IMREAD_COLOR)
azure.storage.blob.BlockBlobService. , , , , . Azure Python .
cin:
blob = BlockBlobService(account_name=..., account_key=...)
sec_p = int((end_date-datetime.datetime.now()).total_seconds())
name = f"{sec_p:09d}-{time.strftime('%Y%m%d-%H%M%S')}"
blob.create_blob_from_bytes("cin",name,body)
. 10 , , . , , . 1 2021 , ( ) , YYYYMMDD-HHMMSS. 2021 ...
Face API , :
cogface = cf.FaceClient(cognitive_endpoint,
CognitiveServicesCredentials(cognitive_key))
res = cogface.face.detect_with_stream(io.BytesIO(body),
return_face_landmarks=True)
if res is not None and len(res)>0:
tr = affine_transform(img,res[0].face_landmarks.as_dict())
body = cv2.imencode('.jpg',tr)[1]
blob.create_blob_from_bytes("cmapped",name,body.tobytes())
affine_transform . , , JPEG- cmapped.
, , 10 cmapped . list_blobs , 10 islice, imdecode numpy- :
imgs = [ imdecode(blob.get_blob_to_bytes("cmapped",x.name).content)
for x in itertools.islice(blob.list_blobs("cmapped"),10) ]
imgs = np.array(imgs).astype(np.float)
, . , float, — np.uint8:
res = (np.average(imgs,axis=0)).astype(np.uint8)
, out, cv2.imencode, create_blob_from_bytes:
b = cv2.imencode('.jpg',res)[1]
r = blob.create_blob_from_bytes("out",f"{name}.jpg",b.tobytes())
result_url = f"https://{act}.blob.core.windows.net/out/{name}.jpg"
return func.HttpResponse(result_url)
— out, , , .
__init__.py, requirements.txt:
azure-functions
opencv-python
azure-cognitiveservices-vision-face
azure-storage-blob==1.5.0
, :
func start
-, URL , Postman. POST, - .
URL function_url UWP-, .
Azure Function , Python Azure Function Azure Portal, Azure CLI:
az functionapp create --resource-group PeopleBlenderBot
--os-type Linux --consumption-plan-location westeurope
--runtime python --runtime-version 3.7
--functions-version 2
--name coportrait --storage-account coportraitstore
Azure Portal , ( ) . Azure CLI , .
, , :
func azure functionapp publish coportrait
Azure Portal, URL:
URL : https://coportrait.azurewebsites.net/api/pdraw?code=geE..e3P==, .. . ( ) function_url UWP-, , !
-
, . — Microsoft Bot Framework -.
, , . , -.
- C# , . - Python, .NET , .
, VS Bot Template Visual Studio, Echo:
PeopleBlenderBot.
Bots\EchoBot.cs, OnMessageActivityAsync. .. turnContext. — Activity, Text, Attachments. , :
if (turnContext.Activity.Attachments?.Count>0)
{
// do the magic
}
else
{
await turnContext.SendActivityAsync("Please send picture");
}
, ContentUrl Attachment. if-, http-:
var http = new HttpClient();
var resp = await http.GetAsync(Attachments[0].ContentUrl);
var str = await resp.Content.ReadAsStreamAsync();
Azure Function, POST-. UWP-:
resp = await http.PostAsync(function_url, new StreamContent(str));
var url = await resp.Content.ReadAsStringAsync();
URL , . , Hero-card, attachment-:
var msg = MessageFactory.Attachment(
(new HeroCard()
{ Images = new CardImage[] { new CardImage(url) } }
).ToAttachment());
await turnContext.SendActivityAsync(msg);
, Bot Framework Emulator:
, , , Azure CLI ARM-. , Azure Portal:
- Web App Bot , Echo Bot . :
- Bot Connector App, , , Telegram
- Bot Web App, -,
appsettings.json PeopleBlenderBot. App Id App Password, bot connector.- Visual Studio
PeopleBlenderBot Publish. Existing Web App, , 1. .
, Bot Connector App Azure Portal, , Web Chat:
— ! , Telegram Channels :
, . , Mechanium, Moscow Maker Faire 2019. , - . — Science Art, — — . , , ! - — pull request ! , , . — , - !
-, , Telegram @peopleblenderbot, . ! , , , , . , , .
! , , !