Developers.io 2017 iPhoneによるAlexa/Lex/Pollyを利用した 音声対応クライアントの作成方法

1. iPhoneによるAlexa/Lex/Pollyを利⽤した ⾳声対応クライアントの作成⽅法

2. ⾃⼰紹介 クラスメソッド（株） アプリケーションエンジニア 平内真⼀ ・モバイルアプリサービス部（iOSエンジニア） ・フリーソフト BlackJumboDog ・Microsoft MVP (2013/1〜)

3. アジェンダ 1. Overview 2. Alexa 3. Lex 4. Polly 5. Summary

4. アジェンダ 1. Overview 2. Alexa 3. Lex 4. Polly 5. Summary

5. 1.概要 Overview

6. 1.概要 Skill Skill Skill Skill Lambda Skill

7. 1.概要 Skill Skill Skill Skill Lambda Skill

8. ゴール 1.Amazonの各種⾳声サービスの概要 2.⾳声対応クライアントの実装⽅法 3.要件に応じたサービスの選択

9. 1.概要 Alexa

10. Alexa アーキテクチャー 今⽇の東京の天気は？ 今⽇の東京の天気は、晴れ時々曇り 最⾼気温は２７度・・・・

11. Alexa アーキテクチャー AWS re:Invent 2016: How Capital One Built a Voice-Based Banking Skill for Amazon Echo (ALX201)

12. Alexa アーキテクチャー AWS re:Invent 2016: How Capital One Built a Voice-Based Banking Skill for Amazon Echo (ALX201)

13. Alexa Alexa Voice Service https://developer.amazon.com/public/solutions/alexa/alexa-voice-service/content/avs-api-overview

14. Alexa Alexa Voice Service http://dev.classmethod.jp/smartphone/alexa-client-friendly-voice-assistant/

15. Alexa • Endpoints/Protocol •Interface •Registration •Authorization

16. Alexa Endpoints/Protocol Region Supported Countries URL North America US https://avs-alexa-na.amazon.com Europe UK, Germany https://avs-alexa-eu.amazon.com

17. Alexa Endpoints/Protocol :method = GET :scheme = https :path = /{{API version}}/directives authorization = Bearer {{YOUR_ACCESS_TOKEN}} ダウンチャンネルストリーム タイマー・アラームなど 接続後１０秒以内 クライアントからの半閉状態で開いたまま 接続の存続期間中はAVSから開く ⻑い休⽌があることは珍しいことではない v20160207

18. Alexa Endpoints/Protocol :method = GET :scheme = https :path = /ping authorization = Bearer {{YOUR_ACCESS_TOKEN}} Ping and Timeout 5分に１回

19. Alexa Endpoints/Protocol :method = POST :scheme = https :path = /{{API version}}/events authorization = Bearer {{YOUR_ACCESS_TOKEN}} content-type = multipart/form-data; boundary={{BOUNDARY_TERM_HERE}}

20. Alexa Endpoints/Protocol https://developer.amazon.com/public/solutions/alexa/alexa-voice-service/docs/avs-http2-requests JSON イベント・ディレクティブ Audio ⾳声

21. Alexa Endpoints/Protocol audio encoded 16bit Linear PCM (LPCM16) 16kHz sample rate Single channel Little endian byte order

22. Alexa • Endpoints/Protocol •Interface •Registration •Authorization

23. Alexa Interface Interface Ddescription SpeechSynthesizer Alexaのスピーチインターフェース SpeechRecognizer AVSのコアインターフェース 各ユーザ発話は、認識イベントを利⽤する Speaker ミュートやミュート解除を含む、デバイスやアプリケーションの⾳量コントロールの ためのインターフェイス Settings ロケールなど、製品のAlexa設定を管理するためのインタフェース PlaybackController ボタンアフォーダンスを介して再⽣キューをナビゲートするためのインターフェイス AudioPlayer オーディオ再⽣を管理および制御するためのインターフェイス Alerts タイマーとアラームの設定、停⽌、および削除のためのインターフェース System Alexaにクライアント情報を提供するためのインタフェース

24. Alexa Interface SpeechRecognizer Interface Event/Directive Recognize Event ユーザ⾳声の送信 StopCapture Directive 録⾳停⽌指⽰ ExpectSpeech Directive 録⾳開始指⽰（会話継続） ExpectSpeechTimedOut Event タイムアウト

25. Alexa Interface SpeechSynthesizer Interface Event/Directive Speak Directive Alexaの送話 SpeechStarted Event 再⽣開始 SpeechFinished Event 再⽣終了

26. Alexa Interface https://developer.amazon.com/public/solutions/alexa/alexa-voice-service/reference/speechrecognizer

27. Alexa Interface Event Code Directive 1 録⾳開始 2 録⾳終了 3 SpeechRecognizer.Recognize + 録⾳⾳声 200 SpeechSynthesizer.Speak + Alexa⾳声 4 SpeechSynthesizer.SpeechStarted 204 会話が継続する場合 SpeechRecognizer.ExpectSpeech 会話が継続しない場合 DIRECTIVEなし 5 再⽣開始 6 再⽣終了 7 SpeechSynthesizer.SpeechFinished 204 SpeechRecognizer.ExpectSpeechが返っている場合は、１に戻る

28. Alexa • Endpoints/Protocol •Interface •Registration •Authorization

29. Alexa Registration https://developer.amazon.com

30. Alexa Registration

31. Alexa Registration

32. Alexa Registration Application Type ID 認証で使⽤されます

33. Alexa Registration

34. Alexa Registration Key / Bundle Id 認証で使⽤されます

35. Alexa Registration

36. Alexa Registration 新規

37. Alexa Registration

38. Alexa Registration 必要なのは以下の３つ ＊Application Type ID（変更できない） ＊Bundle ID（追加可能） ＊Key （追加可能） 削除できない

39. Alexa • Endpoints/Protocol •Interface •Registration •Authorization

40. Alexa Authorization https://developer.amazon.com/sdk-download

41. Alexa Authorization

42. Alexa Authorization Swiftから利⽤するためには、Bridging-Headerを作成 #import <LoginWithAmazon/LoginWithAmazon.h>

43. Alexa Authorization リダイレクトをスキームで受けれるように、URL Schemesで、Bundle Identifireの 前にamzn- を付けたスキームを指定します。 amzn-jp.classmethod.us.ios.Friendly

44. Alexa Authorization 開発者ポータル

45. Alexa Authorization class AppDelegate: UIResponder, UIApplicationDelegate { func application(_ application: UIApplication, open url: URL, sourceApplication: String?, annotation: Any) -> Bool { return AIMobileLib.handleOpen(url, sourceApplication: sourceApplication) }

46. Alexa Authorization @IBAction func tapLoginButton(_ sender: Any) { let SCOPE_DATA = "{alexa:all: { productID:プロダクトID, productInstanceAttributes: { {deviceSerialNumber:デバイスシリアル番号} } } AIMobileLib.authorizeUser(forScopes: ["alexa:all"], delegate: self, options: [kAIOptionScopeData:SCOPE_DATA]) } 開発者ポータル

47. Alexa Authorization

48. Alexa Authorization https://www.amazon.com

49. Alexa Authorization

50. 1.概要 Lex

51. Lex アーキテクチャー AWS Console Lex()

52. Lex アーキテクチャー

53. Lex アーキテクチャー Amazon Polly and Amazon Lex Workshop

54. Lex デモアプリ http://dev.classmethod.jp/smartphone/ios-lex-tap/

55. Lex • Endpoints/API •AWS mobile SDK •Authorization •SessionAttribute

56. Lex Endpoints/API Service Region URL Model building service US East (N.Virginia) https://models.lex.us-east-1.amazonaws.com Runtime service US East (N.Virginia) https://runtime.lex.us-east-1.amazonaws.com

57. Lex Endpoints/API CreateBot DeleteBot DeleteBotAlias GetBotGetBotAlias GetBotAliases GetBuiltinIntent GetBuiltinSlotTypes PutBot PutBotAlias PutIntent PutSlotType Etc. Amazon Lex Model Building Service

58. Lex Endpoints/API PostContent PostText Amazon Lex Runtime Service

59. Lex Endpoints/API PostContent Sends user input (text or speech) to Amazon Lex PostText Sends user input (text-only) to Amazon Lex POST /bot/botName/alias/botAlias/user/userId/content HTTP/1.1 x-amz-lex-session-attributes: sessionAttributes Content-Type: contentType Accept: accept inputStream POST /bot/botName/alias/botAlias/user/userId/text HTTP/1.1 Content-type: application/json { "inputText": "string", "sessionAttributes": { "string" : "string" } }

60. Lex • Endpoints/API •AWS mobile SDK •Authorization •SessionAttribute

61. Lex AWS mobile SDK for iOS

62. Lex AWS mobile SDK for iOS http://dev.classmethod.jp/smartphone/amazon-lex-ios-sdk/ 1. pod install 2. マイク利⽤許可（info.plist） 3. CognitoによるIdentity発⾏とLexの初期化 4. AWSLexVoiceBottn

63. Lex AWS mobile SDK for iOS source 'https://github.com/CocoaPods/Specs.git’ target 'BotSampleApp' do platform :ios, '9.0’ use_frameworks! pod 'AWSLex’ pod 'AWSCognito’ end

64. Lex AWS mobile SDK for iOS <key>NSMicrophoneUsageDescription</key> <string>For interaction with Amazon Lex</string>

65. Lex AWS mobile SDK for iOS let poolId = "us-east-1:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx" let credentialsProvider = AWSCognitoCredentialsProvider(regionType:.USEast1,identityPoolId:poolId) let configuration = AWSServiceConfiguration(region:.USEast1, credentialsProvider:credentialsProvider) AWSServiceManager.default().defaultServiceConfiguration = configuration let BotName = “LexSample" let BotAlias = “$LATEST" let chatConfig = AWSLexInteractionKitConfig.defaultInteractionKitConfig(withBotName: BotName, botAlias: BotAlias) AWSLexInteractionKit.register(with: configuration!, interactionKitConfiguration: chatConfig, forKey: "AWSLexVoiceButton") AWSLexInteractionKit.register(with: configuration!, interactionKitConfiguration: chatConfig, forKey: "chatConfig") cognito Lex

66. Lex AWS mobile SDK for iOS

67. Lex AWS mobile SDK for iOS http://dev.classmethod.jp/smartphone/amazon-lex-ios-sdk/

68. Lex AWS mobile SDK for iOS

69. Lex AWS mobile SDK for iOS

70. Lex AWS mobile SDK for iOS etc.

71. Lex AWS mobile SDK for iOS • AWSLexInteractionDelegate • AWSLexAudioPlayerDelegate • AWSLexMicrophoneDelegate

72. Lex AWS mobile SDK for iOS Fullfillment switchMode InteractionDelegate

73. Lex AWS mobile SDK for iOS started finished AudioPlayerDelegate

74. Lex AWS mobile SDK for iOS start end SoundLevelChaned MicrophoneDelegate

75. Lex • Endpoints/API •AWS mobile SDK •Authorization •SessionAttribute

76. Lex Authorization

77. Lex Authorization

78. Lex • Endpoints/API •AWS mobile SDK •Authorization •SessionAttribute

79. Lex SessionAttribute

80. Lex SessionAttribute

81. Lex SessionAttribute

82. Lex SessionAttribute { "currentIntent": { "name": "intent-name", "slots": {"slot-name": "value",}, "confirmationStatus": "None, Confirmed, or Denied (intent confirmation, if configured)", }, "bot": { "name": "bot-name", "alias": "bot-alias", "version": "bot-version" }, "userId": "User ID specified in the POST request to Amazon Lex.", "inputTranscript": "Text used to process the request", "invocationSource": "FulfillmentCodeHook or DialogCodeHook", "outputDialogMode": "Text or Voice, based on ContentType request header in runtime API request", "sessionAttributes": { "key1": "value1", "key2": "value2" } } Lambda Input sessionAttribute http://docs.aws.amazon.com/ja_jp/lex/latest/dg/lambda-input-response-format.html

83. Lex SessionAttribute { "sessionAttributes": { "key1": "value1", "key2": "value2" }, "dialogAction": { "type": "Close", "fulfillmentState": "Fulfilled or Failed", "message": { "contentType": "PlainText or SSML", "content": "Message to convey to the user. For example, Thanks, your pizza has been ordered." }, "responseCard": { "version": integer-value, "contentType": "application/vnd.amazonaws.card.generic", "genericAttachments": [ { "title":"card-title", "subTitle":"card-sub-title", "imageUrl":"URL of the image to be shown", Lambda Response sessionAttribute http://docs.aws.amazon.com/ja_jp/lex/latest/dg/lambda-input-response-format.html

84. Lex SessionAttribute POST /bot/botName/alias/botAlias/user/userId/text HTTP/1.1 Content-type: application/json { "inputText": "string", "sessionAttributes": { "string" : "string" } } Client POST sessionAttribute http://docs.aws.amazon.com/ja_jp/lex/latest/dg/API_runtime_PostText.html

85. Lex SessionAttribute HTTP/1.1 200 Content-type: application/json { "dialogState": "string", "intentName": "string", "message": "string", "responseCard": { "contentType": "string", "version": "string" }, "sessionAttributes": { "string" : "string" }, "slots": { "string" : "string" }, "slotToElicit": "string" } Client Response sessionAttribute http://docs.aws.amazon.com/ja_jp/lex/latest/dg/API_runtime_PostText.html

86. Lex SessionAttribute "sessionAttributes": { "cards": [ { "title":"Mocha", "subtitle":"Please enjoy the fragrant mocha", "body":"$4.15", "imageUrl":"https://exsample.com/mocha.png", "slotName":"CoffeeType", "slotValue":"Mocha" }, タップされた場合は、 Textで送る

87. Polly

88. Polly アーキテクチャー https://www.slideshare.net/AmazonWebServices/amazon-polly?qid=c9c8e2c4-1c27-41c1-bc5f- 9b1e0c651c21&v=&b=&from_search=11

89. Polly アーキテクチャー https://www.slideshare.net/AmazonWebServices/amazon-polly?qid=c9c8e2c4-1c27-41c1-bc5f- 9b1e0c651c21&v=&b=&from_search=11

90. Polly •AWS mobile SDK •Authorization

91. Polly AWS mobile SMD source 'https://github.com/CocoaPods/Specs.git’ target 'BotSampleApp' do platform :ios, '9.0’ use_frameworks! pod 'AWSPolly’ pod 'AWSCognito’ end

92. Polly AWS mobile SMD let input = AWSPollySynthesizeSpeechURLBuilderRequest() input.text = textView.text input.outputFormat = AWSPollyOutputFormat.mp3 input.voiceId = AWSPollyVoiceId.joanna let builder = AWSPollySynthesizeSpeechURLBuilder.default().getPreSignedURL(input) builder.continueOnSuccessWith { (awsTask: AWSTask<NSURL>) -> Any? in let url = awsTask.result! self.audioPlayer.replaceCurrentItem(with: AVPlayerItem(url: url as URL)) self.audioPlayer.play() return nil } http://dev.classmethod.jp/smartphone/amazon-polly-sdk/ AWSPollySynthesizeSpeechURLBuilderRequest テキスト 声の種類 AVFoundation.AudioPlayer

93. Polly •AWS mobile SDK •Authorization

94. Polly Authorization

95. Polly Authorization

96. Polly AWS mobile SMD ⾔語24 声の種類

97. Summary

98. Summary Skill Skill Skill Skill Intent, Slots Lambda Skill Intent, Slots, SessionAttributes Intent, Slots, SessionAttributes

99. Summary 1.Amazonの各種⾳声サービスの概要 2.⾳声クライアントの実装⽅法 3.要件に応じた選択

100. これからもどうぞ宜しくお願い申し上げます。