本文為「云端智能剪輯」實(shí)踐指南第五期,通過阿里云智能媒體服務(wù)IMS完成數(shù)字人形象訓(xùn)練、人聲克隆定制,并使用Timeline實(shí)現(xiàn)視頻合成及創(chuàng)作,打造一個(gè)“聲形俱佳”的數(shù)字分身。
橙鯉|作者
形象日益逼真、交互更為順暢,虛擬數(shù)字人在視頻內(nèi)容生產(chǎn)環(huán)節(jié)的應(yīng)用越來越多。復(fù)刻一個(gè)不受空間、時(shí)間的限制,具有專屬聲音、形象的“我”,竟如此簡單?
我們先來看一則運(yùn)用數(shù)字人形象定制及人聲克隆后合成的成片效果:https://v.youku.com/v_show/id_XNjI3Mzk1NTM2NA==.html
示例視頻中的主播形象、人物語音,并非是通過拍攝剪輯形成的,而是通過數(shù)字人形象訓(xùn)練和人聲克隆,生成對應(yīng)的形象與聲音模型,并調(diào)用智能媒體服務(wù)的視頻剪輯合成接口生成的最終成片。
本文,我們將逐步介紹數(shù)字人形象訓(xùn)練及人聲克隆的步驟過程,并附以詳細(xì)的剪輯合成Timeline代碼示例,三步在數(shù)字世界復(fù)刻一個(gè)“我”。
首先,我們需要訓(xùn)練出一個(gè)數(shù)字分身,無論是動作、表情、還是口型等,都能夠達(dá)到高還原度的擬真效果。
「數(shù)字人形象定制」主要包含準(zhǔn)備訓(xùn)練素材、提交訓(xùn)練、生成AvatarId三個(gè)步驟。
l 準(zhǔn)備訓(xùn)練素材
1個(gè)訓(xùn)練視頻(Video)和1張頭像圖片(Portrait)
設(shè)備選擇、場地要求、模特形象、錄制過程等具體拍攝指南,可參考:
https://help.aliyun.com/document_detail/2393251.html
l 提交訓(xùn)練
通過控制臺進(jìn)行提交訓(xùn)練,依據(jù)提示,填入數(shù)字人名稱、描述、頭像圖片與訓(xùn)練視頻素材等,點(diǎn)擊「開始定制」,即可開始訓(xùn)練。
控制臺鏈接:
https://ice.console.aliyun.com/production/avatar2d/create
其中,數(shù)字人形象支持傳入帶透明通道(無背景、已摳圖)與不帶透明通道(有背景、未摳圖)兩種不同類型的素材,可以按需傳入。
l 生成AvatarId
當(dāng)訓(xùn)練完成后,您將獲得模型的唯一標(biāo)識 AvatarId,可以使用該ID合成數(shù)字人視頻。
同時(shí),通過點(diǎn)擊「數(shù)字人」列表頁中的「視頻剪輯」按鈕,使用數(shù)字人形象生成視頻,并進(jìn)行二次剪輯創(chuàng)作。
您可以選擇是由「文字驅(qū)動」或是「音頻驅(qū)動」合成數(shù)字人視頻。
以「文字驅(qū)動」為例:輸入文本內(nèi)容,選擇官方人聲效果(亦可選擇通過人聲克隆生成的聲音模型,詳見第二步)即可生成數(shù)字人視頻。
數(shù)字人視頻合成效果如下:https://v.youku.com/v_show/id_XNjI3Mzk1NTc0MA==.html
當(dāng)有了高度擬真的數(shù)字人形象,我們還需要同步擬真的聲音,進(jìn)一步提升“數(shù)字分身”的生動感、完整度。
「人聲克隆」主要包含準(zhǔn)備訓(xùn)練素材、提交訓(xùn)練、生成VoiceId三個(gè)步驟。
l 準(zhǔn)備訓(xùn)練素材
若干訓(xùn)練音頻(Audio)和 1 個(gè)安全認(rèn)證音頻(Authentication)
l 提交訓(xùn)練
通過控制臺提交訓(xùn)練,填入聲音名稱、安全認(rèn)證音頻、訓(xùn)練音頻等,點(diǎn)擊「開始定制」,即可開始訓(xùn)練。
控制臺鏈接:
https://ice.console.aliyun.com/production/voiceCopy/createStandard
l 生成VoiceId
訓(xùn)練完成后,您將獲得模型的唯一標(biāo)識 VoiceId,可以通過點(diǎn)擊「人聲克隆 - 大眾版」列表頁中的「語音合成」按鈕,來使用當(dāng)前聲音模型進(jìn)行語音合成。
人聲克隆-大眾版列表頁
輸入文字內(nèi)容,點(diǎn)擊「開始試聽」按鈕,即可提交智能語音合成。
運(yùn)用人生克隆語音合成后效果如下:語音合成示例音頻 1
我們也可以與訓(xùn)練時(shí)傳入的原始音頻進(jìn)行效果對比:原始訓(xùn)練聲音(截取)
在獲得數(shù)字人形象模型的 AvatarId 與專屬人聲的 VoiceId 后,除了通過文章前述的控制臺來進(jìn)行提交合成,還可以通過時(shí)間線Timeline一次提交生成,完成一鍵“復(fù)刻”。
Timeline示例如下:
{
"VideoTracks": [{
"VideoTrackClips": [{
"Type": "GlobalImage",
"MediaURL": "https://your-bucket.oss-cn-shanghai.aliyuncs.com/image1.jpg",
"Height": 1920,
"Width": 1080
}]
}, {
"VideoTrackClips": [{
"Type": "Image",
"MediaURL": "https://your-bucket.oss-cn-shanghai.aliyuncs.com/image2.png",
"Height": 1920,
"Width": 1080,
"Duration": 2,
"Effects": [{
"Type": "Text",
"Content": "什么是數(shù)字人和人聲克隆?",
"Alignment": "CenterCenter",
"FontSize": 80,
"EffectColorStyle": "CS0001-000001"
}]
},{
"Type": "AI_Avatar",
"AvatarId": "Avatar-******",
"CustomizedVoice": "Voice-******",
"Content": "數(shù)字人和人聲克隆是兩個(gè)前沿的技術(shù)概念\n它們代表著數(shù)字技術(shù)在模擬人類外觀和聲音方面的最新成就",
"ClipId": "avatar1",
"Effects": [{
"Type": "AI_ASR",
"FontSize": 60,
"Alignment": "TopCenter",
"Y": 1670,
"EffectColorStyle": "CS0001-000007",
"AdaptMode": "AutoWrap",
"TextWidth": 0.8
}]
},{
"Type": "Image",
"MediaURL": "https://your-bucket.oss-cn-shanghai.aliyuncs.com/image2.png",
"Height": 1920,
"Width": 1080,
"Duration": 2,
"Effects": [{
"Type": "Text",
"Content": "數(shù)字人",
"Alignment": "CenterCenter",
"FontSize": 150,
"EffectColorStyle": "CS0001-000001"
}]
}, {
"Type": "AI_Avatar",
"AvatarId": "Avatar-******",
"CustomizedVoice": "Voice-******",
"Content": "數(shù)字人\n通常指由計(jì)算機(jī)生成的虛擬人類形象\n這些形象可以是二維(2D)或三維(3D)的\n具有與現(xiàn)實(shí)人類相似的外觀、動作和行為\n隨著圖形渲染技術(shù)的進(jìn)步\n數(shù)字人越來越能夠以高度逼真的方式呈現(xiàn)\n包括復(fù)雜的面部表情、肢體動作\n并能在虛擬環(huán)境中以自然的方式行動",
"ClipId": "avatar2",
"Effects": [{
"Type": "AI_ASR",
"FontSize": 60,
"Alignment": "TopCenter",
"Y": 1670,
"EffectColorStyle": "CS0001-000007",
"AdaptMode": "AutoWrap",
"TextWidth": 0.8
}]
}, {
"Type": "Image",
"MediaURL": "https://your-bucket.oss-cn-shanghai.aliyuncs.com/image2.png",
"Height": 1920,
"Width": 1080,
"Duration": 2,
"Effects": [{
"Type": "Text",
"Content": "數(shù)字人的應(yīng)用",
"Alignment": "CenterCenter",
"FontSize": 120,
"EffectColorStyle": "CS0001-000001"
}]
},{
"Type": "AI_Avatar",
"AvatarId": "Avatar-******",
"CustomizedVoice": "Voice-******",
"Content": "數(shù)字人的應(yīng)用范圍極其廣泛",
"ClipId": "avatar3",
"Effects": [{
"Type": "AI_ASR",
"FontSize": 60,
"Alignment": "TopCenter",
"Y": 1670,
"EffectColorStyle": "CS0001-000007",
"AdaptMode": "AutoWrap",
"TextWidth": 0.8
}]
}, {
"Type": "AI_Avatar",
"AvatarId": "Avatar-******",
"CustomizedVoice": "Voice-******",
"Content": "從電子游戲、電影和電視制作",
"ClipId": "avatar4",
"Effects": [{
"Type": "AI_ASR",
"FontSize": 60,
"Alignment": "TopCenter",
"Y": 1670,
"EffectColorStyle": "CS0001-000007",
"AdaptMode": "AutoWrap",
"TextWidth": 0.8
}]
}, {
"Type": "AI_Avatar",
"AvatarId": "Avatar-******",
"CustomizedVoice": "Voice-******",
"Content": "到增強(qiáng)現(xiàn)實(shí)(AR)和虛擬現(xiàn)實(shí)(VR)體驗(yàn)等",
"ClipId": "avatar5",
"Effects": [{
"Type": "AI_ASR",
"FontSize": 60,
"Alignment": "TopCenter",
"Y": 1670,
"EffectColorStyle": "CS0001-000007",
"AdaptMode": "AutoWrap",
"TextWidth": 0.8
}]
}, {
"Type": "AI_Avatar",
"AvatarId": "Avatar-******",
"CustomizedVoice": "Voice-******",
"Content": "到在線教育、模擬訓(xùn)練、客戶服務(wù)和健康護(hù)理",
"ClipId": "avatar6",
"Effects": [{
"Type": "AI_ASR",
"FontSize": 60,
"Alignment": "TopCenter",
"Y": 1670,
"EffectColorStyle": "CS0001-000007",
"AdaptMode": "AutoWrap",
"TextWidth": 0.8
}]
}, {
"Type": "AI_Avatar",
"AvatarId": "Avatar-******",
"CustomizedVoice": "Voice-******",
"Content": "在這些應(yīng)用中\(zhòng)n數(shù)字人可以作為用戶的虛擬代表\n或者作為虛擬助手和顧問\n提供幫助和咨詢\n在娛樂產(chǎn)業(yè)中",
"ClipId": "avatar7",
"Effects": [{
"Type": "AI_ASR",
"FontSize": 60,
"Alignment": "TopCenter",
"Y": 1670,
"EffectColorStyle": "CS0001-000007",
"AdaptMode": "AutoWrap",
"TextWidth": 0.8
}]
}, {
"Type": "AI_Avatar",
"AvatarId": "Avatar-******",
"CustomizedVoice": "Voice-******",
"Content": "數(shù)字人可以被用來創(chuàng)造虛構(gòu)角色\n甚至在某些情況下代替真實(shí)的演員進(jìn)行表演",
"ClipId": "avatar8",
"Effects": [{
"Type": "AI_ASR",
"FontSize": 60,
"Alignment": "TopCenter",
"Y": 1670,
"EffectColorStyle": "CS0001-000007",
"AdaptMode": "AutoWrap",
"TextWidth": 0.8
}]
}, {
"Type": "AI_Avatar",
"AvatarId": "Avatar-******",
"CustomizedVoice": "Voice-******",
"Content": "而在商業(yè)領(lǐng)域\n數(shù)字人可以用作品牌大使或虛擬員工\n增加用戶互動的吸引力",
"ClipId": "avatar9",
"Effects": [{
"Type": "AI_ASR",
"FontSize": 60,
"Alignment": "TopCenter",
"Y": 1670,
"EffectColorStyle": "CS0001-000007",
"AdaptMode": "AutoWrap",
"TextWidth": 0.8
}]
}, {
"Type": "AI_Avatar",
"AvatarId": "Avatar-******",
"CustomizedVoice": "Voice-******",
"Content": "在教育和訓(xùn)練場景中\(zhòng)n數(shù)字人能夠模擬不同的情境\n提供更加豐富和互動的學(xué)習(xí)體驗(yàn)",
"ClipId": "avatar10",
"Effects": [{
"Type": "AI_ASR",
"FontSize": 60,
"Alignment": "TopCenter",
"Y": 1670,
"EffectColorStyle": "CS0001-000007",
"AdaptMode": "AutoWrap",
"TextWidth": 0.8
}]
},{
"Type": "Image",
"MediaURL": "https://your-bucket.oss-cn-shanghai.aliyuncs.com/image2.png",
"Height": 1920,
"Width": 1080,
"Duration": 2,
"Effects": [{
"Type": "Text",
"Content": "人聲克隆",
"Alignment": "CenterCenter",
"FontSize": 150,
"EffectColorStyle": "CS0001-000001"
}]
}, {
"Type": "AI_Avatar",
"AvatarId": "Avatar-******",
"CustomizedVoice": "Voice-******",
"Content": "人聲克隆是指使用計(jì)算機(jī)算法模仿特定人的聲音\n這項(xiàng)技術(shù)通過分析一個(gè)人的語音記錄\n捕捉其獨(dú)特的聲音特征\n如音高、音色、語速和口音\n并創(chuàng)建一個(gè)可以產(chǎn)生類似聲音的模型\n人聲克隆技術(shù)往往建立在深度學(xué)習(xí)和神經(jīng)網(wǎng)絡(luò)的基礎(chǔ)上\n通過大量的聲音訓(xùn)練數(shù)據(jù)\n使得合成的聲音越來越難以與原聲區(qū)分",
"ClipId": "avatar11",
"Effects": [{
"Type": "AI_ASR",
"FontSize": 60,
"Alignment": "TopCenter",
"Y": 1670,
"EffectColorStyle": "CS0001-000007",
"AdaptMode": "AutoWrap",
"TextWidth": 0.8
}]
}, {
"Type": "Image",
"MediaURL": "https://your-bucket.oss-cn-shanghai.aliyuncs.com/image2.png",
"Height": 1920,
"Width": 1080,
"Duration": 2,
"Effects": [{
"Type": "Text",
"Content": "人聲克隆的應(yīng)用",
"Alignment": "CenterCenter",
"FontSize": 120,
"EffectColorStyle": "CS0001-000001"
}]
}, {
"Type": "AI_Avatar",
"AvatarId": "Avatar-******",
"CustomizedVoice": "Voice-******",
"Content": "人聲克隆可以用在多個(gè)方面",
"ClipId": "avatar12",
"Effects": [{
"Type": "AI_ASR",
"FontSize": 60,
"Alignment": "TopCenter",
"Y": 1670,
"EffectColorStyle": "CS0001-000007",
"AdaptMode": "AutoWrap",
"TextWidth": 0.8
}]
}, {
"Type": "AI_Avatar",
"AvatarId": "Avatar-******",
"CustomizedVoice": "Voice-******",
"Content": "例如為無法親自錄音的藝術(shù)家復(fù)原聲音",
"ClipId": "avatar13",
"Effects": [{
"Type": "AI_ASR",
"FontSize": 60,
"Alignment": "TopCenter",
"Y": 1670,
"EffectColorStyle": "CS0001-000007",
"AdaptMode": "AutoWrap",
"TextWidth": 0.8
}]
}, {
"Type": "AI_Avatar",
"AvatarId": "Avatar-******",
"CustomizedVoice": "Voice-******",
"Content": "或?yàn)檎Z音輔助設(shè)備提供更自然的語音輸出",
"ClipId": "avatar14",
"Effects": [{
"Type": "AI_ASR",
"FontSize": 60,
"Alignment": "TopCenter",
"Y": 1670,
"EffectColorStyle": "CS0001-000007",
"AdaptMode": "AutoWrap",
"TextWidth": 0.8
}]
}, {
"Type": "AI_Avatar",
"AvatarId": "Avatar-******",
"CustomizedVoice": "Voice-******",
"Content": "它也可以幫助有語音障礙的人恢復(fù)他們的聲音",
"ClipId": "avatar15",
"Effects": [{
"Type": "AI_ASR",
"FontSize": 60,
"Alignment": "TopCenter",
"Y": 1670,
"EffectColorStyle": "CS0001-000007",
"AdaptMode": "AutoWrap",
"TextWidth": 0.8
}]
}, {
"Type": "AI_Avatar",
"AvatarId": "Avatar-******",
"CustomizedVoice": "Voice-******",
"Content": "或用于個(gè)性化的語音合成服務(wù)\n如定制語音導(dǎo)航或個(gè)人助理",
"ClipId": "avatar16",
"Effects": [{
"Type": "AI_ASR",
"FontSize": 60,
"Alignment": "TopCenter",
"Y": 1670,
"EffectColorStyle": "CS0001-000007",
"AdaptMode": "AutoWrap",
"TextWidth": 0.8
}]
}, {
"Type": "AI_Avatar",
"AvatarId": "Avatar-******",
"CustomizedVoice": "Voice-******",
"Content": "人聲克隆技術(shù)可以為數(shù)字人提供聲線\n使得虛擬角色不僅在視覺上\n還在聽覺上都顯得栩栩如生\n具有高度自然語音的數(shù)字人能夠提供更加動態(tài)和親切的交互體驗(yàn)\n從而在各種虛擬場景中擔(dān)任重要角色",
"ClipId": "avatar17",
"Effects": [{
"Type": "AI_ASR",
"FontSize": 60,
"Alignment": "TopCenter",
"Y": 1670,
"EffectColorStyle": "CS0001-000007",
"AdaptMode": "AutoWrap",
"TextWidth": 0.8
}]
},{
"Type": "Image",
"MediaURL": "https://your-bucket.oss-cn-shanghai.aliyuncs.com/image2.png",
"Height": 1920,
"Width": 1080,
"Duration": 2,
"Effects": [{
"Type": "Text",
"Content": "總結(jié)",
"Alignment": "CenterCenter",
"FontSize": 150,
"EffectColorStyle": "CS0001-000001"
}]
}, {
"Type": "AI_Avatar",
"AvatarId": "Avatar-******",
"CustomizedVoice": "Voice-******",
"Content": "結(jié)合數(shù)字人和人聲克隆技術(shù)\n我們可以創(chuàng)造出能夠在屏幕上以二維形象出現(xiàn)并以逼真的人聲進(jìn)行溝通的虛擬代表\n這種結(jié)合提供了豐富的用戶體驗(yàn)\n并在教育、娛樂、客戶服務(wù)等各種場景中擁有潛在的應(yīng)用價(jià)值",
"ClipId": "avatar18",
"Effects": [{
"Type": "AI_ASR",
"FontSize": 60,
"Alignment": "TopCenter",
"Y": 1670,
"EffectColorStyle": "CS0001-000007",
"AdaptMode": "AutoWrap",
"TextWidth": 0.8
}]
}]
},{
"VideoTrackClips": [{
"Type": "Image",
"MediaURL": "https://your-bucket.oss-cn-shanghai.aliyuncs.com/image3.png",
"ReferenceClipId": "avatar4"
},{
"Type": "Image",
"MediaURL": "https://your-bucket.oss-cn-shanghai.aliyuncs.com/image4.png",
"ReferenceClipId": "avatar5"
},{
"Type": "Image",
"MediaURL": "https://your-bucket.oss-cn-shanghai.aliyuncs.com/image5.png",
"ReferenceClipId": "avatar6"
},{
"Type": "Image",
"MediaURL": "https://your-bucket.oss-cn-shanghai.aliyuncs.com/image6.png",
"ReferenceClipId": "avatar8"
},{
"Type": "Image",
"MediaURL": "https://your-bucket.oss-cn-shanghai.aliyuncs.com/image7.png",
"ReferenceClipId": "avatar9"
},{
"Type": "Image",
"MediaURL": "https://your-bucket.oss-cn-shanghai.aliyuncs.com/image8.png",
"ReferenceClipId": "avatar10"
},{
"Type": "Image",
"MediaURL": "https://your-bucket.oss-cn-shanghai.aliyuncs.com/image9.png",
"ReferenceClipId": "avatar13"
},{
"Type": "Image",
"MediaURL": "https://your-bucket.oss-cn-shanghai.aliyuncs.com/image10.png",
"ReferenceClipId": "avatar14"
},{
"Type": "Image",
"MediaURL": "https://your-bucket.oss-cn-shanghai.aliyuncs.com/image11.png",
"ReferenceClipId": "avatar15"
},{
"Type": "Image",
"MediaURL": "https://your-bucket.oss-cn-shanghai.aliyuncs.com/image12.png",
"ReferenceClipId": "avatar16"
}]
},{
"VideoTrackClips": [{
"Type": "GlobalImage",
"MediaURL": "https://your-bucket.oss-cn-shanghai.aliyuncs.com/image13.png",
"Width": 312,
"Height": 72,
"X": 0,
"Y": 1848
}]
}],
"AudioTracks": [{
"AudioTrackClips": [{
"MediaURL": "https://your-bucket.oss-cn-shanghai.aliyuncs.com/audio1.wav",
"LoopMode": true,
"Effects": [{
"Type": "Volume",
"Gain": "0.2"
}]
}]
}]
}
各API文檔鏈接:
1.SubmitAvatarVideoJob - 提交數(shù)字人視頻合成任務(wù)
2.SubmitAudioProduceJob - 提交智能語音任務(wù)
3.SubmitMediaProducingJob - 提交剪輯合成任務(wù)
值得一提的是,本示例中用到的口播文案、圖片、音樂等都完全是由各類生成模型生成的,也就是說,該視頻的輸入只有一段話:“請介紹一下數(shù)字人與人聲克隆技術(shù)”。
敬請期待云端智能剪輯即將推出的「一鍵成片」功能,僅需要輸入一個(gè)詞、一段話,和極其簡單的配置,即可生成一段高質(zhì)量的視頻。
智能媒體服務(wù)IMS「云端智能剪輯」是基于云計(jì)算和人工智能技術(shù)的視頻剪輯生產(chǎn)服務(wù),能為用戶提供直播剪輯、視頻剪輯、模版工廠、數(shù)字人制作等核心功能,并可使用 AI 輔助剪輯生產(chǎn)。該產(chǎn)品可廣泛應(yīng)用于互聯(lián)網(wǎng)、文化傳媒、廣告營銷、教育金融等行業(yè),滿足企業(yè)進(jìn)行規(guī)模、高效、便捷、智能的視頻內(nèi)容生產(chǎn)需求。
歡迎加入官方答疑釘釘群咨詢交流:48335001108
