H5开发的视频语音识别分析播放记录
·
需求:
实现一个H5网页,在手机浏览器上一边录视频一边语音录制提问发给模型端,模型端解析的结果两种输出方式,一种解析完毕再返回音频数据,wav格式,第二种是流式传输,一边解析一边传输。目前模型端已经实现。
需求分析:
录音的同时并开始视频,因为视频和录音要调取对应的接口,文件也要尽可能的小,接口限制要不然太大的话传输耗时多。
开始:
H5网页的话获取视频和录音权限的话用到navigator.mediaDevices.getUserMedia,但是因为音频和视频是分开的,这里视频就用到此API,然后音频就换了一个recorder-core插件来控制,
视频流的开启和音频的开始这一步在初始化中就完成,这样在后续使用中不用频繁的操作
里面操控录制视频的分辨率和文件大小的配置真是油盐不进啊!!!
onMounted(() => {
videoType.value = MediaRecorder.isTypeSupported('video/mp4') ? 'mp4' : 'webm';
alert('浏览器允许生成视频格式为:'+videoType.value)
if (navigator.mediaDevices || navigator.getUserMedia) {
//{ facingMode: {exact: "environment"} }
getUserMedia({
video: {
facingMode: { exact: "environment" },
width: { min: 480, ideal: 480, max: 480, exact:480 },
height: { min: 640, ideal: 640, max: 640, exact: 640 },
// width: 480,
// height: 640,
}, audio: false}); // 调用用户媒体设备,访问摄像头、录音
// getUserMedias({audio: true },{videoBitsPerSecond:16000,channelCount:1})
rec.value = Recorder({
type:"wav" //录音格式,可以换成wav等其他格式
,sampleRate:16000 //录音的采样率,越大细节越丰富越细腻
,bitRate:16 //录音的比特率,越大音质越好
,log:false //是否在控制台输出日志
});
// return
rec.value.open(
function (e) {
// e.preventDefault();
//视频流
},
function (msg:string, isUserNotAllow: any) {
// 浏览器不支持录音、用户拒绝麦克风权限、或者非安全环境(非https、file等
console.log((isUserNotAllow ? 'UserNotAllow,' : '') + '无法录音:' + msg)
// fn(“打开失败”)
}
)
} else {
alert("你的浏览器不支持访问用户媒体设备");
}
})
const getUserMedia = (constrains: {video: any, audio: any}) => {
if (navigator.mediaDevices.getUserMedia) {
// 最新标准API
navigator.mediaDevices.getUserMedia(constrains).then(stream => { success(stream); }).catch(err => { error(err); });
}
// else if (navigator.webkitGetUserMedia) {
// // webkit内核浏览器
// navigator.webkitGetUserMedia(constrains).then((stream:any) => { success(stream); }).catch((err:any) => { error(err); });
// } else if (navigator.mozGetUserMedia) {
// // Firefox浏览器
// navigator.mozGetUserMedia(constrains).then((stream:any) => { success(stream); }).catch((err:any) => { error(err); });
// }
else if (navigator.getUserMedia) {
// 旧版API
navigator.getUserMedia(constrains).then((stream:any) => { success(stream); }).catch((err:any) => { error(err); });
}
}
const success = (stream: any) => {
nextTick(() => {
if (video.value) {
video.value.srcObject = stream;
// hmFn()
streamValue.value = stream
video.value.onloadedmetadata = () => {
requestAnimationFrame(fm)
}
// fm()
}
})
}
长按视频流式播放
流式输出遇到很多问题,视频大小分辨率输出ios和鸿蒙兼容问题,分段播放的接受到的是wav文件流转到base64,一开始给每个分段都加了完整的wav头部信息,后来发现分段太小,都不够一个字符读音,最后网上给的灵感,多段合并播放,100段的流,每5段或者10段为一组,经过测试发现可以实现,但是就是毛边撕拉撕拉的杂音,完整拼接也有这个问题。那就先这样吧。模型端也没时间。算了,上完整代码吧还是,。
app.vue
<script setup lang="ts">
import { ref, nextTick, onMounted } from 'vue'
//必须引入的核心
import Recorder from 'recorder-core'
import { asr, qwen2, tts, upload, uploadAudio } from './api/index'
import { blobToBase64, str24, base64ToArrayBuffer } from './utils/blobToBase64'
import './utils/base64.js'
import Loading from './components/loading.vue'
//引入mp3格式支持文件;如果需要多个格式支持,把这些格式的编码引擎js文件放到后面统统引入进来即可
// import 'recorder-core/src/engine/mp3'
// import 'recorder-core/src/engine/mp3-engine'
//录制wav格式的用这一句就行
import 'recorder-core/src/engine/wav'
import { fenduan } from './hooks/useFenduan'
import { useNoFenduan } from './hooks/useNoFenduan'
import { useAudio } from './hooks/useAudio'
//可选的插件支持项,这个是波形可视化插件
// import 'recorder-core/src/extensions/waveview'
import { saveAs } from "file-saver";
const checkeds = ref([])
const isDialog = ref(false)
const isLoadFfmpegCore = ref(false)
const video: any = ref(null)
const rec = ref()
const mediaRecorder = ref()
const videoBlobs = ref([])
const asrResult = ref('')
const audioContent = ref('')
const isRecording = ref(false)
const audio = ref()
const recordContent = ref('')
const type = ref('视频')
const type1 = ref('流式')
const isCanera = ref(false)
const wrapperRef = ref()
const isShow = ref(false)
const imgContent = ref('')
const canvas1 = ref()
// const bufferQueue = ref([])
// const playing = ref(false);
// const chunkCounter = ref(0);
// const receivedChunks = ref([]);
// const audioContext = ref()
// const wavHeader = ref()
const canvas = ref()
const timeArr = ref({
t1: '',
t2: '',
t3: '',
t4: '',
t5: ''
})
const videoType = ref('')
const reset = () => {
timeArr.value = {
t1: '',
t2: '',
t3: '',
t4: '',
t5: ''
}
isRecording.value = false
isShow.value = false
isDialog.value = false
if (type.value == '图片') isCanera.value = true
}
const ttsFn = (str) => {
let t5 = new Date().getTime()
tts({ text: str }).then((res: any) => {
timeArr.value.t5 = '语音播报接口耗时:' + (new Date().getTime() - t5)
if (res.code == 200) {
audioContent.value = 'data:audio/mp3;base64,' + res.data
audio.value.src = audioContent.value
audio.value.play()
isRecording.value = true
audio.value.addEventListener('ended', function () {
reset()
console.log("语音播报结束")
}, false);
}
if(type.value == '图片') isCanera.value = true
}).catch((err) => {
reset()
alert(err)
})
}
const uploadAll = (res, mediaType?: string, time?: any) => {
if(time)timeArr.value.t2 = '前端获取base64到上传数据之前的耗时:' + (new Date().getTime() - time)
let t3 = new Date().getTime()
upload({audioFile:recordContent.value, videoFile:res, fileType: type.value == '图片' ? 'image' : 'video', mediaType}).then((res3: any) => {
if (res3.code == 200) {
timeArr.value.t3 = '上传耗时:' + (new Date().getTime() - t3)
let t4 = new Date().getTime()
useAudio(res3.data, reset, (t) => timeArr.value.t4 = '前端获取数据到开始播放的耗时:' + (new Date().getTime() - t4) )
// audio.value.src = 'data:audio/wav;base64,' + res3.data
// audio.value.play()
// timeArr.value.t4 = '接收数据到开始播放的耗时:' + (new Date().getTime() - t4)
// audio.value.addEventListener('ended', function () {
// reset()
// console.log("语音播报结束")
// }, false);
} else {
reset()
alert(error)
}
// audio.value.src = URL.createObjectURL(res3) //Blob流播放方式
}).catch(error => {
reset()
alert(error)
}).finally(() => {
timeArr.value.t3 = '上传接口耗时:' + (new Date().getTime() - t3)
})
}
const videoArr: any = ref([])
const sseFn = async (res, mediaType?: string, time?: any) => {
if(time)timeArr.value.t2 = '前端获取base64到上传数据之前的耗时:' + (new Date().getTime() - time)
const uid = str24()
const eventSource = await new EventSource('https://aidemo.ysh360.top' + '/sse/createSse?uid=' + uid);
videoArr.value = []
let t3 = new Date().getTime()
uploadAudio({ audioFile: recordContent.value, videoFile: res, uid, fileType: type.value == '图片' ? 'image' : 'video', mediaType }).then(response => {
timeArr.value.t3 = '上传耗时:' + (new Date().getTime() - t3)
// audio.value.src = 'data:audio/wav;base64,' + response.data
// audio.value.play()
// audio.value.addEventListener('ended', function () {
// reset()
// console.log("语音播报结束")
// }, false);
if (response.code == 500 || response.code == 502) {
reset()
}
}).catch(err => {
reset()
// alert(err)
})
;
fenduan(
eventSource,
reset,
(t) => timeArr.value.t3 = '开始获取分段耗时:' + (t - t3),
// (t) => timeArr.value.t4 = '前端获取分段耗时:' + (t - t3)
)
// useNoFenduan(eventSource, reset)
}
const blobToBase64Fn = (superBuffer: any, time: any) => {
blobToBase64(superBuffer).then((res: any) => {
if (type1.value == '流式') {
// alert('视频流式暂时无法使用')
// if (checkeds.value.includes('语音')) {
// ttsFn(asrResult.value)
// } else {
// }
sseFn(res.replace(`data:video/${videoType.value};base64,`, ''),videoType.value, time)
} else {
// qwenFn(res.replace('data:video/mp4;base64,', ''))
uploadAll(res.replace(`data:video/${videoType.value};base64,`, ''), videoType.value, time)
}
})
}
const streamValue = ref(null)
const hmFn = () => {
const isHarmonyOS = /HarmonyOS/i.test(navigator.userAgent);
if (isHarmonyOS) {
video.value.addEventListener('loadedmetadata', () => {
const aspectRatio = video.videoWidth / video.videoHeight;
if (window.innerWidth / window.innerHeight > aspectRatio) {
video.value.style.width = 'auto';
video.value.style.height = '100%';
} else {
video.value.style.width = '100%';
video.value.style.height = 'auto';
}
});
}
}
const lastFrameTime = ref(0)
const frameDuration = ref(24)
const fm = (timestamp) => {
if (!lastFrameTime) lastFrameTime.value = timestamp;
const elapsedTime = timestamp - lastFrameTime.value;
if (elapsedTime >= frameDuration.value) {
canvas1.value.width = 480;
canvas1.value.height = 640;
const canvasContent = canvas1.value.getContext('2d');
canvasContent.drawImage(video.value, 0, 0, 480, 640);
lastFrameTime.value = timestamp;
}
requestAnimationFrame(fm)
}
const success = (stream: any) => {
nextTick(() => {
if (video.value) {
video.value.srcObject = stream;
// hmFn()
streamValue.value = stream
video.value.onloadedmetadata = () => {
requestAnimationFrame(fm)
}
// fm()
}
})
}
const error = (error: any) => {
alert(`访问用户媒体设备失败,${error.name, error.message}`)
}
const getUserMedia = (constrains: {video: any, audio: any}) => {
if (navigator.mediaDevices.getUserMedia) {
// 最新标准API
navigator.mediaDevices.getUserMedia(constrains).then(stream => { success(stream); }).catch(err => { error(err); });
}
// else if (navigator.webkitGetUserMedia) {
// // webkit内核浏览器
// navigator.webkitGetUserMedia(constrains).then((stream:any) => { success(stream); }).catch((err:any) => { error(err); });
// } else if (navigator.mozGetUserMedia) {
// // Firefox浏览器
// navigator.mozGetUserMedia(constrains).then((stream:any) => { success(stream); }).catch((err:any) => { error(err); });
// }
else if (navigator.getUserMedia) {
// 旧版API
navigator.getUserMedia(constrains).then((stream:any) => { success(stream); }).catch((err:any) => { error(err); });
}
}
const recordFlag = ref(false)
const longPressTimer: any = ref(null)
const tatchFn = () => {
if (recordFlag.value) {
rec.value.stop((blob: Blob, duration: any) => {
// blob: 流文件,duration: 持续时间ms,这里可以执行文件播放或者文件上传
// rec.value.close()
recordFlag.value = false
isDialog.value = true
let t1 = new Date().getTime()
console.log(blob,'blob')
blobToBase64(blob).then((res: any) => {
// console.log(res,'res___________________res')
t1 = new Date().getTime() - t1
timeArr.value.t1 = '语音blob流转base64时间:' + t1
recordContent.value = res.replace('data:audio/wav;base64,', '')
let t2 = new Date().getTime()
if (type1.value == '整体') {
if (type.value == '语音') {
asr({audio_data:res.replace('data:audio/wav;base64,','')}).then((res: any) => {
asrResult.value = res.data
timeArr.value.t2 = '语音识别时间:' + (new Date().getTime() - t2)
ttsFn(asrResult.value)
timeArr.value.t3 = 'null'
timeArr.value.t4 = 'null'
}).catch((error) => {
recordFlag.value = false
isDialog.value = false
alert(error)
})
} else if (type.value == '图片') {
timeArr.value.t2 = 'null'
timeArr.value.t3 = 'null'
// qwenFn(imgContent.value.replace('data:image/png;base64,', ''))
uploadAll(imgContent.value.replace('data:image/png;base64,', ''),'png',t2)
}
} else {
if(type.value == '语音') {
asr({ audio_data: res.replace('data:audio/wav;base64,', '') }).then((res: any) => {
asrResult.value = res.data
timeArr.value.t2 = '语音识别时间:' + (new Date().getTime() - t2)
timeArr.value.t3 = 'null'
timeArr.value.t4 = 'null'
ttsFn(asrResult.value)
}).catch((error) => {
recordFlag.value = false
isDialog.value = false
alert(error)
})
} else if (type.value == '图片') {
timeArr.value.t2 = 'null'
timeArr.value.t3 = 'null'
sseFn(imgContent.value.replace('data:image/png;base64,', ''),'png',t2)
}
}
})
},
(s: any) => {
alert('结束出错:' + s)
rec.value.close()
recordFlag.value = false
isDialog.value = false
}
)
mediaRecorder.value.addEventListener('stop', () => {
const superBuffer = new Blob(videoBlobs.value, { type: `video/${videoType.value}` });
// saveAs(superBuffer, `video.${videoType.value}`);
//获取录制的视频流并调整分辨率
// alert('canvas1-width:' + canvas1.value.width +'canvas1-height:' +canvas1.value.height)
// video.value.src = url;
// video.value.onloadeddata = (e) => {
// canvasContent.drawImage(video.value, 0, 0, 480, 360);
// }
let t2 = new Date().getTime()
if (videoType.value == 'webm') {
timeArr.value.t3 = 'null'
blobToBase64Fn(superBuffer, t2)
} else {
timeArr.value.t3 = 'null'
blobToBase64Fn(superBuffer, t2)
}
});
console.log(type.value == '视频', 'type.value == 视频')
if(type.value == '视频')mediaRecorder.value.stop()
} else {
videoBlobs.value = []
// navigator.mediaDevices.getUserMedia({ audio: true }).then(res => {
mediaRecorder.value = new MediaRecorder(canvas1.value.captureStream(),
{
mimeType: `video/${videoType.value}`,
videoBitsPerSecond: 500000
});
if(type.value =='视频'){
mediaRecorder.value.ondataavailable = event => {
if (event.data && event.data.size > 0) {
videoBlobs.value.push(event.data);
}
};
mediaRecorder.value.start();//视频录制开始
}
rec.value.start()
recordFlag.value = true
// })
}
}
// const recFnCancel = () => {
// if (longPressTimer.value) {
// clearTimeout(longPressTimer.value);
// longPressTimer.value = null;
// tatchFn()
// }
// }
const recFn = (e) => {
// longPressTimer.value = setTimeout(function() {
// 执行长按要执行的操作
// if(type.value == '视频'&&type1.value == '流式') return alert('流式视频暂时关闭')
e.preventDefault()
tatchFn()
// }, 1000);
}
const cameraClick = () => {
if (!video.value) return;
isShow.value = true;
// 设置canvas画布
const canvas = document.createElement("canvas");
canvas.width = video.value.videoWidth;
canvas.height = video.value.videoHeight;
// 获取canvas上下文对象
const ctx = canvas.getContext("2d");
// 截图操作
console.log(canvas.height, canvas.width)
ctx?.drawImage(video.value, 0, 0, canvas.width, canvas.height);
// 展示截图
nextTick(() => {
// wrapperRef.value.appendChild(canvas);
imgContent.value = canvas.toDataURL('image/png')
wrapperRef.value.setAttribute("src", imgContent.value);
isCanera.value = false;
})
}
const change = (value) => {
type.value = value.target.value;
isCanera.value = type.value == '图片';
isShow.value = false;
}
const change1 = (value) => {
type1.value = value.target.value;
}
onMounted(() => {
videoType.value = MediaRecorder.isTypeSupported('video/mp4') ? 'mp4' : 'webm';
alert('浏览器允许生成视频格式为:'+videoType.value)
if (navigator.mediaDevices || navigator.getUserMedia) {
//{ facingMode: {exact: "environment"} }
getUserMedia({
video: {
facingMode: { exact: "environment" },
width: { min: 480, ideal: 480, max: 480, exact:480 },
height: { min: 640, ideal: 640, max: 640, exact: 640 },
// width: 480,
// height: 640,
}, audio: false}); // 调用用户媒体设备,访问摄像头、录音
// getUserMedias({audio: true },{videoBitsPerSecond:16000,channelCount:1})
rec.value = Recorder({
type:"wav" //录音格式,可以换成wav等其他格式
,sampleRate:16000 //录音的采样率,越大细节越丰富越细腻
,bitRate:16 //录音的比特率,越大音质越好
,log:false //是否在控制台输出日志
});
// return
rec.value.open(
function (e) {
// e.preventDefault();
//视频流
},
function (msg:string, isUserNotAllow: any) {
// 浏览器不支持录音、用户拒绝麦克风权限、或者非安全环境(非https、file等
console.log((isUserNotAllow ? 'UserNotAllow,' : '') + '无法录音:' + msg)
// fn(“打开失败”)
}
)
} else {
alert("你的浏览器不支持访问用户媒体设备");
}
})
</script>
useFendaun
import { ref } from "vue";
import { saveAs } from "file-saver";
export const fenduan = (
eventSource: any,
reset: Function,
callback: Function
) => {
const bufferQueue = ref([]);
const playing = ref(false);
const chunkCounter = ref(0);
const receivedChunks = ref([]);
const audioContext = ref();
const wavHeader = ref();
const num = ref(0);
const chunk_all = ref([]);
const base64ToUint8Array = (base64String) => {
const binaryString = atob(base64String);
const len = binaryString.length;
const bytes = new Uint8Array(len);
for (let i = 0; i < len; i++) {
bytes[i] = binaryString.charCodeAt(i);
}
return bytes;
};
const concatenateUint8Arrays = (arrays) => {
// console.log(arrays, "arrs________________Arrs");
let totalLength = arrays.reduce((sum, value) => sum + value.length, 0);
let result = new Uint8Array(totalLength);
let offset = 0;
for (let array of arrays) {
result.set(array, offset);
offset += array.length;
}
return result;
};
const processChunkGroup = () => {
const combinedChunks = concatenateUint8Arrays(receivedChunks.value);
const chunkWithHeader = concatenateUint8Arrays([
wavHeader.value,
combinedChunks,
]);
bufferQueue.value.push(chunkWithHeader);
receivedChunks.value = [];
};
const playStreamedAudio = async () => {
while (bufferQueue.value.length > 0) {
const chunk = bufferQueue.value.shift();
try {
const audioBuffer = await audioContext.value.decodeAudioData(
chunk.buffer
);
const source = audioContext.value.createBufferSource();
source.buffer = audioBuffer;
source.connect(audioContext.value.destination);
const playTime = audioContext.value.currentTime - 0.05;
source.start(playTime);
await new Promise((resolve) => (source.onended = resolve));
} catch (error) {
console.error("解码音频数据时出错:", error);
}
}
playing.value = bufferQueue.value.length > 0;
};
if (!audioContext.value) {
audioContext.value = new (window.AudioContext ||
window.webkitAudioContext)();
}
if (audioContext.value.state === "suspended") {
audioContext.value.resume();
}
eventSource.addEventListener("message", async (event) => {
// const chunk_one = atob(event.data)
// .split("")
// .map((c) => c.charCodeAt(0));
// chunk_all.value.push(new Uint8Array(chunk_one));
if (event.data && num.value == 0) {
num.value++;
callback(new Date().getTime());
}
const base64Chunk = event.data.trim();
const byteArray = base64ToUint8Array(base64Chunk);
if (!wavHeader.value) {
// 从第一段提取WAV头部
wavHeader.value = byteArray.slice(0, 44);
receivedChunks.value.push(byteArray.slice(44));
} else {
receivedChunks.value.push(byteArray);
}
chunkCounter.value++;
// 每十个片段为一组处理
if (chunkCounter.value >= 5) {
processChunkGroup();
chunkCounter.value = 0;
}
if (!playing.value) {
playing.value = true;
playStreamedAudio();
}
});
eventSource.addEventListener("end", () => {
if (receivedChunks.value.length > 0) {
processChunkGroup();
}
eventSource.close();
});
eventSource.addEventListener("error", (event) => {
console.error("SSE 错误:", event);
if (receivedChunks.value.length > 0) {
processChunkGroup();
}
// const blob = new Blob(chunk_all.value, { type: "audio/wav" });
// saveAs(blob, "audio.wav");
eventSource.close();
reset();
});
};
useAudio
import { ref } from "vue";
export const useAudio = (base64, reset, callback) => {
const source = ref();
const audioContext = ref();
audioContext.value = new (window.AudioContext || window.webkitAudioContext)();
const base64ToUint8Array = () => {
const binaryString = atob(base64);
const len = binaryString.length;
const bytes = new Uint8Array(len);
for (let i = 0; i < len; i++) {
bytes[i] = binaryString.charCodeAt(i);
}
return bytes.buffer;
};
const playAudio = async () => {
callback(new Date().getTime());
// if (audioContext.value.state === "suspended") {
// audioContext.value.resume();
// }
const audioBuffer = await audioContext.value.decodeAudioData(
base64ToUint8Array()
);
source.value = audioContext.value.createBufferSource();
source.value.buffer = audioBuffer;
source.value.connect(audioContext.value.destination);
source.value.start(0);
source.value.onended = () => {
reset();
};
};
playAudio();
};
这些还好,让人头疼的是兼容问题,最让人头疼的是没有测试机,我用的安卓,周边也用到都是小米系列,领导倒是用的ios。
兼容问题
- ios端部分机型无法播放(全是厚脸皮找的其他组的手机测试的),最后用的web audio API解决的
- ios端视频文件大小怎么压缩都没有作用 就采取了折中的法子,视频的实时流用canvas承接一下,实时推到video上,那么获取到就是canvas操作过的视频文件了。
更多推荐


所有评论(0)