需求:

        实现一个H5网页,在手机浏览器上一边录视频一边语音录制提问发给模型端,模型端解析的结果两种输出方式,一种解析完毕再返回音频数据,wav格式,第二种是流式传输,一边解析一边传输。目前模型端已经实现。

需求分析:

录音的同时并开始视频,因为视频和录音要调取对应的接口,文件也要尽可能的小,接口限制要不然太大的话传输耗时多。

开始:

H5网页的话获取视频和录音权限的话用到navigator.mediaDevices.getUserMedia但是因为音频和视频是分开的,这里视频就用到此API,然后音频就换了一个recorder-core插件来控制

视频流的开启和音频的开始这一步在初始化中就完成,这样在后续使用中不用频繁的操作

里面操控录制视频的分辨率和文件大小的配置真是油盐不进啊!!!

onMounted(() => {
  videoType.value = MediaRecorder.isTypeSupported('video/mp4') ? 'mp4' : 'webm';
  alert('浏览器允许生成视频格式为:'+videoType.value)
  if (navigator.mediaDevices || navigator.getUserMedia) {
    //{ facingMode: {exact: "environment"} }
    getUserMedia({
      video: {
        facingMode: { exact: "environment" },
        width: { min: 480, ideal: 480, max: 480, exact:480 },
        height: { min: 640, ideal: 640, max: 640, exact: 640 },
        // width: 480,
        // height: 640,
      }, audio: false}); // 调用用户媒体设备,访问摄像头、录音
    // getUserMedias({audio: true },{videoBitsPerSecond:16000,channelCount:1})
    rec.value = Recorder({
      type:"wav" //录音格式,可以换成wav等其他格式
      ,sampleRate:16000 //录音的采样率,越大细节越丰富越细腻
      ,bitRate:16 //录音的比特率,越大音质越好
      ,log:false //是否在控制台输出日志
    });
    // return
    rec.value.open(
      function (e) {
        // e.preventDefault();
        //视频流
        
      },
      function (msg:string, isUserNotAllow: any) {
        // 浏览器不支持录音、用户拒绝麦克风权限、或者非安全环境(非https、file等
        console.log((isUserNotAllow ? 'UserNotAllow,' : '') + '无法录音:' + msg)
        // fn(“打开失败”)
      }
    )
  } else {
    alert("你的浏览器不支持访问用户媒体设备");
  }
})
const getUserMedia = (constrains: {video: any, audio: any}) => { 
      if (navigator.mediaDevices.getUserMedia) {
        // 最新标准API
        navigator.mediaDevices.getUserMedia(constrains).then(stream => { success(stream); }).catch(err => { error(err); });
      }
  //     else if (navigator.webkitGetUserMedia) {
  //       // webkit内核浏览器
  //       navigator.webkitGetUserMedia(constrains).then((stream:any) => { success(stream); }).catch((err:any) => { error(err); });
  //     } else if (navigator.mozGetUserMedia) {
  //       // Firefox浏览器
  //       navigator.mozGetUserMedia(constrains).then((stream:any) => { success(stream); }).catch((err:any) => { error(err); });
  // } 
      else if (navigator.getUserMedia) {
        // 旧版API
        navigator.getUserMedia(constrains).then((stream:any) => { success(stream); }).catch((err:any) => { error(err); });
      }
}
const success = (stream: any) => {
  nextTick(() => {
    if (video.value) {
      video.value.srcObject = stream;
      // hmFn()
      
      streamValue.value = stream
      video.value.onloadedmetadata = () => {
        requestAnimationFrame(fm)
      }
      // fm()
    }
  })

}

长按视频流式播放

流式输出遇到很多问题,视频大小分辨率输出ios和鸿蒙兼容问题,分段播放的接受到的是wav文件流转到base64,一开始给每个分段都加了完整的wav头部信息,后来发现分段太小,都不够一个字符读音,最后网上给的灵感,多段合并播放,100段的流,每5段或者10段为一组,经过测试发现可以实现,但是就是毛边撕拉撕拉的杂音,完整拼接也有这个问题。那就先这样吧。模型端也没时间。算了,上完整代码吧还是,。

app.vue

<script setup lang="ts">
import { ref, nextTick, onMounted } from 'vue'
//必须引入的核心
import Recorder from 'recorder-core'
import { asr, qwen2, tts, upload, uploadAudio } from './api/index'
import { blobToBase64, str24, base64ToArrayBuffer } from './utils/blobToBase64'
import './utils/base64.js'
import Loading from './components/loading.vue'
//引入mp3格式支持文件;如果需要多个格式支持,把这些格式的编码引擎js文件放到后面统统引入进来即可
// import 'recorder-core/src/engine/mp3'
// import 'recorder-core/src/engine/mp3-engine'
//录制wav格式的用这一句就行
import 'recorder-core/src/engine/wav'
import { fenduan } from './hooks/useFenduan'
import { useNoFenduan } from './hooks/useNoFenduan'
import { useAudio } from './hooks/useAudio'
//可选的插件支持项,这个是波形可视化插件
// import 'recorder-core/src/extensions/waveview'
import { saveAs } from "file-saver";
const checkeds = ref([])
const isDialog = ref(false)
const isLoadFfmpegCore = ref(false)
const video: any = ref(null)
const rec = ref()
const mediaRecorder = ref()
const videoBlobs = ref([])
const asrResult = ref('')
const audioContent = ref('')
const isRecording = ref(false)
const audio = ref()
const recordContent = ref('')
const type = ref('视频')
const type1 = ref('流式')
const isCanera = ref(false)
const wrapperRef = ref()
const isShow = ref(false)
const imgContent = ref('')
const canvas1 = ref()
// const bufferQueue = ref([])
// const playing = ref(false);
// const chunkCounter = ref(0);
// const receivedChunks = ref([]);
// const audioContext = ref()
// const wavHeader = ref()
const canvas = ref()
const timeArr = ref({
  t1: '',
  t2: '',
  t3: '',
  t4: '',
  t5: ''
})
const videoType = ref('')
const reset = () => {
  timeArr.value = {
    t1: '',
    t2: '',
    t3: '',
    t4: '',
    t5: ''
  }
  isRecording.value = false
  isShow.value = false
  isDialog.value = false
  if (type.value == '图片') isCanera.value = true
}
const ttsFn = (str) => {
  let t5 = new Date().getTime()
  tts({ text: str }).then((res: any) => {
    timeArr.value.t5 = '语音播报接口耗时:' + (new Date().getTime() - t5)
    if (res.code == 200) {
      audioContent.value = 'data:audio/mp3;base64,' + res.data
      audio.value.src = audioContent.value
      audio.value.play()
      isRecording.value = true
      audio.value.addEventListener('ended', function () {
        reset()
        console.log("语音播报结束")
      }, false);
    }
    
    if(type.value == '图片') isCanera.value = true
  }).catch((err) => {
    reset()
    alert(err)
  })
}
const uploadAll = (res, mediaType?: string, time?: any) => {
  if(time)timeArr.value.t2 = '前端获取base64到上传数据之前的耗时:' + (new Date().getTime() - time)
  let t3 = new Date().getTime()
  upload({audioFile:recordContent.value, videoFile:res, fileType: type.value == '图片' ? 'image' : 'video', mediaType}).then((res3: any) => {
    if (res3.code == 200) {
      timeArr.value.t3 = '上传耗时:' + (new Date().getTime() - t3)
      let t4 = new Date().getTime()
      useAudio(res3.data, reset, (t) => timeArr.value.t4 = '前端获取数据到开始播放的耗时:' + (new Date().getTime() - t4) )
      // audio.value.src = 'data:audio/wav;base64,' + res3.data
      
      // audio.value.play()
      // timeArr.value.t4 = '接收数据到开始播放的耗时:' + (new Date().getTime() - t4)
      //  audio.value.addEventListener('ended', function () {
      //     reset()
      //     console.log("语音播报结束")
      //   }, false);
    } else {
       reset()
      alert(error)
    }
    // audio.value.src = URL.createObjectURL(res3)  //Blob流播放方式
    
  }).catch(error => {
    reset()
    alert(error)
  }).finally(() => {
    timeArr.value.t3 = '上传接口耗时:' + (new Date().getTime() - t3)
    
  })
}
const videoArr: any = ref([])

const sseFn = async (res, mediaType?: string, time?: any) => {
  if(time)timeArr.value.t2 = '前端获取base64到上传数据之前的耗时:' + (new Date().getTime() - time)
  const uid = str24()
  const eventSource = await new EventSource('https://aidemo.ysh360.top' + '/sse/createSse?uid=' + uid);
  videoArr.value = []
  let t3 = new Date().getTime()
  uploadAudio({ audioFile: recordContent.value, videoFile: res, uid, fileType: type.value == '图片' ? 'image' : 'video', mediaType }).then(response => {
      timeArr.value.t3 = '上传耗时:' + (new Date().getTime() - t3)
    // audio.value.src = 'data:audio/wav;base64,' + response.data
    // audio.value.play()
    //   audio.value.addEventListener('ended', function () {
    //     reset()
    //     console.log("语音播报结束")
    //   }, false);
    if (response.code == 500 || response.code == 502) {
      reset()
    }
  }).catch(err => {
    reset()
    // alert(err)
  })
    ;
  fenduan(
    eventSource,
    reset,
    (t) => timeArr.value.t3 = '开始获取分段耗时:' + (t - t3),
    // (t) => timeArr.value.t4 = '前端获取分段耗时:' + (t - t3)
  )
  // useNoFenduan(eventSource, reset)
  
}
const blobToBase64Fn = (superBuffer: any, time: any) => {
  blobToBase64(superBuffer).then((res: any) => {
    
    if (type1.value == '流式') {
      // alert('视频流式暂时无法使用')
      // if (checkeds.value.includes('语音')) {
      //   ttsFn(asrResult.value)
      // } else {
      // }
      sseFn(res.replace(`data:video/${videoType.value};base64,`, ''),videoType.value, time)
    } else {
      // qwenFn(res.replace('data:video/mp4;base64,', ''))
      uploadAll(res.replace(`data:video/${videoType.value};base64,`, ''), videoType.value, time)
    }
  })
}
const streamValue = ref(null)
const hmFn = () => {
  const isHarmonyOS = /HarmonyOS/i.test(navigator.userAgent);
  if (isHarmonyOS) {
    video.value.addEventListener('loadedmetadata', () => {
      const aspectRatio = video.videoWidth / video.videoHeight;
      if (window.innerWidth / window.innerHeight > aspectRatio) {
        video.value.style.width = 'auto';
        video.value.style.height = '100%';
      } else {
        video.value.style.width = '100%';
        video.value.style.height = 'auto';
      }
    });
  }
}
const lastFrameTime = ref(0)
const frameDuration = ref(24)
const fm = (timestamp) => {

  if (!lastFrameTime) lastFrameTime.value = timestamp;

  const elapsedTime = timestamp - lastFrameTime.value;

  if (elapsedTime >= frameDuration.value) {
    canvas1.value.width = 480;
    canvas1.value.height = 640;
    const canvasContent = canvas1.value.getContext('2d');
    canvasContent.drawImage(video.value, 0, 0, 480, 640);
    lastFrameTime.value = timestamp;
  }




  requestAnimationFrame(fm)
}
const success = (stream: any) => {
  nextTick(() => {
    if (video.value) {
      video.value.srcObject = stream;
      // hmFn()
      
      streamValue.value = stream
      video.value.onloadedmetadata = () => {
        requestAnimationFrame(fm)
      }
      // fm()
    }
  })

}
const error = (error: any) => {
  alert(`访问用户媒体设备失败,${error.name, error.message}`)
}
const getUserMedia = (constrains: {video: any, audio: any}) => { 
      if (navigator.mediaDevices.getUserMedia) {
        // 最新标准API
        navigator.mediaDevices.getUserMedia(constrains).then(stream => { success(stream); }).catch(err => { error(err); });
      }
  //     else if (navigator.webkitGetUserMedia) {
  //       // webkit内核浏览器
  //       navigator.webkitGetUserMedia(constrains).then((stream:any) => { success(stream); }).catch((err:any) => { error(err); });
  //     } else if (navigator.mozGetUserMedia) {
  //       // Firefox浏览器
  //       navigator.mozGetUserMedia(constrains).then((stream:any) => { success(stream); }).catch((err:any) => { error(err); });
  // } 
      else if (navigator.getUserMedia) {
        // 旧版API
        navigator.getUserMedia(constrains).then((stream:any) => { success(stream); }).catch((err:any) => { error(err); });
      }

}
const recordFlag = ref(false)
const longPressTimer: any = ref(null)
const tatchFn = () => {
  if (recordFlag.value) {
    rec.value.stop((blob: Blob, duration: any) => {
      // blob: 流文件,duration: 持续时间ms,这里可以执行文件播放或者文件上传
      // rec.value.close()
      recordFlag.value = false
      isDialog.value = true
      let t1 = new Date().getTime()
      console.log(blob,'blob')
      blobToBase64(blob).then((res: any) => { 
        // console.log(res,'res___________________res')
        t1 = new Date().getTime() - t1
        timeArr.value.t1 = '语音blob流转base64时间:' + t1
        recordContent.value = res.replace('data:audio/wav;base64,', '')
        let t2 = new Date().getTime()
        if (type1.value == '整体') {
          if (type.value == '语音') {
            asr({audio_data:res.replace('data:audio/wav;base64,','')}).then((res: any) => {
              asrResult.value = res.data
              timeArr.value.t2 = '语音识别时间:' + (new Date().getTime() - t2)
              ttsFn(asrResult.value)
              timeArr.value.t3 = 'null'
              timeArr.value.t4 = 'null'
          }).catch((error) => {
              recordFlag.value = false
              isDialog.value = false
              alert(error)
            })
          } else if (type.value == '图片') {
            timeArr.value.t2 = 'null'
            timeArr.value.t3 = 'null'
            // qwenFn(imgContent.value.replace('data:image/png;base64,', ''))
            uploadAll(imgContent.value.replace('data:image/png;base64,', ''),'png',t2)
          }
          
            
        } else {
          if(type.value == '语音') {
            asr({ audio_data: res.replace('data:audio/wav;base64,', '') }).then((res: any) => {
              asrResult.value = res.data
              timeArr.value.t2 = '语音识别时间:' + (new Date().getTime() - t2)
              timeArr.value.t3 = 'null'
                timeArr.value.t4 = 'null'
                ttsFn(asrResult.value)
              
            }).catch((error) => {
              recordFlag.value = false
              isDialog.value = false
              alert(error)
            })
          } else if (type.value == '图片') {
            timeArr.value.t2 = 'null'
            timeArr.value.t3 = 'null'
            sseFn(imgContent.value.replace('data:image/png;base64,', ''),'png',t2)
          }
            
        }
        
      })
    },
    (s: any) => {
      alert('结束出错:' + s)
      rec.value.close()
      recordFlag.value = false
      isDialog.value = false
    }
    )
    mediaRecorder.value.addEventListener('stop', () => {
      const superBuffer = new Blob(videoBlobs.value, { type: `video/${videoType.value}` });
      // saveAs(superBuffer, `video.${videoType.value}`);
      //获取录制的视频流并调整分辨率
      
      // alert('canvas1-width:' + canvas1.value.width +'canvas1-height:' +canvas1.value.height)
      // video.value.src = url;
      // video.value.onloadeddata = (e) => {
      //   canvasContent.drawImage(video.value, 0, 0, 480, 360);
      // }
      let t2 = new Date().getTime()
      if (videoType.value == 'webm') {
        timeArr.value.t3 = 'null'
        blobToBase64Fn(superBuffer, t2)
      } else {
        timeArr.value.t3 = 'null'
        blobToBase64Fn(superBuffer, t2)
      }
    });
    console.log(type.value == '视频', 'type.value == 视频')
    if(type.value == '视频')mediaRecorder.value.stop()
  
  } else {
    videoBlobs.value = []
    // navigator.mediaDevices.getUserMedia({ audio: true }).then(res => {
    mediaRecorder.value = new MediaRecorder(canvas1.value.captureStream(),
      {
        mimeType: `video/${videoType.value}`,
        videoBitsPerSecond: 500000
      });
      
      if(type.value =='视频'){
          mediaRecorder.value.ondataavailable = event => {
            if (event.data && event.data.size > 0) {
              videoBlobs.value.push(event.data);
            }
          };
          mediaRecorder.value.start();//视频录制开始
        }
        rec.value.start()
        recordFlag.value = true
    // })
  }
}
// const recFnCancel = () => {
//   if (longPressTimer.value) {
//     clearTimeout(longPressTimer.value);
//     longPressTimer.value = null;
//     tatchFn()
//   }
// }
const recFn = (e) => {
  // longPressTimer.value = setTimeout(function() {
      // 执行长按要执行的操作
  // if(type.value == '视频'&&type1.value == '流式') return alert('流式视频暂时关闭')
    e.preventDefault()
    tatchFn()
  // }, 1000);
  
}
const cameraClick = () =>  {
  if (!video.value) return;
  isShow.value = true;
  // 设置canvas画布  
  const canvas = document.createElement("canvas");
  canvas.width = video.value.videoWidth;
  canvas.height = video.value.videoHeight;
  // 获取canvas上下文对象
  const ctx = canvas.getContext("2d");
  // 截图操作
  console.log(canvas.height, canvas.width)
  ctx?.drawImage(video.value, 0, 0, canvas.width, canvas.height);
  // 展示截图  
  nextTick(() => {
    // wrapperRef.value.appendChild(canvas);
    imgContent.value = canvas.toDataURL('image/png')
    wrapperRef.value.setAttribute("src", imgContent.value);
    isCanera.value = false;
  })
  
}
const change = (value) => {
  type.value = value.target.value;
  isCanera.value = type.value == '图片';
  isShow.value = false;
}
const change1 = (value) => {
  type1.value = value.target.value;
}
onMounted(() => {
  videoType.value = MediaRecorder.isTypeSupported('video/mp4') ? 'mp4' : 'webm';
  alert('浏览器允许生成视频格式为:'+videoType.value)
  if (navigator.mediaDevices || navigator.getUserMedia) {
    //{ facingMode: {exact: "environment"} }
    getUserMedia({
      video: {
        facingMode: { exact: "environment" },
        width: { min: 480, ideal: 480, max: 480, exact:480 },
        height: { min: 640, ideal: 640, max: 640, exact: 640 },
        // width: 480,
        // height: 640,
      }, audio: false}); // 调用用户媒体设备,访问摄像头、录音
    // getUserMedias({audio: true },{videoBitsPerSecond:16000,channelCount:1})
    rec.value = Recorder({
      type:"wav" //录音格式,可以换成wav等其他格式
      ,sampleRate:16000 //录音的采样率,越大细节越丰富越细腻
      ,bitRate:16 //录音的比特率,越大音质越好
      ,log:false //是否在控制台输出日志
    });
    // return
    rec.value.open(
      function (e) {
        // e.preventDefault();
        //视频流
        
      },
      function (msg:string, isUserNotAllow: any) {
        // 浏览器不支持录音、用户拒绝麦克风权限、或者非安全环境(非https、file等
        console.log((isUserNotAllow ? 'UserNotAllow,' : '') + '无法录音:' + msg)
        // fn(“打开失败”)
      }
    )
  } else {
    alert("你的浏览器不支持访问用户媒体设备");
  }
})
  
</script>

useFendaun

import { ref } from "vue";
import { saveAs } from "file-saver";
export const fenduan = (
  eventSource: any,
  reset: Function,
  callback: Function
) => {
  const bufferQueue = ref([]);
  const playing = ref(false);
  const chunkCounter = ref(0);
  const receivedChunks = ref([]);
  const audioContext = ref();
  const wavHeader = ref();
  const num = ref(0);
  const chunk_all = ref([]);
  const base64ToUint8Array = (base64String) => {
    const binaryString = atob(base64String);
    const len = binaryString.length;
    const bytes = new Uint8Array(len);
    for (let i = 0; i < len; i++) {
      bytes[i] = binaryString.charCodeAt(i);
    }
    return bytes;
  };
  const concatenateUint8Arrays = (arrays) => {
    // console.log(arrays, "arrs________________Arrs");
    let totalLength = arrays.reduce((sum, value) => sum + value.length, 0);
    let result = new Uint8Array(totalLength);
    let offset = 0;
    for (let array of arrays) {
      result.set(array, offset);
      offset += array.length;
    }
    return result;
  };
  const processChunkGroup = () => {
    const combinedChunks = concatenateUint8Arrays(receivedChunks.value);
    const chunkWithHeader = concatenateUint8Arrays([
      wavHeader.value,
      combinedChunks,
    ]);
    bufferQueue.value.push(chunkWithHeader);
    receivedChunks.value = [];
  };
  const playStreamedAudio = async () => {
    while (bufferQueue.value.length > 0) {
      const chunk = bufferQueue.value.shift();
      try {
        const audioBuffer = await audioContext.value.decodeAudioData(
          chunk.buffer
        );
        const source = audioContext.value.createBufferSource();
        source.buffer = audioBuffer;
        source.connect(audioContext.value.destination);
        const playTime = audioContext.value.currentTime - 0.05;
        source.start(playTime);
        await new Promise((resolve) => (source.onended = resolve));
      } catch (error) {
        console.error("解码音频数据时出错:", error);
      }
    }
    playing.value = bufferQueue.value.length > 0;
  };
  if (!audioContext.value) {
    audioContext.value = new (window.AudioContext ||
      window.webkitAudioContext)();
  }

  if (audioContext.value.state === "suspended") {
    audioContext.value.resume();
  }
  eventSource.addEventListener("message", async (event) => {
    // const chunk_one = atob(event.data)
    //   .split("")
    //   .map((c) => c.charCodeAt(0));
    // chunk_all.value.push(new Uint8Array(chunk_one));
    if (event.data && num.value == 0) {
      num.value++;
      callback(new Date().getTime());
    }
    const base64Chunk = event.data.trim();
    const byteArray = base64ToUint8Array(base64Chunk);

    if (!wavHeader.value) {
      // 从第一段提取WAV头部
      wavHeader.value = byteArray.slice(0, 44);
      receivedChunks.value.push(byteArray.slice(44));
    } else {
      receivedChunks.value.push(byteArray);
    }

    chunkCounter.value++;

    // 每十个片段为一组处理
    if (chunkCounter.value >= 5) {
      processChunkGroup();
      chunkCounter.value = 0;
    }

    if (!playing.value) {
      playing.value = true;
      playStreamedAudio();
    }
  });

  eventSource.addEventListener("end", () => {
    if (receivedChunks.value.length > 0) {
      processChunkGroup();
    }
    eventSource.close();
  });

  eventSource.addEventListener("error", (event) => {
    console.error("SSE 错误:", event);
    if (receivedChunks.value.length > 0) {
      processChunkGroup();
    }
    // const blob = new Blob(chunk_all.value, { type: "audio/wav" });
    // saveAs(blob, "audio.wav");
    eventSource.close();
    reset();
  });
};

useAudio

import { ref } from "vue";
export const useAudio = (base64, reset, callback) => {
  const source = ref();
  const audioContext = ref();
  audioContext.value = new (window.AudioContext || window.webkitAudioContext)();
  const base64ToUint8Array = () => {
    const binaryString = atob(base64);
    const len = binaryString.length;
    const bytes = new Uint8Array(len);
    for (let i = 0; i < len; i++) {
      bytes[i] = binaryString.charCodeAt(i);
    }
    return bytes.buffer;
  };
  const playAudio = async () => {
    callback(new Date().getTime());
    // if (audioContext.value.state === "suspended") {
    //   audioContext.value.resume();
    // }
    const audioBuffer = await audioContext.value.decodeAudioData(
      base64ToUint8Array()
    );
    source.value = audioContext.value.createBufferSource();
    source.value.buffer = audioBuffer;
    source.value.connect(audioContext.value.destination);
    source.value.start(0);
    source.value.onended = () => {
      reset();
    };
  };
  playAudio();
};

这些还好,让人头疼的是兼容问题,最让人头疼的是没有测试机,我用的安卓,周边也用到都是小米系列,领导倒是用的ios。

兼容问题

  1. ios端部分机型无法播放(全是厚脸皮找的其他组的手机测试的),最后用的web audio API解决的
  2. ios端视频文件大小怎么压缩都没有作用 就采取了折中的法子,视频的实时流用canvas承接一下,实时推到video上,那么获取到就是canvas操作过的视频文件了。

更多推荐