forked from aws-amplify/amplify-js
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat(@aws-amplify/ui-components): Add Chatbot UI to main (aws-amplify…
…#6684) * feat(@aws-amplify/ui-components): Add ChatBot Component Fixes: aws-amplify#5024 * amplify-chatbot initial import * Use interface and comment out test * Expose additional css variables and add icon variant to button * Update snapshot * Clean up code * Remove unused test case * Add snapshot testing * Apply comments from @ashika01 * Rename --icon-color to --icon-fill * Remove unused class css * Update css for compatibility with existing components * Set default height * Integrate Interactions text message * Update snapshots * Simplify code * Add audiorecorder and integrate voice chat * Use interface over type * Reorder functions and add byte descriptions * Add loading animation * Update interaction types * Scroll to bottom * set methods private * Rename css class * Update snapshot * Add error handling and reorder functions * Refactor error handling * Refactor chatbot functions * Cleanup * Update snapshot * Expose width css variable from amplify-button * px to rem * Expose width and height variable; Control height at top level * Add header slot * Add listening animation * Cleanup * Update angular module * Move visualization to helper and downsample data array * Separate animation scss * Remove console.logs * Control width / height at host; expose message colors * Use I18n with strings * Fix typo * Use enum for chat state * Revert width back to 100% * Rename updateProps to validateProps * Separate out interaction enum strings * Move MIME type string to constants file * Use async/await pattern in recorder.ts * Check isBrowser and add silence props * Separate init from recorder for async control * Remove fieldId * Add try catch around Interactions.send * Remove requestId * Update snapshot * Expose Interactions types * Remove duplicate logic * Use enum to describe where the message is from * Clean up css and set enum value * Add slot description * Simplify import * Default noop to visualizer * Comment AudioRecorder and separate constants * Update snapshot * Reorder css * Enable conversationModeOn prop * Update packages/amplify-ui-components/src/common/audio-control/helper.ts Co-authored-by: Ashika <35131273+ashika01@users.noreply.github.com> * Move error strings to translations * Remove trailing comma * Wrap audioContext resume with error logger * Try catch `resume` and make startRecording async * Use callback based decode for safari Co-authored-by: Ashika <35131273+ashika01@users.noreply.github.com> * ci: enable preview release from ui-components/main (aws-amplify#6648) * Enable publish from ui-preview branch * Revert checkout Co-authored-by: Jordan Ranz <jordanmranz@gmail.com> * fix(@aws-amplify/ui-components): Update scss and reset chat state upon finish (aws-amplify#6652) * Move width/height control to container level * Add min-height to footer * Reset chat state upon session finish * Remove trailing comma * Handle error based on whether it's recoverable or not * Put a different placeholder if only voice is enabled * Make dot color customizable * Fix typo in translations * Let users stop audio and remove speaking chat state * Add --amplify-blue as bg chat color * Remove console.error * ci: add interactions integ test (aws-amplify#6678) Co-authored-by: Ashika <35131273+ashika01@users.noreply.github.com> Co-authored-by: Jordan Ranz <jordanmranz@gmail.com>
- Loading branch information
Showing
57 changed files
with
1,452 additions
and
130 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
127 changes: 127 additions & 0 deletions
127
packages/amplify-ui-components/src/common/audio-control/helper.ts
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,127 @@ | ||
| import { RECORDER_EXPORT_MIME_TYPE } from './settings'; | ||
|
|
||
| /** | ||
| * Merges multiple buffers into one. | ||
| */ | ||
| const mergeBuffers = (bufferArray: Float32Array[], recLength: number) => { | ||
| const result = new Float32Array(recLength); | ||
| let offset = 0; | ||
| for (let i = 0; i < bufferArray.length; i++) { | ||
| result.set(bufferArray[i], offset); | ||
| offset += bufferArray[i].length; | ||
| } | ||
| return result; | ||
| }; | ||
|
|
||
| /** | ||
| * Downsamples audio to desired export sample rate. | ||
| */ | ||
| const downsampleBuffer = (buffer: Float32Array, recordSampleRate: number, exportSampleRate: number) => { | ||
| if (exportSampleRate === recordSampleRate) { | ||
| return buffer; | ||
| } | ||
| const sampleRateRatio = recordSampleRate / exportSampleRate; | ||
| const newLength = Math.round(buffer.length / sampleRateRatio); | ||
| const result = new Float32Array(newLength); | ||
| let offsetResult = 0; | ||
| let offsetBuffer = 0; | ||
| while (offsetResult < result.length) { | ||
| const nextOffsetBuffer = Math.round((offsetResult + 1) * sampleRateRatio); | ||
| let accum = 0, | ||
| count = 0; | ||
| for (let i = offsetBuffer; i < nextOffsetBuffer && i < buffer.length; i++) { | ||
| accum += buffer[i]; | ||
| count++; | ||
| } | ||
| result[offsetResult] = accum / count; | ||
| offsetResult++; | ||
| offsetBuffer = nextOffsetBuffer; | ||
| } | ||
| return result; | ||
| }; | ||
|
|
||
| /** | ||
| * converts raw audio values to 16 bit pcm. | ||
| */ | ||
| const floatTo16BitPCM = (output: DataView, offset: number, input: Float32Array) => { | ||
| let byteOffset = offset; | ||
| for (let i = 0; i < input.length; i++, byteOffset += 2) { | ||
| const s = Math.max(-1, Math.min(1, input[i])); | ||
| output.setInt16(byteOffset, s < 0 ? s * 0x8000 : s * 0x7fff, true); | ||
| } | ||
| }; | ||
|
|
||
| /** | ||
| * Write given strings in big-endian order. | ||
| */ | ||
| const writeString = (view: DataView, offset: number, string: string) => { | ||
| for (let i = 0; i < string.length; i++) { | ||
| view.setUint8(offset + i, string.charCodeAt(i)); | ||
| } | ||
| }; | ||
|
|
||
| /** | ||
| * Encodes raw pcm audio into a wav file. | ||
| */ | ||
| const encodeWAV = (samples: Float32Array, exportSampleRate?: number) => { | ||
| /** | ||
| * WAV file consists of three parts: RIFF header, WAVE subchunk, and data subchunk. We precompute the size of them. | ||
| */ | ||
|
|
||
| const audioSize = samples.length * 2; // We use 16-bit samples, so we have (2 * sampleLength) bytes. | ||
| const fmtSize = 24; // Byte size of the fmt subchunk: 24 bytes that the audio information that we'll set below. | ||
| const dataSize = 8 + audioSize; // Byte size of the data subchunk: raw sound data plus 8 bytes for the subchunk descriptions. | ||
|
|
||
| const totalByteSize = 12 + fmtSize + dataSize; // Byte size of the whole file, including the chunk header / descriptor. | ||
|
|
||
| // create DataView object to write byte values into | ||
| const buffer = new ArrayBuffer(totalByteSize); // buffer to write the chunk values in. | ||
| const view = new DataView(buffer); | ||
|
|
||
| /** | ||
| * Start writing the .wav file. We write top to bottom, so byte offset (first numeric argument) increases strictly. | ||
| */ | ||
| // RIFF header | ||
| writeString(view, 0, 'RIFF'); // At offset 0, write the letters "RIFF" | ||
| view.setUint32(4, fmtSize + dataSize, true); // At offset 4, write the size of fmt and data chunk size combined. | ||
| writeString(view, 8, 'WAVE'); // At offset 8, write the format type "WAVE" | ||
|
|
||
| // fmt subchunk | ||
| writeString(view, 12, 'fmt '); //chunkdId 'fmt ' | ||
| view.setUint32(16, fmtSize - 8, true); // fmt subchunk size below this value. We set 8 bytes already, so subtract 8 bytes from fmtSize. | ||
| view.setUint16(20, 1, true); // Audiio format code, which is 1 for PCM. | ||
| view.setUint16(22, 1, true); // Number of audio channels. We use mono, ie 1. | ||
| view.setUint32(24, exportSampleRate, true); // Sample rate of the audio file. | ||
| view.setUint32(28, exportSampleRate * 2, true); // Data rate, or # of data bytes per second. Since each sample is 2 bytes, this is 2 * sampleRate. | ||
| view.setUint16(32, 2, true); // block align, # of bytes per sample including all channels, ie. 2 bytes. | ||
| view.setUint16(34, 16, true); // bits per sample, ie. 16 bits | ||
|
|
||
| // data subchunk | ||
| writeString(view, 36, 'data'); // write the chunkId 'data' | ||
| view.setUint32(40, audioSize, true); // Audio byte size | ||
| floatTo16BitPCM(view, 44, samples); // raw pcm values then go here. | ||
| return view; | ||
| }; | ||
|
|
||
| /** | ||
| * Given arrays of raw pcm audio, downsamples the audio to desired sample rate and encodes it to a wav audio file. | ||
| * | ||
| * @param recBuffer {Float32Array[]} - 2d float array containing the recorded raw audio | ||
| * @param recLength {number} - total length of recorded audio | ||
| * @param recordSampleRate {number} - sample rate of the recorded audio | ||
| * @param exportSampleRate {number} - desired sample rate of the exported file | ||
| */ | ||
| export const exportBuffer = ( | ||
| recBuffer: Float32Array[], | ||
| recLength: number, | ||
| recordSampleRate: number, | ||
| exportSampleRate: number, | ||
| ) => { | ||
| const mergedBuffers = mergeBuffers(recBuffer, recLength); | ||
| const downsampledBuffer = downsampleBuffer(mergedBuffers, recordSampleRate, exportSampleRate); | ||
| const encodedWav = encodeWAV(downsampledBuffer, exportSampleRate); | ||
| const audioBlob = new Blob([encodedWav], { | ||
| type: RECORDER_EXPORT_MIME_TYPE, | ||
| }); | ||
| return audioBlob; | ||
| }; |
3 changes: 3 additions & 0 deletions
3
packages/amplify-ui-components/src/common/audio-control/index.ts
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| export * from './recorder'; | ||
| export * from './helper'; | ||
| export * from './visualizer'; |
Oops, something went wrong.