In this tutorial we are going to be covering Blendshapes, what they are, what it's used for and how to use them in Unity and we will cover integrating that with Unity's audio API to get the desired result.
So now you got a feel as to what we will be making.
Blendshapes are just like transitions that the vertices go through to go from being in one position in space to another with the help of a transition value. All the major 3D modelling software from Blender to Maya has it.
As you can see that the model has 2 shape keys, one called MinValue and another called MaxValue, these names don't carry over into unity but the index of the shape key ( blend shape ) matters.
We will be manipulating the MaxValue to make changes to the mesh.
This is how the model looks when it has it's MaxValue set at zero.
You may have guessed now that it is just a basic linear interpolation between these two values.
You can get the blend file I have used from HERE.
The .blend file can be directly imported into unity.
When imported into Unity and dragged into the scene you may have noticed something odd about the components that the game object has. Instead of the usual Mesh Renderer we now have a Skinned Mesh Renderer.
We need a Skinned Mesh Renderer when dealing with meshes that can be animated with bones or Blend shapes.
We can also see that the MaxValue blend shape came in the inspector and that it's value is set at 1.
MinValue is not there because that's the basis for which this blend shape was defined with.
Lets look at the Skinned Mesh Renderer API that Unity provides.
skinnedMeshRenderer.SetBlendShapeWeight(0,100.0f) It takes in two parameters the index of blend shape and it's weight ( the amount of strength for that shape ) values for the weight go from 0 to 100.
skinnedMeshRenderer.GetBlendShapeWeight(0); It returns a float value of the weight of the blend shape at that index (int).
So now we will look at Unity's Audio API, not the entire thing just the part that allows us to do stuff like this.
This is a representation of the audio data based on it's frequency and the how loud (amplitude) that particular part of the frequency spectrum is.
Let's start coding the thing.
We will make class called AudioVisualizer.
Member variables:
[Range(1.0f,4500.0f)]
public float multiplier;
public int minRange = 0;
public int maxRange = 64;
private SkinnedMeshRenderer skinnedMeshRenderer;
private float prevAvg = 0.0f;Nothing special here, we will get to know what these are used for as we go on.Start function:
void Start ()
{
skinnedMeshRenderer = GetComponent<SkinnedMeshRenderer> ();
} Update Function:
void Update ()
{
float[] spectrum = new float[64];
AudioListener.GetSpectrumData(spectrum, 0, FFTWindow.BlackmanHarris);
if (maxRange < minRange)
maxRange = minRange + 1;
minRange = Mathf.Clamp(minRange, 0, 63);
maxRange = Mathf.Clamp(maxRange, 0, 63);
float avg = 0;
for (int i = minRange; i < maxRange; i++)
avg += Mathf.Abs(spectrum[i]);
avg = avg / (float)Mathf.Abs(maxRange - minRange);
if (avg - prevAvg > 0.0012f)
avg = prevAvg + 0.0012f;
else if (avg - prevAvg < -0.0012f)
avg = prevAvg - 0.0012f;
skinnedMeshRenderer.SetBlendShapeWeight(0, avg*multiplier);
prevAvg = avg;
} Now time to break it down.
float[] spectrum = new float[64];
AudioListener.GetSpectrumData(spectrum, 0, FFTWindow.BlackmanHarris); We allocate an array of 64 floats.
We then pass in that array as parameter to GetSpectrumData.GetSpectrumData requires 3 parameters -
- The float array (has be power of 2 in size eg :32,64,128 etc).
- Which audio channel it is. In case of stereo audio 2 channels exist, we will use the first one.
- Type of FFT Window, We will use BlackmanHarris which is more expensive to compute when compared to Window type of Triangles or Hamming but has better quality.
So each index of the float[] is a higher frequency bracket.
With index 0 being lowest frequency bracket and 63 being highest frequency bracket.
The float value at each index represents how much amplitude that frequency had.
if (maxRange < minRange)
maxRange = minRange + 1;
minRange = Mathf.Clamp(minRange, 0, 63);
maxRange = Mathf.Clamp(maxRange, 0, 63); The above code keep parameters within the correct range.
float avg = 0;
for (int i = minRange; i < maxRange; i++)
avg += Mathf.Abs(spectrum[i]);
avg = avg / (float)Mathf.Abs(maxRange - minRange);
if (avg - prevAvg > 0.0012f)
avg = prevAvg + 0.0012f;
else if (avg - prevAvg < -0.0012f)
avg = prevAvg - 0.0012f;I am going through the specified range of frequencies and averaging them and also limiting how much change can happen each frame so that the motion is no that erratic.
skinnedMeshRenderer.SetBlendShapeWeight(0, avg*multiplier);
prevAvg = avg;Here we are setting the blend shape weight and assigning prevAvg as the currently calculated average.
That's it! Here is the source code:
1using UnityEngine;
2
3[RequireComponent(typeof(SkinnedMeshRenderer))]
4public class AudioVisualizer : MonoBehaviour
5{
6 [Range(1.0f,4500.0f)]
7 public float multiplier;
8 public int minRange = 0;
9 public int maxRange = 64;
10 private SkinnedMeshRenderer skinnedMeshRenderer;
11 private float prevAvg = 0.0f;
12 void Start ()
13 {
14 skinnedMeshRenderer = GetComponent<SkinnedMeshRenderer> ();
15 }
16 void Update ()
17 {
18 float[] spectrum = new float[64];
19 AudioListener.GetSpectrumData(spectrum, 0, FFTWindow.BlackmanHarris);
20 if (maxRange < minRange)
21 maxRange = minRange + 1;
22 minRange = Mathf.Clamp(minRange, 0, 63);
23 maxRange = Mathf.Clamp(maxRange, 0, 63);
24 float avg = 0;
25 for (int i = minRange; i < maxRange; i++)
26 avg += Mathf.Abs(spectrum[i]);
27 avg = avg / (float)Mathf.Abs(maxRange - minRange);
28 if (avg - prevAvg > 0.0012f)
29 avg = prevAvg + 0.0012f;
30 else if (avg - prevAvg < -0.0012f)
31 avg = prevAvg - 0.0012f;
32 skinnedMeshRenderer.SetBlendShapeWeight(0, avg*multiplier);
33 prevAvg = avg;
34 }
35}