Lip Sync

Ddavidsvson · 17 Th12 2016

I'm going to need to do lip sync in my current project, but am completley new to the subject.

I found this: https://github.com/DanielSWolf/rhubarb-lip-sync

Any better sugestions? Where whould I start to get this going with spine and Unity?

Nate · 17 Th12 2016

I've not see Rhubarb. It looks nice, that is probably how I would design such a tool. There is also Papagayo and Annosoft, though I have no experience with those either. Someone wrote a Papagayo to Spine exporter you might find useful. Otherwise, I would load the Rhubarb data and programmatically generate an animation which has an AttachmentTimeline that hides/shows the appropriate attachments at the right times. I'd probably do this at runtime rather than convert Rhubarb output to Spine JSON.

Ddavidsvson · 17 Th12 2016

Papagayo looks interesting, Annosoft looks :think: ... o wow that is alot of $$ :whew:

But I really like the simpleness of rhubarb. Thanks for the tips. I'll try to figure out how to generate an animation programmatically then.

DanielWolf · 10 Th01 2017

Hi, I'm the author of Rhubarb. Converting Rhubarb's lip animation to Spine sounds interesting. Let me know if there's any way I can help!

Ddavidsvson · 19 Th01 2017

Both papagayo and rhubarb can output at textfile with the times (papagayo: frames) when to attach the different attachments (different mouthshapes).

Example rhubarb (time in seconds first then witch attachment):

0.00   X
0.05   D
0.27   C
0.31   B
0.43   X
0.47   X

Example Papagayo (witch frame first (30f/s) then witch attachment):

1 etc
4 O
7 etc
16 E
19 MBP
22 etc

How do I generate an animation programmatically and show/hide the different attachments?

DanielWolf · 27 Th01 2017

I believe the most elegant solution would be to add Spine support right into Rhubarb. I'm prepared to do that, but I'd like some feedback first. Here are my thoughts:

Typically, people have an existing character in Spine and want to add dialog animations to it. So rather than generating a new Spine file from scratch, the user should specify an existing Spine JSON file. This file should contain a rigged skeleton with a slot for the mouth. This slot contains multiple image attachments for the various mouth shapes.

The user would now call Rhubarb with a WAVE file and the existing Spine JSON file. Rhubarb would then generate a new Spine JSON file that's a copy of the original. In addition, it contains a new animation that's named after the WAVE file, with a prefix like "say_". So If you call Rhubarb with a recording named "hi_there.wav", you'd get a new animation called "say_hi_there".

I think this would be pretty straightforward to implement. But I'm new to Spine, so I'm not sure if it's the best solution from a user perspective. Please comment:

Is the behavior I describe above what you're looking for? Is there a better approach?

Nate · 31 Th01 2017

Welcome, DanielWolf!

You're right that generating an entire skeleton would make it difficult to get the animation into an existing skeleton. It could be possible using Import Project, which allows importing an animation from another project file. Rhubarb would write a skeleton JSON file, then you'd need to Import Data to get it into a project file, then you could Import Project to merge in an animation. Note this requires both skeletons to have the same objects that were keyed (slots, attachments, events, etc).

Your proposal would work fine and is simpler than the above juggling. Users would export their project to JSON, add animations to it with Rhubarb, then they can use the data as is or import it back into a Spine project for further editing. I don't think there is a better approach.

DanielWolf · 31 Th01 2017

Thanks for your detailed explanation. That's exactly what I was assuming, so I'll implement it that way.

I'm not sure how soon I will find the time to implement the exporter. As soon as there are any news, I'll update this topic.

Ddavidsvson · 9 Th03 2017

So, I try to generate a animation programmatically from the papagayo output as it can be seen above. But no success, I get no error messages but no attachments seems to get set on slot "Mouth". Any suggestions? Here is my code:

    public TextAsset speechTxt;
    private int speechTxtFrameRate = 24;

void Talk() {
     var lines = speechTxt.text.Split("\n"[0]);     

     AttachmentTimeline timeLine = new AttachmentTimeline (lines.Length-1);
     timeLine.slotIndex = skeletonAnimation.skeleton.data.FindSlotIndex("Mouth");     

 for (int frameIndex = 1; frameIndex < lines.Length-1; frameIndex++) {
      var frameData = lines[frameIndex].Split(" "[0]);
      timeLine.SetFrame (frameIndex - 1, float.Parse (frameData [0]) / speechTxtFrameRate, "Mouths/"+frameData[1]);
 }

 ExposedList<Timeline> timelines = new ExposedList<Timeline>();
 timelines.Add(timeLine);
 Spine.Animation anim = new Spine.Animation("talk", timelines, Speech.Length);
 skeletonAnimation.skeleton.data.animations.Add(anim);
 
 skeletonAnimation.state.SetAnimation (1, "talk", false);
}

Nate · 9 Th03 2017

It looks OK. You'll have to debug it further. Does the AttachmentTimeline try to set the attachment when it should? If so, when it sets the attachment is it able to find the attachment with the Mouths/xxx name?

Ddavidsvson · 9 Th03 2017

Thank you for looking!
I have been tearing my hair, but I just figured out that it was my line endings in the text file that was my problem :bang: ! when I converted them everything worked :rofl: !

Jjamqdlaty · 1 Th08 2017

DanielWolf đã viết
Thanks for your detailed explanation. That's exactly what I was assuming, so I'll implement it that way.

I'm not sure how soon I will find the time to implement the exporter. As soon as there are any news, I'll update this topic.

Hi. do you still plan to implement the exporter?

DanielWolf · 1 Th08 2017

do you still plan to implement the exporter?

Absolutely. I've been postponing this feature for some time, mainly because I heard that Esoteric Software was planning to add audio support to Spine. I felt that it made sense to wait for that, since it makes things easier for me.

Now that Spine 3.7 beta is out, I'll soon start implementing Spine support in Rhubarb Lip Sync.

Erika · 1 Th08 2017

DanielWolf đã viết
I heard that Esoteric Software was planning to add audio support to Spine.

And now we have it!
Thanks for doing this, I myself would love to use it in some projects!

DanielWolf · 1 Th08 2017

Erikari đã viết
I myself would love to use it in some projects!

That's great to hear! And it definitely makes more sense now that Spine has audio support.

By the way: Integrating third-party tools such as Rhubarb with Spine would be much easier if Spine had a plugin interface. I recently integrated Rhubarb with Adobe After Effects, and the UX is much better than what I'll be able to provide for Spine. Is there a chance to see a plugin/scripting system for Spine any time soon?

Raeleus · 5 Th08 2017

Rhubarb looks great! I would love to see integration of this tool in one way or another.

AlaNintendo · 29 Th08 2017

following!

DanielWolf · 13 Th09 2017

I've been busy with other stuff, but now I'm ready to add Spine support to Rhubarb Lip Sync!

Now that Spine has audio support, I'm thinking of the following rough workflow:

The user creates a Spine project with a skeleton. They give the skeleton a slot for the mouth. This slot contains multiple image attachments for the various mouth shapes, named using a fixed schema.
The user has one or more audio file to lip-sync. For each audio file, they create an event.
This step is optional: Rhubarb Lip Sync can create good mouth animation from audio alone. However, the results are even better if Rhubarb is told the dialog text for each sound file. If the user wants to, they may set the event's default string value to the dialog text.
The user exports the skeleton in JSON format.
The user starts "Rhubarb Lip Sync for Spine", which will be a cross-platform GUI application. There, they open the exported JSON file.
The user now has the following controls:
a dropdown to select the slot that represents the mouth
a list of all events with associated audio file. Each event has a checkbox that can be checked.
At the click of a button, automatic mouth animation is performed for all checked events:
For each checked event, a new animation is created that's named after the event, with an added prefix like "say_". So if the user created the event 'hi_there' based on the file 'hi_there.wav', there will now be the animation 'say_hi_there'.
Each generated animation contains the matching audio event at frame 0, plus the actual mouth animation.
The original JSON file is overwritten to contain the new animations.
The user imports the JSON file back into Spine to get the new animations.

I don't expect any technical problems, but I'm concerned about the user experience. My experience with Spine is very limited, so I'd like to hear from some power users:

Does this workflow make sense to you?
Is there aspect that may be inconvenient or unnecessarily complicated?

Any feedback is welcome!

Nate · 13 Th09 2017

Sounds like a good plan!