Skip to content

Conversation

@neph1
Copy link

@neph1 neph1 commented Apr 25, 2025

(Not sure why it's diffing some lines that are the same. Edit: Might be my soft reset)

A little experiment to use different prompts for different segments.

In this PR, you can set one prompt per segment generated (roughly 1.5 seconds). Prompts are semi-colon (;) separated. In natural order. If you specify less prompts than segments, it will reuse the first one. Each section has no knowledge of the other prompts, so you must be explicit in the description, hence the double "robot chef"

Example:
"the robot chef gestures with its arms; the robot chef walks to the pot on the right side and stirs it"

250425_190901_125_9597_18.mp4

Now, there are some limitations with this method.
Since FramePack starts with the last segment, the amount of movement you can do is limited. In this case it will start with the image and the prompt " the robot chef walks to the pot on the right side and stirs it". Technically, you might be able to have it enter from the left side, but with the image being what it is, it might be difficult. I've only done a couple of tests so far, so I have no idea how well it works on more complex things.

I afterwards (or mid) realized there is a more advanced project here, which utilizes time-stamps for the prompts : https://github.com/colinurbs/FramePack-Studio

@neph1 neph1 marked this pull request as ready for review April 25, 2025 17:51
@neph1 neph1 changed the title Multiple prompts Sequential prompts Apr 25, 2025
@raziel2001au
Copy link

I think specifying a different prompt for each segment isn't a great solution. What about simply doing a dynamic prompt where every segment it checks if the prompt has changed and if so, applies the new prompt? That way you can monitor the generation and dynamically alter the prompt if required. The alternative would be to have some way of setting the number of segments that should use each prompt, but I still think simply making the prompt dynamic is a fairly good solution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants