[User Story] - Embeddable chat supporting Human In the loop #13
Description
Description
Provide a full featured chat experience with support for implementing human in the loop for conversational patterns:
- Notify: let the user know some event is important, but not take any actions. This is useful in flagging events that user should see, but where the agent is not empowered to act them. An example is agent progress notification in chat based agents.
- Review or Approval: review an action the agent wants to take. Some actions are "dangerous" enough that it may be worth hard coding a review for any action the agent wants to take. The human can either approve the action, or give direct feedback to the agent on how to change it.
Current Behavior
Current chat experience is basic with no HITL support.
Acceptance Criteria
- Tool execution result are notified to the chat.
- Tool approval request are shown in the chat and user can approve or decline
Design Considerations
- Consider to use AG-UI protocol to standardize chat implementation and communication with backend agents: https://github.com/ag-ui-protocol/ag-ui.
- Agent framework has also recently introduced support for chatKit from Open AI. there is a demo here. Compared to ag-ui it also 'standardize' the way to manage attachments (which is required by the banking app for payment flow) and thread conversation.
- ChatAgent in agent frameworks provide support for streaming generated content or user input requests. It seems it doesn't have support for streaming other internal events like ToolCallStart, ToolCallEnd. The only way to implement that is using a custom middleware ToolNotificationMiddleware that intercept function calls and send data to the UI output stream in a dedicated SSE session.
A better option would be to simply wrap the current supervisor agent in a workflow.
Main Affected Modules and/or Classes
Will start with a spike using chatKit demo from agent-framework
References
ag-ui - https://github.com/ag-ui-protocol/ag-ui
workflow events ( might not be updated) - https://learn.microsoft.com/en-us/agent-framework/user-guide/workflows/core-concepts/events?pivots=programming-language-python