add pipeline

Author: oshaikh13

public/final_pipeline.jpg

@@ -13,7 +13,7 @@ const App = ({ carouselData, suggestionsData, activeChats, setActiveChats }) =>
           marginBottom: '20px',
         }}
       >
-        GUMBO
+        GUMBOs are proactive assistants enabled by GUMs
       </h2>
 
       {/* App Section */}

src/App.js

@@ -43,10 +43,6 @@ if __name__ == "__main__":
     asyncio.run(main())
 `;
 
-  // const abstractText = "Human-computer interaction has long imagined technology that understands us—from our preferences and habits, to the timing and purpose of our everyday actions. Yet current user models remain fragmented, narrowly tailored to specific applications, and incapable of the flexible, cross-context reasoning required to fulfill these visions. This paper presents an architecture for a general user model (GUM) that learns about you by observing any interaction you have with your computer. The GUM takes as input any unstructured observation of a user (e.g., device screenshots) and constructs confidence-weighted natural language propositions that capture that user's behavior, knowledge, beliefs, and preferences. GUMs can infer that a user is preparing for a wedding they're attending from a message thread with a friend. Or recognize that a user is struggling with a collaborator's feedback on a draft paper by observing multiple stalled edits and a switch to reading related work. GUMs introduce an architecture that infers new propositions about a user from multimodal observations, retrieves related propositions for context, and continuously revises existing propositions. To illustrate the breadth of applications that GUMs enable, we demonstrate how they augment chat-based assistants with contextual understanding, manage OS notifications to surface important information only when needed, and enable interactive agents that adapt to user preferences across applications. We also instantiate a new class of proactive assistants (GUMBOs) that discover and execute useful suggestions on a user's behalf based on the their GUM. In our evaluations, we find that GUMs make calibrated and accurate inferences about users, and that assistants built on GUMs proactively identify and perform actions of meaningful value that users wouldn't think to request explicitly. From observing a user coordinating a move with their roommate, GUMBO worked backward from the user's move-in date and budget, generated a personalized schedule with logistical to-dos, and recommended helpful moving services. Altogether, GUMs introduce new methods that leverage large multimodal models to understand unstructured user context—enabling both long-standing visions of HCI and entirely new interactive systems that anticipate user needs.";
-
-  // const abstractPreview = abstractText.split('. ').slice(0, 3).join('. ') + '.';
-
   return (
 
     <div style={{margin: '0 auto', paddingLeft: '5%', paddingRight: '5%', paddingTop: '20px', paddingBottom: '20px' }}>
@@ -90,12 +86,12 @@ if __name__ == "__main__":
       </div>
 
       <div style={{ display: 'flex', justifyContent: 'center', gap: '15px', marginBottom: '20px' }}>
-        <a href="https://arxiv.org" target="_blank" rel="noopener noreferrer" className="start-chat-button" style={{ padding: '12px 12px', fontSize: '16px' }}>
-          <FaFileAlt style={{ marginRight: '0.5rem', position: 'relative', top: '2px', fontSize: '18px' }} /> Paper
+        <a href="https://arxiv.org" target="_blank" rel="noopener noreferrer" className="start-chat-button" style={{ padding: '12px 12px', fontSize: '16px', display: 'flex', alignItems: 'center' }}>
+          <FaFileAlt style={{ marginRight: '0.5rem', fontSize: '18px' }} /> Paper
         </a>
 
-        <a href="https://github.com/generalusermodels/gum" target="_blank" rel="noopener noreferrer" className="start-chat-button" style={{ padding: '12px 12px', fontSize: '16px' }}>
-          <FaGithub style={{ marginRight: '0.5rem', position: 'relative', top: '2px', fontSize: '18px' }} /> GitHub
+        <a href="https://github.com/generalusermodels/gum" target="_blank" rel="noopener noreferrer" className="start-chat-button" style={{ padding: '12px 12px', fontSize: '16px', display: 'flex', alignItems: 'center' }}>
+          <FaGithub style={{ marginRight: '0.5rem', fontSize: '18px' }} /> GitHub
         </a>
       </div>
 
@@ -228,13 +224,16 @@ if __name__ == "__main__":
               setActiveChats={setActiveChats}
             />
           </DynamicDataProvider>
-        </div>
+          <p style={{ 
+            marginTop: '15px',
+          }}>
+            Above is an example instantiation of GUMBO based on the user's current GUM. Feel free to click on and explore suggestions. Dragging the slider will update the GUMBO's suggestions based on the user's changing GUM.  
+          </p>
+        </div>        
       </div>
 
-
-
       <div style={{ 
-        margin: '30px 0px 0px 0px', 
+        margin: '14px 0px 0px 0px', 
         padding: '25px 30px', 
         borderLeft: '4px solid var(--chat-button-bg)',
         borderRadius: '6px',
@@ -292,12 +291,33 @@ if __name__ == "__main__":
         }}>
           How it works
         </h3>
+        
+        <div style={{ 
+          display: 'flex', 
+          justifyContent: 'center', 
+          margin: '30px auto',
+          width: '70%',
+          backgroundColor: 'white',
+          padding: '20px',
+          borderRadius: '8px',
+          boxShadow: '0 2px 8px rgba(0, 0, 0, 0.2)'
+        }}>
+          <img 
+            src="/final_pipeline.jpg" 
+            alt="GUM Pipeline Architecture" 
+            style={{
+              maxWidth: '100%',
+              height: 'auto',
+            }}
+          />
+        </div>
+
         <p style={{ 
           lineHeight: '1.6',
           margin: '0',
           fontSize: '15px'
         }}>
-          placeholder placeholder placeholder...
+          A Propose module translates unstructured observations into confidence-weighted propositions about the user's preferences, context, and intent. A Retrieve module indexes and searches these propositions to return the most contextually relevant subset for a given query. Finally, using results from Retrieve, a Revise module reevaluates and refines propositions as new observations arrive. Each module is parameterized by a large multimodal model (in our case, a vision and language model, or VLM).
         </p>
 
         <h3 style={{ 
@@ -315,7 +335,7 @@ if __name__ == "__main__":
           margin: '0',
           fontSize: '15px'
         }}>
-          placeholder placeholder placeholder...
+          For GUMs, privacy guarantees are critical from the start. Our general engineering principle here is to rely primarily on open-source models for our study. While closed-source models are more performant, we expect open-source models to be owned by individual users and eventually distilled to be run on local devices. Our study was deployed and run with open-source models. As gaps between closed and open sourced models close and as models become cheaper for inference, model's will become more performant and feasible on commodity hardware. Our implementation is open-source (available on <a href="https://github.com/generalusermodels/gum" target="_blank" rel="noopener noreferrer" style={{ color: '#ff9d9d' }}>GitHub</a>) and uses the OpenAI Completions API. Open source inference platforms like vLLM support the Completions API, and work with systems like GUM.
         </p>