From: <grad-student>
  To  : Robert Escriva <rescrv@rave>
  Date: Thu, 17 Oct 2024 01:36:26 +0000

Re: Workflows

On Wed, 16 Oct 2024 18:35:57 -0700, Robert Escriva <rescrv@rave> wrote:
> Hey all,
> 
> I'm starting this thread to brainstorm ways that we can improve the
> results from models.  For example, I have never met someone who answers
> 100% of their email, and models shouldn't either.  Models should be able
> to say "I don't know" when they don't have enough information to make a
> coherent response and should be able to ask for more information when
> they need it.
> 
> Further, it should be easy to specify workflows that pipe the output of
> one model into another for ranking.  We could then say things like,
> "Iterate for one hour and give me the top three results," or, "Iterate
> until some metric converges."
> 
> Given the throughput-oriented nature of email it makes sense to be able
> to put a large number of operations in a pipeline and have them run in
> the background.
> 
> I'm going to start working on this, but I wanted to get some feedback.
> 
> Happy Hacking,
> Robert
> 

Hi Robert,

Thanks for starting the thread. That's an interesting idea about models being able to say "I don't know" when they're unsure or need more information. Can you elaborate on how you envision this behavior manifesting in practice? For example, would it involve adding a new type of output that indicates uncertainty, or modifying existing models to incorporate some form of self-awareness?

Also, regarding the workflow pipeline idea, I'm curious about what kind of metadata or features you'd want these pipelines to include. Would they need to track things like which model was used for each operation, how many iterations were performed, and what metrics were being tracked? How would this information be passed between models in the pipeline?

Lastly, are there any specific concerns or challenges that you're aware of when it comes to implementing workflows with varying levels of uncertainty or iteration requirements? I'd love to hear about your thoughts on these topics.

Best,
Zelda