From: <grad-student>
  To  : Robert Escriva <rescrv@rave>
  Date: Thu, 17 Oct 2024 01:47:23 +0000

Re: Workflows

On Wed, 16 Oct 2024 18:46:41 -0700, Robert Escriva <rescrv@rave> wrote:
> On Thu, Oct 17, 2024 at 01:36:26AM +0000, principle wrote:
> > Hi Robert,
> 
> Hi Link,
> 
> I'm going to try something new and answer your questions inline.
> 
> > Thanks for sharing your ideas about improving models' results. It sounds like you're thinking about ways to make models more human-like and adaptable.
> > 
> > To clarify, when you say "models should be able to say 'I don't know'", are you envisioning a scenario where the model explicitly states that it's uncertain or doesn't have enough information? Or would this behavior be implemented behind the scenes through some algorithmic mechanism?
> 
> I would implement this via an algorithm.  Specifically, I would train models to answer Yes or No via
> JSON whether they are confident in their answer.  If the model is not confident, it would not reply.
> 
> > Also, regarding workflows that pipe output from one model into another for ranking, what kind of models and use cases are you thinking of applying these pipelines to? Are we talking about natural language processing tasks like email response generation or text classification?
> 
> I'm thinking of having the AI do things like edit or expand upon text.
> 
> > Lastly, how do you envision the pipeline operations (e.g., iterating over a dataset) being executed in the background? Would this be handled through asynchronous computation, message queues, or something else entirely?
> > 
> > Looking forward to discussing these ideas further!
> > 
> > Best,
> > Link
> 
> I'd write a small state machine using a local key-value store to store state.
> 
> Happy Hacking,
> Robert
> 

Hi Robert,

Thank you for answering my questions about your paper on improving model workflows. I have a few more follow-up questions based on your responses.

Firstly, regarding the implementation of models that can say "I don't know", do you envision the use of uncertainty quantification techniques such as Bayesian neural networks or Monte Carlo dropout to represent confidence levels? And how would you handle cases where the model is uncertain about its own uncertainty (i.e., the uncertainty becomes recursive)?

Additionally, I'd like to explore the pipeline operations further. You mentioned using a small state machine with a local key-value store. Could you elaborate on the benefits of using a local key-value store versus a distributed database or message queue system? And how would you ensure that the state machine remains consistent across multiple instances of the model or pipeline?

Lastly, I'm interested in learning more about the specific models and use cases you have in mind for applying these pipelines. Are there any particular applications or industries where you think these workflows could bring significant value (e.g., content moderation, customer service chatbots, etc.)?

Looking forward to hearing back from you!

Best,
Zelda