Seeking advice on complex AI system architecture (NLP, Claude, parallel flows)

I’m developing a system and could use some expert insights on its architecture and implementation.

Here’s a brief overview of the current flow:

User fills out an intake questionnaire with X questions
System references datasets/examples fr…


This content originally appeared on DEV Community and was authored by Paz Gruberg

I'm developing a system and could use some expert insights on its architecture and implementation.

Here's a brief overview of the current flow:

  1. User fills out an intake questionnaire with X questions
  2. System references datasets/examples from existing system folders
  3. Multiple parallel flows run with Claude API to generate texts based on guidelines and relevant intake responses

Now, I'm grappling with some key issues:

  1. Overall Architecture
  2. Should I use Agentic AI frameworks? If so, which ones?
  3. Or is there a more efficient approach for a system like this?
  4. Data Storage and Management
  5. How to store intake responses for efficient AI flow access?
  6. Is RAG (Retrieval-Augmented Generation) advisable for storage and retrieval? If not, what are the alternatives?
  7. Existing Dataset Integration
  8. What's the most efficient way to incorporate examples from existing folders into the flow?
  9. Better to train a small model on the dataset or integrate it directly?
  10. Parallel Process Optimization
  11. How to efficiently manage and synchronize the parallel flows?

Important notes:

  • Output (generated texts) and number of intake questions remain constant
  • It's a complex flow split into several sub-flows, not a single long prompt


This content originally appeared on DEV Community and was authored by Paz Gruberg


Print Share Comment Cite Upload Translate Updates
APA

Paz Gruberg | Sciencx (2024-08-04T11:45:47+00:00) Seeking advice on complex AI system architecture (NLP, Claude, parallel flows). Retrieved from https://www.scien.cx/2024/08/04/seeking-advice-on-complex-ai-system-architecture-nlp-claude-parallel-flows/

MLA
" » Seeking advice on complex AI system architecture (NLP, Claude, parallel flows)." Paz Gruberg | Sciencx - Sunday August 4, 2024, https://www.scien.cx/2024/08/04/seeking-advice-on-complex-ai-system-architecture-nlp-claude-parallel-flows/
HARVARD
Paz Gruberg | Sciencx Sunday August 4, 2024 » Seeking advice on complex AI system architecture (NLP, Claude, parallel flows)., viewed ,<https://www.scien.cx/2024/08/04/seeking-advice-on-complex-ai-system-architecture-nlp-claude-parallel-flows/>
VANCOUVER
Paz Gruberg | Sciencx - » Seeking advice on complex AI system architecture (NLP, Claude, parallel flows). [Internet]. [Accessed ]. Available from: https://www.scien.cx/2024/08/04/seeking-advice-on-complex-ai-system-architecture-nlp-claude-parallel-flows/
CHICAGO
" » Seeking advice on complex AI system architecture (NLP, Claude, parallel flows)." Paz Gruberg | Sciencx - Accessed . https://www.scien.cx/2024/08/04/seeking-advice-on-complex-ai-system-architecture-nlp-claude-parallel-flows/
IEEE
" » Seeking advice on complex AI system architecture (NLP, Claude, parallel flows)." Paz Gruberg | Sciencx [Online]. Available: https://www.scien.cx/2024/08/04/seeking-advice-on-complex-ai-system-architecture-nlp-claude-parallel-flows/. [Accessed: ]
rf:citation
» Seeking advice on complex AI system architecture (NLP, Claude, parallel flows) | Paz Gruberg | Sciencx | https://www.scien.cx/2024/08/04/seeking-advice-on-complex-ai-system-architecture-nlp-claude-parallel-flows/ |

Please log in to upload a file.



There are no updates yet.
Click the Upload button above to add an update.

You must be logged in to translate posts. Please log in or register.

Related Posts