Knowledge Bases

Knowledge Bases are a critical component of workflows in Lleverage, enabling the storage and retrieval of vector embeddings for efficient AI-driven search and retrieval tasks.

• Automatic Vectorization: When you upload documents—such as PDFs, CSVs, Word documents, or PowerPoint presentations—Lleverage automatically creates vector embeddings for these files using best-in-class embeddings model selected specifically for your use case. This allows for powerful similarity search and retrieval based on the content of your documents.

• Supported Formats: You can upload various document types, including:

  • PDFs

  • CSVs

  • Word documents (DOCX)

  • PowerPoint presentations (PPTX)

• Preprocessing with Unstructured.io: Before storing vectors, Lleverage leverages the capabilities of Unstructured.io, a best-in-class platform for document preprocessing. This ensures that documents are properly cleaned, chunked, and transformed into a format suitable for creating embeddings. Additionally, Unstructured.io generates summaries, making your document content ready for retrieval-augmented generation (RAG) workflows.

1. Document Upload: You can upload your documents (e.g., PDFs, CSVs, DOCX, PPTX) to Knowledge Base within your project.

2. Preprocessing and Chunking: Using Unstructured.io, documents are automatically cleaned, transformed, and split into smaller, manageable chunks to optimize the embedding process.

3. Embedding Generation: Lleverage selects the appropriate model to create vector representations of your document chunks. These embeddings capture the semantic meaning of the text, enabling advanced similarity searches.

4. Storage: The resulting vectors are stored in your personal Weaviate Vector Database Collection, where they can be queried for similarity search or used in workflows to retrieve relevant documents based on input queries.

Use Cases for Knowledge Bases

• Semantic Search: Query your Knowledge Base using text-based input to find documents or document segments that are semantically similar to the query.

• Retrieval-Augmented Generation (RAG): Use your knowledge base to retrieve relevant information during a workflow execution and integrate it into responses from large language models (LLMs).

• Data Organization: Automatically structure large amounts of unstructured data by transforming them into searchable, vectorized content, making it easier to retrieve insights from vast document collections.

Knowledge Bases integrate seamlessly into your Lleverage workflows. By connecting the Search Knowledge node, you can query your vectorized documents based on input text and retrieve relevant results, which can then be passed into downstream nodes such as LLMs or prompts.

Knowledge Bases are a core component of Lleverage workflows, enabling intelligent document processing and semantic search capabilities. Through an intuitive node in the workflow builder, you can easily upload and manage documents that are automatically processed and made searchable using advanced AI technology.

Key Features

• Automatic Document Processing: Upload various document types including PDFs, Word documents, Excel spreadsheets, and PowerPoint presentations.

• Smart Chunking: Documents are processed using an intelligent AI pipeline that analyzes content structure and determines the most effective chunking strategy, ensuring optimal retrieval results.

• Built-in Vector Store: Documents are automatically vectorized and stored in a Weaviate vector store, enabling powerful semantic search capabilities without any additional configuration.

• Unstructured Data Processing: Leverages Unstructured.io for robust document preprocessing, ensuring high-quality text extraction across different document formats.

Using Knowledge Base in Workflows

1. Adding Documents:

   • Drag the searchKnowledge node into your workflow

   • Upload documents directly through the node interface

   • Documents are automatically processed and made searchable

2. Searching Content:

   • Use natural language queries to search across your documents (e.g. by adding a variable that is called 'Query' and using that variable in the searchKnowledge node

   • Results are ranked by semantic relevance

   • Retrieved content can be seamlessly integrated with other workflow nodes

3. Integration with AI Models:

   • Combine Knowledge Base search results with LLM nodes for enhanced content generation

   • Create workflows that leverage both document knowledge and AI capabilities

Best Practices

• Organize related documents within the same Knowledge node for better context

• Use specific queries to retrieve relevant information

• Consider document structure when designing your workflows

• Test searches with different query formulations to optimize retrieval

Supported Document Types

20 different document types, including:

• PDFs

• Word documents (DOCX)

• Excel spreadsheets (XLSX)

• PowerPoint presentations (PPTX)

• Plain text files (TXT)

Note: Vector Stores functionality is being deprecated in favor of the new Knowledge Base system. Existing Vector Store implementations will continue to function but are recommended to migrate to the new Knowledge Base system.