graph_rag_server

RAG System

A production-ready Retrieval-Augmented Generation (RAG) system built with FastAPI, featuring clean architecture, comprehensive testing, and CI/CD integration.

🚀 Features

• Clean Architecture: Domain-driven design with clear separation of concerns
• Multiple Vector Databases: Support for Infinity and Elasticsearch
• Document Processing: PDF, image, and text document parsing
• Hybrid Search: Combined vector and full-text search capabilities
• Knowledge Base Management: Organize documents into knowledge bases
• Comprehensive Testing: Unit, integration, and end-to-end tests
• CI/CD Pipeline: Automated testing and deployment with GitHub Actions
• High Test Coverage: Maintained at 80%+ coverage
• Structured Logging: JSON-formatted logs with request tracking
• Type Safety: Full type hints and mypy validation

📋 Table of Contents

• Quick Start
• Installation
• Configuration
• Running Tests
• CI/CD Pipeline
• Architecture
• Documentation
• Contributing

🏃 Quick Start

# Clone the repository
git clone https://github.com/YOUR_USERNAME/rag-system.git
cd rag-system

# Install dependencies
pip install -r requirements.txt

# Set up environment variables
cp .env.example .env
# Edit .env with your configuration

# Run tests
pytest

# Start the application
python main.py

📦 Installation

Prerequisites

• Python 3.11 or 3.12
• pip (Python package manager)
• Git

Development Setup

1. Create a virtual environment:

   python -m venv venv
   source venv/bin/activate  # On Windows: venv\Scripts\activate
   

2. Install dependencies:

   pip install -r requirements.txt
   

3. Install development dependencies:

   pip install pytest pytest-asyncio pytest-cov hypothesis httpx
   pip install flake8 black isort mypy  # For linting
   pip install safety bandit  # For security scanning
   

4. Set up pre-commit hooks (optional):

   pip install pre-commit
   pre-commit install
   

⚙️ Configuration

Configuration is managed through environment variables and the .env file.

Environment Variables

Copy .env.example to .env and configure:

# Application
APP_NAME=RAG System
DEBUG=false

# Database
DB_HOST=localhost
DB_PORT=5432
DB_DATABASE=rag_system
DB_USERNAME=your_username
DB_PASSWORD=your_password

# Vector Database (infinity or elasticsearch)
VECTOR_DB_TYPE=infinity

# Infinity Configuration
INFINITY_HOST=localhost
INFINITY_PORT=23817

# Elasticsearch Configuration
ES_HOST=localhost
ES_PORT=9200

See Configuration Guide for detailed configuration options.

🧪 Running Tests

Quick Test Commands

# Run all tests
pytest

# Run with coverage
pytest --cov=src --cov-report=html

# Run specific test types
pytest -m unit          # Unit tests only
pytest -m integration   # Integration tests only
pytest -m e2e          # End-to-end tests only

# Run fast tests only
pytest -m "not slow"

Using the Test Runner Script

# Run all tests with CI configuration
python scripts/run_tests.py

# Run with coverage report
python scripts/run_tests.py --coverage --html

# Run all checks (tests + lint + security)
python scripts/run_tests.py --all

# Run in parallel
python scripts/run_tests.py --parallel

Performance Testing

# Run simulated performance test (quick)
python scripts/benchmark_simple.py

# Run real performance test (requires running server)
python main.py  # In one terminal
python scripts/benchmark.py  # In another terminal

See Performance Guide for detailed performance testing documentation.

Test Organization

Tests are organized by type:

• tests/unit/: Fast, isolated unit tests
• tests/integration/: Component interaction tests
• tests/e2e/: Complete workflow tests

See Testing Guide for detailed testing documentation.

🔄 CI/CD Pipeline

The project uses GitHub Actions for continuous integration and deployment.

Pipeline Jobs

1. Test: Runs all tests with coverage measurement
2. Lint: Code quality checks (flake8, black, isort, mypy)
3. Security: Security scans (safety, bandit)
4. Build Status: Aggregates results

Coverage Reporting

Coverage reports are automatically uploaded to Codecov on every push.

Setting Up CI/CD

1. Enable GitHub Actions: Already configured in .github/workflows/test.yml

2. Set up Codecov:

   • Sign up at codecov.io
   • Add your repository
   • Add CODECOV_TOKEN to GitHub Secrets

See CI/CD Guide for detailed pipeline documentation.

🏗️ Architecture

The system follows a clean, layered architecture:

┌─────────────────────────────────────┐
│      Presentation Layer             │  FastAPI routes, middleware
├─────────────────────────────────────┤
│      Application Layer              │  Use cases, handlers, DTOs
├─────────────────────────────────────┤
│      Domain Layer                   │  Entities, value objects, services
├─────────────────────────────────────┤
│      Infrastructure Layer           │  Databases, external services
└─────────────────────────────────────┘

Directory Structure

rag_system/
├── src/
│   ├── domain/              # Domain layer (business logic)
│   ├── application/         # Application layer (use cases)
│   ├── infrastructure/      # Infrastructure layer (databases, APIs)
│   ├── presentation/        # Presentation layer (API routes)
│   ├── config/             # Configuration management
│   └── shared/             # Shared utilities
├── tests/
│   ├── unit/               # Unit tests
│   ├── integration/        # Integration tests
│   └── e2e/               # End-to-end tests
├── docs/                   # Documentation
├── scripts/               # Utility scripts
└── docker/                # Docker configuration

See Architecture Documentation for detailed design information.

📚 Documentation

• Architecture Documentation - System design and architecture
• Configuration Guide - Environment and configuration setup
• Directory Structure - Project organization
• Logging Guide - Logging configuration and usage
• CI/CD Guide - Pipeline and deployment
• Testing Guide - Testing practices and commands
• Performance Guide - Performance testing and optimization

🤝 Contributing

We welcome contributions! Please follow these guidelines:

Development Workflow

1. Fork the repository
2. Create a feature branch: git checkout -b feature/your-feature
3. Write tests: Ensure your code is well-tested
4. Run tests: python scripts/run_tests.py --all
5. Commit changes: Use clear, descriptive commit messages
6. Push to your fork: git push origin feature/your-feature
7. Create a Pull Request

Code Standards

• Style: Follow PEP 8 (enforced by flake8 and black)
• Type Hints: Use type hints for all functions
• Documentation: Add docstrings to all public APIs
• Testing: Maintain 80%+ test coverage
• Commits: Use conventional commit messages

Running Quality Checks

# Format code
black src tests
isort src tests

# Lint code
flake8 src tests

# Type check
mypy src

# Run all checks
python scripts/run_tests.py --all

📊 Test Coverage

Current coverage by layer:

Layer	Target	Current
Domain Layer	90%	TBD
Application Layer	85%	TBD
Infrastructure Layer	70%	TBD
Presentation Layer	75%	TBD
Overall	80%	TBD

🔒 Security

• Dependency Scanning: Automated with safety
• Code Scanning: Automated with bandit
• Security Reports: Available in CI/CD artifacts

Report security vulnerabilities to: security@example.com

📝 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• FastAPI for the excellent web framework
• pytest for the comprehensive testing framework
• Codecov for coverage reporting
• GitHub Actions for CI/CD automation

📞 Support

• Documentation: docs/
• Issues: GitHub Issues
• Discussions: GitHub Discussions

🗺️ Roadmap

• Phase 1: Infrastructure setup
• Phase 2: Domain layer refactoring
• Phase 3: Application layer refactoring
• Phase 4: Infrastructure layer migration
• Phase 5: Presentation layer migration
• Phase 6: Documentation and cleanup

See Implementation Plan for detailed roadmap.

---

Built with ❤️ using Clean Architecture principles