Project Overview

Project Purpose and Objectives

This project aims to facilitate the exploration, analysis, and visualization of embeddings generated from source code files. It focuses on converting code into semantically meaningful vector representations and applying advanced dimensionality reduction and clustering techniques to uncover latent structure and relationships within the codebase. The core objectives include:

• Embedding Computation: Generate high-quality semantic embeddings of source code chunks using pretrained transformer models.

• Semantic Splitting: Split source code files into coherent chunks based on semantic similarity for more granular analysis.

• Dimensionality Reduction: Apply algorithms such as PCA, t-SNE, and UMAP to reduce high-dimensional embeddings into 2D or 3D for visualization.

• Clustering: Implement multiple clustering algorithms (KMeans, AgglomerativeClustering, DBSCAN variants, SpectralClustering) to group similar code chunks or files.

• Interactive Visualization: Present embeddings and clustering results through an interactive web interface allowing exploration and insight discovery.

• Database Integration: Fetch source code metadata and embeddings from a PostgreSQL database and manage data persistently.

The system is designed to support developers and analysts in understanding large codebases by revealing structural and semantic patterns in the code, aiding tasks such as code review, refactoring, and architecture analysis.

Example Workflows and Use Cases

Workflow 1: Generating and Visualizing Embeddings for a Codebase

1. Load Source Code Files
   The system fetches file overviews and metadata from the PostgreSQL database.

2. Semantic Splitting
   Each source code file is split into semantically coherent chunks using the semantic_split function in tools.py, which leverages sentence embeddings and cosine similarity.

3. Embedding Computation
   Each chunk is embedded using a pretrained transformer model (via HuggingFace in tools.py), producing vector representations that capture semantic content.

4. Dimensionality Reduction & Clustering
   Using algorithms like UMAP or t-SNE from visualizer.py or main.py, embeddings are projected into a lower-dimensional space. Clustering algorithms such as HDBSCAN or KMeans group similar chunks.

5. Interactive Visualization
   The clustered and reduced embeddings are visualized in an interactive Plotly scatter plot within a Streamlit app (visualizer.py), where users can hover over points to view metadata or cluster assignment.

Workflow 2: Exploring Existing Embeddings via Web App

1. Upload Embeddings & Metadata
   Users upload pickled embeddings and metadata files through the Streamlit sidebar.

2. Select Reduction and Clustering Methods
   The app allows configuration of reduction (PCA, t-SNE, UMAP) and clustering (various sklearn and HDBSCAN algorithms) parameters interactively.

3. Visual Exploration
   Results update dynamically, presenting visual clusters and data distributions for in-depth investigation.

Stack and Technologies

• Python 3.8+ — Primary programming language for backend logic, data processing, and web app.

• Streamlit — Framework for building the interactive web application interface rapidly.

• Plotly Express — Used for creating rich, interactive scatter plots for embedding visualization.

• Transformers (HuggingFace) — Provides pretrained transformer models and tokenizers for embedding computation.

• PyTorch — Backend for transformer model inference.

• PostgreSQL — Database system storing source code file metadata and embeddings.

• psycopg — PostgreSQL driver for Python to connect and query the database.

• Scikit-learn — Implements dimensionality reduction (PCA, t-SNE), clustering algorithms (KMeans, AgglomerativeClustering, SpectralClustering).

• UMAP-learn — Alternative dimensionality reduction algorithm optimized for preserving local and global data structure.

• HDBSCAN — Density-based clustering algorithm for discovering clusters of varying density.

• NLTK — Natural language toolkit used for sentence tokenization during semantic splitting.

• NumPy — Numerical computing library for array and matrix operations.

**Rationale:** This technology stack balances cutting-edge NLP embedding capabilities with rich clustering and visualization tools. Streamlit and Plotly enable rapid development of interactive frontends. PostgreSQL ensures robust metadata and embedding storage. The use of multiple reduction and clustering methods provides flexibility to adapt to different dataset characteristics.

High-Level Architecture

+----------------------+       +---------------------+       +---------------------+
|                      |       |                     |       |                     |
|    Source Code Files  | ----> |   Semantic Splitting | ----> |  Embedding Computa- |
|  (Codebase + Metadata)|       |  (tools.py)          |       |  tion (tools.py)    |
|                      |       |                     |       |  (transformers +    |
+----------------------+       +---------------------+       |   PyTorch)          |
                                                                  +---------------------+
                                                                       |
                                                                       v
                                                        +-----------------------------+
                                                        | Dimensionality Reduction &   |
                                                        | Clustering (main.py &        |
                                                        | visualizer.py)               |
                                                        | - PCA, t-SNE, UMAP           |
                                                        | - KMeans, Agglomerative,     |
                                                        |   DBSCAN, HDBSCAN, OPTICS    |
                                                        +-----------------------------+
                                                                       |
                                                                       v
                                                        +-----------------------------+
                                                        | Interactive Visualization    |
                                                        | (visualizer.py using Streamlit|
                                                        | and Plotly)                  |
                                                        +-----------------------------+
                                                                       |
                                                                       v
                                                        +-----------------------------+
                                                        | PostgreSQL Database          |
                                                        | - Stores file metadata       |
                                                        | - Stores embeddings          |
                                                        +-----------------------------+

**Component Interactions:**

• Data Ingestion: The backend (main.py) connects to PostgreSQL to retrieve code file metadata.

• Processing: Source code files are split semantically (tools.py) and embedded with transformer models.

• Analysis: Embeddings are dimensionality reduced and clustered using sklearn, UMAP, and HDBSCAN.

• Visualization: visualizer.py powers a Streamlit app that loads embeddings and metadata, runs reduction and clustering, and displays interactive plots.

• Persistence: Embeddings and metadata are serialized and saved as pickle files or stored/retrieved from the database.

Developer Navigation

Frontend Developers Start Here

• File: visualizer.py

• Focus: Streamlit UI components, Plotly visualization setup, user configuration of dimensionality reduction and clustering.

• Skills: Python UI with Streamlit, interactive data visualization with Plotly, integration of machine learning results into user experience.

Backend Developers Focus Here

• File: main.py

• Focus: Database connections (PostgreSQL via psycopg), embedding computation using pretrained transformers, clustering logic, and orchestration of data flow.

• Skills: Database querying, environment variable management, handling model inference with HuggingFace and PyTorch, managing file I/O and serialization.

NLP / Embedding Specialists

• File: tools.py

• Focus: Semantic text splitting based on sentence embeddings, embedding generation using transformer tokenizers and models, cosine similarity calculations.

• Skills: NLP preprocessing, HuggingFace transformers usage, PyTorch tensor operations, similarity metrics.

Data Scientists / Algorithm Developers

• Files: visualizer.py, main.py

• Focus: Experimenting with dimensionality reduction (PCA, t-SNE, UMAP) and clustering algorithms (KMeans, AgglomerativeClustering, DBSCAN variants).

• Skills: Understanding sklearn and UMAP APIs, tuning clustering parameters, interpreting embedding visualizations.

This overview provides a concise yet thorough roadmap for contributors to quickly comprehend project goals, architecture, workflows, and entry points for development. It balances technical depth with clarity suitable for both new and experienced developers.