Audio-Classifier/.claude-todo.md

Audio Classifier - Technical Implementation TODO

Phase 1: Project Structure & Dependencies

1.1 Root structure

• Create root .gitignore
• Create root README.md with setup instructions
• Create docker-compose.yml (PostgreSQL + pgvector)
• Create .env.example

1.2 Backend structure (Python/FastAPI)

• Create backend/ directory
• Create backend/requirements.txt:
  • fastapi==0.109.0
  • uvicorn[standard]==0.27.0
  • sqlalchemy==2.0.25
  • psycopg2-binary==2.9.9
  • pgvector==0.2.4
  • librosa==0.10.1
  • essentia-tensorflow==2.1b6.dev1110
  • pydantic==2.5.3
  • pydantic-settings==2.1.0
  • python-multipart==0.0.6
  • mutagen==1.47.0
  • numpy==1.24.3
  • scipy==1.11.4
• Create backend/pyproject.toml (optional, for poetry users)
• Create backend/.env.example
• Create backend/Dockerfile
• Create backend/src/__init__.py

1.3 Backend core modules structure

• backend/src/core/__init__.py
• backend/src/core/audio_processor.py - librosa feature extraction
• backend/src/core/essentia_classifier.py - Essentia models (genre/mood/instruments)
• backend/src/core/analyzer.py - Main orchestrator
• backend/src/core/file_scanner.py - Recursive folder scanning
• backend/src/core/waveform_generator.py - Peaks extraction for visualization

1.4 Backend database modules

• backend/src/models/__init__.py
• backend/src/models/database.py - SQLAlchemy engine + session
• backend/src/models/schema.py - SQLAlchemy models (AudioTrack)
• backend/src/models/crud.py - CRUD operations
• backend/src/alembic/ - Migration setup
• backend/src/alembic/versions/001_initial_schema.py - CREATE TABLE + pgvector extension

1.5 Backend API structure

• backend/src/api/__init__.py
• backend/src/api/main.py - FastAPI app + CORS + startup/shutdown events
• backend/src/api/routes/__init__.py
• backend/src/api/routes/tracks.py - GET /tracks, GET /tracks/{id}, DELETE /tracks/{id}
• backend/src/api/routes/search.py - GET /search?q=...&genre=...&mood=...
• backend/src/api/routes/analyze.py - POST /analyze/folder, GET /analyze/status/{job_id}
• backend/src/api/routes/audio.py - GET /audio/stream/{id}, GET /audio/download/{id}, GET /audio/waveform/{id}
• backend/src/api/routes/similar.py - GET /tracks/{id}/similar
• backend/src/api/routes/stats.py - GET /stats (total tracks, genres distribution)

1.6 Backend utils

• backend/src/utils/__init__.py
• backend/src/utils/config.py - Pydantic Settings for env vars
• backend/src/utils/logging.py - Logging setup
• backend/src/utils/validators.py - Audio file validation

1.7 Frontend structure (Next.js 14)

• npx create-next-app@latest frontend --typescript --tailwind --app --no-src-dir
• cd frontend && npm install
• Install deps: shadcn-ui, @tanstack/react-query, zustand, axios, lucide-react, recharts
• npx shadcn-ui@latest init
• Add shadcn components: button, input, slider, select, card, dialog, progress, toast

1.8 Frontend structure details

• frontend/app/layout.tsx - Root layout with QueryClientProvider
• frontend/app/page.tsx - Main library view
• frontend/app/tracks/[id]/page.tsx - Track detail page
• frontend/components/SearchBar.tsx
• frontend/components/FilterPanel.tsx
• frontend/components/TrackCard.tsx
• frontend/components/TrackDetails.tsx
• frontend/components/AudioPlayer.tsx
• frontend/components/WaveformDisplay.tsx
• frontend/components/BatchScanner.tsx
• frontend/components/SimilarTracks.tsx
• frontend/lib/api.ts - Axios client with base URL
• frontend/lib/types.ts - TypeScript interfaces
• frontend/hooks/useSearch.ts
• frontend/hooks/useTracks.ts
• frontend/hooks/useAudioPlayer.ts
• frontend/.env.local.example

---

Phase 2: Database Schema & Migrations

2.1 PostgreSQL setup

• docker-compose.yml: service postgres with pgvector image pgvector/pgvector:pg16
• Expose port 5432
• Volume for persistence: postgres_data:/var/lib/postgresql/data
• Init script: backend/init-db.sql with CREATE EXTENSION vector

2.2 SQLAlchemy models

• Define AudioTrack model in schema.py:
  • id: UUID (PK)
  • filepath: String (unique, indexed)
  • filename: String
  • duration_seconds: Float
  • file_size_bytes: Integer
  • format: String (mp3/wav)
  • analyzed_at: DateTime
  • tempo_bpm: Float
  • key: String
  • time_signature: String
  • energy: Float
  • danceability: Float
  • valence: Float
  • loudness_lufs: Float
  • spectral_centroid: Float
  • zero_crossing_rate: Float
  • genre_primary: String (indexed)
  • genre_secondary: ARRAY[String]
  • genre_confidence: Float
  • mood_primary: String (indexed)
  • mood_secondary: ARRAY[String]
  • mood_arousal: Float
  • mood_valence: Float
  • instruments: ARRAY[String]
  • has_vocals: Boolean
  • vocal_gender: String (nullable)
  • embedding: Vector(512) (nullable, for future CLAP)
  • embedding_model: String (nullable)
  • metadata: JSON
• Create indexes: filepath, genre_primary, mood_primary, tempo_bpm

2.3 Alembic migrations

• alembic init backend/src/alembic
• Configure alembic.ini with DB URL
• Create initial migration with schema above
• Add pgvector extension in migration

---

Phase 3: Core Audio Processing

3.1 audio_processor.py - Librosa feature extraction

• Function load_audio(filepath: str) -> Tuple[np.ndarray, int]
• Function extract_tempo(y, sr) -> float - librosa.beat.tempo
• Function extract_key(y, sr) -> str - librosa.feature.chroma_cqt + key detection
• Function extract_spectral_features(y, sr) -> dict:
  • spectral_centroid
  • zero_crossing_rate
  • spectral_rolloff
  • spectral_bandwidth
• Function extract_mfcc(y, sr) -> np.ndarray
• Function extract_chroma(y, sr) -> np.ndarray
• Function extract_energy(y, sr) -> float - RMS energy
• Function extract_all_features(filepath: str) -> dict - orchestrator

3.2 essentia_classifier.py - Essentia TensorFlow models

• Download Essentia models (mtg-jamendo):
  • genre: https://essentia.upf.edu/models/classification-heads/mtg_jamendo_genre/mtg_jamendo_genre-discogs-effnet-1.pb
  • mood: https://essentia.upf.edu/models/classification-heads/mtg_jamendo_moodtheme/mtg_jamendo_moodtheme-discogs-effnet-1.pb
  • instrument: https://essentia.upf.edu/models/classification-heads/mtg_jamendo_instrument/mtg_jamendo_instrument-discogs-effnet-1.pb
• Store models in backend/models/ directory
• Class EssentiaClassifier:
  • __init__(): load models
  • predict_genre(audio_path: str) -> dict: returns {primary, secondary[], confidence}
  • predict_mood(audio_path: str) -> dict: returns {primary, secondary[], arousal, valence}
  • predict_instruments(audio_path: str) -> List[dict]: returns [{name, confidence}, ...]
• Add model metadata files (class labels) in JSON

3.3 waveform_generator.py

• Function generate_peaks(filepath: str, num_peaks: int = 800) -> List[float]
  • Load audio with librosa
  • Downsample to num_peaks points
  • Return normalized amplitude values
• Cache peaks in JSON file next to audio (optional)

3.4 file_scanner.py

• Function scan_folder(path: str, recursive: bool = True) -> List[str]
  • Walk directory tree
  • Filter by extensions: .mp3, .wav, .flac, .m4a, .ogg
  • Return list of absolute paths
• Function get_file_metadata(filepath: str) -> dict
  • Use mutagen for ID3 tags
  • Return: filename, size, format

3.5 analyzer.py - Main orchestrator

• Class AudioAnalyzer:
  • __init__()
  • analyze_file(filepath: str) -> AudioAnalysis:
    1. Validate file exists and is audio
    2. Extract features (audio_processor)
    3. Classify genre/mood/instruments (essentia_classifier)
    4. Get file metadata (file_scanner)
    5. Return structured AudioAnalysis object
  • analyze_folder(path: str, recursive: bool, progress_callback) -> List[AudioAnalysis]:
    • Scan folder
    • Parallel processing with ThreadPoolExecutor (num_workers=4)
    • Progress updates
• Pydantic model AudioAnalysis matching JSON schema from architecture

---

Phase 4: Database CRUD Operations

4.1 crud.py - CRUD functions

• create_track(session, analysis: AudioAnalysis) -> AudioTrack
• get_track_by_id(session, track_id: UUID) -> Optional[AudioTrack]
• get_track_by_filepath(session, filepath: str) -> Optional[AudioTrack]
• get_tracks(session, skip: int, limit: int, filters: dict) -> List[AudioTrack]
  • Support filters: genre, mood, bpm_min, bpm_max, energy_min, energy_max, has_vocals
• search_tracks(session, query: str, filters: dict, limit: int) -> List[AudioTrack]
  • Full-text search on: genre_primary, mood_primary, instruments, filename
  • Combined with filters
• get_similar_tracks(session, track_id: UUID, limit: int) -> List[AudioTrack]
  • If embeddings exist: vector similarity with pgvector
  • Fallback: similar genre + mood + BPM range
• delete_track(session, track_id: UUID) -> bool
• get_stats(session) -> dict
  • Total tracks
  • Genres distribution
  • Moods distribution
  • Average BPM
  • Total duration

---

Phase 5: FastAPI Backend Implementation

5.1 config.py - Settings

• class Settings(BaseSettings):
  • DATABASE_URL: str
  • CORS_ORIGINS: List[str]
  • ANALYSIS_USE_CLAP: bool = False
  • ANALYSIS_NUM_WORKERS: int = 4
  • ESSENTIA_MODELS_PATH: str
  • AUDIO_LIBRARY_PATH: str (optional default scan path)
• Load from .env

5.2 main.py - FastAPI app

• Create FastAPI app with metadata (title, version, description)
• Add CORS middleware (allow frontend origin)
• Add startup event: init DB engine, load Essentia models
• Add shutdown event: cleanup
• Include routers from routes/
• Health check endpoint: GET /health

5.3 routes/tracks.py

• GET /api/tracks:
  • Query params: skip, limit, genre, mood, bpm_min, bpm_max, energy_min, energy_max, has_vocals, sort_by
  • Return paginated list of tracks
  • Include total count
• GET /api/tracks/{track_id}:
  • Return full track details
  • 404 if not found
• DELETE /api/tracks/{track_id}:
  • Soft delete or hard delete (remove from DB only, keep file)
  • Return success

5.4 routes/search.py

• GET /api/search:
  • Query params: q (search query), genre, mood, bpm_min, bpm_max, limit
  • Full-text search + filters
  • Return matching tracks

5.5 routes/audio.py

• GET /api/audio/stream/{track_id}:
  • Get track from DB
  • Return FileResponse with media_type audio/mpeg
  • Support Range requests for seeking (Accept-Ranges: bytes)
  • headers: Content-Disposition: inline
• GET /api/audio/download/{track_id}:
  • Same as stream but Content-Disposition: attachment
• GET /api/audio/waveform/{track_id}:
  • Get track from DB
  • Generate or load cached peaks (waveform_generator)
  • Return JSON: {peaks: [], duration: float}

5.6 routes/analyze.py

• POST /api/analyze/folder:
  • Body: {path: str, recursive: bool}
  • Validate path exists
  • Start background job (asyncio Task or Celery)
  • Return job_id
• GET /api/analyze/status/{job_id}:
  • Return job status: {status: "pending|running|completed|failed", progress: int, total: int, errors: []}
• Background worker implementation:
  • Scan folder
  • For each file: analyze, save to DB (skip if already exists by filepath)
  • Update job status
  • Store job state in-memory dict or Redis

5.7 routes/similar.py

• GET /api/tracks/{track_id}/similar:
  • Query params: limit (default 10)
  • Get similar tracks (CRUD function)
  • Return list of tracks

5.8 routes/stats.py

• GET /api/stats:
  • Get stats (CRUD function)
  • Return JSON with counts, distributions

---

Phase 6: Frontend Implementation

6.1 API client (lib/api.ts)

• Create axios instance with baseURL from env var (NEXT_PUBLIC_API_URL)
• API functions:
  • getTracks(params: FilterParams): Promise<{tracks: Track[], total: number}>
  • getTrack(id: string): Promise<Track>
  • deleteTrack(id: string): Promise<void>
  • searchTracks(query: string, filters: FilterParams): Promise<Track[]>
  • getSimilarTracks(id: string, limit: number): Promise<Track[]>
  • analyzeFolder(path: string, recursive: boolean): Promise<{jobId: string}>
  • getAnalyzeStatus(jobId: string): Promise<JobStatus>
  • getStats(): Promise<Stats>

6.2 TypeScript types (lib/types.ts)

• interface Track matching AudioTrack model
• interface FilterParams
• interface JobStatus
• interface Stats

6.3 Hooks

• hooks/useTracks.ts:
  • useQuery for fetching tracks with filters
  • Pagination state
  • Mutation for delete
• hooks/useSearch.ts:
  • Debounced search query
  • Combined filters state
• hooks/useAudioPlayer.ts:
  • Current track state
  • Play/pause/seek controls
  • Volume control
  • Queue management (optional)

6.4 Components - UI primitives (shadcn)

• Install shadcn components: button, input, slider, select, card, dialog, badge, progress, toast, dropdown-menu, tabs

6.5 SearchBar.tsx

• Input with search icon
• Debounced onChange (300ms)
• Clear button
• Optional: suggestions dropdown

6.6 FilterPanel.tsx

• Genre multi-select (fetch available genres from API or hardcode)
• Mood multi-select
• BPM range slider (min/max)
• Energy range slider
• Has vocals checkbox
• Sort by dropdown (Latest, BPM, Duration, Name)
• Clear all filters button

6.7 TrackCard.tsx

• Props: track: Track, onPlay, onDelete
• Display: filename, duration, BPM, genre, mood, instruments (badges)
• Inline AudioPlayer component
• Buttons: Play, Download, Similar, Details
• Hover effects

6.8 AudioPlayer.tsx

• Props: trackId, filename, duration
• HTML5 audio element with ref
• WaveformDisplay child component
• Progress slider (seek support)
• Play/Pause button
• Volume slider with icon
• Time display (current / total)
• Download button (calls /api/audio/download/{id})

6.9 WaveformDisplay.tsx

• Props: trackId, currentTime, duration
• Fetch peaks from /api/audio/waveform/{id}
• Canvas rendering:
  • Draw bars for each peak
  • Color played portion differently (blue vs gray)
  • Click to seek
• Loading state while fetching peaks

6.10 TrackDetails.tsx (Modal/Dialog)

• Props: trackId, open, onClose
• Fetch full track details
• Display all metadata in organized sections:
  • Audio info: duration, format, file size
  • Musical features: tempo, key, time signature, energy, danceability, valence
  • Classification: genre (primary + secondary), mood (primary + secondary + arousal/valence), instruments
  • Spectral features: spectral centroid, zero crossing rate, loudness
• Similar tracks section (preview)
• Download button

6.11 SimilarTracks.tsx

• Props: trackId, limit
• Fetch similar tracks
• Display as list of mini TrackCards
• Click to navigate or play

6.12 BatchScanner.tsx

• Input for folder path
• Recursive checkbox
• Scan button
• Progress bar (poll /api/analyze/status/{jobId})
• Status messages (pending, running X/Y, completed, errors)
• Error list if any

6.13 Main page (app/page.tsx)

• SearchBar at top
• FilterPanel in sidebar or collapsible
• BatchScanner in header or dedicated section
• TrackCard grid/list
• Pagination controls (Load More or page numbers)
• Total tracks count
• Loading states
• Empty state if no tracks

6.14 Track detail page (app/tracks/[id]/page.tsx)

• Fetch track by ID
• Large AudioPlayer
• Full metadata display (similar to TrackDetails modal)
• SimilarTracks section
• Back to library button

6.15 Layout (app/layout.tsx)

• QueryClientProvider setup
• Toast provider (for notifications)
• Global styles
• Header with app title and nav

---

Phase 7: Docker & Deployment

7.1 docker-compose.yml

• Service: postgres
  • image: pgvector/pgvector:pg16
  • environment: POSTGRES_USER, POSTGRES_PASSWORD, POSTGRES_DB
  • ports: 5432:5432
  • volumes: postgres_data, init-db.sql
• Service: backend
  • build: ./backend
  • depends_on: postgres
  • environment: DATABASE_URL
  • ports: 8000:8000
  • volumes: audio files mount (read-only)
• Service: frontend (optional, or dev mode only)
  • build: ./frontend
  • ports: 3000:3000
  • environment: NEXT_PUBLIC_API_URL=http://localhost:8000

7.2 Backend Dockerfile

• FROM python:3.11-slim
• Install system deps: ffmpeg, libsndfile1
• COPY requirements.txt
• RUN pip install -r requirements.txt
• COPY src/
• Download Essentia models during build or on startup
• CMD: uvicorn src.api.main:app --host 0.0.0.0 --port 8000

7.3 Frontend Dockerfile (production build)

• FROM node:20-alpine
• COPY package.json, package-lock.json
• RUN npm ci
• COPY app/, components/, lib/, hooks/, public/
• RUN npm run build
• CMD: npm start

---

Phase 8: Documentation & Scripts

8.1 Root README.md

• Project description
• Features list
• Tech stack
• Prerequisites (Docker, Node, Python)
• Quick start:
  • Clone repo
  • Copy .env.example to .env
  • docker-compose up
  • Access frontend at localhost:3000
• Development setup
• API documentation link (FastAPI /docs)
• Architecture diagram (optional)

8.2 Backend README.md

• Setup instructions
• Environment variables documentation
• Essentia models download instructions
• API endpoints list
• Database schema
• Running migrations

8.3 Frontend README.md

• Setup instructions
• Environment variables
• Available scripts (dev, build, start)
• Component structure

8.4 Scripts

• scripts/download-essentia-models.sh - Download Essentia models
• scripts/init-db.sh - Run migrations
• backend/src/cli.py - CLI for manual analysis (optional)

---

Phase 9: Testing & Validation

9.1 Backend tests (optional but recommended)

• Test audio_processor.extract_all_features with sample file
• Test essentia_classifier with sample file
• Test CRUD operations
• Test API endpoints with pytest + httpx

9.2 Frontend tests (optional)

• Test API client functions
• Test hooks
• Component tests with React Testing Library

9.3 Integration test

• Full flow: analyze folder -> save to DB -> search -> play -> download

---

Phase 10: Optimizations & Polish

10.1 Performance

• Add database indexes
• Cache waveform peaks
• Optimize audio loading (lazy loading for large libraries)
• Add compression for API responses

10.2 UX improvements

• Loading skeletons
• Error boundaries
• Toast notifications for actions
• Keyboard shortcuts (space to play/pause, arrows to seek)
• Dark mode support

10.3 Backend improvements

• Rate limiting
• Request validation with Pydantic
• Logging (structured logs)
• Error handling middleware

---

Implementation order priority

1. Phase 2 (Database) - Foundation
2. Phase 3 (Audio processing) - Core logic
3. Phase 4 (CRUD) - Data layer
4. Phase 5.1-5.2 (FastAPI setup) - API foundation
5. Phase 5.3-5.8 (API routes) - Complete backend
6. Phase 6.1-6.3 (Frontend setup + API client + hooks) - Frontend foundation
7. Phase 6.4-6.12 (Components) - UI implementation
8. Phase 6.13-6.15 (Pages) - Complete frontend
9. Phase 7 (Docker) - Deployment
10. Phase 8 (Documentation) - Final polish

---

Notes for implementation

• Use type hints everywhere in Python
• Use TypeScript strict mode in frontend
• Handle errors gracefully (try/catch, proper HTTP status codes)
• Add logging at key points (file analysis start/end, DB operations)
• Validate file paths (security: prevent path traversal)
• Consider file locking for concurrent analysis
• Add progress updates for long operations
• Use environment variables for all config
• Keep audio files outside Docker volumes for performance
• Consider caching Essentia predictions (expensive)
• Add retry logic for failed analyses
• Support cancellation for long-running jobs

Files to download/prepare before starting

1. Essentia models (3 files):
   • mtg_jamendo_genre-discogs-effnet-1.pb
   • mtg_jamendo_moodtheme-discogs-effnet-1.pb
   • mtg_jamendo_instrument-discogs-effnet-1.pb
2. Class labels JSON for each model
3. Sample audio files for testing

External dependencies verification

• librosa: check version compatibility with numpy
• essentia-tensorflow: verify CPU-only build works
• pgvector: verify PostgreSQL extension installation
• FFmpeg: required by librosa for audio decoding

Security considerations

• Validate all file paths (no ../ traversal)
• Sanitize user input in search queries
• Rate limit API endpoints
• CORS: whitelist frontend origin only
• Don't expose full filesystem paths in API responses
• Consider adding authentication later (JWT)

Future enhancements (not in current scope)

• CLAP embeddings for semantic search
• Batch export to CSV/JSON
• Playlist creation
• Audio trimming/preview segments
• Duplicate detection (audio fingerprinting)
• Tag editing (write back to files)
• Multi-user support with authentication
• WebSocket for real-time analysis progress
• Audio visualization (spectrogram, chromagram)