Audio embeddings are numerical representations of audio data that capture the essential acoustic characteristics of a sound file in a format machines can work with. Instead of storing audio as a waveform — a long series of amplitude values — an embedding compresses everything meaningful about a sound into a compact, fixed-length array of numbers called a vector.
This vector can be stored in a database, compared mathematically against other vectors, and searched at scale. Two sounds that are acoustically similar will produce vectors that are mathematically close to each other — regardless of their filenames, metadata, or format. This is the property that makes audio embeddings the foundation of modern AI-powered audio applications.
This guide explains what audio embeddings are, exactly how they are generated from raw audio, what types of information they encode, and the full range of applications they enable — from speech recognition to music recommendation to semantic SFX search.
What Audio Embeddings Capture
A well-trained audio embedding model encodes far more about a sound than any keyword tag can describe. The vector it produces captures:
- Pitch and tonality — the fundamental frequency and its harmonics. A C major chord and a C minor chord will produce different vectors even if they share the same tempo and instrumentation.
- Timbre — the tonal quality that distinguishes a piano from a guitar playing the same note, or a male voice from a female voice saying the same word.
- Rhythm and tempo — the periodic structure of a sound over time. A 120 BPM techno kick and an 80 BPM hip-hop kick are acoustically different and will produce different vectors.
- Spectral texture — how energy is distributed across the frequency spectrum. A bright, high-frequency synthesizer pad has a completely different spectral profile from a warm, low-frequency bass drone.
- Contextual and semantic meaning — for models like CLAP that are trained with language supervision, the embedding also captures semantic relationships. "Rain on a tin roof" and "heavy rain on metal" will produce similar vectors because the model learned to associate these descriptions with similar acoustic patterns.
How Audio Embeddings Are Generated
Generating an audio embedding from a raw audio file involves four main stages. Understanding each stage helps you choose the right model and diagnose problems when embeddings don't behave as expected.
Stage 1 — Pre-processing
Raw audio comes in many formats, sample rates, bit depths, and channel configurations. Before any feature extraction or neural network inference can happen, the audio must be normalized to a consistent format.
The audio is converted to a standard sample rate — typically 16,000 Hz (16 kHz) for speech models or 44,100 Hz for music models. CLAP operates at 44,100 Hz to preserve musical detail across the full audible frequency range.
Stereo audio is typically mixed down to mono before embedding generation. This ensures consistent input regardless of whether the source was recorded in stereo, mid-side, or mono.
The audio signal is normalized so that loudness differences between files don't dominate the embedding. A quiet version and a loud version of the same recording should produce similar vectors. For audio files in mixed formats, batch converting them to WAV or FLAC on Mac beforehand ensures consistent preprocessing input with no quality loss.
Stage 2 — Segmentation
Neural networks expect inputs of a fixed size, but audio files vary in length from milliseconds to hours. The audio is divided into overlapping windows or frames — short segments of typically 25–50 milliseconds — that are analyzed individually before being aggregated into a single embedding for the full file.
For short audio files (loops, one-shots, short samples), a single window may cover the entire file. For longer audio, AudioVector processes overlapping segments and averages the resulting embeddings to produce a single representative vector for the full duration.
Stage 3 — Feature Extraction
Raw waveforms are not directly fed into most neural networks. Instead, the waveform is first transformed into a feature representation that is more informative for the model to process. The most widely used feature for audio embedding models is the mel spectrogram.
| Feature Type | What It Captures | Typical Use |
|---|---|---|
| Mel Spectrogram | Frequency content over time, scaled to match human hearing perception | Input to most modern audio neural networks including CLAP |
| MFCCs (Mel-Frequency Cepstral Coefficients) |
The shape of the spectral envelope — captures timbre and vowel-like qualities | Traditional speech recognition, legacy audio classification systems |
| Chroma Features | Pitch class content — which notes of the chromatic scale are present | Music key detection, chord recognition, harmonic similarity |
| Spectral Centroid | The "center of mass" of the frequency spectrum — relates to perceived brightness | Timbre description, mood classification |
| Onset Strength | The strength and timing of note or beat onsets over time | Rhythm analysis, tempo estimation |
Modern embedding models like CLAP use the mel spectrogram as input because it simultaneously captures frequency content, time structure, and perceptual scale — giving the neural network rich, multi-dimensional information to work with.
Stage 4 — Neural Network Inference
The extracted features are passed through a trained neural network — the audio encoder — which processes them through multiple learned layers and outputs the final embedding vector. The architecture varies by model:
- CNNs (Convolutional Neural Networks) — treat the spectrogram like an image and apply convolutional filters to detect local patterns in frequency and time. VGGish and PANNs use CNN-based encoders.
- Transformers — process the spectrogram as a sequence using self-attention mechanisms that can capture long-range dependencies across the full duration of the audio. CLAP uses a transformer-based audio encoder (HTSAT — Hierarchical Token-Semantic Audio Transformer).
- RNNs (Recurrent Neural Networks) — process the spectrogram frame by frame, maintaining a hidden state that accumulates context over time. Less common in modern architectures, largely replaced by transformers.
The output of the final encoder layer is the embedding vector. For CLAP, this is a 512-dimensional vector that encodes everything the model learned about the audio's acoustic content and semantic meaning.
How Audio Embeddings Are Stored and Searched
Once generated, audio embeddings are stored in a vector database — a database optimized for storing high-dimensional vectors and performing fast nearest-neighbor searches. To find the most similar audio files to a reference, you compute the distance between the reference embedding and every stored embedding, then return the closest matches.
The most common distance metrics for audio embeddings are:
- Cosine similarity — measures the angle between two vectors. A score of 1.0 = identical; 0.0 = completely dissimilar. The standard for CLAP embeddings.
- Euclidean distance — the straight-line distance between two points in the embedding space. Lower is more similar.
- Dot product — used when embeddings are normalized (equivalent to cosine similarity for unit vectors).
At scale, vector databases use approximate nearest-neighbor (ANN) algorithms — such as HNSW (Hierarchical Navigable Small World) used by Qdrant and Pinecone — to search millions of vectors in milliseconds rather than computing exact distances against every entry.
Applications of Audio Embeddings
Audio embeddings unlock a category of applications that is impossible to build reliably with keyword tags alone.
Music Recommendation
Generate embeddings for every track in a catalog. At playback time, find the N most similar embeddings to the current track and surface them as recommendations. The same approach powers "Similar Artists" features at scale.
Speech Recognition
Embeddings trained on speech data (like Wav2Vec 2.0) capture phoneme structure, speaker identity, and language patterns — forming the foundation of modern ASR systems that generalize across accents and environments.
Sound Classification
Train a classifier on top of pre-computed embeddings to categorize sounds into labels — "dog bark", "car engine", "piano note" — without processing raw audio. The embedding does the heavy lifting; the classifier only needs a small labeled dataset.
SFX Library Search
Vectorize a sound effects library and let editors drag a reference file to find the 20 most acoustically similar sounds in the archive — replacing hours of folder browsing with an instant similarity query.
Duplicate Detection
Find re-encoded, pitch-shifted, or slightly edited duplicates of existing files across a large catalog. Audio embeddings detect acoustic similarity even when filenames and metadata are completely different.
Automatic Clustering
Apply k-means or DBSCAN clustering to a set of audio embeddings to automatically group similar sounds — drum hits, synth pads, vocal samples — without writing a single category label.
Generating Audio Embeddings with AudioVector on Mac
AudioVector is a native macOS application that runs the complete CLAP embedding pipeline locally — audio decoding, mel spectrogram conversion, transformer inference, and JSON export — with no Python environment, no terminal, and no internet connection required.
It is the fastest way to go from a folder of audio files to a set of vector database-ready JSON embeddings on a Mac.
Drag a single audio file or an entire folder into AudioVector. Supported formats: MP3, WAV, FLAC, AIFF, M4A, AAC. Mixed-format folders work in batch — no conversion required.
AudioVector decodes the audio, converts it to a mel spectrogram, and passes it through the bundled CLAP audio encoder. All computation happens on your Mac — Apple Silicon Macs use the Neural Engine for significantly faster throughput.
AudioVector writes one JSON file per audio source containing the filename, duration, and the full 512-dimensional CLAP embedding vector. The output is immediately compatible with Pinecone, Qdrant, Postgres pgvector, Weaviate, and Chroma.
Upsert the exported JSONs into your vector database. Your audio catalog is now searchable by acoustic similarity — any query against the database returns the most acoustically similar files to the reference embedding.
What the JSON output contains
| Field | Type | Description |
|---|---|---|
| filename | String | The original audio file name — used as the record identifier in your database. |
| duration_seconds | Float | Total audio duration in seconds — useful as a metadata filter in vector queries. |
| embedding | Array[512] of floats | The 512-dimensional CLAP audio vector — the acoustic fingerprint of the file. |
Audio Embeddings vs. Traditional Audio Features: Key Differences
MFCCs, chroma features, and spectral centroids are hand-crafted features — mathematical transformations designed by audio researchers to capture specific acoustic properties. They are deterministic, interpretable, and fast to compute.
Audio embeddings are learned representations — a neural network processed millions of examples and discovered which feature combinations are most useful for the task at hand. The result is a denser, more expressive representation that captures relationships no hand-crafted feature can encode.
| Hand-crafted Features (MFCCs) | Neural Embeddings (CLAP) | |
|---|---|---|
| Design | Designed by researchers based on acoustic theory | Learned from millions of audio examples by a neural network |
| Expressiveness | Captures specific acoustic properties (e.g., timbre shape) | Captures complex relationships across all acoustic dimensions simultaneously |
| Semantic understanding | None — purely acoustic signal processing | Yes (for CLAP) — aligns acoustic content with natural language meaning |
| Generalization | Fixed — only captures what it was designed to capture | Zero-shot — generalizes to audio types not seen during training |
| Similarity search quality | Good for specific tasks (e.g., speaker ID with MFCCs) | Superior for general-purpose cross-genre audio similarity |