Vector Quantization

The level, Large Language Models are a class of neural network architectures designed to understand, generate, and manipulate human language at an unprecedented scale. To define them precisely, one must dissect the three constituent words of their name. "Large" refers not merely to the volume of text data on which they are trained—often petabytes of books, articles, websites, and code—but also to the number of parameters they contain. Parameters are the internal, adjustable weights within the neural network that store learned knowledge. If a small language model might have millions of parameters, a large language model typically operates in the billions or even trillions. For context, GPT-3 (2020) had 175 billion parameters, while more recent models like GPT-4 (reportedly over 1.7 trillion parameters) and Google’s PaLM (540 billion parameters) represent orders-of-magnitude leaps in scale.

A Large Language Model (LLM) is a type of neural network trained on vast amounts of text data to understand, generate, and manipulate human language with a degree of fluency and coherence that was unimaginable just a decade ago. At their core, LLMs are probabilistic systems. They do not “think” or “understand” in the human sense; rather, they learn statistical patterns—distributions of words, phrases, sentences, and even broader discourse structures—from hundreds of billions of tokens (a token can be a word, part of a word, or a character). When given an input prompt, an LLM predicts the most likely sequence of tokens to follow, based on the patterns it has internalized during training. The defining characteristic of an LLM is not merely its size, but the emergent capabilities that arise from scale. Traditional language models, such as n-gram models or early recurrent neural networks (RNNs), operated with limited context windows and parameter counts in the millions. LLMs, by contrast, contain billions or even trillions of parameters—the weights and biases that shape the model’s predictions. For example, GPT-3 (2020) had 175 billion parameters; GPT-4 (2023) is estimated to have over 1.7 trillion parameters across a mixture-of-experts architecture. This scaling, combined with training on corpora that include much of the publicly available internet (books, articles, code, forums, scientific papers), yields models that can perform tasks they were never explicitly trained for, such as translation, summarization, question answering, code generation, and even rudimentary reasoning

This paper presents a compression scheme for digital still images, by using the Kohonen's neural network algorithm, not only for its vector quantization feature, but also for its topological property. This property allows an increase of about 80% for the compression rate. Compared to the JPEG standard, this compression scheme shows better performances (in terms of PSNR) for compression rates higher than 30.

The self-organizing Kohonen map is a reliable and efficient way to achieve vector quantization. Typical application of such algorithm is image compression. Moreover, Kohonen networks realize a mapping between an input and an output space that preserves topology. This feature can be used to build new compression schemes which allow to obtain better compression rate than with classical method as JPEG without reducing the image quality. Compared to JPEG, our lossy compression scheme shows better performances (in terms of PSNR) for compression rates higher than 30 [C. Amerijckx, M. Verleysen, P. Thissen and J.-D. . Image compression by self-organized Kohonen map. IEEE Transactions on Neural Networks, 9(3), 503-507.]. For lossless compression, this rate is about 2.7 for standard images.

The minimum number of misclassications in a multi-class classier is reached when the borders between classes are set according to the Bayes criterion. Unfortunately, this criterion necessitates the knowledge of the probability density function of each class of data, which i s unknown in practical problems. The theory of kernel estimators (Parzen windows) provides a way to estimate these probability densities, given a set of data in each class. The computational complexity of these estimators is however much too large in most practical applications; we propose here a neural network aimed to estimate the probability density function underlying a set of data, in a sub-optimal way (while performances are quite similar to those in the optimal case), but with a strongly reduced complexity which makes the method useful in practical situations. The algorithm is based on a \competitive learning" vector quantization of the data, and on the choice of optimal widths for the kernels. We study the inuence of this factor on the classication error rate, and provide examples of the use of the algorithm on real-world data.

Modern centralized video platforms suffer from a systemic discoverability crisis, frequently mischaracterized as algorithmic "filter bubbles." This paper argues that the suppression of long-tail content is not a psychological phenomenon, but a rigid architectural limitation driven by Content Delivery Network (CDN) caching economics. To minimize origin-server egress costs, platforms are structurally incentivized to serve cached, popular content from edge nodes, reducing search interfaces to curated storefronts. We propose a Hybrid-Edge Search and Delivery Architecture that decouples indexing from physical storage. By utilizing a Browser-First Orchestrator powered by WebAssembly (Wasm) and compiled from memory-safe systems languages (e.g., Rust, Zig), we shift metadata vectorization and search execution entirely to the local client. Coupled with an Asynchronous Long-Tail Protocol (ALTP) based on distributed hash routing for peer-to-peer media fragmentation, this local-first architecture bypasses central bandwidth costs, restoring deterministic algorithmic transparency to digital video discovery.

Page 1. Novel Algorithms for Optimal Compression Using Classification Metrics Bei Xie and Tamal Bose Erzsebet Merenyi Wireless ( Virginia Tech Electrical and Computer Engineering Bradley Department of Electrical and Computer Engineering Rice University ...

The series "Advances in Intelligent Systems and Computing" contains publications on theory, applications, and design methods of Intelligent Systems and Intelligent Computing. Virtually all disciplines such as engineering, natural sciences, computer and information science, ICT, economics, business, e-commerce, environment, healthcare, life science are covered. The list of topics spans all the areas of modern intelligent systems and computing. The publications within "Advances in Intelligent Systems and Computing" are primarily textbooks and proceedings of important conferences, symposia and congresses. They cover significant recent developments in the field, both of a foundational and applicable character. An important characteristic feature of the series is the short publication time and world-wide distribution. This permits a rapid and broad dissemination of research results.

The ISO/IEC MPEG-4 Audio standard includes the TwinVQ encoding tool. This tool is suitable for low-bit-rate general audio coding, but drawback is the computational complexity of the encoder. To develop a faster TwinVQ encoder, new fast vector quantization algorithms -area localized pre-selection and hit zone masking -are introduced. These algorithms exploit pre-and main-selection procedure scheme of the conjugate structure vector quantization which is used in the TwinVQ. The improvement is evaluated by measuring the encoding speed and the sound quality of reproduction.

A novel encoding technique is proposed for the recognition of patterns using four different techniques for training artificial neural networks (ANNs) of the Kohonen type. Each template or model pattern is overlaid on a radial grid of appropriate size, and converted to a two-dimensional feature array which then acts as the training input to the ANN. The first technique employs Kohonen's self-organizing network, each neuron of which is assigned, after training, the label of the model pattern. It is found that a graphical plot of the labels of the neurons exhibits clusters (which means in effect that the feature array pertaining to distorted versions of the same pattern belongs to a specific cluster), thereby justifying the coding strategy used in this paper. When the new, unknown pattern is input to the network, it is classified to have the same label of the neuron whose corresponding model pattern is closest to the given pattern. In an attempt to reduce the computational time and the size of the network, and simultaneously improve accuracy in recognition, Kohonen's learning vector quantization (LVQ) algorithm is used to train the ANN. To further improve the network's performance and to realize a network of minimum size, two constructive learning algorithms, both based on LVQ, are proposed: (1) multi-step learning vector quantization (MLVQ), and (2) thermal multi-step learning vector quantization (TLVQ). When the proposed algorithms are applied to the classification of noiseless and noisy (and distorted) patterns, the results demonstrate that the pattern encoding strategy and the suggested training techniques for ANNs are efficient and robust. For lack of space, only the most essential results are presented here. For details, see Ganesh .

We demonstrate that wave-like properties in highdimensional embeddings arise from quantization of value distributions rather than dimensional ordering. Through systematic dimension randomization experiments, we show that spectral signatures remain invariant under permutation, establishing wave-likeness as a distributional property determined by finite discrete value sets. We formalize this through a Riemann-Lebesgue framework distinguishing value-domain (Riemann) from frequencydomain (Lebesgue) quantization, with optimal strategy depending critically on embedding skewness percentage. Comprehensive evaluation on GloVe embeddings reveals that low-skew embeddings (≤5% skewed dimensions) benefit substantially from Lebesgue quantization achieving up to 6.3% improvement with 8× compression (467 MB → 55 MB), while high-skew embeddings (≥10% skewed) perform better with simple uniform quantization. We identify 𝑘 = 32 (corresponding to 2 5 Walsh function orders, representing 5 bits per dimension) as nearly universal optimal across most dimensions, revealing that optimal quantization reflects information density. * This work is a product of a productive collaboration among Claude Sonnet 4.5, Claude Opus 4.5, and the author, the last of whom deeply regrets that current academic conventions categorically prohibit generative AI systems from being recognized as co-authors despite their substantial intellectual contributions. Our collaboration proceeded as follows: the author proposed initial ideas and possibilities as an architect, many of which were speculative or incomplete, and the AI systems, as an engineer, interpreted them, critically evaluated them, formalized them mathematically, and implemented them computationally, and we did so though intense interaction.

Automatic Music Transcription is the extraction of an acceptable notation from performed music. One important task in this problem is rhythm quantization which refers to categorization of note durations. Although quantization of a pure mechanical performance is rather straightforward, the task becomes increasingly difficult in presence of musical expression, i.e. systematic variations in timing of notes and tempo changes. For quantization of natural performances, we employ a framework based on Bayesian statistics. Expressive deviations are modelled by a probabilistic performance model from which the corresponding optimal quantizer can be derived by Bayes theorem. We demonstrate that some simple quantization schemata can be derived in this framework by simple assumptions about timing deviations. A general quantization method, which can be derived in this framework, is vector quantization (VQ). The algorithm operates on short groups of onsets and is thus flexible in capturing the structure of timing deviations between neighbouring onsets and thus performs better than simple rounding methods. Finally, we present some results on simple examples.

Automatic Music Transcription is the extraction of an acceptable notation from performed music. One important task in this problem is rhythm quantization which refers to categorization of note durations. Although quantization of a pure mechanical performance is rather straightforward, the task becomes increasingly difficult in presence of musical expression, i.e. systematic variations in timing of notes and tempo changes. For quantization of natural performances, we employ a framework based on Bayesian statistics. Expressive deviations are modelled by a probabilistic performance model from which the corresponding optimal quantizer can be derived by Bayes theorem. We demonstrate that some simple quantization schemata can be derived in this framework by simple assumptions about timing deviations. A general quantization method, which can be derived in this framework, is vector quantization (VQ). The algorithm operates on short groups of onsets and is thus flexible in capturing the structure of timing deviations between neighbouring onsets and thus performs better than simple rounding methods. Finally, we present some results on simple examples.

Vector quantization, a central topic in data compression, deals with the problem of encoding an information source or a sample of data vectors by means of a finite codebook, such that the average distortion is minimized. We introduce a common framework, based on maximum entropy inference to derive a deterministic annealing algorithm for robust vector quantization. The objective function for codebook design is extended to take channel noise and bandwidth limitations into account. Formulated as an on-line problem it is possible to derive learning rules for competitive neural networks. The resulting update rule is a generalization of the 'neural gas' model. The foundation in coding theory allows us to specify an optimality criterion for the 'neural gas' update rule.

Blosc2 is a high-performance compression library and data format designed for binary data such as numerical arrays, tensors, and other structured types. In this proposal, we aim to develop two new codecs for Blosc2 by leveraging its extensible pluginbased codec framework. Specifically, we propose integrating archetypal analysis (AA) and fuzzy clustering as novel codecs within Blosc2. Our proposal has been assessed by using the Olivetti faces dataset and results have demonstrated that AA outperforms fuzzy clustering in preserving fine-grained data details, achieving substantially higher Structural Similarity Index. These results underscore AA's capability to capture the complete data distribution, including its extreme values, which is an essential property for achieving high-fidelity compression.

Advances in technology are now increasing bringing people towards digital and mobile applications. To determine the owner of a handwriting, one of the manual techniques commonly used by humans that can be facilitated by mobile application technology is handwriting recognition. Learning Vector Quantization is one of the machine learning methods used to perform handwriting recognition and is one of the Artificial Neural Network (ANN) methods. This study aims to build a handwriting recognition system using the Learning Vector Quantization method on a mobile application, with feature extraction as the basic step in interpreting and classifying images. The results obtained from testing the prediction of learning vector quantization from 16 new data with a total of 80 tests. The results showed that 54 data were correct and 26 were incorrect, so the accuracy was 67.5%. Then obtained precision = 75.17%, recall = 67.5%, learning rate = 0.005, alpha value = 0.05 and iteration = 100

Compression of a noisy source is usually a two stage problem, involving the operations of estimation (denoising) and quantization. A survey of literature on this problem reveals that for the squared error distortion measure, the best possible compression strategy is to subject the noisy source to an optimal estimator followed by an optimal quantizer for the estimate. What we present in this paper is a simple but sub-optimal vector quantization (VQ) strategy that combines estimation and compression in one efficient step. The idea is to train a VQ on pairs of noisy and clean images. When presented with a noisy image, our VQ-based system estimates the noise variance and then performs joint denoising and compression. Simulations performed on images corrupted by additive, white, Gaussian noise show significant denoising at various bit rates. Results also indicate that our system is robust enough to handle a wide range of noise variations, while designed for a particular noise variance.

Speaker Recognition, Speaker Identification, MFCC, and Feature Extraction. Speaker Recognition Defined by the process of recognizing a person by his\her voice through specific features that extract from his\her voice signal. An Automatic Speaker recognition (ASP) is a biometric authentication system. In the last decade, many advances in the speaker recognition field have been attained, along with many techniques in feature extraction and modeling phases. In this paper, we present an overview of the most recent works in ASP technology. The study makes an effort to discuss several modeling ASP techniques like Gaussian Mixture Model GMM, Vector Quantization (VQ), and Clustering Algorithms. Also, several feature extraction techniques like Linear Predictive Coding (LPC) and Mel frequency cepstral coefficients (MFCC) are examined. Finally, as a result of this study, we found MFCC and GMM methods could be considered as the most successful techniques in the field of speaker recognition so f...

We investigate the use of fuzzy logic as applied to feature selection and classification. Fuzzy logic, a generalization of Aristotelian logic, can be useful in situations where there is imprecision or vagueness in the problem domain. Fuzzy logic is applied to transform input data into fuzzy sets that are then suitable for processing by a feature selection algorithm. A fuzzy entropy measure is used to perform classification using a similarity classifier. SAS/IML® was used to perform all computations.

Quick and accurate quantification of lake water quality (WQ) is essential for its management and improvement. Use of geotechnology (remote sensing, GIS, and GPS) applications is a step forward in improving our ability to effectively quantify and manage the WQ of ungauged lakes. Beaver Reservoir, a drinking water source for over 280,000 people in northwest Arkansas, is facing increased chlorophyll-a (Chl-a) and suspended matter (SM) content in the lake. This study is designed to qualitatively predict the Chl-a and SM content in the lake on a spatial basis from Landsat-TM image digital information. A Learning Vector Quantization (LVQ) classification neural model was used to predict the qualitative (oligotrophic, moderately oligotrophic, and mildly trophic) classes for several spatial positions in the lake. The geostatistical tool in ArcGIS was used to spatially map the Chl-a and SM extent around the lake. The LVQ classification model predicted the Chl-a extent with more than 90% accur...

The architecture of a very large scale integration (VLSI) vector-quantization processor (VQP) has been optimized to develop a general-purpose intelligent query-search agent. The agent performs a similarity-based search in a large-volume database. Although similarity-based search processing is computationally very expensive, latency-free searches have become possible due to the highly parallel maximum-likelihood search architecture of the VQP chip. Three architectures of the VQP chip have been studied and their performances are compared. In order to give reasonable searching results according to the different policies, the concept of penalty function has been introduced into the VQP. An E-commerce real-estate agency system has been developed using the VQP chip implemented in a field-programmable gate array (FPGA) and the effectiveness of such an agency system has been demonstrated.

We devise and explore an iterative optimization procedure for controlling particle populations in particle-incell (PIC) codes via merging and splitting of computational macro-particles. Our approach, is to compute an optimal representation of the global particle phase space structure while decreasing or increasing the entire particle population, based on k-means clustering of the data. In essence the procedure amounts to merging or splitting particles by statistical means, throughout the entire simulation volume in question, while minimizing a 6-dimensional total distance measure to preserve the physics. Particle merging is by far the most demanding procedure when considering conservation laws of physics; it amounts to lossy compression of particle phase space data. We demonstrate that our k-means approach conserves energy and momentum to high accuracy, even for high compression ratios, R ≈ 3 -i.e., N f 0.33N i . Interestingly, we find that an accurate particle splitting step can be performed using k-means as well; this from an argument of symmetry. The split solution, using k-means, places splitted particles optimally, to obtain maximal spanning on the phase space manifold. Implementation and testing is done using an electromagnetic PIC code, the Photon-Plasma code. Nonetheless, the k-means framework is general; it is not limited to Vlasov-Maxwell type PIC codes. We discuss advantages and drawbacks of this optimal phase space reconstruction.