Gibbs randomness-compression proposition: An efficient deep learning
arXiv:2505.23869v3 Announce Type: replace
Abstract: A proposition that connects randomness and compression is put forward via Gibbs entropy over set of measurement vectors associated with a compression process. The proposition states that a lossy compression process is equivalent to {it directed randomness} that preserves information content. The proposition originated from the observed behavior in newly proposed {it Dual Tomographic Compression} (DTC) compress-train framework. This is akin to tomographic reconstruction of layer weight matrices via building compressed sensed projections, via so-called {it weight rays}. This tomographic approach is applied to previous and next layers in a dual fashion, that triggers neuronal-level pruning. This novel model compress-train scheme appears in iterative fashion and acts as a smart neural architecture search: also called {it compression aware training}. The experiments demonstrated the utility of this dual-tomography during training: method accelerates and supports lottery ticket hypothesis. However, random compress-train iterations having similar performance demonstrated the connection between randomness and compression from statistical physics perspective, we formulated the so-called {it Gibbs randomness-compression proposition}, signifying randomness-compression relationship via Gibbs entropy. The proposition is supported with the experimental evidence, resulting in very high correlation between learning performance vs. Gibbs entropy over compression ratios. Practically, the DTC framework provides a promising approach for massively energy- and resource-efficient deep learning training.
