Information Thermodynamics via Gibbs Free Energy

---

tags: - colorclass/a thermodynamic theory of statistical learning ---# Information Thermodynamic Framework

Free Energy Decomposition

The information processing capacity expressed as Gibbs free energy:

$$\Delta G_{info}=\Delta H_{info}-T\Delta S_{info}$$

where: - $\Delta H_{info}=-{\sum }_i{p}_i\log \frac{p_i}{q_i}$ (relative entropy between learned and true distributions) - $\Delta S_{info}=-{\sum }_i{p}_i\log p_i$ (Shannon entropy of parameter distribution)

Multi-Dataset Competition

For competing datasets, the chemical potential of information:

$${\mu }_i=\frac{\partial G_{info}}{\partial N_i}=k_{B}T\log \left(\frac{{\theta }_i}{1-{\theta }_i}\right)+\sum _j{\varphi }_{ij}{\theta }_j$$

where ${\varphi }_{ij}$ represents information interaction potentials between datasets.

Equilibrium Conditions

At equilibrium:

$$\sum _i{\mu }_i\delta N_i=0$$

with constraint:

$$\sum _i{\theta }_i\le C_{capacity}$$

where $C_{capacity}$ is the model’s information capacity.

Information Flow

The rate of information transfer:

$$J_i=L_{ij}(\nabla {\mu }_j-T\nabla s_j)$$

where $L_{ij}$ is the Onsager coefficient matrix and $s_j$ is the specific entropy.