Transfer entropy

Transfer entropy is a non-parametric statistic measuring the amount of directed (time-asymmetric) transfer of information between two random processes.^[1][2][3] Transfer entropy from a process X to another process Y is the amount of uncertainty reduced in future values of Y by knowing the past values of X given past values of Y. More specifically, if ${\displaystyle X_{t}}$ and ${\displaystyle Y_{t}}$ for ${\displaystyle t\in \mathbb {N} }$ denote two random processes and the amount of information is measured using Shannon's entropy, the transfer entropy can be written as:

${\displaystyle T_{X\rightarrow Y}=H\left(Y_{t}\mid Y_{t-1:t-L}\right)-H\left(Y_{t}\mid Y_{t-1:t-L},X_{t-1:t-L}\right),}$

where H(X) is Shannon's entropy of X. The above definition of transfer entropy has been extended by other types of entropy measures such as Rényi entropy.^[3][4]

Transfer entropy is conditional mutual information,^[5][6] with the history of the influenced variable ${\displaystyle Y_{t-1:t-L}}$ in the condition:

${\displaystyle T_{X\rightarrow Y}=I(Y_{t};X_{t-1:t-L}\mid Y_{t-1:t-L}).}$

Transfer entropy reduces to Granger causality for vector auto-regressive processes.^[7] Hence, it is advantageous when the model assumption of Granger causality doesn't hold, for example, analysis of non-linear signals.^[8][9] However, it usually requires more samples for accurate estimation.^[10] The probabilities in the entropy formula can be estimated using different approaches (binning, nearest neighbors) or, in order to reduce complexity, using a non-uniform embedding.^[11] While it was originally defined for bivariate analysis, transfer entropy has been extended to multivariate forms, either conditioning on other potential source variables^[12] or considering transfer from a collection of sources,^[13] although these forms require more samples again.

Transfer entropy has been used for estimation of functional connectivity of neurons,^[13][14][15] social influence in social networks^[8] and statistical causality between armed conflict events.^[16] Transfer entropy is a finite version of the directed information which was defined in 1990 by James Massey^[17] as ${\displaystyle I(X^{n}\to Y^{n})=\sum _{i=1}^{n}I(X^{i};Y_{i}|Y^{i-1})}$, where ${\displaystyle X^{n}}$ denotes the vector ${\displaystyle X_{1},X_{2},...,X_{n}}$ and ${\displaystyle Y^{n}}$ denotes ${\displaystyle Y_{1},Y_{2},...,Y_{n}}$. The directed information places an important role in characterizing the fundamental limits (channel capacity) of communication channels with or without feedback^[18][19] and gambling with causal side information.^[20]

• Directed information
• Mutual information
• Conditional mutual information
• Causality
• Causality (physics)
• Structural equation modeling
• Rubin causal model

• "Transfer Entropy Toolbox". Google Code., a toolbox, developed in C++ and MATLAB, for computation of transfer entropy between spike trains.
• "Java Information Dynamics Toolkit (JIDT)". GitHub. 2019-01-16., a toolbox, developed in Java and usable in MATLAB, GNU Octave and Python, for computation of transfer entropy and related information-theoretic measures in both discrete and continuous-valued data.
• "Multivariate Transfer Entropy (MuTE) toolbox". GitHub. 2019-01-09., a toolbox, developed in MATLAB, for computation of transfer entropy with different estimators.