Definition D1
For a matrix W, the map x ↦ Wx is called a linear layer.
Proof.
(1)A linear layer is a linear map. That is, it satisfies W(ax + by) = a(Wx) + b(Wy).A1
∎
Definition D2
Applying linear layers f₁, …, fₙ in turn, the composition fₙ∘⋯∘f₁ is called an n-layer network without activation functions.
Proof.
(1)Since (g∘f)(x) = g(f(x)), the value of the previous layer becomes the input of the next.A9
∎
Definition D3
For a matrix W and a vector b, the map x ↦ Wx + b is called an affine layer. b is called the bias.
Proof.
(1)If b = 0, then Wx + 0 = Wx, the same as a linear layer.D1, A7
∎
Theorem T1
The composition of two linear layers is one linear layer. That is, W₂(W₁x) = (W₂W₁)x
Proof.
(1)f₁(x) = W₁x and f₂(y) = W₂y are linear maps.D1, A1
(2)(f₂∘f₁)(x) = W₂(W₁x)A9, D1
(3)f₂∘f₁ is also a linear map.(1), A3
(4)Its matrix is W₂W₁, so (f₂∘f₁)(x) = (W₂W₁)x(3), A2, D1
(5)Therefore the two layers are the same function as a single layer.(2), (4)
∎
Theorem T2
If n ≥ 1, then Wₙ(⋯(W₁x)⋯) = (Wₙ⋯W₁)x. That is, n layers are one layer.
Proof.
(1)n = 1 is the statement itself.D1
(2)Suppose it holds at n and set M = Wₙ⋯W₁; then Wₙ(⋯(W₁x)) = Mxassumption
(3)The value of the (n+1)-layer network is Wₙ₊₁(Mx)(2), D2, A9
(4)By T1, Wₙ₊₁(Mx) = (Wₙ₊₁M)x(3), T1
(5)By associativity, Wₙ₊₁M can be written without parentheses as Wₙ₊₁Wₙ⋯W₁.(4), A4
(6)By induction the statement holds for all n ≥ 1.(1), (5), A10
∎
Theorem T3
The composition of n affine layers is also one affine layer. That is, there exist a matrix W and a vector c with value Wx + c.
Proof.
(1)n = 1 is the statement itself.D3
(2)Stacking two affine layers gives W₂(W₁x + b₁) + b₂D3, A9
(3)W₂ is linear, so W₂(W₁x + b₁) = W₂(W₁x) + W₂b₁(2), A1, A8
(4)By T1, W₂(W₁x) = (W₂W₁)x(3), T1
(5)Grouping by associativity of addition gives (W₂W₁)x + (W₂b₁ + b₂)(2), (3), (4), A6
(6)Therefore the two affine layers are one affine layer with W = W₂W₁, c = W₂b₁ + b₂.(5), D3
(7)If the statement holds at n, then n+1 layers are a composition of two affine layers, so (6) applies directly.assumption, (6)
(8)By induction the statement holds for all n ≥ 1.(1), (7), A10
∎
Corollary C
Without activation functions, stacking layers does not enlarge the family of representable functions.
Proof.
(1)Every function obtained from n layers is of the form x ↦ Wx + c.T3
(2)Conversely, x ↦ Wx + c is also an n-layer network with the remaining layers set to W = I, b = 0.D3, A5, A7
(3)Therefore the family of functions obtained from n layers equals the family obtained from 1 layer.(1), (2)
(4)Even inserting a function σ between layers, if σ satisfies A1, then σ is a linear layer and by (3) everything folds again.(3), A1, A2, D1
(5)Therefore, to prevent the folding, σ must fail A1. That is the place of the activation function.(4)
∎