Have you ever had a dataset, and asked: Does this model learn something different from that model? This is the question that Nguyen et. al. covered in their paper “Do Wide And Deep Networks Learn The Same Things?” [1].

Introduction

We apply CKA (centered kernel alignment) to measure the similarity of the hidden representations of different neural network architectures, finding that representations in wide or deep models exhibit a characteristic structure, which we term the block structure. We study how the block structure varies across different training runs, and uncover a connection between block structure and model overparametrization — block structure primarily appears in overparameterized models.

#neural-networks #paper #tensorflow #python