Deriving convolution from first principles

Basics Of Deep Learning

TL;DR:_ Have you even wondered what is so special about convolution? In this post, I derive the convolution from first principles and show that it naturally emerges from translational symmetry._

Image for post

La connoissance de certains principes supplée facilement à la connoissance de certains faits. (Claude Adrien Helvétius)

During my undergraduate studies, which I did in Electrical Engineering at the Technion in Israel, I was always appalled that such an important concept as convolution [1] just landed out of nowhere. This seemingly arbitrary definition disturbed the otherwise beautiful picture of the signal processing world like a grain of sand in one’s eye. How nice would it be to have the convolution emerge from first principles rather than have it postulated! As I will show in this post, such first principles are the notion of translational invariance or symmetry.

Let me start with the formula taught in basic signal processing courses defining the discrete convolution [2] of two n-dimensional vectors x and w:

Image for post

Here, for convenience, I assume that all the indices run from zero to _n_−1 and are modulo n; it is convenient to think of vectors as defined on a circle. Writing the above formula as a matrix-vector multiplication leads to a very special matrix that is called circulant:

Image for post

A circulant matrix has multi-diagonal structure, with elements on each diagonal having the same value. It can be formed by stacking together shifted (modulo n) versions of a vector [3]; for this reason, I use the notation C(w) referring to a circulant matrix formed by the vector w. Since any convolution xwcan beequivalently represented as a multiplication by the circulant matrix C(w)x, I will use the two terms interchangeably.

#ai & machine learning #convolution #convolutional newral net #deep learning #deep learning

What is GEEK

Buddha Community

Deriving convolution from first principles

Deriving convolution from first principles

Basics Of Deep Learning

TL;DR:_ Have you even wondered what is so special about convolution? In this post, I derive the convolution from first principles and show that it naturally emerges from translational symmetry._

Image for post

La connoissance de certains principes supplée facilement à la connoissance de certains faits. (Claude Adrien Helvétius)

During my undergraduate studies, which I did in Electrical Engineering at the Technion in Israel, I was always appalled that such an important concept as convolution [1] just landed out of nowhere. This seemingly arbitrary definition disturbed the otherwise beautiful picture of the signal processing world like a grain of sand in one’s eye. How nice would it be to have the convolution emerge from first principles rather than have it postulated! As I will show in this post, such first principles are the notion of translational invariance or symmetry.

Let me start with the formula taught in basic signal processing courses defining the discrete convolution [2] of two n-dimensional vectors x and w:

Image for post

Here, for convenience, I assume that all the indices run from zero to _n_−1 and are modulo n; it is convenient to think of vectors as defined on a circle. Writing the above formula as a matrix-vector multiplication leads to a very special matrix that is called circulant:

Image for post

A circulant matrix has multi-diagonal structure, with elements on each diagonal having the same value. It can be formed by stacking together shifted (modulo n) versions of a vector [3]; for this reason, I use the notation C(w) referring to a circulant matrix formed by the vector w. Since any convolution xwcan beequivalently represented as a multiplication by the circulant matrix C(w)x, I will use the two terms interchangeably.

#ai & machine learning #convolution #convolutional newral net #deep learning #deep learning

Nat  Kutch

Nat Kutch

1596633180

Deriving convolution from first principles

TL;DR:_ Have you even wondered what is so special about convolution? In this post, I derive the convolution from first principles and show that it naturally emerges from translational symmetry._

Image for post

La connoissance de certains principes supplée facilement à la connoissance de certains faits. (Claude Adrien Helvétius)

D

uring my undergraduate studies, which I did in Electrical Engineering at the Technion in Israel, I was always appalled that such an important concept as convolution [1] just landed out of nowhere. This seemingly arbitrary definition disturbed the otherwise beautiful picture of the signal processing world like a grain of sand in one’s eye. How nice would it be to have the convolution emerge from first principles rather than have it postulated! As I will show in this post, such first principles are the notion of translational invariance or symmetry.

Let me start with the formula taught in basic signal processing courses defining the discrete convolution [2] of two n-dimensional vectors x and w:

Image for post

Here, for convenience, I assume that all the indices run from zero to _n_−1 and are modulo n; it is convenient to think of vectors as defined on a circle. Writing the above formula as a matrix-vector multiplication leads to a very special matrix that is called circulant:

Image for post

Image for post

A circulant matrix has multi-diagonal structure, with elements on each diagonal having the same value. It can be formed by stacking together shifted (modulo n) versions of a vector [3]; for this reason, I use the notation C(w) referring to a circulant matrix formed by the vector w. Since any convolution xwcan beequivalently represented as a multiplication by the circulant matrix C(w)x, I will use the two terms interchangeably.

One of the first things we are taught in linear algebra is that matrix multiplication is non-commutative, i.e.,in general, ABBA. However, circulant matrices are very special exception:

Circulant matrices commute,

or in other words, C(w)C(u)=C(u)C(w). This is true for any circulant matrix, or any choice of u and w. Equivalently, we can say that the convolution is a commutative operation, xw=wx.

Image for post

A particular choice of w=[0,1,0…,0] yields a special circulant matrix that shifts vectors to the right by one position. This matrix is called the (right) _shift operator _[4] and denoted by S. The transpose of the right shift operator is the left shift operator. Obviously, shifting left and then right (or vice versa) does not do anything, which means S is an orthogonal matrix:

Image for post

Circulant matrices can be characterised by their commutativity property. It appears to be sufficient to show only commutativity with shift (Lemma 3.1 in [5]):

A matrix is circulant if and only if it commutes with shift.

The first direction of this “if and only if” statement leads to a very important property called translation or _shift equivariance _[6]: the convolution’s commutativity with shift implies that it does not matter whether we first shift a vector and then convolve it, or first convolve and then shift — the result will be the same.

Image for post

The second direction allows us to define convolution as the shift-equivariant linear operation: in order to commute with shift, a matrix must have the circulant structure. This is exactly what we aspired to from the beginning, to have the convolution emerge from the first principles of translational symmetry [7]. Instead of being given a formula of the convolution and proving its shift equivariance property, as it is typically done in signal processing books, we can start from the requirement of shift equivariance and arrive at the formula of the convolution as the only possible linear operation satisfying it.

Image for post

Illustration of shift equivariance as the interchangeability of shift and blur operations.

A nother important fact taught in signal processing courses is the connection between the convolution and the Fourier transform [8]. Here as well, the Fourier transform lands out of the blue, and then one is shown that it diagonalises the convolution operation, allowing to perform convolution of two vectors in the frequency domain as element-wise product of their Fourier transforms. Nobody ever explains where these sines and cosines come from and what is so special about them.

#deep-learning #convolutional-neural-net #data-science #machine-learning #convolution #deep learning

A reliable Cryptocurrency Derivatives Exchange Development company

Antier Solutions is consider as one of the best Cryptocurrency Derivatives Exchange Development company that is specialized in offering world’s best cryptocurrency Derivatives Exchange Development Services that enables the following 4 key types- Futures trading, Forwards trading, Options trading, and Perpetual trading. Our skilled team of blockchain developers build Derivative exchange on both centralized and decentralized platform. It’s your choice to build the best one for you. Exchange owners can leverage the potential of a cryptocurrency derivatives exchange to reach more investors. A Crypto derivative trading platform is more flexible than Spot Margin trading and opens access to otherwise unavailable markets. To know more about us, feel free to visit us.

#cryptocurrency derivatives exchange #crypto derivatives exchange development #cryptocurrency derivatives exchange development #derivatives exchange development services #derivative exchange development company

Dejah  Reinger

Dejah Reinger

1599921480

API-First, Mobile-First, Design-First... How Do I Know Where to Start?

Dear Frustrated,

I understand your frustration and I have some good news and bad news.

Bad News First (First joke!)
  • Stick around another 5-10 years and there will be plenty more firsts to add to your collection!
  • Definitions of these Firsts can vary from expert to expert.
  • You cannot just pick a single first and run with it. No first is an island. You will probably end up using a lot of these…

Good News

While there are a lot of different “first” methodologies out there, some are very similar and have just matured just as our technology stack has.

Here is the first stack I recommend looking at when you are starting a new project:

1. Design-First (Big Picture)

Know the high-level, big-picture view of what you are building. Define the problem you are solving and the requirements to solve it. Are you going to need a Mobile app? Website? Something else?

Have the foresight to realize that whatever you think you will need, it will change in the future. I am not saying design for every possible outcome but use wisdom and listen to your experts.

2. API First

API First means you think of APIs as being in the center of your little universe. APIs run the world and they are the core to every (well, almost every) technical product you put on a user’s phone, computer, watch, tv, etc. If you break this first, you will find yourself in a world of hurt.

Part of this First is the knowledge that you better focus on your API first, before you start looking at your web page, mobile app, etc. If you try to build your mobile app first and then go back and try to create an API that matches the particular needs of that one app, the above world of hurt applies.

Not only this but having a working API will make design/implementation of your mobile app or website MUCH easier!

Another important point to remember. There will most likely be another client that needs what this API is handing out so take that into consideration as well.

3. API Design First and Code-First

I’ve grouped these next two together. Now I know I am going to take a lot of flak for this but hear me out.

Code-First

I agree that you should always design your API first and not just dig into building it, However, code is a legitimate design tool, in the right hands. Not everyone wants to use some WYSIWYG tool that may or may not take add eons to your learning curve and timetable. Good Architects (and I mean GOOD!) can design out an API in a fraction of the time it takes to use some API design tools. I am NOT saying everyone should do this but don’t rule out Code-First because it has the word “Code” in it.

You have to know where to stop though.

Designing your API with code means you are doing design-only. You still have to work with the technical and non-technical members of your team to ensure that your API solves your business problem and is the best solution. If you can’t translate your code-design into some visual format that everyone can see and understand, DON’T use code.

#devops #integration #code first #design first #api first #api

Julie  Donnelly

Julie Donnelly

1602394680

S.O.L.I.D. Software Design Principles To Help You Become A Better Developer

S.O.L.I.D design principle comes from Object oriented programming guidelines. It is designed to develop software that can be easily maintained and extended; prevents code smells; easy to refractor; promotes agility and finally incorporates rapid + frequent changes quickly without bugs.

Generally, technical debt is the result of prioritising speedy delivery over perfect code. To keep it under control - use SOLID principles, during development.

Robert Martin, is credited with writing the SOLID principles and stated 4 major software issues if S.O.L.I.D is not followed diligently. They are :

Rigidity:

Implementing even a small change is difficult since it’s likely to translate into a cascade of changes.

Fragility:

Any change tends to break the software in many places, even in areas not conceptually related to the change.

Immobility:

We’re unable to reuse modules from other projects or within the same project because those modules have lots of dependencies.

Viscosity:

Difficult to implement new features the right way.

SOLID is a guideline and not a rule. It is important to understand the crux of it and incorporate it with a crisp judgement. There can be a case when only few principles out of all is required.

#typescript #javascript #solid-in-javascript #what-are-solid-principles #what-is-a-solid-principle #design-patterns #design-principles #object-oriented-design