1624256378

PyTorch implementation for Graph Contrastive Learning Automated [talk] [poster] [appendix]

Yuning You, Tianlong Chen, Yang Shen, Zhangyang Wang

In ICML 2021.

In this repository, we propose a principled framework named joint augmentation selection (JOAO), to automatically, adaptively and dynamically select augmentations during GraphCL training. Sanity check shows that the selection aligns with previous “best practices”, as shown in Figure 2.

- torch-geometric >= 1.6.0
- ogb == 1.2.4

- Semi-supervised learning [TU Datasets] [OGB]
- Unsupervised representation learning [TU Datasets]
- Transfer learning [MoleculeNet and PPI]

If you use this code for you research, please cite our paper.

```
@article{you2021graph,
title={Graph Contrastive Learning Automated},
author={You, Yuning and Chen, Tianlong and Shen, Yang and Wang, Zhangyang},
journal={arXiv preprint arXiv:2106.07594},
year={2021}
}
```

**Author:** Shen-Lab

**Download Link:** Download The Source Code

**Official Website:** https://github.com/Shen-Lab/GraphCL_Automated

#pytorch #python #machine-learning

1624256378

PyTorch implementation for Graph Contrastive Learning Automated [talk] [poster] [appendix]

Yuning You, Tianlong Chen, Yang Shen, Zhangyang Wang

In ICML 2021.

In this repository, we propose a principled framework named joint augmentation selection (JOAO), to automatically, adaptively and dynamically select augmentations during GraphCL training. Sanity check shows that the selection aligns with previous “best practices”, as shown in Figure 2.

- torch-geometric >= 1.6.0
- ogb == 1.2.4

- Semi-supervised learning [TU Datasets] [OGB]
- Unsupervised representation learning [TU Datasets]
- Transfer learning [MoleculeNet and PPI]

If you use this code for you research, please cite our paper.

```
@article{you2021graph,
title={Graph Contrastive Learning Automated},
author={You, Yuning and Chen, Tianlong and Shen, Yang and Wang, Zhangyang},
journal={arXiv preprint arXiv:2106.07594},
year={2021}
}
```

**Author:** Shen-Lab

**Download Link:** Download The Source Code

**Official Website:** https://github.com/Shen-Lab/GraphCL_Automated

#pytorch #python #machine-learning

1600632000

Pytorch is a Deep Learning Library Devoloped by Facebook. it can be used for various purposes such as Natural Language Processing , Computer Vision, etc

Python, Numpy, Pandas and Matplotlib

**What is a tensor ?**

A Tensor is a n-dimensional array of elements. In pytorch, everything is a defined as a tensor.

#pytorch #pytorch-tutorial #pytorch-course #deep-learning-course #deep-learning

1620805745

Want to try automated inventory management system for small businesses? Originscale automation software automate your data flow across orders, inventory, and purchasing. TRY FOR FREE

#automation #automation software #automated inventory management #automated inventory management system #automation management system #inventory automation

1593880440

**TL;DR:** *In this post, I discuss how to design local and computationally efficient provably powerful graph neural networks that are not based on the Weisfeiler-Lehman tests hierarchy. This is the second in the series of posts on the expressivity of graph neural networks. See Part 1 describing the relation between graph neural networks and the Weisfeiler-Lehman graph isomorphism test. In Part 3, I will argue why we should abandon the graph isomorphism problem altogether.*

Recent groundbreaking papers [1–2] established the connection between graph neural networks and the graph isomorphism tests, observing the analogy between the message passing mechanism and the Weisfeiler-Lehman (WL) test [3]. WL test is a general name for a hierarchy of graph-theoretical polynomial-time iterative algorithms for determining graph isomorphism. The *k*-WL test recolours *k*-tuples of vertices of a graph at each step according to some neighbourhood aggregation rules and stops upon reaching a stable colouring. If the histograms of colours of the two graphs are not the same, the graphs are deemed not isomorphic; otherwise, the graphs are possibly (but not necessarily) isomorphic.

Message passing neural networks are at most as powerful as the 1-WL test (also known as node colour refinement), and thus unable to distinguish between even very simple instances of non-isomorphic graphs. For example, message passing neural networks cannot count triangles [4], a motif known to play an important role in social networks where it is associated with the clustering coefficient indicative of how “tightly knit” the users are [5]. It is possible to design more expressive graph neural networks that replicate the increasingly more powerful *k*-WL tests [2,6]. However, such architectures result in high complexity and large number of parameters, but most importantly, typically require non-local operations that make them impractical.

Examples of non-isomorphic graphs that cannot be distinguished by 1-WL but can be distinguished by 3-WL due to its capability of counting triangles.

Thus, provably powerful graph neural networks based on the Weisfeiler-Lehman hierarchy are either not very powerful but practical, or powerful but impractical [7]. I argue that there is a different simple way to design efficient and provably powerful graph neural networks, which we proposed in a new paper with Giorgos Bouritsas and Fabrizio Frasca [8].

**Graph Substructure Networks. **The idea is actually very simple and conceptually similar to positional encoding or graphlet descriptors [9]: we make the message passing mechanism aware of the local graph structure, allowing for computing messages differently depending on the topological relationship between the endpoint nodes. This is done by passing to message passing functions additional structural descriptors associated with each node [10], which are constructed by subgraph isomorphism counting. In this way, we can partition the nodes of the graph into different equivalence classes reflecting topological characteristics that are shared both between nodes in each graph individually and across different graphs.

We call this architecture Graph Substructure Network (GSN). It has the same algorithmic design and memory and computational complexity as standard message passing neural networks, with an additional pre-computation step in which the structural descriptors are constructed. The choice of the substructures to count is crucial both to the expressive power of GSNs and the computational complexity of the pre-computation step.

The worst-case complexity of counting substructures of size *k* in a graph with *n* nodes is 𝒪(*nᵏ*). Thus, it is similar to high-order graph neural network models or Morris [2] and Maron [6]. However, GSN has several advantages over these methods. First, for some types of substructures such as paths and cycles the counting can be done with significantly lower complexity. Secondly, the computationally expensive step is done only once as preprocessing and thus does not affect network training and inference that remain linear, the same way as in message-passing neural networks. The memory complexity in training and inference is linear as well. Thirdly and most importantly, the expressive power of GSN is different from *k*-WL tests and in some cases is stronger.

**How powerful are GSNs?** The substructure counting endows GSN with more expressive power than the standard message-passing neural networks. First, it is important to clarify that the expressive power of GSN depends on the graph substructures used. Same way as we have a hierarchy of *k*-WL tests, we might have different variants of GSNs based on counting one or more structures. Using structures more complex than star graphs, GSNs can be made strictly more powerful than 1-WL (or the equivalent 2-WL) and thus also more powerful than standard message passing architectures. With 4-cliques, GSN is at least no less powerful than 3-WL, as shown by the following example of strongly regular graphs on which GSN succeeds while 3-WL fails:

Example of non-isomorphic strongly regular graphs with 16 vertices and node degree 6, where every two adjacent vertices have 2 mutual neighbours, and every two non-adjacent vertices also have 2 mutual neighbours. The 3-WL test fails on this example, while GSN with 4-clique structure can distinguish between them. In the graph on the left (known as the Rook’s graph) each node participates in exactly one 4-clique. The graph on the right (Shrikhande graph) has maximum cliques of size 3 (triangles). Figure from [8].

More generally speaking, for various substructures of 𝒪(1) size, as long as they cannot be counted by 3-WL, there exist graphs where GSN succeeds and 3-WL fails [11]. While we could not find examples to the contrary, they might in principle exist — that is why our statement about the power of GSN is of a weak form, “at least not less powerful”.

This holds for larger *k* as well; a generalisation of strongly regular graphs in the above figure, called *k*-*isoregular*, are instances on which the (*k*+1)-WL test fails [12]. These examples can also be distinguished by GSN with appropriate structures. The expressive power of GSNs can thus be captured by the following figure:

GSN is outside the Weisfeiler-Lehman hierarchy. With the appropriate structure (e.g. cliques or cycles of certain size), it is likely to be made at least not less powerful than k-WL.

How powerful can GSN be in principle? This is still an open question. The Graph Reconstruction Conjecture [13] postulates the possibility of recovering a graph from all its node-deleted substructures. Thus, if the Reconstruction Conjecture is correct, a GSN with substructures of size _n_−1 would be able to correctly test isomorphism of any graphs. However, the Reconstruction Conjecture is currently proven only for graphs of size _n≤_11 [14], and second, such large structures would be impractical.

The more interesting question is whether a similar result exists for “small” structures (of 𝒪(1) size independent of the number of nodes *n*). Our empirical results show that GSN with small substructures such as cycles, paths, and cliques work for strongly regular graphs, which are known to be a tough nut for the Weisfeiler-Lehman tests.

#geometric-deep-learning #deep-learning #graph-neural-networks #graph-theory #machine-learning #deep learning

1598002800

In a previous blog post, we implemented the **SimCLR framework in PyTorch**. It was a fun exercise to understand and implement it on a simple dataset of 5 categories with a total of just 1250 training images. From the SimCLR paper, we saw how the framework **benefits from larger models and larger batch sizes** and can produce results comparable to those of supervised models if enough computing power is available. But these requirements make the framework quite computation-heavy. Wouldn’t it be wonderful if we could have **the simplicity and power of this framework and have fewer compute requirements** so that this can become accessible to everyone? **Moco-v2** comes to the rescue.

We will **implement Moco-v2 in PyTorch** on much bigger datasets this time and train our model on Google Colab. We will work with the **Imagenette and Imagewoof datasets** this time, made by Jeremy Howard from Fast.AI.

Some images from the Imagenette datase

Some images from the Imagewoof dataset

A quick summary of these datasets (more info is here):

- Imagenette consists of
**10 easily classified classes**from Imagenet with a total of 9479 training and 3935 validation set images. - Imagewoof is a dataset of
**10 difficult classes**from Imagenet — difficult because all classes are dog breeds. There’re a total of 9035 training, and 3939 validation set images.

The way contrastive learning works in self-supervised learning is based on the idea that **we want different outlooks of images from the same category to have similar representations**. But since we don’t know which images belong to the same category, what is generally done is that **representations of different outlooks of the same image are brought closer to each other**. These different views taken pairwise are called as positive pairs.

#fastai #self-supervised-learning #contrastive-learning #deep-learning #pytorch #deep learning