How to Make a Snappy IDE with Three Different Ways

rust-analyzer is a new “IDE backend” for the Rust programming language. Support rust-analyzer on Open Collective.

In this post, we’ll learn how to make a snappy IDE, in three different ways :-) It was inspired by this excellent article about using datalog for semantic analysis: https://petevilter.me/post/datalog-typechecking/ The post describes only the highest-level architecture. There’s much more to implementing a full-blown IDE.

Specifically, we’ll look at the backbone infrastructure of an IDE which serves two goals:

Quickly accepting new edits to source files.
Providing type information about currently opened files for highlighting, completion, etc.

Map Reduce

The first architecture is reminiscent of the map-reduce paradigm. The idea is to split analysis into relatively simple indexing phase, and a separate full analysis phase.

The core constraint of indexing is that it runs on a per-file basis. The indexer takes the text of a single file, parses it, and spits out some data about the file. The indexer can’t touch other files.

Full analysis can read other files, and it leverages information from the index to save work.

This all sounds way too abstract, so let’s look at a specific example — Java. In Java, each file starts with a package declaration. The indexer concatenates the name of the package with a class name to get a fully-qualified name (FQN). It also collects the set of methods declared in the class, the list of superclasses and interfaces, etc.

Per-file data is merged into an index which maps FQNs to classes. Note that constructing this mapping is an embarrassingly parallel task — all files are parsed independently. Moreover, this map is cheap to update. When a file change arrives, this file’s contribution from the index is removed, the text of the file is changed and the indexer runs on the new text and adds the new contributions. The amount of work to do is proportional to the number of changed files, and is independent from the total number of files.

Let’s see how FQN index can be used to quickly provide completion.

// File ./mypackage/Foo.java
package mypackage;

import java.util.*;

public class Foo {
    public static Bar f() {
        return new Bar();
    }
}

// File ./mypackage/Bar.java
package mypackage;

public class Bar {
    public void g() {}
}

// File ./Main.java
import mypackage.Foo;

public class Main {
    public static void main(String[] args) {
        Foo.f().
    }
}

The user has just typed Foo.f()., and we need to figure out that the type of receiver expression is Bar, and suggest g as a completion.

First, as the file Main.java is modified, we run the indexer on this single file. Nothing has changed (the file still contains the class Main with a static main method), so we don’t need to update the FQN index.

Next, we need to resolve the name Foo. We parse the file, notice an import and look up mypackage.Foo in the FQN index. In the index, we also find that Foo has a static method f, so we resolve the call as well. The index also stores the return type of f, but, and this is crucial, it stores it as a string "Bar", and not as a direct reference to the class Bar.

The reason for that is import java.util.* in Foo.java. Bar can refer either to java.util.Bar or to mypackage.Bar. The indexer doesn’t know which one, because it can look only at the text of Foo.java. In other words, while the index does store the return types of methods, it stores them in an unresolved form.

The next step is to resolve the identifier Bar in the context of Foo.java. This uses the FQN index, and lands in the class mypackage.Bar. There the desired method g is found.

Altogether, only three files were touched during completion. The FQN index allowed us to completely ignore all the other files in the project.

One problem with the approach described thus far is that resolving types from the index requires a non-trivial amount of work. This work might be duplicated if, for example, Foo.f is called several times. The fix is to add a cache. Name resolution results are memoized, so that the cost is paid only once. The cache is blown away completely on any change — with an index, reconstructing the cache is not that costly.

To sum up, the first approach works like this:

Each file is being indexed, independently and in parallel, producing a “stub” — a set of visible top-level declarations, with unresolved types.
All stubs are merged into a single index data structure.
Name resolution and type inference work primarily off the stubs.
Name resolution is lazy (we only resolve a type from the stub when we need it) and memoized (each type is resolved only once).
The caches are completely invalidated on every change
The index is updated incrementally:

if the edit doesn’t change the file’s stub, no change to the index is required.
otherwise, old keys are removed and new keys are added

Note an interesting interplay between “dumb” indexes which can be updated incrementally, and “smart” caches, which are re-computed from scratch.

This approach combines simplicity and stellar performance. The bulk of work is the indexing phase, and you can parallelize and even distribute it across several machine. Two examples of this architecture are IntelliJ and Sorbet.

The main drawback of this approach is that it works only when it works — not every language has a well-defined FQN concept. I think overall it’s a good idea to design name resolution and module systems (mostly boring parts of a language) such that they work well with the map-reduce paradigm.

Require package declarations or infer them from the file-system layout
Forbid meta-programming facilities which add new top-level declarations, or restrict them in such way that they can be used by the indexer. For example, preprocessor-like compiler plugins that access a single file at a time might be fine.
Make sure that each source element corresponds to a single semantic element. For example, if the language supports conditional compilation, make sure that it works during name resolution (like Kotlin’s expect/actual) and not during parsing (like conditional compilation in most other languages). Otherwise, you’d have to index the same file with different conditional compilation settings, and that is messy.
Make sure that FQNs are enough for most of the name resolution.

The last point is worth elaborating. Let’s look at the following Rust example:

// File: ./foo.rs
trait T {
    fn f(&self) {}
}
// File: ./bar.rs
struct S;

// File: ./somewhere/else.rs
impl T for S {}

// File: ./main.s
use foo::T;
use bar::S

fn main() {
    let s = S;
    s.f();
}

Here, we can easily find the S struct and the T trait (as they are imported directly). However, to make sure that s.f indeed refers to f from T, we also need to find the corresponding impl, and that can be roughly anywhere!

Leveraging Headers

The second approach places even more restrictions on the language. It requires:

a “declaration before use” rule,
headers or equivalent interface files.

Two such languages are C++ and OCaml.

The idea of the approach is simple — just use a traditional compiler and snapshot its state immediately after imports for each compilation unit. An example:

#include <iostream>

void main() {
    std::cout << "Hello, World!" << std::
}

Here, the compiler fully processes iostream (and any further headers included), snapshots its state and proceeds with parsing the program itself. When the user types more characters, the compiler restarts from the point just after the include. As the size of each compilation unit itself is usually reasonable, the analysis is fast.

If the user types something into the header file, then the caches need to be invalidated. However, changes to headers are comparatively rare, most of the code lives in .cpp files.

In a sense, headers correspond to the stubs of the first approach, with two notable differences:

It’s the user who is tasked with producing a stub, not the tool.
Unlike stubs, headers can’t be mutually recursive. Stubs store unresolved types, but includes can be snapshotted after complete analysis.

The two examples of this approach are Merlin of OCaml and clangd.

The huge benefit of this approach is that it allows re-use of an existing batch compiler. The two other approaches described in this article typically result in compiler re-writes. The drawback is that almost nobody likes headers and forward declarations.

Intermission: Laziness vs Incrementality

Note how neither of the two approaches is incremental in any interesting way. It is mostly “if something has changed, let’s clear the caches completely”. There’s a tiny bit of incrementality in the index update in the first approach, but it is almost trivial — remove old keys, add new keys.

This is because it’s not the incrementality that makes and IDE fast. Rather, it’s laziness — the ability to skip huge swaths of code altogether.

With map-reduce, the index tells us exactly which small set of files is used from the current file and is worth looking at. Headers shield us from most of the implementation code.

Query-based Compiler

Welcome to my world…

Rust fits the described approaches like a square peg into a round hole.

Here’s a small example:

#[macro_use]
extern crate bitflags;

bitflags! {
    struct Flags: u32 {
        const A = 0b00000001;
        const B = 0b00000010;
        const C = 0b00000100;
        const ABC = Self::A.bits | Self::B.bits | Self::C.bits;
    }
}

bitflags is macro which comes from another crate and defines a top-level declaration. We can’t put the results of macro expansion into the index, because it depends on a macro definition in another file. We can put the macro call itself into an index, but that is mostly useless, as the items, declared by the macro, would miss the index.

Here’s another one:

mod foo;

#[path = "foo.rs"]
mod bar;

Modules foo and bar refer to the same file, foo.rs, which effectively means that items from foo.rs are duplicated. If foo.rs contains the declaration struct S;, then foo::S and bar::S are different types. You also can’t fit that into an index, because those mod declarations are in a different file.

The second approach doesn’t work either. In C++, the compilation unit is a single file. In Rust, the compilation unit is a whole crate, which consists of many files and is typically much bigger. And Rust has procedural macros, which means that even surface analysis of code can take an unbounded amount of time. And there are no header files, so the IDE has to process the whole crate. Additionally, intra-crate name resolution is much more complicated (declaration before use vs. fixed point iteration intertwined with macro expansion).

It seems that purely laziness based models do not work for Rust. The minimal feasible unit of laziness, a crate, is still too big.

For this reason, in rust-analyzer we resort to a smart solution. We compensate for the deficit of laziness with incrementality. Specifically, we use a generic framework for incremental computation — salsa.

The idea behind salsa is rather simple — all function calls inside the compiler are instrumented to record which other functions were called during their execution. The recorded traces are used to implement fine-grained incrementality. If after modification the results of all of the dependencies are the same, the old result is reused.

There’s also an additional, crucial, twist — if a function is re-executed due to a change in dependency, the new result is compared with the old one. If despite a different input they are the same, the propagation of invalidation stops.

Using this engine, we were able to implement a rather fancy update strategy. Unlike the map reduce approach, our indices can store resolved types, which are invalidated only when a top-level change occurs. Even after a top-level change, we are able to re-use results of most macro expansions. And typing inside of a top-level macro also doesn’t invalidate caches unless the expansion of the macro introduces a different set of items.

The main benefit of this approach is generality and correctness. If you have an incremental computation engine at your disposal, it becomes relatively easy to experiment with the way you structure the computation. The code looks mostly like a boring imperative compiler, and you are immune to cache invalidation bugs (we had one, due to procedural macros being non-deterministic).

The main drawback is extra complexity, slower performance (fine-grained tracking of dependencies takes time and memory) and a feeling that this is a somewhat uncharted territory yet :-)

Links

How IntelliJ works

https://jetbrains.org/intellij/sdk/docs/basics/indexing_and_psi_stubs.html

How Sorbet works

https://www.youtube.com/watch?v=Gdx6by6tcvw

How clangd works

https://clangd.llvm.org/design/

How Merlin works

https://arxiv.org/abs/1807.06702

How rust-analyzer works

https://github.com/rust-analyzer/rust-analyzer/tree/master/docs/dev

#rust #developer

What is GEEK

Buddha Community

How to Make a Snappy IDE with Three Different Ways

Mike doctor

1624309200

4 Top Ways to Make Passive Income with Crypto (I Earn $2,685 per Month)

In this video, I’m showing you the 4 top ways to make passive income with cryptocurrencies in 2021. If you want to know how to make money with crypto, try out these 4 strategies that anyone can do! And make sure to watch until the end for a full explanation and walkthrough of how these methods work.
I personally made $2,685 passive income last month using crypto (previous months even more), and I wanted to reveal the best ways for you to do the same. Whether you have bitcoin, ethereum, litecoin, GUSD, or pretty much any other coin, you can make money. And even you currently don’t have any crypto, 2 of these methods will still work!

0:00 - Introduction
00:17 - Crypto Affiliate marketing (my strategies)
00:41 - Walkthrough of the Coinbase refer a friend program
01:38 - Walkthrough of the Coinbase official affiliate program
03:38 - Crypto lending using BlockFi (up to 8.6% APY)
04:21 - Walkthrough of BlockFi’s website and interest rates
06:34 - My BlockFi dashboard and earnings
07:43 - Bitcoin mining
10:14 - Staking
10:40 - Taking you through crypto.com’s staking rewards page
12:22 - How much I have made and outro
📺 The video in this post was made by Charlie Chang
The origin of the article: https://www.youtube.com/watch?v=58WKjp57TXs
🔺 DISCLAIMER: The article is for information sharing. The content of this video is solely the opinions of the speaker who is not a licensed financial advisor or registered investment advisor. Not investment advice or legal advice.
Cryptocurrency trading is VERY risky. Make sure you understand these risks and that you are responsible for what you do with your money
🔥 If you’re a beginner. I believe the article below will be useful to you ☞ What You Should Know Before Investing in Cryptocurrency - For Beginner
⭐ ⭐ ⭐The project is of interest to the community. Join to Get free ‘GEEK coin’ (GEEKCASH coin)!
☞ **-----CLICK HERE-----**⭐ ⭐ ⭐
Thanks for visiting and watching! Please don’t forget to leave a like, comment and share!

#bitcoin #crypto #ways to make passive income with crypto #$2,685 per month #4 top ways to make passive income with crypto (i earn $2,685 per month) #4 top ways to make passive income with crypto

josephine alberto

1579945937

How does tinder make money?

Essential information regarding how do dating apps make money and how does tinder make money. Moreover, we present unique ways to make money through dating apps.

#how does tinder make money #how does bumble make money #how much money do dating apps make #how dating apps make money #how do dating apps make money

jiga sata

1620140009

Different Ways NumPy Creating ArThere are three different ways to create Numpy arrays

Different Ways to Create Numpy Arrays

At the heart of a Numpy library is the array object or the ndarray object (n-dimensional array). You will use Numpy arrays to perform logical, statistical, and Fourier transforms. As part of working with Numpy, one of the first things you will do is create Numpy arrays. The main objective of this guide is to inform a data professional, you, about the different tools available to create Numpy arrays.

There are three different ways to create Numpy arrays:

Using Numpy functions
Conversion from other Python structures like lists
Using special library functions
Using Numpy functions
Numpy has built-in functions for creating arrays. We will cover some of them in this guide.

Creating a One-dimensional Array
First, let’s create a one-dimensional array or an array with a rank 1. arange is a widely used function to quickly create an array. Passing a value 20 to the arange function creates an array with values ranging from 0 to 19.

1
2
3
import Numpy as np
array = np.arange(20)
array
python
Output:

1
2
3
4
array([0, 1, 2, 3, 4,
5, 6, 7, 8, 9,
10, 11, 12, 13, 14,
15, 16, 17, 18, 19])
To verify the dimensionality of this array, use the shape property.

1
array.shape
python
Output:

1
(20,)
Since there is no value after the comma, this is a one-dimensional array. To access a value in this array, specify a non-negative index. As in other programming languages, the index starts from zero. So to access the fourth element in the array, use the index 3.

1
array[3]
python
Output:

1
3
Numpy Arrays are mutable, which means that you can change the value of an element in the array after an array has been initialized. Use the print function to view the contents of the array.

1
2
array[3] = 100
print(array)
python
Output:

1
2
3
4
5
[ 0 1 2 100
4 5 6 7
8 9 10 11
12 13 14 15
16 17 18 19]
Unlike Python lists, the contents of a Numpy array are homogenous. So if you try to assign a string value to an element in an array, whose data type is int, you will get an error.

1
array[3] =‘Numpy’
python
Output:

1
ValueError: invalid literal for int() with base 10: ‘Numpy’
Creating a Two-dimensional Array
Let’s talk about creating a two-dimensional array. If you only use the arange function, it will output a one-dimensional array. To make it a two-dimensional array, chain its output with the reshape function.

1
2
array = np.arange(20).reshape(4,5)
array
python
Output:

1
2
3
4
array([[ 0, 1, 2, 3, 4],
[ 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14],
[15, 16, 17, 18, 19]])
First, 20 integers will be created and then it will convert the array into a two-dimensional array with 4 rows and 5 columns. Let’s check the dimensionality of this array.

1
array.shape
python
Output:

1
(4, 5)
Since we get two values, this is a two-dimensional array. To access an element in a two-dimensional array, you need to specify an index for both the row and the column.

1
array[3][4]
python
Output:

1
19
Creating a Three-dimensional Array and Beyond
To create a three-dimensional array, specify 3 parameters to the reshape function.

1
2
array = np.arange(27).reshape(3,3,3)
array
python
Output:

1
2
3
4
5
6
7
8
9
10
11
array([[[ 0, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8]],

   [[ 9, 10, 11],
    [12, 13, 14],
    [15, 16, 17]],

   [[18, 19, 20],
    [21, 22, 23],
    [24, 25, 26]]])

Just a word of caution: The number of elements in the array (27) must be the product of its dimensions (333). To cross-check if it is a three-dimensional array, you can use the shape property.

1
array.shape
python
Output:

1
(3, 3, 3)
Also, using the arange function, you can create an array with a particular sequence between a defined start and end values

1
np.arange(10, 35, 3)
python
Output:

1
array([10, 13, 16, 19, 22, 25, 28, 31, 34])
Using Other Numpy Functions
Other than arange function, you can also use other helpful functions like zerosand ones to quickly create and populate an array.

Use the zeros function to create an array filled with zeros. The parameters to the function represent the number of rows and columns (or its dimensions).

1
np.zeros((2,4))
python
Output:

1
2
array([[0., 0., 0., 0.],
[0., 0., 0., 0.]])
Use the ones function to create an array filled with ones.

1
np.ones((3,4))
python
Output:

1
2
3
array([[1., 1., 1., 1.],
[1., 1., 1., 1.],
[1., 1., 1., 1.]])
The empty function creates an array. Its initial content is random and depends on the state of the memory.

1
np.empty((2,3))
python
Output:

1
2
array([[0.65670626, 0.52097334, 0.99831087],
[0.07280136, 0.4416958 , 0.06185705]])
The full function creates a n * n array filled with the given value.

1
np.full((2,2), 3)
python
Output:

1
2
array([[3, 3],
[3, 3]])
The eye function lets you create a n * n matrix with the diagonal 1s and the others 0.

1
np.eye(3,3)
python
Output:

1
2
3
array([[1., 0., 0.],
[0., 1., 0.],
[0., 0., 1.]])
The function linspace returns evenly spaced numbers over a specified interval. For example, the below function returns four equally spaced numbers between the interval 0 and 10.

1
np.linspace(0, 10, num=4)
python
Output:

1
array([ 0., 3.33333333, 6.66666667, 10.])
Conversion from Python Lists
Other than using Numpy functions, you can also create an array directly from a Python list. Pass a Python list to the array function to create a Numpy array:

1
2
array = np.array([4,5,6])
array
python
Output:

1
array([4, 5, 6])
You can also create a Python list and pass its variable name to create a Numpy array.

1
2
list = [4,5,6]
list
python
Output:

1
[4, 5, 6]
1
2
array = np.array(list)
array
python
Output:

1
array([4, 5, 6])
You can confirm that both the variables, array and list, are a of type Python list and Numpy array respectively.

1
type(list)
python
list

1
type(array)
python
Numpy.ndarray

To create a two-dimensional array, pass a sequence of lists to the array function.

1
2
array = np.array([(1,2,3), (4,5,6)])
array
python
Output:

1
2
array([[1, 2, 3],
[4, 5, 6]])
1
array.shape
python
Output:

1
(2, 3)
Using Special Library Functions
You can also use special library functions to create arrays. For example, to create an array filled with random values between 0 and 1, use random function. This is particularly useful for problems where you need a random state to get started.

1
np.random.random((2,2))
python
Output:

1
2
array([[0.1632794 , 0.34567049],
[0.03463241, 0.70687903]])

#different ways

Felix Kling

1595295292