Riley Lambert

Riley Lambert

1565797933

Building a mobile chat app with Nest.js and Ionic 4

In this first part of this tutorial series we will learn how to build a fullstack mobile application using cutting edge technologies like Ionic 4, Nest.js and Pusher Chatkit. Ionic will handle our front end, we will build the backend of the app using Nest.js and use Pusher Chatkit to handle messaging.

The application that you’ll be building is a group chat application that will allow users to register, login and then chat with a group of other users.

The aim of this tutorial is to show you how you can use Pusher’s Chatkit to easily add chat features in your mobile applications built with Ionic 4, Angular and Nest.js.

You can find the source code for the first part from this GitHub repository.

Since we are not going to create the mobile UI in this part, we’ll be using cURL to interact with our application. This is a GIF image that shows how to send to POST request to the /register endpoint to register a user in then send a POST request to the /login endpoint to get a JWT token:

Prerequisites

You need to have a basic understanding of TypeScript and Node.js to follow this tutorial. TypeScript is a superset of JavaScript that adds static types to the language.

Both Nest.js and Ionic 4 (based on Angular) requires TypeScript so you need to be familiar with it.

You also need to have a recent version of Node.js (v8.11.2) and NPM (v5.6.0) installed on your machine, if they are not installed on your system, you simply need to head to the official website and grab the binaries for your system or refer to your operating system instructions for installing Node.js via the official package manager of your system.

Why use Chatkit?

Chatkit is a Pusher hosted API that allows developers to build apps with chat features without re-inventing the wheel. The available features include:

  • Group chat
  • One-to-one chat
  • Private chat
  • Typing indicators
  • “Who’s online” presence
  • Read receipts
  • Photo, video, and audio messages

The set of features covers the most needed chat features in most apps, which means you can focus on building the features that are specific to your app and let Pusher take care of the commonly needed chat features including managing chat state and data, scaling and infrastructure.

Configuring Chatkit

Setting up Chatkit is quite easy, first you need to have an account with Pusher, next, simply head to the dashboard. Under the CHATKIT box, click on the CREATE button to create your own Chatkit instance:

Enter a name for your instance and click on CREATE:

After creating your Chatkit instance, head to the Credentials tab and take note of your Instance Locator and Secret Key. You will need them later to connect your application to your Chatkit instance.

You also need to manually create a room where we’ll add new users once they sign up in our application but to interact with the instance you first need to create a user:

Add the required information and create you first Chatkit user:

You’ll be taken to this interface:

Under User Actions, click on Create and join a room. Give a name to your room and click on CREATE ROOM:

Take note of the room ID because we’ll need it later in this tutorial series.

Note: Please note that except for this first user that we manually created, all the other users will be created from the SDK and added to the room when new users register via our application interface.
Chatkit handles all the chat data and features but we need a server to create users and add authentication. For this matter, we’ll use Nest.js for setting up an authentication server.

Installing the Nest.js CLI

Before creating a Nest.js project we first need to install Nest.js CLI which makes it easy to create and manage Nest.js projects. The CLI helps you from the first step of creating a project to the final step of building a production version of your final app. It’s based on the @angular-devkit package and provides its own schematics for Nest.js development which is @nestjs/schematics.

You can install Nest.js CLI from npm via the following command:

    $ npm install -g @nestjs/cli


Note: Please note that you may need to use sudo on Debian based system or macOS or an elevated administrator command prompt on Windows to install Node.js globally on your system depending on your npm configuration. As the time of this writing, Nest.js CLI v5.6.3 will be installed.
You can also create a Nest.js project by pulling the nestjs/cli[:version] Docker image or cloning the <a href="https://github.com/nestjs/nest-cli.git" target="_blank">https://github.com/nestjs/nest-cli.git</a> repository and installing dependencies. For more information, you can see the official repository.

Creating a new Nest.js project

After installing Nest.js CLI, you can now run the following command to easily create a new project:

    $ mkdir chatkit-nestjs-ionic
    $ cd chatkit-nestjs-ionic
    $ nest new server


The CLI will ask you for a bunch of information like the description and author and which package manager to use for installing packages, either npm or Yarn, enter the required information then hit Enter to start generating your project files and installing dependencies:

Wait a little to finish the installation process:

Then you can navigate inside your project’s folder and run a local development server:

    $ cd server
    $ npm run start


As you can see from the screenshot, this command allows you to start a development server on the port configured inside the src/main.ts file.

Your server is now running, you can simply open your browser and navigate to localhost:3000. You should see the Hello world! message.

Since we use Chatkit for adding all chat features, we will not need to implement any feature in the server except for JWT authentication and user management.

Setting up TypeORM and creating a database

For storing and registering users we need a database.

Nest.js supports TypeORM which is considered the most mature Object Relational Mapper (ORM) available in TypeScript. It’s available from the @nestjs/typeorm package.

Let’s start by installing the required dependencies:

    $ npm install --save @nestjs/typeorm typeorm sqlite3


For the sake of simplicity, we’ll use an SQLite database, but TypeORM supports all major databases like MySQL, PostgreSQL, MSSQL, Oracle, and MongoDB.

Note: Since an ORM abstracts away any direct operation with the underlying database system, you can later switch to use a fully fledged system like MySQL for production without changing anything in your code. But for now, let’s keep it simple and use SQLite.
After finishing up with installing the dependencies, you need to import the TypeOrmModule into the root ApplicationModule module. Open the src/app.module.ts file and add the following changes:

    // server/src/app.module.ts
    import { Module } from '@nestjs/common';
    import { TypeOrmModule } from '@nestjs/typeorm';

    import { AppController } from './app.controller';
    import { AppService } from './app.service';

    @Module({
      imports: [
       TypeOrmModule.forRoot({
          type: 'sqlite',
          database: 'my.db',
          entities: [__dirname + '/**/*.entity{.ts,.js}'],
          synchronize: true,
       }),
    ],
      controllers: [AppController],
      providers: [AppService],
    })
    export class AppModule {}

We import TypeOrmModule and we call the forRoot() method which takes the same configuration object as the standard createConnection() method of TypeORM.

In the configuration object, we specify:

  • The sqlite string for type so we can use SQLite as the database,
  • The my.db string for the database file (SQLite uses files to store the database),
  • The entities array which refers to all files that end with .entity.ts or .entity.js extensions. These files are created by developers and contain the ORM entities.
  • The synchronize option which takes true or false and allows you to automatically sync your database tables with the entities each time you run the app. In development, you can set it to true but it’s not preferable in production.

Note: Now, you can inject the Connection and EntityManager services anywhere you want to access them.
Next, let’s create a User entity which corresponds to a user in the database. Create a src/models/user.entity.ts file and add the following class:

    // server/src/models/user.entity.ts
    import { Entity, Column, PrimaryGeneratedColumn } from 'typeorm';
    @Entity()
    export class User {
      @PrimaryGeneratedColumn()
      id: number;

      @Column()
      name: string;

      @Column()
      email: string;

      @Column()
      password: string;
    }

You need to import the User entity and add it in the imports array of the module using the forFeature method:

    // server/src/app.module.ts
    import { User } from './models/user.entity';
    ...

    @Module({
    imports: [
    ...
    TypeOrmModule.forFeature([User]),

Next, let’s create a UserService that encapsulates all database operations that we need to perform against the User model.

Head back to your terminal and run the following command to generate a service:

    $ nest g s user


This command will create the src/user/user.service.ts file that contains the actual service code and the src/user/user.service.spec.ts file that contains the unit tests for the service. And also update the src/app.module.ts file by including UserService in the providers array.

Next, let’s add the create and findByEmail methods in the src/user/user.service.ts file which will be used respectively to persist a user and find a user by its email in the database.

    // server/src/user/user.service.ts
    import { Injectable } from '@nestjs/common';
    import { User } from '../models/user.entity';
    import { Repository } from 'typeorm';
    import { InjectRepository } from '@nestjs/typeorm';

    @Injectable()
    export class UserService {
        constructor(
            @InjectRepository(User)
            private userRepository: Repository<User>,
        ) { }

        async  findByEmail(email: string): Promise<User> {
            return await this.userRepository.findOne({
                where: {
                    email: email,
                }
            });
        }

        async  create(user: User): Promise<User> {
            return await this.userRepository.save(user);
        }
    }

First we import User, Repository and InjectRepository, next, inject the User repository via the service’s constructor and finally we define our methods.

The findByEmail method simply calls the findOne method of the injected repository to search for a user by the passed email in the database.

The create method calls the save method of the injected repository to save a user in the database.

Adding JWT authentication

Authentication is important for most web applications. You can follow different ways and approaches to implement user authentication. In this tutorial, we’ll implement authentication with JSON Web Tokens (JWTs).

First, you need to install the JWT utilities module for Nest.js using :

    $ npm install --save @nestjs/jwt


Next, open the /src/app.module.ts file and include the module in the imports array:

    // server/src/app.module.ts
    import { JwtModule } from  '@nestjs/jwt';
    // [...]

    JwtModule.register({
        secretOrPrivateKey:  'secret123'
    })

We also provided a private secret key that will be used to sign the JWT payload.

To interact with Chatkit, you also need valid JWT tokens that will be obtained by the client by using a token provider and will be sent with every request that the client makes to Chatkit.

Chatkit provides a test token provider that can be used to quickly start testing the chat features but it should be only used for testing. For production, you need to create your own token provider which can be done in two ways:

  • Either, by using the provided server SDKs.
  • Or without the help of the server SDKs using a JWT library or your own custom JWT implementation. See this link for more information.

In this tutorial, we’ll use the Node.js SDK for Chatkit to add a token provider in our Nest.js project so head back to your terminal and run the following command from the root of your project to install it:

    $ npm install @pusher/chatkit-server --save


Next, let’s create the AuthService class that will encapsulate the code for implementing JWT authentication in our application.

Using Nest.js CLI run the following command to generate a service:

    $ nest g s auth


This command will add the /src/auth/auth.service.ts file that contains the service and the /src/auth/auth.service.spec.ts file that contains the tests for the service and will update the main app module contained in the /src/app.module.ts file to include the generated service.

If you open the main module file at this stage, you can see that the JwtauthService was imported and included in the providers array:

    // server/src/app.module.ts
    import { Module } from '@nestjs/common';
    import { AppController } from './app.controller';
    import { AppService } from './app.service';
    import { AuthService } from './auth/auth.service';
    // [...]

    @Module({
      imports: [/* [...] */],
      controllers: [AppController],
      providers: [AppService, UserService,AuthService],
    })
    export class AppModule {}

Now, after creating the service, you need to import the Chatkit server SDK, JwtService , UserService , the User entity and the AuthenticationResponse. Open the src/auth/auth.service.ts file and add the following import:

    // server/src/auth/auth.service.ts
    import Chatkit from '@pusher/chatkit-server';
    import { JwtService } from  '@nestjs/jwt';
    import { UserService } from  '../user/user.service';
    import { User } from  '../models/user.entity';
    import  Chatkit, { AuthenticationResponse } from  '@pusher/chatkit-server';

Next, you need to add the following code:

    // server/src/auth/auth.service.ts
    @Injectable()
    export class AuthService {
      chatkit: Chatkit;
      constructor(
        private readonly userService: UserService,
        private readonly jwtService: JwtService
      ) {
        this.chatkit = new Chatkit({
          instanceLocator: YOUR_INSTANCE_LOCATOR,
          key: YOUR_SECRET_KEY
        })    
      }

We add a member variable to the service that holds the Chatkit instance. Next we inject UserService and JwtService via the constructor and inside it, we create the Chatkit instance.

Replace YOUR_INSTANCE_LOCATOR and YOUR_SECRET_KEY with the credentials from the dashboard. When a user connects to Chatkit, a request will be sent to a /token endpoint (that will be created later in this tutorial) to authenticate the user. Your server has to send a response that contains a token using the Chatkit.authenticate method if the request is valid.

Now, you need to define and implement the following methods:

  • getToken: It’s used to create and return a valid JWT token. This method will simply use the authenticate method of the Chatkit instance to generate a valid token.
  • validateUser: It’s used to validate the user. This method will use the findByEmail method of UserService to check if the user exists in the database.
  • createUser: It’s used to create a user in the local database and then in the Chatkit instance.

Let’s start with the createUser method which takes a parameter of the User type:

    // server/src/auth/auth.service.ts
    private async createUser(userData: User): Promise<User>{
        return this.userService.create(userData).then(user =>{
          const userId = `${user.name}${user.id}`;
          const roomId = "YOUR_ROOM_ID";
          const avatarURL = "https://image.flaticon.com/icons/png/128/149/149071.png";

          return this.chatkit.createUser({id: userId, 
             name: user.name,
             avatarURL: avatarURL
          }).then(()=>{

            return this.chatkit.addUsersToRoom({ roomId: roomId,
              userIds: [userId]}).then(()=>{
                return user;
            });

          })

        });
    }

Replace YOUR_ROOM_ID with the room id from the dashboard.

This method calls the create method of UserService to persist the user in the database then when the Promise successfully resolves with a user object that has a unique identifier in the database we use the id and name to create a corresponding user in the Chatkit instance by calling the createUser method of the instance and finally we add the user to the room by calling the addUsersToRoom method.

The createUser method of the Chatkit instance requires a unique user identifier and a user name. We construct the user id by concatenating the name with the database id of the user. This way we make sure the Chatkit user id is unique. We also provide a user avatar for testing using the <a href="https://image.flaticon.com/icons/png/128/149/149071.png" target="_blank">https://image.flaticon.com/icons/png/128/149/149071.png</a> URL.

Note: In a production application, you need to provide your users with a way to upload their avatars then associate them with the Chatkit user. You also need to hash passwords before storing them in the database using a tool like bcrypt.
Let’s now define the getToken method. It takes a user ID and returns an AuthenticationResponse:

    // server/src/auth/auth.service.ts
    public getToken(userId:  string): AuthenticationResponse {
        return this.chatkit.authenticate({ userId: userId });
    }  

The getToken method is simply a wrapper around the authenticate method of the Chatkit instance which returns a valid JWT token that can be used by the client to access Chatkit APIs. The authenticate method takes a userId that we specify when we create the user in the Chatkit instance (a concatenation of the word name and the database identifier of the user).

Another method that we need to define is the validateUser method which takes a parameter of the User type:

    // server/src/auth/auth.service.ts
    private async validateUser(userData: User): Promise<User> {
        return await this.userService.findByEmail(userData.email);
    }

This method calls the findByEmail method of UserService which checks if the user with the email exists in the database. If it exists the user object is returned otherwise a null object is returned.

After defining these methods, we’ll use them to define two public methods in the same service which are:

  • register for registering users,
  • login for login users.

This is the implementation of the two methods:

    // server/src/auth/auth.service.ts
    public async login(user: User): Promise<any | {status: number}>{
        return this.validateUser(user).then((userInfo)=>{
          if(!userInfo){
            return { status: 404 };
          }
          let userId = `${userInfo.name}${userInfo.id}`;
          const accessToken = this.jwtService.sign(userId);
          return {
             expires_in: 3600,
             access_token: accessToken,
             user_id: userId,
             status: 200
          };

        });
    }

    public async register(user: User): Promise<any>{
        return this.createUser(user)
    }

In the login method, we first use the validateUser method to make sure the user exists in the database then we call the sign method of JwtService to create an access token from the user id and name payload. Finally, we return an object containing the expires_in, access_token, user_id and status properties.

In the register method, we simply call the previously-defined createUser method to create a user in the database and then in the remote Chatkit instance.

Creating endpoints

After implementing the login and register methods, it’s time to create the corresponding endpoints in our application that handle user authentication. We also need to create a /token endpoint that will be used by the Chatkit client SDK to request JWT tokens from our server.

Open the existing src/app.controller.ts file and update it accordingly:

    // server/src/app.controller.ts
    import { Post, Body,Request, Controller} from '@nestjs/common';
    import { AuthService } from './auth/auth.service';
    import { User } from './models/user.entity';

    @Controller()
    export class AppController {
      constructor(private readonly authService: AuthService) {}

      @Post('token')
      async token(@Request() req): Promise<any> {
        return this.authService.getToken(req.query.user_id).body;
      }

      @Post('login')
      async login(@Body() userData: User): Promise<any> {
        return this.authService.login(userData);
      }  

      @Post('register')
      async register(@Body() userData: User): Promise<any> {
        return this.authService.register(userData);
      }    
    }

We start by importing the Post, Request and Body decorators, and also AuthService and the User entity. Next, we inject AuthService as an authService instance via the controller’s constructor.

Finally, we instruct Nest.js to create the three /token, /login and /register routes that accept a POST request by decorating their methods with the @Post decorator (the route is passed as a parameter).

For the login and register methods, we use the @Body() decorator to instruct Nest.js to inject the body of the received request in the endpoint handler as userData.

For the token method we need the full request so we use the @Request decorator instead.

Note: We could also create a controller for handling authentication using nest g controller auth but since our Nest.js app has only one task which is to handle JWT auth we can simply use the existing application controller.## Testing our auth endpoints

After creating the authentication endpoints, let’s use cURL to test them before we create our front-end mobile application in the next tutorial.

First, run the following command from the root of your project to start the Nest.js development server:

    $ npm start 


Next, make sure you have cURL installed on your system and run the following command from your terminal:

    curl -X POST -H 'content-type: application/json'  -d  '{ "email": "<a href="/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="03626b6e666743646e626a6f2d606c6e">[email protected]</a>", "name": "ahmed", "password": "pass001" }' localhost:3000/register


This will create a user in your SQLite database and a Chatkit user that you can see from the Console/INSTANCE INSPECTOR tab in your Chatkit dashboard. The endpoint returns the created Chatkit user with the id, name, created_at and updated_at fields.

You can also test the /login endpoint using:

    curl -X POST -H 'content-type: application/json'  -d  '{ "email": "<a href="/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="52333a3f373612353f333b3e7c313d3f">[email protected]</a>", "password": "pass001"}' localhost:3000/login


This should return a response object with an access token and a user id.

Enabling CORS

Since we’ll be using Ionic for creating the mobile app that will interact with this server and we’ll do most Ionic development on the browser we need to setup CORS (Cross Origin Resource Sharing). Otherwise, the browsers will block the requests to the server due to the same origin policy.

You can easily enable CORS in Nest.js by opening the src/main.ts file and calling the app.enableCors method:

    // server/src/main.ts
    import { NestFactory } from '@nestjs/core';
    import { AppModule } from './app.module';

    async function bootstrap() {
      const app = await NestFactory.create(AppModule);
      app.enableCors();
      await app.listen(3000);
    }
    bootstrap();

Conclusion

In this tutorial, we’ve seen how to create a server for JWT authentication using Nest.js and Chatkit’s SDK for Node.js.

In the next tutorial, we’ll continue developing our mobile application that uses this server for authentication and Chatkit for implementing the chat features.

You can find the source code for the first part from this GitHub repository.

Building a mobile app with Nest.js, Ionic 4 and Chatkit - Part 2: Build the frontend

In the previous tutorial we have created the server side of our chat application that uses TypeScript, Nest.js and Chatkit. Now, in the second tutorial, we are going to create the frontend mobile application using Ionic 4 and Pusher Chatkit.

Chatkit provides developers with client and server side SDKs for working with the API. In the previous tutorial, we used the Node.js SDK. In this tutorial, we’ll be using the JavaScript SDK.

You can find the source code for this part in this GitHub repository.

Prerequisites

To follow along with this tutorial you need to have these prerequisites:

  • Working knowledge of TypeScript,
  • Working knowledge of Angular (we’ll be using Ionic with Angular),
  • Recent versions of Node.js (v8.11.2) and npm (v5.6.0) installed on your system.

You can check the installed versions of Node.js and npm using the following commands from your terminal:

$ node --version
$ npm --version


What we’ll be building

In this part, we’ll build a chat mobile application with several chat features like:

  • User registration and login with email and password,
  • Listing users in a room with their online status,
  • Group chatting.

Introducing Ionic 4

Ionic 4 is a mobile UI library built on top of modern web technologies like web components and CSS variables. It’s the latest version of the most popular UI framework (now just a library) for building hybrid mobile applications with JavaScript and the web.

Ionic 4 aims to become just a UI mobile library that can be used by developers with any preferred client side library or framework like for example Angular, Vue, or React. You can also use it with plain JavaScript to build mobile applications.

Being framework-agnostic doesn’t mean that support for Angular is dropped. In fact, the Ionic team is also working on the Ionic-Angular v4 package that could be seen as the next version of Ionic 3 (which could be only used with Angular).

Installing the Ionic CLI v4

Now if you have Node.js and npm installed, you can go ahead and install Ionic CLI v4 by running the following command from your terminal:

$ npm install -g ionic


Note: Please note that depending on how you configured npm in your system you may need to open an Admin command prompt on Windows or run the command with sudo in macOS and Linux if you want to install packages globally. As the time of this writing, Ionic v4.5.0 is installed.## Creating an Ionic/Angular v4 project

You can create Ionic projects using the Ionic CLI and you can also specify which starter or base template you want to use for your project:

  • The blank starter: it provides a base blank project with one page.
  • The tabs starter: it provides a base project with tabs.
  • The side menu starter: it provides a base project with a side menu.

Now head back to your terminal and run the ionic start command to generate your project based on the blank template:

$ cd chatkit-nestjs-ionic
$ ionic start frontend blank --type=angular


Note: You also need to specify the type of framework to use with the --type=angular option, which is new in Ionic CLI v4 (For now the CLI supports only Angular).
The Ionic CLI will prompt you if you want to integrate Cordova.

  • Yes if you need to target native iOS and Android or,
  • No if you only need the web version of your project. For example, to develop a Progressive Web App.

You can enter No for now since we’ll be using the browser for testing.

You can also enter No for Install the free Ionic Appflow SDK and connect your app? If you don’t want to install Ionic Appflow SDK.

Wait for your project to be generated and the dependencies to get installed then run the following command to serve your project locally:

$ cd frontend
$ ionic serve


Your application will be running from the localhost:8100 address.

Installing the Chatkit client side SDK

Let’s now start implementing Chatkit by installing the JavaScript client side SDK using the following command:

$ cd frontend
$ npm install @pusher/chatkit-client --save


We’ll be importing this library in our project in the next section.

Setting up HttpClient, Forms and Ionic Storage

We’ll be using Angular HttpClient for sending requests to Chatkit and to our Nest.js server so we need to set it up in the project. Open the src/app/app.module.ts file, import HttpClientModule and FormsModule then add them to the imports array:

// frontend/src/app/app.module.ts

import { NgModule } from '@angular/core';
import { BrowserModule } from '@angular/platform-browser';
import { RouteReuseStrategy } from '@angular/router';
import { IonicModule, IonicRouteStrategy } from '@ionic/angular';
import { SplashScreen } from '@ionic-native/splash-screen/ngx';
import { StatusBar } from '@ionic-native/status-bar/ngx';
import { AppComponent } from './app.component';
import { AppRoutingModule } from './app-routing.module';

import { HttpClientModule } from '@angular/common/http';
import { FormsModule } from '@angular/forms';

@NgModule({
  declarations: [AppComponent],
  entryComponents: [],
  imports: [BrowserModule, IonicModule.forRoot(), AppRoutingModule,
    HttpClientModule,FormsModule
  ],
  providers: [
    StatusBar,
    SplashScreen,
    { provide: RouteReuseStrategy, useClass: IonicRouteStrategy }
  ],
  bootstrap: [AppComponent]
})
export class AppModule {}


We’ll use the Ionic storage module for working with the browser’s local storage so we first need to install it via npm:

$ npm install  --save  @ionic/storage


Next, add IonicStorageModule.forRoot() in the imports array:

// frontend/src/app/app.module.ts
// [...]
import { IonicStorageModule } from '@ionic/storage';

@NgModule({
  // [...]
  imports: [/* [...] */,IonicStorageModule.forRoot()],
  providers: [
    // [...]
  ],
  bootstrap: [AppComponent]
})
export class AppModule {}


Note: At this point, you can start your development server with ionic serve to make sure you don’t have any problems starting your app.## Creating the authentication service

After installing the client SDK, we’ll create an Angular service that handles authentication in the mobile application.

First, in you terminal, create a User interface using the following command:

$ ionic g interface user


This will create a src/app/user.ts file. Open it and update it as follows:

// frontend/src/app/user.ts
export interface User {
    id: number;
    name: string;
    email: string;
    password: string;
}


Again, in your terminal, run the following command to generate a service:

$ ionic g service auth


This will create the src/app/auth.service.ts and src/app/auth.service.spec.ts files. Open the src/app/auth.service.ts file and start by adding the following imports:

// frontend/src/app/auth.service.ts
import { HttpClient } from  '@angular/common/http';
import { tap } from  'rxjs/operators';
import { Observable } from  'rxjs';
import { Storage } from  '@ionic/storage';
import { User } from './user';


Next create the AUTH_SERVER variable:

// frontend/src/app/auth.service.ts
AUTH_SERVER:  string  =  'http://localhost:3000';


AUTH_SERVER holds the address of our authentication (Nest.js) server and authState.

Next, inject HttpClient and Storage via the service constructor:

// frontend/src/app/auth.service.ts
@Injectable({
  providedIn: 'root'
})
export class AuthService {

    constructor(private  httpClient:  HttpClient, private  storage:  Storage) {}
}


Next, add three methods to the src/app/auth.service.ts file for registration, and login:

// frontend/src/app/auth.service.ts
register(userInfo:  User):  Observable<User>{
    return  this.httpClient.post<User>(`${this.AUTH_SERVER}/register`,userInfo);
}

login(userInfo: User): Observable<any>{
    return this.httpClient.post(`${this.AUTH_SERVER}/login`,userInfo).pipe(
    tap( async (res: { status: number,  access_token, expires_in, user_id })=>{
      if(res.status !== 404){
        await this.storage.set("ACCESS_TOKEN", res.access_token);
        await this.storage.set("EXPIRES_IN", res.expires_in);
        await this.storage.set("USER_ID", res.user_id);
      }
      })
    );    
}


At this point, here is how the complete src/app/auth.service.ts file looks like:

// frontend/src/app/auth.service.ts
import { Injectable } from '@angular/core';
import { HttpClient } from '@angular/common/http';
import { tap } from 'rxjs/operators';
import { Observable } from 'rxjs';
import { Storage } from '@ionic/storage';
import { User } from './user';
@Injectable({
  providedIn: 'root'
})
export class AuthService {
  AUTH_SERVER: string = 'http://localhost:3000';
  constructor(private httpClient: HttpClient, private storage: Storage) { }
  register(userInfo: User): Observable<User> {
    return this.httpClient.post<User>(`${this.AUTH_SERVER}/register`, userInfo);
  }
  login(userInfo: User): Observable<any> {
    return this.httpClient.post(`${this.AUTH_SERVER}/login`, userInfo).pipe(
      tap(async (res: { status: number, access_token, expires_in, user_id }) => {
        if (res.status !== 404) {
          await this.storage.set("ACCESS_TOKEN", res.access_token);
          await this.storage.set("EXPIRES_IN", res.expires_in);
          await this.storage.set("USER_ID", res.user_id);
        }
      })
    );
  }
}


That’s it, we’ve finished with the authentication service. In the next section, we’ll see how you can use this service for adding authentication to your application.

Creating the registration and login pages

After creating the authentication service we can now create the register and login pages that will allow the users to either register or login. In your terminal, run the following command:

$ ionic g page login


This will generate a src/app/login folder with the following files:

  • src/app/login/login.module.ts
  • src/app/login/login.page.scss
  • src/app/login/login.page.html
  • src/app/login/login.page.spec.ts
  • src/app/login/login.page.ts

And will update the src/app/app-routing.module.ts file that holds the routing information by adding the following route:

// frontend/src/app/app-routing.module.ts
{ path:  'login', loadChildren:  './login/login.module#LoginPageModule' }


That means we can access the login page from the /login path. You also need to generate a page for registering users using:

$ ionic g page register


This will generate a src/app/register folder with similar files to the login page and will add the following route:

// frontend/src/app/app-routing.module.ts
{ path:  'register', loadChildren:  './register/register.module#RegisterPageModule' }


That means we can access this page from the /register path.

Implementing the register page

Let’s start adding the code for registering users. Open the src/app/register/register.page.ts file then import and inject AuthService and Router:

// frontend/src/app/register/register.page.ts
import { Component, OnInit } from '@angular/core';
import { Router } from  "@angular/router";
import { AuthService } from '../auth.service';

@Component({
  selector: 'app-register',
  templateUrl: './register.page.html',
  styleUrls: ['./register.page.scss'],
})
export class RegisterPage implements OnInit {
  constructor(private  authService:  AuthService, private  router:  Router) { }
  ngOnInit() {
  }
}


Next, add the following method:

// frontend/src/app/register/register.page.ts
import { Component, OnInit } from '@angular/core';
import { Router } from "@angular/router";
import { AuthService } from '../auth.service';

@Component({
  selector: 'app-register',
  templateUrl: './register.page.html',
  styleUrls: ['./register.page.scss'],
})
export class RegisterPage implements OnInit {
  constructor(private authService: AuthService, private router: Router) { }
  ngOnInit() {
  }
  register(form) {
    this.authService.register(form.value).subscribe((res) => {
      this.router.navigateByUrl('login');
    });
  }
}


We simply call the register() method of the authentication service and we pass the form value then we subscribe to the returned observable. After registration is successfully done we navigate to the login page.

Next open the src/app/register/register.page.html and add a form inside <ion-content> to get the user’s information:

// frontend/src/app/register/register.page.html
<ion-header>
  <ion-toolbar color="primary">
    <ion-title>Chatkit Demo</ion-title>
  </ion-toolbar>
</ion-header>
<ion-content padding>
  <form #f="ngForm" (ngSubmit)="register(f)">
    <ion-grid>
      <ion-row justify-content-center>
        <ion-col align-self-center size-md="6" size-lg="5" size-xs="12">
          <div text-center>
            <h3>Register</h3>
          </div>
          <div padding>
            <ion-item>
              <ion-input name="name" type="text" placeholder="Name" ngModel required></ion-input>
            </ion-item>
            <ion-item>
              <ion-input name="email" type="email" placeholder="<a href="/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="bdc4d2c8cffdd8d0dcd4d193ded2d0">[email protected]</a>" ngModel required></ion-input>
            </ion-item>
            <ion-item>
              <ion-input name="password" type="password" placeholder="Password" ngModel required></ion-input>
            </ion-item>
            <ion-item>
              <ion-input name="confirm" type="password" placeholder="Password again" ngModel required></ion-input>
            </ion-item>
          </div>
          <div padding>
            <ion-button size="large" type="submit" [disabled]="f.invalid" expand="block">Register</ion-button>
          </div>
        </ion-col>
      </ion-row>
    </ion-grid>
  </form>
</ion-content>


In the form, we bind the register method we created before with the ngSubmit event so the method gets called when the user submits the form.

If you run your application and go to the <a href="http://localhost:8100/register" target="_blank">http://localhost:8100/register</a> address, you should see this page:

Implementing the login page

Next, let’s implement the login page. Open the src/app/login/login.page.ts file then import AuthService and Router:

// frontend/src/app/login/login.page.ts
import { Router } from  "@angular/router";
import { AuthService } from  '../auth.service';


Next inject them via the service constructor:

// frontend/src/app/login/login.page.ts
constructor(private  authService:  AuthService, private  router:  Router) { }


Next, add a showError variable:

// frontend/src/app/login/login.page.ts
export  class  LoginPage  implements  OnInit {
    showError:  boolean  =  false;


Finally, add the login method:

// frontend/src/app/login/login.page.ts
login(form){
    this.authService.login(form.value).subscribe((res)=>{
      if(res.status == 200){
        this.showError = false;
        this.router.navigateByUrl(`home/${res.user_id}`);
      }
      else{
        this.showError = true;
      }
    });
}


In this method, we call the login() method of the authentication server and we pass the form data (email and password) using the .value member variable of the form instance. Next, we subscribe to the observable and check the status of the returned response. If the request is successful i.e res.status == 200 we navigate to the home page of the application and we pass user_id as a parameter to the home/ URL. Otherwise, we simply set showError to true.

At this point, this is how the complete src/app/login/login.page.ts looks like:

// frontend/src/app/login/login.page.ts
import { Component, OnInit } from '@angular/core';
import { Router } from "@angular/router";
import { AuthService } from '../auth.service';

@Component({
  selector: 'app-login',
  templateUrl: './login.page.html',
  styleUrls: ['./login.page.scss'],
})
export class LoginPage implements OnInit {
  showError: boolean = false;
  constructor(private authService: AuthService, private router: Router) { }
  ngOnInit() {
  }
  login(form) {
    this.authService.login(form.value).subscribe((res) => {
      if (res.status == 200) {
        this.showError = false;
        this.router.navigateByUrl(`home/${res.user_id}`);
      }
      else {
        this.showError = true;
      }
    });
  }
}


Let’s now add the form to get the user’s email and password in the login page. Open the src/app/login/login.page.html file and add a form inside <ion-content>:

// frontend/src/app/login/login.page.html
<ion-header>
  <ion-toolbar color="primary">
    <ion-title>Chatkit Demo</ion-title>
  </ion-toolbar>
</ion-header>
<ion-content padding>
  <form #f="ngForm" (ngSubmit)="login(f)">
    <ion-grid>
      <ion-row justify-content-center>
        <div *ngIf="showError">Error! Please try again</div>
      </ion-row>
      <ion-row justify-content-center>
        <ion-col align-self-center size-md="6" size-lg="5" size-xs="12">
          <div text-center>
            <h3>Login</h3>
          </div>
          <div padding>
            <ion-item>
              <ion-input name="email" type="email" placeholder="<a href="/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="354c5a4047755058545c591b565a58">[email protected]</a>" ngModel required></ion-input>
            </ion-item>
            <ion-item>
              <ion-input name="password" type="password" placeholder="Password" ngModel required></ion-input>
            </ion-item>
          </div>
          <div padding>
            <ion-button size="large" type="submit" [disabled]="f.invalid" expand="block">Login</ion-button>
          </div>
        </ion-col>
      </ion-row>
    </ion-grid>
  </form>
</ion-content>


If you visit the <a href="http://localhost:8100/login" target="_blank">http://localhost:8100/login</a> address, you should see this page:

That’s it for the authentication part, next we’ll be working with Chatkit to add chat features to our application.

Creating the chat service and connecting to Chatkit

After implementing user authentication in our application, we’ll create an Angular service that will encapsulate all the code required to connect with Chatkit and call any chat features so head back to your terminal and run this command:

$ ionic g service chat


The command will create the src/app/chat.service.ts file that contains the actual code for the service and the src/app/chat.service.spec.ts file that contains the specifications or test units.

Now open the src/app/chat.service.ts file, and let’s add the code to link our application to Chatkit.

Start by importing ChatManager, TokenProvider and BehaviorSubject:

// frontend/src/app/chat.service.ts
import { Observable, BehaviorSubject } from  'rxjs';
import { ChatManager, TokenProvider } from  '@pusher/chatkit-client';


Next, add the following variables to the service:

// frontend/src/app/chat.service.ts
AUTH_URL = 'http://localhost:3000/token';
INSTANCE_LOCATOR = 'YOUR_INSTANCE_LOCATOR';
GENERAL_ROOM_ID = 'YOUR_ROOM_ID';
GENERAL_ROOM_INDEX = 0;

chatManager: ChatManager;
currentUser;
messages = [];

usersSubject = new BehaviorSubject([]);
messagesSubject = new BehaviorSubject([]);


Make sure you create a room in your dashboard then replace YOUR_INSTANCE_LOCATOR and YOUR_ROOM_ID with your own values. Next, add the following method that allows you to connect to your Chatkit instance and subscribe to your room:

// frontend/src/app/chat.service.ts
async connectToChatkit(userId: string){
    this.chatManager = new ChatManager({
      instanceLocator: this.INSTANCE_LOCATOR,
      userId: userId,
      tokenProvider: new TokenProvider({ url: this.AUTH_URL})
    })

    this.currentUser = await this.chatManager.connect();

    await this.currentUser.subscribeToRoom({
      roomId: this.GENERAL_ROOM_ID,
      hooks: {},
      messageLimit: 20
    });
}


We create an instance of [ChatManager](<a href="https://docs.pusher.com/chatkit/reference/javascript#chat-manager)" target="_blank">https://docs.pusher.com/chatkit/reference/javascript#chat-manager)</a> and we assign it to chatManager variable. We pass in an instance locator, a user ID and a token provider.

We then use its [connect](<a href="https://docs.pusher.com/chatkit/reference/javascript#connect)" target="_blank">https://docs.pusher.com/chatkit/reference/javascript#connect)</a> method to connect to Chatkit and retrieve a current user object that we’ll use to interact with our Chatkit instance, rooms, messages, and users. The connect method returns a promise that resolves with a Current User object.

Finally, we use the subscribeToRoom method to subscribe to our room. This will allow us to be notified when new messages or users are added to the room. We pass in the room ID, a hooks object and a message limit number.

In the hooks object of the subscribeToRoom method, we need to provide subscription hooks that will be called when a new message or a user is added to the room or an event like user typing is triggered:

// frontend/src/app/chat.service.ts
hooks: {
    onMessage: message => {
          this.messages.push(message);
          this.messagesSubject.next(this.messages);
    }
},


In our case, we are using the onMessage hook that gets called for new messages.

Next after calling the subscribeToRoom method add the following code:

// frontend/src/app/chat.service.ts
const  users  =  this.currentUser.rooms[this.GENERAL_ROOM_INDEX].users;
this.usersSubject.next(users);


This will allow us to get the list of users in the room.

Next, add the following methods which return the behavior subjects from the service:

// frontend/src/app/chat.service.ts
getUsers(){
    return  this.usersSubject;
}

getMessages(){
    return  this.messagesSubject;
}


Next, add the sendMessage that is used to send a message to the room:

// frontend/src/app/chat.service.ts
sendMessage(message){
    return this.currentUser.sendMessage({
      text: message.text,
      roomId: message.roomId || this.GENERAL_ROOM_ID
    })   
}


Finally, we add a couple of other needed methods for checking the status of the user and return the current user:

// frontend/src/app/chat.service.ts
isUserOnline(user): boolean {
    return user.presence.state == 'online';
}

getCurrentUser(){
    return this.currentUser;
}


This is how the complete src/app/chat.service.ts file looks like:

// frontend/src/app/chat.service.ts
import { Injectable } from '@angular/core';
import { Observable, BehaviorSubject } from 'rxjs';
import { ChatManager, TokenProvider } from '@pusher/chatkit-client';

@Injectable({
  providedIn: 'root'
})
export class ChatService {
  AUTH_URL = 'http://localhost:3000/token';
  INSTANCE_LOCATOR = 'YOUR_INSTANCE_LOCATOR';
  GENERAL_ROOM_ID = 'YOUR_ROOM_ID';
  GENERAL_ROOM_INDEX = 0;
  chatManager: ChatManager;
  currentUser;
  messages = [];

  usersSubject = new BehaviorSubject([]);
  messagesSubject = new BehaviorSubject([]);
  constructor() { }
  async connectToChatkit(userId: string) {
    this.chatManager = new ChatManager({
      instanceLocator: this.INSTANCE_LOCATOR,
      userId: userId,
      tokenProvider: new TokenProvider({ url: this.AUTH_URL })
    })
    this.currentUser = await this.chatManager.connect();
    await this.currentUser.subscribeToRoom({
      roomId: this.GENERAL_ROOM_ID,
      hooks: {
        onMessage: message => {
          this.messages.push(message);
          this.messagesSubject.next(this.messages);
        }
      },
      messageLimit: 20
    });

    const users = this.currentUser.rooms[this.GENERAL_ROOM_INDEX].users;
    this.usersSubject.next(users);
  }

  getUsers() {
    return this.usersSubject;
  }
  getMessages() {
    return this.messagesSubject;
  }
  sendMessage(message) {
    return this.currentUser.sendMessage({
      text: message.text,
      roomId: message.roomId || this.GENERAL_ROOM_ID
    })
  }
  isUserOnline(user): boolean {
    return user.presence.state == 'online';
  }
  getCurrentUser() {
    return this.currentUser;
  }
}


That’s it for our chat service, next we’ll use this service to implement group chat in our mobile app.

Displaying room users on the home page

When a user registers in our application a Chatkit user is created behind the scenes. Open the src/app/home/home.page.ts file and start by adding the following imports:

// frontend/src/app/home/home.page.ts
import { OnInit } from '@angular/core';
import { Router, ActivatedRoute } from  '@angular/router';
import { ChatService } from  '../chat.service';
import { User } from  '../user';


Next, add the following variables to the component:

// frontend/src/app/home/home.page.ts
export  class  HomePage  implements  OnInit{
    userId:  string  =  '';
    userList:  any  = [];


Next inject ChatService and ActivatedRoute which is used to get route parameters:

// frontend/src/app/home/home.page.ts
constructor(private  chatService:  ChatService, private  route:  ActivatedRoute){}


Finally when the component is initialized we connect to our Chatkit instance and we retrieve the room users:

// frontend/src/app/home/home.page.ts
ngOnInit(){
    this.userId = this.route.snapshot.params.id;
    this.chatService.connectToChatkit(this.userId);
    this.chatService.getUsers().subscribe((users)=>{
        this.userList = users;
    });
}


On the ngOnInit life-cycle event of the component we first retrieve the user id from the route path then we call the connectToChatkit method of ChatServiceto connect the Chatkit.

Note: HomePage needs to implement OnInit i.e export class HomePage implements OnInit.
Finally, we subscribe to the getUsers method of ChatService to get the room’s users and add them to userList.

One more method that we need in our component is:

// frontend/src/app/home/home.page.ts
isOnline(user){
    return  this.chatService.isUserOnline(user);
}


This will allow us to check if the user is online.

This is the full content of the the src/app/home/home.page.ts:

// frontend/src/app/home/home.page.ts
import { Component } from '@angular/core';
import { OnInit } from '@angular/core';
import { Router, ActivatedRoute } from '@angular/router';
import { ChatService } from '../chat.service';
import { User } from '../user';

@Component({
  selector: 'app-home',
  templateUrl: 'home.page.html',
  styleUrls: ['home.page.scss'],
})
export class HomePage implements OnInit {
  userId: string = '';
  userList: any = [];
  constructor(private chatService: ChatService, private route: ActivatedRoute) { }
  ngOnInit() {
    this.userId = this.route.snapshot.params.id;
    this.chatService.connectToChatkit(this.userId);
    this.chatService.getUsers().subscribe((users) => {
      this.userList = users;
    });
  }
  isOnline(user) {
    return this.chatService.isUserOnline(user);
  }
}


Since we pass the user ID from the login page to the home page, we need to change the path of the home component to accept a parameter. Open the src/app/home/home.module.ts and change the path as follows:

// frontend/src/app/home/home.module.ts
RouterModule.forChild([
{
    path:  ':id',
    component:  HomePage
}
])


Now we need to display the list of users in the room and a button to start chatting. Open the src/app/home/home.page.html file, remove what’s inside <ion-content> and add:

// frontend/src/app/home/home.page.html
<div text-center>
    <ion-button  color="light" outline size="large" [routerLink]="'/chat'">
        <ion-icon name="chatbubbles"></ion-icon>
        Start chatting
    </ion-button>
</div>


This will add a button that takes the user to a chat page (that will be creating next). Next inside <ion-content>, add an Ionic spinner to create a loading animation:

// frontend/src/app/home/home.page.html
<h4>Users</h4>
<ion-spinner  name="dots" *ngIf="userList.length === 0"></ion-spinner>


Now, add the list to display users:

// frontend/src/app/home/home.page.html
<ion-list>
    <ion-item class="user-item" *ngFor="let user of userList">
        <div class="user-avatar">
            <img  [src]="user.avatarURL" alt="">
        </div>
        <ion-label class="user-name">
           {{ user.name }}
        </ion-label>
        <div class="user-presence">
            <ion-icon [class.user-online]="isOnline(user)" name="radio-button-on"></ion-icon>
        </div>  
    </ion-item>
</ion-list>


We simply loop over userList and display each user’s name, avatar and online status.

Also change the title of the page and the color of the toolbar:

// frontend/src/app/home/home.page.html
<ion-header>
  <ion-toolbar color="primary">
    <ion-title>
      Chatkit Demo
    </ion-title>
  </ion-toolbar>
</ion-header>


This is how the complete src/app/home/home.page.html file looks like:

// frontend/src/app/home/home.page.html
<ion-header>
  <ion-toolbar color="primary">
    <ion-title>
      Chatkit Demo
    </ion-title>
  </ion-toolbar>
</ion-header>
<ion-content padding>
  <div text-center>
    <ion-button  color="light" outline size="large" [routerLink]="'/chat'">
        <ion-icon name="chatbubbles"></ion-icon>
        Start chatting
    </ion-button>
</div>
<h4>Users</h4>
<ion-spinner  name="dots" *ngIf="userList.length === 0"></ion-spinner>
<ion-list>
  <ion-item class="user-item" *ngFor="let user of userList">
      <div class="user-avatar">
          <img  [src]="user.avatarURL" alt="">
      </div>
      <ion-label class="user-name">
         {{ user.name }}
      </ion-label>
      <div class="user-presence">
          <ion-icon [class.user-online]="isOnline(user)" name="radio-button-on"></ion-icon>
      </div>  
  </ion-item>
</ion-list>
</ion-content>


Finally let’s add some styling. Open the src/app/home/home.page.scss file and add the following styles:

// frontend/src/app/home/home.page.scss
.user-item {
    display: flex;
    .user-avatar{
        flex : 1; 
        img{
            width: 30px;
            height: 30px;
        }
    } 
    .user-name{
        flex: 2;
    }
    .user-presence{
        flex: 1;
    }
    .user-online{
        color: #32eb32;
    }
}


We use CSS Flex layout to style each list item.

Creating the chat page

When the user clicks on START CHATTING he will be taking to a chat page that we’re going to create in this section. Head back to your terminal and run the following command:

$ ng generate page chat


Open the src/app/chat/chat.page.ts file and first add the following imports:

// frontend/src/app/chat/chat.page.ts
import { Router } from  '@angular/router';
import { ChatService } from  '../chat.service';
import { User } from  '../user';


Next inject Router and ChatService:

// frontend/src/app/chat/chat.page.ts
constructor(private  router:  Router, private  chatService:  ChatService) { }


Next add the following variables to the component:

// frontend/src/app/chat/chat.page.ts
messageList:  any[] = [ ];
chatMessage:  string  ="";


Next on the ngOnInit life-cycle event get the messages and assign them to messageList:

// frontend/src/app/chat/chat.page.ts
ngOnInit() {
    this.chatService.getMessages().subscribe(messages  =>{
        this.messageList  =  messages;
    });
}


Finally, add the method to send a message to the room members:

// frontend/src/app/chat/chat.page.ts
sendMessage(){
    this.chatService.sendMessage({text:this.chatMessage}).then(()=>{
        this.chatMessage  =  "";
     });
}


This is the full content of the src/app/chat/chat.page.ts file:

// frontend/src/app/chat/chat.page.ts
import { Component, OnInit } from '@angular/core';
import { Router } from '@angular/router';
import { ChatService } from '../chat.service';
import { User } from '../user';

@Component({
  selector: 'app-chat',
  templateUrl: './chat.page.html',
  styleUrls: ['./chat.page.scss'],
})
export class ChatPage implements OnInit {
  messageList: any[] = [];
  chatMessage: string = "";
  constructor(private router: Router, private chatService: ChatService) { }

  ngOnInit() {
    this.chatService.getMessages().subscribe(messages => {
      this.messageList = messages;
    });
  }
  sendMessage() {
    this.chatService.sendMessage({ text: this.chatMessage }).then(() => {
      this.chatMessage = "";
    });
  }
}


Now, open the src/app/chat/chat.page.html file and let’s create a chat UI. First, add the list for displaying messages inside <ion-content>:

// frontend/src/app/chat/chat.page.html
<ion-content padding>
  <div class="container">
    <div *ngFor="let msg of messageList" class="message left">
      <img class="user-img" [src]="msg.sender.avatarURL" alt="" src="">
      <div class="msg-detail">
        <div class="msg-info">
          <p>
            {{msg.sender.name}}
          </p>
        </div>
        <div class="msg-content">
          <span class="triangle"></span>
          <p class="line-breaker ">{{msg.text}}</p>
        </div>
      </div>
    </div>
  </div>
</ion-content>


We loop through messageList using the *ngFor directive and display each message text, and sender information like name and avatar.

Next, add a textarea to enter the user’s message and a button to send it below <ion-content>:

// frontend/src/app/chat/chat.page.html
<ion-footer no-border>
      <div class="input-wrap">        
        <textarea #messageInput
                  placeholder="Enter your message!"
                  [(ngModel)]="chatMessage"
                  (keyup.enter)="sendMessage()">
        </textarea>
        <button ion-button clear icon-only item-right (click)="sendMessage()">
          <ion-icon name="ios-send" ios="ios-send" md="md-send"></ion-icon>
        </button>
      </div>     
</ion-footer>


We bind the sendMessage method to both the text-area enter and the button click events which allows users to send a message by either pressing Enter or clicking on the button.

Also change the text of the title and the color of the toolbar:

// frontend/src/app/chat/chat.page.html
<ion-header>
  <ion-toolbar color="primary">
    <ion-title>Chat Room</ion-title>
  </ion-toolbar>
</ion-header>


This is the full content of the src/app/chat/chat.page.html file:

// frontend/src/app/chat/chat.page.html
<ion-header>
  <ion-toolbar color="primary">
    <ion-title>Chat Room</ion-title>
  </ion-toolbar>
</ion-header>
<ion-content padding>
  <div class="container">
    <div *ngFor="let msg of messageList" class="message left">
      <img class="user-img" [src]="msg.sender.avatarURL" alt="" src="">
      <div class="msg-detail">
        <div class="msg-info">
          <p>
            {{msg.sender.name}}
          </p>
        </div>
        <div class="msg-content">
          <span class="triangle"></span>
          <p class="line-breaker ">{{msg.text}}</p>
        </div>
      </div>
    </div>
  </div>
</ion-content>
<ion-footer no-border>
  <div class="input-wrap">
    <textarea #messageInput placeholder="Enter your message!" [(ngModel)]="chatMessage" (keyup.enter)="sendMessage()">
    </textarea>
    <button ion-button clear icon-only item-right (click)="sendMessage()">
      <ion-icon name="ios-send" ios="ios-send" md="md-send"></ion-icon>
    </button>
  </div>
</ion-footer>


Next, open the src/app/chat/chat.page.scss file and add the following styles:

// frontend/src/app/chat/chat.page.scss
.input-wrap {
    padding: 5px;
    display: flex;
    textarea {
      flex: 3;
      border: 0;
      border-bottom: 1px #000;
      border-style: solid;
    }
    button {
        flex: 1;
    }
}
ion-footer {
    box-shadow: 0 0 4px rgba(0, 0, 0, 0.11);
    background-color: #fff;
}
ion-content .scroll-content {
    background-color: #f5f5f5;
}
.line-breaker {
    white-space: pre-line;
}
.container {
  .message {
    position: relative;
    padding: 7px 0;
    .msg-content {
        color: #343434;
        background-color: #ddd;
        float: left;        
    }
    .user-img {
        position: absolute;
        border-radius: 45px;
        width: 45px;
        height: 45px;
        box-shadow: 0 0 2px rgba(0, 0, 0, 0.36);
    }
    .msg-detail {
        width: 100%;
        padding-left: 60px;
        display: inline-block;
        p {
          margin: 0;
        }
        .msg-info {
          p {
            font-size: .8em;
            color: #888;
          }
        }
    }
  }
}


This is a screenshot of the chat UI:

Note: Please note that you first need to register by visiting localhost:8100/register where you need to enter your name, email and password. After registering you’ll be redirected to the /login page where you need to enter your email and password. If login is successful, you’ll be redirected to the home page where you have the START CHATTING button that you need to click on in order to navigate to the chat page.## Conclusion

In this tutorial, we’ve created a simple group chat app using Nest.js, Ionic 4 and Chatkit. You can also leverage Chatkit to allow private and direct chats between users by creating rooms that have only two members and you can allow users to create their own chat rooms and invite users they want and other features that you can read about from the docs.

You can find the source code for this part in this GitHub repository.

*Originally published by Ahmed Bouchefra at *https://pusher.com

Learn More

Learn Swift 4: From Beginner to Advanced

Build a Basic App with Spring Boot and JPA using PostgreSQL

Build a Video Chat App with ASP.NET Core, Angular

5 ways to build real-time apps with JavaScript

Full Stack Developers: Everything You Need to Know

Android Studio for beginners

Create and use private rooms in an Angular 7 chat app with Chatkit

#mobile-apps #angular #cordova #android #ios #javascript #typescript

What is GEEK

Buddha Community

Building a mobile chat app with Nest.js and Ionic 4
Franz  Becker

Franz Becker

1648803600

Plpgsql Check: Extension That Allows to Check Plpgsql Source Code.

plpgsql_check

I founded this project, because I wanted to publish the code I wrote in the last two years, when I tried to write enhanced checking for PostgreSQL upstream. It was not fully successful - integration into upstream requires some larger plpgsql refactoring - probably it will not be done in next years (now is Dec 2013). But written code is fully functional and can be used in production (and it is used in production). So, I created this extension to be available for all plpgsql developers.

If you like it and if you would to join to development of this extension, register yourself to postgresql extension hacking google group.

Features

  • check fields of referenced database objects and types inside embedded SQL
  • using correct types of function parameters
  • unused variables and function argumens, unmodified OUT argumens
  • partially detection of dead code (due RETURN command)
  • detection of missing RETURN command in function
  • try to identify unwanted hidden casts, that can be performance issue like unused indexes
  • possibility to collect relations and functions used by function
  • possibility to check EXECUTE stmt agaist SQL injection vulnerability

I invite any ideas, patches, bugreports.

plpgsql_check is next generation of plpgsql_lint. It allows to check source code by explicit call plpgsql_check_function.

PostgreSQL PostgreSQL 10, 11, 12, 13 and 14 are supported.

The SQL statements inside PL/pgSQL functions are checked by validator for semantic errors. These errors can be found by plpgsql_check_function:

Active mode

postgres=# CREATE EXTENSION plpgsql_check;
LOAD
postgres=# CREATE TABLE t1(a int, b int);
CREATE TABLE

postgres=#
CREATE OR REPLACE FUNCTION public.f1()
RETURNS void
LANGUAGE plpgsql
AS $function$
DECLARE r record;
BEGIN
  FOR r IN SELECT * FROM t1
  LOOP
    RAISE NOTICE '%', r.c; -- there is bug - table t1 missing "c" column
  END LOOP;
END;
$function$;

CREATE FUNCTION

postgres=# select f1(); -- execution doesn't find a bug due to empty table t1
  f1 
 ────
   
 (1 row)

postgres=# \x
Expanded display is on.
postgres=# select * from plpgsql_check_function_tb('f1()');
─[ RECORD 1 ]───────────────────────────
functionid │ f1
lineno     │ 6
statement  │ RAISE
sqlstate   │ 42703
message    │ record "r" has no field "c"
detail     │ [null]
hint       │ [null]
level      │ error
position   │ 0
query      │ [null]

postgres=# \sf+ f1
    CREATE OR REPLACE FUNCTION public.f1()
     RETURNS void
     LANGUAGE plpgsql
1       AS $function$
2       DECLARE r record;
3       BEGIN
4         FOR r IN SELECT * FROM t1
5         LOOP
6           RAISE NOTICE '%', r.c; -- there is bug - table t1 missing "c" column
7         END LOOP;
8       END;
9       $function$

Function plpgsql_check_function() has three possible formats: text, json or xml

select * from plpgsql_check_function('f1()', fatal_errors := false);
                         plpgsql_check_function                         
------------------------------------------------------------------------
 error:42703:4:SQL statement:column "c" of relation "t1" does not exist
 Query: update t1 set c = 30
 --                   ^
 error:42P01:7:RAISE:missing FROM-clause entry for table "r"
 Query: SELECT r.c
 --            ^
 error:42601:7:RAISE:too few parameters specified for RAISE
(7 rows)

postgres=# select * from plpgsql_check_function('fx()', format:='xml');
                 plpgsql_check_function                     
────────────────────────────────────────────────────────────────
 <Function oid="16400">                                        ↵
   <Issue>                                                     ↵
     <Level>error</level>                                      ↵
     <Sqlstate>42P01</Sqlstate>                                ↵
     <Message>relation "foo111" does not exist</Message>       ↵
     <Stmt lineno="3">RETURN</Stmt>                            ↵
     <Query position="23">SELECT (select a from foo111)</Query>↵
   </Issue>                                                    ↵
  </Function>
 (1 row)

Arguments

You can set level of warnings via function's parameters:

Mandatory arguments

  • function name or function signature - these functions requires function specification. Any function in PostgreSQL can be specified by Oid or by name or by signature. When you know oid or complete function's signature, you can use a regprocedure type parameter like 'fx()'::regprocedure or 16799::regprocedure. Possible alternative is using a name only, when function's name is unique - like 'fx'. When the name is not unique or the function doesn't exists it raises a error.

Optional arguments

relid DEFAULT 0 - oid of relation assigned with trigger function. It is necessary for check of any trigger function.

fatal_errors boolean DEFAULT true - stop on first error

other_warnings boolean DEFAULT true - show warnings like different attributes number in assignmenet on left and right side, variable overlaps function's parameter, unused variables, unwanted casting, ..

extra_warnings boolean DEFAULT true - show warnings like missing RETURN, shadowed variables, dead code, never read (unused) function's parameter, unmodified variables, modified auto variables, ..

performance_warnings boolean DEFAULT false - performance related warnings like declared type with type modificator, casting, implicit casts in where clause (can be reason why index is not used), ..

security_warnings boolean DEFAULT false - security related checks like SQL injection vulnerability detection

anyelementtype regtype DEFAULT 'int' - a real type used instead anyelement type

anyenumtype regtype DEFAULT '-' - a real type used instead anyenum type

anyrangetype regtype DEFAULT 'int4range' - a real type used instead anyrange type

anycompatibletype DEFAULT 'int' - a real type used instead anycompatible type

anycompatiblerangetype DEFAULT 'int4range' - a real type used instead anycompatible range type

without_warnings DEFAULT false - disable all warnings

all_warnings DEFAULT false - enable all warnings

newtable DEFAULT NULL, oldtable DEFAULT NULL - the names of NEW or OLD transitive tables. These parameters are required when transitive tables are used.

Triggers

When you want to check any trigger, you have to enter a relation that will be used together with trigger function

CREATE TABLE bar(a int, b int);

postgres=# \sf+ foo_trg
    CREATE OR REPLACE FUNCTION public.foo_trg()
         RETURNS trigger
         LANGUAGE plpgsql
1       AS $function$
2       BEGIN
3         NEW.c := NEW.a + NEW.b;
4         RETURN NEW;
5       END;
6       $function$

Missing relation specification

postgres=# select * from plpgsql_check_function('foo_trg()');
ERROR:  missing trigger relation
HINT:  Trigger relation oid must be valid

Correct trigger checking (with specified relation)

postgres=# select * from plpgsql_check_function('foo_trg()', 'bar');
                 plpgsql_check_function                 
--------------------------------------------------------
 error:42703:3:assignment:record "new" has no field "c"
(1 row)

For triggers with transitive tables you can set a oldtable or newtable parameters:

create or replace function footab_trig_func()
returns trigger as $$
declare x int;
begin
  if false then
    -- should be ok;
    select count(*) from newtab into x; 

    -- should fail;
    select count(*) from newtab where d = 10 into x;
  end if;
  return null;
end;
$$ language plpgsql;

select * from plpgsql_check_function('footab_trig_func','footab', newtable := 'newtab');

Mass check

You can use the plpgsql_check_function for mass check functions and mass check triggers. Please, test following queries:

-- check all nontrigger plpgsql functions
SELECT p.oid, p.proname, plpgsql_check_function(p.oid)
   FROM pg_catalog.pg_namespace n
   JOIN pg_catalog.pg_proc p ON pronamespace = n.oid
   JOIN pg_catalog.pg_language l ON p.prolang = l.oid
  WHERE l.lanname = 'plpgsql' AND p.prorettype <> 2279;

or

SELECT p.proname, tgrelid::regclass, cf.*
   FROM pg_proc p
        JOIN pg_trigger t ON t.tgfoid = p.oid 
        JOIN pg_language l ON p.prolang = l.oid
        JOIN pg_namespace n ON p.pronamespace = n.oid,
        LATERAL plpgsql_check_function(p.oid, t.tgrelid) cf
  WHERE n.nspname = 'public' and l.lanname = 'plpgsql'

or

-- check all plpgsql functions (functions or trigger functions with defined triggers)
SELECT
    (pcf).functionid::regprocedure, (pcf).lineno, (pcf).statement,
    (pcf).sqlstate, (pcf).message, (pcf).detail, (pcf).hint, (pcf).level,
    (pcf)."position", (pcf).query, (pcf).context
FROM
(
    SELECT
        plpgsql_check_function_tb(pg_proc.oid, COALESCE(pg_trigger.tgrelid, 0)) AS pcf
    FROM pg_proc
    LEFT JOIN pg_trigger
        ON (pg_trigger.tgfoid = pg_proc.oid)
    WHERE
        prolang = (SELECT lang.oid FROM pg_language lang WHERE lang.lanname = 'plpgsql') AND
        pronamespace <> (SELECT nsp.oid FROM pg_namespace nsp WHERE nsp.nspname = 'pg_catalog') AND
        -- ignore unused triggers
        (pg_proc.prorettype <> (SELECT typ.oid FROM pg_type typ WHERE typ.typname = 'trigger') OR
         pg_trigger.tgfoid IS NOT NULL)
    OFFSET 0
) ss
ORDER BY (pcf).functionid::regprocedure::text, (pcf).lineno

Passive mode

Functions should be checked on start - plpgsql_check module must be loaded.

Configuration

plpgsql_check.mode = [ disabled | by_function | fresh_start | every_start ]
plpgsql_check.fatal_errors = [ yes | no ]

plpgsql_check.show_nonperformance_warnings = false
plpgsql_check.show_performance_warnings = false

Default mode is by_function, that means that the enhanced check is done only in active mode - by plpgsql_check_function. fresh_start means cold start.

You can enable passive mode by

load 'plpgsql'; -- 1.1 and higher doesn't need it
load 'plpgsql_check';
set plpgsql_check.mode = 'every_start';

SELECT fx(10); -- run functions - function is checked before runtime starts it

Limits

plpgsql_check should find almost all errors on really static code. When developer use some PLpgSQL's dynamic features like dynamic SQL or record data type, then false positives are possible. These should be rare - in well written code - and then the affected function should be redesigned or plpgsql_check should be disabled for this function.

CREATE OR REPLACE FUNCTION f1()
RETURNS void AS $$
DECLARE r record;
BEGIN
  FOR r IN EXECUTE 'SELECT * FROM t1'
  LOOP
    RAISE NOTICE '%', r.c;
  END LOOP;
END;
$$ LANGUAGE plpgsql SET plpgsql.enable_check TO false;

A usage of plpgsql_check adds a small overhead (in enabled passive mode) and you should use it only in develop or preprod environments.

Dynamic SQL

This module doesn't check queries that are assembled in runtime. It is not possible to identify results of dynamic queries - so plpgsql_check cannot to set correct type to record variables and cannot to check a dependent SQLs and expressions.

When type of record's variable is not know, you can assign it explicitly with pragma type:

DECLARE r record;
BEGIN
  EXECUTE format('SELECT * FROM %I', _tablename) INTO r;
  PERFORM plpgsql_check_pragma('type: r (id int, processed bool)');
  IF NOT r.processed THEN
    ...

Attention: The SQL injection check can detect only some SQL injection vulnerabilities. This tool cannot be used for security audit! Some issues should not be detected. This check can raise false alarms too - probably when variable is sanitized by other command or when value is of some compose type. 

Refcursors

plpgsql_check should not to detect structure of referenced cursors. A reference on cursor in PLpgSQL is implemented as name of global cursor. In check time, the name is not known (not in all possibilities), and global cursor doesn't exist. It is significant break for any static analyse. PLpgSQL cannot to set correct type for record variables and cannot to check a dependent SQLs and expressions. A solution is same like dynamic SQL. Don't use record variable as target when you use refcursor type or disable plpgsql_check for these functions.

CREATE OR REPLACE FUNCTION foo(refcur_var refcursor)
RETURNS void AS $$
DECLARE
  rec_var record;
BEGIN
  FETCH refcur_var INTO rec_var; -- this is STOP for plpgsql_check
  RAISE NOTICE '%', rec_var;     -- record rec_var is not assigned yet error

In this case a record type should not be used (use known rowtype instead):

CREATE OR REPLACE FUNCTION foo(refcur_var refcursor)
RETURNS void AS $$
DECLARE
  rec_var some_rowtype;
BEGIN
  FETCH refcur_var INTO rec_var;
  RAISE NOTICE '%', rec_var;

Temporary tables

plpgsql_check cannot verify queries over temporary tables that are created in plpgsql's function runtime. For this use case it is necessary to create a fake temp table or disable plpgsql_check for this function.

In reality temp tables are stored in own (per user) schema with higher priority than persistent tables. So you can do (with following trick safetly):

CREATE OR REPLACE FUNCTION public.disable_dml()
RETURNS trigger
LANGUAGE plpgsql AS $function$
BEGIN
  RAISE EXCEPTION SQLSTATE '42P01'
     USING message = format('this instance of %I table doesn''t allow any DML operation', TG_TABLE_NAME),
           hint = format('you should to run "CREATE TEMP TABLE %1$I(LIKE %1$I INCLUDING ALL);" statement',
                         TG_TABLE_NAME);
  RETURN NULL;
END;
$function$;

CREATE TABLE foo(a int, b int); -- doesn't hold data ever
CREATE TRIGGER foo_disable_dml
   BEFORE INSERT OR UPDATE OR DELETE ON foo
   EXECUTE PROCEDURE disable_dml();

postgres=# INSERT INTO  foo VALUES(10,20);
ERROR:  this instance of foo table doesn't allow any DML operation
HINT:  you should to run "CREATE TEMP TABLE foo(LIKE foo INCLUDING ALL);" statement
postgres=# 

CREATE TABLE
postgres=# INSERT INTO  foo VALUES(10,20);
INSERT 0 1

This trick emulates GLOBAL TEMP tables partially and it allows a statical validation. Other possibility is using a [template foreign data wrapper] (https://github.com/okbob/template_fdw)

You can use pragma table and create ephemeral table:

BEGIN
   CREATE TEMP TABLE xxx(a int);
   PERFORM plpgsql_check_pragma('table: xxx(a int)');
   INSERT INTO xxx VALUES(10);

Dependency list

A function plpgsql_show_dependency_tb can show all functions, operators and relations used inside processed function:

postgres=# select * from plpgsql_show_dependency_tb('testfunc(int,float)');
┌──────────┬───────┬────────┬─────────┬────────────────────────────┐
│   type   │  oid  │ schema │  name   │           params           │
╞══════════╪═══════╪════════╪═════════╪════════════════════════════╡
│ FUNCTION │ 36008 │ public │ myfunc1 │ (integer,double precision) │
│ FUNCTION │ 35999 │ public │ myfunc2 │ (integer,double precision) │
│ OPERATOR │ 36007 │ public │ **      │ (integer,integer)          │
│ RELATION │ 36005 │ public │ myview  │                            │
│ RELATION │ 36002 │ public │ mytable │                            │
└──────────┴───────┴────────┴─────────┴────────────────────────────┘
(4 rows)

Profiler

The plpgsql_check contains simple profiler of plpgsql functions and procedures. It can work with/without a access to shared memory. It depends on shared_preload_libraries config. When plpgsql_check was initialized by shared_preload_libraries, then it can allocate shared memory, and function's profiles are stored there. When plpgsql_check cannot to allocate shared momory, the profile is stored in session memory.

Due dependencies, shared_preload_libraries should to contains plpgsql first

postgres=# show shared_preload_libraries ;
┌──────────────────────────┐
│ shared_preload_libraries │
╞══════════════════════════╡
│ plpgsql,plpgsql_check    │
└──────────────────────────┘
(1 row)

The profiler is active when GUC plpgsql_check.profiler is on. The profiler doesn't require shared memory, but if there are not shared memory, then the profile is limmitted just to active session.

When plpgsql_check is initialized by shared_preload_libraries, another GUC is available to configure the amount of shared memory used by the profiler: plpgsql_check.profiler_max_shared_chunks. This defines the maximum number of statements chunk that can be stored in shared memory. For each plpgsql function (or procedure), the whole content is split into chunks of 30 statements. If needed, multiple chunks can be used to store the whole content of a single function. A single chunk is 1704 bytes. The default value for this GUC is 15000, which should be enough for big projects containing hundred of thousands of statements in plpgsql, and will consume about 24MB of memory. If your project doesn't require that much number of chunks, you can set this parameter to a smaller number in order to decrease the memory usage. The minimum value is 50 (which should consume about 83kB of memory), and the maximum value is 100000 (which should consume about 163MB of memory). Changing this parameter requires a PostgreSQL restart.

The profiler will also retrieve the query identifier for each instruction that contains an expression or optimizable statement. Note that this requires pg_stat_statements, or another similar third-party extension), to be installed. There are some limitations to the query identifier retrieval:

  • if a plpgsql expression contains underlying statements, only the top level query identifier will be retrieved
  • the profiler doesn't compute query identifier by itself but relies on external extension, such as pg_stat_statements, for that. It means that depending on the external extension behavior, you may not be able to see a query identifier for some statements. That's for instance the case with DDL statements, as pg_stat_statements doesn't expose the query identifier for such queries.
  • a query identifier is retrieved only for instructions containing expressions. This means that plpgsql_profiler_function_tb() function can report less query identifier than instructions on a single line.

Attention: A update of shared profiles can decrease performance on servers under higher load.

The profile can be displayed by function plpgsql_profiler_function_tb:

postgres=# select lineno, avg_time, source from plpgsql_profiler_function_tb('fx(int)');
┌────────┬──────────┬───────────────────────────────────────────────────────────────────┐
│ lineno │ avg_time │                              source                               │
╞════════╪══════════╪═══════════════════════════════════════════════════════════════════╡
│      1 │          │                                                                   │
│      2 │          │ declare result int = 0;                                           │
│      3 │    0.075 │ begin                                                             │
│      4 │    0.202 │   for i in 1..$1 loop                                             │
│      5 │    0.005 │     select result + i into result; select result + i into result; │
│      6 │          │   end loop;                                                       │
│      7 │        0 │   return result;                                                  │
│      8 │          │ end;                                                              │
└────────┴──────────┴───────────────────────────────────────────────────────────────────┘
(9 rows)

The profile per statements (not per line) can be displayed by function plpgsql_profiler_function_statements_tb:

        CREATE OR REPLACE FUNCTION public.fx1(a integer)
         RETURNS integer
         LANGUAGE plpgsql
1       AS $function$
2       begin
3         if a > 10 then
4           raise notice 'ahoj';
5           return -1;
6         else
7           raise notice 'nazdar';
8           return 1;
9         end if;
10      end;
11      $function$

postgres=# select stmtid, parent_stmtid, parent_note, lineno, exec_stmts, stmtname
             from plpgsql_profiler_function_statements_tb('fx1');
┌────────┬───────────────┬─────────────┬────────┬────────────┬─────────────────┐
│ stmtid │ parent_stmtid │ parent_note │ lineno │ exec_stmts │    stmtname     │
╞════════╪═══════════════╪═════════════╪════════╪════════════╪═════════════════╡
│      0 │             ∅ │ ∅           │      2 │          0 │ statement block │
│      1 │             0 │ body        │      3 │          0 │ IF              │
│      2 │             1 │ then body   │      4 │          0 │ RAISE           │
│      3 │             1 │ then body   │      5 │          0 │ RETURN          │
│      4 │             1 │ else body   │      7 │          0 │ RAISE           │
│      5 │             1 │ else body   │      8 │          0 │ RETURN          │
└────────┴───────────────┴─────────────┴────────┴────────────┴─────────────────┘
(6 rows)

All stored profiles can be displayed by calling function plpgsql_profiler_functions_all:

postgres=# select * from plpgsql_profiler_functions_all();
┌───────────────────────┬────────────┬────────────┬──────────┬─────────────┬──────────┬──────────┐
│        funcoid        │ exec_count │ total_time │ avg_time │ stddev_time │ min_time │ max_time │
╞═══════════════════════╪════════════╪════════════╪══════════╪═════════════╪══════════╪══════════╡
│ fxx(double precision) │          1 │       0.01 │     0.01 │        0.00 │     0.01 │     0.01 │
└───────────────────────┴────────────┴────────────┴──────────┴─────────────┴──────────┴──────────┘
(1 row)

There are two functions for cleaning stored profiles: plpgsql_profiler_reset_all() and plpgsql_profiler_reset(regprocedure).

Coverage metrics

plpgsql_check provides two functions:

  • plpgsql_coverage_statements(name)
  • plpgsql_coverage_branches(name)

Note

There is another very good PLpgSQL profiler - https://bitbucket.org/openscg/plprofiler

My extension is designed to be simple for use and practical. Nothing more or less.

plprofiler is more complex. It build call graphs and from this graph it can creates flame graph of execution times.

Both extensions can be used together with buildin PostgreSQL's feature - tracking functions.

set track_functions to 'pl';
...
select * from pg_stat_user_functions;

Tracer

plpgsql_check provides a tracing possibility - in this mode you can see notices on start or end functions (terse and default verbosity) and start or end statements (verbose verbosity). For default and verbose verbosity the content of function arguments is displayed. The content of related variables are displayed when verbosity is verbose.

postgres=# do $$ begin perform fx(10,null, 'now', e'stěhule'); end; $$;
NOTICE:  #0 ->> start of inline_code_block (Oid=0)
NOTICE:  #2   ->> start of function fx(integer,integer,date,text) (Oid=16405)
NOTICE:  #2        call by inline_code_block line 1 at PERFORM
NOTICE:  #2       "a" => '10', "b" => null, "c" => '2020-08-03', "d" => 'stěhule'
NOTICE:  #4     ->> start of function fx(integer) (Oid=16404)
NOTICE:  #4          call by fx(integer,integer,date,text) line 1 at PERFORM
NOTICE:  #4         "a" => '10'
NOTICE:  #4     <<- end of function fx (elapsed time=0.098 ms)
NOTICE:  #2   <<- end of function fx (elapsed time=0.399 ms)
NOTICE:  #0 <<- end of block (elapsed time=0.754 ms)

The number after # is a execution frame counter (this number is related to deep of error context stack). It allows to pair start end and of function.

Tracing is enabled by setting plpgsql_check.tracer to on. Attention - enabling this behaviour has significant negative impact on performance (unlike the profiler). You can set a level for output used by tracer plpgsql_check.tracer_errlevel (default is notice). The output content is limited by length specified by plpgsql_check.tracer_variable_max_length configuration variable.

In terse verbose mode the output is reduced:

postgres=# set plpgsql_check.tracer_verbosity TO terse;
SET
postgres=# do $$ begin perform fx(10,null, 'now', e'stěhule'); end; $$;
NOTICE:  #0 start of inline code block (oid=0)
NOTICE:  #2 start of fx (oid=16405)
NOTICE:  #4 start of fx (oid=16404)
NOTICE:  #4 end of fx
NOTICE:  #2 end of fx
NOTICE:  #0 end of inline code block

In verbose mode the output is extended about statement details:

postgres=# do $$ begin perform fx(10,null, 'now', e'stěhule'); end; $$;
NOTICE:  #0            ->> start of block inline_code_block (oid=0)
NOTICE:  #0.1       1  --> start of PERFORM
NOTICE:  #2              ->> start of function fx(integer,integer,date,text) (oid=16405)
NOTICE:  #2                   call by inline_code_block line 1 at PERFORM
NOTICE:  #2                  "a" => '10', "b" => null, "c" => '2020-08-04', "d" => 'stěhule'
NOTICE:  #2.1       1    --> start of PERFORM
NOTICE:  #2.1                "a" => '10'
NOTICE:  #4                ->> start of function fx(integer) (oid=16404)
NOTICE:  #4                     call by fx(integer,integer,date,text) line 1 at PERFORM
NOTICE:  #4                    "a" => '10'
NOTICE:  #4.1       6      --> start of assignment
NOTICE:  #4.1                  "a" => '10', "b" => '20'
NOTICE:  #4.1              <-- end of assignment (elapsed time=0.076 ms)
NOTICE:  #4.1                  "res" => '130'
NOTICE:  #4.2       7      --> start of RETURN
NOTICE:  #4.2                  "res" => '130'
NOTICE:  #4.2              <-- end of RETURN (elapsed time=0.054 ms)
NOTICE:  #4                <<- end of function fx (elapsed time=0.373 ms)
NOTICE:  #2.1            <-- end of PERFORM (elapsed time=0.589 ms)
NOTICE:  #2              <<- end of function fx (elapsed time=0.727 ms)
NOTICE:  #0.1          <-- end of PERFORM (elapsed time=1.147 ms)
NOTICE:  #0            <<- end of block (elapsed time=1.286 ms)

Special feature of tracer is tracing of ASSERT statement when plpgsql_check.trace_assert is on. When plpgsql_check.trace_assert_verbosity is DEFAULT, then all function's or procedure's variables are displayed when assert expression is false. When this configuration is VERBOSE then all variables from all plpgsql frames are displayed. This behaviour is independent on plpgsql.check_asserts value. It can be used, although the assertions are disabled in plpgsql runtime.

postgres=# set plpgsql_check.tracer to off;
postgres=# set plpgsql_check.trace_assert_verbosity TO verbose;

postgres=# do $$ begin perform fx(10,null, 'now', e'stěhule'); end; $$;
NOTICE:  #4 PLpgSQL assert expression (false) on line 12 of fx(integer) is false
NOTICE:   "a" => '10', "res" => null, "b" => '20'
NOTICE:  #2 PL/pgSQL function fx(integer,integer,date,text) line 1 at PERFORM
NOTICE:   "a" => '10', "b" => null, "c" => '2020-08-05', "d" => 'stěhule'
NOTICE:  #0 PL/pgSQL function inline_code_block line 1 at PERFORM
ERROR:  assertion failed
CONTEXT:  PL/pgSQL function fx(integer) line 12 at ASSERT
SQL statement "SELECT fx(a)"
PL/pgSQL function fx(integer,integer,date,text) line 1 at PERFORM
SQL statement "SELECT fx(10,null, 'now', e'stěhule')"
PL/pgSQL function inline_code_block line 1 at PERFORM

postgres=# set plpgsql.check_asserts to off;
SET
postgres=# do $$ begin perform fx(10,null, 'now', e'stěhule'); end; $$;
NOTICE:  #4 PLpgSQL assert expression (false) on line 12 of fx(integer) is false
NOTICE:   "a" => '10', "res" => null, "b" => '20'
NOTICE:  #2 PL/pgSQL function fx(integer,integer,date,text) line 1 at PERFORM
NOTICE:   "a" => '10', "b" => null, "c" => '2020-08-05', "d" => 'stěhule'
NOTICE:  #0 PL/pgSQL function inline_code_block line 1 at PERFORM
DO

Attention - SECURITY

Tracer prints content of variables or function arguments. For security definer function, this content can hold security sensitive data. This is reason why tracer is disabled by default and should be enabled only with super user rights plpgsql_check.enable_tracer.

Pragma

You can configure plpgsql_check behave inside checked function with "pragma" function. This is a analogy of PL/SQL or ADA language of PRAGMA feature. PLpgSQL doesn't support PRAGMA, but plpgsql_check detects function named plpgsql_check_pragma and get options from parameters of this function. These plpgsql_check options are valid to end of group of statements.

CREATE OR REPLACE FUNCTION test()
RETURNS void AS $$
BEGIN
  ...
  -- for following statements disable check
  PERFORM plpgsql_check_pragma('disable:check');
  ...
  -- enable check again
  PERFORM plpgsql_check_pragma('enable:check');
  ...
END;
$$ LANGUAGE plpgsql;

The function plpgsql_check_pragma is immutable function that returns one. It is defined by plpgsql_check extension. You can declare alternative plpgsql_check_pragma function like:

CREATE OR REPLACE FUNCTION plpgsql_check_pragma(VARIADIC args[])
RETURNS int AS $$
SELECT 1
$$ LANGUAGE sql IMMUTABLE;

Using pragma function in declaration part of top block sets options on function level too.

CREATE OR REPLACE FUNCTION test()
RETURNS void AS $$
DECLARE
  aux int := plpgsql_check_pragma('disable:extra_warnings');
  ...

Shorter syntax for pragma is supported too:

CREATE OR REPLACE FUNCTION test()
RETURNS void AS $$
DECLARE r record;
BEGIN
  PERFORM 'PRAGMA:TYPE:r (a int, b int)';
  PERFORM 'PRAGMA:TABLE: x (like pg_class)';
  ...

Supported pragmas

echo:str - print string (for testing)

status:check,status:tracer, status:other_warnings, status:performance_warnings, status:extra_warnings,status:security_warnings

enable:check,enable:tracer, enable:other_warnings, enable:performance_warnings, enable:extra_warnings,enable:security_warnings

disable:check,disable:tracer, disable:other_warnings, disable:performance_warnings, disable:extra_warnings,disable:security_warnings

type:varname typename or type:varname (fieldname type, ...) - set type to variable of record type

table: name (column_name type, ...) or table: name (like tablename) - create ephereal table

Pragmas enable:tracer and disable:tracerare active for Postgres 12 and higher

Compilation

You need a development environment for PostgreSQL extensions:

make clean
make install

result:

[pavel@localhost plpgsql_check]$ make USE_PGXS=1 clean
rm -f plpgsql_check.so   libplpgsql_check.a  libplpgsql_check.pc
rm -f plpgsql_check.o
rm -rf results/ regression.diffs regression.out tmp_check/ log/
[pavel@localhost plpgsql_check]$ make USE_PGXS=1 all
clang -O2 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fpic -I/usr/local/pgsql/lib/pgxs/src/makefiles/../../src/pl/plpgsql/src -I. -I./ -I/usr/local/pgsql/include/server -I/usr/local/pgsql/include/internal -D_GNU_SOURCE   -c -o plpgsql_check.o plpgsql_check.c
clang -O2 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fpic -I/usr/local/pgsql/lib/pgxs/src/makefiles/../../src/pl/plpgsql/src -shared -o plpgsql_check.so plpgsql_check.o -L/usr/local/pgsql/lib -Wl,--as-needed -Wl,-rpath,'/usr/local/pgsql/lib',--enable-new-dtags  
[pavel@localhost plpgsql_check]$ su root
Password: *******
[root@localhost plpgsql_check]# make USE_PGXS=1 install
/usr/bin/mkdir -p '/usr/local/pgsql/lib'
/usr/bin/mkdir -p '/usr/local/pgsql/share/extension'
/usr/bin/mkdir -p '/usr/local/pgsql/share/extension'
/usr/bin/install -c -m 755  plpgsql_check.so '/usr/local/pgsql/lib/plpgsql_check.so'
/usr/bin/install -c -m 644 plpgsql_check.control '/usr/local/pgsql/share/extension/'
/usr/bin/install -c -m 644 plpgsql_check--0.9.sql '/usr/local/pgsql/share/extension/'
[root@localhost plpgsql_check]# exit
[pavel@localhost plpgsql_check]$ make USE_PGXS=1 installcheck
/usr/local/pgsql/lib/pgxs/src/makefiles/../../src/test/regress/pg_regress --inputdir=./ --psqldir='/usr/local/pgsql/bin'    --dbname=pl_regression --load-language=plpgsql --dbname=contrib_regression plpgsql_check_passive plpgsql_check_active plpgsql_check_active-9.5
(using postmaster on Unix socket, default port)
============== dropping database "contrib_regression" ==============
DROP DATABASE
============== creating database "contrib_regression" ==============
CREATE DATABASE
ALTER DATABASE
============== installing plpgsql                     ==============
CREATE LANGUAGE
============== running regression test queries        ==============
test plpgsql_check_passive    ... ok
test plpgsql_check_active     ... ok
test plpgsql_check_active-9.5 ... ok

=====================
 All 3 tests passed. 
=====================

Compilation on Ubuntu

Sometimes successful compilation can require libicu-dev package (PostgreSQL 10 and higher - when pg was compiled with ICU support)

sudo apt install libicu-dev

Compilation plpgsql_check on Windows

You can check precompiled dll libraries http://okbob.blogspot.cz/2015/02/plpgsqlcheck-is-available-for-microsoft.html

or compile by self:

  1. Download and install PostgreSQL for Win32 from http://www.enterprisedb.com
  2. Download and install Microsoft Visual C++ Express
  3. Lern tutorial http://blog.2ndquadrant.com/compiling-postgresql-extensions-visual-studio-windows
  4. Build plpgsql_check.dll
  5. Install plugin
  6. copy plpgsql_check.dll to PostgreSQL\14\lib
  7. copy plpgsql_check.control and plpgsql_check--2.1.sql to PostgreSQL\14\share\extension

Checked on

  • gcc on Linux (against all supported PostgreSQL)
  • clang 3.4 on Linux (against PostgreSQL 10)
  • for success regress tests the PostgreSQL 10 or higher is required

Compilation against PostgreSQL 10 requires libICU!

Licence

Copyright (c) Pavel Stehule (pavel.stehule@gmail.com)

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Note

If you like it, send a postcard to address

Pavel Stehule
Skalice 12
256 01 Benesov u Prahy
Czech Republic

I invite any questions, comments, bug reports, patches on mail address pavel.stehule@gmail.com


Author: okbob
Source Code: https://github.com/okbob/plpgsql_check
License: View license

#postgresql 

Plpgsql Check: Extension That Allows to Check Plpgsql Source Code.

plpgsql_check

I founded this project, because I wanted to publish the code I wrote in the last two years, when I tried to write enhanced checking for PostgreSQL upstream. It was not fully successful - integration into upstream requires some larger plpgsql refactoring - probably it will not be done in next years (now is Dec 2013). But written code is fully functional and can be used in production (and it is used in production). So, I created this extension to be available for all plpgsql developers.

If you like it and if you would to join to development of this extension, register yourself to postgresql extension hacking google group.

Features

  • check fields of referenced database objects and types inside embedded SQL
  • using correct types of function parameters
  • unused variables and function argumens, unmodified OUT argumens
  • partially detection of dead code (due RETURN command)
  • detection of missing RETURN command in function
  • try to identify unwanted hidden casts, that can be performance issue like unused indexes
  • possibility to collect relations and functions used by function
  • possibility to check EXECUTE stmt agaist SQL injection vulnerability

I invite any ideas, patches, bugreports.

plpgsql_check is next generation of plpgsql_lint. It allows to check source code by explicit call plpgsql_check_function.

PostgreSQL PostgreSQL 10, 11, 12, 13 and 14 are supported.

The SQL statements inside PL/pgSQL functions are checked by validator for semantic errors. These errors can be found by plpgsql_check_function:

Active mode

postgres=# CREATE EXTENSION plpgsql_check;
LOAD
postgres=# CREATE TABLE t1(a int, b int);
CREATE TABLE

postgres=#
CREATE OR REPLACE FUNCTION public.f1()
RETURNS void
LANGUAGE plpgsql
AS $function$
DECLARE r record;
BEGIN
  FOR r IN SELECT * FROM t1
  LOOP
    RAISE NOTICE '%', r.c; -- there is bug - table t1 missing "c" column
  END LOOP;
END;
$function$;

CREATE FUNCTION

postgres=# select f1(); -- execution doesn't find a bug due to empty table t1
  f1 
 ────
   
 (1 row)

postgres=# \x
Expanded display is on.
postgres=# select * from plpgsql_check_function_tb('f1()');
─[ RECORD 1 ]───────────────────────────
functionid │ f1
lineno     │ 6
statement  │ RAISE
sqlstate   │ 42703
message    │ record "r" has no field "c"
detail     │ [null]
hint       │ [null]
level      │ error
position   │ 0
query      │ [null]

postgres=# \sf+ f1
    CREATE OR REPLACE FUNCTION public.f1()
     RETURNS void
     LANGUAGE plpgsql
1       AS $function$
2       DECLARE r record;
3       BEGIN
4         FOR r IN SELECT * FROM t1
5         LOOP
6           RAISE NOTICE '%', r.c; -- there is bug - table t1 missing "c" column
7         END LOOP;
8       END;
9       $function$

Function plpgsql_check_function() has three possible formats: text, json or xml

select * from plpgsql_check_function('f1()', fatal_errors := false);
                         plpgsql_check_function                         
------------------------------------------------------------------------
 error:42703:4:SQL statement:column "c" of relation "t1" does not exist
 Query: update t1 set c = 30
 --                   ^
 error:42P01:7:RAISE:missing FROM-clause entry for table "r"
 Query: SELECT r.c
 --            ^
 error:42601:7:RAISE:too few parameters specified for RAISE
(7 rows)

postgres=# select * from plpgsql_check_function('fx()', format:='xml');
                 plpgsql_check_function                     
────────────────────────────────────────────────────────────────
 <Function oid="16400">                                        ↵
   <Issue>                                                     ↵
     <Level>error</level>                                      ↵
     <Sqlstate>42P01</Sqlstate>                                ↵
     <Message>relation "foo111" does not exist</Message>       ↵
     <Stmt lineno="3">RETURN</Stmt>                            ↵
     <Query position="23">SELECT (select a from foo111)</Query>↵
   </Issue>                                                    ↵
  </Function>
 (1 row)

Arguments

You can set level of warnings via function's parameters:

Mandatory arguments

  • function name or function signature - these functions requires function specification. Any function in PostgreSQL can be specified by Oid or by name or by signature. When you know oid or complete function's signature, you can use a regprocedure type parameter like 'fx()'::regprocedure or 16799::regprocedure. Possible alternative is using a name only, when function's name is unique - like 'fx'. When the name is not unique or the function doesn't exists it raises a error.

Optional arguments

relid DEFAULT 0 - oid of relation assigned with trigger function. It is necessary for check of any trigger function.

fatal_errors boolean DEFAULT true - stop on first error

other_warnings boolean DEFAULT true - show warnings like different attributes number in assignmenet on left and right side, variable overlaps function's parameter, unused variables, unwanted casting, ..

extra_warnings boolean DEFAULT true - show warnings like missing RETURN, shadowed variables, dead code, never read (unused) function's parameter, unmodified variables, modified auto variables, ..

performance_warnings boolean DEFAULT false - performance related warnings like declared type with type modificator, casting, implicit casts in where clause (can be reason why index is not used), ..

security_warnings boolean DEFAULT false - security related checks like SQL injection vulnerability detection

anyelementtype regtype DEFAULT 'int' - a real type used instead anyelement type

anyenumtype regtype DEFAULT '-' - a real type used instead anyenum type

anyrangetype regtype DEFAULT 'int4range' - a real type used instead anyrange type

anycompatibletype DEFAULT 'int' - a real type used instead anycompatible type

anycompatiblerangetype DEFAULT 'int4range' - a real type used instead anycompatible range type

without_warnings DEFAULT false - disable all warnings

all_warnings DEFAULT false - enable all warnings

newtable DEFAULT NULL, oldtable DEFAULT NULL - the names of NEW or OLD transitive tables. These parameters are required when transitive tables are used.

Triggers

When you want to check any trigger, you have to enter a relation that will be used together with trigger function

CREATE TABLE bar(a int, b int);

postgres=# \sf+ foo_trg
    CREATE OR REPLACE FUNCTION public.foo_trg()
         RETURNS trigger
         LANGUAGE plpgsql
1       AS $function$
2       BEGIN
3         NEW.c := NEW.a + NEW.b;
4         RETURN NEW;
5       END;
6       $function$

Missing relation specification

postgres=# select * from plpgsql_check_function('foo_trg()');
ERROR:  missing trigger relation
HINT:  Trigger relation oid must be valid

Correct trigger checking (with specified relation)

postgres=# select * from plpgsql_check_function('foo_trg()', 'bar');
                 plpgsql_check_function                 
--------------------------------------------------------
 error:42703:3:assignment:record "new" has no field "c"
(1 row)

For triggers with transitive tables you can set a oldtable or newtable parameters:

create or replace function footab_trig_func()
returns trigger as $$
declare x int;
begin
  if false then
    -- should be ok;
    select count(*) from newtab into x; 

    -- should fail;
    select count(*) from newtab where d = 10 into x;
  end if;
  return null;
end;
$$ language plpgsql;

select * from plpgsql_check_function('footab_trig_func','footab', newtable := 'newtab');

Mass check

You can use the plpgsql_check_function for mass check functions and mass check triggers. Please, test following queries:

-- check all nontrigger plpgsql functions
SELECT p.oid, p.proname, plpgsql_check_function(p.oid)
   FROM pg_catalog.pg_namespace n
   JOIN pg_catalog.pg_proc p ON pronamespace = n.oid
   JOIN pg_catalog.pg_language l ON p.prolang = l.oid
  WHERE l.lanname = 'plpgsql' AND p.prorettype <> 2279;

or

SELECT p.proname, tgrelid::regclass, cf.*
   FROM pg_proc p
        JOIN pg_trigger t ON t.tgfoid = p.oid 
        JOIN pg_language l ON p.prolang = l.oid
        JOIN pg_namespace n ON p.pronamespace = n.oid,
        LATERAL plpgsql_check_function(p.oid, t.tgrelid) cf
  WHERE n.nspname = 'public' and l.lanname = 'plpgsql'

or

-- check all plpgsql functions (functions or trigger functions with defined triggers)
SELECT
    (pcf).functionid::regprocedure, (pcf).lineno, (pcf).statement,
    (pcf).sqlstate, (pcf).message, (pcf).detail, (pcf).hint, (pcf).level,
    (pcf)."position", (pcf).query, (pcf).context
FROM
(
    SELECT
        plpgsql_check_function_tb(pg_proc.oid, COALESCE(pg_trigger.tgrelid, 0)) AS pcf
    FROM pg_proc
    LEFT JOIN pg_trigger
        ON (pg_trigger.tgfoid = pg_proc.oid)
    WHERE
        prolang = (SELECT lang.oid FROM pg_language lang WHERE lang.lanname = 'plpgsql') AND
        pronamespace <> (SELECT nsp.oid FROM pg_namespace nsp WHERE nsp.nspname = 'pg_catalog') AND
        -- ignore unused triggers
        (pg_proc.prorettype <> (SELECT typ.oid FROM pg_type typ WHERE typ.typname = 'trigger') OR
         pg_trigger.tgfoid IS NOT NULL)
    OFFSET 0
) ss
ORDER BY (pcf).functionid::regprocedure::text, (pcf).lineno

Passive mode

Functions should be checked on start - plpgsql_check module must be loaded.

Configuration

plpgsql_check.mode = [ disabled | by_function | fresh_start | every_start ]
plpgsql_check.fatal_errors = [ yes | no ]

plpgsql_check.show_nonperformance_warnings = false
plpgsql_check.show_performance_warnings = false

Default mode is by_function, that means that the enhanced check is done only in active mode - by plpgsql_check_function. fresh_start means cold start.

You can enable passive mode by

load 'plpgsql'; -- 1.1 and higher doesn't need it
load 'plpgsql_check';
set plpgsql_check.mode = 'every_start';

SELECT fx(10); -- run functions - function is checked before runtime starts it

Limits

plpgsql_check should find almost all errors on really static code. When developer use some PLpgSQL's dynamic features like dynamic SQL or record data type, then false positives are possible. These should be rare - in well written code - and then the affected function should be redesigned or plpgsql_check should be disabled for this function.

CREATE OR REPLACE FUNCTION f1()
RETURNS void AS $$
DECLARE r record;
BEGIN
  FOR r IN EXECUTE 'SELECT * FROM t1'
  LOOP
    RAISE NOTICE '%', r.c;
  END LOOP;
END;
$$ LANGUAGE plpgsql SET plpgsql.enable_check TO false;

A usage of plpgsql_check adds a small overhead (in enabled passive mode) and you should use it only in develop or preprod environments.

Dynamic SQL

This module doesn't check queries that are assembled in runtime. It is not possible to identify results of dynamic queries - so plpgsql_check cannot to set correct type to record variables and cannot to check a dependent SQLs and expressions.

When type of record's variable is not know, you can assign it explicitly with pragma type:

DECLARE r record;
BEGIN
  EXECUTE format('SELECT * FROM %I', _tablename) INTO r;
  PERFORM plpgsql_check_pragma('type: r (id int, processed bool)');
  IF NOT r.processed THEN
    ...

Attention: The SQL injection check can detect only some SQL injection vulnerabilities. This tool cannot be used for security audit! Some issues should not be detected. This check can raise false alarms too - probably when variable is sanitized by other command or when value is of some compose type. 

Refcursors

plpgsql_check should not to detect structure of referenced cursors. A reference on cursor in PLpgSQL is implemented as name of global cursor. In check time, the name is not known (not in all possibilities), and global cursor doesn't exist. It is significant break for any static analyse. PLpgSQL cannot to set correct type for record variables and cannot to check a dependent SQLs and expressions. A solution is same like dynamic SQL. Don't use record variable as target when you use refcursor type or disable plpgsql_check for these functions.

CREATE OR REPLACE FUNCTION foo(refcur_var refcursor)
RETURNS void AS $$
DECLARE
  rec_var record;
BEGIN
  FETCH refcur_var INTO rec_var; -- this is STOP for plpgsql_check
  RAISE NOTICE '%', rec_var;     -- record rec_var is not assigned yet error

In this case a record type should not be used (use known rowtype instead):

CREATE OR REPLACE FUNCTION foo(refcur_var refcursor)
RETURNS void AS $$
DECLARE
  rec_var some_rowtype;
BEGIN
  FETCH refcur_var INTO rec_var;
  RAISE NOTICE '%', rec_var;

Temporary tables

plpgsql_check cannot verify queries over temporary tables that are created in plpgsql's function runtime. For this use case it is necessary to create a fake temp table or disable plpgsql_check for this function.

In reality temp tables are stored in own (per user) schema with higher priority than persistent tables. So you can do (with following trick safetly):

CREATE OR REPLACE FUNCTION public.disable_dml()
RETURNS trigger
LANGUAGE plpgsql AS $function$
BEGIN
  RAISE EXCEPTION SQLSTATE '42P01'
     USING message = format('this instance of %I table doesn''t allow any DML operation', TG_TABLE_NAME),
           hint = format('you should to run "CREATE TEMP TABLE %1$I(LIKE %1$I INCLUDING ALL);" statement',
                         TG_TABLE_NAME);
  RETURN NULL;
END;
$function$;

CREATE TABLE foo(a int, b int); -- doesn't hold data ever
CREATE TRIGGER foo_disable_dml
   BEFORE INSERT OR UPDATE OR DELETE ON foo
   EXECUTE PROCEDURE disable_dml();

postgres=# INSERT INTO  foo VALUES(10,20);
ERROR:  this instance of foo table doesn't allow any DML operation
HINT:  you should to run "CREATE TEMP TABLE foo(LIKE foo INCLUDING ALL);" statement
postgres=# 

CREATE TABLE
postgres=# INSERT INTO  foo VALUES(10,20);
INSERT 0 1

This trick emulates GLOBAL TEMP tables partially and it allows a statical validation. Other possibility is using a [template foreign data wrapper] (https://github.com/okbob/template_fdw)

You can use pragma table and create ephemeral table:

BEGIN
   CREATE TEMP TABLE xxx(a int);
   PERFORM plpgsql_check_pragma('table: xxx(a int)');
   INSERT INTO xxx VALUES(10);

Dependency list

A function plpgsql_show_dependency_tb can show all functions, operators and relations used inside processed function:

postgres=# select * from plpgsql_show_dependency_tb('testfunc(int,float)');
┌──────────┬───────┬────────┬─────────┬────────────────────────────┐
│   type   │  oid  │ schema │  name   │           params           │
╞══════════╪═══════╪════════╪═════════╪════════════════════════════╡
│ FUNCTION │ 36008 │ public │ myfunc1 │ (integer,double precision) │
│ FUNCTION │ 35999 │ public │ myfunc2 │ (integer,double precision) │
│ OPERATOR │ 36007 │ public │ **      │ (integer,integer)          │
│ RELATION │ 36005 │ public │ myview  │                            │
│ RELATION │ 36002 │ public │ mytable │                            │
└──────────┴───────┴────────┴─────────┴────────────────────────────┘
(4 rows)

Profiler

The plpgsql_check contains simple profiler of plpgsql functions and procedures. It can work with/without a access to shared memory. It depends on shared_preload_libraries config. When plpgsql_check was initialized by shared_preload_libraries, then it can allocate shared memory, and function's profiles are stored there. When plpgsql_check cannot to allocate shared momory, the profile is stored in session memory.

Due dependencies, shared_preload_libraries should to contains plpgsql first

postgres=# show shared_preload_libraries ;
┌──────────────────────────┐
│ shared_preload_libraries │
╞══════════════════════════╡
│ plpgsql,plpgsql_check    │
└──────────────────────────┘
(1 row)

The profiler is active when GUC plpgsql_check.profiler is on. The profiler doesn't require shared memory, but if there are not shared memory, then the profile is limmitted just to active session.

When plpgsql_check is initialized by shared_preload_libraries, another GUC is available to configure the amount of shared memory used by the profiler: plpgsql_check.profiler_max_shared_chunks. This defines the maximum number of statements chunk that can be stored in shared memory. For each plpgsql function (or procedure), the whole content is split into chunks of 30 statements. If needed, multiple chunks can be used to store the whole content of a single function. A single chunk is 1704 bytes. The default value for this GUC is 15000, which should be enough for big projects containing hundred of thousands of statements in plpgsql, and will consume about 24MB of memory. If your project doesn't require that much number of chunks, you can set this parameter to a smaller number in order to decrease the memory usage. The minimum value is 50 (which should consume about 83kB of memory), and the maximum value is 100000 (which should consume about 163MB of memory). Changing this parameter requires a PostgreSQL restart.

The profiler will also retrieve the query identifier for each instruction that contains an expression or optimizable statement. Note that this requires pg_stat_statements, or another similar third-party extension), to be installed. There are some limitations to the query identifier retrieval:

  • if a plpgsql expression contains underlying statements, only the top level query identifier will be retrieved
  • the profiler doesn't compute query identifier by itself but relies on external extension, such as pg_stat_statements, for that. It means that depending on the external extension behavior, you may not be able to see a query identifier for some statements. That's for instance the case with DDL statements, as pg_stat_statements doesn't expose the query identifier for such queries.
  • a query identifier is retrieved only for instructions containing expressions. This means that plpgsql_profiler_function_tb() function can report less query identifier than instructions on a single line.

Attention: A update of shared profiles can decrease performance on servers under higher load.

The profile can be displayed by function plpgsql_profiler_function_tb:

postgres=# select lineno, avg_time, source from plpgsql_profiler_function_tb('fx(int)');
┌────────┬──────────┬───────────────────────────────────────────────────────────────────┐
│ lineno │ avg_time │                              source                               │
╞════════╪══════════╪═══════════════════════════════════════════════════════════════════╡
│      1 │          │                                                                   │
│      2 │          │ declare result int = 0;                                           │
│      3 │    0.075 │ begin                                                             │
│      4 │    0.202 │   for i in 1..$1 loop                                             │
│      5 │    0.005 │     select result + i into result; select result + i into result; │
│      6 │          │   end loop;                                                       │
│      7 │        0 │   return result;                                                  │
│      8 │          │ end;                                                              │
└────────┴──────────┴───────────────────────────────────────────────────────────────────┘
(9 rows)

The profile per statements (not per line) can be displayed by function plpgsql_profiler_function_statements_tb:

        CREATE OR REPLACE FUNCTION public.fx1(a integer)
         RETURNS integer
         LANGUAGE plpgsql
1       AS $function$
2       begin
3         if a > 10 then
4           raise notice 'ahoj';
5           return -1;
6         else
7           raise notice 'nazdar';
8           return 1;
9         end if;
10      end;
11      $function$

postgres=# select stmtid, parent_stmtid, parent_note, lineno, exec_stmts, stmtname
             from plpgsql_profiler_function_statements_tb('fx1');
┌────────┬───────────────┬─────────────┬────────┬────────────┬─────────────────┐
│ stmtid │ parent_stmtid │ parent_note │ lineno │ exec_stmts │    stmtname     │
╞════════╪═══════════════╪═════════════╪════════╪════════════╪═════════════════╡
│      0 │             ∅ │ ∅           │      2 │          0 │ statement block │
│      1 │             0 │ body        │      3 │          0 │ IF              │
│      2 │             1 │ then body   │      4 │          0 │ RAISE           │
│      3 │             1 │ then body   │      5 │          0 │ RETURN          │
│      4 │             1 │ else body   │      7 │          0 │ RAISE           │
│      5 │             1 │ else body   │      8 │          0 │ RETURN          │
└────────┴───────────────┴─────────────┴────────┴────────────┴─────────────────┘
(6 rows)

All stored profiles can be displayed by calling function plpgsql_profiler_functions_all:

postgres=# select * from plpgsql_profiler_functions_all();
┌───────────────────────┬────────────┬────────────┬──────────┬─────────────┬──────────┬──────────┐
│        funcoid        │ exec_count │ total_time │ avg_time │ stddev_time │ min_time │ max_time │
╞═══════════════════════╪════════════╪════════════╪══════════╪═════════════╪══════════╪══════════╡
│ fxx(double precision) │          1 │       0.01 │     0.01 │        0.00 │     0.01 │     0.01 │
└───────────────────────┴────────────┴────────────┴──────────┴─────────────┴──────────┴──────────┘
(1 row)

There are two functions for cleaning stored profiles: plpgsql_profiler_reset_all() and plpgsql_profiler_reset(regprocedure).

Coverage metrics

plpgsql_check provides two functions:

  • plpgsql_coverage_statements(name)
  • plpgsql_coverage_branches(name)

Note

There is another very good PLpgSQL profiler - https://bitbucket.org/openscg/plprofiler

My extension is designed to be simple for use and practical. Nothing more or less.

plprofiler is more complex. It build call graphs and from this graph it can creates flame graph of execution times.

Both extensions can be used together with buildin PostgreSQL's feature - tracking functions.

set track_functions to 'pl';
...
select * from pg_stat_user_functions;

Tracer

plpgsql_check provides a tracing possibility - in this mode you can see notices on start or end functions (terse and default verbosity) and start or end statements (verbose verbosity). For default and verbose verbosity the content of function arguments is displayed. The content of related variables are displayed when verbosity is verbose.

postgres=# do $$ begin perform fx(10,null, 'now', e'stěhule'); end; $$;
NOTICE:  #0 ->> start of inline_code_block (Oid=0)
NOTICE:  #2   ->> start of function fx(integer,integer,date,text) (Oid=16405)
NOTICE:  #2        call by inline_code_block line 1 at PERFORM
NOTICE:  #2       "a" => '10', "b" => null, "c" => '2020-08-03', "d" => 'stěhule'
NOTICE:  #4     ->> start of function fx(integer) (Oid=16404)
NOTICE:  #4          call by fx(integer,integer,date,text) line 1 at PERFORM
NOTICE:  #4         "a" => '10'
NOTICE:  #4     <<- end of function fx (elapsed time=0.098 ms)
NOTICE:  #2   <<- end of function fx (elapsed time=0.399 ms)
NOTICE:  #0 <<- end of block (elapsed time=0.754 ms)

The number after # is a execution frame counter (this number is related to deep of error context stack). It allows to pair start end and of function.

Tracing is enabled by setting plpgsql_check.tracer to on. Attention - enabling this behaviour has significant negative impact on performance (unlike the profiler). You can set a level for output used by tracer plpgsql_check.tracer_errlevel (default is notice). The output content is limited by length specified by plpgsql_check.tracer_variable_max_length configuration variable.

In terse verbose mode the output is reduced:

postgres=# set plpgsql_check.tracer_verbosity TO terse;
SET
postgres=# do $$ begin perform fx(10,null, 'now', e'stěhule'); end; $$;
NOTICE:  #0 start of inline code block (oid=0)
NOTICE:  #2 start of fx (oid=16405)
NOTICE:  #4 start of fx (oid=16404)
NOTICE:  #4 end of fx
NOTICE:  #2 end of fx
NOTICE:  #0 end of inline code block

In verbose mode the output is extended about statement details:

postgres=# do $$ begin perform fx(10,null, 'now', e'stěhule'); end; $$;
NOTICE:  #0            ->> start of block inline_code_block (oid=0)
NOTICE:  #0.1       1  --> start of PERFORM
NOTICE:  #2              ->> start of function fx(integer,integer,date,text) (oid=16405)
NOTICE:  #2                   call by inline_code_block line 1 at PERFORM
NOTICE:  #2                  "a" => '10', "b" => null, "c" => '2020-08-04', "d" => 'stěhule'
NOTICE:  #2.1       1    --> start of PERFORM
NOTICE:  #2.1                "a" => '10'
NOTICE:  #4                ->> start of function fx(integer) (oid=16404)
NOTICE:  #4                     call by fx(integer,integer,date,text) line 1 at PERFORM
NOTICE:  #4                    "a" => '10'
NOTICE:  #4.1       6      --> start of assignment
NOTICE:  #4.1                  "a" => '10', "b" => '20'
NOTICE:  #4.1              <-- end of assignment (elapsed time=0.076 ms)
NOTICE:  #4.1                  "res" => '130'
NOTICE:  #4.2       7      --> start of RETURN
NOTICE:  #4.2                  "res" => '130'
NOTICE:  #4.2              <-- end of RETURN (elapsed time=0.054 ms)
NOTICE:  #4                <<- end of function fx (elapsed time=0.373 ms)
NOTICE:  #2.1            <-- end of PERFORM (elapsed time=0.589 ms)
NOTICE:  #2              <<- end of function fx (elapsed time=0.727 ms)
NOTICE:  #0.1          <-- end of PERFORM (elapsed time=1.147 ms)
NOTICE:  #0            <<- end of block (elapsed time=1.286 ms)

Special feature of tracer is tracing of ASSERT statement when plpgsql_check.trace_assert is on. When plpgsql_check.trace_assert_verbosity is DEFAULT, then all function's or procedure's variables are displayed when assert expression is false. When this configuration is VERBOSE then all variables from all plpgsql frames are displayed. This behaviour is independent on plpgsql.check_asserts value. It can be used, although the assertions are disabled in plpgsql runtime.

postgres=# set plpgsql_check.tracer to off;
postgres=# set plpgsql_check.trace_assert_verbosity TO verbose;

postgres=# do $$ begin perform fx(10,null, 'now', e'stěhule'); end; $$;
NOTICE:  #4 PLpgSQL assert expression (false) on line 12 of fx(integer) is false
NOTICE:   "a" => '10', "res" => null, "b" => '20'
NOTICE:  #2 PL/pgSQL function fx(integer,integer,date,text) line 1 at PERFORM
NOTICE:   "a" => '10', "b" => null, "c" => '2020-08-05', "d" => 'stěhule'
NOTICE:  #0 PL/pgSQL function inline_code_block line 1 at PERFORM
ERROR:  assertion failed
CONTEXT:  PL/pgSQL function fx(integer) line 12 at ASSERT
SQL statement "SELECT fx(a)"
PL/pgSQL function fx(integer,integer,date,text) line 1 at PERFORM
SQL statement "SELECT fx(10,null, 'now', e'stěhule')"
PL/pgSQL function inline_code_block line 1 at PERFORM

postgres=# set plpgsql.check_asserts to off;
SET
postgres=# do $$ begin perform fx(10,null, 'now', e'stěhule'); end; $$;
NOTICE:  #4 PLpgSQL assert expression (false) on line 12 of fx(integer) is false
NOTICE:   "a" => '10', "res" => null, "b" => '20'
NOTICE:  #2 PL/pgSQL function fx(integer,integer,date,text) line 1 at PERFORM
NOTICE:   "a" => '10', "b" => null, "c" => '2020-08-05', "d" => 'stěhule'
NOTICE:  #0 PL/pgSQL function inline_code_block line 1 at PERFORM
DO

Attention - SECURITY

Tracer prints content of variables or function arguments. For security definer function, this content can hold security sensitive data. This is reason why tracer is disabled by default and should be enabled only with super user rights plpgsql_check.enable_tracer.

Pragma

You can configure plpgsql_check behave inside checked function with "pragma" function. This is a analogy of PL/SQL or ADA language of PRAGMA feature. PLpgSQL doesn't support PRAGMA, but plpgsql_check detects function named plpgsql_check_pragma and get options from parameters of this function. These plpgsql_check options are valid to end of group of statements.

CREATE OR REPLACE FUNCTION test()
RETURNS void AS $$
BEGIN
  ...
  -- for following statements disable check
  PERFORM plpgsql_check_pragma('disable:check');
  ...
  -- enable check again
  PERFORM plpgsql_check_pragma('enable:check');
  ...
END;
$$ LANGUAGE plpgsql;

The function plpgsql_check_pragma is immutable function that returns one. It is defined by plpgsql_check extension. You can declare alternative plpgsql_check_pragma function like:

CREATE OR REPLACE FUNCTION plpgsql_check_pragma(VARIADIC args[])
RETURNS int AS $$
SELECT 1
$$ LANGUAGE sql IMMUTABLE;

Using pragma function in declaration part of top block sets options on function level too.

CREATE OR REPLACE FUNCTION test()
RETURNS void AS $$
DECLARE
  aux int := plpgsql_check_pragma('disable:extra_warnings');
  ...

Shorter syntax for pragma is supported too:

CREATE OR REPLACE FUNCTION test()
RETURNS void AS $$
DECLARE r record;
BEGIN
  PERFORM 'PRAGMA:TYPE:r (a int, b int)';
  PERFORM 'PRAGMA:TABLE: x (like pg_class)';
  ...

Supported pragmas

echo:str - print string (for testing)

status:check,status:tracer, status:other_warnings, status:performance_warnings, status:extra_warnings,status:security_warnings

enable:check,enable:tracer, enable:other_warnings, enable:performance_warnings, enable:extra_warnings,enable:security_warnings

disable:check,disable:tracer, disable:other_warnings, disable:performance_warnings, disable:extra_warnings,disable:security_warnings

type:varname typename or type:varname (fieldname type, ...) - set type to variable of record type

table: name (column_name type, ...) or table: name (like tablename) - create ephereal table

Pragmas enable:tracer and disable:tracerare active for Postgres 12 and higher

Compilation

You need a development environment for PostgreSQL extensions:

make clean
make install

result:

[pavel@localhost plpgsql_check]$ make USE_PGXS=1 clean
rm -f plpgsql_check.so   libplpgsql_check.a  libplpgsql_check.pc
rm -f plpgsql_check.o
rm -rf results/ regression.diffs regression.out tmp_check/ log/
[pavel@localhost plpgsql_check]$ make USE_PGXS=1 all
clang -O2 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fpic -I/usr/local/pgsql/lib/pgxs/src/makefiles/../../src/pl/plpgsql/src -I. -I./ -I/usr/local/pgsql/include/server -I/usr/local/pgsql/include/internal -D_GNU_SOURCE   -c -o plpgsql_check.o plpgsql_check.c
clang -O2 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fpic -I/usr/local/pgsql/lib/pgxs/src/makefiles/../../src/pl/plpgsql/src -shared -o plpgsql_check.so plpgsql_check.o -L/usr/local/pgsql/lib -Wl,--as-needed -Wl,-rpath,'/usr/local/pgsql/lib',--enable-new-dtags  
[pavel@localhost plpgsql_check]$ su root
Password: *******
[root@localhost plpgsql_check]# make USE_PGXS=1 install
/usr/bin/mkdir -p '/usr/local/pgsql/lib'
/usr/bin/mkdir -p '/usr/local/pgsql/share/extension'
/usr/bin/mkdir -p '/usr/local/pgsql/share/extension'
/usr/bin/install -c -m 755  plpgsql_check.so '/usr/local/pgsql/lib/plpgsql_check.so'
/usr/bin/install -c -m 644 plpgsql_check.control '/usr/local/pgsql/share/extension/'
/usr/bin/install -c -m 644 plpgsql_check--0.9.sql '/usr/local/pgsql/share/extension/'
[root@localhost plpgsql_check]# exit
[pavel@localhost plpgsql_check]$ make USE_PGXS=1 installcheck
/usr/local/pgsql/lib/pgxs/src/makefiles/../../src/test/regress/pg_regress --inputdir=./ --psqldir='/usr/local/pgsql/bin'    --dbname=pl_regression --load-language=plpgsql --dbname=contrib_regression plpgsql_check_passive plpgsql_check_active plpgsql_check_active-9.5
(using postmaster on Unix socket, default port)
============== dropping database "contrib_regression" ==============
DROP DATABASE
============== creating database "contrib_regression" ==============
CREATE DATABASE
ALTER DATABASE
============== installing plpgsql                     ==============
CREATE LANGUAGE
============== running regression test queries        ==============
test plpgsql_check_passive    ... ok
test plpgsql_check_active     ... ok
test plpgsql_check_active-9.5 ... ok

=====================
 All 3 tests passed. 
=====================

Compilation on Ubuntu

Sometimes successful compilation can require libicu-dev package (PostgreSQL 10 and higher - when pg was compiled with ICU support)

sudo apt install libicu-dev

Compilation plpgsql_check on Windows

You can check precompiled dll libraries http://okbob.blogspot.cz/2015/02/plpgsqlcheck-is-available-for-microsoft.html

or compile by self:

  1. Download and install PostgreSQL for Win32 from http://www.enterprisedb.com
  2. Download and install Microsoft Visual C++ Express
  3. Lern tutorial http://blog.2ndquadrant.com/compiling-postgresql-extensions-visual-studio-windows
  4. Build plpgsql_check.dll
  5. Install plugin
  6. copy plpgsql_check.dll to PostgreSQL\14\lib
  7. copy plpgsql_check.control and plpgsql_check--2.1.sql to PostgreSQL\14\share\extension

Checked on

  • gcc on Linux (against all supported PostgreSQL)
  • clang 3.4 on Linux (against PostgreSQL 10)
  • for success regress tests the PostgreSQL 10 or higher is required

Compilation against PostgreSQL 10 requires libICU!

Licence

Copyright (c) Pavel Stehule (pavel.stehule@gmail.com)

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Note

If you like it, send a postcard to address

Pavel Stehule
Skalice 12
256 01 Benesov u Prahy
Czech Republic

I invite any questions, comments, bug reports, patches on mail address pavel.stehule@gmail.com


Author: okbob
Source Code: https://github.com/okbob/plpgsql_check
License: View license

#postgresql 

Veronica  Roob

Veronica Roob

1653475560

A Pure PHP Implementation Of The MessagePack Serialization Format

msgpack.php

A pure PHP implementation of the MessagePack serialization format.

Features

Installation

The recommended way to install the library is through Composer:

composer require rybakit/msgpack

Usage

Packing

To pack values you can either use an instance of a Packer:

$packer = new Packer();
$packed = $packer->pack($value);

or call a static method on the MessagePack class:

$packed = MessagePack::pack($value);

In the examples above, the method pack automatically packs a value depending on its type. However, not all PHP types can be uniquely translated to MessagePack types. For example, the MessagePack format defines map and array types, which are represented by a single array type in PHP. By default, the packer will pack a PHP array as a MessagePack array if it has sequential numeric keys, starting from 0 and as a MessagePack map otherwise:

$mpArr1 = $packer->pack([1, 2]);               // MP array [1, 2]
$mpArr2 = $packer->pack([0 => 1, 1 => 2]);     // MP array [1, 2]
$mpMap1 = $packer->pack([0 => 1, 2 => 3]);     // MP map {0: 1, 2: 3}
$mpMap2 = $packer->pack([1 => 2, 2 => 3]);     // MP map {1: 2, 2: 3}
$mpMap3 = $packer->pack(['a' => 1, 'b' => 2]); // MP map {a: 1, b: 2}

However, sometimes you need to pack a sequential array as a MessagePack map. To do this, use the packMap method:

$mpMap = $packer->packMap([1, 2]); // {0: 1, 1: 2}

Here is a list of type-specific packing methods:

$packer->packNil();           // MP nil
$packer->packBool(true);      // MP bool
$packer->packInt(42);         // MP int
$packer->packFloat(M_PI);     // MP float (32 or 64)
$packer->packFloat32(M_PI);   // MP float 32
$packer->packFloat64(M_PI);   // MP float 64
$packer->packStr('foo');      // MP str
$packer->packBin("\x80");     // MP bin
$packer->packArray([1, 2]);   // MP array
$packer->packMap(['a' => 1]); // MP map
$packer->packExt(1, "\xaa");  // MP ext

Check the "Custom types" section below on how to pack custom types.

Packing options

The Packer object supports a number of bitmask-based options for fine-tuning the packing process (defaults are in bold):

NameDescription
FORCE_STRForces PHP strings to be packed as MessagePack UTF-8 strings
FORCE_BINForces PHP strings to be packed as MessagePack binary data
DETECT_STR_BINDetects MessagePack str/bin type automatically
  
FORCE_ARRForces PHP arrays to be packed as MessagePack arrays
FORCE_MAPForces PHP arrays to be packed as MessagePack maps
DETECT_ARR_MAPDetects MessagePack array/map type automatically
  
FORCE_FLOAT32Forces PHP floats to be packed as 32-bits MessagePack floats
FORCE_FLOAT64Forces PHP floats to be packed as 64-bits MessagePack floats

The type detection mode (DETECT_STR_BIN/DETECT_ARR_MAP) adds some overhead which can be noticed when you pack large (16- and 32-bit) arrays or strings. However, if you know the value type in advance (for example, you only work with UTF-8 strings or/and associative arrays), you can eliminate this overhead by forcing the packer to use the appropriate type, which will save it from running the auto-detection routine. Another option is to explicitly specify the value type. The library provides 2 auxiliary classes for this, Map and Bin. Check the "Custom types" section below for details.

Examples:

// detect str/bin type and pack PHP 64-bit floats (doubles) to MP 32-bit floats
$packer = new Packer(PackOptions::DETECT_STR_BIN | PackOptions::FORCE_FLOAT32);

// these will throw MessagePack\Exception\InvalidOptionException
$packer = new Packer(PackOptions::FORCE_STR | PackOptions::FORCE_BIN);
$packer = new Packer(PackOptions::FORCE_FLOAT32 | PackOptions::FORCE_FLOAT64);

Unpacking

To unpack data you can either use an instance of a BufferUnpacker:

$unpacker = new BufferUnpacker();

$unpacker->reset($packed);
$value = $unpacker->unpack();

or call a static method on the MessagePack class:

$value = MessagePack::unpack($packed);

If the packed data is received in chunks (e.g. when reading from a stream), use the tryUnpack method, which attempts to unpack data and returns an array of unpacked messages (if any) instead of throwing an InsufficientDataException:

while ($chunk = ...) {
    $unpacker->append($chunk);
    if ($messages = $unpacker->tryUnpack()) {
        return $messages;
    }
}

If you want to unpack from a specific position in a buffer, use seek:

$unpacker->seek(42); // set position equal to 42 bytes
$unpacker->seek(-8); // set position to 8 bytes before the end of the buffer

To skip bytes from the current position, use skip:

$unpacker->skip(10); // set position to 10 bytes ahead of the current position

To get the number of remaining (unread) bytes in the buffer:

$unreadBytesCount = $unpacker->getRemainingCount();

To check whether the buffer has unread data:

$hasUnreadBytes = $unpacker->hasRemaining();

If needed, you can remove already read data from the buffer by calling:

$releasedBytesCount = $unpacker->release();

With the read method you can read raw (packed) data:

$packedData = $unpacker->read(2); // read 2 bytes

Besides the above methods BufferUnpacker provides type-specific unpacking methods, namely:

$unpacker->unpackNil();   // PHP null
$unpacker->unpackBool();  // PHP bool
$unpacker->unpackInt();   // PHP int
$unpacker->unpackFloat(); // PHP float
$unpacker->unpackStr();   // PHP UTF-8 string
$unpacker->unpackBin();   // PHP binary string
$unpacker->unpackArray(); // PHP sequential array
$unpacker->unpackMap();   // PHP associative array
$unpacker->unpackExt();   // PHP MessagePack\Type\Ext object

Unpacking options

The BufferUnpacker object supports a number of bitmask-based options for fine-tuning the unpacking process (defaults are in bold):

NameDescription
BIGINT_AS_STRConverts overflowed integers to strings [1]
BIGINT_AS_GMPConverts overflowed integers to GMP objects [2]
BIGINT_AS_DECConverts overflowed integers to Decimal\Decimal objects [3]

1. The binary MessagePack format has unsigned 64-bit as its largest integer data type, but PHP does not support such integers, which means that an overflow can occur during unpacking.

2. Make sure the GMP extension is enabled.

3. Make sure the Decimal extension is enabled.

Examples:

$packedUint64 = "\xcf"."\xff\xff\xff\xff"."\xff\xff\xff\xff";

$unpacker = new BufferUnpacker($packedUint64);
var_dump($unpacker->unpack()); // string(20) "18446744073709551615"

$unpacker = new BufferUnpacker($packedUint64, UnpackOptions::BIGINT_AS_GMP);
var_dump($unpacker->unpack()); // object(GMP) {...}

$unpacker = new BufferUnpacker($packedUint64, UnpackOptions::BIGINT_AS_DEC);
var_dump($unpacker->unpack()); // object(Decimal\Decimal) {...}

Custom types

In addition to the basic types, the library provides functionality to serialize and deserialize arbitrary types. This can be done in several ways, depending on your use case. Let's take a look at them.

Type objects

If you need to serialize an instance of one of your classes into one of the basic MessagePack types, the best way to do this is to implement the CanBePacked interface in the class. A good example of such a class is the Map type class that comes with the library. This type is useful when you want to explicitly specify that a given PHP array should be packed as a MessagePack map without triggering an automatic type detection routine:

$packer = new Packer();

$packedMap = $packer->pack(new Map([1, 2, 3]));
$packedArray = $packer->pack([1, 2, 3]);

More type examples can be found in the src/Type directory.

Type transformers

As with type objects, type transformers are only responsible for serializing values. They should be used when you need to serialize a value that does not implement the CanBePacked interface. Examples of such values could be instances of built-in or third-party classes that you don't own, or non-objects such as resources.

A transformer class must implement the CanPack interface. To use a transformer, it must first be registered in the packer. Here is an example of how to serialize PHP streams into the MessagePack bin format type using one of the supplied transformers, StreamTransformer:

$packer = new Packer(null, [new StreamTransformer()]);

$packedBin = $packer->pack(fopen('/path/to/file', 'r+'));

More type transformer examples can be found in the src/TypeTransformer directory.

Extensions

In contrast to the cases described above, extensions are intended to handle extension types and are responsible for both serialization and deserialization of values (types).

An extension class must implement the Extension interface. To use an extension, it must first be registered in the packer and the unpacker.

The MessagePack specification divides extension types into two groups: predefined and application-specific. Currently, there is only one predefined type in the specification, Timestamp.

Timestamp

The Timestamp extension type is a predefined type. Support for this type in the library is done through the TimestampExtension class. This class is responsible for handling Timestamp objects, which represent the number of seconds and optional adjustment in nanoseconds:

$timestampExtension = new TimestampExtension();

$packer = new Packer();
$packer = $packer->extendWith($timestampExtension);

$unpacker = new BufferUnpacker();
$unpacker = $unpacker->extendWith($timestampExtension);

$packedTimestamp = $packer->pack(Timestamp::now());
$timestamp = $unpacker->reset($packedTimestamp)->unpack();

$seconds = $timestamp->getSeconds();
$nanoseconds = $timestamp->getNanoseconds();

When using the MessagePack class, the Timestamp extension is already registered:

$packedTimestamp = MessagePack::pack(Timestamp::now());
$timestamp = MessagePack::unpack($packedTimestamp);

Application-specific extensions

In addition, the format can be extended with your own types. For example, to make the built-in PHP DateTime objects first-class citizens in your code, you can create a corresponding extension, as shown in the example. Please note, that custom extensions have to be registered with a unique extension ID (an integer from 0 to 127).

More extension examples can be found in the examples/MessagePack directory.

To learn more about how extension types can be useful, check out this article.

Exceptions

If an error occurs during packing/unpacking, a PackingFailedException or an UnpackingFailedException will be thrown, respectively. In addition, an InsufficientDataException can be thrown during unpacking.

An InvalidOptionException will be thrown in case an invalid option (or a combination of mutually exclusive options) is used.

Tests

Run tests as follows:

vendor/bin/phpunit

Also, if you already have Docker installed, you can run the tests in a docker container. First, create a container:

./dockerfile.sh | docker build -t msgpack -

The command above will create a container named msgpack with PHP 8.1 runtime. You may change the default runtime by defining the PHP_IMAGE environment variable:

PHP_IMAGE='php:8.0-cli' ./dockerfile.sh | docker build -t msgpack -

See a list of various images here.

Then run the unit tests:

docker run --rm -v $PWD:/msgpack -w /msgpack msgpack

Fuzzing

To ensure that the unpacking works correctly with malformed/semi-malformed data, you can use a testing technique called Fuzzing. The library ships with a help file (target) for PHP-Fuzzer and can be used as follows:

php-fuzzer fuzz tests/fuzz_buffer_unpacker.php

Performance

To check performance, run:

php -n -dzend_extension=opcache.so \
-dpcre.jit=1 -dopcache.enable=1 -dopcache.enable_cli=1 \
tests/bench.php

Example output

Filter: MessagePack\Tests\Perf\Filter\ListFilter
Rounds: 3
Iterations: 100000

=============================================
Test/Target            Packer  BufferUnpacker
---------------------------------------------
nil .................. 0.0030 ........ 0.0139
false ................ 0.0037 ........ 0.0144
true ................. 0.0040 ........ 0.0137
7-bit uint #1 ........ 0.0052 ........ 0.0120
7-bit uint #2 ........ 0.0059 ........ 0.0114
7-bit uint #3 ........ 0.0061 ........ 0.0119
5-bit sint #1 ........ 0.0067 ........ 0.0126
5-bit sint #2 ........ 0.0064 ........ 0.0132
5-bit sint #3 ........ 0.0066 ........ 0.0135
8-bit uint #1 ........ 0.0078 ........ 0.0200
8-bit uint #2 ........ 0.0077 ........ 0.0212
8-bit uint #3 ........ 0.0086 ........ 0.0203
16-bit uint #1 ....... 0.0111 ........ 0.0271
16-bit uint #2 ....... 0.0115 ........ 0.0260
16-bit uint #3 ....... 0.0103 ........ 0.0273
32-bit uint #1 ....... 0.0116 ........ 0.0326
32-bit uint #2 ....... 0.0118 ........ 0.0332
32-bit uint #3 ....... 0.0127 ........ 0.0325
64-bit uint #1 ....... 0.0140 ........ 0.0277
64-bit uint #2 ....... 0.0134 ........ 0.0294
64-bit uint #3 ....... 0.0134 ........ 0.0281
8-bit int #1 ......... 0.0086 ........ 0.0241
8-bit int #2 ......... 0.0089 ........ 0.0225
8-bit int #3 ......... 0.0085 ........ 0.0229
16-bit int #1 ........ 0.0118 ........ 0.0280
16-bit int #2 ........ 0.0121 ........ 0.0270
16-bit int #3 ........ 0.0109 ........ 0.0274
32-bit int #1 ........ 0.0128 ........ 0.0346
32-bit int #2 ........ 0.0118 ........ 0.0339
32-bit int #3 ........ 0.0135 ........ 0.0368
64-bit int #1 ........ 0.0138 ........ 0.0276
64-bit int #2 ........ 0.0132 ........ 0.0286
64-bit int #3 ........ 0.0137 ........ 0.0274
64-bit int #4 ........ 0.0180 ........ 0.0285
64-bit float #1 ...... 0.0134 ........ 0.0284
64-bit float #2 ...... 0.0125 ........ 0.0275
64-bit float #3 ...... 0.0126 ........ 0.0283
fix string #1 ........ 0.0035 ........ 0.0133
fix string #2 ........ 0.0094 ........ 0.0216
fix string #3 ........ 0.0094 ........ 0.0222
fix string #4 ........ 0.0091 ........ 0.0241
8-bit string #1 ...... 0.0122 ........ 0.0301
8-bit string #2 ...... 0.0118 ........ 0.0304
8-bit string #3 ...... 0.0119 ........ 0.0315
16-bit string #1 ..... 0.0150 ........ 0.0388
16-bit string #2 ..... 0.1545 ........ 0.1665
32-bit string ........ 0.1570 ........ 0.1756
wide char string #1 .. 0.0091 ........ 0.0236
wide char string #2 .. 0.0122 ........ 0.0313
8-bit binary #1 ...... 0.0100 ........ 0.0302
8-bit binary #2 ...... 0.0123 ........ 0.0324
8-bit binary #3 ...... 0.0126 ........ 0.0327
16-bit binary ........ 0.0168 ........ 0.0372
32-bit binary ........ 0.1588 ........ 0.1754
fix array #1 ......... 0.0042 ........ 0.0131
fix array #2 ......... 0.0294 ........ 0.0367
fix array #3 ......... 0.0412 ........ 0.0472
16-bit array #1 ...... 0.1378 ........ 0.1596
16-bit array #2 ........... S ............. S
32-bit array .............. S ............. S
complex array ........ 0.1865 ........ 0.2283
fix map #1 ........... 0.0725 ........ 0.1048
fix map #2 ........... 0.0319 ........ 0.0405
fix map #3 ........... 0.0356 ........ 0.0665
fix map #4 ........... 0.0465 ........ 0.0497
16-bit map #1 ........ 0.2540 ........ 0.3028
16-bit map #2 ............. S ............. S
32-bit map ................ S ............. S
complex map .......... 0.2372 ........ 0.2710
fixext 1 ............. 0.0283 ........ 0.0358
fixext 2 ............. 0.0291 ........ 0.0371
fixext 4 ............. 0.0302 ........ 0.0355
fixext 8 ............. 0.0288 ........ 0.0384
fixext 16 ............ 0.0293 ........ 0.0359
8-bit ext ............ 0.0302 ........ 0.0439
16-bit ext ........... 0.0334 ........ 0.0499
32-bit ext ........... 0.1845 ........ 0.1888
32-bit timestamp #1 .. 0.0337 ........ 0.0547
32-bit timestamp #2 .. 0.0335 ........ 0.0560
64-bit timestamp #1 .. 0.0371 ........ 0.0575
64-bit timestamp #2 .. 0.0374 ........ 0.0542
64-bit timestamp #3 .. 0.0356 ........ 0.0533
96-bit timestamp #1 .. 0.0362 ........ 0.0699
96-bit timestamp #2 .. 0.0381 ........ 0.0701
96-bit timestamp #3 .. 0.0367 ........ 0.0687
=============================================
Total                  2.7618          4.0820
Skipped                     4               4
Failed                      0               0
Ignored                     0               0

With JIT:

php -n -dzend_extension=opcache.so \
-dpcre.jit=1 -dopcache.jit_buffer_size=64M -dopcache.jit=tracing -dopcache.enable=1 -dopcache.enable_cli=1 \
tests/bench.php

Example output

Filter: MessagePack\Tests\Perf\Filter\ListFilter
Rounds: 3
Iterations: 100000

=============================================
Test/Target            Packer  BufferUnpacker
---------------------------------------------
nil .................. 0.0005 ........ 0.0054
false ................ 0.0004 ........ 0.0059
true ................. 0.0004 ........ 0.0059
7-bit uint #1 ........ 0.0010 ........ 0.0047
7-bit uint #2 ........ 0.0010 ........ 0.0046
7-bit uint #3 ........ 0.0010 ........ 0.0046
5-bit sint #1 ........ 0.0025 ........ 0.0046
5-bit sint #2 ........ 0.0023 ........ 0.0046
5-bit sint #3 ........ 0.0024 ........ 0.0045
8-bit uint #1 ........ 0.0043 ........ 0.0081
8-bit uint #2 ........ 0.0043 ........ 0.0079
8-bit uint #3 ........ 0.0041 ........ 0.0080
16-bit uint #1 ....... 0.0064 ........ 0.0095
16-bit uint #2 ....... 0.0064 ........ 0.0091
16-bit uint #3 ....... 0.0064 ........ 0.0094
32-bit uint #1 ....... 0.0085 ........ 0.0114
32-bit uint #2 ....... 0.0077 ........ 0.0122
32-bit uint #3 ....... 0.0077 ........ 0.0120
64-bit uint #1 ....... 0.0085 ........ 0.0159
64-bit uint #2 ....... 0.0086 ........ 0.0157
64-bit uint #3 ....... 0.0086 ........ 0.0158
8-bit int #1 ......... 0.0042 ........ 0.0080
8-bit int #2 ......... 0.0042 ........ 0.0080
8-bit int #3 ......... 0.0042 ........ 0.0081
16-bit int #1 ........ 0.0065 ........ 0.0095
16-bit int #2 ........ 0.0065 ........ 0.0090
16-bit int #3 ........ 0.0056 ........ 0.0085
32-bit int #1 ........ 0.0067 ........ 0.0107
32-bit int #2 ........ 0.0066 ........ 0.0106
32-bit int #3 ........ 0.0063 ........ 0.0104
64-bit int #1 ........ 0.0072 ........ 0.0162
64-bit int #2 ........ 0.0073 ........ 0.0174
64-bit int #3 ........ 0.0072 ........ 0.0164
64-bit int #4 ........ 0.0077 ........ 0.0161
64-bit float #1 ...... 0.0053 ........ 0.0135
64-bit float #2 ...... 0.0053 ........ 0.0135
64-bit float #3 ...... 0.0052 ........ 0.0135
fix string #1 ....... -0.0002 ........ 0.0044
fix string #2 ........ 0.0035 ........ 0.0067
fix string #3 ........ 0.0035 ........ 0.0077
fix string #4 ........ 0.0033 ........ 0.0078
8-bit string #1 ...... 0.0059 ........ 0.0110
8-bit string #2 ...... 0.0063 ........ 0.0121
8-bit string #3 ...... 0.0064 ........ 0.0124
16-bit string #1 ..... 0.0099 ........ 0.0146
16-bit string #2 ..... 0.1522 ........ 0.1474
32-bit string ........ 0.1511 ........ 0.1483
wide char string #1 .. 0.0039 ........ 0.0084
wide char string #2 .. 0.0073 ........ 0.0123
8-bit binary #1 ...... 0.0040 ........ 0.0112
8-bit binary #2 ...... 0.0075 ........ 0.0123
8-bit binary #3 ...... 0.0077 ........ 0.0129
16-bit binary ........ 0.0096 ........ 0.0145
32-bit binary ........ 0.1535 ........ 0.1479
fix array #1 ......... 0.0008 ........ 0.0061
fix array #2 ......... 0.0121 ........ 0.0165
fix array #3 ......... 0.0193 ........ 0.0222
16-bit array #1 ...... 0.0607 ........ 0.0479
16-bit array #2 ........... S ............. S
32-bit array .............. S ............. S
complex array ........ 0.0749 ........ 0.0824
fix map #1 ........... 0.0329 ........ 0.0431
fix map #2 ........... 0.0161 ........ 0.0189
fix map #3 ........... 0.0205 ........ 0.0262
fix map #4 ........... 0.0252 ........ 0.0205
16-bit map #1 ........ 0.1016 ........ 0.0927
16-bit map #2 ............. S ............. S
32-bit map ................ S ............. S
complex map .......... 0.1096 ........ 0.1030
fixext 1 ............. 0.0157 ........ 0.0161
fixext 2 ............. 0.0175 ........ 0.0183
fixext 4 ............. 0.0156 ........ 0.0185
fixext 8 ............. 0.0163 ........ 0.0184
fixext 16 ............ 0.0164 ........ 0.0182
8-bit ext ............ 0.0158 ........ 0.0207
16-bit ext ........... 0.0203 ........ 0.0219
32-bit ext ........... 0.1614 ........ 0.1539
32-bit timestamp #1 .. 0.0195 ........ 0.0249
32-bit timestamp #2 .. 0.0188 ........ 0.0260
64-bit timestamp #1 .. 0.0207 ........ 0.0281
64-bit timestamp #2 .. 0.0212 ........ 0.0291
64-bit timestamp #3 .. 0.0207 ........ 0.0295
96-bit timestamp #1 .. 0.0222 ........ 0.0358
96-bit timestamp #2 .. 0.0228 ........ 0.0353
96-bit timestamp #3 .. 0.0210 ........ 0.0319
=============================================
Total                  1.6432          1.9674
Skipped                     4               4
Failed                      0               0
Ignored                     0               0

You may change default benchmark settings by defining the following environment variables:

NameDefault
MP_BENCH_TARGETSpure_p,pure_u, see a list of available targets
MP_BENCH_ITERATIONS100_000
MP_BENCH_DURATIONnot set
MP_BENCH_ROUNDS3
MP_BENCH_TESTS-@slow, see a list of available tests

For example:

export MP_BENCH_TARGETS=pure_p
export MP_BENCH_ITERATIONS=1000000
export MP_BENCH_ROUNDS=5
# a comma separated list of test names
export MP_BENCH_TESTS='complex array, complex map'
# or a group name
# export MP_BENCH_TESTS='-@slow' // @pecl_comp
# or a regexp
# export MP_BENCH_TESTS='/complex (array|map)/'

Another example, benchmarking both the library and the PECL extension:

MP_BENCH_TARGETS=pure_p,pure_u,pecl_p,pecl_u \
php -n -dextension=msgpack.so -dzend_extension=opcache.so \
-dpcre.jit=1 -dopcache.enable=1 -dopcache.enable_cli=1 \
tests/bench.php

Example output

Filter: MessagePack\Tests\Perf\Filter\ListFilter
Rounds: 3
Iterations: 100000

===========================================================================
Test/Target            Packer  BufferUnpacker  msgpack_pack  msgpack_unpack
---------------------------------------------------------------------------
nil .................. 0.0031 ........ 0.0141 ...... 0.0055 ........ 0.0064
false ................ 0.0039 ........ 0.0154 ...... 0.0056 ........ 0.0053
true ................. 0.0038 ........ 0.0139 ...... 0.0056 ........ 0.0044
7-bit uint #1 ........ 0.0061 ........ 0.0110 ...... 0.0059 ........ 0.0046
7-bit uint #2 ........ 0.0065 ........ 0.0119 ...... 0.0042 ........ 0.0029
7-bit uint #3 ........ 0.0054 ........ 0.0117 ...... 0.0045 ........ 0.0025
5-bit sint #1 ........ 0.0047 ........ 0.0103 ...... 0.0038 ........ 0.0022
5-bit sint #2 ........ 0.0048 ........ 0.0117 ...... 0.0038 ........ 0.0022
5-bit sint #3 ........ 0.0046 ........ 0.0102 ...... 0.0038 ........ 0.0023
8-bit uint #1 ........ 0.0063 ........ 0.0174 ...... 0.0039 ........ 0.0031
8-bit uint #2 ........ 0.0063 ........ 0.0167 ...... 0.0040 ........ 0.0029
8-bit uint #3 ........ 0.0063 ........ 0.0168 ...... 0.0039 ........ 0.0030
16-bit uint #1 ....... 0.0092 ........ 0.0222 ...... 0.0049 ........ 0.0030
16-bit uint #2 ....... 0.0096 ........ 0.0227 ...... 0.0042 ........ 0.0046
16-bit uint #3 ....... 0.0123 ........ 0.0274 ...... 0.0059 ........ 0.0051
32-bit uint #1 ....... 0.0136 ........ 0.0331 ...... 0.0060 ........ 0.0048
32-bit uint #2 ....... 0.0130 ........ 0.0336 ...... 0.0070 ........ 0.0048
32-bit uint #3 ....... 0.0127 ........ 0.0329 ...... 0.0051 ........ 0.0048
64-bit uint #1 ....... 0.0126 ........ 0.0268 ...... 0.0055 ........ 0.0049
64-bit uint #2 ....... 0.0135 ........ 0.0281 ...... 0.0052 ........ 0.0046
64-bit uint #3 ....... 0.0131 ........ 0.0274 ...... 0.0069 ........ 0.0044
8-bit int #1 ......... 0.0077 ........ 0.0236 ...... 0.0058 ........ 0.0044
8-bit int #2 ......... 0.0087 ........ 0.0244 ...... 0.0058 ........ 0.0048
8-bit int #3 ......... 0.0084 ........ 0.0241 ...... 0.0055 ........ 0.0049
16-bit int #1 ........ 0.0112 ........ 0.0271 ...... 0.0048 ........ 0.0045
16-bit int #2 ........ 0.0124 ........ 0.0292 ...... 0.0057 ........ 0.0049
16-bit int #3 ........ 0.0118 ........ 0.0270 ...... 0.0058 ........ 0.0050
32-bit int #1 ........ 0.0137 ........ 0.0366 ...... 0.0058 ........ 0.0051
32-bit int #2 ........ 0.0133 ........ 0.0366 ...... 0.0056 ........ 0.0049
32-bit int #3 ........ 0.0129 ........ 0.0350 ...... 0.0052 ........ 0.0048
64-bit int #1 ........ 0.0145 ........ 0.0254 ...... 0.0034 ........ 0.0025
64-bit int #2 ........ 0.0097 ........ 0.0214 ...... 0.0034 ........ 0.0025
64-bit int #3 ........ 0.0096 ........ 0.0287 ...... 0.0059 ........ 0.0050
64-bit int #4 ........ 0.0143 ........ 0.0277 ...... 0.0059 ........ 0.0046
64-bit float #1 ...... 0.0134 ........ 0.0281 ...... 0.0057 ........ 0.0052
64-bit float #2 ...... 0.0141 ........ 0.0281 ...... 0.0057 ........ 0.0050
64-bit float #3 ...... 0.0144 ........ 0.0282 ...... 0.0057 ........ 0.0050
fix string #1 ........ 0.0036 ........ 0.0143 ...... 0.0066 ........ 0.0053
fix string #2 ........ 0.0107 ........ 0.0222 ...... 0.0065 ........ 0.0068
fix string #3 ........ 0.0116 ........ 0.0245 ...... 0.0063 ........ 0.0069
fix string #4 ........ 0.0105 ........ 0.0253 ...... 0.0083 ........ 0.0077
8-bit string #1 ...... 0.0126 ........ 0.0318 ...... 0.0075 ........ 0.0088
8-bit string #2 ...... 0.0121 ........ 0.0295 ...... 0.0076 ........ 0.0086
8-bit string #3 ...... 0.0125 ........ 0.0293 ...... 0.0130 ........ 0.0093
16-bit string #1 ..... 0.0159 ........ 0.0368 ...... 0.0117 ........ 0.0086
16-bit string #2 ..... 0.1547 ........ 0.1686 ...... 0.1516 ........ 0.1373
32-bit string ........ 0.1558 ........ 0.1729 ...... 0.1511 ........ 0.1396
wide char string #1 .. 0.0098 ........ 0.0237 ...... 0.0066 ........ 0.0065
wide char string #2 .. 0.0128 ........ 0.0291 ...... 0.0061 ........ 0.0082
8-bit binary #1 ........... I ............. I ........... F ............. I
8-bit binary #2 ........... I ............. I ........... F ............. I
8-bit binary #3 ........... I ............. I ........... F ............. I
16-bit binary ............. I ............. I ........... F ............. I
32-bit binary ............. I ............. I ........... F ............. I
fix array #1 ......... 0.0040 ........ 0.0129 ...... 0.0120 ........ 0.0058
fix array #2 ......... 0.0279 ........ 0.0390 ...... 0.0143 ........ 0.0165
fix array #3 ......... 0.0415 ........ 0.0463 ...... 0.0162 ........ 0.0187
16-bit array #1 ...... 0.1349 ........ 0.1628 ...... 0.0334 ........ 0.0341
16-bit array #2 ........... S ............. S ........... S ............. S
32-bit array .............. S ............. S ........... S ............. S
complex array ............. I ............. I ........... F ............. F
fix map #1 ................ I ............. I ........... F ............. I
fix map #2 ........... 0.0345 ........ 0.0391 ...... 0.0143 ........ 0.0168
fix map #3 ................ I ............. I ........... F ............. I
fix map #4 ........... 0.0459 ........ 0.0473 ...... 0.0151 ........ 0.0163
16-bit map #1 ........ 0.2518 ........ 0.2962 ...... 0.0400 ........ 0.0490
16-bit map #2 ............. S ............. S ........... S ............. S
32-bit map ................ S ............. S ........... S ............. S
complex map .......... 0.2380 ........ 0.2682 ...... 0.0545 ........ 0.0579
fixext 1 .................. I ............. I ........... F ............. F
fixext 2 .................. I ............. I ........... F ............. F
fixext 4 .................. I ............. I ........... F ............. F
fixext 8 .................. I ............. I ........... F ............. F
fixext 16 ................. I ............. I ........... F ............. F
8-bit ext ................. I ............. I ........... F ............. F
16-bit ext ................ I ............. I ........... F ............. F
32-bit ext ................ I ............. I ........... F ............. F
32-bit timestamp #1 ....... I ............. I ........... F ............. F
32-bit timestamp #2 ....... I ............. I ........... F ............. F
64-bit timestamp #1 ....... I ............. I ........... F ............. F
64-bit timestamp #2 ....... I ............. I ........... F ............. F
64-bit timestamp #3 ....... I ............. I ........... F ............. F
96-bit timestamp #1 ....... I ............. I ........... F ............. F
96-bit timestamp #2 ....... I ............. I ........... F ............. F
96-bit timestamp #3 ....... I ............. I ........... F ............. F
===========================================================================
Total                  1.5625          2.3866        0.7735          0.7243
Skipped                     4               4             4               4
Failed                      0               0            24              17
Ignored                    24              24             0               7

With JIT:

MP_BENCH_TARGETS=pure_p,pure_u,pecl_p,pecl_u \
php -n -dextension=msgpack.so -dzend_extension=opcache.so \
-dpcre.jit=1 -dopcache.jit_buffer_size=64M -dopcache.jit=tracing -dopcache.enable=1 -dopcache.enable_cli=1 \
tests/bench.php

Example output

Filter: MessagePack\Tests\Perf\Filter\ListFilter
Rounds: 3
Iterations: 100000

===========================================================================
Test/Target            Packer  BufferUnpacker  msgpack_pack  msgpack_unpack
---------------------------------------------------------------------------
nil .................. 0.0001 ........ 0.0052 ...... 0.0053 ........ 0.0042
false ................ 0.0007 ........ 0.0060 ...... 0.0057 ........ 0.0043
true ................. 0.0008 ........ 0.0060 ...... 0.0056 ........ 0.0041
7-bit uint #1 ........ 0.0031 ........ 0.0046 ...... 0.0062 ........ 0.0041
7-bit uint #2 ........ 0.0021 ........ 0.0043 ...... 0.0062 ........ 0.0041
7-bit uint #3 ........ 0.0022 ........ 0.0044 ...... 0.0061 ........ 0.0040
5-bit sint #1 ........ 0.0030 ........ 0.0048 ...... 0.0062 ........ 0.0040
5-bit sint #2 ........ 0.0032 ........ 0.0046 ...... 0.0062 ........ 0.0040
5-bit sint #3 ........ 0.0031 ........ 0.0046 ...... 0.0062 ........ 0.0040
8-bit uint #1 ........ 0.0054 ........ 0.0079 ...... 0.0062 ........ 0.0050
8-bit uint #2 ........ 0.0051 ........ 0.0079 ...... 0.0064 ........ 0.0044
8-bit uint #3 ........ 0.0051 ........ 0.0082 ...... 0.0062 ........ 0.0044
16-bit uint #1 ....... 0.0077 ........ 0.0094 ...... 0.0065 ........ 0.0045
16-bit uint #2 ....... 0.0077 ........ 0.0094 ...... 0.0063 ........ 0.0045
16-bit uint #3 ....... 0.0077 ........ 0.0095 ...... 0.0064 ........ 0.0047
32-bit uint #1 ....... 0.0088 ........ 0.0119 ...... 0.0063 ........ 0.0043
32-bit uint #2 ....... 0.0089 ........ 0.0117 ...... 0.0062 ........ 0.0039
32-bit uint #3 ....... 0.0089 ........ 0.0118 ...... 0.0063 ........ 0.0044
64-bit uint #1 ....... 0.0097 ........ 0.0155 ...... 0.0063 ........ 0.0045
64-bit uint #2 ....... 0.0095 ........ 0.0153 ...... 0.0061 ........ 0.0045
64-bit uint #3 ....... 0.0096 ........ 0.0156 ...... 0.0063 ........ 0.0047
8-bit int #1 ......... 0.0053 ........ 0.0083 ...... 0.0062 ........ 0.0044
8-bit int #2 ......... 0.0052 ........ 0.0080 ...... 0.0062 ........ 0.0044
8-bit int #3 ......... 0.0052 ........ 0.0080 ...... 0.0062 ........ 0.0043
16-bit int #1 ........ 0.0089 ........ 0.0097 ...... 0.0069 ........ 0.0046
16-bit int #2 ........ 0.0075 ........ 0.0093 ...... 0.0063 ........ 0.0043
16-bit int #3 ........ 0.0075 ........ 0.0094 ...... 0.0062 ........ 0.0046
32-bit int #1 ........ 0.0086 ........ 0.0122 ...... 0.0063 ........ 0.0044
32-bit int #2 ........ 0.0087 ........ 0.0120 ...... 0.0066 ........ 0.0046
32-bit int #3 ........ 0.0086 ........ 0.0121 ...... 0.0060 ........ 0.0044
64-bit int #1 ........ 0.0096 ........ 0.0149 ...... 0.0060 ........ 0.0045
64-bit int #2 ........ 0.0096 ........ 0.0157 ...... 0.0062 ........ 0.0044
64-bit int #3 ........ 0.0096 ........ 0.0160 ...... 0.0063 ........ 0.0046
64-bit int #4 ........ 0.0097 ........ 0.0157 ...... 0.0061 ........ 0.0044
64-bit float #1 ...... 0.0079 ........ 0.0153 ...... 0.0056 ........ 0.0044
64-bit float #2 ...... 0.0079 ........ 0.0152 ...... 0.0057 ........ 0.0045
64-bit float #3 ...... 0.0079 ........ 0.0155 ...... 0.0057 ........ 0.0044
fix string #1 ........ 0.0010 ........ 0.0045 ...... 0.0071 ........ 0.0044
fix string #2 ........ 0.0048 ........ 0.0075 ...... 0.0070 ........ 0.0060
fix string #3 ........ 0.0048 ........ 0.0086 ...... 0.0068 ........ 0.0060
fix string #4 ........ 0.0050 ........ 0.0088 ...... 0.0070 ........ 0.0059
8-bit string #1 ...... 0.0081 ........ 0.0129 ...... 0.0069 ........ 0.0062
8-bit string #2 ...... 0.0086 ........ 0.0128 ...... 0.0069 ........ 0.0065
8-bit string #3 ...... 0.0086 ........ 0.0126 ...... 0.0115 ........ 0.0065
16-bit string #1 ..... 0.0105 ........ 0.0137 ...... 0.0128 ........ 0.0068
16-bit string #2 ..... 0.1510 ........ 0.1486 ...... 0.1526 ........ 0.1391
32-bit string ........ 0.1517 ........ 0.1475 ...... 0.1504 ........ 0.1370
wide char string #1 .. 0.0044 ........ 0.0085 ...... 0.0067 ........ 0.0057
wide char string #2 .. 0.0081 ........ 0.0125 ...... 0.0069 ........ 0.0063
8-bit binary #1 ........... I ............. I ........... F ............. I
8-bit binary #2 ........... I ............. I ........... F ............. I
8-bit binary #3 ........... I ............. I ........... F ............. I
16-bit binary ............. I ............. I ........... F ............. I
32-bit binary ............. I ............. I ........... F ............. I
fix array #1 ......... 0.0014 ........ 0.0059 ...... 0.0132 ........ 0.0055
fix array #2 ......... 0.0146 ........ 0.0156 ...... 0.0155 ........ 0.0148
fix array #3 ......... 0.0211 ........ 0.0229 ...... 0.0179 ........ 0.0180
16-bit array #1 ...... 0.0673 ........ 0.0498 ...... 0.0343 ........ 0.0388
16-bit array #2 ........... S ............. S ........... S ............. S
32-bit array .............. S ............. S ........... S ............. S
complex array ............. I ............. I ........... F ............. F
fix map #1 ................ I ............. I ........... F ............. I
fix map #2 ........... 0.0148 ........ 0.0180 ...... 0.0156 ........ 0.0179
fix map #3 ................ I ............. I ........... F ............. I
fix map #4 ........... 0.0252 ........ 0.0201 ...... 0.0214 ........ 0.0167
16-bit map #1 ........ 0.1027 ........ 0.0836 ...... 0.0388 ........ 0.0510
16-bit map #2 ............. S ............. S ........... S ............. S
32-bit map ................ S ............. S ........... S ............. S
complex map .......... 0.1104 ........ 0.1010 ...... 0.0556 ........ 0.0602
fixext 1 .................. I ............. I ........... F ............. F
fixext 2 .................. I ............. I ........... F ............. F
fixext 4 .................. I ............. I ........... F ............. F
fixext 8 .................. I ............. I ........... F ............. F
fixext 16 ................. I ............. I ........... F ............. F
8-bit ext ................. I ............. I ........... F ............. F
16-bit ext ................ I ............. I ........... F ............. F
32-bit ext ................ I ............. I ........... F ............. F
32-bit timestamp #1 ....... I ............. I ........... F ............. F
32-bit timestamp #2 ....... I ............. I ........... F ............. F
64-bit timestamp #1 ....... I ............. I ........... F ............. F
64-bit timestamp #2 ....... I ............. I ........... F ............. F
64-bit timestamp #3 ....... I ............. I ........... F ............. F
96-bit timestamp #1 ....... I ............. I ........... F ............. F
96-bit timestamp #2 ....... I ............. I ........... F ............. F
96-bit timestamp #3 ....... I ............. I ........... F ............. F
===========================================================================
Total                  0.9642          1.0909        0.8224          0.7213
Skipped                     4               4             4               4
Failed                      0               0            24              17
Ignored                    24              24             0               7

Note that the msgpack extension (v2.1.2) doesn't support ext, bin and UTF-8 str types.

License

The library is released under the MIT License. See the bundled LICENSE file for details.

Author: rybakit
Source Code: https://github.com/rybakit/msgpack.php
License: MIT License

#php 

Autumn  Blick

Autumn Blick

1594770710

How To Succeed In Mobile App Wireframe Design?

In the world of overrated terms “web development”, a mobile app wireframe design is one of the most underrated terms. The design of wireframes is considered when people look for the bigger picture.

While designing the UI-UX, people forget the simple norm of general to specific shifting. As the complexity increases and so does the approach become more difficult, this is where the designing of the wireframes comes in handy.

Before diving into the “How to”, let’s first see why we need them in the first place.

What are mobile app wireframes?

Wireframes are the skeletal layouts of an application or a website that is being designed. The specificity comes into play, the elements and the features have to be placed at specific locations. Take a building, in the process of making it, first the foundation is laid and then pieces are fitted together from the skeleton structure on a piece of paper, wireframes do the same for the website or application structure such as a smart home application.

The designing of wireframes is commonly known as wireframing. For the construction of a building, the framework or the skeletal structure is important while designing a web application or mobile application, wireframing is important to make it user-friendly. This entirely and solely works to make the journey smooth and destination easy to reach.

As for the building, the layers of cementing and painting is done later to increase the visual appeal, the visual contents and appealing stuff are added after wireframing. The simpler it sounds after the definition, the complex it gets when it is being done.

It is a very goal-oriented procedure, one has to keep in mind is the goal of the product or the destination of the service. The main focus should be on UX. The arrangement of the elements and their interaction with each other and with the user is the utmost important task in mobile app wireframing.

What not to do while designing the mobile app wireframe?

  • Do not even think of skipping the process.
  • Do not beautify (visually appealing designs added first) and then get into the wireframing business).
  • Do not do it just for the sake of doing it.

One has to keep in mind that skipping this entirely can lead to the failure of the entire process of web and mobile app development at the end.

Again taking the example of the construction of a building, the foundation must be laid first based on the skeletal framework that has been prepared, then only you can jump to beautify your building, as a designer one has to understand and follow the steps where designing the mobile app wireframe comes first and then the visually appealing content is added next not the other way round.

For the most part, people do not understand the importance and come up with some trashy design of wireframes and the main foundation becomes faulty, hence the entire designing at later stages becomes faulty. If one wants to skip the reworking part, mobile app wireframing must never be ignored.

#android app #ios app #minimum viable product (mvp) #mobile app development #app designing #mobile app wireframe designing #mobile app wireframing #mobile application wireframing #mobile wireframing #web app wireframing #wireframe designing

Jones Brianna

Jones Brianna

1614154249

List Of The Top Pittsburgh Mobile App Development Companies

https://clutch.co/app-developers/pittsburgh
Let’s look at the list of top list of the top Pittsburgh mobile app development companies which are known for providing top-notch services globally. They are great developers who provide quality services for all your needs.

#mobile app developers #mobile app development services #mobile app development #mobile app developers #mobile apps #mobile app development solutions