Morioh
Login
Vegard Sandnes

Vegard Sandnes

5 years ago

Image Object Detection using TensorFlowjs

In this post, we will build an image object detection system with Tensorflow-js with the pre-trained model.

To start with, there are lots of ways to deploy TensorFlow in webpage one way is to include ml5js. Visit https://ml5js.org/. Its a wrapper around tf.js a tensor flow and p5.js library used for doing operations in Html element.

But, We will like to keep the power on the backend part so that I can try and run these models over backend with API's backend processes and so on.

Therefore, In the first half of the post, we will create a UI using React.js and Material-UI and in the second half will we create an API in Node.js to power the UI.

Let's start with building a sample React project.

FRONTEND PART

If you have followed along with my previous article the react project will seem to be fairly easy to build.

  1. Open the terminal and do
create-react-app image_classification_react_ui

This will create a react project to work with.

2. Let’s install the dependency required

npm install @material-ui/core npm install - save isomorphic-fetch es6-promise
Note: isomorphic-fetch is required to call the object detection API endpoint from React code.

3. Open the project in your favorite editor and let’s create 2 folders

  1. container — This will contain a file — ImageOps.jsx , which have all frontend UI code.
  2. utils — This will contain a file Api.js , which is used to call the object detection endpoint.
└── src 
    ├── containers 
        ├── ImageOps.jsx 
    ├── utils 
        ├── Api.js

Let’s look into the ImageOps.jsxcode and understand it.

import React from 'react';
 
import Container from '@material-ui/core/Container';
import Grid from '@material-ui/core/Grid';
 
import Card from '@material-ui/core/Card';
import CardContent from '@material-ui/core/CardContent';
import Typography from '@material-ui/core/Typography';
import Button from '@material-ui/core/Button';
import { red } from '@material-ui/core/colors';
 
import {api} from '../utils/Api';
 
import Table from '@material-ui/core/Table';
import TableBody from '@material-ui/core/TableBody';
import TableCell from '@material-ui/core/TableCell';
import TableHead from '@material-ui/core/TableHead';
import TableRow from '@material-ui/core/TableRow';
import Paper from '@material-ui/core/Paper';
import CircularProgress from '@material-ui/core/CircularProgress';
 
 
export default class ImageOps extends React.Component {
  
  constructor(props) {
    super(props);
 
    this.state = {
      image_object: null,
      image_object_details: {},
      active_type: null
    }
  }
 
  updateImageObject(e) {
    const file = e.target.files[0];
    const reader = new FileReader();
    
    reader.readAsDataURL(file);
    reader.onload = () => {
      this.setState({image_object: reader.result, image_object_details: {}, active_type: null});
    };
 
  }
 
  processImageObject(type) {
 
    this.setState({active_type: type}, () => {
 
      if(!this.state.image_object_details[this.state.active_type]) {
        api("detect_image_objects", {
          type,
          data: this.state.image_object
        }).then((response) => {
          
          const filtered_data = response;
          const image_details = this.state.image_object_details;
    
          image_details[filtered_data.type] = filtered_data.data;
    
          this.setState({image_object_details: image_details });
        });
      }
    });
  }
 
  render() {
    return (
      <Container maxWidth="md">
        <Grid container spacing={2}>
          <Grid item xs={12}>
            <CardContent>
              <Typography variant="h4" color="textPrimary" component="h4">
                Object Detection Tensorflow
              </Typography>
            </CardContent>
          </Grid>
          <Grid item xs={12}>
            {this.state.image_object &&
              <img src={this.state.image_object} alt="" height="500px"/>
            }
          </Grid>
          <Grid item xs={12}>
            <Card>
              <CardContent>
                <Button variant="contained"
                  component='label' // <-- Just add me!
                  >
                  Upload Image
                  <input accept="image/jpeg" onChange={(e) => this.updateImageObject(e)} type="file" style={{ display: 'none' }} />
                </Button>
              </CardContent>
            </Card>
          </Grid>
          <Grid item xs={3}>
            <Grid container justify="center" spacing={3}>
              <Grid item >
                {this.state.image_object && <Button onClick={() => this.processImageObject("imagenet")}variant="contained" color="primary">
                  Get objects with ImageNet
                </Button>}
              </Grid>
              <Grid item>
                {this.state.image_object && <Button onClick={() => this.processImageObject("coco-ssd")}variant="contained" color="secondary">
                  Get objects with Coco SSD
                </Button>}
              </Grid>
            </Grid>
          </Grid>
          <Grid item xs={9}>
            <Grid container justify="center">
              {this.state.active_type && this.state.image_object_details[this.state.active_type] &&
                <Grid item xs={12}>
                  <Card>
                    <CardContent>
                      <Typography variant="h4" color="textPrimary" component="h4">
                        {this.state.active_type.toUpperCase()}
                      </Typography>
                      <ImageDetails type={this.state.active_type} data = {this.state.image_object_details[this.state.active_type]}></ImageDetails>
                    </CardContent>
                  </Card>
                </Grid>
              }
              {this.state.active_type && !this.state.image_object_details[this.state.active_type] &&
                <Grid item xs={12}>
                  <CircularProgress
                    color="secondary"
                  />
                </Grid>
              }
            </Grid>
          </Grid>
        </Grid>
      </Container>
    )
  }
}
 
class ImageDetails extends React.Component {
  
  render() {
 
    console.log(this.props.data);
 
    return (
      <Grid item xs={12}>
        <Paper>
          <Table>
          <TableHead>
            <TableRow>
            <TableCell>Objects</TableCell>
            <TableCell align="right">Probability</TableCell>
            </TableRow>
          </TableHead>
          <TableBody>
            {this.props.data.map((row) => {
              if (this.props.type === "imagenet") {
                return (
                  <TableRow key={row.className}>
                    <TableCell component="th" scope="row">
                    {row.className}
                    </TableCell>
                    <TableCell align="right">{row.probability.toFixed(2)}</TableCell>
                  </TableRow>
                )
              } else if(this.props.type === "coco-ssd") {
                return (
                  <TableRow key={row.className}>
                    <TableCell component="th" scope="row">
                    {row.class}
                    </TableCell>
                    <TableCell align="right">{row.score.toFixed(2)}</TableCell>
                  </TableRow>
                )
              }
              })
            }
          </TableBody>
          </Table>
        </Paper>
       
      </Grid>
    )
  }
}
 
}
Note: Here is the Github repo link of above — https://github.com/overflowjs-com/image_object_detction_react_ui . If you find understanding above diffcult then i highly recommend to read our Part 2 and Part 1.

In render, we have created a Grid of three rows with first row containing heading

Second, containing the image to display

<Grid item xs={12}>
 {this.state.image_object &&
  <img src={this.state.image_object} alt="" height="500px"/>}         
</Grid>

Here we are displaying an image if the image has been uploaded or image object is available in the state

Next grid contains a button to upload a file and update uploaded file to the current state.

<Grid item xs={12}>
  <Card>
    <CardContent>
      <Button variant="contained"
        component='label' // <-- Just add me!
        >
        Upload Image
        <input accept="image/jpeg" onChange={(e) => this.updateImageObject(e)} type="file" style={{ display: 'none' }} />
      </Button>
    </CardContent>
  </Card>
</Grid>

On Button to upload an image on change event we have called a function updateImageto update the currently selected image on the state.

updateImageObject(e) {
    const file = e.target.files[0];
    const reader = new FileReader();
    
    reader.readAsDataURL(file);
    reader.onload = () => {
      this.setState({image_object: reader.result, image_object_details: {}, active_type: null
      });
    };
}

In the above code, we are reading the current file object from file input uploader and loading its data on the current state. As the new image is getting uploaded we are resetting image_object_details and active_type so that fresh operations can be applied on uploaded image

Below is the next grid that contains code for two buttons for each model.

<Grid item xs={3}>
    <Grid container justify="center" spacing={3}>
      <Grid item >
        {this.state.image_object && <Button onClick={() => this.processImageObject("imagenet")}variant="contained" color="primary">
          Get objects with ImageNet
        </Button>}
      </Grid>
      <Grid item> 
        {this.state.image_object && <Button onClick={() => this.processImageObject("coco-ssd")}variant="contained" color="secondary">
          Get objects with Coco SSD
        </Button>}
      </Grid>
    </Grid>
  </Grid>
  <Grid item xs={9}>
    <Grid container justify="center">
      {this.state.active_type && this.state.image_object_details[this.state.active_type] &&
        <Grid item xs={12}>
          <Card>
            <CardContent>
              <Typography variant="h4" color="textPrimary" component="h4">
                {this.state.active_type.toUpperCase()}
              </Typography>
              <ImageDetails data = {this.state.image_object_details[this.state.active_type]}></ImageDetails>
            </CardContent>
          </Card>
        </Grid>
      }
      {this.state.active_type && !this.state.image_object_details[this.state.active_type] && 
        <Grid item xs={12}>
          <CircularProgress
            color="secondary"
          />
        </Grid>
      }
   </Grid>
</Grid>

Here we are dividing Grid into two parts 3 columns and 9 columns from 12 columns parent.

First Grid with 3 columns contains two Grid having two buttons

<Grid container justify="center" spacing={3}>
  <Grid item >
    {this.state.image_object && <Button onClick={() => this.processImageObject("imagenet")}variant="contained" color="primary">
      Get objects with ImageNet
    </Button>}
  </Grid>
  <Grid item> 
    {this.state.image_object && <Button onClick={() => this.processImageObject("coco-ssd")}variant="contained" color="secondary">
      Get objects with Coco SSD
    </Button>}
  </Grid>
</Grid>
We are analyzing image detection using ImageNet and Coco SSD Models and compare the outputs.

Each button has an action event onClick and it is calling a function processImageObject()which takes the name of the model as a parameter.

processImageObject(type) {
this.setState({active_type: type}, () => {
    api("detect_image_objects", {
      type,
      data: this.state.image_object
    }).then((response) => {
       
      const filtered_data = response;
      const image_details = this.state.image_object_details;
image_details[filtered_data.type] = filtered_data.data;
this.setState({image_object_details: image_details });
    });
  });
}

We are setting the state object action_typewith currently selected modal.

The Process image object function will take the current image from state and send it to API function which I will show you next and API will be called detect_image_objectsand in response, we will process and show in UI.

The response from API will be fetched and it will be set in stage image_object_details.

We are setting each API response based on the type of model (imagenet/coco-ssd)

This Buttons will only be visible when image_object is present in the state.

{
 this.state.image_object && 
 <Button onClick={() => this.processImageObject()} variant="contained" color="primary">Process Image 
 </Button>
}

Below is another grid we have created:

<Grid item xs={9}>
  <Grid container justify="center">
    {this.state.active_type && this.state.image_object_details[this.state.active_type] &&
      <Grid item xs={12}>
        <Card>
          <CardContent>
            <Typography variant="h4" color="textPrimary" component="h4">
              {this.state.active_type.toUpperCase()}
            </Typography>
            <ImageDetails type={this.state.active_type} data = {this.state.image_object_details[this.state.active_type]}></ImageDetails>
          </CardContent>
        </Card>
      </Grid>
    }
    {this.state.active_type && !this.state.image_object_details[this.state.active_type] && 
      <Grid item xs={12}>
        <CircularProgress
          color="secondary"
        />
      </Grid>
    }
  </Grid>
</Grid>

Here we have checked whether current action_typemodal is selected or not then if API has processed details it shows object details. For this, we have created a component ImageDetails.

Let’s look into ImageDetails component code which is easy to understand.

class ImageDetails extends React.Component {
  
  render() {
 
    console.log(this.props.data);
 
    return (
      <Grid item xs={12}>
        <Paper>
          <Table>
          <TableHead>
            <TableRow>
            <TableCell>Objects</TableCell>
            <TableCell align="right">Probability</TableCell>
            </TableRow>
          </TableHead>
          <TableBody>
            {this.props.data.map((row) => {
              if (this.props.type === "imagenet") {
                return (
                  <TableRow key={row.className}>
                    <TableCell component="th" scope="row">
                    {row.className}
                    </TableCell>
                    <TableCell align="right">{row.probability.toFixed(2)}</TableCell>
                  </TableRow>
                )
              } else if(this.props.type === "coco-ssd") {
                return (
                  <TableRow key={row.className}>
                    <TableCell component="th" scope="row">
                    {row.class}
                    </TableCell>
                    <TableCell align="right">{row.score.toFixed(2)}</TableCell>
                  </TableRow>
                )
              }
              })
            }
          </TableBody>
          </Table>
        </Paper>
       
      </Grid>
    )
  }
}

This component will show details received from modal Name of Object and their probability. Based on the type of modal we are working with we can display two different outputs which are handled in this class.

4. The last step is to write the API.js wrapper to do a server-side call.

import fetch from 'isomorphic-fetch';

const BASE_API_URL = “http://localhost:4000/api/”

 

export function api(api_end_point, data) {

 

  return fetch(BASE_API_URL+api_end_point,

    {

      method: ‘POST’,

      headers: {

        ‘Content-Type’: ‘application/json’

      },

      body:JSON.stringify(data)

    }).then((response) => {

      return response.json();

    });

}

In this sample code, we are providing a wrapper over fetch API function will take API endpoint and data and it will construct complete URL and return response sent from API.

Final UI will look like this


Object detection using Tensorflow.js

Github code link for UI- https://github.com/overflowjs-com/image_object_detction_react_ui

Github code link for Backend- https://github.com/overflowjs-com/image_object_detection_tensor_api

Happy coding!

#tensorflow #node-js #reactjs #javascript #web-development

Image Object Detection using TensorFlowjs
4 Likes53.25 GEEK