Conor  Grady

Conor Grady

1671075360

How to Handle Conversions From Go To DynamoDB Types

In this Golang article we will learn about How to Handle Conversions From Go To DynamoDB Types. Explore practical code samples in this demonstration to learn how to handle conversions between Go types in your application and DynamoDB.

DynamoDB provides a rich set of data types including Strings, Numbers, Sets, Lists, Maps etc. In the Go SDK for DynamoDB, the types package contains Go representations of these data types and the attributevalue module provides functions to work with Go and DynamoDB types.

This blog post will demonstrate how to handle conversions between Go types in your application and DynamoDB. We will start off with simple code snippets to introduce some of the API constructs and wrap up with an example of how to use these Go SDK features in the context of a complete application (including a code walk though).

You can refer to the complete code on GitHub.

To begin with, go through a few examples.

Please note: Error handling has been purposely omitted in the below code snippets to keep them concise.

Converting From Go to DynamoDB types

The Marshal family of functions takes care of this. It works with basic scalars (int, uint, float, bool, string), maps, slices, and structs.

To work with scalar types, just use the (generic) Marshal function:

func marshalScalars() {
    av, err := attributevalue.Marshal("foo")
    log.Println(av.(*types.AttributeValueMemberS).Value)

    av, err = attributevalue.Marshal(true)
    log.Println(av.(*types.AttributeValueMemberBOOL).Value)

    av, err = attributevalue.Marshal(42)
    log.Println(av.(*types.AttributeValueMemberN).Value)

    av, err = attributevalue.Marshal(42.42)
    log.Println(av.(*types.AttributeValueMemberN).Value)
}

 

Marshal converts a Go data type into an AttributeValue. But AttributeValue itself is just an interface and requires you to cast it to a concrete type such as AttributeValueMemberS (for string), AttributeValueMemberBOOL (for boolean) etc.

If you try to cast incompatible types, the SDK responds with a helpful error message. For example, panic: interface conversion: types.AttributeValue is *types.AttributeValueMemberN, not *types.AttributeValueMemberS

When working with slices and maps, you are better off using specific functions such as MarshalList and MarshalMap:

func marshalSlicesAndMaps() {
    avl, err := attributevalue.MarshalList([]string{"foo", "bar"})

    for _, v := range avl {
        log.Println(v.(*types.AttributeValueMemberS).Value)
    }

    avm, err := attributevalue.MarshalMap(map[string]interface{}{"foo": "bar", "boo": "42"})

    for k, v := range avm {
        log.Println(k, "=", v.(*types.AttributeValueMemberS).Value)
    }
}

 

The above examples gave you a sense of how to work with simple data types in isolation. In a real-world application, you will make use of composite data types to represent your domain model - most likely they will be in the form of Go structs. So let's look at a few examples of that.

Working With Go Structs

Here is a simple one:

type User struct {
    Name string
    Age  string
}

func marshalStruct() {
    user := User{Name: "foo", Age: "42"}

    av, err := attributevalue.Marshal(user)

    avm := av.(*types.AttributeValueMemberM).Value
    log.Println("name", avm["Name"].(*types.AttributeValueMemberS).Value)
    log.Println("age", avm["Age"].(*types.AttributeValueMemberS).Value)

    avMap, err := attributevalue.MarshalMap(user)

    for name, value := range avMap {
        log.Println(name, "=", value.(*types.AttributeValueMemberS).Value)
    }
}

So far, it seems like we can handle simple use cases, but we can do better. This example had a homogenous data type; i.e., the struct had only string type making it easy to iterate over the result map and cast the value to a *types.AttributeValueMemberS. If that were not the case, you would have to iterate over each and every attribute value type and typecast it to the appropriate Go type. This will be evident when working with the rest of the DynamoDB APIs. For example, the result of a GetItem invocation (GetItemOutput) contains a map[string]types.AttributeValue.

The SDK provides a way for us to make this much easier!

Converting From DynamoDB To Go Types

The Unmarshal family of functions takes care of this. Here is another example:

type AdvancedUser struct {
    Name         string
    Age          int
    IsOnline     bool
    Favourites   []string
    Contact      map[string]string
    RegisteredOn time.Time
}

func marshalUnmarshal() {
    user := AdvancedUser{
        Name:         "abhishek",
        Age:          35,
        IsOnline:     false,
        Favourites:   []string{"Lost In Translation, The Walking Dead"},
        Contact:      map[string]string{"mobile": "+919718861200", "email": "abhirockzz@gmail.com"},
        RegisteredOn: time.Now(),
    }

    avMap, err := attributevalue.MarshalMap(user)

    var result AdvancedUser
    err = attributevalue.UnmarshalMap(avMap, &result)

    log.Println("\nname", result.Name, "\nage", result.Age, "\nfavs", result.Favourites)
}

 

With MarshalMap, we converted an instance of AdvancedUser struct into a map[string]types.AttributeValue (imagine you get this as a response to a GetItem API call). Now, instead of iterating over individual AttributeValues, we simply use UnmarshalMap to convert it back a Go struct.

There is more! Utility functions like UnmarshalListOfMaps make it convenient to work with multiple slices of Go structs.

func marshalUnmarshalMultiple() {
    user1 := User{Name: "user1", Age: "42"}

    user1Map, err := attributevalue.MarshalMap(user1)
    if err != nil {
        log.Fatal(err)
    }

    user2 := User{Name: "user2", Age: "24"}

    user2Map, err := attributevalue.MarshalMap(user2)
    if err != nil {
        log.Fatal(err)
    }

    var users []User

    err = attributevalue.UnmarshalListOfMaps([]map[string]types.AttributeValue{user1Map, user2Map}, &users)
    if err != nil {
        log.Fatal(err)
    }

    for _, user := range users {
        log.Println("name", user.Name, "age", user.Age)
    }
}

 

Using struct Tags for Customization

Marshal and Unmarshal functions support the dynamodbav struct tag to control conversion between Go types and DynamoDB AttributeValue. Consider the following struct:

type User struct {
    Email string `dynamodbav:"email" json:"user_email"`
    Age   int    `dynamodbav:"age,omitempty" json:"age,omitempty"`
    City  string `dynamodbav:"city" json:"city"`
}

 

Here are a couple of common scenarios where the dynamodbav comes in handy:

Customize Attribute Name

Say we have a table with email as the partition key. Without the dynamodbav:"email" tag, when we marshal the User struct and try to save in the table, it will use Email (upper case) as the attribute name. DynamoDB will not accept this since attribute names are case sensitive: "All names must be encoded using UTF-8, and are case-sensitive."

Handle Missing Attributes

DynamoDB is a NoSQL database and tables don't have a fixed schema (except for partition key and an optional sort key). For example, a user item may not include the age attribute. By using dynamodbav:"age,omitempty", if the Age field is missing, it won't be sent to DynamoDB (it will be ignored).

To look at all the usage patterns of this struct tag, refer to the Marshal API documentation.

As promised before, let's explore how to put all these APIs to use within the scope of an...

End-To-End Example

We will look at a Go application that exposes a REST API with a few endpoints. It combines the CRUD APIs (PutItem, GetItem, etc.) together with all the functions/APIs mentioned above.

Test the Application

Before we see the code, let's quickly review and test the endpoints exposed by the application. You will need to have Go installed, clone the application, and change to the right directory.

git clone https://github.com/abhirockzz/dynamodb-go-sdk-type-conversion
cd dynamodb-go-sdk-type-conversion

 

First, create a DynamoDB table (you can name it users). Use city as the Partition key, email as the Sort key.

Create a DynamoDB table

You need some test data. You can do so manually, but I have included a simple utility to seed some test data during application startup. To use it, simply set the SEED_TEST_DATA variable at application startup:

export SEED_TEST_DATA=true

go run main.go

# output
started http server...

 

This will create 100 items. Check DynamoDB table to confirm:

100 items created

 

Your application should be available at port 8080. You can use curl or any other HTTP client to invoke the endpoints:

# to get all users
curl -i http://localhost:8080/users/

# to get all users in a particular city
curl -i http://localhost:8080/users/London

# to get a specific user
curl -i "http://localhost:8080/user?city=London&email=user11@foo.com"

 

To better understand how the above APIs are used, let's briefly review key parts of the code:

Code Walkthrough

Add New Item to a DynamoDB Table

Starting with the HTTP handler for adding a User:

func (h Handler) CreateUser(rw http.ResponseWriter, req *http.Request) {
    var user model.User

    err := json.NewDecoder(req.Body).Decode(&user)
    if err != nil {// handle error}

    err = h.d.Save(user)
    if err != nil {// handle error}

    err = json.NewEncoder(rw).Encode(user.Email)
    if err != nil {// handle error}
}

 

First, we convert the JSON payload into a User struct which we then pass to the Save function.

func (d DB) Save(user model.User) error {

    item, err := attributevalue.MarshalMap(user)

    if err != nil {// handle error}

    _, err = d.client.PutItem(context.Background(), &dynamodb.PutItemInput{
        TableName: aws.String(d.table),
        Item:      item})

    if err != nil {// handle error}

    return nil
}

 

Notice how MarshalMap is used to convert the User struct to a map[string]types.AttributeValue that the PutItem API can accept:

Get a Single Item From DynamoDB

Since our table has a composite primary key (city is the partition key and email is the sort key), we will need to provide both of them to locate a specific user item:

func (h Handler) FetchUser(rw http.ResponseWriter, req *http.Request) {

    email := req.URL.Query().Get("email")
    city := req.URL.Query().Get("city")

    log.Println("getting user with email", email, "in city", city)

    user, err := h.d.GetOne(email, city)
    if err != nil {// handle error}


    err = json.NewEncoder(rw).Encode(user)
    if err != nil {// handle error}
}

 

We extract the email and city from the query parameters in the HTTP request and pass it on to the database layer (GetOne function).

func (d DB) GetOne(email, city string) (model.User, error) {

    result, err := d.client.GetItem(context.Background(),
        &dynamodb.GetItemInput{
            TableName: aws.String(d.table),
            Key: map[string]types.AttributeValue{
                "email": &types.AttributeValueMemberS{Value: email},
                "city":  &types.AttributeValueMemberS{Value: city}},
        })

    if err != nil {// handle error}

    if result.Item == nil {
        return model.User{}, ErrNotFound
    }

    var user model.User

    err = attributevalue.UnmarshalMap(result.Item, &user)
    if err != nil {// handle error}

    return user, nil
}

 

We invoke GetItem API and get back the result in form of a map[string]types.AttributeValue (via the Item attribute in GetItemOutput). This is converted back into the Go (User) struct using UnmarshalMap.

Notice: The Key attribute in GetItemInput also accepts a map[string]types.AttributeValue, but we don't use MarshalMap to create it.

Fetch Multiple Items

We can choose to query for all users in a specific city - this is a perfectly valid access pattern since city is the partition key.

The HTTP handler function accepts the city as a path parameter, which is passed on to the database layer.

func (h Handler) FetchUsers(rw http.ResponseWriter, req *http.Request) {
    city := mux.Vars(req)["city"]
    log.Println("city", city)

    log.Println("getting users in city", city)

    users, err := h.d.GetMany(city)

    if err != nil {
        http.Error(rw, err.Error(), http.StatusInternalServerError)
        return
    }

    err = json.NewEncoder(rw).Encode(users)
    if err != nil {
        http.Error(rw, err.Error(), http.StatusInternalServerError)
        return
    }
}

 

From there on, GetMany function does all the work:

func (d DB) GetMany(city string) ([]model.User, error) {

    kcb := expression.Key("city").Equal(expression.Value(city))
    kce, _ := expression.NewBuilder().WithKeyCondition(kcb).Build()

    result, err := d.client.Query(context.Background(), &dynamodb.QueryInput{
        TableName:                 aws.String(d.table),
        KeyConditionExpression:    kce.KeyCondition(),
        ExpressionAttributeNames:  kce.Names(),
        ExpressionAttributeValues: kce.Values(),
    })

    if err != nil {
        log.Println("Query failed with error", err)
        return []model.User{}, err
    }

    users := []model.User{}

    if len(result.Items) == 0 {
        return users, nil
    }

    err = attributevalue.UnmarshalListOfMaps(result.Items, &users)
    if err != nil {
        log.Println("UnmarshalMap failed with error", err)
        return []model.User{}, err
    }

    return users, nil
}

 

Pay attention to two things:

  • How a KeyConditionExpression is being used (this is from the expressions package).
  • More interestingly, the usage of UnmarshalListOfMaps function to directly convert a []map[string]types.AttributeValue (slice of items from DynamoDB) into a slice of User struct: If not for this function, we would need to extract each item from the result; i.e., a map[string]types.AttributeValue and call UnmarshalMap for each of them. So this is pretty handy!

Finally: Just Get Everything!

The GetAll function uses Scan operation to retrieve all the records in the DynamoDB table.

func (d DB) GetAll() ([]model.User, error) {

    result, err := d.client.Scan(context.Background(), &dynamodb.ScanInput{
        TableName: aws.String(d.table),
    })

    if err != nil {
        log.Println("Scan failed with error", err)
        return []model.User{}, err
    }

    users := []model.User{}

    err = attributevalue.UnmarshalListOfMaps(result.Items, &users)

    if err != nil {
        log.Println("UnmarshalMap failed with error", err)
        return []model.User{}, err
    }

    return users, nil
}

Wrap Up

I hope you found this useful and now you are aware of the APIs in DynamoDB Go SDK to work with simple Go types as well as structs, maps, slices, etc. I would encourage you to explore some of the other nuances such as how to customize the Marshal and Unmarshal features using MarshalWithOptions and UnmarshalWithOptions respectively.

Original article sourced at: https://medium.com

#go #golang 

What is GEEK

Buddha Community

How to Handle Conversions From Go To DynamoDB Types
Fannie  Zemlak

Fannie Zemlak

1599854400

What's new in the go 1.15

Go announced Go 1.15 version on 11 Aug 2020. Highlighted updates and features include Substantial improvements to the Go linker, Improved allocation for small objects at high core counts, X.509 CommonName deprecation, GOPROXY supports skipping proxies that return errors, New embedded tzdata package, Several Core Library improvements and more.

As Go promise for maintaining backward compatibility. After upgrading to the latest Go 1.15 version, almost all existing Golang applications or programs continue to compile and run as older Golang version.

#go #golang #go 1.15 #go features #go improvement #go package #go new features

A GUI for Local DynamoDB- Dynamodb-Admin

Quick Start Guide

1. Install the package globally from npm.

$ npm install -g dynamodb-admin

2. Run DynamoDB locally inside a Docker container

Make sure you have Docker installed on your system. Instructions are here.

Now pull and run the Docker dynamodb-local image to spin up your very own DynamoDB instance running on port 8000.

$ docker pull amazon/dynamodb-local
$ docker run -p 8000:8000 amazon/dynamodb-local

3. Start dynamodb-admin (with defaults)

MacOS/Linux

$ dynamodb-admin

Windows

> export DYNAMO_ENDPOINT=http://localhost:8000
> dynamodb-admin

After these steps you will have:

The next step is to create a table and start reading/writing to it!

Advanced Setup

You may need to override regions, endpoints and/or credentials to peek inside local DynamoDB instances you have spun up to replicate a production environment.

If so, just override the defaults when starting the service. You can override some, or all of these, as required.

DYNAMO_ENDPOINT=http://localhost:<PORT> AWS_REGION=<AWS-REGION> AWS_ACCESS_KEY_ID=<YOUR-ACCESS-KEY> AWS_SECRET_ACCESS_KEY=<YOUR-SECRET> dynamodb-admin

Setting up your Tables

The easy way — for simple use cases

Here we are going to create your table using the dynamodb-admin GUI. This is most likely going to be appropriate for your use-case.

Clicking on ‘Create table’ takes us to a screen where we can define how our table should look. Remember that DynamoDB is effectively a key-value store, meaning to get started we only need to define a table name and a hash attribute (the primary key). If you expect that you’ll need to perform lookups based on another attribute of your data, you may want to add some Secondary Indices.

In this example, I’ve named the table ‘Cats’ and given it a primary index ‘name’ and a secondary index ‘owner’. Both indices are of type String and we must give the secondary index a name — which can be different from the name of the attribute.

‘name’ is not a good choice of primary index in practice, as it means only one cat with a given name can be present in the table. Instead, it would be better to give each cat a unique id and use that as the primary index.

Finally, add your first item to the table by clicking the ‘Create item’ link on the top right. This will take you to a new screen where you can enter the json which defines the record. The only requirement for each entry is that the primary key is included.

The Hard(er) Way — for more complex table structures

Using the GUI to set up tables is fine for simple tables, or when you’re just exploring how your data storage might be structured. However, if you are trying to mimic a complex table, or you want to stand-up tables quickly for testing you may want to use the command line to create the table(s) for you.

1. Install the AWS CLI

Instructions for installing (for Mac) via the command line are here. This is a very powerful utility tool. I’d recommend installing it if you work with AWS even if you don’t opt to use it here.

2. Create a table schema

If you already have a table schema you can skip this and move along to Step 3.

$ aws dynamodb create-table --generate-cli-skeleton > dynamo_table_def.json

This will create a basic table schema and pipe it into a file named dynamo_table_def.json.

You can now open the json file and edit it to fit your desired table schema. This is great because you can now use this same schema file when you need the table, rather than manually setting it up each time via the GUI.

If you’re just curious what the schema should look like, or you need some inspiration for your own — here is the schema for the Cats table.

{
  "AttributeDefinitions": [
    {
      "AttributeName": "name",
      "AttributeType": "S"
    },
    {
      "AttributeName": "owner",
      "AttributeType": "S"
    }
  ],
  "TableName": "Cats",
  "KeySchema": [
    {
      "AttributeName": "name",
      "KeyType": "HASH"
    }
  ],
  "ProvisionedThroughput": {
    "ReadCapacityUnits": 3,
    "WriteCapacityUnits": 3
  },
  "GlobalSecondaryIndexes": [
    {
      "IndexName": "idx_owner",
      "KeySchema": [
        {
          "AttributeName": "owner",
          "KeyType": "HASH"
        }
      ],
      "Projection": {
        "ProjectionType": "ALL"
      },
      "ProvisionedThroughput": {
        "ReadCapacityUnits": 3,
        "WriteCapacityUnits": 3
      }
    }
  ]
}

3. Use the schema to create the table

Finally, create the table locally

$ aws dynamodb create-table --cli-input-json file://dynamo_table_def.json --endpoint-url http://localhost:8000

Remember the --endpoint-url parameter, otherwise a real table will be created in whatever region your AWS CLI defaults to.

After running this command, go back to dynamodb-admin in your browser. You’ll see your table has been created. Now it’s time to use it!

Use Cases

I’ve picked 3 examples to show how dynamodb-admin can help you develop and test your applications.

Running locally alongside an application

This example is super simple. Let’s say you’re developing a Python application which reads from a DynamoDB table of movies. You may want run a local DynamoDB instance for development and tests, to avoid standing up unnecessary infrastructure. Dynamodb-local is a godsend for this. However it can be fiddly to put data in the table, from the command line.

You could write code to put the correct items in the table. Indeed for tests this might be ideal, as you absolutely should test the logic you’re using to read and write from the table.

However to quickly test some code path or to build out a feature, when the remote infrastructure or data is not present, it’s typically much easier to put the data into the DynamoDB table manually.

You can set up your table and add some movies, using the GUI, as described above. Then read from the table like so:

from pprint import pprint
import boto3
from botocore.exceptions import ClientError


def get_movie(title, year, dynamodb=None):
    if not dynamodb:
        dynamodb = boto3.resource('dynamodb', endpoint_url="http://localhost:8000")

    table = dynamodb.Table('Movies')

    try:
        response = table.get_item(Key={'year': year, 'title': title})
    except ClientError as e:
        print(e.response['Error']['Message'])
    else:
        return response['Item']


if __name__ == '__main__':
    movie = get_movie("The Big New Movie", 2015,)
    if movie:
        print("Get movie succeeded:")
        pprint(movie, sort_dicts=False)

Creating the table and putting items in it, using dynamodb-admin, lets you focus on the business logic. When you’re happy with how your logic looks, you can focus on writing to the table.

Using dynamodb-admin as a library

Since dynamodb-admin is a Node library, we can use it inside our Node projects. This is again great for local development, as each time you run the service you have what is effectively the AWS console ready to view and manipulate the data.

const AWS = require('aws-sdk');
const {createServer} = require('dynamodb-admin');
 
const dynamodb = new AWS.DynamoDB();
const dynClient = new AWS.DynamoDB.DocumentClient({service: dynamodb});
 
const app = createServer(dynamodb, dynClient);
 
const port = 8001;
const server = app.listen(port);
server.on('listening', () => {
  const address = server.address();
  console.log(`  listening on http://0.0.0.0:${address.port}`);
});

Using dynamodb-admin with AWS Amplify

AWS Amplify is a development framework that deals with a lot of the common problems when building a mobile or web application, setting up the infrastructure required for you. Each part of the framework deserves a blog post of its own, but here we are going to be looking at mocking the DynamoDB tables Amplify creates based on your GraphQL API definition.

If you’d like to find out how to use Amplify to create a GraphQL API the documentation is here

Amplify let’s you mock services used by your app with the Amplify CLI tool by running

$ amplify mock <service>

If you have used Amplify to create a GraphQL API to serve as the backend for your project you can run

$ amplify mock api

This will do two things:

  1. Starts a mock Appsync API endpoint on port 20002
  2. Creates a DynamoDB instance on port 62224

We can now use dynamodb-admin to take a peek inside the tables Amplify has created, based on our API’s requirements by running

$ AWS_REGION=us-fake-1 AWS_ACCESS_KEY_ID=fake AWS_SECRET_ACCESS_KEY=fake DYNAMO_ENDPOINT=http://localhost:62224  dynamodb-admin

I’ve personally found this really useful to test locally, before committing to pushing my API changes.

Originally published by https://medium.com/swlh 

#dynamodb #aws #code #dynamodb-admin #dynamodb-local

Conor  Grady

Conor Grady

1671075360

How to Handle Conversions From Go To DynamoDB Types

In this Golang article we will learn about How to Handle Conversions From Go To DynamoDB Types. Explore practical code samples in this demonstration to learn how to handle conversions between Go types in your application and DynamoDB.

DynamoDB provides a rich set of data types including Strings, Numbers, Sets, Lists, Maps etc. In the Go SDK for DynamoDB, the types package contains Go representations of these data types and the attributevalue module provides functions to work with Go and DynamoDB types.

This blog post will demonstrate how to handle conversions between Go types in your application and DynamoDB. We will start off with simple code snippets to introduce some of the API constructs and wrap up with an example of how to use these Go SDK features in the context of a complete application (including a code walk though).

You can refer to the complete code on GitHub.

To begin with, go through a few examples.

Please note: Error handling has been purposely omitted in the below code snippets to keep them concise.

Converting From Go to DynamoDB types

The Marshal family of functions takes care of this. It works with basic scalars (int, uint, float, bool, string), maps, slices, and structs.

To work with scalar types, just use the (generic) Marshal function:

func marshalScalars() {
    av, err := attributevalue.Marshal("foo")
    log.Println(av.(*types.AttributeValueMemberS).Value)

    av, err = attributevalue.Marshal(true)
    log.Println(av.(*types.AttributeValueMemberBOOL).Value)

    av, err = attributevalue.Marshal(42)
    log.Println(av.(*types.AttributeValueMemberN).Value)

    av, err = attributevalue.Marshal(42.42)
    log.Println(av.(*types.AttributeValueMemberN).Value)
}

 

Marshal converts a Go data type into an AttributeValue. But AttributeValue itself is just an interface and requires you to cast it to a concrete type such as AttributeValueMemberS (for string), AttributeValueMemberBOOL (for boolean) etc.

If you try to cast incompatible types, the SDK responds with a helpful error message. For example, panic: interface conversion: types.AttributeValue is *types.AttributeValueMemberN, not *types.AttributeValueMemberS

When working with slices and maps, you are better off using specific functions such as MarshalList and MarshalMap:

func marshalSlicesAndMaps() {
    avl, err := attributevalue.MarshalList([]string{"foo", "bar"})

    for _, v := range avl {
        log.Println(v.(*types.AttributeValueMemberS).Value)
    }

    avm, err := attributevalue.MarshalMap(map[string]interface{}{"foo": "bar", "boo": "42"})

    for k, v := range avm {
        log.Println(k, "=", v.(*types.AttributeValueMemberS).Value)
    }
}

 

The above examples gave you a sense of how to work with simple data types in isolation. In a real-world application, you will make use of composite data types to represent your domain model - most likely they will be in the form of Go structs. So let's look at a few examples of that.

Working With Go Structs

Here is a simple one:

type User struct {
    Name string
    Age  string
}

func marshalStruct() {
    user := User{Name: "foo", Age: "42"}

    av, err := attributevalue.Marshal(user)

    avm := av.(*types.AttributeValueMemberM).Value
    log.Println("name", avm["Name"].(*types.AttributeValueMemberS).Value)
    log.Println("age", avm["Age"].(*types.AttributeValueMemberS).Value)

    avMap, err := attributevalue.MarshalMap(user)

    for name, value := range avMap {
        log.Println(name, "=", value.(*types.AttributeValueMemberS).Value)
    }
}

So far, it seems like we can handle simple use cases, but we can do better. This example had a homogenous data type; i.e., the struct had only string type making it easy to iterate over the result map and cast the value to a *types.AttributeValueMemberS. If that were not the case, you would have to iterate over each and every attribute value type and typecast it to the appropriate Go type. This will be evident when working with the rest of the DynamoDB APIs. For example, the result of a GetItem invocation (GetItemOutput) contains a map[string]types.AttributeValue.

The SDK provides a way for us to make this much easier!

Converting From DynamoDB To Go Types

The Unmarshal family of functions takes care of this. Here is another example:

type AdvancedUser struct {
    Name         string
    Age          int
    IsOnline     bool
    Favourites   []string
    Contact      map[string]string
    RegisteredOn time.Time
}

func marshalUnmarshal() {
    user := AdvancedUser{
        Name:         "abhishek",
        Age:          35,
        IsOnline:     false,
        Favourites:   []string{"Lost In Translation, The Walking Dead"},
        Contact:      map[string]string{"mobile": "+919718861200", "email": "abhirockzz@gmail.com"},
        RegisteredOn: time.Now(),
    }

    avMap, err := attributevalue.MarshalMap(user)

    var result AdvancedUser
    err = attributevalue.UnmarshalMap(avMap, &result)

    log.Println("\nname", result.Name, "\nage", result.Age, "\nfavs", result.Favourites)
}

 

With MarshalMap, we converted an instance of AdvancedUser struct into a map[string]types.AttributeValue (imagine you get this as a response to a GetItem API call). Now, instead of iterating over individual AttributeValues, we simply use UnmarshalMap to convert it back a Go struct.

There is more! Utility functions like UnmarshalListOfMaps make it convenient to work with multiple slices of Go structs.

func marshalUnmarshalMultiple() {
    user1 := User{Name: "user1", Age: "42"}

    user1Map, err := attributevalue.MarshalMap(user1)
    if err != nil {
        log.Fatal(err)
    }

    user2 := User{Name: "user2", Age: "24"}

    user2Map, err := attributevalue.MarshalMap(user2)
    if err != nil {
        log.Fatal(err)
    }

    var users []User

    err = attributevalue.UnmarshalListOfMaps([]map[string]types.AttributeValue{user1Map, user2Map}, &users)
    if err != nil {
        log.Fatal(err)
    }

    for _, user := range users {
        log.Println("name", user.Name, "age", user.Age)
    }
}

 

Using struct Tags for Customization

Marshal and Unmarshal functions support the dynamodbav struct tag to control conversion between Go types and DynamoDB AttributeValue. Consider the following struct:

type User struct {
    Email string `dynamodbav:"email" json:"user_email"`
    Age   int    `dynamodbav:"age,omitempty" json:"age,omitempty"`
    City  string `dynamodbav:"city" json:"city"`
}

 

Here are a couple of common scenarios where the dynamodbav comes in handy:

Customize Attribute Name

Say we have a table with email as the partition key. Without the dynamodbav:"email" tag, when we marshal the User struct and try to save in the table, it will use Email (upper case) as the attribute name. DynamoDB will not accept this since attribute names are case sensitive: "All names must be encoded using UTF-8, and are case-sensitive."

Handle Missing Attributes

DynamoDB is a NoSQL database and tables don't have a fixed schema (except for partition key and an optional sort key). For example, a user item may not include the age attribute. By using dynamodbav:"age,omitempty", if the Age field is missing, it won't be sent to DynamoDB (it will be ignored).

To look at all the usage patterns of this struct tag, refer to the Marshal API documentation.

As promised before, let's explore how to put all these APIs to use within the scope of an...

End-To-End Example

We will look at a Go application that exposes a REST API with a few endpoints. It combines the CRUD APIs (PutItem, GetItem, etc.) together with all the functions/APIs mentioned above.

Test the Application

Before we see the code, let's quickly review and test the endpoints exposed by the application. You will need to have Go installed, clone the application, and change to the right directory.

git clone https://github.com/abhirockzz/dynamodb-go-sdk-type-conversion
cd dynamodb-go-sdk-type-conversion

 

First, create a DynamoDB table (you can name it users). Use city as the Partition key, email as the Sort key.

Create a DynamoDB table

You need some test data. You can do so manually, but I have included a simple utility to seed some test data during application startup. To use it, simply set the SEED_TEST_DATA variable at application startup:

export SEED_TEST_DATA=true

go run main.go

# output
started http server...

 

This will create 100 items. Check DynamoDB table to confirm:

100 items created

 

Your application should be available at port 8080. You can use curl or any other HTTP client to invoke the endpoints:

# to get all users
curl -i http://localhost:8080/users/

# to get all users in a particular city
curl -i http://localhost:8080/users/London

# to get a specific user
curl -i "http://localhost:8080/user?city=London&email=user11@foo.com"

 

To better understand how the above APIs are used, let's briefly review key parts of the code:

Code Walkthrough

Add New Item to a DynamoDB Table

Starting with the HTTP handler for adding a User:

func (h Handler) CreateUser(rw http.ResponseWriter, req *http.Request) {
    var user model.User

    err := json.NewDecoder(req.Body).Decode(&user)
    if err != nil {// handle error}

    err = h.d.Save(user)
    if err != nil {// handle error}

    err = json.NewEncoder(rw).Encode(user.Email)
    if err != nil {// handle error}
}

 

First, we convert the JSON payload into a User struct which we then pass to the Save function.

func (d DB) Save(user model.User) error {

    item, err := attributevalue.MarshalMap(user)

    if err != nil {// handle error}

    _, err = d.client.PutItem(context.Background(), &dynamodb.PutItemInput{
        TableName: aws.String(d.table),
        Item:      item})

    if err != nil {// handle error}

    return nil
}

 

Notice how MarshalMap is used to convert the User struct to a map[string]types.AttributeValue that the PutItem API can accept:

Get a Single Item From DynamoDB

Since our table has a composite primary key (city is the partition key and email is the sort key), we will need to provide both of them to locate a specific user item:

func (h Handler) FetchUser(rw http.ResponseWriter, req *http.Request) {

    email := req.URL.Query().Get("email")
    city := req.URL.Query().Get("city")

    log.Println("getting user with email", email, "in city", city)

    user, err := h.d.GetOne(email, city)
    if err != nil {// handle error}


    err = json.NewEncoder(rw).Encode(user)
    if err != nil {// handle error}
}

 

We extract the email and city from the query parameters in the HTTP request and pass it on to the database layer (GetOne function).

func (d DB) GetOne(email, city string) (model.User, error) {

    result, err := d.client.GetItem(context.Background(),
        &dynamodb.GetItemInput{
            TableName: aws.String(d.table),
            Key: map[string]types.AttributeValue{
                "email": &types.AttributeValueMemberS{Value: email},
                "city":  &types.AttributeValueMemberS{Value: city}},
        })

    if err != nil {// handle error}

    if result.Item == nil {
        return model.User{}, ErrNotFound
    }

    var user model.User

    err = attributevalue.UnmarshalMap(result.Item, &user)
    if err != nil {// handle error}

    return user, nil
}

 

We invoke GetItem API and get back the result in form of a map[string]types.AttributeValue (via the Item attribute in GetItemOutput). This is converted back into the Go (User) struct using UnmarshalMap.

Notice: The Key attribute in GetItemInput also accepts a map[string]types.AttributeValue, but we don't use MarshalMap to create it.

Fetch Multiple Items

We can choose to query for all users in a specific city - this is a perfectly valid access pattern since city is the partition key.

The HTTP handler function accepts the city as a path parameter, which is passed on to the database layer.

func (h Handler) FetchUsers(rw http.ResponseWriter, req *http.Request) {
    city := mux.Vars(req)["city"]
    log.Println("city", city)

    log.Println("getting users in city", city)

    users, err := h.d.GetMany(city)

    if err != nil {
        http.Error(rw, err.Error(), http.StatusInternalServerError)
        return
    }

    err = json.NewEncoder(rw).Encode(users)
    if err != nil {
        http.Error(rw, err.Error(), http.StatusInternalServerError)
        return
    }
}

 

From there on, GetMany function does all the work:

func (d DB) GetMany(city string) ([]model.User, error) {

    kcb := expression.Key("city").Equal(expression.Value(city))
    kce, _ := expression.NewBuilder().WithKeyCondition(kcb).Build()

    result, err := d.client.Query(context.Background(), &dynamodb.QueryInput{
        TableName:                 aws.String(d.table),
        KeyConditionExpression:    kce.KeyCondition(),
        ExpressionAttributeNames:  kce.Names(),
        ExpressionAttributeValues: kce.Values(),
    })

    if err != nil {
        log.Println("Query failed with error", err)
        return []model.User{}, err
    }

    users := []model.User{}

    if len(result.Items) == 0 {
        return users, nil
    }

    err = attributevalue.UnmarshalListOfMaps(result.Items, &users)
    if err != nil {
        log.Println("UnmarshalMap failed with error", err)
        return []model.User{}, err
    }

    return users, nil
}

 

Pay attention to two things:

  • How a KeyConditionExpression is being used (this is from the expressions package).
  • More interestingly, the usage of UnmarshalListOfMaps function to directly convert a []map[string]types.AttributeValue (slice of items from DynamoDB) into a slice of User struct: If not for this function, we would need to extract each item from the result; i.e., a map[string]types.AttributeValue and call UnmarshalMap for each of them. So this is pretty handy!

Finally: Just Get Everything!

The GetAll function uses Scan operation to retrieve all the records in the DynamoDB table.

func (d DB) GetAll() ([]model.User, error) {

    result, err := d.client.Scan(context.Background(), &dynamodb.ScanInput{
        TableName: aws.String(d.table),
    })

    if err != nil {
        log.Println("Scan failed with error", err)
        return []model.User{}, err
    }

    users := []model.User{}

    err = attributevalue.UnmarshalListOfMaps(result.Items, &users)

    if err != nil {
        log.Println("UnmarshalMap failed with error", err)
        return []model.User{}, err
    }

    return users, nil
}

Wrap Up

I hope you found this useful and now you are aware of the APIs in DynamoDB Go SDK to work with simple Go types as well as structs, maps, slices, etc. I would encourage you to explore some of the other nuances such as how to customize the Marshal and Unmarshal features using MarshalWithOptions and UnmarshalWithOptions respectively.

Original article sourced at: https://medium.com

#go #golang 

Arvel  Parker

Arvel Parker

1593156510

Basic Data Types in Python | Python Web Development For Beginners

At the end of 2019, Python is one of the fastest-growing programming languages. More than 10% of developers have opted for Python development.

In the programming world, Data types play an important role. Each Variable is stored in different data types and responsible for various functions. Python had two different objects, and They are mutable and immutable objects.

Table of Contents  hide

I Mutable objects

II Immutable objects

III Built-in data types in Python

Mutable objects

The Size and declared value and its sequence of the object can able to be modified called mutable objects.

Mutable Data Types are list, dict, set, byte array

Immutable objects

The Size and declared value and its sequence of the object can able to be modified.

Immutable data types are int, float, complex, String, tuples, bytes, and frozen sets.

id() and type() is used to know the Identity and data type of the object

a**=25+**85j

type**(a)**

output**:<class’complex’>**

b**={1:10,2:“Pinky”****}**

id**(b)**

output**:**238989244168

Built-in data types in Python

a**=str(“Hello python world”)****#str**

b**=int(18)****#int**

c**=float(20482.5)****#float**

d**=complex(5+85j)****#complex**

e**=list((“python”,“fast”,“growing”,“in”,2018))****#list**

f**=tuple((“python”,“easy”,“learning”))****#tuple**

g**=range(10)****#range**

h**=dict(name=“Vidu”,age=36)****#dict**

i**=set((“python”,“fast”,“growing”,“in”,2018))****#set**

j**=frozenset((“python”,“fast”,“growing”,“in”,2018))****#frozenset**

k**=bool(18)****#bool**

l**=bytes(8)****#bytes**

m**=bytearray(8)****#bytearray**

n**=memoryview(bytes(18))****#memoryview**

Numbers (int,Float,Complex)

Numbers are stored in numeric Types. when a number is assigned to a variable, Python creates Number objects.

#signed interger

age**=**18

print**(age)**

Output**:**18

Python supports 3 types of numeric data.

int (signed integers like 20, 2, 225, etc.)

float (float is used to store floating-point numbers like 9.8, 3.1444, 89.52, etc.)

complex (complex numbers like 8.94j, 4.0 + 7.3j, etc.)

A complex number contains an ordered pair, i.e., a + ib where a and b denote the real and imaginary parts respectively).

String

The string can be represented as the sequence of characters in the quotation marks. In python, to define strings we can use single, double, or triple quotes.

# String Handling

‘Hello Python’

#single (') Quoted String

“Hello Python”

# Double (") Quoted String

“”“Hello Python”“”

‘’‘Hello Python’‘’

# triple (‘’') (“”") Quoted String

In python, string handling is a straightforward task, and python provides various built-in functions and operators for representing strings.

The operator “+” is used to concatenate strings and “*” is used to repeat the string.

“Hello”+“python”

output**:****‘Hello python’**

"python "*****2

'Output : Python python ’

#python web development #data types in python #list of all python data types #python data types #python datatypes #python types #python variable type

Unconvert: Remove Unnecessary Type Conversions From Go Source

About

The unconvert program analyzes Go packages to identify unnecessary type conversions; i.e., expressions T(x) where x already has type T.

Install

$ go get github.com/mdempsky/unconvert

Usage

$ unconvert -v bytes fmt
GOROOT/src/bytes/reader.go:117:14: unnecessary conversion
                abs = int64(r.i) + offset
                           ^
GOROOT/src/fmt/print.go:411:21: unnecessary conversion
        p.fmt.integer(int64(v), 16, unsigned, udigits)
                           ^

Flags

Using the -v flag, unconvert will also print the source line and a caret to indicate the unnecessary conversion's position therein.

Using the -apply flag, unconvert will rewrite the Go source files without the unnecessary type conversions.

Using the -all flag, unconvert will analyze the Go packages under all possible GOOS/GOARCH combinations, and only identify conversions that are unnecessary in all cases.

E.g., syscall.Timespec's Sec and Nsec fields are int64 under linux/amd64 but int32 under linux/386. An int64(ts.Sec) conversion that appears in a linux/amd64-only file will be identified as unnecessary, but it will be preserved if it occurs in a file that's compiled for both linux/amd64 and linux/386.

Author: Mdempsky
Source Code: https://github.com/mdempsky/unconvert 
License: BSD-3-Clause license

#go #golang #type