Connecting Redis Sentinel with Spring

In this article, I am going to make a basic example of how to connect to a “Redis Sentinel” configuration in a Spring application.

You might also enjoy:  Getting Started with Python, Redis, and Nginx

First of all, for Maven dependencies, “spring-boot-starter-data-redis” is needed in our pom.xml file. This library includes LettuceConnectionFactory, which we will use to connect “Redis Sentinel”.

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-redis</artifactId>
            <version>2.0.5.RELEASE</version>
        </dependency>

We will use application.yml file for connection and cache configurations. Below there is an example of application.yml file. Under cache.config.defaultConfig there is a list of predefined cache configurations. You can map these configurations to service cache definitions which are defined under cache.appCaches.appCacheMap.

cache:
    config:
        defaultConfigs:
        -  cacheName: ONE_HOUR_CACHE         
           timeToLiveSeconds: 3600 
        -  cacheName: TEN_MINUTES_CACHE
           timeToLiveSeconds: 600

  appCaches:
    appCacheMap:
        serviceOne_cache: ONE_HOUR_CACHE
        serviceTwo_cache: TEN_MINUTES_CACHE   

spring:
  cache:
    type: redis
  redis:
    port: 6666
    password: 123pwd
    sentinel:
      master: masterredis
      nodes:
        - 10.0.0.16
        - 10.0.0.17
        - 10.0.0.18
    lettuce:
      shutdown-timeout: 200ms             

In order to read these configuration parameters easily, we will need a CacheSettingsModel and two classes annotated with ConfigurationProperties. One for the defaultConfigs and another for the appCacheMap. For redis configs, which are under spring.redis, we will use RedisProperties class, which is included in spring-boot-starter-data-redis. You can see these classes below:

package com.example.sentinel.config.model;

public class CacheSettingsModel {
private String cacheName;
private String timeToLiveSeconds;

public final String getCacheName() {
return cacheName;
}

public final void setCacheName(String cacheName) {
this.cacheName = cacheName;
}

public final String getTimeToLiveSeconds() {
return timeToLiveSeconds;
}

public final void setTimeToLiveSeconds(String timeToLiveSeconds) {
this.timeToLiveSeconds = timeToLiveSeconds;
}
}

package com.example.sentinel.config;

import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;

import org.apache.commons.collections4.CollectionUtils;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.stereotype.Component;

import com.example.sentinel.config.model.CacheSettingsModel;

@Component
@ConfigurationProperties( prefix="cache.config")
public class CacheSettings {

private List<CacheSettingsModel> defaultConfigs;

public List<CacheSettingsModel> getDefaultConfigs() {
return defaultConfigs;
}

public void setDefaultConfigs(List<CacheSettingsModel> defaultConfigs) {
this.defaultConfigs = defaultConfigs;
}

public Map<String,CacheSettingsModel> getCacheConfigAsMap() {
return CollectionUtils.emptyIfNull(defaultConfigs).stream().
          collect(Collectors.toMap(CacheSettingsModel::getCacheName, c -> c));
}
}

package com.example.sentinel.config;

import java.util.Map;

import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.stereotype.Component;

@Component
@ConfigurationProperties( prefix="cache.appCaches")
public class AppCacheSettings {

private Map<String,String> appCacheMap;

public Map<String, String> getAppCacheMap() {
return appCacheMap;
}

public void setAppCacheMap(Map<String, String> appCacheMap) {
this.appCacheMap = appCacheMap;
}
}

Now we are going to create a connection to “Redis Sentinel” with the help of those configuration classes. As I told you before, we are going to use Lettuce to connect to Redis. We are going to create a new instance of RedisSentinelConfiguration class with the help of RedisProperties class and use it in the constructor of LettuceConnectionFactory.

package com.example.sentinel.config;

import java.time.Duration;
import java.util.HashMap;
import java.util.Map;

import javax.inject.Inject;

import org.springframework.boot.autoconfigure.data.redis.RedisProperties;
import org.springframework.cache.annotation.EnableCaching;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.redis.cache.RedisCacheConfiguration;
import org.springframework.data.redis.cache.RedisCacheManager;
import org.springframework.data.redis.connection.RedisPassword;
import org.springframework.data.redis.connection.RedisSentinelConfiguration;
import org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory;

import com.example.sentinel.config.AppCacheSettings;
import com.example.sentinel.config.CacheSettings;
import com.example.sentinel.config.model.CacheSettingsModel;

@Configuration
@EnableCaching
public class CacheConfig {

    @Inject
    private CacheSettings cacheSettings;
    @Inject
    private AppCacheSettings appCacheSettings;
    @Inject
    private RedisProperties redisProperties;

    @Bean
    public LettuceConnectionFactory redisConnectionFactory() {

        RedisSentinelConfiguration sentinelConfig = new RedisSentinelConfiguration()
                .master(redisProperties.getSentinel().getMaster());

        redisProperties.getSentinel().getNodes().forEach(s -> sentinelConfig.sentinel(s, Integer.valueOf(redisProperties.getPort())));
        sentinelConfig.setPassword(RedisPassword.of(redisProperties.getPassword()));

        return new LettuceConnectionFactory(sentinelConfig);
    }

    @Bean
    public RedisCacheConfiguration cacheConfiguration() {
        return RedisCacheConfiguration.defaultCacheConfig()
                .entryTtl(Duration.ofSeconds(600))
                .disableCachingNullValues();
    }

    private RedisCacheConfiguration buildRedisCacheConfig(CacheSettingsModel cachesProperties) {
        return RedisCacheConfiguration.defaultCacheConfig()
                .entryTtl(Duration.ofSeconds(Long.parseLong(cachesProperties.getTimeToLiveSeconds())))
                .disableCachingNullValues();
    }

    @Bean
    public RedisCacheManager cacheManager() {

        Map<String, RedisCacheConfiguration> cacheConfigs = new HashMap<>();

        Map<String, CacheSettingsModel> cacheConfigMap = cacheSettings.getCacheConfigAsMap();
        Map<String, String> appCacheMap = appCacheSettings.getAppCacheMap();

        appCacheMap.forEach((key, value) -> cacheConfigs.put(key, buildRedisCacheConfig(cacheConfigMap.get(value))));

        return RedisCacheManager.builder(redisConnectionFactory())
                .cacheDefaults(cacheConfiguration())
                .withInitialCacheConfigurations(cacheConfigs)
                .transactionAware()
                .build();
    }
}

At the end, we are going to create a CacheManager bean. We are going to use RedisCacheManager.builder. With the help of AppCacheSettings and CacheSettings classes we are going to set initial cache configurations of cacheManager bean. If you want to enable transactional operations on caches you should set transactionAware to true (default is false). I’ll just copy the definition of transactionAware from the Spring API document.

Set this to “true” to synchronize cache put/evict operations with ongoing Spring-managed transactions, performing the actual cache put/evict operation only in the after-commit phase of a successful transaction.

We are almost done. To use Redis cache in our application, we only need to put @Cacheable (org.springframework.cache.annotation.Cacheable) annotation to the services that we want to cache. As a Value parameter, you can use one of the predefined cache names under appCacheMap properties (in application.yml file) to set cache properties like timeToLive.

 @Cacheable("serviceOne_cache")
 public List<String> serviceOne(String param) {
 ...
 }

That’s it. Spring will do the rest!

Thank for visiting and reading this post.! We highly appreciate your actions! Please share if you liked it!

Further Reading

Database Caching With Redis and Java

How to use Python with Redis

#spring-boot #redis #database #web-development

What is GEEK

Buddha Community

Redis Transactions & Long-Running Lua Scripts

Redis offers two mechanisms for handling transactions – MULTI/EXEC based transactions and Lua scripts evaluation. Redis Lua scripting is the recommended approach and is fairly popular in usage.

Our Redis™ customers who have Lua scripts deployed often report this error – “BUSY Redis is busy running a script. You can only call SCRIPT KILL or SHUTDOWN NOSAVE”. In this post, we will explain the Redis transactional property of scripts, what this error is about, and why we must be extra careful about it on Sentinel-managed systems that can failover.

Transactional Nature of Redis Lua Scripts

Redis “transactions” aren’t really transactions as understood conventionally – in case of errors, there is no rollback of writes made by the script.

“Atomicity” of Redis scripts is guaranteed in the following manner:

• Once a script begins executing, all other commands/scripts are blocked until the script completes. So, other clients either see the changes made by the script or they don’t. This is because they can only execute either before the script or after the script.
• However, Redis doesn’t do rollbacks, so on an error within a script, any changes already made by the script will be retained and future commands/scripts will see those partial changes.
• Since all other clients are blocked while the script executes, it is critical that the script is well-behaved and finishes in time.

The ‘lua-time-limit’ Value

It is highly recommended that the script complete within a time limit. Redis enforces this in a weak manner with the ‘lua-time-limit’ value. This is the maximum allowed time (in ms) that the script is allowed to run. The default value is 5 seconds. This is a really long time for CPU-bound activity (scripts have limited access and can’t run commands that access the disk).

However, the script is not killed when it executes beyond this time. Redis starts accepting client commands again, but responds to them with a BUSY error.

If you must kill the script at this point, there are two options available:

• SCRIPT KILL command can be used to stop a script that hasn’t yet done any writes.
• If the script has already performed writes to the server and must still be killed, use the SHUTDOWN NOSAVE to shutdown the server completely.

It is usually better to just wait for the script to complete its operation. The complete information on methods to kill the script execution and related behavior are available in the documentation.

#cloud #database #developer #high availability #howto #redis #scalegrid #lua-time-limit #redis diagram #redis master #redis scripts #redis sentinel #redis servers #redis transactions #sentinel-managed #server failures

PostgreSQL Connection Pooling: Part 4 – PgBouncer vs. Pgpool-II

In our previous posts in this series, we spoke at length about using PgBouncer  and Pgpool-II , the connection pool architecture and pros and cons of leveraging one for your PostgreSQL deployment. In our final post, we will put them head-to-head in a detailed feature comparison and compare the results of PgBouncer vs. Pgpool-II performance for your PostgreSQL hosting !

The bottom line – Pgpool-II is a great tool if you need load-balancing and high availability. Connection pooling is almost a bonus you get alongside. PgBouncer does only one thing, but does it really well. If the objective is to limit the number of connections and reduce resource consumption, PgBouncer wins hands down.

It is also perfectly fine to use both PgBouncer and Pgpool-II in a chain – you can have a PgBouncer to provide connection pooling, which talks to a Pgpool-II instance that provides high availability and load balancing. This gives you the best of both worlds!

PostgreSQL Connection Pooling: Part 4 – PgBouncer vs. Pgpool-II

CLICK TO TWEET

Performance Testing

While PgBouncer may seem to be the better option in theory, theory can often be misleading. So, we pitted the two connection poolers head-to-head, using the standard pgbench tool, to see which one provides better transactions per second throughput through a benchmark test. For good measure, we ran the same tests without a connection pooler too.

Testing Conditions

All of the PostgreSQL benchmark tests were run under the following conditions:

1. Initialized pgbench using a scale factor of 100.
2. Disabled auto-vacuuming on the PostgreSQL instance to prevent interference.
3. No other workload was working at the time.
4. Used the default pgbench script to run the tests.
5. Used default settings for both PgBouncer and Pgpool-II, except max_children*. All PostgreSQL limits were also set to their defaults.
6. All tests ran as a single thread, on a single-CPU, 2-core machine, for a duration of 5 minutes.
7. Forced pgbench to create a new connection for each transaction using the -C option. This emulates modern web application workloads and is the whole reason to use a pooler!

We ran each iteration for 5 minutes to ensure any noise averaged out. Here is how the middleware was installed:

• For PgBouncer, we installed it on the same box as the PostgreSQL server(s). This is the configuration we use in our managed PostgreSQL clusters. Since PgBouncer is a very light-weight process, installing it on the box has no impact on overall performance.
• For Pgpool-II, we tested both when the Pgpool-II instance was installed on the same machine as PostgreSQL (on box column), and when it was installed on a different machine (off box column). As expected, the performance is much better when Pgpool-II is off the box as it doesn’t have to compete with the PostgreSQL server for resources.

Throughput Benchmark

Here are the transactions per second (TPS) results for each scenario across a range of number of clients:

#database #developer #performance #postgresql #connection control #connection pooler #connection pooler performance #connection queue #high availability #load balancing #number of connections #performance testing #pgbench #pgbouncer #pgbouncer and pgpool-ii #pgbouncer vs pgpool #pgpool-ii #pooling modes #postgresql connection pooling #postgresql limits #resource consumption #throughput benchmark #transactions per second #without pooling

Spring Data Redis: Support to increase Productivity When using Redis

Spring Data Redis 

The primary goal of the Spring Data project is to make it easier to build Spring-powered applications that use new data access technologies such as non-relational databases, map-reduce frameworks, and cloud based data services.

This modules provides integration with the Redis store.

Features

• Connection package as low-level abstraction across multiple Redis drivers (Lettuce and Jedis).
• Exception translation to Spring’s portable Data Access exception hierarchy for Redis driver exceptions
• RedisTemplate that provides a high level abstraction for performing various Redis operations, exception translation and serialization support.
• Pubsub support (such as a MessageListenerContainer for message-driven POJOs).
• Redis Sentinel and Redis Cluster support.
• Reactive API using the Lettuce driver.
• JDK, String, JSON and Spring Object/XML mapping serializers.
• JDK Collection implementations on top of Redis.
• Atomic counter support classes.
• Sorting and Pipelining functionality.
• Dedicated support for SORT, SORT/GET pattern and returned bulk values.
• Redis implementation for Spring 3.1 cache abstraction.
• Automatic implementation of Repository interfaces including support for custom finder methods using @EnableRedisRepositories.
• CDI support for repositories.

Code of Conduct

This project is governed by the Spring Code of Conduct. By participating, you are expected to uphold this code of conduct. Please report unacceptable behavior to spring-code-of-conduct@pivotal.io.

Getting Started

Here is a quick teaser of an application using Spring Data Redis in Java:

public class Example {

    // inject the actual template
    @Autowired
    private RedisTemplate<String, String> template;

    // inject the template as ListOperations
    // can also inject as Value, Set, ZSet, and HashOperations
    @Resource(name="redisTemplate")
    private ListOperations<String, String> listOps;

    public void addLink(String userId, URL url) {
        listOps.leftPush(userId, url.toExternalForm());
        // or use template directly
        redisTemplate.boundListOps(userId).leftPush(url.toExternalForm());
    }
}

@Configuration
class ApplicationConfig {

  @Bean
  public RedisConnectionFactory redisConnectionFactory() {
    return new LettuceConnectionFactory();
  }
}

Maven configuration

Add the Maven dependency:

<dependency>
  <groupId>org.springframework.data</groupId>
  <artifactId>spring-data-redis</artifactId>
  <version>${version}</version>
</dependency>

If you’d rather like the latest snapshots of the upcoming major version, use our Maven snapshot repository and declare the appropriate dependency version.

<dependency>
  <groupId>org.springframework.data</groupId>
  <artifactId>spring-data-redis</artifactId>
  <version>${version}-SNAPSHOT</version>
</dependency>

<repository>
  <id>spring-libs-snapshot</id>
  <name>Spring Snapshot Repository</name>
  <url>https://repo.spring.io/libs-snapshot</url>
</repository>

Getting Help

Having trouble with Spring Data? We’d love to help!

• Check the reference documentation, and Javadocs.
• Learn the Spring basics – Spring Data builds on Spring Framework, check the spring.io web-site for a wealth of reference documentation. If you are just starting out with Spring, try one of the guides.
• If you are upgrading, check out the changelog for “new and noteworthy” features.
• Ask a question - we monitor stackoverflow.com for questions tagged with spring-data-redis. You can also chat with the community on Gitter.
• Report bugs with Spring Data Redis at github.com/spring-projects/spring-data-redis.

Reporting Issues

Spring Data uses Github as issue tracking system to record bugs and feature requests. If you want to raise an issue, please follow the recommendations below:

• Before you log a bug, please search the issue tracker to see if someone has already reported the problem.
• If the issue does not already exist, create a new issue.
• Please provide as much information as possible with the issue report, we like to know the version of Spring Data that you are using, the JVM version, Stacktrace, etc.
• If you need to paste code, or include a stack trace use Markdown code fences ```.
• If possible try to create a test-case or project that replicates the issue. Attach a link to your code or a compressed file containing your code.

Building from Source

You don’t need to build from source to use Spring Data (binaries in repo.spring.io), but if you want to try out the latest and greatest, Spring Data can be easily built with the maven wrapper. You also need JDK 1.8.

 $ ./mvnw clean install

If you want to build with the regular mvn command, you will need Maven v3.5.0 or above.

Also see CONTRIBUTING.adoc if you wish to submit pull requests, and in particular please sign the Contributor’s Agreement before your first non-trivial change.

Building reference documentation

Building the documentation builds also the project without running tests.

 $ ./mvnw clean install -Pdistribute

The generated documentation is available from target/site/reference/html/index.html.

Guides

The spring.io site contains several guides that show how to use Spring Data step-by-step:

Messaging with Redis: Learn how to use Redis as a message broker.

Accessing Data Reactively with Redis: Learn how to reactively interface with Redis and Spring Data.

Examples

Spring Data Examples contains example projects that explain specific features in more detail.

Download Details:
Author: spring-projects
Source Code: https://github.com/spring-projects/spring-data-redis
License: Apache-2.0 License

#spring #spring-framework #spring-boot #java 

Redis Master Slave Configuration and Tested in Spring Boot

This article talks about master-slave configuration set up in local docker setup and we will verify read replica of slave nodes when master is unavailable or down. We can read data of not when the master node is down. This sample is explained with Spring Boot.

In realtime, the application has to process and fetch from multiple tables through joining to get complex data every time. To improve performance, if we could cache the response which does not change frequently, so that performance will be faster.

When cache is implemented there will be a 100% probability that a single instance can be hit with maximum load on a single instance. For Ex: an app service or back-end system continuously sends requests to a single instance then there might be chances where the connections in Redis instance may be exhausted.

To solve this, We can run multiple instances of Redis with master-slave architecture, where the master would be the write node and slaves would act as read-only nodes like scaling horizontally. Any updates to the master would be automatically synced with slave nodes asynchronously. Any write attempt to the slave nodes would be rejected.

#architecure #master-slave #redis #spring-boot-2 #redis master slave configuration and tested in spring boot #redis master slave configuration

Spring vs Spring BooDifference Between Spring and Spring Boot

As an extension of the Spring Framework, Spring Boot is widely used to make development on Spring faster, more efficient and convenient. In this article, we will look at some of the parameters were using Spring Boot can drastically reduce the time and effort required in application development.

What is Spring?

Spring Boot

Difference between Spring and Spring Boot

Advantages of Spring Boot over Spring

Conclusion

#full stack development #spring #spring and spring boot #spring boot