Connecting Redis Sentinel with Spring

In this article, I am going to make a basic example of how to connect to a “Redis Sentinel” configuration in a Spring application.

You might also enjoy:  Getting Started with Python, Redis, and Nginx

First of all, for Maven dependencies, “spring-boot-starter-data-redis” is needed in our pom.xml file. This library includes LettuceConnectionFactory, which we will use to connect “Redis Sentinel”.

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-redis</artifactId>
            <version>2.0.5.RELEASE</version>
        </dependency>

We will use application.yml file for connection and cache configurations. Below there is an example of application.yml file. Under cache.config.defaultConfig there is a list of predefined cache configurations. You can map these configurations to service cache definitions which are defined under cache.appCaches.appCacheMap.

cache:
    config:
        defaultConfigs:
        -  cacheName: ONE_HOUR_CACHE         
           timeToLiveSeconds: 3600 
        -  cacheName: TEN_MINUTES_CACHE
           timeToLiveSeconds: 600

  appCaches:
    appCacheMap:
        serviceOne_cache: ONE_HOUR_CACHE
        serviceTwo_cache: TEN_MINUTES_CACHE   

spring:
  cache:
    type: redis
  redis:
    port: 6666
    password: 123pwd
    sentinel:
      master: masterredis
      nodes:
        - 10.0.0.16
        - 10.0.0.17
        - 10.0.0.18
    lettuce:
      shutdown-timeout: 200ms             

In order to read these configuration parameters easily, we will need a CacheSettingsModel and two classes annotated with ConfigurationProperties. One for the defaultConfigs and another for the appCacheMap. For redis configs, which are under spring.redis, we will use RedisProperties class, which is included in spring-boot-starter-data-redis. You can see these classes below:

package com.example.sentinel.config.model;

public class CacheSettingsModel {
private String cacheName;
private String timeToLiveSeconds;

public final String getCacheName() {
return cacheName;
}

public final void setCacheName(String cacheName) {
this.cacheName = cacheName;
}

public final String getTimeToLiveSeconds() {
return timeToLiveSeconds;
}

public final void setTimeToLiveSeconds(String timeToLiveSeconds) {
this.timeToLiveSeconds = timeToLiveSeconds;
}
}

package com.example.sentinel.config;

import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;

import org.apache.commons.collections4.CollectionUtils;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.stereotype.Component;

import com.example.sentinel.config.model.CacheSettingsModel;

@Component
@ConfigurationProperties( prefix="cache.config")
public class CacheSettings {

private List<CacheSettingsModel> defaultConfigs;

public List<CacheSettingsModel> getDefaultConfigs() {
return defaultConfigs;
}

public void setDefaultConfigs(List<CacheSettingsModel> defaultConfigs) {
this.defaultConfigs = defaultConfigs;
}

public Map<String,CacheSettingsModel> getCacheConfigAsMap() {
return CollectionUtils.emptyIfNull(defaultConfigs).stream().
          collect(Collectors.toMap(CacheSettingsModel::getCacheName, c -> c));
}
}

package com.example.sentinel.config;

import java.util.Map;

import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.stereotype.Component;

@Component
@ConfigurationProperties( prefix="cache.appCaches")
public class AppCacheSettings {

private Map<String,String> appCacheMap;

public Map<String, String> getAppCacheMap() {
return appCacheMap;
}

public void setAppCacheMap(Map<String, String> appCacheMap) {
this.appCacheMap = appCacheMap;
}
}

Now we are going to create a connection to “Redis Sentinel” with the help of those configuration classes. As I told you before, we are going to use Lettuce to connect to Redis. We are going to create a new instance of RedisSentinelConfiguration class with the help of RedisProperties class and use it in the constructor of LettuceConnectionFactory.

package com.example.sentinel.config;

import java.time.Duration;
import java.util.HashMap;
import java.util.Map;

import javax.inject.Inject;

import org.springframework.boot.autoconfigure.data.redis.RedisProperties;
import org.springframework.cache.annotation.EnableCaching;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.redis.cache.RedisCacheConfiguration;
import org.springframework.data.redis.cache.RedisCacheManager;
import org.springframework.data.redis.connection.RedisPassword;
import org.springframework.data.redis.connection.RedisSentinelConfiguration;
import org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory;

import com.example.sentinel.config.AppCacheSettings;
import com.example.sentinel.config.CacheSettings;
import com.example.sentinel.config.model.CacheSettingsModel;

@Configuration
@EnableCaching
public class CacheConfig {

    @Inject
    private CacheSettings cacheSettings;
    @Inject
    private AppCacheSettings appCacheSettings;
    @Inject
    private RedisProperties redisProperties;

    @Bean
    public LettuceConnectionFactory redisConnectionFactory() {

        RedisSentinelConfiguration sentinelConfig = new RedisSentinelConfiguration()
                .master(redisProperties.getSentinel().getMaster());

        redisProperties.getSentinel().getNodes().forEach(s -> sentinelConfig.sentinel(s, Integer.valueOf(redisProperties.getPort())));
        sentinelConfig.setPassword(RedisPassword.of(redisProperties.getPassword()));

        return new LettuceConnectionFactory(sentinelConfig);
    }

    @Bean
    public RedisCacheConfiguration cacheConfiguration() {
        return RedisCacheConfiguration.defaultCacheConfig()
                .entryTtl(Duration.ofSeconds(600))
                .disableCachingNullValues();
    }

    private RedisCacheConfiguration buildRedisCacheConfig(CacheSettingsModel cachesProperties) {
        return RedisCacheConfiguration.defaultCacheConfig()
                .entryTtl(Duration.ofSeconds(Long.parseLong(cachesProperties.getTimeToLiveSeconds())))
                .disableCachingNullValues();
    }

    @Bean
    public RedisCacheManager cacheManager() {

        Map<String, RedisCacheConfiguration> cacheConfigs = new HashMap<>();

        Map<String, CacheSettingsModel> cacheConfigMap = cacheSettings.getCacheConfigAsMap();
        Map<String, String> appCacheMap = appCacheSettings.getAppCacheMap();

        appCacheMap.forEach((key, value) -> cacheConfigs.put(key, buildRedisCacheConfig(cacheConfigMap.get(value))));

        return RedisCacheManager.builder(redisConnectionFactory())
                .cacheDefaults(cacheConfiguration())
                .withInitialCacheConfigurations(cacheConfigs)
                .transactionAware()
                .build();
    }
}

At the end, we are going to create a CacheManager bean. We are going to use RedisCacheManager.builder. With the help of AppCacheSettings and CacheSettings classes we are going to set initial cache configurations of cacheManager bean. If you want to enable transactional operations on caches you should set transactionAware to true (default is false). I’ll just copy the definition of transactionAware from the Spring API document.

Set this to “true” to synchronize cache put/evict operations with ongoing Spring-managed transactions, performing the actual cache put/evict operation only in the after-commit phase of a successful transaction.

We are almost done. To use Redis cache in our application, we only need to put @Cacheable (org.springframework.cache.annotation.Cacheable) annotation to the services that we want to cache. As a Value parameter, you can use one of the predefined cache names under appCacheMap properties (in application.yml file) to set cache properties like timeToLive.

 @Cacheable("serviceOne_cache")
 public List<String> serviceOne(String param) {
 ...
 }

That’s it. Spring will do the rest!

Thank for visiting and reading this post.! We highly appreciate your actions! Please share if you liked it!

Further Reading

Database Caching With Redis and Java

How to use Python with Redis

#spring-boot #redis #database #web-development

What is GEEK

Buddha Community

Redis Transactions & Long-Running Lua Scripts

Redis offers two mechanisms for handling transactions – MULTI/EXEC based transactions and Lua scripts evaluation. Redis Lua scripting is the recommended approach and is fairly popular in usage.

Our Redis™ customers who have Lua scripts deployed often report this error – “BUSY Redis is busy running a script. You can only call SCRIPT KILL or SHUTDOWN NOSAVE”. In this post, we will explain the Redis transactional property of scripts, what this error is about, and why we must be extra careful about it on Sentinel-managed systems that can failover.

Transactional Nature of Redis Lua Scripts

Redis “transactions” aren’t really transactions as understood conventionally – in case of errors, there is no rollback of writes made by the script.

“Atomicity” of Redis scripts is guaranteed in the following manner:

• Once a script begins executing, all other commands/scripts are blocked until the script completes. So, other clients either see the changes made by the script or they don’t. This is because they can only execute either before the script or after the script.
• However, Redis doesn’t do rollbacks, so on an error within a script, any changes already made by the script will be retained and future commands/scripts will see those partial changes.
• Since all other clients are blocked while the script executes, it is critical that the script is well-behaved and finishes in time.

The ‘lua-time-limit’ Value

It is highly recommended that the script complete within a time limit. Redis enforces this in a weak manner with the ‘lua-time-limit’ value. This is the maximum allowed time (in ms) that the script is allowed to run. The default value is 5 seconds. This is a really long time for CPU-bound activity (scripts have limited access and can’t run commands that access the disk).

However, the script is not killed when it executes beyond this time. Redis starts accepting client commands again, but responds to them with a BUSY error.

If you must kill the script at this point, there are two options available:

• SCRIPT KILL command can be used to stop a script that hasn’t yet done any writes.
• If the script has already performed writes to the server and must still be killed, use the SHUTDOWN NOSAVE to shutdown the server completely.

It is usually better to just wait for the script to complete its operation. The complete information on methods to kill the script execution and related behavior are available in the documentation.

#cloud #database #developer #high availability #howto #redis #scalegrid #lua-time-limit #redis diagram #redis master #redis scripts #redis sentinel #redis servers #redis transactions #sentinel-managed #server failures

PostgreSQL Connection Pooling: Part 4 – PgBouncer vs. Pgpool-II

In our previous posts in this series, we spoke at length about using PgBouncer  and Pgpool-II , the connection pool architecture and pros and cons of leveraging one for your PostgreSQL deployment. In our final post, we will put them head-to-head in a detailed feature comparison and compare the results of PgBouncer vs. Pgpool-II performance for your PostgreSQL hosting !

The bottom line – Pgpool-II is a great tool if you need load-balancing and high availability. Connection pooling is almost a bonus you get alongside. PgBouncer does only one thing, but does it really well. If the objective is to limit the number of connections and reduce resource consumption, PgBouncer wins hands down.

It is also perfectly fine to use both PgBouncer and Pgpool-II in a chain – you can have a PgBouncer to provide connection pooling, which talks to a Pgpool-II instance that provides high availability and load balancing. This gives you the best of both worlds!

PostgreSQL Connection Pooling: Part 4 – PgBouncer vs. Pgpool-II

CLICK TO TWEET

Performance Testing

While PgBouncer may seem to be the better option in theory, theory can often be misleading. So, we pitted the two connection poolers head-to-head, using the standard pgbench tool, to see which one provides better transactions per second throughput through a benchmark test. For good measure, we ran the same tests without a connection pooler too.

Testing Conditions

All of the PostgreSQL benchmark tests were run under the following conditions:

1. Initialized pgbench using a scale factor of 100.
2. Disabled auto-vacuuming on the PostgreSQL instance to prevent interference.
3. No other workload was working at the time.
4. Used the default pgbench script to run the tests.
5. Used default settings for both PgBouncer and Pgpool-II, except max_children*. All PostgreSQL limits were also set to their defaults.
6. All tests ran as a single thread, on a single-CPU, 2-core machine, for a duration of 5 minutes.
7. Forced pgbench to create a new connection for each transaction using the -C option. This emulates modern web application workloads and is the whole reason to use a pooler!

We ran each iteration for 5 minutes to ensure any noise averaged out. Here is how the middleware was installed:

• For PgBouncer, we installed it on the same box as the PostgreSQL server(s). This is the configuration we use in our managed PostgreSQL clusters. Since PgBouncer is a very light-weight process, installing it on the box has no impact on overall performance.
• For Pgpool-II, we tested both when the Pgpool-II instance was installed on the same machine as PostgreSQL (on box column), and when it was installed on a different machine (off box column). As expected, the performance is much better when Pgpool-II is off the box as it doesn’t have to compete with the PostgreSQL server for resources.

Throughput Benchmark

Here are the transactions per second (TPS) results for each scenario across a range of number of clients:

#database #developer #performance #postgresql #connection control #connection pooler #connection pooler performance #connection queue #high availability #load balancing #number of connections #performance testing #pgbench #pgbouncer #pgbouncer and pgpool-ii #pgbouncer vs pgpool #pgpool-ii #pooling modes #postgresql connection pooling #postgresql limits #resource consumption #throughput benchmark #transactions per second #without pooling

Redis Master Slave Configuration and Tested in Spring Boot

This article talks about master-slave configuration set up in local docker setup and we will verify read replica of slave nodes when master is unavailable or down. We can read data of not when the master node is down. This sample is explained with Spring Boot.

In realtime, the application has to process and fetch from multiple tables through joining to get complex data every time. To improve performance, if we could cache the response which does not change frequently, so that performance will be faster.

When cache is implemented there will be a 100% probability that a single instance can be hit with maximum load on a single instance. For Ex: an app service or back-end system continuously sends requests to a single instance then there might be chances where the connections in Redis instance may be exhausted.

To solve this, We can run multiple instances of Redis with master-slave architecture, where the master would be the write node and slaves would act as read-only nodes like scaling horizontally. Any updates to the master would be automatically synced with slave nodes asynchronously. Any write attempt to the slave nodes would be rejected.

#architecure #master-slave #redis #spring-boot-2 #redis master slave configuration and tested in spring boot #redis master slave configuration

Spring vs Spring BooDifference Between Spring and Spring Boot

As an extension of the Spring Framework, Spring Boot is widely used to make development on Spring faster, more efficient and convenient. In this article, we will look at some of the parameters were using Spring Boot can drastically reduce the time and effort required in application development.

What is Spring?

Spring Boot

Difference between Spring and Spring Boot

Advantages of Spring Boot over Spring

Conclusion

#full stack development #spring #spring and spring boot #spring boot

Spring Native turns Spring apps into native executables

Spring Native beta release leverages GraalVM to compile Spring Java and Kotlin applications to native images, reducing startup time and memory overhead compared to the JVM.

Spring Native, for compiling Spring Java applications to standalone executables called native images, is now available as a beta release. Native images promise faster startup times and lower runtime memory overhead compared to the JVM.

Launched March 11 and available on start.spring.io, the Spring Native beta compiles Spring applications to native images using the GraalVM multi-language runtime. These standalone executables offer benefits including nearly instant startup (typically fewer than 100ms), instant peak performance, and lower memory consumption, at the cost of longer build times and fewer runtime optimizations than the JVM.

#spring native turns spring apps into native executables #spring native #spring #native executables #spring apps