1655877894
Heterogeneous migration: reducing Dangdang’s customer system RTO 60x!
Apache ShardingSphere helps Dangdang rebuild its customer system with 350 million users, and seamlessly transition from a PHP+SQL Server technology stack to a Java+ShardingSphere+MySQL stack. The performance, availability, and maintainability of its customer system have been significantly improved, which is the best practice of ShardingSphere’s heterogeneous migration.
Dangdang’s customer system
Dangdang’s customer system is mainly responsible for account registration, login, and privacy data maintenance. Its previous technology stack was based on PHP and SQL Server, which means a standard centralized architecture, as shown in the figure below.
Before the rebuild project started, several business modules of the customer system had encountered multiple problems and technical challenges, such as logical decentralization, low throughput, and high operation & maintenance costs.
To improve customers’ shopping experience, Dangdang’s technical team decided to optimize the business logic and underlying data architecture to achieve the availability, scalability, and comprehensive improvement of the customer system in multiple scenarios. The rebuild also introduced many technological innovations such as cross-data source double write, read/write splitting, intelligent gateway, and gray release.
Dangdang’s technical team completed the system rebuild within half a year, from demand design, sharding planning, logic optimization, and stress testing to its official launch.
The project used Java to reconstruct more than ten modules, build distributed database solutions through ShardingSphere & MySQL, and finally complete the online migration of heterogeneous databases. The project boasts the following highlights:
- Reconstruct PHP business code using Java language.
- Replace SQL Server with ShardingSphere & MySQL.
- Complete online data migration of 350 million users.
- Complete a seamless launch through the data double-write scheme.
Pain points & challenges
Business pain points
At the business level, the registration and login logic of some modules of the customer system was scattered at different ends. This resulted in high maintenance costs, and the old technical architecture was limited in terms of performance improvement and high availability.
- Maintenance difficulty: the registration and login logic of multiple platforms is scattered, so business maintenance is complicated.
- Limited performance: the PHP & SQL Server, a centralized technical architecture, had insufficient throughput.
- Poor availability and security: If the active/standby status of SQL Server changes, the subscription database becomes invalid and the reconfiguration takes a window of time. The security of SQL Server running on Windows Server is poor due to viruses, and the upgrading takes a long time (>30min) after the patch is installed.
Challenges
- Data integrity: the customer system involves data of more than 350 million users. It is necessary to ensure data consistency and integrity after migrating from SQL Server to MySQL.
- API transparency: the API is transparent to the caller to ensure that the caller does not change and to minimize the change of interface.
- Seamless switch: the business system must be seamlessly switched over without impact on business.
- Time is short: the system will be blocked before and after “618 (aka JD.com Day) and 11.11 (aka Singles Day)” (two online shopping festivals in China), so we need to switch it between the two shopping promotions in a limited window of time, and then undergo the tests to prepare for the 11.11 shopping festival.
Solutions
Overall planning
To improve the maintainability, availability, and performance of the customer system, the R&D team reorganized the customer system architecture.
At the application layer, the goal was to unify the function logic of all terminals and improve business maintainability.
At the database layer, the centralized architecture was transformed into a distributed database architecture to improve performance and availability, which is exactly the open-source distributed solution built by ShardingSphere & MySQL.
- Application layer: As Dangdang’s overall technology stack changed, its business development language changed from PHP to Java.
- Middleware: As a mature open-source database middleware, ShardingSphere, was used to achieve data sharding.
- Database: Multiple MySQL clusters were used to replace SQL Server databases.
The overall architecture design introduced multiple schemes, such as distributed primary-key generation strategy, shard management, data migration verification, and gray release.
Distributed primary-key generation strategy
Distributed primary-key generation strategy is the first problem to be solved if database architecture is to be transformed from a centralized architecture to a distributed one based on middleware.
During the system rebuild, we chose to build two or more database ID-generating servers. Each server had a Sequence table that records the current ID of each table. The step size of ID that increases in the Sequence table is the number of servers. The starting values are staggered so that the ID generation is hashed to each server node.
Implementing sharding (Apache ShardingSphere）
During the customer system rebuild, database sharding was completed through Apache ShardingSphere, and the read/write splitting function was also enabled.
Due to the requirements of the customer system for high concurrency and low latency, the access end chose ShardingSphere-JDBC, which is positioned as a lightweight Java framework and provides additional services in Java’s JDBC layer.
It connects directly to the database via the client and provides services in the form of a jar package without additional deployment and dependence. It can be viewed as an enhanced version of the JDBC driver, fully compatible with JDBC and various ORM frameworks.
Sharding: ShardingSphere supports a complete set of sharding algorithms, including modulo operation, hash, range, time, and customized algorithms. Customers use the modulo sharding algorithm to split large tables.
Read-write splitting: in addition to Sharding, ShardingSphere’s read/write splitting function is also enabled to make full use of MHA cluster resources and improve system throughput capacity.
Double-write & data synchronization
Data synchronization runs through the whole rebuild project, and the integrity and consistency of data migration are vital to the rebuild.
This example periodically synchronizes SQL Server’s historical data to MySQL based on Elastic-Job synchronization. During the database switchover, a backup scheme is used to double-write the database to ensure data consistency. The process consists of:
Step 1: implement the double-write mechanism
Disconnect link 1, get through links 2, 3, 4, and then 9, 10.
Step 2: switch the login service
Disconnect links 9,10, get through link 7 and disconnect link 5.
Step 3: switch read service
Get through link 8 and disconnect link 6.
Step 4: cancel the double-write mechanism
Disconnect link 2 and complete the switchover.
Data verification is performed periodically on both the service side and the database side. Different frequencies are used in different time periods to sample or fully check data integrity. COUNT/SUM is also verified on the database side.
Customer system reconstruction adopts an apollo-based gray release. In the process of new login processing, configuration items are gradually released and sequential cutover within a small range is implemented to ensure the launch success rate. The rebuilt system architecture is shown in the following figure.
Advantages
After the rebuild, the response speed of Dangdang’s customer system is significantly improved, and the daily operation & maintenance costs are also reduced.
The distributed solution provided by ShardingSphere plays a big part in this. The solution is suitable for various high-traffic Internet platform services, as well as e-commerce platforms and other data-processing systems.
- Performance improvement: response speed increased by more than 20%.
- High availability: RTO is reduced to less than 30s owing to the ShardingSphere & MySQL design.
- Easy to maintain: business logic and database maintainability are significantly improved.
- Seamless migration: complete online cutover of each module within 6 months, and window time is zero.
Conclusion
This is ShardingSphere’s second implementation by Dangdang, following the previous one we shared in the post “Asia’s E-Commerce Giant Dangdang Increases Order Processing Speed by 30% — Saves Over Ten Million in Technology Budget with Apache ShardingSphere”.
Apache ShardingSphere provides strong support for enterprise systems, as the project strives for simplicity and perfection, to achieve simpler business logic and maximum performance.
Apache ShardingSphere Project Links:
What is GEEK
Buddha Community
1622049211
Fynzo Customer Feedback Software For Cafes, Hotels, Saloons, Spa!
Customer Feedback Tool | Fynzo online customer feedback comes with Android, iOS app. Collect feedback from your customers with tablets or send them feedback links.
Visit page for more information: https://www.fynzo.com/feedback
#CustomerFeedbackSystem
#PowerfulCustomerFeedbackSystem
#freecustomerfeedbacktools
#automatedcustomerfeedbacksystem
#customerfeedbacktools
#customerratingsystem
#Customerfeedbackmanagement
#customer feedback system #powerful customer feedback system #free customer feedback tools #automated customer feedback system #customer feedback tools #customer rating system
1656151740
Test_cov_console: Flutter Console Coverage Test
Flutter Console Coverage Test
This small dart tools is used to generate Flutter Coverage Test report to console
How to install
Add a line like this to your package's pubspec.yaml (and run an implicit flutter pub get):
dev_dependencies:
test_cov_console: ^0.2.2
How to run
run the following command to make sure all flutter library is up-to-date
flutter pub get
Running "flutter pub get" in coverage... 0.5s
run the following command to generate lcov.info on coverage directory
flutter test --coverage
00:02 +1: All tests passed!
run the tool to generate report from lcov.info
flutter pub run test_cov_console
---------------------------------------------|---------|---------|---------|-------------------|
File |% Branch | % Funcs | % Lines | Uncovered Line #s |
---------------------------------------------|---------|---------|---------|-------------------|
lib/src/ | | | | |
print_cov.dart | 100.00 | 100.00 | 88.37 |...,149,205,206,207|
print_cov_constants.dart | 0.00 | 0.00 | 0.00 | no unit testing|
lib/ | | | | |
test_cov_console.dart | 0.00 | 0.00 | 0.00 | no unit testing|
---------------------------------------------|---------|---------|---------|-------------------|
All files with unit testing | 100.00 | 100.00 | 88.37 | |
---------------------------------------------|---------|---------|---------|-------------------|
Optional parameter
If not given a FILE, "coverage/lcov.info" will be used.
-f, --file=<FILE> The target lcov.info file to be reported
-e, --exclude=<STRING1,STRING2,...> A list of contains string for files without unit testing
to be excluded from report
-l, --line It will print Lines & Uncovered Lines only
Branch & Functions coverage percentage will not be printed
-i, --ignore It will not print any file without unit testing
-m, --multi Report from multiple lcov.info files
-c, --csv Output to CSV file
-o, --output=<CSV-FILE> Full path of output CSV file
If not given, "coverage/test_cov_console.csv" will be used
-t, --total Print only the total coverage
Note: it will ignore all other option (if any), except -m
-p, --pass=<MINIMUM> Print only the whether total coverage is passed MINIMUM value or not
If the value >= MINIMUM, it will print PASSED, otherwise FAILED
Note: it will ignore all other option (if any), except -m
-h, --help Show this help
example run the tool with parameters
flutter pub run test_cov_console --file=coverage/lcov.info --exclude=_constants,_mock
---------------------------------------------|---------|---------|---------|-------------------|
File |% Branch | % Funcs | % Lines | Uncovered Line #s |
---------------------------------------------|---------|---------|---------|-------------------|
lib/src/ | | | | |
print_cov.dart | 100.00 | 100.00 | 88.37 |...,149,205,206,207|
lib/ | | | | |
test_cov_console.dart | 0.00 | 0.00 | 0.00 | no unit testing|
---------------------------------------------|---------|---------|---------|-------------------|
All files with unit testing | 100.00 | 100.00 | 88.37 | |
---------------------------------------------|---------|---------|---------|-------------------|
report for multiple lcov.info files (-m, --multi)
It support to run for multiple lcov.info files with the followings directory structures:
1. No root module
<root>/<module_a>
<root>/<module_a>/coverage/lcov.info
<root>/<module_a>/lib/src
<root>/<module_b>
<root>/<module_b>/coverage/lcov.info
<root>/<module_b>/lib/src
...
2. With root module
<root>/coverage/lcov.info
<root>/lib/src
<root>/<module_a>
<root>/<module_a>/coverage/lcov.info
<root>/<module_a>/lib/src
<root>/<module_b>
<root>/<module_b>/coverage/lcov.info
<root>/<module_b>/lib/src
...
You must run test_cov_console on <root> dir, and the report would be grouped by module, here is
the sample output for directory structure 'with root module':
flutter pub run test_cov_console --file=coverage/lcov.info --exclude=_constants,_mock --multi
---------------------------------------------|---------|---------|---------|-------------------|
File |% Branch | % Funcs | % Lines | Uncovered Line #s |
---------------------------------------------|---------|---------|---------|-------------------|
lib/src/ | | | | |
print_cov.dart | 100.00 | 100.00 | 88.37 |...,149,205,206,207|
lib/ | | | | |
test_cov_console.dart | 0.00 | 0.00 | 0.00 | no unit testing|
---------------------------------------------|---------|---------|---------|-------------------|
All files with unit testing | 100.00 | 100.00 | 88.37 | |
---------------------------------------------|---------|---------|---------|-------------------|
---------------------------------------------|---------|---------|---------|-------------------|
File - module_a - |% Branch | % Funcs | % Lines | Uncovered Line #s |
---------------------------------------------|---------|---------|---------|-------------------|
lib/src/ | | | | |
print_cov.dart | 100.00 | 100.00 | 88.37 |...,149,205,206,207|
lib/ | | | | |
test_cov_console.dart | 0.00 | 0.00 | 0.00 | no unit testing|
---------------------------------------------|---------|---------|---------|-------------------|
All files with unit testing | 100.00 | 100.00 | 88.37 | |
---------------------------------------------|---------|---------|---------|-------------------|
---------------------------------------------|---------|---------|---------|-------------------|
File - module_b - |% Branch | % Funcs | % Lines | Uncovered Line #s |
---------------------------------------------|---------|---------|---------|-------------------|
lib/src/ | | | | |
print_cov.dart | 100.00 | 100.00 | 88.37 |...,149,205,206,207|
lib/ | | | | |
test_cov_console.dart | 0.00 | 0.00 | 0.00 | no unit testing|
---------------------------------------------|---------|---------|---------|-------------------|
All files with unit testing | 100.00 | 100.00 | 88.37 | |
---------------------------------------------|---------|---------|---------|-------------------|
Output to CSV file (-c, --csv, -o, --output)
flutter pub run test_cov_console -c --output=coverage/test_coverage.csv
#### sample CSV output file:
File,% Branch,% Funcs,% Lines,Uncovered Line #s
lib/,,,,
test_cov_console.dart,0.00,0.00,0.00,no unit testing
lib/src/,,,,
parser.dart,100.00,100.00,97.22,"97"
parser_constants.dart,100.00,100.00,100.00,""
print_cov.dart,100.00,100.00,82.91,"29,49,51,52,171,174,177,180,183,184,185,186,187,188,279,324,325,387,388,389,390,391,392,393,394,395,398"
print_cov_constants.dart,0.00,0.00,0.00,no unit testing
All files with unit testing,100.00,100.00,86.07,""Installing
Use this package as an executable
Install it
You can install the package from the command line:
dart pub global activate test_cov_consoleUse it
The package has the following executables:
$ test_cov_console
Use this package as a library
Depend on it
Run this command:
With Dart:
$ dart pub add test_cov_consoleWith Flutter:
$ flutter pub add test_cov_consoleThis will add a line like this to your package's pubspec.yaml (and run an implicit dart pub get):
dependencies:
test_cov_console: ^0.2.2Alternatively, your editor might support dart pub get or flutter pub get. Check the docs for your editor to learn more.
Import it
Now in your Dart code, you can use:
import 'package:test_cov_console/test_cov_console.dart';example/lib/main.dart
import 'package:flutter/material.dart';
void main() {
runApp(MyApp());
}
class MyApp extends StatelessWidget {
// This widget is the root of your application.
@override
Widget build(BuildContext context) {
return MaterialApp(
title: 'Flutter Demo',
theme: ThemeData(
// This is the theme of your application.
//
// Try running your application with "flutter run". You'll see the
// application has a blue toolbar. Then, without quitting the app, try
// changing the primarySwatch below to Colors.green and then invoke
// "hot reload" (press "r" in the console where you ran "flutter run",
// or simply save your changes to "hot reload" in a Flutter IDE).
// Notice that the counter didn't reset back to zero; the application
// is not restarted.
primarySwatch: Colors.blue,
// This makes the visual density adapt to the platform that you run
// the app on. For desktop platforms, the controls will be smaller and
// closer together (more dense) than on mobile platforms.
visualDensity: VisualDensity.adaptivePlatformDensity,
),
home: MyHomePage(title: 'Flutter Demo Home Page'),
);
}
}
class MyHomePage extends StatefulWidget {
MyHomePage({Key? key, required this.title}) : super(key: key);
// This widget is the home page of your application. It is stateful, meaning
// that it has a State object (defined below) that contains fields that affect
// how it looks.
// This class is the configuration for the state. It holds the values (in this
// case the title) provided by the parent (in this case the App widget) and
// used by the build method of the State. Fields in a Widget subclass are
// always marked "final".
final String title;
@override
_MyHomePageState createState() => _MyHomePageState();
}
class _MyHomePageState extends State<MyHomePage> {
int _counter = 0;
void _incrementCounter() {
setState(() {
// This call to setState tells the Flutter framework that something has
// changed in this State, which causes it to rerun the build method below
// so that the display can reflect the updated values. If we changed
// _counter without calling setState(), then the build method would not be
// called again, and so nothing would appear to happen.
_counter++;
});
}
@override
Widget build(BuildContext context) {
// This method is rerun every time setState is called, for instance as done
// by the _incrementCounter method above.
//
// The Flutter framework has been optimized to make rerunning build methods
// fast, so that you can just rebuild anything that needs updating rather
// than having to individually change instances of widgets.
return Scaffold(
appBar: AppBar(
// Here we take the value from the MyHomePage object that was created by
// the App.build method, and use it to set our appbar title.
title: Text(widget.title),
),
body: Center(
// Center is a layout widget. It takes a single child and positions it
// in the middle of the parent.
child: Column(
// Column is also a layout widget. It takes a list of children and
// arranges them vertically. By default, it sizes itself to fit its
// children horizontally, and tries to be as tall as its parent.
//
// Invoke "debug painting" (press "p" in the console, choose the
// "Toggle Debug Paint" action from the Flutter Inspector in Android
// Studio, or the "Toggle Debug Paint" command in Visual Studio Code)
// to see the wireframe for each widget.
//
// Column has various properties to control how it sizes itself and
// how it positions its children. Here we use mainAxisAlignment to
// center the children vertically; the main axis here is the vertical
// axis because Columns are vertical (the cross axis would be
// horizontal).
mainAxisAlignment: MainAxisAlignment.center,
children: <Widget>[
Text(
'You have pushed the button this many times:',
),
Text(
'$_counter',
style: Theme.of(context).textTheme.headline4,
),
],
),
),
floatingActionButton: FloatingActionButton(
onPressed: _incrementCounter,
tooltip: 'Increment',
child: Icon(Icons.add),
), // This trailing comma makes auto-formatting nicer for build methods.
);
}
}Author: DigitalKatalis
Source Code: https://github.com/DigitalKatalis/test_cov_console
License: BSD-3-Clause license
1621965612
Fynzo Customer Feedback Software For Cafes, Hotels, Saloons, Spa!
Powerful Customer Feedback Software - Fynzo
Wonder what your customers think about your services or products? Unveil it with our customer feedback system. Delight your customers by taking their feedback.
For more info visit: https://www.fynzo.com/feedback
#customer feedback system #powerful customer feedback system #free customer feedback tools #automated customer feedback system #customer feedback management
1594166040
What are the benefits of cloud migration? Reasons you should migrate
The moving of applications, databases and other business elements from the local server to the cloud server called cloud migration. This article will deal with migration techniques, requirement and the benefits of cloud migration.
In simple terms, moving from local to the public cloud server is called cloud migration. Gartner says 17.5% revenue growth as promised in cloud migration and also has a forecast for 2022 as shown in the following image.
#cloud computing services #cloud migration #all #cloud #cloud migration strategy #enterprise cloud migration strategy #business benefits of cloud migration #key benefits of cloud migration #benefits of cloud migration #types of cloud migration
1620633584
System Databases in SQL Server
Introduction
In SSMS, we many of may noticed System Databases under the Database Folder. But how many of us knows its purpose?. In this article lets discuss about the System Databases in SQL Server.
System Database
Fig. 1 System Databases
There are five system databases, these databases are created while installing SQL Server.
- Master
- Model
- MSDB
- Tempdb
- Resource
Master
- This database contains all the System level Information in SQL Server. The Information in form of Meta data.
- Because of this master database, we are able to access the SQL Server (On premise SQL Server)
Model
- This database is used as a template for new databases.
- Whenever a new database is created, initially a copy of model database is what created as new database.
MSDB
- This database is where a service called SQL Server Agent stores its data.
- SQL server Agent is in charge of automation, which includes entities such as jobs, schedules, and alerts.
TempDB
- The Tempdb is where SQL Server stores temporary data such as work tables, sort space, row versioning information and etc.
- User can create their own version of temporary tables and those are stored in Tempdb.
- But this database is destroyed and recreated every time when we restart the instance of SQL Server.
Resource
- The resource database is a hidden, read only database that holds the definitions of all system objects.
- When we query system object in a database, they appear to reside in the sys schema of the local database, but in actually their definitions reside in the resource db.
#sql server #master system database #model system database #msdb system database #sql server system databases #ssms #system database #system databases in sql server #tempdb system database