1675337640
Learn how to write performant and safe apps quickly in Rust. This post guides you through designing and implementing an HTTP Tunnel, and covers the basics of creating robust, scalable, and observable applications.
About a year ago, I started to learn Rust. The first two weeks were quite painful. Nothing compiled; I didn’t know how to do basic operations; I couldn’t make a simple program run. But step by step, I started to understand what the compiler wanted. Even more, I realized that it forces the right thinking and correct behavior.
Yes, sometimes, you have to write seemingly redundant constructs. But it’s better not to compile a correct program than to compile an incorrect one. This makes making mistakes more difficult.
Soon after, I became more or less productive and finally could do what I wanted. Well, most of the time.
Out of curiosity, I decided to take on a slightly more complex challenge: implement an HTTP Tunnel in Rust. It turned out to be surprisingly easy to do and took about a day, which is quite impressive. I stitched together tokio, clap, serde, and several other very useful crates. Let me share the knowledge I gained during this exciting challenge and elaborate on why I organized the app this way. I hope you’ll enjoy it.
Simply put, it’s a lightweight VPN that you can set up with your browser so your internet provider cannot block or track your activity, and web servers won’t see your IP address.
If you’d like, you can test it with your browser locally, e.g., with Firefox, (otherwise, just skip this section for now).
$ cargo install http-tunnel
$ http-tunnel --bind 0.0.0.0:8080 http
You can also check the HTTP-tunnel GitHub repository for the build/installation instructions.
Now, you can go to your browser and set the HTTP Proxy
to localhost:8080
. For instance, in Firefox, search for proxy
in the preferences section:
Find the proxy settings, specify it for HTTP Proxy
, and check it for HTTPS
:
Set the proxy to just built http_tunnel
. You can visit several web pages and check the ./logs/application.log
file—all your traffic was going via the tunnel. For example:
Now, let’s walk through the process from the beginning.
Each application starts with a design, which means we need to define the following:
We need to follow the specification outlined in the here:
Negotiate a target with an HTTP CONNECT
request. For example, if the client wants to create a tunnel to Wikipedia’s website, the request will look like this:
CONNECT www.wikipedia.org:443 HTTP/1.1
...
Followed by a response like below:
HTTP/1.1 200 OK
After this point, just relay TCP traffic both ways until one of the sides closes it or an I/O error happens.
The HTTP Tunnel should work for both HTTP
and HTTPS
.
We also should be able to manage access/block targets (e.g., to block-list trackers).
The service shouldn’t log any information that identifies users.
It should have high throughput and low latency (it should be unnoticeable for users and relatively cheap to run).
Ideally, we want it to be resilient to traffic spikes, provide noisy neighbor isolation, and resist basic DDoS attacks.
Error messaging should be developer-friendly. We want the system to be observable to troubleshoot and tune it in production at a massive scale.
When designing components, we need to break down the app into a set of responsibilities. First, let’s see what our flow diagram looks like:
To implement this, we can introduce the following four main components:
HTTP CONNECT
negotiatorTCP/TLS acceptor
When we roughly know how to organize the app, it’s time to decide which dependencies we should use. For Rust, the best I/O library I know is tokio. In the tokio
family, there are many libraries including tokio-tls
, which makes things much simpler. So the TCP acceptor code would look like this:
let mut tcp_listener = TcpListener::bind(&proxy_configuration.bind_address)
.await
.map_err(|e| {
error!(
"Error binding address {}: {}",
&proxy_configuration.bind_address, e
);
e
})?;
And then the whole acceptor loop + launching asynchronous connection handlers would be:
loop {
// Asynchronously wait for an inbound socket.
let socket = tcp_listener.accept().await;
let dns_resolver_ref = dns_resolver.clone();
match socket {
Ok((stream, _)) => {
let config = config.clone();
// handle accepted connections asynchronously
tokio::spawn(async move { tunnel_stream(&config, stream, dns_resolver_ref).await });
}
Err(e) => error!("Failed TCP handshake {}", e),
}
}
Let’s break down what’s happening here. We accept a connection. If the operation was successful, use tokio::spawn
to create a new task that will handle that connection. Memory/thread-safety management happens behind the scenes. Handling futures is hidden by the async/await
syntax sugar.
However, there is one question. TcpStream
and TlsStream
are different objects, but handling both is precisely the same. Can we reuse the same code? In Rust, abstraction is achieved via Traits
, which are super handy:
/// Tunnel via a client connection.
async fn tunnel_stream<C: AsyncRead + AsyncWrite + Send + Unpin + 'static>(
config: &ProxyConfiguration,
client_connection: C,
dns_resolver: DnsResolver,
) -> io::Result<()> {...}
The stream must implement:
AsyncRead /Write
: Allows us to read/write it asynchronously.Send
: To be able to send between threads.Unpin
: To be moveable (otherwise, we won’t be able to do async move
and tokio::spawn
to create an async
task).'static
: To denote that it may live until the application shutdown and doesn’t depend on any other object’s destruction.Which our TCP/TLS
streams exactly are. However, now we can see that it doesn’t have to be TCP/TLS
streams. This code would work for UDP
, QUIC
, or ICMP
. For example, it can wrap any protocol within any other protocol or itself.
In other words, this code is reusable, extendable, and ready for migration, which happens sooner or later.
HTTP connect negotiator and target connector
Let’s pause for a second and think at a higher level. What if we can abstract from HTTP Tunnel, and need to implement a generic tunnel?
A target could be, for instance, another tunnel. Also, we can support different protocols. The core would stay the same.
We already saw that the tunnel_stream
method already works with any L4 Client<->Tunnel
connection.
#[async_trait]
pub trait TunnelTarget {
type Addr;
fn addr(&self) -> Self::Addr;
}
#[async_trait]
pub trait TargetConnector {
type Target: TunnelTarget + Send + Sync + Sized;
type Stream: AsyncRead + AsyncWrite + Send + Sized + 'static;
async fn connect(&mut self, target: &Self::Target) -> io::Result<Self::Stream>;
}
Here, we specify two abstractions:
TunnelTarget
is just something that has an Addr
— whatever it is.TargetConnector
— can connect to that Addr
and needs to return a stream that supports async I/O.Okay, but what about the target negotiation? The tokio-utils
crate already has an abstraction for that, named Framed
streams (with corresponding Encoder/Decoder
traits). We need to implement them for HTTP CONNECT
(or any other proxy protocol). You can find the implementation here.
Relay
We only have one major component remaining — that relays data after the tunnel negotiation is done. tokio
provides a method to split a stream into two halves: ReadHalf
and WriteHalf
. We can split client and target connections and relay them in both directions:
let (client_recv, client_send) = io::split(client);
let (target_recv, target_send) = io::split(target);
let upstream_task =
tokio::spawn(
async move {
upstream_relay.relay_data(client_recv, target_send).await
});
let downstream_task =
tokio::spawn(
async move {
downstream_relay.relay_data(target_recv, client_send).await
});
Where the relay_data(…)
definition requires nothing more than implementing the abstractions mentioned above. For example, it can connect any two halves of a stream:
/// Relays data in a single direction. E.g.
pub async fn relay_data<R: AsyncReadExt + Sized, W: AsyncWriteExt + Sized>(
self,
mut source: ReadHalf<R>,
mut dest: WriteHalf<W>,
) -> io::Result<RelayStats> {...}
And finally, instead of a simple HTTP Tunnel, we have an engine that can be used to build any type of tunnels or a chain of tunnels (e.g., for onion routing) over any transport and proxy protocols:
/// A connection tunnel.
///
/// # Parameters
/// * `<H>` - proxy handshake codec for initiating a tunnel.
/// It extracts the request message, which contains the target, and, potentially policies.
/// It also takes care of encoding a response.
/// * `<C>` - a connection from from client.
/// * `<T>` - target connector. It takes result produced by the codec and establishes a connection
/// to a target.
///
/// Once the target connection is established, it relays data until any connection is closed or an
/// error happens.
impl<H, C, T> ConnectionTunnel<H, C, T>
where
H: Decoder<Error = EstablishTunnelResult> + Encoder<EstablishTunnelResult>,
H::Item: TunnelTarget + Sized + Display + Send + Sync,
C: AsyncRead + AsyncWrite + Sized + Send + Unpin + 'static,
T: TargetConnector<Target = H::Item>,
{...}
The implementation is almost trivial in basic cases, but we want our app to handle failures, and that’s the focus of the next section.
The amount of time engineers deal with failures is proportional to the scale of a system. It’s easy to write happy-case code. Still, if it enters an irrecoverable state on the very first error, it’s painful to use. Besides that, your app will be used by other engineers, and there are very few things more irritating than cryptic/misleading error messages. If your code runs as a part of a large service, some people need to monitor and support it (e.g., SREs or DevOps), and it should be a pleasure for them to deal with your service.
What kind of failures may an HTTP Tunnel encounter?
It’s a good idea to enumerate all error codes that your app returns to the client. So it’s clear why a request failed if the operation can be tried again (or shouldn’t) if it’s an integration bug, or just network noise:
pub enum EstablishTunnelResult {
/// Successfully connected to target.
Ok,
/// Malformed request
BadRequest,
/// Target is not allowed
Forbidden,
/// Unsupported operation, however valid for the protocol.
OperationNotAllowed,
/// The client failed to send a tunnel request timely.
RequestTimeout,
/// Cannot connect to target.
BadGateway,
/// Connection attempt timed out.
GatewayTimeout,
/// Busy. Try again later.
TooManyRequests,
/// Any other error. E.g. an abrupt I/O error.
ServerError,
}
Dealing with delays is crucial for a network app. If your operations don’t have timeouts, it’s a matter of time until all of your threads will be “Waiting for Godot,” or your app will exhaust all available resources and become unavailable. Here we delegate the timeout definition to RelayPolicy
:
let read_result = self
.relay_policy
.timed_operation(source.read(&mut buffer))
.await;
if read_result.is_err() {
shutdown_reason = RelayShutdownReasons::ReaderTimeout;
break;
}
let n = match read_result.unwrap() {
Ok(n) if n == 0 => {
shutdown_reason = RelayShutdownReasons::GracefulShutdown;
break;
}
Ok(n) => n,
Err(e) => {
error!(
"{} failed to read. Err = {:?}, CTX={}",
self.name, e, self.tunnel_ctx
);
shutdown_reason = RelayShutdownReasons::ReadError;
break;
}
};
The relay policy can be configured like this:
relay_policy:
idle_timeout: 10s
min_rate_bpm: 1000
max_rate_bps: 10000
max_lifetime: 100s
max_total_payload: 100mb
So we can limit activity per connection with max_rate_bps
and detect idle clients with min_rate_bpm
(so they don’t consume system resources than can be utilized more productively). A connection lifetime and total traffic may be bounded as well.
It goes without saying that each failure mode needs to be tested. It’s straightforward to do that in Rust, in general, and with tokio-test
in particular:
#[tokio::test]
async fn test_timed_operation_timeout() {
let time_duration = 1;
let data = b"data on the wire";
let mut mock_connection: Mock = Builder::new()
.wait(Duration::from_secs(time_duration * 2))
.read(data)
.build();
let relay_policy: RelayPolicy = RelayPolicyBuilder::default()
.min_rate_bpm(1000)
.max_rate_bps(100_000)
.idle_timeout(Duration::from_secs(time_duration))
.build()
.unwrap();
let mut buf = [0; 1024];
let timed_future = relay_policy
.timed_operation(mock_connection.read(&mut buf))
.await;
assert!(timed_future.is_err());
}
The same goes for I/O errors:
#[tokio::test]
async fn test_timed_operation_failed_io() {
let mut mock_connection: Mock = Builder::new()
.read_error(Error::from(ErrorKind::BrokenPipe))
.build();
let relay_policy: RelayPolicy = RelayPolicyBuilder::default()
.min_rate_bpm(1000)
.max_rate_bps(100_000)
.idle_timeout(Duration::from_secs(5))
.build()
.unwrap();
let mut buf = [0; 1024];
let timed_future = relay_policy
.timed_operation(mock_connection.read(&mut buf))
.await;
assert!(timed_future.is_ok()); // no timeout
assert!(timed_future.unwrap().is_err()); // but io-error
}
I haven’t seen an application that failed only in ways anticipated by its developers. I’m not saying there are no such applications. Still, chances are, your app is going to encounter something you didn’t expect: data races, specific traffic patterns, dealing with traffic bursts, and legacy clients.
But, one of the most common types of failures is human failures, such as pushing bad code or configuration, which are inevitable in large projects. Anyway, we need to be able to deal with something we didn’t foresee. So we emit enough information that would allow us to detect failures and troubleshoot.
So we’d better log every error and important event with meaningful information and relevant context as well as statistics:
/// Stats after the relay is closed. Can be used for telemetry/monitoring.
#[derive(Builder, Clone, Debug, Serialize)]
pub struct RelayStats {
pub shutdown_reason: RelayShutdownReasons,
pub total_bytes: usize,
pub event_count: usize,
pub duration: Duration,
}
/// Statistics. No sensitive information.
#[derive(Serialize)]
pub struct TunnelStats {
tunnel_ctx: TunnelCtx,
result: EstablishTunnelResult,
upstream_stats: Option<RelayStats>,
downstream_stats: Option<RelayStats>,
}
Note: the tunnel_ctx: TunnelCtx
field, which can be used to correlate metric records with log messages:
Last but not least, we’d like to be able to run our tunnel in different modes with different parameters. Here’s where serde
and clap
become handy:
#[derive(Parser, Debug)]
#[clap(author, version, about, long_about = None)]
#[clap(propagate_version = true)]
struct Cli {
/// Configuration file.
#[clap(long)]
config: Option<String>,
/// Bind address, e.g. 0.0.0.0:8443.
#[clap(long)]
bind: String,
#[clap(subcommand)]
command: Commands,
}
#[derive(Subcommand, Debug)]
enum Commands {
Http(HttpOptions),
Https(HttpsOptions),
Tcp(TcpOptions),
}
In my opinion, clap
makes dealing with command line parameters pleasant. Extraordinarily expressive and easy to maintain.
Configuration files can be easily handled with serde-yaml
:
target_connection:
dns_cache_ttl: 60s
allowed_targets: "(?i)(wikipedia|rust-lang)\\.org:443$"
connect_timeout: 10s
relay_policy:
idle_timeout: 10s
min_rate_bpm: 1000
max_rate_bps: 10000
Which corresponds to Rust structs:
#[derive(Deserialize, Clone)]
pub struct TargetConnectionConfig {
#[serde(with = "humantime_serde")]
pub dns_cache_ttl: Duration,
#[serde(with = "serde_regex")]
pub allowed_targets: Regex,
#[serde(with = "humantime_serde")]
pub connect_timeout: Duration,
pub relay_policy: RelayPolicy,
}
#[derive(Builder, Deserialize, Clone)]
pub struct RelayPolicy {
#[serde(with = "humantime_serde")]
pub idle_timeout: Duration,
/// Min bytes-per-minute (bpm)
pub min_rate_bpm: u64,
// Max bytes-per-second (bps)
pub max_rate_bps: u64,
}
It doesn’t need any additional comments to make it readable and maintainable, which is beautiful.
As you can see from this quick overview, the Rust ecosystem already provides many building blocks, so you can focus on what you need to do rather than how. You didn’t see any memory/resources management or explicit thread safety (which often comes at the expense of concurrency) with impressive performance. Abstraction mechanisms are fantastic, so your code can be highly reusable. This task was a lot of fun, so I’ll try to take on the next challenge.
Original article sourced at: https://medium.com
1675339514
Hello Guys, I just completed my divorce with my cheating husband of 10 years with two beautiful kids when i got a solid evidence of his unfaithfulness on extra marital affairs and his infidelity lifestyle, And the various applications he used to hide chats and lot of secret on his cell phone. Then, i decided to hire (Henryclarkethicalhacker@gmail com), an hacker and a PI just to be 100% sure because i don’t want to confront him or take any kinda step without proof, fact and figures because that could leads to defamation… It was with this great, honest, professional and trustworthy Man i got to know my husband is a professional cheater and has been using this app to hide most of his chats. This great hacker helped me broke to into his cell phone activities and wired everything he does on his cell phone directly to my phone and i was able to monitor and track him directly from my phone remotely without him knowing, whatsapp, text, call, contact Henry via on +17736092741, or 1201-430-5865.
1656151740
Flutter Console Coverage Test
This small dart tools is used to generate Flutter Coverage Test report to console
Add a line like this to your package's pubspec.yaml (and run an implicit flutter pub get):
dev_dependencies:
test_cov_console: ^0.2.2
flutter pub get
Running "flutter pub get" in coverage... 0.5s
flutter test --coverage
00:02 +1: All tests passed!
flutter pub run test_cov_console
---------------------------------------------|---------|---------|---------|-------------------|
File |% Branch | % Funcs | % Lines | Uncovered Line #s |
---------------------------------------------|---------|---------|---------|-------------------|
lib/src/ | | | | |
print_cov.dart | 100.00 | 100.00 | 88.37 |...,149,205,206,207|
print_cov_constants.dart | 0.00 | 0.00 | 0.00 | no unit testing|
lib/ | | | | |
test_cov_console.dart | 0.00 | 0.00 | 0.00 | no unit testing|
---------------------------------------------|---------|---------|---------|-------------------|
All files with unit testing | 100.00 | 100.00 | 88.37 | |
---------------------------------------------|---------|---------|---------|-------------------|
If not given a FILE, "coverage/lcov.info" will be used.
-f, --file=<FILE> The target lcov.info file to be reported
-e, --exclude=<STRING1,STRING2,...> A list of contains string for files without unit testing
to be excluded from report
-l, --line It will print Lines & Uncovered Lines only
Branch & Functions coverage percentage will not be printed
-i, --ignore It will not print any file without unit testing
-m, --multi Report from multiple lcov.info files
-c, --csv Output to CSV file
-o, --output=<CSV-FILE> Full path of output CSV file
If not given, "coverage/test_cov_console.csv" will be used
-t, --total Print only the total coverage
Note: it will ignore all other option (if any), except -m
-p, --pass=<MINIMUM> Print only the whether total coverage is passed MINIMUM value or not
If the value >= MINIMUM, it will print PASSED, otherwise FAILED
Note: it will ignore all other option (if any), except -m
-h, --help Show this help
flutter pub run test_cov_console --file=coverage/lcov.info --exclude=_constants,_mock
---------------------------------------------|---------|---------|---------|-------------------|
File |% Branch | % Funcs | % Lines | Uncovered Line #s |
---------------------------------------------|---------|---------|---------|-------------------|
lib/src/ | | | | |
print_cov.dart | 100.00 | 100.00 | 88.37 |...,149,205,206,207|
lib/ | | | | |
test_cov_console.dart | 0.00 | 0.00 | 0.00 | no unit testing|
---------------------------------------------|---------|---------|---------|-------------------|
All files with unit testing | 100.00 | 100.00 | 88.37 | |
---------------------------------------------|---------|---------|---------|-------------------|
It support to run for multiple lcov.info files with the followings directory structures:
1. No root module
<root>/<module_a>
<root>/<module_a>/coverage/lcov.info
<root>/<module_a>/lib/src
<root>/<module_b>
<root>/<module_b>/coverage/lcov.info
<root>/<module_b>/lib/src
...
2. With root module
<root>/coverage/lcov.info
<root>/lib/src
<root>/<module_a>
<root>/<module_a>/coverage/lcov.info
<root>/<module_a>/lib/src
<root>/<module_b>
<root>/<module_b>/coverage/lcov.info
<root>/<module_b>/lib/src
...
You must run test_cov_console on <root> dir, and the report would be grouped by module, here is
the sample output for directory structure 'with root module':
flutter pub run test_cov_console --file=coverage/lcov.info --exclude=_constants,_mock --multi
---------------------------------------------|---------|---------|---------|-------------------|
File |% Branch | % Funcs | % Lines | Uncovered Line #s |
---------------------------------------------|---------|---------|---------|-------------------|
lib/src/ | | | | |
print_cov.dart | 100.00 | 100.00 | 88.37 |...,149,205,206,207|
lib/ | | | | |
test_cov_console.dart | 0.00 | 0.00 | 0.00 | no unit testing|
---------------------------------------------|---------|---------|---------|-------------------|
All files with unit testing | 100.00 | 100.00 | 88.37 | |
---------------------------------------------|---------|---------|---------|-------------------|
---------------------------------------------|---------|---------|---------|-------------------|
File - module_a - |% Branch | % Funcs | % Lines | Uncovered Line #s |
---------------------------------------------|---------|---------|---------|-------------------|
lib/src/ | | | | |
print_cov.dart | 100.00 | 100.00 | 88.37 |...,149,205,206,207|
lib/ | | | | |
test_cov_console.dart | 0.00 | 0.00 | 0.00 | no unit testing|
---------------------------------------------|---------|---------|---------|-------------------|
All files with unit testing | 100.00 | 100.00 | 88.37 | |
---------------------------------------------|---------|---------|---------|-------------------|
---------------------------------------------|---------|---------|---------|-------------------|
File - module_b - |% Branch | % Funcs | % Lines | Uncovered Line #s |
---------------------------------------------|---------|---------|---------|-------------------|
lib/src/ | | | | |
print_cov.dart | 100.00 | 100.00 | 88.37 |...,149,205,206,207|
lib/ | | | | |
test_cov_console.dart | 0.00 | 0.00 | 0.00 | no unit testing|
---------------------------------------------|---------|---------|---------|-------------------|
All files with unit testing | 100.00 | 100.00 | 88.37 | |
---------------------------------------------|---------|---------|---------|-------------------|
flutter pub run test_cov_console -c --output=coverage/test_coverage.csv
#### sample CSV output file:
File,% Branch,% Funcs,% Lines,Uncovered Line #s
lib/,,,,
test_cov_console.dart,0.00,0.00,0.00,no unit testing
lib/src/,,,,
parser.dart,100.00,100.00,97.22,"97"
parser_constants.dart,100.00,100.00,100.00,""
print_cov.dart,100.00,100.00,82.91,"29,49,51,52,171,174,177,180,183,184,185,186,187,188,279,324,325,387,388,389,390,391,392,393,394,395,398"
print_cov_constants.dart,0.00,0.00,0.00,no unit testing
All files with unit testing,100.00,100.00,86.07,""
You can install the package from the command line:
dart pub global activate test_cov_console
The package has the following executables:
$ test_cov_console
Run this command:
With Dart:
$ dart pub add test_cov_console
With Flutter:
$ flutter pub add test_cov_console
This will add a line like this to your package's pubspec.yaml (and run an implicit dart pub get
):
dependencies:
test_cov_console: ^0.2.2
Alternatively, your editor might support dart pub get
or flutter pub get
. Check the docs for your editor to learn more.
Now in your Dart code, you can use:
import 'package:test_cov_console/test_cov_console.dart';
example/lib/main.dart
import 'package:flutter/material.dart';
void main() {
runApp(MyApp());
}
class MyApp extends StatelessWidget {
// This widget is the root of your application.
@override
Widget build(BuildContext context) {
return MaterialApp(
title: 'Flutter Demo',
theme: ThemeData(
// This is the theme of your application.
//
// Try running your application with "flutter run". You'll see the
// application has a blue toolbar. Then, without quitting the app, try
// changing the primarySwatch below to Colors.green and then invoke
// "hot reload" (press "r" in the console where you ran "flutter run",
// or simply save your changes to "hot reload" in a Flutter IDE).
// Notice that the counter didn't reset back to zero; the application
// is not restarted.
primarySwatch: Colors.blue,
// This makes the visual density adapt to the platform that you run
// the app on. For desktop platforms, the controls will be smaller and
// closer together (more dense) than on mobile platforms.
visualDensity: VisualDensity.adaptivePlatformDensity,
),
home: MyHomePage(title: 'Flutter Demo Home Page'),
);
}
}
class MyHomePage extends StatefulWidget {
MyHomePage({Key? key, required this.title}) : super(key: key);
// This widget is the home page of your application. It is stateful, meaning
// that it has a State object (defined below) that contains fields that affect
// how it looks.
// This class is the configuration for the state. It holds the values (in this
// case the title) provided by the parent (in this case the App widget) and
// used by the build method of the State. Fields in a Widget subclass are
// always marked "final".
final String title;
@override
_MyHomePageState createState() => _MyHomePageState();
}
class _MyHomePageState extends State<MyHomePage> {
int _counter = 0;
void _incrementCounter() {
setState(() {
// This call to setState tells the Flutter framework that something has
// changed in this State, which causes it to rerun the build method below
// so that the display can reflect the updated values. If we changed
// _counter without calling setState(), then the build method would not be
// called again, and so nothing would appear to happen.
_counter++;
});
}
@override
Widget build(BuildContext context) {
// This method is rerun every time setState is called, for instance as done
// by the _incrementCounter method above.
//
// The Flutter framework has been optimized to make rerunning build methods
// fast, so that you can just rebuild anything that needs updating rather
// than having to individually change instances of widgets.
return Scaffold(
appBar: AppBar(
// Here we take the value from the MyHomePage object that was created by
// the App.build method, and use it to set our appbar title.
title: Text(widget.title),
),
body: Center(
// Center is a layout widget. It takes a single child and positions it
// in the middle of the parent.
child: Column(
// Column is also a layout widget. It takes a list of children and
// arranges them vertically. By default, it sizes itself to fit its
// children horizontally, and tries to be as tall as its parent.
//
// Invoke "debug painting" (press "p" in the console, choose the
// "Toggle Debug Paint" action from the Flutter Inspector in Android
// Studio, or the "Toggle Debug Paint" command in Visual Studio Code)
// to see the wireframe for each widget.
//
// Column has various properties to control how it sizes itself and
// how it positions its children. Here we use mainAxisAlignment to
// center the children vertically; the main axis here is the vertical
// axis because Columns are vertical (the cross axis would be
// horizontal).
mainAxisAlignment: MainAxisAlignment.center,
children: <Widget>[
Text(
'You have pushed the button this many times:',
),
Text(
'$_counter',
style: Theme.of(context).textTheme.headline4,
),
],
),
),
floatingActionButton: FloatingActionButton(
onPressed: _incrementCounter,
tooltip: 'Increment',
child: Icon(Icons.add),
), // This trailing comma makes auto-formatting nicer for build methods.
);
}
}
Author: DigitalKatalis
Source Code: https://github.com/DigitalKatalis/test_cov_console
License: BSD-3-Clause license
1643176207
Serde
*Serde is a framework for serializing and deserializing Rust data structures efficiently and generically.*
You may be looking for:
#[derive(Serialize, Deserialize)]
Click to show Cargo.toml. Run this code in the playground.
[dependencies]
# The core APIs, including the Serialize and Deserialize traits. Always
# required when using Serde. The "derive" feature is only required when
# using #[derive(Serialize, Deserialize)] to make Serde work with structs
# and enums defined in your crate.
serde = { version = "1.0", features = ["derive"] }
# Each data format lives in its own crate; the sample code below uses JSON
# but you may be using a different one.
serde_json = "1.0"
use serde::{Serialize, Deserialize};
#[derive(Serialize, Deserialize, Debug)]
struct Point {
x: i32,
y: i32,
}
fn main() {
let point = Point { x: 1, y: 2 };
// Convert the Point to a JSON string.
let serialized = serde_json::to_string(&point).unwrap();
// Prints serialized = {"x":1,"y":2}
println!("serialized = {}", serialized);
// Convert the JSON string back to a Point.
let deserialized: Point = serde_json::from_str(&serialized).unwrap();
// Prints deserialized = Point { x: 1, y: 2 }
println!("deserialized = {:?}", deserialized);
}
Serde is one of the most widely used Rust libraries so any place that Rustaceans congregate will be able to help you out. For chat, consider trying the #rust-questions or #rust-beginners channels of the unofficial community Discord (invite: https://discord.gg/rust-lang-community), the #rust-usage or #beginners channels of the official Rust Project Discord (invite: https://discord.gg/rust-lang), or the #general stream in Zulip. For asynchronous, consider the [rust] tag on StackOverflow, the /r/rust subreddit which has a pinned weekly easy questions post, or the Rust Discourse forum. It's acceptable to file a support issue in this repo but they tend not to get as many eyes as any of the above and may get closed without a response after some time.
Download Details:
Author: serde-rs
Source Code: https://github.com/serde-rs/serde
License: View license
1636360749
The std
library provides many custom types which expands drastically on the primitives
. Some of these include:
String
s like: "hello world"
[1, 2, 3]
Option<i32>
Result<i32, i32>
Box<i32>
All values in Rust are stack allocated by default. Values can be boxed (allocated on the heap) by creating a Box<T>
. A box is a smart pointer to a heap allocated value of type T
. When a box goes out of scope, its destructor is called, the inner object is destroyed, and the memory on the heap is freed.
Boxed values can be dereferenced using the *
operator; this removes one layer of indirection.
use std::mem;
#[allow(dead_code)]
#[derive(Debug, Clone, Copy)]
struct Point {
x: f64,
y: f64,
}
// A Rectangle can be specified by where its top left and bottom right
// corners are in space
#[allow(dead_code)]
struct Rectangle {
top_left: Point,
bottom_right: Point,
}
fn origin() -> Point {
Point { x: 0.0, y: 0.0 }
}
fn boxed_origin() -> Box<Point> {
// Allocate this point on the heap, and return a pointer to it
Box::new(Point { x: 0.0, y: 0.0 })
}
fn main() {
// (all the type annotations are superfluous)
// Stack allocated variables
let point: Point = origin();
let rectangle: Rectangle = Rectangle {
top_left: origin(),
bottom_right: Point { x: 3.0, y: -4.0 }
};
// Heap allocated rectangle
let boxed_rectangle: Box<Rectangle> = Box::new(Rectangle {
top_left: origin(),
bottom_right: Point { x: 3.0, y: -4.0 },
});
// The output of functions can be boxed
let boxed_point: Box<Point> = Box::new(origin());
// Double indirection
let box_in_a_box: Box<Box<Point>> = Box::new(boxed_origin());
println!("Point occupies {} bytes on the stack",
mem::size_of_val(&point));
println!("Rectangle occupies {} bytes on the stack",
mem::size_of_val(&rectangle));
// box size == pointer size
println!("Boxed point occupies {} bytes on the stack",
mem::size_of_val(&boxed_point));
println!("Boxed rectangle occupies {} bytes on the stack",
mem::size_of_val(&boxed_rectangle));
println!("Boxed box occupies {} bytes on the stack",
mem::size_of_val(&box_in_a_box));
// Copy the data contained in `boxed_point` into `unboxed_point`
let unboxed_point: Point = *boxed_point;
println!("Unboxed point occupies {} bytes on the stack",
mem::size_of_val(&unboxed_point));
}
Vectors are re-sizable arrays. Like slices, their size is not known at compile time, but they can grow or shrink at any time. A vector is represented using 3 parameters:
The capacity indicates how much memory is reserved for the vector. The vector can grow as long as the length is smaller than the capacity. When this threshold needs to be surpassed, the vector is reallocated with a larger capacity.
fn main() {
// Iterators can be collected into vectors
let collected_iterator: Vec<i32> = (0..10).collect();
println!("Collected (0..10) into: {:?}", collected_iterator);
// The `vec!` macro can be used to initialize a vector
let mut xs = vec![1i32, 2, 3];
println!("Initial vector: {:?}", xs);
// Insert new element at the end of the vector
println!("Push 4 into the vector");
xs.push(4);
println!("Vector: {:?}", xs);
// Error! Immutable vectors can't grow
collected_iterator.push(0);
// FIXME ^ Comment out this line
// The `len` method yields the number of elements currently stored in a vector
println!("Vector length: {}", xs.len());
// Indexing is done using the square brackets (indexing starts at 0)
println!("Second element: {}", xs[1]);
// `pop` removes the last element from the vector and returns it
println!("Pop last element: {:?}", xs.pop());
// Out of bounds indexing yields a panic
println!("Fourth element: {}", xs[3]);
// FIXME ^ Comment out this line
// `Vector`s can be easily iterated over
println!("Contents of xs:");
for x in xs.iter() {
println!("> {}", x);
}
// A `Vector` can also be iterated over while the iteration
// count is enumerated in a separate variable (`i`)
for (i, x) in xs.iter().enumerate() {
println!("In position {} we have value {}", i, x);
}
// Thanks to `iter_mut`, mutable `Vector`s can also be iterated
// over in a way that allows modifying each value
for x in xs.iter_mut() {
*x *= 3;
}
println!("Updated vector: {:?}", xs);
}
More Vec
methods can be found under the std::vec module
There are two types of strings in Rust: String
and &str
.
A String
is stored as a vector of bytes (Vec<u8>
), but guaranteed to always be a valid UTF-8 sequence. String
is heap allocated, growable and not null terminated.
&str
is a slice (&[u8]
) that always points to a valid UTF-8 sequence, and can be used to view into a String
, just like &[T]
is a view into Vec<T>
.
fn main() {
// (all the type annotations are superfluous)
// A reference to a string allocated in read only memory
let pangram: &'static str = "the quick brown fox jumps over the lazy dog";
println!("Pangram: {}", pangram);
// Iterate over words in reverse, no new string is allocated
println!("Words in reverse");
for word in pangram.split_whitespace().rev() {
println!("> {}", word);
}
// Copy chars into a vector, sort and remove duplicates
let mut chars: Vec<char> = pangram.chars().collect();
chars.sort();
chars.dedup();
// Create an empty and growable `String`
let mut string = String::new();
for c in chars {
// Insert a char at the end of string
string.push(c);
// Insert a string at the end of string
string.push_str(", ");
}
// The trimmed string is a slice to the original string, hence no new
// allocation is performed
let chars_to_trim: &[char] = &[' ', ','];
let trimmed_str: &str = string.trim_matches(chars_to_trim);
println!("Used characters: {}", trimmed_str);
// Heap allocate a string
let alice = String::from("I like dogs");
// Allocate new memory and store the modified string there
let bob: String = alice.replace("dog", "cat");
println!("Alice says: {}", alice);
println!("Bob says: {}", bob);
}
More str
/String
methods can be found under the std::str and std::string modules
There are multiple ways to write string literals with special characters in them. All result in a similar &str
so it's best to use the form that is the most convenient to write. Similarly there are multiple ways to write byte string literals, which all result in &[u8; N]
.
Generally special characters are escaped with a backslash character: \
. This way you can add any character to your string, even unprintable ones and ones that you don't know how to type. If you want a literal backslash, escape it with another one: \\
String or character literal delimiters occuring within a literal must be escaped: "\""
, '\''
.
fn main() {
// You can use escapes to write bytes by their hexadecimal values...
let byte_escape = "I'm writing \x52\x75\x73\x74!";
println!("What are you doing\x3F (\\x3F means ?) {}", byte_escape);
// ...or Unicode code points.
let unicode_codepoint = "\u{211D}";
let character_name = "\"DOUBLE-STRUCK CAPITAL R\"";
println!("Unicode character {} (U+211D) is called {}",
unicode_codepoint, character_name );
let long_string = "String literals
can span multiple lines.
The linebreak and indentation here ->\
<- can be escaped too!";
println!("{}", long_string);
}
Sometimes there are just too many characters that need to be escaped or it's just much more convenient to write a string out as-is. This is where raw string literals come into play.
fn main() {
let raw_str = r"Escapes don't work here: \x3F \u{211D}";
println!("{}", raw_str);
// If you need quotes in a raw string, add a pair of #s
let quotes = r#"And then I said: "There is no escape!""#;
println!("{}", quotes);
// If you need "# in your string, just use more #s in the delimiter.
// There is no limit for the number of #s you can use.
let longer_delimiter = r###"A string with "# in it. And even "##!"###;
println!("{}", longer_delimiter);
}
Want a string that's not UTF-8? (Remember, str
and String
must be valid UTF-8). Or maybe you want an array of bytes that's mostly text? Byte strings to the rescue!
use std::str;
fn main() {
// Note that this is not actually a `&str`
let bytestring: &[u8; 21] = b"this is a byte string";
// Byte arrays don't have the `Display` trait, so printing them is a bit limited
println!("A byte string: {:?}", bytestring);
// Byte strings can have byte escapes...
let escaped = b"\x52\x75\x73\x74 as bytes";
// ...but no unicode escapes
// let escaped = b"\u{211D} is not allowed";
println!("Some escaped bytes: {:?}", escaped);
// Raw byte strings work just like raw strings
let raw_bytestring = br"\u{211D} is not escaped here";
println!("{:?}", raw_bytestring);
// Converting a byte array to `str` can fail
if let Ok(my_str) = str::from_utf8(raw_bytestring) {
println!("And the same as text: '{}'", my_str);
}
let _quotes = br#"You can also use "fancier" formatting, \
like with normal raw strings"#;
// Byte strings don't have to be UTF-8
let shift_jis = b"\x82\xe6\x82\xa8\x82\xb1\x82\xbb"; // "ようこそ" in SHIFT-JIS
// But then they can't always be converted to `str`
match str::from_utf8(shift_jis) {
Ok(my_str) => println!("Conversion successful: '{}'", my_str),
Err(e) => println!("Conversion failed: {:?}", e),
};
}
For conversions between character encodings check out the encoding crate.
A more detailed listing of the ways to write string literals and escape characters is given in the 'Tokens' chapter of the Rust Reference.
Option
Sometimes it's desirable to catch the failure of some parts of a program instead of calling panic!
; this can be accomplished using the Option
enum.
The Option<T>
enum has two variants:
None
, to indicate failure or lack of value, andSome(value)
, a tuple struct that wraps a value
with type T
.// An integer division that doesn't `panic!`
fn checked_division(dividend: i32, divisor: i32) -> Option<i32> {
if divisor == 0 {
// Failure is represented as the `None` variant
None
} else {
// Result is wrapped in a `Some` variant
Some(dividend / divisor)
}
}
// This function handles a division that may not succeed
fn try_division(dividend: i32, divisor: i32) {
// `Option` values can be pattern matched, just like other enums
match checked_division(dividend, divisor) {
None => println!("{} / {} failed!", dividend, divisor),
Some(quotient) => {
println!("{} / {} = {}", dividend, divisor, quotient)
},
}
}
fn main() {
try_division(4, 2);
try_division(1, 0);
// Binding `None` to a variable needs to be type annotated
let none: Option<i32> = None;
let _equivalent_none = None::<i32>;
let optional_float = Some(0f32);
// Unwrapping a `Some` variant will extract the value wrapped.
println!("{:?} unwraps to {:?}", optional_float, optional_float.unwrap());
// Unwrapping a `None` variant will `panic!`
println!("{:?} unwraps to {:?}", none, none.unwrap());
}
Result
We've seen that the Option
enum can be used as a return value from functions that may fail, where None
can be returned to indicate failure. However, sometimes it is important to express why an operation failed. To do this we have the Result
enum.
The Result<T, E>
enum has two variants:
Ok(value)
which indicates that the operation succeeded, and wraps the value
returned by the operation. (value
has type T
)Err(why)
, which indicates that the operation failed, and wraps why
, which (hopefully) explains the cause of the failure. (why
has type E
)mod checked {
// Mathematical "errors" we want to catch
#[derive(Debug)]
pub enum MathError {
DivisionByZero,
NonPositiveLogarithm,
NegativeSquareRoot,
}
pub type MathResult = Result<f64, MathError>;
pub fn div(x: f64, y: f64) -> MathResult {
if y == 0.0 {
// This operation would `fail`, instead let's return the reason of
// the failure wrapped in `Err`
Err(MathError::DivisionByZero)
} else {
// This operation is valid, return the result wrapped in `Ok`
Ok(x / y)
}
}
pub fn sqrt(x: f64) -> MathResult {
if x < 0.0 {
Err(MathError::NegativeSquareRoot)
} else {
Ok(x.sqrt())
}
}
pub fn ln(x: f64) -> MathResult {
if x <= 0.0 {
Err(MathError::NonPositiveLogarithm)
} else {
Ok(x.ln())
}
}
}
// `op(x, y)` === `sqrt(ln(x / y))`
fn op(x: f64, y: f64) -> f64 {
// This is a three level match pyramid!
match checked::div(x, y) {
Err(why) => panic!("{:?}", why),
Ok(ratio) => match checked::ln(ratio) {
Err(why) => panic!("{:?}", why),
Ok(ln) => match checked::sqrt(ln) {
Err(why) => panic!("{:?}", why),
Ok(sqrt) => sqrt,
},
},
}
}
fn main() {
// Will this fail?
println!("{}", op(1.0, 10.0));
}
?
Chaining results using match can get pretty untidy; luckily, the ?
operator can be used to make things pretty again. ?
is used at the end of an expression returning a Result
, and is equivalent to a match expression, where the Err(err)
branch expands to an early Err(From::from(err))
, and the Ok(ok)
branch expands to an ok
expression.
mod checked {
#[derive(Debug)]
enum MathError {
DivisionByZero,
NonPositiveLogarithm,
NegativeSquareRoot,
}
type MathResult = Result<f64, MathError>;
fn div(x: f64, y: f64) -> MathResult {
if y == 0.0 {
Err(MathError::DivisionByZero)
} else {
Ok(x / y)
}
}
fn sqrt(x: f64) -> MathResult {
if x < 0.0 {
Err(MathError::NegativeSquareRoot)
} else {
Ok(x.sqrt())
}
}
fn ln(x: f64) -> MathResult {
if x <= 0.0 {
Err(MathError::NonPositiveLogarithm)
} else {
Ok(x.ln())
}
}
// Intermediate function
fn op_(x: f64, y: f64) -> MathResult {
// if `div` "fails", then `DivisionByZero` will be `return`ed
let ratio = div(x, y)?;
// if `ln` "fails", then `NonPositiveLogarithm` will be `return`ed
let ln = ln(ratio)?;
sqrt(ln)
}
pub fn op(x: f64, y: f64) {
match op_(x, y) {
Err(why) => panic!("{}", match why {
MathError::NonPositiveLogarithm
=> "logarithm of non-positive number",
MathError::DivisionByZero
=> "division by zero",
MathError::NegativeSquareRoot
=> "square root of negative number",
}),
Ok(value) => println!("{}", value),
}
}
}
fn main() {
checked::op(1.0, 10.0);
}
Be sure to check the documentation, as there are many methods to map/compose Result
.
panic!
The panic!
macro can be used to generate a panic and start unwinding its stack. While unwinding, the runtime will take care of freeing all the resources owned by the thread by calling the destructor of all its objects.
Since we are dealing with programs with only one thread, panic!
will cause the program to report the panic message and exit.
// Re-implementation of integer division (/)
fn division(dividend: i32, divisor: i32) -> i32 {
if divisor == 0 {
// Division by zero triggers a panic
panic!("division by zero");
} else {
dividend / divisor
}
}
// The `main` task
fn main() {
// Heap allocated integer
let _x = Box::new(0i32);
// This operation will trigger a task failure
division(3, 0);
println!("This point won't be reached!");
// `_x` should get destroyed at this point
}
Let's check that panic!
doesn't leak memory.
$ rustc panic.rs && valgrind ./panic
==4401== Memcheck, a memory error detector
==4401== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==4401== Using Valgrind-3.10.0.SVN and LibVEX; rerun with -h for copyright info
==4401== Command: ./panic
==4401==
thread '<main>' panicked at 'division by zero', panic.rs:5
==4401==
==4401== HEAP SUMMARY:
==4401== in use at exit: 0 bytes in 0 blocks
==4401== total heap usage: 18 allocs, 18 frees, 1,648 bytes allocated
==4401==
==4401== All heap blocks were freed -- no leaks are possible
==4401==
==4401== For counts of detected and suppressed errors, rerun with: -v
==4401== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
Where vectors store values by an integer index, HashMap
s store values by key. HashMap
keys can be booleans, integers, strings, or any other type that implements the Eq
and Hash
traits. More on this in the next section.
Like vectors, HashMap
s are growable, but HashMaps can also shrink themselves when they have excess space. You can create a HashMap with a certain starting capacity using HashMap::with_capacity(uint)
, or use HashMap::new()
to get a HashMap with a default initial capacity (recommended).
use std::collections::HashMap;
fn call(number: &str) -> &str {
match number {
"798-1364" => "We're sorry, the call cannot be completed as dialed.
Please hang up and try again.",
"645-7689" => "Hello, this is Mr. Awesome's Pizza. My name is Fred.
What can I get for you today?",
_ => "Hi! Who is this again?"
}
}
fn main() {
let mut contacts = HashMap::new();
contacts.insert("Daniel", "798-1364");
contacts.insert("Ashley", "645-7689");
contacts.insert("Katie", "435-8291");
contacts.insert("Robert", "956-1745");
// Takes a reference and returns Option<&V>
match contacts.get(&"Daniel") {
Some(&number) => println!("Calling Daniel: {}", call(number)),
_ => println!("Don't have Daniel's number."),
}
// `HashMap::insert()` returns `None`
// if the inserted value is new, `Some(value)` otherwise
contacts.insert("Daniel", "164-6743");
match contacts.get(&"Ashley") {
Some(&number) => println!("Calling Ashley: {}", call(number)),
_ => println!("Don't have Ashley's number."),
}
contacts.remove(&"Ashley");
// `HashMap::iter()` returns an iterator that yields
// (&'a key, &'a value) pairs in arbitrary order.
for (contact, &number) in contacts.iter() {
println!("Calling {}: {}", contact, call(number));
}
}
For more information on how hashing and hash maps (sometimes called hash tables) work, have a look at Hash Table Wikipedia
Any type that implements the Eq
and Hash
traits can be a key in HashMap
. This includes:
bool
(though not very useful since there is only two possible keys)int
, uint
, and all variations thereofString
and &str
(protip: you can have a HashMap
keyed by String
and call .get()
with an &str
)Note that f32
and f64
do not implement Hash
, likely because floating-point precision errors would make using them as hashmap keys horribly error-prone.
All collection classes implement Eq
and Hash
if their contained type also respectively implements Eq
and Hash
. For example, Vec<T>
will implement Hash
if T
implements Hash
.
You can easily implement Eq
and Hash
for a custom type with just one line: #[derive(PartialEq, Eq, Hash)]
The compiler will do the rest. If you want more control over the details, you can implement Eq
and/or Hash
yourself. This guide will not cover the specifics of implementing Hash
.
To play around with using a struct
in HashMap
, let's try making a very simple user logon system:
use std::collections::HashMap;
// Eq requires that you derive PartialEq on the type.
#[derive(PartialEq, Eq, Hash)]
struct Account<'a>{
username: &'a str,
password: &'a str,
}
struct AccountInfo<'a>{
name: &'a str,
email: &'a str,
}
type Accounts<'a> = HashMap<Account<'a>, AccountInfo<'a>>;
fn try_logon<'a>(accounts: &Accounts<'a>,
username: &'a str, password: &'a str){
println!("Username: {}", username);
println!("Password: {}", password);
println!("Attempting logon...");
let logon = Account {
username,
password,
};
match accounts.get(&logon) {
Some(account_info) => {
println!("Successful logon!");
println!("Name: {}", account_info.name);
println!("Email: {}", account_info.email);
},
_ => println!("Login failed!"),
}
}
fn main(){
let mut accounts: Accounts = HashMap::new();
let account = Account {
username: "j.everyman",
password: "password123",
};
let account_info = AccountInfo {
name: "John Everyman",
email: "j.everyman@email.com",
};
accounts.insert(account, account_info);
try_logon(&accounts, "j.everyman", "psasword123");
try_logon(&accounts, "j.everyman", "password123");
}
Consider a HashSet
as a HashMap
where we just care about the keys ( HashSet<T>
is, in actuality, just a wrapper around HashMap<T, ()>
).
"What's the point of that?" you ask. "I could just store the keys in a Vec
."
A HashSet
's unique feature is that it is guaranteed to not have duplicate elements. That's the contract that any set collection fulfills. HashSet
is just one implementation. (see also: BTreeSet
)
If you insert a value that is already present in the HashSet
, (i.e. the new value is equal to the existing and they both have the same hash), then the new value will replace the old.
This is great for when you never want more than one of something, or when you want to know if you've already got something.
But sets can do more than that.
Sets have 4 primary operations (all of the following calls return an iterator):
union
: get all the unique elements in both sets.
difference
: get all the elements that are in the first set but not the second.
intersection
: get all the elements that are only in both sets.
symmetric_difference
: get all the elements that are in one set or the other, but not both.
Try all of these in the following example:
use std::collections::HashSet;
fn main() {
let mut a: HashSet<i32> = vec![1i32, 2, 3].into_iter().collect();
let mut b: HashSet<i32> = vec![2i32, 3, 4].into_iter().collect();
assert!(a.insert(4));
assert!(a.contains(&4));
// `HashSet::insert()` returns false if
// there was a value already present.
assert!(b.insert(4), "Value 4 is already in set B!");
// FIXME ^ Comment out this line
b.insert(5);
// If a collection's element type implements `Debug`,
// then the collection implements `Debug`.
// It usually prints its elements in the format `[elem1, elem2, ...]`
println!("A: {:?}", a);
println!("B: {:?}", b);
// Print [1, 2, 3, 4, 5] in arbitrary order
println!("Union: {:?}", a.union(&b).collect::<Vec<&i32>>());
// This should print [1]
println!("Difference: {:?}", a.difference(&b).collect::<Vec<&i32>>());
// Print [2, 3, 4] in arbitrary order.
println!("Intersection: {:?}", a.intersection(&b).collect::<Vec<&i32>>());
// Print [1, 5]
println!("Symmetric Difference: {:?}",
a.symmetric_difference(&b).collect::<Vec<&i32>>());
}
(Examples are adapted from the documentation.)
Rc
When multiple ownership is needed, Rc
(Reference Counting) can be used. Rc
keeps track of the number of the references which means the number of owners of the value wrapped inside an Rc
.
Reference count of an Rc
increases by 1 whenever an Rc
is cloned, and decreases by 1 whenever one cloned Rc
is dropped out of the scope. When an Rc
's reference count becomes zero, which means there are no owners remained, both the Rc
and the value are all dropped.
Cloning an Rc
never performs a deep copy. Cloning creates just another pointer to the wrapped value, and increments the count.
use std::rc::Rc;
fn main() {
let rc_examples = "Rc examples".to_string();
{
println!("--- rc_a is created ---");
let rc_a: Rc<String> = Rc::new(rc_examples);
println!("Reference Count of rc_a: {}", Rc::strong_count(&rc_a));
{
println!("--- rc_a is cloned to rc_b ---");
let rc_b: Rc<String> = Rc::clone(&rc_a);
println!("Reference Count of rc_b: {}", Rc::strong_count(&rc_b));
println!("Reference Count of rc_a: {}", Rc::strong_count(&rc_a));
// Two `Rc`s are equal if their inner values are equal
println!("rc_a and rc_b are equal: {}", rc_a.eq(&rc_b));
// We can use methods of a value directly
println!("Length of the value inside rc_a: {}", rc_a.len());
println!("Value of rc_b: {}", rc_b);
println!("--- rc_b is dropped out of scope ---");
}
println!("Reference Count of rc_a: {}", Rc::strong_count(&rc_a));
println!("--- rc_a is dropped out of scope ---");
}
// Error! `rc_examples` already moved into `rc_a`
// And when `rc_a` is dropped, `rc_examples` is dropped together
// println!("rc_examples: {}", rc_examples);
// TODO ^ Try uncommenting this line
}
When shared ownership between threads is needed, Arc
(Atomic Reference Counted) can be used. This struct, via the Clone
implementation can create a reference pointer for the location of a value in the memory heap while increasing the reference counter. As it shares ownership between threads, when the last reference pointer to a value is out of scope, the variable is dropped.
fn main() {
use std::sync::Arc;
use std::thread;
// This variable declaration is where its value is specified.
let apple = Arc::new("the same apple");
for _ in 0..10 {
// Here there is no value specification as it is a pointer to a reference
// in the memory heap.
let apple = Arc::clone(&apple);
thread::spawn(move || {
// As Arc was used, threads can be spawned using the value allocated
// in the Arc variable pointer's location.
println!("{:?}", apple);
});
}
}
Original article source at https://doc.rust-lang.org
#rust #programming #developer
1640144506
In this video we're taking a look at the String, &String and &str types in Rust!
Exercise solutions: https://github.com/PascalPrecht/rustlings/commits/solutions
---
0:00 Intro
0:09 Exercise 1
4:47 Exercise 2
10:38 Outro
There are two types of strings in Rust: String
and &str
.
A String
is stored as a vector of bytes (Vec<u8>
), but guaranteed to always be a valid UTF-8 sequence. String
is heap allocated, growable and not null terminated.
&str
is a slice (&[u8]
) that always points to a valid UTF-8 sequence, and can be used to view into a String
, just like &[T]
is a view into Vec<T>
.
fn main() {
// (all the type annotations are superfluous)
// A reference to a string allocated in read only memory
let pangram: &'static str = "the quick brown fox jumps over the lazy dog";
println!("Pangram: {}", pangram);
// Iterate over words in reverse, no new string is allocated
println!("Words in reverse");
for word in pangram.split_whitespace().rev() {
println!("> {}", word);
}
// Copy chars into a vector, sort and remove duplicates
let mut chars: Vec<char> = pangram.chars().collect();
chars.sort();
chars.dedup();
// Create an empty and growable `String`
let mut string = String::new();
for c in chars {
// Insert a char at the end of string
string.push(c);
// Insert a string at the end of string
string.push_str(", ");
}
// The trimmed string is a slice to the original string, hence no new
// allocation is performed
let chars_to_trim: &[char] = &[' ', ','];
let trimmed_str: &str = string.trim_matches(chars_to_trim);
println!("Used characters: {}", trimmed_str);
// Heap allocate a string
let alice = String::from("I like dogs");
// Allocate new memory and store the modified string there
let bob: String = alice.replace("dog", "cat");
println!("Alice says: {}", alice);
println!("Bob says: {}", bob);
}
More str
/String
methods can be found under the std::str and std::string modules
There are multiple ways to write string literals with special characters in them. All result in a similar &str
so it's best to use the form that is the most convenient to write. Similarly there are multiple ways to write byte string literals, which all result in &[u8; N]
.
Generally special characters are escaped with a backslash character: \
. This way you can add any character to your string, even unprintable ones and ones that you don't know how to type. If you want a literal backslash, escape it with another one: \\
String or character literal delimiters occuring within a literal must be escaped: "\""
, '\''
.
fn main() {
// You can use escapes to write bytes by their hexadecimal values...
let byte_escape = "I'm writing \x52\x75\x73\x74!";
println!("What are you doing\x3F (\\x3F means ?) {}", byte_escape);
// ...or Unicode code points.
let unicode_codepoint = "\u{211D}";
let character_name = "\"DOUBLE-STRUCK CAPITAL R\"";
println!("Unicode character {} (U+211D) is called {}",
unicode_codepoint, character_name );
let long_string = "String literals
can span multiple lines.
The linebreak and indentation here ->\
<- can be escaped too!";
println!("{}", long_string);
}
Sometimes there are just too many characters that need to be escaped or it's just much more convenient to write a string out as-is. This is where raw string literals come into play.
fn main() {
let raw_str = r"Escapes don't work here: \x3F \u{211D}";
println!("{}", raw_str);
// If you need quotes in a raw string, add a pair of #s
let quotes = r#"And then I said: "There is no escape!""#;
println!("{}", quotes);
// If you need "# in your string, just use more #s in the delimiter.
// There is no limit for the number of #s you can use.
let longer_delimiter = r###"A string with "# in it. And even "##!"###;
println!("{}", longer_delimiter);
}
Want a string that's not UTF-8? (Remember, str
and String
must be valid UTF-8). Or maybe you want an array of bytes that's mostly text? Byte strings to the rescue!
use std::str;
fn main() {
// Note that this is not actually a `&str`
let bytestring: &[u8; 21] = b"this is a byte string";
// Byte arrays don't have the `Display` trait, so printing them is a bit limited
println!("A byte string: {:?}", bytestring);
// Byte strings can have byte escapes...
let escaped = b"\x52\x75\x73\x74 as bytes";
// ...but no unicode escapes
// let escaped = b"\u{211D} is not allowed";
println!("Some escaped bytes: {:?}", escaped);
// Raw byte strings work just like raw strings
let raw_bytestring = br"\u{211D} is not escaped here";
println!("{:?}", raw_bytestring);
// Converting a byte array to `str` can fail
if let Ok(my_str) = str::from_utf8(raw_bytestring) {
println!("And the same as text: '{}'", my_str);
}
let _quotes = br#"You can also use "fancier" formatting, \
like with normal raw strings"#;
// Byte strings don't have to be UTF-8
let shift_jis = b"\x82\xe6\x82\xa8\x82\xb1\x82\xbb"; // "ようこそ" in SHIFT-JIS
// But then they can't always be converted to `str`
match str::from_utf8(shift_jis) {
Ok(my_str) => println!("Conversion successful: '{}'", my_str),
Err(e) => println!("Conversion failed: {:?}", e),
};
}
For conversions between character encodings check out the encoding crate.
A more detailed listing of the ways to write string literals and escape characters is given in the 'Tokens' chapter of the Rust Reference.
#rust #programming #developer
1675337640
Learn how to write performant and safe apps quickly in Rust. This post guides you through designing and implementing an HTTP Tunnel, and covers the basics of creating robust, scalable, and observable applications.
About a year ago, I started to learn Rust. The first two weeks were quite painful. Nothing compiled; I didn’t know how to do basic operations; I couldn’t make a simple program run. But step by step, I started to understand what the compiler wanted. Even more, I realized that it forces the right thinking and correct behavior.
Yes, sometimes, you have to write seemingly redundant constructs. But it’s better not to compile a correct program than to compile an incorrect one. This makes making mistakes more difficult.
Soon after, I became more or less productive and finally could do what I wanted. Well, most of the time.
Out of curiosity, I decided to take on a slightly more complex challenge: implement an HTTP Tunnel in Rust. It turned out to be surprisingly easy to do and took about a day, which is quite impressive. I stitched together tokio, clap, serde, and several other very useful crates. Let me share the knowledge I gained during this exciting challenge and elaborate on why I organized the app this way. I hope you’ll enjoy it.
Simply put, it’s a lightweight VPN that you can set up with your browser so your internet provider cannot block or track your activity, and web servers won’t see your IP address.
If you’d like, you can test it with your browser locally, e.g., with Firefox, (otherwise, just skip this section for now).
$ cargo install http-tunnel
$ http-tunnel --bind 0.0.0.0:8080 http
You can also check the HTTP-tunnel GitHub repository for the build/installation instructions.
Now, you can go to your browser and set the HTTP Proxy
to localhost:8080
. For instance, in Firefox, search for proxy
in the preferences section:
Find the proxy settings, specify it for HTTP Proxy
, and check it for HTTPS
:
Set the proxy to just built http_tunnel
. You can visit several web pages and check the ./logs/application.log
file—all your traffic was going via the tunnel. For example:
Now, let’s walk through the process from the beginning.
Each application starts with a design, which means we need to define the following:
We need to follow the specification outlined in the here:
Negotiate a target with an HTTP CONNECT
request. For example, if the client wants to create a tunnel to Wikipedia’s website, the request will look like this:
CONNECT www.wikipedia.org:443 HTTP/1.1
...
Followed by a response like below:
HTTP/1.1 200 OK
After this point, just relay TCP traffic both ways until one of the sides closes it or an I/O error happens.
The HTTP Tunnel should work for both HTTP
and HTTPS
.
We also should be able to manage access/block targets (e.g., to block-list trackers).
The service shouldn’t log any information that identifies users.
It should have high throughput and low latency (it should be unnoticeable for users and relatively cheap to run).
Ideally, we want it to be resilient to traffic spikes, provide noisy neighbor isolation, and resist basic DDoS attacks.
Error messaging should be developer-friendly. We want the system to be observable to troubleshoot and tune it in production at a massive scale.
When designing components, we need to break down the app into a set of responsibilities. First, let’s see what our flow diagram looks like:
To implement this, we can introduce the following four main components:
HTTP CONNECT
negotiatorTCP/TLS acceptor
When we roughly know how to organize the app, it’s time to decide which dependencies we should use. For Rust, the best I/O library I know is tokio. In the tokio
family, there are many libraries including tokio-tls
, which makes things much simpler. So the TCP acceptor code would look like this:
let mut tcp_listener = TcpListener::bind(&proxy_configuration.bind_address)
.await
.map_err(|e| {
error!(
"Error binding address {}: {}",
&proxy_configuration.bind_address, e
);
e
})?;
And then the whole acceptor loop + launching asynchronous connection handlers would be:
loop {
// Asynchronously wait for an inbound socket.
let socket = tcp_listener.accept().await;
let dns_resolver_ref = dns_resolver.clone();
match socket {
Ok((stream, _)) => {
let config = config.clone();
// handle accepted connections asynchronously
tokio::spawn(async move { tunnel_stream(&config, stream, dns_resolver_ref).await });
}
Err(e) => error!("Failed TCP handshake {}", e),
}
}
Let’s break down what’s happening here. We accept a connection. If the operation was successful, use tokio::spawn
to create a new task that will handle that connection. Memory/thread-safety management happens behind the scenes. Handling futures is hidden by the async/await
syntax sugar.
However, there is one question. TcpStream
and TlsStream
are different objects, but handling both is precisely the same. Can we reuse the same code? In Rust, abstraction is achieved via Traits
, which are super handy:
/// Tunnel via a client connection.
async fn tunnel_stream<C: AsyncRead + AsyncWrite + Send + Unpin + 'static>(
config: &ProxyConfiguration,
client_connection: C,
dns_resolver: DnsResolver,
) -> io::Result<()> {...}
The stream must implement:
AsyncRead /Write
: Allows us to read/write it asynchronously.Send
: To be able to send between threads.Unpin
: To be moveable (otherwise, we won’t be able to do async move
and tokio::spawn
to create an async
task).'static
: To denote that it may live until the application shutdown and doesn’t depend on any other object’s destruction.Which our TCP/TLS
streams exactly are. However, now we can see that it doesn’t have to be TCP/TLS
streams. This code would work for UDP
, QUIC
, or ICMP
. For example, it can wrap any protocol within any other protocol or itself.
In other words, this code is reusable, extendable, and ready for migration, which happens sooner or later.
HTTP connect negotiator and target connector
Let’s pause for a second and think at a higher level. What if we can abstract from HTTP Tunnel, and need to implement a generic tunnel?
A target could be, for instance, another tunnel. Also, we can support different protocols. The core would stay the same.
We already saw that the tunnel_stream
method already works with any L4 Client<->Tunnel
connection.
#[async_trait]
pub trait TunnelTarget {
type Addr;
fn addr(&self) -> Self::Addr;
}
#[async_trait]
pub trait TargetConnector {
type Target: TunnelTarget + Send + Sync + Sized;
type Stream: AsyncRead + AsyncWrite + Send + Sized + 'static;
async fn connect(&mut self, target: &Self::Target) -> io::Result<Self::Stream>;
}
Here, we specify two abstractions:
TunnelTarget
is just something that has an Addr
— whatever it is.TargetConnector
— can connect to that Addr
and needs to return a stream that supports async I/O.Okay, but what about the target negotiation? The tokio-utils
crate already has an abstraction for that, named Framed
streams (with corresponding Encoder/Decoder
traits). We need to implement them for HTTP CONNECT
(or any other proxy protocol). You can find the implementation here.
Relay
We only have one major component remaining — that relays data after the tunnel negotiation is done. tokio
provides a method to split a stream into two halves: ReadHalf
and WriteHalf
. We can split client and target connections and relay them in both directions:
let (client_recv, client_send) = io::split(client);
let (target_recv, target_send) = io::split(target);
let upstream_task =
tokio::spawn(
async move {
upstream_relay.relay_data(client_recv, target_send).await
});
let downstream_task =
tokio::spawn(
async move {
downstream_relay.relay_data(target_recv, client_send).await
});
Where the relay_data(…)
definition requires nothing more than implementing the abstractions mentioned above. For example, it can connect any two halves of a stream:
/// Relays data in a single direction. E.g.
pub async fn relay_data<R: AsyncReadExt + Sized, W: AsyncWriteExt + Sized>(
self,
mut source: ReadHalf<R>,
mut dest: WriteHalf<W>,
) -> io::Result<RelayStats> {...}
And finally, instead of a simple HTTP Tunnel, we have an engine that can be used to build any type of tunnels or a chain of tunnels (e.g., for onion routing) over any transport and proxy protocols:
/// A connection tunnel.
///
/// # Parameters
/// * `<H>` - proxy handshake codec for initiating a tunnel.
/// It extracts the request message, which contains the target, and, potentially policies.
/// It also takes care of encoding a response.
/// * `<C>` - a connection from from client.
/// * `<T>` - target connector. It takes result produced by the codec and establishes a connection
/// to a target.
///
/// Once the target connection is established, it relays data until any connection is closed or an
/// error happens.
impl<H, C, T> ConnectionTunnel<H, C, T>
where
H: Decoder<Error = EstablishTunnelResult> + Encoder<EstablishTunnelResult>,
H::Item: TunnelTarget + Sized + Display + Send + Sync,
C: AsyncRead + AsyncWrite + Sized + Send + Unpin + 'static,
T: TargetConnector<Target = H::Item>,
{...}
The implementation is almost trivial in basic cases, but we want our app to handle failures, and that’s the focus of the next section.
The amount of time engineers deal with failures is proportional to the scale of a system. It’s easy to write happy-case code. Still, if it enters an irrecoverable state on the very first error, it’s painful to use. Besides that, your app will be used by other engineers, and there are very few things more irritating than cryptic/misleading error messages. If your code runs as a part of a large service, some people need to monitor and support it (e.g., SREs or DevOps), and it should be a pleasure for them to deal with your service.
What kind of failures may an HTTP Tunnel encounter?
It’s a good idea to enumerate all error codes that your app returns to the client. So it’s clear why a request failed if the operation can be tried again (or shouldn’t) if it’s an integration bug, or just network noise:
pub enum EstablishTunnelResult {
/// Successfully connected to target.
Ok,
/// Malformed request
BadRequest,
/// Target is not allowed
Forbidden,
/// Unsupported operation, however valid for the protocol.
OperationNotAllowed,
/// The client failed to send a tunnel request timely.
RequestTimeout,
/// Cannot connect to target.
BadGateway,
/// Connection attempt timed out.
GatewayTimeout,
/// Busy. Try again later.
TooManyRequests,
/// Any other error. E.g. an abrupt I/O error.
ServerError,
}
Dealing with delays is crucial for a network app. If your operations don’t have timeouts, it’s a matter of time until all of your threads will be “Waiting for Godot,” or your app will exhaust all available resources and become unavailable. Here we delegate the timeout definition to RelayPolicy
:
let read_result = self
.relay_policy
.timed_operation(source.read(&mut buffer))
.await;
if read_result.is_err() {
shutdown_reason = RelayShutdownReasons::ReaderTimeout;
break;
}
let n = match read_result.unwrap() {
Ok(n) if n == 0 => {
shutdown_reason = RelayShutdownReasons::GracefulShutdown;
break;
}
Ok(n) => n,
Err(e) => {
error!(
"{} failed to read. Err = {:?}, CTX={}",
self.name, e, self.tunnel_ctx
);
shutdown_reason = RelayShutdownReasons::ReadError;
break;
}
};
The relay policy can be configured like this:
relay_policy:
idle_timeout: 10s
min_rate_bpm: 1000
max_rate_bps: 10000
max_lifetime: 100s
max_total_payload: 100mb
So we can limit activity per connection with max_rate_bps
and detect idle clients with min_rate_bpm
(so they don’t consume system resources than can be utilized more productively). A connection lifetime and total traffic may be bounded as well.
It goes without saying that each failure mode needs to be tested. It’s straightforward to do that in Rust, in general, and with tokio-test
in particular:
#[tokio::test]
async fn test_timed_operation_timeout() {
let time_duration = 1;
let data = b"data on the wire";
let mut mock_connection: Mock = Builder::new()
.wait(Duration::from_secs(time_duration * 2))
.read(data)
.build();
let relay_policy: RelayPolicy = RelayPolicyBuilder::default()
.min_rate_bpm(1000)
.max_rate_bps(100_000)
.idle_timeout(Duration::from_secs(time_duration))
.build()
.unwrap();
let mut buf = [0; 1024];
let timed_future = relay_policy
.timed_operation(mock_connection.read(&mut buf))
.await;
assert!(timed_future.is_err());
}
The same goes for I/O errors:
#[tokio::test]
async fn test_timed_operation_failed_io() {
let mut mock_connection: Mock = Builder::new()
.read_error(Error::from(ErrorKind::BrokenPipe))
.build();
let relay_policy: RelayPolicy = RelayPolicyBuilder::default()
.min_rate_bpm(1000)
.max_rate_bps(100_000)
.idle_timeout(Duration::from_secs(5))
.build()
.unwrap();
let mut buf = [0; 1024];
let timed_future = relay_policy
.timed_operation(mock_connection.read(&mut buf))
.await;
assert!(timed_future.is_ok()); // no timeout
assert!(timed_future.unwrap().is_err()); // but io-error
}
I haven’t seen an application that failed only in ways anticipated by its developers. I’m not saying there are no such applications. Still, chances are, your app is going to encounter something you didn’t expect: data races, specific traffic patterns, dealing with traffic bursts, and legacy clients.
But, one of the most common types of failures is human failures, such as pushing bad code or configuration, which are inevitable in large projects. Anyway, we need to be able to deal with something we didn’t foresee. So we emit enough information that would allow us to detect failures and troubleshoot.
So we’d better log every error and important event with meaningful information and relevant context as well as statistics:
/// Stats after the relay is closed. Can be used for telemetry/monitoring.
#[derive(Builder, Clone, Debug, Serialize)]
pub struct RelayStats {
pub shutdown_reason: RelayShutdownReasons,
pub total_bytes: usize,
pub event_count: usize,
pub duration: Duration,
}
/// Statistics. No sensitive information.
#[derive(Serialize)]
pub struct TunnelStats {
tunnel_ctx: TunnelCtx,
result: EstablishTunnelResult,
upstream_stats: Option<RelayStats>,
downstream_stats: Option<RelayStats>,
}
Note: the tunnel_ctx: TunnelCtx
field, which can be used to correlate metric records with log messages:
Last but not least, we’d like to be able to run our tunnel in different modes with different parameters. Here’s where serde
and clap
become handy:
#[derive(Parser, Debug)]
#[clap(author, version, about, long_about = None)]
#[clap(propagate_version = true)]
struct Cli {
/// Configuration file.
#[clap(long)]
config: Option<String>,
/// Bind address, e.g. 0.0.0.0:8443.
#[clap(long)]
bind: String,
#[clap(subcommand)]
command: Commands,
}
#[derive(Subcommand, Debug)]
enum Commands {
Http(HttpOptions),
Https(HttpsOptions),
Tcp(TcpOptions),
}
In my opinion, clap
makes dealing with command line parameters pleasant. Extraordinarily expressive and easy to maintain.
Configuration files can be easily handled with serde-yaml
:
target_connection:
dns_cache_ttl: 60s
allowed_targets: "(?i)(wikipedia|rust-lang)\\.org:443$"
connect_timeout: 10s
relay_policy:
idle_timeout: 10s
min_rate_bpm: 1000
max_rate_bps: 10000
Which corresponds to Rust structs:
#[derive(Deserialize, Clone)]
pub struct TargetConnectionConfig {
#[serde(with = "humantime_serde")]
pub dns_cache_ttl: Duration,
#[serde(with = "serde_regex")]
pub allowed_targets: Regex,
#[serde(with = "humantime_serde")]
pub connect_timeout: Duration,
pub relay_policy: RelayPolicy,
}
#[derive(Builder, Deserialize, Clone)]
pub struct RelayPolicy {
#[serde(with = "humantime_serde")]
pub idle_timeout: Duration,
/// Min bytes-per-minute (bpm)
pub min_rate_bpm: u64,
// Max bytes-per-second (bps)
pub max_rate_bps: u64,
}
It doesn’t need any additional comments to make it readable and maintainable, which is beautiful.
As you can see from this quick overview, the Rust ecosystem already provides many building blocks, so you can focus on what you need to do rather than how. You didn’t see any memory/resources management or explicit thread safety (which often comes at the expense of concurrency) with impressive performance. Abstraction mechanisms are fantastic, so your code can be highly reusable. This task was a lot of fun, so I’ll try to take on the next challenge.
Original article sourced at: https://medium.com