1557908248
Since we introduced the Model Optimization Toolkit — a suite of techniques that developers, both novice and advanced, can use to optimize machine learning models — we have been busy working on our roadmap to add several new approaches and tools. Today, we are happy to share the new weight pruning API.
Optimizing machine learning programs can take very different forms. Fortunately, neural networks have proven resilient to different transformations aimed at this goal.
One such family of optimizations aims to reduce the number of parameters and operations involved in the computation by removing connections, and thus parameters, in between neural network layers.
The weight pruning API is built on top of Keras, so it will be very easy for developers to apply this technique to any existing Keras training program. This API will be part of a new GitHub repository for the model optimization toolkit, along with many upcoming optimization techniques.
import tensorflow_model_optimization as tfmot model = build_your_model() pruning_schedule = tfmot.sparsity.keras.PolynomialDecay( initial_sparsity=0.0, final_sparsity=0.5, begin_step=2000, end_step=4000) model_for_pruning = tfmot.sparsity.keras.prune_low_magnitude(model, pruning_schedule=pruning_schedule) ... model_for_pruning.fit(...)
Weight pruning means eliminating unnecessary values in the weight tensors. We are practically setting the neural network parameters’ values to zero to remove what we estimate are unnecessary connections between the layers of a neural network. This is done during the training process to allow the neural network to adapt to the changes.
An immediate benefit from this work is disk compression: sparse tensors are amenable to compression. Thus, by applying simple file compression to the pruned TensorFlow checkpoint, or the converted TensorFlow Lite model, we can reduce the size of the model for its storage and/or transmission. For example, in the tutorial, we show how a 90% sparse model for MNIST can be compressed from 12MB to 2MB.
Moreover, across several experiments, we found that weight pruning is compatible with quantization, resulting in compound benefits. In the same tutorial, we show how we can further compress the pruned model from 2MB to just 0.5MB by applying post-training quantization.
In the future, TensorFlow Lite will add first-class support for sparse representation and computation, thus expanding the compression benefit to the runtime memory and unlocking performance improvements, since sparse tensors allow us to skip otherwise unnecessary computations involving the zeroed values.
In our experiments, we have validated that this technique can be successfully applied to different types of models across distinct tasks, from image processing convolutional-based neural networks to speech processing ones using recurrent neural networks. The following table shows a subset of some of these experimental results.
Sparsity results across different models and tasks.
Our Keras-based weight pruning API uses a straightforward, yet broadly applicable algorithm designed to iteratively remove connections based on their magnitude during training. Fundamentally, a final target sparsity is specified (e.g. 90%), along with a schedule to perform the pruning (e.g. start pruning at step 2,000, stop at step 10,000, and do it every 100 steps), and an optional configuration for the pruning structure (e.g. apply to individual values or blocks of values in certain shape).
Example of tensors with no sparsity (left), sparsity in blocks of 1x1 (center), and sparsity in blocks of 1x2 (right).
As training proceeds, the pruning routine will be scheduled to execute, eliminating (i.e. setting to zero) the weights with the lowest magnitude values (i.e. those closest to zero) until the current sparsity target is reached. Every time the pruning routine is scheduled to execute, the current sparsity target is recalculated, starting from 0% until it reaches the final target sparsity at the end of the pruning schedule by gradually increasing it according to a smooth ramp-up function.
Example of sparsity ramp-up function with a schedule to start pruning from step 0 until step 100, and a final target sparsity of 90%.
Just like the schedule, the ramp-up function can be tweaked as needed. For example, in certain cases, it may be convenient to schedule the training procedure to start after a certain step when some convergence level has been achieved, or end pruning earlier than the total number of training steps in your training program to further fine-tune the system at the final target sparsity level. For more details on these configurations, please refer to our tutorial and documentation.
At the end of the training procedure, the tensors corresponding to the “pruned” Keras layers will contain zeros according to the final sparsity target for the layer.
Animation of pruning applied to a tensor. Black cells indicate where the non-zero weights exist. Sparsity increases as training proceeds.
As mentioned earlier, the weight pruning API will be part of a new GitHub project and repository aimed at techniques that make machine learning models more efficient to execute and/or represent. This is a great project to star if you are interested in this exciting area of machine learning or just want to have the resources to optimize your models.
Given the importance of this area, we are also creating a new sub-site under tensorflow.org/model_optimization with relevant documentation and resources. We encourage you to give this a try right away and welcome your feedback.
Thanks for reading ❤
If you liked this post, share it with all of your programming buddies!
Follow us on Facebook | Twitter
☞ Complete Guide to TensorFlow for Deep Learning with Python
☞ Tensorflow Bootcamp For Data Science in Python
☞ Python for Data Science and Machine Learning Bootcamp
☞ 9 Things You Should Know About TensorFlow
☞ TensorFlow is dead, long live TensorFlow!
☞ How to Image Classification with TensorFlow 2.0?
☞ Introduction to Tensorflow for Java
☞ Machine Learning Tutorial - Image Processing using Python, OpenCV, Keras and TensorFlow
Originally published on https://medium.com
#tensorflow #deep-learning #machine-learning
1595396220
As more and more data is exposed via APIs either as API-first companies or for the explosion of single page apps/JAMStack, API security can no longer be an afterthought. The hard part about APIs is that it provides direct access to large amounts of data while bypassing browser precautions. Instead of worrying about SQL injection and XSS issues, you should be concerned about the bad actor who was able to paginate through all your customer records and their data.
Typical prevention mechanisms like Captchas and browser fingerprinting won’t work since APIs by design need to handle a very large number of API accesses even by a single customer. So where do you start? The first thing is to put yourself in the shoes of a hacker and then instrument your APIs to detect and block common attacks along with unknown unknowns for zero-day exploits. Some of these are on the OWASP Security API list, but not all.
Most APIs provide access to resources that are lists of entities such as /users
or /widgets
. A client such as a browser would typically filter and paginate through this list to limit the number items returned to a client like so:
First Call: GET /items?skip=0&take=10
Second Call: GET /items?skip=10&take=10
However, if that entity has any PII or other information, then a hacker could scrape that endpoint to get a dump of all entities in your database. This could be most dangerous if those entities accidently exposed PII or other sensitive information, but could also be dangerous in providing competitors or others with adoption and usage stats for your business or provide scammers with a way to get large email lists. See how Venmo data was scraped
A naive protection mechanism would be to check the take count and throw an error if greater than 100 or 1000. The problem with this is two-fold:
skip = 0
while True: response = requests.post('https://api.acmeinc.com/widgets?take=10&skip=' + skip), headers={'Authorization': 'Bearer' + ' ' + sys.argv[1]}) print("Fetched 10 items") sleep(randint(100,1000)) skip += 10
To secure against pagination attacks, you should track how many items of a single resource are accessed within a certain time period for each user or API key rather than just at the request level. By tracking API resource access at the user level, you can block a user or API key once they hit a threshold such as “touched 1,000,000 items in a one hour period”. This is dependent on your API use case and can even be dependent on their subscription with you. Like a Captcha, this can slow down the speed that a hacker can exploit your API, like a Captcha if they have to create a new user account manually to create a new API key.
Most APIs are protected by some sort of API key or JWT (JSON Web Token). This provides a natural way to track and protect your API as API security tools can detect abnormal API behavior and block access to an API key automatically. However, hackers will want to outsmart these mechanisms by generating and using a large pool of API keys from a large number of users just like a web hacker would use a large pool of IP addresses to circumvent DDoS protection.
The easiest way to secure against these types of attacks is by requiring a human to sign up for your service and generate API keys. Bot traffic can be prevented with things like Captcha and 2-Factor Authentication. Unless there is a legitimate business case, new users who sign up for your service should not have the ability to generate API keys programmatically. Instead, only trusted customers should have the ability to generate API keys programmatically. Go one step further and ensure any anomaly detection for abnormal behavior is done at the user and account level, not just for each API key.
APIs are used in a way that increases the probability credentials are leaked:
If a key is exposed due to user error, one may think you as the API provider has any blame. However, security is all about reducing surface area and risk. Treat your customer data as if it’s your own and help them by adding guards that prevent accidental key exposure.
The easiest way to prevent key exposure is by leveraging two tokens rather than one. A refresh token is stored as an environment variable and can only be used to generate short lived access tokens. Unlike the refresh token, these short lived tokens can access the resources, but are time limited such as in hours or days.
The customer will store the refresh token with other API keys. Then your SDK will generate access tokens on SDK init or when the last access token expires. If a CURL command gets pasted into a GitHub issue, then a hacker would need to use it within hours reducing the attack vector (unless it was the actual refresh token which is low probability)
APIs open up entirely new business models where customers can access your API platform programmatically. However, this can make DDoS protection tricky. Most DDoS protection is designed to absorb and reject a large number of requests from bad actors during DDoS attacks but still need to let the good ones through. This requires fingerprinting the HTTP requests to check against what looks like bot traffic. This is much harder for API products as all traffic looks like bot traffic and is not coming from a browser where things like cookies are present.
The magical part about APIs is almost every access requires an API Key. If a request doesn’t have an API key, you can automatically reject it which is lightweight on your servers (Ensure authentication is short circuited very early before later middleware like request JSON parsing). So then how do you handle authenticated requests? The easiest is to leverage rate limit counters for each API key such as to handle X requests per minute and reject those above the threshold with a 429 HTTP response.
There are a variety of algorithms to do this such as leaky bucket and fixed window counters.
APIs are no different than web servers when it comes to good server hygiene. Data can be leaked due to misconfigured SSL certificate or allowing non-HTTPS traffic. For modern applications, there is very little reason to accept non-HTTPS requests, but a customer could mistakenly issue a non HTTP request from their application or CURL exposing the API key. APIs do not have the protection of a browser so things like HSTS or redirect to HTTPS offer no protection.
Test your SSL implementation over at Qualys SSL Test or similar tool. You should also block all non-HTTP requests which can be done within your load balancer. You should also remove any HTTP headers scrub any error messages that leak implementation details. If your API is used only by your own apps or can only be accessed server-side, then review Authoritative guide to Cross-Origin Resource Sharing for REST APIs
APIs provide access to dynamic data that’s scoped to each API key. Any caching implementation should have the ability to scope to an API key to prevent cross-pollution. Even if you don’t cache anything in your infrastructure, you could expose your customers to security holes. If a customer with a proxy server was using multiple API keys such as one for development and one for production, then they could see cross-pollinated data.
#api management #api security #api best practices #api providers #security analytics #api management policies #api access tokens #api access #api security risks #api access keys
1601381326
We’ve conducted some initial research into the public APIs of the ASX100 because we regularly have conversations about what others are doing with their APIs and what best practices look like. Being able to point to good local examples and explain what is happening in Australia is a key part of this conversation.
The method used for this initial research was to obtain a list of the ASX100 (as of 18 September 2020). Then work through each company looking at the following:
With regards to how the APIs are shared:
#api #api-development #api-analytics #apis #api-integration #api-testing #api-security #api-gateway
1604399880
I’ve been working with Restful APIs for some time now and one thing that I love to do is to talk about APIs.
So, today I will show you how to build an API using the API-First approach and Design First with OpenAPI Specification.
First thing first, if you don’t know what’s an API-First approach means, it would be nice you stop reading this and check the blog post that I wrote to the Farfetchs blog where I explain everything that you need to know to start an API using API-First.
Before you get your hands dirty, let’s prepare the ground and understand the use case that will be developed.
If you desire to reproduce the examples that will be shown here, you will need some of those items below.
To keep easy to understand, let’s use the Todo List App, it is a very common concept beyond the software development community.
#api #rest-api #openai #api-first-development #api-design #apis #restful-apis #restful-api
1602682740
In the API economy, a successful service can gain popularity and be utilized in ways unpredicted and often inconceivable by its original owners. The very flexible nature of the technology opens many doors, including business collaborations, reuse in third-party products or even conquering hardware barriers by reaching a spectrum of devices.
Taking the builder’s perspective
Important note: Most of the time API consumers are not the end-users but rather the app developers. Any new venture ought to be supported with excellent learning resources and descriptive documentation. These things combined will ensure a top-notch developer experience and encourage adoption of your product, increasing its visibility in the market.
More than the revenue
While in the simplest scenario, the most popular API business model is revenue via service charges, there are several other goals:
#api #api-development #api-integration #restful-api #api-based-business-model #api-first-development #automation #rest-api
1598083582
As more companies realize the benefits of an API-first mindset and treating their APIs as products, there is a growing need for good API product management practices to make a company’s API strategy a reality. However, API product management is a relatively new field with little established knowledge on what is API product management and what a PM should be doing to ensure their API platform is successful.
Many of the current practices of API product management have carried over from other products and platforms like web and mobile, but API products have their own unique set of challenges due to the way they are marketed and used by customers. While it would be rare for a consumer mobile app to have detailed developer docs and a developer relations team, you’ll find these items common among API product-focused companies. A second unique challenge is that APIs are very developer-centric and many times API PMs are engineers themselves. Yet, this can cause an API or developer program to lose empathy for what their customers actually want if good processes are not in place. Just because you’re an engineer, don’t assume your customers will want the same features and use cases that you want.
This guide lays out what is API product management and some of the things you should be doing to be a good product manager.
#api #analytics #apis #product management #api best practices #api platform #api adoption #product managers #api product #api metrics