Introducing GitPlus: Version Control extension for Jupyter

There’s no easy way to version control notebooks from Jupyter UI. Of course you can drop down to command line & learn a bunch of git commands to version control your notebooks. But not everyone using Jupyter is proficient at git. Hence I built GitPlus, a JupyterLab extension that provides the ability to commit notebooks & create GitHub pull requests directly from JupyterLab UI.

How to version control Jupyter Notebooks

When GitPlus extension is installed, it provides a new menu item Git-Plus in JupyterLab UI. From there, you can commit notebook files or create a GitHub pull request as shown in demo videos below.

How does it work?

• There’s a server component of the extension that uses gitpython & GitHub APIs to push commits and create pull requests.
• There’s a client component of the extension that provides the UI and calls appropriate endpoint on the server extension.
• The client component looks at all the open files in JupyterLab and determines which of those files are under a GitHub repository.
• It lets you select the repository to which you want to push commit or create pull request.
• It captures the list of files to commit, commit message and pushes the changes as a commit to remote repository.
• In case of pull requests, it creates a new branch and pushes the changes there. This newly created branch is compared to default branch (typically master) of your repository.
• In case of forked repositories, pull request is created on the parent repository.

#data-science #git #version-control #mlops #jupyter-notebook

What is GEEK

Buddha Community

Introducing GitPlus: Version Control extension for Jupyter

There’s no easy way to version control notebooks from Jupyter UI. Of course you can drop down to command line & learn a bunch of git commands to version control your notebooks. But not everyone using Jupyter is proficient at git. Hence I built GitPlus, a JupyterLab extension that provides the ability to commit notebooks & create GitHub pull requests directly from JupyterLab UI.

How to version control Jupyter Notebooks

When GitPlus extension is installed, it provides a new menu item Git-Plus in JupyterLab UI. From there, you can commit notebook files or create a GitHub pull request as shown in demo videos below.

How does it work?

• There’s a server component of the extension that uses gitpython & GitHub APIs to push commits and create pull requests.
• There’s a client component of the extension that provides the UI and calls appropriate endpoint on the server extension.
• The client component looks at all the open files in JupyterLab and determines which of those files are under a GitHub repository.
• It lets you select the repository to which you want to push commit or create pull request.
• It captures the list of files to commit, commit message and pushes the changes as a commit to remote repository.
• In case of pull requests, it creates a new branch and pushes the changes there. This newly created branch is compared to default branch (typically master) of your repository.
• In case of forked repositories, pull request is created on the parent repository.

#data-science #git #version-control #mlops #jupyter-notebook

How To Install Git on Ubuntu 16.04 LTS

Git is one of the most popular tools used as a distributed version control system (VCS). Git is commonly used for source code management (SCM) and has become more used than old VCS systems like SVN. In this article, we’ll show you how to install Git on your Ubuntu 16.04 dedicated server.

Installing Git on Ubuntu 16.04 LTS

We have also created a convenient video tutorial that outlines how to install Git on a Ubuntu 16.04 Server.

Now, let’s get started on that install…

Preflight Check

• You should be running a server with any VPS Ubuntu LTS release.
• You will need to log in to SSH via the root user.

First, as always, we should start out by running general OS and package updates. On Ubuntu we’ll do this by running:

apt-get update

After you have run the general updates on the server you can get started with installing Git.

1. Install Git

2. apt-get install git-core

3. You may be asked to confirm the download and installation; simply enter y to confirm. It’s that simple, Git should be installed and ready to use!

4. Confirm Git the installation

5. With the main installation done, first check to ensure the executable file is set up and accessible. The best way to do this is simply to run Git with the version command.

6. git --version

7. git version 2.7.4

8. Configure Git’s settings (for the root user)

9. It’s a good idea to setup your user for git now, to prevent any commit errors later. We’ll setup the user testuser with the e-mail address testuser@example.com.

10. git config --global user.name "testuser" git config --global user.email "testuser@example.com"

Note:

It’s important to know that git configs work on a user by user basis. For example, if you have a ‘david’ Linux user and they will be working with git then David should run the same commands from his user account. By doing this the commits made by the ‘david’ Linux user will be done under his details in Git.

#distributed version control #git #linux #scm #source code management #tools #ubuntu #ubuntu 16.04 #vcs #version control #version control system

How to version control Jupyter Notebooks

Jupyter notebooks are fantastic in many ways but collaboration is not so easy with them. In this article we’ll look at all the tools you can leverage to make notebooks play nicely with modern version control systems like git!

Why is Jupyter version control so hard?

The software world has converged on git as it’s version control tool of choice. Git is designed to work primarily for human-readable text files. Whereas Jupyter is a rich JSON document with source code, markdown, HTML, images all rolled into a single .ipynb file.

Git doesn’t handle rich documents like notebooks very well. E.g. git merge for long nested JSON documents is humanly impossible, git diff for binary image string is horrible (shown below).

What’s required from notebook version control?

Here’s what we need from a modern version control system -

• Ability to create checkpoints / commits
• Quickly checkout any of the past notebook versions
• See what changed from one version to another (a.k.a visual diff for notebooks)
• Multiple people can work on a single notebook with easy merge conflict resolution
• Ability to provide feedback & ask questions about a specific notebook cell

That’s our wishlist! This blogpost is going to introduce you to all the important tools that can help you achieve these.

Disclaimer: I’m the author of two of the tools listed below (ReviewNB & GitPlus) but this is an unbiased review of all the useful tools in this space.

nbdime

nbdime is an open source library for diffing and merging notebooks locally. You can set this up to work with local git client so that git diff & git merge commands use nbdime for .ipynb files. With nbdime you can —

• Run git diff to see how notebook has changed before committing
• Easily merge remote changes with your locally edited notebook

JupyterLab Git Extensions

Following JupyterLab extensions are useful for notebook version control. You can install these on your local JupyterLab.

• jupyterlab-git can be used to browse GitHub repositories, look at visual diffs of changed files, and push your commits
• GitPlus can be used to push commits and create pull requests on GitHub directly from JupyterLab UI

#data-science #jupyter-notebook #version-control #git #github

10 Open Source/Commercial Control Panels For Virtual Machines (VM's) Management

Automatic creation and management of virtual machines is a topical issue for any company that provides VPS services. If you manage a large number of machines, a command line is definitely not the only tool you may need to perform various operations including client tasks, because such operations may be time-consuming.

In order to simplify routine tasks of server administrators and users, various companies develop control panels for virtual machines management, including interface-based solutions.

Don’t Miss: 20 Open Source/Commercial Control Panels to Manage Linux Servers

A control panel empowers you to perform any operation with a mouse click, whereas it would take you a good deal of time to complete the same task in the console. With a control panel, you will save your time and effort. However, it’s not all that simple.

Nowadays, VMmanager is the most popular software product for small and medium-sized businesses. VMware, in its turn, is a leading solution for large organizations. Both software products are commercial and rather expensive.

They deliver a large number of functions, however, some companies, especially, startups may need them. Besides, many of them cannot afford such an expensive product. For example, startups and companies in times of crisis may experience financial difficulties. Moreover, one can find interesting, outstanding solutions integrated with billing systems including tools for VM management.

How not to get lost among a great number of offers? We decided to help our users and wrote the following article, in which they will find answers to this question.

In this article, we will describe control panels for virtual machines management, both commercial and open-source, and help you choose the right solution to meet your personal needs.

1. VMmanager

VMmanager is one of the most popular commercial server virtualizations platforms based on QEMU/KVM technology. The solution has a reach feature set, that can suit both IT infrastructure owners and VPS services providers’ needs.

Virtual servers can be created within 2 minutes. Many routine tasks are performed automatically: including migration, cloning, reinstalling the OS, backups, adding and deleting interfaces, virtual server image creation, monitoring, statistics collection, server provisioning, etc.

The main advantages of VMmanager are:

• Centralized management of various clusters.
• Fault tolerance due to a microservice architecture.
• Overselling, which helps to improve VPS provider’s equipment efficiency.
• Complete control of the infrastructure thanks to a robust system of metrics collection.
• A modern and intuitive interface.

VMmanager – Virtualization Management Platform

2. VMware vSphere

VMware vSphere is the world’s leading server virtualization platform for building cloud infrastructure. With tons of its different powerful features, vSphere is a truely state-of-the-art software virtual machines management software. It is an ideal solution for large VPS providers with appropriate budgets and professional staff.

#control panels #virtualization #hosting control panel #linux control panels #virtual mahine control panels #linux

Rollup Plugin Glsl Optimize: Import GLSL Source Files As Strings

rollup-plugin-glsl-optimize 

Import GLSL source files as strings. Pre-processed, validated and optimized with Khronos Group SPIRV-Tools.

Primary use-case is processing WebGL2 / GLSL ES 300 shaders.

import frag from './shaders/myShader.frag';
console.log(frag);

Features

GLSL Optimizer

For WebGL2 / GLSL ES >= 300

With optimize: true (default) shaders will be compiled to SPIR-V (opengl semantics) and optimized for performance using the Khronos SPIR-V Tools Optimizer before being cross-compiled back to GLSL.

Shader Preprocessor

Shaders are preprocessed and validated using the Khronos Glslang Validator.

Macros are run at build time with support for C-style #include directives: *

#version 300 es
#include "postProcessingShared.glsl"
#include "dofCircle.glsl"

void main() {
  outColor = CircleDof(UVAndScreenPos, Color, ColorCoc);
}

* Via the GL_GOOGLE_include_directive extension. But an #extension directive is not required nor recommended in your final inlined code.

Supports glslify

Specify glslify: true to process shader sources with glslify (a node.js-style module system for GLSL).

And install glslify in your devDependencies with npm i -D glslify

Installation

npm i rollup-plugin-glsl-optimize -D

Khronos tools

This plugin uses the Khronos Glslang Validator, Khronos SPIRV-Tools Optimizer and Khronos SPIRV Cross compiler.

Binaries are automatically installed for:

• Windows 64bit (MSVC 2017)
• MacOS x86_64 (clang)
• Ubuntu Trusty / Debian Buster amd64 (clang)

Paths can be manually provided / overridden with the GLSLANG_VALIDATOR, GLSLANG_OPTIMIZER, GLSLANG_CROSS environment variables.

Usage

// rollup.config.mjs
import {default as glslOptimize} from 'rollup-plugin-glsl-optimize';

export default {
    // ...
    plugins: [
        glslOptimize(),
    ]
};

Shader stages

The following shader stages are supported by the Khronos tools and recognized by file extension:

Shader Stage	File Extensions
Vertex	.vs, .vert, .vs.glsl, .vert.glsl
Fragment	.fs, .frag, .fs.glsl, .frag.glsl
Geometry*	.geom, .geom.glsl
Compute*	.comp, .comp.glsl
Tess Control*	.tesc, .tesc.glsl
Tess Eval*	.tese, .tese.glsl

* Unsupported in WebGL2

Options

• include : PathFilter (default table above) File extensions within rollup to include. Though this option can be reconfigured, shader stage detection still operates based on the table above.
• exclude : PathFilter (default undefined) File extensions within rollup to exclude.
• includePaths : string[] (default undefined) Additional search paths for #include directive (source file directory is always searched)

Features

• optimize : boolean (default true) Optimize via SPIR-V as described in the Optimization section [requires WebGL2 / GLSL ES >= 300]. When disabled simply runs the preprocessor [all supported GLSL versions].
• compress : boolean (default true) Strip all whitespace in the sources

Debugging

• sourceMap : boolean (default true) Emit source maps. These contain the final preprocessed/optimized GLSL source (but not stripped of whitespace) to aid debugging.
• emitLineDirectives : boolean (default false) Emit #line NN "original.file" directives for debugging - useful with #include. Note this requires the GL_GOOGLE_cpp_style_line_directive extension so the shader will fail to run in drivers that lack support.

Preprocessor

• optimizerPreserveUnusedBindings : boolean (default true) Ensure that the optimizer preserves all declared bindings, even when those bindings are unused.
• preamble : string (default undefined) Prepended to the shader source (after the #version directive, before the preprocessor runs)

glslify

• glslify : boolean (default false) Process sources using glslify prior to all preprocessing, validation and optimization.
• glslifyOptions (default undefined) When glslify enabled, pass these additional options to glslify.compile().

Advanced Options

• optimizerDebugSkipOptimizer : boolean (default false) When optimize enabled, skip the SPIR-V optimizer - compiles to SPIR-V then cross-compiles back to GLSL immediately.
• suppressLineExtensionDirective : boolean (default false) When emitLineDirectives enabled, suppress the GL_GOOGLE_cpp_style_line_directive directive.
• extraValidatorParams, extraOptimizerParams, extraCrossParams : string[] (default undefined) Additional parameters for the Khronos Glslang Validator here, the Khronos SPIR-V Optimizer here, and the Khronos SPIR-V Cross compiler here.
• glslangValidatorPath, glslangOptimizerPath, glslangCrossPath : string (default undefined) Provide / override binary tool paths.\

It's recommended to instead use the environment variables GLSLANG_VALIDATOR, GLSLANG_OPTIMIZER, GLSLANG_CROSS where needed. They always take precedence if set.

Changelog

Available in CHANGES.md.

Caveats & Known Issues

• This plugin handles glsl and glslify by itself. Use with conflicting plugins (e.g. rollup-plugin-glsl, rollup-plugin-glslify) will cause unpredictable results.
• Optimizer: lowp precision qualifier - emitted as mediump
  SPIR-V has a single RelaxedPrecision decoration for 16-32bit precision. However most implementations actually treat mediump and lowp equivalently, hence the lack of need for it in SPIR-V.

License

Released under the MIT license.
Strip whitespace function adapted from code by Vincent Wochnik (rollup-plugin-glsl).

Khronos tool binaries (built by the upstream projects) are distributed and installed with this plugin under the terms of the Apache License Version 2.0. See the corresponding LICENSE files installed in the bin folder and the binary releases.

---

Author: docd27
Source code: https://github.com/docd27/rollup-plugin-glsl-optimize
License: MIT license

#javascript #Rollup