macOS

A SwiftUI View for Displaying Markdown with Customizable Appearances

Parma

Display Markdown using pure SwiftUI components. Taking advantages of ViewBuilder to make custom appearances for Text and View.

Example

import Parma

struct ContentView: View {
    var markdown = "I'm **Strong**."
    
    var body: some View {
        Parma(markdown)
    }
}

For more examples, please refer to demo app.

Markdown Support

Already Supported

• Heading level 1-6
• Paragraph
• Multi-level bullet list
• Multi-level ordered list
  • Period delimiter
  • Parenthesis delimiter
• Image (Needs extra configurations)
• Inline text
  • Strong
  • Emphasis
  • Code

Possibly Support in Future Versions

• Divider
• Block quote
• Code block

Unsupported

• Inline hyperlink

Installation

Requirement

• Xcode 11.0 or later
• Swift 5 or later
• iOS 13.0 / macOS 10.15 or later deployment targets

Swift Package Manager

Swift Package Manager is a tool for managing the distribution of Swift code. It’s integrated with the Swift build system to automate the process of downloading, compiling, and linking dependencies on all platforms.

Adding Parma as a dependency by using Xcode’s GUI, the package url is https://github.com/dasautoooo/Parma .

CocoaPods

CocoaPods is a dependency manager for Cocoa projects. For usage and installation instructions, visit their website. To integrate Parma into your Xcode project using CocoaPods, specify it in your Podfile:

pod 'Parma'

Appearance Customization

To customize Text styles and Views, create a new render which conform to protocol ParmaRenderable, and only reimplement those that fit your purposes. Finally, assign the customized render as a new render when create Parma view.

import Parma

struct ContentView: View {
    var markdown = "I'm **Strong**."
    
    var body: some View {
        Parma(markdown, render: MyRender())
    }
}

struct MyRender: ParmaRenderable {
    ...
}

There's a DemoApp that modified some of these delegate methods below for everyone to take as a reference.

/// Define the heading text style.
/// - Parameters:
///   - level: The level of heading.
///   - textView: The textView generated from captured heading string.
func heading(level: HeadingLevel?, textView: Text) -> Text

/// Define the paragraph text style.
/// - Parameter text: The text string captured from paragraph.
func paragraph(text: String) -> Text

/// Define the text style for plain text. Do NOT recommend to alter this if there's no special purpose.
/// - Parameter text: The text string captured from markdown.
func plainText(_ text: String) -> Text

/// Define the strong text style.
/// - Parameter textView: The textView generated from captured strong string.
func strong(textView: Text) -> Text

/// Define the emphasis text style.
/// - Parameter textView: The textView generated from captured emphasis string.
func emphasis(textView: Text) -> Text

/// Define the link text style.
/// - Parameters:
///   - textView: The textView generated from captured link string.
///   - destination: The destination of the link.
func link(textView: Text, destination: String?) -> Text

/// Define the code text style.
/// - Parameter text: The text string captured from code.
func code(_ text: String) -> Text

/// Define the style of heading view.
/// - Parameters:
///   - level: The level of heading.
///   - view: The view contains heading text.
func headingBlock(level: HeadingLevel?, view: AnyView) -> AnyView

/// Define the style of paragraph view.
/// - Parameter view: The view contains view(s) which belong(s) to this paragraph.
func paragraphBlock(view: AnyView) -> AnyView

/// Define the style of list item.
/// - Parameter attributes: Attributes of the list containing the item. Those must be considered for proper item rendering.
/// - Parameter index: Normalized index of the list item. For exemple, the index of the third item of a one level list would be `[2]` and the second item of a sublist appearing fourth in it's parent list would be `[3, 1]`.
/// - Parameter view: The view contains view(s) which belong(s) to this item.
func listItem(attributes: ListAttributes, index: [Int], view: AnyView) -> AnyView

/// Define the style of image view.
/// - Parameter urlString: The url string for this image view.
/// - Parameter altTextView: The view contains alt text.
func imageView(with urlString: String, altTextView: AnyView?) -> AnyView

Name Origin

Parma is a city in the northern Italy, which is famous for its architecture, music and art. The reason of choosing this city name as the project name is Giambattista Bodoni, a famous typographer, who spent most his lifetime living and working in this city.

Bodoni was an Italian typographer, type-designer in Parma. During his lifespan, he designed many typefaces that known as Bodoni nowadays. Each Mac has Bodoni font installed, and free to use.

Credit

The package is built upon Down, which is a markdown parser in Swift.

Download Details:

Author: Dasautoooo
Source Code: https://github.com/dasautoooo/Parma 
License: MIT license

#swift #macos #markdown #ios 

Watch the latest and greatest conference videos on your Mac

Conferences.digital

Conferences.digital is the best way to watch the latest and greatest videos from your favourite developer conferences for free on your Mac. Either search specifically for conferences, talks, speakers or topics or simply browse through the catalog - you can add talks to your watchlist to save for later, favourite or continue watching where you left off.

Overview

Features

•  Search
•  Updates via Sparkle
•  Watchlist
•  Continue watching / Track progress

Installation

Some of you will see the warning in the screenshot below about the app "can't be opened because it is from an unidentified developer" when you first try to run it.

This is because I am not registered with Apple as part of their developer program. This is the first MacOS app that I've ever built, and I don't plan to release it in the Mac App Store so I don't want to pay the fees to register with Apple.

To bypass Apple's restriction, right-click the app and select "Open" the first time you use it or open your System Preferences application, click on the Security & Privacy selection, and there will be an "Open Anyway" button you can click.

If you have any other issues check out #9.

Contributing

If you have any ideas how to improve the app or you wish yourself a new feature you can submit an pull request or create an issue here.

If you want to submit an pull request make sure you read the contribution guidelines first.

Bugs

Please be advised that this project is still in an early stage and work in progress.

If you encounter any bugs please create an issue here and be as descriptive as possible.

Requirements

Conferences.digital is written in Swift 5. Compatible with macOS 10.12.2+ 

As soon as new conferences/talks have been added it will be announced on twitter.

Download the latest release here.

Download Details:

Author: Zagahr
Source Code: https://github.com/zagahr/Conferences.digital 
License: BSD-2-Clause license

#swift #learning #video #macos 

The MacOS Color Picker As an App with More Features

System Color Picker

The familiar color picker supercharged

The macOS color picker as an app with lots of extra features.

Features

• Quickly copy, paste, and convert colors in Hex, HSL, RGB, LCH format
• Show as a normal app or in the menu bar
• Pick a color or toggle the window from anywhere with a global keyboard shortcut
• Make the window stay on top of all other windows
• Launch it at login (when in the menu bar)
• Recently picked colors
• Shortcuts support
• Hide menu bar icon

Tips

• Press the Space key while using the color sampler to show the RGB values. The color sampler is a system component and it can unfortunately not show other kinds of color values.
• Press the Option key when copying the Hex color to invert whether to include #.

Keyboard shortcuts

You can use the following keyboard shortcuts in the app:

• Pick color: Command p
• Copy as Hex: Shift Command h
• Copy as HSL: Shift Command s
• Copy as RGB: Shift Command r
• Copy as LCH: Shift Command l
• Paste color: Shift Command v (In the format Hex, HSL, RGB, or LCH)

Plugins

The built-in color picker supports plugins:

• Scala Color
• Pro Picker
• Material Design
• Color Picker Plus

Screenshots

FAQ

The app does not show up in the menu bar

macOS hides menu bar apps when there is no space left in the menu bar. This is a common problem on MacBooks with a notch. Try quitting some menu bar apps to free up space. If this does not solve it, try quitting Bartender if you have it installed.

What is LCH color?

It's a more human-friendly color format.

Note that the LCH color is currently clamped to sRGB range.

The color changes if I copy and then paste it

That is because the default color space in the picker is Display P3, which is part of CSS Color 4, but the color space used for the legacy CSS color formats is sRGB (browsers are starting to handle color spaces but they are not all there yet).

How do I change the color space?

Right-click the color wheel. You probably want to select “sRGB”.

Note that the color strings will always be converted to sRGB color space.

Can you support SwiftUI.Color / UIColor / NSColor formats?

The best practice is to use Asset Catalog for colors instead of hard-coding the values in code. If you really want to hard-code colors, the Scala color picker plugin supports UIColor and NSColor.

Can I contribute localizations?

I don't have any immediate plans to localize the app.

Built with

• Defaults - Swifty and modern UserDefaults
• Regex - Swifty regular expressions
• KeyboardShortcuts - Add user-customizable global keyboard shortcuts to your macOS app
• LaunchAtLogin - Add “Launch at Login” functionality to your macOS app

Other apps

• Gifski - Convert videos to high-quality GIFs
• Plash - Make any website your Mac desktop wallpaper
• More apps…

Download

Requires macOS 12 or later.

Older versions

• Last macOS 11 compatible version (1.9.6)

Download Details:

Author: Sindresorhus
Source Code: https://github.com/sindresorhus/System-Color-Picker 
License: MIT license

#swift #macos #app #color #picker 

AuroraEditor: A IDE Built By The Community, for The Community

Aurora Editor

Aurora Editor is a IDE built by the community, for the community, and written in Swift for the best native performance and feel for macOS.

It will support everything you could expect from an Xcode-inspired application, including deep integration with your selected git provider, project planning, and your favourite built in editor tools.

Motivation

Developers should be able to use an editor that feels snappy and fast.

Most comparable editors are built on Electron, This is a huge disadvantage because it utilize a lot of unneccecary system resources. Electron requires a Chromium instance to run. This can mean massive performance losses and high RAM usage even for small apps built on it. Additionally, the overall code footprint is much larger and animations are slower. More frames are lost and things like window resizing feels laggy. Native apps are smooth as butter and utilize system resources much more efficiently for better performance and reliability.

Xcode is a great native editor for developers on Mac, but unfortunately it doesn't support creating a multitude of projects in different programming languages, and this is where Aurora Editor comes in. Aurora Editor wants to give developers the possibility of creating their desired projects in their desired language on a editor that is native and gives a similar experience, performance and feel than that of Xcode on Mac.

Included Repositories

Community

Join our community on Discord or Slack where we discuss and collaborate on all things of Aurora Editor. Don't be shy, jump right in and be part of the discussion!

Community Details

We would love to hear what kind of great ideas you as the community have. If you have an idea or a feature request for Aurora Editor feel free to add it to the Ideas Discussion.

If you created an awesome feature for Aurora Editor why not Show and tell, and celebrate with the community and developers of Aurora Editor.

When we have some awesome news or a big annoucement to make, we will be making it in the Annoucement Discussion. So stay tuned for any future annoucments.

Difference between AuroraEditor and CodeEdit

Please read https://github.com/AuroraEditor/AuroraEditor/discussions/286 to read about the difference between our projects.

Mental Health Awareness

We are aware how difficult and overwhelming it can be sometimes for developers when working on a big or small project. If you feel like you are getting overwhelmed when working on a certain feature or bug on Aurora Editor don't be afraid to let us know and we'll be able to help you out and takeover what you have been working on if you feel comfortable letting someone else do it. If you just feel the need to talk about certain issues feel free to talk about it in the Mental Health Channel or if just need advice on something ask in the Advice Channel.

Contributing

Be part of the next revolution in code editing by contributing to the project. This is a community-led effort, so we welcome as many contributors who can help! Please read the following for more information.

• What to do/add to the editor
• Contribution Guide
• Architecture Guide
• Developer Guide
• Troubeshooting Guide

Editor Localization

We want users to feel comfortable using Aurora Editor in their own speaking language, help us by translating Aurora Editor for you and the rest of the community.

Translate Aurora Editor Now

Download Details:

Author: AuroraEditor
Source Code: https://github.com/AuroraEditor/AuroraEditor 
License: MIT license

#editor #macos #swift #ide 

iOS: Most Usable tools for iOS Penetration Testing

iOS/macOS penetration testing cheatsheet

macOS Quick Look plugin for iOS & OSX developers

https://github.com/ealeksandrov/ProvisionQL – Generate amazing preview for .ipa .app .appex .mobileprovision .provisionprofile

iOS / macOS obfuscation

https://github.com/obfuscator-llvm/obfuscator/wiki – ollvm

Static analyze

Jailbreak

Little h4ck for sslpinning bypass (help in some cases when sslkillswitch useless)

• Configure burp proxy on iOS device – Visit [your_proxy_adress]:[proxy_port]/mobileassistant.deb – Download file and install
  • Via iFile
  • Via ssh like `dpkg -i path/to/mobileassistant.deb
• Respring
• Launch Mobile Assistant
• Add app in bottom panel
• Turn-on switcher next to app
• Launch your app
• Congrats

More info here NB! in some cases you may face with lack of libraries, do not replace anything manually in iOS, it may lead to infinity loop)

AppSign / Rebuild / Resign / Inject / Useful tools

Download and decrypt

Extract data

All in one (Inject > Repack > Resign > Upload)

Inject framework

Repack and resign binary

Upload and run on device

Useful tools

Slides and articles and links

Download Details:

Author: ansjdnakjdnajkd
Source Code: https://github.com/ansjdnakjdnajkd/iOS 
License: Apache-2.0 license

#macos #swift #security #ios #apple 

PixelKit: Live Graphics in Swift & Metal

PixelKit

Live Graphics for iOS, macOS and tvOS

Runs on RenderKit, powered by Metal

PixelKit combines custom shaders, metal performance shaders, core image filters and vision to create tools for real-time rendering.

Examples: Camera Effects - Green Screen 
Info: Coordinate Space - Blend Operators - Effect Convenience Funcs - High Bit Mode

CameraPIX	DepthCameraPIX	ImagePIX	VideoPIX	ScreenCapturePIX	StreamInPIX	SlopePIX

ColorPIX	CirclePIX	RectanglePIX	PolygonPIX	ArcPIX	LinePIX	GradientPIX	StackPIX

NoisePIX	TextPIX	MetalPIX	TwirlPIX	FeedbackPIX	DelayPIX	SharpenPIX	StreamOutPIX

LevelsPIX	BlurPIX	EdgePIX	ThresholdPIX	QuantizePIX	TransformPIX	KaleidoscopePIX

ChannelMixPIX	ChromaKeyPIX	CornerPinPIX	ColorShiftPIX	FlipFlopPIX	RangePIX	StarPIX

SepiaPIX	ConvertPIX	ReducePIX	ClampPIX	FreezePIX	FlarePIX	AirPlayPIX	RecordPIX

BlendPIX	CrossPIX	LookupPIX	DisplacePIX	RemapPIX	ReorderPIX	ResolutionPIX	CropPIX

BlendsPIX	LumaLevelsPIX	LumaBlurPIX	LumaTransformPIX	TimeMachinePIX	ArrayPIX

Install

Swift Package

.package(url: "https://github.com/heestand-xyz/PixelKit", from: "3.0.0")

Setup

SwiftUI

import SwiftUI
import PixelKit

struct ContentView: View {
    
    @StateObject var circlePix = CirclePIX()
    @StateObject var blurPix = BlurPIX()
    
    var body: some View {
        PixelView(pix: blurPix)
            .onAppear {
                blurPix.input = circlePix
                blurPix.radius = 0.25
            }
    }
}

UIKit

import UIKit
import PixelKit

class ViewController: UIViewController {

    override func viewDidLoad() {
        super.viewDidLoad()

        let circlePix = CirclePIX()

        let blurPix = BlurPIX()
        blurPix.input = circlePix
        blurPix.radius = 0.25

        let finalPix: PIX = blurPix
        finalPix.view.frame = view.bounds
        view.addSubview(finalPix.view)
    }
}

Resolution

In PixelKit all PIXs have a resolution. Some PIXs have defined resolutions (default to .auto) and some PIXs have derived resolutions.

The .auto resolution will fill up the view and get the correct resolution based on the view size. If a view is 100x100 points, the resolution will be 200x200 pixels on macOS and 300x300 pixels on iPhone.

Import the resolution package to work with resolutions:

import Resolution

You can multiply and divide resolutions with a CGFloat or Int.

There are predefined resolutions like ._1080p & ._4K.

Rendered Image

.renderedImage // UIImage or NSImage
.renderedTexture // MTLTexture

Example: Camera Effects

import SwiftUI
import PixelKit

class ViewModel: ObservableObject {
    
    let camera: CameraPIX
    let levels: LevelsPIX
    let colorShift: ColorShiftPIX
    let blur: BlurPIX
    let circle: CirclePIX
    
    let finalPix: PIX
    
    init() {
        
        camera = CameraPIX()
        camera.cameraResolution = ._1080p

        levels = LevelsPIX()
        levels.input = camera
        levels.brightness = 1.5
        levels.gamma = 0.5

        colorShift = ColorShiftPIX()
        colorShift.input = levels
        colorShift.saturation = 0.5

        blur = BlurPIX()
        blur.input = colorShift
        blur.radius = 0.25

        circle = CirclePIX(at: .square(1080))
        circle.radius = 0.45
        circle.backgroundColor = .clear

        finalPix = blur & (camera * circle)
    }
}

struct ContentView: View {
    
    @StateObject var viewModel = ViewModel()
    
    var body: some View {
        PixelView(pix: viewModel.finalPix)
    }
}

This can also be done with Effect Convenience Funcs:
 

let pix = CameraPIX().pixBrightness(1.5).pixGamma(0.5).pixSaturation(0.5).pixBlur(0.25)

Remeber to add NSCameraUsageDescription to your Info.plist

Example: Green Screen

import RenderKit import PixelKit

let cityImage = ImagePIX()
cityImage.image = UIImage(named: "city")

let supermanVideo = VideoPIX()
supermanVideo.load(fileNamed: "superman", withExtension: "mov")

let supermanKeyed = ChromaKeyPIX()
supermanKeyed.input = supermanVideo
supermanKeyed.keyColor = .green

let blendPix = BlendPIX()
blendPix.blendingMode = .over
blendPix.inputA = cityImage
blendPix.inputB = supermanKeyed

let finalPix: PIX = blendPix
finalPix.view.frame = view.bounds
view.addSubview(finalPix.view)

This can also be done with Blend Operators and Effect Convenience Funcs:
 

let pix = cityImage & supermanVideo.pixChromaKey(.green)

Example: Depth Camera

import RenderKit import PixelKit

let cameraPix = CameraPIX()
cameraPix.camera = .front

let depthCameraPix = DepthCameraPIX.setup(with: cameraPix)

let levelsPix = LevelsPIX()
levelsPix.input = depthCameraPix
levelsPix.inverted = true

let lumaBlurPix = cameraPix.pixLumaBlur(pix: levelsPix, radius: 0.1)

let finalPix: PIX = lumaBlurPix
finalPix.view.frame = view.bounds
view.addSubview(finalPix.view)

The DepthCameraPIX was added in PixelKit v0.8.4 and requires an iPhone X or newer.

Note to use the setup(with:filter:) method of DepthCameraPIX.
It will take care of orientation, color and enable depth on the CameraPIX.

To gain access to depth values ouside of the 0.0 and 1.0 bounds,
enable 16 bit mode like this: PixelKit.main.render.bits = ._16

Example: Multi Camera

let cameraPix = CameraPIX()
cameraPix.camera = .back

let multiCameraPix = MultiCameraPIX.setup(with: cameraPix, camera: .front)

let movedMultiCameraPix = multiCameraPix.pixScale(by: 0.25).pixTranslate(x: 0.375 * (9 / 16), y: 0.375)

let finalPix: PIX = camearPix & movedMultiCameraPix
finalPix.view.frame = view.bounds
view.addSubview(finalPix.view)

Note MultiCameraPIX requires iOS 13.

Coordinate Space

The PixelKit coordinate space is normailzed to the vertical axis (1.0 in height) with the origin (0.0, 0.0) in the center.
Note that compared to native UIKit and SwiftUI views the vertical axis is flipped and origin is moved, this is more convinent when working with graphics in PixelKit. A full rotation is defined by 1.0

Center: CGPoint(x: 0, y: 0)
Bottom Left: CGPoint(x: -0.5 * aspectRatio, y: -0.5)
Top Right: CGPoint(x: 0.5 * aspectRatio, y: 0.5)
 

Tip: Resolution has an .aspect property:
let aspectRatio: CGFloat = Resolution._1080p.aspect

Blend Operators

A quick and convenient way to blend PIXs
These are the supported BlendingMode operators:

let blendPix = (CameraPIX() !** NoisePIX(at: .fullHD(.portrait))) * CirclePIX(at: .fullHD(.portrait))

The default global blend operator fill mode is .fit, change it like this:
PIX.blendOperators.globalPlacement = .fill

Effect Convenience Funcs

• .pixScaleResolution(to: ._1080p * 0.5) -> ResolutionPIX
• .pixScaleResolution(by: 0.5) -> ResolutionPIX
• .pixBrightness(0.5) -> LevelsPIX
• .pixDarkness(0.5) -> LevelsPIX
• .pixContrast(0.5) -> LevelsPIX
• .pixGamma(0.5) -> LevelsPIX
• .pixInvert() -> LevelsPIX
• .pixOpacity(0.5) -> LevelsPIX
• .pixBlur(0.5) -> BlurPIX
• .pixEdge() -> EdgePIX
• .pixThreshold(at: 0.5) -> ThresholdPIX
• .pixQuantize(by: 0.5) -> QuantizePIX
• .pixPosition(at: CGPoint(x: 0.5, y: 0.5)) -> TransformPIX
• .pixRotatate(by: 0.5) -> TransformPIX
• .pixRotatate(byRadians: .pi) -> TransformPIX
• .pixRotatate(byDegrees: 180) -> TransformPIX
• .pixScale(by: 0.5) -> TransformPIX
• .pixKaleidoscope() -> KaleidoscopePIX
• .pixTwirl(0.5) -> TwirlPIX
• .pixSwap(.red, .blue) -> ChannelMixPIX
• .pixChromaKey(.green) -> ChromaKeyPIX
• .pixHue(0.5) -> ColorShiftPIX
• .pixSaturation(0.5) -> ColorShiftPIX
• .pixCrop(CGRect(x: 0.25, y 0.25, width: 0.5, height: 0.5)) -> CropPIX
• .pixFlipX() -> FlipFlopPIX
• .pixFlipY() -> FlipFlopPIX
• .pixFlopLeft() -> FlipFlopPIX
• .pixFlopRight() -> FlipFlopPIX
• .pixRange(inLow: 0.0, inHigh: 0.5, outLow: 0.5, outHigh: 1.0) -> RangePIX
• .pixRange(inLow: .clear, inHigh: .gray, outLow: .gray, outHigh: .white) -> RangePIX
• .pixSharpen() -> SharpenPIX
• .pixSlope() - > SlopePIX
• .pixVignetting(radius: 0.5, inset: 0.25, gamma: 0.5) -> LumaLevelsPIX
• .pixLookup(pix: pixB, axis: .x) -> LookupPIX
• .pixLumaBlur(pix: pixB, radius: 0.5) -> LumaBlurPIX
• .pixLumaLevels(pix: pixB, brightness: 2.0) -> LumaLevelsPIX
• .pixDisplace(pix: pixB, distance: 0.5) -> DisplacePIX
• .pixRemap(pix: pixB) -> RemapPIX

Keep in mind that these funcs will create new PIXs.
Be careful of overloading GPU memory, some funcs create several PIXs.

High Bit Mode

Some effects like DisplacePIX and SlopePIX can benefit from a higher bit depth.
The default is 8 bits. Change it like this: PixelKit.main.render.bits = ._16

Enable high bit mode before you create any PIXs.

Note resources do not support higher bits yet.
There is currently there is some gamma offset with resources.

MetalPIXs

let metalPix = MetalPIX(at: ._1080p, code:
    """
    pix = float4(u, v, 0.0, 1.0);
    """
)

let metalEffectPix = MetalEffectPIX(code:
    """
    float gamma = 0.25;
    pix = pow(input, 1.0 / gamma);
    """
)
metalEffectPix.input = CameraPIX()

let metalMergerEffectPix = MetalMergerEffectPIX(code:
    """
    pix = pow(inputA, 1.0 / inputB);
    """
)
metalMergerEffectPix.inputA = CameraPIX()
metalMergerEffectPix.inputB = ImagePIX("img_name")

let metalMultiEffectPix = MetalMultiEffectPIX(code:
    """
    float4 inPixA = inTexs.sample(s, uv, 0);
    float4 inPixB = inTexs.sample(s, uv, 1);
    float4 inPixC = inTexs.sample(s, uv, 2);
    pix = inPixA + inPixB + inPixC;
    """
)
metalMultiEffectPix.inputs = [ImagePIX("img_a"), ImagePIX("img_b"), ImagePIX("img_c")]

Uniforms:

var lumUniform = MetalUniform(name: "lum")
let metalPix = MetalPIX(at: ._1080p, code:
    """
    pix = float4(in.lum, in.lum, in.lum, 1.0);
    """,
    uniforms: [lumUniform]
)
lumUniform.value = 0.5

Notes:

• To gain camera access, on macOS, check Camera in the App Sandbox in your Xcode project settings under Capabilities.

inspired by TouchDesigner created by Anton Heestand XYZ

Download Details:

Author: Heestand-xyz
Source Code: https://github.com/heestand-xyz/PixelKit 
License: MIT license

#swift #macos #ios #graphic 

ChatGPT: ChatGPT Desktop Application (Mac, Windows and Linux)

ChatGPT

ChatGPT Desktop Application (Mac, Windows and Linux)

📦 Install

• 📝 Update Log
• 🕒 History versions...

Windows

ChatGPT_0.10.1_x64_en-US.msi: Direct download installer

Use winget:

# install the latest version
winget install --id=lencx.ChatGPT -e

# install the specified version
winget install --id=lencx.ChatGPT -e --version 0.10.0

Note: If the installation path and application name are the same, it will lead to conflict (#142)

Mac

• ChatGPT_0.10.1_x64.dmg: Direct download installer
• ChatGPT.app.tar.gz: Download the .app installer
• Homebrew
  Or you can install with Homebrew (Cask):

brew tap lencx/chatgpt https://github.com/lencx/ChatGPT.git
brew install --cask chatgpt --no-quarantine

• Also, if you keep a Brewfile, you can add something like this:

repo = "lencx/chatgpt"
tap repo, "https://github.com/#{repo}.git"
cask "chatgpt", args: { "no-quarantine": true }

Linux

• chat-gpt_0.10.1_amd64.deb: Download .deb installer, advantage small size, disadvantage poor compatibility
• chat-gpt_0.10.1_amd64.AppImage: Works reliably, you can try it if .deb fails to run
• Available on AUR with the package name chatgpt-desktop-bin, and you can use your favourite AUR package manager to install it.

📢 Announcement

ChatGPT Prompts!

This is a major and exciting update. It works like a Telegram bot command and helps you quickly populate custom models to make chatgpt work the way you want it to. This project has taken a lot of my spare time, so if it helps you, please help spread the word or star it would be a great encouragement to me. I hope I can keep updating it and adding more interesting features.

How does it work?

You can look at awesome-chatgpt-prompts to find interesting features to import into the app. You can also use Sync Prompts to sync all in one click, and if you don't want certain prompts to appear in your slash commands, you can disable them.

• In the chatgpt text input area, type a character starting with / to bring up the command prompt, press the spacebar, and it will fill the input area with the text associated with the command by default (note: if it contains multiple command prompts, it will only select the first one as the fill, you can keep typing until the first prompted command is the one you want, then press the spacebar.
• Or use the mouse to click on one of the multiple commands). When the fill is complete, you simply press the Enter key.
• Under the slash command, use the tab key to modify the contents of the {q} tag (only single changes are supported #54). Use the keyboard ⇧ (arrow up) and ⇩ (arrow down) keys to select the slash command.

✨ Features

• Multi-platform: macOS Linux Windows
• Export ChatGPT history (PNG, PDF and Markdown)
• The main window and system tray support custom URLs to wrap any website into a desktop application
• Automatic application upgrade notification
• Common shortcut keys
• System tray hover window
• Powerful menu items
• Support for slash commands and their configuration (can be configured manually or synchronized from a file #55)
• Customize global shortcuts (#108)
• Pop-up Search (#122 mouse selected content, no more than 400 characters): The application is built using Tauri, and due to its security restrictions, some of the action buttons will not work, so we recommend going to your browser.

#️⃣ MenuItem

• Preferences
  • Theme - Light, Dark, System (Only macOS and Windows are supported).
  • Stay On Top: The window is stay on top of other windows.
  • Titlebar: Whether to display the titlebar, supported by macOS only.
  • Hide Dock Icon (#35): Hide application icons from the Dock(support macOS only).
    • Right-click on the SystemTray to open the menu, then click Show Dock Icon in the menu item to re-display the application icon in the Dock (SystemTrayMenu -> Show Dock Icon).
  • Inject Script: Using scripts to modify pages.
  • Control Center: The control center of ChatGPT application, it will give unlimited imagination to the application.
    • Theme, Stay On Top, Titlebar, ...
    • User Agent (#17): Custom user agent, which may be required in some scenarios. The default value is the empty string.
    • Switch Origin (#14): Switch the site source address, the default is https://chat.openai.com, please make sure the mirror site UI is the same as the original address. Otherwise, some functions may not be available.
  • Go to Config: Open the configuration file directory (path: ~/.chatgpt/*).
  • Clear Config: Clear the configuration file (path: ~/.chatgpt/*), dangerous operation, please backup the data in advance.
  • Restart ChatGPT: Restart the application, for example: the program is stuck or the injection script can take effect by restarting the application after editing.
  • Awesome ChatGPT: Recommended Related Resources.
• Edit - Undo, Redo, Cut, Copy, SelectAll, ...
• View - Go Back, Go Forward, Scroll to Top of Screen, Scroll to Bottom of Screen, Refresh the Screen, ...
• Help
  • Update Log: ChatGPT changelog.
  • Report Bug: Report a bug or give feedback.
  • Toggle Developer Tools: Developer debugging tools.

⚙️ Application Configuration

• [.chatgpt] - application configuration root folder
  • chat.conf.json - preferences configuration
  • chat.awesome.json - Custom URL lists, similar to browser bookmarks. Any URL can be used as the main window or tray window (Control Conter -> Awesome)
  • chat.model.json - prompts configuration，contains three parts:
    • user_custom - Requires manual data entry (Control Conter -> Language Model -> User Custom)
    • sync_prompts - Synchronizing data from f/awesome-chatgpt-prompts (Control Conter -> Language Model -> Sync Prompts)
    • sync_custom - Synchronize custom json and csv file data, support local and remote (Control Conter -> Language Model -> Sync Custom)
  • chat.model.cmd.json - filtered (whether to enable) and sorted slash commands
  • [cache_model] - caching model data
    • chatgpt_prompts.json - Cache sync_prompts data
    • user_custom.json - Cache user_custom data
    • ae6cf32a6f8541b499d6bfe549dbfca3.json - Randomly generated file names, cache sync_custom data
    • 4f695d3cfbf8491e9b1f3fab6d85715c.json - Randomly generated file names, cache sync_custom data
    • bd1b96f15a1644f7bd647cc53073ff8f.json - Randomly generated file names, cache sync_custom data

Sync Custom

Currently, only json and csv are supported for synchronizing custom files, and the following formats need to be met, otherwise the application will be abnormal：

JSON format:

[
  {
    "cmd": "a",
    "act": "aa",
    "prompt": "aaa aaa aaa"
  },
  {
    "cmd": "b",
    "act": "bb",
    "prompt": "bbb bbb bbb"
  }
]

CSV format

"cmd","act","prompt"
"a","aa","aaa aaa aaa"
"b","bb","bbb bbb bbb"

📌 TODO

• Control Center enhancement
• Pop-up Search enhancement
• ...

👀 Preview

❓FAQ

Can't open ChatGPT

If you cannot open the application after the upgrade, please try to clear the configuration file, which is in the ~/.chatgpt/* directory.

Out of sync login status between multiple windows

If you have already logged in in the main window, but the system tray window shows that you are not logged in, you can fix it by restarting the application (Menu -> Preferences -> Restart ChatGPT).

Is it safe?

It's safe, just a wrapper for OpenAI ChatGPT website, no other data transfer exists (you can check the source code).

Developer cannot be verified?

• Open a Mac app from an unidentified developer

How do I build it?

PreInstall

• Rust (Required)
• Node.js (Required)
• VS Code (Optional)
  • rust-analyzer
  • tauri

Start

# step1:
git clone https://github.com/lencx/ChatGPT.git

# step2:
cd ChatGPT

# step3: install deps
yarn

# step4:
yarn dev

# step5:
# bundle path: src-tauri/target/release/bundle
yarn build

• The distDir configuration is set to "../dist" but this path doesn't exist
• Error A public key has been found, but no private key. Make sure to set TAURI_PRIVATE_KEY environment variable.

❤️ Thanks

• The core implementation of the share button code was copied from the @liady extension with some modifications.
• Thanks to the Awesome ChatGPT Prompts repository for inspiring the custom command function for this application.

Download Details:

Author: lencx
Source Code: https://github.com/lencx/ChatGPT 
License: Apache-2.0 license

#desktopapp #windows #macos #linux #rust #application 

Rome: Carthage Cache for S3, Minio, Ceph, Google Storage, Artifactory

Rome 

Rome is a tool that allows developers on Apple platforms to use:

• Amazon's S3
• Minio
• Ceph
• other S3 compatible object stores
• or/and a local folder
• your own custom engine

as a shared cache for frameworks built with Carthage.

Get Rome

Using Homebrew

$ brew tap tmspzz/tap https://github.com/tmspzz/homebrew-tap.git
$ brew install tmspzz/homebrew-tap/rome

Using CocoaPods

Simply add the following line to your Podfile:

pod 'Rome'

This will download Rome to the Pods/ folder during your next pod install execution and will allow you to invoke it via ${PODS_ROOT}/Rome/rome in your Script Build Phases.

Manual

The Rome binary is also attached as a zip to each release on the releases page here on GitHub.

Using Rome? Let me know by opening an issue and I will gladly add you to the user list.

Use Rome with fastlane

You can integrate Rome into your fastlane automation with the fastlane plugin for Rome.

The problem

Suppose you're working a number of frameworks for your project and want to share those with your team. A great way to do so is to use Carthage and have team members point the Cartfile to the new framework version (or branch, tag, commit) and run carthage update.

Unfortunately this will require them to build from scratch the new framework. This is particularly annoying if the dependency tree for that framework is big and / or takes a long time to build.

The solution

Use a cache. The first team member (or a CI) can build the framework and share it, while all other developers can get it from the cache with no waiting time.

Workflow

The Rome's workflow changes depending if you are the producer (i.e. the first person in your team to build the framework) or the consumer.

Producer workflow

$ vi Cartfile # point to the new version of the framework
$ carthage update && rome upload

If you are running Rome in the context of a framework and want to upload the current framework see CurrentMap.

Consumer workflow

$ vi Cartfile # point to the new version of the framework if necessary
$ carthage update --no-build && rome download

or

$ vi Cartfile.resolved # point to the new version of the framework
$ rome download

If you are running Rome in the context of a framework and want to download the current framework see CurrentMap.

CI workflow

A CI can be both consumer and producer.

A simple workflow for using Rome on a continuous integration should resemble the following:

• get available artifacts
• check if any artifacts are missing
• build missing artifacts if any
• upload build artifacts to the cache if needed

You can use the fastlane plugin for Rome to implement a CI workflow too.

If you are running Rome in the context of a framework and want to upload or download the current framework see CurrentMap.

--cache-builds workflow (recommended)

This workflow relies on .version files being produced by Carthage. This means that you have to invoke carthage with --cache-builds for this to work.

In code:

#!/bin/bash
rome download --platform iOS # download missing frameworks (or copy from local cache)
carthage bootstrap --platform iOS --cache-builds # build dependencies missing a .version file or that where not found in the cache
rome list --missing --platform iOS | awk '{print $1}' | xargs -I {} rome upload "{}" --platform iOS # upload what is missing

List workflow

This workflow relies on querying the cache to check for missing dependencies and then selectively telling Carthage what to build. This flow is more fragile as Carthage will refuse to build indirect dependencies.

In code:

#!/bin/bash
rome download --platform iOS # download missing frameworks (or copy from local cache)
rome list --missing --platform iOS | awk '{print $1}' | xargs -I {} carthage bootstrap "{}" --platform iOS --cache-builds # list what is missing and update/build if needed
rome list --missing --platform iOS | awk '{print $1}' | xargs -I {} rome upload "{}" --platform iOS # upload what is missing

If no frameworks are missing, the awk pipe to carthage will fail and the rest of the command will not be executed. This avoids rebuilding all dependencies or uploading artifacts already present in the cache.

Set up

If you plan to use Amazon's S3 as a cache, then follow the next three steps:

• First you need a .aws/credentials file in your home folder. This is used to specify your AWS Credentials.
• Second you need a .aws/config file in your home folder. This is used to specify the AWS Region.
• Third you need a Romefile in the project where you want to use Rome. At the same level where the Cartfile is.

If you just want to use only a local folder as a cache then:

• You need a Romefile in the project where you want to use Rome. At the same level where the Cartfile is. Since 0.17.1.49, if you want to place the Romefile elsewhere or name it differently use --romefile <path-to-romefile> when running rome <COMMAND>.

Setting up AWS credentials

Since version 0.2.0.0 Rome will expect to find credentials either as environment variables $AWS_ACCESS_KEY_ID and $AWS_SECRET_ACCESS_KEY or in a file at .aws/credentials. This aligns Rome behavior to other tools that use Amazon's SDK. See Amazon's blogpost on the topic.

In your home folder create a .aws/credentials like the following

[default]
aws_access_key_id = ACCESS_KEY
aws_secret_access_key = SECRET_KEY

this should look something like

[default]
aws_access_key_id = AGIAJQARMD67CE3DTKHA
aws_secret_access_key = TedRV2/dFkBr1H3D7xuPsF9+CBHTjK0NKrJuoVs8

these will be the credentials that Rome will use to access S3 on your behalf. To use configurations other than the default profile set the $AWS_PROFILE environment variable to your desired profile.

Since version 0.21.0.58 Rome also supports privilege escalation via Amazon STS by specifying role_arn and source_profile in ~/.aws/config

Selecting the AWS Region

In your home folder create a .aws/config like the following

[default]
region = us-east-1

To use configurations other than the default profile set the $AWS_PROFILE environment variable to your desired profile.

Alternatively the AWS Region can also be specified by setting an AWS_REGION environment variable.

Setting up endpoint override

To your .aws/config in the profile section you wish to use, add an endpoint key like so

[default]
region = us-east-1
endpoint = https://my.minio.host:9091

Do not remove the region key.

Default port for https endpoints is 443 if the port is left unspecified.

Default port for http endpoints is 9000 if the port is left unspecified.

Alternatively the endpoint can also be specified by setting an AWS_ENDPOINT environment variable.

Custom Engine

You can write your own script that Rome will use as engine to execute upload/download/list commands. You start by specifying the path to a script or executable in your Romefile as shown in the example structure. Rome will invoke the specified script or executable with three commands and different parameters based on the action to perform:

• ./script.sh upload local-path remote-path
• ./script.sh download remote-path local-path
• ./script.sh list remote-path

For example, if your Romefile specifies engine: script.sh, Rome will execute the following command when uploading/downloading/listing a framework:

./script.sh upload Alamofire/iOS/Alamofire.framework-4.8.2.zip Alamofire/iOS/Alamofire.framework-4.8.2.zip
./script.sh download Alamofire/iOS/Alamofire.framework-4.8.2.zip Alamofire/iOS/Alamofire.framework-4.8.2.zip
./script.sh list Alamofire/iOS/Alamofire.framework-4.8.2.zip

The script should take the given remote-path, carry out its logic to retrieve the artifact and place it at local-path. Please refer to the cache structure definition for more information on the cache is constructed.

For an example of a custom engine, take a look at engine.sh which is used in the integration tests to simply copy artifacts in a different directory. Infinite uses cases are opened by using a custom engine, such as uploading artifacts to any non-compatible S3 storage system.

Other example engines:

• Google Storage

Romefile

About the format

Since version 0.17.0.48 the Romefile is in YAML format. Rome can still read the INI Romefile, for now.

Sucessive release might abandon compatibility.

Feature support that require additions or changes to the Romefile won't be supported in INI.

You can migrate your Romefile to YAML by running rome utils migrate-romefile.

If you are looking for the documention prior to 0.17.0.48, check the wiki

Purpose

The Romefile has three purposes:

1. Specifies what caches to use - cache key. This key is required.
2. Allows to use custom name mappings between repository names and framework names - repositoryMap key. This key is optional and can be omitted.
3. Allows to ignore certain framework names - ignoreMap key. This key is optional and can be omitted.
4. Allows to specify the current framework's name(s) - currentMap key. This key is optional and can be omitted.

Structure

A Romefile is made of 4 objects, of which only one, the cache, is required.

• A cache definition object
• A repositoryMap made of a list of Romefile Entry
• An ignoreMap made of a list of Romefile Entry
• A currentMap made of a list of Romefile Entry

Each Romefile Entry is made of:

• A name
• A type which can be static or dynamic
• A set of supported platforms including iOS, Mac, tvOS, watchOS

A Romefile looks like this:

cache: # required
  # at least one of the following is required:
  local: ~/Library/Caches/Rome # optional and can be combined with either a `s3Bucket` or `engine`
  s3Bucket: ios-dev-bucket # optional and can be combined with `local`
  engine: script.sh # optional and can be combined with `local`
                    
repositoryMap: # optional
- better-dog-names: # entry that does not follow
                    # the "Organization/FrameworkName" convention.
  - name: DogFramework # required
    type: static # optional, defaults to dynamic
- HockeySDK-iOS:
  - name: HockeySDK
    platforms: [iOS] # optional, all platforms if empty
- awesome-framework-for-cat-names:
  - name: CatFramework
    type: dynamic
ignoreMap:
- GDCWebServer:
  - name: GDCWebServer
currentMap:
- animal-names-framework:
  - name: AnimalNames

Cache

The cache must contain at least one between:

• the name of the S3 Bucket to upload/download to/from. The key s3Bucket is optional.
• the path to local directory to use as an additional cache. The key local is optional.
• the path to a custom engine to use as an additional cache. The key engine is optional.

cache: # required
  local: ~/Library/Caches/Rome # optional
                               # at least one between `local`, `s3bucket` and `engine` is required
  s3Bucket: ios-dev-bucket # optional
                           # at least one between `local`, `s3bucket` and `engine` is required
  engine: script.sh        # optional
                           # at least one between `local`, `s3bucket` and `engine` is required

This is already a viable Romefile.

RepositoryMap

This contains the mappings of repository and framework names. This is particularly useful if dependecies are not on GitHub or don't respect the "Organization/FrameworkName" convention.

Example:

Suppose you have the following in your Cartfile

github "Alamofire/Alamofire" ~> 4.3.0
github "bitstadium/HockeySDK-iOS" "3.8.6"
git "http://stash.myAnimalStartup.com/scm/iossdk/awesome-framework-for-cat-names.git" ~> 3.3.1
git "http://stash.myAnimalStartup.com/scm/iossdk/better-dog-names.git" ~> 0.4.4

which translates to the following Cartfile.resolved

github "Alamofire/Alamofire" "4.3.0"
github "bitstadium/HockeySDK-iOS" "3.8.6"
git "http://stash.myAnimalStartup.com/scm/iossdk/awesome-framework-for-cat-names.git" "3.3.1"
git "http://stash.myAnimalStartup.com/scm/iossdk/better-dog-names.git" "0.4.4"

but your framework names are actually HockeySDK, CatFramework and DogFramework as opposed to HockeySDK-iOS, awesome-framework-for-cat-names and better-dog-names.

simply add a repositoryMap key to your Romefile and specify the following mapping:

cache:
  local: ~/Library/Caches/Rome 
repositoryMap:
- better-dog-names: # this is the Romefile Entry for  `better-dog-names`
  - name: DogFramework
    type: static
    platforms: [iOS, Mac]
- HockeySDK-iOS: # this is the Romefile Entry for  `HockeySDK-iOS`
  - name: HockeySDK
    platforms: [iOS]
- awesome-framework-for-cat-names:  # this is the Romefile Entry for  `awesome-framework-for-cat-names`
  - name: CatFramework
  - type: dynamic

Note that it was not necessary to add Alamofire as it respects the "Organization/FrameworkName" convention.

IgnoreMap

This contains the mappings of repository and framework names that should be ignored. This is particularly useful in case not all your Cartfile.resolved entries produce a framework.

Some repositories use Carthage as a simple mechanism to include other git repositories that do not produce frameworks. Even Carthage itself does this, to include xcconfigs.

Example:

Suppose you have the following in your Cartfile

github "Quick/Nimble"
github "jspahrsummers/xcconfigs"

xcconfigs can be ignored by Rome by adding an ignoreMap key in the Romefile

ignoreMap:
- xcconfigs:
  - name: xcconfigs

Each entry in the IgnoreMap is also a Romefile Entry and supports all keys.

CurrentMap

By default the currentMap is not used. Specify --no-skip-current as a command line option to use it. It is supported by Rome versions greater than 0.18.x.y and can be specified only in YAML.

The currentMap contains the mappings of repository and framework name(s) that describe the current framework. This is particularly useful if you want to use Rome in the context of a framework. It is the equivalent of Carthage's --no-skip-current.

currentMap:
- Alamofire:
  - name: Alamofire

Each entry in the currentMap is also a Romefile Entry and supports all keys.

The currentMap is subject to the ignores specified in the ignoreMap. If you explicitly specify names of frameworks to upload, download on the command line, you don't need to pass --no-skip-current to use the currentMap. Just specify the name(s) of the current framework.

The version of the current framework is determined by

git describe --tags --exact-match `git rev-parse HEAD`

If the commands does not resolve to any tag, the HEAD commit hash from git rev-parse HEAD is used as version.

In order for --no-skip-current to work, make sure to run carthage archive to create an artifact to cache.

Multiple Aliases

Suppose you have a framework Framework that builds two targets, t1 and t2, Rome can handle both targets by specifying

repositoryMap:
- Framework:
  - name: t1
  - name: t2

Note: if ANY of the aliases is missing on S3, the entire entry will be reported as missing when running rome list [--missing]

Multiple aliases are supported in ignoreMap too

Static Frameworks

Since version 0.30.1 Carthage has support for Static Frameworks. To indicate that one of the aliases is a Static Framework, modify the repositoryMap like so:

repositoryMap:
- Framework:
  - name: t1
    type: static
  - name: t2

If left unspecified, an alias is a Dynamic Framework by default.

Platforms

Since version 0.17.1.49 Rome allows you to specify what platforms are supported for a specific Romefile Entry. This serves a differet purpose than the command line option --platforms.

repositoryMap:
- Framework: 
  - name: t1
    type: static
    platforms: [iOS, Mac]
  - name: t2

The above means that t1 is only available for iOS and Mac. The --platforms command line options can be used to futher limit the Rome command to a specific subset of the supported platfroms.

Cache Structure

The following describes the structure of the cache that Rome creates and manages.

By default frameworks, dSYMs and .bcsymbolmaps are placed in the cache (local and/or remote) according to the following convention:

<git-repository-name>/<platform>/<framework-name>.framework(.dSYM)-(static-)<version-hash>.zip
<git-repository-name>/<platform>/<bcsymbolmap-hash>.bcsymbolmap-(static-)<version-hash>.zip

Carthage version files are placed at:

<git-repository-name>/.<framework-name>.version-(static-)<version-hash>

For example the cache for the Cartfile.resolved in RepositoryMap would look like the following

/Users/blender/Library/Caches/Rome/
├── HockeySDK-iOS
│   └── iOS
│       ├── HockeySDK.framework-3.8.6.zip
│       ├── HockeySDK.framework.dSYM-3.8.6.zip
│       └── D034377A-B469-3819-97A7-1DC0AA293AC3.bcsymbolmap
├── awesome-framework-for-cat-names
│        ├── iOS
│        │   ├── CatFramework.framework-883eea474e3932607988d4e74bf50c9799bfd99a.zip
│        │   └── CatFramework.framework.dSYM-883eea474e3932607988d4e74bf50c9799bfd99a.zip
│        ├── tvOS
│        │   ├── CatFramework.framework-883eea474e3932607988d4e74bf50c9799bfd99a.zip
│        │   └── CatFramework.framework.dSYM-883eea474e3932607988d4e74bf50c9799bfd99a.zip
│        └── .CatFramework.version-883eea474e3932607988d4e74bf50c9799bfd99a
└─── better-dog-names
        ├── iOS
        │   ├── DogFramework.framework-v4.0.0.zip
        │   └── DogFramework.framework.dSYM-v4.0.0.zip
        ├── Mac
        │   ├── DogFramework.framework-v4.0.0.zip
        │   └── DogFramework.framework.dSYM-v4.0.0.zip
        └── .DogFramework.version-v4.0.0

Cache Prefix

Since version 0.12.0.31 Rome supports prefixes for top level directories in your caches. You can append --cache-prefix MY_PREFIX to all commands. This simply means that both the framework/dSYM and .version file conventional locations can be prefixed by another directory of your choosing. Thus the conventions become:

<MY_PREFIX>/<git-repository-name>/<platform>/<framework-name>.framework(.dSYM)-<version-hash>.zip

and

<MY_PREFIX>/<git-repository-name>/.<framework-name>.version-<version-hash>

This is particularly useful when the need to cache frameworks at the same version but build with different versions of the compiler arises.

Suppose you want to cache v4.0.0 of DogFramework build for Swift2.1/Swift3.1/Swift3.2/Swift4. Once built you can upload each build with the same version number to a separate top level directory in the cache via the --cache-prefix option.

Thus running for the Swift2.1 build

$ rome upload better-dog-names --platform iOS # note there is no prefix here

and running for the the Swift3.2 build

$ rome upload better-dog-names --platform iOS --cache-prefix Swift_3_2

would lead to the following cache structure

/Users/blender/Library/Caches/Rome/
├── better-dog-names
│   ├── iOS
│   │   ├── DogFramework.framework-v4.0.0.zip
│   │   └── DogFramework.framework.dSYM-v4.0.0.zip
│   └── .DogFramework.version-v4.0.0-iOS
└── Swift_3_2
    └── better-dog-names
        ├── iOS
        │   ├── DogFramework.framework-v4.0.0.zip
        │   └── DogFramework.framework.dSYM-v4.0.0.zip
        └── .DogFramework.version-v4.0.0-iOS

Usage

Getting help:

$ rome --help
Cache tool for Carthage

Usage: rome COMMAND [-v]

Available options:
  -h,--help                Show this help text
  --version                Prints the version information
  -v                       Show verbose output

Available commands:
  upload                   Uploads frameworks and dSYMs contained in the local
                           Carthage/Build/<platform> to S3, according to the
                           local Cartfile.resolved
  download                 Downloads and unpacks in Carthage/Build/<platform>
                           frameworks and dSYMs found in S3, according to the
                           local Cartfile.resolved
  list                     Lists frameworks in the cache and reports cache
                           misses/hits, according to the local
                           Cartfile.resolved. Ignores dSYMs.
  utils                    A series of utilities to make life easier. `rome
                           utils --help` to know more

Uploading

Uploading one or more frameworks, corresponding dSYMs, .bcsymbolmaps and Carthage version files if present (an empty list of frameworks will upload all frameworks found in Cartfile.resolved):

Referring to the Cartfile.resolved in RepositoryMap

$ rome upload Alamofire
Uploaded Alamofire to: Alamofire/iOS/Alamofire.framework-4.3.0.zip
Uploaded Alamofire.dSYM to: Alamofire/iOS/Alamofire.framework.dSYM-4.3.0.zip
Uploaded Alamofire to: Alamofire/tvOS/Alamofire.framework-4.3.0.zip
Uploaded Alamofire.dSYM to: Alamofire/tvOS/Alamofire.framework.dSYM-4.3.0.zip
Uploaded Alamofire to: Alamofire/watchOS/Alamofire.framework-4.3.0.zip
Uploaded Alamofire.dSYM to: Alamofire/watchOS/Alamofire.framework.dSYM-4.3.0.zip

Uploading for a specific platform (all platforms are uploaded by default):

$ rome upload --platform ios Alamofire
Uploaded Alamofire to: Alamofire/iOS/Alamofire.framework-4.3.0.zip
Uploaded Alamofire.dSYM to: Alamofire/iOS/Alamofire.framework.dSYM-4.3.0.zip

If a local cache is specified in your Romefile and you wish to ignore it pass --skip-local-cache on the command line.

Since version 0.20.0.56, if you are on a fast Internet connection you can use the --concurrently flag to maximise concurrency for the operation and maximise bandwith use. Using the --concurrently flag should result in a x3 speedup.

Downloading

Downloading one or more frameworks, corresponding dSYMs, .bcsymbolmaps and Carthage version files if present (an empty list of frameworks will download all frameworks found in Cartfile.resolved):

Referring to the Cartfile.resolved in RepositoryMap

$ rome download Alamofire
Downloaded Alamofire from: Alamofire/iOS/Alamofire.framework-4.3.0.zip
Downloaded Alamofire.dSYM from: Alamofire/iOS/Alamofire.framework.dSYM-4.3.0.zip
Error downloading Alamofire : The specified key does not exist.
Error downloading Alamofire.dSYM : The specified key does not exist.
Downloaded Alamofire from: Alamofire/tvOS/Alamofire.framework-4.3.0.zip
Downloaded Alamofire.dSYM from: Alamofire/tvOS/Alamofire.framework.dSYM-4.3.0.zip
Downloaded Alamofire from: Alamofire/watchOS/Alamofire.framework-4.3.0.zip
Downloaded Alamofire.dSYM from: Alamofire/watchOS/Alamofire.framework.dSYM-4.3.0.zip

Downloading for a specific platform (all platforms are downloaded by default):

$ rome download --platform ios,watchos Alamofire
Downloaded Alamofire from: Alamofire/iOS/Alamofire.framework-4.3.0.zip
Downloaded Alamofire.dSYM from: Alamofire/iOS/Alamofire.framework.dSYM-4.3.0.zip
Downloaded Alamofire from: Alamofire/watchOS/Alamofire.framework-4.3.0.zip
Downloaded Alamofire.dSYM from: Alamofire/watchOS/Alamofire.framework.dSYM-4.3.0.zip

If a local cache is specified in your Romefile and you wish to ignore it pass --skip-local-cache on the command line.

Since version 0.20.0.56, if you are on a fast Internet connection you can use the --concurrently flag to maximise concurrency for the operation and maximise bandwith use. Using the --concurrently flag should result in a x3 speedup.

Listing

Listing frameworks and reporting on their availability:

$ rome list
Alamofire 4.3.0 : +iOS -macOS +tvOS +watchOS
ResearchKit 1.4.1 : +iOS -macOS -tvOS -watchOS

Listing only frameworks present in the cache:

$ rome list --present
Alamofire 4.3.0 : +iOS +tvOS +watchOS
ResearchKit 1.4.1 : +iOS

Listing only frameworks missing from the cache:

$ rome list --missing
Alamofire 4.3.0 : -macOS
ResearchKit 1.4.1 : -macOS -tvOS -watchOS

Listing frameworks missing for specific platforms:

$ rome list --missing --platform watchos,tvos
ResearchKit 1.4.1 : -tvOS -watchOS

Forwarding a list of missing frameworks to Carthage for building:

$ rome list --missing --platform ios | awk '{print $1}' | xargs -I {} carthage build "{}" --platform ios
*** xcodebuild output can be found in ...

Since version 0.13.0.33 list results can also be printed as JSON by specifying --print-format=JSON

Note: list completely ignores dSYMs, bcsymbolmap and Carthage version files. If a dSYM or a Carthage version file is missing, the corresponding framework is still reported as present.

Utils

A collection of utilities to make life easier.

migrate-romefile

Migrate the Romefile from INI to YAML in place, by running:

rome utils migrate-romefile

Troubleshooting & FAQ

Getting "Image not found" when running an application using binaries

Implicit dependencies of frameworks when using binaries are not copied over by Xcode automatically despite "Always Embed Standard Libraries" set to YES (see 56).

Here is an example with ReactiveCocoa, which depends on CoreLocation and MapKit. If ReactiveCocoa is built via Carthage or as a Xcode subproject, CoreLocation and MapKit are copied into the app's bundle. On the other hand, when using the binary, Xcode has no clue of that and does not copy the necessary frameworks even if "Always Embed Standard Libraries" is set to yes.

To fix that, add an explicit import statement to one of your files:

// Implicit ReactiveCocoa Dependencies

import CoreLocation
import MapKit

Supporting multiple Swift Versions

Storing artifacts or a the same famework at different Swift versions can be achieved by specifying a cache prefix when using any Rome command like so:

$ rome upload --platform iOS --cache-prefix Swift3 Alamofire
$ rome download --platform iOS --cache-prefix Swift3 Alamofire
$ rome list --platform iOS --cache-prefix Swift3

If you prefer a more accurate way of generating cache prefixes for different swift versions consider using the following:

--cache-prefix `xcrun swift --version | head -1 | sed 's/.*\((.*)\).*/\1/' | tr -d "()" | tr " " "-"`

The specified prefix is prepended to the git repository name in the caches. Using a local cache path like ~/Library/Caches/Rome will store Alamofire from the example above at ~/Library/Caches/Rome/Swift3/Alamofire

See Cache Structure and Cache Prefix for an in depth explanation.

Developing

1. Install Stack via homebrew brew install stack
2. Clone the repo git clone https://github.com/tmspzz/Rome.git
3. cd Rome && stack build
4. Optional: Install brittany via stack install brittany
5. Optional: Install hlint via stack install hlint

IDE

1. Optional: If you use VIM install haskell-vim-how
2. Optional: If you use Visual Studio Code install Haskero

Releasing

1. Increase the version number in Rome.cabal
2. Increase the version number in app/Main.hs
3. Increase the version number in Rome.podspec
4. Commit
5. Create a new pre-release on Github
6. Attach the zipped binary
7. Promote to release
8. Run pod trunk push Rome.podspec
9. Update the homebrew formula
10. Run bundle exec github_changelog_generator -u tmspzz -p Rome -t <YourGitHubToken>
11. Commit CHANGELOG.md

Presentations and Tutorials

Video tutorial on Rome given at CocoaHeads Berlin and slides

• Features
• Usage
• Romefile

AppUnite article - comparison of popular approaches to building dependencies with Carthage by Szymon Mrozek

Download Details:

Author: tmspzz
Source Code: https://github.com/tmspzz/Rome 
License: MIT license

#swift #macos #aws #ios #framework 

SwiftCoroutine: Swift Coroutines for IOS, MacOS and Linux

SwiftCoroutine

Many languages, such as Kotlin, Go, JavaScript, Python, Rust, C#, C++ and others, already have coroutines support that makes the async/await pattern implementation possible. This feature is not yet supported in Swift, but this can be improved by a framework without the need to change the language.

Main features

• It is the first implementation of coroutines for Swift with iOS, macOS and Linux support.
• It includes futures and channels that supplement coroutines for more flexibility.
• It is completely lock-free and uses only atomic primitives for synchronizations.

Motivation

Asynchronous programming is usually associated with callbacks. It is quite convenient until there are too many of them and they start nesting. Then it's called a pyramid of doom or even callback hell.

Another problem of asynchronous programming is error handling, because Swift's natural error handling mechanism cannot be used.

What about Rx and other such frameworks?

There are many other frameworks that make it easy to use asynchronous code, such as Combine, RxSwift, PromiseKit and so on. They use other approaches that have some drawbacks:

• Similar to callbacks, you also need to create chained calls, that’s why you can’t normally use loops, exception handling, etc.
• Usually you need to learn a complex new API with hundreds of methods.
• Instead of working with the actual data, you need to operate with some wrappers all the time.
• Chaining of errors can be really complicated to handle.

Async/await

The async/await pattern is an alternative that allows an asynchronous, non-blocking function to be structured in a way similar to an ordinary synchronous function.

It is already well-established in other programming languages and is an evolution in asynchronous programming. The implementation of this pattern is possible thanks to coroutines.

Let’s have a look at the example with coroutine inside of which await() suspends it and resumes when the result is available without blocking the thread.

//executes coroutine on the main thread
DispatchQueue.main.startCoroutine {
    
    //extension that returns CoFuture<(data: Data, response: URLResponse)>
    let dataFuture = URLSession.shared.dataTaskFuture(for: imageURL)
    
    //await CoFuture result that suspends coroutine and doesn't block the thread
    let data: Data = try dataFuture.await().data

    //create UIImage from the data
    guard let image = UIImage(data: data) else { return }
    
    //execute heavy task on global queue and await the result without blocking the thread
    let thumbnail: UIImage = try DispatchQueue.global().await { image.makeThumbnail() }

    //set image in UIImageView on the main thread
    self.imageView.image = thumbnail
    
}

Documentation

API documentation

Requirements

• Support only 64-bit architectures
• iOS 10+ / macOS 10.12+ / Ubuntu
• Xcode 10.4+
• Swift 5.2+

Installation

• It's available through the Swift Package Manager for iOS, macOS and Linux.
• It's available through CocoaPods for iOS and macOS.

Working with SwiftCoroutine

Coroutines

A coroutine is a computation that can be suspended and resumed at a later time without blocking a thread. Coroutines build upon regular functions and can be executed on any scheduler with a possibility to switch among them during execution.

Key benefits

• Suspend instead of block. The main advantage of coroutines is the ability to suspend their execution at some point without blocking a thread and resuming later on.
• Fast context switching. Switching between coroutines is much faster than switching between threads as it does not require the involvement of operating system.
• Asynchronous code in synchronous manner. The use of coroutines allows an asynchronous, non-blocking function to be structured in a manner similar to an ordinary synchronous function. And even though coroutines can run in multiple threads, your code will still look consistent and therefore easy to understand.

Usage

The coroutines API design is as minimalistic as possible. It consists of the CoroutineScheduler protocol that describes how to schedule coroutines (DispatchQueue already conforms it), and the Coroutine structure with utility methods. This API is enough to do amazing things.

The following example shows the usage of await() inside a coroutine to wrap asynchronous calls.

//execute coroutine on the main thread
DispatchQueue.main.startCoroutine {
    
    //await URLSessionDataTask response without blocking the thread
    let (data, response, error) = try Coroutine.await { callback in
        URLSession.shared.dataTask(with: url, completionHandler: callback).resume()
    }
    
    . . . use response on the main thread . . . 
}

Here's how we can conform NSManagedObjectContext to CoroutineScheduler for launching coroutines on it.

extension NSManagedObjectContext: CoroutineScheduler {

    func scheduleTask(_ task: @escaping () -> Void) {
        perform(task)
    }
    
}

//execute coroutine on the main thread
DispatchQueue.main.startCoroutine {
    let context: NSManagedObjectContext //context with privateQueueConcurrencyType
    let request: NSFetchRequest<NSDictionary> //some complex request

    //execute request on the context without blocking the main thread
    let result: [NSDictionary] = try context.await { try context.fetch(request) }
}

Futures and Promises

A future is a read-only holder for a result that will be provided later and the promise is the provider of this result. They represent the eventual completion or failure of an asynchronous operation.

The futures and promises approach itself has become an industry standart. It is a convenient mechanism to synchronize asynchronous code. But together with coroutines, it takes the usage of asynchronous code to the next level and has become a part of the async/await pattern. If coroutines are a skeleton, then futures and promises are its muscles.

Main features

• Performance. It is much faster than most of other futures and promises implementations.
• Awaitable. You can await the result inside the coroutine.
• Cancellable. You can cancel the whole chain as well as handle it and complete the related actions.

Usage

Futures and promises are represented by the corresponding CoFuture class and its CoPromise subclass.

//wraps some async func with CoFuture
func makeIntFuture() -> CoFuture<Int> {
    let promise = CoPromise<Int>()
    someAsyncFunc { int in
        promise.success(int)
    }
    return promise
}

It allows to start multiple tasks in parallel and synchronize them later with await().

//create CoFuture<Int> that takes 2 sec. from the example above 
let future1: CoFuture<Int> = makeIntFuture()

//execute coroutine on the global queue and returns CoFuture<Int> with future result
let future2: CoFuture<Int> = DispatchQueue.global().coroutineFuture {
    try Coroutine.delay(.seconds(3)) //some work that takes 3 sec.
    return 6
}

//execute coroutine on the main thread
DispatchQueue.main.startCoroutine {
    let sum: Int = try future1.await() + future2.await() //will await for 3 sec.
    self.label.text = "Sum is \(sum)"
}

It's very easy to transform or compose CoFutures into a new one.

let array: [CoFuture<Int>]

//create new CoFuture<Int> with sum of future results
let sum = CoFuture { try array.reduce(0) { try $0 + $1.await() } }

Channels

Futures and promises provide a convenient way to transfer a single value between coroutines. Channels provide a way to transfer a stream of values. Conceptually, a channel is similar to a queue that allows to suspend a coroutine on receive if it is empty, or on send if it is full.

This non-blocking primitive is widely used in such languages as Go and Kotlin, and it is another instrument that improves working with coroutines.

Usage

To create channels, use the CoChannel class.

//create a channel with a buffer which can store only one element
let channel = CoChannel<Int>(capacity: 1)

DispatchQueue.global().startCoroutine {
    for i in 0..<100 {
        //imitate some work
        try Coroutine.delay(.seconds(1))
        //sends a value to the channel and suspends coroutine if its buffer is full
        try channel.awaitSend(i)
    }
    
    //close channel when all values are sent
    channel.close()
}

DispatchQueue.global().startCoroutine {
    //receives values until closed and suspends a coroutine if it's empty
    for i in channel.makeIterator() {
        print("Receive", i)
    }
    
    print("Done")
}

Scope

All launched coroutines, CoFutures and CoChannels, usually do not need to be referenced. They are deinited after their execution. But often there is a need to complete them earlier, when they are no longer needed. For this, CoFuture and CoChannel have methods for canceling.

CoScope makes it easier to manage the life cycle of these objects. It allows you to keep weak references to them and cancel if necessary or on deinit.

Usage

You can add coroutines, CoFutures, CoChannels and other CoCancellable to CoScope to cancel them when they are no longer needed or on deinit.

class ViewController: UIViewController {

    let scope = CoScope() //will cancel all objects on `cancel()` or deinit
    
    func performSomeWork() {
        //create new `CoChannel` and add to `CoScope`
        let channel = makeSomeChannel().added(to: scope)
        
        //execute coroutine and add to `CoScope`
        DispatchQueue.main.startCoroutine(in: scope) { [weak self] in
            for item in channel.makeIterator() {
                try self?.performSomeWork(with: item)
            }
        }
    }
    
    func performSomeWork(with item: Item) throws {
        //create new `CoFuture` and add to `CoScope`
        let future = makeSomeFuture(item).added(to: scope)
        
        let result = try future.await()
        . . . do some work using result . . .
    }

}

Download Details:

Author: Belozierov
Source Code: https://github.com/belozierov/SwiftCoroutine 
License: MIT license

#swift #macos #linux #ios #async 

FlexibleImage: A Simple Way to Play with The Image!

FlexibleImage

FlexibleImage is implemented with the hope that anyone could easily develop an app that provides features such as Camera Filter and Theme. When you write code in the "Method Chaining" style, the effect is applied in the appropriate order.

You may want to see Examples section first if you'd like to see the actual code.

💡 Usage

Code

import UIKit

import FlexibleImage

/// Generate Example
let image1 = UIImage
    .circle(
        color: UIColor.blue,
        size: CGSize(width: 100, height: 100)
    )!
    
    .adjust()
    .offset(CGPoint(x: 25, y: 0))
    .margin(UIEdgeInsets(top: 5, left: 5, bottom: 5, right: 5))
    .padding(UIEdgeInsets(top: 15, left: 15, bottom: 15, right: 15))
    .normal(color: UIColor.white)
    .border(color: UIColor.red, lineWidth: 5, radius: 50)
    .image()!
    
    .adjust()
    .background(color: UIColor.darkGray)
    .image()


/// Effect Example
let image2 = UIImage(named: "macaron.jpg")!
    .adjust()
    .outputSize(CGSize(width: 250, height: 250))
    .exclusion(color: UIColor(red: 0, green: 0, blue: 0.352941176, alpha: 1.0))
    .linearDodge(color: UIColor(red: 0.125490196, green: 0.058823529, blue: 0.192156863, alpha: 1.0))
    .hardMix(color: UIColor(red: 0.3, green: 0.3, blue: 0.3, alpha: 1.0))
    .image()


/// Mix Example
let image3 = image2!.adjust()
    .append(
        image1!.adjust()
            .outputSize(CGSize(width: 250, height: 250))
            .alpha(0.5)
    )
    .image()

/// Clipping Example
let image4 = image3!.adjust()
    .corner(CornerType(25))
    .image()

/// Pipeline
let pipeline = ImagePipeline()
        .exclusion(color: UIColor(red: 0, green: 0, blue: 0.352941176, alpha: 1.0))
        .linearDodge(color: UIColor(red: 0.125490196, green: 0.058823529, blue: 0.192156863, alpha: 1.0))

let image5 = pipeline.image(image2)
let image6 = pipeline.image(image1)

Playground

Use CocoaPods command $ pod try FlexibleImage to try Playground!

🏗 Installation

CocoaPods (For iOS 8+ projects)

KWDrawerController is available on CocoaPods. Add the following to your Podfile:

/// Swift 3
pod 'FlexibleImage', '~> 1.7'

/// Swift 4
pod 'FlexibleImage', '~> 1.9'

Carthage (For iOS 8+ projects)

github "kawoou/FlexibleImage" ~> 1.9

Manually

You can either simply drag and drop the Sources folder into your existing project.

📕 Supported Features

Common

Filter

Blend

Post-processing

Generate

Pipeline (ImagePipeline class)

🎁 Example

• iOS APP Example
• iOS Playground Example
• macOS App Example
• macOS Playground Example
• tvOS App Example
• tvOS Playground Example

🏷 Changelog

• 1.0
  • First Release.
• 1.1
  • Add to clipping corner radius.
• 1.2
  • Support tvOS, macOS.
• 1.3
  • Support watchOS.
  • Added monochrome, sepia, vibrance, solarize, posterize filters.
  • Update resize methods.
• 1.4
  • Add blur filter.
  • Optimize build time.
  • Setup TravisCI
  • Support carthage.
• 1.5
  • Support Metal depending on the situation.
  • Added brightness, chromaKey, swizzling, contrast, gamma filters.
• 1.6 (Hotfix!)
  • Fix issue Metal library path on Cocoapods.
• 1.7
  • Pipelined implementation for stream processing.
  • Fix rendering bug due to image orientation (Thanks to Kwonyoon Kang)
• 1.8
  • Support for Swift 4 and Xcode 9
• 1.9
  • Support for Swift 4.1 and Xcode 9.3
• 1.10
  • Support for Swift 4.2 and Xcode 10

💻 Requirements

• iOS 8.0+
• tvOS 9.0+
• macOS 10.10+
• watchOS 2.0+
• Swift 3.0+

Download Details:

Author: Kawoou
Source Code: https://github.com/kawoou/FlexibleImage 
License: MIT license

#swift #macos #ios

Master The Command Line, in one Page

The Art of Command Line

Note: I'm planning to revise this and looking for a new co-author to help with expanding this into a more comprehensive guide. While it's very popular, it could be broader and a bit deeper. If you like to write and are close to being an expert on this material and willing to consider helping, please drop me a note at josh (0x40) holloway.com. –jlevy, Holloway. Thank you!

Fluency on the command line is a skill often neglected or considered arcane, but it improves your flexibility and productivity as an engineer in both obvious and subtle ways. This is a selection of notes and tips on using the command-line that we've found useful when working on Linux. Some tips are elementary, and some are fairly specific, sophisticated, or obscure. This page is not long, but if you can use and recall all the items here, you know a lot.

This work is the result of many authors and translators. Some of this originally appeared on Quora, but it has since moved to GitHub, where people more talented than the original author have made numerous improvements. Please submit a question if you have a question related to the command line. Please contribute if you see an error or something that could be better!

Meta

Scope:

• This guide is for both beginners and experienced users. The goals are breadth (everything important), specificity (give concrete examples of the most common case), and brevity (avoid things that aren't essential or digressions you can easily look up elsewhere). Every tip is essential in some situation or significantly saves time over alternatives.
• This is written for Linux, with the exception of the "macOS only" and "Windows only" sections. Many of the other items apply or can be installed on other Unices or macOS (or even Cygwin).
• The focus is on interactive Bash, though many tips apply to other shells and to general Bash scripting.
• It includes both "standard" Unix commands as well as ones that require special package installs -- so long as they are important enough to merit inclusion.

Notes:

• To keep this to one page, content is implicitly included by reference. You're smart enough to look up more detail elsewhere once you know the idea or command to Google. Use apt, yum, dnf, pacman, pip or brew (as appropriate) to install new programs.
• Use Explainshell to get a helpful breakdown of what commands, options, pipes etc. do.

Basics

Learn basic Bash. Actually, type man bash and at least skim the whole thing; it's pretty easy to follow and not that long. Alternate shells can be nice, but Bash is powerful and always available (learning only zsh, fish, etc., while tempting on your own laptop, restricts you in many situations, such as using existing servers).

Learn at least one text-based editor well. The nano editor is one of the simplest for basic editing (opening, editing, saving, searching). However, for the power user in a text terminal, there is no substitute for Vim (vi), the hard-to-learn but venerable, fast, and full-featured editor. Many people also use the classic Emacs, particularly for larger editing tasks. (Of course, any modern software developer working on an extensive project is unlikely to use only a pure text-based editor and should also be familiar with modern graphical IDEs and tools.)

Finding documentation:

• Know how to read official documentation with man (for the inquisitive, man man lists the section numbers, e.g. 1 is "regular" commands, 5 is files/conventions, and 8 are for administration). Find man pages with apropos.
• Know that some commands are not executables, but Bash builtins, and that you can get help on them with help and help -d. You can find out whether a command is an executable, shell builtin or an alias by using type command.
• curl cheat.sh/command will give a brief "cheat sheet" with common examples of how to use a shell command.

Learn about redirection of output and input using > and < and pipes using |. Know > overwrites the output file and >> appends. Learn about stdout and stderr.

Learn about file glob expansion with * (and perhaps ? and [...]) and quoting and the difference between double " and single ' quotes. (See more on variable expansion below.)

Be familiar with Bash job management: &, ctrl-z, ctrl-c, jobs, fg, bg, kill, etc.

Know ssh, and the basics of passwordless authentication, via ssh-agent, ssh-add, etc.

Basic file management: ls and ls -l (in particular, learn what every column in ls -l means), less, head, tail and tail -f (or even better, less +F), ln and ln -s (learn the differences and advantages of hard versus soft links), chown, chmod, du (for a quick summary of disk usage: du -hs *). For filesystem management, df, mount, fdisk, mkfs, lsblk. Learn what an inode is (ls -i or df -i).

Basic network management: ip or ifconfig, dig, traceroute, route.

Learn and use a version control management system, such as git.

Know regular expressions well, and the various flags to grep/egrep. The -i, -o, -v, -A, -B, and -C options are worth knowing.

Learn to use apt-get, yum, dnf or pacman (depending on distro) to find and install packages. And make sure you have pip to install Python-based command-line tools (a few below are easiest to install via pip).

Everyday use

In Bash, use Tab to complete arguments or list all available commands and ctrl-r to search through command history (after pressing, type to search, press ctrl-r repeatedly to cycle through more matches, press Enter to execute the found command, or hit the right arrow to put the result in the current line to allow editing).

In Bash, use ctrl-w to delete the last word, and ctrl-u to delete the content from current cursor back to the start of the line. Use alt-b and alt-f to move by word, ctrl-a to move cursor to beginning of line, ctrl-e to move cursor to end of line, ctrl-k to kill to the end of the line, ctrl-l to clear the screen. See man readline for all the default keybindings in Bash. There are a lot. For example alt-. cycles through previous arguments, and alt-* expands a glob.

Alternatively, if you love vi-style key-bindings, use set -o vi (and set -o emacs to put it back).

For editing long commands, after setting your editor (for example export EDITOR=vim), ctrl-x ctrl-e will open the current command in an editor for multi-line editing. Or in vi style, escape-v.

To see recent commands, use history. Follow with !n (where n is the command number) to execute again. There are also many abbreviations you can use, the most useful probably being !$ for last argument and !! for last command (see "HISTORY EXPANSION" in the man page). However, these are often easily replaced with ctrl-r and alt-..

Go to your home directory with cd. Access files relative to your home directory with the ~ prefix (e.g. ~/.bashrc). In sh scripts refer to the home directory as $HOME.

To go back to the previous working directory: cd -.

If you are halfway through typing a command but change your mind, hit alt-# to add a # at the beginning and enter it as a comment (or use ctrl-a, #, enter). You can then return to it later via command history.

Use xargs (or parallel). It's very powerful. Note you can control how many items execute per line (-L) as well as parallelism (-P). If you're not sure if it'll do the right thing, use xargs echo first. Also, -I{} is handy. Examples:

      find . -name '*.py' | xargs grep some_function
      cat hosts | xargs -I{} ssh root@{} hostname

pstree -p is a helpful display of the process tree.

Use pgrep and pkill to find or signal processes by name (-f is helpful).

Know the various signals you can send processes. For example, to suspend a process, use kill -STOP [pid]. For the full list, see man 7 signal

Use nohup or disown if you want a background process to keep running forever.

Check what processes are listening via netstat -lntp or ss -plat (for TCP; add -u for UDP) or lsof -iTCP -sTCP:LISTEN -P -n (which also works on macOS).

See also lsof and fuser for open sockets and files.

See uptime or w to know how long the system has been running.

Use alias to create shortcuts for commonly used commands. For example, alias ll='ls -latr' creates a new alias ll.

Save aliases, shell settings, and functions you commonly use in ~/.bashrc, and arrange for login shells to source it. This will make your setup available in all your shell sessions.

Put the settings of environment variables as well as commands that should be executed when you login in ~/.bash_profile. Separate configuration will be needed for shells you launch from graphical environment logins and cron jobs.

Synchronize your configuration files (e.g. .bashrc and .bash_profile) among various computers with Git.

Understand that care is needed when variables and filenames include whitespace. Surround your Bash variables with quotes, e.g. "$FOO". Prefer the -0 or -print0 options to enable null characters to delimit filenames, e.g. locate -0 pattern | xargs -0 ls -al or find / -print0 -type d | xargs -0 ls -al. To iterate on filenames containing whitespace in a for loop, set your IFS to be a newline only using IFS=$'\n'.

In Bash scripts, use set -x (or the variant set -v, which logs raw input, including unexpanded variables and comments) for debugging output. Use strict modes unless you have a good reason not to: Use set -e to abort on errors (nonzero exit code). Use set -u to detect unset variable usages. Consider set -o pipefail too, to abort on errors within pipes (though read up on it more if you do, as this topic is a bit subtle). For more involved scripts, also use trap on EXIT or ERR. A useful habit is to start a script like this, which will make it detect and abort on common errors and print a message:

     set -euo pipefail
      trap "echo 'error: Script failed: see failed command above'" ERR

• In Bash scripts, subshells (written with parentheses) are convenient ways to group commands. A common example is to temporarily move to a different working directory, e.g.

      # do something in current dir
      (cd /some/other/dir && other-command)
      # continue in original dir

In Bash, note there are lots of kinds of variable expansion. Checking a variable exists: ${name:?error message}. For example, if a Bash script requires a single argument, just write input_file=${1:?usage: $0 input_file}. Using a default value if a variable is empty: ${name:-default}. If you want to have an additional (optional) parameter added to the previous example, you can use something like output_file=${2:-logfile}. If $2 is omitted and thus empty, output_file will be set to logfile. Arithmetic expansion: i=$(( (i + 1) % 5 )). Sequences: {1..10}. Trimming of strings: ${var%suffix} and ${var#prefix}. For example if var=foo.pdf, then echo ${var%.pdf}.txt prints foo.txt.

Brace expansion using {...} can reduce having to re-type similar text and automate combinations of items. This is helpful in examples like mv foo.{txt,pdf} some-dir (which moves both files), cp somefile{,.bak} (which expands to cp somefile somefile.bak) or mkdir -p test-{a,b,c}/subtest-{1,2,3} (which expands all possible combinations and creates a directory tree). Brace expansion is performed before any other expansion.

The order of expansions is: brace expansion; tilde expansion, parameter and variable expansion, arithmetic expansion, and command substitution (done in a left-to-right fashion); word splitting; and filename expansion. (For example, a range like {1..20} cannot be expressed with variables using {$a..$b}. Use seq or a for loop instead, e.g., seq $a $b or for((i=a; i<=b; i++)); do ... ; done.)

The output of a command can be treated like a file via <(some command) (known as process substitution). For example, compare local /etc/hosts with a remote one:

      diff /etc/hosts <(ssh somehost cat /etc/hosts)

• When writing scripts you may want to put all of your code in curly braces. If the closing brace is missing, your script will be prevented from executing due to a syntax error. This makes sense when your script is going to be downloaded from the web, since it prevents partially downloaded scripts from executing:

{
      # Your code here
}

• A "here document" allows redirection of multiple lines of input as if from a file:

cat <<EOF
input
on multiple lines
EOF

In Bash, redirect both standard output and standard error via: some-command >logfile 2>&1 or some-command &>logfile. Often, to ensure a command does not leave an open file handle to standard input, tying it to the terminal you are in, it is also good practice to add </dev/null.

Use man ascii for a good ASCII table, with hex and decimal values. For general encoding info, man unicode, man utf-8, and man latin1 are helpful.

Use screen or tmux to multiplex the screen, especially useful on remote ssh sessions and to detach and re-attach to a session. byobu can enhance screen or tmux by providing more information and easier management. A more minimal alternative for session persistence only is dtach.

In ssh, knowing how to port tunnel with -L or -D (and occasionally -R) is useful, e.g. to access web sites from a remote server.

It can be useful to make a few optimizations to your ssh configuration; for example, this ~/.ssh/config contains settings to avoid dropped connections in certain network environments, uses compression (which is helpful with scp over low-bandwidth connections), and multiplex channels to the same server with a local control file:

      TCPKeepAlive=yes
      ServerAliveInterval=15
      ServerAliveCountMax=6
      Compression=yes
      ControlMaster auto
      ControlPath /tmp/%r@%h:%p
      ControlPersist yes

A few other options relevant to ssh are security sensitive and should be enabled with care, e.g. per subnet or host or in trusted networks: StrictHostKeyChecking=no, ForwardAgent=yes

Consider mosh an alternative to ssh that uses UDP, avoiding dropped connections and adding convenience on the road (requires server-side setup).

To get the permissions on a file in octal form, which is useful for system configuration but not available in ls and easy to bungle, use something like

      stat -c '%A %a %n' /etc/timezone

For interactive selection of values from the output of another command, use percol or fzf.

For interaction with files based on the output of another command (like git), use fpp (PathPicker).

For a simple web server for all files in the current directory (and subdirs), available to anyone on your network, use: python -m SimpleHTTPServer 7777 (for port 7777 and Python 2) and python -m http.server 7777 (for port 7777 and Python 3).

For running a command as another user, use sudo. Defaults to running as root; use -u to specify another user. Use -i to login as that user (you will be asked for your password).

For switching the shell to another user, use su username or su - username. The latter with "-" gets an environment as if another user just logged in. Omitting the username defaults to root. You will be asked for the password of the user you are switching to.

Know about the 128K limit on command lines. This "Argument list too long" error is common when wildcard matching large numbers of files. (When this happens alternatives like find and xargs may help.)

For a basic calculator (and of course access to Python in general), use the python interpreter. For example,

>>> 2+3
5

Processing files and data

To locate a file by name in the current directory, find . -iname '*something*' (or similar). To find a file anywhere by name, use locate something (but bear in mind updatedb may not have indexed recently created files).

For general searching through source or data files, there are several options more advanced or faster than grep -r, including (in rough order from older to newer) ack, ag ("the silver searcher"), and rg (ripgrep).

To convert HTML to text: lynx -dump -stdin

For Markdown, HTML, and all kinds of document conversion, try pandoc. For example, to convert a Markdown document to Word format: pandoc README.md --from markdown --to docx -o temp.docx

If you must handle XML, xmlstarlet is old but good.

For JSON, use jq. For interactive use, also see jid and jiq.

For YAML, use shyaml.

For Excel or CSV files, csvkit provides in2csv, csvcut, csvjoin, csvgrep, etc.

For Amazon S3, s3cmd is convenient and s4cmd is faster. Amazon's aws and the improved saws are essential for other AWS-related tasks.

Know about sort and uniq, including uniq's -u and -d options -- see one-liners below. See also comm.

Know about cut, paste, and join to manipulate text files. Many people use cut but forget about join.

Know about wc to count newlines (-l), characters (-m), words (-w) and bytes (-c).

Know about tee to copy from stdin to a file and also to stdout, as in ls -al | tee file.txt.

For more complex calculations, including grouping, reversing fields, and statistical calculations, consider datamash.

Know that locale affects a lot of command line tools in subtle ways, including sorting order (collation) and performance. Most Linux installations will set LANG or other locale variables to a local setting like US English. But be aware sorting will change if you change locale. And know i18n routines can make sort or other commands run many times slower. In some situations (such as the set operations or uniqueness operations below) you can safely ignore slow i18n routines entirely and use traditional byte-based sort order, using export LC_ALL=C.

You can set a specific command's environment by prefixing its invocation with the environment variable settings, as in TZ=Pacific/Fiji date.

Know basic awk and sed for simple data munging. See One-liners for examples.

To replace all occurrences of a string in place, in one or more files:

      perl -pi.bak -e 's/old-string/new-string/g' my-files-*.txt

• To rename multiple files and/or search and replace within files, try repren. (In some cases the rename command also allows multiple renames, but be careful as its functionality is not the same on all Linux distributions.)

      # Full rename of filenames, directories, and contents foo -> bar:
      repren --full --preserve-case --from foo --to bar .
      # Recover backup files whatever.bak -> whatever:
      repren --renames --from '(.*)\.bak' --to '\1' *.bak
      # Same as above, using rename, if available:
      rename 's/\.bak$//' *.bak

• As the man page says, rsync really is a fast and extraordinarily versatile file copying tool. It's known for synchronizing between machines but is equally useful locally. When security restrictions allow, using rsync instead of scp allows recovery of a transfer without restarting from scratch. It also is among the fastest ways to delete large numbers of files:

mkdir empty && rsync -r --delete empty/ some-dir && rmdir some-dir

For monitoring progress when processing files, use pv, pycp, pmonitor, progress, rsync --progress, or, for block-level copying, dd status=progress.

Use shuf to shuffle or select random lines from a file.

Know sort's options. For numbers, use -n, or -h for handling human-readable numbers (e.g. from du -h). Know how keys work (-t and -k). In particular, watch out that you need to write -k1,1 to sort by only the first field; -k1 means sort according to the whole line. Stable sort (sort -s) can be useful. For example, to sort first by field 2, then secondarily by field 1, you can use sort -k1,1 | sort -s -k2,2.

If you ever need to write a tab literal in a command line in Bash (e.g. for the -t argument to sort), press ctrl-v [Tab] or write $'\t' (the latter is better as you can copy/paste it).

The standard tools for patching source code are diff and patch. See also diffstat for summary statistics of a diff and sdiff for a side-by-side diff. Note diff -r works for entire directories. Use diff -r tree1 tree2 | diffstat for a summary of changes. Use vimdiff to compare and edit files.

For binary files, use hd, hexdump or xxd for simple hex dumps and bvi, hexedit or biew for binary editing.

Also for binary files, strings (plus grep, etc.) lets you find bits of text.

For binary diffs (delta compression), use xdelta3.

To convert text encodings, try iconv. Or uconv for more advanced use; it supports some advanced Unicode things. For example:

      # Displays hex codes or actual names of characters (useful for debugging):
      uconv -f utf-8 -t utf-8 -x '::Any-Hex;' < input.txt
      uconv -f utf-8 -t utf-8 -x '::Any-Name;' < input.txt
      # Lowercase and removes all accents (by expanding and dropping them):
      uconv -f utf-8 -t utf-8 -x '::Any-Lower; ::Any-NFD; [:Nonspacing Mark:] >; ::Any-NFC;' < input.txt > output.txt

To split files into pieces, see split (to split by size) and csplit (to split by a pattern).

Date and time: To get the current date and time in the helpful ISO 8601 format, use date -u +"%Y-%m-%dT%H:%M:%SZ" (other options are problematic). To manipulate date and time expressions, use dateadd, datediff, strptime etc. from dateutils.

Use zless, zmore, zcat, and zgrep to operate on compressed files.

File attributes are settable via chattr and offer a lower-level alternative to file permissions. For example, to protect against accidental file deletion the immutable flag: sudo chattr +i /critical/directory/or/file

Use getfacl and setfacl to save and restore file permissions. For example:

   getfacl -R /some/path > permissions.txt
   setfacl --restore=permissions.txt

• To create empty files quickly, use truncate (creates sparse file), fallocate (ext4, xfs, btrfs and ocfs2 filesystems), xfs_mkfile (almost any filesystems, comes in xfsprogs package), mkfile (for Unix-like systems like Solaris, Mac OS).

System debugging

For web debugging, curl and curl -I are handy, or their wget equivalents, or the more modern httpie.

To know current cpu/disk status, the classic tools are top (or the better htop), iostat, and iotop. Use iostat -mxz 15 for basic CPU and detailed per-partition disk stats and performance insight.

For network connection details, use netstat and ss.

For a quick overview of what's happening on a system, dstat is especially useful. For broadest overview with details, use glances.

To know memory status, run and understand the output of free and vmstat. In particular, be aware the "cached" value is memory held by the Linux kernel as file cache, so effectively counts toward the "free" value.

Java system debugging is a different kettle of fish, but a simple trick on Oracle's and some other JVMs is that you can run kill -3 <pid> and a full stack trace and heap summary (including generational garbage collection details, which can be highly informative) will be dumped to stderr/logs. The JDK's jps, jstat, jstack, jmap are useful. SJK tools are more advanced.

Use mtr as a better traceroute, to identify network issues.

For looking at why a disk is full, ncdu saves time over the usual commands like du -sh *.

To find which socket or process is using bandwidth, try iftop or nethogs.

The ab tool (comes with Apache) is helpful for quick-and-dirty checking of web server performance. For more complex load testing, try siege.

For more serious network debugging, wireshark, tshark, or ngrep.

Know about strace and ltrace. These can be helpful if a program is failing, hanging, or crashing, and you don't know why, or if you want to get a general idea of performance. Note the profiling option (-c), and the ability to attach to a running process (-p). Use trace child option (-f) to avoid missing important calls.

Know about ldd to check shared libraries etc — but never run it on untrusted files.

Know how to connect to a running process with gdb and get its stack traces.

Use /proc. It's amazingly helpful sometimes when debugging live problems. Examples: /proc/cpuinfo, /proc/meminfo, /proc/cmdline, /proc/xxx/cwd, /proc/xxx/exe, /proc/xxx/fd/, /proc/xxx/smaps (where xxx is the process id or pid).

When debugging why something went wrong in the past, sar can be very helpful. It shows historic statistics on CPU, memory, network, etc.

For deeper systems and performance analyses, look at stap (SystemTap), perf, and sysdig.

Check what OS you're on with uname or uname -a (general Unix/kernel info) or lsb_release -a (Linux distro info).

Use dmesg whenever something's acting really funny (it could be hardware or driver issues).

If you delete a file and it doesn't free up expected disk space as reported by du, check whether the file is in use by a process: lsof | grep deleted | grep "filename-of-my-big-file"

One-liners

A few examples of piecing together commands:

• It is remarkably helpful sometimes that you can do set intersection, union, and difference of text files via sort/uniq. Suppose a and b are text files that are already uniqued. This is fast, and works on files of arbitrary size, up to many gigabytes. (Sort is not limited by memory, though you may need to use the -T option if /tmp is on a small root partition.) See also the note about LC_ALL above and sort's -u option (left out for clarity below).

      sort a b | uniq > c   # c is a union b
      sort a b | uniq -d > c   # c is a intersect b
      sort a b b | uniq -u > c   # c is set difference a - b

• Pretty-print two JSON files, normalizing their syntax, then coloring and paginating the result:

      diff <(jq --sort-keys . < file1.json) <(jq --sort-keys . < file2.json) | colordiff | less -R

Use grep . * to quickly examine the contents of all files in a directory (so each line is paired with the filename), or head -100 * (so each file has a heading). This can be useful for directories filled with config settings like those in /sys, /proc, /etc.

Summing all numbers in the third column of a text file (this is probably 3X faster and 3X less code than equivalent Python):

     awk '{ x += $3 } END { print x }' myfile

• To see sizes/dates on a tree of files, this is like a recursive ls -l but is easier to read than ls -lR:

     find . -type f -ls

• Say you have a text file, like a web server log, and a certain value that appears on some lines, such as an acct_id parameter that is present in the URL. If you want a tally of how many requests for each acct_id:

      egrep -o 'acct_id=[0-9]+' access.log | cut -d= -f2 | sort | uniq -c | sort -rn

To continuously monitor changes, use watch, e.g. check changes to files in a directory with watch -d -n 2 'ls -rtlh | tail' or to network settings while troubleshooting your wifi settings with watch -d -n 2 ifconfig.

Run this function to get a random tip from this document (parses Markdown and extracts an item):

      function taocl() {
        curl -s https://raw.githubusercontent.com/jlevy/the-art-of-command-line/master/README.md |
          sed '/cowsay[.]png/d' |
          pandoc -f markdown -t html |
          xmlstarlet fo --html --dropdtd |
          xmlstarlet sel -t -v "(html/body/ul/li[count(p)>0])[$RANDOM mod last()+1]" |
          xmlstarlet unesc | fmt -80 | iconv -t US
      }

Obscure but useful

expr: perform arithmetic or boolean operations or evaluate regular expressions

m4: simple macro processor

yes: print a string a lot

cal: nice calendar

env: run a command (useful in scripts)

printenv: print out environment variables (useful in debugging and scripts)

look: find English words (or lines in a file) beginning with a string

cut, paste and join: data manipulation

fmt: format text paragraphs

pr: format text into pages/columns

fold: wrap lines of text

column: format text fields into aligned, fixed-width columns or tables

expand and unexpand: convert between tabs and spaces

nl: add line numbers

seq: print numbers

bc: calculator

factor: factor integers

gpg: encrypt and sign files

toe: table of terminfo entries

nc: network debugging and data transfer

socat: socket relay and tcp port forwarder (similar to netcat)

slurm: network traffic visualization

dd: moving data between files or devices

file: identify type of a file

tree: display directories and subdirectories as a nesting tree; like ls but recursive

stat: file info

time: execute and time a command

timeout: execute a command for specified amount of time and stop the process when the specified amount of time completes.

lockfile: create semaphore file that can only be removed by rm -f

logrotate: rotate, compress and mail logs.

watch: run a command repeatedly, showing results and/or highlighting changes

when-changed: runs any command you specify whenever it sees file changed. See inotifywait and entr as well.

tac: print files in reverse

comm: compare sorted files line by line

strings: extract text from binary files

tr: character translation or manipulation

iconv or uconv: conversion for text encodings

split and csplit: splitting files

sponge: read all input before writing it, useful for reading from then writing to the same file, e.g., grep -v something some-file | sponge some-file

units: unit conversions and calculations; converts furlongs per fortnight to twips per blink (see also /usr/share/units/definitions.units)

apg: generates random passwords

xz: high-ratio file compression

ldd: dynamic library info

nm: symbols from object files

ab or wrk: benchmarking web servers

strace: system call debugging

mtr: better traceroute for network debugging

cssh: visual concurrent shell

rsync: sync files and folders over SSH or in local file system

wireshark and tshark: packet capture and network debugging

ngrep: grep for the network layer

host and dig: DNS lookups

lsof: process file descriptor and socket info

dstat: useful system stats

glances: high level, multi-subsystem overview

iostat: Disk usage stats

mpstat: CPU usage stats

vmstat: Memory usage stats

htop: improved version of top

last: login history

w: who's logged on

id: user/group identity info

sar: historic system stats

iftop or nethogs: network utilization by socket or process

ss: socket statistics

dmesg: boot and system error messages

sysctl: view and configure Linux kernel parameters at run time

hdparm: SATA/ATA disk manipulation/performance

lsblk: list block devices: a tree view of your disks and disk partitions

lshw, lscpu, lspci, lsusb, dmidecode: hardware information, including CPU, BIOS, RAID, graphics, devices, etc.

lsmod and modinfo: List and show details of kernel modules.

fortune, ddate, and sl: um, well, it depends on whether you consider steam locomotives and Zippy quotations "useful"

macOS only

These are items relevant only on macOS.

Package management with brew (Homebrew) and/or port (MacPorts). These can be used to install on macOS many of the above commands.

Copy output of any command to a desktop app with pbcopy and paste input from one with pbpaste.

To enable the Option key in macOS Terminal as an alt key (such as used in the commands above like alt-b, alt-f, etc.), open Preferences -> Profiles -> Keyboard and select "Use Option as Meta key".

To open a file with a desktop app, use open or open -a /Applications/Whatever.app.

Spotlight: Search files with mdfind and list metadata (such as photo EXIF info) with mdls.

Be aware macOS is based on BSD Unix, and many commands (for example ps, ls, tail, awk, sed) have many subtle variations from Linux, which is largely influenced by System V-style Unix and GNU tools. You can often tell the difference by noting a man page has the heading "BSD General Commands Manual." In some cases GNU versions can be installed, too (such as gawk and gsed for GNU awk and sed). If writing cross-platform Bash scripts, avoid such commands (for example, consider Python or perl) or test carefully.

To get macOS release information, use sw_vers.

Windows only

These items are relevant only on Windows.

Ways to obtain Unix tools under Windows

Access the power of the Unix shell under Microsoft Windows by installing Cygwin. Most of the things described in this document will work out of the box.

On Windows 10, you can use Windows Subsystem for Linux (WSL), which provides a familiar Bash environment with Unix command line utilities.

If you mainly want to use GNU developer tools (such as GCC) on Windows, consider MinGW and its MSYS package, which provides utilities such as bash, gawk, make and grep. MSYS doesn't have all the features compared to Cygwin. MinGW is particularly useful for creating native Windows ports of Unix tools.

Another option to get Unix look and feel under Windows is Cash. Note that only very few Unix commands and command-line options are available in this environment.

Useful Windows command-line tools

You can perform and script most Windows system administration tasks from the command line by learning and using wmic.

Native command-line Windows networking tools you may find useful include ping, ipconfig, tracert, and netstat.

You can perform many useful Windows tasks by invoking the Rundll32 command.

Cygwin tips and tricks

Install additional Unix programs with the Cygwin's package manager.

Use mintty as your command-line window.

Access the Windows clipboard through /dev/clipboard.

Run cygstart to open an arbitrary file through its registered application.

Access the Windows registry with regtool.

Note that a C:\ Windows drive path becomes /cygdrive/c under Cygwin, and that Cygwin's / appears under C:\cygwin on Windows. Convert between Cygwin and Windows-style file paths with cygpath. This is most useful in scripts that invoke Windows programs.

More resources

• awesome-shell: A curated list of shell tools and resources.
• awesome-osx-command-line: A more in-depth guide for the macOS command line.
• Strict mode for writing better shell scripts.
• shellcheck: A shell script static analysis tool. Essentially, lint for bash/sh/zsh.
• Filenames and Pathnames in Shell: The sadly complex minutiae on how to handle filenames correctly in shell scripts.
• Data Science at the Command Line: More commands and tools helpful for doing data science, from the book of the same name

Disclaimer

With the exception of very small tasks, code is written so others can read it. With power comes responsibility. The fact you can do something in Bash doesn't necessarily mean you should! ;)

🌍 Čeština ∙ Deutsch ∙ Ελληνικά ∙ English ∙ Español ∙ Français ∙ Indonesia ∙ Italiano ∙ 日本語 ∙ 한국어 ∙ polski ∙ Português ∙ Română ∙ Русский ∙ Slovenščina ∙ Українська ∙ 简体中文 ∙ 繁體中文

Download Details:

Author: jlevy
Source Code: https://github.com/jlevy/the-art-of-command-line 
License: Attribution-ShareAlike 4.0 International

#windows #macos #linux #bash #documentation #unix 

ChatGPT: ChatGPT Desktop Application (Mac, Windows and Linux)

ChatGPT

ChatGPT Desktop Application (Mac, Windows and Linux)

📦 Install

• 📝 Update Log
• 🕒 History versions...

Windows

• ChatGPT_0.9.2_x64_en-US.msi: Direct download installer
• Use winget:

# install the latest version
winget install --id=lencx.ChatGPT -e

# install the specified version
winget install --id=lencx.ChatGPT -e --version 0.9.0

Note: If the installation path and application name are the same, it will lead to conflict (#142)

Mac

• ChatGPT_0.9.2_x64.dmg: Direct download installer
• ChatGPT.app.tar.gz: Download the .app installer
• Homebrew
  Or you can install with Homebrew (Cask):

brew tap lencx/chatgpt https://github.com/lencx/ChatGPT.git
brew install --cask chatgpt --no-quarantine

• Also, if you keep a Brewfile, you can add something like this:

repo = "lencx/chatgpt"
tap repo, "https://github.com/#{repo}.git"
cask "chatgpt", args: { "no-quarantine": true }

Linux

• chat-gpt_0.9.2_amd64.deb: Download .deb installer, advantage small size, disadvantage poor compatibility
• chat-gpt_0.9.2_amd64.AppImage: Works reliably, you can try it if .deb fails to run
• Available on AUR with the package name chatgpt-desktop-bin, and you can use your favourite AUR package manager to install it.

📢 Announcement

ChatGPT Prompts!

This is a major and exciting update. It works like a Telegram bot command and helps you quickly populate custom models to make chatgpt work the way you want it to. This project has taken a lot of my spare time, so if it helps you, please help spread the word or star it would be a great encouragement to me. I hope I can keep updating it and adding more interesting features.

How does it work?

You can look at awesome-chatgpt-prompts to find interesting features to import into the app. You can also use Sync Prompts to sync all in one click, and if you don't want certain prompts to appear in your slash commands, you can disable them.

• In the chatgpt text input area, type a character starting with / to bring up the command prompt, press the spacebar, and it will fill the input area with the text associated with the command by default (note: if it contains multiple command prompts, it will only select the first one as the fill, you can keep typing until the first prompted command is the one you want, then press the spacebar.
• Or use the mouse to click on one of the multiple commands). When the fill is complete, you simply press the Enter key.
• Under the slash command, use the tab key to modify the contents of the {q} tag (only single changes are supported #54). Use the keyboard ⇧ (arrow up) and ⇩ (arrow down) keys to select the slash command.

✨ Features

• Multi-platform: macOS Linux Windows
• Export ChatGPT history (PNG, PDF and Markdown)
• Automatic application upgrade notification
• Common shortcut keys
• System tray hover window
• Powerful menu items
• Support for slash commands and their configuration (can be configured manually or synchronized from a file #55)
• Customize global shortcuts (#108)
• Pop-up Search (#122 mouse selected content, no more than 400 characters): The application is built using Tauri, and due to its security restrictions, some of the action buttons will not work, so we recommend going to your browser.

#️⃣ MenuItem

• Preferences
  • Theme - Light, Dark, System (Only macOS and Windows are supported).
  • Stay On Top: The window is stay on top of other windows.
  • Titlebar: Whether to display the titlebar, supported by macOS only.
  • Hide Dock Icon (#35): Hide application icons from the Dock(support macOS only).
    • Right-click on the SystemTray to open the menu, then click Show Dock Icon in the menu item to re-display the application icon in the Dock (SystemTrayMenu -> Show Dock Icon).
  • Inject Script: Using scripts to modify pages.
  • Control Center: The control center of ChatGPT application, it will give unlimited imagination to the application.
    • Theme, Stay On Top, Titlebar, ...
    • User Agent (#17): Custom user agent, which may be required in some scenarios. The default value is the empty string.
    • Switch Origin (#14): Switch the site source address, the default is https://chat.openai.com, please make sure the mirror site UI is the same as the original address. Otherwise, some functions may not be available.
  • Go to Config: Open the configuration file directory (path: ~/.chatgpt/*).
  • Clear Config: Clear the configuration file (path: ~/.chatgpt/*), dangerous operation, please backup the data in advance.
  • Restart ChatGPT: Restart the application, for example: the program is stuck or the injection script can take effect by restarting the application after editing.
  • Awesome ChatGPT: Recommended Related Resources.
• Edit - Undo, Redo, Cut, Copy, SelectAll, ...
• View - Go Back, Go Forward, Scroll to Top of Screen, Scroll to Bottom of Screen, Refresh the Screen, ...
• Help
  • Update Log: ChatGPT changelog.
  • Report Bug: Report a bug or give feedback.
  • Toggle Developer Tools: Developer debugging tools.

⚙️ Application Configuration

• [.chatgpt] - application configuration root folder
  • chat.conf.json - preferences configuration
  • chat.model.json - prompts configuration，contains three parts:
    • user_custom - Requires manual data entry (Control Conter -> Language Model -> User Custom)
    • sync_prompts - Synchronizing data from f/awesome-chatgpt-prompts (Control Conter -> Language Model -> Sync Prompts)
    • sync_custom - Synchronize custom json and csv file data, support local and remote (Control Conter -> Language Model -> Sync Custom)
  • chat.model.cmd.json - filtered (whether to enable) and sorted slash commands
  • [cache_model] - caching model data
    • chatgpt_prompts.json - Cache sync_prompts data
    • user_custom.json - Cache user_custom data
    • ae6cf32a6f8541b499d6bfe549dbfca3.json - Randomly generated file names, cache sync_custom data
    • 4f695d3cfbf8491e9b1f3fab6d85715c.json - Randomly generated file names, cache sync_custom data
    • bd1b96f15a1644f7bd647cc53073ff8f.json - Randomly generated file names, cache sync_custom data

Sync Custom

Currently, only json and csv are supported for synchronizing custom files, and the following formats need to be met, otherwise the application will be abnormal：

JSON format:

[
  {
    "cmd": "a",
    "act": "aa",
    "prompt": "aaa aaa aaa"
  },
  {
    "cmd": "b",
    "act": "bb",
    "prompt": "bbb bbb bbb"
  }
]

CSV format

"cmd","act","prompt"
"a","aa","aaa aaa aaa"
"b","bb","bbb bbb bbb"

📌 TODO

• Control Center enhancement
• Pop-up Search enhancement
• ...

👀 Preview

❓FAQ

Can't open ChatGPT

If you cannot open the application after the upgrade, please try to clear the configuration file, which is in the ~/.chatgpt/* directory.

Out of sync login status between multiple windows

If you have already logged in in the main window, but the system tray window shows that you are not logged in, you can fix it by restarting the application (Menu -> Preferences -> Restart ChatGPT).

Is it safe?

It's safe, just a wrapper for OpenAI ChatGPT website, no other data transfer exists (you can check the source code).

Developer cannot be verified?

• Open a Mac app from an unidentified developer

How do I build it?

PreInstall

• Rust (Required)
• Node.js (Required)
• VS Code (Optional)
  • rust-analyzer
  • tauri

Start

# step1:
git clone https://github.com/lencx/ChatGPT.git

# step2:
cd ChatGPT

# step3: install deps
yarn

# step4:
yarn dev

# step5:
# bundle path: src-tauri/target/release/bundle
yarn build

• The distDir configuration is set to "../dist" but this path doesn't exist
• Error A public key has been found, but no private key. Make sure to set TAURI_PRIVATE_KEY environment variable.

❤️ Thanks

• The core implementation of the share button code was copied from the @liady extension with some modifications.
• Thanks to the Awesome ChatGPT Prompts repository for inspiring the custom command function for this application.

Supporter Benefits (visible to your supporters and members only) - Share tips for using ChatGPT desktop application and next steps.

Download Details:

Author: lencx
Source Code: https://github.com/lencx/ChatGPT 
License: Apache-2.0 license

#rust #desktop #app #windows #macos #linux 

Clipboard: Cut, Copy, Paste anything, Anywhere, All From The Terminal

Clipboard

🚀 Clipboard is a power tool that saves you time and effort. Previously, you've always had to think about where exactly you want to move text and files. This increases your mental workload and makes some scenarios simply impractical. Now, you can have a unified clipboard to use anywhere in the command line, just as if you were using a GUI.

• Quick. Zero configuration needed to use.
• Easy. Friendly to newbies and power users alike.
• Compatible. Works on any system that supports C++23. Really!
• Unified. Functions exactly the same everywhere.
• Universal. Supports English, Spanish, Portuguese, and Turkish.
• Integrated. Connects with many native GUI clipboards.
• Tiny. Mere tens of kilobytes in size.

All Except Windows

curl -sSL https://github.com/Slackadays/Clipboard/raw/main/src/install.sh | bash

Windows

(Invoke-WebRequest -UseBasicParsing https://github.com/Slackadays/Clipboard/raw/main/src/install.ps1).Content | powershell

Install Manually

Get the latest release instead by adding --branch 0.2.1r2 right after git clone.... Change the installation prefix by adding -DINSTALL_PREFIX=/CUSTOM/PREFIX to cmake ...

git clone https://github.com/slackadays/Clipboard 
cd Clipboard/build
cmake -DCMAKE_BUILD_TYPE=MinSizeRel ..
cmake --build .
cmake --install .

Uninstall

Remove all the files in install_manifest.txt. If you're not using Windows, you can also do xargs rm < install_manifest.txt.

Premade Builds

You can also download Clipboard directly from GitHub Actions.

In all commands, you can substitute cb for clipboard. Add a number to the end of the action to choose which clipboard you want to use (the default is 0) or _ to use a persistent clipboard.

Copy   clipboard ([--]copy|[-]cp)[(num)|_(id)] (file) [files]

Cut   clipboard ([--]cut|[-]ct)[(num)|_(id)] (file) [files]

Paste   clipboard ([--]paste|[-]p)[(num)|_(id)]

Pipe In   (something) | clipboard [([--]copy|[-]cp)][(num)|_(id)]

Pipe Out   clipboard [([--]paste|[-]p][(num)|_(id)] | (something) or clipboard [([--]paste|[-]p)][(num)|_(id)] > (some file)

Show Contents   clipboard ([--]show|[-]sh)[(num)|_(id)]

Clear Contents   clipboard ([--]clear|[-]clr)[(num)|_(id)]

Examples

cb copy foo.txt launchcodes.doc
clipboard cut1 MyDirectory
cb cp800 bar.conf AnotherDirectory baz.txt

Environment Variables

CI   Set this to make Clipboard overwrite existing items without a user prompt when pasting. This variable is intended for Continuous Integration scripts where a live human is not present to make decisions.

FORCE_COLOR   Set this to make Clipboard always show color regardless of what you set NO_COLOR to.

TMPDIR   Set this to the directory that Clipboard will use to hold the items you cut or copy into a temporary directory. Other programs use TMPDIR as well, so be careful about changing this.

CLIPBOARD_TMPDIR   Set this to the directory that only Clipboard will use to hold the items you cut or copy into a temporary directory.

CLIPBOARD_PERSISTDIR   Set this to the directory that only Clipboard will use to hold the items you cut or copy into a persistent directory.

CLIPBOARD_ALWAYS_PERSIST   Set this to make Clipboard always use persistent clipboards.

CLIPBOARD_NOGUI   Set this to disable integration with GUI clipboards.

NO_COLOR   Set this to make Clipboard not show any colors.

Flags

--fast-copy, -fc   Add this to use links when copying, cutting, or pasting. If you modify the items that you used with this flag, then the items you paste will have the same changes.

Go to the Clipboard Wiki for more information, or join our Discord group!

Thank you to all the contributors who have helped make Clipboard great.

Download Details:

Author: Slackadays
Source Code: https://github.com/Slackadays/Clipboard 
License: GPL-3.0 license

#cpluplus #windows #macos #linux #cli

Docker-OSX: Run macOS VM in a Docker!

Docker-OSX · 

Run Mac OS X in Docker with near-native performance! X11 Forwarding! iMessage security research! iPhone USB working! macOS in a Docker container!

Conduct Security Research on macOS using both Linux & Windows!

Docker-OSX now has a Discord server & Telegram!

The Discord is active on docker-osx and anyone is welcome to come and ask questions, ideas, etc.

Quick Start Docker-OSX

Video setup tutorial is also available here: https://www.youtube.com/watch?v=wLezYl77Ll8

Windows users: click here to see the notes below!

First time here? try initial setup, otherwise try the instructions below to use either Catalina or Big Sur.

Any questions, ideas, or just want to hang out?

https://discord.gg/sickchat

Catalina

docker run -it \
    --device /dev/kvm \
    -p 50922:10022 \
    -v /tmp/.X11-unix:/tmp/.X11-unix \
    -e "DISPLAY=${DISPLAY:-:0.0}" \
    sickcodes/docker-osx:latest

# docker build -t docker-osx .

Big Sur

docker run -it \
    --device /dev/kvm \
    -p 50922:10022 \
    -v /tmp/.X11-unix:/tmp/.X11-unix \
    -e "DISPLAY=${DISPLAY:-:0.0}" \
    sickcodes/docker-osx:big-sur

# docker build -t docker-osx --build-arg SHORTNAME=big-sur .

Monterey

docker run -it \
    --device /dev/kvm \
    -p 50922:10022 \
    -v /tmp/.X11-unix:/tmp/.X11-unix \
    -e "DISPLAY=${DISPLAY:-:0.0}" \
    -e GENERATE_UNIQUE=true \
    -e MASTER_PLIST_URL='https://raw.githubusercontent.com/sickcodes/osx-serial-generator/master/config-custom.plist' \
    sickcodes/docker-osx:monterey

# docker build -t docker-osx --build-arg SHORTNAME=monterey .

Ventura

docker run -it \
    --device /dev/kvm \
    -p 50922:10022 \
    -v /tmp/.X11-unix:/tmp/.X11-unix \
    -e "DISPLAY=${DISPLAY:-:0.0}" \
    -e GENERATE_UNIQUE=true \
    -e MASTER_PLIST_URL='https://raw.githubusercontent.com/sickcodes/osx-serial-generator/master/config-custom.plist' \
    sickcodes/docker-osx:ventura

# docker build -t docker-osx --build-arg SHORTNAME=ventura .

Run Catalina Pre-Installed

# 40GB disk space required: 20GB original image 20GB your container.
docker pull sickcodes/docker-osx:auto

# boot directly into a real OS X shell with a visual display [NOT HEADLESS]
docker run -it \
    --device /dev/kvm \
    -p 50922:10022 \
    -v /tmp/.X11-unix:/tmp/.X11-unix \
    -e "DISPLAY=${DISPLAY:-:0.0}" \
    -e GENERATE_UNIQUE=true \
    sickcodes/docker-osx:auto

# username is user
# passsword is alpine

Older Systems

High Sierra

docker run -it \
    --device /dev/kvm \
    -p 50922:10022 \
    -v /tmp/.X11-unix:/tmp/.X11-unix \
    -e "DISPLAY=${DISPLAY:-:0.0}" \
    sickcodes/docker-osx:high-sierra

# docker build -t docker-osx --build-arg SHORTNAME=high-sierra .

Mojave

docker run -it \
    --device /dev/kvm \
    -p 50922:10022 \
    -v /tmp/.X11-unix:/tmp/.X11-unix \
    -e "DISPLAY=${DISPLAY:-:0.0}" \
    sickcodes/docker-osx:mojave

# docker build -t docker-osx --build-arg SHORTNAME=mojave .

Download the image manually and use it in Docker

This is a particularly good way for downloading the container, in case Docker's CDN (or your connection) happens to be slow.

wget https://images2.sick.codes/mac_hdd_ng_auto.img

docker run -it \
    --device /dev/kvm \
    -p 50922:10022 \
    -v "${PWD}/mac_hdd_ng_auto.img:/image" \
    -v /tmp/.X11-unix:/tmp/.X11-unix \
    -e "DISPLAY=${DISPLAY:-:0.0}" \
    -e GENERATE_UNIQUE=true \
    -e MASTER_PLIST_URL=https://raw.githubusercontent.com/sickcodes/Docker-OSX/master/custom/config-nopicker-custom.plist \
    sickcodes/docker-osx:naked

Use your own image and manually and automatically log into a shell

Enable SSH in network sharing inside the guest first. Change -e "USERNAME=user" and -e "PASSWORD=password" to your credentials. The container will add itself to ~/.ssh/authorized_keys

Since you can't see the screen, use the PLIST with nopicker, for example:

# Catalina
# wget https://images2.sick.codes/mac_hdd_ng_auto.img
# Monterey
wget https://images.sick.codes/mac_hdd_ng_auto_monterey.img

docker run -it \
    --device /dev/kvm \
    -p 50922:10022 \
    -v "${PWD}/mac_hdd_ng_auto_monterey.img:/image" \
    -v /tmp/.X11-unix:/tmp/.X11-unix \
    -e "DISPLAY=${DISPLAY:-:0.0}" \
    -e "USERNAME=user" \
    -e "PASSWORD=alpine" \
    -e GENERATE_UNIQUE=true \
    -e MASTER_PLIST_URL=https://raw.githubusercontent.com/sickcodes/Docker-OSX/master/custom/config-nopicker-custom.plist \
    sickcodes/docker-osx:naked-auto

Share directories, sharing files, shared folder, mount folder

The easiest and most secure way is sshfs

# on Linux/Windows
mkdir ~/mnt/osx
sshfs user@localhost:/ -p 50922 ~/mnt/osx
# wait a few seconds, and ~/mnt/osx will have full rootfs mounted over ssh, and in userspace
# automated: sshpass -p <password> sshfs user@localhost:/ -p 50922 ~/mnt/osx

(VFIO) iPhone USB passthrough (VFIO)

If you have a laptop see the next usbfluxd section.

If you have a desktop PC, you can use @Silfalion's instructions : https://github.com/Silfalion/Iphone_docker_osx_passthrough

(USBFLUXD) iPhone USB -> Network style passthrough OSX-KVM Docker-OSX

Video setup tutorial for usbfluxd is also available here: https://www.youtube.com/watch?v=kTk5fGjK_PM

This method WORKS on laptop, PC, anything!

Thank you @nikias for usbfluxd via https://github.com/corellium!

This is done inside Linux.

Open 3 terminals on Linux

Connecting your device over USB on Linux allows you to expose usbmuxd on port 5000 using https://github.com/corellium/usbfluxd to another system on the same network.

Ensure usbmuxd, socat and usbfluxd are installed.

sudo pacman -S libusbmuxd usbmuxd avahi socat

Available on the AUR: https://aur.archlinux.org/packages/usbfluxd/

yay usbfluxd

Plug in your iPhone or iPad.

Terminal 1

sudo systemctl start usbmuxd
sudo avahi-daemon

Terminal 2:

# on host
sudo systemctl restart usbmuxd
sudo socat tcp-listen:5000,fork unix-connect:/var/run/usbmuxd

Terminal 3:

sudo usbfluxd -f -n

Connect to a host running usbfluxd

This is done inside macOS.

Install homebrew.

172.17.0.1 is usually the Docker bridge IP, which is your PC, but you can use any IP from ip addr...

macOS Terminal:

# on the guest
brew install make automake autoconf libtool pkg-config gcc libimobiledevice usbmuxd

git clone https://github.com/corellium/usbfluxd.git
cd usbfluxd

./autogen.sh
make
sudo make install

Accept the USB over TCP connection, and appear as local:

(you may need to change 172.17.0.1 to the IP address of the host. e.g. check ip addr)

# on the guest
sudo launchctl start usbmuxd
export PATH=/usr/local/sbin:${PATH}
sudo usbfluxd -f -r 172.17.0.1:5000

Close apps such as Xcode and reopen them and your device should appear!

If you need to start again on Linux, wipe the current usbfluxd, usbmuxd, and socat:

sudo killall usbfluxd
sudo systemctl restart usbmuxd
sudo killall socat

Make container FASTER using https://github.com/sickcodes/osx-optimizer

SEE commands in https://github.com/sickcodes/osx-optimizer!

• Skip the GUI login screen (at your own risk!)
• Disable spotlight indexing on macOS to heavily speed up Virtual Instances.
• Disable heavy login screen wallpaper
• Disable updates (at your own risk!)

Increase disk space by moving /var/lib/docker to external drive, block storage, NFS, or any other location conceivable.

Move /var/lib/docker, following the tutorial below

• Cheap large physical disk storage instead using your server's disk, or SSD.
• Block Storage, NFS, etc.

Tutorial here: https://sick.codes/how-to-run-docker-from-block-storage/

Only follow the above tutorial if you are happy with wiping all your current Docker images/layers.

Safe mode: Disable docker temporarily so you can move the Docker folder temporarily.

• Do NOT do this until you have moved your image out already https://github.com/dulatello08/Docker-OSX/#quick-start-your-own-image-naked-container-image

killall dockerd
systemctl disable --now docker
systemctl disable --now docker.socket
systemctl stop docker
systemctl stop docker.socket

Now, that Docker daemon is off, move /var/lib/docker somewhere

Then, symbolicly link /var/lib/docker somewhere:

mv /var/lib/docker /run/media/user/some_drive/docker
ln -s /run/media/user/some_drive/docker /var/lib/docker

# now check if /var/lib/docker is working still
ls /var/lib/docker

If you see folders, then it worked. You can restart Docker, or just reboot if you want to be sure.

Important notices:

2021-11-14 - Added High Sierra, Mojave

Pick one of these while building, irrelevant when using docker pull:

--build-arg SHORTNAME=high-sierra 
--build-arg SHORTNAME=mojave
--build-arg SHORTNAME=catalina
--build-arg SHORTNAME=big-sur
--build-arg SHORTNAME=monterey
--build-arg SHORTNAME=ventura

Technical details

There currently multiple images, each with different use cases (explained below):

• High Sierra
• Mojave
• Catalina
• Big Sur
• Monterey
• Ventura
• Auto (pre-made Catalina)
• Naked (use your own .img)
• Naked-Auto (user your own .img and SSH in)

High Sierra:

Mojave:

Catalina:

Big-Sur:

Monterey make your own image:

Pre-made Catalina system by Sick.Codes: username: user, password: alpine

Naked: Bring-your-own-image setup (use any of the above first):

Naked Auto: same as above but with -e USERNAME & -e PASSWORD and -e OSX_COMMANDS="put your commands here"

Capabilities

• use iPhone OSX KVM on Linux using usbfluxd!
• macOS Monterey VM on Linux!
• Folder sharing-
• USB passthrough (hotplug too)
• SSH enabled (localhost:50922)
• VNC enabled (localhost:8888) if using ./vnc version
• iMessage security research via serial number generator!
• X11 forwarding is enabled
• runs on top of QEMU + KVM
• supports Big Sur, custom images, Xvfb headless mode
• you can clone your container with docker commit

Requirements

• 20GB+++ disk space for bare minimum installation (50GB if using Xcode)
• virtualization should be enabled in your BIOS settings
• a x86_64 kvm-capable host
• at least 50 GBs for :auto (half for the base image, half for your runtime image

TODO

• documentation for security researchers
• gpu acceleration
• support for virt-manager

Docker

Images built on top of the contents of this repository are also available on Docker Hub for convenience: https://hub.docker.com/r/sickcodes/docker-osx

A comprehensive list of the available Docker images and their intended purpose can be found in the Instructions.

Kubernetes

Docker-OSX supports Kubernetes.

Kubernetes Helm Chart & Documentation can be found under the helm directory.

Thanks cephasara for contributing this major contribution.

Support

Small questions & issues

Feel free to open an issue, should you come across minor issues with running Docker-OSX or have any questions.

Resolved issues

Before you open an issue, however, please check the closed issues and confirm that you're using the latest version of this repository — your issues may have already been resolved! You might also see your answer in our questions and answers section below.

Feature requests and updates

Follow @sickcodes!

Professional support

For more sophisticated endeavours, we offer the following support services:

• Enterprise support, business support, or casual support.
• Custom images, custom scripts, consulting (per hour available!)
• One-on-one conversations with you or your development team.

In case you're interested, contact @sickcodes on Twitter or click here.

License/Contributing

Docker-OSX is licensed under the GPL v3+. Contributions are welcomed and immensely appreciated. You are in-fact permitted to use Docker-OSX as a tool to create proprietary software.

Other cool Docker/QEMU based projects

• Run Android in a Docker Container with Dock Droid
• Run Android fully native on the host!
• Run iOS 12 in a Docker container with Docker-eyeOS - https://github.com/sickcodes/Docker-eyeOS
• Run iMessage relayer in Docker with Bluebubbles.app - Getting started wiki

Disclaimer

If you are serious about Apple Security, and possibly finding 6-figure bug bounties within the Apple Bug Bounty Program, then you're in the right place! Further notes: Is Hackintosh, OSX-KVM, or Docker-OSX legal?

Product names, logos, brands and other trademarks referred to within this project are the property of their respective trademark holders. These trademark holders are not affiliated with our repository in any capacity. They do not sponsor or endorse this project in any way.

Instructions

Container images

Already set up or just looking to make a container quickly? Check out our quick start or see a bunch more use cases under our container creation examples section.

There are several different Docker-OSX images available which are suitable for different purposes.

• sickcodes/docker-osx:latest - I just want to try it out.
• sickcodes/docker-osx:latest - I want to use Docker-OSX to develop/secure apps in Xcode (sign into Xcode, Transporter)
• sickcodes/docker-osx:naked - I want to use Docker-OSX for CI/CD-related purposes (sign into Xcode, Transporter)

Create your personal image using :latest or big-sur. Then, pull the image out the image. Afterwards, you will be able to duplicate that image and import it to the :naked container, in order to revert the container to a previous state repeatedly.

sickcodes/docker-osx:auto - I'm only interested in using the command line (useful for compiling software or using Homebrew headlessly).

sickcodes/docker-osx:naked - I need iMessage/iCloud for security research.

sickcodes/docker-osx:big-sur - I want to run Big Sur.

sickcodes/docker-osx:monterey - I want to run Monterey.

sickcodes/docker-osx:ventura - I want to run Ventura.

sickcodes/docker-osx:high-sierra - I want to run High Sierra.

sickcodes/docker-osx:mojave - I want to run Mojave.

Initial setup

Before you do anything else, you will need to turn on hardware virtualization in your BIOS. Precisely how will depend on your particular machine (and BIOS), but it should be straightforward.

Then, you'll need QEMU and some other dependencies on your host:

# ARCH
sudo pacman -S qemu libvirt dnsmasq virt-manager bridge-utils flex bison iptables-nft edk2-ovmf

# UBUNTU DEBIAN
sudo apt install qemu qemu-kvm libvirt-clients libvirt-daemon-system bridge-utils virt-manager libguestfs-tools

# CENTOS RHEL FEDORA
sudo yum install libvirt qemu-kvm

Then, enable libvirt and load the KVM kernel module:

sudo systemctl enable --now libvirtd
sudo systemctl enable --now virtlogd

echo 1 | sudo tee /sys/module/kvm/parameters/ignore_msrs

sudo modprobe kvm

I'd like to run Docker-OSX on Windows

Running Docker-OSX on Windows is possible using WSL2 (Windows 11 + Windows Subsystem for Linux).

You must have Windows 11 installed with build 22000+ (21H2 or higher).

First, install WSL on your computer by running this command in an administrator powershell. For more info, look here.

This will install Ubuntu by default.

wsl --install

You can confirm WSL2 is enabled using wsl -l -v in PowerShell. To see other distributions that are available, use wsl -l -o.

If you have previously installed WSL1, upgrade to WSL 2. Check this link to upgrade from WSL1 to WSL2.

After WSL installation, go to C:/Users/<Your_Name>/.wslconfig and add nestedVirtualization=true to the end of the file (If the file doesn't exist, create it). For more information about the .wslconfig file check this link. Verify that you have selected "Show Hidden Files" and "Show File Extensions" in File Explorer options. The result should be like this:

[wsl2]
nestedVirtualization=true

Go into your WSL distro (Run wsl in powershell) and check if KVM is enabled by using the kvm-ok command. The output should look like this:

INFO: /dev/kvm exists
KVM acceleration can be used

Now download and install Docker for Windows if it is not already installed.

After installation, go into Settings and check these 2 boxes:

General -> "Use the WSL2 based engine";
Resources -> WSL Integration -> "Enable integration with my default WSL distro", 

Ensure x11-apps is installed. Use the command sudo apt install x11-apps -y to install it if it isn't.

Finally, there are 3 ways to get video output:

• WSLg: This is the simplest and easiest option to use. There may be some issues such as the keyboard not being fully passed through or seeing a second mouse on the desktop - Issue on WSLg - but this option is recommended.

To use WSLg's built-in X-11 server, change these two lines in the docker run command to point Docker-OSX to WSLg.

-e "DISPLAY=${DISPLAY:-:0.0}" \
-v /mnt/wslg/.X11-unix:/tmp/.X11-unix \

Or try:

-e "DISPLAY=${DISPLAY:-:0}" \
-v /mnt/wslg/.X11-unix:/tmp/.X11-unix \

For Ubuntu 20.x on Windows, see https://github.com/sickcodes/Docker-OSX/discussions/458

• VNC: See the VNC section for more information. You could also add -vnc argument to qemu. Connect to your mac VM via a VNC Client. Here is a how to
• Desktop Environment: This will give you a full desktop linux experience but it will use a bit more of the computer's resources. Here is an example guide, but there are other guides that help set up a desktop environment. DE Example

Additional boot instructions for when you are creating your container

Boot the macOS Base System (Press Enter)

Click Disk Utility

Erase the BIGGEST disk (around 200gb default), DO NOT MODIFY THE SMALLER DISKS.

-- if you can't click erase, you may need to reduce the disk size by 1kb

(optional) Create a partition using the unused space to house the OS and your files if you want to limit the capacity. (For Xcode 12 partition at least 60gb.)

Click Reinstall macOS

The system may require multiple reboots during installation

Troubleshooting

Routine checks

This is a great place to start if you are having trouble getting going, especially if you're not that familiar with Docker just yet.

Just looking to make a container quickly? Check out our container creation examples section.

More specific/advanced troubleshooting questions and answers may be found in More Questions and Answers. You should also check out the closed issues. Someone else might have gotten a question like yours answered already even if you can't find it in this document!

Confirm that your CPU supports virtualization

See initial setup.

Docker Unknown Server OS error

docker: unknown server OS: .
See 'docker run --help'.

This means your docker daemon is not running.

pgrep dockerd should return nothing

Therefore, you have a few choices.

sudo dockerd for foreground Docker usage. I use this.

Or

sudo systemctl --start dockerd to start dockerd this now.

Or

sudo systemctl --enable --now dockerd for start dockerd on every reboot, and now.

Use more CPU Cores/SMP

Examples:

-e EXTRA='-smp 6,sockets=3,cores=2'

-e EXTRA='-smp 8,sockets=4,cores=2'

-e EXTRA='-smp 16,sockets=8,cores=2'

Note, unlike memory, CPU usage is shared. so you can allocate all of your CPU's to the container.

Confirm your user is part of the the Docker group, KVM group, libvirt group

Add yourself to the Docker group

If you use sudo dockerd or dockerd is controlled by systemd/systemctl, then you must be in the Docker group. If you are not in the Docker group:

sudo usermod -aG docker "${USER}"

and also add yourself to the kvm and libvirt groups if needed:

sudo usermod -aG libvirt "${USER}"
sudo usermod -aG kvm "${USER}"

See also: initial setup.

Is the docker daemon enabled?

# run ad hoc
sudo dockerd

# or daemonize it
sudo nohup dockerd &

# enable it in systemd (it will persist across reboots this way)
sudo systemctl enable --now docker

# or just start it as your user with systemd instead of enabling it
systemctl start docker

More Questions and Answers

Big thank you to our contributors who have worked out almost every conceivable issue so far!

https://github.com/sickcodes/Docker-OSX/blob/master/CREDITS.md

Start the same container later (persistent disk)

Created a container with docker run and want to reuse the underlying image again later?

NB: see container creation examples first for how to get to the point where this is applicable.

This is for when you want to run the SAME container again later. You may need to use docker commit to save your container before you can reuse it. Check if your container is persisted with docker ps --all.

If you don't run this you will have a new image every time.

# look at your recent containers and copy the CONTAINER ID
docker ps --all

# docker start the container ID
docker start -ai abc123xyz567

# if you have many containers, you can try automate it with filters like this
# docker ps --all --filter "ancestor=sickcodes/docker-osx"
# for locally tagged/built containers
# docker ps --all --filter "ancestor=docker-osx"

You can also pull the .img file out of the container, which is stored in /var/lib/docker, and supply it as a runtime argument to the :naked Docker image.

See also: here.

I have used Docker-OSX before and want to restart a container that starts automatically

Containers that use sickcodes/docker-osx:auto can be stopped while being started.

# find last container
docker ps -a

# docker start old container with -i for interactive, -a for attach STDIN/STDOUT
docker start -ai -i <Replace this with your ID>

LibGTK errors "connection refused"

You may see one or more libgtk-related errors if you do not have everything set up for hardware virtualisation yet. If you have not yet done so, check out the initial setup section and the routine checks section as you may have missed a setup step or may not have all the needed Docker dependencies ready to go.

See also: here.

Permissions denied error

If you have not yet set up xhost, try the following:

echo $DISPLAY

# ARCH
sudo pacman -S xorg-xhost

# UBUNTU DEBIAN
sudo apt install x11-xserver-utils

# CENTOS RHEL FEDORA
sudo yum install xorg-x11-server-utils

# then run
xhost +

RAM over-allocation

You cannot allocate more RAM than your machine has. The default is 3 Gigabytes: -e RAM=3.

If you are trying to allocate more RAM to the container than you currently have available, you may see an error like the following: cannot set up guest memory 'pc.ram': Cannot allocate memory. See also: here, here.

For example (below) the buff/cache already contains 20 Gigabytes of allocated RAM:

[user@hostname ~]$ free -mh
               total        used        free      shared  buff/cache   available
Mem:            30Gi       3.5Gi       7.0Gi       728Mi        20Gi        26Gi
Swap:           11Gi          0B        11Gi

Clear the buffer and the cache:

sudo tee /proc/sys/vm/drop_caches <<< 3

Now check the RAM again:

[user@hostname ~]$ free -mh
               total        used        free      shared  buff/cache   available
Mem:            30Gi       3.3Gi        26Gi       697Mi       1.5Gi        26Gi
Swap:           11Gi          0B        11Gi

PulseAudio

Use PulseAudio for sound

Note: AppleALC, alcid and VoodooHDA-OC do not have codec support. However, IORegistryExplorer does show the controller component working.

docker run \
    --device /dev/kvm \
    -e AUDIO_DRIVER=pa,server=unix:/tmp/pulseaudio.socket \
    -v "/run/user/$(id -u)/pulse/native:/tmp/pulseaudio.socket" \
    -v /tmp/.X11-unix:/tmp/.X11-unix \
    sickcodes/docker-osx

PulseAudio debugging

docker run \
    --device /dev/kvm \
    -e AUDIO_DRIVER=pa,server=unix:/tmp/pulseaudio.socket \
    -v "/run/user/$(id -u)/pulse/native:/tmp/pulseaudio.socket" \
    -v /tmp/.X11-unix:/tmp/.X11-unix \
    -e PULSE_SERVER=unix:/tmp/pulseaudio.socket \
    sickcodes/docker-osx pactl list

PulseAudio with WSLg

docker run \
    --device /dev/kvm \
    -e AUDIO_DRIVER=pa,server=unix:/tmp/pulseaudio.socket \
    -v /mnt/wslg/runtime-dir/pulse/native:/tmp/pulseaudio.socket \
    -v /mnt/wslg/.X11-unix:/tmp/.X11-unix \
    sickcodes/docker-osx

Forward additional ports (nginx hosting example)

It's possible to forward additional ports depending on your needs. In this example, we'll use Mac OSX to host nginx:

host:10023 <-> 10023:container:10023 <-> 80:guest

On the host machine, run:

docker run -it \
    --device /dev/kvm \
    -p 50922:10022 \
    -e ADDITIONAL_PORTS='hostfwd=tcp::10023-:80,' \
    -p 10023:10023 \
    sickcodes/docker-osx:auto

In a Terminal session running the container, run:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

brew install nginx
sudo sed -i -e 's/8080/80/' /usr/local/etc/nginx/nginx.confcd
# sudo nginx -s stop
sudo nginx

nginx should now be reachable on port 10023.

Additionally, you can string multiple statements together, for example:

    -e ADDITIONAL_PORTS='hostfwd=tcp::10023-:80,hostfwd=tcp::10043-:443,'
    -p 10023:10023 \
    -p 10043:10043 \

Bridged networking

You might not need to do anything with the default setup to enable internet connectivity from inside the container. Additionally, curl may work even if ping doesn't.

See discussion here and here and here.

Enable IPv4 forwarding for bridged network connections for remote installations

This is not required for LOCAL installations.

Additionally note it may cause the host to leak your IP, even if you're using a VPN in the container.

However, if you're trying to connect to an instance of Docker-OSX remotely (e.g. an instance of Docker-OSX hosted in a datacenter), this may improve your performance:

# enable for current session
sudo sysctl -w net.ipv4.ip_forward=1

# OR
# sudo tee /proc/sys/net/ipv4/ip_forward <<< 1

# enable permanently
sudo touch /etc/sysctl.conf
sudo tee -a /etc/sysctl.conf <<EOF
net.ipv4.ip_forward = 1
EOF

# or edit manually with the editor of your choice
nano /etc/sysctl.conf || vi /etc/sysctl.conf || vim /etc/sysctl.conf

# now reboot

Share folder with Docker-OSX QEMU macOS

Sharing a folder with guest is quite simple.

Your folder, will go to /mnt/hostshare inside the Arch container which is then passed over QEMU.

Then mount using sudo -S mount_9p hostshare from inside the mac.

For example,

FOLDER=~/somefolder

    -v "${FOLDER}:/mnt/hostshare" \
    -e EXTRA="-virtfs local,path=/mnt/hostshare,mount_tag=hostshare,security_model=passthrough,id=hostshare" \

Full example:

# stat mac_hdd_ng.img
SHARE=~/somefolder

docker run -it \
    --device /dev/kvm \
    -p 50922:10022 \
    -v /tmp/.X11-unix:/tmp/.X11-unix \
    -e "DISPLAY=${DISPLAY:-:0.0}" \
    -v "${PWD}/mac_hdd_ng.img:/home/arch/OSX-KVM/mac_hdd_ng.img" \
    -v "${SHARE}:/mnt/hostshare" \
    -e EXTRA="-virtfs local,path=/mnt/hostshare,mount_tag=hostshare,security_model=passthrough,id=hostshare" \
    sickcodes/docker-osx:latest

# !!! Open Terminal inside macOS and run the following command to mount the virtual file system
# sudo -S mount_9p hostshare

Share Linux NFS Drive into macOS

To share a folder using NFS, setup a folder for on the host machine, for example, /srv/nfs/share and then append to /etc/exports:

/srv/nfs/share      127.0.0.1/0(insecure,rw,all_squash,anonuid=1000,anongid=985,no_subtree_check)

You may need to reload exports now, which will begin sharing that directory.

# reload shared folders
sudo exportfs -arv

Source & Explanation

Give permissions on the shared folder for the anonuid and anongid, where anonuid and anongid matches that of your linux user; id -u

id -u ; id -g will print userid:groupid

chown 1000:985 /srv/nfs/share
chmod u+rwx /srv/nfs/share

Start the Docker-OSX container with the additional flag --network host

Create and mount the nfs folder from the mac terminal:

mkdir -p ~/mnt
sudo mount_nfs -o locallocks 10.0.2.2:/srv/nfs/share ~/mnt

Share USB Drive into macOS over QEMU

Mount USB Drive (Hotplug/Hot Plug USB)

Start your container.

Pick a port, for example, 7700.

lsusb to get vid:pid

On Linux: sudo usbredirserver -p 7700 1e3d:2096

Now, in the Docker window hit Enter to see the (qemu) console.

You can add/remove the disk using commands like this, even once the machine is started:

chardev-add socket,id=usbredirchardev1,port=7700,host=172.17.0.1

device_add usb-redir,chardev=usbredirchardev1,id=usbredirdev1,debug=4

Mount USB Drive inside macOS at boot Docker OSX

PORT=7700
IP_ADDRESS=172.17.0.1

-e EXTRA="-chardev socket,id=usbredirchardev1,port=${PORT},host=${IP_ADDRESS} -device usb-redir,chardev=usbredirchardev1,id=usbredirdev1,debug=4" \`

Fedora: enable internet connectivity with a bridged network

Fedora's default firewall settings may prevent Docker's network interface from reaching the internet. In order to reoslve this, you will need to whitelist the interface in your firewall:

# Set the docker0 bridge to the trusted zone
sudo firewall-cmd --permanent --zone=trusted --add-interface=docker0
sudo firewall-cmd --reload

Nested Hardware Virtualization

Check if your machine has hardware virtualization enabled:

sudo tee /sys/module/kvm/parameters/ignore_msrs <<< 1

egrep -c '(svm|vmx)' /proc/cpuinfo

Virtual network adapters

Fast internet connectivity

-e NETWORKING=vmxnet3

Slow internet connectivity

-e NETWORKING=e1000-82545em

CI/CD Related Improvements

Tips for reducing the size of the image

• Start the container as usual, and remove unnecessary files. A useful way to do this is to use du -sh * starting from the / directory, and find large directories where files can be removed. E.g. unnecessary cached files, Xcode platforms, etc.
• Once you are satisfied with the amount of free space, enable trim with sudo trimforce enable, and reboot.
• Zero out the empty space on the disk with dd if=/dev/zero of=./empty && rm -f empty
• Shut down the VM and copy out the qcow image with docker cp stoppedcontainer:/home/arch/OSX-KVM/mac_hdd_ng.img .
• Run qemu-img check -r all mac_hdd_ng.img to fix any errors.
• Run qemu-img convert -O qcow2 mac_hdd_ng.img deduped.img and check for errors again
• OPTIONAL: Run qemu-img convert -c -O qcow2 deduped.img compressed.img to further compress the image. This may reduce the runtime speed though, but it should reduce the size by roughly 25%.
• Check for errors again, and build a fresh docker image. E.g. with this Dockerfile

FROM sickcodes/docker-osx
USER arch
COPY --chown=arch ./deduped.img /home/arch/OSX-KVM/mac_hdd_ng.img

Run Docker-OSX headlessly with Telnet

First make sure autoboot is enabled

Next, you will want to set up SSH to be automatically started.

sudo systemsetup -setremotelogin on

Make sure to commit the new docker image and save it, or rebuild as described in the section on reducing disk space.

Then run it with these arguments.

# Run with the -nographic flag, and enable a telnet interface
  docker run \
    --device /dev/kvm \
    -p 50922:10022 \
    -e "DISPLAY=${DISPLAY:-:0.0}" \
    -e EXTRA="-monitor telnet::45454,server,nowait -nographic -serial null" \
    mycustomimage

What mirrors are appropriate to use to build Docker-OSX locally?

If you are building Docker-OSX locally, you'll probably want to use Arch Linux's mirrors.

Mirror locations can be found here (uses two-letter country codes): https://archlinux.org/mirrorlist/all/

docker build -t docker-osx:latest \
    --build-arg RANKMIRRORS=true \
    --build-arg MIRROR_COUNTRY=US \
    --build-arg MIRROR_COUNT=10 \
    --build-arg SHORTNAME=catalina \
    --build-arg SIZE=200G .

Custom QEMU Arguments (passthrough devices)

Pass any devices/directories to the Docker container & the QEMU arguments using the handy runtime argument provider option -e EXTRA=.

# example customizations
docker run \
    -e RAM=4 \
    -e SMP=4 \
    -e CORES=4 \
    -e EXTRA='-usb -device usb-host,hostbus=1,hostaddr=8' \
    -e INTERNAL_SSH_PORT=23 \
    -e MAC_ADDRESS="$(xxd -c1 -p -l 6 /dev/urandom | tr '\n' ':' | cut -c1-17)" \
    -e AUDIO_DRIVER=alsa \
    -e IMAGE_PATH=/image \
    -e SCREEN_SHARE_PORT=5900 \
    -e DISPLAY=:0 \
    -e NETWORKING=vmxnet3 \
    --device /dev/kvm \
    --device /dev/snd \
    -v /tmp/.X11-unix:/tmp/.X11-unix \
    docker-osx:latest

Generating serial numbers

Generate serial numbers in ./custom OR make docker generate them at runtime (see below).

At any time, verify your serial number before logging into iCloud, etc.

# this is a quick way to check your serial number via cli inside OSX
ioreg -l | grep IOPlatformSerialNumber

# test some commands
sshpass -p 'alpine' ssh user@localhost -p 50922 'ping google.com'

# check your serial number
sshpass -p 'alpine' ssh user@localhost -p 50922 'ioreg -l | grep IOPlatformSerialNumber'

Getting started with serial numbers

# ARCH
pacman -S libguestfs

# UBUNTU DEBIAN
apt install libguestfs -y

# RHEL FEDORA CENTOS
yum install libguestfs -y

Inside the ./custom folder you will find 4 scripts.

• config-nopicker-custom.plist
• opencore-image-ng.sh

These two files are from OSX-KVM.

You don't need to touch these two files.

The config.plist has 5 values replaced with placeholders. Click here to see those values for no reason.

• generate-unique-machine-values.sh This script will generate serial numbers, with Mac Addresses, plus output to CSV/TSV, plus make a bootdisk image.

You can create hundreds, ./custom/generate-unique-machine-values.sh --help

./custom/generate-unique-machine-values.sh \
    --count 1 \
    --tsv ./serial.tsv \
    --bootdisks \
    --output-bootdisk OpenCore.qcow2 \
    --output-env source.env.sh

Or if you have some specific serial numbers...

• generate-specific-bootdisk.sh

This example generates a random set of serial numbers at runtime, headlessly

# proof of concept only, generates random serial numbers, headlessly, and quits right after.
docker run --rm -it \
    --device /dev/kvm \
    -p 50922:10022 \
    -e NOPICKER=true \
    -e GENERATE_UNIQUE=true \
    -e DEVICE_MODEL="iMacPro1,1" \
    sickcodes/docker-osx:auto

# -e OSX_COMMANDS='ioreg -l | grep IOPlatformSerialNumber' \

This example generates a specific set of serial numbers at runtime

# run the same as above 17gb auto image, with SSH, with nopicker, and save the bootdisk for later.
# you don't need to save the bootdisk IF you supply specific serial numbers!

docker run -it \
    --device /dev/kvm \
    -p 50922:10022 \
    -e NOPICKER=true \
    -e GENERATE_SPECIFIC=true \
    -e DEVICE_MODEL="iMacPro1,1" \
    -e SERIAL="C02TW0WAHX87" \
    -e BOARD_SERIAL="C027251024NJG36UE" \
    -e UUID="5CCB366D-9118-4C61-A00A-E5BAF3BED451" \
    -e MAC_ADDRESS="A8:5C:2C:9A:46:2F" \
    -e OSX_COMMANDS='ioreg -l | grep IOPlatformSerialNumber' \
    sickcodes/docker-osx:auto

This example generates a specific set of serial numbers at runtime, with your existing image, at 1000x1000 display resolution

# run an existing image in current directory, with a screen, with SSH, with nopicker.

stat mac_hdd_ng.img # make sure you have an image if you're using :naked

docker run -it \
    -v "${PWD}/mac_hdd_ng.img:/image" \
    --device /dev/kvm \
    -e "DISPLAY=${DISPLAY:-:0.0}" \
    -v /tmp/.X11-unix:/tmp/.X11-unix \
    -p 50922:10022 \
    -e NOPICKER=true \
    -e GENERATE_SPECIFIC=true \
    -e DEVICE_MODEL="iMacPro1,1" \
    -e SERIAL="C02TW0WAHX87" \
    -e BOARD_SERIAL="C027251024NJG36UE" \
    -e UUID="5CCB366D-9118-4C61-A00A-E5BAF3BED451" \
    -e MAC_ADDRESS="A8:5C:2C:9A:46:2F" \
    -e WIDTH=1000 \
    -e HEIGHT=1000 \
    sickcodes/docker-osx:naked

If you want to generate serial numbers, either make them at runtime using -e GENERATE_UNIQUE=true \

Or you can generate them inside the ./custom folder. And then use:

    -e GENERATE_SPECIFIC=true \
    -e SERIAL="" \
    -e BOARD_SERIAL="" \
    -e UUID="" \
    -e MAC_ADDRESS="" \

Making serial numbers persist across reboots

stat mac_hdd_ng_testing.img
touch ./output.env

# generate fresh random serial numbers, with a screen, using your own image, and save env file with your new serial numbers for later.

docker run -it \
    --device /dev/kvm \
    -e "DISPLAY=${DISPLAY:-:0.0}" \
    -v /tmp/.X11-unix:/tmp/.X11-unix \
    -p 50922:10022 \
    -e NOPICKER=true \
    -e GENERATE_UNIQUE=true \
    -e GENERATE_SPECIFIC=true \
    -e DEVICE_MODEL="iMacPro1,1" \
    -v "${PWD}/output.env:/env" \
    -v "${PWD}/mac_hdd_ng_testing.img:/image" \
    sickcodes/docker-osx:naked

To use iMessage or iCloud you need to change 5 values.

• SERIAL
• BOARD_SERIAL
• UUID
• MAC_ADDRESS

ROM is just the lowercased mac address, without : between each word.

You can tell the container to generate them for you using -e GENERATE_UNIQUE=true

Or tell the container to use specific ones using -e GENERATE_SPECIFIC=true

    -e GENERATE_SPECIFIC=true \
    -e DEVICE_MODEL="iMacPro1,1" \
    -e SERIAL="C02TW0WAHX87" \
    -e BOARD_SERIAL="C027251024NJG36UE" \
    -e UUID="5CCB366D-9118-4C61-A00A-E5BAF3BED451" \
    -e MAC_ADDRESS="A8:5C:2C:9A:46:2F" \

Changing display resolution

The display resolution is controlled by this line:

https://github.com/sickcodes/Docker-OSX/blob/master/custom/config-nopicker-custom.plist#L819

Instead of mounting that disk, Docker-OSX will generate a new OpenCore.qcow2 by using this one cool trick:

-e GENERATE_UNIQUE=true \
-e WIDTH=800 \
-e HEIGHT=600 \

To use WIDTH/HEIGHT, you must use with either -e GENERATE_UNIQUE=true or -e GENERATE_SPECIFIC=true.

It will take around 30 seconds longer to boot because it needs to make a new boot partition using libguestfs.

-e GENERATE_SPECIFIC=true \
-e WIDTH=1920 \
-e HEIGHT=1080 \
-e SERIAL="" \
-e BOARD_SERIAL="" \
-e UUID="" \
-e MAC_ADDRESS="" \

Change Docker-OSX Resolution Examples

# using an image in your current directory
stat mac_hdd_ng.img

docker run -it \
    --device /dev/kvm \
    -p 50922:10022 \
    -v "${PWD}/mac_hdd_ng.img:/image" \
    -v /tmp/.X11-unix:/tmp/.X11-unix \
    -e "DISPLAY=${DISPLAY:-:0.0}" \
    -e GENERATE_SPECIFIC=true \
    -e DEVICE_MODEL="iMacPro1,1" \
    -e SERIAL="C02TW0WAHX87" \
    -e BOARD_SERIAL="C027251024NJG36UE" \
    -e UUID="5CCB366D-9118-4C61-A00A-E5BAF3BED451" \
    -e MAC_ADDRESS="A8:5C:2C:9A:46:2F" \
    -e MASTER_PLIST_URL=https://raw.githubusercontent.com/sickcodes/Docker-OSX/master/custom/config-nopicker-custom.plist \
    -e WIDTH=1600 \
    -e HEIGHT=900 \
    sickcodes/docker-osx:naked

# generating random serial numbers, using the DIY installer, along with the screen resolution changes.
docker run -it \
    --device /dev/kvm \
    -p 50922:10022 \
    -v /tmp/.X11-unix:/tmp/.X11-unix \
    -e "DISPLAY=${DISPLAY:-:0.0}" \
    -e GENERATE_UNIQUE=true \
    -e WIDTH=800 \
    -e HEIGHT=600 \
    sickcodes/docker-osx:latest

Here's a few other resolutions! If you resolution is invalid, it will default to 800x600.

    -e WIDTH=800 \
    -e HEIGHT=600 \

    -e WIDTH=1280 \
    -e HEIGHT=768 \

    -e WIDTH=1600 \
    -e HEIGHT=900 \

    -e WIDTH=1920 \
    -e HEIGHT=1080 \

    -e WIDTH=2560 \
    -e HEIGHT=1600 \

This example shows how to change resolution after the container is created.

First step is to stop the docker daemon

sudo systemctl stop docker

The second step is to change container config in

/var/lib/docker/containers/[container-id]/config.v2.json

(Suppose your original WIDTH is 1024 and HEIGHT is 768, you can search 1024 and replace it with the new value. Same for 768.)

The last step is to restart the docker daemon

sudo systemctl restart docker

Mounting physical disks in Mac OSX

Pass the disk into the container as a volume and then pass the disk again into QEMU command line extras with.

Use the config-custom.plist because you probably want to see the boot menu, otherwise omit the first line:

DISK_TWO="${PWD}/mount_me.img"

-e MASTER_PLIST_URL='https://raw.githubusercontent.com/sickcodes/osx-serial-generator/master/config-custom.plist' \
-v "${DISK_TWO}:/disktwo" \
-e EXTRA='-device ide-hd,bus=sata.5,drive=DISK-TWO -drive id=DISK-TWO,if=none,file=/disktwo,format=qcow2' \

Physical disk mounting example

OSX_IMAGE="${PWD}/mac_hdd_ng_xcode_bigsur.img"
DISK_TWO="${PWD}/mount_me.img"

docker run -it \
    --device /dev/kvm \
    -e "DISPLAY=${DISPLAY:-:0.0}" \
    -v /tmp/.X11-unix:/tmp/.X11-unix \
    -e MASTER_PLIST_URL='https://raw.githubusercontent.com/sickcodes/osx-serial-generator/master/config-custom.plist' \
    -v "${OSX_IMAGE}":/image \
    -v "${DISK_TWO}":/disktwo \
    -e EXTRA='-device ide-hd,bus=sata.5,drive=DISK-TWO -drive id=DISK-TWO,if=none,file=/disktwo,format=qcow2' \
    sickcodes/docker-osx:naked

See also: here.

Extracting the APFS disk on Linux

In Docker-OSX, we are using qcow2 images.

This means the image grows as you use it, but the guest OS thinks you have 200GB available.

READ ONLY

# mount the qemu image like a real disk
sudo modprobe nbd max_part=8
sudo qemu-nbd --connect=/dev/nbd0 ./image.img
sudo fdisk /dev/nbd0 -l

mkdir -p ./mnt
sudo mount /dev/nbd0p1 ./mnt

# inspect partitions (2 partitions)
sudo fdisk /dev/nbd0 -l

# mount using apfs-linux-rw OR apfs-fuse
mkdir -p ./part

sudo mount /dev/nbd0p2 ./part
sudo apfs-fuse -o allow_other /dev/nbd0p2 ./part

When you are finishing looking at your disk, you can unmount the partition, the disk, and remove the loopback device:

sudo umount ./part
sudo umount ./mnt
sudo qemu-nbd --disconnect /dev/nbd0
sudo rmmod nbd

USB Passthrough

Firstly, QEMU must be started as root.

It is also potentially possible to accomplish USB passthrough by changing the permissions of the device in the container. See here.

For example, create a new Dockerfile with the following

FROM sickcodes/docker-osx
USER arch
RUN sed -i -e s/exec\ qemu/exec\ sudo\ qemu/ ./Launch.sh
COPY --chown=arch ./new_image.img /home/arch/OSX-KVM/mac_hdd_ng.img

Where new_image.img is the qcow2 image you extracted. Then rebuild with docker build .

Next we need to find out the bus and port numbers of the USB device we want to pass through to the VM:

lsusb -t
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/6p, 5000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/12p, 480M
    |__ Port 2: Dev 5, If 0, Class=Human Interface Device, Driver=usbhid, 12M
    |__ Port 2: Dev 5, If 1, Class=Chip/SmartCard, Driver=, 12M
    |__ Port 3: Dev 2, If 0, Class=Wireless, Driver=, 12M
    |__ Port 3: Dev 2, If 1, Class=Wireless, Driver=, 12M
    |__ Port 5: Dev 3, If 0, Class=Video, Driver=uvcvideo, 480M
    |__ Port 5: Dev 3, If 1, Class=Video, Driver=uvcvideo, 480M

In this example, we want to pass through a smartcard device. The device we want is on bus 1 and port 2.

There may also be differences if your device is usb 2.0 (ehci) vs usb 3.0 (xhci). See here for more details.

# hostbus and hostport correspond to the numbers from lsusb
# runs in privileged mode to enable access to the usb devices.
docker run \
  --privileged \
  --device /dev/kvm \
  -e RAM=4 \
  -p 50922:10022 \
  -e "DISPLAY=${DISPLAY:-:0.0}" \
  -e EXTRA="-device virtio-serial-pci -device usb-host,hostbus=1,hostport=2" \
  mycustomimage

You should see the device show up when you do system_profiler SPUSBDataType in the MacOS shell.

Important Note: this will cause the host system to lose access to the USB device while the VM is running!

Container creation examples

Quick Start your own image (naked container image)

This is my favourite container. You can supply an existing disk image as a Docker command line argument.

Pull images out using sudo find /var/lib/docker -size +10G | grep mac_hdd_ng.img

Supply your own local image with the command argument -v "${PWD}/mac_hdd_ng.img:/image" and use sickcodes/docker-osx:naked when instructing Docker to create your container.

Naked image is for booting any existing .img file, e.g in the current working directory ($PWD)

By default, this image has a variable called NOPICKER which is "true". This skips the disk selection menu. Use -e NOPICKER=false or any other string than the word true to enter the boot menu.

This lets you use other disks instead of skipping the boot menu, e.g. recovery disk or disk utility.

docker pull sickcodes/docker-osx:naked

# run your own image + SSH
# change mac_hdd_ng.img
docker run -it \
    --device /dev/kvm \
    -p 50922:10022 \
    -v "${PWD}/mac_hdd_ng.img:/image" \
    -v /tmp/.X11-unix:/tmp/.X11-unix \
    -e "DISPLAY=${DISPLAY:-:0.0}" \
    sickcodes/docker-osx:naked

# run local copy of the auto image + SSH + Boot menu
docker run -it \
    --device /dev/kvm \
    -p 50922:10022 \
    -v "${PWD}/mac_hdd_ng_auto.img:/image" \
    -v /tmp/.X11-unix:/tmp/.X11-unix \
    -e "DISPLAY=${DISPLAY:-:0.0}" \
    -e "NOPICKER=false" \
    sickcodes/docker-osx:naked

Building an OSX container with video output

The Quick Start command should work out of the box, provided that you keep the following lines. Works in auto & naked machines:

    -v /tmp/.X11-unix:/tmp/.X11-unix \
    -e "DISPLAY=${DISPLAY:-:0.0}" \

Prebuilt image with arbitrary command line arguments

-e OSX_COMMANDS lets you run any commands inside the container

docker pull sickcodes/docker-osx:auto

# boot to OS X shell + display + specify commands to run inside OS X!
docker run -it \
    --device /dev/kvm \
    -p 50922:10022 \
    -v /tmp/.X11-unix:/tmp/.X11-unix \
    -e "DISPLAY=${DISPLAY:-:0.0}" \
    -e "OSX_COMMANDS=/bin/bash -c \"put your commands here\"" \
    sickcodes/docker-osx:auto

# Boots in a minute or two!

OR if you have an image already and just want to log in and execute arbitrary commands:

docker pull sickcodes/docker-osx:naked-auto

# boot to OS X shell + display + specify commands to run inside OS X!
docker run -it \
    --device /dev/kvm \
    -p 50922:10022 \
    -v /tmp/.X11-unix:/tmp/.X11-unix \
    -e "DISPLAY=${DISPLAY:-:0.0}" \
    -e USERNAME=yourusername \
    -e PASSWORD=yourpassword \
    -e "OSX_COMMANDS=/bin/bash -c \"put your commands here\"" \
    sickcodes/docker-osx:naked-auto

# Boots in a minute or two!

Further examples

There's a myriad of other potential use cases that can work perfectly with Docker-OSX, some of which you'll see below!

Building a headless OSX container

For a headless container, remove the following two lines from your docker run command:

    # -v /tmp/.X11-unix:/tmp/.X11-unix \
    # -e "DISPLAY=${DISPLAY:-:0.0}" \

Building a headless container from a custom image

This is particularly helpful for CI/CD pipelines.

# run your own image headless + SSH
docker run -it \
    --device /dev/kvm \
    -p 50922:10022 \
    -v "${PWD}/mac_hdd_ng.img:/image" \
    sickcodes/docker-osx:naked

Building a headless container which allows insecure VNC on localhost (!for local use only!)

Must change -it to -i to be able to interact with the QEMU console

To exit a container using -i you must docker kill <containerid>. For example, to kill everything, docker ps | xargs docker kill.

Native QEMU VNC example

docker run -i \
    --device /dev/kvm \
    -p 50922:10022 \
    -p 5999:5999 \
    -v /tmp/.X11-unix:/tmp/.X11-unix \
    -e "DISPLAY=${DISPLAY:-:0.0}" \
    -e EXTRA="-display none -vnc 0.0.0.0:99,password=on" \
    sickcodes/docker-osx:big-sur

# type `change vnc password myvncusername` into the docker terminal and set a password
# connect to localhost:5999 using VNC
# qemu 6 seems to require a username for vnc now

NOT TLS/HTTPS Encrypted at all!

Or ssh -N root@1.1.1.1 -L 5999:127.0.0.1:5999, where 1.1.1.1 is your remote server IP.

(Note: if you close port 5999 and use the SSH tunnel, this becomes secure.)

Building a headless container to run remotely with secure VNC

Add the following line:

-e EXTRA="-display none -vnc 0.0.0.0:99,password=on"

In the Docker terminal, press enter until you see (qemu).

Type change vnc password someusername

Enter a password for your new vnc username^.

You also need the container IP: docker inspect <containerid> | jq -r '.[0].NetworkSettings.IPAddress'

Or ip n will usually show the container IP first.

Now VNC connect using the Docker container IP, for example 172.17.0.2:5999

Remote VNC over SSH: ssh -N root@1.1.1.1 -L 5999:172.17.0.2:5999, where 1.1.1.1 is your remote server IP and 172.17.0.2 is your LAN container IP.

Now you can direct connect VNC to any container built with this command!

I'd like to use SPICE instead of VNC

Optionally, you can enable the SPICE protocol, which allows use of remote-viewer to access your OSX container rather than VNC.

Note: -disable-ticketing will allow unauthenticated access to the VM. See the spice manual for help setting up authenticated access ("Ticketing").

  docker run \
    --device /dev/kvm \
    -p 3001:3001 \
    -p 50922:10022 \
    -e "DISPLAY=${DISPLAY:-:0.0}" \
    -e EXTRA="-monitor telnet::45454,server,nowait -nographic -serial null -spice disable-ticketing,port=3001" \
    mycustomimage

Then simply do remote-viewer spice://localhost:3001 and add --spice-debug for debugging.

Creating images based on an already configured and set up container

# You can create an image of an already configured and setup container.
# This allows you to effectively duplicate a system.
# To do this, run the following commands

# make note of your container id
docker ps --all
docker commit containerid newImageName

# To run this image do the following
docker run \
    --device /dev/kvm \
    --device /dev/snd \
    -v /tmp/.X11-unix:/tmp/.X11-unix \
    newImageName

docker pull sickcodes/docker-osx:auto

# boot directly into a real OS X shell with no display (Xvfb) [HEADLESS]
docker run -it \
    --device /dev/kvm \
    -p 50922:10022 \
    sickcodes/docker-osx:auto

# username is user
# passsword is alpine
# Wait 2-3 minutes until you drop into the shell.

Run the original version of Docker-OSX

docker pull sickcodes/docker-osx:latest

docker run -it \
    --device /dev/kvm \
    --device /dev/snd \
    -v /tmp/.X11-unix:/tmp/.X11-unix \
    -e "DISPLAY=${DISPLAY:-:0.0}" \
    sickcodes/docker-osx:latest

# press CTRL + G if your mouse gets stuck
# scroll down to troubleshooting if you have problems
# need more RAM and SSH on localhost -p 50922?

Run but enable SSH in OS X (Original Version)!

docker run -it \
    --device /dev/kvm \
    --device /dev/snd \
    -p 50922:10022 \
    -v /tmp/.X11-unix:/tmp/.X11-unix \
    -e "DISPLAY=${DISPLAY:-:0.0}" \
    sickcodes/docker-osx:latest

# turn on SSH after you've installed OS X in the "Sharing" settings.
ssh user@localhost -p 50922

Autoboot into OS X after you've installed everything

Add the extra option -e NOPICKER=true.

Old machines:

# find your containerID
docker ps

# move the no picker script on top of the Launch script
# NEW CONTAINERS
docker exec containerID mv ./Launch-nopicker.sh ./Launch.sh

# VNC-VERSION-CONTAINER
docker exec containerID mv ./Launch-nopicker.sh ./Launch_custom.sh

# LEGACY CONTAINERS
docker exec containerID bash -c "grep -v InstallMedia ./Launch.sh > ./Launch-nopicker.sh
chmod +x ./Launch-nopicker.sh
sed -i -e s/OpenCore\.qcow2/OpenCore\-nopicker\.qcow2/ ./Launch-nopicker.sh
"

The big-sur image starts slowly after installation. Is this expected?

Automatic updates are still on in the container's settings. You may wish to turn them off. We have future plans for development around this.

What is ${DISPLAY:-:0.0}?

$DISPLAY is the shell variable that refers to your X11 display server.

${DISPLAY} is the same, but allows you to join variables like this:

• e.g. ${DISPLAY}_${DISPLAY} would print :0.0_:0.0
• e.g. $DISPLAY_$DISPLAY would print :0.0

...because $DISPLAY_ is not $DISPLAY

${variable:-fallback} allows you to set a "fallback" variable to be substituted if $variable is not set.

You can also use ${variable:=fallback} to set that variable (in your current terminal).

In Docker-OSX, we assume, :0.0 is your default $DISPLAY variable.

You can see what yours is

echo $DISPLAY

That way, ${DISPLAY:-:0.0} will use whatever variable your X11 server has set for you, else :0.0

What is -v /tmp/.X11-unix:/tmp/.X11-unix?

-v is a Docker command-line option that lets you pass a volume to the container.

The directory that we are letting the Docker container use is a X server display socket.

/tmp/.X11-unix

If we let the Docker container use the same display socket as our own environment, then any applications you run inside the Docker container will show up on your screen too! https://www.x.org/archive/X11R6.8.0/doc/RELNOTES5.html

ALSA errors on startup or container creation

You may when initialising or booting into a container see errors from the (qemu) console of the following form: ALSA lib blahblahblah: (function name) returned error: no such file or directory. These are more or less expected. As long as you are able to boot into the container and everything is working, no reason to worry about these.

See also: here.

generate-specific-bootdisk.sh \
  --model "${DEVICE_MODEL}" \
  --serial "${SERIAL}" \
  --board-serial "${BOARD_SERIAL}" \
  --uuid "${UUID}" \
  --mac-address "${MAC_ADDRESS}" \
  --output-bootdisk OpenCore-nopicker.qcow2

Follow @sickcodes on Twitter

Click to join the Discord server https://discord.gg/sickchat

Click to join the Telegram server https://t.me/sickcodeschat

Or reach out via Linkedin if it's private: https://www.linkedin.com/in/sickcodes

Or via https://sick.codes/contact/

Download Details:

Author: Sickcodes
Source Code: https://github.com/sickcodes/Docker-OSX 
License: GPL-3.0 license

#docker #macos #container #os 

ChangeMenuBarColor: Change Menu Bar Color in MacOS Big Sur & Monterey

Change menu bar color in macOS Big Sur and Monterey

Simple utility to change macOS Big Sur and Monterey menu bar color by appending a solid color or gradient rectangle to a wallpaper image.

Motivation

Big Sur and Monterey changed the way the menu bar is displayed. It now adopts the color of the wallpaper, which may not always be what you would like. This utility allows you to specify the solid color or gradient of the menu bar you want to use.

Example

Imagine you have a dark wallpaper (here is mine). This dark wallpaper results in the menu bar being black even though you use the Light mode

With this utility you can generate a new wallpaper that makes the menu bar being shown in any color you want, like a nice tone of gray from Catalina.

Or a custom gradient

Installation

Mint

The easiest and preferred way to install and run the tools is with Mint.

Due to Swift Package Manager limitations you need to have Xcode installed and opened at least once for the installation from source code to work.

After you installed Xcode install Mint with Homebrew

brew install mint

Next run

mint run igorkulman/ChangeMenuBarColor

and if everything is OK you can use my utility be downloaded and run for the first time. You are ready to go!

Binary release

If you do not want to install Xcode and Mint you can download the latest binary release.

The binary release is not signed so it might not work on all systems.

Usage

All the commands shown are for the utility being installed via Mint. If you downloaded the latest binary release replace mint run igorkulman/ChangeMenuBarColorSolidColor with path/to/unzipped/ChangeMenuBarColorSolidColor.

Solid color

To set a new wallpaper file with a solid color rectangle that matches the menu bar, run

mint run igorkulman/ChangeMenuBarColorSolidColor "desired_hex_color" "optional_path_to_your_wallpaper" 

So for example

mint run igorkulman/ChangeMenuBarColor SolidColor "#CCCCCC" "/Users/igorkulman/wallpaper.jpg"

If you do not provide the wallpaper path

mint run igorkulman/ChangeMenuBarColor SolidColor "#CCCCCC"

the currently set wallpaper will be used.

Gradient

To set a new wallpaper file with a gradient rectangle at the top, run

mint run igorkulman/ChangeMenuBarColor Gradient "start_hex_color" "end_hex_color" "optional_path_to_your_wallpaper"

So for example

mint run igorkulman/ChangeMenuBarColor Gradient "#FF0000" "#00FF00" "/Users/igorkulman/wallpaper.jpg"

If you do not provide the wallpaper path

mint run igorkulman/ChangeMenuBarColor Gradient "#FF0000" "#00FF00"

the currently set wallpaper will be used.

Multiple displays

If you use multiple displays and want the wallpaper generated for all of them, add the --all-displays flag at the end of the command, so for example ./ChangeMenuBarColor Gradient "#FF0000" "#00FF00" --all-displays.

Uninstall

• manually set any new wallpaper
• run mint uninstall igorkulman/ChangeMenuBarColor if you installed the utility via mint or delete the folder with the utility if you downloaded it manually.

Support the project

Known issues

Make sure Automatically hide and show the menu bar is disabled. This setting causes the utility to incorrectly detect the menu bar size.

Dynamic wallpapers are not supported at the moment. If you use a dynamic wallpaper the utility will convert it to a static .jpg image.

Catalina support

The utility builds and runs on Catalina but the menu bar on Catalina works in a different way that on Big Sur and Monterey so the result will never be the same as on Big Sur and Monterey.

Contributing

All contributions are welcomed!

Download Details:

Author: igorkulman
Source Code: https://github.com/igorkulman/ChangeMenuBarColor 
License: MIT license

#swift #xcode #macos