Rupert  Beatty

Rupert Beatty


MochiDiffusion: Run Stable Diffusion on Mac Natively

Mochi Diffusion

Run Stable Diffusion on Mac natively



This app uses Apple's Core ML Stable Diffusion implementation to achieve maximum performance and speed on Apple Silicon based Macs while reducing memory requirements. It also runs on Intel based Macs too.


  • Extremely fast and memory efficient (~150MB with Neural Engine)
  • Runs well on all Apple Silicon Macs by fully utilizing Neural Engine
  • Generate images locally and completely offline
  • Generated images are saved with prompt info inside EXIF metadata (view in Finder's Get Info window)
  • Convert generated images to high resolution (using RealESRGAN)
  • Use custom Stable Diffusion Core ML models
  • No worries about pickled models
  • macOS native app using SwiftUI


Download the latest version from the releases page.


When using a model for the very first time, it may take up to 2 minutes for the Neural Engine to compile a cached version. Afterwards, subsequent generations will be much faster.

Compute Unit

  • CPU & Neural Engine provides a good balance between speed and low memory usage
  • CPU & GPU may be faster on M1 Max, Ultra and later but will use more memory

Depending on the option chosen, you will need to use the correct model version (see Models section for details).

Intel Macs uses CPU & GPU as it doesn't have Neural Engine.


You will need to convert or download Core ML models in order to use Mochi Diffusion.

A few models have been converted and uploaded here.

  1. Convert or download Core ML models
    • split_einsum version is compatible with all compute unit options including Neural Engine
    • original version is only compatible with CPU & GPU option
  2. By default, the app's model folder will be created under the Documents folder. This location can be customized under Settings
  3. In the model folder, create a new folder with the name you'd like displayed in the app then move or extract the converted models here
  4. Your directory should look like this:
└── MochiDiffusion/
    └── models/
        ├── stable-diffusion-2-1_split-einsum_compiled/
        │   ├── merges.txt
        │   ├── TextEncoder.mlmodelc
        │   ├── Unet.mlmodelc
        │   ├── VAEDecoder.mlmodelc
        │   └── vocab.json
        ├── ...
        └── ...


  • Apple Silicon (M1 and later) or Intel Mac (high performance CPU & GPU required)
  • macOS Ventura 13.1 and later
  • Xcode 14.2 (to build)


All generation happens locally and absolutely nothing is sent to the cloud.


Mochi Diffusion is always looking for contributions, whether it's through bug reports, code, or new translations.

If you find a bug, or would like to suggest a new feature or enhancement, try searching for your problem first as it helps avoid duplicates. If you can't find your issue, feel free to create a new issue. Don't create an issue for your question as those are for bugs and feature requests only.

If you're looking to contribute code, feel free to open a Pull Request. I recommend installing SwiftLint to catch lint issues.

If you'd like to translate Mochi Diffusion to your language, please visit the project page on Crowdin. You can create an account for free and start translating and/or approving.


English, 한국어, 中文

Download Details:

Author: Godly-devotion
Source Code: 
License: GPL-3.0 license

#swift #macos #apple #coreml 

MochiDiffusion: Run Stable Diffusion on Mac Natively
Rupert  Beatty

Rupert Beatty


The MobileNet Neural Network using Apple's New CoreML Framework

MobileNet with CoreML

This is the MobileNet neural network architecture from the paper MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications implemented using Apple's shiny new CoreML framework.

This uses the pretrained weights from shicai/MobileNet-Caffe.

There are two demo apps included:

Cat Demo. Shows the prediction for a cat picture. Open the project in Xcode 9 and run it on a device with iOS 11 or on the simulator.

Camera Demo. Runs from a live video feed and performs a prediction as often as it can manage. (You'll need to run this app on a device, it won't work in the simulator.)

The cat demo app

Note: Also check out Forge, my neural net library for iOS 10 that comes with a version of MobileNet implemented in Metal.

Converting the weights

The repo already includes a fully-baked MobileNet.mlmodel, so you don't have to follow the steps in this section. However, in case you're curious, here's how I converted the original Caffe model into this .mlmodel file:

Note: You don't have to download mobilenet_deploy.prototxt. There's already one included in this repo. (I added a Softmax layer at the end, which is missing from the original.)

  • From a Terminal, do the following:
$ virtualenv -p /usr/bin/python2.7 env
$ source env/bin/activate
$ pip install tensorflow
$ pip install keras==1.2.2
$ pip install coremltools

It's important that you set up the virtual environment using /usr/bin/python2.7. If you use another version of Python, the conversion script will crash with Fatal Python error: PyThreadState_Get: no current thread. You also need to use Keras 1.2.2 and not the newer 2.0.

  • Run the script to do the conversion:
$ python

This creates the MobileNet.mlmodel file.

  • Clean up by deactivating the virtualenv:
$ deactivate


Download Details:

Author: Hollance
Source Code: 

#swift #ios #machinelearning #mobilenet #coreml 

The MobileNet Neural Network using Apple's New CoreML Framework
Royce  Reinger

Royce Reinger


Core ML tools Contain Supporting Tools for Core ML Model Conversion

Core ML Tools

Use coremltools to convert machine learning models from third-party libraries to the Core ML format. This Python package contains the supporting tools for converting models from training libraries such as the following:

With coremltools, you can do the following:

  • Convert trained models to the Core ML format.
  • Read, write, and optimize Core ML models.
  • Verify conversion/creation (on macOS) by making predictions using Core ML.

After conversion, you can integrate the Core ML models with your app using Xcode.

Version 6

The coremltools 6 package offers new features to optimize the model conversion process. For details, see New in coremltools.

For a full list of changes, see Release Notes.

To install coremltools 6.0 use the following command:

pip install coremltools

Version 5

The coremltools 5 package offers several performance improvements over previous versions, including new features. For details, see New in coremltools.

Core ML

Core ML is an Apple framework to integrate machine learning models into your app. Core ML provides a unified representation for all models. Your app uses Core ML APIs and user data to make predictions, and to fine-tune models, all on the user’s device. Core ML optimizes on-device performance by leveraging the CPU, GPU, and Neural Engine while minimizing its memory footprint and power consumption. Running a model strictly on the user’s device removes any need for a network connection, which helps keep the user’s data private and your app responsive.


To install coremltools, see the “Installation“ page. For more information, see the following:

Download Details:

Author: Apple
Source Code: 
License: BSD-3-Clause license

#machinelearning #tensorflow #pytorch #coreml 

Core ML tools Contain Supporting Tools for Core ML Model Conversion
Rupert  Beatty

Rupert Beatty


CoreMLHelpers: Types & Functions That Make It A Little Easier to Work


This is a collection of types and functions that make it a little easier to work with Core ML in Swift.

Some of the things CoreMLHelpers has to offer:

Experimental features:

Let me know if there's anything else you'd like to see added to this library!

💡 Tip: Get the Core ML Survival Guide

If Core ML is giving you trouble --- or if you want to learn more about using the Core ML and Vision APIs --- then check out my book Core ML Survival Guide. It has 400+ pages of Core ML tips and tricks.

I wrote the Core ML Survival Guide because the same questions kept coming up on Stack Overflow, on the Apple Developer Forums, and on this GitHub repo. Core ML may appear easy-to-use at first --- but if you want to go beyond the basics, the learning curve suddenly becomes very steep. My goal with this book is to make the advanced features of Core ML accessible to everyone too.

The Core ML Survival Guide currently has over 80 chapters and includes pretty much everything I know about Core ML. As I learn new things I'll keep updating the book, so you'll always have access to the most up-to-date knowledge about Core ML. Cheers!

How to use CoreMLHelpers

Copy the source files from the CoreMLHelpers folder into your project. You probably don't need all of them, so just pick the files you require and ignore the rest.

Note: A lot of the code in CoreMLHelpers is only intended as a demonstration of how to approach a certain problem. There's often more than one way to do it. It's quite likely you will need to customize the code for your particular situation, so consider these routines a starting point.

Why can't I use this with Swift Package Manager / Carthage / CocoaPods?

I believe a proper framework should have a well-thought-out API but CoreMLHelpers is a hodgepodge of helper functions that isn't particularly well-organized. Putting this into a package makes things more complicated than necessary. Just copy the one or two source files you need into your project, and adapt them to your needs.

Read more about Core ML


  • proper unit tests
  • add more numpy-like functionality to MultiArray (and fix the bugs!)

Download Details:

Author: Hollance
Source Code: 
License: MIT license

#swift #ios #machinelearning #coreml 

CoreMLHelpers: Types & Functions That Make It A Little Easier to Work

Awesome CoreML Models: Largest List Of Models for Core ML

Since iOS 11, Apple released Core ML framework to help developers integrate machine learning models into applications. The official documentation

We've put up the largest collection of machine learning models in Core ML format, to help iOS, macOS, tvOS, and watchOS developers experiment with machine learning techniques.

If you've converted a Core ML model, feel free to submit a pull request.

Recently, we've included visualization tools. And here's one Netron


Image - Metadata/Text

Models that take image data as input and output useful information about the image.

Image - Image

Models that transform images.

Text - Metadata/Text

Models that process text data


Visualization Tools

Tools that help visualize CoreML Models

Supported formats

List of model formats that could be converted to Core ML with examples

The Gold

Collections of machine learning models that could be converted to Core ML

Individual machine learning models that could be converted to Core ML. We'll keep adjusting the list as they become converted.

  • LaMem Score the memorability of pictures.
  • ILGnet The aesthetic evaluation of images.
  • Colorization Automatic colorization using deep neural networks.
  • Illustration2Vec Estimating a set of tags and extracting semantic feature vectors from given illustrations.
  • CTPN Detecting text in natural image.
  • Image Analogy Find semantically-meaningful dense correspondences between two input images.
  • iLID Automatic spoken language identification.
  • Fashion Detection Cloth detection from images.
  • Saliency The prediction of salient areas in images has been traditionally addressed with hand-crafted features.
  • Face Detection Detect face from image.
  • mtcnn Joint Face Detection and Alignment.
  • deephorizon Single image horizon line estimation.

Contributing and License

  • See the guide
  • Distributed under the MIT license. See LICENSE for more information.

Download Details:

Author: likedan
Source code:

License: MIT license


Awesome CoreML Models: Largest List Of Models for Core ML

How to Split Conference Semantics using Core ML


This project is Object Segmentation on iOS with Core ML.
If you are interested in iOS + Machine Learning, visit here you can see various DEMOs.


How it works

When use Metal



  • Xcode 10.2+
  • iOS 12.0+
  • Swift 5



Download model from apple's model page.


NameInputOutputSizeiOS version+Download
DeepLabV3Image (Color 513 × 513)MultiArray (Int32 513 × 513)8.6 MBiOS 12.0+link
DeepLabV3FP16Image (Color 513 × 513)MultiArray (Int32 513 × 513)4.3 MBiOS 12.0+link
DeepLabV3Int8LUTImage (Color 513 × 513)MultiArray (Int32 513 × 513)2.3 MBiOS 12.0+link
FaceParsingImage (Color 512 × 512)MultiArray (Int32) 512 × 51252.7 MBiOS 14.0+link

Inference Time − DeepLabV3

DeviceInference TimeTotal Time (GPU)Total Time (CPU)
iPhone 12 Pro29 ms29 ms240 ms
iPhone 12 Pro Max
iPhone 1230 ms31 ms253 ms
iPhone 12 Mini29 ms30 ms226 ms
iPhone 11 Pro39 ms40 ms290 ms
iPhone 11 Pro Max35 ms36 ms280 ms
iPhone 11
iPhone SE (2nd)
iPhone XS Max
iPhone XS54 ms55 ms327 ms
iPhone XR133 ms402 ms
iPhone X137 ms143 ms376 ms
iPhone 8+140 ms146 ms420 ms
iPhone 8189 ms529 ms
iPhone 7+240 ms667 ms
iPhone 7192 ms208 ms528 ms
iPhone 6S +309 ms1015 ms

need to measure

Inference Time − FaceParsing

DeviceInference TimeTotal Time (GPU)Total Time (CPU)
iPhone 12 Pro
iPhone 11 Pro37 ms37 ms

Labels − DeepLabV3

# total 21
["background", "aeroplane", "bicycle", "bird", "boat", 
"bottle", "bus", "car", "cat", "chair", 
"cow", "diningtable", "dog", "horse", "motorbike", 
"person", "pottedplant", "sheep", "sofa", "train", 

Labels − FaceParsing

# total 19
["background", "skin", "l_brow", "r_brow", "l_eye", 
"r_eye", "eye_g", "l_ear", "r_ear", "ear_r", 
"nose", "mouth", "u_lip", "l_lip", "neck", 
"neck_l", "cloth", "hair", "hat"]

See also

Download Details:

Author: tucan9389
Download Link: Download The Source Code
Official Website: 
License: MIT license

#machinelearning  #coreml 

How to Split Conference Semantics using Core ML
Vida  Herzog

Vida Herzog


How to Use Turi Create Tool To Train Object Detection Model

Video contains tutorial on how to use Turi Create tool to train CoreML Object Detection Model using One Shot Learning technique (just 1 sample image). Created model is exported to CoreML format and used in iOS app, which detects this object on provided photos.

Turi Create:


How to Use Turi Create Tool To Train Object Detection Model
Dominic  Feeney

Dominic Feeney


Consistency of Bilinear Upsampling Layer

The code for TensorFlow-Keras 2.5 and Apple Core ML which generates consistent results through training to prediction


It is well known among deep-learning manias that bilinear upsampling layers in TensorFlow have pixel-offset issues. This has been partly fixed by adding an ‘align_corner’ attribute to them in TensorFlow 2.x. But the problem remains to cause inconsistent computation flow when exporting a trained model in TensorFlow into another DL framework through various versions.

In my case, a neural network model with bilinear upsampling layers showed weird behavior when converting the trained model from TensorFlow 2.5 to Apple Core ML by using coremltools 3.4. After uncountable coding, trials, and delete-delete-delete, I nearly gave up the consistent results of the upsampling layer between TensorFlow and Core ML.

I wanted to use Keras in the latest TensorFlow 2.5 for training in Windows PC, and I wanted to use the previous coremltools 3.4 for converting the trained model to Core ML for my macOS laptop. This is because the version 2.5 has stable Automatic Mixed Precision computing, and because I could not use coremltools 4.x for TF 2.5 due to dependency errors in anaconda and pip in macOS.

What is the pixel-offset issue?

Some respectful programmers provide good explanations of this (troublesome) specification defined in TensorFlow. They are helpful for me, and maybe for you too:

#tensorflow #coreml #deep-learning #keras #consistency #consistency of bilinear upsampling layer

Consistency of Bilinear Upsampling Layer

Converting PyTorch & TensorFlow Models Into Apple Core ML Using CoreMLTools

Core ML is an Apple framework that allows developers to integrate machine learning/deep learning models into their applications. However, it does not support model creation and training, i.e., you first need to create the model in a framework like TensorFlow or PyTorch, then you can convert and use it. There are two ways you can convert your machine learning model from the framework of your choice to the Core ML model format: through an intermediary model format like ONNX or by using Apple’s own CoreMLTools Python library.

Although ONNX works just fine for the conversion, CoreMLTools offers other useful functionalities like model optimization. Also, you’ll need to use CoreMLTools for the final conversion from ONNX format to Core ML format anyway. Currently, it supports the conversion of models created using the following libraries:

  • PyTorch
  • TensorFlow 1.x & 2.x
  • TensorFlow’s Keras APIs
  • scikit-learn
  • XGBoost
  • LibSVM

#developers corner #apple ai #apple machine learning #core ml3 #coreml #coremltools #model conversion #python libraries

Converting PyTorch & TensorFlow Models Into Apple Core ML Using CoreMLTools
Hertha  Mayer

Hertha Mayer


Core ML + ARKit: Annotating objects in Augmented Reality

Isn’t it magic to annotate objects in AR?

I was inspired by this example of Core ML + ARKit. But I found one significant disadvantage — it doesn’t place annotations on objects automatically. Instead, you need to center the object in your camera view and use your finger to place an annotation.

In my opinion, this destroys user expectations, so I decided to fix that and build a more immersive user experience using object detection in augmented reality.

To follow this tutorial, you should be aware of the basics of iOS development using Swift and be familiar (at least somewhat) with Core ML and ARKit. Also, you need an iPhone or iPad with iOS 13+ or iPadOS 13+, respectively.

Get started

  1. Download the source code.
  2. Open the project and navigate to Signing & Capabilities.
  3. Select your Apple Developer account, add the bundle identifier, and create a provisioning profile (I recommend using Automatically manage signing).
  4. Connect your device.
  5. Build and run.
  6. Enjoy!

Now let’s see what happens under the hood

Out app has two main entities. The first one is the object detection service (shown below): it takes an image as input and returns a bounding box and class label of the recognized object. The second is the ViewController, the place where all the AR magic happens:

Below are the steps as identified in the inline comments in the code block above

  1. Initialize the Core ML model: In this case, we’re working with a _Swifty_representation of a pre-trained machine learning model. Apple provides dozens of open-source models here. Also, you can create your own by using tools like Create ML.
  2. Initialize an image analysis request, which uses the Core ML model to process input images.
  3. detect method that instantiates the handler to perform Vision requests on a single image. It uses a Core Video pixel buffer because it can be easily taken from the current ARFrame, since pixel buffer doesn’t store information about the current image orientation. We take the current device orientation and map it into an Exif orientation format.

#mobile-machine-learning #coreml #heartbeat #augmented-reality #ios-app-development

Core ML + ARKit: Annotating objects in Augmented Reality
Carmen  Grimes

Carmen Grimes


Use Apple new ML Compute framework to accelerate ML training and inferencing

With the recent new wave of operating system versions (BigSur, iOS 14 etc.), announced at recent WWDC, Apple quite silently introduced a new ML framework to accelerate training of neural networks across the CPU or one or more available GPUs.

ML Compute is not properly a new ML framework but new API’s that utilizes the high performance BNNS primitives made available by the Accelerate framework for the CPU and Metal Performance Shaders for the GPU.

After looking at the documentation and starting to use it on an iOS/macOS application, I understood that this is not really a simple, high level framework but something probably targeting the acceleration of existing third-party ML library like the ONNX Runtime or the TensorFlow Lite framework on the Apple platform.

Even if Apple documentation is pretty good, I would say these APIs are not really developer friendly and swifty for doing generic ML on iOS/macOS. The tensor API, for example, is really rough and it requires dealing with unmanaged pointers in Swift. Basically, you are responsible for managing by yourself the ownership and lifetime of memory allocation of objects like tensors, nodes and graphs that you pass to these API.

More generally, ML Compute does not provide by definition ML APIs like Keras, PyTorch or Swift for TensorFlow to simplify building and training ML models but low level API to build computing graph and manage low level training loop.

For general ML coding on iOS/macOS, I would suggest continuing to use Core ML with tools like CoreMLTools to import model from other framework (TensorFlow, PyTorch etc.) or eventually give a try to the SwiftCoreMLTools library I developed if you want to completely build and/or train model locally on devices avoiding any Python code.

Anyway my personal opinion, after playing with that, is that ML Compute could become potentially really powerful even for regular Swift ML developer adding for example on top of that a Swift Function Builder (DSL) high level API, like the one I developed for SwiftCoreMLTools, and a high level Tensor Swift API, hopefully, integrated with Swift Numerics multi dimensional array.

To quickly test the capability of these APIs, I decided to develop and illustrate a PoC App to train and inference with ML Compute on both iOS and macOS a simple shallow model for the MNIST dataset.

Image for post

#swift #mlcompute #machine-learning #coreml #gpu #cpu

Use Apple new ML Compute framework to accelerate ML training and inferencing

Creating artistic live video filters with deep neural networks

In a previous project, I worked on replicating fast neural style transfer, transforming an image by taking the artistic styling from one image and applying it to another image through deep neural networks. While transforming images in a python notebook works well, it is not very accessible to the average user. I wanted to deploy the model on an iOS device similar to the Prisma app made popular a few years ago. More than that, I wanted to test the limits of a generative model and transform frames on live video feed. The goal of this project was to run a generative model on real time video, exploring what is possible given the current boundaries of the technology. There are a few things that made this possible — 1) scaling inputs, 2) utilizing the device’s GPU, and 3) simplifying the model. Since this builds upon the previous project, some familiarity with the previous post will be helpful.

Scaling Inputs

Many phones today can take stunning 4k videos, including the iPhone XS that I developed on. While the A12 chip in the device is powerful, it would be far too slow to use a deep neural network on every frame of that size. Usually video frames are downscaled for image recognition on devices and the model is run on a subset of frames. For instance, an object recognition app may run a model every second on a 224 x 244 frame, instead of 30 times per second on a 4096 x 2160 frame. That works in an object detection use case, as objects don’t change that much between frames.

This obviously won’t work for stylizing video frames. Having only a single stylized frame flicker every second would not be appealing to a user. However, there are some takeaways from this. First, it is completely reasonable to downscale the frame size. It is common for video to be streamed at 360p and scaled up to the device’s 1080p screen. Second, perhaps running a model on 30 frames per second is not necessary and a slower frame rate would be sufficient.

There is a trade-off between model resolution and frame rate, as there are a limited number of computations the GPU can make in a second. You may see some video chat platforms have a slower frame rate or more buffering when using convolutions for video effects (i.e. changing the background). To get a sense of what different frame rates and input shapes looked like, I created a few stylized videos on a computer with the original neural network and OpenCV. I settled on a goal frame rate of 15 fps with 480 x 853 inputs. I found these to still be visually appealing as well as recognizable numbers for benchmark testing.

Utilizing the GPU

I used tfcoreml and coremltools to transform the Tensorflow model to a CoreML model. A gist of the complete method can be found below. There were a couple of considerations with this. First, I moved to batch normalization instead of instance normalization. This was because CoreML does not have an instance normalization layer out of the box, and this simplified the implementation since only one frame would be in each batch at interference time. A custom method could also be used in tfcoreml to convert the instance normalization layer.

#machine-learning #style-transfer #coreml #iphone #deep-learning #deep learning

Creating artistic live video filters with deep neural networks
Fredy  Larson

Fredy Larson


The Heartbeat Newsletter: Volume 1

Greetings Heartbeat Readers & Community,

Our sincere thanks for your continued support, readership, and contributions, which have made the last two years truly something special to be a part of.

We’ve grown a lot and undergone a number of exciting changes during that time, but one thing has remained consistent—the incredible blog posts that our contributing writers continue to share with us. We’ve seen so much growth in both those contributors and in the technologies we’re fascinated by: Machine learning, mobile development, and the intersection of the two.

So we want to provide you with a weekly look at some of the incredible new work we’re sharing, some classics that we think are worth revisiting, and some fun surprises along the way. So without further ado, your first issue of The Heartbeat Newsletter.

Happy Reading,

Austin & the Heartbeat Team

#machine-learning #coreml #mobile-machine-learning #snapchat #mobile-app-development

The Heartbeat Newsletter: Volume 1
Amara  Reichel

Amara Reichel


Core ML Convolutional Neural Network On-Device

Did you know you can fully train a LeNet convolutional neural network (CNN) directly on iOS devices? And that the performance isn’t bad at all ?!


 Core ML Convolutional Neural Network On-Device