square-zero-labs/modnet

MODNet: Trimap-Free Portrait Matting in Real Time

For more information, check out the official repository and example colab.

Usage (Transformers.js)

If you haven't already, you can install the Transformers.js JavaScript library from NPM using:

npm i @huggingface/transformers

You can then use the model for portrait matting, as follows:

import { pipeline } from '@huggingface/transformers';

const segmenter = await pipeline('background-removal', 'Xenova/modnet', { dtype: 'fp32' });
const url = 'https://images.pexels.com/photos/5965592/pexels-photo-5965592.jpeg?auto=compress&cs=tinysrgb&w=1024';
const output = await segmenter(url);
output[0].save('mask.png');
// You can also use `output[0].toCanvas()` or `await output[0].toBlob()` if you would like to access the output without saving.

Or with the AutoModel and AutoProcessor APIs:

import { AutoModel, AutoProcessor, RawImage } from '@huggingface/transformers';

// Load model and processor
const model = await AutoModel.from_pretrained('Xenova/modnet', { dtype: 'fp32' });
const processor = await AutoProcessor.from_pretrained('Xenova/modnet');

// Load image from URL
const url = 'https://images.pexels.com/photos/5965592/pexels-photo-5965592.jpeg?auto=compress&cs=tinysrgb&w=1024';
const image = await RawImage.fromURL(url);

// Pre-process image
const { pixel_values } = await processor(image);

// Predict alpha matte
const { output } = await model({ input: pixel_values });

// Save output mask
const mask = await RawImage.fromTensor(output[0].mul(255).to('uint8')).resize(image.width, image.height);
mask.save('mask.png');

Input image	Output mask

Note: Having a separate repo for ONNX weights is intended to be a temporary solution until WebML gains more traction. If you would like to make your models web-ready, we recommend converting to ONNX using 🤗 Optimum and structuring your repo like this one (with ONNX weights located in a subfolder named onnx).