Question: How to perform OCR on Camera's live frame data?

[Kommandat]

Really more of a question (or a feature request), but is it possible to perform OCR on the live frame data from the Camera?

In my app I was trying to get around this by repeatedly taking pictures at regular intervals and scanning them to show red bounding boxes for the text (video below). This would've worked fine, except iOS makes a loud shutter noise when you take a picture (and a shutter noise every 500ms is really really annoying).

It looks like expo-camera (which is what I'm using) doesn't access the live frame data but there are some other options out there that do, but I don't know how to get them to work with MLKit:

• https://js.tensorflow.org/api_react_native/0.2.1/#cameraWithTensors
• https://mrousavy.com/react-native-vision-camera/docs/guides/frame-processors-plugins-ios

Thanks! Appreciate this package a ton and understand what I'm asking here might be difficult to do. If so, I'm also curious to learn more and help out.

RPReplay_Final1632580413.mov

[agoldis]

Good question! Let's break it down, here's the pipeline:

1. Get a feed of images from the camera > 2. Input the images into MLKit (this library) > 3. augment the preview screen with OCR data.

Seems like https://github.com/tensorflow/tfjs/blob/d8a8afeeb9218e39655712c3d8d26977371054bf/tfjs-react-native/src/camera/camera_stream.tsx#L369 would be a good reference for 1 and 3.

I would start with 1 and 3 - i.e. to make sure that I am able to receive a feed of images and augment them (with a static text). 1 would produce a stream of in-memory, platform-specific abstractions of "image". We'll need to feed those images into mlkit. This library need an enhancement - to be able to receive the "image" object instead of a file URL. It is already supported in native packages, we'll just need to expose those methods to RN.

• https://developers.google.com/ml-kit/vision/text-recognition/v2/ios#objc
• https://developers.google.com/ml-kit/vision/text-recognition/v2/android#using-a-bytebuffer-or-bytearray

[Kommandat]

Thanks for outlining the steps! I don't have time to do this now, but I'm definitely interested in tackling this and making the enhancement to this library in ~1 month or so.

A few questions:

• The Tensorflow Camera Stream returns an iterable of Tensor3Ds. How would this be converted into a CMSampleBufferRef or ByteBuffer / ByteArray that MLKit needs? Do you think Tensor3D.buffer() is sufficient?
• Which files in this library should I familiarize myself with to understand what would be needed for this enhancement?
• I'm not too familiar with how to integrate native code with React native (so far I've just been working with React Native & Expo, using libraries like this one when I need native functionality). Do you have any recommended resources to learn more on this topic?

Thanks!

[Noitidart]

Thanks all for sharing this, this is complicated and I was also trying frame processor, but was having a tough time.

[agoldis]

• The Tensorflow Camera Stream returns an iterable of Tensor3Ds. How would this be converted into a CMSampleBufferRef or ByteBuffer / ByteArray that MLKit needs? Do you think Tensor3D.buffer() is sufficient?

I am not too familiar with Tensor and its terms. The referenced file https://github.com/tensorflow/tfjs/blob/d8a8afeeb9218e39655712c3d8d26977371054bf/tfjs-react-native/src/camera/camera_stream.tsx#L291 integrates with Expo Camera package. It "renders" two React elements;

return [cameraComp, glViewComponent];

The first one "starts" expo camera, the seconds one shows a GLView that render augmented camera output. Seems like all the magic is happening in nextFrameGenerator.

Replacing Tensor logic with mlkit logic would produce the desired effect I think.

Which files in this library should I familiarize myself with to understand what would be needed for this enhancement?

I'm not too familiar with how to integrate native code with React native (so far I've just been working with React Native & Expo, using libraries like this one when I need native functionality). Do you have any recommended resources to learn more on this topic?

Just follow the official RN guide - it's quite a detailed documentation and this package is very "usual" in terms of implementation - it's just one class for each platform.