How to Make HTML5 Barcode Reader with Desktop and Mobile Cameras

Recently, I noticed someone asked whether Dynamsoft Barcode Reader SDK can work for mobile devices. Because Dynamsoft Barcode Reader SDK does not support ARM yet, we can’t use it to develop native mobile apps. However, don’t be frustrated. Inspired by the article – Face detection using HTML5, javascript, webrtc, websockets, Jetty and OpenCV, I figured out the alternative solution. You can transmit the real-time camera stream to a remote server using HTML5 in Web browsers and decode the barcode image with Dynamsoft Barcode Reader SDK on the server that running on Windows, Linux or Mac. In this article, I will illustrate how to implement an HTML5 Barcode Reader with Cameras for PC and mobile devices step by step.

Read more

How to Use Edge.js to Empower WebSocket Solutions in JavaScript

A few days ago, I read the article – Run .NET and Node.js code in-process with Edge.js, which inspired me to try a new way of WebSocket solutions in JavaScript. It is known that Node.js makes applications, which written in JavaScript, run on server-side. Let’s see how we can quickly implement image transmission between servers and clients in a few lines of code.

Read more

Document Web Scanning in HTML5 and Java

In this tutorial, I am about to make a more complicated solution – Web-based document imaging capture, combining the implementation of Java TWAIN and Java WebSocket with Dynamic .NET TWAIN library, C#, JNI, Java Swing, Jetty and HTML5 WebSocket. Here is the workflow:

Java Web TWAIN workflow

Prerequisites

You need to read the following articles first and try relevant sample code.

How to Run

  1. In Eclipse, run the project as Java Application, and select the class UIMain.jtwain server

    When you see the window, the WebSocket server has already been initialized.

    •  The button Load is used to load images from the local disk.
    • The button Send is used to push images from the WebSocket server to Web clients.
    • The button Scan is used to scan documents locally and automatically send captured images to remote Web clients.
  2. Open a Web client in Chrome, and select a source to scan.jtwain source
  3. Display the captured image.jtwain scan

The Anatomy of Web-based Document Imaging Capture

Import all relevant libraries to the new project.

jtwain libs

Modify the source code of Java TWAIN and Java Websocket to make them parts of the new project.

Create a class named SourceManager, and add all methods from Java TWAIN project:

package com.data;
import java.awt.event.ActionEvent;
import java.awt.event.ActionListener;
import java.io.File;
import java.util.ArrayList;

import javatwain.DotNetScanner;
import javatwain.IJavaProxy;
import javatwain.INativeProxy;

import javax.swing.Timer;

import com.server.WSHandler;
import com.util.ImageUtil;

import net.sf.jni4net.Bridge;

public class SourceManager implements INativeProxy {
	private IJavaProxy mScanner;
	private String[] mSources;
	private ScanAction mScanAction;

    public SourceManager() {
		initTWAIN();
		mScanAction = new ScanAction();
    }

	private void initTWAIN() {
		try {
			Bridge.init();
			Bridge.LoadAndRegisterAssemblyFrom(new java.io.File("libs\\jni\\JavaTwain.j4n.dll"));
		}
		catch (Exception e) {
            e.printStackTrace();
        }

		mScanner = new DotNetScanner();
		mScanner.RegisterListener(this);
		mSources = mScanner.GetSources();
	}

	public String[]	getSources() {
		return mSources;
	}

	public synchronized void acquireImage(int index) {
		mScanAction.setIndex(index);
		mScanAction.start();
	}

    @Override
	public boolean Notify(String message, String value) {
    	ArrayList<WSHandler> sessions = WSHandler.getAllSessions();
    	for (WSHandler session : sessions) {
    		session.sendImage(ImageUtil.getImageBytes(new File(value)));
    	}

        return true;
    }

    public class ScanAction {
    	private int mIndex;
    	private int mDelay = 1;
    	private Timer mTimer;

    	public ScanAction() {
    		mTimer = new Timer(mDelay, mTaskPerformer);
    		mTimer.setRepeats(false);
    	}

    	private ActionListener mTaskPerformer = new ActionListener() {
            @Override
    		public void actionPerformed(ActionEvent evt) {
        		mScanner.AcquireImage(mIndex);
            	ActionListener taskPerformer = new ActionListener() {
                    @Override
        			public void actionPerformed(ActionEvent evt) {
        				mScanner.CloseSource();
                    }
                };
        		int delay = 1; 
                Timer timer = new Timer(delay, taskPerformer);
                timer.setRepeats(false);
                timer.start();
            }
        };

        public void setIndex(int index) {
        	mIndex = index;
        }

        public void start() {
        	mTimer.start();
        }
    }
}

The format of the message transferred between Java WebSocket Server and JavaScript client is JSON.

Send JSON data in Java:

        JsonObject jsonObj = new JsonObject();
        JsonArray jsonArray = new JsonArray();

        String[] sources = mSourceManager.getSources();
        if (sources != null) {
        	for (String source : sources) {
        		jsonArray.add(new JsonPrimitive(source));
        	}
        }

        jsonObj.add(Msg.MSG_SOURCES, jsonArray);

        String s = jsonObj.toString();

        try {
			session.getRemote().sendString(s);
		} catch (IOException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}

Parse and read JSON data in Java:

public void onMessage(String message) {
    	JsonParser parser = new JsonParser();
    	boolean isJSON = true;
    	JsonElement element = null;
    	try {
    		element =  parser.parse(message);
    	}
    	catch (JsonParseException e) {
    		System.out.println("exception: " + e);
    		isJSON = false;
    	}

        if (isJSON && element != null) {
        	JsonObject obj = element.getAsJsonObject();
        	element = obj.get(Msg.MSG_MESSAGE);
        	if (element != null) {
        		switch (element.getAsString()) {
        		case Msg.MSG_SOURCE:
        			int index = obj.get(Msg.MSG_INDEX).getAsInt();
        			mSourceManager.acquireImage(index);
        			break;
        		}
        	}
        }

        System.out.println("Message: " + message);
    }

Send JSON data in JavaScript:

var json = {};
json.Message = MSG_SOURCE;
json.Index = i;
var msg = JSON.stringify(json);
ws.send(msg);

Parse and read JSON data in JavaScript:

ws.onmessage = function (evt) { 
    var data = evt.data;
    	var json = JSON.parse(data);
    	var value = json[MSG_SOURCES];
    	showSources(value);
    }

Create a class ScanAction which includes a Timer in UI thread. The Timer is used to trigger image capture event.

public class ScanAction {
    	private int mIndex;
    	private int mDelay = 1;
    	private Timer mTimer;

    	public ScanAction() {
    		mTimer = new Timer(mDelay, mTaskPerformer);
    		mTimer.setRepeats(false);
    	}

    	private ActionListener mTaskPerformer = new ActionListener() {
            @Override
    		public void actionPerformed(ActionEvent evt) {
        		mScanner.AcquireImage(mIndex);
            	ActionListener taskPerformer = new ActionListener() {
                    @Override
        			public void actionPerformed(ActionEvent evt) {
        				mScanner.CloseSource();
                    }
                };
        		int delay = 1; 
                Timer timer = new Timer(delay, taskPerformer);
                timer.setRepeats(false);
                timer.start();
            }
        };

        public void setIndex(int index) {
        	mIndex = index;
        }

        public void start() {
        	mTimer.start();
        }
    }

When the image data is available, send it to all Web clients:

public boolean Notify(String message, String value) {
    	ArrayList<WSHandler> sessions = WSHandler.getAllSessions();
    	for (WSHandler session : sessions) {
    		session.sendImage(ImageUtil.getImageBytes(new File(value)));
    	}

        return true;
    }

Source Code

JavaWebTWAIN

How to Implement a Java WebSocket Server for Image Transmission with Jetty

In the previous articles, I shared how to implement a .Net WebSocket server with SuperWebSocket. Today, I’d like to continue the series WebSocket: How-to, talking about how to implement a WebSocket server in Java.

What is Jetty?

“Jetty provides a Web server andjavax.servlet container, plus support forSPDY, WebSocket, OSGi, JMX, JNDI, JAASand many other integrations.” from http://www.eclipse.org/jetty/

Quickly Setup a Java WebSocket Server

Create a class WSHandler which extends the class WebSocketHandler:

import org.eclipse.jetty.websocket.api.Session;
import org.eclipse.jetty.websocket.api.annotations.OnWebSocketClose;
import org.eclipse.jetty.websocket.api.annotations.OnWebSocketConnect;
import org.eclipse.jetty.websocket.api.annotations.OnWebSocketError;
import org.eclipse.jetty.websocket.api.annotations.OnWebSocketMessage;
import org.eclipse.jetty.websocket.api.annotations.WebSocket;
import org.eclipse.jetty.websocket.server.WebSocketHandler;
import org.eclipse.jetty.websocket.servlet.WebSocketServletFactory;

@WebSocket
public class WSHandler extends WebSocketHandler {

    @OnWebSocketClose
    public void onClose(int statusCode, String reason) {
    }

    @OnWebSocketError
    public void onError(Throwable t) {
    }

    @OnWebSocketConnect
    public void onConnect(Session session) {
    }

    @OnWebSocketMessage
    public void onMessage(String message) {
    }

	@Override
	public void configure(WebSocketServletFactory factory) {
		// TODO Auto-generated method stub
		factory.register(WSHandler.class);
	}
}

Start the server with a port and a handler:

    public static void main(String[] args) throws Exception {
        Server server = new Server(2014);
        server.setHandler(new WSHandler());
        server.setStopTimeout(0);
        server.start();
        server.join();
    }

Done.

JavaScript Client for WebSocket Connection

We can create a simple web client for testing.

Index.htm:

<!DOCTYPE html>
<html>
    <body>
        <script src="websocket.js"></script>
    </body>
</html>

Websocket.js:

var ws = new WebSocket("ws://127.0.0.1:2014/");

ws.onopen = function() {
    alert("Opened");
    ws.send("I'm client");
};

ws.onmessage = function (evt) { 
};

ws.onclose = function() {
    alert("Closed");
};

ws.onerror = function(err) {
    alert("Error: " + err);
};

Run the app and check the message.

Image Transmission Between WebSocket Server & Web Clients

What I’m going to do: Read more

How to Remotely Scan Documents from PCs and Mobile Devices with HTML5 WebSocket and Dynamic .NET TWAIN

In this tutorial, I want to make a more complicated WebSocket demo, which enables me to capture images from scanners on server side, and view the images in desktop web browsers or mobile web browsers. Here is the flowchart:

 flow chart

Prerequisites

Scanning Documents through WebSocket Connection

Initialize Dynamic .NET TWAIN component:

            dynamicDotNetTwain = new Dynamsoft.DotNet.TWAIN.DynamicDotNetTwain(); // create Dynamic .NET TWAIN component
            dynamicDotNetTwain.OnPostAllTransfers += new Dynamsoft.DotNet.TWAIN.Delegate.OnPostAllTransfersHandler(this.dynamicDotNetTwain_OnPostAllTransfers); 
            dynamicDotNetTwain.MaxImagesInBuffer = 64;
            dynamicDotNetTwain.IfAppendImage = true;
            dynamicDotNetTwain.IfThrowException = true;
            dynamicDotNetTwain.IfShowUI = false;

When a client is connected to WebSocket server, query all available scanner sources on server side, and serialize them as JSON data:

            int iIndex;
            dynamicDotNetTwain.OpenSourceManager();

            StringBuilder sb = new StringBuilder();
            StringWriter sw = new StringWriter(sb);
            using (JsonWriter writer = new JsonTextWriter(sw))
            {
                writer.Formatting = Formatting.Indented;

                writer.WriteStartObject();
                writer.WritePropertyName("Sources");
                writer.WriteStartArray();
                for (iIndex = 0; iIndex < dynamicDotNetTwain.SourceCount; iIndex++)
                {
                    writer.WriteValue(dynamicDotNetTwain.SourceNameItems(Convert.ToInt16(iIndex)));
                }
                writer.WriteEnd();
                writer.WriteEndObject();
            }

            String msg = sw.ToString();
            session.Send(msg);

Send the JSON data to the Web client, and de-serialize the JSON data in JavaScript:

                        var json = JSON.parse(data);
                        var jValue;
                        for (jProperty in json) {
                            jValue = json[jProperty];
                            switch (jProperty) {
                                case "Sources":
                                    showSources(jValue);
                                    break;
                                case "Draw":
                                    var w = jValue[0];
                                    var h = jValue[1];
                                    imageWidth = parseInt(w);
                                    imageHeight = parseInt(h);
                                    break;
                            }
                            break;
                        }

Show the sources in drop-down list with HTML5 elements <select> and <option>:

                var sources = document.getElementById('sources');
                var option;

                var count = values.length;
                if (count == 0) {
                    option = document.createElement('option');
                    option.text = "N/A";
                    sources.appendChild(option);
                }
                else {
                    for (var i = 0; i < 3; i++) {
                        option = document.createElement('option');
                        option.text = values[i];
                        sources.appendChild(option);
                    };
                }

Acquire document images from scanners:

        private void appServer_NewMessageReceived(WebSocketSession session, string message)
        {   
            int iIndex = Int32.Parse(message);

            try
            {
                dynamicDotNetTwain.CloseSource();
                bool success = dynamicDotNetTwain.SelectSourceByIndex(Convert.ToInt16(iIndex));
                dynamicDotNetTwain.OpenSource();
                dynamicDotNetTwain.AcquireImage();
            }
            catch (Dynamsoft.DotNet.TWAIN.TwainException exp)
            {
                String errorstr = "";
                errorstr += "Error " + exp.Code + "\r\n" + "Description: " + exp.Message + "\r\nPosition: " + exp.TargetSite + "\r\nHelp: " + exp.HelpLink + "\r\n";
                MessageBox.Show(errorstr);
            }
            catch (Exception exp)
            {
                String errorstr = "";
                errorstr += "ErrorMessage: " + exp.Message + "\r\n";
                MessageBox.Show(errorstr);
            }
        }

When the image data is ready, send it to the Web client:

private void dynamicDotNetTwain_OnPostAllTransfers()
        {
            if (dynamicDotNetTwain.MaxImagesInBuffer < 1)
            {
                MessageBox.Show("no image");
                return;
            }

            Image img = dynamicDotNetTwain.GetImage(0);

            ImageData imageData = load_image(img);

            /* send width & height in JSON */
            StringBuilder sb = new StringBuilder();
            StringWriter sw = new StringWriter(sb);
            using (JsonWriter writer = new JsonTextWriter(sw))
            {
                writer.Formatting = Formatting.Indented;

                writer.WriteStartObject();
                writer.WritePropertyName("Draw");
                writer.WriteStartArray();
                writer.WriteValue(imageData.Width);
                writer.WriteValue(imageData.Height);
                writer.WriteEnd();
                writer.WriteEndObject();
            }
            String msg = sw.ToString();

            IEnumerable<WebSocketSession> sessions = appServer.GetAllSessions();
            foreach (WebSocketSession session in sessions) 
            {
                session.Send(msg);
                session.Send(imageData.Data, 0, imageData.Data.Length);
            }

            imageData = null;
        }

Finally, I can view the captured document images in Windows Chrome:

scan document with websocket

Based on HTML5, the solution is platform-independent, which means it supports not only Windows, but also Linux, Mac OS, iOS, Android and so forth. Thus I can also access the page and remotely operate scanners in Android Chrome:


android_scan_sources

android_scan_document

Now, you can try to make your own cloud scanner application with Dynamic .NET TWAIN and HTML5 WebSocket.

Source Code

WebSocketScanDocument

 

Barcode Generator with HTML5 WebSocket

Last week, I shared how to transmit an image between a WebSocket server and a Web client. Based on that, we can make a little bit of improvement. Today, I would like to share how to generate barcode on .Net server side and display it on web browser.

websocket barcode

Prerequisites

Remote Barcode Generator

  • In your WinForms project, add DynamicBarcode.dll and set its property Copy to Output Directory as Copy always. As so, the Dynamic .NET TWAIN component is able to automatically load the barcode library.

barcode library

  • Encode the received message, and draw generated barcode onto a background image:
            float scale = 3;
            short sImageIndex = 0;
            dynamicDotNetTwain.MaxImagesInBuffer = 1;
            bool isLoaded = dynamicDotNetTwain.LoadImage("test.png");
            dynamicDotNetTwain.AddBarcode(sImageIndex, Dynamsoft.DotNet.TWAIN.Enums.Barcode.BarcodeFormat.QR_CODE, message, "", 0, 0, scale);
  • Send image width, height and data to web client:
            Image img = dynamicDotNetTwain.GetImage(sImageIndex);

            ImageData imageData = load_image(img);
            session.Send(imageData.Width + "," + imageData.Height);
            session.Send(imageData.Data, 0, imageData.Data.Length);
            imageData = null;
  • Display the barcode image on JavaScript web client:
               ws.onmessage = function (evt) {
                    var data = evt.data;
                    if (!data)
                        return;

                    if (data instanceof ArrayBuffer) {
                        drawImage(evt.data);
                    }
                    else {
                        var len = data.length;
                        var index = data.indexOf(",");
                        var w = data.substring(0, index);
                        var h = data.substring(index + 1, len);
                        imageWidth = parseInt(w);
                        imageHeight = parseInt(h);
                    }
                };

Now, you can have fun with the remote barcode generator via all browsers (Chrome, Firefox, IE etc.) which support HTML5 WebSocket.

Source Code

WebSocketBarcodeGenerator

 

Image Transmission between a HTML5 WebSocket Server and a Web Client

HTML5 WebSocket facilitates the communication between web browsers and local/remote servers. If you want to learn a simple websocket example, creating a WebSocket Server in C# and a Web client in JavaScript, you can refer to SuperWebSocket, which is a .NET implementation of Web Socket Server.

In this article, I would like to share how I implemented a simple WebSocket solution for image transmission based on the basic sample code of SuperWebSocket and Dynamic .NET TWAIN, especially what problem I have solved.

Here is a quick look of the WebSocket server and the HTML5 & JavaScript client.

WebSocket ServerWebSocket Client

Prerequisites

Create a .NET WebSocket Server

Create a new WinForms project, and add the following references which are located at the folder of SuperWebSocket.

WebSocket References

Add the required namespaces:

using Dynamsoft.DotNet.TWAIN;

using SuperSocket.SocketBase;
using SuperWebSocket;

In the sample code, the server is launched with port 2012, no IP specified by default. You can specify the IP address for remote access. For example:

if (!appServer.Setup("192.168.8.84", 2012)) //Setup with listening port
{
     MessageBox.Show("Failed to setup!");
     return;
}

You can use Dynamic Web TWAIN component to do some image operation. With two lines of code, I can load an image file:

bool isLoad = dynamicDotNetTwain.LoadImage("dynamsoft_logo_black.png"); // load an image
Image img = dynamicDotNetTwain.GetImage(0);

Be careful of the image format. It is PNG. If you want to display it in a Web browser, you need to convert the format from PNG to BMP:

            byte[] result;
            using (System.IO.MemoryStream stream = new System.IO.MemoryStream())
            {
                img.Save(stream, System.Drawing.Imaging.ImageFormat.Bmp);   // convert png to bmp
                result = stream.GetBuffer();
            }

It’s not done yet. The byte array contains some extra image information, which is 54 bytes in length. So the actual data length is:

int iRealLen = result.Length - 54;
byte[] image = new byte[iRealLen];

Here are some tricky things that if you just send the subset of the byte array to your Web browser, you will find the displayed image is upside-down, and the color is also incorrect.

To fix the position issue, you need to sort the bytes of original data array from bottom to top. As to the color, exchange the position of blue and red. See the code:

            int iIndex = 0;
            int iRowIndex = 0;
            int iWidth = width * 4;
            for (int i = height - 1; i >= 0; --i)
            {
                iRowIndex = i * iWidth;
                for (int j = 0; j < iWidth; j += 4)
                {
                    // RGB to BGR
                    image[iIndex++] = result[iRowIndex + j + 2 + 54]; // B
                    image[iIndex++] = result[iRowIndex + j + 1 + 54]; // G
                    image[iIndex++] = result[iRowIndex + j + 54];     // R
                    image[iIndex++] = result[iRowIndex + j + 3 + 54]; // A
                }   
            }

Now, you can send the data via:

session.Send(imageData.Data, 0, imageData.Data.Length);

Create a JavaScript Client

To receive the image as ArrayBuffer on the client side, you have to specify the binaryType after creating a WebSocket:

ws.binaryType = "arraybuffer";

Once the image data is received, draw all bytes onto a new canvas, and finally create an image element to display the canvas data:

                var imageWidth = 73, imageHeight = 73; // hardcoded width & height. 
                var byteArray = new Uint8Array(data);

                var canvas = document.createElement('canvas');
                canvas.height = imageWidth;
                canvas.width = imageHeight;
                var ctx = canvas.getContext('2d');

                var imageData = ctx.getImageData(0, 0, imageWidth, imageHeight); // total size: imageWidth * imageHeight * 4; color format BGRA
                var dataLen = imageData.data.length;
                for (var i = 0; i < dataLen; i++)
                {
                    imageData.data[i] = byteArray[i];
                }
                ctx.putImageData(imageData, 0, 0);

                // create a new element and add it to div
                var image = document.createElement('img');
                image.width = imageWidth;
                image.height = imageHeight;
                image.src = canvas.toDataURL();

                var div = document.getElementById('img');
                div.appendChild(image);

Download Sample

WebSocketImageSendSource