Artificial intelligence (AI) is becoming more common practice within the workforce, and many large corporations are joining the bandwagon as they create and transform businesses with document scanning AI software.
In this article we will compare some popular document information extraction software/services (sometimes referred to as intelligent OCR or document AI) and look at results on a sample document. We will evaluate AI based on generic document parsing that means AI will be evaluated based on extraction of key-value pairs and tables. Typically another level of AI is needed to classify these Key Value Pairs into useful information. As an example Key Value Pairs need to be further identified as document specific information like Invoice date, due date, due amount etc. for Accounts Payable Invoice.
Before we jump into the detailed comparison of document extraction accuracy, the table below summarizes the overall features.
Feature Comparison
As seen in the image above, various providers and their services are compared in a qualitative table. The providers are only compatible and run on the cloud platforms of Microsoft ® Azure Cloud, Google Cloud Platform (GCP), and Amazon AWS respectively. Various providers are being analyzed and use well-known cloud services, however; their capabilities are still lacking when compared to Deep Cognition’s PaperEntry AI Module.
Here are the detailed ten features listed in the comparative table:
- Accuracy — PaperEntry is 99.0{29fe85292aceb8cf4c6c5bf484e3bcf0e26120073821381a5855b08e43d3ac09} accurate in recognizing Key Value Pairs within a document.
- Classification of Key Value Pairs- PaperEntry has the capability to classify Key Value Pairs and normalize them before integration into any TMS/ERP software.
- Table Extraction- Our competitors have the ability to extract information from tables, however they do not have the capability to correct and align columns.
- Smart Table Entry Extraction – Deep Cognition PaperEntry AI can extract and separate values embedded in columns like descriptions (e.g. P.O. # mentioned in a description will be separated out).
- AI Validator Software- When there is an error in data extraction, users can correct the output easily using our validation software.
- Labeling Software- Deep Cognition provides a specialized labeling software that can be used by users to label their documents and eventually use those for retraining of the AI.
- Re-Trainability- Practice makes perfect, with PaperEntry, training is a necessity when it comes to improving accuracy. Users can re-train their AI when there are mistakes made in document extraction.
- Document Compatibility- Many competitors offer different types of documents, however; they are all standardized (i.e. 1040, account invoices) and do not offer flexibility to create their own custom document format and extract information from outside of templates.
- Security & Privacy- Privacy is of the utmost importance, especially when dealing with sensitive information, PaperEntry not only waives the worry by offering on-premise deployment, but also training to use the program offline or on-cloud.
- Custom Options- PaperEntry is 100{29fe85292aceb8cf4c6c5bf484e3bcf0e26120073821381a5855b08e43d3ac09} customizable to the needs of any organization, large or small. Deep Cognition is the only provider with a human validation service option, ensuring that all information on forwarded documents is extracted correctly.
Accuracy Comparison
Trials are important when deciding whether or not to choose a service provider. Many providers offer free trials, contact us for a free demo or click here for an in-depth video representation.
In this section, screen captures of the Key Value Pairs and Tables from various providers are shown and analyzed on a qualitative and qualitative basis.
In order to maintain consistency, the same invoice was used for testing each text extraction provider’s service.
Within the invoice, there are a total of 23 Key Values Pairs (Marked in Red), and 1 Table with 24 values (Marked in Blue). Below is a comparison of how well the various providers’ software extracted the information within the example invoice. The information is organized by what information was correctly extracted, and what were some issues that arose.
Provider-1 AI
Provider-1 has one of the more advanced modules for document extraction, it allows for you to custom build and program your own software. Provider-1’s solution only works with Microsoft ® Azure Cloud.
- Correctly Identified Key Value Pairs- The program identified 11/23 Key Values, of the values detected, there is a fair amount of accuracy as presented in the screen capture. The Key Values correctly identified relate to destination and numbers. It is to be noted that Provider-1’s solution only tried to identify invoice related fields and this might be the main reason for low number of key-value pair identification.
- Incorrectly Identified Key Value Pairs- The program missed many of the Key Values, as well as extracted the information incorrectly. There are a few mistakes in the keys, as some were not detected correctly and there is grouping of words.
- Correctly Identified Table Values- The program correctly identified the numerical values within the table, as well as the headers in the table. From the screen capture, you can see what was used to extract the values, the subtotal is left out from the table.
- Incorrectly Identified Table Values- The program mistook the descriptions as rows of its own, and created rows that were not originally in the table. Additional rows of information were included in the table that were one block in the original table, as well as blank cells of information where there is spacing between values.
Provider-2 AI
Provider-2’s solution only works with Google Cloud Platform (GCP). From the screen captured image, the identified Key Value Pairs are evaluated from the example invoice. Provider-2 is capable of identifying all the words on the invoice. Users may not notice at first glance, and may assume that since all the information is selected within the various tabs that everything they need for integration would be available, but since everything selected is not necessary, integration would be more difficult. PaperEntry, fixes any issues that may arise, with the optional human validation, everything needed to be selected and integrated will be.
- Correctly Identified Key Value Pairs- The program identified 19/23 Key Values Pairs, of the 19 key values identified, some of the table values were included in the key values, the program was able to clearly distinguish between bolded and non-bolded information.
- Incorrectly Identified Key Value Pairs- The program identified information in a table as a key value, depending on the purpose of the document, the table information can be formatted differently than a key value. Another issue is the program does not identify a Key Value Pair if there is not a value associated with the key.
- Correctly Identified Table Values- The program correctly identified the correct table and information in the invoice needed for entry into any TMS or ERP system. All the values are correct and none of the text is missing.
- Incorrectly Identified Table Values- The program identified 2 tables from the invoice, however there is only one table that is needed for processing, in terms of data entry. The first table is bolded key headings and their values. The only correct and necessary table is the 2nd one. The program also identified blank values from the table and created additional cells.
Provider-3’s AI
Provider-3’s solution only works with Amazon AWS. Provider-3 has a more simple, user friendly interface compared to the previous competitor. When it comes to identifying Tables, the program creates additional columns and rows for blank values. Many would pass on this option, as the information is not correctly identified, this may be why Provider-3 has implemented a human validation service, so that the information will not be incorrectly interpreted in the future and will be more accurate.
- Correctly Identified Key Value Pairs- The program identified 21/23 Keys and Values, compared to the previous program there is a significant difference in what is identified as a Key and Value. Everything identified is clearly laid out and legible for users to understand, the interface is great for those who aren’t big on clustered information.
- Incorrectly Identified Key Value Pairs-The program incorrectly identified some of the values, as seen above, some values are missing. Also, the organization of the information is laid out in no certain order and can be difficult to find what you are looking for.
- Correctly Identified Table Values- The program identified the correct table values. Most of the values are correctly identified and organized in an easy to understand row and column structure.
- Incorrectly Identified Table Values- The program incorrectly identified the table headers as a value itself, rather than as a title. There are additional columns and rows created in place of the spacing within the example invoice, that are left as blank values. With the addition of blank cells, the accuracy of the program goes down.
Deep Cognition’s PaperEntry AI
Deep Cognition strives to provide the best service needed for each client, with their flexibility and adaptability of PaperEntry, any issues that arise can be easily resolved and re-trained for accuracy. Not to mention, with human validation, these issues are quickly addressed without notice. The PaperEntry software aims to be as close as possible to to 100{29fe85292aceb8cf4c6c5bf484e3bcf0e26120073821381a5855b08e43d3ac09} accurate for integrating all company invoices into any programs used. With any mistakes, any user can easily highlight and address the issue, so it does not happen again in the future.
- Correctly Identified Key Value Pairs- The program identified 26/23 Key Values, the program extracted general information that is important when integrating a document into any invoice software. This includes bolded and non-bolded information. However, the program offers a service where users can go in and correct the information, in turn retraining the AI.
- Incorrectly Identified Key Value Pairs- The program incorrectly identified table headers in with the Key Value Pairs, and is missing some of the values to the keys. There are missing and incomplete values as well.
- Correctly Identified Table Values- The program correctly identified the table information within the document, and organized the data in any easy to read and understand format. There is no additional column or row in the table and it is exported as you would expect an excel table to look.
- Incorrectly Identified Table Values- Depending on the user preference, one could say that the subtotals are missing from the table, however; the program correctly extracted the table information without errors.
Final Breakdown
In this breakdown, you can see the total amount of errors made by each provider. Of the 23 total Key Value Pairs and 24 Table values, the graph above can be interpreted like this
- PaperEntry
- Key Value Points
- 22 True Positives, detected and correctly identified
- 1 False Positive, detected, but incorrectly identified
- 1 False Negative, not detected and incorrectly identified
- Precision: 0.96
- Recall: 0.96
- Table
- 24 True Positives, detected and correctly identified
- 0 False Positive, detected, but incorrectly identified
- 0 False Negative, not detected and incorrectly identified
- Precision: 1.00
- Recall: 1.00
- Key Value Points
- Provider 1 *Precision and recall for KVPs was determined based upon the program’s ability to extract only invoice information from the document, rather than the document as a whole.
- Key Value Points
- 7 True Positives, detected and correctly identified
- 0 False Positive, detected, but incorrectly identified
- 0 False Negative, not detected and incorrectly identified
- Precision: 1.00
- Recall: 1.00
- Table
- 21 True Positives, detected and correctly identified
- 45 False Positive, detected, but incorrectly identified
- 3 False Negative, not detected and incorrectly identified
- Precision: 0.32
- Recall: 0.88
- Key Value Points
- Provider 2
- Key Value Points
- 16 True Positives, detected and correctly identified
- 1 False Positive, detected, but incorrectly identified
- 1 False Negative, not detected and incorrectly identified
- Precision: 0.94
- Recall: 0.94
- Table
- 23 True Positives, detected and correctly identified
- 26 False Positive, detected, but incorrectly identified
- 1 False Negative, not detected and incorrectly identified
- Precision: 0.47
- Recall: 0.96
- Key Value Points
- Provider 3
- Key Value Points
- 21 True Positives, detected and correctly identified
- 1 False Positive, detected, but incorrectly identified
- 1 False Negative, not detected and incorrectly identified
- Precision: 0.95
- Recall: 0.95
- Table
- 24 True Positives, detected and correctly identified
- 11 False Positive, detected, but incorrectly identified
- 1 False Negative, not detected and incorrectly identified
- Precision: 0.69
- Recall: 0.96
- Key Value Points
Markets
Currently, other programs outside of PaperEntry are not equipped for use within various industries, their document extraction software is not able to identify the important keys/values and tables. On the other hand, PaperEntry is available for deployment within the following markets and industries:
- Accounts Payable- Invoices
- Logistics- Commercial Invoices Entry, integrates into CargoWise or any TMS
- Logistics- AP Invoices, Bill of Lading, Waybill, Arrival Notices
- Pharma- Bill of Lading, AP Invoices
- Healthcare- Patient fax orders, patient insurance cards etc.
- Banking- Loan document processing, tax document processing
With PaperEntry the possibilities are endless, processing work invoices is easier than before. Anything is possible, from Accounting to Medicine, any type of invoice or document can be read and integrated with PaperEntry AI.
Conclusion
After analyzing the various AI Document processing programs, various pros and cons can be concluded. When extracting data from a presented document, Provider-1 was difficult to set up and did not extract as much information as the others, and Provider-2 had a tendency to collude all information as Keys and Values. Provider-3 was more simplified in the extraction of information, but did not identify as much information from the document as Provider-2 and PaperEntry. PaperEntry was overall more efficient in extracting information from the document, as it identified Keys and Values needed to be identified on an invoice. The PaperEntry AI Module was able to process a given document and extract the proper information needed for integration into any program.
Finally the fact that PaperEntry provides labeling and training software to end customers so that they can train on their own documents sets it apart. This means that customers can train AI continuously as it makes mistakes and reach superhuman accuracy.
More Information About PaperEntry- PaperEntry AI module extracts information from forwarded or uploaded documents of your choice, there is no complex set up or templates involved. With quick AI training, the service is operational within a week. Multiple integration options are available for many popular ERP or TMS systems. Worried about security and privacy? PaperEntry offers on-premises deployment in addition to a cloud based version. Click here to read more in-depth about PaperEntry.
Setup your personalized demo today by contacting us at sales@deepcognition.ai .
Disclaimers- Any product names, logos, brands, and other trademarks or images featured or referred to within this article and/or on any social media forum are the property of their respective trademark holders.
These trademark holders are not affiliated with Deep Cognition, Inc. or it’s website. These trademark holders do not sponsor or endorse Deep Cognition, Inc. or any of it’s products or comments.
Further, Deep Cognition, Inc. declares no affiliation, sponsorship, nor any partnerships with any registered trademarks unless otherwise stated.
“Microsoft”, “Azure” and “Microsoft Azure” are registered trademarks of Microsoft Corporation
“Google”, and “Google Cloud Platform” are registered trademarks of Alphabet Inc.
Amazon”, and “Amazon AWS” are registered trademarks of Amazon.com, Inc.
