DETAILED ACTION
Claim Rejections - 35 USC § 103
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Prior arts cited in this office action:
Osindero (US 20170004374 A1, hereinafter “Osindero”)
Hu et al. (WO 2021161095 A1, hereinafter “Hu”)
Liu et al. (CN 113221890 A, hereinafter “Liu”)
Kumar et al. (US 20210374040 A1, hereinafter “Kumar”)
Licata et al. (US 20130073583 A1, hereinafter “Licata”)
Ren et al. (CN 109388807 B, hereinafter “Ren”)
Response to Arguments
Applicant’s Arguments/Remarks filed on 03/25/2026 have been fully considered but they are not persuasive.
Applicant’s Arguments/Remarks: applicant argues that Osindero do not each contain at least one of the
one or more text lines formed from the set of the plurality of pixels in the image-based
document. Determining the location of text lines, as described in Hu, does not teach or suggest identifying a set of bounding boxes within the image based document or that each bounding box specifies a position of each of the one or more text lines formed from the set of the plurality of pixels in the image-based document. Specifically, the text line areas of Hu do not specify a position of each of the one or more text lines, formed from the set of the plurality of pixels detected in the image-based document, within the image based document.
Examiner’s Response: Examiner disagrees with applicant assertion above that the combination of the cited prior arts does not teach or suggest applicant invention as claimed. Applicant’s argument seems to be solely based on Licata when the rejection is based on the combination of cited references.
Applicant is reminded that the test for obviousness is not whether the features of a secondary reference may be bodily incorporated into the structure of the primary reference; nor is it that the claimed invention must be expressly suggested in any one or all of the references. Rather, the test is what the combined teachings of the references would have suggested to those of ordinary skill in the art. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981).
First, in an image any selection area is formed by a group of pixels and Osindero teaches The portions of an image comprising the text data are initially identified and the text imaged by the pixels of that image portion is extracted in textual data format (Abstract). As a result, the area selected by the box in Osindero and the text lines are formed form set of pixels. Hu further teaches in other implementations, the text line detector 210 may determine one or more target text line areas in the image 170 with other sizes, dimensions, and/or orientations. For example, if a line of text in the image 170 is arranged in other ways (for example, with a plurality of characters arranged in a curved line instead of being positioned vertically or horizontally), the text line detector 210 may also detect a target text line area 212 with the line of text included therein (Hu [0034]). In other words, not only the text line area is determined but also the position and orientation of each line is detected. In figure 3, for example, how the lines are detected if the position is not determined. The position of each line is determined with respect to each other and the page in order to distinguish them with each other such as 212-1, 212-2, 212-3 and 212-4 for example.
Applicant’s Arguments/Remarks: applicant argues that there is no teaching that the bounding the one or more text lines from the set of the plurality of pixels in the image-based document in a bounding box is based at least in part on comparing a value of a particular pixel to a threshold value of a pixel being text.
Osindero teaches the further image regions 354, 356, 358 and the like are analyzed sequentially or in parallel to determine the presence of characters in these image portions. Weighted pixel contributions to the presence of a character within smaller regions such as the region 362 of the selected region 352 are further determined by the pixel weighing module 304 (Osindero [0006], [0051]-[0054]). Therefore, one of ordinary skill in the art can see that the determination of a presence of a character in a region of an image is based on the value of the pixel or pixels in that region. Hu also teaches in determining the target text line area 212, the text line detector 210 is not required to recognize the specific characters or texts in the image 170, but to determine whether a group of pixels or larger image units in the image 170 may have a text present therein. The probability distribution information indicates a conditional probability of each possible character model element belonging to the target text line area 212. In some implementations, the text decoder 230 uses the single character model 220 to determine a sequence of character model elements with maximum occurrence probabilities in the target text line area 212 (Hu [0032], [0043], fig. 3).
As a result, examiner maintains that the combination of the cited prior arts teaches or suggests applicant invention as claimed. Therefore claims 1-13, 15-16, 18-22 are not allowable over the cited prior arts.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-9, 11, 15-16, 18 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Osindero (US 20170004374 A1, hereinafter “Osindero”) in view of Hu et al. (WO 2021161095 A1, hereinafter “Hu”) and in view of Licata et al. (US 20130073583 A1, hereinafter “Licata”).
Regarding claims 1, 8 and 15:
Osindero teaches a method comprising:
receiving, by a computer system, an image-based document comprised of a plurality of pixels (Osindero [0001], [0006], [0008], [0022], [0034], fig. 2, where Osindero teaches obtaining character predictions by the processor, further comprises obtaining, by the processor, a respective weighted sampling of the pixels in each of a plurality of image regions comprised in the portion of the image and obtaining, by the processor, the character predictions based on the weighted sampling of the pixels in the plurality of image regions);
detecting, by the computer system, one or more text lines formed from a set of the plurality of pixels in the image-based document (Osindero [0034], [0045], [0053]-[0055], [0065], fig. 3, where Osindero teaches Weighted pixel contributions to the presence of a character within smaller regions such as the region 362 of the selected region 352 are further determined by the pixel weighing module 304);
identifying, by the computer system, a set of square boxes within the image-based document, each square box in the set of square boxes containing at least one of the detected one or more text lines, each square box specifying a position of each of the one or more text lines in the image-based document, formed from the set of pixels, within the image based documents; processing, by the computer system, pixels in each of the set of square boxes to identify a set of text characters (Osindero [0034], [0052]-[0055], fig. 3, where Osindero teaches In Eq. (1), ρ.sub.ij indicates a pixel value within the region 362 having the center μ, width or radius σ and lying within the area bounded by coordinates (x.sub.A.sup.j,y.sub.B.sup.j) as shown at 370. The pixels closest to the center μ or center 364 carry the greatest weight and the weight decreases rapidly as the distance of the pixels from the center 364 increases as indicated by the exponential factor in Eq. (1), thereby providing scale invariance of the image pixels to the character prediction);
determining, by the computer system, a first primary script classification, from a plurality of primary script classifications, for the set of text characters in each of the set of square boxes, each of the plurality of primary script classifications corresponding to one or more natural languages, wherein the first primary script classification corresponds to more than one natural language (Osindero [0063]; where Osindero teaches In some embodiments, text matching techniques commonly known in the art can be used for the language identification by the language identification module 402. As different natural languages like English, Arabic, Chinese and the like have very dissimilar characters, a broad identification of a language of interest for an image based on the character predictions can be obtained); and
responsive to determining the first primary script classification, the computer system: processing the pixels in each of the set of bounding boxes to generate a set of text strings corresponding the set of square boxes (Osindero [0063]-[0065], where Osindero teaches in some embodiments, the text from an image can comprise a single character or it can comprise a string of characters that form a word or a combination of words that form a sentence);
processing the set of text strings to determine a language, for the set of text strings, that corresponds to the first primary script classification (Osindero [0063]-[0066], where Osindero teaches; and
storing an output of the determined language for the image-based document (Osindero [0063]-[0065], where Osindero teaches the text output module 406 provides the text 256 from the image 250 as one or more of tokens that can be associated with the image to be stored in the image text database 110) .
Osindero does not use the words bounding box in his disclosure, and that the one or more text lines formed from the set of the plurality of pixels in the image-based document, identifying the set of bounding boxes within the image-based document.
classifying, by a direction classification module of the computer system, a direction of the image-based document to determine an orientation of text lines of the image-based document;
the detected one or more text lines having a longest length text line among text lines from the image-based document;
However, the square box can be interpreted as bounding box as claim by the applicant. Nonetheless, for sake or completeness one can take a look at Hu figure 3 where Liu teaches a plurality dotted line that represent bounding box that encompass different objects or line of texts. Fig. 3 shows an example of target text line areas 212 determined from the image 170. As shown in the figure, the text line detector 210 may determine a plurality of target text line areas 212-1, 212-2, ... 212-10 in the image 170 (collectively or individually referred to as the target text line areas 212), each being expected to include a text to be recognized. It should be appreciated that the division of target text line areas shown in Fig. 3 is only an example. In other implementations, the text line detector 210 may determine one or more target text line areas in the image 170 with other sizes, dimensions, and/or orientations. For example, if a line of text in the image 170 is arranged in other ways (for example, with a plurality of characters arranged in a curved line instead of being positioned vertically or horizontally), the text line detector 210 may also detect a target text line area 212 with the line of text included therein. the text line detector 210 is not required to recognize the specific characters or texts in the image 170, but to determine whether a group of pixels or larger image units in the image 170 may have a text present therein.
Hu further teaches for example, a group of characters may be arranged in an image in a vertical direction (for example, the Chinese text ^ ” in the image 170 of Fig. 1), and another group of characters may be arranged in a horizontal direction (for example, the English text “Vegetables” and other texts in the image of Fig. 170). In addition, mixed- language texts are very common in many practical applications, such as in commercial documents, store signboards, and restaurant menus, e.g., ”in the image 170 of Fig. 1. Considering the character arrangement directions and the difference of characters in different languages, multiple dedicated text recognition models are designed for respective orientations and multiple dedicated text recognition models are designed for different languages to process the corresponding image areas. Among a plurality of predetermined languages, the number of commonly-used textual elements and the lengths of characters included therein in the different languages may be varied greatly. For example, words in Latin languages and words in Eastern languages are different. Therefore, in order to balance the size of sets of textual elements in different languages and to enable recognition of some out-of-vocabulary words that are not included in the lexicon 520, in some implementations, sub-words (also referred to as “textual sub-elements”) may be used as basic elements of the language model and the predetermined lexicon of some languages. (Hu [0025], [0032]-[0034, [0043], [0059], fig. 3).
Therefore, taking the teachings of Osindero and HU as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the application to used bounding box to select each text lines that can collectively or individually referred to as the target text line areas, to determine the direction of the of the group of characters (line) and the length of the characters included, in order to break down the process by allowing for processing each line individually which would render the system more efficient and facilitate the determination of the language the text is written in which would facilitate also the recognition of the text and other features .
The combination above fails to teach explicitly wherein said identifying comprises determining a number of bounding boxes to include in the set of bounding boxes within the image-based document.
However, Hu teaches in figure 3 that each line is encompassed withing a bounding box and while 212-1 and 212-2 are separated and stand individually boxes 212-3 to 212-10 can be combined or grouped into one to form a particular region. It should be appreciated that the division of target text line areas shown in Fig. 3 is only an example. In other implementations, the text line detector 210 may determine one or more target text line areas in the image 170 with other sizes, dimensions, and/or orientations. The probability distribution information indicates a conditional probability of each possible character model element belonging to the target text line area 212. In some implementations, the text decoder 230 uses the single character model 220 to determine a sequence of character model elements with maximum occurrence probabilities in the target text line area 212 (Hu [0033]-[0034], [0043], fig. 3). Furthermore, Licata teaches the process continues to step 305 in which the context/content processing platform 103 causes, at least in part, a creation of one or more bounding boxes around one or more estimated text portions of the sensor data. The bounding boxes, as discussed above, may be of any shape around an estimated text region. Then, in step 307, the context/content processing platform 103 causes, at least in part, an optical character recognition of the sensor data within the one or more bounding boxes. For example, if the sensor data is an image that includes a series of text regions such as a name and telephone number on a business card, bounding boxes are created around each of the estimated text regions which may be, for example, three separate boxes for a phone number 777-777-7777. Because conducting OCR on each individual box is essentially useless, aside from perhaps conducting a search for an area code, the process continues to step 309 in which the context/content processing platform 103 processes and/or facilitates a processing of the one or more bounding boxes to cause, at least in part, an expansion of the one or more bounding boxes. The expansion may be in any direction that may suit the image. But, if the image is one that the context/content processing platform 103 determines includes what could be interpreted as lines of text, the process continues to step 311 in which the context/content processing platform 103 causes, at least in part, the expansion of the one or more bounding boxes to encompass a line of text. For example, the context/content processing platform 103 detects text lines for each bounding box and assigns text line number ID to each of the BBX. The context/content processing platform 103 groups the BBX according to text lines in an array and geometrically merges and joins each of the BBX by text line number ID. The context/content processing platform 103 then re-ranks the grouped BBX from a top of the image to a bottom and outputs a new four coordinate systems and a new number of BBX (Licata [0041], [0062], figs. 4B and 5A). Licata discloses that in case of in case of business cart the name of the company the name of the person and the phone number is included each within their corresponding bounding box, which comprises group of pixels or pixel set. Further, one can see that each bounding box forms a line to indicate a text line formed from the set of the plurality of pixels in the business card.
Therefore, taking the teachings of Osindero, Hu and Licata as a whole, it would have been obvious to one of ordinary skill in the art at the time of the effective filing date of the application to, for example, using a bounding box for each character, a bounding box for each letter, a bounding box for each group of characters or a bounding box for each group of letters to form a line of text, and group them into meaningful information that would facilitate recovery of the actual text such as with phone numbers, names, etc.
Regarding claim 2:
Osindero in view of Hu and in view of Licata teaches wherein detecting of the one or more of text lines in the image-based document further includes predicting a probability of each pixel in the image-based document comprising text (Osindero [0009], [0054]-[0056]; Hu [0035], [0071], [0078]).
Regarding claims 3 and 16:
Osindero in view of Hu and in view of Licata teaches wherein each bounding box includes a set of coordinates corresponding to each corner of the bounding box, wherein the set of coordinates specify the position of each bounding box relative to a coordinate plane for the image-based document (Osindero [0009], [0053]-[0055], [0059]-[0060]; Hu [0035], [0040], [0071], [0078]).
Regarding claim 4:
Osindero in view of Hu and in view of Licata teaches further comprising:
identifying an outer bound of a text area by processing pixels in the image-based document; and
assigning a first coordinate point of the set of coordinates specifying the outer bound of the bounding box relative to the coordinate plane for the image-based document (Osindero [0009], [0053]-[0055], [0059]-[0060]; Hu [0032]-[0034], [0040], [0078], fig. 3).
Regarding claims 5 and 17:
Osindero in view of Hu and in view of Licata teaches wherein identifying the set of bounding boxes to be sampled further comprises:
deriving, for each bounding box, a length value and a width value using the set of coordinates for each bounding box; and
selecting the set of bounding boxes as bounding boxes with a greatest width value (Osindero [0009], [0053]-[0055], [0059]-[0060]; where a region preferred is selected where selection based on the size or the bounded coordinates can be easily derived by one having ordinary skill in the art).
Regarding claims 6, 9 and 20:
Osindero in view of Hu and in view of Licata teaches further comprising:
responsive to determining that the first primary script classification corresponds to only one language, storing the output of the determined language for the image-based document as the one language associated with the first primary script classification (Osindero [0063]).
Regarding claims 7 and 11:
Osindero in view of Hu and in view of Licata teaches wherein processing the text in each of the set of bounding boxes to be sampled to identify the first primary script classification further includes:
identifying, for each text line in each of the set of bounding boxes, a text line primary script classification identifying a primary script classification for each text line; and
aggregating each identified text line primary script classification for each text line, wherein the first primary script classification is identified as a primary script classification with a greatest number of aggregated identified text line primary script classification instances for text lines in each of the set of bounding boxes (Osindero [0055]-[0056], fig. 3A; Hu [0032]-[0036], [0040], [0078], fig. 3).
Regarding claim 16:
Osindero in view of Hu and in view of Licata teaches wherein each bounding box includes a set of coordinates corresponding to each corner of the bounding box, wherein the set of coordinates specify a location of each bounding box relative to a coordinate plane for the image-based document, and wherein identifying the set of bounding boxes to be sampled further comprises:
deriving, for each bounding box, a length value and a width value using the set of coordinates for each bounding box; and selecting the set of bounding boxes as bounding boxes with a greatest width value (Osindero [0009], [0053]-[0055], [0059]-[0060]; Hu [0034], [0039], where a region preferred is selected where selection based on the size or the bounded coordinates can be easily derived by one having ordinary skill in the art).
Regarding claim 18:
Osindero in view of Hu and in view of Licata teaches wherein the process further comprises:
processing sets of coordinates for each of the set of bounding boxes to derive an orientation of the image-based document; and
responsive to determining that the orientation of the image-based document is not in an upright orientation, rotating the orientation of the image-based document to the upright orientation (Osindero [0009], [0053]-[0055], [0059]-[0060]; Hu [0032]-[0035], [0040], [0078], fig. 3).
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Osindero (US 20170004374 A1, hereinafter “Osindero”) in view of Hu et al. (WO 2021161095 A1, hereinafter “Hu”) in view of Licata et al. (US 20130073583 A1, hereinafter “Licata”) and in view of Kumar et al. (US 20210374040 A1, hereinafter “Kumar”).
Regarding claim 10:
The combination above fails to teach explicitly wherein the text string comprises a Unicode format.
However, Kumar in the same line of endeavor teaches in one embodiment, the ATG Intelligent Mapper 310 may search images and convert the images to optical character recognition (OCR) and hash code using the image processing 314. OCR engines, such as Tesseract OCR with packages containing an OCR engine and a command line program for example, support Unicode and may recognize many languages as well as many output formats (e.g. HTML, PDF, plain text, TSV, invisible-text-only PDF) (Kumar [0055]).
Therefore, taking the teachings of Osindero , Hu, Licata and Kumar as whole, it would have been obvious to one or ordinary skill in the art before the effective filing date of the application to generate computer file such as text string form OCR that comprises Unicode since it is well known format and allow for easy computer processing (Kumar [0055]).
Claims 12-13, 19 and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Osindero (US 20170004374 A1, hereinafter “Osindero”) in view of Hu et al. (WO 2021161095 A1, hereinafter “Hu”) in view of Licata et al. (US 20130073583 A1, hereinafter “Licata”) and in view of Liu et al. (CN 113221890 A, hereinafter “Liu”).
Regarding claims 12 and 19:
Osindero in view of Hu and in view of Licata teaches explicitly all the limitation of this claim except wherein the plurality of text lines in the image-based document are detected using a distributed brokered networking (DBNet) neural network model using a multi-layer convolutional neural network model backbone.
However, Liu in the same line of endeavor teaches a text detection algorithm wherein OCR text line detection using DBNet text line detection algorithm, the algorithm comprises the following steps:
The text detection algorithm in the invention uses DBNet text line detection algorithm, which can quickly and accurately detect the text line in the image (Liu claim 3);
Therefore, taking the teaching of Osindero, Licata and Liu as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the application to use a distributed brokered networking (DBNet) neural network model using a multi-layer convolutional neural network model backbone to detect the plurality of text lines since it is a well-known technique in the art when applied performs well in text line detection and provide predictable result.
Regarding claims 13:
Osindero in view of Hu, in view of Licata and in view of Liu teaches wherein the first primary script classification of multiple primary script classifications is identified by a convolutional neural network model extracting a sequence of visual features from the image-based document and a recurrent neural network model predicting a text sequence from the extracted sequence of visual features (Osindero [0057]-[00561]; Liu claim 4),
wherein the determined language for the image-based document is determined using a bi-directional long short-term memory (LSTM) neural network model comprising an embedding layer to transform a character sequence of each text line into a numeric sequence and bi-directional LSTM layers extracting contextual features in the numeric sequence to determine the language (Osindero [0057]-[0061]; Liu claim 4).
Regarding claim 21:
Osindero in view of Hu, in view of Licata and in view of Liu teaches further comprising bounding the one or more text lines from the set of the plurality of pixels in the image-based document in a bounding box based at least in part on comparing a value of a particular pixel to a threshold value of a pixel being text (Osindero [0006], [0013], [0051]-[0056], where Osindero teaches determining where to look and the weight of the pixel to determine whether character is present or not (threshold) in the region selected; Hu [0043]).
Claim 22 is rejected under 35 U.S.C. 103 as being unpatentable over Osindero (US 20170004374 A1, hereinafter “Osindero”) in view of Hu et al. (WO 2021161095 A1, hereinafter “Hu”) and in view of Licata et al. (US 20130073583 A1, hereinafter “Licata”) and in view of Ren et al. (CN 109388807 B, hereinafter “Ren”).
Regarding claim 22:
Osindero in view of Hu, in view of Licata and in view of Liu fails to explicitly teach wherein the determining the first primary script classification comprises transforming the set of text characters in each of the set of bounding boxes to numeric sequences to determine a language for each text line.
However, Ren teaches firstly, obtaining the character sequence in the current history content of the electronic medical record of the named entity to be identified; The electronic medical record naming entity identification method provided by this embodiment is realized by combining the convolution network model (CNN) and bidirectional long term memory network model (Bi-LSTM), and the network model only can process the input of the numerical value type; Therefore, when obtaining the character sequence of the electronic medical record of named entity to be identified, it needs to convert it into vector form ([0066]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the application to convert each character into its numerical representation sequence before being inputted into the bidirectional LSTM, in order for the network can better process the information and be more efficient
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WEDNEL CADEAU whose telephone number is (571)270-7843. The examiner can normally be reached Mon-Fri 9:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chieh Fan can be reached at 571-272-3042. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WEDNEL CADEAU/Primary Examiner, Art Unit 2632 April 9, 2026
Prosecution Insights