DETAILED ACTION
This Office action is responsive to the Request for Continued Examination (RCE) filed under 37 CFR §1.53(d) for the instant application on May 27, 2026. The Applicants have properly set forth the RCE, which has been entered into the application, and an examination on the merits follows herewith.
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on May 27, 2026 has been considered by the examiner.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent No. 10,572,215 to Cooper et al. (“Cooper”), over U.S. Patent Application Publication No. 2015/0073907 to Purves et al. (“Purves”), and also over U.S. Patent Application Publication No. 2014/0247278 to Samara et al. (“Samara”).
Regarding claim 1, Cooper describes an augmented reality (AR) system in which a computing device is enabled to analyze information, such as sensor data captured by one or more sensors of the computing device, to identify one or more objects represented in the information and to obtain and present supplemental content based on the identified objects (see e.g. column 1, line 62 – column 2, line 41). Like claimed, Cooper particularly teaches that the computing device can be a mobile computing device (see e.g. column 12, lines 24-38) comprising:
an image sensor configured to capture one or more images comprising one or more barcodes from a field of view of the image sensor (see e.g. column 2, lines 21-26; column 2, line 56 – column 3, line 10; column 8, lines 9-24; and column 12, lines 24-38: Cooper discloses that the mobile computing device can comprise one or more sensors, including a camera for acquiring images and/or video from a field of view of the camera. The camera is considered an image sensor like claimed. Cooper further discloses that the captured images can comprise one or more objects, such as barcodes – see e.g. see e.g. column 2, lines 11-33; and column 6, lines 15-35.);
a visual display (see e.g. column 3, lines 6-12; column 8, lines 9-24; and column 13, lines 30-37: Cooper discloses that the mobile computing device comprises a display screen. Such a display screen is considered a visual display like claimed.);
a memory for storing a software program comprising a plurality of software plug-ins (see e.g. column 13, lines 50-60: Cooper discloses that the mobile computing device includes memory and storage for storing processor-executable instructions. Cooper discloses that the processing of the AR system can be implemented via an application executed on the mobile computing device – see e.g. column 16, lines 36-45. Cooper further discloses that the AR system comprises an extensible architecture in which users can dynamically install and/or activate plug-ins to enable the identification of new types of objects or supplemental content relating to objects – see e.g. column 2, lines 4-9; column 4, lines 20-43; and column 16, lines 6-35. Cooper teaches that the plug-ins can be partially or entirely implemented on the mobile computing device – see e.g. column 19, line 22-56; column 21, lines 6-46; column 23, line 19 – column 24, line 10; and column 26, lines 4-36. In such embodiments, the memory/storage of the mobile computing device would necessarily comprise a software program comprising a plurality of software plug-ins, i.e. the AR application and its plug-ins.); and
a processor (see e.g. column 13, lines 50-60: Cooper discloses that the mobile computing device comprises a processor/controller for executing processor-executable instructions), the processor configured to:
display, in real time, the one or more images captured by the image sensor on the visual display (see e.g. column 3, lines 6-15; and column 8, lines 9-24: Cooper discloses that the display screen of the mobile computing device can display a “live” view of image frames captured by the camera. Cooper further discloses that the augmented reality system operates in real-time to provide the user with the supplemental content and/or functionality while engaging with the primary content captured by the sensors – see e.g. column 2, lines 32-41.);
select one or more software plug-ins of the plurality of software plug-ins that is capable of recognizing one or more input information from the one or more barcodes in the one or more images (see e.g. column 2, lines 11-32; and column 8, lines 9-61: like noted above, Cooper teaches that the mobile computing device is enabled to analyze captured sensor information to identify one or more objects represented therein. Cooper particularly discloses that a recognition module performs recognition on the content captured by the mobile device to identify one or more objects therein – see e.g. column 10, line 65 – column 11, line 15; and column 17, lines 5-37. Moreover, Cooper teaches that the augmented reality system can be extended via one or more plug-ins, e.g. “matchers,” to identify particular objects within the sensor data – see e.g. column 16, lines 6-22; column 17, lines 17-37; column 20, line 60 – column 21, line 5; and column 26, lines 4-36. Cooper suggests that the plug-ins/matchers can include plug-ins/matchers capable of recognizing input information from one or more barcodes within image data captured by the mobile computing device – see e.g. column 17, line 31 – column 18, line 5. The plug-ins employed to recognize information can be selected automatically or by a user – see e.g. column 4, lines 34-43, column 21, lines 27-46; and column 23, line 43 – column 24, line 7. Cooper further teaches that such functionality, including the plug-ins, can be partially or entirely implemented on the mobile computing device – see e.g. column 4, lines 2-6; column 9, lines 15-24; column 10, lines 5-11; column 14, lines 12-34; column 21, lines 6-46; and column 26, lines 4-36. In such embodiments, the processor selects one or more software plug-ins of the plurality of plug-ins to recognize one or more input information, e.g. from one or more barcodes, in the images captured by the camera of the mobile computing device.);
input one or more images of the document to each of the selected one or more software plug-ins (see e.g. column 16, lines 6-22; column 17, line 17 – column 18, line 5; column 20, line 60 – column 21, line 5; and column 26, lines 4-36: like noted above, Cooper teaches that the augmented reality system can be extended via one or more plug-ins, e.g. “matchers,” to identify particular objects such as barcodes within the sensor data. To recognize the particular objects, sensor data such as images are provided to each of the plug-ins/matchers – see e.g. column 17, lines 5-37; column 21, lines 6-60; column 22, lines 4-25; and column 26, lines 4-36.);
utilize a first software plug-in of the one or more software plug-ins to recognize first input information contained within the one or more barcodes from the one or more images, and utilize a second software plug-in of the one or more software plug-ins to recognize second input information contained within the one or more images, wherein the first software plug-in is different from the second software plug-in and wherein the first input information is different from the second input information (see e.g. column 17, line 5 – column 18, line 5; column 21, lines 6-38; column 22, lines 4-25; and column 26, lines 4-36: Cooper teaches that the sensor data can be provided to multiple plug-ins/matchers in parallel, wherein each plug-in/matcher is configured to recognize particular objects within the sensor data, and if a matcher/plug-in recognizes an object within the sensor data, it returns a digital entity with facet data indicating the recognized object. Like noted above, Cooper suggests that the plug-ins/matchers can include plug-ins/matchers capable of recognizing input information from one or more barcodes within image data captured by the mobile computing device – see e.g. column 17, line 17 – column 18, line 5. Other matchers/plug-ins can recognize other types of input information such as text within the image data – see e.g. column 17, line 17- column 18, line 5. Cooper further suggests that multiple objects can be recognized within a document from the image of the document utilizing multiple respective plug-ins – see e.g. column 1, line 62 – column 2, line 10; column 2, line 56 – column 3, line 28; column 4, lines 20-43; column 5, line 48 – column 6, line 14; column 7, lines 30-63; and FIGS. 2A-D. In such embodiments, a first plug-in/matcher is utilized to recognize first input information, e.g. input information contained in one or more barcodes, from the one or more images captured by the mobile computing device; a second plug-in/matcher is utilized to recognize a second input information, e.g. text, from the one or more images, and wherein the first plug-in/matcher is different from the second plug-in/matcher and wherein the first input information recognized by the first plug-in/matcher is different from the second input information recognized by the second plug-in/matcher.);
generate first recognized input information from the first input information, and generate second recognized input information from the second input information (see e.g. column 17, line 5 – column 18, line 5; column 21, lines 6-38; column 22, lines 4-25; and column 26, lines 4-36: like noted above, Cooper teaches that the sensor data can be provided to multiple plug-ins/matchers in parallel, wherein each plug-in/matcher is configured to recognize particular objects within the sensor data, and if a plug-in/matcher recognizes an object within the sensor data, it returns a digital entity with facet data indicating the recognized object. Like further noted above, Cooper suggests that multiple objects can be recognized within a document from the image of the document utilizing multiple respective plug-ins – see e.g. column 1, line 62 – column 2, line 10; column 2, line 56 – column 3, line 28; column 4, lines 20-43; column 5, line 48 – column 6, line 14; column 7, lines 30-63; and FIGS. 2A-D. In such embodiments where multiple objects, e.g. first and second input information, are recognized in a document by multiple respective plug-ins/matchers, a first digital entity with facet data indicating a first recognized object would be generated by a first plug-in/matcher, and a second digital entity with facet data indicating a second recognized object would be generated by a second plug-in/matcher. The first digital entity with facet data indicating the first recognized object is considered first recognized input information generated from the first input information, and the second digital entity with facet data indicating the second recognized object is considered second recognized input information generated from the second input information like claimed.);
combine the first recognized input information and the second recognized input information obtained from the first software plug-in and the second software plug-in (As described above, Cooper teaches embodiments where multiple objects are recognized in a document by multiple respective plug-ins/matchers, wherein a first digital entity with facet data indicating a first recognized object is generated by a first plug-in/matcher, and a second digital entity with facet data indicating a second recognized object is generated by a second plug-in/matcher; the first digital entity with facet data indicating the first recognized object is considered first recognized input information, and the second digital entity with facet data indicating the second recognized object is considered second recognized input information. Cooper suggests that the digital entity and facet data are stored in memory, e.g. so that “resolvers” can identify supplemental content and/or functions corresponding to the entities – see e.g. column 17, line 34 – column 18, line 45; and column 26, lines 19-63. Accordingly, in embodiments where multiple objects are recognized in a document by multiple respective plug-ins/matchers, the entity and facet data provided by the different plug-ins/matchers would understandably be stored in memory. This can be considered combining, e.g. into the same memory, the entity and facet data provided by the different plug-ins/matchers. Cooper thus further teaches combining the first recognized input information and the second recognized input information, i.e. the first and second entity and facet data, obtained from the first plug-in/matcher and the second plug-in/matcher.);
convert the combined first recognized input information and second recognized input information into one or more contextual messages (As described above, Cooper teaches embodiments where multiple objects are recognized in a document by multiple respective plug-ins/matchers, wherein a first digital entity with facet data indicating a first recognized object is generated by a first plug-in/matcher, and a second digital entity with facet data indicating a second recognized object is generated by a second plug-in/matcher. Like further described above, Cooper suggests that the digital entity and facet data are stored in memory, e.g. so that “resolvers” can identify supplemental content and/or functions corresponding to the entities – see e.g. column 17, line 34 – column 18, line 45; and column 26, lines 19-63. Such supplemental content and/or functions can be considered contextual messages like claimed. Accordingly, Cooper is considered to teach converting the combined first recognized input information and second recognized input information, i.e. the first and second entity and facet data stored in memory, into one or more contextual messages, i.e. into supplemental content and/or functions.); and
present the one or more contextual messages, corresponding to the first software plug-in and the second software plug-in, on the visual display (see e.g. column 2, lines 32-41: Cooper discloses that the augmented reality system operates in real-time to provide the user with the supplemental content while engaging with the primary content captured by the sensors. Like noted above, Cooper discloses that the supplemental content can comprise information and/or selectable functions associated with each object identified in the sensor information – see e.g. column 6, lines 15-57; column 8, lines 24-61; and column 18, line 46 – column 19, line 3. Cooper particularly demonstrates that the display screen of the mobile device presents the supplemental content and/or selectable functions, i.e. the one or more contextual messages corresponding to the first software plug-in/matcher and the second software plug-in/matcher, via “ribbons” overlaid on the image frames captured by the camera of the mobile device – see e.g. column 5, line 48 – column 6, line 57; column 7, lines 30-63; column 18, lines 46-66; column 26, lines 37-63; and FIGS. 2A-D.).
Accordingly, Cooper teaches a mobile computing device similar to that of claim 1, but does not explicitly disclose that the one or more barcodes are comprised in an identification document captured in the one or more images, wherein the first input information (i.e. the information that is contained within the one or more barcodes) and the second input information are contained within the identification document, and wherein the one or more software plug-ins are capable of verifying information contained within the identification document, as is required by claim 1. Cooper also does not disclose that the one or more contextual messages are presented at least partly superimposed over the identification document to indicate an association between each of the one or more contextual messages and the one or more barcodes in the identification document, as is further required by claim 1.
Purves nevertheless describes a mobile computing device that (i) displays one or more images of an identification document (e.g. a sales receipt) comprising one or more barcodes, (ii) recognizes input information associated with the one or more barcodes of the identification document, including by verifying information (e.g. merchant information, product information) contained within the identification document, and (iii) presents one or more contextual messages (e.g. virtual layers) related to the recognized input information (see e.g. paragraphs 0199-0201, and FIGS. 7A-B).
It would have been obvious to one of ordinary skill in the art, having the teachings of Cooper and Purves before the effective filing date of the claimed invention, to modify the software plug-ins taught by Cooper, which recognize input information in one or more images, so as to particularly be applicable to images of an identification document comprising one or more barcodes like taught by Purves, whereby the first input information (i.e. the information that is contained within the one or more barcodes) and the second input information would be contained within the identification document, and the one or more software plug-ins would be capable of verifying information contained within the identification document. It would have been advantageous to one of ordinary skill to utilize such a combination because it would enable relevant contextual messages to be presented to the user regarding the identification document, as is demonstrated by Purves (see e.g. paragraphs 0199-0201, and FIGS. 7A-B). Accordingly, Cooper and Purves are considered to teach a mobile computing device similar to that of claim 1, but do not explicitly teach that the one or more contextual messages are presented at least partly superimposed over the identification document to indicate an association between each of the one or more contextual messages and the one or more barcodes in the identification document, as is further required by claim 1.
Samara nevertheless generally teaches presenting one or more contextual messages (i.e. within an “augment”) at least partly superimposed over a barcode to indicate an association between each of the one or more contextual messages and the barcode (see e.g. paragraphs 0003-0004, 0006-0007, 0044-0045 and 0066-0073, and FIGS. 7A-I).
It would have been obvious to one of ordinary skill in the art, having the teachings of Cooper, Purves and Samara before the effective filing date of the claimed invention, to modify the mobile computing device taught by Cooper and Purves so as to present the one or more contextual messages (i.e. the contextual messages corresponding to the first software plug-in and the second software plug-in) at least partly superimposed over the one or more barcodes (i.e. the barcodes within the identification document) to indicate an association between each of the one or more contextual messages and the one or more barcodes, as is taught by Samara. It would have been advantageous to one of ordinary skill to utilize such a combination because it can provide the user a more immersive experience, as is suggested by Samara (see e.g. paragraphs 0006-0007). Accordingly, Cooper, Purves and Samara are considered to teach, to one of ordinary skill in the art, a mobile computing device like that of claim 1.
As per claim 2, Cooper further teaches that the one or more input information comprises at least one of a recognizable object or a symbol (see e.g. column 2, lines 11-32; and column 8, lines 9-61: like noted above, Cooper teaches that mobile computing device is enabled to analyze captured sensor information to identify one or more objects represented therein.). Purves provides a similar teaching (see e.g. paragraphs 0209-0210, 0216, 0218-0222 and 0250-0252, and FIGS. 9A-B and 11-13). Accordingly, the above-described combination of Cooper, Purves and Samara further teaches a mobile computing device like that of claim 2.
As per claim 3, Cooper further teaches that the first software plug-in and the second software plug-in utilized by the processor are based on user input, and wherein the user input is indicative of user behavior (see e.g. column 23, lines 43-63: Cooper discloses that certain plug-ins can be dynamically activated based on user behavior, e.g. when the user leaves his or her home. Additionally, Cooper discloses that specific plug-ins for certain sensor data can be selected by the user – see e.g. column 21, lines 27-37. Cooper also discloses that the identification of objects within the sensor data, e.g. by the plug-ins, can be performed in response to user input indicative of user behavior, e.g. in response to gestures – see e.g. column 5, lines 35-47.). Purves similarly teaches that the identification of objects within the sensor data can be performed in response to user input indicative of user behavior (e.g. pointing gestures) (see paragraphs 0209-0210, 0216, 0218-0222 and 0250-0252). Like noted above, Cooper teaches that such recognition can be performed by one or more plug-ins (see e.g. column 16, lines 6-22; column 17, lines 17-37; column 20, line 60 – column 21, line 5; and column 26, lines 4-36). Accordingly, the above-described combination of Cooper, Purves and Samara further teaches a mobile computing device like that of claim 3, in which the first software plug-in and the second software plug-in utilized by the processor are based on user input, and wherein the user input is indicative of user behavior.
As per claim 4, Cooper further teaches that the processor is configured to utilize the first software plug-in and the second software plug-in to generate one or more information elements from the one or more input information (see e.g. column 16, lines 6-22; column 17, lines 17-37; column 20, line 60 – column 21, line 5; and column 26, lines 4-36: like noted above, Cooper teaches that one or more plug-ins, e.g. “matchers,” can be utilized to identify particular objects within the sensor data. The processor thus executes the one or more plug-ins, i.e. the first software plug-in and the second software plug-in, to generate one or more information elements from the input information, i.e. to generate indications of identified objects in the sensor data.). Accordingly, the above-described combination of Cooper, Purves and Samara further teaches a mobile computing device like that of claim 4.
As per claim 5, Cooper teaches that the processor is configured to convert the one or more information elements to the one or more contextual messages (see e.g. column 11, line 16 – column 12, line 23: Cooper discloses that a supplemental content module and a function module operate to identify supplemental content and functions related to the objects identified in the sensor content. Cooper also teaches that one or more resolver plug-ins can be utilized to provide the supplemental content and/or functions associated with the identified objects – see e.g. column 16, lines 22-35; column 18, lines 6-45; column 19, lines 22-56; and column 26, lines 37-63. The processor thus executes the supplemental content module, the function module and/or different plug-ins to convert the one or more information elements, i.e. the identified one or more objects, into the one or more contextual messages, i.e. supplemental content and/or functions.). Accordingly, the above-described combination of Cooper, Purves and Samara further teaches a mobile computing device like that of claim 5.
As per claim 6, Cooper further teaches decoding a symbol to generate the one or more information elements in an instance in which the one or more input information includes the symbol, wherein the symbol corresponds to a barcode (see e.g. column 2, lines 21-41; column 6, lines 26-39; column 17, line 65 – column 18, line 5; and column 18, lines 22-32: Cooper discloses that the objects recognized by the AR system can include bar codes, whereby the supplemental information and/or functions relate to the object. In such circumstances, the AR system understandably decodes a symbol to generate one or more information elements, wherein the symbol corresponds to a bar code.). Accordingly, the above-described combination of Cooper, Purves and Samara further teaches a mobile computing device like that of claim 6.
Regarding claims 7 and 13, Cooper describes an augmented reality (AR) system in which a computing device is enabled to analyze information, such as sensor data captured by one or more sensors of the computing device, to identify one or more objects represented in the information and to obtain and present supplemental content based on the identified objects (see e.g. column 1, line 62 – column 2, line 41). Like claimed, Cooper particularly teaches:
receiving, by a processor, one or more images comprising one or more barcodes from a field of view of an image sensor (see e.g. column 2, lines 21-26; column 2, line 56 – column 3, line 10; column 8, lines 9-24; and column 12, lines 24-38: Cooper describes a mobile computing device that comprises one or more sensors, including a camera for acquiring images and/or video from a field of view of the camera. Cooper further discloses that the captured images can comprise one or more objects, such as barcodes – see e.g. see e.g. column 2, lines 11-33; and column 6, lines 15-35. Moreover, Cooper discloses that the mobile computing device also comprises a processor/controller for executing processor-executable instructions for gathering and processing the sensor data – see e.g. column 13, lines 50-65.);
displaying, by the processor in real-time, the one or more images captured by the image sensor (see e.g. column 3, lines 6-15; and column 8, lines 9-24: Cooper discloses that the mobile computing device comprises a display screen that can display a “live” view of image frames captured by the camera. Cooper further discloses that the augmented reality system operates in real-time to provide the user with the supplemental content and/or functionality while engaging with the primary content captured by the sensors – see e.g. column 2, lines 32-41.);
selecting, by the processor, one or more software plug-ins of a plurality of software plug-ins that is capable of recognizing one or more input information from the one or more barcodes in the one or more images (see e.g. column 2, lines 11-32; and column 8, lines 9-61: Cooper teaches that the mobile computing device is enabled to analyze captured sensor information to identify one or more objects represented therein. Cooper particularly discloses that a recognition module performs recognition on the content captured by the mobile device to identify one or more objects therein – see e.g. column 10, line 65 – column 11, line 15; and column 17, lines 5-37. Moreover, Cooper teaches that the augmented reality system can be extended via one or more plug-ins, e.g. “matchers,” to identify particular objects within the sensor data – see e.g. column 16, lines 6-22; column 17, lines 17-37; column 20, line 60 – column 21, line 5; and column 26, lines 4-36. Cooper suggests that the plug-ins/matchers can include plug-ins/matchers capable of recognizing input information from one or more barcodes within image data captured by the mobile computing device – see e.g. column 17, line 31 – column 18, line 5. The plug-ins employed to recognize information can be selected automatically or by a user – see e.g. column 4, lines 34-43, column 21, lines 27-46; and column 23, line 43 – column 24, line 7. Cooper further teaches that such functionality, including the plug-ins, can be partially or entirely implemented on the mobile computing device, i.e. via the processor thereof – see e.g. column 4, lines 2-6; column 9, lines 15-24; column 10, lines 5-11; column 14, lines 12-34; column 21, lines 6-46; and column 26, lines 4-36. In such embodiments, the processor selects one or more software plug-ins of the plurality of plug-ins to recognize one or more input information, i.e. from one or more barcodes, in the images captured by the camera of the mobile computing device.);
inputting, by the processor, the one or more images to each of the selected one or more software plug-ins (see e.g. column 16, lines 6-22; column 17, line 17 – column 18, line 5; column 20, line 60 – column 21, line 5; and column 26, lines 4-36: like noted above, Cooper teaches that the augmented reality system can be extended via one or more plug-ins, e.g. “matchers,” to identify particular objects such as barcodes within the sensor data. To recognize the particular objects, sensor data such as images are provided to each of the plug-ins/matchers – see e.g. column 17, lines 5-37; column 21, lines 6-60; column 22, lines 4-25; and column 26, lines 4-36.);
utilizing, by the processor, a first software plug-in of the one or more software plug-ins to recognize a first input information contained within the one or more barcodes from the one or more images, and a second software plug-in of the one or more software plug-ins to recognize second input information contained within the one or more images, wherein the first software plug-in is different from the second software plug-in and wherein the first input information is different from the second input information (see e.g. column 17, line 5 – column 18, line 5; column 21, lines 6-38; column 22, lines 4-25; and column 26, lines 4-36: Cooper teaches that the sensor data can be provided to multiple plug-ins/matchers in parallel, wherein each plug-in/matcher is configured to recognize particular objects within the sensor data, and if a matcher/plug-in recognizes an object within the sensor data, it returns a digital entity with facet data indicating the recognized object. Like noted above, Cooper suggests that the plug-ins/matchers can include plug-ins/matchers capable of recognizing input information from one or more barcodes within image data captured by the mobile computing device – see e.g. column 17, line 17 – column 18, line 5. Other matchers/plug-ins can recognize other types of input information such as text within the image data – see e.g. column 17, line 17- column 18, line 5. Cooper further suggests that multiple objects can be recognized within the image by utilizing multiple respective plug-ins – see e.g. column 1, line 62 – column 2, line 10; column 2, line 56 – column 3, line 28; column 4, lines 20-43; column 5, line 48 – column 6, line 14; column 7, lines 30-63; and FIGS. 2A-D. In such embodiments, a first plug-in/matcher is utilized to recognize first input information, e.g. information contained in one or more barcodes, from the one or more images captured by the mobile computing device; a second plug-in/matcher is utilized to recognize second input information, e.g. text, from the one or more images, and wherein the first plug-in/matcher is different from the second plug-in/matcher and wherein the first input information recognized by the first plug-in/matcher is different from the second input information recognized by the second plug-in/matcher.);
generating, by the processor, first recognized input information from the first input information, and second recognized input information from the second input information (see e.g. column 17, line 5 – column 18, line 5; column 21, lines 6-38; column 22, lines 4-25; and column 26, lines 4-36: like noted above, Cooper teaches that the sensor data can be provided to multiple plug-ins/matchers in parallel, wherein each plug-in/matcher is configured to recognize particular objects within the sensor data, and if a plug-in/matcher recognizes an object within the sensor data, it returns a digital entity with facet data indicating the recognized object. Cooper further suggests that multiple objects can be recognized within a document from the image of the document utilizing multiple respective plug-ins – see e.g. column 1, line 62 – column 2, line 10; column 2, line 56 – column 3, line 28; column 4, lines 20-43; column 5, line 48 – column 6, line 14; column 7, lines 30-63; and FIGS. 2A-D. In such embodiments where multiple objects, e.g. first and second input information, are recognized in a document by multiple respective plug-ins/matchers, a first digital entity with facet data indicating a first recognized object would be generated by a first plug-in/matcher, and a second digital entity with facet data indicating a second recognized object would be generated by a second plug-in/matcher. The first digital entity with facet data indicating the first recognized object is considered first recognized input information generated from the first input information, and the second digital entity with facet data indicating the second recognized object is considered second recognized input information generated from the second input information like claimed.);
combining, by the processor, the first recognized input information and the second recognized input information obtained from the first software plug-in and the second software plug-in (As described above, Cooper teaches embodiments where multiple objects are recognized in a document by multiple respective plug-ins/matchers, wherein a first digital entity with facet data indicating a first recognized object is generated by a first plug-in/matcher, and a second digital entity with facet data indicating a second recognized object is generated by a second plug-in/matcher; the first digital entity with facet data indicating the first recognized object is considered first recognized input information, and the second digital entity with facet data indicating the second recognized object is considered second recognized input information. Cooper suggests that the digital entity and facet data are stored in memory, e.g. so that “resolvers” can identify supplemental content and/or functions corresponding to the entities – see e.g. column 17, line 34 – column 18, line 45; and column 26, lines 19-63. Accordingly, in embodiments where multiple objects are recognized in a document by multiple respective plug-ins/matchers, the entity and facet data provided by the different plug-ins/matchers would understandably be stored in memory. This can be considered combining, e.g. into the same memory, the entity and facet data provided by the different plug-ins/matchers. Cooper thus further teaches combining the first recognized input information and the second recognized input information, i.e. the first and second entity and facet data, obtained from the first plug-in/matcher and the second plug-in/matcher.);
converting, by the processor, the combined first recognized input information and second recognized input information into one or more contextual messages by using the first software plug-in and the second software plug-in (As described above, Cooper teaches embodiments where multiple objects are recognized in a document by multiple respective plug-ins/matchers, wherein a first digital entity with facet data indicating a first recognized object is generated by a first plug-in/matcher, and a second digital entity with facet data indicating a second recognized object is generated by a second plug-in/matcher. Like further described above, Cooper suggests that the digital entity and facet data are stored in memory, e.g. so that “resolvers” can identify supplemental content and/or functions corresponding to the entities – see e.g. column 17, line 34 – column 18, line 45; and column 26, lines 19-63. Such supplemental content and/or functions can be considered contextual messages like claimed. Accordingly, Cooper is considered to teach converting the combined first recognized input information and second recognized input information, i.e. the first and second entity and facet data stored in memory by using the first software plug-in and the second software plug-in, into one or more contextual messages, i.e. into supplemental content and/or functions.); and
presenting, by the processor, the one or more contextual messages on a user interface displayed on a visual display (see e.g. column 2, lines 32-41: Cooper discloses that the augmented reality system operates in real-time to provide the user with the supplemental content while engaging with the primary content captured by the sensors. Like noted above, Cooper discloses that the supplemental content can comprise information and/or selectable functions associated with each object identified in the sensor information – see e.g. column 6, lines 15-57; column 8, lines 24-61; and column 18, line 46 – column 19, line 3. Cooper particularly demonstrates that the display screen of the mobile device presents the supplemental content and/or selectable functions, i.e. the one or more contextual messages corresponding to the first software plug-in/matcher and the second software plug-in/matcher, via “ribbons” overlaid on the image frames captured by the camera of the mobile device – see e.g. column 5, line 48 – column 6, line 57; column 7, lines 30-63; column 18, lines 46-66; column 26, lines 37-63; and FIGS. 2A-D.).
Accordingly, Cooper teaches a method similar to that of claim 7. Cooper discloses that such teachings can be implemented via processor-executable instructions stored on non-transitory computer-readable media, e.g. the storage of a mobile device (see e.g. column 10, lines 5-20; column 13, lines 50-60; column 26, lines 4-63; and column 28, lines 5-67). Such non-transitory computer-readable media comprising computer-executable instructions for implementing the above-described teachings of Cooper is considered a non-transitory computer-readable medium similar to that of claim 13. Cooper, however, does not explicitly disclose that the one or more barcodes are comprised in an identification document captured in the one or more images, wherein the first input information (i.e. the information that is contained within the one or more barcodes) and the second input information are contained within the identification document, and wherein the one or more software plug-ins are capable of verifying information contained within the identification document, as is required by claims 7 and 13. Cooper also does not disclose that the one or more contextual messages are presented at least partly superimposed over the identification document to indicate an association between each of the one or more contextual messages and the one or more barcodes in the identification document, as is further required by claims 7 and 13.
Purves nevertheless describes a mobile computing device that (i) displays one or more images of an identification document (e.g. a sales receipt) comprising one or more barcodes, (ii) recognizes input information associated with the one or more barcodes of the identification document, including by verifying information (e.g. merchant information, product information) contained within the identification document, and (iii) presents one or more contextual messages (e.g. virtual layers) related to the recognized input information (see e.g. paragraphs 0199-0201, and FIGS. 7A-B).
It would have been obvious to one of ordinary skill in the art, having the teachings of Cooper and Purves before the effective filing date of the claimed invention, to modify the software plug-ins taught by Cooper, which recognize input information in one or more images, so as to particularly be applicable to images of an identification document comprising one or more barcodes like taught by Purves, whereby the first input information (i.e. the information that is contained within the one or more barcodes) and the second input information would be contained within the identification document, and the one or more software plug-ins would be capable of verifying information contained within the identification document. It would have been advantageous to one of ordinary skill to utilize such a combination because it would enable relevant contextual messages to be presented to the user regarding the identification document, as is demonstrated by Purves (see e.g. paragraphs 0199-0201, and FIGS. 7A-B). Accordingly, Cooper and Purves are considered to teach a method similar to that of claim 7 and a non-transitory computer-readable medium similar to that of claim 13, but do not explicitly teach that the one or more contextual messages are presented at least partly superimposed over the identification document to indicate an association between each of the one or more contextual messages and the one or more barcodes in the identification document, as is further required by claims 7 and 13.
Samara nevertheless generally teaches presenting one or more contextual messages (i.e. within an “augment”) at least partly superimposed over a barcode to indicate an association between each of the one or more contextual messages and the barcode (see e.g. paragraphs 0003-0004, 0006-0007, 0044-0045 and 0066-0073, and FIGS. 7A-I).
It would have been obvious to one of ordinary skill in the art, having the teachings of Cooper, Purves and Samara before the effective filing date of the claimed invention, to modify the mobile computing device taught by Cooper and Purves so as to present the one or more contextual messages (i.e. the contextual messages corresponding to the first software plug-in and the second software plug-in) at least partly superimposed over the one or more barcodes (i.e. the barcodes within the identification document) to indicate an association between each of the one or more contextual messages and the one or more barcodes, as is taught by Samara. It would have been advantageous to one of ordinary skill to utilize such a combination because it can provide the user a more immersive experience, as is suggested by Samara (see e.g. paragraphs 0006-0007). Accordingly, Cooper, Purves and Samara are considered to teach, to one of ordinary skill in the art, a method like that of claim 7 and a non-transitory computer-readable medium like that of claim 13.
As per claims 8 and 14, Cooper further teaches that the one or more input information comprises at least one of a recognizable object or a symbol (see e.g. column 2, lines 11-32; and column 8, lines 9-61: like noted above, Cooper teaches that mobile computing device is enabled to analyze captured sensor information to identify one or more objects represented therein.). Purves provides a similar teaching (see e.g. paragraphs 0209-0210, 0216, 0218-0222 and 0250-0252, and FIGS. 9A-B and 11-13). Accordingly, the above-described combination of Cooper, Purves and Samara further teaches a method and non-transitory computer-readable medium like in claims 8 and 14, respectively.
As per claims 9 and 15, Cooper further teaches that the first software plug-in and the second software plug-in utilized by the processor is based on user input, and wherein the user input is indicative of user behavior (see e.g. column 23, lines 43-63: Cooper discloses that certain plug-ins can be dynamically activated based on user behavior, e.g. when the user leaves his or her home. Additionally, Cooper discloses that specific plug-ins for certain sensor data can be selected by the user – see e.g. column 21, lines 27-37. Cooper also discloses that the identification of objects within the sensor data, e.g. by the plug-ins, can be performed in response to user input indicative of user behavior, e.g. in response to gestures – see e.g. column 5, lines 35-47.). Purves similarly teaches that the identification of objects within the sensor data can be performed in response to user input indicative of user behavior (e.g. pointing gestures) (see paragraphs 0209-0210, 0216, 0218-0222 and 0250-0252). Like noted above, Cooper teaches that such recognition can be performed by one or more plug-ins (see e.g. column 16, lines 6-22; column 17, lines 17-37; column 20, line 60 – column 21, line 5; and column 26, lines 4-36). Accordingly, the above-described combination of Cooper, Purves and Samara further teaches a method and non-transitory computer-readable medium like in claims 9 and 15, respectively.
As per claims 10 and 16, Cooper further teaches that the processor is configured to utilize the first software plug-in and the second software plug-in to generate one or more information elements from the one or more input information (see e.g. column 16, lines 6-22; column 17, lines 17-37; column 20, line 60 – column 21, line 5; and column 26, lines 4-36: like noted above, Cooper teaches that one or more plug-ins, e.g. “matchers,” can be utilized to identify particular objects within the sensor data. The processor thus executes the one or more plug-ins, i.e. the first software plug-in and the second software plug-in, to generate one or more information elements from the input information, i.e. to generate indications of identified objects in the sensor data.). Accordingly, the above-described combination of Cooper, Purves and Samara further teaches a method and non-transitory computer-readable medium like in claims 10 and 16, respectively.
As per claims 11 and 17, Cooper teaches that the processor is configure to convert the one or more information elements to the one or more contextual messages (see e.g. column 11, line 16 – column 12, line 23: like noted above, Cooper discloses that a supplemental content module and a function module operate to identify supplemental content and functions related to the objects identified in the sensor content. Cooper also teaches that one or more plug-ins can be utilized to provide the supplemental content and/or functions associated with the identified objects – see e.g. column 16, lines 22-35; column 18, lines 6-45; column 19, lines 22-56; and column 26, lines 37-63. The processor thus executes the supplemental content module, the function module and/or different plug-ins to convert the one or more information elements, i.e. the identified one or more objects, to the one or more contextual messages, i.e. supplemental content and/or functions.). Accordingly, the above-described combination of Cooper, Purves and Samara further teaches a method and non-transitory computer-readable medium like in claims 11 and 17, respectively.
As per claims 12 and 18, Cooper further teaches decoding a symbol to generate the one or more information elements in an instance in which the one or more input information includes the symbol, wherein the symbol corresponds to a barcode (see e.g. column 2, lines 21-41; column 6, lines 26-39; column 17, line 65 – column 18, line 5; and column 18, lines 22-32: Cooper discloses that the objects recognized by the AR system can include bar codes, whereby the supplemental information and/or functions relate to the object. In such circumstances, the AR system understandably decodes a symbol to generate one or more information elements, wherein the symbol corresponds to a bar code.). Accordingly, the above-described combination of Cooper, Purves and Samara further teaches a method and non-transitory computer-readable medium like in claims 12 and 18, respectively.
As per claim 19, it would have been obvious, as is described above, to modify the software plug-ins taught by Cooper, which recognize input information in a document, so as to particularly be applicable to an identification document like taught by Purves. Purves suggests that the identification document can be a driver’s license (see e.g. paragraphs 0219 and 0252, and FIG. 11). Accordingly, the above-described combination of Cooper, Purves and Samara further teaches a mobile computing device like that of claim 19.
As per claim 20, it would have been obvious to modify the software plug-ins taught by Cooper, which recognize input information in one or more images, so as to particularly be applicable to images of an identification document comprising one or more barcodes like taught by Purves, whereby the first input information (i.e. the information that is contained within the one or more barcodes) and the second input information would be contained within the identification document, and the one or more software plug-ins would be capable of verifying information contained within the identification document. Cooper further teaches verifying recognized input information contained within the identification document with a predefined requirement (i.e. verifying if associated virtual labels have been configured by the user to be displayed), and, on successfully verifying that the recognized input information contained within the identification document satisfies the predetermined requirement (i.e. that the virtual labels have been configured to be displayed), presenting the one or more contextual messages (i.e. virtual labels) associated with the recognized input information (see e.g. paragraphs 0201-0202, and FIGS 7B-C). As further noted above, it would have been obvious to modify the mobile computing device taught by Cooper and Purves so as to present the one or more contextual messages (i.e. the contextual messages corresponding to the first software plug-in and the second software plug-in) at least partly superimposed over the one or more barcodes (i.e. the barcodes within the identification document) to indicate an association between each of the one or more contextual messages and the one or more barcodes, as is taught by Samara. Accordingly, Cooper, Purves and Samara are considered to teach, to one of ordinary skill in the art: (i) verifying the first recognized input information and the second recognized input information contained within the identification document with a predefined requirement; and (ii) on successfully verifying that the first recognized input information and the second recognized input information contained within the identification document satisfy the predefined requirement, presenting the one or more contextual messages, corresponding to the first software plug-in and the second software plug-in, at least partly superimposed over the identification document on the visual display to indicate the association between each of the one or more contextual messages and the one or more barcodes in the identification document, as is required by claim 20.
Response to Arguments
The Examiner acknowledges the Applicant’s amendments to claims 1, 7 and 13, and addition of new claim 20. In response to these amendments, the objections presented in the previous Office Action to claims 1-19 are respectfully withdrawn.
The Applicant’s arguments concerning the 35 U.S.C. § 103 rejections presented in the previous Office Action have been considered, but are moot in view of the new grounds of rejection presented above. Similarly, the Applicant’s arguments concerning new claim 20 have been considered, but are moot in view of the above new grounds of rejection.
Conclusion
The prior art made of record on form PTO-892 and not relied upon is considered pertinent to applicant’s disclosure. The applicant is required under 37 C.F.R. §1.111(C) to consider these references fully when responding to this action. In particular, the U.S. Patent Application Publication to Whelihan cited therein demonstrates an augmented reality display that presents a contextual message adjacent a barcode in order to indicate an association between the contextual message and the barcode.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BLAINE T BASOM whose telephone number is (571)272-4044. The examiner can normally be reached Monday-Friday, 9:00 am - 5:30 pm, EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matt Ell can be reached at (571)270-3264 . The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BTB/
6/27/2026
/MATTHEW ELL/Supervisory Patent Examiner, Art Unit 2141
Prosecution Insights