DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendments / Arguments
Applicant’s arguments filed on July 20, 2026 with respect to claims 1, 12, 20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Kocharlakota et al. (US 2020/0005542 A1, hereinafter “Kocharlakota”) in view of Rhoads et al. (US 20150016712 A1, hereinafter “Rhoads”).
Regarding claim 1,
Kocharlakota discloses:
A computer-implemented method, comprising:
obtaining identifying information for an object, detected in a field of view of an AR device (Kocharlakota: par. 7, ". . .a method for providing a personalized augmented reality (AR) display includes obtaining, at an augmented reality apparatus, image data of a camera field of view, . . . , identifying a first object in the camera field of view based on the image data)
and the identifying information comprising implicit signals that represent contextual data associated with the object (NOTE 1A: The implicit signals that represent contextual data associated with the objects are the image data obtained at an AR apparatus such as GPS data (¶105), binary descriptors or other data associated with the identification of an object (¶55), location data assigning a specific geospatial coordinate (¶51), object signatures (¶45, 80), user identifier (¶45), visibility data (¶45), features in the image data (¶55), image patches associated with points in the feature space to recognize objects within the space (¶86-88)),
and responsive to determining that the identified object instance (NOTE: the claimed “the object instance” is Kocharlakota’s augmentation target object, which is an object instance within the field of view) is associated with the first object record, presenting, via the AR device, AR content that is specific to the first object record (Kocharlakota: par. 8, "and responsive to determining that the first object is the augmentation target, display, on the internally-facing display, an item of AR content associated with the augmentation target in the AR field of view").
Although Kocharlakota further teaches identifying a parent object to another object in the camera field of view based on image data received from the camera, Kocharlakota fails to explicitly disclose: wherein the implicit signals include visual features of at least one of: a background scene containing the object; or nearby objects located in a vicinity of the object that are depicted in images of the object’s surroundings and determined based on image analysis of the images; identifying a unique instance of the object based on determining a match between visual features of the object’s surroundings and stored identifying information of a known instance of the object.
The analogous art Rhoads teaches:
wherein the implicit signals include visual features of one or more nearby objects located in a vicinity of the object that are depicted in images of the object’s surroundings and determined based on image analysis of the images (Rhoads: par. 225, “the auxiliary information includes information derived by processing . . . portions of the query data that do not represent the aforementioned object-of-interest. In one implementation, the query data is processed to identify one or more objects surrounding the object-of-interest and perform an object recognition process on the identified objects. In this case, the object recognition process could be performed at a higher level of abstraction than the object recognition process that would typically be performed to recognize the object-of-interest. For example, the query data would be processed to estimate whether an identified surrounding object was a tree . . . to estimate whether an identified surrounding object was a car . . . etc. In another implementation, the query data is processed to estimate a spatial relationship between an identified surrounding object and the object-of-interest and/or estimate a spatial relationship between different identified surrounding objects. In another implementation, the query data is processed to estimate a size of the identified surrounding object relative to the object-of-interest. In another implementation, the query data is processed to estimate any colors, textures, shapes, etc., associated with the identified surrounding object. . .”; NOTE: The object recognition process constitute the claimed image analysis. The colors, textures, shapes, etc. associated with the objects surrounding the object of interest are the visual features representing contextual data of one or more nearby objects located in the vicinity of the object.))
identifying a unique instance of the object based on determining a match between visual features of the object’s surroundings and stored identifying information of a known instance of the object (Rhoads: ¶216, “. . . querying an object registry containing information regarding a (potentially vast) multitude of registered or enrolled objects, all in an effort to identify which of the registered/enrolled objects most closely matches the discerned aspects. In some embodiments of the present technology, the total number of records to be searched in an object recognition process can be reduced by reference to one or more items auxiliary information. Such auxiliary information can thus be used to filter out records stored within the object registry that would otherwise represent an invalid or unlikely candidate match . . . ”; par. 225, “. . . The object registry may then be queried to identify any stored record(s) associated with object identifying information, location information, descriptive information, related object information, similar object information, etc., that corresponds to the aforementioned auxiliary information that was discerned, estimated or otherwise recognized by any suitable technique . . .”; par. 148, “. . . allows, e.g., two tea pots that are identical in shape, but different in ornamentation, to be distinguished and separately identified. . .”; par. 166, “. . . the subject tea pot--which may have come from a paint-your-own-pots craft shop, can be distinguished from all other such pots--despite the fact that their shapes may all be identical . . .”; NOTE: Each enrolled objects are known instances of objects in a registry, or database, for example teapots. Each of these known instances of teapots in registry have corresponding auxiliary information. Such information includes visual features of surrounding objects such as color, shape, texture, etc. as cited above in reference to paragraph 55. The system then uses these auxiliary information to match, or query the registry to distinguish a specific teapot from all other teapots, constituting to a unique instance of that teapot.)
wherein the stored identifying information is stored in the first object record corresponding to the known instance of the object and comprises identifiers of one or more objects located in a vicinity of the known instance of the object (Rhoads: par. 187, “. . . The object metadata associated with a particular registered or enrolled object can also include related object information identifying (e.g., by one or more items of object identifying information) any other object that is (or is likely to be, or has in the past been, etc.) related to the particular registered or enrolled object. . .”; par. 225, “. . . Rhoads: par. 225, “the auxiliary information includes information derived by processing . . . portions of the query data that do not represent the aforementioned object-of-interest. . . . to identify one or more objects surrounding the object-of-interest and perform an object recognition process on the identified objects. . . to estimate any colors, textures, shapes, etc., associated with the identified surrounding object. . .” NOTE: The metadata for an enrolled object, constitutes the identifying information corresponding to the known instance of the object. The metadata comprise identifiers that identifies colors, textures, shapes of objects surrounding the object of interest.)
It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to combine Kocharlakota, and Rhoads to include: wherein the implicit signals include visual features of one or more nearby objects located in a vicinity of the object that are depicted in images of the object’s surroundings and determined based on image analysis of the images; identifying a unique instance of the object based on determining a match between visual features of the object’s surroundings and stored identifying information of a known instance of the object, wherein the stored identifying information is stored in the first object record corresponding to the known instance of the object and comprises identifiers of one or more objects located in a vicinity of the known instance of the object.
The reason for doing so is to “query data to discern one or more aspects associated with an object-of-interest and thereafter querying an object registry containing information regarding a (potentially vast) multitude of registered or enrolled objects, all in an effort to identify which of the registered/enrolled objects most closely matches the discerned aspects” and to filter out records stored within the object registry that would otherwise represent an invalid or unlikely candidate match” (Rhoads paragraph 216).
Regarding claim 2, depending on 1,
The combination of Kocharlakota and Rhoads discloses:
The method of claim 1,
Kocharlakota further teaches:
wherein the identified object instance is a real object of a first class of objects and wherein determining that the identified object instance is associated with the first object record comprises distinguishing the identified object instance from at least one other object instance of the same class based on the implicit signals (Kocharlakota: par. 122, "According to some embodiments, at operation 1225, a determination of whether the object identified at operation 1215 is an augmentation target is performed. In certain embodiments, this determination is performed by comparing an object signature of the object identified at operation 1215 against a set of object signatures (for example, object signatures maintained in a PCAR database utilizing database schema 600 in FIG. 6). In various embodiments, the determination of whether an object comprises an augmentation target is based on whether the object belongs to a defined class of objects which can be augmentation targets (for example, non-transitory items, such as buildings or pieces of furniture”; par. 81, “. . . object location data can be used to distinguish between objects which are generic augmentation targets (for example, any instance of a particular model of automobile), and objects which are specific augmentation targets (for example, a specific instance of a recognized refrigerator at a particular location, which is a user's or viewer's “home fridge.”) . . .”; NOTE: Also see, Fig. 8E. It shows two refrigerators, the system identified the refrigerator on the right to display the AR content.)".
Regarding claim 3, depending on claim 1,
The combination of Kocharlakota and Rhoads discloses:
The method of claim 1,
Kocharlakota further teaches:
wherein obtaining the identifying information for the object comprises performing object detection based on parsing a video depicting the object using an object detection model (Kocharlakota: par. 50, "image data 215 comprises data output by from one or more externally-oriented sensors (for example, a CMOS video camera or a dynamic vision sensor (DVS), configured to produce a comparatively less data-intensive representation of a field of view by capturing changes in the intensity of received light at the pixels of the sensor").
(Kocharlakota: ¶56, “According to various embodiments, signature engine 225 receives descriptors of recognized objects in a field of view (for example, a camera field of view of an AR apparatus operating as client platform 205) from object recognition engine 227. In certain embodiments, object recognition engine 227 operates continuously, and continuously scans image data from a field of view for objects to recognize.
(Kocharlakota: ¶55, “. . .The object recognition engine 227 performs object recognition using a binary descriptor based object recognition technique, including, for example, the binary robust independent elementary features (BRIEF), binary robust invariant scalable keypoints (BRISK), fast retina keypoints (FREAK), or oriented fast and rotated BRIEF (ORB). According to various embodiments, object recognition engine 227 identifies one or more objects within the field of view of an externally facing sensor (such as an CMOS camera or DVS sensor) and provides a descriptor or other data associated with the identification of the first object. . .“
NOTE 3A: Kocharlakota discloses that the AR apparatus employs a CMOS video camera, or a DVS Dynamic Vision Sensor which are both capable of obtaining video. The object recognition engine continuously scans image data (parsing a video) from a field of view for objects to recognize.
Regarding claim 4, depending on claim 3,
The combination of Kocharlakota and Rhoads discloses:
The method of claim 3,
Kocharlakota further teaches:
wherein obtaining the identifying information for the object comprises performing image analysis on frames of the video (Kocharlakota: par. 50, "image data 215 comprises data output by from one or more externally-oriented sensors (for example, a CMOS video camera or a dynamic vision sensor (DVS), configured to produce a comparatively less data-intensive representation of a field of view by capturing changes in the intensity of received light at the pixels of the sensor").
Regarding claim 5, depending on claim 4,
The combination of Kocharlakota and Rhoads discloses:
The method of claim 4,
Kocharlakota further teaches:
wherein the identifying information for the object comprises at least one of a barcode, a QR code, an NFC tag, or a serial number (Kocharlakota: par. 107, "In certain embodiments according to the present disclosure, at operation 1020, one or more components (for example, an augmentation engine, such as augmentation engine 229 in FIG. 2 and an object recognition engine, such as object recognition engine 227 in FIG. 2) determines whether there are objects within a given sensor range (for example, the field of view of an externally-oriented camera of an AR apparatus) which comprise augmentation targets. According to various embodiments, the determination of whether there are augmentation targets, and in particular, augmentation targets associated with items of AR content for the present user is performed by applying one or more object recognition algorithms to image data from the externally-oriented camera. In certain embodiments, the determination of whether there are augmentation targets associated with AR content for a present user may be assisted by the augmentation target itself. For example, the augmentation target may have one or more surfaces with a coded medium (for example, a barcode or QR code), bypassing the need for recognition of the object as the augmentation target. As another example, the augmentation target may also have a beacon (for example, a flashing infrared beacon) advertising itself as an augmentation target”; also see ¶107, “the augmentation target may have one or more surfaces with a coded medium”).
(Kocharlakota: ¶46, “In certain embodiments, the associations between item(s) of AR content and objects are determined by a user (for example, as shown in FIGS. 8A through 8F of this disclosure). In various embodiments, the logic for creating associations between objects and item(s) of content reside within the AR apparatus itself. As a non-limiting example, a PCAR service running on client platform 205 could use information of the client platform itself (for example, an international mobile equipment identifier (IMEI) associated with client platform, or system information, such as an identifier of the operating system) to create associations between objects (for example, a charger for the device) and item(s) of AR content. . .”).
Regarding claim 6, depending on claim 1,
The combination of Kocharlakota and Rhoads discloses:
The method of claim 1,
Kocharlakota further teaches:
wherein obtaining the identifying information for the object comprises obtaining sensor output of at least one sensor associated with the AR device (Kocharlakota: Fig. 2,205,215,217, par. 50, "image data 215 comprises data output by from one or more externally-oriented sensors (for example, a CMOS video camera or a dynamic vision sensor (DVS), configured to produce a comparatively less data-intensive representation of a field of view by capturing changes in the intensity of received light at the pixels of the sensor"; par. 56, "signature engine 225 also receives location data 217 from one or more entities (such as a GPS sensor, or wireless communication unit)".
Regarding claim 7, depending on claim 6,
The combination of Kocharlakota and Rhoads discloses:
The method of claim 6,
Kocharlakota further teaches:
wherein the at least one sensor comprises at least one of GPS sensor, LIDAR scanner, or image sensors (Kocharlakota: par. 50, "image data 215 comprises data output by from one or more externally-oriented sensors (for example, a CMOS video camera or a dynamic vision sensor (DVS), configured to produce a comparatively less data-intensive representation of a field of view by capturing changes in the intensity of received light at the pixels of the sensor"; par. 56, "signature engine 225 also receives location data 217 from one or more entities (such as a GPS sensor, or wireless communication unit)"; par. 65, "In certain embodiments, data output from IMU 315 may be used for positioning (for example, to confirm a geospatial position of AR apparatus 300).
Regarding claim 8, depending on claim 1,
The combination of Kocharlakota and Rhoads discloses:
The method of claim 1,
Kocharlakota further teaches:
wherein the identifying information for the object comprises at least one of:
geolocation of a user associated with the AR device at a time of detecting the object; or LIDAR data indicating a specific location associated with the object. (Kocharlakota: par. 56, "signature engine 225 also receives location data 217 from one or more entities (such as a GPS sensor, or wireless communication unit)"; par. 65, "In certain embodiments, data output from IMU 315 may be used for positioning (for example, to confirm a geospatial position of AR apparatus 300), or to obtain image stabilization data (for example, data indicating the direction and periodicity of a camera shake) to facilitate object recognition”; par. 92, “In certain embodiments, the AR apparatus's identification of objects within scene 800 can be assisted by data provided by the PCAR framework and/or the objects themselves. In one non-limiting example, the PCAR framework can provide an AR apparatus with additional object information (for example, an identification of objects near the current location of the AR apparatus”).
NOTE 8A: Kocharlakota teaches a stereoscopic of cameras to generate image data comprising depth estimation (similar to LIDAR data) indicating a specific location associated with the object.
Regarding claim 9, depending on claim 1,
The combination of Kocharlakota and Rhoads discloses:
The method of claim 1,
Kocharlakota further teaches:
further comprising: obtaining a user identifier associated with the AR device (“Kocharlakota: par. 45, “According to various embodiments, PCAR database 203 comprises, at a minimum, a repository of information associating items of augmented reality (AR) content with identifiers of objects. According to some embodiments, PCAR database 203 also includes data corresponding to a schema comprising object signatures (for example, identifiers of a particular object—for example, a chair at a known location), a user identifier (for example, an ID of a user who created an association between item(s) of AR content and an object, visibility data (for example, data specifying the permissions of other user to view the AR content whose association was created by a user identified in the user identifier field, object location, and the expiration time of the association between the item of AR content and the object”);
and verifying that a user associated with the user identifier is permitted to access AR content associated with the identified object instance (Kocharlakota: ¶45, “. . . visibility data (for example, data specifying the permissions of other user to view the AR content. . .)
wherein the AR content presented via the AR device is specific to the user associated with the user identifier (Kocharlakota: par. 80 , “In certain embodiments according to this disclosure, schema 600 includes a visibility field 620, which is an attribute of the association between an object (as identified by its object signature) and one or more items of augmentation data. As shown in the non-limiting example of FIG. 6, values in visibility field 620 correspond to identifiers of users who can access the augmentation data associated with a particular object signature. For example, visibility value 619 specifies that “Martha” is the user with permission to see item(s) of AR content based on augmentation data 617, in response to an AR apparatus associated with Martha recognizing, based on obtained image data, an object having object signature 607”).
Regarding claim 10, depending on claim 1,
The combination of Kocharlakota and Rhoads discloses:
The method of claim 1,
Kocharlakota further teaches:
wherein the stored identifying information of the known instance of the object comprise at least one of:
geolocation of previous users of an object associated with the first object record (Kocharlakota: Fig. 6, par. 106, "According to various embodiments, at operation 1015, the PCAR framework performs an initial determination of whether there are any augmentations associated with the user device at a current device location (as determined from location data received in operation 1010). According to certain embodiments, the determination performed at operation 1015 comprises sending a query to a host platform (for example host platform 201 in FIG. 2) which maintains an index or other data structure associating locations, user devices and items of AR content").;
NOTE 10: The host platform maintains an index or other data structure associating locations (location data), user devices and items of AR content, suggesting previous data is maintained including location data if there are augmentation history associated with the user device at a current device location.
identifiers of objects located in a vicinity of the object associated with the first object record (Kocharlakota: par. 55, “object recognition engine 227 identifies one or more objects within the field of view of an externally facing sensor (such as an CMOS camera or DVS sensor) and provides a descriptor or other data associated with the identification of the first object to augmentation engine 229 or signature engine 225” par. 56, “In the non-limiting example of FIG. 2, signature engine 225 periodically performs comparisons of generated signatures against signatures maintained in a PCAR database 203 maintained on host platform 201. According to various embodiments, when a match between a signature at client platform 205 and host platform 201 is determined, signature engine 225 receives augmentation data. According to various embodiments, augmentation data encompasses, at a minimum, data associated with an AR display, the AR display being associated with the object for which a signature match was determined”);
or an indication of an indoor location associated with the object associated with the first object record (Kocharlakota: Fig. 6, par. 81, "For example, object location value 627 indicates that the object associated with object signature 607 is located indoors at a specific set of GPS coordinates. In this way, a PCAR database or a PCAR framework utilize object location 627 in determining whether to present an item of AR content based on augmentation data 617 in response to recognizing an object having object signature 607").
Regarding claim 11, depending on claim 1,
The combination of Kocharlakota and Rhoads discloses:
The method of claim 1, wherein the AR content comprises graphical representation of supplementary information associated with the object (Kocharlakota: Fig. 5, 9C, 12, par. 123, "at operation 1230, responsive to determining that the object identified at operation 1215 comprises an augmentation target, an item of AR content associated with the augmentation target is displayed on an internally-facing display in the AR field of view of the AR apparatus. For example, the item of AR content may be a frame or overlay which can be selected (for example, first highlighting frame 810a in FIG. 8B). In certain embodiments, the item of AR content may be a pre-associated item of AR content").
Regarding claim 12,
The combination of Kocharlakota and Rhoads discloses:
a computing system (Kocharlakota: Fig 1, par. 8, “an augmented reality apparatus”), comprising: a processor (Kocharlakota: Fig 1, 140, par. 34 – 36, “The main processor 140”);
a memory coupled to the processor, (Kocharlakota: Fig 1, 160, par. 38, “The memory 160 is coupled to the main processor 140”)
the memory storing computer-executable instructions that (Kocharlakota: ¶8, “. . . a memory containing instructions, which when executed by the processor, cause the processor to”), when executed by the processor, configure the processor to do the corresponding method of using same as claimed in amended claim 1. Therefore, amended apparatus claim 12 corresponds to amended method claim 1, and is rejected for the same reasons of obviousness as used above.
Regarding claims 13 - 19,
apparatus claims 13 - 19 are drawn to the apparatus corresponding to the methods of using same as claimed in method claims 2 – 8 respectively. Therefore, apparatus claims 13 - 19 correspond to method claims 2 – 8 respectively, and are rejected for the same reasons of obviousness as used above.
Regarding claim 20,
The combination of Kocharlakota and Rhoads discloses:
A non-transitory processor-readable medium storing processor-executable instructions (Kocharlakota: par. 9, “a non-transitory computer-readable medium includes program code”).
CRM claim 20 is drawn to a non-transitory processor-readable storage medium including instructions for causing a processor to execute the amended method of claim 1. Therefore, claim 20 corresponds to amended method claim 1, and is rejected for the same reasons of obviousness as used above.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PATRICK GALERA whose telephone number is (571)272-5070. The examiner can normally be reached Mon-Fri 0800-1700 ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Poon can be reached at 571-270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PATRICK P GALERA/Examiner, Art Unit 2617 /KING Y POON/Supervisory Patent Examiner, Art Unit 2617
