DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendments / Arguments
Applicant’s arguments filed on February 2, 2026 with respect to claims 1, 12, 20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Kocharlakota et al. (US 2020/0005542 A1, hereinafter “Kocharlakota”) in view of Rus et al. (US 2021/0133480 A1, hereinafter “Rus”).
Regarding claim 1,
Kocharlakota discloses:
A computer-implemented method, comprising:
obtaining identifying information for an object, detected in a field of view of an AR device (Kocharlakota: par. 7, ". . .a method for providing a personalized augmented reality (AR) display includes obtaining, at an augmented reality apparatus, image data of a camera field of view, . . . , identifying a first object in the camera field of view based on the image data)
and the identifying information comprising implicit signals that represent contextual data associated with the object (NOTE 1A: The implicit signals that represent contextual data associated with the objects are the image data obtained at an AR apparatus such as GPS data (¶105), binary descriptors or other data associated with the identification of an object (¶55), location data assigning a specific geospatial coordinate (¶51), object signatures (¶45, 80), user identifier (¶45), visibility data (¶45), features in the image data (¶55), image patches associated with points in the feature space to recognize objects within the space (¶86-88)),
and responsive to determining that the identified object instance (NOTE: the claimed “the object instance” is Kocharlakota’s augmentation target object, which is an object instance within the field of view) is associated with the first object record, presenting, via the AR device, AR content that is specific to the first object record (Kocharlakota: par. 8, "and responsive to determining that the first object is the augmentation target, display, on the internally-facing display, an item of AR content associated with the augmentation target in the AR field of view").
Although Kocharlakota further teaches identifying a parent object to another object in the camera field of view based on image data received from the camera, Kocharlakota fails to explicitly disclose: wherein the implicit signals include visual features of at least one of: a background scene containing the object; or nearby objects located in a vicinity of the object that are depicted in images of the object’s surroundings and determined based on image analysis of the images; identifying a unique instance of the object based on determining a match between visual features of the object’s surroundings and stored identifying information of a known instance of the object.
The analogous art Rus teaches:
wherein the implicit signals include visual features of at least one of: a background scene containing the object; or nearby objects located in a vicinity of the object that are depicted in images of the object’s surroundings and determined based on image analysis of the images (Rus: par. 30, “the object locator/descriptor generator 108 performs a sequence of two steps: first it makes a determination of what objects (i.e., classes of objects) it can locate in the image (step 322); then it processes a part of the image associated with each located object to determine the descriptor for the instance of that object (step 324) producing one descriptor for each object that is located in the previous step. . .”; par. 58, “. . . make use of 3D geometric relationships between objects. Such geometric considerations, for example, generate higher scores if the objects are in compatible geometric relationships in a reference image and an unknown image. . .”;
NOTE: In the applicant’s specification paragraph 72, “visual features” includes “nearby objects”, “. . . the AR engine may determine visual features, such as a background, nearby objects, etc., that may be used for limiting the possible matches of object instances. . .”, therefore, a nearby object is a visual feature. The implicit signal including visual feature Rus uses to determine corresponding object instances include nearby objects located in a vicinity of the object that are depicted in images of the object’s surroundings by identifying 3D geometrical relationship between objects. Since the objects having a determined 3D geometrical relationship is considered for compatibility between multiple images (analysis of reference image and unknown image), therefore, the nearby object is depicted in images of the object’s surroundings.)
identifying a unique instance of the object based on determining a match between visual features of the object’s surroundings and stored identifying information of a known instance of the object (Rus: par. 58-60, “. . . make use of 3D geometric relationships between objects. Such geometric considerations, for example, generate higher scores if the objects are in compatible geometric relationships in a reference image and an unknown image. . . reference database”; NOTE: Generating a high compatibility score constitute identifying a unique instance of the object. The object instance is unique because it is specific and compatible to a reference image with specific object relationships based on the 3D geometric relationships between objects (visual features such as nearby objects). The object instance is unique because the system matched to determine a high compatibility score specific and unique to that known record instance of the object stored in a reference database.)
It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to combine Kocharlakota and Rus to include: wherein the implicit signals include visual features of at least one of: a background scene containing the object; or nearby objects located in a vicinity of the object that are depicted in images of the object’s surroundings and determined based on image analysis of the images; identifying a unique instance of the object based on determining a match between visual features of the object’s surroundings and stored identifying information of a known instance of the object.
The reason for doing so is “to preserve distinctions between different instances of a type of object, as well as distinctions between entirely different types of objects” (Rus: Abstract).
Regarding claim 2, depending on 1,
The combination of Kocharlakota and Rus discloses:
The method of claim 1,
Kocharlakota further teaches:
wherein the identified object instance is a real object of a first class of objects and wherein determining that the identified object instance is associated with the first object record comprises distinguishing the identified object instance from at least one other object instance of the same class based on the implicit signals (Kocharlakota: par. 122, "According to some embodiments, at operation 1225, a determination of whether the object identified at operation 1215 is an augmentation target is performed. In certain embodiments, this determination is performed by comparing an object signature of the object identified at operation 1215 against a set of object signatures (for example, object signatures maintained in a PCAR database utilizing database schema 600 in FIG. 6). In various embodiments, the determination of whether an object comprises an augmentation target is based on whether the object belongs to a defined class of objects which can be augmentation targets (for example, non-transitory items, such as buildings or pieces of furniture)".
Regarding claim 3, depending on claim 1,
The combination of Kocharlakota and Rus discloses:
The method of claim 1,
Kocharlakota further teaches:
wherein obtaining the identifying information for the object comprises performing object detection based on parsing a video depicting the object using an object detection model (Kocharlakota: par. 50, "image data 215 comprises data output by from one or more externally-oriented sensors (for example, a CMOS video camera or a dynamic vision sensor (DVS), configured to produce a comparatively less data-intensive representation of a field of view by capturing changes in the intensity of received light at the pixels of the sensor").
(Kocharlakota: ¶56, “According to various embodiments, signature engine 225 receives descriptors of recognized objects in a field of view (for example, a camera field of view of an AR apparatus operating as client platform 205) from object recognition engine 227. In certain embodiments, object recognition engine 227 operates continuously, and continuously scans image data from a field of view for objects to recognize.
(Kocharlakota: ¶55, “. . .The object recognition engine 227 performs object recognition using a binary descriptor based object recognition technique, including, for example, the binary robust independent elementary features (BRIEF), binary robust invariant scalable keypoints (BRISK), fast retina keypoints (FREAK), or oriented fast and rotated BRIEF (ORB). According to various embodiments, object recognition engine 227 identifies one or more objects within the field of view of an externally facing sensor (such as an CMOS camera or DVS sensor) and provides a descriptor or other data associated with the identification of the first object. . .“
NOTE 3A: Kocharlakota discloses that the AR apparatus employs a CMOS video camera, or a DVS Dynamic Vision Sensor which are both capable of obtaining video. The object recognition engine continuously scans image data (parsing a video) from a field of view for objects to recognize.
Regarding claim 4, depending on claim 3,
The combination of Kocharlakota and Rus discloses:
The method of claim 3,
Kocharlakota further teaches:
wherein obtaining the identifying information for the object comprises performing image analysis on frames of the video (Kocharlakota: par. 50, "image data 215 comprises data output by from one or more externally-oriented sensors (for example, a CMOS video camera or a dynamic vision sensor (DVS), configured to produce a comparatively less data-intensive representation of a field of view by capturing changes in the intensity of received light at the pixels of the sensor").
Regarding claim 5, depending on claim 4,
The combination of Kocharlakota and Rus discloses:
The method of claim 4,
Kocharlakota further teaches:
wherein the identifying information for the object comprises at least one of a barcode, a QR code, an NFC tag, or a serial number (Kocharlakota: par. 107, "In certain embodiments according to the present disclosure, at operation 1020, one or more components (for example, an augmentation engine, such as augmentation engine 229 in FIG. 2 and an object recognition engine, such as object recognition engine 227 in FIG. 2) determines whether there are objects within a given sensor range (for example, the field of view of an externally-oriented camera of an AR apparatus) which comprise augmentation targets. According to various embodiments, the determination of whether there are augmentation targets, and in particular, augmentation targets associated with items of AR content for the present user is performed by applying one or more object recognition algorithms to image data from the externally-oriented camera. In certain embodiments, the determination of whether there are augmentation targets associated with AR content for a present user may be assisted by the augmentation target itself. For example, the augmentation target may have one or more surfaces with a coded medium (for example, a barcode or QR code), bypassing the need for recognition of the object as the augmentation target. As another example, the augmentation target may also have a beacon (for example, a flashing infrared beacon) advertising itself as an augmentation target”; also see ¶107, “the augmentation target may have one or more surfaces with a coded medium”).
(Kocharlakota: ¶46, “In certain embodiments, the associations between item(s) of AR content and objects are determined by a user (for example, as shown in FIGS. 8A through 8F of this disclosure). In various embodiments, the logic for creating associations between objects and item(s) of content reside within the AR apparatus itself. As a non-limiting example, a PCAR service running on client platform 205 could use information of the client platform itself (for example, an international mobile equipment identifier (IMEI) associated with client platform, or system information, such as an identifier of the operating system) to create associations between objects (for example, a charger for the device) and item(s) of AR content. . .”).
Regarding claim 6, depending on claim 1,
The combination of Kocharlakota and Rus discloses:
The method of claim 1,
Kocharlakota further teaches:
wherein obtaining the identifying information for the object comprises obtaining sensor output of at least one sensor associated with the AR device (Kocharlakota: Fig. 2,205,215,217, par. 50, "image data 215 comprises data output by from one or more externally-oriented sensors (for example, a CMOS video camera or a dynamic vision sensor (DVS), configured to produce a comparatively less data-intensive representation of a field of view by capturing changes in the intensity of received light at the pixels of the sensor"; par. 56, "signature engine 225 also receives location data 217 from one or more entities (such as a GPS sensor, or wireless communication unit)".
Regarding claim 7, depending on claim 6,
The combination of Kocharlakota and Rus discloses:
The method of claim 6,
Kocharlakota further teaches:
wherein the at least one sensor comprises at least one of GPS sensor, LIDAR scanner, or image sensors (Kocharlakota: par. 50, "image data 215 comprises data output by from one or more externally-oriented sensors (for example, a CMOS video camera or a dynamic vision sensor (DVS), configured to produce a comparatively less data-intensive representation of a field of view by capturing changes in the intensity of received light at the pixels of the sensor"; par. 56, "signature engine 225 also receives location data 217 from one or more entities (such as a GPS sensor, or wireless communication unit)"; par. 65, "In certain embodiments, data output from IMU 315 may be used for positioning (for example, to confirm a geospatial position of AR apparatus 300).
Regarding claim 8, depending on claim 1,
The combination of Kocharlakota and Rus discloses:
The method of claim 1,
Kocharlakota further teaches:
wherein the identifying information for the object comprises at least one of:
geolocation of a user associated with the AR device at a time of detecting the object; or LIDAR data indicating a specific location associated with the object. (Kocharlakota: par. 56, "signature engine 225 also receives location data 217 from one or more entities (such as a GPS sensor, or wireless communication unit)"; par. 65, "In certain embodiments, data output from IMU 315 may be used for positioning (for example, to confirm a geospatial position of AR apparatus 300), or to obtain image stabilization data (for example, data indicating the direction and periodicity of a camera shake) to facilitate object recognition”; par. 92, “In certain embodiments, the AR apparatus's identification of objects within scene 800 can be assisted by data provided by the PCAR framework and/or the objects themselves. In one non-limiting example, the PCAR framework can provide an AR apparatus with additional object information (for example, an identification of objects near the current location of the AR apparatus”).
NOTE 8A: Kocharlakota teaches a stereoscopic of cameras to generate image data comprising depth estimation (similar to LIDAR data) indicating a specific location associated with the object.
Regarding claim 9, depending on claim 1,
The combination of Kocharlakota and Rus discloses:
The method of claim 1,
Kocharlakota further teaches:
further comprising: obtaining a user identifier associated with the AR device (“Kocharlakota: par. 45, “According to various embodiments, PCAR database 203 comprises, at a minimum, a repository of information associating items of augmented reality (AR) content with identifiers of objects. According to some embodiments, PCAR database 203 also includes data corresponding to a schema comprising object signatures (for example, identifiers of a particular object—for example, a chair at a known location), a user identifier (for example, an ID of a user who created an association between item(s) of AR content and an object, visibility data (for example, data specifying the permissions of other user to view the AR content whose association was created by a user identified in the user identifier field, object location, and the expiration time of the association between the item of AR content and the object”);
and verifying that a user associated with the user identifier is permitted to access AR content associated with the identified object instance (Kocharlakota: ¶45, “. . . visibility data (for example, data specifying the permissions of other user to view the AR content. . .)
wherein the AR content presented via the AR device is specific to the user associated with the user identifier (Kocharlakota: par. 80 , “In certain embodiments according to this disclosure, schema 600 includes a visibility field 620, which is an attribute of the association between an object (as identified by its object signature) and one or more items of augmentation data. As shown in the non-limiting example of FIG. 6, values in visibility field 620 correspond to identifiers of users who can access the augmentation data associated with a particular object signature. For example, visibility value 619 specifies that “Martha” is the user with permission to see item(s) of AR content based on augmentation data 617, in response to an AR apparatus associated with Martha recognizing, based on obtained image data, an object having object signature 607”).
Regarding claim 10, depending on claim 1,
The combination of Kocharlakota and Rus discloses:
The method of claim 1,
Kocharlakota further teaches:
wherein the stored identifying information of the known instance of the object comprise at least one of:
geolocation of previous users of an object associated with the first object record (Kocharlakota: Fig. 6, par. 106, "According to various embodiments, at operation 1015, the PCAR framework performs an initial determination of whether there are any augmentations associated with the user device at a current device location (as determined from location data received in operation 1010). According to certain embodiments, the determination performed at operation 1015 comprises sending a query to a host platform (for example host platform 201 in FIG. 2) which maintains an index or other data structure associating locations, user devices and items of AR content").;
NOTE 10: The host platform maintains an index or other data structure associating locations (location data), user devices and items of AR content, suggesting previous data is maintained including location data if there are augmentation history associated with the user device at a current device location.
identifiers of objects located in a vicinity of the object associated with the first object record (Kocharlakota: par. 55, “object recognition engine 227 identifies one or more objects within the field of view of an externally facing sensor (such as an CMOS camera or DVS sensor) and provides a descriptor or other data associated with the identification of the first object to augmentation engine 229 or signature engine 225” par. 56, “In the non-limiting example of FIG. 2, signature engine 225 periodically performs comparisons of generated signatures against signatures maintained in a PCAR database 203 maintained on host platform 201. According to various embodiments, when a match between a signature at client platform 205 and host platform 201 is determined, signature engine 225 receives augmentation data. According to various embodiments, augmentation data encompasses, at a minimum, data associated with an AR display, the AR display being associated with the object for which a signature match was determined”);
or an indication of an indoor location associated with the object associated with the first object record (Kocharlakota: Fig. 6, par. 81, "For example, object location value 627 indicates that the object associated with object signature 607 is located indoors at a specific set of GPS coordinates. In this way, a PCAR database or a PCAR framework utilize object location 627 in determining whether to present an item of AR content based on augmentation data 617 in response to recognizing an object having object signature 607").
Regarding claim 11, depending on claim 1,
The combination of Kocharlakota and Rus discloses:
The method of claim 1, wherein the AR content comprises graphical representation of supplementary information associated with the object (Kocharlakota: Fig. 5, 9C, 12, par. 123, "at operation 1230, responsive to determining that the object identified at operation 1215 comprises an augmentation target, an item of AR content associated with the augmentation target is displayed on an internally-facing display in the AR field of view of the AR apparatus. For example, the item of AR content may be a frame or overlay which can be selected (for example, first highlighting frame 810a in FIG. 8B). In certain embodiments, the item of AR content may be a pre-associated item of AR content").
Regarding claim 12,
The combination of Kocharlakota and Rus discloses:
a computing system (Kocharlakota: Fig 1, par. 8, “an augmented reality apparatus”), comprising: a processor (Kocharlakota: Fig 1, 140, par. 34 – 36, “The main processor 140”);
a memory coupled to the processor, (Kocharlakota: Fig 1, 160, par. 38, “The memory 160 is coupled to the main processor 140”)
the memory storing computer-executable instructions that (Kocharlakota: ¶8, “. . . a memory containing instructions, which when executed by the processor, cause the processor to”), when executed by the processor, configure the processor to do the corresponding method of using same as claimed in amended claim 1. Therefore, amended apparatus claim 12 corresponds to amended method claim 1, and is rejected for the same reasons of obviousness as used above.
Regarding claims 13 - 19,
apparatus claims 13 - 19 are drawn to the apparatus corresponding to the methods of using same as claimed in method claims 2 – 8 respectively. Therefore, apparatus claims 13 - 19 correspond to method claims 2 – 8 respectively, and are rejected for the same reasons of obviousness as used above.
Regarding claim 20,
The combination of Kocharlakota and Rus discloses:
A non-transitory processor-readable medium storing processor-executable instructions (Kocharlakota: par. 9, “a non-transitory computer-readable medium includes program code”).
CRM claim 20 is drawn to a non-transitory processor-readable storage medium including instructions for causing a processor to execute the amended method of claim 1. Therefore, claim 20 corresponds to amended method claim 1, and is rejected for the same reasons of obviousness as used above.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PATRICK GALERA whose telephone number is (571)272-5070. The examiner can normally be reached Mon-Fri 0800-1700 ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Poon can be reached at 571-270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PATRICK P GALERA/Examiner, Art Unit 2617 /KING Y POON/Supervisory Patent Examiner, Art Unit 2617
Prosecution Insights