DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments with respect to claims 1-22 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, 4-10, 13, 14, 16, 18-22 are rejected under 35 U.S.C. 103 as being unpatentable over by Chojnacka et al. (US 20200342668) in view of Seroussi et al. (US 20250037395).
Claim 1. Chojnacka et al. (Chojnacka) teaches a computer-implemented method for detecting physical objects in a physical environment based on analysis of image frames captured by an image sensor of a computing device and recommending items, in particular, germane, pertinent, contextual items, for virtual placement, the method comprising:
receiving, from a camera of an augmented reality (AR) device, a stream of images of a scene including the object [0019];
identifying the object in the stream of images [0020];
retrieving a set of recommendations for the object from an object recommendation data store [0044];
extracting contextual information from the stream of images [0047];
selecting a recommendation from the set of recommendations based at least in part on the contextual information [0045]; and
sending to the AR device the recommendation for presenting in AR on a display of the AR device [0024].
Chojnacka does not teach that the recommendation includes information about the identified object, the information being associated with at least one of care, appropriate use, or a warning instruction for the object, and wherein the information is directed at the user as an end-user of the object.
Seroussi et al. (Seroussi) teaches a computer-implemented method of providing an AR technology-based decoration system. The image input is processed to identify 3D features which define the scene boundary (floor, wall, obstacles, objects etc.) and objects present therein to classify the scene type, style, theme (indoor, outdoor, room, bedroom, nursery, living room, outdoor veranda etc., the materials, color scheme and design style of the real world environment, e.g., modern, classic, industrial, gothic, etc.) [0089].
Upon selection of the products for one or more of the designable areas, as seen in FIG. 5, the user is directed to an online shopping cart interface where they can proceed to purchasing one or more of the recommended products suitable to the user preferences. The listed items in the cart are optionally further provided with a “details” button which upon activation provides a pop-up screen with e.g., details about each item, instructions for care, placement, reviews, grading by other users, etc. [0122].
It would have been obvious to one of ordinary skill before the effective filing date of the claimed invention to modify Chojnacka to include that the recommendation includes information about the identified object, the information being associated with at least one of care, appropriate use, or a warning instruction for the object, and wherein the information is directed at the user as an end-user of the object, as disclosed in Seroussi, because it would advantageously allow the user to visualize the real world environment with the proposed or chosen products prior to actual purchasing or installation, as taught by Seroussi [0123].
Claim 2. Chojnacka teaches said method, wherein the contextual information can include one or more of: (1) a setting; (2) a geographic location; (3) a lifecycle period of the object; (4) an environment type; (5) object defect; (6) object use or misuse; (7) object safety issue; or (8) any combination of (1), (2), (3), (4), (5), (6), and (7) [0044].
Claim 4. Chojnacka teaches said method, wherein the AR device determines a surface of the object to present the recommendation in AR, and wherein the recommendation is presented attached to the determined surface [0028].
Claim 5. Chojnacka teaches said method, wherein the surface is determined by: determining a pose of the object and non-occluded surfaces of the object based on the stream of images of the scene; and determining the surface based on the determined pose of the object and the determined non-occluded surfaces of the object [0028].
Claim 6. Chojnacka teaches said method, wherein the surface is further determined by: determining a distance to the object and lighting conditions of the scene, wherein determining the surface is based further on the distance to the object and the lighting conditions of the scene [0034].
Claim 7. Chojnacka teaches said method, wherein selecting the recommendation comprises: ranking the set of recommendations, wherein the ranking is based at least in part on the contextual information and the selected recommendation is the highest ranked recommendation [0021].
Claim 8. Chojnacka teaches said method, wherein ranking the set of recommendations comprises: processing the set of recommendations with a machine learning model to score the set of recommendations based at least in part on the contextual information [0021].
Claim 9. Chojnacka teaches said method, wherein the recommendation selected includes an object warning when the extracted contextual information includes an identified type of room in the scene that is not the type of room recommended for the object [0040].
Claim 10. Chojnacka teaches said method, the method further comprising: identifying at least two surfaces of the object; selecting a surface recommendation for each of the at least two surfaces; and sending the surface recommendations to the AR device, wherein each of the surface recommendations is presented in AR attached to a corresponding surface of the at least two surfaces [0028].
Claim 13. Chojnacka teaches said method further comprising: identifying surfaces of the object; comparing the surfaces on the object to models stored in a database to identify a defect of a surface of the surfaces; and sending a message to the AR device to present a recommendation related to the defect in AR attached to the surface [0028].
Claim 14. Chojnacka teaches said method, wherein the recommendation can include: (1) object materials; (2) object dimensions; (3) object care instructions; (4) disposal information; (5) sustainability information; (6) assembly information; (7) installation error information; (8) consumable/replacement information; (9) object intended use information; or (10) any combination of (1), (2), (3), (4), (5), (6), (7), (8), and (9) [0028].
Claim 16. Chojnacka teaches said method, wherein the recommendation is presented in AR with a 3D graphic [0019].
Claim 18. Chojnacka teaches an augmented reality (AR) device comprising:
a camera; a display; at least one processor; and a memory device, the memory device storing instructions which when executed by the at least one processor cause the AR device [0025] to:
capture with the camera a stream of images of a scene [0025];
provide the stream of images to a server operating a recommendation engine, wherein the recommendation engine processes the stream of images to identify an object and extract contextual information, retrieves a set of recommendations for the object [0019]-[0021], and
selects a recommendation from the set of recommendations based at least in part on the contextual information [0047];
receive the recommendation from the server [0021]; and
present the recommendation in AR on the display [0024].
Chojnacka does not teach that the recommendation includes information about the identified object, the information being associated with at least one of care, appropriate use, or a warning instruction for the object, and wherein the information is directed at the user as an end-user of the object.
Seroussi et al. (Seroussi) teaches a computer-implemented method of providing an AR technology-based decoration system. The image input is processed to identify 3D features which define the scene boundary (floor, wall, obstacles, objects etc.) and objects present therein to classify the scene type, style, theme (indoor, outdoor, room, bedroom, nursery, living room, outdoor veranda etc., the materials, color scheme and design style of the real world environment, e.g., modern, classic, industrial, gothic, etc.) [0089].
Upon selection of the products for one or more of the designable areas, as seen in FIG. 5, the user is directed to an online shopping cart interface where they can proceed to purchasing one or more of the recommended products suitable to the user preferences. The listed items in the cart are optionally further provided with a “details” button which upon activation provides a pop-up screen with e.g., details about each item, instructions for care, placement, reviews, grading by other users, etc. [0122].
It would have been obvious to one of ordinary skill before the effective filing date of the claimed invention to modify Chojnacka to include that the recommendation includes information about the identified object, the information being associated with at least one of care, appropriate use, or a warning instruction for the object, and wherein the information is directed at the user as an end-user of the object, as disclosed in Seroussi, because it would advantageously allow the user to visualize the real world environment with the proposed or chosen products prior to actual purchasing or installation, as taught by Seroussi [0123].
Claim 19. Chojnacka teaches said device further comprising: a global positioning system (GPS) receiver configured to receive global position system data to determine a location of the AR device, wherein at the contextual information is further based on the location of the AR device [0044], [0060].
Claim 20. Chojnacka teaches said device, wherein the AR device is one of smart glasses, a smart phone, or a computing tablet [0049].
Claim 21. Chojnacka teaches a virtual reality (VR) device comprising:
a display; at least one processor; and a memory device, the memory device storing instructions which when executed by the at least one processor cause the VR device [0025] to:
present a virtual scene with an object with an object ID;
send the object ID to a server operating a recommendation engine, wherein the recommendation engine extracts contextual information, retrieves a set of recommendations for the object via the object ID, and selects a recommendation from the set of recommendations based at least in part on the contextual information [0019]-[0021];
receive the recommendation; and present the recommendation in VR on the display [0024].
Chojnacka does not teach that the recommendation includes information about the identified object, the information being associated with at least one of care, appropriate use, or a warning instruction for the object, and wherein the information is directed at the user as an end-user of the object.
Seroussi et al. (Seroussi) teaches a computer-implemented method of providing an AR technology-based decoration system. The image input is processed to identify 3D features which define the scene boundary (floor, wall, obstacles, objects etc.) and objects present therein to classify the scene type, style, theme (indoor, outdoor, room, bedroom, nursery, living room, outdoor veranda etc., the materials, color scheme and design style of the real world environment, e.g., modern, classic, industrial, gothic, etc.) [0089].
Upon selection of the products for one or more of the designable areas, as seen in FIG. 5, the user is directed to an online shopping cart interface where they can proceed to purchasing one or more of the recommended products suitable to the user preferences. The listed items in the cart are optionally further provided with a “details” button which upon activation provides a pop-up screen with e.g., details about each item, instructions for care, placement, reviews, grading by other users, etc. [0122].
It would have been obvious to one of ordinary skill before the effective filing date of the claimed invention to modify Chojnacka to include that the recommendation includes information about the identified object, the information being associated with at least one of care, appropriate use, or a warning instruction for the object, and wherein the information is directed at the user as an end-user of the object, as disclosed in Seroussi, because it would advantageously allow the user to visualize the real world environment with the proposed or chosen products prior to actual purchasing or installation, as taught by Seroussi [0123].
Claim 22. Same reasoning applied to claim 21.
Claims 3, 12, 15 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Chojnacka and Greenberger, in view of Bronicki et al. (US 20210374836).
Claim 3. The combination of Schojnacka and Greenberger teaches all the limitations of claim 3 except wherein the contextual information includes an object lifecycle period, the object lifecycle period including one of: (1) the object on display for sale; (2) the object during installation; (3) the object during use; and (4) the object at end of life.
Bronicki et al. (Bronicki) teaches a computer-implemented method for displaying navigation assistance augmented-reality-view user interface wherein when the image in augmented reality display area changes (for example, due to movement of a device displaying user interface, due to movement of a camera capturing the image, etc.), the position of the overlays may change according to the changes in the image. For example, the location of destination overlay indicator may change to a location adjacent to the location of a selected item (such as a product, a shelf, a label, etc.) in the image, the location of route overlay indicator 2508 may change to a location of the route [0377].
It would have been obvious to one of ordinary skill in the art before the inventive filing date of the claimed invention to include wherein the contextual information includes an object lifecycle period, the object lifecycle period including one of: (1) the object on display for sale; (2) the object during installation; (3) the object during use; and (4) the object at end of life as taught by Bronicki in the system of Schojnacka and Greenberger, since the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable. KSR, 127 S.Ct. at 1740, 82 USPQ2d at 1396.
Claim 12. The combination of Schojnacka and Greenberger teaches all the limitations of claim 12 except identifying an installation error based at least on the stream of images, the object, and the contextual information.
Bronicik et al. teaches a computer-implemented method for displaying navigation assistance augmented-reality-view user interface wherein if it is determined that the product type of the newly added items does not match the product type associated with the peg-hook, processing device may transmit a notification, via communications network, to server and/or user device. The notification may alert a user (e.g., a store associate of retail store) about the mismatch of product types and/or prompt the user to reorganize the items to ensure that items of the correct product type are hanging on a peg-hook [0312].
It would have been obvious to one of ordinary skill in the art before the inventive filing date of the claimed invention to include identifying an installation error based at least on the stream of images, the object, and the contextual information, as taught by Bronicki in the system of Schojnacka and Greenberger, since the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable. KSR, 127 S.Ct. at 1740, 82 USPQ2d at 1396.
Claim 15. The combination of Schojnacka and Greenberger teaches all the limitations of claim 15 except wherein the object is a product sold by a retailer.
Bronicik teaches a computer-implemented method for displaying navigation assistance augmented-reality-view user interface destination overlay indicator and/or route overlay indicator may be presented using an augmented reality system. For example, destination overlay indicator may be presented in a location adjacent to or over the location of a selected item (such as a product, a shelf, a label, etc.), route overlay indicator may be presented in a location indicative of the route [0379].
It would have been obvious to one of ordinary skill in the art before the inventive filing date of the claimed invention to include wherein the object is a product sold by a retailer, as taught by Bronicki in the system of Schojnacka and Greenberger, since the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable. KSR, 127 S.Ct. at 1740, 82 USPQ2d at 1396.
Claims 11 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Chojnacka and Greenberger, in view of Official Notice.
Claim 11. The combination of Schojnacka and Greenberger teaches all the limitations of claim 11 except wherein each of the at least two surfaces are made of a different material and the recommendation selected for each of the at least two surfaces includes care information for each of the different materials. Official Notice is taken that it is old and well known to provide recommendations based on customer reviews. It would have been obvious to one of ordinary skill in the art before the inventive filing date of the claimed invention to include wherein each of the at least two surfaces are made of a different material and the recommendation selected for each of the at least two surfaces includes care information for each of the different materials in the system of Schojnacka and Greenberger, since the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable. KSR, 127 S.Ct. at 1740, 82 USPQ2d at 1396.
Claim 17. The combination of Schojnacka and Greenberger teaches all the limitations of claim 17 except that the recommendation is selected further based on at least one of customer reviews for the object and/or object return data. Official Notice is taken that it is old and well known to provide recommendations based on customer reviews. It would have been obvious to one of ordinary skill in the art before the inventive filing date of the claimed invention to include that the recommendation is selected further based on at least one of customer reviews for the object and/or object return data in the system of Schojnacka and Greenberger, since the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable. KSR, 127 S.Ct. at 1740, 82 USPQ2d at 1396.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MILA AIRAPETIAN whose telephone number is (571)272-3202. The examiner can normally be reached Monday-Friday 8:30 am-6:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jeffrey A. Smith can be reached on (571) 272-6763. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MILA AIRAPETIAN/Primary Examiner, Art Unit 3688
Prosecution Insights