DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Drawings
The drawings are objected to because fig. 5 is missing reference elements that would explain exactly what is what (and a sufficient description thereof) in that illustration. Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Rejections - 35 USC § 101
Note: claim 15 is not rejected under 35 U.S.C. 101 despite reciting a “computer program product” because paragraph 15 of applicant’s specification recites “A computer program product embodiment ("CPP embodiment" or “CPP”) is a term used in the present disclosure to describe any set of one, or more, storage media… computer readable storage medium, as that term is used in the present disclosure, is not to be construed as storage in the form of transitory signals per se”. Therefore, the computer program product of claim 15 is being interpreted as a non-transitory computer readable storage medium accordingly.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 5, 7-8, 12, 14-15 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Cannefax et al. (U.S. Patent Application Publication No. 2021/0375065), hereinafter referenced as Cannefax, in view of Chen et al. (U.S. Patent Application Publication No. 2014/0192159), hereinafter referenced as Chen.
Regarding claim 1, Cannefax teaches A processor-implemented method for mixed reality, the method comprising: identifying one or more physical visuals in a physical environment (fig. 2, step 202 teaches “segmenting the environment data into a plurality of segments, wherein each segment is referred to herein as a context, and associating an ID with each context”); contexts here show physical visuals in physical environment and ID shows the identifying of such; comparing the identified one or more physical visuals in the physical environment to one or more digital visuals in a mixed reality simulated environment (paragraph 23 teaches “the term ‘Digital Island’ used throughout the description herein should be understood to include a digital object that occupies a volume of space in which a coordinate system is stable and which can include additional digital objects and which can be placed and manipulated within an MR environment. In accordance with an embodiment, a digital island may be a digital object” and paragraph 42 teaches “evaluating one or more conditions of the plurality of digital islands (e.g., evaluating all conditions of the plurality of digital islands) against values of traits determined from each of the plurality of contexts (e.g., as determined in operation 204)”); evaluating conditions of digital islands (which are digital visuals in mixed reality (MR) environment) against contexts (identified physical visuals as aforementioned), shows a comparison of the two; determining one or more object matches between the identified one or more physical visuals in the physical environment and the one or more digital visuals in the mixed reality simulated environment (paragraph 42 teaches “the MR context solver module 124 may perform the following operations too determine the optimal match: evaluating one or more conditions of the plurality of digital islands (e.g., evaluating all conditions of the plurality of digital islands) against values of traits determined from each of the plurality of contexts (e.g., as determined in operation 204) (the evaluating may include generating a score (e.g., a number from 0 to 1) representing a measure of quality of fit of a determined value of a trait and a condition of the one or more conditions)… and evaluating relations against possible pairings produced by the matches for each context”); this shows determining object matches between the identified physical visuals (contexts) and digital visuals/islands in the mixed reality (MR) environment;
However, Cannefax fails to teach and removing one or more corresponding digital visuals of the one or more determined object matches from the mixed reality simulated environment.
However, Chen teaches and removing one or more corresponding digital visuals of the one or more determined object matches from the mixed reality simulated environment (Chen, paragraph 11 teaches “FIG. 4E illustrates addition, deletion, or modification of some matched pairs of features from a virtual image and a real image according to various embodiments of the invention” and paragraph 32 teaches “deleted or modified from, the corresponding virtual or real images.”); deletion/removing of matched pairs of features from virtual image shows removing one or more digital/virtual visuals/features of determined object matches (matched pairs) and when viewed in combination this would be the object matches from the mixed reality simulated environment as aforementioned in Cannefax. Chen is considered to be analogous art because it is reasonably pertinent to the problem faced by the inventor of digital object removal according to object matches in virtual and physical space. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Cannefax's invention with the virtual object deletion techniques of Chen to ensure providing better situation awareness of the coverage area of the real camera in a 3-D environment (Chen, paragraph 22). This would ensure a user is more aware of their surroundings and lead to a more realistic experience when using MR.
Regarding claim 5, the combination of Cannefax and Chen teaches further comprising: displaying the mixed reality simulated environment and the one or more digital visuals in the mixed reality simulated environment (Cannefax, paragraph 27 teaches “configured to display and manipulate digital content within an MR environment via a MR-capable device... a user (e.g., a wearer of an HMD, or someone holding a smartphone, tablet, or other MR-capable device) experiences the MR environment as presented by the MR context solver system via an MR device”); this shows to display MR environment (mixed reality simulated environment) and digital content (includes digital visuals/islands) within such.
Regarding claim 7, the combination of Cannefax and Chen teaches wherein the determining of the one or more object matches between the identified one or more physical visuals in the physical environment and the one or more digital visuals in the mixed reality simulated environment comprises identifying similarities between locations and dimensions of the one or more identified physical visuals and the one or more digital visuals, in the mixed reality simulated environment (Cannefax, paragraph 21 teaches “ a condition can include… location criteria (e.g., including location of the digital object with respect to other objects in the MR environment….orientation criteria (e.g., an angle criteria between the digital object and other objects or surfaces), alignment criteria (e.g., including horizontal and vertical alignment of a part of the digital object with one or more objects (or parts of the objects) in the MR environment, alignment between the digital object and other objects), object dimension criteria (e.g., including criteria regarding the digital object size)… a set of conditions associated with a digital object specify where and how the digital object can be spawned (e.g., placed) within an MR environment when using the method 200 described below for matching a digital object with a segment (e.g., segment the MR environment, wherein each segment may be referred to herein as a context as described below) within the environment” and paragraph 33 teaches “a context may include one or more detected items in the environment such as surfaces and objects, as well as properties of the detected items (e.g., size, orientation,”); this shows the aforementioned determination of object matches comprises using conditions which include location criteria (to compare and identify similarities in location of digital island/visuals and physical visuals (other objects and contexts)) as well as size in the two type of objects/visuals which would correspond to identifying similarities in dimensions since comparing size would mean comparing dimensions as size is overall extent/magnitude of object determined by dimensions.
Regarding claim 8, the system claim 8 recites similar limitations as method claim 1, and thus is rejected under similar rationale. In addition, Cannefax, fig. 1 teaches computer system 100 for mixed reality, processors 104/106, memory 110, and claim 9 teaches “non-transitory computer-readable storage medium storing a set of instructions that, when executed by one or more computer processors, cause the one or more computer processors to perform operations, the operations comprising: accessing data describing a three-dimensional environment;”; this shows a computer-readable tangible storage medium which would execute instructions stored thereof using processor via at least one memory as one of ordinary skill in the art would understand.
Regarding claim 12, the system claim 12 recites similar limitations as method claim 5, and thus is rejected under similar rationale.
Regarding claim 14, the system claim 14 recites similar limitations as method claim 7, and thus is rejected under similar rationale.
Regarding claim 15, the computer program product claim 15 recites similar limitations as method claim 1, and thus is rejected under similar rationale. In addition, Cannefax, paragraph 11 teaches “computing machine program products that comprise illustrative embodiments of the disclosure” and claim 9 teaches “non-transitory computer-readable storage medium storing a set of instructions that, when executed by one or more computer processors, cause the one or more computer processors to perform operations, the operations comprising: accessing data describing a three-dimensional environment;”; this shows the computer-readable tangible storage medium and program instructions stored on at least one of the one or more tangible storage medium (which would be comprised in computer program product as one of ordinary skill in the art would understand) being executable by processor(s).
Regarding claim 19, the computer program product claim 19 recites similar limitations as method claim 5, and thus is rejected under similar rationale.
Claim(s) 2-3, 9-10, and 16-17 is/are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Cannefax and Chen as applied to claims 1, 8 and 15 above, and further in view of Gupta et al. (U.S. Patent No. 11,776,206), hereinafter referenced as Gupta.
Regarding claim 2, the combination of Cannefax and Chen teaches wherein the identifying of the one or more physical visuals in the physical environment comprises using one or more … and a mixed reality device (Cannefax, fig. 1 shows MR device 102 in implementation of system 100 which performs the aforementioned tasks inclusive of identifying).
However, the combination of Cannefax and Chen fails to teach identifying using one or more Internet of things devices.
However, Gupta teaches identifying using one or more Internet of things devices (Gupta, col. 10, lines 50-54 teach “the positions and other relevant attributes of assets, such as state and sensor readings for IoT enabled devices as well as audio and video transmission, are monitored and recorded through persistent communication between the assets, real and virtual, and server”); this shows the aforementioned identifying of physical visuals in the physical environment would be done by using Internet of things devices. Gupta is considered to be analogous art because it is reasonably pertinent to the problem faced by the inventor of using IoT devices and digital twin in mixed reality environment alongside collecting/identifying real data while also having virtual data. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Cannefax and Chen with the IoT and digital twin in MR techniques of Gupta to improve object/machine placement, resulting in streamlined user flow within the space (Gupta, col. 13, lines 34-35). This would enhance user experience due to improved object accuracy.
Regarding claim 3, the combination of Cannefax, Chen and Gupta teaches wherein the comparing of the identified one or more physical visuals in the physical environment to the one or more digital visuals in the mixed reality simulated environment is performed using a digital twin (Gupta, col. 1, lines 23-24 teach “digital twins are virtual objects that represent a real object (real twin), and can be in the form of a digital model”, col. 52, lines 10-13 teach “FIG. 15 illustrates an example detailed block diagram of the XR method 1500 for digital twins, performed by the mapping device 108” and col. 52, lines 21-28 teach “At step 1504, the mapping device 108 generates, using i) the at least one image of the real asset, ii) the anchor points in the real world coordinates of the real 3D space, and iii) the real location of the real asset in the real world coordinates of the real 3D space: a 3D map in an XR application which includes a virtual asset of the real asset with a virtual state which matches the real state”); this shows comparing by mapping identified physical visuals and digital visuals (real asset versus virtual asset) in XR/MR performed using a digital twin. The same motivations used in claim 2 apply here in claim 3.
Regarding claim 9, the system claim 9 recites similar limitations as method claim 2, and thus is rejected under similar rationale.
Regarding claim 10, the system claim 10 recites similar limitations as method claim 3, and thus is rejected under similar rationale.
Regarding claim 16, the computer program product claim 16 recites similar limitations as method claim 2, and thus is rejected under similar rationale.
Regarding claim 17, the computer program product claim 17 recites similar limitations as method claim 3, and thus is rejected under similar rationale.
Claim(s) 4, 11, and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Cannefax and Chen as applied to claims 1, 8 and 15 above, and further in view of Bergen (U.S. Patent Application Publication No. 2020/0013211), hereinafter referenced as Bergen.
Regarding claim 4, the combination of Cannefax and Chen teaches wherein the identifying of the one or more physical visuals in the physical environment comprises using one or more machine learning models (Cannefax, paragraph 33 teaches “analysis for segmentation may be performed using…machine learning methods”); this shows machine learning would be used for the identifying of physical visuals (contexts that result from segmentation) in the physical environment.
However, the combination of Cannefax and Chen fails to teach identifying …comprises… and one or more natural language processing algorithms.
However, Bergen teaches identifying …comprises… and one or more natural language processing algorithms (Bergen, paragraph 6 teaches “program instructions are further adapted to perform natural language processing the different words in the memory to determine different physical and functional attributes of the real world object”); this shows, when viewed in combination, the identifying of real world (physical visuals) object would be done using natural language processing (and algorithms thereof as one of ordinary skill in the art would understand). Bergen is considered to be analogous art because it is reasonably pertinent to the problem faced by the inventor of natural language processing algorithms alongside physical/real objects/visuals. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Cannefax and Chen with the natural language processing techniques of Bergen so that the virtual artifact operates in the virtual environment similar to the real world object would operate in the real world (Bergen, paragraph 15). Improved realism would result from this (results in better user experience) and this is tied to the natural language processing keywords associated with real objects.
Regarding claim 11, the system claim 11 recites similar limitations as method claim 4, and thus is rejected under similar rationale.
Regarding claim 18, the computer program product claim 18 recites similar limitations as method claim 4, and thus is rejected under similar rationale.
Claim(s) 6, 13 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Cannefax and Chen as applied to claims 1, 8 and 15 above, and further in view of Nuernberger et al. (U.S. Patent Application Publication No. 2017/0287218), hereinafter referenced as Nu.
Regarding claim 6, the combination of Cannefax and Chen fails to teach further comprising: guiding directionality of the one or more identified physical visuals in the mixed reality simulated environment.
However, Nu teaches further comprising: guiding directionality of the one or more identified physical visuals in the mixed reality simulated environment (Nu, paragraph 122 teaches “display via the display device an indication of the corresponding candidate anchor feature at its corresponding physical feature within the real world physical environment..output a directional notification that indicates” paragraph 91 teaches “broadcast a directional sound “Clink” that the user 200 perceives to originate from behind and above him at approximately the location of the virtual box” and paragraph 92 teaches “controller may generate a sound or vibration to signal that the virtual anchor feature of the virtual object has aligned with the corresponding candidate anchor feature of the corresponding physical feature.”); indication of corresponding physical feature, alongside direction notification to indicate and sound from a specific location of physical feature, all show guiding directionality of the physical features (physical visuals in MR simulated environment from the above combination). Nu is considered to be analogous art because it is reasonably pertinent to the problem faced by the inventor of directionality being generated for physical visuals/features/objects. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Cannefax and Chen with the directionality indication techniques of Nu so users may utilize the real world physical objects, surfaces and geometries around them to easily position virtual objects in their environment (Nu, paragraph 19). This would allow user to more easily know locations as well as make the invention more interactive thus boosted user engagement.
Regarding claim 13, the system claim 13 recites similar limitations as method claim 6, and thus is rejected under similar rationale.
Regarding claim 20, the computer program product claim 20 recites similar limitations as method claim 6, and thus is rejected under similar rationale.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Kari et al. (U.S. Patent Application Publication No. 2024/0338908) paragraph 52 teaches “the user 103 remove the transparent chair digital twin 102c via the other user interaction, the AR device 1200 and/or the other device communicatively coupled with the AR device 1200 maintains visual coherence of a sixth representation of the real-world scene”); this show both utilization of digital twin as well as removing corresponding digital visual which would be matched to physical visual.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NAUMAN U AHMAD whose telephone number is (703)756-5306. The examiner can normally be reached Monday - Friday 9:00am - 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at (571) 272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/N.U.A./Examiner, Art Unit 2611
/KEE M TUNG/Supervisory Patent Examiner, Art Unit 2611
Prosecution Insights