DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The amendment filed on May 18, 2026 has been entered.
In view of the amendment to the specification, the Title of the specification has been acknowledged.
In view of the amendment to the claims, the amendment of claims 1, 7, 9, 12-13 and 17-23 have been acknowledged.
In view of the amendment of claims 12, 13, 18, 19 and 20, Applicant amended claims 12 and 19 to describe “3D”; amended claim 13 to use “application”; amended claim 18 and 20 to describe “AI”. Accordingly, the 35 U.S.C. 112 rejections of claims 12-20 have been withdrawn.
Response to Arguments
Applicant’s arguments, see pages 13-17 of Remarks, filed May 18, 2026 have been fully considered. Applicant’s arguments are directed to the amended claims. Upon further consideration and additional prior art reference search, a new ground(s) of rejection is made in view of addition prior art.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 4-5, 8-9, 12, 14-15 and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Guo et al (U.S. Patent Application Publication 2017/0109940 A1) in view of KUMAR et al (U.S. Patent Application Publication 2024/0038488 A1).
Regarding claim 1, Guo discloses a method comprising:
rendering one or more virtual images (FIG. 3; paragraph [0075], the electronic device 322 may identify a region of interest (ROI) in the one or more images (e.g., combined image) ... the electronic device 322 (e.g., image visualization adjuster 336) may generate a magnifying window on the display 340 that presents a zoomed version of the ROI) from one or more perspectives using a magnified portion (Paragraph [0075], a magnifying window on the display 340 that presents a zoomed version of the ROI) of a virtual three-dimensional (3D) representation (Paragraph [0055], the image visualization adjuster 336 may generate and/or adjust an image visualization based on the depth information ... More specifically, the depth information may be used to adjust the shape of a 3D display and/or to generate a 3D surround view for display ...) of an environment associated with a machine (Paragraph [0087], one or more of the elements 562 described in connection with FIG. 5 may be implemented on the electronic device 322 described in connection with FIG. 3; paragraph [0089], the four cameras 546 may capture images 548 of a scene surrounding the vehicle; paragraph [0095], the image visualization adjuster 536 may set the rendering geometry 556 (e.g., bowl or other 3D model) to fit the environment), the magnified portion of the virtual 3D representation corresponding to one or more locations in the environment (Paragraph [0096], the image visualization adjuster 536 may identify an ROI (based on object detection, for example). The image visualization adjuster 536 may optionally generate a magnifying window that presents a zoomed version of the ROI); and
performing one or more control operations associated with the machine in the environment based at least on the one or more locations (Paragraph [0080], the electronic device 322 (e.g., processor 330) may optionally be coupled to, be part of (e.g., be integrated into), include and/or implement an advanced driver assistance system (ADAS). For example, the electronic device 322 may present the image visualization in a vehicle (on an in-dash display, a console display, a windshield projector, a heads-up display, optical head mounted display, etc.). This may assist the driver in situational awareness and/or collision avoidance).
However, Guo does not specifically disclose first predicted locations in the environment;
obtaining, based at least on applying the one or more virtual images to one or more machine learning models, one or more second predicted locations in the environment; and
performing one or more control operations associated with the machine in the environment based at least on the one or more second predicted locations.
In additional, KUMAR discloses (FIG. 1; paragraph [0070], system 100 may implement one or more steps to identify representative images captured at high-magnification levels, suitable for 3D biological structure modeling, from images captured at low-magnification levels; paragraph [0050], the user could obtain the model (e.g., the cryo-EM model described herein) and plug it into supported data acquisition software stacks; paragraph [0071], sample 130 being captured by microscope 110 may result in the images of a cryo-EM grid including holes within it. As described herein, a “grid” or “cryo-EM grid” refers to a substrate with which cryo-EM samples (e.g., samples 130) are placed for imaging ...; paragraph [0026], Cryo-EM's ability to elucidate 3D structures of previously uncharacterized, complex molecules has made it a powerful tool to define binding modes of novel compounds in protein targets) first predicted locations in the environment (Paragraph [0071], a location of where to focus to capture first images 112);
obtaining, based at least on applying the one or more virtual images (Paragraph [0071], first images 112 captured at a first magnification level ... the first image montage comprising first images 112 may depict a cryogenic electron microscopy grid comprising a plurality of cryogenic electron microscopy grid units) to one or more machine learning models (Paragraph [0072], computing system 120 may be configured to execute a first machine learning (ML) model 122), one or more second predicted locations in the environment (Paragraph [0072], the first image montage comprising first images 112 may be input to first ML model 122 ...; paragraph [0076], the first machine learning model (e.g., first ML model 122) may be configured to determine a predicted location of a center of each of the cryogenic electron microscopy grid units within a corresponding training image from the first plurality of training images); and
performing one or more control operations associated with the object Paragraph [0073], computing system 120 may be configured to detect, using first ML model 122, one or more cryogenic electron microscopy units (e.g., identified squares 132) that satisfy a first quality condition. In one or more example, the cryo-EM grid units may use first ML model 122 to generate a first score indicating a quality of the cryogenic electron microscopy grid units).
It's noted that KUMAR describes “a location of where to focus to capture first images” and does not describe “one or more first predicted locations in the environment”. However, the invention of KUMAR describes that the computing system uses machine learning models to determine a predicted location. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the system for predicting a first location in the environment by using one of machine learning model.
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the system and method for producing an image visualization taught by Guo incorporate the teachings of KUMAR, and applying the system for identifying representative images captured at each magnification levels taught by KUMAR to provide machine learning models to determine a predicted location in the virtual 3D environment and generate the magnified portion of the virtual 3D representation to assist the driver in situational awareness. Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify Guo according to the relied-upon teachings of KUMAR to obtain the invention as specified in claim.
Regarding claim 2, the combination of Guo in view of KUMAR discloses everything claimed as applied above (see claim 1).
However, Guo does not specifically disclose further comprising applying, to the one or more machine learning models substantially contemporaneously with the one or more virtual images, one or more token embeddings corresponding to a structured language command, wherein the obtaining of the one or more second predicted locations is further based at least on the applying of the one or more token embeddings.
In additional, KUMAR disclose further comprising applying, to the one or more machine learning models substantially contemporaneously with the one or more virtual images (FIG. 1; paragraph [0072], the first image montage comprising first images 112 may be input to first ML model 122; paragraph [0079], the second image montage comprising second images 114 may be input to second ML model 124 ...), one or more token embeddings corresponding to a structured language command (Paragraph [0076], the first machine learning model (e.g., first ML model 122) may be configured to determine a predicted boundary of each of the cryogenic electron microscopy grid units within a corresponding training image from the first plurality of training images. In some embodiments, the predicted boundary may comprise a bounding box enclosing a boundary of each square of the grid ...; paragraph [0079], second ML model 124 may be configured to detect one or more apertures 134 within the cryogenic electron microscopy grid unit that satisfy a second quality condition based on the second image montage. As described herein, a “hole” refers to an aperture within the squares of the cryo-EM grid), wherein the obtaining of the one or more second predicted locations is further based at least on the applying of the one or more token embeddings (Paragraph [0073], computing system 120 may be configured to detect, using first ML model 122, one or more cryogenic electron microscopy units (e.g., identified squares 132) that satisfy a first quality condition ...).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the system and method for producing an image visualization taught by Guo incorporate the teachings of KUMAR, and applying the system for identifying representative images captured at each magnification levels taught by KUMAR to provide machine learning models to determine a predicted location in the virtual 3D environment and generate the magnified portion of the virtual 3D representation to assist the driver in situational awareness. Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify Guo according to the relied-upon teachings of KUMAR to obtain the invention as specified in claim.
Regarding claim 4, the combination of Guo in view of KUMAR discloses everything claimed as applied above (see claim 1).
However, Guo does not specifically disclose wherein the one or more second predicted locations correspond to one or more refined versions of the one or more first predicted locations such that one or more first confidence scores associated with the one or more first predicted locations are less than one or more second confidences scores associated with the one or more second predicted locations.
In additional, KUMAR disclose wherein the one or more second predicted locations correspond to one or more refined versions of the one or more first predicted locations (FIG. 1; paragraph [0077], computing system 120 may be configured to generate a comparison score. The comparison score may indicate how similar the predicted location is to the predetermined location, how similar the predicted boundary is to the predetermined boundary, or both. In some embodiments, computing system 120 may be configured to update a first set of parameters of the first machine learning model based on the comparisons) such that one or more first confidence scores associated with the one or more first predicted locations (Paragraph [0073], computing system 120 may be configured to detect, using first ML model 122, one or more cryogenic electron microscopy units (e.g., identified squares 132) that satisfy a first quality condition. In one or more example, the cryo-EM grid units may use first ML model 122 to generate a first score indicating a quality of the cryogenic electron microscopy grid units; paragraph [0078], the first magnification level may comprise 1×, 2×, 3×, etc.) are less than one or more second confidences scores (Paragraph [0080], second ML model 124 may evaluate second images 124 to determine a second score for each aperture. Example images of apertures within a square are illustrated in FIG. 3B. Computing system 120 may input second images 114 into second machine learning model 124 to obtain a second score indicating a likelihood that at least one of a biological structure suspended within each of a plurality of apertures depicted by second images 114; paragraph [0078], the second magnification level may comprise 10×, 20×, 30×, etc.) associated with the one or more second predicted locations (Paragraph [0079], second ML model 124 may be configured to detect one or more apertures 134 within the cryogenic electron microscopy grid unit that satisfy a second quality condition based on the second image montage. As described herein, a “hole” refers to an aperture within the squares of the cryo-EM grid in which vitreous ice and protein are suspended. Identified holes 134 may be provided to microscope 110 with instructions to capture third images 116 (e.g., micrographs) captured at a third magnification level).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the system and method for producing an image visualization taught by Guo incorporate the teachings of KUMAR, and applying the system for identifying representative images captured at each magnification levels taught by KUMAR to provide machine learning models to determine a predicted location in the virtual 3D environment and generate the magnified portion of the virtual 3D representation to assist the driver in situational awareness. Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify Guo according to the relied-upon teachings of KUMAR to obtain the invention as specified in claim.
Regarding claim 5, the combination of Guo in view of KUMAR discloses everything claimed as applied above (see claim 1).
However, Guo does not specifically disclose wherein the one or more machine learning models include one or more convex upsampling layers to increase one or more spatial dimensions of one or more feature maps corresponding to the one or more virtual images.
In additional, KUMAR disclose wherein the one or more machine learning models include one or more convex upsampling layers to increase one or more spatial dimensions of one or more feature maps corresponding to the one or more virtual images (FIG. 1; paragraph [0086], computing system 120 may receive third images 116 depicting sample 130 captured at the third magnification level. In one or more examples, the third magnification may comprise 100×, 200×, 300×, etc., and the second magnification level may comprise 10×, 20×, 30×, etc.; FIGS. 3A-3C show images captured using electron microscopy techniques at various magnification levels).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the system and method for producing an image visualization taught by Guo incorporate the teachings of KUMAR, and applying the system for identifying representative images captured at each magnification levels taught by KUMAR to provide machine learning models for increasing one or more spatial dimensions of one or more feature maps corresponding to the one or more virtual images. Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify Guo according to the relied-upon teachings of KUMAR to obtain the invention as specified in claim.
Regarding claim 8, the combination of Guo in view of KUMAR discloses everything claimed as applied above (see claim 1), and Guo disclose wherein the one or more first predicted locations and the one or more second predicted locations correspond to at least one of:
one or more objects in the environment (FIG, 2; paragraph [0027], paragraph [0028], the surround view image visualization 202 makes the ground around the vehicle 206 (e.g., vehicle model, representation, etc.) appear flat, while other objects in the image have a sense of height. For example, a house 216, another vehicle 214 and a fire hydrant 218 each appear to have height (e.g., 3 dimensions, height, depth) in the surround view image visualization 202); or
one or more positions associated with one or more key poses of the machine.
Regarding claim 9, the combination of Guo in view of KUMAR discloses everything claimed as applied above (see claim 1), and Guo disclose further comprising:
generating the virtual 3D representation of the environment based at least on applying one or more images depicting the environment (FIG. 3; paragraph [0055], the image visualization adjuster 336 may generate and/or adjust an image visualization based on the depth information ... More specifically, the depth information may be used to adjust the shape of a 3D display and/or to generate a 3D surround view for display ...; paragraph [0087], one or more of the elements 562 described in connection with FIG. 5 may be implemented on the electronic device 322 described in connection with FIG. 3; paragraph [0089], the four cameras 546 may capture images 548 of a scene surrounding the vehicle; paragraph [0095], the image visualization adjuster 536 may set the rendering geometry 556 (e.g., bowl or other 3D model) to fit the environment).
However, Guo does not specifically disclose generating the virtual 3D representation of the environment to a neural network; and
obtaining the one or more first predicted locations in the environment based at least on applying, to one or more second machine learning models, one or more second virtual images depicting the virtual 3D representation of the environment from one or more second perspectives.
In additional, KUMAR discloses generating the virtual 3D representation of the environment (FIG. 1; computing system 120 may be configured to generate a 3D representation of the biological structure (e.g., sample 130)) to a neural network (Paragraph [0071], sample 130 being captured by microscope 110 ...; paragraph [0075], the first ML model may be a convolutional neural network; paragraph [0084], the second ML model may be a convolutional neural network); and
obtaining the one or more first predicted locations in the environment based at least on applying, to one or more second machine learning models (Paragraph [0076], the first machine learning model (e.g., first ML model 122) may be configured to determine a predicted location of a center of each of the cryogenic electron microscopy grid units ...), one or more second virtual images depicting the virtual 3D representation of the environment from one or more second perspectives (Paragraph [0078], second images 114 may be captured by microscope 110 at a second magnification level).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the system and method for producing an image visualization taught by Guo incorporate the teachings of KUMAR, and applying the system for identifying representative images captured at each magnification levels taught by KUMAR to provide machine learning models to determine a predicted location in the virtual 3D environment and generate the magnified portion of the virtual 3D representation to assist the driver in situational awareness. Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify Guo according to the relied-upon teachings of KUMAR to obtain the invention as specified in claim.
Regarding claim 12, Guo discloses a system comprising:
one or more processors (FIG. 3; paragraph [0034], the electronic device 322 may include a processor 330) to:
generate an updated version of a three-dimensional (3D} representation (Paragraph [0055], the image visualization adjuster 336 may generate and/or adjust an image visualization based on the depth information ... More specifically, the depth information may be used to adjust the shape of a 3D display and/or to generate a 3D surround view for display ...) of an environment (Paragraph [0087], one or more of the elements 562 described in connection with FIG. 5 may be implemented on the electronic device 322 described in connection with FIG. 3; paragraph [0089], the four cameras 546 may capture images 548 of a scene surrounding the vehicle; paragraph [0095], the image visualization adjuster 536 may set the rendering geometry 556 (e.g., bowl or other 3D model) to fit the environment), the updated version including a magnified portion of the 3D representation (Paragraph [0075], the electronic device 322 may identify a region of interest (ROI) in the one or more images (e.g., combined image) ... the electronic device 322 (e.g., image visualization adjuster 336) may generate a magnifying window on the display 340 that presents a zoomed version of the ROI);
render one or more images depicting the magnified portion of the 3D representation from one or more perspectives (Paragraph [0075], the electronic device 322 may identify a region of interest (ROI) in the one or more images (e.g., combined image) ... the electronic device 322 (e.g., image visualization adjuster 336) may generate a magnifying window on the display 340 that presents a zoomed version of the ROI);
perform one or more operations associated with a machine in the environment (Paragraph [0080], the electronic device 322 (e.g., processor 330) may optionally be coupled to, be part of (e.g., be integrated into), include and/or implement an advanced driver assistance system (ADAS). For example, the electronic device 322 may present the image visualization in a vehicle (on an in-dash display, a console display, a windshield projector, a heads-up display, optical head mounted display, etc.). This may assist the driver in situational awareness and/or collision avoidance).
However, Guo does not specifically disclose a magnified portion of the 3D representation based at least on one or more first predictions associated with the magnified portion;
apply, to one or more machine learning models, the one or more images; and
perform one or more operations associated with an object
In additional, KUMAR discloses (FIG. 1; paragraph [0040], computing system 120 may execute these machine learning models utilizing one or more processing devices (e.g., computing system 1500 discussed below with respect to FIG. 15) that may include hardware (e.g., a general purpose processor, a graphic processing unit (GPU)) to:
a magnified portion of the 3D representation (Paragraph [0050], the user could obtain the model (e.g., the cryo-EM model described herein) and plug it into supported data acquisition software stacks; paragraph [0071], sample 130 being captured by microscope 110 may result in the images of a cryo-EM grid including holes within it. As described herein, a “grid” or “cryo-EM grid” refers to a substrate with which cryo-EM samples (e.g., samples 130) are placed for imaging ...; paragraph [0026], Cryo-EM's ability to elucidate 3D structures of previously uncharacterized, complex molecules has made it a powerful tool to define binding modes of novel compounds in protein targets; paragraph [0072], computing system 120 may be configured to execute a first machine learning (ML) model 122 ... the first image montage comprising first images 112 may be input to first ML model 122 ...; paragraph [0076], the first machine learning model (e.g., first ML model 122) may be configured to determine a predicted location of a center of each of the cryogenic electron microscopy grid units within a corresponding training image from the first plurality of training images) based at least on one or more first predictions (Paragraphs [0072]-[0073], the first image montage comprising first images 112 may be input to first ML model 122 ... computing system 120 may be configured to detect, using first ML model 122, one or more cryogenic electron microscopy units (e.g., identified squares 132). As shown in FIG. 1, the identified squares 132 is provided to microscope 110 with instructions to capture second images 114) associated with the magnified portion (Paragraph [0071], first images 112 captured at a first magnification level);
apply, to one or more machine learning models (Paragraph [0079], computing system 120 may be configured to execute a second machine learning (ML) model 124), the one or more images (Paragraph [0078], computing system 120 may be configured to, for each of the one or more cryogenic electron microscopy grid units, receive a second image montage comprising second images of the cryogenic electron microscopy grid unit captured at a second magnification level); and
perform one or more operations associated with an object Paragraph [0079], the second image montage comprising second images 114 may be input to second ML model 124. Second ML model 124 may be configured to detect one or more apertures 134 within the cryogenic electron microscopy grid unit that satisfy a second quality condition based on the second image montage. As described herein, a “hole” refers to an aperture within the squares of the cryo-EM grid in which vitreous ice and protein are suspended) based at least on one or more second predictions obtained using the one or more machine learning models (Paragraph [0084], the second machine learning model may be configured to determine a predicted location of a center of each of the apertures (e.g., holes) ...).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the system and method for producing an image visualization taught by Guo incorporate the teachings of KUMAR, and applying the system for identifying representative images captured at each magnification levels taught by KUMAR to provide machine learning models to determine a predicted location in the virtual 3D environment and generate the magnified portion of the virtual 3D representation to assist the driver in situational awareness. Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify Guo according to the relied-upon teachings of KUMAR to obtain the invention as specified in claim.
Regarding claim 14, the combination of Guo in view of KUMAR discloses everything claimed as applied above (see claim 12).
However, Guo does not specifically disclose wherein the one or more second predictions correspond to one or more refined versions of the one or more first predictions.
In additional, KUMAR disclose wherein the one or more second predictions correspond to one or more refined versions of the one or more first predictions (FIG. 1; paragraph [0077], computing system 120 may be configured to generate a comparison score. The comparison score may indicate how similar the predicted location is to the predetermined location, how similar the predicted boundary is to the predetermined boundary, or both. In some embodiments, computing system 120 may be configured to update a first set of parameters of the first machine learning model based on the comparisons).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the system and method for producing an image visualization taught by Guo incorporate the teachings of KUMAR, and applying the system for identifying representative images captured at each magnification levels taught by KUMAR to provide machine learning models to determine a predicted location in the virtual 3D environment and generate the magnified portion of the virtual 3D representation to assist the driver in situational awareness. Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify Guo according to the relied-upon teachings of KUMAR to obtain the invention as specified in claim.
Regarding claim 15, the combination of Guo in view of KUMAR discloses everything claimed as applied above (see claim 1).
However, Guo does not specifically disclose wherein the one or more second predictions are associated with one or more greater confidence scores than the one or more first predictions.
In additional, KUMAR disclose wherein the one or more second predictions are associated with one or more greater confidence scores (FIG. 1; paragraph [0080], second ML model 124 may evaluate second images 124 to determine a second score for each aperture. Example images of apertures within a square are illustrated in FIG. 3B. Computing system 120 may input second images 114 into second machine learning model 124 to obtain a second score indicating a likelihood that at least one of a biological structure suspended within each of a plurality of apertures depicted by second images 114; paragraph [0078], the second magnification level may comprise 10×, 20×, 30×, etc. ) than the one or more first predictions (Paragraph [0073], computing system 120 may be configured to detect, using first ML model 122, one or more cryogenic electron microscopy units (e.g., identified squares 132) that satisfy a first quality condition. In one or more example, the cryo-EM grid units may use first ML model 122 to generate a first score indicating a quality of the cryogenic electron microscopy grid units; paragraph [0078], the first magnification level may comprise 1×, 2×, 3×, etc.).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the system and method for producing an image visualization taught by Guo incorporate the teachings of KUMAR, and applying the system for identifying representative images captured at each magnification levels taught by KUMAR to provide machine learning models to determine a predicted location in the virtual 3D environment and generate the magnified portion of the virtual 3D representation to assist the driver in situational awareness. Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify Guo according to the relied-upon teachings of KUMAR to obtain the invention as specified in claim.
Regarding claim 18, the combination of Guo in view of KUMAR discloses everything claimed as applied above (see claim 12), and Guo disclose wherein the system is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing one or more simulation operations;
a system for performing one or more digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing one or more deep learning operations;
a system implemented using an edge device;
a system implemented using a robot;
a system for performing one or more generative artificial intelligence (AI) operations;
a system for performing operations using a large language model;
a system for performing operations using one or more vision language models (VLMs);
a system for performing operations using one or more multi-modal language models;
a system for performing one or more conversational AI operations;
a system for generating synthetic data;
a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content (FIG. 3; paragraph [0055], the image visualization adjuster 336 may adjust a rendering geometry based on the depth information. More specifically, the depth information may be used to adjust the shape of a 3D display and/or to generate a 3D surround view for display, virtual reality and/or augmented reality);
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.
Regarding claim 19, Guo discloses at least one processor (FIG. 3; paragraph [0034], the electronic device 322 may include a processor 330) comprising:
processing circuitry (Paragraph [0033], The electronic device 322 may include one or more components or elements. One or more of the components or elements may be implemented in hardware (e.g., circuitry)) to perform one or more operations associated with a machine in an environment (Paragraph [0080], the electronic device 322 (e.g., processor 330) may optionally be coupled to, be part of (e.g., be integrated into), include and/or implement an advanced driver assistance system (ADAS). For example, the electronic device 322 may present the image visualization in a vehicle (on an in-dash display, a console display, a windshield projector, a heads-up display, optical head mounted display, etc.). This may assist the driver in situational awareness and/or collision avoidance) using one or more updated Paragraph [0075], the electronic device 322 may identify a region of interest (ROI) in the one or more images (e.g., combined image) ... the electronic device 322 (e.g., image visualization adjuster 336) may generate a magnifying window on the display 340 that presents a zoomed version of the ROI), one or more images depicting a magnified portion of a three-dimensional (3D} representation of the environment (Paragraph [0075], the electronic device 322 may identify a region of interest (ROI) in the one or more images (e.g., combined image) ... the electronic device 322 (e.g., image visualization adjuster 336) may generate a magnifying window on the display 340 that presents a zoomed version of the ROI) from one or more virtual sensor perspectives (Paragraph [0075], the electronic device 322 may identify a region of interest (ROI) in the one or more images (e.g., combined image)).
However, Guo does not specifically disclose the one or more updated predictions generated based at least on applying, to one or more machine learning models, one or more images depicting a magnified portion of a three-dimensional (3D} representation of the environment, the magnified portion corresponding to one or more locations associated with one or more initial predictions.
In additional, KUMAR discloses (FIG. 1; paragraph [0040], computing system 120 may execute these machine learning models utilizing one or more processing devices (e.g., computing system 1500 discussed below with respect to FIG. 15) that may include hardware (e.g., a general purpose processor, a graphic processing unit (GPU), an application-specific integrated circuit (ASIC), a system-on-chip (SoC)) the one or more updated predictions generated (Paragraph [0050], the user could obtain the model (e.g., the cryo-EM model described herein) and plug it into supported data acquisition software stacks; paragraph [0071], sample 130 being captured by microscope 110 may result in the images of a cryo-EM grid including holes within it. As described herein, a “grid” or “cryo-EM grid” refers to a substrate with which cryo-EM samples (e.g., samples 130) are placed for imaging ...; paragraph [0026], Cryo-EM's ability to elucidate 3D structures of previously uncharacterized, complex molecules has made it a powerful tool to define binding modes of novel compounds in protein targets; paragraph [0079], the second image montage comprising second images 114 may be input to second ML model 124. Second ML model 124 may be configured to detect one or more apertures 134 within the cryogenic electron microscopy grid unit that satisfy a second quality condition based on the second image montage. As described herein, a “hole” refers to an aperture within the squares of the cryo-EM grid in which vitreous ice and protein are suspended) based at least on applying, to one or more machine learning models (Paragraph [0079], computing system 120 may be configured to execute a second machine learning (ML) model 124), one or more images depicting a magnified portion of a three-dimensional (3D} representation of the environment (Paragraph [0026], Cryo-EM's ability to elucidate 3D structures of previously uncharacterized, complex molecules has made it a powerful tool to define binding modes of novel compounds in protein targets ...; paragraph [0078], computing system 120 may be configured to, for each of the one or more cryogenic electron microscopy grid units, receive a second image montage comprising second images of the cryogenic electron microscopy grid unit captured at a second magnification level), the magnified portion corresponding to one or more locations associated with one or more initial predictions (Paragraphs [0072]-[0073], the first image montage comprising first images 112 may be input to first ML model 122 ... computing system 120 may be configured to detect, using first ML model 122, one or more cryogenic electron microscopy units (e.g., identified squares 132). As shown in FIG. 1, the identified squares 132 is provided to microscope 110 with instructions to capture second images 114).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the system and method for producing an image visualization taught by Guo incorporate the teachings of KUMAR, and applying the system for identifying representative images captured at each magnification levels taught by KUMAR to provide machine learning models to determine a predicted location in the virtual 3D environment and generate the magnified portion of the virtual 3D representation to assist the driver in situational awareness. Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify Guo according to the relied-upon teachings of KUMAR to obtain the invention as specified in claim.
Regarding claim 20, the combination of Guo in view of KUMAR discloses everything claimed as applied above (see claim 19), and Guo disclose wherein the processor is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing one or more simulation operations;
a system for performing one or more digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing one or more deep learning operations;
a system implemented using an edge device;
a system implemented using a robot; a system for performing one or more generative AI operations;
a system for performing operations using a large language model;
a system for performing operations using one or more vision language models (VLMs);
a system for performing operations using one or more multi-modal language models;
a system for performing one or more conversational artificial intelligence (AI) operations;
a system for generating synthetic data;
a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content (FIG. 3; paragraph [0055], the image visualization adjuster 336 may adjust a rendering geometry based on the depth information. More specifically, the depth information may be used to adjust the shape of a 3D display and/or to generate a 3D surround view for display, virtual reality and/or augmented reality);
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.
Claims 3, 11 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Guo et al (U.S. Patent Application Publication 2017/0109940 A1) in view of KUMAR et al (U.S. Patent Application Publication 2024/0038488 A1) in view of Chatzikalymnios et al (U.S. Patent Application Publication 2025/0068228 A1).
Regarding claim 3, the combination of Guo in view of KUMAR discloses everything claimed as applied above (see claim 1).
However, Guo does not specifically disclose further comprising obtaining, based at least on the applying of the one or more virtual images to the one or more machine learning models, one or more heatmaps indicative of the one or more second predicted locations.
In additional, Chatzikalymnios discloses (Paragraph [0029], An XR device can receive a sequence of images and track one or more objects depicted in the images in a 3D space. The XR device may utilize various parameters to track an object ... and predictive information (e.g., using a machine learning model to predict object motion)) further comprising obtaining, based at least on the applying of the one or more virtual images to the one or more machine learning models (FIG. 2; paragraph [0064], The multi-camera object tracking system 216 may implement two phases of object tracking: ... Various algorithms, including algorithms implemented by object tracking machine learning models), one or more heatmaps indicative of the one or more second predicted locations (FIG. 3; paragraph [0086], The inference component 314 may be used to generate tracking estimates or predictions, e.g., to predict the location or pose of a tracked object. As mentioned, the XR device 110 may utilize one or more object tracking machine learning models for this purpose ... The machine learning model may, in some examples, be known as a core tracker. A core tracker is used in computer vision systems to track the movement of an object in a sequence of images or videos ...; paragraph [0137], the multi-camera object tracking system may be configured to generate a heatmap across the relevant image that returns a score for each projected location) .
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the system and method for producing an image visualization taught by Guo in view of KUMAR incorporate the teachings of Chatzikalymnios, and applying the multi-camera object tracking system taught by Chatzikalymnios to provide compatibility with the system for using the machine learning model to generate the heatmaps based on the predicted location. Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify Guo in view of KUMAR according to the relied-upon teachings of Chatzikalymnios to obtain the invention as specified in claim.
Regarding claim 11, the combination of Guo in view of KUMAR discloses everything claimed as applied above (see claim 1).
However, Guo does not specifically disclose wherein the one or more second predicted locations include one or more two-dimensional (2D) space predictions corresponding to virtual images of the one or more virtual images, the method further comprising:
mapping the 2D space predictions into a 3D space;
generating, based at least on the mapping, one or more 3D space predictions; and
performing one or more second control operations associated with the machine based at least on the one or more 3D space predictions.
In additional, Chatzikalymnios discloses (Paragraph [0029], An XR device can receive a sequence of images and track one or more objects depicted in the images in a 3D space. The XR device may utilize various parameters to track an object ... and predictive information (e.g., using a machine learning model to predict object motion)) wherein the one or more second predicted locations include one or more two-dimensional (2D) space predictions (FIG. 2; paragraph [0118], the multi-camera object tracking system 216 then uses the predicted location of the object to execute a distance algorithm for determining which subset of the image sensors 208 ... the multi-camera object tracking system 216 may translate the predicted location of the object in the real world onto a camera view image, or onto an image plane associated with the particular camera, to obtain the predicted location as it would appear on an image captured by the particular camera. This may be referred to as a 2D projected location) corresponding to virtual images of the one or more virtual images (FIF. 3; paragraph [0086], The inference component 314 may be used to generate tracking estimates or predictions, e.g., to predict the location or pose of a tracked object. As mentioned, the XR device 110 may utilize one or more object tracking machine learning models for this purpose ... The machine learning model may, in some examples, be known as a core tracker. A core tracker is used in computer vision systems to track the movement of an object in a sequence of images or videos), the method further comprising:
mapping the 2D space predictions into a 3D space (Paragraph [0118], he multi-camera object tracking system 216 may utilize the pose of the XR device 110 to calculate where a particular 3D point should appear on the 2D plane of a camera. The 2D projected location may be obtained by applying, for example, a projection matrix to the 3D coordinates of the relevant object);
generating, based at least on the mapping, one or more 3D space predictions (Paragraph [0132], it will be appreciated that predetermined calibrations and/or transformations stored for each respective camera may be used by the multi-camera object tracking system to generate or update these 2D projections, together with other data such as the pose of the XR device. In some examples, the predicted 3D location of the physical object 108 in the real-world environment 102 is a 3D position in a defined coordinate system that can be related in a specific manner to each respective camera view based on predefined or predetermined calibrations, e.g., during setup of the cameras of the multi-camera object tracking system or in some offline calibration step); and
performing one or more second control operations associated with the machine based at least on the one or more 3D space predictions (FIG. 1; paragraph [0131], the 3D location of the physical object 108 in the real-world environment 102, as predicted by the multi-camera object tracking system for the current frame, is then projected onto a left camera view image 902 associated with the left camera and a right camera view image 904 associated with the right camera).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the system and method for producing an image visualization taught by Guo in view of KUMAR incorporate the teachings of Chatzikalymnios, and applying the multi-camera object tracking system taught by Chatzikalymnios to provide compatibility with the system for using the machine learning model to map the 2D space predictions into a 3D space and project the projected location of the object onto the image. Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify Guo in view of KUMAR according to the relied-upon teachings of Chatzikalymnios to obtain the invention as specified in claim.
Regarding claim 13, the combination of Guo in view of KUMAR discloses everything claimed as applied above (see claim 12).
However, Guo does not specifically disclose the one or more processors further to obtain, based at least on application of the one or more images to the one or more machine learning models, one or more heatmaps indicative of one or more locations corresponding to the one or more second predictions.
In additional, Chatzikalymnios discloses (Paragraph [0029], An XR device can receive a sequence of images and track one or more objects depicted in the images in a 3D space. The XR device may utilize various parameters to track an object ... and predictive information (e.g., using a machine learning model to predict object motion)) the one or more processors further to obtain, based at least on application of the one or more images to the one or more machine learning models (FIG. 2; paragraph [0064], The multi-camera object tracking system 216 may implement two phases of object tracking: ... Various algorithms, including algorithms implemented by object tracking machine learning models), one or more heatmaps indicative of one or more locations corresponding to the one or more second predictions (FIG. 3; paragraph [0086], The inference component 314 may be used to generate tracking estimates or predictions, e.g., to predict the location or pose of a tracked object. As mentioned, the XR device 110 may utilize one or more object tracking machine learning models for this purpose ... The machine learning model may, in some examples, be known as a core tracker. A core tracker is used in computer vision systems to track the movement of an object in a sequence of images or videos ...; paragraph [0137], the multi-camera object tracking system may be configured to generate a heatmap across the relevant image that returns a score for each projected location).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the system and method for producing an image visualization taught by Guo in view of KUMAR incorporate the teachings of Chatzikalymnios, and applying the multi-camera object tracking system taught by Chatzikalymnios to provide compatibility with the system for using the machine learning model to generate the heatmaps based on the predicted location. Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify Guo in view of KUMAR according to the relied-upon teachings of Chatzikalymnios to obtain the invention as specified in claim.
Claims 6-7 and 16-17 are rejected under 35 U.S.C. 103 as being unpatentable over Guo et al (U.S. Patent Application Publication 2017/0109940 A1) in view of KUMAR et al (U.S. Patent Application Publication 2024/0038488 A1) in view of Carroll et al (U.S. Patent Application Publication 2020/0117952 A1).
Regarding claim 6, the combination of Guo in view of KUMAR discloses everything claimed as applied above (see claim 1).
However, Guo does not specifically disclose wherein the one or more virtual images are rendered such that one or more sizes associated with the one or more virtual images are rationally divisible by one or more patch sizes associated with the one or more machine learning models.
In additional, Carroll discloses (Abstract, a computer-implemented method for target object position prediction includes receiving, via an RGB camera a plurality of images depicting one or more persons positioned on a floor ...) wherein the one or more virtual images are rendered (Paragraph [0034], the bottom portion of FIG. 3B shows how the annotated images 380 are created. The input images 305 are pre-processed by the preprocessing module 310, as described above with respect to FIG. 3A) such that one or more sizes associated with the one or more virtual images are rationally divisible by one or more patch sizes (Paragraphs [0026]-[0029], FIG. 3A illustrates an example technique for training a FP CNN ... In addition to the input images 303 and the person location labels 355, a set of camera coordinates 340 is supplied that describes the position of the cameras 302 in real world space. For example in some embodiments the camera coordinates 340 include latitude and longitude values of the camera, as well as directional information ... the directional information also includes spherical coordinates (i.e., declination and degree of rotation relative to world coordinates) .. A preprocessing module 310 performs any pre-processing necessary to prepare the images for further processing. This pre-processing may include, for example, cropping the input images 303 to a preferred size or to focus on target objects, denoising images, or converting the input images 303 from color to black and white (or vice versa) ...) associated with the one or more machine learning models (Paragraph [0021], upon determining that one or more images comprise the person 110, the one or more input images are fed into the PE model. For the purposes of this discussion, it is assumed that the PE model is a convolutional neural network (CNN)).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the system and method for producing an image visualization taught by Guo in view of KUMAR incorporate the teachings of Carroll, and applying the method for target object position prediction taught by Carroll to provide compatibility with the system for using the machine learning model to render the image based on the degree of rotation and a preferred size. Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify Guo in view of KUMAR according to the relied-upon teachings of Carroll to obtain the invention as specified in claim.
Regarding claim 7, the combination of Guo in view of KUMAR discloses everything claimed as applied above (see claim 1) and the combination of Guo in view of KUMAR discloses a machine in the environment (see claim 1).
However, Guo does not specifically disclose further comprising:
determining, based at least on one or more local features corresponding to the one or more second predicted locations, a degree of rotation associated with manipulating an end-effector of the machine,
wherein the one or more control operations include rotating the end-effector of the machine based at least on the degree of rotation.
In additional, Carroll discloses (Abstract, a computer-implemented method for target object position prediction includes receiving, via an RGB camera a plurality of images depicting one or more persons positioned on a floor ...) further comprising:
determining, based at least on one or more local features corresponding to the one or more second predicted locations (Paragraph [0005], techniques disclosed herein include identifying and calculating body part key points of a person included in an input image and predicting a position of the person's feet using the image and the calculated body part key points), a degree of rotation (Paragraphs [0026]-[0029], FIG. 3A illustrates an example technique for training a FP CNN ... In addition to the input images 303 and the person location labels 355, a set of camera coordinates 340 is supplied that describes the position of the cameras 302 in real world space. For example in some embodiments the camera coordinates 340 include latitude and longitude values of the camera, as well as directional information ... the directional information also includes spherical coordinates (i.e., declination and degree of rotation relative to world coordinates) .. A preprocessing module 310 performs any pre-processing necessary to prepare the images for further processing. This pre-processing may include, for example, cropping the input images 303 to a preferred size or to focus on target objects, denoising images, or converting the input images 303 from color to black and white (or vice versa) ...) associated with manipulating an end-effector of the Paragraph [0021], upon determining that one or more images comprise the person 110, the one or more input images are fed into the PE model. For the purposes of this discussion, it is assumed that the PE model is a convolutional neural network (CNN)),
wherein the one or more control operations include rotating the end-effector of the Paragraph [0034], the bottom portion of FIG. 3B shows how the annotated images 380 are created. The input images 305 are pre-processed by the preprocessing module 310, as described above with respect to FIG. 3A).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the system and method for producing an image visualization taught by Guo in view of KUMAR incorporate the teachings of Carroll, and applying the method for target object position prediction taught by Carroll to provide compatibility with the system for using the machine learning model to render the image based on the degree of rotation and a preferred size. Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify Guo in view of KUMAR according to the relied-upon teachings of Carroll to obtain the invention as specified in claim.
Regarding claim 16, the combination of Guo in view of KUMAR discloses everything claimed as applied above (see claim 12).
However, Guo does not specifically disclose wherein the one or more images are generated such that one or more sizes associated with the one or more images are rationally divisible by one or more patch sizes associated with the one or more machine learning models.
In additional, Carroll discloses (Abstract, a computer-implemented method for target object position prediction includes receiving, via an RGB camera a plurality of images depicting one or more persons positioned on a floor ...) wherein the one or more images are generated (Paragraph [0034], the bottom portion of FIG. 3B shows how the annotated images 380 are created. The input images 305 are pre-processed by the preprocessing module 310, as described above with respect to FIG. 3A) such that one or more sizes associated with the one or more images are rationally divisible by one or more patch sizes (Paragraphs [0026]-[0029], FIG. 3A illustrates an example technique for training a FP CNN ... In addition to the input images 303 and the person location labels 355, a set of camera coordinates 340 is supplied that describes the position of the cameras 302 in real world space. For example in some embodiments the camera coordinates 340 include latitude and longitude values of the camera, as well as directional information ... the directional information also includes spherical coordinates (i.e., declination and degree of rotation relative to world coordinates) .. A preprocessing module 310 performs any pre-processing necessary to prepare the images for further processing. This pre-processing may include, for example, cropping the input images 303 to a preferred size or to focus on target objects, denoising images, or converting the input images 303 from color to black and white (or vice versa) ...) associated with the one or more machine learning models (Paragraph [0021], upon determining that one or more images comprise the person 110, the one or more input images are fed into the PE model. For the purposes of this discussion, it is assumed that the PE model is a convolutional neural network (CNN)).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the system and method for producing an image visualization taught by Guo in view of KUMAR incorporate the teachings of Carroll, and applying the method for target object position prediction taught by Carroll to provide compatibility with the system for using the machine learning model to render the image based on the degree of rotation and a preferred size. Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify Guo in view of KUMAR according to the relied-upon teachings of Carroll to obtain the invention as specified in claim.
Regarding claim 17, the combination of Guo in view of KUMAR discloses everything claimed as applied above (see claim 12) and the combination of Guo in view of KUMAR discloses a machine in the environment (see claim 12).
However, Guo does not specifically disclose the one or more processors further to:
determine, based at least on one or more local features corresponding to the one or more second predictions, a degree of rotation associated with manipulating an end-effector of the machine; and
wherein the one or more operations include rotating the end-effector of the machine based at least on the degree of rotation.
In additional, Carroll discloses (Abstract, a computer-implemented method for target object position prediction includes receiving, via an RGB camera a plurality of images depicting one or more persons positioned on a floor ...) the one or more processors further to:
determine, based at least on one or more local features corresponding to the one or more second predictions (Paragraph [0005], techniques disclosed herein include identifying and calculating body part key points of a person included in an input image and predicting a position of the person's feet using the image and the calculated body part key points), a degree of rotation (Paragraphs [0026]-[0029], FIG. 3A illustrates an example technique for training a FP CNN ... In addition to the input images 303 and the person location labels 355, a set of camera coordinates 340 is supplied that describes the position of the cameras 302 in real world space. For example in some embodiments the camera coordinates 340 include latitude and longitude values of the camera, as well as directional information ... the directional information also includes spherical coordinates (i.e., declination and degree of rotation relative to world coordinates) .. A preprocessing module 310 performs any pre-processing necessary to prepare the images for further processing. This pre-processing may include, for example, cropping the input images 303 to a preferred size or to focus on target objects, denoising images, or converting the input images 303 from color to black and white (or vice versa) ...) associated with manipulating an end-effector of the Paragraph [0021], upon determining that one or more images comprise the person 110, the one or more input images are fed into the PE model. For the purposes of this discussion, it is assumed that the PE model is a convolutional neural network (CNN)); and
wherein the one or more operations include rotating the end-effector of the Paragraph [0034], the bottom portion of FIG. 3B shows how the annotated images 380 are created. The input images 305 are pre-processed by the preprocessing module 310, as described above with respect to FIG. 3A).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the system and method for producing an image visualization taught by Guo in view of KUMAR incorporate the teachings of Carroll, and applying the method for target object position prediction taught by Carroll to provide compatibility with the system for using the machine learning model to render the image based on the degree of rotation and a preferred size. Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify Guo in view of KUMAR according to the relied-upon teachings of Carroll to obtain the invention as specified in claim.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Guo et al (U.S. Patent Application Publication 2017/0109940 A1) in view of KUMAR et al (U.S. Patent Application Publication 2024/0038488 A1) in view of Bhatnagar et al (U.S. Patent Application Publication 2025/0341941 A1).
Regarding claim 10, the combination of Guo in view of KUMAR discloses everything claimed as applied above (see claim 9).
However, Guo does not specifically disclose wherein a first zoom factor associated with the one or more virtual images is greater than a second zoom factor associated with the one or more second virtual images.
In additional, Bhatnagar discloses (Abstract, while a view of a three-dimensional environment is visible via a display generation component, a computer system automatically detects an object in the three-dimensional environment ...; paragraph [0231], FIG. 7C illustrates a virtual magnifier 7032 magnifying (or increasing the size of) a first portion of application user interface 7030, according to some embodiments. For example, virtual magnifier 7032 magnifies search field 7036, among other user interface elements. In some embodiments, virtual magnifier 7032 magnifies portions of the view of the three-dimensional environment 7000′ on which user 7002 is focused) wherein a first zoom factor associated with the one or more virtual images is greater than a second zoom factor associated with the one or more second virtual images (Paragraph [0232], Virtual magnifier 7032 includes slider 7034, which is a user interface element for adjusting a zoom level of the virtual magnifier 7032. A user can increase or decrease the zoom or magnification level by interacting (e.g., directly, or indirectly) with slider 7034 ... If the zoom level is decreased in response to user's 7002 input directed at slider 7034, the size of the displayed search field and/or other user interface elements (displayed by or within virtual magnifier 7032) decreases in accordance with the decreased zoom level. Thus, the second zoom factor is less than the first zoom factor when the zoom level is decreased in response to user's 7002 input).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the system and method for producing an image visualization taught by Guo in view of KUMAR incorporate the teachings of Bhatnagar, and applying the view of a three-dimensional environment taught by Bhatnagar to provide zoom level of the virtual magnifier for controlling the size of the displayed object in the three-dimensional environment. Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify Guo in view of KUMAR according to the relied-upon teachings of Bhatnagar to obtain the invention as specified in claim.
Claims 21-23 are rejected under 35 U.S.C. 103 as being unpatentable over Guo et al (U.S. Patent Application Publication 2017/0109940 A1) in view of Bhatnagar et al (U.S. Patent Application Publication 2025/0341941 A1) in view of KUMAR et al (U.S. Patent Application Publication 2024/0038488 A1).
Regarding claim 21, Guo discloses a method comprising:
determining an area of interest (FIG. 3; paragraph [0075], the electronic device 322 may identify a region of interest (ROI) in the one or more images (e.g., combined image) ... the electronic device 322 (e.g., image visualization adjuster 336) may generate a magnifying window on the display 340 that presents a zoomed version of the ROI) surrounding an object in a virtual environment representing a real environment (Paragraph [0087], one or more of the elements 562 described in connection with FIG. 5 may be implemented on the electronic device 322 described in connection with FIG. 3; paragraph [0089], the four cameras 546 may capture images 548 of a scene surrounding the vehicle; paragraph [0095], the image visualization adjuster 536 may set the rendering geometry 556 (e.g., bowl or other 3D model) to fit the environment);
generating, based at least on zooming-in to magnify a view of the area of interest (Paragraph [0066], adjusting the image visualization based on depth information may include adjusting the viewpoint, rotation, and/or shift of the image visualization. In some approaches, when one or more objects are nearer, the viewpoint may be adjusted to “zoom in” more closely to the electronic device (e.g., vehicle)), an image depicting a magnified view of the area of interest in the virtual environment (Paragraph [0075], the electronic device 322 may identify a region of interest (ROI) in the one or more images (e.g., combined image) ... the electronic device 322 (e.g., image visualization adjuster 336) may generate a magnifying window on the display 340 that presents a zoomed version of the ROI).
However, Guo does not specifically disclose a virtual camera and based at least on zooming-in the virtual camera;
determining, using a machine learning model to analyze the image, a location of the object; and
causing a machine to manipulate the object based at least on the location.
In additional, Bhatnagar discloses (Abstract, while a view of a three-dimensional environment is visible via a display generation component, a computer system automatically detects an object in the three-dimensional environment ...; FIG. 7B; paragraph [0229], a view of a three-dimensional environment 7000' ...; paragraph [0230], application user interface 7030 includes one or more user interface elements, including a search field 7036 ...) a virtual camera (Paragraph [0047], a computer system displays a ray extending in the respective direction away from a reference point associated with the user, such as the user's viewpoint, and displays a cursor moving automatically along the ray. In response to one or more additional inputs from the user to stop the movement of the cursor, the cursor is stopped at a particular position along the ray, and a target corresponding to the particular position of the cursor is selected for further interaction) and based at least on zooming-in the virtual camera (FIG. 7C; paragraph [0232], Virtual magnifier 7032 includes slider 7034, which is a user interface element for adjusting a zoom level of the virtual magnifier 7032. A user can increase or decrease the zoom or magnification level by interacting (e.g., directly, or indirectly) with slider 7034 ... If the zoom level is decreased in response to user's 7002 input directed at slider 7034, the size of the displayed search field and/or other user interface elements (displayed by or within virtual magnifier 7032) decreases in accordance with the decreased zoom level. Thus, the second zoom factor is less than the first zoom factor when the zoom level is decreased in response to user's 7002 input; paragraph [00232], if the zoom level is decreased in response to user's 7002 input directed at slider 7034, the size of the displayed search field and/or other user interface elements (displayed by or within virtual magnifier 7032) decreases in accordance with the decreased zoom level. Thus, the second zoom factor is less than the first zoom factor when the zoom level is decreased in response to user's 7002 input).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the system and method for producing an image visualization taught by Guo incorporate the teachings of Bhatnagar, and applying the system viewing a three-dimensional environment taught by Bhatnagar to provide the virtual camera in the virtual environment and use zoom-in the virtual camera to view the virtual 3D representation. Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify Guo according to the relied-upon teachings of Bhatnagar to obtain the invention as specified in claim.
However, the combination of Guo in view of Bhatnagar does not specifically disclose determining, using a machine learning model to analyze the image, a location of the object; and
causing a machine to manipulate the object based at least on the location.
In additional, KUMAR discloses (Paragraph [0050], the user could obtain the model (e.g., the cryo-EM model described herein) and plug it into supported data acquisition software stacks; FIG. 1; paragraph [0071], sample 130 being captured by microscope 110 may result in the images of a cryo-EM grid including holes within it. As described herein, a “grid” or “cryo-EM grid” refers to a substrate with which cryo-EM samples (e.g., samples 130) are placed for imaging ...; paragraph [0026], Cryo-EM's ability to elucidate 3D structures of previously uncharacterized, complex molecules has made it a powerful tool to define binding modes of novel compounds in protein targets) determining, using a machine learning model to analyze the image (Paragraph [0071], first images 112 captured at a first magnification level ... the first image montage comprising first images 112 may depict a cryogenic electron microscopy grid comprising a plurality of cryogenic electron microscopy grid units; paragraph [0072], the first image montage comprising first images 112 may be input to first ML model 122 ...;), a location of the object (Paragraph [0076], the first machine learning model (e.g., first ML model 122) may be configured to determine a predicted location of a center of each of the cryogenic electron microscopy grid units within a corresponding training image from the first plurality of training images); and
causing a machine (Paragraph [0040], computing system 120 may execute these machine learning models utilizing one or more processing devices (e.g., computing system 1500 discussed below with respect to FIG. 15) that may include hardware (e.g., a general purpose processor, a graphic processing unit (GPU) ...) to manipulate the object based at least on the location (Paragraph [0073], computing system 120 may be configured to detect, using first ML model 122, one or more cryogenic electron microscopy units (e.g., identified squares 132) that satisfy a first quality condition. In one or more example, the cryo-EM grid units may use first ML model 122 to generate a first score indicating a quality of the cryogenic electron microscopy grid units).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the system and method for producing an image visualization taught by Guo in view of Bhatnagar incorporate the teachings of KUMAR, and applying the computing system taught by KUMAR to provide machine learning models for predicting the location in the environment and perform the control operations associated with the object. Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify Guo in view of Bhatnagar according to the relied-upon teachings of KUMAR to obtain the invention as specified in claim.
Regarding claim 22, the combination of Guo in view of Bhatnagar in view of KUMAR discloses everything claimed as applied above (see claim 21).
However, Guo does not specifically disclose wherein the determining the area of interest comprises:
applying a second image depicting the virtual environment to a second machine learning model;
determining, using the second machine learning model, a predicted location of the object; and
determining the area of interest based at least on the predicted location.
In additional, KUMAR discloses wherein the determining the area of interest comprises:
applying a second image depicting the virtual environment to a second machine learning model (FIG. 1; paragraphs [0078]-[0079], second images 114 may be captured by microscope 110 at a second magnification level ... computing system 120 may be configured to execute a second machine learning (ML) model 124. The second image montage comprising second images 114 may be input to second ML model 124);
determining, using the second machine learning model, a predicted location of the object (Paragraph [0085], training the second machine learning model may include inputting each of the second plurality of training images to the second machine learning model to obtain an output of a predicted location of a center of each of the apertures (e.g., the holes), or a predicted boundary of each of the apertures (e.g., the holes)); and
determining the area of interest based at least on the predicted location (Paragraph [0079], second ML model 124 may be configured to detect one or more apertures 134 within the cryogenic electron microscopy grid unit that satisfy a second quality condition based on the second image montage. As described herein, a “hole” refers to an aperture within the squares of the cryo-EM grid in which vitreous ice and protein are suspended).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the system and method for producing an image visualization taught by Guo in view of Bhatnagar incorporate the teachings of KUMAR, and applying the computing system taught by KUMAR to provide machine learning models for predicting the location in the environment and perform the control operations associated with the object. Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify Guo in view of Bhatnagar according to the relied-upon teachings of KUMAR to obtain the invention as specified in claim.
Regarding claim 23, the combination of Guo in view of Bhatnagar in view of KUMAR discloses everything claimed as applied above (see claim 21).
However, Guo does not specifically disclose further comprising:
generating, using a second virtual camera and based at least on zooming-in the second virtual camera to magnify a second view of the area of interest, a second image depicting a second magnified view of the area of interest from a different perspective than the image,
wherein the determining the location of the object is based at least on using the machine learning model to analyze the image and the second image.
In additional, Bhatnagar discloses further comprising:
generating, using a second virtual camera (Paragraph [0068], a viewpoint of a user determines what content is visible in the viewport, a viewpoint generally specifies a location and a direction relative to the three-dimensional environment, and as the viewpoint shifts, the view of the three-dimensional environment will also shift in the viewport. For a head mounted device, a viewpoint is typically based on a location and direction of the head, face, and/or eyes of a user to provide a view of the three-dimensional environment that is perceptually accurate and provides an immersive experience when the user is using the head-mounted device. For a handheld or stationed device, the viewpoint shifts as the handheld or stationed device is moved and/or as a position of a user relative to the handheld or stationed device changes (e.g., a user moving toward, away from, up, down, to the right, and/or to the left of the device) ... the viewpoint of the user moves as the field of view of the one or more cameras moves (and the appearance of one or more virtual objects displayed via the one or more display generation components is updated based on the viewpoint of the user (e.g., displayed positions and poses of the virtual objects are updated based on the movement of the viewpoint of the user)); paragraph [0223], ... changes the viewpoint of the user in the three-dimensional environment in accordance with the movement of the display generation component relative to the user's head or face or relative to the physical environment) and based at least on zooming-in the second virtual camera to magnify a second view of the area of interest (FIG. 7B; paragraph [0229], a view of a three-dimensional environment 7000' ...; paragraph [0230], application user interface 7030 includes one or more user interface elements, including a search field 7036 and control 7038 ...; paragraph [0234], virtual magnifier 7032 is moved from left to right while the distance between the virtual magnifier 7032 and application user interface 7030 remains the same. The second portion of application user interface 7030 that is magnified incudes a magnified version 7038′ of control 7038 for initiating a video call), a second image depicting a second magnified view of the area of interest from a different perspective than the image (Paragraph [0231], virtual magnifier 7032 magnifies portions of the view of the three-dimensional environment 7000′ on which user 7002 is focused, where the focus of the user can be determined in a number of ways (e.g., a location of a gaze, a cursor, a location pointed at by a controller, the user's field of view, or other ways); paragraph [0234], FIG. 7D shows the virtual magnifier 7032 at a position different from the position at which the virtual magnifier 7032 was in FIG. 7C ... The second portion of application user interface 7030 that is magnified incudes a magnified version 7038′ of control 7038 for initiating a video call).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the system and method for producing an image visualization taught by Guo incorporate the teachings of Bhatnagar, and applying the system viewing a three-dimensional environment taught by Bhatnagar to provide the virtual camera in the virtual environment and use zoom-in the virtual camera to view the virtual 3D representation. Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify Guo according to the relied-upon teachings of Bhatnagar to obtain the invention as specified in claim.
However, Guo does not specifically disclose wherein the determining the location of the object is based at least on using the machine learning model to analyze the image and the second image.
In additional, KUMAR discloses wherein the determining the location of the object is based at least on using the machine learning model to analyze the image and the second image (FIG. 1; paragraph [0072], the first image montage comprising first images 112 may be input to first ML model 122. First ML model 122 may be configured to detect one or more cryogenic electron microscopy grid units (e.g., squares) that satisfy a first quality condition based on the first image montage ... as shown in FIG. 1, the identified squares 132 is provided to microscope 110 with instructions to capture second images 114; paragraph [0079], the second image montage comprising second images 114 may be input to second ML model 124. Second ML model 124 may be configured to detect one or more apertures 134 within the cryogenic electron microscopy grid unit that satisfy a second quality condition based on the second image montage. As described herein, a “hole” refers to an aperture within the squares of the cryo-EM grid in which vitreous ice and protein are suspended).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the system and method for producing an image visualization taught by Guo in view of Bhatnagar incorporate the teachings of KUMAR, and applying the computing system taught by KUMAR to provide machine learning models for predicting the location in the environment and perform the control operations associated with the object. Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify Guo in view of Bhatnagar according to the relied-upon teachings of KUMAR to obtain the invention as specified in claim.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Xilin Guo whose telephone number is (571)272-5786. The examiner can normally be reached Monday - Friday 9:00 AM-5:30 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Hajnik can be reached at 571-272-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/XILIN GUO/ Primary Examiner, Art Unit 2616
Prosecution Insights