DETAILED ACTION
Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
2. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
3. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
4. Claims 1-2, 4-9, 11-16, 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Hettige (Hettige, Kethmi Hirushini, et al. "Airphynet: Harnessing physics-guided neural networks for air quality prediction." arXiv preprint arXiv:2402.03784 (2024)), hereinafter Hettige, in view of Appel (Appel, Marius. "Efficient data-driven gap filling of satellite image time series using deep neural networks with partial convolutions." arXiv preprint arXiv:2208.08781 (2022)), hereinafter Appel.
Regarding claim 1, Hettige teaches a computer-implemented method for dynamic image reconstruction, the method comprising: training a physics-informed neural network (PINN) using received training data (Section 4.1-4.2, using training data for training a neural network; Section 1 paragraphs 4-5, wherein the neural network is defined as a physics guided neural network, which is interpreted as a PINN); receiving wind field velocity measurements in a geographical domain from one or more weather information sources (Section 4.1, wherein input datasets is obtained from air quality and weather monitoring stations which includes wind speed and wind direction, which is interpreted as wind field velocity measurements) and air pollutant concentration measurements of one or more air pollutants from the geographical domain using a sparse sensor network to produce a plurality of input data (Section 4.1, wherein input datasets is obtained from air quality monitoring stations which measure concentrations of major pollutants, and datasets having missing values suggests it is obtained with a sparse sensor network); and performing the dynamic image reconstruction on the processed plurality of input data using the trained PINN to generate a reconstructed continuous spatial map of the geographical domain (Section 4.5, Figs 3-4, wherein visualizing a heatmap of predicted pollutants and wind direction outputted by the neural network over multiple timesteps is interpreted as performing dynamic image reconstruction to generate a reconstructed continuous spatial map of the geographical domain).
Hettige does not teach processing the plurality of input data through multiple physics-informed masked convolutional layers in the trained PINN to perform a three-dimensional partial convolution process.
Appel teaches processing the plurality of input data through multiple physics-informed masked convolutional layers in the trained PINN to perform a three-dimensional partial convolution process (Section 2.2, Fig. 1, processing three-dimensional input through a masked partial convolution process, wherein the neural network is defined as being capable of mapping physical properties of processes, which is interpreted as a PINN).
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hettige to incorporate the teachings of Appel for this method for image reconstruction for pollutants and wind field measurements. Hettige discusses a physics guided neural network that takes in wind velocity and air pollutant measurements as inputs in order to more accurately generate air quality forecasts. Appel discusses a neural network for mapping physical processes which utilizes three-dimensional partial convolutions in order to fill in gaps of data, in order to facilitate more accurate and computationally efficient predictions with sparser data. While the neural network described in Appel is not explicitly stated to be a physics informed neural network, Appel discusses using neural networks for modelling physical properties such as advection and diffusion, and modeling carbon monoxide, a common pollutant. Similarly, Hettige describes a physics guided neural network for modelling advection and diffusion, as well as air quality and air pollutants. As both references discuss neural networks capable of modelling similar processes, it would be obvious to combine them.
Regarding claim 2, Hettige in view of Appel discloses the method of claim 1. Additionally, Appel teaches the method of claim 1, wherein performing the three-dimensional partial convolution process further comprises: applying a three-dimensional partial convolution mask to the plurality of input data during each pass through each of the multiple physics-informed masked convolutional layers (Section 2.1-2.2, wherein a binary mask with the same shape as the three-dimensional input is applied for each partial convolutional layer); and updating the three-dimensional partial convolution mask during each pass of the plurality of input data through each of the multiple physics-informed masked convolutional layers (Section 2.1-2.2, Fig. 1, wherein the partial convolution mask is updated as it’s passed through the network alongside the input).
The motivation to combine would be the same as that set forth for claim 1.
Regarding claim 4, Hettige in view of Appel discloses the method of claim 1. Additionally, Hettige teaches the method of claim 1, wherein training the PINN using the received training data comprises propagating a physics-informed neural network loss contribution through the PINN (Section 3.4, wherein the physics guided neural network is trained through back-propagation by minimizing the mean absolute error).
Regarding claim 5, Hettige in view of Appel discloses the method of claim 1. Additionally, Hettige teaches the method of claim 1, wherein performing the three-dimensional partial convolution process further comprises: generating an emissions characteristic vector (Section 3.4, paragraph 6, wherein the output from the neural network representing the latent space for generating predictions on pollutants is interpreted as generating emissions characteristics, and the output can be reshaped to produce a desired output format with the correct dimensionality suggests the output can be a vector), wherein the emissions characteristic vector represents one or more source locations of the one or more air pollutants in the geographical domain and emission magnitudes of the one or more air pollutants at one or more locations in the geographical domain (Fig. 3-4, wherein the heatmap visualized from the generated predictions represent concentrations of air pollutants and emission magnitudes in the geographical domain).
Regarding claim 6, Hettige in view of Appel disclose the method of claim 1. Additionally, Hettige teaches the method of claim 1, wherein the PINN comprises: an encoder and two decoders (Fig. 1, wherein the physics guided neural network consists of an encoder and decoder; Section 3.3, wherein wind features are transformed using a multi-layer perceptron (MLP) as part of the neural network, and receiving output from an MLP suggests the MLP is another decoder); and wherein another one of the two decoders comprises a multi-layered perceptron (Section 3.3, Fig. 1, wherein wind features are transformed using an MLP as part of the neural network, and receiving output from an MLP suggests the MLP is a decoder).
Hettige does not teach wherein the encoder and one of the two decoders comprise the multiple physics-informed masked convolutional layers.
Appel teaches wherein the encoder and one of the two decoders comprise the multiple physics-informed masked convolutional layers (Section 2.2, Fig. 1, wherein the model architecture consists of partial convolutional layers of encoder and decoder parts).
The motivation to combine would be the same as that set forth for claim 1.
Regarding claim 7, Hettige in view of Appel discloses the method of claim 1. Additionally, Hettige teaches the method of claim 1, wherein training the PINN using the received training data comprises incorporating atmospheric diffusion into the multiple physics-informed masked convolutional layers using an Advection-Diffusion Equation (Section 3.3, Fig. 1, wherein a Diffusion-Advection Differential Equation Function is used as part of the process of incorporating diffusion into the convolution process of the physics guided neural network).
Regarding claim 8, Hettige teaches a computer system for dynamic image reconstruction, the computer system comprising: one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage medium, and program instructions stored on at least one of the one or more tangible storage medium (Section 4.2, wherein the neural network model is implemented using a GPU and via PyTorch, which suggests being run on a non-transitory computer-readable storage medium) for execution by at least one of the one or more processors via at least one of the one or more memories, wherein the computer system is capable of performing a method comprising: training a physics-informed neural network (PINN) using received training data (Section 4.1-4.2, using training data for training a neural network; Section 1 paragraphs 4-5, wherein the neural network is defined as a physics guided neural network, which is interpreted as a PINN); receiving wind field velocity measurements in a geographical domain from one or more weather information sources (Section 4.1, wherein input datasets is obtained from air quality and weather monitoring stations which includes wind speed and wind direction, which is interpreted as wind field velocity measurements) and air pollutant concentration measurements of one or more air pollutants from the geographical domain using a sparse sensor network to produce a plurality of input data (Section 4.1, wherein input datasets is obtained from air quality monitoring stations which measure concentrations of major pollutants, and datasets having missing values suggests it is obtained with a sparse sensor network); and performing the dynamic image reconstruction on the processed plurality of input data using the trained PINN to generate a reconstructed continuous spatial map of the geographical domain (Section 4.5, Figs 3-4, wherein visualizing a heatmap of predicted pollutants and wind direction outputted by the neural network over multiple timesteps is interpreted as performing dynamic image reconstruction to generate a reconstructed continuous spatial map of the geographical domain).
Hettige does not teach processing the plurality of input data through multiple physics-informed masked convolutional layers in the trained PINN to perform a three-dimensional partial convolution process.
Appel teaches processing the plurality of input data through multiple physics-informed masked convolutional layers in the trained PINN to perform a three-dimensional partial convolution process (Section 2.2, Fig. 1, processing three-dimensional input through a masked partial convolution process, wherein the neural network is defined as being capable of mapping physical properties of processes, which is interpreted as a PINN).
The motivation to combine would be the same as that set forth for claim 1.
Regarding claim 9, Hettige in view of Appel discloses the system of claim 8. Additionally, Appel teaches the system of claim 8, wherein performing the three-dimensional partial convolution process further comprises: applying a three-dimensional partial convolution mask to the plurality of input data during each pass through each of the multiple physics-informed masked convolutional layers (Section 2.1-2.2, wherein a binary mask with the same shape as the three-dimensional input is applied for each partial convolutional layer); and updating the three-dimensional partial convolution mask during each pass of the plurality of input data through each of the multiple physics-informed masked convolutional layers (Section 2.1-2.2, Fig. 1, wherein the partial convolution mask is updated as it’s passed through the network alongside the input).
The motivation to combine would be the same as that set forth for claim 1.
Regarding claim 11, Hettige in view of Appel discloses the system of claim 8. Additionally, Hettige teaches the system of claim 8, wherein training the PINN using the received training data comprises propagating a physics-informed neural network loss contribution through the PINN (Section 3.4, wherein the physics guided neural network is trained through back-propagation by minimizing the mean absolute error).
Regarding claim 12, Hettige in view of Appel discloses the system of claim 8. Additionally, Hettige teaches the system of claim 8, wherein performing the three-dimensional partial convolution process further comprises: generating an emissions characteristic vector (Section 3.4, paragraph 6, wherein the output from the neural network representing the latent space for generating predictions on pollutants is interpreted as generating emissions characteristics, and the output can be reshaped to produce a desired output format with the correct dimensionality suggests the output can be a vector), wherein the emissions characteristic vector represents one or more source locations of the one or more air pollutants in the geographical domain and emission magnitudes of the one or more air pollutants at one or more locations in the geographical domain (Fig. 3-4, wherein the heatmap visualized from the generated predictions represent concentrations of air pollutants and emission magnitudes in the geographical domain).
Regarding claim 13, Hettige in view of Appel disclose the system of claim 8. Additionally, Hettige teaches the system of claim 8, wherein the PINN comprises: an encoder and two decoders (Fig. 1, wherein the physics guided neural network consists of an encoder and decoder; Section 3.3, wherein wind features are transformed using a multi-layer perceptron (MLP) as part of the neural network, and receiving output from an MLP suggests the MLP is another decoder); and wherein another one of the two decoders comprises a multi-layered perceptron (Section 3.3, Fig. 1, wherein wind features are transformed using an MLP as part of the neural network, and receiving output from an MLP suggests the MLP is a decoder).
Hettige does not teach wherein the encoder and one of the two decoders comprise the multiple physics-informed masked convolutional layers.
Appel teaches wherein the encoder and one of the two decoders comprise the multiple physics-informed masked convolutional layers (Section 2.2, Fig. 1, wherein the model architecture consists of partial convolutional layers of encoder and decoder parts).
The motivation to combine would be the same as that set forth for claim 1.
Regarding claim 14, Hettige in view of Appel discloses the system of claim 8. Additionally, Hettige teaches the system of claim 8, wherein training the PINN using the received training data comprises incorporating atmospheric diffusion into the multiple physics-informed masked convolutional layers using an Advection-Diffusion Equation (Section 3.3, Fig. 1, wherein a Diffusion-Advection Differential Equation Function is used as part of the process of incorporating diffusion into the convolution process of the physics guided neural network).
Regarding claim 15, Hettige teaches a computer program product for dynamic image reconstruction, the computer program product comprising: one or more computer-readable tangible storage medium, and program instructions stored on at least one of the one or more tangible storage medium, the program instructions executable by a processor (Section 4.2, wherein the neural network model is implemented using a GPU and via PyTorch, which suggests being run on a non-transitory computer-readable storage medium) to cause the processor to perform a method comprising: training a physics-informed neural network (PINN) using received training data (Section 4.1-4.2, using training data for training a neural network; Section 1 paragraphs 4-5, wherein the neural network is defined as a physics guided neural network, which is interpreted as a PINN); receiving wind field velocity measurements in a geographical domain from one or more weather information sources (Section 4.1, wherein input datasets is obtained from air quality and weather monitoring stations which includes wind speed and wind direction, which is interpreted as wind field velocity measurements) and air pollutant concentration measurements of one or more air pollutants from the geographical domain using a sparse sensor network to produce a plurality of input data (Section 4.1, wherein input datasets is obtained from air quality monitoring stations which measure concentrations of major pollutants, and datasets having missing values suggests it is obtained with a sparse sensor network); and performing the dynamic image reconstruction on the processed plurality of input data using the trained PINN to generate a reconstructed continuous spatial map of the geographical domain (Section 4.5, Figs 3-4, wherein visualizing a heatmap of predicted pollutants and wind direction outputted by the neural network over multiple timesteps is interpreted as performing dynamic image reconstruction to generate a reconstructed continuous spatial map of the geographical domain).
Hettige does not teach processing the plurality of input data through multiple physics-informed masked convolutional layers in the trained PINN to perform a three-dimensional partial convolution process.
Appel teaches processing the plurality of input data through multiple physics-informed masked convolutional layers in the trained PINN to perform a three-dimensional partial convolution process (Section 2.2, Fig. 1, processing three-dimensional input through a masked partial convolution process, wherein the neural network is defined as being capable of mapping physical properties of processes, which is interpreted as a PINN).
The motivation to combine would be the same as that set forth for claim 1.
Regarding claim 16, Hettige in view of Appel discloses the computer program product of claim 15. Additionally, Appel teaches the computer program product of claim 15, wherein performing the three-dimensional partial convolution process further comprises: applying a three-dimensional partial convolution mask to the plurality of input data during each pass through each of the multiple physics-informed masked convolutional layers (Section 2.1-2.2, wherein a binary mask with the same shape as the three-dimensional input is applied for each partial convolutional layer); and updating the three-dimensional partial convolution mask during each pass of the plurality of input data through each of the multiple physics-informed masked convolutional layers (Section 2.1-2.2, Fig. 1, wherein the partial convolution mask is updated as it’s passed through the network alongside the input).
The motivation to combine would be the same as that set forth for claim 1.
Regarding claim 18, Hettige in view of Appel discloses the computer program product of claim 15. Additionally, Hettige teaches the computer program product of claim 15, wherein training the PINN using the received training data comprises propagating a physics-informed neural network loss contribution through the PINN (Section 3.4, wherein the physics guided neural network is trained through back-propagation by minimizing the mean absolute error).
Regarding claim 19, Hettige in view of Appel discloses the computer program product of claim 15. Additionally, Hettige teaches the computer program product of claim 15, wherein performing the three-dimensional partial convolution process further comprises: generating an emissions characteristic vector (Section 3.4, paragraph 6, wherein the output from the neural network representing the latent space for generating predictions on pollutants is interpreted as generating emissions characteristics, and the output can be reshaped to produce a desired output format with the correct dimensionality suggests the output can be a vector), wherein the emissions characteristic vector represents one or more source locations of the one or more air pollutants in the geographical domain and emission magnitudes of the one or more air pollutants at one or more locations in the geographical domain (Fig. 3-4, wherein the heatmap visualized from the generated predictions represent concentrations of air pollutants and emission magnitudes in the geographical domain).
Regarding claim 20, Hettige in view of Appel disclose the computer program product of claim 15. Additionally, Hettige teaches the computer program product of claim 15, wherein the PINN comprises: an encoder and two decoders (Fig. 1, wherein the physics guided neural network consists of an encoder and decoder; Section 3.3, wherein wind features are transformed using a multi-layer perceptron (MLP) as part of the neural network, and receiving output from an MLP suggests the MLP is another decoder); and wherein another one of the two decoders comprises a multi-layered perceptron (Section 3.3, Fig. 1, wherein wind features are transformed using an MLP as part of the neural network, and receiving output from an MLP suggests the MLP is a decoder).
Hettige does not teach wherein the encoder and one of the two decoders comprise the multiple physics-informed masked convolutional layers.
Appel teaches wherein the encoder and one of the two decoders comprise the multiple physics-informed masked convolutional layers (Section 2.2, Fig. 1, wherein the model architecture consists of partial convolutional layers of encoder and decoder parts).
The motivation to combine would be the same as that set forth for claim 1.
Allowable Subject Matter
5. Claims 3, 10, and 17 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
6. Any inquiry concerning this communication or earlier communications from the examiner should be directed to JORDAN W YICK whose telephone number is (571)272-4063. The examiner can normally be reached M-F 8-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Said Broome can be reached at (571) 272-2931. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JORDAN WAN YICK/Examiner, Art Unit 2612
/Said Broome/Supervisory Patent Examiner, Art Unit 2612