DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Responsive to communication filed on 20 Apr 2023.
Claims 1-20 are presented for examination.
Information Disclosure Statement
The Information Disclosure Statement dated 20 Apr 2023 has been reviewed. See attached.
Drawings
The drawings dated 20 Apr 2023 have been reviewed. They are accepted.
Specification
The abstract dated 20 Apr 2023 has been reviewed. It contains 136 words and 10 lines and has no legal phraseology. It is accepted.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception without significantly more. The following section follows the 2019 Patent Eligibility Guidance (PEG) for analyzing subject matter eligibility:
Step 1 - Statutory Category:
Step 1 of the PEG analysis entails considering whether the claimed subject matter falls within
the four statutory categories of patentable subject matter identified by 35 U.S.C. 101 (process, machine,
manufacture, or composition of matter).
Step 2A Prong One - Judicial exception:
In Step 2A Prong 1, examiners evaluate whether the claim recites a judicial exception (an
abstract idea, law of nature, or a natural phenomenon).
Step 2A Prong Two - Integration into a practical application:
If claims recite a judicial exception, the claim requires further analysis in Step 2A Prong 2. In Step
2A Prong 2, examiners evaluate whether the claim as a whole integrates the exception into a practical application. This evaluation considers any additional elements in the claim beyond any recited judicial
exceptions.
Step 2B - Significantly More:
If the additional elements identified in Step 2A Prong 2 do not integrate the exception into a
practical application, then the claim is directed to the recited judicial exception and requires further
analysis under Step 2B- Significantly More. This evaluation is to evaluate if the additional elements of
the claim provide an inventive concept.
As noted in the MPEP 2106.05(II): The identification of the additional element(s) in the claim
from Step 2A Prong 2, as well as the conclusions from Step 2A Prong 2 on the considerations discussed
in MPEP 2106.05(a) -(c), (e), (f), and (h) are to be carried over. Claim limitations identified as Insignificant
Extra-Solution Activities are re-evaluated to determine if the elements are beyond what is well -
understood, routine, and conventional (WURC) activity, as dictated by MPEP 2106.05(II).
The additional elements are evaluated to determine if any additional element or combination of
elements are other than what is well-understood, routine, conventional activity in the field, or simply
append well-understood, routine, conventional activities previously known to the industry, specified at a
high level of generality, to the judicial exception, per MPEP § 2106.05(d).
Claims 1-4, 6-10
Claim 1:
Step 1: Claim 1 and its dependent claims 2-10 are direct to a system which falls within one of the four statutory categories of a machine.
Step 2A Prong 1: Claim 1 recites a judicial exception, noted in bold:
“stylize the content, This limitation can be reasonably read to entail a human applying a certain style to content. This task can be practically performed in the human mind or using an assistive physical aid. Therefore, this claim limitation includes the recitation of the judicial exception of abstract ideas of a mental process.
“output the stylized content having the desired stylization.” This limitation can be reasonably read to entail a human producing a piece of stylized art with a pen and paper. This task can be practically performed in the human mind or using an assistive physical aid. Therefore, this claim limitation includes the recitation of the judicial exception of abstract ideas of a mental process.
Step 2A Prong 2: Claim 1 additionally recites the limitations:
“a hardware processor;” This limitation has been identified as mere instructions to apply the judicial exception (MPEP 2106.05(f)).
“a system memory storing a software code and a machine learning (ML) model trained to apply a stylization to an image;” This limitation has been identified as mere instructions to apply the judicial exception (MPEP 2106.05(f)).
“receive a first sequence of images and style data describing a desired stylization of content depicted by the first sequence of images;” This limitation has been identified as insignificant extra-solution activity as it is mere data gathering (MPEP 2106.05(g)).
“using the ML model” This limitation has been identified as mere instructions to apply the judicial exception (MPEP 2106.05(f)).
“wherein stylizing includes applying an exponential moving average (EMA) temporal smoothing algorithm to sequential image pairs of the first sequence of images to generate a second sequence of images providing a depiction of the content having the desired stylization;” This limitation has been identified as mere instructions to apply the judicial exception (MPEP 2106.05(f)).
Step 2B: The courts have found that limitations that amount to insignificant extra-solution activity are not enough to qualify the claim as significantly more than the abstract idea. The limitation, identified above as being insignificant extra-solution activity, is directed to mere data gathering (MPEP 2106.05(g)(3)) as the limitation merely recites to gather a sequence of images and a desired style to apply to the images. Therefore, the claim does not include additional elements, alone or in the ordered combination that are sufficient to amount to significantly more than the recited judicial exception.
Claim 2:
Step 1: Regarding dependent claim 2, the judicial exception of independent claim 1 is further incorporated. The claim falls within the corresponding statutory category as stated previously.
Step 2A Prong 1: No further judicial limitations have been incorporated in claim 2.
Step 2A Prong 2: Claim 2 additionally recites the limitation “the first sequence of images comprise two-dimensional (2D) images, and wherein the stylized content is three-dimensional (3D)”. This limitation has been identified as Field of Use and Technological Environment (MPEP 2106.05(h)) as it is simply limiting the input images to being 2D and the output to being 3D.
Step 2B: As there are no limitations that fall within the Insignificant Extra-Solution Activity category, no further evaluation is required at this step.
Claim 3:
Step 1: Regarding dependent claim 3, the judicial exception of independent claim 1 is further incorporated. The claim falls within the corresponding statutory category as stated previously.
Step 2A Prong 1: No further judicial limitations have been incorporated in claim 3.
Step 2A Prong 2: Claim 3 additionally recites the limitation “the ML model comprises a neural network”. This limitation has been identified as Field of Use and Technological Environment (MPEP 2106.05(h)) as it is simply limiting the ML model to one that contains a neural network.
Step 2B: As there are no limitations that fall within the Insignificant Extra-Solution Activity category, no further evaluation is required at this step.
Claim 4:
Step 1: Regarding dependent claim 4, the judicial exception of independent claim 1 is further incorporated. The claim falls within the corresponding statutory category as stated previously.
Step 2A Prong 1: Claim 4 recites a judicial exception, noted in bold:
“stylizing further comprises use of a transport function having a first-order Euler integrator” This limitation can reasonably seen as use of a mathematical calculation (the Euler method; see reference “Euler's Method for First-Order ODE” in conclusion. Therefore, this claim limitation includes the recitation of the judicial exception of use of a mathematical calculation.
Step 2A Prong 2: All claim limitations have been shown to be a judicial exception, thus there are no additional elements that would integrate the claim into a practical application.
Step 2B: As there are no limitations that fall within the Insignificant Extra-Solution Activity category, no further evaluation is required at this step.
Claim 6:
Step 1: Regarding dependent claim 6, the judicial exception of independent claim 1 is further incorporated. The claim falls within the corresponding statutory category as stated previously.
Step 2A Prong 1: No further judicial limitations have been incorporated in claim 6.
Step 2A Prong 2: Claim 6 additionally recites the limitation “stylizing limits modulations to an input density of image content included in each of the first sequence of images to multiplication by a scaling factor”. This limitation has been identified as Mere Instructions to Apply an Exception (MPEP 2106.05(f)) as it is simply applying multiplication by a scaling factor to the input density of image content to stylize the images.
Step 2B: As there are no limitations that fall within the Insignificant Extra-Solution Activity category, no further evaluation is required at this step.
Claim 7:
Step 1: Regarding dependent claim 7, the judicial exception of independent claim 1 is further incorporated. The claim falls within the corresponding statutory category as stated previously.
Step 2A Prong 1: Claim 7 recites a judicial exception, noted in bold:
“transform the second sequence of images, This limitation can be reasonably read to entail a human, using a pen and paper, transforming (e.g. stylizing, re-sizing, etc.) the images to a separate sequence of images. This task can be practically performed in the human mind or using an assistive physical aid. Therefore, this claim limitation includes the recitation of the judicial exception of abstract ideas of a mental process.
Step 2A Prong 2: Claim 7 additionally recites the limitations:
“using another ML model”. This limitation has been identified as Mere Instructions to Apply an Exception (MPEP 2106.05(f)).
“the stylized content includes the view-independent sequence of images”. This limitation has been identified as Mere Instructions to Apply an Exception (MPEP 2106.05(f)).
Step 2B: As there are no limitations that fall within the Insignificant Extra-Solution Activity category, no further evaluation is required at this step.
Claim 8:
Step 1: Regarding dependent claim 8, the judicial exception of independent claim 1 is further incorporated. The claim falls within the corresponding statutory category as stated previously.
Step 2A Prong 1: No further judicial limitations have been incorporated in claim 8.
Step 2A Prong 2: Claim 8 additionally recites the limitation “the another ML model comprises a feed- forward convolutional neural network”. This limitation has been identified as Field of Use and Technological Environment (MPEP 2106.05(h)) as it is simply limiting the ML model to one that contains a feed-forward convolutional neural network.
Step 2B: As there are no limitations that fall within the Insignificant Extra-Solution Activity category, no further evaluation is required at this step.
Claim 9:
Step 1: Regarding dependent claim 9, the judicial exception of independent claim 1 is further incorporated. The claim falls within the corresponding statutory category as stated previously.
Step 2A Prong 1: No further judicial limitations have been incorporated in claim 9.
Step 2A Prong 2: Claim 9 additionally recites the limitation “the content comprises a simulation of at least one of a fluid or a suspension of airborne particulates, in motion”. This limitation has been identified as Field of Use and Technological Environment (MPEP 2106.05(h)) as it is simply limiting the content to a specific field.
Step 2B: As there are no limitations that fall within the Insignificant Extra-Solution Activity category, no further evaluation is required at this step.
Claim 10:
Step 1: Regarding dependent claim 10, the judicial exception of independent claim 1 is further incorporated. The claim falls within the corresponding statutory category as stated previously.
Step 2A Prong 1: No further judicial limitations have been incorporated in claim 10.
Step 2A Prong 2: Claim 10 additionally recites the limitation “the content comprises a simulation of at least one of a fluid or a suspension of airborne particulates, in motion”. This limitation has been identified as Field of Use and Technological Environment (MPEP 2106.05(h)) as it is simply limiting the content to a specific field.
Step 2B: As there are no limitations that fall within the Insignificant Extra-Solution Activity category, no further evaluation is required at this step.
Claims 11-14, 16-20:
Claims 11-14, 16-20 are direct to a method, which falls within one of the four statutory categories of a process.
Claims 11-14, 16-20 are similar in claim structure and limitations to claims 1-4, 6-10. Thus the rationale for rejection for claims 1-4, 6-10 will apply to claims 11-14, 16-20.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 6, 9-14, 16, 19, 20 are rejected under 35 U.S.C. 103 as being unpatentable over Kim_2019 (“Transport-Based Neural Style Transfer of Smoke Simulations”) in view of Kozlowski_2023 (US 11847737 B2).
Claim 1. Kim_2019 teaches “A system comprising: a hardware processor; and a system memory storing a software code (Col. 3 Line 2: “The system further comprises a display module, a processor, and memory storing one or more programs configured for execution by the processor”) and a machine learning (ML) model rained to apply a stylization to an image; (Page 4 Col 1: “Our method employs pre-trained CNNs for natural image classification as both feature extractor and synthesizer.” NOTE: CNN = Convolutional Neural Network, the examiner reads a neural network as a machine learning model) the hardware processor configured to execute the software code to: receive a first sequence of images (Fig. 3: The top row are the first sequence of images) and style data describing a desired stylization of content depicted by the first sequence of images; (Pg. 5 Col. 2: “… our method allows the incorporation of a given input image style…”) stylize the content, using the ML model, to provide a stylized content having the desired stylization […] to sequential image pairs of the first sequence of images to generate a second sequence of images providing a depiction of the content having the desired stylization; and output the stylized content having the desired stylization.” (Fig. 2: Rows 2 and 3 of this figure display the output of stylized images based on the input images.)
While Kim_2019 teaches a system to stylize a sequence of images and output the results, Kim_2019 does not teach that the stylizing includes “applying an exponential moving average (EMA) temporal smoothing algorithm”.
Kozlowski_2023, however, teaches “applying an exponential moving average (EMA) temporal smoothing algorithm…” (Col. 2 Line 47: “In typical instances, an exponential moving average (EMA) may be utilized to find a value for a pixel based on previous values for the pixel, thus smoothing out the value of the pixel over time…”).
Kim_2019 and Kozlowski_2023 are analogous art because they are from the same field of endeavor called image generation. Before the effective filing date it would have been obvious to a person of ordinary skill in the art to combine Kim_2019 and Kozlowski_2023. The rationale for doing so would have been that the EMA from Kozlowski_2023 is utilized to determine display values for pixels in an image over time based on previous display values of said pixels. This could be utilized in the style transfer of Kim_2019 to assist in creating the stylized images while also decreasing computational time. Therefore, it would have been obvious to one of ordinary skill in the art to combine the stylizing system of Kim_2019 and the EMA of Kozlowski_2023 to create a stylizing system that requires less computing power and memory resources. (Kozlowski_2023 Col. 2 Line 54: “… EMA is a relatively inexpensive averaging method that requires less computing power and memory.”)
Claim 2. Kim_2019 and Kozlowski_2023 teach the limitations of claim 1. Kim_2019 also teaches “the first sequence of images comprise two- dimensional (2D) images, and wherein the stylized content is three-dimensional (3D)” (Pg. 10 Col. 2: “Even though we are using 2D CNNs, our differentiable renderer allows the recreation of 3D volumetric structures from [a] small set of views”).
Claim 3. Kim_2019 and Kozlowski_2023 teach the limitations of claim 1. Kim_2019 further teaches “the ML model comprises a neural network”. (Page 4 Col 1: “Our method employs pre-trained CNNs for natural image classification as both feature extractor and synthesizer.”)
Claim 4. Kim_2019 and Kozlowski_2023 teach the limitations of claim 1. Kim_2019 additionally teaches “stylizing further comprises use of a transport function (Pg. 5 Col. 1: “…the transport function
τ
(
d
,
v
)
advects d by v.” having a first-order Euler integrator.” (Pg. 6 Col. 2: “… we approximate the solution of Equation (10) by first evaluating Equation (4) to find a set of stylization velocities computed for a single frame… This is performed iteratively for all simulation frames of a sequence…” NOTE: This process of approximation is representative of the Euler method for approximating first-order differential equations. The Euler method comprises determining an initial value for a point in time (Pg. 6 Col. 2: “first evaluating Equation (4) to find a set of stylization velocities computed for a single frame”), adding the initial value, along with other values relating to the initial value, to approximate subsequent values (Pg. 6 Col. 2: “Then, we merge the velocities per-frame individually using Equation (9)”), and repeating this process until the desired equation has been approximated for all values (Pg. 6 Col. 2: “This is performed iteratively for all simulation frames of a sequence…”). For further reference, see “Euler’s Method for First-Order ODE” cited in the conclusion).
Claim 6. Kim_2019 and Kozlowski_2023 teach the limitations of claim 1. Kim_2019 also teaches “stylizing limits modulations to an input density of image content included in each of the first sequence of images to multiplication by a factor.” (Pg. 7 Col 2: “The smoke density is linearly mapped to extinction using the scaling factor γ…”).
Claim 9. Kim_2019 and Kozlowski_2023 teach the limitations of claim 1. Kim_2019 further teaches “the content comprises a simulation of at least one of a fluid or a suspension of airborne particulates, in motion.” (Fig. 2).
Claim 10. Kim_2019 and Kozlowski_2023 teach the limitations of claim 9. Kim_2019 additionally teaches “the at least one of the fluid or the suspension of airborne particulates comprises smoke.” (Fig. 2).
Claims 11-14, 16, 19, and 20 are similar in claim structure and limitations to claims 1-4, 6, 9, and 10, thus the rationale for rejection for claims 1-4, 6, 9, and 10 will apply to claims 11-14, 16, 19, and 20, respectively.
Claims 5 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Kim_2019 in view of Kozlowski_2023 in view of Platkevič_2021 (“Fluidymation: Stylizing Animations Using Natural Dynamics of Artistic Media”).
Claim 5. Kim_2019 and Kozlowski_2023 teach the limitations of claim 1.
Kim_2019 and Kozlowski_2023 do not teach “the first sequence of images comprises up to one hundred and sixty images, and wherein the stylized content is output in less than three minutes from receiving the first sequence of images.”
Platkevič_2021, however, teaches “the first sequence of images comprises up to one hundred and sixty images, and wherein the stylized content is output in less than three minutes from receiving the first sequence of images.” (Pg. 19 Table 2)
Kim_2019, Kozlowski_2023, and Platkevič_2021 are analogous art because they are from the same field of endeavor of image manipulation. Before the effective filing date it would have been obvious to a person of ordinary skill in the art to combine Kim_2019, Kozlowski_2023, and Platkevič_2021. The rationale for doing so would have been that stylized content from previous methods do not look natural. (Platkevič_2021 Pg. 22 Col. 1: “Their drawback is that the stylized content looks glued on the moving objects, which breaks the impression of being painted frame by frame.”) Therefore, it would have been obvious to one of ordinary skill in the art that methods to obtain the claimed image limit and time limit are known and may be utilized to achieve said image and time limit.
Claim 15 is similar in claim structure and limitations to claim 5, thus the rationale for rejection for claim 5 will apply to claim 15.
Claims 7, 8, 17, 18 are rejected under 35 U.S.C. 103 as being unpatentable over Kim_2019 in view of Kozlowski_2023 in view of Jaderberg_2015 (“Spatial Transformer Networks”).
Claim 7. Kim_2019 and Kozlowski_2023 teach the limitations of claim 1. Kim_2019 then teaches “the hardware processor is further configured to execute the software code to: transform the second sequence of images, (Pg. 8 Section 5.2: “Although our method is based on 2D representations of the smoke data…”) […] to a view-independent sequence of images; (Pg. 8 Section 5.2: “… we can reliably cover multiple viewing directions without introducing bias towards certain views.”) wherein the stylized content includes the view-independent sequence of images.” (Pg. 8 Section 5.2: “Figure 3 shows the bunny smoke example stylized with a spiral pattern from different viewpoints.”)
Kim_2019 and Kozlowski_2023 do not explicitly teach that the view-independent sequence of images are being created with “another ML model”.
Jaderberg_2015, however, teaches “another ML model” (Pg. 8 Sect. 4.3: “We then train a spatial transformer network, ST-CNN, which contains 2 or 4 parallel spatial transformers…”)
Kim_2019, Kozlowski_2023, and Jaderberg_2015 are analogous art because they are from the same field of endeavor called image manipulation. Before the effective filing date it would have been obvious to a person of ordinary skill in the art to combine Kim_2019, Kozlowski_2023, and Jaderberg_2015. The rationale for doing so would have been that Kim_2019 utilizes the spatial transformer network of Jaderberg_2015 (Kim_2019 Pg. 8 Col. 1: “Thus, we adopted the spatial transformer network (STN) of Jaderberg et al. [2015]”). Therefore, it would have been obvious for one of ordinary skill in the art to combine the STN of Jaderberg_2015 with the stylizing system of Kim_2019 and Kozlowski_2023.
Claim 8. Kim_2019, Kozlowski_2023, and Jaderberg_2015 teach the limitations of claim 7. Jaderberg_2015 also teaches “the another ML model comprises a feed-forward convolutional neural network.” (Pg. 9 Sect. 5: “While we only explore feed-forward networks in this work…”)
Claims 17 and 18 are similar in claim structure and limitations to claims 7 and 8, thus the rationale for rejection for claims 7 and 8 will apply to claims 17 and 18, respectively.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Department of Mathematics, Oregon State University [1996] “Euler’s Method for First-Order ODE” discloses the method of and reasoning behind Euler’s Method of solving first-order differential equations.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALEXANDER W LYON whose telephone number is (571)270-0757. The examiner can normally be reached Monday-Thursday 8:30-5:30, Friday 08:30-12:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emerson Puente can be reached at (571) 272-3652. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/A.W.L./Examiner, Art Unit 2187
/JOHN E JOHANSEN/Examiner, Art Unit 2187
Prosecution Insights