Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
The present application has a provisional application No. 63/168,499 filed on March 31, 2021.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-36 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding Claim 1
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 1 is a method claim thus it falls into one of the four categories of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
Regarding independent claim 1, following limitations recite a judicial exception:
“receiving sensory data”
[Mental Process] – receiving sensory data can be done with a person looking into the data that is possibly written on a paper which involves observations, evaluations, judgments, and opinions that is capable of being performed in the human mind with the assistance of paper and pen
“detecting one or more objects in the sensory data”
[Mental Process] – detecting objects in the sensory data requires comparing and analyzing the data which involves observations, evaluations, judgments, and opinions that is capable of being performed in the human mind with the assistance of paper and pen
“initializing both a simulator geometry of the one or more objects in a simulator and simulator parameters used in the simulator”
[Mental Process] – initializing the geometry and the parameters is simply giving these items starting values which involves observations, evaluations, judgments, and opinions that is capable of being performed in the human mind with the assistance of paper and pen
“predicting the simulator geometry using the simulator parameters”
[Mathematical Calculations] – predicting the simulator geometry using such parameters requires mathematical computation which recites to an abstract idea.
“computing predicted sensory data from the predicted simulator geometry”
[Mathematical Calculations] – computing the predicted sensory data using the geometry requires mathematical computation which recites to an abstract idea.
“computing a loss between the predicted sensory data and the received sensory data”
[Mathematical Calculations] – computing the loss between two data requires mathematical computation which recites to an abstract idea.
“updating the simulator geometry and the simulator parameters by minimizing the computed loss”
[Mathematical Calculations] – updating the numbers by minimizing the loss requires mathematical computation which recites to an abstract idea.
“repeating (i)-(viii) if new sensory data is received”
[Mental Process] – As it goes through all the steps from (i) to (viii) which recite to Mental Process, this limitation also recites to Mental Process
[Mathematical Calculations] – As it goes through all the steps from (i) to (viii) which recite to Mathematical Calculations, this limitation also recites to Mathematical Calculations
“providing a simulation of the one or more objects using the updated simulator geometry and the updated simulator parameters”
[Mathematical Calculations] – providing a simulation of objects using the updated numbers requires mathematical computation which recites to an abstract idea.
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
Regarding Claim 1, the claim recites additional elements of
“simulator”
A simulator, a mere computer program, is recited at a high level of generality and is merely adding words “apply it” to the judicial exception. (see MPEP 2106.05(f))
[Even when viewed in combination, the additional elements do no more than automate the mental processes that a person could perform, using computer components as a tool, thus the claim as a whole does not integrate into a practical application.]
Step 2B – whether the claim as a whole amount to significantly more than the judicial exception? I.e. Are there any additional elements (features/limitations/step) recited in the claim beyond the abstract idea?
The claim does not provide an inventive concept (significantly more than the abstract idea). The claim is ineligible.
As explained above, the additional element [1] is considered a mere instruction to apply an exception to the generic computer components or machine-learning components that simply run mathematical calculations and mental processes (see MPEP 2106.05(f)). This limitation remains a mere instruction to apply an exception even upon reconsideration. Even when considered in combination, the additional element represents a mere instruction to apply an exception, which cannot provide an inventive concept.
Regarding Claim 2
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 2 is a dependent claim of 1, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
Regarding dependent claim 2, following limitations recite a judicial exception:
“receiving kinematic information of the robot”
[Mental Process] – receiving information of the robot can be done with a person looking into the data that is possibly written on a paper which involves observations, evaluations, judgments, and opinions that is capable of being performed in the human mind with the assistance of paper and pen
“receiving robot action information concerning actions performed by the robot manipulating the one or more objects”
[Mental Process] – receiving robot action information can be done with a person looking into the data that is possibly written on a paper which involves observations, evaluations, judgments, and opinions that is capable of being performed in the human mind with the assistance of paper and pen
“wherein receiving the sensory data includes receiving sensory data concerning the one or more objects being manipulated by the actions performed by the robot and wherein predicting the simulator geometry also uses the robot action information”
[Mental Process] – receiving sensory data can be done with a person looking into the data that is possibly written on a paper which involves observations, evaluations, judgments, and opinions that is capable of being performed in the human mind with the assistance of paper and pen
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
Regarding Claim 2, the claim recites additional elements of
“simulator”
A simulator, a mere computer program, is recited at a high level of generality and is merely adding words “apply it” to the judicial exception. (see MPEP 2106.05(f))
[Even when viewed in combination, the additional elements do no more than automate the mental processes that a person could perform, using computer components as a tool, thus the claim as a whole does not integrate into a practical application.]
Step 2B – whether the claim as a whole amount to significantly more than the judicial exception? I.e. Are there any additional elements (features/limitations/step) recited in the claim beyond the abstract idea?
The claim does not provide an inventive concept (significantly more than the abstract idea). The claim is ineligible.
As explained above, the additional element [1] is considered a mere instruction to apply an exception to the generic computer components or machine-learning components that simply run mathematical calculations and mental processes (see MPEP 2106.05(f)). This limitation remains a mere instruction to apply an exception even upon reconsideration. Even when considered in combination, the additional element represents a mere instruction to apply an exception, which cannot provide an inventive concept.
Regarding Claim 3
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 3 is a dependent claim of 1, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
Regarding dependent claim 3, following limitations recite a judicial exception:
“minimizing the computed loss includes minimizing the computed loss uses a minimization technique selected from the group consisting of gradient descent, a Levenberg-Marquardt algorithm, a Trust Region Optimization technique, and a Gauss-Newton algorithm”
[Mathematical Calculations] – minimizing the loss using listed techniques requires mathematical computation which recites to an abstract idea.
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
The claim 3 does not recite any additional elements other than abstract ideas, so it does not integrate into a practical application. Thus, this claim is directed to the abstract idea.
Regarding Claim 4
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 4 is a dependent claim of 3, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
Regarding dependent claim 4, following limitations recite a judicial exception:
“a derivative for the minimization technique is computed using auto-differentiation, finite difference, adjoint method or is analytically derived”
[Mathematical Calculations] –the listed techniques requires mathematical computation which recites to an abstract idea.
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
The claim 4 does not recite any additional elements other than abstract ideas, so it does not integrate into a practical application. Thus, this claim is directed to the abstract idea.
Regarding Claim 5
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 5 is a dependent claim of 2, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
Regarding dependent claim 5, following limitations recite a judicial exception:
“wherein receiving robot action includes receiving robot joint angle, velocity and/or torque measurement information.”
[Mental Process] – receiving robot action information can be done with a person looking into the data that is possibly written on a paper which involves observations, evaluations, judgments, and opinions that is capable of being performed in the human mind with the assistance of paper and pen
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
The claim 5 does not recite any additional elements other than abstract ideas, so it does not integrate into a practical application. Thus, this claim is directed to the abstract idea.
Regarding Claim 6
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 6 is a dependent claim of 1, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
As Claim 6 does not have any abstract idea by itself, thus uses all the limitations of Claim 1.
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
Regarding Claim 6, the claim recites additional elements of
“the simulator is a position-based dynamics simulator”
The simulator, a mere computer program, is recited at a high level of generality and is merely adding words “apply it” to the judicial exception. (see MPEP 2106.05(f))
[Even when viewed in combination, the additional elements do no more than automate the mental processes that a person could perform, using computer components as a tool, thus the claim as a whole does not integrate into a practical application.]
Step 2B – whether the claim as a whole amount to significantly more than the judicial exception? I.e. Are there any additional elements (features/limitations/step) recited in the claim beyond the abstract idea?
The claim does not provide an inventive concept (significantly more than the abstract idea). The claim is ineligible.
As explained above, the additional element [1] is considered a mere instruction to apply an exception to the generic computer components or machine-learning components that simply run mathematical calculations and mental processes (see MPEP 2106.05(f)). This limitation remains a mere instruction to apply an exception even upon reconsideration. Even when considered in combination, the additional element represents a mere instruction to apply an exception, which cannot provide an inventive concept.
Regarding Claim 7
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 7 is a dependent claim of 1, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
As Claim 7 does not have any abstract idea by itself, thus uses all the limitations of Claim 1.
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
Regarding Claim 7, the claim recites additional elements of
“the simulator is a rigid body dynamics simulator”
The simulator, a mere computer program, is recited at a high level of generality and is merely adding words “apply it” to the judicial exception. (see MPEP 2106.05(f))
[Even when viewed in combination, the additional elements do no more than automate the mental processes that a person could perform, using computer components as a tool, thus the claim as a whole does not integrate into a practical application.]
Step 2B – whether the claim as a whole amount to significantly more than the judicial exception? I.e. Are there any additional elements (features/limitations/step) recited in the claim beyond the abstract idea?
The claim does not provide an inventive concept (significantly more than the abstract idea). The claim is ineligible.
As explained above, the additional element [1] is considered a mere instruction to apply an exception to the generic computer components or machine-learning components that simply run mathematical calculations and mental processes (see MPEP 2106.05(f)). This limitation remains a mere instruction to apply an exception even upon reconsideration. Even when considered in combination, the additional element represents a mere instruction to apply an exception, which cannot provide an inventive concept.
Regarding Claim 8
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 8 is a dependent claim of 1, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
As Claim 8 does not have any abstract idea by itself, thus uses all the limitations of Claim 1.
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
Regarding Claim 8, the claim recites additional elements of
“the simulator is an articulated rigid body dynamics simulator”
The simulator, a mere computer program, is recited at a high level of generality and is merely adding words “apply it” to the judicial exception. (see MPEP 2106.05(f))
[Even when viewed in combination, the additional elements do no more than automate the mental processes that a person could perform, using computer components as a tool, thus the claim as a whole does not integrate into a practical application.]
Step 2B – whether the claim as a whole amount to significantly more than the judicial exception? I.e. Are there any additional elements (features/limitations/step) recited in the claim beyond the abstract idea?
The claim does not provide an inventive concept (significantly more than the abstract idea). The claim is ineligible.
As explained above, the additional element [1] is considered a mere instruction to apply an exception to the generic computer components or machine-learning components that simply run mathematical calculations and mental processes (see MPEP 2106.05(f)). This limitation remains a mere instruction to apply an exception even upon reconsideration. Even when considered in combination, the additional element represents a mere instruction to apply an exception, which cannot provide an inventive concept.
Regarding Claim 9
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 9 is a dependent claim of 1, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
As Claim 9 does not have any abstract idea by itself, thus uses all the limitations of Claim 1.
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
Regarding Claim 9, the claim recites additional elements of
“the simulator is a smooth particular hydrodynamics simulator”
The simulator, a mere computer program, is recited at a high level of generality and is merely adding words “apply it” to the judicial exception. (see MPEP 2106.05(f))
[Even when viewed in combination, the additional elements do no more than automate the mental processes that a person could perform, using computer components as a tool, thus the claim as a whole does not integrate into a practical application.]
Step 2B – whether the claim as a whole amount to significantly more than the judicial exception? I.e. Are there any additional elements (features/limitations/step) recited in the claim beyond the abstract idea?
The claim does not provide an inventive concept (significantly more than the abstract idea). The claim is ineligible.
As explained above, the additional element [1] is considered a mere instruction to apply an exception to the generic computer components or machine-learning components that simply run mathematical calculations and mental processes (see MPEP 2106.05(f)). This limitation remains a mere instruction to apply an exception even upon reconsideration. Even when considered in combination, the additional element represents a mere instruction to apply an exception, which cannot provide an inventive concept.
Regarding Claim 10
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 10 is a dependent claim of 1, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
As Claim 10 does not have any abstract idea by itself, thus uses all the limitations of Claim 1.
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
Regarding Claim 10, the claim recites additional elements of
“the simulator is a finite element method-based dynamics simulator”
The simulator, a mere computer program, is recited at a high level of generality and is merely adding words “apply it” to the judicial exception. (see MPEP 2106.05(f))
[Even when viewed in combination, the additional elements do no more than automate the mental processes that a person could perform, using computer components as a tool, thus the claim as a whole does not integrate into a practical application.]
Step 2B – whether the claim as a whole amount to significantly more than the judicial exception? I.e. Are there any additional elements (features/limitations/step) recited in the claim beyond the abstract idea?
The claim does not provide an inventive concept (significantly more than the abstract idea). The claim is ineligible.
As explained above, the additional element [1] is considered a mere instruction to apply an exception to the generic computer components or machine-learning components that simply run mathematical calculations and mental processes (see MPEP 2106.05(f)). This limitation remains a mere instruction to apply an exception even upon reconsideration. Even when considered in combination, the additional element represents a mere instruction to apply an exception, which cannot provide an inventive concept.
Regarding Claim 11
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 11 is a dependent claim of 1, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
As Claim 11 does not have any abstract idea by itself, thus uses all the limitations of Claim 1.
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
Regarding Claim 11, the claim recites additional elements of
“the simulator is a projective dynamics simulator”
The simulator, a mere computer program, is recited at a high level of generality and is merely adding words “apply it” to the judicial exception. (see MPEP 2106.05(f))
[Even when viewed in combination, the additional elements do no more than automate the mental processes that a person could perform, using computer components as a tool, thus the claim as a whole does not integrate into a practical application.]
Step 2B – whether the claim as a whole amount to significantly more than the judicial exception? I.e. Are there any additional elements (features/limitations/step) recited in the claim beyond the abstract idea?
The claim does not provide an inventive concept (significantly more than the abstract idea). The claim is ineligible.
As explained above, the additional element [1] is considered a mere instruction to apply an exception to the generic computer components or machine-learning components that simply run mathematical calculations and mental processes (see MPEP 2106.05(f)). This limitation remains a mere instruction to apply an exception even upon reconsideration. Even when considered in combination, the additional element represents a mere instruction to apply an exception, which cannot provide an inventive concept.
Regarding Claim 12
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 12 is a dependent claim of 1, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
As Claim 12 does not have any abstract idea by itself, thus uses all the limitations of Claim 1.
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
Regarding Claim 12, the claim recites additional elements of
“the simulator is an energy projection-based dynamics simulator”
The simulator, a mere computer program, is recited at a high level of generality and is merely adding words “apply it” to the judicial exception. (see MPEP 2106.05(f))
[Even when viewed in combination, the additional elements do no more than automate the mental processes that a person could perform, using computer components as a tool, thus the claim as a whole does not integrate into a practical application.]
Step 2B – whether the claim as a whole amount to significantly more than the judicial exception? I.e. Are there any additional elements (features/limitations/step) recited in the claim beyond the abstract idea?
The claim does not provide an inventive concept (significantly more than the abstract idea). The claim is ineligible.
As explained above, the additional element [1] is considered a mere instruction to apply an exception to the generic computer components or machine-learning components that simply run mathematical calculations and mental processes (see MPEP 2106.05(f)). This limitation remains a mere instruction to apply an exception even upon reconsideration. Even when considered in combination, the additional element represents a mere instruction to apply an exception, which cannot provide an inventive concept.
Regarding Claim 13
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 13 is a dependent claim of 1, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
As Claim 13 does not have any abstract idea by itself, thus uses all the limitations of Claim 1.
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
Regarding Claim 13, the claim recites additional elements of
“sensory data includes image data, CT/MRI scans, ultrasound, depth image data, and/or point cloud data”
Listed data types are merely data gathering recited at a high level of generality, thus is insignificant extra-solution activity (See MPEP 2106.05(g)).
[Even when viewed in combination, the additional elements do no more than automate the mental processes that a person could perform, using computer components as a tool, thus the claim as a whole does not integrate into a practical application.]
Step 2B – whether the claim as a whole amount to significantly more than the judicial exception? I.e. Are there any additional elements (features/limitations/step) recited in the claim beyond the abstract idea?
The claim does not provide an inventive concept (significantly more than the abstract idea). The claim is ineligible.
As explained above, the additional element [1] is considered an insignificant extra solution activity and at best the equivalent of a mere data gathering recited at a high level of generality and amount to receiving or transmitting data over a network, which is well-understood, routine, conventional activity (See MPEP 2106.05(d), subsection II).
This limitation remains insignificant extra-solution activity even upon reconsideration. Even when considered in combination, the additional element represents insignificant extra-solution activity, which cannot provide an inventive concept.
Regarding Claim 14
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 14 is a dependent claim of 1, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
Regarding dependent claim 14, following limitations recite a judicial exception:
“multiple iterations of simulation time steps”
[Mathematical Calculations] – simulation itself is being considered as a series of mathematical computation to mimic the real-world environment which recites to an abstract idea.
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
Regarding Claim 14, the claim recites additional elements of
“the sensory data is expanded over a predetermined time window encompassing multiple iterations of simulation time steps”
Collecting Sensory data over a period of time is merely data gathering recited at a high level of generality, thus is insignificant extra-solution activity (See MPEP 2106.05(g)).
[Even when viewed in combination, the additional elements do no more than automate the mental processes that a person could perform, using computer components as a tool, thus the claim as a whole does not integrate into a practical application.]
Step 2B – whether the claim as a whole amount to significantly more than the judicial exception? I.e. Are there any additional elements (features/limitations/step) recited in the claim beyond the abstract idea?
The claim does not provide an inventive concept (significantly more than the abstract idea). The claim is ineligible.
As explained above, the additional element [1] is considered an insignificant extra solution activity and at best the equivalent of a mere data gathering recited at a high level of generality and amount to receiving or transmitting data over a network, which is well-understood, routine, conventional activity (See MPEP 2106.05(d), subsection II).
This limitation remains insignificant extra-solution activity even upon reconsideration. Even when considered in combination, the additional element represents insignificant extra-solution activity, which cannot provide an inventive concept.
Regarding Claim 15
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 15 is a dependent claim of 1, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
As Claim 15 does not have any abstract idea by itself, thus uses all the limitations of Claim 1.
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
The claim 15 does not recite any additional elements other than abstract ideas, so it does not integrate into a practical application. Thus, this claim is directed to the abstract idea.
Regarding Claim 16
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 16 is a dependent claim of 1, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
As Claim 16 does not have any abstract idea by itself, thus uses all the limitations of Claim 1.
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
The claim 16 does not recite any additional elements other than abstract ideas, so it does not integrate into a practical application. Thus, this claim is directed to the abstract idea.
Regarding Claim 17
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 17 is a dependent claim of 1, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
As Claim 17 does not have any abstract idea by itself, thus uses all the limitations of Claim 1.
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
The claim 17 does not recite any additional elements other than abstract ideas, so it does not integrate into a practical application. Thus, this claim is directed to the abstract idea.
Regarding Claim 18
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 18 is a dependent claim of 1, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
As Claim 18 does not have any abstract idea by itself, thus uses all the limitations of Claim 1.
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
The claim 18 does not recite any additional elements other than abstract ideas, so it does not integrate into a practical application. Thus, this claim is directed to the abstract idea.
Regarding Claim 19
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 19 is a dependent claim of 18, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
As Claim 19 does not have any abstract idea by itself, thus uses all the limitations of Claim 1.
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
The claim 19 does not recite any additional elements other than abstract ideas, so it does not integrate into a practical application. Thus, this claim is directed to the abstract idea.
Regarding Claim 20
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 20 is a dependent claim of 1, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
As Claim 20 does not have any abstract idea by itself, thus uses all the limitations of Claim 1.
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
The claim 20 does not recite any additional elements other than abstract ideas, so it does not integrate into a practical application. Thus, this claim is directed to the abstract idea.
Regarding Claim 21
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 21 is a dependent claim of 1, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
As Claim 21 does not have any abstract idea by itself, thus uses all the limitations of Claim 1.
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
The claim 21 does not recite any additional elements other than abstract ideas, so it does not integrate into a practical application. Thus, this claim is directed to the abstract idea.
Regarding Claim 22
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 22 is a dependent claim of 2, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
Regarding dependent claim 22, following limitations recite a judicial exception:
“manipulating the one or more objects in accordance with the simulation so that a physical geometry of the one or more objects aligns with a goal geometry”
[Mental Process Calculations] – manipulating the objects following the simulation computed to align the goal geometry can be done by comparing the goal geometry and the simulation to match the physical geometry of robot which involves observations, evaluations, judgments, and opinions that is capable of being performed in the human mind with the assistance of paper and pen
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
The claim 22 does not recite any additional elements other than abstract ideas, so it does not integrate into a practical application. Thus, this claim is directed to the abstract idea.
Regarding Claim 23
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 23 is a dependent claim of 22, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
Regarding dependent claim 23, following limitations recite a judicial exception:
“the simulation is updated during manipulation of the one or more objects to provide closed-loop control”
[Mathematical Calculations] – Updating the simulation during the manipulation to provide closed-loop control requires to constant mathematical computation which recites to an abstract idea.
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
The claim 23 does not recite any additional elements other than abstract ideas, so it does not integrate into a practical application. Thus, this claim is directed to the abstract idea.
Regarding Claim 24
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 24 is a dependent claim of 22, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
As Claim 24 does not have any abstract idea by itself, thus uses all the limitations of Claim 22.
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
The claim 24 does not recite any additional elements other than abstract ideas, so it does not integrate into a practical application. Thus, this claim is directed to the abstract idea.
Regarding Claim 25
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 25 is a dependent claim of 22, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
Regarding dependent claim 25, following limitations recite a judicial exception:
“computing a control loss between the goal geometry and the simulator geometry and minimizing the control loss to compute a sequence of robot actions that are used to manipulate the one or more objects”
[Mathematical Calculations] – computing a control loss between the goal geometry and simulator geometry and minimizing the loss requires mathematical computation which recites to an abstract idea.
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
The claim 25 does not recite any additional elements other than abstract ideas, so it does not integrate into a practical application. Thus, this claim is directed to the abstract idea.
Regarding Claim 26
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 26 is a dependent claim of 22, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
As Claim 26 does not have any abstract idea by itself, thus uses all the limitations of Claim 22.
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
Regarding Claim 26, the claim recites additional elements of
“executing the sequence of robot actions to manipulate the one or more objects such that the physical geometry of the one or more objects aligns with the goal geometry”
Executing the sequence of robot actions is simply executing the data computed which is ta types are merely data gathering recited at a high level of generality and is merely adding words “apply it” to the judicial exception. (see MPEP 2106.05(f))
[Even when viewed in combination, the additional elements do no more than automate the mental processes that a person could perform, using computer components as a tool, thus the claim as a whole does not integrate into a practical application.]
Step 2B – whether the claim as a whole amount to significantly more than the judicial exception? I.e. Are there any additional elements (features/limitations/step) recited in the claim beyond the abstract idea?
The claim does not provide an inventive concept (significantly more than the abstract idea). The claim is ineligible.
As explained above, the additional element [1] is considered merely computer components that are just to store and execute code-based instructions which are considered a mere instruction to apply an exception and amount to storing and receiving information in memory, which is well-understood, routine, conventional activity (See MPEP 2106.05(d), subsection II).
This limitation remains a mere instruction to apply an exception even upon reconsideration. Even when considered in combination, the additional element represents a mere instruction to apply an exception, which cannot provide an inventive concept.
Regarding Claims 27, 28, 36
Claims 27, 28, 36 have similar limitations of Claims 3, 4 and 1, respectively. For the reasons described above with respect to Claims 3, 4 and 1, these judicial exceptions are not meaningfully integrated into a practical application, or significantly more than the abstract ideas. The claims do not provide anything more than the abstract ideas of mental processes and mathematical calculations that are practically capable of being performed with the assistance of pen and paper. Therefore, Claims 27, 28 and 36 also recite abstract ideas that do not integrate into a practical application or amount to significantly more than judicial exception, and thus are rejected under U.S.C. 101.
Regarding Claim 29
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 29 is a dependent claim of 22, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
As Claim 29 does not have any abstract idea by itself, thus uses all the limitations of Claim 22.
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
The claim 29 does not recite any additional elements other than abstract ideas, so it does not integrate into a practical application. Thus, this claim is directed to the abstract idea.
Regarding Claim 30
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 30 is a dependent claim of 25, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
As Claim 30 does not have any abstract idea by itself, thus uses all the limitations of Claim 25.
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
The claim 30 does not recite any additional elements other than abstract ideas, so it does not integrate into a practical application. Thus, this claim is directed to the abstract idea.
Regarding Claim 31
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 31 is a dependent claim of 22, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
As Claim 31 does not have any abstract idea by itself, thus uses all the limitations of Claim 22.
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
The claim 31 does not recite any additional elements other than abstract ideas, so it does not integrate into a practical application. Thus, this claim is directed to the abstract idea.
Regarding Claim 32
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 32 is a dependent claim of 22, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
As Claim 32 does not have any abstract idea by itself, thus uses all the limitations of Claim 22.
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
The claim 32 does not recite any additional elements other than abstract ideas, so it does not integrate into a practical application. Thus, this claim is directed to the abstract idea.
Regarding Claim 33
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 33 is a dependent claim of 22, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
As Claim 33 does not have any abstract idea by itself, thus uses all the limitations of Claim 22.
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
The claim 33 does not recite any additional elements other than abstract ideas, so it does not integrate into a practical application. Thus, this claim is directed to the abstract idea.
Regarding Claim 34
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 34 is a dependent claim of 33, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
As Claim 34 does not have any abstract idea by itself, thus uses all the limitations of Claim 33.
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
The claim 34 does not recite any additional elements other than abstract ideas, so it does not integrate into a practical application. Thus, this claim is directed to the abstract idea.
Regarding Claim 35
Step 1 – whether the claim falls within any statutory category. See MPEP 2016.03
Claim 35 is a dependent claim of 22, thus it falls within the same category of statutory subject matter.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
As Claim 35 does not have any abstract idea by itself, thus uses all the limitations of Claim 22.
Step 2A Prong 2 – whether the claim recites additional elements that integrate the exception into a practical application of the exception?
The claim 35 does not recite any additional elements other than abstract ideas, so it does not integrate into a practical application. Thus, this claim is directed to the abstract idea.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-5, 13-15, 22-24, 28, 29, 36 are rejected under 35 U.S.C. 103 as being unpatentable over Valencia et al. (Valencia), Non-Patent Literature, “Combining Self-Organizing and Graph Neural Networks for Modeling Deformable Objects in Robotic Manipulation”, Published in December 2020, 11 Pages, in view of Lassner et al. (Lassner), Non-Patent Literature (listed in IDS filed on 09/11/2023), “Pulsar: Efficient Sphere-based Neural Rendering”, Published in December 2020, 13 Pages.
As to independent Claim 1,
Valencia teaches a method for generating and updating a simulation of one or more objects from sensory data, comprising:
i. receiving sensory data; ii. detecting one or more objects in the sensory data (Valencia, Pg5, Section4.1, Lines3-5, "an Intel RealSense SR305 RGB-D sensor mounted overhead on a tripod, and a deformable object placed on the palm of the robotic hand"
Pg6, Section4.2, Lines1-2, "The RGB-D sensor data is processed in a ROS node to detect the object and generate the point cloud data"
Pg6, Section4.2.1, Lines1-3, 11-12, "Classical image segmentation techniques are applied to both aligned color and depth images for the detection of the deformable objects ... to obtain the object of interest. The segmented image is then deprojected to convert the 2D pixels to 3D point clouds", wherein Valencia explicitly discloses the sensor that collects the data to detect the object of interest by applying image segmentation techniques, rendering it functionally equivalent to the claimed invention);
iii. initializing both a simulator geometry of the one or more objects in a simulator and simulator parameters used in the simulator (Valencia, Pg2, Section3.1, Paragraph2, Lines1-7, "A GNG(Growing Neural Gas) model (Fritzke, 1995) produces a graph representation G=(O, R) from a data distribution P of size N. Where, O={o_i}_i=1:N_o is the set of nodes with N_o cardinality, and R = {rk,uk,vk}_k=1:N_r is the set of edges with N_R cardinality, which connects an unordered pair of nodes uk and vk. Also, each node has an associated feature vector o_i = {xi,ei}, which contains the position and spatial error"
Pg2, Section3.1, Paragraph3, Lines4-6, "First, the graph is initialized by creating two nodes with position set to random values and spatial error set to zero"
Pg3, Algorithm1
PNG
media_image1.png
207
289
media_image1.png
Greyscale
Pg4, Algorithm2
PNG
media_image2.png
176
252
media_image2.png
Greyscale
Pg5, Algorithm4, Line3
PNG
media_image3.png
210
280
media_image3.png
Greyscale
Pg2, Right Column, Lines1-3, "Recent learning-based models such as Graph Neural Networks (GNN) have demonstrated the ability to act as a physics engine", wherein Valencia explicitly discloses in Algorithm 1 and 2 about initializing the graph or the geometry using G=(O,R) which represents the geometry of the deformable objects in Mesh/Particle Grid using the input P, which is the sensor data. Also, in the algorithm 4 line3 shows initializing the latent state embedding parameter (the corresponding simulator parameters) to 0 used in the GNN (the corresponding simulator), rendering it functionally equivalent to the claimed invention);
iv. predicting the simulator geometry using the simulator parameters (Valencia, Pg5, Section3.2.2, Lines 1-4, "An IN(Interaction Network) model is trained to learn the transition dynamics of the object state. It takes the ojbect-action graph at a certain time step G_t and outputs a prediction of the nodes position of the graph for the next time step G_t+1"
Pg5, Algorithm4
PNG
media_image3.png
210
280
media_image3.png
Greyscale
, wherein Valencia discloses about predicting the simulator geometry G_t+1 in GNN-based IN using the initialized parameters in Algorithm4. Algorithm 4 is designed to predict G_t+1, rendering it functionally equivalent to the claimed invention);
Valencia teaches, as mentioned above, about predicting the simulator geometry, but is silent about computing predicted sensory data from it. From the same field of endeavor, Lassner teaches this limitation:
v. computing predicted sensory data from the predicted simulator geometry (Lassner, Pg4, Section3.2, Lines1-5, "Our differentiable renderer implements a mapping F = R(S,R,t,K) that maps from the 3D sphere-based scene representation S to a rendered feature image F based on the image formation model defined by the camera rotation R, translation t, and intrinsic parameter K"
Pg4, Section3.2, Paragraph2, Lines1-7, "The rendering operation R has to compute the channel values for each pixel of the feature image in a differentiable manner. To this end, we propose a blending function that combines the channel information based on the position, radius, and opacity of the spheres that are intersected by the camera ray associated with each pixel"
Pg4, Equation1,
PNG
media_image4.png
45
292
media_image4.png
Greyscale
, wherein Lassner explicitly discloses outputting a rendered feature image F (the corresponding predicted sensory data) by inputting the predicted geometry, S, into a differentiable renderer, R, which is functionally equivalent to the claimed invention);
Lassner further teaches
vi. computing a loss between the predicted sensory data and the received sensory data;
vii. updating the simulator geometry and the simulator parameters by minimizing the computed loss (Lassner, Pg2, Left Column, Lines2-3, "The resulting image can be compared to ground truth observations to inform an optimization process"
Pg5, Section3.4,
PNG
media_image5.png
112
332
media_image5.png
Greyscale
Pg5, Section3.4, Lines10-11, "We use ADAM(Adaptive Moment Estimation) in all experiments to solve this optimization problem", wherein Lassner explicitly discloses the optimization process (the corresponding loss calculation) that minimizes the loss from the predicted sensory data and the received sensory data (the sensory data received by using RGB-D sensor mentioned by Valencia which is used as the ground truth data) using Lassner's ADAM backpropagation process to update the corresponding simulator geometry and parameters, rendering it functionally equivalent to the claimed invention.)
Valencia further teaches
viii. repeating (i)-(viii) if new sensory data is received (Valencia, Pg4, Algorithm2
PNG
media_image2.png
176
252
media_image2.png
Greyscale
Pg9, Section5.2, Lines22-26, "Given that the modeling and prediction framework is meant to be part of the robotic hand control loop, new RGB-D data is made available to update the deformable object representation, and provide an updated prediction, at the same frame rate as the robot controller"
Pg3, Figure1
PNG
media_image6.png
305
779
media_image6.png
Greyscale
, wherein Valencia explcitly discloses that when new sensor data P comes in after t = 1, the system in figure 1 (the loop arrow after "Shape Estimation") and in algorithm2 shows that it loops back to G_t-1 to for iteratively executes the process, rendering it functionally equivalent to the claimed invention); and
ix. providing a simulation of the one or more objects using the updated simulator geometry and the updated simulator parameters (Valencia, Pg9, Section5.2, Lines22-26, "Given that the modeling and prediction framework is meant to be part of the robotic hand control loop, new RGB-D data is made available to update the deformable object representation, and provide an updated prediction, at the same frame rate as the robot controller"
Pg1, Introduction, Lines6-8, "an object model that integrates shape representation and prediction is required in order to perform a variety of tasks with deformable objects"
Pg4, Section3.2, Lines5-7, "With the objective to support the requirements of path planning and dynamic interaction of a robotic hand with a deformable object..."
Pg3, Figure1, wherein Valencia discloses providing a simulation of objects as in Figure 1 (the arrow outputting G_t+1), which uses the updated simulator geometry and the parameters as mentioned above, that can be used in controlling a robotic hand, rendering it functionally equivalent to the claimed invention.)
Valencia and Lassner are analogous to the claimed invention as they are from the same field of endeavor of 3D scene representation, dynamic object shape estimation, and rendering from sensor observations for computer vision and robotic applications. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the GNN-based 3D deformable object shape estimation and prediction framework of Valencia with the sphere-based differentiable rendering engine and photometric loss optimization loop of Lassner. The motivation is as recited by Lassner (Lassner, Pg2, Left Column, Paragraph1, Lines16-19, "This makes it easy to handle point cloud data from 3D sensors directly, allows for the optimization of the scene representation without problems of changing topology" and Pg2, First Bullet, "...enables end-to-end training of deep models with geometry and projection components”) such that the combination enables the system to render 2D predicted sensory images, compares them directly against received RGB-D sensor observations to compute a photometric loss, and perform end-to-end gradient descent updates on the 3D simulator geometry and parameters without topological constraints.
As to dependent Claim 2,
The combination of Valencia and Lassner teaches, as mentioned above, all the limitations of Claim 1. It teaches about overall architecture of using the sensory data to predict an object’s simulator geometry by minimizing its loss function which then provides a simulation that can be executed by a robot hand to manipulate the object.
Valencia further teaches the method of claim 1, wherein a robot manipulates the one or more objects and further comprising:
receiving kinematic information of the robot (Valencia, Pg4, Section3.2.1, Paragraph2, Lines4-6, "This means that new nodes are added to the graph with their feature corresponding to the position components of the fingertips pose"
Pg6, Section 4.2.2, Lines1-2, "The data captured on the fingertips correspond to the pose (position and orientation) of each tip", wherein Valencia explicitly discloses the robot fingers position and orientation and the data is captured as nodes that is used by the engine or the simulator, rendering it functionally equivalent to the claimed invention);
receiving robot action information concerning actions performed by the robot manipulating the one or more objects (Valencia, Pg4, Section3.2.1, Lines1-3, "A new representation is created to jointly capture the object shape and the manipulation actions. This is defined as a directed graph G = <O, R>"
Pg4, Section3.2.1, Paragraph2, Lines6-9, "Also, edges are created when physical interactions are detected between the fingertips and the object, thus assigning action nodes for the fingertips as senders, and object nodes as receivers in the directed graph"
Pg6, Right Column, Paragraph1, Lines13-15, "Each file stores the data generated in synchronization with the execution of the fingers trajectory...", wherein Valencia explicitly discloses of receiving manipulation actions and actions nodes (the corresponding robot action information) that is collected from the finger trajectory and the interaction between the robot fingers and the objects, rendering it functionally equivalent to the claimed invention); and
wherein receiving the sensory data includes receiving sensory data concerning the one or more objects being manipulated by the actions performed by the robot and wherein predicting the simulator geometry also uses the robot action information (Valencia, Pg5, Section4.1, Lines1-5, "The configuration of the real robotic environment is shown in Figure 2, which consists of a Barrett BH8-280 robotic hand resting on a flat table, an Intel RealSense SR305 RGB-D sensor mounted overhead on a tripod, and a deformable object placed on the palm of the robotic hand"
Pg6, Section4.2, Lines1-2, "The RGB-D sensor data is processed in a ROS node to detect the object and generate the point cloud data"
Pg4, Section3.2, Lines7-11, "Graph Neural Network (GNN) based models are also adapted in our framework to predict the future object state using the information of the current object state and the manipulation actions of the robotic hand"
Pg5, Section3.2.2, Lines 1-4, "An IN(Interaction Network) model is trained to learn the transition dynamics of the object state. It takes the ojbect-action graph at a certain time step G_t and outputs a prediction of the nodes position of the graph for the next time step G_t+1", wherein Valencia explicitly discloses that the system receives point cloud data by the RGB-D sensor mounted on the robot hand when it is manipulating the deformable object, which the combined graph G_t of the object's information and the robot's action information is inputted into the GNN/IN model to predict the simulator geometry of the object, G_t+1, rendering it functionally equivalent to the claimed invention.)
Valencia and Lassner are analogous to the claimed invention as they are from the same field of endeavor of 3D scene representation, dynamic object shape estimation, and rendering from sensor observations for computer vision and robotic applications. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the GNN-based 3D deformable object shape estimation and prediction framework of Valencia with the sphere-based differentiable rendering engine and photometric loss optimization loop of Lassner. The motivation is as recited by Lassner (Lassner, Pg2, Left Column, Paragraph1, Lines16-19, "This makes it easy to handle point cloud data from 3D sensors directly, allows for the optimization of the scene representation without problems of changing topology" and Pg2, First Bullet, "...enables end-to-end training of deep models with geometry and projection components”) such that the combination enables the system to render 2D predicted sensory images, compares them directly against received RGB-D sensor observations to compute a photometric loss, and perform end-to-end gradient descent updates on the 3D simulator geometry and parameters without topological constraints.
As to dependent Claim 3,
The combination of Valencia and Lassner teaches, as mentioned above, all the limitations of Claim 1. It teaches about overall architecture of using the sensory data to predict an object’s simulator geometry by minimizing its loss function which then provides a simulation that can be executed by a robot hand to manipulate the object.
Lassner further teaches the method of claim 1, wherein
minimizing the computed loss includes minimizing the computed loss uses a minimization technique selected from the group consisting of gradient descent, a Levenberg-Marquardt algorithm, a Trust Region Optimization technique, and a Gauss-Newton algorithm (Lassner, Pg11, Section B.2. First Bullet, Lines6-9, "However, from an optimization point of view, we found the gradients normalized by the number of pixels are much better suited for stable loss reduction with gradient descent techniques"
Pg5, Section3.4, Lines10-11, "We use ADAM in all experiments to solve this optimization problem", wherein Lassner explicitly discloses that the minimizing loss uses gradient descent technique which ADAM relies on, rendering it functionally equivalent to the claimed invention.)
Valencia and Lassner are analogous to the claimed invention as they are from the same field of endeavor of 3D scene representation, dynamic object shape estimation, and rendering from sensor observations for computer vision and robotic applications. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the GNN-based 3D deformable object shape estimation and prediction framework of Valencia with the sphere-based differentiable rendering engine and photometric loss optimization loop of Lassner. The motivation is as recited by Lassner (Lassner, Pg2, Left Column, Paragraph1, Lines16-19, "This makes it easy to handle point cloud data from 3D sensors directly, allows for the optimization of the scene representation without problems of changing topology" and Pg2, First Bullet, "...enables end-to-end training of deep models with geometry and projection components”) such that the combination enables the system to render 2D predicted sensory images, compares them directly against received RGB-D sensor observations to compute a photometric loss, and perform end-to-end gradient descent updates on the 3D simulator geometry and parameters without topological constraints.
As to dependent Claim 4,
The combination of Valencia and Lassner teaches, as mentioned above, all the limitations of Claim 3. It teaches about overall architecture of using the sensory data to predict an object’s simulator geometry by minimizing its loss function and the minimizing uses gradient descent technique.
Lassner further teaches the method of claim 3, wherein a derivative for the minimization technique is computed using auto-differentiation, finite difference, adjoint method or is analytically derived (Lassner, Pg2, Left Column, Paragraph1, Lines30-32, "Lastly, we integrate Pulsar with the PyTorch [29] optimization framework to make use of auto-differentiation and ease the integration with deep learning models.")
Valencia and Lassner are analogous to the claimed invention as they are from the same field of endeavor of 3D scene representation, dynamic object shape estimation, and rendering from sensor observations for computer vision and robotic applications. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the GNN-based 3D deformable object shape estimation and prediction framework of Valencia with the sphere-based differentiable rendering engine and photometric loss optimization loop of Lassner. The motivation is as recited by Lassner (Lassner, Pg2, Left Column, Paragraph1, Lines16-19, "This makes it easy to handle point cloud data from 3D sensors directly, allows for the optimization of the scene representation without problems of changing topology" and Pg2, First Bullet, "...enables end-to-end training of deep models with geometry and projection components”) such that the combination enables the system to render 2D predicted sensory images, compares them directly against received RGB-D sensor observations to compute a photometric loss, and perform end-to-end gradient descent updates on the 3D simulator geometry and parameters without topological constraints.
As to dependent Claim 5,
The combination of Valencia and Lassner teaches, as mentioned above, all the limitations of Claim 2. It teaches about controlling robot by receiving the robot’s information to manipulate the object using the simulator geometry and the sensory data.
Valencia further teaches the method of claim 2, wherein receiving robot action includes receiving robot joint angle, velocity and/or torque measurement information (Valencia, Pg6, Right Column, Paragraph1, Lines2-7, "The base joints range is limited to (-90Deg,90Deg), whereas the spread joint is limited to (-45Deg,45Deg). Each trajectory is generated taking as final configuration a random joint position within the available moving range for each robotic finger, and using a linear interpolation with 50 points beginning from a predefined rest position of the hand"
Pg4, Section3.2.1, Lines4-6, "...associated feature vector o_i = {x_i,v_i}, which contains the object-action state defined as position and velocity"
Pg4, Section3.2.1, Paragraph2, Lines12-14, "Furthermore, the velocity feature is computed by differentiating the signal obtained by the position feature of the object-action nodes", wherein Valencia explicitly discloses that the robot action includes the corresponding joint angles and the corresponding velocity, which is functionally equivalent to the claimed invention.)
Valencia and Lassner are analogous to the claimed invention as they are from the same field of endeavor of 3D scene representation, dynamic object shape estimation, and rendering from sensor observations for computer vision and robotic applications. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the GNN-based 3D deformable object shape estimation and prediction framework of Valencia with the sphere-based differentiable rendering engine and photometric loss optimization loop of Lassner. The motivation is as recited by Lassner (Lassner, Pg2, Left Column, Paragraph1, Lines16-19, "This makes it easy to handle point cloud data from 3D sensors directly, allows for the optimization of the scene representation without problems of changing topology" and Pg2, First Bullet, "...enables end-to-end training of deep models with geometry and projection components”) such that the combination enables the system to render 2D predicted sensory images, compares them directly against received RGB-D sensor observations to compute a photometric loss, and perform end-to-end gradient descent updates on the 3D simulator geometry and parameters without topological constraints.
As to dependent Claim 13,
The combination of Valencia and Lassner teaches, as mentioned above, all the limitations of Claim 1. It teaches about overall architecture of using the sensory data to predict an object’s simulator geometry by minimizing its loss function which then provides a simulation that can be executed by a robot hand to manipulate the object.
Valencia further teaches the method of claim 1, wherein the sensory data includes image data, CT/MRI scans, ultrasound, depth image data, and/or point cloud data (Valencia, Pg6, Section4.2, Lines1-2, "The RGB-D sensor data is processed in a ROS node to detect the object and generate the point cloud data.")
Valencia and Lassner are analogous to the claimed invention as they are from the same field of endeavor of 3D scene representation, dynamic object shape estimation, and rendering from sensor observations for computer vision and robotic applications. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the GNN-based 3D deformable object shape estimation and prediction framework of Valencia with the sphere-based differentiable rendering engine and photometric loss optimization loop of Lassner. The motivation is as recited by Lassner (Lassner, Pg2, Left Column, Paragraph1, Lines16-19, "This makes it easy to handle point cloud data from 3D sensors directly, allows for the optimization of the scene representation without problems of changing topology" and Pg2, First Bullet, "...enables end-to-end training of deep models with geometry and projection components”) such that the combination enables the system to render 2D predicted sensory images, compares them directly against received RGB-D sensor observations to compute a photometric loss, and perform end-to-end gradient descent updates on the 3D simulator geometry and parameters without topological constraints.
As to dependent Claim 14,
The combination of Valencia and Lassner teaches, as mentioned above, all the limitations of Claim 1. It teaches about overall architecture of using the sensory data to predict an object’s simulator geometry by minimizing its loss function which then provides a simulation that can be executed by a robot hand to manipulate the object.
The combination further teaches the method of claim 1, wherein the sensory data is expanded over a predetermined time window encompassing multiple iterations of simulation time steps (Valencia teaches about collecting a sequence of 800 frames of point cloud (the corresponding sensory data over a predetermined time window), which will be used during the simulation time steps(Valencia, Pg6, Right Column, Paragraph1, Lines12-15, "A dataset is created which consists of a file with 800 samples, using a sampling rate of 30 Hz. Each file stores the data generated in synchronization with the execution of the finger trajectory, which takes approximately 27 s to complete"). Whereas Lassner teaches that the minimizing loss function uses N sequential images (i from 0 to N) or frames to execute the backpropagation (Pg5, Section3.4,
PNG
media_image5.png
112
332
media_image5.png
Greyscale
), rendering it functionally equivalent when they are combined.
Valencia and Lassner are analogous to the claimed invention as they are from the same field of endeavor of 3D scene representation, dynamic object shape estimation, and rendering from sensor observations for computer vision and robotic applications. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the GNN-based 3D deformable object shape estimation and prediction framework of Valencia with the sphere-based differentiable rendering engine and photometric loss optimization loop of Lassner. The motivation is as recited by Lassner (Lassner, Pg2, Left Column, Paragraph1, Lines16-19, "This makes it easy to handle point cloud data from 3D sensors directly, allows for the optimization of the scene representation without problems of changing topology" and Pg2, First Bullet, "...enables end-to-end training of deep models with geometry and projection components”) such that the combination enables the system to render 2D predicted sensory images, compares them directly against received RGB-D sensor observations to compute a photometric loss, and perform end-to-end gradient descent updates on the 3D simulator geometry and parameters without topological constraints.
As to dependent Claim 15,
The combination of Valencia and Lassner teaches, as mentioned above, all the limitations of Claim 1. It teaches about overall architecture of using the sensory data to predict an object’s simulator geometry by minimizing its loss function which then provides a simulation that can be executed by a robot hand to manipulate the object.
Valencia further teaches the method of claim 1, wherein the one or more objects includes at least one deformable object (Valencia, Pg5, Section4.1, Lines3-5, "an Intel RealSense SR305 RGB-D sensor mounted overhead on a tripod, and a deformable object placed on the palm of the robotic hand.")
Valencia and Lassner are analogous to the claimed invention as they are from the same field of endeavor of 3D scene representation, dynamic object shape estimation, and rendering from sensor observations for computer vision and robotic applications. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the GNN-based 3D deformable object shape estimation and prediction framework of Valencia with the sphere-based differentiable rendering engine and photometric loss optimization loop of Lassner. The motivation is as recited by Lassner (Lassner, Pg2, Left Column, Paragraph1, Lines16-19, "This makes it easy to handle point cloud data from 3D sensors directly, allows for the optimization of the scene representation without problems of changing topology" and Pg2, First Bullet, "...enables end-to-end training of deep models with geometry and projection components”) such that the combination enables the system to render 2D predicted sensory images, compares them directly against received RGB-D sensor observations to compute a photometric loss, and perform end-to-end gradient descent updates on the 3D simulator geometry and parameters without topological constraints.
As to dependent Claim 22,
The combination of Valencia and Lassner teaches, as mentioned above, all the limitations of Claim 2. It teaches about controlling robot by receiving the robot’s information to manipulate the object using the simulator geometry and the sensory data.
Valencia further teaches the method of claim 2, further comprising manipulating the one or more objects in accordance with the simulation so that a physical geometry of the one or more objects aligns with a goal geometry (Valencia, Pg5, Section4.1, Lines1-5, "The configuration of the real robotic environment is shown in Figure 2, which consists of a Barrett BH8-280 robotic hand resting on a flat table, an Intel RealSense SR305 RGB-D sensor mounted overhead on a tripod, and a deformable object placed on the palm of the robotic hand"
Pg2, Right Column, Lines7-8, "...where a real robotic gripper performs a shape control task on a deformable object"
Pg4, Section3.2, Lines5-7, "With the objective to support the requirements of path planning and dynamic interaction of a robotic hand with a deformable object...", wherein Valencia teaches that the estimated and predicted shape models are utilized with the goal to plan and support the manipulation actions of a robotic hand and to satisfy the requirements of path planning and dynamic interaction during shape control tasks on deformable objects, rendering it functionally equivalent to the claimed invention of manipulating the object to match a desired goal geometry using the state predictions derived from the simulation framework.)
Valencia and Lassner are analogous to the claimed invention as they are from the same field of endeavor of 3D scene representation, dynamic object shape estimation, and rendering from sensor observations for computer vision and robotic applications. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the GNN-based 3D deformable object shape estimation and prediction framework of Valencia with the sphere-based differentiable rendering engine and photometric loss optimization loop of Lassner. The motivation is as recited by Lassner (Lassner, Pg2, Left Column, Paragraph1, Lines16-19, "This makes it easy to handle point cloud data from 3D sensors directly, allows for the optimization of the scene representation without problems of changing topology" and Pg2, First Bullet, "...enables end-to-end training of deep models with geometry and projection components”) such that the combination enables the system to render 2D predicted sensory images, compares them directly against received RGB-D sensor observations to compute a photometric loss, and perform end-to-end gradient descent updates on the 3D simulator geometry and parameters without topological constraints.
As to dependent Claim 23,
The combination of Valencia and Lassner teaches, as mentioned above, all the limitations of Claim 22. It teaches about manipulating the actions of robotic hand to manipulate the objects to match its physical geometry with the goal geometry.
Valencia further teaches the method of claim 22, wherein the simulation is updated during manipulation of the one or more objects to provide closed-loop control (Valencia, Pg1, Introduction, Lines1-2, "In the context of robotic manipulation, object models are used to provide feedback signals that a robot can control when performing a specific task"
Pg9, Section5.2, Lines22-26, "Given that the modeling and prediction framework is meant to be part of the robotic hand control loop, new RGB-D data is made available to update the deformable object representation, and provide an updated prediction, at the same frame rate as the robot controller", wherein Valencia discloses updating the model/simulation during manipulation to provide closed-loop control by providing the feedback signals to the robot, rendering it functionally equivalent to the claimed invention.)
Valencia and Lassner are analogous to the claimed invention as they are from the same field of endeavor of 3D scene representation, dynamic object shape estimation, and rendering from sensor observations for computer vision and robotic applications. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the GNN-based 3D deformable object shape estimation and prediction framework of Valencia with the sphere-based differentiable rendering engine and photometric loss optimization loop of Lassner. The motivation is as recited by Lassner (Lassner, Pg2, Left Column, Paragraph1, Lines16-19, "This makes it easy to handle point cloud data from 3D sensors directly, allows for the optimization of the scene representation without problems of changing topology" and Pg2, First Bullet, "...enables end-to-end training of deep models with geometry and projection components”) such that the combination enables the system to render 2D predicted sensory images, compares them directly against received RGB-D sensor observations to compute a photometric loss, and perform end-to-end gradient descent updates on the 3D simulator geometry and parameters without topological constraints.
As to dependent Claim 24,
The combination of Valencia and Lassner teaches, as mentioned above, all the limitations of Claim 22. It teaches about manipulating the actions of robotic hand to manipulate the objects to match its physical geometry with the goal geometry.
Valencia further teaches the method of claim 22, wherein the simulation is used to provide open-loop control (Valencia, Pg9, Left Column, Lines3-7, "Multi-Step Predictions: The nodes position are predicted for every frame but with updates from observed data fed into the model at different frames (t>1), which involves a longer-term prediction before new data is made available to the GNN models", wherein Valencia discloses using the model/simulation to provide open-loop control by performing multi-step predictions without requiring continuous sensor data updates at every frame, which is functionally equivalent to the claimed invention's open-loop control.)
Valencia and Lassner are analogous to the claimed invention as they are from the same field of endeavor of 3D scene representation, dynamic object shape estimation, and rendering from sensor observations for computer vision and robotic applications. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the GNN-based 3D deformable object shape estimation and prediction framework of Valencia with the sphere-based differentiable rendering engine and photometric loss optimization loop of Lassner. The motivation is as recited by Lassner (Lassner, Pg2, Left Column, Paragraph1, Lines16-19, "This makes it easy to handle point cloud data from 3D sensors directly, allows for the optimization of the scene representation without problems of changing topology" and Pg2, First Bullet, "...enables end-to-end training of deep models with geometry and projection components”) such that the combination enables the system to render 2D predicted sensory images, compares them directly against received RGB-D sensor observations to compute a photometric loss, and perform end-to-end gradient descent updates on the 3D simulator geometry and parameters without topological constraints.
As to dependent Claim 28,
The combination of Valencia and Lassner teaches, as mentioned above, all the limitations of Claim 24. It teaches about the simulation is used to provide open-loop control.
Lassner further teaches the method of claim 24, wherein a derivative for the minimization technique is computed using auto-differentiation, finite difference, adjoint method or is analytically derived (Lassner, Pg2, Left Column, Paragraph1, Lines30-32, "Lastly, we integrate Pulsar with the PyTorch [29] optimization framework to make use of auto-differentiation and ease the integration with deep learning models.")
Valencia and Lassner are analogous to the claimed invention as they are from the same field of endeavor of 3D scene representation, dynamic object shape estimation, and rendering from sensor observations for computer vision and robotic applications. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the GNN-based 3D deformable object shape estimation and prediction framework of Valencia with the sphere-based differentiable rendering engine and photometric loss optimization loop of Lassner. The motivation is as recited by Lassner (Lassner, Pg2, Left Column, Paragraph1, Lines16-19, "This makes it easy to handle point cloud data from 3D sensors directly, allows for the optimization of the scene representation without problems of changing topology" and Pg2, First Bullet, "...enables end-to-end training of deep models with geometry and projection components”) such that the combination enables the system to render 2D predicted sensory images, compares them directly against received RGB-D sensor observations to compute a photometric loss, and perform end-to-end gradient descent updates on the 3D simulator geometry and parameters without topological constraints.
As to dependent Claim 29,
The combination of Valencia and Lassner teaches, as mentioned above, all the limitations of Claim 22. It teaches about manipulating the actions of robotic hand to manipulate the objects to match its physical geometry with the goal geometry.
Valencia further teaches the method of claim 1, wherein the one or more objects includes at least one deformable object (Valencia, Pg5, Section4.1, Lines3-5, "an Intel RealSense SR305 RGB-D sensor mounted overhead on a tripod, and a deformable object placed on the palm of the robotic hand.")
Valencia and Lassner are analogous to the claimed invention as they are from the same field of endeavor of 3D scene representation, dynamic object shape estimation, and rendering from sensor observations for computer vision and robotic applications. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the GNN-based 3D deformable object shape estimation and prediction framework of Valencia with the sphere-based differentiable rendering engine and photometric loss optimization loop of Lassner. The motivation is as recited by Lassner (Lassner, Pg2, Left Column, Paragraph1, Lines16-19, "This makes it easy to handle point cloud data from 3D sensors directly, allows for the optimization of the scene representation without problems of changing topology" and Pg2, First Bullet, "...enables end-to-end training of deep models with geometry and projection components”) such that the combination enables the system to render 2D predicted sensory images, compares them directly against received RGB-D sensor observations to compute a photometric loss, and perform end-to-end gradient descent updates on the 3D simulator geometry and parameters without topological constraints.
As to dependent Claim 36,
it is a non-transitory computer-readable medium claim that contains similar limitations of Claim 1 and thus rejected under the same rationale.
Claims 6, 7, 9, 10, 20, 35 are rejected under 35 U.S.C. 103 as being unpatentable over Valencia and Lassner as mentioned in Claim 1, in further view of Zhang et al. (Zhang), Non-Patent Literature (listed in IDS filed on 09/11/2023), “Deformable Models for Surgical Simulation: A Survey”, Published in December 2017, 22 Pages.
As to dependent Claim 6,
The combination of Valencia and Lassner teaches, as mentioned above, all the limitations of Claim 1. It teaches about overall architecture of using the sensory data to predict an object’s simulator geometry by minimizing its loss function which then provides a simulation that can be executed by a robot hand to manipulate the object.
Valencia teaches about using the physics engine such as GNN, however is silent about the following limitations about the simulator itself. From the same field of endeavor, Zhang teaches
the method of claim 1, wherein the simulator is a position-based dynamics simulator (Zhang, Pg4, Right Column, Section 4), Lines1-3, "Other deformable modeling methods such as the shape matching technique coupled with position-based solver [66] were also studied for soft tissue deformation", wherein Zhang explicitly discloses the position-based solver (the corresponding position-based dynamics simulator), which is functionally equivalent to the claimed invention.)
Valencia, Lassner and Zhang are analogous to the claimed invention as they are from the same field of endeavor of 3D computer graphics, dynamic object modeling, and real-time physical simulation. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the graph-based dynamic shape estimation and tracking mechanism for deformable objects of Valencia, the sphere-based differentiable rendering and scene optimization engine of Lassner with the soft tissue deformation modeling of Zhang. The motivation is as recited by Zhang (Zhang, Pg15, Section VII, Lines1-5, "Among various deformable models proposed for modeling of soft tissue deformation, it is obvious that there is no single deformable model that can address both requirements of realistic and real-time surgical simulation. Instead, they were developed in different ways to meet specific needs") such that the integration helps overcome the inherent trade-off between computational rendering speed and realistic physical accuracy, thereby establishing a unified 3D simulation framework capable of processing and rendering complex deformable object interactions in real time.
As to dependent Claim 7,
The combination of Valencia and Lassner teaches, as mentioned above, all the limitations of Claim 1. It teaches about overall architecture of using the sensory data to predict an object’s simulator geometry by minimizing its loss function which then provides a simulation that can be executed by a robot hand to manipulate the object.
Valencia teaches about using the physics engine such as GNN, however is silent about the following limitations about the simulator itself. From the same field of endeavor, Zhang teaches
the method of claim 1, wherein the simulator is a rigid body dynamics simulator (Zhang, Pg4, Right Column, Section 4), Lines3-6, "Shape matching is a geometrically-motivated approach based on finding the least squares optimal rigid transformations between two sets of points", wherein Zhang explicitly discloses the shape matching rigid transformation (the corresponding rigid body dynamics simulator), rendering it functionally equivalent to the claimed invention.)
Valencia, Lassner and Zhang are analogous to the claimed invention as they are from the same field of endeavor of 3D computer graphics, dynamic object modeling, and real-time physical simulation. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the graph-based dynamic shape estimation and tracking mechanism for deformable objects of Valencia, the sphere-based differentiable rendering and scene optimization engine of Lassner with the soft tissue deformation modeling of Zhang. The motivation is as recited by Zhang (Zhang, Pg15, Section VII, Lines1-5, "Among various deformable models proposed for modeling of soft tissue deformation, it is obvious that there is no single deformable model that can address both requirements of realistic and real-time surgical simulation. Instead, they were developed in different ways to meet specific needs") such that the integration helps overcome the inherent trade-off between computational rendering speed and realistic physical accuracy, thereby establishing a unified 3D simulation framework capable of processing and rendering complex deformable object interactions in real time.
As to dependent Claim 9,
The combination of Valencia and Lassner teaches, as mentioned above, all the limitations of Claim 1. It teaches about overall architecture of using the sensory data to predict an object’s simulator geometry by minimizing its loss function which then provides a simulation that can be executed by a robot hand to manipulate the object.
Valencia teaches about using the physics engine such as GNN, however is silent about the following limitations about the simulator itself. From the same field of endeavor, Zhang teaches
The method of claim 1, wherein the simulator is a smooth particular hydrodynamics simulator (Zhang, Pg5, Left Column, SectionB, Lines16-17, "smoothed particle hydrodynamics (SPH) in the category of meshless approach"
Pg8, Seciton 2.3), Lines2-5, "particle-based methods such as the smoothed particle hydrodynamics (SPH) and point-collocation-based method of finite spheres (PCMFS) were also studied for soft tissue deformation.")
Valencia, Lassner and Zhang are analogous to the claimed invention as they are from the same field of endeavor of 3D computer graphics, dynamic object modeling, and real-time physical simulation. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the graph-based dynamic shape estimation and tracking mechanism for deformable objects of Valencia, the sphere-based differentiable rendering and scene optimization engine of Lassner with the soft tissue deformation modeling of Zhang. The motivation is as recited by Zhang (Zhang, Pg15, Section VII, Lines1-5, "Among various deformable models proposed for modeling of soft tissue deformation, it is obvious that there is no single deformable model that can address both requirements of realistic and real-time surgical simulation. Instead, they were developed in different ways to meet specific needs") such that the integration helps overcome the inherent trade-off between computational rendering speed and realistic physical accuracy, thereby establishing a unified 3D simulation framework capable of processing and rendering complex deformable object interactions in real time.
As to dependent Claim 10,
The combination of Valencia and Lassner teaches, as mentioned above, all the limitations of Claim 1. It teaches about overall architecture of using the sensory data to predict an object’s simulator geometry by minimizing its loss function which then provides a simulation that can be executed by a robot hand to manipulate the object.
Valencia teaches about using the physics engine such as GNN, however is silent about the following limitations about the simulator itself. From the same field of endeavor, Zhang teaches
the method of claim 1, wherein the simulator is a finite element method-based dynamics simulator (Zhang, Pg5, Setion1.1), Lines1-5, "FEM(Finite Element Method) is a typical method for simulation and analysis of soft tissue deformation in surgical simulation, which requires explicit construction of the object mesh to approximate the constitutive laws governing the mechanical behaviors of soft tissues.")
Valencia, Lassner and Zhang are analogous to the claimed invention as they are from the same field of endeavor of 3D computer graphics, dynamic object modeling, and real-time physical simulation. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the graph-based dynamic shape estimation and tracking mechanism for deformable objects of Valencia, the sphere-based differentiable rendering and scene optimization engine of Lassner with the soft tissue deformation modeling of Zhang. The motivation is as recited by Zhang (Zhang, Pg15, Section VII, Lines1-5, "Among various deformable models proposed for modeling of soft tissue deformation, it is obvious that there is no single deformable model that can address both requirements of realistic and real-time surgical simulation. Instead, they were developed in different ways to meet specific needs") such that the integration helps overcome the inherent trade-off between computational rendering speed and realistic physical accuracy, thereby establishing a unified 3D simulation framework capable of processing and rendering complex deformable object interactions in real time.
As to dependent Claim 20,
The combination of Valencia and Lassner teaches, as mentioned above, all the limitations of Claim 1. It teaches about overall architecture of using the sensory data to predict an object’s simulator geometry by minimizing its loss function which then provides a simulation that can be executed by a robot hand to manipulate the object.
Valencia teaches about manipulating deformable objects, but does not teach the following limitation about the object. From the same field of endeavor, Zhang teaches
the method of claim 1, wherein the one or more objects includes at least one liquid (Zhang, Pg9, Right Column, Lines23-26, "Costa [134] presented a fast deformation model based on the principle of Pascal and the conservation of volume to simulate deformation of soft tissues formed by fibers and fluid", wherein Zhang discloses soft tissues that consists fibers and fluid or the liquid, which is functionally equivalent to the claimed invention.)
Valencia, Lassner and Zhang are analogous to the claimed invention as they are from the same field of endeavor of 3D computer graphics, dynamic object modeling, and real-time physical simulation. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the graph-based dynamic shape estimation and tracking mechanism for deformable objects of Valencia, the sphere-based differentiable rendering and scene optimization engine of Lassner with the soft tissue deformation modeling of Zhang. The motivation is as recited by Zhang (Zhang, Pg15, Section VII, Lines1-5, "Among various deformable models proposed for modeling of soft tissue deformation, it is obvious that there is no single deformable model that can address both requirements of realistic and real-time surgical simulation. Instead, they were developed in different ways to meet specific needs") such that the integration helps overcome the inherent trade-off between computational rendering speed and realistic physical accuracy, thereby establishing a unified 3D simulation framework capable of processing and rendering complex deformable object interactions in real time.
As to dependent Claim 35,
The combination of Valencia and Lassner teaches, as mentioned above, all the limitations of Claim 22. It teaches about manipulating the actions of robotic hand to manipulate the objects to match its physical geometry with the goal geometry.
Valencia teaches about manipulating deformable objects, but does not teach the following limitation about the object. From the same field of endeavor, Zhang teaches
the method of claim 1, wherein the one or more objects includes at least one liquid (Zhang, Pg9, Right Column, Lines23-26, "Costa [134] presented a fast deformation model based on the principle of Pascal and the conservation of volume to simulate deformation of soft tissues formed by fibers and fluid", wherein Zhang discloses soft tissues that consists fibers and fluid or the liquid, which is functionally equivalent to the claimed invention.)
Valencia, Lassner and Zhang are analogous to the claimed invention as they are from the same field of endeavor of 3D computer graphics, dynamic object modeling, and real-time physical simulation. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the graph-based dynamic shape estimation and tracking mechanism for deformable objects of Valencia, the sphere-based differentiable rendering and scene optimization engine of Lassner with the soft tissue deformation modeling of Zhang. The motivation is as recited by Zhang (Zhang, Pg15, Section VII, Lines1-5, "Among various deformable models proposed for modeling of soft tissue deformation, it is obvious that there is no single deformable model that can address both requirements of realistic and real-time surgical simulation. Instead, they were developed in different ways to meet specific needs") such that the integration helps overcome the inherent trade-off between computational rendering speed and realistic physical accuracy, thereby establishing a unified 3D simulation framework capable of processing and rendering complex deformable object interactions in real time.
Claims 8, 11, 12, 16, 17, 21, 31, 32 are rejected under 35 U.S.C. 103 as being unpatentable over Valencia and Lassner as mentioned in Claim 1, in further view of Li et al. (Li), Non-Patent Literature, “Soft Articulated Characters in Projective Dynamics”, Published in July 2020, 12 Pages.
As to dependent Claim 8,
The combination of Valencia and Lassner teaches, as mentioned above, all the limitations of Claim 1. It teaches about overall architecture of using the sensory data to predict an object’s simulator geometry by minimizing its loss function which then provides a simulation that can be executed by a robot hand to manipulate the object.
Valencia teaches about using the physics engine such as GNN, however is silent about the following limitations about the simulator itself. From the same field of endeavor, Li teaches
the method of claim 1, wherein the simulator is an articulated rigid body dynamics simulator (Li, Pg1385, Abstract, Lines1-2, "We propose a fast and robust solver to simulate continuum-based deformable models with constraints, in particular, rigid body and joint constraints useful for soft articulated characters", wherein Li explicitly discloses the rigid body and joint constraints (the corresponding articulated rigid body) simulations, rendering it functionally equivalent to the claimed invention.)
Valencia, Lassner and Li are analogous to the claimed invention as they are from the same field of endeavor of 3D computer graphics, dynamic object modeling, and real-time physical simulation. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the graph-based dynamic shape estimation and tracking mechanism for deformable objects of Valencia, the sphere-based differentiable rendering and scene optimization engine of Lassner with the Projective Dynamics framework for two-way coupling of rigid and deformable bodies of Li. The motivation is as recited by Li (Li, Pg1385, Introduction, Lines15-17, "The mechanical interplay between rigid and deformable bodies results in numerically challenging simulation problems") such that the integration helps to resolve the numerical challenges and computational bottlenecks inherent in soft articulated body interactions, thereby establishing a highly stable, scalable, and computationally efficient real-time simulation system capable of modeling and rendering complex multi-object mechanical behaviors.
As to dependent Claim 11,
The combination of Valencia and Lassner teaches, as mentioned above, all the limitations of Claim 1. It teaches about overall architecture of using the sensory data to predict an object’s simulator geometry by minimizing its loss function which then provides a simulation that can be executed by a robot hand to manipulate the object.
Valencia teaches about using the physics engine such as GNN, however is silent about the following limitations about the simulator itself. From the same field of endeavor, Li teaches
the method of claim 1, wherein the simulator is a projective dynamics simulator (Li, Pg1385, Introduction, Paragraph2, Lines1-4, "Projective Dynamics [1] is an implicit Euler solver used in real-time deformable object simulation. It exploits a special potential energy structure which enables an efficient local/global solver...")
Valencia, Lassner and Li are analogous to the claimed invention as they are from the same field of endeavor of 3D computer graphics, dynamic object modeling, and real-time physical simulation. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the graph-based dynamic shape estimation and tracking mechanism for deformable objects of Valencia, the sphere-based differentiable rendering and scene optimization engine of Lassner with the Projective Dynamics framework for two-way coupling of rigid and deformable bodies of Li. The motivation is as recited by Li (Li, Pg1385, Introduction, Lines15-17, "The mechanical interplay between rigid and deformable bodies results in numerically challenging simulation problems") such that the integration helps to resolve the numerical challenges and computational bottlenecks inherent in soft articulated body interactions, thereby establishing a highly stable, scalable, and computationally efficient real-time simulation system capable of modeling and rendering complex multi-object mechanical behaviors.
As to dependent Claim 12,
The combination of Valencia and Lassner teaches, as mentioned above, all the limitations of Claim 1. It teaches about overall architecture of using the sensory data to predict an object’s simulator geometry by minimizing its loss function which then provides a simulation that can be executed by a robot hand to manipulate the object.
Valencia teaches about using the physics engine such as GNN, however is silent about the following limitations about the simulator itself. From the same field of endeavor, Li teaches
the method of claim 1, wherein the simulator is an energy projection-based dynamics simulator (Li, Pg1385, Introduction, Paragraph2, Lines1-4, "Projective Dynamics [1] is an implicit Euler solver used in real-time deformable object simulation. It exploits a special potential energy structure which enables an efficient local/global solver..."
Pg1388, Section 3.2, Lines4-6, "For each element i, the Projective Dynamics energy Ei(x,p_i)..."
Pg1389, Right Column, Paragraph1, Lines1-3, "Local step, Our first nonlinear component is contained in p which is a stacked vector of projections from the deformation gradient of each deformable element", wherein Li discloses that projective dynamics itself uses potential energy E(x) which is projected in the local step with the condition of deformation gradient, rendering it functionally equivalent to the claimed invention.)
Valencia, Lassner and Li are analogous to the claimed invention as they are from the same field of endeavor of 3D computer graphics, dynamic object modeling, and real-time physical simulation. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the graph-based dynamic shape estimation and tracking mechanism for deformable objects of Valencia, the sphere-based differentiable rendering and scene optimization engine of Lassner with the Projective Dynamics framework for two-way coupling of rigid and deformable bodies of Li. The motivation is as recited by Li (Li, Pg1385, Introduction, Lines15-17, "The mechanical interplay between rigid and deformable bodies results in numerically challenging simulation problems") such that the integration helps to resolve the numerical challenges and computational bottlenecks inherent in soft articulated body interactions, thereby establishing a highly stable, scalable, and computationally efficient real-time simulation system capable of modeling and rendering complex multi-object mechanical behaviors.
As to dependent Claim 16,
The combination of Valencia and Lassner teaches, as mentioned above, all the limitations of Claim 1. It teaches about overall architecture of using the sensory data to predict an object’s simulator geometry by minimizing its loss function which then provides a simulation that can be executed by a robot hand to manipulate the object.
Valencia teaches about manipulating deformable objects, but does not teach the following limitation about the object. From the same field of endeavor, Li teaches
the method of claim 1, wherein the one or more objects includes at least one rigid body (Li, Pg1385, Introduction, Lines3-5, "The animation of articulated characters often models the body as a collection of rigid bodies (“Ragdoll physics”)", wherein Li explicitly discloses objects include rigid bodies, rendering it functionally equivalent to the claimed invention.)
Valencia, Lassner and Li are analogous to the claimed invention as they are from the same field of endeavor of 3D computer graphics, dynamic object modeling, and real-time physical simulation. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the graph-based dynamic shape estimation and tracking mechanism for deformable objects of Valencia, the sphere-based differentiable rendering and scene optimization engine of Lassner with the Projective Dynamics framework for two-way coupling of rigid and deformable bodies of Li. The motivation is as recited by Li (Li, Pg1385, Introduction, Lines15-17, "The mechanical interplay between rigid and deformable bodies results in numerically challenging simulation problems") such that the integration helps to resolve the numerical challenges and computational bottlenecks inherent in soft articulated body interactions, thereby establishing a highly stable, scalable, and computationally efficient real-time simulation system capable of modeling and rendering complex multi-object mechanical behaviors.
As to dependent Claim 17,
The combination of Valencia and Lassner teaches, as mentioned above, all the limitations of Claim 1. It teaches about overall architecture of using the sensory data to predict an object’s simulator geometry by minimizing its loss function which then provides a simulation that can be executed by a robot hand to manipulate the object.
Valencia teaches about manipulating deformable objects, but does not teach the following limitation about the object. From the same field of endeavor, Li teaches
the method of claim 1, wherein the one or more objects includes at least one articulated rigid body (Li, Pg1385, Abstract, Lines2-3, "Our method embeds the degrees of freedom of both articulated rigid bodies and deformable bodies.”)
Valencia, Lassner and Li are analogous to the claimed invention as they are from the same field of endeavor of 3D computer graphics, dynamic object modeling, and real-time physical simulation. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the graph-based dynamic shape estimation and tracking mechanism for deformable objects of Valencia, the sphere-based differentiable rendering and scene optimization engine of Lassner with the Projective Dynamics framework for two-way coupling of rigid and deformable bodies of Li. The motivation is as recited by Li (Li, Pg1385, Introduction, Lines15-17, "The mechanical interplay between rigid and deformable bodies results in numerically challenging simulation problems") such that the integration helps to resolve the numerical challenges and computational bottlenecks inherent in soft articulated body interactions, thereby establishing a highly stable, scalable, and computationally efficient real-time simulation system capable of modeling and rendering complex multi-object mechanical behaviors.
As to dependent Claim 21,
The combination of Valencia and Lassner teaches, as mentioned above, all the limitations of Claim 1. It teaches about overall architecture of using the sensory data to predict an object’s simulator geometry by minimizing its loss function which then provides a simulation that can be executed by a robot hand to manipulate the object.
Valencia teaches about manipulating deformable objects, but does not teach the following limitation about the object. From the same field of endeavor, Li teaches
the method of claim 1, wherein the one or more objects includes at least two different objects that interact with one another (Li, Pg1385, Abstract, Lines6-7, "rigid-body parts (bones) being correctly coupled with deformable parts (flesh)"
Pg1385, Introduction, Lines15-17, "The mechanical interplay between rigid and deformable bodies results in numerically challenging simulation problems"
Pg1386, Section2.3, Lines1-5, "Coupling rigid and deformable bodies is an interesting topic that has been explored by many authors [29], [30], [31], [32], [33]. One common approach is to simulate each subsystem with specialized technique and then bridge the two together", wherein Li explicitly discloses two objects such as the rigid bodies and deformable parts interacting with one another, which is functionally equivalent to the claimed invention.)
Valencia, Lassner and Li are analogous to the claimed invention as they are from the same field of endeavor of 3D computer graphics, dynamic object modeling, and real-time physical simulation. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the graph-based dynamic shape estimation and tracking mechanism for deformable objects of Valencia, the sphere-based differentiable rendering and scene optimization engine of Lassner with the Projective Dynamics framework for two-way coupling of rigid and deformable bodies of Li. The motivation is as recited by Li (Li, Pg1385, Introduction, Lines15-17, "The mechanical interplay between rigid and deformable bodies results in numerically challenging simulation problems") such that the integration helps to resolve the numerical challenges and computational bottlenecks inherent in soft articulated body interactions, thereby establishing a highly stable, scalable, and computationally efficient real-time simulation system capable of modeling and rendering complex multi-object mechanical behaviors.
As to dependent Claim 31,
The combination of Valencia and Lassner teaches, as mentioned above, all the limitations of Claim 22. It teaches about manipulating the actions of robotic hand to manipulate the objects to match its physical geometry with the goal geometry.
Valencia teaches about manipulating deformable objects, but does not teach the following limitation about the object. From the same field of endeavor, Li teaches
the method of claim 22, wherein the one or more objects includes at least one rigid body (Li, Pg1385, Introduction, Lines3-5, "The animation of articulated characters often models the body as a collection of rigid bodies (“Ragdoll physics”)", wherein Li explicitly discloses objects include rigid bodies, rendering it functionally equivalent to the claimed invention.)
Valencia, Lassner and Li are analogous to the claimed invention as they are from the same field of endeavor of 3D computer graphics, dynamic object modeling, and real-time physical simulation. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the graph-based dynamic shape estimation and tracking mechanism for deformable objects of Valencia, the sphere-based differentiable rendering and scene optimization engine of Lassner with the Projective Dynamics framework for two-way coupling of rigid and deformable bodies of Li. The motivation is as recited by Li (Li, Pg1385, Introduction, Lines15-17, "The mechanical interplay between rigid and deformable bodies results in numerically challenging simulation problems") such that the integration helps to resolve the numerical challenges and computational bottlenecks inherent in soft articulated body interactions, thereby establishing a highly stable, scalable, and computationally efficient real-time simulation system capable of modeling and rendering complex multi-object mechanical behaviors.
As to dependent Claim 32,
The combination of Valencia and Lassner teaches, as mentioned above, all the limitations of Claim 22. It teaches about manipulating the actions of robotic hand to manipulate the objects to match its physical geometry with the goal geometry.
Valencia teaches about manipulating deformable objects, but does not teach the following limitation about the object. From the same field of endeavor, Li teaches
the method of claim 22, wherein the one or more objects includes at least one articulated rigid body (Li, Pg1385, Abstract, Lines2-3, "Our method embeds the degrees of freedom of both articulated rigid bodies and deformable bodies.”)
Valencia, Lassner and Li are analogous to the claimed invention as they are from the same field of endeavor of 3D computer graphics, dynamic object modeling, and real-time physical simulation. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the graph-based dynamic shape estimation and tracking mechanism for deformable objects of Valencia, the sphere-based differentiable rendering and scene optimization engine of Lassner with the Projective Dynamics framework for two-way coupling of rigid and deformable bodies of Li. The motivation is as recited by Li (Li, Pg1385, Introduction, Lines15-17, "The mechanical interplay between rigid and deformable bodies results in numerically challenging simulation problems") such that the integration helps to resolve the numerical challenges and computational bottlenecks inherent in soft articulated body interactions, thereby establishing a highly stable, scalable, and computationally efficient real-time simulation system capable of modeling and rendering complex multi-object mechanical behaviors.
Claims 18, 19, 33, 34 are rejected under 35 U.S.C. 103 as being unpatentable over Valencia and Lassner as mentioned in Claim 1, in further view of Chi et al. (Chi), Non-Patent Literature, “Occlusion-robust Deformable Object Tracking without Physics Simulation”, Published in November 2019, 8 Pages.
As to dependent Claim 18,
The combination of Valencia and Lassner teaches, as mentioned above, all the limitations of Claim 1. It teaches about overall architecture of using the sensory data to predict an object’s simulator geometry by minimizing its loss function which then provides a simulation that can be executed by a robot hand to manipulate the object.
Valencia teaches about manipulating deformable objects, but does not teach the following limitation about the object. From the same field of endeavor, Chi teaches
the method of claim 1, wherein the one or more objects includes at least one deformable linear object (Chi, Pg6443, Introduction, Lines1-3, "Tracking the geometry of deformable objects such as rope and cloth is difficult due to the continuous nature of the object"
Pg6448, Section VI, Lines1-3, "We conducted several experiments tracking rope and cloth to test the performance of our algorithm both quantitatively and qualitatively", wherein Chi explicitly discloses a rope (the corresponding deformable linear object) as a deformable object and it is being used in tracking the geometry, rendering it functionally equivalent to the claimed invention.)
Valencia, Lassner and Chi are analogous to the claimed invention as they are from the same field of endeavor of 3D computer graphics, dynamic object modeling, and real-time physical simulation. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the graph-based dynamic shape estimation and tracking mechanism for deformable objects of Valencia, the sphere-based differentiable rendering and scene optimization engine of Lassner with the occlusion-robust tracking framework enforcing topological consistency and distance constraints for deformable linear objects of Chi. The motivation is as recited by Chi (Chi, Pg6443, Introduction, Lines1-10, "Tracking the geometry of deformable objects such as rope and cloth is difficult due to the continuous nature of the object (i.e. an infinite number of degrees of freedom), the lack of knowledge of the physical parameters of the object, and often a lack of visual features to track. In addition to these challenges, to be useful for robotic manipulation of deformable objects, a tracking algorithm must operate within a small computational budget and must be able to handle object self-occlusion (e.g. folding) and occlusion by objects in the environment") such that the combination helps overcome tracking failures caused by severe occlusions and high degrees of freedom, thereby establishing a real-time, computationally efficient, and robust 3D tracking system capable of maintaining model topology for dynamic linear structures like ropes during manipulation.
As to dependent Claim 19,
The combination of Valencia, Lassner and Chi teaches, as mentioned above, all the limitations of Claim 18. It teaches about overall architecture of using the sensory data to predict an object’s simulator geometry, wherein the objects can be deformable linear object. by minimizing its loss function which then provides a simulation that can be executed by a robot hand to manipulate the object.
Chi further teaches the method of claim 18, wherein the at least one deformable linear object is selected from the group consisting of rope, suture thread and tendons (Chi, Pg6443, Introduction, Lines1-3, "Tracking the geometry of deformable objects such as rope and cloth is difficult due to the continuous nature of the object"
Pg6448, Section VI, Lines1-3, "We conducted several experiments tracking rope and cloth to test the performance of our algorithm both quantitatively and qualitatively.")
Valencia, Lassner and Chi are analogous to the claimed invention as they are from the same field of endeavor of 3D computer graphics, dynamic object modeling, and real-time physical simulation. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the graph-based dynamic shape estimation and tracking mechanism for deformable objects of Valencia, the sphere-based differentiable rendering and scene optimization engine of Lassner with the occlusion-robust tracking framework enforcing topological consistency and distance constraints for deformable linear objects of Chi. The motivation is as recited by Chi (Chi, Pg6443, Introduction, Lines1-10, "Tracking the geometry of deformable objects such as rope and cloth is difficult due to the continuous nature of the object (i.e. an infinite number of degrees of freedom), the lack of knowledge of the physical parameters of the object, and often a lack of visual features to track. In addition to these challenges, to be useful for robotic manipulation of deformable objects, a tracking algorithm must operate within a small computational budget and must be able to handle object self-occlusion (e.g. folding) and occlusion by objects in the environment") such that the combination helps overcome tracking failures caused by severe occlusions and high degrees of freedom, thereby establishing a real-time, computationally efficient, and robust 3D tracking system capable of maintaining model topology for dynamic linear structures like ropes during manipulation.
As to dependent Claim 33,
The combination of Valencia and Lassner teaches, as mentioned above, all the limitations of Claim 22. It teaches about manipulating the actions of robotic hand to manipulate the objects to match its physical geometry with the goal geometry.
Valencia teaches about manipulating deformable objects, but does not teach the following limitation about the object. From the same field of endeavor, Chi teaches
the method of claim 22, wherein the one or more objects includes at least one deformable linear object (Chi, Pg6443, Introduction, Lines1-3, "Tracking the geometry of deformable objects such as rope and cloth is difficult due to the continuous nature of the object"
Pg6448, Section VI, Lines1-3, "We conducted several experiments tracking rope and cloth to test the performance of our algorithm both quantitatively and qualitatively", wherein Chi explicitly discloses a rope (the corresponding deformable linear object) as a deformable object and it is being used in tracking the geometry, rendering it functionally equivalent to the claimed invention.)
Valencia, Lassner and Chi are analogous to the claimed invention as they are from the same field of endeavor of 3D computer graphics, dynamic object modeling, and real-time physical simulation. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the graph-based dynamic shape estimation and tracking mechanism for deformable objects of Valencia, the sphere-based differentiable rendering and scene optimization engine of Lassner with the occlusion-robust tracking framework enforcing topological consistency and distance constraints for deformable linear objects of Chi. The motivation is as recited by Chi (Chi, Pg6443, Introduction, Lines1-10, "Tracking the geometry of deformable objects such as rope and cloth is difficult due to the continuous nature of the object (i.e. an infinite number of degrees of freedom), the lack of knowledge of the physical parameters of the object, and often a lack of visual features to track. In addition to these challenges, to be useful for robotic manipulation of deformable objects, a tracking algorithm must operate within a small computational budget and must be able to handle object self-occlusion (e.g. folding) and occlusion by objects in the environment") such that the combination helps overcome tracking failures caused by severe occlusions and high degrees of freedom, thereby establishing a real-time, computationally efficient, and robust 3D tracking system capable of maintaining model topology for dynamic linear structures like ropes during manipulation.
As to dependent Claim 34,
The combination of Valencia and Lassner teaches, as mentioned above, all the limitations of Claim 33. It teaches about manipulating objects such that its physical geometry and the goal geometry matches, wherein the objects can be deformable linear object.
Chi further teaches the method of claim 33, wherein the at least one deformable linear object is selected from the group consisting of rope, suture thread and tendons (Chi, Pg6443, Introduction, Lines1-3, "Tracking the geometry of deformable objects such as rope and cloth is difficult due to the continuous nature of the object"
Pg6448, Section VI, Lines1-3, "We conducted several experiments tracking rope and cloth to test the performance of our algorithm both quantitatively and qualitatively.")
Valencia, Lassner and Chi are analogous to the claimed invention as they are from the same field of endeavor of 3D computer graphics, dynamic object modeling, and real-time physical simulation. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the graph-based dynamic shape estimation and tracking mechanism for deformable objects of Valencia, the sphere-based differentiable rendering and scene optimization engine of Lassner with the occlusion-robust tracking framework enforcing topological consistency and distance constraints for deformable linear objects of Chi. The motivation is as recited by Chi (Chi, Pg6443, Introduction, Lines1-10, "Tracking the geometry of deformable objects such as rope and cloth is difficult due to the continuous nature of the object (i.e. an infinite number of degrees of freedom), the lack of knowledge of the physical parameters of the object, and often a lack of visual features to track. In addition to these challenges, to be useful for robotic manipulation of deformable objects, a tracking algorithm must operate within a small computational budget and must be able to handle object self-occlusion (e.g. folding) and occlusion by objects in the environment") such that the combination helps overcome tracking failures caused by severe occlusions and high degrees of freedom, thereby establishing a real-time, computationally efficient, and robust 3D tracking system capable of maintaining model topology for dynamic linear structures like ropes during manipulation.
Claims 25, 26, 27, 30 are rejected under 35 U.S.C. 103 as being unpatentable over Valencia and Lassner as mentioned in Claim 1, in further view of Wu et al. (Wu), Non-Patent Literature, “Amphibious Robot’s Trajectory Tracking with DNN-Based Nonlinear Model Predictive Control”, Published in July 2020, 6 Pages.
As to dependent Claim 25,
The combination of Valencia and Lassner teaches, as mentioned above, all the limitations of Claim 22. It teaches about manipulating the actions of robotic hand to manipulate the objects to match its physical geometry with the goal geometry.
Valencia discloses predicting object shape changes to plan and support robotic manipulation actions (Valencia, Pg1, Abstract, Lines10-11, "with the goal to plan and support the manipulation actions of a robotic hand", Pg4, Section3.2, Lines5-7, "With the objective to support the requirements of path planning and dynamic interaction of a robotic hand with a deformable object..."). Although Valencia does not explicitly recite the specific objective function formula for calculating a control loss, Wu from the same field of endeavor teaches computing a control loss between the goal geometry and the simulator geometry and minimizing the control loss to compute a sequence of robot actions that are used to manipulate the one or more objects (Wu, Pg2021, Section IV, Lines1-9, "Based on the DNN-based model (3) given in Eq.(3), a model predict controller is designed to obtain an optimal control sequence over a given prediction period by minimizing a predefined objective function. From Fig.6 (b), the objective function is formulated as :
PNG
media_image7.png
188
379
media_image7.png
Greyscale
"
Pg2020, Section III, Lines1-3, "Model predictive control is a kind of model-based control method of obtaining optimal control sequences by minimizing an objective function under some constraints", wherein Wu explicitly teaches a model predictive control (MPC) framework that computes a control loss (cost function J(k) between a desired goal state X_d (the corresponding goal geometry) and a predicted model state X (the corresponding simulator geometry), and minimizes said loss to compute an optimal sequence of robot control actions (u_k+j), rendering it functionally equivalent to the claimed invention.)
Valencia, Lassner and Wu are analogous to the claimed invention as they are from the same field of endeavor of model-based robotic manipulation, physics-based simulation, and visual-perception-driven control tracking. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the graph-based dynamic shape estimation and tracking mechanism for deformable objects of Valencia, the sphere-based differentiable rendering and scene optimization engine of Lassner with the nonlinear model predictive control framework that calculates a control loss between predicted states and goal states to determine an optimal control sequence of Wu. The motivation is as recited by Wu (Wu, Pg2019, Abstract, Lines3-5, "Efficient and accurate control strategies can guarantee high performance of the robot’s trajectory tracking tasks") such that integrating Valencia's physics simulation dynamics and Lassner's visual rendering optimization into Wu's control loss minimization framework enables to continuously update simulator parameters against real-time sensory image data and derive precise sequences of robot control actions to achieve a target goal geometry.
As to dependent Claim 26,
The combination of Valencia and Lassner teaches, as mentioned above, all the limitations of Claim 22. It teaches about manipulating the actions of robotic hand to manipulate the objects to match its physical geometry with the goal geometry.
Valencia discloses executing real-time control actions using a robotic gripper on a deformable object to accomplish a shape control task (Valencia, Pg2, Right Column, Lines7-8, "...where a real robotic gripper performs a shape control task on a deformable object"). From the same field of endeavor, Wu further elaborates this by explicitly disclosing the algorithm used to convert the control sequence which the sequence actually executes to move the robots as computed (Wu, Pg2022, Algorithm1, Line12 "C_b = convert control sequence to robot's command (S_b)", which is functionally equivalent to the claimed invention when they are combined.)
Valencia, Lassner and Wu are analogous to the claimed invention as they are from the same field of endeavor of model-based robotic manipulation, physics-based simulation, and visual-perception-driven control tracking. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the graph-based dynamic shape estimation and tracking mechanism for deformable objects of Valencia, the sphere-based differentiable rendering and scene optimization engine of Lassner with the nonlinear model predictive control framework that calculates a control loss between predicted states and goal states to determine an optimal control sequence of Wu. The motivation is as recited by Wu (Wu, Pg2019, Abstract, Lines3-5, "Efficient and accurate control strategies can guarantee high performance of the robot’s trajectory tracking tasks") such that integrating Valencia's physics simulation dynamics and Lassner's visual rendering optimization into Wu's control loss minimization framework enables to continuously update simulator parameters against real-time sensory image data and derive precise sequences of robot control actions to achieve a target goal geometry.
As to dependent Claim 27,
The combination of Valencia, Lassner and Wu teaches, as mentioned above, all the limitations of Claim 25. It teaches about executing the actual robot to manipulate the object such that objects can match its physical geometry with the goal geometry.
Wu further teaches the method of claim 25, wherein minimizing the control loss uses a minimization technique selected from the group consisting of gradient descent, a Levenberg-Marquardt algorithm, a Trust Region Optimization technique, and a Gauss-Newton algorithm (Wu, Pg2022, Left Column, Paragraph2, Lines1-3, "Typically, the minimum value of objection function can be found iteratively using some nonlinear optimization methods like constrained Gauss-Newton algorithm", wherein Wu explicitly discloses the optimization uses the corresponding Gauss-Newton algorithm, which is functionally equivalent to the claimed invention.)
Valencia, Lassner and Wu are analogous to the claimed invention as they are from the same field of endeavor of model-based robotic manipulation, physics-based simulation, and visual-perception-driven control tracking. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the graph-based dynamic shape estimation and tracking mechanism for deformable objects of Valencia, the sphere-based differentiable rendering and scene optimization engine of Lassner with the nonlinear model predictive control framework that calculates a control loss between predicted states and goal states to determine an optimal control sequence of Wu. The motivation is as recited by Wu (Wu, Pg2019, Abstract, Lines3-5, "Efficient and accurate control strategies can guarantee high performance of the robot’s trajectory tracking tasks") such that integrating Valencia's physics simulation dynamics and Lassner's visual rendering optimization into Wu's control loss minimization framework enables to continuously update simulator parameters against real-time sensory image data and derive precise sequences of robot control actions to achieve a target goal geometry.
As to dependent Claim 30,
The combination of Valencia, Lassner and Wu teaches, as mentioned above, all the limitations of Claim 25. It teaches about executing the actual robot to manipulate the object such that objects can match its physical geometry with the goal geometry.
Valencia further teaches the method of claim 1, wherein the one or more objects includes at least one deformable object (Valencia, Pg5, Section4.1, Lines3-5, "an Intel RealSense SR305 RGB-D sensor mounted overhead on a tripod, and a deformable object placed on the palm of the robotic hand.")
Valencia, Lassner and Wu are analogous to the claimed invention as they are from the same field of endeavor of model-based robotic manipulation, physics-based simulation, and visual-perception-driven control tracking. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the graph-based dynamic shape estimation and tracking mechanism for deformable objects of Valencia, the sphere-based differentiable rendering and scene optimization engine of Lassner with the nonlinear model predictive control framework that calculates a control loss between predicted states and goal states to determine an optimal control sequence of Wu. The motivation is as recited by Wu (Wu, Pg2019, Abstract, Lines3-5, "Efficient and accurate control strategies can guarantee high performance of the robot’s trajectory tracking tasks") such that integrating Valencia's physics simulation dynamics and Lassner's visual rendering optimization into Wu's control loss minimization framework enables to continuously update simulator parameters against real-time sensory image data and derive precise sequences of robot control actions to achieve a target goal geometry.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DONG YOON JUNG whose telephone number is (571)270-0198. The examiner can normally be reached 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Cesar Paula can be reached at (571) 272-4128. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DONG YOON JUNG/Examiner, Art Unit 2145
/CESAR B PAULA/Supervisory Patent Examiner, Art Unit 2145
Prosecution Insights