DETAILED ACTION
This Office Action is in response to the claims filed on 08/16/2023.
Claims 1-20 are pending.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Examiner Notes
Examiner cites particular columns, paragraphs, figures and line numbers in the
references as applied to the claims below for the convenience of the applicant. Although
the specified citations are representative of the teachings in the art and are applied to
the specific limitations within the individual claim, other passages and figures may apply
as well. Examiner may also include cited interpretations encompassed within parenthesis, e.g. (Examiner’s interpretation), for clarity. It is respectfully requested that, in preparing responses, the applicant fully consider the references in their entirety as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior art or disclosed by the examiner. The entire reference is considered to provide disclosure relating to the claimed invention. The claims & only the claims form the metes & bounds of the invention. Office personnel are to give the claims their broadest reasonable interpretation in light of the supporting disclosure. Unclaimed limitations appearing in the specification are not read into the claim. Prior art was referenced using terminology familiar to one of ordinary skill in the art. Such an approach is broad in concept and can be either explicit or implicit in meaning. Examiner's Notes are provided with the cited references to assist the applicant to better understand how the examiner interprets the applied prior art. Such comments are entirely consistent with the intent & spirit of compact prosecution.
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 9/15/2023, 11/05/2024, 2/25/2025, 10/8/2025, and 3/25/226 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention recites a judicial exception, is directed to that judicial exception (an abstract idea), as it has not been integrated into a practical application and the claim(s) further do/does not recite significantly more than the judicial exception. Examiner has evaluated the claim(s) under the framework provided in MPEP 2106 and has provided such analysis below.
To determine if a claim is directed to patent ineligible subject matter, the Court
has guided the Office to apply the Alice/Mayo test, which requires:
Step 1. Determining if the claim falls within a statutory category of a Process, Machine, Manufacture, or a Composition of Matter (see MPEP 2106.03);
Step 2A. Determining if the claim is directed to a patent ineligible judicial exception consisting of a law of nature, a natural phenomenon, or abstract idea (MPEP 2106.04);
Step 2A is a two-prong inquiry. MPEP 2106.04(II)(A).
Under the first prong, examiners evaluate whether a law of nature, natural phenomenon, or abstract idea is set forth or described in the claim. Abstract ideas include mathematical concepts, certain methods of organizing human activity, and mental processes. MPEP 2106.04(a)(2).
The second prong is an inquiry into whether the claim integrates a judicial exception into a practical application. MPEP 2106.04(d).
Step 2B. If the claim is directed to a judicial exception, determining if the claim recites limitations or elements that amount to significantly more than the judicial exception. (See MPEP 2106).
Step 1:
Claims 1-18 are directed to a device, as such these claims fall within the statutory category of a machine.
Claim 19 is directed to a method, as such the claim falls within the statutory category of a process.
Claim 20 is directed to a non-transitory computer readable medium, as such the claim falls within the statutory category of manufacture.
Step 2A, Prong 1:
The examiner submits that the foregoing claim limitations constitute abstract ideas, as the claims cover Mental Processes, given the broadest reasonable interpretation.
In order to apply Step 2A, a recitation of claims is copied below. The limitations of those claims which describe an abstract idea are bolded.
As per claim 1, the claim recites the limitations of:
generate, based on the measured data, an actual shape model representing
a three-dimensional real shape of the machine system (As drafted and under its broadest reasonable interpretation, this limitation amounts to Mental Processes (MPEP 2106.04(a)(2)(III)) which are defined as concepts that can practically be performed in the human mind (e.g. observations, evaluations, judgments, opinions), or by a human using pen and paper as a physical aid. For instance, a person can reasonably evaluate measurement data and then draw (i.e. generate) a 3D real shape of a machine system.);
correct the simulation model of the machine system based on a comparison between the simulation model and the actual shape model (As drafted and under its broadest reasonable interpretation, this limitation amounts to Mental Processes (MPEP 2106.04(a)(2)(III)). For instance, a person can reasonably compare (i.e. observe, evaluate, judge) two models and then correct a model based on that comparison, with/without the aid of pen/paper.)
Step 2A, Prong 2:
As per claim 1, this judicial exception is not integrated into a practical application because the additional claim limitations outside the abstract idea only present Insignificant Extra Solution Activity. In particular, the claim recites the additional limitations:
store a simulation model of a machine system including a robot, the
simulation model generated to simulate a three-dimensional real shape of the machine system (The additional element amounts to Insignificant Extra-solution Activity (mere data gathering, pre-solution activity) per MPEP 2106.05(g). The term "extra-solution activity" can be understood as activities incidental to the primary process or product that are merely a nominal or tangential addition to the claim. Extra-solution activity includes both pre-solution and post-solution activity. An example of pre-solution activity is a step of gathering data for use in a claimed process. Note: the act of storing a simulation model does not meaningfully limit the claim, as storing data is considered well understood, routine, and conventional (see MPEP 2106.05(d));
receive measured data acquired by measuring the machine system in a real
space (The additional element amounts to Insignificant Extra-solution Activity (mere data gathering, pre-solution activity) per MPEP 2106.05(g). The term "extra-solution activity" can be understood as activities incidental to the primary process or product that are merely a nominal or tangential addition to the claim. Extra-solution activity includes both pre-solution and post-solution activity. An example of pre-solution activity is a step of gathering data for use in a claimed process.);
Accordingly, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea when considered as an ordered combination and as a whole.
Step 2B:
For step 2B of the analysis, the Examiner must consider whether each claim limitation individually or as an ordered combination amounts to significantly more than the abstract idea. This analysis includes determining whether an inventive concept is furnished by an element or a combination of elements that are beyond the judicial exception. For limitations that were categorized as “apply it” or generally linking the use of the abstract idea to a particular technological environment or field of use, the analysis is the same.
The additional elements as described in Step 2A Prong 2 are not sufficient to amount to significantly more than the judicial exception because the additional limitations are directed towards Insignificant Extra Solution Activity (MPEP 2106.05(g)) and are considered well known, don’t impose meaningful limits on the claim, and simply amount to necessary data gathering/outputting. Per MPEP 2106.05(d), “[t]he courts have recognized the following computer functions as well‐understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity. i. Receiving or transmitting data over a network, ii. Performing repetitive calculations, iii. Electronic recordkeeping, iv. Storing and retrieving information in memory, v. Electronically scanning or extracting data from a physical document”.
For the foregoing reasons, claim 1 is directed to an abstract idea without significantly more and is rejected as not patent eligible under 35 U.S.C. 101.
Independent claims 19 and 20 recite significantly the same subject matter as claim 1 and are rejected under similar rationale and further failure to add significantly more.
Claim 2 recites wherein the machine system includes a plurality of objects including the robot, wherein the simulation model includes a plurality of object models respectively corresponding to the plurality of objects, and wherein the circuitry is configured to correct the simulation model by individually matching each of the plurality of object models to the actual shape model. The additional elements elaborate on the machine system (i.e. 3D model), the simulation model, and model correction from claim 1, thus further amounting to Mental Process per MPEP 2106.04(a)(2)(III). Therefore, the claim is rejected as not patent eligible under 35 U.S.C. §101.
Claim 3 recites wherein the circuitry is configured to correct the simulation model by repeating a matching process that includes: selecting one matching target model from the plurality of object models; and matching the matching target model to the actual shape model. The additional elements elaborate on the model correction from claim 2, thus further amounting to Mental Process per MPEP 2106.04(a)(2)(III). Therefore, the claim is rejected as not patent eligible under 35 U.S.C. §101.
Claim 4 recites wherein the matching process further includes excluding, from the actual shape model, a part that has matched the matching target model, and wherein circuitry is configured to match, in the matching process, the matching target model to the actual shape model from which one or more parts that has matched one or more other object models are excluded. The additional elements elaborate on the model correction from claim 3, thus further amounting to Mental Process per MPEP 2106.04(a)(2)(III). Therefore, the claim is rejected as not patent eligible under 35 U.S.C. §101.
Claim 5 recites wherein circuitry is configured to select, as the matching target model, a largest object model among all object models of the plurality of object models that have not yet been selected as the matching target model in the matching process. The additional elements elaborate on the model correction from claim 4, thus further amounting to Mental Process per MPEP 2106.04(a)(2)(III). Therefore, the claim is rejected as not patent eligible under 35 U.S.C. §101.
Claim 6 recites wherein the circuitry is further configured to: extract, from the actual shape model, one or more parts each of which does not match any object model after the matching process is completed for all of the plurality of object models; and add one or more new object models to the simulation model based on the extracted one or more parts of the actual shape model. The additional elements as drafted, under broadest reasonable interpretation, amount to Mental Processes per MPEP 2106.04(a)(2) since a person can reasonably evaluate model parts to determine which parts don’t match, extract from the actual shape model those parts determined to not match, and then add one or more object models to the simulation model with/without the aid of pen/paper. Therefore, the claim is rejected as not patent eligible under 35 U.S.C. §101.
Claim 7 recites wherein the circuitry is further configured to: extract, from the simulation model, one or more virtual parts each of which does not match the actual shape model after the matching process is completed for all of the plurality of object models; and delete the extracted one or more virtual parts from the simulation model. The additional elements as drafted, under broadest reasonable interpretation, amount to Mental Processes per MPEP 2106.04(a)(2) since a person can reasonably evaluate simulation model virtual parts, extract those parts from the simulation model, and then delete/remove those virtual parts from the simulation model. Therefore, the claim is rejected as not patent eligible under 35 U.S.C. §101.
Claim 8 recites wherein the circuitry is further configured to: generate the actual shape model based on the measured data that includes a three-dimensional real image of the machine system acquired by measuring the machine system by a three-dimensional camera in the real space (The additional element elaborates on generating the shape model from claim 1, thus further amounting to Mental Process per MPEP 2106.04(a)(2)(III)); generate a pre-processed model by excluding, from the simulation model, one or more virtual hidden parts that has not been measured by the three-dimensional camera (The additional element amounts to Mental Processes per MPEP 2106.04(a)(2) since a person can reasonably generate (i.e. draw) a pre-processed model by excluding one or more virtual hidden parts not previously measured by the camera.); and correct the simulation model based on a comparison between the pre-processed model and the actual shape model (The additional element elaborates on correcting the simulation model from claim 1, thus further amounting to Mental Process per MPEP 2106.04(a)(2)(III)).
Therefore, the claim is rejected as not patent eligible under 35 U.S.C. §101.
Claim 9 recites wherein the circuitry is further configured to: generate the actual shape model based on the measured data that includes a three-dimensional real image of the machine system acquired by measuring the machine system by a three-dimensional camera (The additional element elaborates on generating the shape model, thus further amounting to Mental Processes per MPEP 2106.04(a)(2)); generate a pre-processed model by excluding, from the simulation model, one or more virtual hidden parts included in one or more areas that has not been measured by the three-dimensional camera (The additional element amounts to Mental Processes per MPEP 2106.04(a)(2) since a person can reasonably generate (i.e. draw) a pre-processed model by excluding one or more virtual hidden parts not previously measured by the camera.); divide the pre-processed model into a plurality of pre-processed object models respectively corresponding to the plurality of objects (The additional element amounts to Mental Processes per MPEP 2106.04(a)(2) since a person can reasonably evaluate a pre-processed model and then divide the model into a plurality of pre-processed object models corresponding to the plurality of objects, with/without the aid of pen/paper.); and individually match each of the plurality of object models to the actual shape model based on a comparison of a corresponding pre- processed object model and the actual shape model (The additional element amounts to Mental Processes per MPEP 2106.04(a)(2) since a person can reasonably compare models and then individually match each of the plurality of object models to the actual shape model, with/without the aid of pen/paper.)
Therefore, the claim is rejected as not patent eligible under 35 U.S.C. §101.
Claim 10 recites wherein the circuitry further configured to: calculate a position of a three-dimensional virtual camera corresponding to the three-dimensional camera to match a three-dimensional virtual image with the three-dimensional real image, the three-dimensional virtual image being acquired by virtually measuring the simulation model by the three-dimensional virtual camera in a virtual space; and calculate the one or more virtual hidden parts based on the position of the three-dimensional virtual camera and the simulation model. The additional elements amount to Mental Processes and/or Mathematical Concepts per MPEP 2106.04(a)(2)(III)/(I), respectively. The mathematical concepts grouping is defined as mathematical relationships, mathematical formulas or equations, and mathematical calculations. A mathematical relationship is a relationship between variables or numbers. A mathematical relationship may be expressed in words or using mathematical symbols. A claim that recites a mathematical calculation, when the claim is given its broadest reasonable interpretation in light of the specification, will be considered as falling within the "mathematical concepts" grouping. Therefore, the claim is rejected as not patent eligible under 35 U.S.C. §101.
Claim 11 recites wherein the circuitry is configured to calculate the position of the three-dimensional virtual camera to match one or more virtual calibration parts corresponding to one or more predetermined calibration objects in the three-dimensional virtual image to one or more parts corresponding to the one or more predetermined calibration objects in the three-dimensional real image. The additional elements elaborate on calculating the position of the 3D virtual camera, thus further amounting to Mental Processes and/or Mathematical Concepts per MPEP 2106.04(a)(2)(III)/(I), respectively. Therefore, the claim is rejected as not patent eligible under 35 U.S.C. §101.
Claim 12 recites wherein the circuitry is configured to: acquire the measured data that includes a plurality of three-dimensional real images from a plurality of three-dimensional cameras including the three-dimensional camera (The additional element amounts to Insignificant Extra-solution Activity (mere data gathering, pre-solution activity) per MPEP 2106.05(g). The term "extra-solution activity" can be understood as activities incidental to the primary process or product that are merely a nominal or tangential addition to the claim. Extra-solution activity includes both pre-solution and post-solution activity. An example of pre-solution activity is a step of gathering data for use in a claimed process.); generate the actual shape model by combining the plurality of three-dimensional real images (The additional element amounts to Mental Processes per MPEP 2106.04(a)(2) since a person can reasonably evaluate a plurality of 3D images and then draw a shape model, with/without the aid of pen/paper.); and generate the pre-processed model by excluding, from the simulation model, one or more virtual overlapping hidden parts that has not been measured by any of the plurality of three-dimensional cameras (The additional element amounts to Mental Processes per MPEP 2106.04(a)(2) since a person can reasonably draw a pre-processed model by excluding one or more virtual overlapping hidden parts not previously measured, with/without the aid of pen/paper.)
Therefore, the claim is rejected as not patent eligible under 35 U.S.C. §101.
Claim 13 recites wherein the circuitry is configured to: acquire the plurality of three-dimensional real images each of which includes an image of a common synthesis object from the plurality of three-dimensional cameras (The additional element amounts to Insignificant Extra-solution Activity (mere data gathering, pre-solution activity) per MPEP 2106.05(g).); and combine the plurality of three-dimensional real images to generate the actual shape model to match a part corresponding to the synthesis object in each of the plurality of three-dimensional real images to a predetermined shape of the synthesis object (The additional element amounts to Mental Processes per MPEP 2106.04(a)(2) since a person can reasonably evaluate a plurality of 3D images and then draw (i.e. generate) the actual shape model to match a part corresponding to the synthesis object in each of the plurality of three-dimensional real images to a predetermined shape of the synthesis object.)
Therefore, the claim is rejected as not patent eligible under 35 U.S.C. §101.
Claim 14 recites wherein the circuitry is further configured to: calculate positions of a plurality of three-dimensional virtual cameras respectively corresponding to the plurality of three-dimensional cameras to match a plurality of three-dimensional virtual images acquired by capturing the simulation model using the plurality of three-dimensional virtual cameras to the plurality of three-dimensional real images; and calculate the virtual overlapping hidden part based on the positions of the plurality of three-dimensional virtual cameras and the simulation model. The additional elements further amount to Mental Processes and/or Mathematical Concepts per MPEP 2106.04(a)(2)(III)/(I), respectively. Therefore, the claim is rejected as not patent eligible under 35 U.S.C. §101.
Claim 15 recites wherein the circuitry is configured to: generate the actual shape model representing the three-dimensional real shape of the machine system by point cloud data; and generate the pre-processed model representing a three-dimensional virtual shape of the simulation model by virtual point cloud data. The additional elements amount to Insignificant Extra-Solution Activity (selecting a particular data source or type of data to be manipulated) per MPEP 2106.05(g). Therefore, the claim is rejected as not patent eligible under 35 U.S.C. §101.
Claim 16 recites wherein the circuitry is further configured to: generate the actual shape model representing a three-dimensional real shape of the machine system by point cloud data; generate a pre-processed model representing a three-dimensional virtual shape of the simulation model by virtual point cloud data (The additional elements amount to Insignificant Extra-Solution Activity (selecting a particular data source or type of data to be manipulated) per MPEP 2106.05(g).); and correct the simulation model based on a comparison between the pre-processed model and the actual shape model (The additional element amounts to Mental Processes per MPEP 2106.04(a)(2) since a person can reasonably compare models and then correct the simulation model therefrom.)
Therefore, the claim is rejected as not patent eligible under 35 U.S.C. §101.
Claim 17 recites wherein the circuitry is further configured to simulate an operation of the machine system based on the corrected simulation model. The additional element amounts to Mere Instructions to Apply an Exception per MPEP 2106.05(f). Specifically, the claim recites only the idea of a solution or outcome i.e., the claim fails to recite details of how a solution to a problem is accomplished and/or invokes computers or other machinery merely as a tool to perform an existing process. Therefore, the claim is rejected as not patent eligible under 35 U.S.C. §101.
Claim 18 recites A control system comprising: the simulation device according to claim 17; and a control circuitry configured to control the machine system based on a simulation of the operation of the machine system based on the corrected simulation model. The additional element amounts to Mere Instructions to Apply an Exception per MPEP 2106.05(f). Specifically, the claim recites only the idea of a solution or outcome i.e., the claim fails to recite details of how a solution to a problem is accomplished and/or invokes computers or other machinery merely as a tool to perform an existing process. Therefore, the claim is rejected as not patent eligible under 35 U.S.C. §101.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham V. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103(a) are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
In the event the determination of the status of the application as subject to
AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Claims 1-7 and 16-20 are rejected under 35 U.S.C. 103 as being unpatentable over Wakayama et al. US Patent No. 12014470 B2 (hereinafter referred to as “Wakayama”) in view of Watanabe et al. US Pub. No. US 20030090483 A1 (hereinafter referred to as “Watanabe”).
Regarding claim 1, Wakayama discloses A simulation device comprising circuitry configured to (“a model generation apparatus capable of generating a model for implementing a more precise simulation” Wakayama [Abstract]): store a simulation model of a machine system including a robot (“FIG. 11 is a table showing an example of information stored in a parameter database 17” Wakayama [Col.4 Ln.8]. Fig.11 is interpreted as a simulation model due to Applicant’s disclosure “The simulation model includes at least arrangement information of the objects 3 and structure and dimension information of each of the objects 3.” [Spec. P.0024]. Additionally, Fig.11 is interpreted to include a robot because “In order to reproduce environmental information of the real-world in these robot simulators, it is necessary to, in addition to the modeling of objects to be reproduced, take physical characteristics in the disposition of the objects into consideration.” Wakayama [Col.2 Ln.5]), the simulation model generated to simulate a three-dimensional real shape of the machine system “the model generation apparatus [ ] can reproduce effects of contact, collision, and the like between objects, and can generate a 3D model with which a more precise simulation can be carried out.” Wakayama [Col.5 Ln.10]); generate, based on the measured data, an actual shape model representing a three-dimensional real shape of the machine system (“the model generation apparatus 1 extracts an object to be reconstructed on a 3D model from 3D image information through the operation of the object extraction means 11, and also acquires, from among a plurality of object models available on the 3D model, an object model having the highest shape conformity degree with the object while associating the object model with size information and disposed-place information of the object.” Wakayama [Col.4 Ln.57]); .
Wakayama fails to specifically disclose receive measured data acquired by measuring the machine system in a real space, and correct the simulation model of the machine system based on a comparison between the simulation model and the actual shape model.
However, Watanabe discloses receive measured data acquired by measuring the machine system in a real space (“a sensor is used to measure the layout of the actual peripheral equipment and workpiece” Watanabe [P.0036]); and correct the simulation model of the machine system based on a comparison between the simulation model and the actual shape model (“Then, the layout of the three dimensional models is corrected on the basis of the result of measurement of such components of the actual system.” Watanabe [P.0036]).
Wakayama and Watanabe are analogous art as they both relate to working machines, such as robots, and the accurate simulation thereof. Wakayama discloses “[i]n order to reproduce environmental information of the real-world in these robot simulators, it is necessary to, in addition to the modeling of objects to be reproduced, take physical characteristics in the disposition of the objects into consideration.” [Col.2 Ln.5]. And Watanabe discloses “a simulation apparatus for performing a simulation of a working machine such as a robot and a machine tool, and more particularly, to a simulation apparatus, which provides a simulation by matching a model used in the simulation with an actual system through the use of a sensor” [P.0002]. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Wakayama’s model generation system/method to include receiving measured data simulation model correction, as Watanabe discloses, in order to provide an “accurate layout of the three-dimensional models in a system, thereby allowing the system to be accurately simulated” Watanabe [P.0042].
Regarding claim 2, Wakayama further discloses wherein the machine system includes a plurality of objects including the robot (“In order to reproduce environmental information of the real-world in these robot simulators, it is necessary to, in addition to the modeling of objects to be reproduced, take physical characteristics in the disposition of the objects into consideration.” Wakayama [Col.2 Ln.5]), wherein the simulation model includes a plurality of object models respectively corresponding to the plurality of objects (“In this way, it is possible to generate a 3D model with which a more precise simulation, in which effects of contact, collision, and the like between objects are reproduced, can be carried out.” Wakayama [Col.5 Ln.22]), and wherein the circuitry is configured to correct the simulation model by individually matching each of the plurality of object models to the actual shape model (“the object extracting means 11 extracts an object to be reconstructed on a 3D model from the acquired 3D image information, and acquires, from among a plurality of object models stored in the model group database 15, an object model having the highest shape conformity degree with the object (step S102). For example, as a specific example of the method for acquiring an object model, the object extracting means 11 performs a matching process between each of the object models stored in the model group database 15 and each of the objects included in the 3D image information” Wakayama [Col.6 Ln.50]).
Regarding claim 3, the simulation device of claim 2, Wakayama further discloses wherein the circuitry is configured to correct the simulation model by repeating a matching process that includes: selecting one matching target model from the plurality of object models (“the object extracting means 11 performs a matching process between each of the object models stored in the model group database 15 and each of the objects included in the 3D image” Wakayama [Col.6 Ln.57]); and matching the matching target model to the actual shape model (“the object extracting means 11 extracts an object to be reconstructed on a 3D model from the acquired 3D image information, and acquires, from among a plurality of object models stored in the model group database 15, an object model having the highest shape conformity degree with the object” Wakayama [Col.6 Ln.50]).
Regarding claim 4, the simulation device of claim 3, Wakayama further discloses wherein the matching process further includes excluding, from the actual shape model, a part that has matched the matching target model (“referring to FIG. 4 (see below) [ ] as a specific example of the method for acquiring an object model (i.e. actual shape model), the object extracting means 11 performs a matching process between each of the object models stored in the model group database 15 and each of the objects included in the 3D image information, and thereby extracts a pair whose conformity degree is larger than a predetermined value as an object model. Note that the object extracting means 11 may be equipped with a known image recognition function.” Wakayama [Col.6 Ln.49]), and wherein circuitry is configured to match, in the matching process, the matching target model to the actual shape model from which one or more parts that has matched one or more other object models are excluded (“After disposing the object model, the process returns to the step S105, and the processes are repeated until there is no longer any object that has not yet been disposed. Then, when there is no longer any object that has not yet been disposed (NO in Step S105), the 3D model output means 14 outputs the created 3D model (step S108)” Wakayama [Col.8 Ln.64]).
PNG
media_image1.png
826
527
media_image1.png
Greyscale
Regarding claim 5, the simulation device of claim 4, Wakayama further discloses wherein circuitry is configured to select, as the matching target model, a largest object model among all object models of the plurality of object models that have not yet been selected as the matching target model in the matching process (“the object extracting means 11 performs a matching process between each of the object models stored in the model group database 15 and each of the objects included in the 3D image information, and thereby extracts a pair whose conformity degree is larger than a predetermined value as an object model.” Wakayama [Col.6 Ln.57]. The predetermined value is interpreted to include object model size because “size information of the object has a value by which the size of the object is uniquely expressed (e.g. largest object model).” Wakayama [Col.7 Ln.12]).
Regarding claim 6, the simulation device of claim 3, Wakayama further discloses wherein the circuitry is further configured to: extract, from the actual shape model, one or more parts each of which does not match any object model after the matching process is completed for all of the plurality of object models (“When no other object (i.e. part) can be extracted (i.e. doesn’t match any object model – see Fig.4 and/or Fig.12 S102) (NO in Step S103), it is considered that the extraction of objects has been completed, and for each of the detected objects, the object model is edited and disposed on the 3D model. Firstly, when there is a detected object that has not yet been disposed in the 3D model (YES in Step S105), the model editing means 12 edits the object model so that it conforms to the size information of the object included in the output data from the object extracting means 11 (step S106)” Wakayama [Col.7 Ln.48]. The objects are interpreted as parts because “ each of the parts (i.e., components) that constitute an object model” Wakayama [Col.10 Ln.35]); and add one or more new object models to the simulation model based on the extracted one or more parts of the actual shape model (“the model disposing means 13a disposes an object model on the 3D model (i.e. simulation model) (Step S107, since it is similar to that in the second example embodiment, the description thereof is omitted), and associates the acquired physical parameters with the disposed object (Step S112). Specifically, the 3D model data is edited, and the physical property values of the object model are set to the acquired physical parameter values. After the process has been carried out for one object model, the process returns to the Step S105. When there is an object that has not yet been disposed (YES in Step S105), similar processes are repeated.” Wakayama [Col.11 Ln.17]. See Fig.12 below for clarity. S107 is interpreted as adding one or more new object models to the simulation model.).
PNG
media_image2.png
809
550
media_image2.png
Greyscale
Regarding claim 7, the simulation device of claim 3, Wakayama further discloses wherein the circuitry is further configured to: extract, from the simulation model, one or more virtual parts each of which does not match the actual shape model (“The description of the operation is continued by referring to FIG. 4 (see below) again. Next, the object extracting means 11 extracts an object to be reconstructed on a 3D model from the acquired 3D image information, and acquires, from among a plurality of object models stored in the model group database 15, an object model having the highest shape conformity degree with the object (step S102). For example, as a specific example of the method for acquiring an object model, the object extracting means 11 performs a matching process between each of the object models stored in the model group database 15 and each of the objects included in the 3D image information” Wakayama [Col.6 Ln.50]) after the matching process is completed for all of the plurality of object models (“After disposing the object model, the process returns to the step S105, and the processes are repeated until there is no longer any object that has not yet been disposed. Then, when there is no longer any object that has not yet been disposed (NO in Step S105), the 3D model output means 14 outputs the created 3D model (step S108)” Wakayama [Col.8 Ln.64]; and delete the extracted one or more virtual parts from the simulation model (As seen in Fig.4 below, an object extracted from 3D image information that doesn’t match an object model simply isn’t added/included in the output 3D model (i.e. simulation model), which is equivalent to deleting the extracted virtual parts from the simulation model. The path taken for an extracted object which doesn’t match an actual shape model is highlighted for clarity.)
PNG
media_image3.png
769
491
media_image3.png
Greyscale
Regarding claim 16, Wakayama further discloses wherein the circuitry is further configured to: generate the actual shape model representing a three-dimensional real shape of the machine system by point cloud data (“3D image information is acquired by performing real-space sensing through a depth camera and converting a result of the sensing into point-cloud information.” Wakayama [Col.6 Ln.37], “the object extracting means 11 extracts an object to be reconstructed on a 3D model from the acquired 3D image information” Wakayama [Col.6 Ln.50]); generate a pre-processed model representing a three-dimensional virtual shape of the simulation model by virtual point cloud data (“The model group database 15 is a database in which a group of candidates for object models (i.e. pre-processed models) that can be used on the 3D model output by the model generation apparatus 1 is stored. The object model is data that defines the shape and the size of an object, for example, data expressed in a wire frame model, a polygon mesh, or a point cloud.” Wakayama [Col.6 Ln.4]); and correct the simulation model based on a comparison between the pre-processed model and the actual shape model (“In order to construct a 3D model that is close to the real environment and conforms with physical constraints in a simulation, the 3D image information (i.e. actual shape model) is corrected based on the physical constraints when the object model is disposed.” Wakayama [Col.8 Ln.25])
Regarding claim 17, Wakayama fails to specifically disclose wherein the circuitry is further configured to simulate an operation of the machine system based on the corrected simulation. However, Watanabe discloses wherein the circuitry is further configured to simulate an operation of the machine system based on the corrected simulation model (“as shown in FIGS. 3A and 3B, a sensor 42 is mounted to a robot 41 (an actual object), which is a part of a working machine to be simulated, to measure a peripheral object 43 (which is a table in this embodiment), thereby correcting the layout on the screen. Based on the result of this measurement, a display position (e.g., an on-screen layout position) of a three-dimensional model 46 of the table on the screen 44 of the simulation apparatus is corrected. Incidentally, reference numeral 45 denotes a three-dimensional model of the robot” Watanabe [P.0037]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Wakayama’s model generation system/method to include simulating the operation of the machine system based on a corrected simulation model, as Watanabe discloses, since “[s]imulation without correcting such a mismatch may cause inaccurate simulation” Watanabe [P.0036].
Regarding claim 18, Wakayama in view of Watanabe disclose : the simulation device according to claim 17; .
Wakayama fails to specifically disclose A control system comprising: a control circuitry configured to control the machine system based on a simulation of the operation of the machine system based on the corrected simulation model. However, Watanabe further discloses A control system comprising: a control circuitry configured to control the machine system based on a simulation of the operation of the machine system based on the corrected simulation model (“The procedure as described above may be summarized in a flow chart shown in FIG. 11 (see below)” Watanabe [P.0060])
PNG
media_image4.png
642
395
media_image4.png
Greyscale
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Wakayama’s model generation system/method to include simulating the operation of the machine system based on a corrected simulation model, as Watanabe discloses, since “[s]imulation without correcting such a mismatch may cause inaccurate simulation” Watanabe [P.0036].
Claim 19 recites substantially the same subject matter as claim 1 and is rejected under similar rationale.
Regarding claim 20, Wakayama discloses A non-transitory memory device having instructions stored thereon that, in response to execution by a processing device, cause the processing device to perform operations (“the program can be stored and provided to a computer using any type of non-transitory computer readable media. Non-transitory computer readable media include any type of tangible storage media.” Wakayama [Col.13 Ln.18]). The additional limitations recite substantially the same subject matter as claim 1 and are rejected under similar rationale.
Claims 8-12 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Wakayama et al. US Patent No. 12014470 B2 (hereinafter referred to as “Wakayama”) in view of Watanabe et al. US Pub. No. US 20030090483 A1 (hereinafter referred to as “Watanabe”), in further view of Atohira US Pub No. 20190255706 A1 (hereinafter referred to as “Atohira”).
Regarding claim 8, Wakayama further discloses wherein the circuitry is further configured to: generate the actual shape model based on the measured data that includes a three-dimensional real image of the machine system acquired by measuring the machine system by a three-dimensional camera in the real space (“3D image information is acquired by performing real-space sensing (i.e. measuring) through a depth camera and converting a result of the sensing into point-cloud information.” Wakayama [Col.6 Ln.37], “the object extracting means 11 extracts an object to be reconstructed on a 3D model from the acquired 3D image information” Wakayama [Col.6 Ln.50]);
Wakayama fails to specifically disclose generate a pre-processed model by excluding, from the simulation model, one or more virtual hidden parts that has not been measured by the three-dimensional camera, and correct the simulation model based on a comparison between the pre-processed model and the actual shape model.
However, Atohira discloses generate a pre-processed model by excluding, from the simulation model, one or more virtual hidden parts that has not been measured by the three-dimensional camera (“The exclusion unit 38 calculates an overlap region RD, which is a region in which the imaging range RC1, the imaging range RC2, and the projection range RA overlap. The workpiece model 85M is arranged inside the overlap region RD, and thus the exclusion unit 38 does not exclude the setting points PC1 arranged on the surfaces of the workpiece model 85M. The workpiece model 88M is arranged outside the overlap region RD, and thus the exclusion unit 38 excludes the setting points PC11 (settings points are used for measurement) arranged on the surfaces of the workpiece model 88M (i.e. virtual hidden part).” Atohira [P.0079]. See Fig.18 below.);
PNG
media_image5.png
658
474
media_image5.png
Greyscale
and correct the simulation model based on a comparison between the pre-processed model and the actual shape model (the distance correction unit 39 can control the correction of the distance Z with respect to the camera model 61M and the camera model 62M. For example, the distance correction unit 39 can correct the distance Z with respect to the camera model 61M, and subsequently correct the distance Z with respect to the camera model 62M.” Atohira [P.0088]. The distance corrections are interpreted as comparing and correcting a virtual model (e.g. simulation model, pre-processed model, etc.) to an actual shape model because “The relative positions of the models of the members in the robot device and the workpiece model 85M correspond to the relative positions of the actual members in the robot device 9 and the actual workpiece 85.” Atohira [P.0044]).
Atohira is analogous art as it relates to robot operation simulation. Atohira discloses a “simulation device simulates a robot device, making it possible to determine whether or not the robot device performs a desired operation. Further, the simulation device can evaluate an operation program which drives the robot device before the robot device performs the actual operation” [P.0005]. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Wakayama’s model generation system/method to include a simulation model that excludes hidden object parts not measured by 3D cameras and model correction, such as that disclosed by Atohira, in order to “implement a more accurate simulation.” Atohira [P.0082].
Regarding claim 9, the simulation device of claim 2, Wakayama further discloses wherein the circuitry is further configured to: generate the actual shape model based on the measured data that includes a three-dimensional real image of the machine system acquired by measuring the machine system by a three-dimensional camera (“3D image information is acquired by performing real-space sensing (i.e. measuring) through a depth camera and converting a result of the sensing into point-cloud information.” Wakayama [Col.6 Ln.37], “the object extracting means 11 extracts an object to be reconstructed on a 3D model from the acquired 3D image information” Wakayama [Col.6 Ln.50]); divide the pre-processed model into a plurality of pre-processed object models respectively corresponding to the plurality of objects; and individually match each of the plurality of object models to the actual shape model based on a comparison of a corresponding pre- processed object model and the actual shape model (“as a specific example of the method for acquiring an object model, the object extracting means 11 performs a matching process between each of the object models (i.e. pre-processed) stored in the model group database 15 and each of the objects (i.e. actual shape models) included in the 3D image information, and thereby extracts a pair whose conformity degree is larger than a predetermined value as an object model.” Wakayama [Col.6 Ln.56]. The examiner interprets Wakayama’s matching process between each of the object models and each of the objects as equivalent to dividing the pre-processed model into a plurality of pre-processed object models.)
Wakayama fails to specifically disclose generate a pre-processed model by excluding, from the simulation model, one or more virtual hidden parts included in one or more areas that has not been measured by the three-dimensional camera.
However, Atohira discloses generate a pre-processed model by excluding, from the simulation model, one or more virtual hidden parts included in one or more areas that has not been measured by the three-dimensional camera (“The exclusion unit 38 calculates an overlap region RD, which is a region in which the imaging range RC1, the imaging range RC2, and the projection range RA overlap. The workpiece model 85M is arranged inside the overlap region RD, and thus the exclusion unit 38 does not exclude the setting points PC1 arranged on the surfaces of the workpiece model 85M. The workpiece model 88M is arranged outside the overlap region RD, and thus the exclusion unit 38 excludes the setting points PC11 (settings points are used for measurement) arranged on the surfaces of the workpiece model 88M (i.e. virtual hidden part).” Atohira [P.0079]. See Fig.18 below.)
PNG
media_image5.png
658
474
media_image5.png
Greyscale
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Wakayama’s model generation system/method to include a simulation model that excludes hidden object parts not measured by 3D cameras, such as that disclosed by Atohira, in order to “implement a more accurate simulation.” Atohira [P.0082].
Regarding claim 10, the simulation device of claim 8, Wakayama fails to specifically disclose wherein the circuitry further configured to: calculate a position of a three-dimensional virtual camera corresponding to the three-dimensional camera to match a three-dimensional virtual image with the three-dimensional real image, the three-dimensional virtual image being acquired by virtually measuring the simulation model by the three-dimensional virtual camera in a virtual space; and calculate the one or more virtual hidden parts based on the position of the three-dimensional virtual camera and the simulation model.
However, Atohira discloses wherein the circuitry further configured to:
calculate a position of a three-dimensional virtual camera corresponding to the three-dimensional camera to match a three-dimensional virtual image (“In the virtual space, the positions of the camera models 61M, 62M are predetermined. A distance D between the two camera models 61M, 62M is predetermined. Further, an optical axis LA1 of the camera model 61M and an optical axis LA2 of the camera model 62M are also predetermined. Thus, the distance calculation unit 36 can calculate an angle 01 formed by a line of sight LV1 from the camera model 61M toward the setting point PC1, and the optical axis LA1.” Atohira [P.0056]) with the three-dimensional real image (“The relative positions of the models of the members in the robot device and the workpiece model 85M correspond to the relative positions of the actual members in the robot device 9 and the actual workpiece 85.” Atohira [P.0044]),
the three-dimensional virtual image being acquired by virtually measuring the simulation model by the three-dimensional virtual camera in a virtual space (“The range sensor 6 of the present embodiment captures images (i.e. virtual images) of the workpiece 85 by using the two two-dimensional cameras 61, 62. The controller 2 has a function of processing the images captured by the two-dimensional cameras 61, 62. The controller 2 can generate three-dimensional information of the workpiece 85 by a stereo method [ ] The three-dimensional information includes information related to a position of a predetermined point and a distance from the range sensor to the predetermined point (i.e. virtually measuring).” Atohira [P.0036-0037]); and calculate the one or more virtual hidden parts based on the position of the three-dimensional virtual camera and the simulation model (“the simulation device 3 includes a distance calculation unit 36 which calculates distances Z from the range sensor model 6M (i.e. 3D virtual camera) to the setting point PC1 [ ] the distance calculation unit 36 calculates the distance Z from the range sensor model 6M to the setting point PC1 for each setting point PC1” Atohira [P.0055]. Reference Figs. 11, 12, and 13 for virtual hidden parts.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Wakayama’s model generation system/method to include object/part measurement and position determination, such as that disclosed by Atohira, in order to “implement a more accurate simulation.” Atohira [P.0082].
Regarding claim 11, the simulation device of claim 10, Wakayama fails to specifically disclose wherein the circuitry is configured to calculate the position of the three-dimensional virtual camera to match one or more virtual calibration parts corresponding to one or more predetermined calibration objects in the three-dimensional virtual image to one or more parts corresponding to the one or more predetermined calibration objects in the three-dimensional real image.
However, Atohira discloses wherein the circuitry is configured to calculate the position of the three-dimensional virtual camera to match one or more virtual calibration parts corresponding to one or more predetermined calibration objects in the three-dimensional virtual image to one or more parts corresponding to the one or more predetermined calibration objects in the three-dimensional real image (“In the virtual space, the positions of the camera models 61M, 62M are predetermined. A distance D between the two camera models 61M, 62M is predetermined. Further, an optical axis LA1 of the camera model 61M and an optical axis LA2 of the camera model 62M are also predetermined. Thus, the distance calculation unit 36 can calculate an angle θ1 formed by a line of sight LV1 from the camera model 61M toward the setting point PC1, and the optical axis LA1.” Atohira [P.0056]. See Fig.11 below for reference.)
PNG
media_image6.png
630
451
media_image6.png
Greyscale
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Wakayama’s model generation system/method to include object/part measurement and position determination, such as that disclosed by Atohira, in order to “implement a more accurate simulation.” Atohira [P.0082].
Regarding claim 12, the simulation device of claim 8, Wakayama further discloses wherein the circuitry is configured to: acquire the measured data that includes a plurality of three- dimensional real images from a plurality of three-dimensional cameras including the three-dimensional camera (“The image information (i.e. measurement data) acquiring means 10 acquires 3D image information, which is information three-dimensionally representing the presence of objects in a real space, from real-space sensing information [ ] the image information acquiring means 10 may further include sensing means such as a depth camera, a 3D camera, or 3D LiDAR (Light Detection and Ranging), and acquire the real-space sensing information from the sensing means, or may acquire the real-space sensing information from an external apparatus.” Wakayama [Col.5 Ln.53]; generate the actual shape model by combining the plurality of three-dimensional real images (“A model generation apparatus configured to generate a 3D model from 3D image information” Wakayama [Col.13 Ln.48]); .
Wakayama fails to specifically disclose generate the pre-processed model by excluding, from the simulation model, one or more virtual overlapping hidden parts that has not been measured by any of the plurality of three-dimensional cameras.
However, Atohira discloses generate the pre-processed model by excluding, from the simulation model, one or more virtual overlapping hidden parts that has not been measured by any of the plurality of three-dimensional cameras (“The simulation device includes a three-dimensional information generating unit which generates three-dimensional information including positions of the setting points and distances from the three-dimensional sensor model to the setting points. The simulation device includes an exclusion unit which excludes, among the setting points arranged on the surface of the workpiece model, the setting point which is not visible from at least one of the camera models. The simulation device includes a workpiece position calculation unit which calculates a position and orientation of the workpiece model on the basis of the three-dimensional information.” Atohira [P.0008]. See Figs. 13, 15, and 18 regarding “overlapping” parts.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Wakayama’s model generation system/method to include a simulation model that excludes hidden object parts not measured by 3D cameras and model correction, such as that disclosed by Atohira, in order to “implement a more accurate simulation.” Atohira [P.0082].
Regarding claim 15, the simulation device of claim 8, Wakayama further discloses wherein the circuitry is configured to: generate the actual shape model representing the three- dimensional real shape of the machine system by point cloud data (“3D image information is acquired by performing real-space sensing through a depth camera and converting a result of the sensing into point-cloud information.” Wakayama [Col.6 Ln.37], “the object extracting means 11 extracts an object to be reconstructed on a 3D model from the acquired 3D image information” Wakayama [Col.6 Ln.50]); and generate the pre-processed model representing a three-dimensional virtual shape of the simulation model by virtual point cloud data (“The model group database 15 is a database in which a group of candidates for object models that can be used on the 3D model output by the model generation apparatus 1 is stored. The object model is data that defines the shape and the size of an object, for example, data expressed in a wire frame model, a polygon mesh, or a point cloud.” Wakayama [Col.6 Ln.4]).
Claims 13-14 are rejected under 35 U.S.C. 103 as being unpatentable over Wakayama et al. US Patent No. 12014470 B2 (hereinafter referred to as “Wakayama”) in view of Watanabe et al. US Pub. No. US 20030090483 A1 (hereinafter referred to as “Watanabe”), in further view of Atohira US Pub No. 20190255706 A1 (hereinafter referred to as “Atohira”), and in further view of Kawahara US Pub. No. 20190335162 A1 (hereinafter referred to as “Kawahara”).
Regarding claim 13, the simulation device of claim 12, Wakayama fails to specifically disclose wherein the circuitry is configured to: acquire the plurality of three-dimensional real images each of which includes an image of a common synthesis object from the plurality of three-dimensional cameras, and combine the plurality of three-dimensional real images to generate the actual shape model to match a part corresponding to the synthesis object in each of the plurality of three-dimensional real images to a predetermined shape of the synthesis object.
However, Kawahara discloses wherein the circuitry is configured to: acquire the plurality of three-dimensional real images each of which includes an image of a common synthesis object from the plurality of three-dimensional cameras (“at S703, the control device 300 instructs the image capturing module 110 to perform image capturing (actual image capturing) of multiple viewpoint images, which are a source of a virtual viewpoint image [ ] each image capturing module 110 performs actual image capturing and the data of the images (multiple viewpoint images) captured by each camera 112 is transmitted from the camera adaptor 120 to the server 270.” Kawahara [P.0049]. See Fig.9 below for reference.);
PNG
media_image7.png
487
482
media_image7.png
Greyscale
and combine the plurality of three-dimensional real images to generate the actual shape model to match a part corresponding to the synthesis object in each of the plurality of three-dimensional real images to a predetermined shape of the synthesis object (“For the ball 902 (i.e. synthesis object), which is the foreground, a texture-attached foreground three-dimensional model representing the ball 902 three-dimensionally is created from the foreground three-dimensional model source information and the foreground image obtained by each of the image capturing modules 110a to 110f” Kawahara [P.0056]).
Kawahara is analogous art as it relates to virtual viewpoint image generation technology that synthesizes images from multiple real cameras to create images from arbitrary virtual viewpoints. It aims to reduce processing load and improve image quality by dynamically updating three-dimensional background shape data based on differences detected between simulated images and actual captured images. Kawahara discloses “[t]he present invention relates to a technique to generate an image from a virtual viewpoint based on images captured from a plurality of viewpoints” [P.0001]. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Wakayama’s model generation system/method to include a plurality of 3D real images to include a common synthesis object and combine those real images, as Kawahara discloses, in order “to improve the image quality of the portion corresponding to the specific object in the virtual viewpoint image to be generated” Kawahara [P.0028].
Regarding claim 14, the simulation device of claim 12, Wakayama fails to specifically disclose wherein the circuitry is further configured to: calculate positions of a plurality of three-dimensional virtual cameras respectively corresponding to the plurality of three-dimensional cameras to match a plurality of three-dimensional virtual images acquired by capturing the simulation model using the plurality of three- dimensional virtual cameras to the plurality of three-dimensional real images; and calculate the virtual overlapping hidden part based on the positions of the plurality of three-dimensional virtual cameras and the simulation model.
However, Kawahara discloses wherein the circuitry is further configured to: calculate positions of a plurality of three-dimensional virtual cameras respectively corresponding to the plurality of three-dimensional cameras to match a plurality of three-dimensional virtual images acquired by capturing the simulation model using the plurality of three- dimensional virtual cameras to the plurality of three-dimensional real images (“At S701, the control device 300 performs system check processing aiming at the check of whether each device within the image processing system 100 operates normally and necessary adjustment [ ] In the adjustment of the camera 111, matching of angle of view, color matching, adjustment to match each of the coordinates of the camera 112a to 112z with the world coordinates (i.e. determining positions of each camera), and the like are included.” Kawahara [P.0047]); and calculate the virtual overlapping hidden part based on the positions of the plurality of three-dimensional virtual cameras and the simulation model (“details of the image check processing, which is the feature of the present embodiment, are explained by using the flowchart in FIG. 8” Kawahara [P.0053]. See Kawahara [P.0053-0062] and Figs.9-12. From the figures, soccer goal 901 is interpreted as having overlapping hidden parts based on each camera position.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Wakayama’s model generation system/method to include calculating camera positions and virtual overlapping parts, as Kawahara discloses, in order “to obtain a virtual viewpoint image (=simulation image) in a case where the actual viewpoint and the actual angle of view in a real camera are taken to be a virtual viewpoint and a virtual angle of view” Kawahara [P.0055].
Conclusion
The prior art made of record, listed on form PTO-892, and not relied upon is
considered pertinent to applicant's disclosure:
Yamazaki (Simulation Device Using Three-dimensional Position Information Obtained From Output From Vision Sensor – US Pat. No 12600040 B2). “a simulation implementation unit that simulates a movement of a robot device, and estimates a movement path of a robot. The simulation device comprises a position information generation unit that generates three-dimensional position information for a surface of a workpiece on the basis of the output of a vision sensor, which has imaged an actual workpiece. A display unit displays the three-dimensional position information for the surface of the workpiece, superimposed on an image of a robot device model, an image of a workpiece model, and the movement path of the robot.” [Abstract]
Chizek et al. (Integration Of Auxiliary Sensors With Point Cloud-based Haptic Rendering And Virtual Fixtures – US Pat. No 10394327 B2). “A computing device receives first data about an environment from a first group of one or more sensors. The computing device models the environment as a virtual environment based on the first data. The computing device determines whether to obtain additional data to model the environment. After determining to obtain additional data to model the environment, the computing device receives second data about the environment and model the environment as the virtual environment based on at least the second data. The computing device generates a display of the virtual environment.” [Col.2 Ln.65]
Busey (Pattern-triggered Object Modification In Augmented Reality System – US Pat. No 11908149 B2). “a system configured to obtain a set of images via a camera of the computing device, input the set of images into a neural network, and detect a target physical object with the neural network. The system may determine a contour of the target physical object and a first three-dimensional reconstruction of the target physical object. The system may generate a virtual representation and a virtual object based on attributes of the virtual representation, where a first attribute of the set of attributes includes the first three-dimensional reconstruction. The system may associate the virtual object with the virtual representation and displays the virtual object at pixel coordinates of a display that at least partially occlude at least part of the target physical object, where a position of the virtual object is computed based on the contour.” [Abstract]
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Anthony Chavez whose telephone number is (571) 272-1036. The examiner can normally be reached Monday - Thursday, 8 a.m. - 5 p.m. ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Renee Chavez can be reached at (571) 270-1104 The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANTHONY CHAVEZ/ Examiner, Art Unit 2186
/RENEE D CHAVEZ/Supervisory Patent Examiner, Art Unit 2186
Prosecution Insights