Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendments and Remarks
Applicant's arguments/amendments filed 2/25/26 have been fully considered as follows:
Schultz is cited below to address the amended subject matter.
Schultz discloses in the same context “The identification of the automotive component and the anomaly detection involve a controller subject to artificial intelligence driven training and learning. The controller determines presence of anomaly and a location of anomaly if any.” (abstract).
The amended claim does not clarify how artificial intelligence/machine learning/deep learning model is used in the comparison.
In the same context, Schutz considers:
“ the server 5 may issue notification identifying the anomaly, providing an image of the model with estimated position of anomaly, or other suitable notification…In other forms, the server 5 is trained to learn different types of anomalies. The server 5 includes neural networks that learn and recognize different types of defects by training. In some forms, training data for training the server 5 may include the estimated inlier templates. The training data may include outlier models which indicate different types of defects. The training data may include CAD images of components, scanned pictures of components with or without different types of defects and the like. Based on learning by the training data, the server 5 is trained to learn and classify different types of defects” (¶ 59).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Use of indicates a limitation is not explicitly disclosed by the reference alone.
Claim(s) 1-3, 8-13, 18-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Schultz (US 2022/0327688) in view of Wang (US 2024/0104714)
Claim 1
Examiner’s Interpretation:
Interpretation of “provide a characterization of the deviation”:
Applicant’s specification gives examples of characterization information but does not explicitly define the term:
“ In particular, model comparison module 620 operates to highlight the deviation, and to provide qualitative and quantitative analysis of the deviation.” (Specification, ¶ 78)
“characterization information related to the environmental conditions, such as temperature, atmospheric pressure, humidity, or the like, at the time of the deviation.” (Specification, ¶ 78)
“Such characterization information may further include timestamp information or the like.” (Specification, ¶ 78)
Accordingly, characterization is interpreted as covering qualitative and quantitative analysis and the like.
Claim Mapping:
Schultz discloses a structure inspection system (¶ 6: “the present disclosure is directed towards an inspection system”), comprising:
PNG
media_image1.png
333
418
media_image1.png
Greyscale
a hardware image conversion device configured to receive a plurality of first images of a physical structure (¶ 49: “In some forms, the cameras 40, 45 include a 2D digital camera (color cameras and/or monochrome cameras) that capture two dimensional (2D) images 42, 48, which are provided to the server 5, respectively. In one form, the cameras 40, 45 are configured to detect small surface defects in large patches of the surface area as needed. In at least one variation, one or more of the cameras 40, 45 may include a 3D camera. The 3D cameras may capture and produce a 3D image by taking a set of pictures of an object from different angles and converting the set of pictures into a 3D model with appropriate software.”), and to render a first real time three-dimensional (3D) model (¶ 52: “takes 3D point cloud information, detects anomalies, and identifies their locations substantially in real time”) of the physical structure based on the first images (¶ 49: “converting the set of pictures into a 3D model with appropriate software.”),
a processor (¶ 56: “memory 15 is also configured to store one or more template CAD models of the selected automotive component.”)
wherein the image conversion device is further configured to compare the first real time 3D model with the baseline 3D model (¶ 61: “For example, if a 3D model reconstructing a front door panel may have scratches and burrs, the server 5 determines these anomalies by comparing the 3D model with 3D CAD models relevant to scratches and burrs.”), and to provide an indication when the first real time 3D model deviates from the baseline 3D model and to wherein in comparing the first real time 3D mode with the baseline model, the image conversion device utilizes an artificial intelligence/machine learning/deep learning model(¶ 59, 65: “ the server 5 may issue notification identifying the anomaly, providing an image of the model with estimated position of anomaly, or other suitable notification…In other forms, the server 5 is trained to learn different types of anomalies. The server 5 includes neural networks that learn and recognize different types of defects by training. In some forms, training data for training the server 5 may include the estimated inlier templates. The training data may include outlier models which indicate different types of defects. The training data may include CAD images of components, scanned pictures of components with or without different types of defects and the like. Based on learning by the training data, the server 5 is trained to learn and classify different types of defects”).
Schultz does not explicitly disclose, but suggests wherein the image conversion device is located in a far edge of the structure inspection system that is proximate to the physical structure (¶ 70: “The server 450 is arranged in a network proximity to the conveyor structure 430 in order to prevent a potential network delay or buffering. In some forms, the server 450 and the imaging system 420 are arranged to be physically close to the conveyor structure 430 but the teachings of the present disclosure are not limited thereto.”); and
located in a near edge of the structure inspection system that is remove from the physical structure (¶ 70: “The server 450 is arranged in a network proximity to the conveyor structure 430 in order to prevent a potential network delay or buffering. In some forms, the server 450 and the imaging system 420 are arranged to be physically close to the conveyor structure 430 but the teachings of the present disclosure are not limited thereto.”);
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to locate the server and capture system in different locations.
As suggested by Schultz, one of ordinary skill in the art would have recognized that the inspection and server side could be located either together (in order to reduce latency) or in a different configuration (e.g. not co-located)(“In some forms, the server 450 and the imaging system 420 are arranged to be physically close to the conveyor structure 430 but the teachings of the present disclosure are not limited thereto.”)(¶ 70). One of ordinary skill in the art would have had a reasonable expectation of success because there are a limited number of configuration options (e.g. co-location or not) depending on the acceptable level of latency.
Schultz does not explicitly disclose, but Wang discloses structure inspection (abstract: “A construction inspection method includes establishing a first construction model based on detection data at a construction site, comparing the first construction model with a previously established second construction model based on construction design data to acquire a comparison result, confirming an inspection result of a construction in accordance with the comparison result, and sharing the inspection result with a user”)
provide a characterization of the deviation (¶ 121: “When there is an inconsistent part between the two, the ID of the corresponding component in the BIM model is recorded, and the cause of inconsistency is also recorded, for example, the mismatch of the inclination angle, the mismatch of the position, or the misuse of the material… Then, the owner or the construction team may view the inspection result in real time, which facilitates subsequent corrections for the inspection result and accelerates the inspection procedure.”)
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to apply the techniques in Schultz to structure inspection and characterization.
One of ordinary skill in the art would have motivation to verify construction against an ideal model, which is appliable to both manufacturing and construction applications. One of ordinary skill in the art would have had a reasonable expectation of success because both methods use model comparison to determine deficiencies.
Claim 2
Schultz discloses wherein the image conversion device includes a repository to store the first real time 3D model and the baseline 3D model (Schultz, ¶ 56: “Once difference(s) between the template CAD model and the 3D model are recognized, different templates relating to different types of anomalies are aligned with respect to the 3D model, at 370. The anomalies of automotive components may include as splits, burrs, scratches, slug marks, etc. In some forms, different types of defects may be classified and templates corresponding to different defects are generated and prestored in the memory of the server 5. For example, one or more templates relating to splits are generated and prestored. Likewise, one or more templates relating to burrs, scratches, slug marks, etc. may be prestored. In some forms, pattern recognition techniques may be used and the server 5 is trained to learn different defects by using training data.”)
Claim 3
Schultz does not disclose, but Wang discloses wherein the image conversion device utilizes the baseline 3D model from the repository in rendering the first real time 3D model (“At Step 101, the first construction model is established based on detection data at the construction site….Step 106: positioning the second construction model at the construction site to superimpose the second construction model on a construction area and displaying the second construction model by the augmented reality device. Furthermore, the first construction model and the comparison result may be displayed by the augmented reality device.”).
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to use both models as claimed.
One of ordinary skill in the art would have motivation to provide improved comparison (“and the difference between the two models, i.e., the comparison result, may be intuitively observed.”)(¶ 76). One of ordinary skill in the art would have had a reasonable expectation of success because both techniques use model comparison.
Claim 8
Schultz does not disclose, but Wang discloses wherein in the determination of whether the first real time 3D model deviates from the baseline 3D model, the image conversion device is further configured to determine that the first real time 3D model violates a regulation governing the physical structure (¶ 103: “For example, the quality inspection items of the air conditioner may include a device model number, spatial position, horizontal angle, and vertical angle. Here, it is assumed that the position mismatch of the air conditioner is ±20 cm or less and the angle mismatch is ±10° or less. The liquid pipe and the gas pipe are requested to be provided with a thermal insulation material, the position mismatch of the liquid pipe and the gas pipe is ±30 cm or less, and the inclination angle is requested to meet the design requirement and not to exceed 10% beyond the range.”)
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to analyze design regulations.
One of ordinary skill in the art would have motivation to provide improved comparison and identify in context defects (“and the difference between the two models, i.e., the comparison result, may be intuitively observed.”)(¶ 76). One of ordinary skill in the art would have had a reasonable expectation of success because both techniques use model comparison.
Claim 9
Schultz does not disclose, but Wang discloses wherein the indication identifies the regulation (¶ 92: “For example, the ID of the component having a mismatch may be recorded. In addition, specific content of the mismatch may be recorded.”)
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to analyze design regulations.
One of ordinary skill in the art would have motivation to provide improved comparison and identify in context defects (“and the difference between the two models, i.e., the comparison result, may be intuitively observed.”)(¶ 76). One of ordinary skill in the art would have had a reasonable expectation of success because both techniques use model comparison.
Claim 10
Schultz discloses wherein the image conversion device includes an image capture device configured to generate the image ((¶ 49: “In some forms, the cameras 40, 45 include a 2D digital camera (color cameras and/or monochrome cameras) that capture two dimensional (2D) images 42, 48, which are provided to the server 5, respectively. In one form, the cameras 40, 45 are configured to detect small surface defects in large patches of the surface area as needed. In at least one variation, one or more of the cameras 40, 45 may include a 3D camera. The 3D cameras may capture and produce a 3D image by taking a set of pictures of an object from different angles and converting the set of pictures into a 3D model with appropriate software.”)
Claim 11
The same teachings and rationales in claim 1 are applicable to claim 11.
Claim 12
The same teachings and rationales in claim 2 are applicable to claim 12.
Claim 13
The same teachings and rationales in claim 3 are applicable to claim 13.
Claim 18
The same teachings and rationales in claim 8 are applicable to claim 18.
Claim 19
The same teachings and rationales in claim 9 are applicable to claim 19.
Claim 20
The same teachings and rationales in claim 1 are applicable to claim 20 with Schultz further disclosing an image conversion device including an image capture device and a repository, the image capture device to generate an image of a physical structure (¶ 49: “In some forms, the cameras 40, 45 include a 2D digital camera (color cameras and/or monochrome cameras) that capture two dimensional (2D) images 42, 48, which are provided to the server 5, respectively. In one form, the cameras 40, 45 are configured to detect small surface defects in large patches of the surface area as needed. In at least one variation, one or more of the cameras 40, 45 may include a 3D camera. The 3D cameras may capture and produce a 3D image by taking a set of pictures of an object from different angles and converting the set of pictures into a 3D model with appropriate software.”)
Claim(s) 4-7, 14-17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Schultz (US 2022/0327688) in view of Wang (US 2024/0104714) and Santarone (US 2022/0382929)
Claim 4
Schultz does not disclose but Santarone discloses in the same filed of endeavor (“As Built data is collected that quantifies details of how a specific physical structure was actually constructed. According to the present invention, a Processing Facility is designed and modeled in a 3D virtual setting. As Built data is combined with a design model in a virtual setting to generate an AVM. As Built data may reflect one or more of: fabrication of the Processing Facility; repair; maintenance; upgrades; improvements; and work order execution associated with the Processing Facility.”) wherein the near edge processor is further configured to provide a lifecycle log of the physical structure (Santarone, ¶ 26: “As Built data may include modifications to a Commercial Property that are made during a construction phase, and/or during a Deployment phase, of a Commercial Property life cycle. Similarly, as Deployed data may include details quantifying one or more of: machine operators, production quantity, yield, quality level, usage, maintenance, repairs and improvements performed on the Commercial Property.”)
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to consider life cycle.
One of ordinary skill in the art would have motivation to track expected conditions over time. One of ordinary skill in the art would have had a reasonable expectation of success because both methods use model comparison.
Claim 5
Schultz does not disclose but Santarone discloses wherein the lifecycle log includes the first real time 3D model and the baseline 3D model (Santarone, ¶ 56: “An Augmented Virtual Model includes a three or four dimensional model in a virtual environment that exists parallel to physical embodiments modeled in the Augmented Virtual Model. Details of one or more physical structures and other features within a commercial real estate parcel are generated and quantified and represented in the Augmented Virtual Model. The Augmented Virtual Model exists in parallel to a physical structure in that the AVM includes virtual representations of physical structures and additionally receives and aggregates data relevant to the structures over time. The aggregation of data may be one or more of: a) according to an episode (i.e. onsite inspection, repair, improvement etc.); b) periodic; and c) in real time (without built in delay).”)
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to consider life cycle.
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to consider life cycle.
One of ordinary skill in the art would have motivation to track expected conditions over time. One of ordinary skill in the art would have had a reasonable expectation of success because both methods use model comparison.
Claim 6
Schultz does not disclose but Santarone discloses wherein the lifecycle log further includes a second real time 3D model of the physical structure (Santarone, ¶ 56: “An Augmented Virtual Model includes a three or four dimensional model in a virtual environment that exists parallel to physical embodiments modeled in the Augmented Virtual Model. Details of one or more physical structures and other features within a commercial real estate parcel are generated and quantified and represented in the Augmented Virtual Model. The Augmented Virtual Model exists in parallel to a physical structure in that the AVM includes virtual representations of physical structures and additionally receives and aggregates data relevant to the structures over time. The aggregation of data may be one or more of: a) according to an episode (i.e. onsite inspection, repair, improvement etc.); b) periodic; and c) in real time (without built in delay).”)
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to consider life cycle.
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to consider life cycle.
One of ordinary skill in the art would have motivation to track expected conditions over time. One of ordinary skill in the art would have had a reasonable expectation of success because both methods use model comparison.
Claim 7
Schultz does not disclose but Santarone discloses wherein the second real time 3D model is rendered from a plurality of second images of the physical structure by the image conversion device (Santarone, ¶ 12: “In general, As Built and Experiential Data generated according to the present invention includes one or more of: image data, measurements, component specifications of placement; solid state; electrical; and electromechanical devices (or combination thereof); generate data capturing conditions experienced by a structure.”)
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to consider using additional images.
One of ordinary skill in the art would have motivation to track expected conditions over time. One of ordinary skill in the art would have had a reasonable expectation of success because both methods use model comparison.
Claim 14
The same teachings and rationales in claim 4 are applicable to claim 14.
Claim 15
The same teachings and rationales in claim 5 are applicable to claim 15.
Claim 16
The same teachings and rationales in claim 6 are applicable to claim 16.
Claim 17
The same teachings and rationales in claim 7 are applicable to claim 17.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RYAN M GRAY whose telephone number is (571)272-4582. The examiner can normally be reached on Monday through Friday, 9:00am-5:30pm (EST).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached on (571)272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RYAN M GRAY/Primary Examiner, Art Unit 2611