CTNF 19/073,278 CTNF 91058 DETAILED ACTION Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. Double Patenting 08-33 AIA The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg , 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman , 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi , 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum , 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel , 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington , 418 F.2d 528, 163 USPQ 644 (CCPA 1969). A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA. A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b). The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13. The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA/25, or PTO/AIA/26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer. 08-34 AIA Claim s 1-20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claim s 1-20 of U.S. Patent No. 12283136 . Although the claims at issue are not identical, they are not patentably distinct from each other because the claims recite nearly identical limitations that overlap in scope, as detailed in claim 1 below. Similarly, claims 2-20, while not identical to claims 2-20 of U.S. Patent No. 12283136, are also not patentably distinct from each other because the claims recite nearly identical limitations that overlap in scope, and thus are further rejected for substantially the same reasons, mutatis mutandis . U.S. Patent No. 12283136 (Current App) No 19/073,278 A method, comprising: A method, comprising: receiving a model of an interior of a vehicle, the model defining locations of components in the interior of the vehicle and component identifie rs; receiving a model of an interior of a vehicle, the model of the interior of the vehicle defining locations of components in the interior of the vehicle and component identifiers; capturing images of the interior of a vehicle using one or more cameras; capturing images of the interior of a vehicle using one or more cameras; identifying components in the images using a machine learning (ML) model; identifying components in the images using a machine learning (ML) model; mapping the components identified by the ML model to the components in the model to generate a digital twin of the vehicle, wherein the digital twin correlates the components identified by the ML model to the component identifiers from the model; mapping the components identified by the ML model to the components in the model to generate a digital twin of the vehicle, wherein the digital twin correlates the components identified by the ML model to the component identifiers from the model; receiving, from a wirelessly-connected user device, an image of the interior of the vehicle depicting a first component needing maintenance; and receiving an image of the interior of the vehicle comprising a first component needing maintenance. receiving, from the user device, information describing a location of the user device in the interior of the vehicle; mapping, based on the location of the user device, the first component needing maintenance to a component identifier derived from the digital twin; displaying, on the user device, a GUI asking the user device to confirm the first component needing maintenance depicted in the image; and dispatching maintenance to fix the first component needing maintenance using the component identifier . Claim Rejections - 35 USC § 103 07-06 AIA 15-10-15 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. 07-21-aia AIA Claim s 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Tyson II (U.S. 11, 132,479) . In re claim 1, Tyson discloses a method, comprising: receiving a model (fig. 6; CAD model 12; [Col. 6, lines 10-20]) of an interior of a vehicle, the model of the interior of the vehicle defining locations of components in the interior of the vehicle and component identifiers (fig. 6; steps 11a-14; an assembled structure may comprise a vehicle…produced from assembling a plurality of components together; [Col. 3, lines 19-27]; the component parts are typically three-dimensional objects that are required to be assembled to form a new structure [Col. 13, lines 15-17]; [Col. 6, lines 25-36; Col. 15, lines 55-67;]); capturing images of the interior of a vehicle using one or more cameras (fig. 6; steps 15-16 and 21; The system preferably includes one or more imaging devices comprised of imaging components, which may include one or more cameras that are configured to capture the images of the work area in real-time; [Col. 13, lines 28-31]; the imaging component comprises a camera with a lens, and has an image sensor that captures image data of the objects in the field of view; [Col. 13, lines 37-39]; also see [Col. 15, line 67-Col. 16, line 1]); identifying components in the images using a machine learning (ML) model (fig. 6; steps 15-17 and 21; the system is configured with a library containing recognition data so that when a component is presented within the field of view, the recognition engine processes the component image information to make a recognition match with a library of components; [Col. 35, lines 67-Col. 36, line 8]); mapping the components identified by the ML model to the components in the model to generate a digital twin (fig. 6; digital duplicate 70) of the vehicle, wherein the digital twin correlates the components identified by the ML model to the component identifiers from the model (fig. 6; elements 40, 50, 60 and 70; [Col. 16, lines 36-47]); and receiving an image (one or more cameras that are configured to capture the images of the work area in real-time; [Col. 13, lines 28-31]; note : the captured images of the work area would necessarily also depict components present within that work area that need to be serviced.) of the interior of the vehicle comprising a first component needing maintenance (the system may be used to track repairs to a portion of the structure. For example, where as part of maintenance , or part of a repair of a failed part, the repaired part or portion of the structure may be marked with a marking arrangement that provides details of the component or structure movements from the as-built structure, or any point in time, including at the time of repair, and thereafter. [Col. 11, lines 46-53]; Here, in [Col. 11, lines 46-53], images of components needing maintenance are captured along with other components present within the image, which may not need repair). Tyson lacks receiving a model of an interior of a vehicle, the model defining locations of components in the interior of the vehicle; identifying components in the images using a machine learning (ML) model; mapping the components identified by the ML model to the components in the model However, it would have been obvious, before the effective filing date of the claimed invention, to a person having ordinary skill in the art that a model can also include structural/non-structural elements and surfaces, such as those present in an interior of a vehicle as well as interior components of that vehicle. Additionally, it also would have been obvious to one having ordinary skill in the art at the time the invention was made to automate the identification and mapping of components using a machine learning (ML) model , since it has been held that broadly providing a mechanical or automatic means (in this case, automation via a machine learning (ML) model) to replace manual activity which has accomplished the same result involves only routine skill in the art. In re Venner, 120 USPQ 192. In re claim 2, Tyson teaches the method of claim 1, wherein mapping the components identified by the ML model to the components in the model comprises: identifying locations associated with the images (fig. 6; element 17; [Col. 16, lines 3-6]); correlating the locations associated with the images to the locations of the components in the model (fig. 6; element 23-24; [Col. 16, lines 7-22]); and assigning the component identifiers in the model to the components identified by the ML model based on correlating the locations associated with the images to the locations of the components in the model (fig. 6; element 27; [Col. 16, lines 35-36]). In re claims 3-4, Tyson teaches the method of claim 1, and further teaches wherein the model is a 3D model of the interior of the vehicle (imaging devices may be arranged and configured to employ photogrammetry to track target dots and locations in 3D space, and digital image correlation (DIC) to track complete surfaces; [Col. 17, line 37-41]); and wherein the model is a 3D computer-aided design (CAD) of the interior of the vehicle (fig. 6; elements 11a-14; [abstract]). In re claim 5, Tyson teaches the method of claim 4, further comprising: receiving the component identifiers of the components in the interior of the vehicle (imaging devices may be arranged and configured to employ photogrammetry to track target dots and locations in 3D space, and digital image correlation (DIC) to track complete surfaces; [Col. 17, line 37-41]; the system is configured with a library containing recognition data so that when a component is presented within the field of view, the recognition engine processes the component image information to make a recognition match with a library of components; [Col. 35, lines 67-Col. 36, line 8]); receiving locations of the components in the interior of the vehicle (imaging devices may be arranged and configured to employ photogrammetry to track target dots and locations in 3D space, and digital image correlation (DIC) to track complete surfaces; [Col. 17, line 37-41]); and generating the 3D CAD based on the component identifiers and the locations of the components ([abstract; Col. 9 Col. 35, line 67-Col. 36, line 8]). In re claim 6, Tyson teaches the method of claim 1, further comprising, after generating the digital twin ( note: digital duplicate 70, as shown in fig. 6 occurs during as part of the construction process, and as such, a maintenance operation/repairs would logically occur after the build has been completed): identifying (see “marking” above) a plurality of components in the image using a first ML model ([Col. 11, lines 46-53]); mapping the plurality of components to a subset of components (part of a repair of a failed part, the repaired part or portion of the structure) in the digital twin based on a location associated with the image ([Col. 11, lines 46-53]); receiving a selection of one of the subset of components (i.e. an individual part) as the first component needing maintenance ([Col. 11, lines 46-53]); and dispatching maintenance to fix the first component needing maintenance using a component identifier for the first component that is derived from the digital twin (field repair and structural health monitoring; [Col. 16, lines 47-54]; additionally, in [Col. 19, ln 41-53], Tyson II teaches “The RVAT may be provided with the knowledge of the assembly operation or step, and may associate one or more tools, components, fastening, or other build steps to generate an automatic build step or grouping of steps. The engineer or other technician may use this feature when planning and setting up the RVAT system for operators to use to build a structure. (See e.g., the exemplary depiction of the set up operation (FIGS. 10 and 11), which may be configured to auto-generate the build steps). According to some embodiments, the engineer or other technician carrying out the set up may change or modify the automatic build steps (as desired), and according to other embodiments, auto-build steps may require review and approval at set up.”. This being the case, it seems that in the case of a field repair or other maintenance/repair operation, that Tyson II at least implicitly teaches that these build steps can also be auto-generated. This being the case, it seems that Tyson II teaches dispatching maintenance to fix/repair a first component needing maintenance). In re claim 7, Tyson teaches the method of claim 6, further comprising, after dispatching maintenance: updating the digital twin (update digital-twin to the current condition; [Col. 12, lines 16-25]) to include at least one of a record indicating the first component was repaired (the system may be used to track repairs to a portion of the structure; [Col. 11, lines 46-53]) or an updated component identifier if the first component was replaced with a different component (update digital-twin to the current condition; [Col. 12, lines 16-25]). In re claim 8, Tyson teaches the method of claim 6, further comprising, after mapping the plurality of components to a subset of components in the digital twin: transmitting for display, on a user device, a plurality of labels for the subset of components, wherein the user device captured the image and provides an augmented reality (AR) experience that permits a user to select from one of the subset of components as the first component needing maintenance ([Col. 9, line 59- Col. 10, line 5]). In re claim 9, Tyson teaches a system, comprising: a processor (system includes one or more computing devices, such as, for example, a computer with a processor ; [Col. 13, line 22-28]); and a memory (memory device; [Col. 22, line 30-34]) including instructions that when executed by the processor enable the system to perform an operation, the operation comprising: receiving a model (fig. 6; CAD model 12; [Col. 6, lines 10-20]) of an interior of a vehicle, the model of the interior of the vehicle defining locations of components in the interior of the vehicle and component identifiers (fig. 6; steps 11a-14; an assembled structure may comprise a vehicle…produced from assembling a plurality of components together; [Col. 3, lines 19-27]; the component parts are typically three-dimensional objects that are required to be assembled to form a new structure [Col. 13, lines 15-17]; [Col. 6, lines 25-36; Col. 15, lines 55-67;]); capturing images of the interior of a vehicle using one or more cameras (fig. 6; steps 15-16 and 21; The system preferably includes one or more imaging devices comprised of imaging components, which may include one or more cameras that are configured to capture the images of the work area in real-time; [Col. 13, lines 28-31]; the imaging component comprises a camera with a lens, and has an image sensor that captures image data of the objects in the field of view; [Col. 13, lines 37-39]; also see [Col. 15, line 67-Col. 16, line 1]); identifying components in the images using a machine learning (ML) model (fig. 6; steps 15-17 and 21; the system is configured with a library containing recognition data so that when a component is presented within the field of view, the recognition engine processes the component image information to make a recognition match with a library of components; [Col. 35, lines 67-Col. 36, line 8]); and mapping the components identified by the ML model to the components in the model to generate a digital twin (fig. 6; digital duplicate 70) of the vehicle, wherein the digital twin correlates the components identified by the ML model to the component identifiers from the model (fig. 6; elements 40, 50, 60 and 70; [Col. 16, lines 36-47]); and receiving an image (one or more cameras that are configured to capture the images of the work area in real-time; [Col. 13, lines 28-31]; note : the captured images of the work area would necessarily also depict components present within that work area that need to be serviced) of the interior of the vehicle comprising a first component needing maintenance (the system may be used to track repairs to a portion of the structure. For example, where as part of maintenance , or part of a repair of a failed part, the repaired part or portion of the structure may be marked with a marking arrangement that provides details of the component or structure movements from the as-built structure, or any point in time, including at the time of repair, and thereafter. [Col. 11, lines 46-53]; Here, in [Col. 11, lines 46-53], images of components needing maintenance are captured along with other components present within the image, which may not need repair). Tyson lacks receiving a model of an interior of a vehicle, the model defining locations of components in the interior of the vehicle; identifying components in the images using a machine learning (ML) model; mapping the components identified by the ML model to the components in the model However, it would have been obvious, before the effective filing date of the claimed invention, to a person having ordinary skill in the art that a model can also include structural/non-structural elements and surfaces, such as those present in an interior of a vehicle as well as interior components of that vehicle. Additionally, it also would have been obvious to one having ordinary skill in the art at the time the invention was made to automate the identification and mapping of components using a machine learning (ML) model , since it has been held that broadly providing a mechanical or automatic means (in this case, automation via a machine learning (ML) model) to replace manual activity which has accomplished the same result involves only routine skill in the art. In re Venner, 120 USPQ 192. In re claim 10, see claims 2 and 9 above. In re claim 11, see claims 4 and 9 above. In re claim 12, see claims 5 and 11 above. In re claim 13, see claims 6 and 9 above. In re claim 14, see claims 7 and 13 above. In re claim 15, see claims 8 and 13 above. In re claim 16, see claims 1 and 6-7 above. In re claim 17, see claims 16 and 6-7 above. In re claim 18, see claims 8 and 16 above. In re claim 19, see claims 2, 8 and 18 above. In re claim 20, see claims 8 and 18. Tyson further teaches a decision tree, and further states that “Those skilled in the art should readily appreciate that functions, operations, decisions…flow charts or block diagrams may be implemented as computer program instructions” ([Col. 60, line 25-32]). It would be have been apparent and obvious to those having ordinary skill in the art that decisions…flow charts or block diagrams are a common visual tool for problem solving, which could also be readily applied to identify a problem with a component needing maintenance, since maintenance is also one of the functions taught be Tyson, as explained above in regard to claim 6. Conclusion The prior art of Tang et al. (U.S. 20200334643), was found that seems like it would be useful to the applicant. Tang uses AI/machine learning and acquires image data to be used with a model to detect issues/components that may be in need of repair/maintenance in addition to identifying parts of the vehicle that have been repaired or replaced. The prior art of Jakka et al. (U.S. 20210279852) discloses a method for using a virtual representation of an indoor environment to identify contents that have been damaged (e.g., by flooding). A virtual representation of a physical scene of an indoor environment is processed to identify a list of contents in the physical scene. The virtual representation may include 2-dimensional representations of the physical scene (e.g., images or video) or a 3-dimensional representation of the physical scene (e.g., 3D digital model). A reference line is determined in the virtual representation that is indicative of a maximum vertical extent of the damage in the physical scene. The position of the reference line is compared with the position of the identified contents in the virtual representation to determine contents that are likely to be damaged. For example, the contents that are at or below a plane represented by the reference line in the virtual representation may be identified as damaged. Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOHN D BAILEY whose telephone number is (571)272-5692. The examiner can normally be reached M-F 8-5. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Logan Kraft can be reached at 571-270-5625. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /JOHN D BAILEY/Examiner, Art Unit 3747 /KURT PHILIP LIETHEN/Primary Examiner, Art Unit 3747 Application/Control Number: 19/073,278 Page 2 Art Unit: 3747 Application/Control Number: 19/073,278 Page 3 Art Unit: 3747 Application/Control Number: 19/073,278 Page 4 Art Unit: 3747 Application/Control Number: 19/073,278 Page 5 Art Unit: 3747 Application/Control Number: 19/073,278 Page 6 Art Unit: 3747 Application/Control Number: 19/073,278 Page 7 Art Unit: 3747 Application/Control Number: 19/073,278 Page 8 Art Unit: 3747 Application/Control Number: 19/073,278 Page 9 Art Unit: 3747 Application/Control Number: 19/073,278 Page 10 Art Unit: 3747 Application/Control Number: 19/073,278 Page 11 Art Unit: 3747 Application/Control Number: 19/073,278 Page 12 Art Unit: 3747 Application/Control Number: 19/073,278 Page 13 Art Unit: 3747 Application/Control Number: 19/073,278 Page 14 Art Unit: 3747 Application/Control Number: 19/073,278 Page 15 Art Unit: 3747 Application/Control Number: 19/073,278 Page 16 Art Unit: 3747 Application/Control Number: 19/073,278 Page 17 Art Unit: 3747
Prosecution Insights