DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 14-17 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Sudharsan-Grober et al. (US Publication Number 2024/0168546 A1, hereinafter “Sudharsan”) in view of Sawhney et al. (US Patent Number 11,514,648 B2, hereinafter “Sawhney”).
(1) regarding claim 1:
As shown in fig. 2, Sudharsan disclosed a portable electronic device for assisting a user in locating a condition of a system within a real-world environment (para. [0073], note that FIG. 1 shows a method for identifying at least one a place of interest on a physical object through its 3D model in augmented reality view using AR device) comprising:
a display (fig. 4, para. [0097], note that [0097] FIG. 4 shows identifiers (shown as arrows 5) positioned around the places of interest as selected by the user. The POIs in this example are holes on a car door frame. At this stage, the display of virtual identifiers 5 (shown in FIG. 4));
an image capture device (para. [0123], note that the photo taking logic involves in its AR view a photo frame 9, as shown in FIG. 8); and
a processor configured to: receive, from the image capture device, image data associated with a representation of the real-world environment (para. [0075], note that in Step S9: a position of the physical object within the physical environment using a spatial awareness technology is determined);
align a three-dimensional virtual model with the real-world environment, wherein the three-dimensional virtual model is a representation of the real-world environment (para. [0076], note that in Step S1: the 3D model with the physical object within the physical environment is aligned in an augmented reality view, [0077] Step S4: Adapting a transparency of the 3D model);
receive information associated with the condition of the system (para. [0087], note that places of Interests (POI) are explained as any area or part of the machine, object, etc. in context, where the user needs to focus on, to perform a particular task);
determine a location in the three-dimensional virtual model corresponding to the condition of the system (para. [0096], note that the point or small area of the intersection is instantiated as POI in the 3D model 2a and it is registered in the 3D map of the physical environment 1. Each of the POIs created this way thus will have its associated positional information).
Sudharsan disclosed most of the subject matter as described as above except for specifically teaching provide a graphical indicator to be displayed over the representation of the real-world environment, wherein the graphical indicator is to be displayed at a location on the display that corresponds to a physical location of the condition of the system in the real-world environment.
However, Sawhney disclosed provide a graphical indicator to be displayed over the representation of the real-world environment, wherein the graphical indicator is to be displayed at a location on the display that corresponds to a physical location of the condition of the system in the real-world environment (col. 7, lines 32-40, note that shown in FIG. 1A, the physical object 116 is illustrated as being a printer unit that is resting on the floor of a room (which in this case is the physical environment 118). Upon receiving the image input data 104 and the model input data 106, the image data annotation system 102 possesses both a video file made up of images of a real-life physical instance of the printer unit in addition to a 3D model (e.g., a STEP-File or other CAD file) that defines geometric features of a virtual instance of the printer unit. As described below, the image data annotation system 102 utilizes these representations of the physical object to generate the annotated image data 110.).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art to teach provide a graphical indicator to be displayed over the representation of the real-world environment, wherein the graphical indicator is to be displayed at a location on the display that corresponds to a physical location of the condition of the system in the real-world environment. The suggestion/motivation for doing so would have been in order to annotate a physical object within individual images frames of an image sequence with relevant object annotations based on a three-dimensional (3D) model of the physical object. Annotating the individual image frames with object annotations includes updating individual image frames within image input data to generate annotated image data that is suitable for reliably training a DNN object detection architecture (abs.). Therefore, it would have been obvious to combine Sudharsan with Sawhney to obtain the invention as specified in claim 1.
(2) regarding claim 2:
Sudharsan further disclosed the portable electronic device of claim 1, wherein the processor is further configured to generate the graphical indicator for display over the representation of the real-world environment (10, fig. 9, para. [0126], FIG. 9 shows taking a picture of previously identified place of interest on a physical object as an example where a technician marks something on a physical object 2b within a physical environment 1 as missing or wrong, especially errors, faults, scratches, dents, and wants to take a picture to protocol it).
(3) regarding claim 3:
Sudharsan further disclosed the portable electronic device of claim 1, wherein the graphical indicator comprises an icon, a symbol, text, or a combination thereof (10, fig. 9, note that an icon is displayed. Also see para. [0126]).
(4) regarding claim 14:
Sudharsan further disclosed the portable electronic device of claim 1, wherein determining of the alignment of the system in the real-world environment with the three-dimensional virtual model of the system further includes determining that at least one virtual object or fiducial of the three-dimensional virtual model is aligned with a corresponding object or fiducial in the real-world environment (para. [0086], note that [0086] FIG. 2 shows an alignment of a 3D model 2a on actual physical object in a physical environment 1. This step (step S1 in FIG. 1) may be necessary when the 3D model 2a needs to be aligned/overlaid in augmented reality onto an existing physical object 2b. Alignment is a common feature required when the overlaid 3D model along with its pre-imposed instructions or annotations are helping a service technician 3 to focus on any place of interest. In FIG. 2 the 3D model is arranged as an 3D outline), and wherein the processor is further configured to provide a notification when the system in the real-world environment is aligned with the three-dimensional virtual model of the system (para. [0086], note that alignment is a common feature required when the overlaid 3D model along with its pre-imposed instructions or annotations are helping a service technician 3 to focus on any place of interest. In FIG. 2 the 3D model is arranged as an 3D outline).
(5) regarding claim 15:
Sudharsan further disclosed the portable electronic device of claim 1, wherein the processor is further configured to: identify the system based on user input (para. [0078], note that the motion data is determined by tracking at least one gesture of a user related to the physical object); and receive information including the three-dimensional virtual model of the system from a computing device (para. [0086], note that FIG. 2 shows an alignment of a 3D model 2a on actual physical object in a physical environment 1. This step (step S1 in FIG. 1) may be necessary when the 3D model 2a needs to be aligned/overlaid in augmented reality onto an existing physical object 2b. Alignment is a common feature required when the overlaid 3D model along with its pre-imposed instructions or annotations are helping a service technician 3 to focus on any place of interest).
(6) regarding claim 16:
Sudharsan further disclosed the portable electronic device of claim 1, wherein the processor is further configured to: provide a representation of one or more components of the system for display on the display, wherein the one or more components are hidden from a view of the user (para. [0110], note that [0110] FIG. 6 shows touching and selecting an identifier 5 of a place of interest of a physical object 2b in a physical environment 1 by hand 3. This is especially usefully for selecting an identifier 5 to perform an inspection or to add inspection information, to mark something as missing or faulty is a common use case in inspections or quality assurance); and provide additional information associated with the one or more components for display on the display (para. [0115], note that FIG. 7 shows an example AR user manual 7 created according to the method step (S13, FIG. 1): Adding instructional data to said at least one place of interest. The AR user manual including instructional data 7 or work instructions. In almost all industry use cases mentioned above, there exists a need for the service technician to refer to a user manual 7 to review steps of instruction to complete a task.).
The proposed rejection of claim 1, renders obvious the steps of the method (fig. 1) claim 17 and the non-transitory computer-readable medium (para. [0070]) claim 20 because these steps occur in the operation of the proposed rejection as discussed above. Thus, the arguments similar to that presented above for claim 1 is equally applicable to claims 17 and 20.
Claim(s) 4-9, 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Sudharsan and Sawhney, and further in view of Yaramasu (NPL, “Aircraft Electrical System Intermittent Arc Fault Detection and Location”, 2012).
(1) regarding claim 4:
Sudharsan disclosed most of the subject matter as described as above except for specifically teaching wherein the condition comprises a wiring fault or error of the system, and wherein the system comprising an electrical wiring system in an installed state.
However, Yaramasu teaches wherein the condition comprises a wiring fault or error of the system, and wherein the system comprising an electrical wiring system in an installed state (page 1, I. Introduction, para. [0001], note that the breached or degraded wire insulation can manifest as intermittent arc faults during harsh in-flight vibrations and weather conditions or under mechanical stresses on the wires).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art to teach wherein the condition comprises a wiring fault or error of the system, and wherein the system comprising an electrical wiring system in an installed state. The suggestion/motivation for doing so would have been in order to detect and locate fault in an aircraft system (Abs.). Therefore, it would have been obvious to combine Sudharsan and Sawhney with Yaramasu to obtain the invention as specified in claim 4.
(2) regarding claim 5:
Sudharsan disclosed most of the subject matter as described as above except for specifically teaching wherein at least a portion of the electrical wiring system is hidden from a view of the user in the real-world environment.
However, Yaramasu teaches wherein at least a portion of the electrical wiring system is hidden from a view of the user in the real-world environment (Abs. note that Intermittent arc faults appear in aircraft power systems in unpredictable ways when the degraded wires are wet, vibrating against metal structures, or under mechanical stresses i.e. the wiring is inside an aircraft’s power system).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art to teach wherein at least a portion of the electrical wiring system is hidden from a view of the user in the real-world environment. The suggestion/motivation for doing so would have been in order to detect and locate fault in an aircraft system (Abs.). Therefore, it would have been obvious to combine Sudharsan and Sawhney with Yaramasu to obtain the invention as specified in claim 5.
(3) regarding claim 6:
Sudharsan disclosed most of the subject matter as described as above except for specifically teaching wherein the processor is further configured to receive information associated with the electrical wiring system.
However, Yaramasu teaches wherein the processor is further configured to receive information associated with the electrical wiring system (Abs. note that the ABCD matrix (or transmission matrix) modeling method is introduced to derive normal and faulty load circuit models with better accuracy and reduced complexity compared to the differential equation approach, and an intermittent arc fault detection method is proposed based on temporary deviations of load circuit model coefficients and wiring parameters).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art to teach wherein the processor is further configured to receive information associated with the electrical wiring system. The suggestion/motivation for doing so would have been in order to detect and locate fault in an aircraft system (Abs.). Therefore, it would have been obvious to combine Sudharsan and Sawhney with Yaramasu to obtain the invention as specified in claim 6.
(4) regarding claim 7:
Sudharsan disclosed most of the subject matter as described as above except for specifically teaching wherein the information includes a measurement of a distance from a starting point in the electrical wiring system to the condition.
However, Yaramasu teaches wherein the information includes a measurement of a distance from a starting point in the electrical wiring system to the condition (page 4, B. Modeling of a Load Circuit with Faulty Wire, para. [0003], note that representation is sufficient when the objective is to find the fault location. In Fig. 5, D1 denotes the distance of the fault location from the SSPC and D2 is that from the load end).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art to teach wherein the information includes a measurement of a distance from a starting point in the electrical wiring system to the condition. The suggestion/motivation for doing so would have been in order to detect and locate fault in an aircraft system (Abs.). Therefore, it would have been obvious to combine Sudharsan and Sawhney with Yaramasu to obtain the invention as specified in claim 7.
(5) regarding claim 8:
Sudharsan disclosed most of the subject matter as described as above except for specifically teaching wherein the starting point is at one end of at least one electrical wire of the electrical wiring system.
However, Yaramasu teaches wherein the starting point is at one end of at least one electrical wire of the electrical wiring system (page 9, B. Results, para. [0001], note that the distance D1 to the fault is 19.8 m under fault condition 1, which is close to load, and D1 is 4.8 m under fault condition 2, which is near the power bus).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art to teach wherein the information includes a measurement of a distance from a starting point in the electrical wiring system to the condition. The suggestion/motivation for doing so would have been in order to detect and locate fault in an aircraft system (Abs.). Therefore, it would have been obvious to combine Sudharsan and Sawhney with Yaramasu to obtain the invention as specified in claim 8.
(6) regarding claim 9:
Sudharsan disclosed most of the subject matter as described as above except for specifically teaching wherein the distance is determining by a fault locator, and wherein the fault locator comprises a time domain reflectometer.
However, Yaramasu teaches wherein the distance is determining by a fault locator, and wherein the fault locator comprises a time domain reflectometer (page 2, I. Introduction, para. [0005], note that Sequence time domain reflectometry (STDR) and spread spectrum time domain reflectometry (SSTDR) have been used to detect and locate intermittent arc faults).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art to teach wherein the distance is determining by a fault locator, and wherein the fault locator comprises a time domain reflectometer. The suggestion/motivation for doing so would have been in order to detect and locate fault in an aircraft system (Abs.). Therefore, it would have been obvious to combine Sudharsan and Sawhney with Yaramasu to obtain the invention as specified in claim 9.
The proposed rejection of claim 4, renders obvious the steps of the method (fig. 1) claim 18 because these steps occur in the operation of the proposed rejection as discussed above. Thus, the arguments similar to that presented above for claim 4 is equally applicable to claim 18.
Allowable Subject Matter
Claims 10-13 and 19 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. The following is a statement of reasons for the indication of allowable subject matter: the prior arts made of record do not teach “wherein the processor is further configured to determine a virtual path along a length of at least one electrical wire of the electrical wiring system, and wherein the virtual path starts from the starting point and extends along the length of the at least one electrical wire until the distance is reached”, as claimed in claims 10 and 19. Claims 11-13 depend on claim 10.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Petrov et al. (US Publication Number 2023/0252736 A1) disclosed a method for surfacing an XR object corresponding to an electronic message. The method includes: obtaining an electronic message from a sender; in response to determining that the electronic message is associated with a real-world object, determining whether a current field-of-view (FOV) of a physical environment includes the real-world object; and in accordance with a determination that the current FOV of the physical environment includes the real-world object, presenting, via the display device, an extended reality (XR) object that corresponds to the electronic message in association with the real-world object.
Rice et al. (NPL, "Extraction, Rendering and Augmented Interaction in the Wire Assembly of Commercial Aircraft", 2016) disclosed the development of a prototype digital wire routing system that adds flexibility and control in the electrical wire assembly of aircraft.
Li et al. (NPL, “An AR-Assisted Deep Learning-Based Approach for Automatic Inspection of Aviation Connectors”, 2021) disclosed an augmented reality (AR)-assisted deep learning-based approach to tackle three major challenges in the aviation connector inspection, including the small pins detection, multi-pins sequencing, and mismatched pins visualization.
Any inquiry concerning this communication or earlier communication from the examiner should be directed to Hilina K Demeter whose telephone number is (571) 270-1676.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Y. Poon could be reached at (571) 270- 0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about PAIR system, see http://pari-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HILINA K DEMETER/Primary Examiner, Art Unit 2617