DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments with respect to claim(s) 1-20 have been considered but they are not persuasive.
On page 9, Applicant argues that Sudharsan and Sawhney fail to disclosed “determining of the virtual locations” as recited in claims 1, 17 and 20.
In response: As specifically explained in para. [0018], Sudharsan disclosed an AR device can offer colliders, which are defining the shape of a 3D object in the 3D scene and includes virtual sensors that detect proximity of intersection among 3D objects i.e. virtual locations, on both the 3D models and the hand tracking are enabled. Also see, para. [0096], note that the point or small area of the intersection is instantiated as POI i.e. virtual location in the 3D model 2a and it is registered in the 3D map of the physical environment 1. Therefore, the stated argument is not persuasive. Claims 10-13 are object to include the allowable subject matter because claim 10 recites “wherein the processor is further configured to determine a virtual path along a length of at least one electrical wire of the electrical wiring system, and wherein the virtual path starts from the starting point and extends along the length of the at least one electrical wire until the distance is reached” in limitations of the intervening/depending claims.
On page 10, Applicant further argued that the combination of the prior is cannot support a prima facie case of obviousness.
In response: In response to applicant’s argument that there is no teaching, suggestion, or motivation to combine the references, the examiner recognizes that obviousness may be established by combining or modifying the teachings of the prior art to produce the claimed invention where there is some teaching, suggestion, or motivation to do so found either in the references themselves or in the knowledge generally available to one of ordinary skill in the art. See In re Fine, 837 F.2d 1071, 5 USPQ2d 1596 (Fed. Cir. 1988), In re Jones, 958 F.2d 347, 21 USPQ2d 1941 (Fed. Cir. 1992), and KSR International Co. v. Teleflex, Inc., 550 U.S. 398, 82 USPQ2d 1385 (2007). In this case, Sudharsan and Sawhney are from the same field of endeavor. The suggestion/motivation for doing so would have been in order to annotate a physical object within individual images frames of an image sequence with relevant object annotations based on a three-dimensional (3D) model of the physical object. Annotating the individual image frames with object annotations includes updating individual image frames within image input data to generate annotated image data that is suitable for reliably training a DNN object detection architecture (abs.). Therefore, it would have been obvious to combine Sudharsan with Sawhney to obtain the invention as specified in the claims.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 14-17 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Sudharsan-Grober et al. (US Publication Number 2024/0168546 A1, hereinafter “Sudharsan”) in view of Sawhney et al. (US Patent Number 11,514,648 B2, hereinafter “Sawhney”).
(1) regarding claim 1:
As shown in fig. 2, Sudharsan disclosed a portable electronic device for assisting a user in locating a condition of a system within a real-world environment (para. [0073], note that FIG. 1 shows a method for identifying at least one a place of interest on a physical object through its 3D model in augmented reality view using AR device) comprising:
a display (fig. 4, para. [0097], note that [0097] FIG. 4 shows identifiers (shown as arrows 5) positioned around the places of interest as selected by the user. The POIs in this example are holes on a car door frame. At this stage, the display of virtual identifiers 5 (shown in FIG. 4));
an image capture device (para. [0123], note that the photo taking logic involves in its AR view a photo frame 9, as shown in FIG. 8); and
a processor configured to: receive, from the image capture device, image data associated with a physical representation of the real-world environment (para. [0075], note that in Step S9: a position of the physical object within the physical environment using a spatial awareness technology is determined);
align a three-dimensional virtual model with the real-world environment, wherein the three-dimensional virtual model is a virtual representation of the real-world environment (para. [0076], note that in Step S1: the 3D model with the physical object within the physical environment is aligned in an augmented reality view, [0077] Step S4: Adapting a transparency of the 3D model);
receive first information associated with the condition of the system (para. [0087], note that places of Interests (POI) are explained as any area or part of the machine, object, etc. in context, where the user needs to focus on, to perform a particular task);
determine a virtual location in the three-dimensional virtual model corresponding to the condition of the system (para. [0096], note that the point or small area of the intersection is instantiated as POI i.e. virtual location in the 3D model 2a and it is registered in the 3D map of the physical environment 1. Each of the POIs created this way thus will have its associated positional information).
Sudharsan disclosed most of the subject matter as described as above except for specifically teaching provide a graphical indicator to be displayed over the virtual representation of the real-world environment, wherein the graphical indicator is to be displayed at a display location on the display that corresponds to a physical location of the condition of the system in the real-world environment.
However, Sawhney disclosed provide a graphical indicator to be displayed over the virtual representation of the real-world environment, wherein the graphical indicator is to be displayed at a display location on the display that corresponds to a physical location of the condition of the system in the real-world environment (col. 7, lines 32-40, note that shown in FIG. 1A, the physical object 116 is illustrated as being a printer unit that is resting on the floor of a room (which in this case is the physical environment 118). Upon receiving the image input data 104 and the model input data 106, the image data annotation system 102 possesses both a video file made up of images of a real-life physical instance of the printer unit in addition to a 3D model (e.g., a STEP-File or other CAD file) that defines geometric features of a virtual instance of the printer unit. As described below, the image data annotation system 102 utilizes these representations of the physical object to generate the annotated image data 110).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art to teach provide a graphical indicator to be displayed over the virtual representation of the real-world environment, wherein the graphical indicator is to be displayed at a display location on the display that corresponds to a physical location of the condition of the system in the real-world environment. The suggestion/motivation for doing so would have been in order to annotate a physical object within individual images frames of an image sequence with relevant object annotations based on a three-dimensional (3D) model of the physical object. Annotating the individual image frames with object annotations includes updating individual image frames within image input data to generate annotated image data that is suitable for reliably training a DNN object detection architecture (abs.). Therefore, it would have been obvious to combine Sudharsan with Sawhney to obtain the invention as specified in claim 1.
(2) regarding claim 2:
Sudharsan further disclosed the portable electronic device of claim 1, wherein the processor is further configured to generate the graphical indicator for display over the representation of the real-world environment (10, fig. 9, para. [0126], FIG. 9 shows taking a picture of previously identified place of interest on a physical object as an example where a technician marks something on a physical object 2b within a physical environment 1 as missing or wrong, especially errors, faults, scratches, dents, and wants to take a picture to protocol it).
(3) regarding claim 3:
Sudharsan further disclosed the portable electronic device of claim 1, wherein the graphical indicator comprises an icon, a symbol, text, or a combination thereof (10, fig. 9, note that an icon is displayed. Also see para. [0126]).
(4) regarding claim 14:
Sudharsan further disclosed the portable electronic device of claim 1, wherein determining of the alignment of the system in the real-world environment with the three-dimensional virtual model of the system further includes determining that at least one virtual object or fiducial of the three-dimensional virtual model is aligned with a corresponding physical object or fiducial in the real-world environment (para. [0086], note that [0086] FIG. 2 shows an alignment of a 3D model 2a on actual physical object in a physical environment 1. This step (step S1 in FIG. 1) may be necessary when the 3D model 2a needs to be aligned/overlaid in augmented reality onto an existing physical object 2b. Alignment is a common feature required when the overlaid 3D model along with its pre-imposed instructions or annotations are helping a service technician 3 to focus on any place of interest. In FIG. 2 the 3D model is arranged as an 3D outline), and wherein the processor is further configured to provide a notification when the system in the real-world environment is aligned with the three-dimensional virtual model of the system (para. [0086], note that alignment is a common feature required when the overlaid 3D model along with its pre-imposed instructions or annotations are helping a service technician 3 to focus on any place of interest. In FIG. 2 the 3D model is arranged as an 3D outline).
(5) regarding claim 15:
Sudharsan further disclosed the portable electronic device of claim 1, wherein the processor is further configured to: identify the system based on user input (para. [0078], note that the motion data is determined by tracking at least one gesture of a user related to the physical object); and receive second information including the three-dimensional virtual model of the system from a computing device (para. [0086], note that FIG. 2 shows an alignment of a 3D model 2a on actual physical object in a physical environment 1. This step (step S1 in FIG. 1) may be necessary when the 3D model 2a needs to be aligned/overlaid in augmented reality onto an existing physical object 2b. Alignment is a common feature required when the overlaid 3D model along with its pre-imposed instructions or annotations are helping a service technician 3 to focus on any place of interest).
(6) regarding claim 16:
Sudharsan further disclosed the portable electronic device of claim 1, wherein the processor is further configured to: provide a graphical representation of one or more components of the system for display on the display, wherein the one or more components are hidden from a view of the user (para. [0110], note that FIG. 6 shows touching and selecting an identifier 5 of a place of interest of a physical object 2b in a physical environment 1 by hand 3. This is especially usefully for selecting an identifier 5 to perform an inspection or to add inspection information, to mark something as missing or faulty is a common use case in inspections or quality assurance); and provide additional information associated with the one or more components for display on the display (para. [0115], note that FIG. 7 shows an example AR user manual 7 created according to the method step (S13, FIG. 1): Adding instructional data to said at least one place of interest. The AR user manual including instructional data 7 or work instructions. In almost all industry use cases mentioned above, there exists a need for the service technician to refer to a user manual 7 to review steps of instruction to complete a task.).
The proposed rejection of claim 1, renders obvious the steps of the method (fig. 1) claim 17 and the non-transitory computer-readable medium (para. [0070]) claim 20 because these steps occur in the operation of the proposed rejection as discussed above. Thus, the arguments similar to that presented above for claim 1 is equally applicable to claims 17 and 20.
Claim(s) 4-9, 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Sudharsan and Sawhney, and further in view of Yaramasu (NPL, “Aircraft Electrical System Intermittent Arc Fault Detection and Location”, 2012).
(1) regarding claim 4:
Sudharsan disclosed most of the subject matter as described as above except for specifically teaching wherein the condition comprises a wiring fault or error of the system, and wherein the system comprising an electrical wiring system in an installed state.
However, Yaramasu teaches wherein the condition comprises a wiring fault or error of the system, and wherein the system comprising an electrical wiring system in an installed state (page 1, I. Introduction, para. [0001], note that the breached or degraded wire insulation can manifest as intermittent arc faults during harsh in-flight vibrations and weather conditions or under mechanical stresses on the wires).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art to teach wherein the condition comprises a wiring fault or error of the system, and wherein the system comprising an electrical wiring system in an installed state. The suggestion/motivation for doing so would have been in order to detect and locate fault in an aircraft system (Abs.). Therefore, it would have been obvious to combine Sudharsan and Sawhney with Yaramasu to obtain the invention as specified in claim 4.
(2) regarding claim 5:
Sudharsan disclosed most of the subject matter as described as above except for specifically teaching wherein at least a portion of the electrical wiring system is hidden from a view of the user in the real-world environment.
However, Yaramasu teaches wherein at least a portion of the electrical wiring system is hidden from a view of the user in the real-world environment (Abs. note that Intermittent arc faults appear in aircraft power systems in unpredictable ways when the degraded wires are wet, vibrating against metal structures, or under mechanical stresses i.e. the wiring is inside an aircraft’s power system).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art to teach wherein at least a portion of the electrical wiring system is hidden from a view of the user in the real-world environment. The suggestion/motivation for doing so would have been in order to detect and locate fault in an aircraft system (Abs.). Therefore, it would have been obvious to combine Sudharsan and Sawhney with Yaramasu to obtain the invention as specified in claim 5.
(3) regarding claim 6:
Sudharsan disclosed most of the subject matter as described as above except for specifically teaching wherein the processor is further configured to receive second information associated with the electrical wiring system.
However, Yaramasu teaches wherein the processor is further configured to receive second information associated with the electrical wiring system (Abs. note that the ABCD matrix (or transmission matrix) modeling method is introduced to derive normal and faulty load circuit models with better accuracy and reduced complexity compared to the differential equation approach, and an intermittent arc fault detection method is proposed based on temporary deviations of load circuit model coefficients and wiring parameters).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art to teach wherein the processor is further configured to receive second information associated with the electrical wiring system. The suggestion/motivation for doing so would have been in order to detect and locate fault in an aircraft system (Abs.). Therefore, it would have been obvious to combine Sudharsan and Sawhney with Yaramasu to obtain the invention as specified in claim 6.
(4) regarding claim 7:
Sudharsan disclosed most of the subject matter as described as above except for specifically teaching wherein the second information includes a measurement of a distance from a starting point in the electrical wiring system to the condition.
However, Yaramasu teaches wherein the second information includes a measurement of a distance from a starting point in the electrical wiring system to the condition (page 4, B. Modeling of a Load Circuit with Faulty Wire, para. [0003], note that representation is sufficient when the objective is to find the fault location. In Fig. 5, D1 denotes the distance of the fault location from the SSPC and D2 is that from the load end).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art to teach wherein the second information includes a measurement of a distance from a starting point in the electrical wiring system to the condition. The suggestion/motivation for doing so would have been in order to detect and locate fault in an aircraft system (Abs.). Therefore, it would have been obvious to combine Sudharsan and Sawhney with Yaramasu to obtain the invention as specified in claim 7.
(5) regarding claim 8:
Sudharsan disclosed most of the subject matter as described as above except for specifically teaching wherein the starting point is at one end of at least one electrical wire of the electrical wiring system.
However, Yaramasu teaches wherein the starting point is at one end of at least one electrical wire of the electrical wiring system (page 9, B. Results, para. [0001], note that the distance D1 to the fault is 19.8 m under fault condition 1, which is close to load, and D1 is 4.8 m under fault condition 2, which is near the power bus).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art to teach wherein the information includes a measurement of a distance from a starting point in the electrical wiring system to the condition. The suggestion/motivation for doing so would have been in order to detect and locate fault in an aircraft system (Abs.). Therefore, it would have been obvious to combine Sudharsan and Sawhney with Yaramasu to obtain the invention as specified in claim 8.
(6) regarding claim 9:
Sudharsan disclosed most of the subject matter as described as above except for specifically teaching wherein the distance is determined by a fault locator, and wherein the fault locator comprises a time domain reflectometer.
However, Yaramasu teaches wherein the distance is determined by a fault locator, and wherein the fault locator comprises a time domain reflectometer (page 2, I. Introduction, para. [0005], note that Sequence time domain reflectometry (STDR) and spread spectrum time domain reflectometry (SSTDR) have been used to detect and locate intermittent arc faults).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art to teach wherein the distance is determined by a fault locator, and wherein the fault locator comprises a time domain reflectometer. The suggestion/motivation for doing so would have been in order to detect and locate fault in an aircraft system (Abs.). Therefore, it would have been obvious to combine Sudharsan and Sawhney with Yaramasu to obtain the invention as specified in claim 9.
The proposed rejection of claim 4, renders obvious the steps of the method (fig. 1) claim 18 because these steps occur in the operation of the proposed rejection as discussed above. Thus, the arguments similar to that presented above for claim 4 is equally applicable to claim 18.
Allowable Subject Matter
Claims 10-13 and 19 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. The following is a statement of reasons for the indication of allowable subject matter: the prior arts made of record do not teach “wherein the processor is further configured to determine a virtual path along a length of at least one electrical wire of the electrical wiring system, and wherein the virtual path starts from the starting point and extends along the length of the at least one electrical wire until the distance is reached”, as claimed in claims 10 and 19. Claims 11-13 depend on claim 10.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
De Crescenzio et al. (NPL, “Augmented Reality for Aircraft Maintenance Training and Operations Support”, 2011) disclosed a prototype system that incorporates four main requirements for efficiency. It’s user-centered, it implements markerless camera pose estimation, it pro vides an efficient authoring procedure, and it keeps interaction simple and easy.
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
11. Any inquiry concerning this communication or earlier communication from the examiner should be directed to Hilina K Demeter whose telephone number is (571) 270-1676.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Y. Poon could be reached at (571) 270- 0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about PAIR system, see http://pari-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HILINA K DEMETER/Primary Examiner, Art Unit 2617
Prosecution Insights