CTNF 19/025,567 CTNF 66123 Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. 07-06 AIA 15-10-15 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. 07-20-02-aia AIA This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention. 07-21-aia AIA Claim s 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over KHAN et al (US 11,301,969) in view of HU et al (20220026987) . As per claim 1, Khan teaches the claimed “wearable electronic device comprising: a camera; a display; at least one sensor; memory storing instructions; and at least one processor operatively connected to the camera, the display, the at least one sensor, and the memory, and configured to execute the instructions” ( Khan, Figure 7 – Head-Mounted Display (MMD) 120 ), wherein the instructions, when executed by the at least one processor, cause the wearable electronic device to: obtain a first image comprising at least one object via the camera; obtain a second image by performing pre-processing that calibrates a distortion area of the first image that is generated based on the camera, based on information related to a type of the camera” ( Khan, column 13, lines 10-26 – In some implementations, correcting the distortion involves applying different distortion correction techniques to different images , e.g., a first distortion correction technique is applied for the first image, a second (different) distortion correction technique is applied for the second image, etc.; column 14, lines 42-60 - Thus the same system in different connection modes can apply different distortion correction schemes ) ( Noted : Khan’s distortion correction computations can be performed on different stages in which its initial state (i.e., the distortion Correction Unit 246 of the controller 110) is equivalent to the claimed “pre-processing that calibrates a distortion area”); “input, into a deep learning model stored in the memory, a default matrix related to a first distance of a first object in a third image, and at least one of: first information input by a user, second information obtained via the at least one sensor in relation to the user, or third information obtained via the at least one sensor in relation to a surrounding environment” ( Khan, column 8, lines 3-55 - the one or more communication buses 304 include circuitry that interconnects and controls communications between system components. In some implementations, the one or more I/O devices and sensors 306 include at least one of an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., blood pressure monitor, heart rate monitor, blood oxygen sensor, blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptics engine, one or more depth sensors (e.g., a structured light, a time-of-flight, or the like), and/or the like. In some implementations, one or more of these I/O devices and sensors 306, such as the IMU, is used to track context. One example application, turns on dynamic distortion correction during high acceleration periods identified by an IMU, because such acceleration may indicate that the device might move relative to the user's face/eyes; column 9, lines 31-51 - In some implementations, the distortion correction unit 346 is configured to use eye characteristics and/or context estimates and predictions to determine distortion corrections for a user experience occurring on the HMD 120. To that end, in various implementations, the distortion correction unit 346 includes instructions and/or logic therefor, and heuristics and metadata therefor ) ( Noted: Khan’s user’s physiological information and environment data suggests the inputted default distance table which contains the default matrix related to fixed conditions (e.g., Khan’s distance of a fixed eye center position) which can be modified to generate an adjusted matrix based on user information (e.g., user’s change in pupil or eye center position) or surrounding environment… Furthermore, Khan’s Distortion Correction Unit 346 can be implemented by a neural network which processes data based on the matrix multiplication of visual parameters ( Hu, [0211] - In at least one embodiment, DLA may be used to run any type of network to enhance control and driving safety, including for example and without limitation, a neural network that outputs a measure of confidence for each object detection; [0460] - In at least one embodiment, additional fixed function logic 3716 can also include machine-learning acceleration logic, such as fixed function matrix multiplication logic , for implementations including optimizations for machine learning training or inferencing; [0217] - In at least one embodiment, processor(s) 1910 may include a video image compositor that may be a processing block (e.g., implemented on a microprocessor) that implements video post-processing functions needed by a video playback application to produce final image for player window. In at least one embodiment, video image compositor may perform lens distortion correction on wide-view camera(s) 1970, surround camera(s) 1974, and/or on in-cabin monitoring camera sensor(s); [0438] - In at least one embodiment, display device 3511 can include a head mounted display (HMD) such as a stereoscopic display device for use in virtual reality (VR) applications or augmented reality (AR) applications ); “obtain, from the deep learning model, a matrix for adjusting a second distance of the at least one object in the second image; obtain a fourth image by adjusting the second distance of the at least one object in the second image based on the matrix; and display, on the display, the fourth image” ( Khan, column 12, line 11 to column 13, line 37 - As a specific example, a first correction technique applied to a subset of the images may calculate a pupil position or an eye center and use that calculated pupil position/eye center to determine the correction , while a second correction technique applied to a different subset of the images uses a fixed (e.g., default) pupil position/eye center and uses that fixed pupil position/eye center to compute the correction… At block 520, the method 500 determines distortion for images to be included in the user experience. The distortion is determined based on a relationship, e.g., distance , relative sizes, relative optical characteristics, etc., between the lens and a display of the HMD. In some implementations, the distortion is based on default eye characteristics (e.g., pupil position, eye center location, gaze direction aligned with optical axis of lens, etc.). At block 530, the method 500 determines corrected distortion by correcting the distortion for the images based on the context during the user experience ). Thus, it would have been obvious, in view of Hu, to configure Khan’s device as claimed by inputting the user’s physiological data and/or environment information to a neural network to implement a circuit for correcting the image distortion. The motivation is to enhance the visual representation of camera-captured image. Claim 2 adds into claim 1 “wherein the instructions, when executed by the at least one processor, cause the wearable electronic device to pre-process the first image by calibrating the distortion area based on a type of a lens included in the camera” ( Khan, column 1, lines 32-35 - The amount of distortion in the images can vary with the strength of the lens , e.g., smaller displays may use more powerful lenses and thus require greater distortion; column 5, lines 4-7 - Context can include, but is not limited to, movement of a device or part of a device, e.g., a display, use of a tunable lens , or other device characteristics and/or states that may have an impact on distortion; column 12, lines 28-35 - At block 520, the method 500 determines distortion for images to be included in the user experience. The distortion is determined based on a relationship, e.g., distance, relative sizes, relative optical characteristics, etc., between the lens and a display of the HMD ). Claim 3 adds into claim 1 “wherein the deep learning model comprises a plurality of sub-deep learning models respectively corresponding to a plurality of pieces of information, and wherein the deep learning model is configured to: identify, from among the plurality of sub-deep learning models, at least one sub-deep learning model respectively corresponding to at least one type of information input into the deep learning model from among the first information, the second information, and the third information; and obtain the matrix by sequentially using the at least one sub-deep learning model” ( Hu, [0112] - A deep neural network (DNN) model includes multiple layers of many connected perceptrons (e.g., nodes) that can be trained with enormous amounts of input data to quickly solve complex problems with hid accuracy. In one example, a first layer of a DNN model breaks down an input image of an automobile into various sections and looks for basic patterns such as lines and angles. Second layer assembles lines to look for higher-level patterns such as wheels, windshields, and mirrors. A next layer identifies a type of vehicle, and a final few layers generate a label for an input image, identifying a model of a specific automobile brand. Once a DNN is trained, this DNN can be deployed and used to identify and classify objects or patterns in a process known as inference; [0514] - In at least one embodiment, tensor cores are configured to perform deep learning matrix arithmetic , such as convolution operations for neural network training and inferencing ). Thus, it would have been obvious, in view of Hu, to configure Khan’s device as claimed by implementing the deep learning model comprising a plurality of sub-deep learning models for different types of information. The motivation is to enhance the visual representation under different data conditions of camera-captured image. Claim 4 adds into claim 3 “wherein the deep learning model comprises a first sub-deep learning model corresponding to the first information, a second sub- deep learning model corresponding to the second information, and a third sub-deep learning model corresponding to the third information, wherein, based on the first information being input into the deep learning model, first output data of the first sub-deep learning model is used as input data of the second sub-deep learning model or the third sub-deep learning model, and wherein, based on the second information being input into the deep learning model, second output data of the second sub-deep learning model is used as input data of the third sub- deep learning model” ( Hu, [0112] - A deep neural network (DNN) model includes multiple layers of many connected perceptrons (e.g., nodes) that can be trained with enormous amounts of input data to quickly solve complex problems with hid accuracy... Once a DNN is trained, this DNN can be deployed and used to identify and classify objects or patterns in a process known as inference ). Thus, it would have been obvious, in view of Hu, to configure Khan’s device as claimed by implementing the deep learning model comprising a plurality of sub-deep learning models for different types of information. The motivation is to enhance the visual representation under different data conditions of camera-captured image. Claim 5 adds into claim 3 “wherein the first information is input by the user and comprises at least one of age information, gender information, height information, eyesight information, or body mass index (BMI) information, and wherein the deep learning model comprises at least one first sub-deep learning model respectively corresponding to at least one of the age information, the gender information, the height information, the eyesight information, or the BMI information” ( Khan, column 8, lines 3-55 - the one or more communication buses 304 include circuitry that interconnects and controls communications between system components. In some implementations, the one or more I/O devices and sensors 306 include at least one of an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., blood pressure monitor, heart rate monitor, blood oxygen sensor, blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptics engine, one or more depth sensors (e.g., a structured light, a time-of-flight, or the like), and/or the like... One example application, turns on dynamic distortion correction during high acceleration periods identified by an IMU, because such acceleration may indicate that the device might move relative to the user's face/eyes ) ( Noted: Khan’s user’s physiological sensors anthropometric and sensory-neurological biometrics, used to measure physical attributes and biological function, suggests the claimed “height information, eyesight information, or body mass index (BMI) information, …”). Thus, it would have been obvious to configure Khan’s physiological data of the user to include “height information, eyesight information, or body mass index (BMI) information, …” as claimed in the correction process of camera-captured distorted image. The motivation is to enhance the visual representation under different data conditions of camera-captured image. Claim 6 adds into claim 3 “wherein the second information comprises at least one of interpupillary distance information or pupil color information obtained by the at least one sensor in relation to the user, and wherein the deep learning model comprises at least one second sub-deep learning model respectively corresponding to at least one of the interpupillary distance information or the pupil color information” ( Khan, column 1, lines 57-61 - For example, a distortion may be initially determined based on the characteristics of the display, the lens, and the relationship between them and a general or default circumstance such as the user's pupil position or gaze direction aligning with the optical axis of the lens; column 2, lines 3-6 - For example, for a given image, the distortion can be determined by obtaining a default distortion and correcting the distortion based on the user's pupil position at the time when the image is to be displayed ). Claim 7 adds into claim 3 “wherein the third information comprises at least one of brightness information, Global Positioning System (GPS)information, horizontality information, or inertia information obtained by the at least one sensor in relation to the surrounding environment, and wherein the deep learning model comprises at least one third sub-deep learning model respectively corresponding to at least one of the brightness information, the GPS information, the horizontality information, or the inertia information” ( Khan, column 6, lines 16-32 - To that end, as a non-limiting example, in some implementations the controller 110 includes one or more processing units 202 (e.g., microprocessors, …, one or more input/output (I/O) devices 206, one or more communication interfaces 208 (e.g., universal serial bus (USB), …, global positioning system (GPS), infrared (IR), BLUETOOTH, ZIGBEE, and/or the like type interface) , one or more programming (e.g., I/O) interfaces 210, a memory 220, and one or more communication buses 204 for interconnecting these and various other components; column 8, lines 3-55 - the one or more communication buses 304 include circuitry that interconnects and controls communications between system components. In some implementations, the one or more I/O devices and sensors 306 include at least one of an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope , a thermometer, one or more physiological sensors (e.g., blood pressure monitor, heart rate monitor, blood oxygen sensor, blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptics engine, one or more depth sensors (e.g., a structured light, a time-of-flight, or the like), and/or the like. In some implementations, one or more of these I/O devices and sensors 306, such as the IMU, is used to track context ). Claim 8 adds into claim 1 “wherein the deep learning model is trained by further using a default distance table related to a fourth distance of at least one object in a fifth image, and wherein the instructions, when executed by the at least one processor, cause the wearable electronic device to obtain the matrix by further inputting the default distance table into the deep learning model” ( Khan, column 12, line 11 to column 13, line 37 - As a specific example, a first correction technique applied to a subset of the images may calculate a pupil position or an eye center and use that calculated pupil position/eye center to determine the correction, while a second correction technique applied to a different subset of the images uses a fixed (e.g., default) pupil position/eye center and uses that fixed pupil position/eye center to compute the correction ) ( Noted: Khan’s use of fixed (e.g., default) pupil position/eye center to compute the correction suggests fixed distance table inputted to the neural network to compute the correction). Furthermore, Khan’s Distortion Correction Unit 346 can be implemented by a neural network which processes data based on the matrix multiplication of visual parameters ( Hu, [0211] - In at least one embodiment, DLA may be used to run any type of network to enhance control and driving safety, including for example and without limitation, a neural network that outputs a measure of confidence for each object detection; [0460] - In at least one embodiment, additional fixed function logic 3716 can also include machine-learning acceleration logic, such as fixed function matrix multiplication logic , for implementations including optimizations for machine learning training or inferencing; [0217] - In at least one embodiment, processor(s) 1910 may include a video image compositor that may be a processing block (e.g., implemented on a microprocessor) that implements video post-processing functions needed by a video playback application to produce final image for player window. In at least one embodiment, video image compositor may perform lens distortion correction on wide-view camera(s) 1970, surround camera(s) 1974, and/or on in-cabin monitoring camera sensor(s); [0438] - In at least one embodiment, display device 3511 can include a head mounted display (HMD) such as a stereoscopic display device for use in virtual reality (VR) applications or augmented reality (AR) applications ; column 12, line 11 to column 13, line 37 - As a specific example, a first correction technique applied to a subset of the images may calculate a pupil position or an eye center and use that calculated pupil position/eye center to determine the correction , while a second correction technique applied to a different subset of the images uses a fixed (e.g., default) pupil position/eye center and uses that fixed pupil position/eye center to compute the correction… At block 520, the method 500 determines distortion for images to be included in the user experience. The distortion is determined based on a relationship, e.g., distance , relative sizes, relative optical characteristics, etc., between the lens and a display of the HMD. In some implementations, the distortion is based on default eye characteristics (e.g., pupil position, eye center location, gaze direction aligned with optical axis of lens, etc.). At block 530, the method 500 determines corrected distortion by correcting the distortion for the images based on the context during the user experience ). Thus, it would have been obvious, in view of Hu, to configure Khan’s device as claimed by inputting the user’s physiological data and/or environment information to a neural network to implement a circuit for correcting the image distortion. The motivation is to enhance the visual representation of camera-captured image. Claim 9 adds into claim 8 “wherein the default matrix is configured to maintain the default distance table unchanged” which would have been obvious because the default distance table contains the default matrix related to a distance of a fixed eye center position which can be modified to generate an adjusted matrix based on user information (e.g., user’s change in pupil or eye center position) or surrounding environment. information. The motivation is to correct the distortion for enhancing the visual representation of the camera-captured image. Claim 10 adds into claim 1 “wherein the instructions, when executed by the at least one processor, cause the wearable electronic device to adjust the second distance by adjusting a pixel value of the second image based on the matrix” ( Khan, column 8, lines 3-55 - the one or more communication buses 304 include circuitry that interconnects and controls communications between system components. In some implementations, the one or more I/O devices and sensors 306 include at least one of an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., blood pressure monitor, heart rate monitor, blood oxygen sensor, blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptics engine, one or more depth sensors (e.g., a structured light, a time-of-flight, or the like), and/or the like. In some implementations, one or more of these I/O devices and sensors 306, such as the IMU, is used to track context. One example application, turns on dynamic distortion correction during high acceleration periods identified by an IMU, because such acceleration may indicate that the device might move relative to the user's face/eyes; column 9, lines 31-51 - In some implementations, the distortion correction unit 346 is configured to use eye characteristics and/or context estimates and predictions to determine distortion corrections for a user experience occurring on the HMD 120. To that end, in various implementations, the distortion correction unit 346 includes instructions and/or logic therefor, and heuristics and metadata therefor ) ( Noted: Khan’s Distortion Correction Unit 346 can be implemented by a neural network which processes data based on the matrix multiplication of visual parameters ( Hu, [0460] - In at least one embodiment, additional fixed function logic 3716 can also include machine-learning acceleration logic, such as fixed function matrix multiplication logic , for implementations including optimizations for machine learning training or inferencing; [0217] - In at least one embodiment, processor(s) 1910 may include a video image compositor that may be a processing block (e.g., implemented on a microprocessor) that implements video post-processing functions needed by a video playback application to produce final image for player window. In at least one embodiment, video image compositor may perform lens distortion correction on wide-view camera(s) 1970, surround camera(s) 1974, and/or on in-cabin monitoring camera sensor(s); [0438] - In at least one embodiment, display device 3511 can include a head mounted display (HMD) such as a stereoscopic display device for use in virtual reality (VR) applications or augmented reality (AR) applications ). Thus, it would have been obvious, in view of Hu, to configure Khan’s device as claimed by inputting the user’s physiological data and/or environment information to a neural network to implement a circuit for correcting the image distortion. The motivation is to enhance the visual representation of camera-captured image. Claim 20 adds into claim 1 “wherein the deep learning model is trained by inputting: at least one of user information or surrounding environment information, and at least one matrix for adjusting a third distance of a second object, the at least one matrix respectively corresponding to at least one of the user information or the surrounding environment information” ( Hu, [0115] - In at least one embodiment training of a neural network might be instructed by a provider associated with provider environment 1406 ; [0460] - In at least one embodiment, additional fixed function logic 3716 can also include machine-learning acceleration logic, such as fixed function matrix multiplication logic , for implementations including optimizations for machine learning training or inferencing ). Thus, it would have been obvious, in view of Hu, to configure Khan’s device as claimed by inputting the user’s physiological data and/or environment information to a neural network to implement a circuit for correcting the image distortion. The motivation is to enhance the visual representation of camera-captured image. Claims 11-18, and 19 add into claims 1-10 and 20; therefore, they are rejected under a similar rationale. Any inquiry concerning this communication or earlier communications from the examiner should be directed to PHU K NGUYEN whose telephone number is (571)272-7645. The examiner can normally be reached M-F 8-5pm. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel F. Hajnik can be reached at (571) 272-7 642 . The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /PHU K NGUYEN/Primary Examiner, Art Unit 2616 Application/Control Number: 19/025,567 Page 2 Art Unit: 2616 Application/Control Number: 19/025,567 Page 3 Art Unit: 2616 Application/Control Number: 19/025,567 Page 4 Art Unit: 2616
Prosecution Insights