DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Ren et al. (US Pub. 2025/0157170), hereinafter Ren, in view of Kim et al. (US Pub. 2022/0182537), hereinafter Kim.
Regarding claim 1, Ren discloses a method comprising: obtaining an image frame using at least one see-through camera of a video see-through (VST) extended reality (XR) device (Fig. 1; Paragraph [0034]: shown in FIG. 1, metadata harmonizing image stitching system 100 may comprise a parameter harmonization function 120 that generates stitched image data 152 (e.g., stitched images) of a three-dimensional (3D) environment (e.g., around an ego-object, such as a vehicle) based on image data 110 captured by one or more image sensors 105 and image metadata parameters 112 associated with the image data 110. Image data 110 may include image frames for a plurality of images. Image sensor(s) 105 may include, for example, RGB, infrared (IR) and/or RGB-IR cameras, and/or other cameras, such as cameras described with respect to the vehicle 700 of FIGS. 7A-7D. Image data 110 is not limited to any particular color space. For example, image data 110 may include images represented using a color space such as, but not limited to YUV, RGB, CIELAB (L*a*b*) or other color space; Paragraph [0065]: the systems and methods described herein may be used for a variety of purposes, by way of example and without limitation, for machine control, machine locomotion, machine driving, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security and surveillance, simulation and digital twinning, autonomous or semi-autonomous machine applications, deep learning, environment simulation, object or actor simulation and/or digital twinning, data center processing, conversational AI, light transport simulation (e.g., ray-tracing, path tracing, etc.), collaborative content creation for 3D assets, cloud computing, generative AI, and/or any other suitable applications; Paragraph [0166]: vehicle 700 may further include the infotainment SoC 730 (e.g., an in-vehicle infotainment system (IVI)). Although illustrated and described as a SoC, the infotainment system may not be a SoC, and may include two or more discrete components. The infotainment SoC 730 may include a combination of hardware and software that may be used to provide audio (e.g., music, a personal digital assistant, navigational instructions, news, radio, etc.), video (e.g., TV, movies, streaming, etc.), phone (e.g., hands-free calling), network connectivity (e.g., LTE, Wi-Fi, etc.), and/or information services (e.g., navigation systems, rear-parking assistance, a radio data system, vehicle related information such as fuel level, total distance covered, brake fuel level, oil level, door open/close, air filter information, etc.) to the vehicle 700. For example, the infotainment SoC 730 may radios, disk players, navigation systems, video players, USB and Bluetooth connectivity, carputers, in-car entertainment, Wi-Fi, steering wheel audio controls, hands free voice control, a heads-up display (HUD), an HMI display 734); applying a second correction to the image frame based on one or more intrinsic parameters of at least one display lens of the VST XR device (Fig. 6; Paragraph [0058]: Method 600, at block B602, includes determining a first metadata parameter value associated with rendering a first image and a second metadata parameter value associated with rendering a second image, wherein the first image at least partially overlaps at an image border with the second image. The first metadata parameter value and the second metadata parameter value may be associated with a rendering parameter such as, but not limited to, white balance, tone mapping, exposure time, brightness, contrast, gamma, hue, noise reduction, saturation, sharpness, color filter array (CFA) pattern, lens shading correction, lens distortion correction, focal length correction, barrel distortion correction, or pincushion distortion. In some embodiments, the first metadata parameter value and/or the second metadata parameter may be based on metadata communicated with the first image); and after applying the first correction and the second correction, displaying the image frame on at least one display visible through the at least one display lens (Fig. 1; Paragraph [0038]: presentation module 160 may cause presentation of a visualization 165 of at least a portion of the stitched image data 152 (e.g., on a monitor visible to an occupant or operator of the ego-object or ego-actor). In some embodiments, the presentation module 160 projects the stitched image data 152, or a portion thereof, onto a 3D representation of the 3D environment (e.g., a 3D bowl that models the 3D environment), renders a view of the projected stitched image data 152 from the perspective of a virtual camera, and/or causes presentation of the rendered view as the visualization 165); and wherein the one or more intrinsic parameters of the at least one display lens are determined by a second machine learning model (Paragraph [0122]: processor(s) 710 may include a video image compositor that may be a processing block (e.g., implemented on a microprocessor) that implements video post-processing functions needed by a video playback application to produce the final image for the player window. The video image compositor may perform lens distortion correction on wide-view camera(s) 770, surround camera(s) 774, and/or on in-cabin monitoring camera sensors. In-cabin monitoring camera sensor is preferably monitored by a neural network running on another instance of the Advanced SoC, configured to identify in cabin events and respond accordingly. An in-cabin system may perform lip reading to activate cellular service and place a phone call, dictate emails, change the vehicle's destination, activate or change the vehicle's infotainment system and settings, or provide voice-activated web surfing. Certain functions are available to the driver only when the vehicle is operating in an autonomous mode, and are disabled otherwise).
Ren does not explicitly disclose applying a first correction to the image frame based on one or more intrinsic parameters of the at least one see-through camera; wherein the one or more intrinsic parameters of the at least one see-through camera are determined using a first machine learning model.
However, Kim teaches image processing including distortion correction (Paragraph [0035]; Paragraph [0064]), further comprising applying a first correction to the image frame based on one or more intrinsic parameters of the at least one see-through camera (Fig. 3; Paragraph [0030]: IP network module 22 according to some example embodiments is a neural network model trained to perform at least one of the image processing operations commonly performed on the image sensor 11 of the camera module 10. Here, the image processing operations may include various operations such as a bad pixel correction (BPC) operation, a lens shading correction (LSC) operation, a crosstalk correction operation, a white balance (WB) correction operation, a remosaic operation, a demosaic operation, a denoise operation, a deblur operation, a gamma correction operation, a high dynamic range (HDR) operation, and/or a tone mapping operation. In some example embodiments, the kinds of the image processing operations are not limited to the above-described example. The image processing operation performed by the IP network module 22 will be described in detail later with reference to FIG. 3; Paragraphs [0064]-[0065]: the IP network module 210a may be a neural network model trained to perform the above-described denoise and deblur operations and to remove the distortion of the image data generated by, in particular, a UDC module. Therefore, although the first tetra data IDTa is generated by the camera module 1100, that is, the UDC module disposed under the display, the IP network module 210a may generate the third tetra data IDTc by removing the distortion caused by the UDC module in the second tetra data IDTb…main processor 300a may receive the third tetra data IDTc from the neural network processor 200a and may generate RGB data IDTd by performing the remosaic operation and the demosaic operation on the third tetra data IDTc. Specifically, the remosaic module 310a may convert the third tetra data IDTc into Bayer data and the demosaic module 320a may convert the Bayer data into RGB data CDT including red, blue, and green channels. In some example embodiments, the inventive concepts are not limited thereto and, according to some example embodiments, the third tetra data IDTc may be converted into YUV data); wherein the one or more intrinsic parameters of the at least one see-through camera are determined using a first machine learning model (Paragraph [0081]: image processing system 1200 or a portion thereof (e.g., the neural network processor 200) may perform the image processing operation on the second image data based on using a neural network model learned (e.g., trained) to perform one or more particular (or, alternatively, predetermined) image processing operations. Specifically, the image processing system 1200 may generate the third image data by performing the image processing operation on the second image data by using the IP network module 220 trained to perform at least one of the remosaic operation, the demosaic operation, the denoise operation, the deblur operation, the HDR operation, or the tone mapping operation. In some example embodiments, the IP network module 220 may be a neural network model trained to perform the above-described image processing operations and to remove the distortion of the image data generated by a UDC module). Kim teaches that this will allow for removal of distortion in image data (Paragraphs [0035]; Paragraphs [0063]-[0065]). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Ren with the features of above as taught by Kim so as to allow for removal of distortion in image data as presented by Kim.
Regarding claim 2, Ren, in view of Kim teaches the method of claim 1, Ren discloses wherein the first correction and the second correction are applied in conjunction with passing the image frame through a processing pipeline for generating an XR display based on the image frame (Paragraphs [0121]-[0122]: processor(s) 710 may further include a high-dynamic range signal processor that may include an image signal processor that is a hardware engine that is part of the camera processing pipeline…processor(s) 710 may include a video image compositor that may be a processing block (e.g., implemented on a microprocessor) that implements video post-processing functions needed by a video playback application to produce the final image for the player window. The video image compositor may perform lens distortion correction on wide-view camera(s) 770, surround camera(s) 774, and/or on in-cabin monitoring camera sensors. In-cabin monitoring camera sensor is preferably monitored by a neural network running on another instance of the Advanced SoC, configured to identify in cabin events and respond accordingly. An in-cabin system may perform lip reading to activate cellular service and place a phone call, dictate emails, change the vehicle's destination, activate or change the vehicle's infotainment system and settings, or provide voice-activated web surfing. Certain functions are available to the driver only when the vehicle is operating in an autonomous mode, and are disabled otherwise).
Regarding claim 3, Ren, in view of Kim teaches the method of claim 2, Ren discloses wherein the first and second corrections are applied prior to or simultaneously with performing a correction for a predicted head pose of a user of the VST XR device (Fig. 6; Paragraph [0077]: one or more of the camera(s) may be used to perform advanced driver assistance systems (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a Multi-Function Mono Camera may be installed to provide functions including lane departure warning, traffic sign assist and intelligent headlamp control. One or more of the camera(s) (e.g., all of the cameras) may record and provide image data (e.g., video) simultaneously.; Paragraph [0084]: Cameras with a field of view that include portions of the interior environment within the cabin of the vehicle 700 (e.g., one or more OMS sensor(s) 701) may be used as part of an occupant monitoring system (OMS) such as, but not limited to, a driver monitoring system (DMS). For example, OMS sensors (e.g., the OMS sensor(s) 701) may be used (e.g., by the controller(s) 736) to track an occupant's and/or driver's gaze direction, head pose, and/or blinking. This gaze information may be used to determine a level of attentiveness of the occupant or driver (e.g., to detect drowsiness, fatigue, and/or distraction), and/or to take responsive action to prevent harm to the occupant or operator. In some embodiments, data from OMS sensors may be used to enable gaze-controlled operations triggered by driver and/or non-driver occupants such as, but not limited to, adjusting cabin temperature and/or airflow, opening and closing windows, controlling cabin lighting, controlling entertainment systems, adjusting mirrors, adjusting seat positions, and/or other operations).
Regarding claim 4, Ren, in view of Kim teaches the method of claim 1, Ren discloses wherein: the image frame comprises image data in each of a plurality of color channels (Paragraph [0021]: attempt to address issues that arise when a set of different cameras use different rendering parameters as they capture images, some image stitching technologies select one camera from the set of cameras as the reference camera and the image from the reference camera as the reference image. With such stitching technologies, often referred to as global color transfer techniques, global color statistics (e.g., color channel mean and/or standard deviation) of images from each of the other cameras of the set are transformed to match the global color statistics of the reference image. However, there are several drawbacks to global color transfer-based techniques. For example, color transfer-based techniques are sensitive to the decision of which camera from the set of cameras is selected as the reference camera. That is, depending on the selection of reference cameras, the resulting stitched images may differ with respect to their general appearances. For example, if a front-view camera of an automobile is selected as the reference, it may capture a brighter scene than a rear-view camera. Global color transfer may be applied between left-view and right-view cameras that produce images that share an overlapping region with the front camera. However, the rear-view camera may not share an overlapping view with the front camera, and therefore either the left- or right-view camera images would be selected as reference for performing global color transfer for the rear-view camera); and the second correction is applied separately for each of the color channels (Paragraphs [0029]-[0036]: in addition to white balance and GTM, other metadata parameters that may be harmonized using this process include, but are not limited to, parameters for exposure, lens shading correction, color correction, noise reduction, sharpening, color filter array (CFA) pattern, focal length correction, lens distortion correction (e.g., barrel distortion correction and pincushion distortion) and/or other rendering parameters that affect how an image is rendered. In some embodiments, a series of harmonizations may be applied to a set of images. While the order in which harmonizations are performed may affect the final overall appearance of the stitched image, the selection of which metadata parameters to harmonize and in what order may be selected based on the specific use case for which the stitched image is being generated…metadata parameters can be communicated by the camera to the ISP parameter harmonization function through a channel separate from channels carrying the image data (e.g., pixel color data). In some instances, one or more metadata parameters may represent static characteristics or settings associated with a camera that may not change with each image captured (e.g., a lens distortion correction and/or sensor or pixel size). Such metadata parameters may be stored to a memory and recalled as needed by the ISP parameter harmonization function rather than communicated by the camera to the ISP parameter harmonization function with each captured image… one or more of image metadata parameters 112 can be communicated by the image sensor(s) 105 to the parameter harmonization function 120 through a channel separate from channels carrying the image data 110. In some instances, one or more metadata parameters may represent static characteristics or settings associated with the image sensor(s) 105 that may not change with each image captured (e.g., a lens distortion correction and/or sensor or pixel size). Such metadata parameters may be stored to a memory 114 and recalled as needed by the parameter harmonization function 120 rather than, or in addition to, being communicated by the image sensor(s) 105 to the parameter harmonization function 120 with the image data 110).
Regarding claim 5, Ren, in view of Kim teaches the method of claim 1, Kim discloses further comprising: determining whether to calibrate the VST XR device for at least one of: distortion in the at least one see-through camera or distortion from the at least one display lens (Fig. 3; Paragraph [0030]: IP network module 22 according to some example embodiments is a neural network model trained to perform at least one of the image processing operations commonly performed on the image sensor 11 of the camera module 10. Here, the image processing operations may include various operations such as a bad pixel correction (BPC) operation, a lens shading correction (LSC) operation, a crosstalk correction operation, a white balance (WB) correction operation, a remosaic operation, a demosaic operation, a denoise operation, a deblur operation, a gamma correction operation, a high dynamic range (HDR) operation, and/or a tone mapping operation. In some example embodiments, the kinds of the image processing operations are not limited to the above-described example. The image processing operation performed by the IP network module 22 will be described in detail later with reference to FIG. 3; Paragraphs [0064]-[0065]: the IP network module 210a may be a neural network model trained to perform the above-described denoise and deblur operations and to remove the distortion of the image data generated by, in particular, a UDC module. Therefore, although the first tetra data IDTa is generated by the camera module 1100, that is, the UDC module disposed under the display, the IP network module 210a may generate the third tetra data IDTc by removing the distortion caused by the UDC module in the second tetra data IDTb…main processor 300a may receive the third tetra data IDTc from the neural network processor 200a and may generate RGB data IDTd by performing the remosaic operation and the demosaic operation on the third tetra data IDTc. Specifically, the remosaic module 310a may convert the third tetra data IDTc into Bayer data and the demosaic module 320a may convert the Bayer data into RGB data CDT including red, blue, and green channels. In some example embodiments, the inventive concepts are not limited thereto and, according to some example embodiments, the third tetra data IDTc may be converted into YUV data); responsive to determining to calibrate the VST XR device, providing the image frame to one or more of the first and second machine learning models (Paragraph [0025]: the neural network model is not limited thereto, and may, for example include other n-layered neural networks like a deep belief network, restricted Boltzmann machine, a deep learning system, deconvolutional neural networks (DCNN), stacked neural networks (SNN), deep belief networks (DBN), generative adversarial networks (GANs), restricted Boltzmann machines (RBM), and/or the like; Paragraph [0081]: image processing system 1200 or a portion thereof (e.g., the neural network processor 200) may perform the image processing operation on the second image data based on using a neural network model learned (e.g., trained) to perform one or more particular (or, alternatively, predetermined) image processing operations. Specifically, the image processing system 1200 may generate the third image data by performing the image processing operation on the second image data by using the IP network module 220 trained to perform at least one of the remosaic operation, the demosaic operation, the denoise operation, the deblur operation, the HDR operation, or the tone mapping operation. In some example embodiments, the IP network module 220 may be a neural network model trained to perform the above-described image processing operations and to remove the distortion of the image data generated by a UDC module); and receiving, from one or more of the first and second machine learning models, at least one of: one or more updated intrinsic parameters of the one or more see-through cameras or one or more updated intrinsic parameters of the at least one display lens (Paragraph [0089]: the IP network module 22 may be learned (e.g., a neural network model used by the IP network module 22 may be trained) based on using the image (e.g., raw image) generated by the UDC module as the input data and by using the corrected image obtained by removing the distortion in the input data as the output data. The neural network model may be trained based on using an image generated by the UDC module (said image being referred to herein interchangeably as first trained data) as input data and using a corrected image obtained based on performing one or more particular (or, alternatively, predetermined) image processing operations on the “first trained data” as output data (said corrected image being referred to herein interchangeably as second trained data). The raw image generated by the UDC module may include distortions such as blur, ghost, haze, and flare. Therefore, the IP network module 22 may be trained to perform the remosaic operation, the demosaic operation, the denoise operation, and the deblur operation while processing the distortions included in the raw image).
Regarding claim 6, Ren, in view of Kim teaches the method of claim 5, Kim discloses wherein at least one of the first and second machine learning models is remote from the VST XR device (Fig. 3; Paragraphs [0047]-[0049]: image sensor 1110 of the camera module 1100 may include a color filter array (CFA) having a particular (or, alternatively, predetermined) pattern, may convert the optical signal of the subject incident through the optical lens LS into an electrical signal by using the CFA, may generate first image data IDTa based on electrical signals, and may output the generated first image data IDTa… the pre-processing operation may include the demosaic operation, the denoise operation, and the deblur operation. The pre-processor 100 may transmit second image data IDTb generated by performing the pre-processing operation to the neural network processor 200. The pre-processor 100 may transmit second image data IDTb generated by performing the pre-processing operation to both the neural network processor 200 and the main processor 300).
Regarding claim 7, Ren, in view of Kim teaches the method of claim 1, Ren discloses wherein the one or more intrinsic parameters of the at least one display lens comprise at least one of: barrel distortion or chromatic aberration (Fig. 6; Paragraph [0058]: Method 600, at block B602, includes determining a first metadata parameter value associated with rendering a first image and a second metadata parameter value associated with rendering a second image, wherein the first image at least partially overlaps at an image border with the second image. The first metadata parameter value and the second metadata parameter value may be associated with a rendering parameter such as, but not limited to, white balance, tone mapping, exposure time, brightness, contrast, gamma, hue, noise reduction, saturation, sharpness, color filter array (CFA) pattern, lens shading correction, lens distortion correction, focal length correction, barrel distortion correction, or pincushion distortion. In some embodiments, the first metadata parameter value and/or the second metadata parameter may be based on metadata communicated with the first image).
Regarding claim 8, the limitations of this claim substantially correspond to the limitations of claim 1 (except for the processor, which is disclosed by Ren, Paragraph [0056]: Each block of method 600, described herein, comprises a computing process that may be performed using any combination of hardware, firmware, and/or software. For instance, various functions may be carried out by a processor executing instructions stored in memory. The methods may also be embodied as computer-usable instructions stored on computer storage media); thus they are rejected on similar grounds.
Regarding claim 9, the limitations of this claim substantially correspond to the limitations of claim 2; thus they are rejected on similar grounds.
Regarding claim 10, the limitations of this claim substantially correspond to the limitations of claim 3; thus they are rejected on similar grounds.
Regarding claim 11, the limitations of this claim substantially correspond to the limitations of claim 4; thus they are rejected on similar grounds.
Regarding claim 12, the limitations of this claim substantially correspond to the limitations of claim 5; thus they are rejected on similar grounds.
Regarding claim 13, the limitations of this claim substantially correspond to the limitations of claim 6; thus they are rejected on similar grounds.
Regarding claim 14, the limitations of this claim substantially correspond to the limitations of claim 7; thus they are rejected on similar grounds.
Regarding claim 15, the limitations of this claim substantially correspond to the limitations of claim 1 (except for the machine-readable medium, which is disclosed by Ren, Paragraph [0056]: Each block of method 600, described herein, comprises a computing process that may be performed using any combination of hardware, firmware, and/or software. For instance, various functions may be carried out by a processor executing instructions stored in memory. The methods may also be embodied as computer-usable instructions stored on computer storage media); thus they are rejected on similar grounds.
Regarding claim 16, the limitations of this claim substantially correspond to the limitations of claim 2; thus they are rejected on similar grounds.
Regarding claim 17, the limitations of this claim substantially correspond to the limitations of claim 4; thus they are rejected on similar grounds.
Regarding claim 18, the limitations of this claim substantially correspond to the limitations of claim 5; thus they are rejected on similar grounds.
Regarding claim 19, the limitations of this claim substantially correspond to the limitations of claim 6; thus they are rejected on similar grounds.
Regarding claim 20, the limitations of this claim substantially correspond to the limitations of claim 7; thus they are rejected on similar grounds.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Sen (US Pub. 2025/0042416) teaches lens distortion correction.
Kaji (US Pub. 2023/0319407) teaches lens distortion correction.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MATTHEW D SALVUCCI whose telephone number is (571)270-5748. The examiner can normally be reached M-F: 7:30-4:00PT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, XIAO WU can be reached at (571) 272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MATTHEW SALVUCCI/Primary Examiner, Art Unit 2613
Prosecution Insights