DETAILED ACTION
The action is in response to the original filing on November 27th, 2024, and the Remarks and Amendments filed on July 20th, 2026. Claims 1-18 are pending and have been considered below. Claims 1-16 have been amended. Claims 17 & 18 have been added.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on May 19th, 2025, is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 8-13, & 15 are rejected under 35 U.S.C. 103 as being unpatentable over Chetan Nanda et al. (Pat. Pub. US-20190362478-A1, herein after “Nanda”) in view of Libin Sun et al. (Pat. Pub. US-11688100-B1, herein after “Sun”), Shusil Dangi et al. (Pat. Pub. JP-2023525702-A, herein after “Dangi”), and C-C Tsai et al. (Pat. Pub. CN-116324878-A, herein after “Tsai2”).
In regard to claims 1 & 10, Nanda teaches [a] method for processing dual-stream images “a movie resulting from the two videos has a consistent color cast such that the target video appears to have been filmed under the same evening lighting conditions as the reference video” (Nanda, ¶ [0013]) processing two videos will be read as dual-stream, the method comprising:
processing a “The color consistency application computes a feature vector for the reference video and the target video” (Nanda, ¶ [0015]) where a first video and first processing technique are taught as a reference video and a feature vector. Nanda does not teach that the field of view for the reference or target video are wide or narrow;
storing, “Each feature vector includes global features. Global features include a histogram of a luminance channel and two color channels” (Nanda, ¶ [0015]) where the feature vector includes luminance and color;
processing a current frame of a “… generates an additional feature vector from the intermediate video and the features of the feature vector that are relevant to the reference video” (Nanda, ¶ [0016]) where a second feature vector based on the previous reference video is taught. Additionally, “using such a feature vector in conjunction with machine learning techniques, the color consistency application facilitates a more accurate matching of color cast between videos” (Nanda, ¶ [0015]) where the use of a neural network, or AI is taught and the feature vector with the information is used in combination with the input video to produce a modified image; and
blending the processed “Together, the exposure parameters and the color and tint parameters form a set of parameters that, when applied to the target video, will result in a color scheme that is consistent with the reference video” (Nanda, ¶ [0017]) where the processing of the two videos is matched to produce a reference video and a target video with identical color parameters. Additionally, “displaying an interface comprising the preview image and a plurality of user interface elements that indicate the color parameters and provide adjustable settings to further adjust the parameters” (Nanda, ¶ [0045]) utilizes a display and display device.
Nanda fails to teach processing a wide field-of-view (FOV) image;
processing a current frame of a narrow FOV image stream using a neural network by providing image data of the current frame to the neural network; and
blending the processed wide FOV image stream and the processed narrow FOV image stream to generate a final image.
Sun teaches processing a wide field-of-view (FOV) image “Image 102 represents an input image from a wider FOV image (which may also be referred to herein as an ‘outer’ image or ‘lower quality’ image)” (Sun, ¶ [0009]) where processing wide field of view images is known in the art;
processing a current frame of a narrow FOV image stream using a neural network by providing image data of the current frame to the neural network “image 104 represents an input image from a narrower FOV image (which may also be referred to herein as an ‘inner image or ‘higher quality’ image)” (Sun, ¶ [0009]) where processing narrow field of view images is known in the art, additionally “the first plurality of interpretable variables may be modified to independently change semantically-distinguishable characteristics of the modified second image, such as texture density, color balance, noise pattern, or noise level” (Sun, ¶ [0010]) teaches modifying semantic characteristics and color balance; and
blending the processed wide FOV image stream and the processed narrow FOV image stream to generate a final image “Image 106 represents an exemplary resultant fused wider FOV image, with enhanced texture, detail, and/or noise representation” (Sun, ¶ [0009]) teaches that the wide and narrow image can be blended for purposes of enhancing image quality.
PNG
media_image1.png
522
735
media_image1.png
Greyscale
Sun, Fig. 1, depicts wide FOV image (item 102) being combined with narrow FOV image (item 104) to produce merged image (item 106).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of processing multiple image streams based on color information of one image stream, and applying it to a different image stream taught by Nanda with the method of having a wide and narrow field of view image, and merging them to produce a complete, synchronized image taught by Sun to create a method of image processing that utilizes a wide and narrow field of view to create one image. The suggestion/motivation to do so would have been to maintain the wide field of view at a higher quality.
Nanda in view of Sun fail to comprehensively teach white balance information, color conversion matrix information, gamma or tone mapping information, and semantic information.
Dangi teaches white balance information, color conversion matrix information, gamma or tone mapping information, and semantic information “the ISP may perform demosaic, gain adjustment, white balance adjustment, color balance or color correction, gamma compression, tone mapping or adjustment, denoise or noise filtering, edge enhancement, contrast adjustment, intensity adjustment (darken), among others” (Dangi, Page 3) where the listed information can be stored or derived from an image. Additionally, Dangi utilizes a machine learning system, such as a plurality of neural networks to apply and store this information to a captured image frame (Page 3).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of processing and merging multiple image streams based FOV and color information taught by Nanda in view of Sun with the method of storing various image information aspects taught by Dangi to have a more comprehensive input to the neural network. The suggestion/motivation to do so would have been to output a more accurate image after combination based on white balance, color conversion, tone mapping, and semantic information.
Nanda in view of Sun and Dangi fail to explicitly teach where the previous frame of the processed wide FOV image stream is stored for the narrow FOV frame to be processed.
Tsai2 teaches where the previous frame of the processed wide FOV image stream is stored for the narrow FOV frame to be processed “the image capture device 102 and 104 may include a capture image and/or video data of the image sensor and/or lens. The image capture devices 102 and 104 may capture images and/or video data having different FOV … the image capture device 102 and 104 may include different image sensor with different FOV … such as wide-angle lens, telephoto lens (e.g., short long focus, long focus, etc.), a standard lens, zoom lens and so on. In some examples, the image capture device 102 may include a type of lens, and image capture device 104 may include different types of lens” (Tsai2, Page 9) where different FOV images are captured and stored to be later processed such as by segmentation “The computing component 110 may perform segmentation using an image having different FOV (e.g., foreground-background segmentation)” (Tsai2, Page 10). The first and second FOV may be either wide or narrow and each is used the process the final output image with the processed target. Finally, “image processing system 100 may perform clipping and upsampling 316 on the probability map 312 to infer a probability diagram 318 of the FOV having the image 304 based on the probability map 312 having the FOV of the image 302” (Tsai2, Page 13) where one FOV image is processed based on the other FOV image
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of processing and merging multiple image streams based FOV and color information taught by Nanda in view of Sun and Dangi with the method of the stored image being one given FOV and the processed image being a separate FOV taught by Tsai2 to have a the input be one preferred FOV and the input information be from another FOV. The suggestion/motivation to do so would have been to pull the information from the wide FOV image instead of the narrow FOV image when being added to the neural network.
In regard to claim 10, claim 1 is substantially similar to claim 10, hence the rejection analysis for claim 1 is also applied to claim 10. Nanda in view of Sun, Dangi, and teach the additional limitations of [a] system for processing dual-stream images “Systems and methods are disclosed herein for using machine learning techniques to automatically apply color characteristics of a reference video to a target video” (Nanda, ¶ [0013]), the system comprising:
a first processing module configured to process a wide field-of-view (FOV) image stream using a first processing technique (Nanda, ¶ [0015]) & (Sun, ¶ [0009]);
a memory module configured to store one, from a previous frame of the processed wide FOV image stream (Tsai2, Pages 10 & 13), at least one of white balance information, color conversion matrix information, gamma or tone mapping information, and semantic information (Dangi, Page 3) & “one or more memory devices 504 stores program data 507 that includes one or more datasets and models described herein” (Nanda, ¶ [0064) where memory holds the relevant data. (Nanda, ¶ [0015]);
a second processing module configured to implement a neural network for processing a current frame of a narrow FOV image stream by providing image data of the current frame to the neural network and providing the stored information to the neural network and providing the stored information to the neural network as one or more additional input layers together with the image data of the current frame “The color consistency application provides the additional feature vector a second predictive model. The second predictive model predicts color temperature and tint parameters” (Nanda, ¶ [0016]) where a second model handles the additional feature vector. (Nanda, ¶ [0015]), (Sun, ¶ [0009]), & (Tsai2, Page 9); and
a blending module configured to blend the processed wide FOV image stream and the processed narrow FOV image stream to generate a final image, for display on a display device “Color consistency application 102 combines exposure parameters 135 and color temperature and tint parameters 136 into combined parameters 140” (Nanda, ¶ [0025]) where an application is used to combine the parameters. (Nanda, ¶ [0017]) & (Sun, ¶ [0009]).
In regard to claims 2 & 11, Nanda in view of Sun, Dangi, and Tsai2 teach [t]he method of claim 1, wherein the stored information comprises white balance information and processing the current frame of the “Color consistency application 102 uses predictive model 121 to predict temperature and tint parameter 136, that if applied to target video 104, results in target video 104 having a similar color temperature and tint to reference video 103” (Nanda, ¶ [0021]) where the temperature of the second stream is adjusted to match the reference stream.
Nanda does not teach that the processed stream is a narrow FOV image stream or that the adjustment is based on a wide FOV image stream.
Sun teaches of a narrow FOV image stream “image 104 represents an input image from a narrower FOV image (which may also be referred to herein as an ‘inner image or ‘higher quality’ image)” (Sun, ¶ [0009]) and a wide FOV image stream “Image 106 represents an exemplary resultant fused wider FOV image, with enhanced texture, detail, and/or noise representation” (Sun, ¶ [0009]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of adjusting the color temperature of a second image stream based on a first image stream taught by Nanda with the method of distinct image streams being wide field of view and narrow field of view taught by Sun to have matching color temperatures in the different field of view images. The suggestion/motivation to do so would have been to create a unified image from multiple different fields of view.
Nanda in view of Sun fail to teach the stored information comprises white balance information.
Dangi teaches white balance information “the ISP may perform demosaic, gain adjustment, white balance adjustment, color balance or color correction, gamma compression, tone mapping or adjustment, denoise or noise filtering, edge enhancement, contrast adjustment, intensity adjustment (darken), among others” (Dangi, Page 3) where white balance adjustments can be applied. The motivation to combine Dangi would be the same as claim 1.
In regard to claim 11, claim 2 is substantially similar to claim 11, hence the rejection analysis for claim 2 is also applied to claim 11. Nanda in view of Sun, Dangi, and Tsai2 teach the additional limitations of [t]he system of claim 10, wherein the stored information comprises white balance information and the second processing module is configured to process the current frame of the narrow FOV image stream by adjusting a color temperature of the current frame based on the white balance information to maintain color consistency with the processed wide FOV image stream (Nanda, ¶ [0021]), (Sun, ¶ [0009]), & (Dangi, Page 3).
In regard to claims 3 & 12, Nanda in view of Sun, Dangi, and Tsai2 teach [a] method of claim 1, wherein the stored information comprises color conversion matrix information and processing the current frame of the “The color consistency application receives an identification of a reference video and a target video. The color consistency application computes a feature vector for the reference video and the target video” (Nanda, ¶ [0015]), and “color consistency application 102 converts the target and reference video videos from an R-G-B color space to an L-a-b color space” (Nanda, ¶ [0029]) where the color space may be converted on a current video frame based on a reference video.
Nanda does not teach that the processed stream is a narrow FOV image stream or that the conversion is based on a wide FOV image stream.
Sun teaches of a narrow FOV image stream “image 104 represents an input image from a narrower FOV image (which may also be referred to herein as an ‘inner image or ‘higher quality’ image)” (Sun, ¶ [0009]) and a wide FOV image stream “Image 106 represents an exemplary resultant fused wider FOV image, with enhanced texture, detail, and/or noise representation” (Sun, ¶ [0009]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of converting the color space of a second image stream based on a first image stream taught by Nanda with the method of distinct image streams being wide field of view and narrow field of view taught by Sun to have the ability to switch between color spaces on the different field of view images. The suggestion/motivation to do so would have been to provide easier modifications on different display devices.
In regard to claim 12, claim 3 is substantially similar to claim 12, hence the rejection analysis for claim 3 is also applied to claim 12. Nanda in view of Sun, Dangi, and Tsai2 teach the additional limitations of [t]he system of claim 10, wherein the stored information comprises color conversion matrix information and the second processing module is configured to process the current frame of the narrow FOV image stream by applying a color conversion matrix, based on the color conversion matrix information to convert a color space of the current frame to maintain color consistency with the wide FOV image stream (Nanda, ¶ [0015] & [0029]) & (Sun, ¶ [0009]).
In regard to claims 4 & 13, Nanda in view of Sun, Dangi, and Tsai2 teach [t]he method of claim 1, wherein the stored information comprises gamma or tone mapping information and processing the current frame of the tone mapping value of the current frame based on the gamma or tone mapping information to maintain dynamic range consistency with the “The statistics for each tonal segment are integrated into the feature vector” (Nanda, ¶ [0035]) where tonal segments are used in the feature vector that is applied to the target video stream based on the reference video stream. Additionally, “tonal segments are the pixels corresponding to ranges of pixels in the lightness channel” (Nanda, ¶ [0032]).
Nanda does not teach that the processed stream is a narrow FOV image stream or that the tone mapping is based on a wide FOV image stream.
Sun teaches of a narrow FOV image stream “image 104 represents an input image from a narrower FOV image (which may also be referred to herein as an ‘inner image or ‘higher quality’ image)” (Sun, ¶ [0009]) and a wide FOV image stream “Image 106 represents an exemplary resultant fused wider FOV image, with enhanced texture, detail, and/or noise representation” (Sun, ¶ [0009]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of adjusting the tone mapping value of a second image stream based on a first image stream taught by Nanda with the method of distinct image streams being wide field of view and narrow field of view taught by Sun to have the ability to switch between tonal segments. The suggestion/motivation to do so would have been to provide easier modifications on different display devices.
Nanda in view of Sun do not efficiently teach the use of gamma or tone mapping information.
Dangi teaches gamma or tone mapping information “the ISP may perform demosaic, gain adjustment, white balance adjustment, color balance or color correction, gamma compression, tone mapping or adjustment, denoise or noise filtering, edge enhancement, contrast adjustment, intensity adjustment (darken), among others” (Dangi, Page 3). The motivation to combine Dangi would be the same as claim 1.
In regard to claim 13, claim 4 is substantially similar to claim 13, hence the rejection analysis for claim 4 is also applied to claim 13. Nanda in view of Sun, Dangi, and Tsai2 teach the additional limitations of [t]he system of claim 10, wherein the stored information comprises gamma or tone mapping and the second processing module is configured to process the current frame of the narrow FOV image stream by adjusting a gamma or tone mapping value of the current frame based on the gamma or tone mapping information to maintain dynamic range consistency with the processed wide FOV image stream (Nanda, ¶ [0032]), (Sun, ¶ [0009]), & (Dangi, Page 3).
In regard to claim 8, Nanda in view of Sun, Dangi, and Tsai2 teach [t]he method of claim 1, further comprising concurrently training the neural network using each newly generated frame of the processed “Training can be an iterative process” (Nanda, ¶ [0051]), it is also noted that training may be performed based on how much data is required, and may take other formats, such as manual, random, or automatically (Nanda, ¶ [0046] – [0052]).
Nanda does not teach that the training is based on a wide FOV image stream.
Sun teaches of a wide FOV image stream “Image 106 represents an exemplary resultant fused wider FOV image, with enhanced texture, detail, and/or noise representation” (Sun, ¶ [0009]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of training a machine learning model using an image stream in an iterative manner taught by Nanda with the use of a wide field of view image stream taught by Sun to have the training based on a wide field of view. The suggestion/motivation to do so would have been to provide the image stream for training from a specific image source.
In regard to claim 9, Nanda in view of Sun, Dangi, and Tsai2 teach [t]he method of claim 1, wherein the first processing technique comprises utilizing one or more of a hardware-based image signal processor (ISP), a software-based ISP and a neural network “different cameras within an image capture system can have different intrinsic properties, e.g., different fields of view, depth of field, spatial resolution, color sensitivity, and/or image signal processor (ISP) tuning” (Sun, ¶ [0006]), additionally, “Pipeline 200 takes as input two images (102/104) with different spatial resolution, synchronously captured by two cameras with different optics (focal length, lens, design, etc.) and sensors, and processed by different ISP tunings” (Sun, ¶ [0016]), furthermore, wider FOV image 102 may be upscaled and/or divided into a number of “tiles” as image 102′ before being processed by the deep neural network 220 (Sun, ¶ [0016]) which teaches the various ISP implementations, and the use of a neural network.
In regard to claim 15, Nanda in view of Sun, Dangi, and Tsai2 teach [t]he system of claim 10, further comprising a training module configured to concurrently train the neural network using each newly generated frame “Training data can be generated in a randomized manner. For example, a training application obtains an image. The training application generates random changes to the exposure, temperature, and tint of the image in order to create the modified image” (Nanda, ¶ [0049]) where the machine learning model may be trained to improve its accuracy to identify characteristics and generation of consistent images.
Nanda does not teach of a processed wide FOV image stream.
Sun teaches of a processed wide FOV image stream “Image 106 represents an exemplary resultant fused wider FOV image, with enhanced texture, detail, and/or noise representation” (Sun, ¶ [0009]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of training an artificial intelligence to process multiple image streams for color consistency taught by Nanda with the method of having one image stream be a wide field of view taught by Sun to create a method of training a neural network to process a wide field of view image. The suggestion/motivation to do so would have been to base the neural network’s training on the used image format.
Claims 5 & 14 are rejected under 35 U.S.C. 103 as being unpatentable over Nanda in view of Sun, Dangi, Tsai2, and Fridman Ofer Et. Al. (Pat. Pub. CA-3029124-A1, herein after “Ofer”).
In regard to claims 5 & 14, Nanda in view of Sun, Dangi, and Tsai2 teach [t]he method of claim 1, wherein the stored information comprises semantic information, wherein the semantic information is utilized to identify scene type for the previous frame of the processed “if a first movie (i.e., the reference video) is filmed such that each scene includes a blue-hued color scheme, the color consistency application can modify a second movie (i.e., the target video) to apply the same blue-hued color scheme to the second movie.” (Nanda, ¶ [0013]) where color schemes for example, may be scene-specific image enhancements used for the target video based on the reference video.
Nanda does not teach wherein the semantic information is utilized to identify scene type nor of the wide FOV image stream or the narrow FOV image stream.
Ofer teaches wherein the semantic information “The vehicles may communicate with a central server (e.g., server 1230) that maintains an up- to-date map including these visual landmarks connected to other significant geometric and semantic information” (Ofer, ¶ [0324]) which teaches semantic information is utilized to identify scene type “The first image capture device 122 may acquire a plurality of first images relative to a scene associated with the vehicle 200” (Ofer, ¶ [0117]) also “[t]he disclosed systems and methods may use semantic landmarks (e.g., traffic signs), since they can be reliably detected from the scene and matched with the landmarks stored in the road model or sparse map” (Ofer, ¶ [0320]) where semantic information may be used to identify road features, signs, stop lights, or other positional information. the wide FOV image stream “According to some embodiments, the FONT of one or more image capture devices 122, 124, and 126 may have a wide angle” (Ofer, ¶ [0134]) and the narrow FOV image stream “In such a configuration, image capture device 122 may provide a narrow field of view” (Ofer, ¶ [0145]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of image enhancements taught by Nanda with the method of semantic information used to identify surroundings, and distinct image streams being wide field of view and narrow field of view taught by Ofer to have wide or narrow field of view image enhancements based on the detected scene. The suggestion/motivation to do so would have been to provide clearer field of view images based on the observed surroundings and setting.
Sun additionally teaches of a narrow FOV image stream “image 104 represents an input image from a narrower FOV image (which may also be referred to herein as an ‘inner image or ‘higher quality’ image)” (Sun, ¶ [0009]) and a wide FOV image stream “Image 106 represents an exemplary resultant fused wider FOV image, with enhanced texture, detail, and/or noise representation” (Sun, ¶ [0009]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of combining images taught by Nanda, the semantic information taught by Ofer with the wide and narrow field of view editing technique taught by Sun to more clearly produce a consistent merged image based on the wide and narrow field of view images. The suggestion/motivation to do so would have been to merge the two FOV images based on the detected semantic information.
In regard to claim 14, claim 5 is substantially similar to claim 14, hence the rejection analysis for claim 5 is also applied to claim 14. Nanda in view of Sun, Dangi, Tsai2, and Ofer teach the additional limitations of [t]he system of claim 10, wherein the stored information comprises semantic information and the semantic information (Ofer, ¶ [0324]) is utilized to identify a scene type (Ofer, ¶ [0117] & [0320]) for the previous frame of the processed wide FOV image stream (Ofer, ¶ [0134]), and wherein the second processing module is configured to process the current frame of the narrow FOV image stream (Ofer, ¶ [0145]) by applying scene-specific image enhancements to the current frame based on the scene type to maintain visual consistency with the wide FOV image stream (Nanda, ¶ [0013]) & (Sun, ¶ [0009]).
Claims 6 & 16 are rejected under 35 U.S.C. 103 as being unpatentable over Nanda in view of Sun, Dangi, Tsai2, Ofer, and Chung-Chi Tsai Et. Al. (Pat. Pub. US-11276177-B1, herein after “Tsai”).
In regard to claim 6, Nanda in view of Sun and Ofer teach [t]he method of claim 5 and extracting the semantic information “the vehicles may also send the server other semantic and geometric information associated with the [feature points]” (Ofer, ¶ [0327]).
Nanda in view of Sun and Ofer do not entirely teach wherein the previous frame of the processed wide FOV image stream is segmented to identify a region corresponding to the current frame of the narrow FOV image stream.
Tsai teaches wherein the previous frame of the processed wide FOV image stream is segmented to identify a region corresponding to the current frame of the narrow FOV image stream “The compute components 110 can perform segmentation (e.g., foreground-background segmentation) using images with different FOVs. For example, the compute components 110 can perform foreground-background segmentation using an image with a first FOV captured by image capturing device 102, and an image with a second FOV captured by image capturing device 104. The foreground-background segmentation can also enable (or can be used in conjunction with) other image adjustments or image processing operations such as … depth-enhanced and object-aware auto exposure, auto white balance, auto-focus, tone mapping” (Tsai, ¶ [0035]) where different field of view images are used in tandem to perform segmentation.
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of extracting semantic information taught by Ofer with the use of segmenting an image corresponding to two different field of view images taught by Tsai to segment different field of view image streams to extract the sematic information. The suggestion/motivation to do so would have been to identify the semantic information easier by removing it from the remaining area.
In regard to claim 16, Nanda in view of Sun teach [t]he system of claim 10.
Nanda in view of Sun fail to teach the display device is a head-mounted display (HMD) of an extended reality (XR) system.
Tsai teaches where the display device is a head-mounted display (HMD) of an extended reality (XR) system “the image processing system 100 can be part of an electronic device (or devices) such as … a smart wearable device, an extended reality (XR) device (e.g., a head-mounted display, smart glasses, etc.)” (Tsai, ¶ [0023]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of training an artificial intelligence to process multiple image streams for color consistency taught by Nanda with the use of a head-mounted display device taught by Tsai2 to create a portable, wearable system. The suggestion/motivation to do so would have been to allow the user to experience the color consistency while in extended reality.
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Nanda in view of Sun, Dangi, Tsai2, Ofer, and Feng Li et al. (Pat. Pub. US-20230098437-A1, herein after “Li”).
In regard to claim 7, Nanda in view of Sun, Dangi, and Tsai2 teach [t]he method of claim 1.
Nanda in view of Sun, Dangi, and Tsai2 fail to teach wherein for stereo image pairs, the stored information comprises a weighted average of corresponding information obtained from two previous frames of the processed wide FOV image stream.
Li teaches wherein for stereo image pairs, the stored information comprises a weighted average of corresponding information obtained from two previous frames of the processed wide FOV image stream “Stereo image stitching has applications in many areas, including image/video composition and video stabilization, which can utilize the increased FOV that is achievable from performing stereo image stitching. As used herein, “stitching” or “fusion” may refer to a process that extends the FOV of one or more of the individual source images or a process that generates an enhanced image with the same FOV as one or more of the source images” (Li, ¶ [0004]), and “a weighted averaging step fuses the HR features based on the similarity scores to achieve a single HR feature for every location in the LR feature set” (Li, ¶ [0054]) where the low resolution features are fused by a weighted averaging step to produce the high resolution product, and the low resolution features are wide FOV.
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of processing multiple image streams taught by Nanda in view of Sun, Dangi, and Tsai2 with the method of generating the stored information based on a weighted average of previous frames taught by Li to make the stored information more consistent and representative of the previous frames. The suggestion/motivation to do so would have been to maintain the wide field of view at a higher quality.
Claims 17 & 18 are rejected under 35 U.S.C. 103 as being unpatentable over in view of Sun, Dangi, Tsai2, and Wim Abbeloos et al. (Pat. Pub. JP-2022051700-A, herein after “Abbeloos”).
In regard to claims 17 & 18, Nanda in view of Sun, Dangi, and Tsai2 teach [t]he method of claim 1, wherein the previous frame of the processed wide FOV image stream is an immediately preceding frame of the processed wide FOV image stream The image capture devices 102 and 104 may capture images and/or video data having different FOV … the image capture device 102 and 104 may include different image sensor with different FOV … such as wide-angle lens, telephoto lens (e.g., short long focus, long focus, etc.), a standard lens, zoom lens and so on. In some examples, the image capture device 102 may include a type of lens, and image capture device 104 may include different types of lens” (Tsai2, Page 9) where a wide FOV image stream is taught.
Nanda in view of Sun, Dangi, and Tsai2 fail to explicitly teach wherein the previous frame of the processed image stream is an immediately preceding frame of the processed image stream.
Abbeloos teaches the previous frame of the processed image stream is an immediately preceding frame of the processed image stream “in order to improve the temporal consistency within the generated second frame (having the second FoV), a part of the first past frame input to the modules 102 and 104 is generated by the frame integration module 104” (Abbeloos, Page 2), “The depth network 302 is configured to receive a plurality of past frames. Past frames may include narrow FoV frames and past wide FoV frames” (Abbeloos, Page 4) where there is a given past frame in the FoV image stream.
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of processing a wide FOV image stream and applying it to a different image stream taught by Nanda in view of Tsai2 with the previous frame being a preceding frame taught by Abbeloos to consecutively apply the image data from a previous frame to the current frame. The suggestion/motivation to do so would have been to maintain the previous wide field of view settings on the current narrow field of view.
In regard to claim 18, claim 17 is substantially similar to claim 18, hence the rejection analysis for claim 17 is also applied to claim 18. Nanda in view of Sun, Dangi, Tsai2 and Abbeloos teach the additional limitations of [t]he system of claim 10, wherein the previous frame of the processed wide FOV image stream is an immediately preceding frame of the processed wide FOV image stream (Tsai2, Page 9) & (Abbeloos, Pages 2 & 4).
Response to Arguments
Applicant’s arguments with respect to claim 1 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Applicant argues on page 9 of the Remarks that no reference teaches the amended claims. The newly cited references Dangi and Tsai2 teach the amended subject matter. Dangi teaches the use of white balance information, color conversion matrix information, gamma or tone mapping information, and semantic information, and Tsai2 teaches the use of one FOV image process based on another, different FOV image. The combination of cited references teach the amended subject matter, specifically information from one wide FOV image being applied with another narrow FOV image in a neural network to produce the processed narrow FOV image.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CAIDEN ALEXANDER USSERY whose telephone number is (571)272-1192. The examiner can normally be reached Monday - Friday* 7:30AM - 5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tammy Goddard can be reached at (571) 272-7773. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/C.A.U./Examiner, Art Unit 2611
/TAMMY GODDARD/Supervisory Patent Examiner, Art Unit 2611
Prosecution Insights