DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on May 19th, 2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 8-13, & 15 are rejected under 35 U.S.C. 103 as being unpatentable over Chetan Nanda Et. Al. (Pat. Pub. US-20190362478-A1, herein after “Nanda”) in view of Libin Sun Et. Al. (Pat. Pub. US-11688100-B1, herein after “Sun”).
In regard to claims 1 & 10, Nanda teaches [a] method for processing dual-stream images “a movie resulting from the two videos has a consistent color cast such that the target video appears to have been filmed under the same evening lighting conditions as the reference video” (Nanda, ¶ [0013]) processing two videos will be read as dual-stream, the method comprising:
processing a “The color consistency application computes a feature vector for the reference video and the target video” (Nanda, ¶ [0015]) where a first video and first processing technique are taught as a reference video and a feature vector. Nanda does not teach that the field of view for the reference or target video are wide or narrow;
storing one or more of color information, gamma or tone mapping information, semantic information from the “Each feature vector includes global features. Global features include a histogram of a luminance channel and two color channels” (Nanda, ¶ [0015]) where the feature vector includes luminance and color;
processing a “… generates an additional feature vector from the intermediate video and the features of the feature vector that are relevant to the reference video” (Nanda, ¶ [0016]) where a second feature vector based on the previous reference video is taught. Additionally, “using such a feature vector in conjunction with machine learning techniques, the color consistency application facilitates a more accurate matching of color cast between videos” (Nanda, ¶ [0015]) where the use of a neural network, or AI is taught; and
blending the processed “Together, the exposure parameters and the color and tint parameters form a set of parameters that, when applied to the target video, will result in a color scheme that is consistent with the reference video” (Nanda, ¶ [0017]) where the processing of the two videos is matched to produce a reference video and a target video with identical color parameters. Additionally, “displaying an interface comprising the preview image and a plurality of user interface elements that indicate the color parameters and provide adjustable settings to further adjust the parameters” (Nanda, ¶ [0045]) utilizes a display and display device.
Nanda fails to teach processing a wide field-of-view (FOV) image;
processing a narrow FOV image stream using a neural network based on the one or more of the color information, the gamma or tone mapping information, and the semantic information; and
blending the processed wide FOV image stream and the processed narrow FOV image stream to generate a final image.
Sun teaches processing a wide field-of-view (FOV) image “Image 102 represents an input image from a wider FOV image (which may also be referred to herein as an ‘outer’ image or ‘lower quality’ image)” (Sun, ¶ [0009]) where processing wide field of view images is known in the art;
processing a narrow FOV image stream using a neural network based on the one or more of the color information, the gamma or tone mapping information, and the semantic information “image 104 represents an input image from a narrower FOV image (which may also be referred to herein as an ‘inner image or ‘higher quality’ image)” (Sun, ¶ [0009]) where processing narrow field of view images is known in the art, additionally “the first plurality of interpretable variables may be modified to independently change semantically-distinguishable characteristics of the modified second image, such as texture density, color balance, noise pattern, or noise level” (Sun, ¶ [0010]) teaches modifying semantic characteristics and color balance; and
blending the processed wide FOV image stream and the processed narrow FOV image stream to generate a final image “Image 106 represents an exemplary resultant fused wider FOV image, with enhanced texture, detail, and/or noise representation” (Sun, ¶ [0009]) teaches that the wide and narrow image can be blended for purposes of enhancing image quality.
PNG
media_image1.png
522
735
media_image1.png
Greyscale
Sun, Fig. 1, depicts wide FOV image (item 102) being combined with narrow FOV image (item 104) to produce merged image (item 106).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of processing multiple image streams based on color information of one image stream, and applying it to a different image stream taught by Nanda with the method of having a wide and narrow field of view image, and merging them to produce a complete, synchronized image taught by Sun to create a method of image processing that utilizes a wide and narrow field of view to create one image. The suggestion/motivation to do so would have been to maintain the wide field of view at a higher quality.
In regard to claim 10, claim 1 is substantially similar to claim 10, hence the rejection analysis for claim 1 is also applied to claim 10. Nanda in view of Sun teach the additional limitations of [a] system for processing dual-stream images “Systems and methods are disclosed herein for using machine learning techniques to automatically apply color characteristics of a reference video to a target video” (Nanda, ¶ [0013]), the system comprising:
a first processing module configured to process a wide field-of-view (FOV) image stream using a first processing technique (Nanda, ¶ [0015]) & (Sun, ¶ [0009]);
a memory module configured to store one or more of color information, gamma or tone mapping information, semantic information from the wide FOV image stream “one or more memory devices 504 stores program data 507 that includes one or more datasets and models described herein” (Nanda, ¶ [0064) where memory holds the relevant data. (Nanda, ¶ [0015]);
a second processing module configured to implement a neural network for processing a narrow FOV image stream based on the one or more of the color information, the gamma or tone mapping information, and the semantic information “The color consistency application provides the additional feature vector a second predictive model. The second predictive model predicts color temperature and tint parameters” (Nanda, ¶ [0016]) where a second model handles the additional feature vector. (Nanda, ¶ [0015]) & (Sun, ¶ [0009]); and
a blending module configured to blend the processed wide FOV image stream and the processed narrow FOV image stream to generate a final image, for display on a display device “Color consistency application 102 combines exposure parameters 135 and color temperature and tint parameters 136 into combined parameters 140” (Nanda, ¶ [0025]) where an application is used to combine the parameters. (Nanda, ¶ [0017]) & (Sun, ¶ [0009]).
In regard to claims 2 & 11, Nanda in view of Sun teach [a] method of claim 1, wherein processing the “Color consistency application 102 uses predictive model 121 to predict temperature and tint parameter 136, that if applied to target video 104, results in target video 104 having a similar color temperature and tint to reference video 103” (Nanda, ¶ [0021]) where the temperature of the second stream is adjusted to match the reference stream.
Nanda does not teach that the processed stream is a narrow FOV image stream or that the adjustment is based on a wide FOV image stream.
Sun teaches of a narrow FOV image stream “image 104 represents an input image from a narrower FOV image (which may also be referred to herein as an ‘inner image or ‘higher quality’ image)” (Sun, ¶ [0009]) and a wide FOV image stream “Image 106 represents an exemplary resultant fused wider FOV image, with enhanced texture, detail, and/or noise representation” (Sun, ¶ [0009]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of adjusting the color temperature of a second image stream based on a first image stream taught by Nanda with the method of distinct image streams being wide field of view and narrow field of view taught by Sun to have matching color temperatures in the different field of view images. The suggestion/motivation to do so would have been to create a unified image from multiple different fields of view.
In regard to claim 11, claim 2 is substantially similar to claim 11, hence the rejection analysis for claim 2 is also applied to claim 11. Nanda in view of Sun teach the additional limitations of [a] system of claim 10, wherein the second processing module is configured to process the narrow FOV image stream by adjusting a color temperature of the narrow FOV image stream based on the color information from a previous frame of the wide FOV image stream to maintain color consistency with the wide FOV image stream (Nanda, ¶ [0021]) & (Sun, ¶ [0009]).
In regard to claims 3 & 12, Nanda in view of Sun teach [a] method of claim 1, wherein processing the “color consistency application 102 converts the target and reference video videos from an R-G-B color space to an L-a-b color space” (Nanda, ¶ [0029]) where the color space may be converted.
Nanda does not teach that the processed stream is a narrow FOV image stream or that the conversion is based on a wide FOV image stream.
Sun teaches of a narrow FOV image stream “image 104 represents an input image from a narrower FOV image (which may also be referred to herein as an ‘inner image or ‘higher quality’ image)” (Sun, ¶ [0009]) and a wide FOV image stream “Image 106 represents an exemplary resultant fused wider FOV image, with enhanced texture, detail, and/or noise representation” (Sun, ¶ [0009]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of converting the color space of a second image stream based on a first image stream taught by Nanda with the method of distinct image streams being wide field of view and narrow field of view taught by Sun to have the ability to switch between color spaces on the different field of view images. The suggestion/motivation to do so would have been to provide easier modifications on different display devices.
In regard to claim 12, claim 3 is substantially similar to claim 12, hence the rejection analysis for claim 3 is also applied to claim 12. Nanda in view of Sun teach the additional limitations of [a] system of claim 10, wherein the second processing module is configured to process the narrow FOV image stream by applying a color conversion matrix to convert a color space of the narrow FOV image stream based on the color information from a previous frame of the wide FOV image stream to maintain color consistency with the wide FOV image stream (Nanda, ¶ [0029]) & (Sun, ¶ [0009]).
In regard to claims 4 & 13, Nanda in view of Sun teach [a] method of claim 1, wherein processing the “The statistics for each tonal segment are integrated into the feature vector” (Nanda, ¶ [0035]) where tonal segments are used in the feature vector that is applied to the target video stream based on the reference video stream. Additionally, “tonal segments are the pixels corresponding to ranges of pixels in the lightness channel” (Nanda, ¶ [0032]).
Nanda does not teach that the processed stream is a narrow FOV image stream or that the tone mapping is based on a wide FOV image stream.
Sun teaches of a narrow FOV image stream “image 104 represents an input image from a narrower FOV image (which may also be referred to herein as an ‘inner image or ‘higher quality’ image)” (Sun, ¶ [0009]) and a wide FOV image stream “Image 106 represents an exemplary resultant fused wider FOV image, with enhanced texture, detail, and/or noise representation” (Sun, ¶ [0009]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of adjusting the tone mapping value of a second image stream based on a first image stream taught by Nanda with the method of distinct image streams being wide field of view and narrow field of view taught by Sun to have the ability to switch between tonal segments. The suggestion/motivation to do so would have been to provide easier modifications on different display devices.
In regard to claim 13, claim 4 is substantially similar to claim 13, hence the rejection analysis for claim 4 is also applied to claim 13. Nanda in view of Ofer teach the additional limitations of [a] system of claim 10, wherein the second processing module is configured to process the narrow FOV image stream by adjusting a gamma or tone mapping value of the narrow FOV image stream based on the gamma or tone mapping information from a previous frame of the wide FOV image stream to maintain dynamic range consistency with the wide FOV image stream (Nanda, ¶ [0032]) & (Sun, ¶ [0009]).
In regard to claim 8, Nanda in view of Sun teach [a] method of claim 1, further comprising concurrently training the neural network using each newly generated frame of the processed “Training can be an iterative process” (Nanda, ¶ [0051]), it is also noted that training may be performed based on how much data is required, and may take other formats, such as manual, random, or automatically (Nanda, ¶ [0046] – [0052]).
Nanda does not teach that the training is based on a wide FOV image stream.
Sun teaches of a wide FOV image stream “Image 106 represents an exemplary resultant fused wider FOV image, with enhanced texture, detail, and/or noise representation” (Sun, ¶ [0009]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of training a machine learning model using an image stream in an iterative manner taught by Nanda with the use of a wide field of view image stream taught by Sun to have the training based on a wide field of view. The suggestion/motivation to do so would have been to provide the image stream for training from a specific image source.
In regard to claim 9, Nanda in view of Sun teach [a] method of claim 1, wherein the first processing technique comprises utilizing one or more of a hardware-based image signal processor (ISP), a software-based ISP, a neural network “different cameras within an image capture system can have different intrinsic properties, e.g., different fields of view, depth of field, spatial resolution, color sensitivity, and/or image signal processor (ISP) tuning” (Sun, ¶ [0006]), additionally, “Pipeline 200 takes as input two images (102/104) with different spatial resolution, synchronously captured by two cameras with different optics (focal length, lens, design, etc.) and sensors, and processed by different ISP tunings” (Sun, ¶ [0016]), furthermore, wider FOV image 102 may be upscaled and/or divided into a number of “tiles” as image 102′ before being processed by the deep neural network 220 (Sun, ¶ [0016]) which teaches the ISP, and the neural network.
In regard to claim 15, Nanda in view of Sun teach [a] system of claim 10, further comprising a training module configured to concurrently train the neural network using each newly generated frame “Training data can be generated in a randomized manner. For example, a training application obtains an image. The training application generates random changes to the exposure, temperature, and tint of the image in order to create the modified image” (Nanda, ¶ [0049]) where the machine learning model may be trained to improve its accuracy to identify characteristics and generation of consistent images.
Nanda does not teach of a processed wide FOV image stream.
Sun teaches of a processed wide FOV image stream “Image 106 represents an exemplary resultant fused wider FOV image, with enhanced texture, detail, and/or noise representation” (Sun, ¶ [0009]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of training an artificial intelligence to process multiple image streams for color consistency taught by Nanda with the method of having one image stream be a wide field of view taught by Sun to create a method of training a neural network to process a wide field of view image. The suggestion/motivation to do so would have been to base the neural network’s training on the used image format.
Claims 5, 7, & 14 are rejected under 35 U.S.C. 103 as being unpatentable over Nanda in view of Sun and Fridman Ofer Et. Al. (Pat. Pub. CA-3029124-A1, herein after “Ofer”).
In regard to claims 5 & 14, Nanda in view of Sun teach [a] method of claim 1, wherein the semantic information is utilized to identify scene type for a previous frame of the “if a first movie (i.e., the reference video) is filmed such that each scene includes a blue-hued color scheme, the color consistency application can modify a second movie (i.e., the target video) to apply the same blue-hued color scheme to the second movie.” (Nanda, ¶ [0013]) where color schemes for example, may be scene-specific image enhancements used for the target video based on the reference video.
Nanda does not teach wherein the semantic information is utilized to identify scene type nor of the wide FOV image stream or the narrow FOV image stream.
Ofer teaches wherein the semantic information “The vehicles may communicate with a central server (e.g., server 1230) that maintains an up- to-date map including these visual landmarks connected to other significant geometric and semantic information” (Ofer, ¶ [0324]) which teaches semantic information is utilized to identify scene type “The first image capture device 122 may acquire a plurality of first images relative to a scene associated with the vehicle 200” (Ofer, ¶ [0117]) also “[t]he disclosed systems and methods may use semantic landmarks (e.g., traffic signs), since they can be reliably detected from the scene and matched with the landmarks stored in the road model or sparse map” (Ofer, ¶ [0320]) where semantic information may be used to identify road features, signs, stop lights, or other positional information. the wide FOV image stream “According to some embodiments, the FONT of one or more image capture devices 122, 124, and 126 may have a wide angle” (Ofer, ¶ [0134]) and the narrow FOV image stream “In such a configuration, image capture device 122 may provide a narrow field of view” (Ofer, ¶ [0145]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of image enhancements taught by Nanda with the method of semantic information used to identify surroundings, and distinct image streams being wide field of view and narrow field of view taught by Ofer to have wide or narrow field of view image enhancements based on the detected scene. The suggestion/motivation to do so would have been to provide clearer field of view images based on the observed surroundings and setting.
Sun additionally teaches of a narrow FOV image stream “image 104 represents an input image from a narrower FOV image (which may also be referred to herein as an ‘inner image or ‘higher quality’ image)” (Sun, ¶ [0009]) and a wide FOV image stream “Image 106 represents an exemplary resultant fused wider FOV image, with enhanced texture, detail, and/or noise representation” (Sun, ¶ [0009]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of enhancing an image by using semantic information to identify surroundings, and distinct image streams being wide field of view and narrow field of view taught by Nanda and Ofer with the wide and narrow field of view editing technique taught by Sun to more clearly produce a consistent merged image based on the wide and narrow field of view images. The suggestion/motivation to do so would have been to merge the two FOV images based on the detected semantic information.
In regard to claim 14, claim 5 is substantially similar to claim 14, hence the rejection analysis for claim 5 is also applied to claim 14. Nanda in view of Ofer and Sun teach the additional limitations of [a] system of claim 10, wherein the semantic information (Ofer, ¶ [0324]) is utilized to identify scene type (Ofer, ¶ [0117] & [0320]) for a previous frame of the wide FOV image stream (Ofer, ¶ [0134]), and wherein the second processing module is configured to process the narrow FOV image stream (Ofer, ¶ [0145]) by applying scene-specific image enhancements to the narrow FOV image stream based on the scene type of the previous frame of the wide FOV image stream to maintain visual consistency with the wide FOV image stream (Nanda, ¶ [0013]) & (Sun, ¶ [0009]).
In regard to claim 7, Nanda in view of Sun teach [a] method of claim 1.
Nanda teaches processing corresponding one or more of the white balance information, the gamma or tone mapping information, the semantic information from two previous frames “Statistical attribute data includes statistics such as a mean or a standard deviation of one or more color channels of a tonal segment. Statistical attribute data can also include a median, variance, or other statistics derived from the tonal segments 241-243” (Nanda, ¶ [0034]) which teaches processing corresponding to the tonal segments.
Nanda in view of Sun do not teach stereo image pairs, or processing the narrow FOV image stream comprises utilizing a weighted average or where the processing is based on two previous frames of the processed wide FOV image stream.
Ofer teaches stereo image pairs “there may be a height difference between the image capture devices 122, 124, and 126, which may provide sufficient parallax information to enable stereo analysis” (Ofer, ¶ [0125]) where the image capturing method may utilize different fields of view or be stereoscopic, processing the narrow FOV image stream comprises utilizing a weighted average “processing unit 110 may make the determination based on a weighted average of the individual analyses performed at step 582” (Ofer, ¶ [0183]) where a weighted average is used to process the images. It is also noted that the images may be any desired focal length such as a wide field of view (Ofer, ¶ [0116]) from the processed wide FOV image stream “image capture device 124 may provide a wide field of view” (Ofer, ¶ [0134]) where the processing performed can be based on a wide FOV image.
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of using tone mapping for image processing taught by Nanda with the use of stereo image pairs, and a weighted average for processing based on the captured image taught by Ofer to produce averaged tone mapping and color consistency for stereo image pairs. The suggestion/motivation to do so would have been to ensure that color or tone characteristics are aligned.
Claims 6 & 16 are rejected under 35 U.S.C. 103 as being unpatentable over Nanda in view of Sun, Ofer, and Chung-Chi Tsai Et. Al. (Pat. Pub. US-11276177-B1, herein after “Tsai”).
In regard to claim 6, Nanda in view of Sun and Ofer teach [a] method of claim 5 and extracting the semantic information “the vehicles may also send the server other semantic and geometric information associated with the [feature points]” (Ofer, ¶ [0327]).
Nanda in view of Sun and Ofer do not entirely teach wherein the wide FOV image stream is segmented corresponding to the narrow FOV image stream.
Tsai teaches wherein the wide FOV image stream is segmented corresponding to the narrow FOV image stream “The compute components 110 can perform segmentation (e.g., foreground-background segmentation) using images with different FOVs. For example, the compute components 110 can perform foreground-background segmentation using an image with a first FOV captured by image capturing device 102, and an image with a second FOV captured by image capturing device 104. The foreground-background segmentation can also enable (or can be used in conjunction with) other image adjustments or image processing operations such as … depth-enhanced and object-aware auto exposure, auto white balance, auto-focus, tone mapping” (Tsai, ¶ [0035]) where different field of view images are used in tandem to perform segmentation.
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of extracting semantic information taught by Ofer with the use of segmenting an image corresponding to two different field of view images taught by Tsai to segment different field of view image streams to extract the sematic information. The suggestion/motivation to do so would have been to identify the semantic information easier by removing it from the remaining area.
In regard to claim 16, Nanda in view of Sun teach [a] system of claim 10.
Nanda in view of Sun fail to teach the display device is a head-mounted display (HMD) as part of an extended reality (XR) system.
Tsai teaches where the display device is a head-mounted display (HMD) as part of an extended reality (XR) system “the image processing system 100 can be part of an electronic device (or devices) such as … a smart wearable device, an extended reality (XR) device (e.g., a head-mounted display, smart glasses, etc.)” (Tsai, ¶ [0023]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of training an artificial intelligence to process multiple image streams for color consistency taught by Nanda with the use of a head-mounted display device taught by Tsai to create a portable, wearable system. The suggestion/motivation to do so would have been to allow the user to experience the color consistency while in extended reality.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Feng Li Et. Al. (Pat. Pub. US-20230098437-A1) teaches enhancing images captured from multi-camera systems, including different fields of view, the reference additionally teaches the use of semantic information, artificial intelligence, and an image signal processor.
Chen Golan Et. Al. (Pat. Pub. US-20120026366-A1) teaches obtaining a requested zoom image, and aligning it with a wide image to allow for switching between the two with a consistent image output.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CAIDEN ALEXANDER USSERY whose telephone number is (571)272-1192. The examiner can normally be reached Monday - Friday* 7:30AM - 5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tammy Goddard can be reached at (571) 272-7773. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/C.A.U./Examiner, Art Unit 2611
/TAMMY GODDARD/Supervisory Patent Examiner, Art Unit 2611
Prosecution Insights