DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Applicant’s response, filed 11 December 2025, to the last office action has been entered and made of record.
In response to the amendments to the claims, they are acknowledged, supported by the original disclosure, and no new matter is added.
Amendments to the independent claims 1, 13, and 16 have necessitated an updated ground of rejection over the applied prior art. Please see below for the updated interpretations and rejections.
Response to Arguments
Applicant's arguments filed 11 December 2025 have been fully considered but they are not persuasive.
In response to Applicant’s request for rejoinder, on p. 11 of Applicant’s reply, for rejoinder of withdrawn claims 2, 10, 14, and 17 should the elected independent claims be found allowable, the Examiner acknowledges the request for rejoinder, which would be pending the patentability of independent claims 1, 13, and 16.
In response to Applicant’s arguments on p. 11-17 of Applicant’s reply, that the combined teachings of Kim, McElvain, and Watanabe fail to render obvious the amended claim features of “performing target recognition separately on the first AI input data to obtain first target recognition results and on the second AI input data to obtain second target recognition results; and generating target information representing a final target recognition result by combining the first target recognition results and the second target recognition results” in amended claims 1, 3-9, 11-13, 15-16, and 18-20, the Examiner respectfully disagrees.
Examiner notes the claims are treated with their broadest reasonable interpretations consistent with the specification. See MPEP 2111. Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993). Furthermore, the test for obviousness is what the combined teachings of the references would have suggested to those of ordinary skill in the art. See In re Keller, 642 F.2d 413, 208 USPQ871 (CCPA 1981).
Kim is relied upon to teach a method for recognizing an object in an image, where received images are preprocessed to improve the recognition rate of an object included in the received image, where deblur, denoise, wide dynamic range (WDR) or high dynamic range (HDR), color tone mapping, and demosaicing image processing may be performed, and that the preprocessed image is input to an input layer of an artificial neural network for object recognition (see Kim [0082]-[0089] and [0092]-[0094]).
McElvain is relied upon to teach a known technique of capturing raw image data, including long exposure, middle exposure, and short exposure pixel values characterized by long, middle, and short exposure times, from a spatially-multiplexed-exposure (SME) high-dynamic-range (HDR) image sensor, preprocessing the long exposure, middle exposure, and short exposure pixel values to remove pixel values that fail to meet one or more quality requirements, and generating an HDR image from the captured image data after iterations of the auto exposure processing (see McElvain [0071]-[0085], [0095]-[0097] and [0105]-[0109]). Notably, McElvain further teaches that long exposure pixel values and short exposure pixel values may be separately filtered to remove long exposure pixel values that are saturated and short exposure pixel values that are significantly affected by noise (see McElvain [0073]).
Watanabe is relied upon to teach a known technique of using a first and second deep neural network to process short and long exposure images in parallel to recognize the subject captured in the respective short and long exposure images and determining a final recognition result information based on both the first and second recognition result information to improve the accuracy in recognizing the subject by avoiding secondarily generated artifacts from the HDR image generation process (see Watanabe [0040]-[0052]). Watanabe is further noted to teach that the disclosed embodiment is understood to process the short exposure image with the first DNN unit and generate first recognition results information in parallel with processing the long exposure image with the second DNN unit and generate the second recognition results (see Watanabe Fig. 2 and [0055]-[0057]), and a final recognition result information is determined based on both the first and second recognition result information (see Watanabe Fig. 2 and [0058]).
The combined teachings of Kim, McElvain, and Watanabe would have suggest to one of ordinary skill in the art that the received images for object recognition, captured with an image sensor as taught by McElvain, includes long, middle, and short exposure pixel data, and that image enhancing processing is performed separately on each of the corresponding long, middle, and short exposure pixel data and used to generate a corresponding HDR image, and to further input the preprocessed long, middle, and short exposure images, along with the HDR preprocessed image, to respective object recognition neural networks to be perform object recognition processing in parallel, and determine a final recognition result based on the combined recognition result information from the respective object recognition neural networks for each preprocessed exposure images, predictably leading to an improved method for recognizing an object by further inputting preprocessed long, middle, and short exposure images to the object recognition neural network to account for possibly generated artifacts in the inputted HDR image and further improving the object recognition rate.
As the combined cited prior art teachings suggest that the image enhancing processing is performed separately on each of the corresponding long, middle, and short exposure pixel data, and that the preprocessed long, middle, and short exposure images, along with the HDR preprocessed image, are processed by respective object recognition neural networks to perform object recognition processing in parallel, where the final recognition result is determined based on the combined recognition result information from the respective object recognition neural networks for each preprocessed exposure images, the combined cited prior art teachings provide a broadest reasonable interpretation for performing target recognition separately on first and second AI input data to obtain respective first and second target recognition results, and that a final target recognition result is generated based on a combination of the first and second target recognition results.
Thus, the further teachings of Kim, McElvain, and Watanabe, when combined, would have suggested to those of ordinary skill in the art the broadest reasonable interpretation of the amended claim subject matter of “performing target recognition separately on the first AI input data to obtain first target recognition results and on the second AI input data to obtain second target recognition results; and generating target information representing a final target recognition result by combining the first target recognition results and the second target recognition results”.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claims 1, 3-9, 11-13, 15-16, and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Kim et al. (US 2022/0083797, effectively filed 28 December 2018), herein Kim, in view of McElvain et al. (US 2021/0243352, effectively filed 7 September 2018), herein McElvain, and Watanabe et al. (US 2022/0232182, effectively filed 10 May 2019), herein Watanabe.
Regarding claim 1, Kim discloses a target recognition method, comprising:
performing first image processing on first image data to obtain first artificial intelligence (AI) input data (see Kim [0082]-[0084] and [0092]-[0094], where image enhancement processing is performed on received images to improve recognition rate of an object included in the received image, where deblur, denoise, wide dynamic range (WDR) or high dynamic range (HDR), color tone mapping, and demosaicing image processing may be performed, and the preprocessed image is input to an input layer of an artificial neural network for object recognition);
performing target recognition on the first AI input data to obtain first target recognition results (see Kim [0087]-[0089] and [0092]-[0094], where the preprocessed image is input to an input layer of an artificial neural network for object recognition).
Kim does not explicitly disclose wherein the first image data comprises first raw image data from an image sensor; performing second image processing on second image data to obtain second AI input data, wherein the second image data comprises second raw image data from the image sensor, and wherein a first exposure duration corresponding to the first AI input data is different from a second exposure duration corresponding to the second AI input data, or a first dynamic range corresponding to the first AI input data is different from a second dynamic range corresponding to the second AI input data.
McElvain teaches in a related and pertinent auto exposure method for a spatially-multiplexed-exposure (SME) high-dynamic-range (HDR) image sensor (see McElvain Abstract), where raw image data, including long exposure, middle exposure, and short exposure pixel values characterized by long, middle, and short exposure times, are retrieved from the SME HDR image sensor and preprocessed to remove pixel values that fail to meet one or more quality requirements (see McElvain [0071]-[0075], [0095]-[0097] and [0105]-[0106]), where long exposure pixel values and short exposure pixel values may be separately filtered to remove long exposure pixel values that are saturated and short exposure pixel values that are significantly affected by noise (see McElvain [0073]), synthesizing the long, middle, and short exposure pixel values into an HDR histogram and iteratively performing processing steps to optimize the exposure times for the image sensor (see McElvain [0075]-[0084] and [0106]-[0109]), and generate an HDR image from the captured image data after iterations of the auto exposure processing (see McElvain [0085]).
At the time of filing, one of ordinary skill in the art would have found it obvious to apply the teachings of McElvain to the teachings of Kim, such that the received images are captured with an image sensor as taught by McElvain, including long, middle, and short exposure pixel data, and that the image enhancing processing is performed separately on each of the corresponding long, middle, and short exposure pixel data and used to generate a corresponding HDR image.
This modification is rationalized as an application of a known technique to a known method ready for improvement to yield predictable results.
In this instance, Kim discloses a base method for recognizing an object in an image, where received images are preprocessed to improve the recognition rate of an object included in the received image, where deblur, denoise, wide dynamic range (WDR) or high dynamic range (HDR), color tone mapping, and demosaicing image processing may be performed, and that the preprocessed image is input to an input layer of an artificial neural network for object recognition.
McElvain teaches a known technique of capturing raw image data, including long exposure, middle exposure, and short exposure pixel values characterized by long, middle, and short exposure times, from a spatially-multiplexed-exposure (SME) high-dynamic-range (HDR) image sensor, preprocessing the long exposure, middle exposure, and short exposure pixel values to remove pixel values that fail to meet one or more quality requirements, and generating an HDR image from the captured image data after iterations of the auto exposure processing.
One of ordinary skill in the art would have recognized that by applying McElvain’s techniques would allow for the method of Kim to capture the received images with an SME HDR image sensor as taught by McElvain, including long, middle, and short exposure pixel data, and that the image enhancing processing is performed separately on each of the corresponding long, middle, and short exposure pixel data and used to generate a corresponding HDR image to be input to the artificial neural network for performing object recognition, predictably leading to an improved method for recognizing an object in an image, where long, middle, and short exposure pixel data are captured by an SME HDR image sensor to be enhanced and used to generate an improved HDR image for improved recognition rate of an object.
While Kim teaches that the preprocessed image is input to an input layer of an artificial neural network for object recognition (see Kim [0087]-[0089] ad [0092]-[0094]); Kim and McElvain do not explicitly disclose performing target recognition separately on the first AI input data to obtain first target recognition results and on the second AI input data to obtain second target recognition results; and generating target information representing a final target recognition result by combining the first target recognition results and the second target recognition results.
Watanabe teaches in a related and pertinent image recognition device and method (see Watanabe Abstract), where a camera image sensor captures short and long exposure images to generate a high dynamic range image and recognize a subject in the captured image (see Watanabe [0024] and [0029]-[0033]), and that a recognition unit, including first and second deep neural network (DNN) units, can process the short and long exposure images to recognize the subject captured in the short and long exposure images to improve the accuracy in recognizing the subject by avoiding secondarily generated artifacts from the HDR image generation process (see Watanabe [0040]-[0052]), where the disclosed embodiment is understood to process the short exposure image with the first DNN unit and generate first recognition results information in parallel with processing the long exposure image with the second DNN unit and generate the second recognition results (see Watanabe Fig. 2 and [0055]-[0057]), and a final recognition result information is determined based on both the first and second recognition result information (see Watanabe Fig. 2 and [0058]).
At the time of filing, one of ordinary skill in the art would have found it obvious to apply the teachings of Watanabe to the teachings of Kim and McElvain, such that the object recognition rate can be further improved by also processing the preprocessed long, middle, and short exposure images by respective object recognition neural networks for each preprocessed exposure images and determining a final recognition result based on the combined recognition result information from the respective object recognition neural networks for each preprocessed exposure images.
This modification is rationalized as an application of a known technique to a known method ready for improvement to yield predictable results.
In this instance, Kim and McElvain disclose a base method for recognizing an object in an image, where received images, including long, middle, and short exposure images are captured with a SME HDR image sensor, and image enhancing processing is performed on the corresponding long, middle, and short exposure pixel data and used to generate a corresponding HDR image to be input to the artificial neural network for performing object recognition.
Watanabe teaches a known technique of using a first and second deep neural network to process short and long exposure images in parallel to recognize the subject captured in the respective short and long exposure images and determining a final recognition result information based on both the first and second recognition result information to improve the accuracy in recognizing the subject by avoiding secondarily generated artifacts from the HDR image generation process.
One of ordinary skill in the art would have recognized that by applying Watanabe’s techniques would allow for the method of Kim and McElvain to further input the preprocessed long, middle, and short exposure images, along with the HDR preprocessed image, to respective object recognition neural networks and determine a final recognition result based on the combined recognition result information from the respective object recognition neural networks for each preprocessed exposure images, predictably leading to an improved method for recognizing an object by further inputting preprocessed long, middle, and short exposure images to the object recognition neural network to account for possibly generated artifacts in the inputted HDR image and further improving the object recognition rate.
Regarding claim 3, please see the above rejection of claim 1. Kim, McElvain, and Watanabe disclose the target recognition method of claim 1, further comprising
receiving, from the image sensor, an image data stream, wherein before performing the first image processing and the second image processing, the target recognition method further comprises
splitting the image data stream to obtain the first image data and the second image data (see McElvain [0170]-[0173], where raw image data can be retrieved from a frame of a video stream captured by the SME HDR image sensor, including long exposure pixel values, short exposure pixel values, and middle exposure pixel values; see Watanabe [0029]-[0030], where the imaging unit can capture, in one frame period, images with different sensitivities, corresponding to short exposure images and long exposure images).
Regarding claim 4, please see the above rejection of claim 1. Kim, McElvain, and Watanabe disclose the target recognition method of claim 1, wherein the first image processing is wide dynamic range (WDR) processing (see Kim [0083]-[0084], where the image preprocessing process includes performing Wide Dynamic Range (WDR) or High Dynamic Range (HDR) on the captured images; see McElvain [0085], where an HDR image is generated from the captured raw image data after auto exposure processing),
wherein the second image processing is linear processing (see McElvain [0073], where long and short exposure images are preprocessed to remove pixel values failing to meet one or more quality requirements; see Kim [0083]-[0084], where the preprocessing module can be configured to perform image signal processing on the image including, deblur, denoise, color tone mapping, and demosaicing),
wherein first image content corresponding to the first image data is the same as second image content corresponding to the second image data (see McElvain [0055] and [0072]-[0073], where the different exposure image data captured by the SME HDR image sensor captures the same scene; see Watanabe [0029]-[0030], where the imaging unit captures short and long exposure images of the imaged subject),
wherein the first exposure duration is different from the second exposure duration (see McElvain [0056], [0072]-[0073], [0093]-[0097], and [0105]-[0106], where the captured image data have different exposure times, e.g. long, middle, and short exposure times, and long exposure time is longer than the short exposure time, and middle exposure time is shorter than long exposure time and longer than short exposure time; see Watanabe [0029]-[0030], where the imaging unit captures short and long exposure images of the imaged subject),
wherein performing the first image processing further comprises performing the first image processing based on the first image data and the second image data to obtain the first AI input data (see Kim [0083]-[0084], where the image preprocessing process includes performing Wide Dynamic Range (WDR) or High Dynamic Range (HDR) on the captured images; see McElvain [0072]-[0075], [0085], and [0095]-[0097], where an HDR image is generated from the captured raw image data of different exposure timings, including the long and short exposure images),
wherein the target recognition method further comprises performing third image processing on the first image data to obtain third AI input data (see McElvain [0071]-[0075], [0095]-[0097] and [0105]-[0106], where raw image data, includes long exposure, middle exposure, and short exposure pixel values characterized by long, middle, and short exposure times, are retrieved from the SME HDR image sensor and preprocessed to remove pixel values that fail to meet one or more quality requirements; see Watanabe [0040]-[0052], where the short and long exposure images are further processed by the DNN units to recognize the subject captured in the short and long exposure images; where the combined teachings suggests to further include preprocessed long exposure images to the object recognition neural network to improve object recognition rates), and
wherein performing the target recognition further comprises performing, based on the first AI input data, the second AI input data, and the third AI input data, the target recognition to determine the target information (see Kim [0087]-[0089] and [0092]-[0094], where the preprocessed image, which the processing includes wide dynamic range (WDR) or high dynamic range (HDR) processing, is input to an input layer of an artificial neural network for object recognition; see Watanabe [0040]-[0052], where the short and long exposure images are further processed by the DNN units to recognize the subject captured in the short and long exposure images; where the combined teachings suggests to include preprocessed long, middle, and short exposure images with the generated HDR image to the object recognition neural network to improve object recognition rates).
Regarding claim 5, please see the above rejection of claim 4. Kim, McElvain, and Watanabe disclose the target recognition method of claim 4, wherein performing the first image processing further comprises
performing the first image processing based on the first image data, the second image data, and third image data to obtain the first AI input data (see Kim [0083]-[0084], where the image preprocessing process includes performing Wide Dynamic Range (WDR) or High Dynamic Range (HDR) on the captured images; see McElvain [0072]-[0075], [0085], and [0093]-[0097], where an HDR image is generated from the captured raw image data of different exposure timings, and further suggests the raw image data includes the long, middle, and short exposure images),
wherein the third image data is third raw image data from the image sensor, wherein third image content corresponding to the third image data is the same as the first image content and the second image content (see McElvain [0055], [0072]-[0073], and [0093]-[0097], where the different exposure image data captured by the SME HDR image sensor captures the same scene, and further suggests middle exposure image data is included as part of the raw image data), and
wherein a third exposure duration corresponding to the third image data is different from the first exposure duration and the second exposure duration (see McElvain [0055], [0072]-[0073], and [0093]-[0097], where the different exposure image data captured by the SME HDR image sensor captures the same scene, and that middle exposure image data is captured and processed corresponding to a middle exposure time that is shorter than long exposure time and longer than short exposure time).
Regarding claim 6, please see the above rejection of claim 5. Kim, McElvain, and Watanabe disclose the target recognition method of claim 5, further comprising performing fourth image processing on the third image data to obtain fourth AI input data (see McElvain [0071]-[0075], [0095]-[0097] and [0105]-[0106], where raw image data, includes long exposure, middle exposure, and short exposure pixel values characterized by long, middle, and short exposure times, are retrieved from the SME HDR image sensor and preprocessed to remove pixel values that fail to meet one or more quality requirements; see Watanabe [0040]-[0052], where the exposure images are further processed by the DNN units to recognize the subject captured in the short and long exposure images; where the combined teachings suggests to further include preprocessed middle exposure images to the object recognition neural network to improve object recognition rates),
wherein performing the target recognition further comprises performing the target recognition based on the first AI input data, the second AI input data, the third AI input data, and the fourth AI input data to determine the target information (see Kim [0087]-[0089] and [0092]-[0094], where the preprocessed image, which the processing includes wide dynamic range (WDR) or high dynamic range (HDR) processing, is input to an input layer of an artificial neural network for object recognition; see Watanabe [0040]-[0052], where the short and long exposure images are further processed by the DNN units to recognize the subject captured in the short and long exposure images; where the combined teachings suggests to include preprocessed long, middle, and short exposure images with the generated HDR image to the object recognition neural network to improve object recognition rates).
Regarding claim 7, please see the above rejection of claim 4. Kim, McElvain, and Watanabe disclose the target recognition method of claim 4, further comprising performing fourth image processing on third image data to obtain fourth AI input data (see McElvain [0071]-[0075], [0095]-[0097] and [0105]-[0106], where raw image data, includes long exposure, middle exposure, and short exposure pixel values characterized by long, middle, and short exposure times, are retrieved from the SME HDR image sensor and preprocessed to remove pixel values that fail to meet one or more quality requirements; see Watanabe [0040]-[0052], where the exposure images are further processed by the DNN units to recognize the subject captured in the short and long exposure images; where the combined teachings suggests to further include preprocessed middle exposure images to the object recognition neural network to improve object recognition rates),
wherein the third image data is third raw image data from the image sensor, wherein third image content corresponding to the third image data is the same as the first image content and the second image content (see McElvain [0055], [0072]-[0073], and [0093]-[0097], where the different exposure image data captured by the SME HDR image sensor captures the same scene, and further suggests middle exposure image data is included as part of the raw image data), and
wherein a third exposure duration corresponding to the third image data is different from the first exposure duration and the second exposure duration corresponding to the second image data (see McElvain [0055], [0072]-[0073], and [0093]-[0097], where the different exposure image data captured by the SME HDR image sensor captures the same scene, and that middle exposure image data is captured and processed corresponding to a middle exposure time that is shorter than long exposure time and longer than short exposure time), and
wherein performing the target recognition further comprises performing the target recognition based on the first AI input data, the second AI input data, the third AI input data, and the fourth AI input data to determine the target information (see Kim [0087]-[0089] and [0092]-[0094], where the preprocessed image, which the processing includes wide dynamic range (WDR) or high dynamic range (HDR) processing, is input to an input layer of an artificial neural network for object recognition; see Watanabe [0040]-[0052], where the short and long exposure images are further processed by the DNN units to recognize the subject captured in the short and long exposure images; where the combined teachings suggests to include preprocessed long, middle, and short exposure images with the generated HDR image to the object recognition neural network to improve object recognition rates).
Regarding claim 8, please see the above rejection of claim 1. Kim, McElvain, and Watanabe disclose the target recognition method of claim 1, further comprising:
sending the first AI input data to a display; or storing the first AI input data and the second AI input data (see Watanabe [0098], where the HDR image generated from combining the short and long exposure images are stored in memory, and the memory stores the HDR image input from the HDR combination unit; see McElvain [0089], where dynamic data storage may store one or more of preprocessed long exposure pixel values and short exposure pixel values).
Regarding claim 9, please see the above rejection of claim 1. Kim, McElvain, and Watanabe disclose the target recognition method of claim 1, wherein after determining the target information, the target recognition method further comprises executing, based on the target information, a target event, and wherein the target event comprises:
displaying the target information in a first image displayed by a display; marking, based on the target information, a target object in a second image displayed by the display; uploading the target information to a cloud; or generating, based on the target information, a notification message (see Watanabe [0126], where detected objects are determined to be a pedestrian or not by performing a pattern matching process on a series of feature points indicating an outline of the object, and determined pedestrians controls a display unit to superimpose a rectangular contour line for emphasis on a recognized pedestrian).
Regarding claim 11, please see the above rejection of claim 4. Kim, McElvain, and Watanabe disclose the target recognition method of claim 4, wherein the first raw image data and the second raw image data correspond to a target scene (see McElvain [0055] and [0072]-[0073], where the different exposure image data captured by the SME HDR image sensor captures the same scene; see Watanabe [0029]-[0030], where the imaging unit captures short and long exposure images of the imaged subject),
wherein the target scene comprises a bright area and a dark area, wherein the bright area has a first brightness greater than a first threshold, wherein the dark area has a second brightness less than a second threshold (see McElvain [0004] and [0056], where for scenes that has insufficient dynamic range to capture both the brightest and darkest areas of the scene, high dynamic range (HDR) imaging is used to extend the dynamic range, where an image with short exposure time is captured to get good image data for bright portions of a scene, and another image with a longer exposure time is captured to get good image data for darker portions of the scene, and that the SME HDR image sensor captures long and short exposure pixel data for generating HDR imagery; see McElvain [0093]-[0097], where middle exposure pixel data is further captured by the SME HDR image sensor to further extend the high dynamic range; see McElvain [0119]-[0121], where long exposure pixel values, corresponding to dark portions of a scene, that exceed a saturation threshold are filtered, short exposure pixel values, corresponding to bright portions of a scene, that are below a noise threshold are filtered, and middle exposure pixel values that are above a relative saturation threshold and below a relative noise threshold are filtered),
wherein the first exposure duration is greater than the second exposure duration (see McElvain [0056], [0072]-[0073], [0093]-[0097], and [0105]-[0106], where the captured image data have different exposure times, e.g. long, middle, and short exposure times, and long exposure time is longer than the short exposure time, and middle exposure time is shorter than long exposure time and longer than short exposure time; see Watanabe [0029]-[0030], where the imaging unit captures short and long exposure images of the imaged subject), and
wherein performing the target recognition comprises:
obtaining, based on the second AI input data, first target information corresponding to the bright area (see Watanabe [0040]-[0052], where the short exposure images are processed by a DNN unit to recognize subject captured in the short exposure image; see McElvain [0004] and [0056], where an image with short exposure time is captured to get good image data for bright portions of a scene, and another image with a longer exposure time is captured to get good image data for darker portions of the scene, and that the SME HDR image sensor captures long and short exposure pixel data for generating HDR imagery; where the combined teachings suggests preprocessed short exposure image data is processed by a DNN to perform object recognition);
obtaining, based on the third AI input data, second target information corresponding to the dark area (see Watanabe [0040]-[0052], where the long exposure images are processed by a DNN unit to recognize subject captured in the long exposure image; see McElvain [0004] and [0056], where an image with a longer exposure time is captured to get good image data for darker portions of the scene, and that the SME HDR image sensor captures long and short exposure pixel data for generating HDR imagery; where the combined teachings suggests preprocessed long exposure image data is processed by a DNN to perform object recognition); and
obtaining, based on the first AI input data, third target information corresponding to an area other than the dark area and the bright area in the target scene (see Kim [0087]-[0089] and [0092]-[0094], where the preprocessed image, including WDR or HDR processing, is input to an input layer of an artificial neural network for object recognition; where the combined teachings suggests HDR preprocessed images generated from long, middle, and short exposure image data is processed by a neural network to perform object recognition).
Regarding claim 12, please see the above rejection of claim 6. Kim, McElvain, and Watanabe disclose the target recognition method of claim 6, wherein the first raw image data and the second raw image data correspond to a target scene,
wherein the target scene comprises a bright area, a dark area, and an intermediate area (see McElvain [0004] and [0056], where for scenes that has insufficient dynamic range to capture both the brightest and darkest areas of the scene, high dynamic range (HDR) imaging is used to extend the dynamic range, where an image with short exposure time is captured to get good image data for bright portions of a scene, and another image with a longer exposure time is captured to get good image data for darker portions of the scene, and that the SME HDR image sensor captures long and short exposure pixel data for generating HDR imagery; see McElvain [0093]-[0097], where middle exposure pixel data is further captured by the SME HDR image sensor to further extend the high dynamic range),
wherein the bright area has a first brightness greater than a first threshold, wherein the dark area has a second brightness less than a second threshold, wherein the intermediate area has a third brightness greater than a third threshold and less than a fourth threshold (see McElvain [0093]-[0097], where middle exposure pixel data correspond to middle exposure time that is shorter than the long exposure time and longer than the short exposure time; see McElvain [0119]-[0121], where long exposure pixel values, corresponding to dark portions of a scene, that exceed a saturation threshold are filtered, short exposure pixel values, corresponding to bright portions of a scene, that are below a noise threshold are filtered, and middle exposure pixel values that are above a relative saturation threshold and below a relative noise threshold are filtered),
wherein the third threshold is greater than or equal to the second threshold, wherein the fourth threshold is less than or equal to the first threshold (see McElvain [0093]-[0097], where middle exposure pixel data correspond to middle exposure time that is shorter than the long exposure time and longer than the short exposure time; see McElvain [0119]-[0121], where middle exposure pixel values that are above a relative saturation threshold and below a relative noise threshold are filtered; where the combined teachings suggests that the relative noise threshold for middle exposure pixel values would be equal or greater than the saturation threshold for long exposure pixels and that the relative saturation threshold for the middle exposure pixel values would be equal or less than the noise threshold for the short exposure pixel values),
wherein the first exposure duration is greater than the third exposure duration, wherein the third exposure duration is greater than the second exposure duration (see McElvain [0093]-[0097], where middle exposure pixel data correspond to middle exposure time that is shorter than the long exposure time and longer than the short exposure time), and
wherein performing the target recognition further comprises:
obtaining, based on the second AI input data, first target information corresponding to the bright area (see Watanabe [0040]-[0052], where the short exposure images are processed by a DNN unit to recognize subject captured in the short exposure image; see McElvain [0004] and [0056], where an image with short exposure time is captured to get good image data for bright portions of a scene, and another image with a longer exposure time is captured to get good image data for darker portions of the scene, and that the SME HDR image sensor captures long and short exposure pixel data for generating HDR imagery; where the combined teachings suggests preprocessed short exposure image data is processed by the object recognition neural network to perform object recognition);
obtaining, based on the third AI input data, second target information corresponding to the dark area (see Watanabe [0040]-[0052], where the long exposure images are processed by a DNN unit to recognize subject captured in the long exposure image; see McElvain [0004] and [0056], where an image with a longer exposure time is captured to get good image data for darker portions of the scene, and that the SME HDR image sensor captures long and short exposure pixel data for generating HDR imagery; where the combined teachings suggests preprocessed long exposure image data is processed by the object recognition neural network to perform object recognition);
obtaining, based on the fourth AI input data, third target information corresponding to the intermediate area (see McElvain [0071]-[0075], [0095]-[0097] and [0105]-[0106], where raw image data, includes long exposure, middle exposure, and short exposure pixel values characterized by long, middle, and short exposure times, are retrieved from the SME HDR image sensor and preprocessed to remove pixel values that fail to meet one or more quality requirements; see Watanabe [0040]-[0052], where the exposure images are further processed by the DNN units to recognize the subject captured in the short and long images; where the combined teachings suggests to further include preprocessed middle exposure images to the object recognition neural network to improve object recognition rates); and
obtaining, based on the first AI input data, fourth target information corresponding to an area other than the bright area, the dark area, and the intermediate area in the target scene (see Kim [0087]-[0089] and [0092]-[0094], where the preprocessed image, including WDR or HDR processing, is input to an input layer of an artificial neural network for object recognition; where the combined teachings suggests HDR preprocessed images generated from long, middle, and short exposure image data is processed by the object recognition neural network to perform object recognition).
Regarding claim 13, it recites an apparatus performing the method of claim 1. Kim, McElvain, and Watanabe teach an apparatus performing the method of claim 1. Please see above for detailed claim analysis, with the exception to the following further limitations:
a memory configured to store instructions (see Kim [0095]-[0102], where the disclosed teachings can be implemented by executing software instructions stored on a computer readable medium, such as random-access memory (RAM), read-only memory (ROM), or other medium that can carry or store desired program code); and
a processor coupled to the memory and configured to execute the instructions to perform the method of claim 1 (see Kim [0095]-[0102], where the instructions may be executable by one or more processors to perform certain aspects of the disclosed teachings)
Please see the above rejection for claim 1, as the rationale to combine the teachings of Kim, McElvain, and Watanabe are similar, mutatis mutandis.
Regarding claim 15, see above rejection for claim 13. It is an apparatus claim reciting similar subject matter as claim 3. Please see above claim 3 for detailed claim analysis as the limitations of claim 15 are similarly rejected.
Regarding claim 16, it recites a computer program product comprising instructions stored on a non-transitory computer-readable medium that, when executed by a processor, cause a target recognition apparatus to perform the method of claim 1. Kim, McElvain, and Watanabe teach a computer program product comprising instructions stored on a non-transitory computer-readable medium that, when executed by a processor, cause an apparatus to perform the method of claim 1 (see Kim [0095]-[0102], where the disclosed teachings can be implemented by executing software instructions stored on a computer readable medium, such as random-access memory (RAM), read-only memory (ROM), or other medium that can carry or store desired program code, by one or more processors). Please see above for detailed claim analysis.
Please see the above rejection for claim 1, as the rationale to combine the teachings of Kim, McElvain, and Watanabe are similar, mutatis mutandis.
Regarding claim 18, see above rejection for claim 16. It is a computer program product claim reciting similar subject matter as claim 3. Please see above claim 3 for detailed claim analysis as the limitations of claim 18 are similarly rejected.
Regarding claim 19, see above rejection for claim 16. It is a computer program product claim reciting similar subject matter as claim 4. Please see above claim 4 for detailed claim analysis as the limitations of claim 19 are similarly rejected.
Regarding claim 20, see above rejection for claim 19. It is a computer program product claim reciting similar subject matter as claim 5. Please see above claim 5 for detailed claim analysis as the limitations of claim 20 are similarly rejected.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TIMOTHY WING HO CHOI whose telephone number is (571)270-3814. The examiner can normally be reached 9:00 AM to 5:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, VINCENT RUDOLPH can be reached at (571) 272-8243. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TIMOTHY CHOI/Examiner, Art Unit 2671
/VINCENT RUDOLPH/Supervisory Patent Examiner, Art Unit 2671